feat: add lossless inference proxy transport

[AnthonyRonning]

Summary

• route models, chat completions, and embeddings through the OpenSecret SDK lossless inference transport
• preserve exact request bytes, query parameters, safe headers, provider-specific JSON fields, final OpenSecret status/error bodies, and SSE framing
• support streaming and non-streaming chat completions without proxy-side request parsing or mutation
• remove the unused typed handler and reconstructed-SSE compatibility layer
• reduce the Rust crate API to Config plus create_app and bump the breaking pre-1.0 release to 0.2.0
• consume the published opensecret 3.4.0 crate and lockfile checksum

Why

Maple Proxy previously deserialized inference requests through closed Rust types. That independently discarded provider-specific controls such as Gemma 4 chat_template_kwargs.enable_thinking even after fixing the SDK. The proxy now forwards caller-owned inference data through the SDK encrypted transport without becoming a schema authority.

The old public typed handlers were no longer used by create_app or Maple. Keeping them would preserve the lossy implementation as dead code, duplicate streaming and error behavior, and leave accidental crate internals as supported API.

HTTP contract and security

• the existing models, chat, embeddings, root, and health routes are unchanged
• only explicit inference routes are exposed; there is no catch-all proxy
• bearer/default-key authentication, attestation client caching, 50 MiB request limit, CORS, response timeout, and stream idle timeout are retained
• authorization, session, cookie, framing, representation, integrity, and hop-by-hop headers remain controlled at the trust boundary
• upstream validation errors, statuses, safe headers, response bodies, and SSE data are preserved instead of being collapsed or reconstructed

Intentional Rust API break

Version 0.2.0 removes the public proxy/config modules, typed endpoint handlers, proxy state, internal error types, and request-limit constant. The supported library embedding surface is:

use maple_proxy::{create_app, Config};

The HTTP API remains compatible and is now more faithful to the OpenAI-compatible backend.

Package cleanup

• remove unused tokio-test and redundant dev dependency declarations
• narrow Tower from full to the util feature actually used
• correct crates.io repository metadata
• package only the Rust crate sources, example, tests, README, license, and manifests rather than unrelated plugin, Nix, Docker, and CI files

Validation

• 19 unit tests and 3 integration tests pass against published opensecret 3.4.0
• exact body/query/header forwarding, error status/body preservation, SSE single-DONE behavior, body limits, cache behavior, and timeout paths are covered
• cargo fmt --all -- --check
• cargo clippy --locked --all-targets --all-features -- -D warnings
• cargo package --locked --allow-dirty verifies maple-proxy 0.2.0 as a 12-file crate

Release order

After merge, publish maple-proxy 0.2.0. Maple will then resolve that registry release and complete the Gemma 4 no-thinking classifier PR.

Summary by CodeRabbit

• New Features
  • Added support for both streaming (SSE) and non-streaming OpenAI-compatible chat completions.
  • Transparently forwards provider-specific chat parameters and JSON fields unchanged.
• Bug Fixes
  • Improved gateway handling for upstream connection/attestation failures with safer, filtered responses.
  • Enhanced request/response header filtering and ensured correct streaming behavior (idle-timeout and [DONE] handling).
• Documentation
  • Updated README to reflect streaming + non-streaming behavior, endpoint wording, and the example port value.
• Breaking Changes
  • Reduced the public exposure of OpenAI error types and deserialization behavior in the library API.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5cb92be8-3f3a-4c19-93d8-3d87d5838c86

📥 Commits

Reviewing files that changed from the base of the PR and between a4e3629 and 40eadb0.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (7)

• Cargo.toml
• README.md
• src/config.rs
• src/lib.rs
• src/main.rs
• src/proxy.rs
• tests/health_test.rs

🚧 Files skipped from review as they are similar to previous changes (7)

• src/main.rs
• src/lib.rs
• README.md
• src/config.rs
• tests/health_test.rs
• src/proxy.rs
• Cargo.toml

---

📝 Walkthrough

Walkthrough

Maple Proxy now routes OpenAI-compatible endpoints through a shared raw proxy handler, adds transport overrides for testing, preserves selected headers and responses, handles request and stream idle timeouts, narrows public APIs, and documents streaming and non-streaming support.

Changes

OpenAI-compatible proxy flow

Layer / File(s)	Summary
Transport abstraction and error mapping src/proxy.rs, src/config.rs	Adds transport selection and test overrides, centralizes client error mapping, and reduces the visibility and deserialization surface of OpenAI error types.
Raw forwarding and timeout handling src/proxy.rs	Forwards methods, URIs, headers, and bodies through the shared handler with filtering, response preservation, safe error mapping, and stream idle-timeout handling; tests cover forwarding, sanitization, timeouts, and API-key extraction.
Shared application routing src/lib.rs, src/proxy.rs, src/main.rs	Shares ProxyState, routes models, chat completions, and embeddings through proxy_openai_request, narrows module exports, and updates startup messaging.
Package and documented compatibility surface Cargo.toml, README.md, tests/health_test.rs	Updates package metadata and dependencies, documents streaming and non-streaming requests, changes the example port, and removes a body-limit implementation assertion from the health tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant proxy_openai_request
  participant InferenceTransport
  Client->>proxy_openai_request: Send OpenAI-compatible request
  proxy_openai_request->>InferenceTransport: Forward filtered request
  InferenceTransport-->>proxy_openai_request: Return response or stream
  proxy_openai_request-->>Client: Return filtered response

Loading

Possibly related PRs

• OpenSecretCloud/maple-proxy#17: Also updates OpenSecret streaming behavior and the opensecret dependency.
• OpenSecretCloud/maple-proxy#34: Relates to per-API-key reuse of OpenSecret clients in ProxyState.
• OpenSecretCloud/maple-proxy#35: Shares backend streaming idle-timeout behavior.

Poem

A rabbit hops through headers bright,
Forwarding streams from day to night.
JSON or SSE, both paths run—
Safe timeouts guard everyone.
Wiggle whiskers, ship and go! 🐇

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding a lossless inference proxy transport.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex-maple-agent-read-permissions-maple-proxy

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.