Research Data Opt-In Proxy

research-data-opt-in-proxy is a standalone FastAPI service that proxies OpenAI-compatible LLM requests to https://llm.chutes.ai while recording opt-in research data for the Harvard prefix-caching research collaboration.

The key privacy requirement is enforced by architecture:

• Normal llm.chutes.ai traffic is unchanged.
• Only users who intentionally call this proxy endpoint are recorded.

Why This Exists

Chutes.ai needs research-grade usage traces for prefix-caching analysis without forcing data capture on all users.

This service provides:

• A dedicated opt-in proxy endpoint.
• Optional upstream discount signaling through a secret header.
• Two independently switchable recording formats:

1. Full raw HTTP request/response logging (headers + bodies).
2. An anonymized Qwen Bailian-style usage trace format.

• Automatic small-batch archival of full request/response bodies to object storage.
• An internal export endpoint protected by a dedicated secret.
• A manual full-text JSONL export utility.

Core Behavior

The service:

• Accepts incoming API calls (for example /v1/chat/completions, /v1/models).
• Forwards method/path/query/body/headers to UPSTREAM_BASE_URL.
• Strips hop-by-hop headers (connection, transfer-encoding, etc.) from both request and response forwarding.
• Injects proxy-managed Chutes headers:

1. X-Chutes-Research-OptIn: <secret token> (discount routing signal)
2. X-Chutes-Trace: true (enable upstream trace envelopes)
3. X-Chutes-Correlation-Id: <uuid> (request correlation)
4. X-Chutes-RealIP: <client-ip> (forwarded real client IP for upstream tracking)

• Generates a fresh correlation ID per proxied request and reuses the same value for:

1. Upstream forwarded X-Chutes-Correlation-Id.
2. Stored raw_http_records.correlation_id.
3. Exported JSONL correlation_id.

• Removes caller-supplied versions of managed headers before forwarding, then sets canonical values.
• Strips X-Chutes-* headers from downstream client responses.
• Returns upstream responses while preserving OpenAI-compatible semantics (including SSE streaming for stream=true).
• Unwraps Chutes trace envelopes so downstream clients still receive OpenAI-compatible payloads.
• Stores recording data in Postgres when enabled.
• Redacts sensitive headers (Authorization, API keys, cookies) in stored recordings.
• Enforces configurable request body size limits.
• Caps SSE stream buffer size to prevent memory exhaustion.
• Optionally rate-limits requests per client IP.
• Stores JSON request/response payloads in compact form when a complete final payload can be reconstructed.
• Skips incomplete streaming exchanges instead of persisting partial request/response pairs.

Recording Formats

1) Raw HTTP (ENABLE_RAW_HTTP_RECORDING)

Stores:

• Request metadata: method, path, query, client IP, timestamp, duration.
• Request headers (with sensitive values redacted) and request body.
• Response status, response headers, full response body.
• Object storage archival metadata: request/response object key+url, SHA-256 checksum, size bytes, archived timestamp.
• Correlation and trace metadata: correlation ID, invocation IDs, selected target instance/uid/hotkey/coldkey, and parsed trace events.
• Stream flag and optional transport error field.

Storage detail:

• JSON requests are compacted into request_json and can be omitted from the large BYTEA column.
• Complete JSON responses are compacted into response_json, including finalized streaming responses reconstructed from SSE.
• Original raw byte sizes and SHA-256 digests are preserved in metadata columns.
• Legacy rows can be backfilled into the compact format through a trusted-shell script, or in small bounded batches through the internal compaction endpoint.

Sensitive headers are removed before storage based on STRIPPED_HEADER_NAMES.

Table: raw_http_records

2) Anonymized Usage Trace (ENABLE_QWEN_TRACE_RECORDING)

Stored fields follow the target schema family:

• chat_id
• parent_chat_id
• timestamp
• input_length
• output_length
• type
• turn
• hash_ids

Implementation details:

• chat_id and parent_chat_id are derived from deterministic conversation context hashes.
• hash_ids are generated by:

1. Tokenizing prompt text with tiktoken (cl100k_base).
2. Grouping token IDs into 16-token blocks.
3. Applying salted SipHash-2-4 (key derived via Blake2b from the salt).
4. Domain-remapping hash outputs to sequential integer IDs via bulk upsert.

• Salt comes from ANONYMIZATION_HASH_SALT (required, must be a real secret).
• Trace metadata is also attached in metadata (correlation ID, invocation IDs, selected target identifiers).

Tables:

• anon_usage_traces
• anon_trace_sessions
• anon_hash_domain_map
• anon_trace_clock

Environment Variables

Required:

• DATABASE_URL - Postgres connection string (required when any recording is enabled)
• ANONYMIZATION_HASH_SALT - secret salt for SipHash token hashing (must be set; placeholder values are rejected)

Production note:

• The service now uses self-hosted Postgres 16 on Hetzner instead of Neon.
• Remote deployments should use the shared old_sandy PgBouncer endpoint on port 6432.
• Neon stays online only as rollback safety until final sign-off.

Main:

• UPSTREAM_BASE_URL (default https://llm.chutes.ai)
• UPSTREAM_DISCOUNT_HEADER_NAME (optional)
• UPSTREAM_DISCOUNT_HEADER_VALUE (optional, recommended high-entropy secret)
• UPSTREAM_TRACE_HEADER_NAME (default X-Chutes-Trace)
• UPSTREAM_TRACE_HEADER_VALUE (default true)
• UPSTREAM_CORRELATION_ID_HEADER_NAME (default X-Chutes-Correlation-Id)
• UPSTREAM_REAL_IP_HEADER_NAME (default X-Chutes-RealIP)
• UPSTREAM_HTTP2_ENABLED (default false; keep disabled unless upstream HTTP/2 is known stable)
• EXPORT_ENDPOINT_SECRET (required to enable /internal/export/raw-http.jsonl)
• EXPORT_ENDPOINT_SECRET_HEADER_NAME (default X-Chutes-Export-Secret)
• ARCHIVE_ENDPOINT_SECRET (required to enable /internal/archive/run)
• ARCHIVE_ENDPOINT_SECRET_HEADER_NAME (default X-Chutes-Archive-Secret)
• ENABLE_RAW_HTTP_RECORDING (default true)
• ENABLE_QWEN_TRACE_RECORDING (default false, keep disabled when collecting full-text traces only)
• ARCHIVE_ON_INGEST (default false; when true and object storage is configured, request/response bodies are uploaded immediately and omitted from DB rows)

Object storage archive:

• ARCHIVE_STORAGE_PROVIDER (auto, s3, vercel_blob)
• ARCHIVE_BATCH_SIZE (default 500)
• ARCHIVE_OBJECT_PREFIX (default raw-http-archive)
• ARCHIVE_S3_BUCKET / ARCHIVE_S3_REGION
• VERCEL_BLOB_READ_WRITE_TOKEN / VERCEL_BLOB_ACCESS

Security:

• STRIPPED_HEADER_NAMES - comma-separated header names removed from recorded header maps (defaults include auth/cookie headers plus sensitive Vercel forwarding headers)

Limits:

• MAX_RECORDED_BODY_BYTES - cap stored body sizes (0 = no truncation)
• MAX_REQUEST_BODY_BYTES - reject requests larger than this (default 10 MiB, 0 = unlimited)
• MAX_STREAM_BUFFER_BYTES - cap SSE recording buffer (default 50 MiB, 0 = unlimited)
• RATE_LIMIT_REQUESTS - max requests per IP per window (0 = disabled)
• RATE_LIMIT_WINDOW_SECONDS - rate limit window (default 60)
• RATE_LIMIT_MAX_TRACKED_CLIENTS - cap in-memory rate-limit client buckets to avoid unbounded growth under many-source floods

Recommended production baseline:

• Set RATE_LIMIT_REQUESTS to a non-zero value sized above your expected per-IP burst rate.
• Rate-limited responses include RateLimit-*, X-RateLimit-*, and Retry-After headers so callers can back off cleanly.
• Keep MAX_REQUEST_BODY_BYTES enabled to reject oversized payload attacks early.
• Keep ARCHIVE_ENDPOINT_SECRET and EXPORT_ENDPOINT_SECRET enabled.
• Set ARCHIVE_ON_INGEST=true when S3 or Vercel Blob is available so full bodies do not accumulate in Postgres.
• Set RETENTION_DAYS to a finite value so metadata rows are cleaned up automatically.

Retention:

• RETENTION_DAYS - auto-delete records older than this many days (0 = keep forever)

Timeouts:

• HTTP_CONNECT_TIMEOUT_SECONDS, HTTP_READ_TIMEOUT_SECONDS, HTTP_WRITE_TIMEOUT_SECONDS, HTTP_POOL_TIMEOUT_SECONDS

See .env.example for full template.

Data Retention

When RETENTION_DAYS is set to a positive value, the /internal/archive/run maintenance path also runs cleanup_old_records() after each archival batch. This lets the same Vercel cron both drain unarchived rows and prune old metadata. The helper can still be called directly from app/db.py if you need an external job.

Example cleanup cron (external):

# Run daily at 3am UTC
0 3 * * * cd /path/to/project && python -c "
import asyncio
from app.db import create_pool, cleanup_old_records
async def main():
    pool = await create_pool('DATABASE_URL_HERE')
    result = await cleanup_old_records(pool, retention_days=90)
    print(result)
    await pool.close()
asyncio.run(main())
"

Local Development

Install:

python -m venv .venv
source .venv/bin/activate
pip install -e .[dev]
cp .env.example .env
# edit .env with real values (especially ANONYMIZATION_HASH_SALT)

Run:

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Example production DSN:

DATABASE_URL=postgresql://<user>:<password>@94.130.222.43:6432/research_proxy?sslmode=require

Health check:

curl http://localhost:8000/health
curl http://localhost:8000/healthz

Manual Full-Text Export

Trusted internal export endpoint:

curl -H "X-Chutes-Export-Secret: $EXPORT_ENDPOINT_SECRET" \
  "http://localhost:8000/internal/export/raw-http.jsonl?limit=100"

The internal endpoint and manual exporter stream rows incrementally from Postgres and object storage, so bounded exports do not need to materialize the full result set in memory before bytes start flowing to the caller.

Legacy row compaction endpoint for small bounded batches:

curl -X POST \
  -H "X-Chutes-Archive-Secret: $ARCHIVE_ENDPOINT_SECRET" \
  "http://localhost:8000/internal/storage/compact-json?limit=25"

This migrates eligible legacy rows into the compact JSON storage columns and, when object storage is available, archives the original raw request/response bytes before clearing the large in-row payloads. On Vercel, keep this endpoint for small maintenance batches only.

Manual backfill from a trusted shell:

python scripts/backfill_compact_json.py \
  --batch-size 25

This is the preferred path for large historical backfills because it runs outside the serverless memory limits while still using the same migration code.

Manual script from a trusted shell:

python scripts/export_full_text.py \
  --output ./exports/raw-http-$(date +%F).jsonl

To resolve already-archived request/response bodies from object storage:

python scripts/export_full_text.py \
  --output ./exports/raw-http-full.jsonl \
  --resolve-archived-bodies

Automatic Archival

Run one small archival batch manually:

curl -X POST \
  -H "X-Chutes-Archive-Secret: $ARCHIVE_ENDPOINT_SECRET" \
  "http://localhost:8000/internal/archive/run?limit=500"

Production can run this via Vercel Cron every minute. Archive endpoint also accepts Authorization: Bearer <ARCHIVE_ENDPOINT_SECRET> for cron-compatible auth.

Optional date window:

python scripts/export_full_text.py \
  --output ./exports/raw-http-window.jsonl \
  --start 2026-03-01T00:00:00+00:00 \
  --end 2026-03-02T00:00:00+00:00

Testing

Unit

pytest tests/unit -m unit

Integration (proxy + recorder + Postgres)

export TEST_DATABASE_URL=postgresql://user:password@host/test_database?sslmode=require
pytest tests/integration -m integration

TEST_DATABASE_URL must point to a dedicated disposable test database. The DB-backed tests truncate tables, intentionally ignore DATABASE_URL by default, and now refuse to run if TEST_DATABASE_URL resolves to the same database target as DATABASE_URL.

E2E (live llm.chutes.ai)

export CHUTES_API_KEY=...
export TEST_DATABASE_URL=postgresql://user:password@host/test_database?sslmode=require
pytest tests/e2e -m e2e -s

Deployment (Vercel)

This repo includes:

• api/index.py (Vercel Python entrypoint)
• vercel.json routing all paths to the FastAPI app

Deploy:

vercel --prod

Then set environment variables in Vercel project settings (or with CLI) before final production traffic.

Custom Domain

Current verified production hosts:

• https://research-data-opt-in-proxy.vercel.app
• https://research-data-opt-in-proxy.chutes.ai

The chutes.ai binding follows the same Vercel-managed pattern used by other Chutes subdomains:

• CNAME research-data-opt-in-proxy -> <project-specific>.vercel-dns-016.com.
• TXT _vercel -> vc-domain-verify=research-data-opt-in-proxy.chutes.ai,<token>

Operational note:

• DNS can propagate before Vercel finishes certificate issuance. During that window dig may already show the correct vercel-dns-* target while HTTPS still fails with TLS handshake or certificate errors. Re-check /health after a few minutes before assuming the app deployment is broken.

Security Notes

• Do not commit secrets (DATABASE_URL, salts, API keys). The .gitignore already excludes .env.* files.
• ANONYMIZATION_HASH_SALT must be a real secret. The app will refuse to start with placeholder values.
• Authorization headers and API keys are automatically redacted in raw HTTP recordings before storage.
• Internal managed headers and upstream X-Chutes-* response headers are stripped from stored header maps.
• Hop-by-hop headers (connection, transfer-encoding, upgrade, etc.) are stripped from both forwarded requests and responses.
• Proxy-managed X-Chutes-* headers are not relayed back to downstream clients.
• Client IP attribution trusts Vercel-managed x-forwarded-for headers and falls back to the socket peer. Caller-supplied x-real-ip is ignored.
• Configure RATE_LIMIT_REQUESTS in production to prevent abuse.
• OPTIONS requests and internal maintenance routes are exempt from the in-memory limiter so CORS preflights and cron/archive flows are not throttled accidentally.
• Set RETENTION_DAYS and run periodic cleanup to manage storage growth.
• For the research endpoint, publish clear user messaging that usage is opt-in and recorded.