Skip to content

refactor(api-types): extract shared cloud wire contract from soth-sync#69

Merged
secxena merged 1 commit into
stagingfrom
refactor/extract-soth-api-types-shared-wire-contract
May 4, 2026
Merged

refactor(api-types): extract shared cloud wire contract from soth-sync#69
secxena merged 1 commit into
stagingfrom
refactor/extract-soth-api-types-shared-wire-contract

Conversation

@secxena

@secxena secxena commented May 4, 2026

Copy link
Copy Markdown
Contributor

The SDK and the proxy were silently drifting on the cloud wire format: the proxy converted in-process soth_core::TelemetryEvent -> wire TelemetryEvent via a private map_event in soth-sync, while the SDK shipper just serde_json::json!'d the raw in-process struct. That worked at first but blew up on the cloud as a 422 (topic_cluster_id expected string, got integer 0) — and any future field divergence would silently drop SDK telemetry without affecting proxy telemetry.

Extract the cloud API types and the in-process -> wire conversion into a new pure-types crate soth-api-types. Both soth-sync (proxy) and soth-sdk-core (SDK) depend on it, so a contract change in either direction now propagates to both at once.

Also fix the latent SDK shipper bug that produced the 422:

  • send X-Soth-Api-Version: 2026-02-01 (matches soth-sync)
  • run each event through soth_api_types::convert::map_event
  • ship a real TelemetryBatchRequest envelope (batch_id, device_id_hash="sdk:", proxy_signature="sdk-unsigned"), matching the proxy envelope shape

SOTH_SHIPPER_VERBOSE=1 opts into stderr lines with the POST status + event count so SDK customers can verify batching cadence at runtime.

Verified against the production cloud endpoint:
20 anthropic calls -> 1 POST /v1/edge/telemetry/batch -> 200 (20 events)
…from both soth-py and soth-node, with https_proxy unset (SDK is genuinely proxy-independent).

Build/test parity (CI mirror, locally green):

  • cargo fmt --all -- --check
  • cargo clippy --workspace --lib --bins --tests
  • cargo test --workspace --lib --bins
  • cargo test -p soth-conformance-tests --test parity (2/2)
  • WASM matrix: soth-detect / soth-classify on wasm32-{wasip1,unknown-unknown}
  • cargo build -p soth-py / soth-node
  • pytest bindings/soth-py/tests (45 passed, 2 skipped)
  • npm test in bindings/soth-node (32/32)

Notes:

  • observation_records on TelemetryBatchRequest becomes Option<Vec<serde_json::Value>> so soth-api-types stays free of the soth-telemetry (rusqlite/tokio) transitive closure. The proxy serializes its typed records to Value at the call site; JSON wire shape is identical.
  • proxy_signature: "sdk-unsigned" is a documented placeholder for the SDK lane — full ed25519 signing of SDK telemetry remains Phase-2 work, called out in shipper.rs.

The SDK and the proxy were silently drifting on the cloud wire format:
the proxy converted in-process `soth_core::TelemetryEvent` -> wire
`TelemetryEvent` via a private `map_event` in soth-sync, while the SDK
shipper just `serde_json::json!`'d the raw in-process struct. That
worked at first but blew up on the cloud as a 422 (`topic_cluster_id`
expected string, got integer 0) — and any future field divergence
would silently drop SDK telemetry without affecting proxy telemetry.

Extract the cloud API types and the in-process -> wire conversion into
a new pure-types crate `soth-api-types`. Both `soth-sync` (proxy) and
`soth-sdk-core` (SDK) depend on it, so a contract change in either
direction now propagates to both at once.

Also fix the latent SDK shipper bug that produced the 422:
  - send `X-Soth-Api-Version: 2026-02-01` (matches soth-sync)
  - run each event through `soth_api_types::convert::map_event`
  - ship a real `TelemetryBatchRequest` envelope (batch_id,
    device_id_hash="sdk:<org>", proxy_signature="sdk-unsigned"),
    matching the proxy envelope shape

`SOTH_SHIPPER_VERBOSE=1` opts into stderr lines with the POST status +
event count so SDK customers can verify batching cadence at runtime.

Verified against the production cloud endpoint:
  20 anthropic calls -> 1 POST /v1/edge/telemetry/batch -> 200 (20 events)
…from both soth-py and soth-node, with `https_proxy` unset (SDK is
genuinely proxy-independent).

Build/test parity (CI mirror, locally green):
  - cargo fmt --all -- --check
  - cargo clippy --workspace --lib --bins --tests
  - cargo test --workspace --lib --bins
  - cargo test -p soth-conformance-tests --test parity (2/2)
  - WASM matrix: soth-detect / soth-classify on wasm32-{wasip1,unknown-unknown}
  - cargo build -p soth-py / soth-node
  - pytest bindings/soth-py/tests (45 passed, 2 skipped)
  - npm test in bindings/soth-node (32/32)

Notes:
  * `observation_records` on `TelemetryBatchRequest` becomes
    `Option<Vec<serde_json::Value>>` so soth-api-types stays free of
    the `soth-telemetry` (rusqlite/tokio) transitive closure. The
    proxy serializes its typed records to Value at the call site;
    JSON wire shape is identical.
  * `proxy_signature: "sdk-unsigned"` is a documented placeholder
    for the SDK lane — full ed25519 signing of SDK telemetry remains
    Phase-2 work, called out in shipper.rs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@secxena
secxena merged commit 3352c80 into staging May 4, 2026
18 checks passed
@secxena
secxena deleted the refactor/extract-soth-api-types-shared-wire-contract branch June 9, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant