From fd2480ed4f2207f049c22dda4acce074e29f7ddd Mon Sep 17 00:00:00 2001 From: TalesOfThales <101120927+userFRM@users.noreply.github.com> Date: Sun, 19 Jul 2026 12:20:15 +0200 Subject: [PATCH 1/6] feat(mdds)!: shard streaming pulls; size shards from request shape The dedicated streaming builders (*_stream) now route through the bulk-fetch sharder like the buffered path; previously they always ran a single stream regardless of bulk_fetch policy or tier concurrency. Shard sizing no longer runs a density probe. plan_query decides shardability from the request shape and cuts the requested time or date range into equal concurrent bands, removing the OHLC sizing probe and its per-endpoint rows_per_event multiplier; that multiplier mis-sized quote chains badly enough that large quote pulls never sharded. BREAKING CHANGE: MarketDataClient::bulk_fetch_plan is now fn -> Option (was async fn -> Result, Error>). --- CHANGELOG.md | 6 + docs-site/docs/articles/bulk-downloads.md | 13 +- .../docs/articles/concurrent-requests.md | 2 +- docs-site/docs/changelog.md | 6 + .../include/config_accessors.hpp.inc | 5 +- thetadatadx-cpp/include/thetadatadx.h | 22 +- thetadatadx-ffi/src/config_accessors.rs | 7 +- .../python/thetadatadx/__init__.pyi | 2 +- .../src/_generated/config_accessors.rs | 7 +- .../build_support/endpoints/render/mdds.rs | 194 +-- .../mdds/stream_method_header.rs.tmpl | 22 + thetadatadx-rs/config.default.toml | 6 +- thetadatadx-rs/config_surface.toml | 26 +- thetadatadx-rs/src/config/mdds.rs | 18 +- thetadatadx-rs/src/lib.rs | 2 +- thetadatadx-rs/src/mdds/client.rs | 8 +- thetadatadx-rs/src/mdds/macros.rs | 30 +- thetadatadx-rs/src/mdds/shard.rs | 1168 +++++------------ thetadatadx-rs/src/mdds/stream.rs | 33 - thetadatadx-ts/index.d.ts | 7 +- .../src/_generated/config_accessors.rs | 7 +- 21 files changed, 593 insertions(+), 998 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c0144a4..68b80fbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Streaming history builders (`*_stream()`) now shard concurrently like the buffered path, and shard sizing cuts the requested time or date range into equal concurrent bands from the request shape.** Under `bulk_fetch = "auto"` (the default), a large chunk-streaming history pull fans out across the same equal bands as its buffered sibling: every band streams as its own concurrent request, and each band's chunks reach the handler as they arrive — every chunk exactly once, with chunks from different bands interleaved in arrival order (use the buffered builder, or `bulk_fetch = "off"`, when the single stream's exact order matters). The split is decided from the request shape alone — a multi-day range cuts into equal date bands, a single day with an intraday window cuts into equal time bands — with no density-probe request and no per-endpoint tuning; provably small pulls still run as a single stream. + +- **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. + ## [0.2.0] - 2026-07-16 ### Added diff --git a/docs-site/docs/articles/bulk-downloads.md b/docs-site/docs/articles/bulk-downloads.md index 44bd3edf..12583ad4 100644 --- a/docs-site/docs/articles/bulk-downloads.md +++ b/docs-site/docs/articles/bulk-downloads.md @@ -27,7 +27,7 @@ Raise them for even fatter streams if you have memory to spare, or lower them wh ## Automatic sharding -A single large request does not have to be one stream. The SDK sizes the request, splits it into balanced pieces along its time range, runs them in parallel across your tier's concurrent-request budget, and reassembles them into exactly the rows one request would have returned. +A single large request does not have to be one stream. The SDK splits the requested time (or date) range into equal concurrent pieces — straight from the shape of the request, with no sizing round-trip — runs them in parallel across your tier's concurrent-request budget, and reassembles them into exactly the rows one request would have returned. The number of pieces is your tier's concurrency: @@ -38,7 +38,7 @@ The number of pieces is your tier's concurrency: | Standard | 4 | | Pro | 8 | -On Pro, a full-day chain that runs in 546 s as one 8 MB stream comes back in about 155 s, a further 3.5x, because eight pieces fetch at once. Combined with the window that is roughly 5.6x faster than the protocol default. It scales down cleanly with tier: four pieces on Standard, two on Value, one on Free. +On Pro, one big query becomes eight pieces fetching at once. How much that shortens the wall clock depends on how much of the fan-out the server actually runs in parallel under current conditions; the table below shows one measured day. It scales down cleanly with tier: four pieces on Standard, two on Value, one on Free. Sharding is on by default. Small pulls stay a single request, so there is nothing to tune for them. If you want control: @@ -77,7 +77,7 @@ let ticks = client ### Streaming: chunks as they arrive -The streaming terminal (`.stream(handler)`) hands you the result in chunks as each piece produces them, so you never hold the whole dataset in memory. It is the fastest path for the largest pulls because it skips building and ordering the full frame. +The streaming terminal (`.stream(handler)`) hands you the result in chunks as each piece produces them, so you never hold the whole dataset in memory. It skips building and ordering the full frame, which makes it the path for results too large to hold whole. ```python rows = 0 @@ -103,22 +103,23 @@ Because pieces stream in parallel, chunks from different pieces interleave. Sort ### Which to use -Both are fast. When the server is the bottleneck they finish in the same time. On a fast link the buffered call trails streaming by about 20 percent, the cost of assembling the complete ordered frame in memory that streaming skips. Pick by the shape of the job, not speed: +Both shard and window the same way. Buffered spends extra client-side work assembling the complete ordered frame; streaming skips that. Pick by the shape of the job: - Buffered when you want the finished dataset, to save it, convert it, or work with it whole, and it fits in memory. - Streaming when the result may not fit in memory, or you want to process rows as they arrive. This is the path for the largest pulls. ## What it adds up to -Measured on a full-day SPXW options chain (125,849,342 rows, `strike="*"`, `right="both"`, tick interval) on a Pro account. Absolute times depend on your tier, your distance to the server, and current load, so read them as one box on one day, not a guarantee. +Measured on a full-day SPXW options chain (125,849,342 rows, `strike="*"`, `right="both"`, tick interval) on a Pro account, using the buffered path. Absolute times depend on your tier, your distance to the server, current load, and how much of the shard fan-out the server ran in parallel that day, so read them as one box on one day, not a guarantee. | Setup | Full-day chain | vs default | |---|---:|---:| | 64 KB window, single stream (protocol default) | 872.7 s | 1x | | 8 MB window, single stream | 546.3 s | 1.60x | -| 8 MB window + 8-way sharding, streaming | 155.0 s | 5.63x | | 8 MB window + 8-way sharding, buffered | 183.8 s | 4.75x | +Streaming shards the same way, and its realized gain rides on the same server-side parallelism, so we quote no wall clock for it. + Every setup returns the same rows. A concrete single contract comes back byte-for-byte identical to a single request. A full chain comes back in a deterministic canonical order (ascending expiration, then strike, then calls before puts, in trading-time order within each contract). Set `bulk_fetch = "off"` if you want the server's own ordering instead. ## Configuration reference diff --git a/docs-site/docs/articles/concurrent-requests.md b/docs-site/docs/articles/concurrent-requests.md index 1dd507d7..4879b60b 100644 --- a/docs-site/docs/articles/concurrent-requests.md +++ b/docs-site/docs/articles/concurrent-requests.md @@ -34,7 +34,7 @@ With a Pro subscription, eight of those requests run concurrently and the rest w ## One giant request, split for you -The pattern above parallelizes work you have already split into many requests. The SDK also does the reverse for you: a **single** large history request is split into balanced pieces, run in parallel across your tier's concurrent-request budget, and reassembled into exactly the rows one request would have returned. You write one ordinary query and it runs at your tier's concurrency. This is on by default. +The pattern above parallelizes work you have already split into many requests. The SDK also does the reverse for you: a **single** large history request has its time or date range split into equal pieces, run in parallel across your tier's concurrent-request budget, and reassembled into exactly the rows one request would have returned. You write one ordinary query and it runs at your tier's concurrency. This is on by default. See [Bulk Downloads](/articles/bulk-downloads) for the full picture: how the split works, buffered versus streaming delivery, the `bulk_fetch` and `shard_concurrency` knobs, API examples, and measured performance. diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index 7c0144a4..68b80fbe 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Streaming history builders (`*_stream()`) now shard concurrently like the buffered path, and shard sizing cuts the requested time or date range into equal concurrent bands from the request shape.** Under `bulk_fetch = "auto"` (the default), a large chunk-streaming history pull fans out across the same equal bands as its buffered sibling: every band streams as its own concurrent request, and each band's chunks reach the handler as they arrive — every chunk exactly once, with chunks from different bands interleaved in arrival order (use the buffered builder, or `bulk_fetch = "off"`, when the single stream's exact order matters). The split is decided from the request shape alone — a multi-day range cuts into equal date bands, a single day with an intraday window cuts into equal time bands — with no density-probe request and no per-endpoint tuning; provably small pulls still run as a single stream. + +- **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. + ## [0.2.0] - 2026-07-16 ### Added diff --git a/thetadatadx-cpp/include/config_accessors.hpp.inc b/thetadatadx-cpp/include/config_accessors.hpp.inc index bcfeb257..70d96633 100644 --- a/thetadatadx-cpp/include/config_accessors.hpp.inc +++ b/thetadatadx-cpp/include/config_accessors.hpp.inc @@ -437,8 +437,9 @@ } /// Set the automatic bulk-fetch sharding policy for history pulls (buffered - /// and chunk-streaming alike): 0=Auto (default; large pulls are split into - /// balanced concurrent sub-requests — buffered pulls merge back into + /// and chunk-streaming alike): 0=Auto (default; a large pull's requested + /// time or date range is split into equal concurrent bands — buffered + /// pulls merge back into /// exactly the single-stream rows, with option-chain pulls in a /// deterministic canonical contract order and every other pull in the /// exact single-stream row order; streaming pulls forward each band's diff --git a/thetadatadx-cpp/include/thetadatadx.h b/thetadatadx-cpp/include/thetadatadx.h index 397382bb..556d772a 100644 --- a/thetadatadx-cpp/include/thetadatadx.h +++ b/thetadatadx-cpp/include/thetadatadx.h @@ -2083,20 +2083,24 @@ void thetadatadx_config_set_market_data_connection_window_size_kb(ThetaDataDxCon int32_t thetadatadx_config_get_market_data_connection_window_size_kb(const ThetaDataDxConfig* config, size_t* out_kb); /** - * Set the automatic bulk-fetch sharding policy for buffered history pulls. + * Set the automatic bulk-fetch sharding policy for history pulls, covering + * both the buffered and the chunk-streaming call paths. * - * Under Auto (the default) a large history pull is sized with a cheap - * density probe and, when worthwhile, split into balanced disjoint - * sub-requests across the account's concurrent-request budget, then merged - * back into exactly the rows of the single-stream response: single-contract, - * stock, and index pulls keep the exact single-stream row order, while + * Under Auto (the default) a large history pull's requested time or date + * range is split into equal concurrent bands across the account's + * concurrent-request budget, decided from the request shape alone (no + * sizing request is issued). Buffered pulls merge the shards back into + * exactly the rows of the single-stream response: single-contract, stock, + * and index pulls keep the exact single-stream row order, while * option-chain pulls come back in a deterministic canonical order * (expiration, strike, right ascending; calls before puts; time-ascending - * within each contract). Small pulls and non-history endpoints are never - * sharded. + * within each contract). Streaming pulls forward each band's chunks to the + * handler as they arrive: every chunk exactly once, but chunks from + * different bands interleave in arrival order rather than the single + * stream's order. Small pulls and non-history endpoints are never sharded. * @param config Config handle to mutate. * @param policy 0 = Auto (default), 1 = Off (single stream per query, in - * the server's own row order). + * the server's own row and chunk order). * @return 0 on success, -1 when policy is outside {0, 1} or config is null * (check thetadatadx_last_error()). */ diff --git a/thetadatadx-ffi/src/config_accessors.rs b/thetadatadx-ffi/src/config_accessors.rs index 2245b5c3..c028a1fa 100644 --- a/thetadatadx-ffi/src/config_accessors.rs +++ b/thetadatadx-ffi/src/config_accessors.rs @@ -1214,9 +1214,10 @@ pub unsafe extern "C" fn thetadatadx_config_get_market_data_connection_window_si /// Set the automatic bulk-fetch sharding policy for history pulls, covering /// both the buffered and the chunk-streaming call paths. /// -/// - `policy = 0`: Auto (default) — large history pulls are sized with a -/// cheap density probe and, when worthwhile, split into balanced disjoint -/// sub-requests across the account's concurrent-request budget. Buffered +/// - `policy = 0`: Auto (default) — a large history pull's requested time +/// or date range is split into equal concurrent bands across the +/// account's concurrent-request budget, decided from the request shape +/// alone (no sizing request is issued). Buffered /// pulls merge the shards back into exactly the rows of the single-stream /// response: single-contract, stock, and index pulls keep the exact /// single-stream row order, while option-chain pulls come back in a diff --git a/thetadatadx-py/python/thetadatadx/__init__.pyi b/thetadatadx-py/python/thetadatadx/__init__.pyi index c629a564..ff7e758a 100644 --- a/thetadatadx-py/python/thetadatadx/__init__.pyi +++ b/thetadatadx-py/python/thetadatadx/__init__.pyi @@ -210,7 +210,7 @@ class Config: market_data_connection_window_size_kb: int """Initial connection-level HTTP/2 flow-control window (KB) for the market-data gRPC channel. A larger window raises the throughput ceiling on bulk streaming pulls before HTTP/2 backpressure kicks in. The value is clamped into ``[64, 2_097_151]`` KB at validate/connect time. The default is ``16384`` (16 MiB).""" bulk_fetch: Literal["auto", "off"] - """Automatic bulk-fetch sharding policy for buffered history pulls. ``"auto"`` (the default) sizes large history pulls with a cheap density probe and, when worthwhile, splits them into balanced concurrent sub-requests across the account's concurrent-request budget, merged back into exactly the rows of the single-stream response — single-contract, stock, and index pulls in the exact single-stream order, option-chain pulls in a deterministic canonical order grouped by expiration, strike, and right; small pulls are never sharded. ``"off"`` runs every buffered query as a single stream, in the server's own row order. The setter accepts either value case-insensitively and raises ``ValueError`` otherwise.""" + """Automatic bulk-fetch sharding policy for history pulls (buffered and chunk-streaming alike). ``"auto"`` (the default) splits a large history pull's requested time or date range into equal concurrent bands across the account's concurrent-request budget — buffered pulls are merged back into exactly the rows of the single-stream response, single-contract, stock, and index pulls in the exact single-stream order and option-chain pulls in a deterministic canonical order grouped by expiration, strike, and right; streaming pulls forward each band's chunks as they arrive, every chunk exactly once but interleaved across bands in arrival order; small pulls are never sharded. ``"off"`` runs every query as a single stream, in the server's own row and chunk order. The setter accepts either value case-insensitively and raises ``ValueError`` otherwise.""" shard_concurrency: Optional[int] """Upper bound on concurrent sub-requests per sharded bulk fetch. ``None`` (the default) uses the account's full concurrent-request budget (the tier-derived channel-pool size resolved at connect time); an ``int`` caps the fan-out. The applied value is clamped into ``[1, pool_size]`` when a plan is built, and validation floors an explicit ``0`` to ``1``.""" reconnect_policy: str diff --git a/thetadatadx-py/src/_generated/config_accessors.rs b/thetadatadx-py/src/_generated/config_accessors.rs index 63e0e645..7c5eb9bd 100644 --- a/thetadatadx-py/src/_generated/config_accessors.rs +++ b/thetadatadx-py/src/_generated/config_accessors.rs @@ -545,9 +545,10 @@ impl Config { } /// Automatic bulk-fetch sharding policy for history pulls (buffered and - /// chunk-streaming alike). Accepts ``"auto"`` (default: large history - /// pulls are split into balanced concurrent sub-requests across the - /// account's concurrent-request budget — buffered pulls are merged back + /// chunk-streaming alike). Accepts ``"auto"`` (default: a large history + /// pull's requested time or date range is split into equal concurrent + /// bands across the account's concurrent-request budget — buffered + /// pulls are merged back /// into exactly the rows of the single-stream response, single-contract, /// stock, and index pulls in the exact single-stream order and /// option-chain pulls in a deterministic canonical order grouped by diff --git a/thetadatadx-rs/build_support/endpoints/render/mdds.rs b/thetadatadx-rs/build_support/endpoints/render/mdds.rs index 521e053e..98f33698 100644 --- a/thetadatadx-rs/build_support/endpoints/render/mdds.rs +++ b/thetadatadx-rs/build_support/endpoints/render/mdds.rs @@ -328,104 +328,148 @@ pub(super) fn generate_mdds_streaming_endpoint(out: &mut String, endpoint: &Gene ) .unwrap(); out.push_str(" let _metrics_start = std::time::Instant::now();\n"); - out.push_str(" let _permit = client.request_semaphore.acquire().await\n"); - out.push_str( - " .map_err(|_| Error::config_internal(\"request semaphore closed\"))?;\n", - ); - out.push_str(" let policy = client.config().retry;\n"); - // Wrap the `FnMut + Send` handler in a `Mutex` so the per-attempt - // closure passed to `run_streaming_retry_loop` gets a fresh mutable - // borrow on each invocation (the closure may run twice on a - // post-refresh restart) and the returned future stays Send. - out.push_str(" let handler_mutex = std::sync::Mutex::new(handler);\n"); - out.push_str(" let handler_mutex = &handler_mutex;\n"); - // Set once a chunk reaches `handler`: the no-resume restart replays - // from chunk zero, so a later transient after delivery began is made - // terminal by `run_streaming_retry_loop` to avoid duplicating rows. - out.push_str(" let delivered = std::sync::atomic::AtomicBool::new(false);\n"); - out.push_str(" let delivered = &delivered;\n"); - out.push_str(" crate::mdds::macros::run_streaming_retry_loop(\n"); - out.push_str(" client.session(),\n"); - out.push_str(" &policy,\n"); - writeln!(out, " {endpoint_name_literal},").unwrap(); - out.push_str(" delivered,\n"); - out.push_str(" move |snap| {\n"); - // Clone per-attempt: the FnMut closure may fire twice (post-refresh - // restart) and the `async move` block would otherwise move each - // captured binding into the first attempt's future, so non-Copy - // params clone fresh each iteration. Copy scalars (`Option` etc.) - // are copied into the future automatically and need no rebind. - for arg in method_params - .iter() - .map(|param| direct_method_arg_name(param)) - { - writeln!(out, " let {arg} = {arg}.clone();").unwrap(); - } - for param in &optional_params { - if matches!( - direct_optional_rust_type(param), - "Option" | "Option" | "Option" - ) { - continue; - } - writeln!( - out, - " let {0} = {0}.clone();", - param.name - ) - .unwrap(); - } - out.push_str(" async move {\n"); - out.push_str(" let qi = client.build_query_info(snap.uuid.clone());\n"); - writeln!( - out, - " let request = proto::{} {{", - endpoint.request_type - ) - .unwrap(); - out.push_str(" query_info: Some(qi),\n"); + // Build the wire parameters once, before the shard branch — every + // dispatch below (single stream, shard fan-out, retries) clones this + // same value, so the wire bytes are identical across paths. Mirrors + // the `parsed_endpoint!` stream arm in `src/mdds/macros.rs`. if endpoint.fields.is_empty() { writeln!( out, - " params: Some(proto::{} {{}}),", + " let params = proto::{} {{}};", endpoint.query_type ) .unwrap(); } else { writeln!( out, - " params: Some(proto::{} {{", + " let params = proto::{} {{", endpoint.query_type ) .unwrap(); for field in &endpoint.fields { let expr = mdds_query_field_expr(endpoint, field, false); if expr == field.name { - writeln!(out, " {expr},").unwrap(); + writeln!(out, " {expr},").unwrap(); } else { - writeln!( - out, - " {}: {expr},", - field.name - ) - .unwrap(); + writeln!(out, " {}: {expr},", field.name).unwrap(); } } - out.push_str(" }),\n"); + out.push_str(" };\n"); + } + // Wrap the `FnMut + Send` handler in a `Mutex` so the per-attempt + // closure — and, under a shard plan, every concurrent band — gets a + // fresh mutable borrow per chunk (the closure may run twice on a + // post-refresh restart) and the returned future stays Send. Defined + // before the shard branch so both arms share the one handler. + out.push_str(" let handler_mutex = std::sync::Mutex::new(handler);\n"); + out.push_str(" let handler_mutex = &handler_mutex;\n"); + // Bulk-fetch auto-sharding, same gate as the buffered `.await` + // builders: policy `Off` and small pulls resolve to `None` — the + // single-stream arm below, exactly the pre-shard behaviour. The + // planner's descriptor table is keyed by BUFFERED endpoint names, so + // the `_stream` suffix is stripped for the `auto_plan` lookup only; + // logs, metrics, and retry labels keep this endpoint's own name. + let buffered_name_literal = format!( + "{:?}", + endpoint + .name + .strip_suffix("_stream") + .expect("streaming endpoint name must end in `_stream`") + ); + out.push_str(" #[allow(unused_mut)] // Reason: endpoints with no shardable fields expand no projection arm.\n"); + out.push_str(" let mut shard_query = crate::mdds::shard::ShardQuery::default();\n"); + for field in &endpoint.fields { + writeln!( + out, + " shard_read_field!(shard_query, params, {});", + field.name + ) + .unwrap(); } - out.push_str(" };\n"); + writeln!( + out, + " match crate::mdds::shard::auto_plan(client, {buffered_name_literal}, &shard_query) {{" + ) + .unwrap(); + // Per-attempt stream body shared by both arms: open the stream via + // the generated stub, then drain chunk-by-chunk through the parser + // into `handler_mutex` — byte-for-byte the pre-shard delivery path. + let stub_call = include_str!("templates/mdds/stub_call_error_arm.rs.tmpl") + .replace("__GRPC_NAME__", &endpoint.grpc_name); + let chunk_body = include_str!("templates/mdds/for_each_chunk_body.rs.tmpl") + .replace("__PARSER_NAME__", &parser_name); + let field_idents = endpoint + .fields + .iter() + .map(|field| field.name.as_str()) + .collect::>() + .join(", "); + out.push_str(" Some(plan) => {\n"); + out.push_str(" sharded_stream_fanout!(\n"); + writeln!( + out, + " client, {}, plan, params,", + endpoint.name + ) + .unwrap(); + writeln!(out, " [{field_idents}],").unwrap(); + out.push_str(" |snap, banded, delivered| async move {\n"); + writeln!( + out, + " let request = proto::{} {{", + endpoint.request_type + ) + .unwrap(); + out.push_str(" query_info: Some(client.build_query_info(snap.uuid.clone())),\n"); + out.push_str(" params: Some(banded.clone()),\n"); + out.push_str(" };\n"); + out.push_str(&stub_call); + out.push_str(&chunk_body); + out.push_str(" }\n"); + out.push_str(" )?;\n"); + out.push_str(" }\n"); + out.push_str(" None => {\n"); + out.push_str(" let _permit = client.request_semaphore.acquire().await\n"); out.push_str( - &include_str!("templates/mdds/stub_call_error_arm.rs.tmpl") - .replace("__GRPC_NAME__", &endpoint.grpc_name) - .replace("__ENDPOINT_NAME_LITERAL__", &endpoint_name_literal), + " .map_err(|_| Error::config_internal(\"request semaphore closed\"))?;\n", ); + out.push_str(" let policy = client.config().retry;\n"); + // Set once a chunk reaches `handler`: the no-resume restart replays + // from chunk zero, so a later transient after delivery began is made + // terminal by `run_streaming_retry_loop` to avoid duplicating rows. out.push_str( - &include_str!("templates/mdds/for_each_chunk_body.rs.tmpl") - .replace("__PARSER_NAME__", &parser_name), + " let delivered = std::sync::atomic::AtomicBool::new(false);\n", ); - out.push_str(" }\n"); - out.push_str(" },\n"); - out.push_str(" ).await?;\n"); + out.push_str(" let delivered = &delivered;\n"); + out.push_str(" crate::mdds::macros::run_streaming_retry_loop(\n"); + out.push_str(" client.session(),\n"); + out.push_str(" &policy,\n"); + writeln!(out, " {endpoint_name_literal},").unwrap(); + out.push_str(" delivered,\n"); + out.push_str(" move |snap| {\n"); + // Clone per-attempt: the FnMut closure may fire twice (post-refresh + // restart) and the proto request takes the params by value. + out.push_str(" let params = params.clone();\n"); + out.push_str(" async move {\n"); + out.push_str( + " let qi = client.build_query_info(snap.uuid.clone());\n", + ); + writeln!( + out, + " let request = proto::{} {{", + endpoint.request_type + ) + .unwrap(); + out.push_str(" query_info: Some(qi),\n"); + out.push_str(" params: Some(params),\n"); + out.push_str(" };\n"); + out.push_str(&stub_call); + out.push_str(&chunk_body); + out.push_str(" }\n"); + out.push_str(" },\n"); + out.push_str(" ).await?;\n"); + out.push_str(" }\n"); + out.push_str(" }\n"); writeln!( out, " metrics::histogram!(\"thetadatadx.grpc.latency_ms\", \"endpoint\" => {endpoint_name_literal})" diff --git a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl b/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl index a99fe553..893c4d3b 100644 --- a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl +++ b/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl @@ -42,6 +42,28 @@ /// Decode and decompress failures are terminal and surfaced /// immediately without retry. /// + /// # Bulk-fetch sharding + /// + /// Under [`crate::config::BulkFetchPolicy::Auto`] (the default), a + /// history pull large enough to clear the fan-out break-even is + /// split into equal disjoint bands — the same plan the buffered + /// sibling builder uses (see `mdds/shard.rs`) — and every band + /// streams concurrently as its own top-level request, forwarding + /// each chunk to `handler` as it arrives. Every chunk is still + /// delivered exactly once and chunks stay intact; only ORDER + /// changes: chunks from different bands interleave in arrival + /// order rather than the single stream's order. Each band of a + /// single-contract / stock pull is internally time-ascending; + /// bands of an option-chain pull each carry the server's own + /// per-band enumeration. The retry / no-replay guard above applies + /// per band. A band error fails the whole call and cancels the + /// sibling streams (chunks already delivered stay delivered, like + /// any mid-stream error), and dropping the future — the deadline + /// path — cancels every band. For the single stream's exact chunk + /// order, use the buffered sibling builder (which merges) or set + /// [`bulk_fetch = Off`](crate::config::BulkFetchPolicy::Off). + /// Small pulls never shard. + /// /// # Errors /// /// Returns [`Error`] if the gRPC call fails terminally or response diff --git a/thetadatadx-rs/config.default.toml b/thetadatadx-rs/config.default.toml index b1fdac41..50742501 100644 --- a/thetadatadx-rs/config.default.toml +++ b/thetadatadx-rs/config.default.toml @@ -21,9 +21,9 @@ keepalive_timeout_secs = 10 max_message_size = 4194304 # Bulk-fetch sharding policy for large history pulls. "auto" (default) splits -# a big pull into balanced concurrent sub-requests across your tier's request -# budget; "off" runs every query as a single stream in the server's own row -# and chunk order. Case-insensitive. +# a big pull's time or date range into equal concurrent bands across your +# tier's request budget; "off" runs every query as a single stream in the +# server's own row and chunk order. Case-insensitive. # bulk_fetch = "auto" # Fan-out cap for bulk_fetch = "auto". Absent (default) uses the account's diff --git a/thetadatadx-rs/config_surface.toml b/thetadatadx-rs/config_surface.toml index 3453c4c2..2db239c8 100644 --- a/thetadatadx-rs/config_surface.toml +++ b/thetadatadx-rs/config_surface.toml @@ -1771,9 +1771,10 @@ doc = """ Set the automatic bulk-fetch sharding policy for history pulls, covering both the buffered and the chunk-streaming call paths. -- `policy = 0`: Auto (default) — large history pulls are sized with a - cheap density probe and, when worthwhile, split into balanced disjoint - sub-requests across the account's concurrent-request budget. Buffered +- `policy = 0`: Auto (default) — a large history pull's requested time + or date range is split into equal concurrent bands across the + account's concurrent-request budget, decided from the request shape + alone (no sizing request is issued). Buffered pulls merge the shards back into exactly the rows of the single-stream response: single-contract, stock, and index pulls keep the exact single-stream row order, while option-chain pulls come back in a @@ -1794,8 +1795,9 @@ out-of-domain enum int surfaces the same typed class across every binding. """ cpp_doc = """ Set the automatic bulk-fetch sharding policy for history pulls (buffered -and chunk-streaming alike): 0=Auto (default; large pulls are split into -balanced concurrent sub-requests — buffered pulls merge back into +and chunk-streaming alike): 0=Auto (default; a large pull's requested +time or date range is split into equal concurrent bands — buffered +pulls merge back into exactly the single-stream rows, with option-chain pulls in a deterministic canonical contract order and every other pull in the exact single-stream row order; streaming pulls forward each band's @@ -1808,9 +1810,10 @@ leaf the FFI error code selects. """ py_doc = """ Automatic bulk-fetch sharding policy for history pulls (buffered and -chunk-streaming alike). Accepts ``"auto"`` (default: large history -pulls are split into balanced concurrent sub-requests across the -account's concurrent-request budget — buffered pulls are merged back +chunk-streaming alike). Accepts ``"auto"`` (default: a large history +pull's requested time or date range is split into equal concurrent +bands across the account's concurrent-request budget — buffered +pulls are merged back into exactly the rows of the single-stream response, single-contract, stock, and index pulls in the exact single-stream order and option-chain pulls in a deterministic canonical order grouped by @@ -1822,9 +1825,10 @@ row and chunk order), case-insensitive. """ ts_doc = """ Automatic bulk-fetch sharding policy for history pulls (buffered and -chunk-streaming alike). Accepts `"auto"` (default: large history pulls -are split into balanced concurrent sub-requests across the account's -concurrent-request budget — buffered pulls are merged back into exactly +chunk-streaming alike). Accepts `"auto"` (default: a large history +pull's requested time or date range is split into equal concurrent +bands across the account's concurrent-request budget — buffered pulls +are merged back into exactly the rows of the single-stream response, single-contract, stock, and index pulls in the exact single-stream order and option-chain pulls in a deterministic canonical order grouped by expiration, strike, and diff --git a/thetadatadx-rs/src/config/mdds.rs b/thetadatadx-rs/src/config/mdds.rs index 06348565..4f525ef1 100644 --- a/thetadatadx-rs/src/config/mdds.rs +++ b/thetadatadx-rs/src/config/mdds.rs @@ -28,12 +28,10 @@ pub(crate) const DEFAULT_REQUEST_TIMEOUT_SECS: u64 = 300; /// `.await` on a history builder and the chunk-streaming `.stream*` /// calls alike. /// -/// Under `Auto` (the default) a large history pull is first sized with -/// a cheap density probe; when the estimated response is large enough -/// that one stream cannot saturate the account's concurrent-request -/// budget, the SDK splits the query into balanced disjoint -/// sub-requests along a filter axis (a date band or a time band) and -/// runs them across the tier's channel pool. +/// Under `Auto` (the default) the SDK splits a large history pull into +/// equal disjoint sub-requests along a filter axis (a date band or a +/// time band) — cut from the request's own shape, with no sizing query +/// — and runs them across the tier's channel pool. /// /// The buffered path merges the results into exactly the rows the /// single stream would have returned. Row order: single-contract, @@ -55,8 +53,8 @@ pub(crate) const DEFAULT_REQUEST_TIMEOUT_SECS: u64 = 300; /// Small pulls and non-history endpoints are never sharded, so `Auto` /// leaves them byte-identical to `Off`. /// -/// `Off` disables the probe and the fan-out entirely: every query runs -/// as today's single stream, in the server's own row and chunk order. +/// `Off` disables the fan-out entirely: every query runs as today's +/// single stream, in the server's own row and chunk order. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] #[non_exhaustive] pub enum BulkFetchPolicy { @@ -209,8 +207,8 @@ pub struct MarketDataConfig { /// `.await` and chunk-streaming `.stream*` alike. /// /// See [`BulkFetchPolicy`]. Default [`BulkFetchPolicy::Auto`]: large - /// history pulls are probed and, when worthwhile, split into balanced - /// concurrent sub-requests across the tier's channel pool. Buffered + /// history pulls are split into equal concurrent sub-requests across + /// the tier's channel pool. Buffered /// pulls merge the shards back into exactly the rows of the /// single-stream response — single-contract, stock, and index pulls /// in the exact single-stream order, option-chain pulls in a diff --git a/thetadatadx-rs/src/lib.rs b/thetadatadx-rs/src/lib.rs index fd3a91ed..9a20f22d 100644 --- a/thetadatadx-rs/src/lib.rs +++ b/thetadatadx-rs/src/lib.rs @@ -368,7 +368,7 @@ pub mod streaming { pub use mdds::{MarketDataClient, SubscriptionTier}; /// Bulk-fetch shard planning (manual mode): describe a history query as a -/// [`ShardQuery`], obtain the balanced [`ShardPlan`] via +/// [`ShardQuery`], obtain the equal-span [`ShardPlan`] via /// [`MarketDataClient::bulk_fetch_plan`], and run the [`ShardBand`] /// sub-requests under your own concurrency. The automatic path /// ([`BulkFetchPolicy::Auto`]) uses exactly these plans. diff --git a/thetadatadx-rs/src/mdds/client.rs b/thetadatadx-rs/src/mdds/client.rs index 6b282b7c..75211d18 100644 --- a/thetadatadx-rs/src/mdds/client.rs +++ b/thetadatadx-rs/src/mdds/client.rs @@ -273,10 +273,16 @@ impl MarketDataClient { // Template with an empty UUID; each attempt stamps the UUID // from its own session snapshot. self.build_query_info(String::new()), - self.channels.size(), ) } + /// Size of the gRPC channel pool — the tier's server-enforced + /// concurrent-request ceiling, which caps how wide a bulk-fetch + /// shard plan may fan out. + pub(crate) fn pool_size(&self) -> usize { + self.channels.size() + } + /// Deterministically tear the market-data client down. /// /// The market-data client owns only the `Arc`-backed gRPC channel pool — diff --git a/thetadatadx-rs/src/mdds/macros.rs b/thetadatadx-rs/src/mdds/macros.rs index d1012529..94734828 100644 --- a/thetadatadx-rs/src/mdds/macros.rs +++ b/thetadatadx-rs/src/mdds/macros.rs @@ -966,7 +966,7 @@ macro_rules! parsed_endpoint { /// /// Under [`crate::config::BulkFetchPolicy::Auto`] (the /// default), a history pull large enough to clear the - /// fan-out break-even is split into balanced disjoint + /// fan-out break-even is split into equal disjoint /// bands — the same plan the buffered `.await` path uses /// (see `mdds/shard.rs`) — and every band streams /// concurrently as its own top-level request, forwarding @@ -1031,11 +1031,11 @@ macro_rules! parsed_endpoint { let handler_mutex = &handler_mutex; // ── Bulk-fetch auto-sharding (see `mdds/shard.rs`) ── // Same gate as the buffered `IntoFuture` arm: policy - // `Off`, non-history endpoints, small pulls, and probe - // failures all resolve to `None` — the single-stream - // arm below, exactly today's behaviour. Chunks are - // forwarded as they arrive, so bands interleave in - // arrival order (see the method docs). + // `Off`, non-history endpoints, and small pulls all + // resolve to `None` — the single-stream arm below, + // exactly today's behaviour. Chunks are forwarded as + // they arrive, so bands interleave in arrival order + // (see the method docs). #[allow(unused_mut)] // Reason: endpoints with no shardable fields expand no projection arm. let mut shard_query = $crate::mdds::shard::ShardQuery::default(); $(shard_read_field!(shard_query, params, $field);)* @@ -1043,7 +1043,7 @@ macro_rules! parsed_endpoint { client, stringify!($name), &shard_query, - ).await { + ) { Some(plan) => { sharded_stream_fanout!( client, $name, plan, params, @@ -1143,7 +1143,7 @@ macro_rules! parsed_endpoint { client, stringify!($name), &shard_query, - ).await { + ) { Some(plan) => { sharded_stream_fanout!( client, $name, plan, params, @@ -1259,7 +1259,7 @@ macro_rules! parsed_endpoint { client, stringify!($name), &shard_query, - ).await { + ) { Some(plan) => { sharded_stream_fanout!( client, $name, plan, params, @@ -1352,7 +1352,7 @@ macro_rules! parsed_endpoint { client, stringify!($name), &shard_query, - ).await { + ) { Some(plan) => { sharded_stream_fanout!( client, $name, plan, params, @@ -1438,10 +1438,10 @@ macro_rules! parsed_endpoint { let params = proto::$query { $($field : $val),* }; // ── Bulk-fetch auto-sharding (see `mdds/shard.rs`) ── // Project the axis-relevant wire fields and ask the - // planner. Policy `Off`, non-history endpoints, - // small pulls, and probe failures all resolve to - // `None` — the single-stream arm below, exactly - // today's behaviour. + // planner. Policy `Off`, non-history endpoints, and + // small pulls all resolve to `None` — the + // single-stream arm below, exactly today's + // behaviour. #[allow(unused_mut)] // Reason: endpoints with no shardable fields expand no projection arm. let mut shard_query = $crate::mdds::shard::ShardQuery::default(); $(shard_read_field!(shard_query, params, $field);)* @@ -1449,7 +1449,7 @@ macro_rules! parsed_endpoint { client, stringify!($name), &shard_query, - ).await { + ) { Some(plan) => { // N independent top-level requests, one per // band. Each spawned task acquires its OWN diff --git a/thetadatadx-rs/src/mdds/shard.rs b/thetadatadx-rs/src/mdds/shard.rs index e5b7ef1b..6ab10ec9 100644 --- a/thetadatadx-rs/src/mdds/shard.rs +++ b/thetadatadx-rs/src/mdds/shard.rs @@ -1,8 +1,13 @@ //! Automatic sharding of large history pulls. //! -//! One logical query becomes N balanced, disjoint sub-requests along a -//! filter axis (date band, time band), dispatched concurrently across the -//! tier's channel pool. The buffered `.await` path merges the shard +//! One logical query becomes N disjoint sub-requests along a filter +//! axis (date band, time band), dispatched concurrently across the +//! tier's channel pool. Bands are cut from the request shape alone — +//! equal wall-clock duration on the time axis, equal day count on the +//! date axis — never from a sizing query. Balance is equal span, not +//! equal rows: a deliberate trade that costs zero extra round-trips and +//! zero per-endpoint tuning, and the concurrency win dominates the +//! residual row imbalance between bands. The buffered `.await` path merges the shard //! responses back into exactly the rows the single-stream response would //! have carried — in the exact server order for single-contract / stock / //! index pulls, in a deterministic canonical contract order for chains @@ -10,18 +15,19 @@ //! each band's chunks to the user handler as they arrive — no merge, no //! materialize — so chunks from different bands interleave in arrival //! order (see `join_streaming_shards`). The server's per-account send -//! rate caps a single stream well below the tier's aggregate budget, so a -//! balanced fan-out multiplies bulk throughput without exceeding the -//! tier's concurrent-request ceiling. +//! rate caps a single stream well below the tier's aggregate budget, so +//! the fan-out multiplies bulk throughput without exceeding the tier's +//! concurrent-request ceiling. //! //! # Flow (buffered `.await` path) //! //! 1. The generated builder assembles its wire parameters once and //! projects them into a [`ShardQuery`]. -//! 2. `auto_plan` applies the [`BulkFetchPolicy`], the per-endpoint -//! descriptor, axis selection, a cheap density probe, and the -//! row-estimate floor. Anything that does not clearly benefit resolves -//! to `None` and runs on today's single stream, byte-identical. +//! 2. `auto_plan` applies the [`BulkFetchPolicy`], the shardable-endpoint +//! gate, axis selection, and the equal-span band cut — pure +//! computation, no request issued. Anything that does not clearly +//! benefit resolves to `None` and runs on today's single stream, +//! byte-identical. //! 3. Each [`ShardBand`] is applied onto a clone of the original wire //! parameters — every shard is a normal terminal-parity query differing //! only in its band fields. @@ -38,11 +44,11 @@ //! //! # Semaphore safety //! -//! The joining driver never holds a request permit: the probe takes and -//! releases one before any shard spawns, and each shard acquires its own -//! permit inside its spawned task. Since the semaphore is sized to the -//! channel pool (== tier cap) and a plan never exceeds the pool size, N -//! shards make progress even at `pool_size == 1` (they simply serialize). +//! The joining driver never holds a request permit: each shard acquires +//! its own permit inside its spawned task. Since the semaphore is sized +//! to the channel pool (== tier cap) and a plan never exceeds the pool +//! size, N shards make progress even at `pool_size == 1` (they simply +//! serialize). use std::sync::Arc; @@ -54,9 +60,8 @@ use crate::grpc::{ChannelLease, ChannelPool}; use crate::proto; use super::client::MarketDataClient; +use super::decode; use super::decode::headers::find_header; -use super::decode::{self}; -use super::stream::fold_stream_chunks; // ─── Date-range split math (shared with the Python binding) ───────────── @@ -272,8 +277,8 @@ pub enum ShardBand { }, } -/// A balanced fan-out plan for one logical history query: the per-shard -/// band overrides, in output order. +/// An equal-span fan-out plan for one logical history query: the +/// per-shard band overrides, in output order. /// /// Power users running their own concurrency can request the plan through /// [`MarketDataClient::bulk_fetch_plan`] and apply each band to a clone of @@ -376,92 +381,62 @@ pub(crate) fn set_wire_str(field: &mut T, v: &str) { field.set_str(v); } -// ─── Per-endpoint descriptor ───────────────────────────────────────────── +// ─── Shardable endpoints ───────────────────────────────────────────────── -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum Family { - Options, - Stock, - Index, -} - -#[derive(Clone, Copy)] -struct EndpointShard { - family: Family, - /// Coarse response-rows-per-probe-event multiplier. The probe counts - /// trades (the OHLCVC `count` column); trade-anchored responses carry - /// ~1 row per trade, while NBBO/greeks streams print an order of - /// magnitude more rows per trade. 15 is the conservative middle of the - /// 10–30× intraday NBBO-to-trade ratio: high enough that genuinely - /// big quote pulls clear the floor, low enough that a modest pull - /// cannot be talked into a fan-out by a busy tape. - rows_per_event: u64, -} - -const fn tick(family: Family, rows_per_event: u64) -> Option { - Some(EndpointShard { - family, - rows_per_event, - }) -} - -/// Shardability descriptor, keyed by the generated endpoint method name. +/// Whether the generated endpoint method named `endpoint` may shard. /// -/// Only intraday (tick / bar / greeks) bulk history endpoints appear: -/// snapshots, lists, at-time queries, and the daily-only EOD / open-interest -/// / greeks-EOD families return bounded responses — or a per-day probe that -/// IS the pull — that a fan-out cannot help. Endpoints not listed here -/// always run on the single-stream path. -fn descriptor(endpoint: &str) -> Option { - use Family::{Index, Options, Stock}; - match endpoint { - // Trade-anchored option history: ~1 row per probe event. +/// Only intraday (tick / bar / greeks) bulk history endpoints qualify: +/// snapshots, lists, at-time queries, and the daily-only EOD / +/// open-interest / greeks-EOD families return bounded responses that a +/// fan-out cannot help. Endpoints not listed here always run on the +/// single-stream path. +fn is_shardable_history_endpoint(endpoint: &str) -> bool { + matches!( + endpoint, "option_history_trade" - | "option_history_ohlc" - | "option_history_trade_greeks_all" - | "option_history_trade_greeks_first_order" - | "option_history_trade_greeks_second_order" - | "option_history_trade_greeks_third_order" - | "option_history_trade_greeks_implied_volatility" - // Trade+quote pairs one NBBO with each trade: still ~1 row per trade. - | "option_history_trade_quote" => tick(Options, 1), - // Quote-anchored option history: NBBO/greeks rows per trade. - "option_history_quote" - | "option_history_greeks_all" - | "option_history_greeks_first_order" - | "option_history_greeks_second_order" - | "option_history_greeks_third_order" - | "option_history_greeks_implied_volatility" => tick(Options, 15), - "stock_history_trade" | "stock_history_trade_quote" | "stock_history_ohlc" => { - tick(Stock, 1) - } - "stock_history_quote" => tick(Stock, 15), - "index_history_price" | "index_history_ohlc" => tick(Index, 1), - _ => None, - } + | "option_history_ohlc" + | "option_history_trade_quote" + | "option_history_quote" + | "option_history_trade_greeks_all" + | "option_history_trade_greeks_first_order" + | "option_history_trade_greeks_second_order" + | "option_history_trade_greeks_third_order" + | "option_history_trade_greeks_implied_volatility" + | "option_history_greeks_all" + | "option_history_greeks_first_order" + | "option_history_greeks_second_order" + | "option_history_greeks_third_order" + | "option_history_greeks_implied_volatility" + | "stock_history_trade" + | "stock_history_trade_quote" + | "stock_history_ohlc" + | "stock_history_quote" + | "index_history_price" + | "index_history_ohlc" + ) } // ─── Pure planning math ────────────────────────────────────────────────── -/// Probe bin width along the time axis (one minute), and the probe's -/// interval parameter. ~390 bins cover a regular session, enough -/// resolution to place cuts within a minute of the ideal equal-work -/// boundary while keeping the probe response tiny next to the pull it -/// sizes. -const PROBE_BIN_MS: i64 = 60_000; -const PROBE_INTERVAL: &str = "1m"; - -/// Estimated-response-rows floor below which a query stays on the single -/// stream. Derived from the measured live envelope (~43 MiB/s steady-state -/// per stream at the 8 MiB window; the server prepares a response for -/// roughly tens of seconds before the first byte, per request): under -/// ~10 M rows the single stream finishes within about one prepare interval -/// of the fan-out, so the probe plus N prepares erase the win; above it -/// the balanced fan-out's multiple wins decisively. -const SHARD_MIN_EST_ROWS: u64 = 10_000_000; - -/// Regular-session window (09:30–16:00 ET) used to cap per-day row -/// estimates when the query itself carries no parsable window. +/// Bar-grid ceiling below which a bounded-interval query stays on the +/// single stream. A single-contract (or stock / index) query at a +/// bounded bar interval cannot return more than one row per grid slot; +/// when that provable ceiling is this small, the single stream finishes +/// within about one server prepare interval of a fan-out, so sharding +/// cannot win. Chain cross-products and tick-interval pulls have no +/// such ceiling and shard on the window alone. +const SHARD_MIN_GRID_ROWS: u64 = 10_000_000; + +/// Minimum wall-clock span of one time band (five minutes). The server +/// spends a roughly fixed prepare interval per request before the first +/// byte, so a band much narrower than this pays fan-out overhead without +/// meaningful work to parallelize; a window that cannot yield at least +/// two such bands stays on the single stream. +const MIN_SHARD_BAND_MS: i64 = 5 * 60_000; + +/// Regular-session window (09:30–16:00 ET): the per-day bar-grid span +/// the small-pull gate assumes when the query itself carries no +/// parsable window. const RTH_WINDOW_MS: i64 = 23_400_000; /// Parse a wire time-of-day (`HH:MM`, `HH:MM:SS`, `HH:MM:SS.mmm`, or bare @@ -513,7 +488,7 @@ fn format_hms(ms: i64) -> String { } /// Interval width in ms. `None` for `tick` (or anything unrecognised, -/// which errs toward the larger tick-density estimate). +/// which errs toward treating the pull as unbounded tick density). fn interval_ms(s: &str) -> Option { let s = s.trim(); if s.is_empty() || s.eq_ignore_ascii_case("tick") { @@ -552,7 +527,7 @@ fn date_span_days(q: &ShardQuery) -> Option { /// cross-product). Only chain pulls have an a-priori-unbounded row count /// per grid slot; a concrete contract — and every stock / index query — /// is capped at one row per bar for bounded intervals, which the -/// probe-free pre-gates in `plan_query` exploit. +/// small-pull gates in `plan_query` exploit. fn chain_cross_product(q: &ShardQuery) -> bool { q.expiration.is_some() && (q.expiration.as_deref() == Some("*") @@ -585,158 +560,42 @@ fn select_axis(q: &ShardQuery) -> Option { None } -/// Place up to `n - 1` cut boundaries over a work histogram so the -/// cumulative work between consecutive boundaries is as equal as the bins -/// allow. -/// -/// Returns strictly increasing bin boundaries in `1..bins.len()` (a -/// boundary `k` cuts between bin `k - 1` and bin `k`). Every band the -/// boundaries induce carries probe work > 0: a boundary is emitted only -/// with work strictly before it (the cumulative target was reached) and -/// strictly after it (boundaries cap at the last non-empty bin). A -/// zero-work band is a shard the server answers with gRPC `NotFound` -/// rather than an empty stream, so it must never be planned. Degenerate -/// shapes therefore resolve conservatively: an empty or all-zero -/// histogram, `n <= 1`, a single bin, or work concentrated in one bin -/// yield no cuts, and the caller falls back to a single stream. -fn equal_work_cuts(bins: &[u64], n: usize) -> Vec { - let total: u64 = bins.iter().sum(); - if total == 0 || n <= 1 || bins.len() < 2 { - return Vec::new(); - } - // Largest boundary that still leaves work after the cut: the index of - // the last non-empty bin (`total > 0` guarantees one exists). A - // boundary past it would isolate a zero-work trailing band — the - // 1-bar-histogram shape a short intraday window produces. - let last_boundary = bins - .iter() - .rposition(|&w| w > 0) - .expect("total > 0 implies a non-empty bin"); - // `max(1)` keeps the target strictly advancing when total < n, so the - // dedup loop below always terminates. - let per = (total / n as u64).max(1); - let mut cuts: Vec = Vec::with_capacity(n - 1); - let mut cum = 0u64; - let mut target = per; - for (i, &w) in bins.iter().enumerate() { - cum += w; - while cum >= target && cuts.len() < n - 1 { - let boundary = i + 1; - // `boundary <= last_boundary` implies `boundary < bins.len()` - // and guarantees work on both sides of the cut. - if boundary <= last_boundary && cuts.last() != Some(&boundary) { - cuts.push(boundary); - } - target = target.saturating_add(per); - } - } - cuts -} - -/// Materialize time bands from minute-bin cut boundaries. +/// Materialize `n` equal-duration time bands over the inclusive window +/// `[start_ms, end_ms]`. Caller guarantees `1 <= n <= window_ms`. /// /// Bands are inclusive `[start, end]` wire windows at millisecond -/// resolution: band `k` ends at `cut_ms - 1` and band `k + 1` starts at -/// `cut_ms`, so every tick lands in exactly one band. The server treats -/// `start_time` / `end_time` as inclusive at millisecond resolution — -/// verified live: adjacent inclusive bands built this way reproduce the -/// single stream's row count exactly on full-day chain pulls. The first -/// band starts at the query's own `start_time` and the last ends at its -/// `end_time`, both verbatim. -fn time_bands(start_ms: i64, end_ms: i64, cuts: &[usize]) -> Vec { - let mut edges: Vec = vec![start_ms]; - for &c in cuts { - let cut_ms = start_ms + (c as i64) * PROBE_BIN_MS; - if cut_ms > *edges.last().expect("edges seeded") && cut_ms <= end_ms { - edges.push(cut_ms); - } - } - edges.push(end_ms + 1); - edges - .windows(2) - .map(|w| ShardBand::Time { - start_time: format_hms(w[0]), - end_time: format_hms(w[1] - 1), +/// resolution: band `k` ends 1 ms before band `k + 1` starts, so every +/// tick lands in exactly one band. The server treats `start_time` / +/// `end_time` as inclusive at millisecond resolution — verified live: +/// adjacent inclusive bands built this way reproduce the single +/// stream's row count exactly on full-day chain pulls. The first band +/// starts at the query's own `start_time` and the last ends at its +/// `end_time`; band durations differ by at most 1 ms. +fn time_bands(start_ms: i64, end_ms: i64, n: i64) -> Vec { + let window = end_ms - start_ms + 1; + (0..n) + .map(|k| ShardBand::Time { + start_time: format_hms(start_ms + window * k / n), + end_time: format_hms(start_ms + window * (k + 1) / n - 1), }) .collect() } -/// Materialize date bands from day-bin cut boundaries. Bands are -/// contiguous inclusive `[start_date, end_date]` sub-ranges covering the -/// query's range exactly once, matching the server's inclusive date -/// semantics. -fn date_bands(start_ord: i64, end_ord: i64, cuts: &[usize]) -> Vec { - let mut edges: Vec = vec![start_ord]; - for &c in cuts { - let cut_ord = start_ord + c as i64; - if cut_ord > *edges.last().expect("edges seeded") && cut_ord <= end_ord { - edges.push(cut_ord); - } - } - edges.push(end_ord + 1); - edges - .windows(2) - .map(|w| ShardBand::Date { - start_date: Ymd::from_ord(w[0]).to_yyyymmdd(), - end_date: Ymd::from_ord(w[1] - 1).to_yyyymmdd(), +/// Materialize `n` equal-day-count date bands over the inclusive day +/// ordinals `[start_ord, end_ord]`. Caller guarantees `1 <= n <= days`. +/// Bands are contiguous inclusive `[start_date, end_date]` sub-ranges +/// covering the query's range exactly once, matching the server's +/// inclusive date semantics; band day counts differ by at most one. +fn date_bands(start_ord: i64, end_ord: i64, n: i64) -> Vec { + let days = end_ord - start_ord + 1; + (0..n) + .map(|k| ShardBand::Date { + start_date: Ymd::from_ord(start_ord + days * k / n).to_yyyymmdd(), + end_date: Ymd::from_ord(start_ord + days * (k + 1) / n - 1).to_yyyymmdd(), }) .collect() } -/// Density-probe fold: per-bin work, probe rows, and probe events. -struct ProbeSummary { - bins: Vec, - rows: u64, - events: u64, -} - -impl ProbeSummary { - fn new(bins: usize) -> Self { - Self { - bins: vec![0; bins], - rows: 0, - events: 0, - } - } - - fn record(&mut self, bin: usize, count: i64) { - // A bar with a zero/absent trade count still stands for at least - // one response row (index bars carry no trade counts at all), so - // work is floored at 1 per probe row. - let work = u64::try_from(count).unwrap_or(0).max(1); - let idx = bin.min(self.bins.len().saturating_sub(1)); - if let Some(slot) = self.bins.get_mut(idx) { - *slot += work; - } - self.rows += 1; - self.events += work; - } -} - -/// Estimate the pull's response rows from the probe. -/// -/// `rows_per_event x events` models tick-density responses; when the query -/// carries a bounded bar interval, `probe_rows x bars_per_probe_row` caps -/// the estimate at the bar grid's own ceiling. `probe_span_ms` is the wire -/// span one probe row stands for — one probe bin (`PROBE_BIN_MS`) on the -/// time axis, one day's intraday window on the date axis — so a probe row -/// expands into at most `probe_span_ms / interval_ms` finer bars. -fn estimate_rows( - desc: EndpointShard, - query_interval: Option<&str>, - probe: &ProbeSummary, - probe_span_ms: i64, -) -> u64 { - let by_events = desc.rows_per_event.saturating_mul(probe.events); - match query_interval.and_then(interval_ms) { - Some(ivl) => { - let bars_per_row = u64::try_from(probe_span_ms / ivl.min(probe_span_ms)).unwrap_or(1); - by_events.min(probe.rows.saturating_mul(bars_per_row.max(1))) - } - None => by_events, - } -} - // ─── Dispatch handle (owned, task-portable) ────────────────────────────── /// Owned, `'static` snapshot of everything one top-level MDDS dispatch @@ -751,7 +610,6 @@ pub(crate) struct ShardDispatch { channels: ChannelPool, pub(crate) retry: RetryPolicy, query_info_template: proto::QueryInfo, - pool_size: usize, } impl ShardDispatch { @@ -761,7 +619,6 @@ impl ShardDispatch { channels: ChannelPool, retry: RetryPolicy, query_info_template: proto::QueryInfo, - pool_size: usize, ) -> Self { Self { semaphore, @@ -769,7 +626,6 @@ impl ShardDispatch { channels, retry, query_info_template, - pool_size, } } @@ -787,240 +643,6 @@ impl ShardDispatch { } } -// ─── Density probe ─────────────────────────────────────────────────────── - -/// Owned probe parameters; rebuilt into a full request per retry attempt. -enum ProbeParams { - OptionOhlc(proto::OptionHistoryOhlcRequestQuery), - StockOhlc(proto::StockHistoryOhlcRequestQuery), - IndexOhlc(proto::IndexHistoryOhlcRequestQuery), - OptionEod(proto::OptionHistoryEodRequestQuery), - StockEod(proto::StockHistoryEodRequestQuery), - IndexEod(proto::IndexHistoryEodRequestQuery), -} - -impl ProbeParams { - async fn open( - &self, - d: &ShardDispatch, - qi: proto::QueryInfo, - ) -> Result, Error> { - use crate::proto::beta_theta_terminal as stub; - // Bind the lease to a local so the pre-dispatch reservation lives - // across the open await (see `ChannelPool::next`). - let lease = d.channel(); - let out = match self { - Self::OptionOhlc(p) => { - let req = proto::OptionHistoryOhlcRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_option_history_ohlc(&lease, req).await - } - Self::StockOhlc(p) => { - let req = proto::StockHistoryOhlcRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_stock_history_ohlc(&lease, req).await - } - Self::IndexOhlc(p) => { - let req = proto::IndexHistoryOhlcRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_index_history_ohlc(&lease, req).await - } - Self::OptionEod(p) => { - let req = proto::OptionHistoryEodRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_option_history_eod(&lease, req).await - } - Self::StockEod(p) => { - let req = proto::StockHistoryEodRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_stock_history_eod(&lease, req).await - } - Self::IndexEod(p) => { - let req = proto::IndexHistoryEodRequest { - query_info: Some(qi), - params: Some(p.clone()), - }; - stub::get_index_history_eod(&lease, req).await - } - }; - out.map_err(Into::into) - } -} - -fn contract_spec_of(q: &ShardQuery) -> Option { - Some(proto::ContractSpec { - symbol: q.symbol.clone().unwrap_or_default(), - expiration: q.expiration.clone().unwrap_or_default(), - strike: q.strike.clone(), - right: q.right.clone(), - }) -} - -/// Intraday OHLCVC probe over one trading day at [`PROBE_INTERVAL`]. -/// -/// The probe forwards the query's contract identity and time window but not -/// its row-reducing filters (`venue`, `strike_range`, `max_dte`): the banded -/// sub-requests carry every filter verbatim, so returned rows stay exact — -/// only the size estimate can skew (a venue- or strike-filtered pull may be -/// mis-sized against the default tape / full chain the probe counts). -fn time_probe_params(family: Family, q: &ShardQuery, day: &str) -> ProbeParams { - match family { - Family::Options => ProbeParams::OptionOhlc(proto::OptionHistoryOhlcRequestQuery { - contract_spec: contract_spec_of(q), - date: Some(day.to_string()), - expiration: String::new(), - interval: PROBE_INTERVAL.to_string(), - start_time: q.start_time.clone(), - end_time: q.end_time.clone(), - strike_range: None, - start_date: None, - end_date: None, - }), - Family::Stock => ProbeParams::StockOhlc(proto::StockHistoryOhlcRequestQuery { - symbol: q.symbol.clone().unwrap_or_default(), - date: Some(day.to_string()), - interval: PROBE_INTERVAL.to_string(), - start_time: q.start_time.clone(), - end_time: q.end_time.clone(), - venue: None, - start_date: None, - end_date: None, - }), - Family::Index => ProbeParams::IndexOhlc(proto::IndexHistoryOhlcRequestQuery { - symbol: q.symbol.clone().unwrap_or_default(), - start_date: day.to_string(), - end_date: day.to_string(), - interval: PROBE_INTERVAL.to_string(), - start_time: q.start_time.clone(), - end_time: q.end_time.clone(), - }), - } -} - -/// Per-day OHLCVC probe (EOD report) over the query's date range. -fn date_probe_params(family: Family, q: &ShardQuery, start: &str, end: &str) -> ProbeParams { - match family { - Family::Options => ProbeParams::OptionEod(proto::OptionHistoryEodRequestQuery { - contract_spec: contract_spec_of(q), - start_date: start.to_string(), - end_date: end.to_string(), - expiration: String::new(), - max_dte: None, - strike_range: None, - }), - Family::Stock => ProbeParams::StockEod(proto::StockHistoryEodRequestQuery { - symbol: q.symbol.clone().unwrap_or_default(), - start_date: start.to_string(), - end_date: end.to_string(), - }), - Family::Index => ProbeParams::IndexEod(proto::IndexHistoryEodRequestQuery { - symbol: q.symbol.clone().unwrap_or_default(), - start_date: start.to_string(), - end_date: end.to_string(), - }), - } -} - -/// How probe rows fold into axis bins. -enum ProbeBinning { - /// Bin OHLC bars by minute offset from the window start. - TimeBins { start_ms: i64 }, - /// Bin EOD rows by day offset from the range start. - DateBins { start_ord: i64 }, -} - -/// Run the density probe as a normal top-level request: its own semaphore -/// permit, the standard retry / refresh shell, chunk-at-a-time decode -/// (probe memory stays bounded by one chunk). The histogram is rebuilt -/// from scratch on a retry attempt, so a restarted stream cannot -/// double-count. -async fn run_probe( - d: &ShardDispatch, - params: ProbeParams, - binning: ProbeBinning, - bins: usize, -) -> Result { - metrics::counter!("thetadatadx.grpc.requests", "endpoint" => "bulk_fetch_probe").increment(1); - let _permit = d - .semaphore - .acquire() - .await - .map_err(|_| Error::config_internal("request semaphore closed"))?; - // `Mutex` (not `RefCell`) so the retry future stays `Send` — the - // buffered builder futures are `Pin>`. Locking - // is uncontended: the fold is strictly sequential per chunk. - let summary = std::sync::Mutex::new(ProbeSummary::new(bins)); - super::macros::run_unary_retry_loop(&d.session, &d.retry, "bulk_fetch_probe", |snap| { - let params = ¶ms; - let binning = &binning; - let summary = &summary; - async move { - // Fresh fold per attempt: a retried stream replays from chunk - // zero, so the previous partial histogram is discarded. - *summary - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) = ProbeSummary::new(bins); - let stream = params.open(d, d.query_info(snap.uuid)).await?; - fold_stream_chunks(stream, |table| { - let mut s = summary - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - match binning { - ProbeBinning::TimeBins { start_ms } => { - for t in decode::parse_ohlc_ticks(table).map_err(Error::from)? { - let bin = (i64::from(t.ms_of_day) - start_ms).max(0) / PROBE_BIN_MS; - s.record(usize::try_from(bin).unwrap_or(usize::MAX), t.count); - } - } - ProbeBinning::DateBins { start_ord } => { - for t in decode::parse_eod_ticks(table).map_err(Error::from)? { - let bin = - ord_of_yyyymmdd_i32(t.date).map_or(0, |o| (o - start_ord).max(0)); - s.record(usize::try_from(bin).unwrap_or(usize::MAX), t.count); - } - } - } - Ok(()) - }) - .await - } - }) - .await?; - Ok(summary - .into_inner() - .unwrap_or_else(std::sync::PoisonError::into_inner)) -} - -/// `YYYYMMDD` integer (as decoded ticks carry it) to a day ordinal. -fn ord_of_yyyymmdd_i32(date: i32) -> Option { - if date <= 0 { - return None; - } - let (y, m, d) = ( - date / 10_000, - (date / 100 % 100) as u32, - (date % 100) as u32, - ); - is_valid_ymd(y, m, d).then(|| { - Ymd { - year: y as u32, - month: m, - day: d, - } - .to_ord() - }) -} - // ─── Plan construction ─────────────────────────────────────────────────── /// Resolved shard width: the configured `shard_concurrency` clamped into @@ -1035,13 +657,11 @@ fn shard_width(config_width: Option, pool_size: usize) -> usize { } } -/// Policy-gated plan for the automatic buffered path. Any reason not to -/// shard — policy `Off`, an unlisted endpoint, no usable axis, a pull -/// under the floor, or a failed probe — resolves to `None`, and the caller -/// runs today's single stream. A probe failure is deliberately swallowed -/// here (at `debug`): the pull itself must never fail because a sizing -/// query did. -pub(crate) async fn auto_plan( +/// Policy-gated plan for the automatic paths. Any reason not to shard — +/// policy `Off`, an unlisted endpoint, no usable axis, or a provably +/// small or too-narrow pull — resolves to `None`, and the caller runs +/// today's single stream. +pub(crate) fn auto_plan( client: &MarketDataClient, endpoint: &'static str, q: &ShardQuery, @@ -1049,165 +669,108 @@ pub(crate) async fn auto_plan( if client.config().market_data.bulk_fetch == BulkFetchPolicy::Off { return None; } - match plan_query(client, endpoint, q).await { - Ok(plan) => { - if let Some(p) = plan.as_ref() { - let axis = match p.bands.first() { - Some(ShardBand::Time { .. }) => "time", - Some(ShardBand::Date { .. }) => "date", - None => "none", - }; - tracing::debug!( - endpoint, - axis, - shards = p.bands.len(), - "bulk fetch sharded across the request pool" - ); - } - plan - } - Err(err) => { - tracing::debug!( - endpoint, - error = %err, - "bulk-fetch density probe failed; falling back to a single stream" - ); - None - } + let width = shard_width( + client.config().market_data.shard_concurrency, + client.pool_size(), + ); + let plan = plan_query(endpoint, q, width); + if let Some(p) = plan.as_ref() { + let axis = match p.bands.first() { + Some(ShardBand::Time { .. }) => "time", + Some(ShardBand::Date { .. }) => "date", + None => "none", + }; + tracing::debug!( + endpoint, + axis, + shards = p.bands.len(), + "bulk fetch sharded across the request pool" + ); } + plan } -/// Build the shard plan the automatic path would use for `endpoint` and -/// `query`, independent of the configured [`BulkFetchPolicy`]. +/// Build the shard plan for `endpoint` and `q` at fan-out width `width`, +/// from the request shape alone — pure computation, no request issued. +/// +/// Sharding splits a large history pull into equal wall-clock bands — +/// equal duration along the time axis, equal day count along the date +/// axis — run concurrently; it never issues a sizing query. Balance is +/// equal span, not equal rows: a deliberate trade that costs zero extra +/// round-trips and zero per-endpoint tuning, and the concurrency win +/// dominates the residual row imbalance between bands. /// /// Shared by [`MarketDataClient::bulk_fetch_plan`] (manual mode) and -/// [`auto_plan`]. Returns `Ok(None)` when the query should stay on a -/// single stream. -async fn plan_query( - client: &MarketDataClient, - endpoint: &str, - q: &ShardQuery, -) -> Result, Error> { - let Some(desc) = descriptor(endpoint) else { - return Ok(None); - }; - let Some(axis) = select_axis(q) else { - return Ok(None); - }; - let d = client.shard_dispatch(); - let width = shard_width(client.config().market_data.shard_concurrency, d.pool_size); +/// [`auto_plan`]. Returns `None` when the query should stay on a single +/// stream: an endpoint outside the shardable set, a shape with no cut +/// axis, `width < 2`, a window too narrow for two bands of +/// [`MIN_SHARD_BAND_MS`], or a bounded-interval single-contract pull +/// whose bar grid provably stays under [`SHARD_MIN_GRID_ROWS`]. +fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Option { + if !is_shardable_history_endpoint(endpoint) { + return None; + } + let axis = select_axis(q)?; if width < 2 { - return Ok(None); + return None; } + let width = i64::try_from(width).unwrap_or(i64::MAX); match axis { ShardAxis::Time => { - let (Some(start_time), Some(end_time)) = - (q.start_time.as_deref(), q.end_time.as_deref()) - else { - return Ok(None); - }; - let (Some(start_ms), Some(end_ms)) = - (parse_ms_of_day(start_time), parse_ms_of_day(end_time)) - else { - return Ok(None); - }; + let start_ms = parse_ms_of_day(q.start_time.as_deref()?)?; + let end_ms = parse_ms_of_day(q.end_time.as_deref()?)?; if end_ms <= start_ms { - return Ok(None); + return None; } - // Probe-free pre-gate: a single-contract (or stock / index) - // query at a bounded bar interval cannot exceed one row per - // grid slot, so a provably-small pull skips the sizing probe - // entirely — `Auto` must add zero latency where it cannot - // help. Chain cross-products and tick-interval pulls are - // unbounded a priori and fall through to the probe (whose - // response is a ~390-bar day for concrete queries — cheap — - // and proportional to the pull for chains). + // Request-shape small-pull gate: a single-contract (or stock + // / index) query at a bounded bar interval cannot exceed one + // row per grid slot, so a provably-small pull never pays a + // fan-out. Chain cross-products and tick-interval pulls are + // unbounded a priori and shard on the window alone. if !chain_cross_product(q) { if let Some(ivl) = q.interval.as_deref().and_then(interval_ms) { let grid_rows = u64::try_from((end_ms - start_ms) / ivl.max(1) + 1).unwrap_or(0); - if grid_rows < SHARD_MIN_EST_ROWS { - return Ok(None); + if grid_rows < SHARD_MIN_GRID_ROWS { + return None; } } } - let day = match (&q.date, &q.start_date) { - (Some(day), _) => day.clone(), - (None, Some(day)) => day.clone(), - (None, None) => return Ok(None), - }; - let bins = usize::try_from((end_ms - start_ms) / PROBE_BIN_MS + 1).unwrap_or(1); - let probe = run_probe( - &d, - time_probe_params(desc.family, q, &day), - ProbeBinning::TimeBins { start_ms }, - bins, - ) - .await?; - if estimate_rows(desc, q.interval.as_deref(), &probe, PROBE_BIN_MS) < SHARD_MIN_EST_ROWS - { - return Ok(None); - } - let cuts = equal_work_cuts(&probe.bins, width); - if cuts.is_empty() { - return Ok(None); - } - Ok(Some(ShardPlan { - bands: time_bands(start_ms, end_ms, &cuts), - })) + // Equal-duration bands, capped so each spans at least + // `MIN_SHARD_BAND_MS`; a window too narrow for two such + // bands stays on the single stream. + let n = ((end_ms - start_ms + 1) / MIN_SHARD_BAND_MS).min(width); + (n >= 2).then(|| ShardPlan { + bands: time_bands(start_ms, end_ms, n), + }) } ShardAxis::Date => { - let (Some(start_date), Some(end_date)) = - (q.start_date.as_deref(), q.end_date.as_deref()) - else { - return Ok(None); - }; - let (Ok(start), Ok(end)) = - (Ymd::from_yyyymmdd(start_date), Ymd::from_yyyymmdd(end_date)) - else { - return Ok(None); - }; - let (start_ord, end_ord) = (start.to_ord(), end.to_ord()); - let window_ms = match ( - q.start_time.as_deref().and_then(parse_ms_of_day), - q.end_time.as_deref().and_then(parse_ms_of_day), - ) { - (Some(s), Some(e)) if e > s => e - s, - _ => RTH_WINDOW_MS, - }; - // Probe-free pre-gate, same reasoning as the time axis: one - // contract at a bounded bar interval is capped at the grid, - // so a provably-small multi-day pull never pays the probe. + let start_ord = Ymd::from_yyyymmdd(q.start_date.as_deref()?).ok()?.to_ord(); + let end_ord = Ymd::from_yyyymmdd(q.end_date.as_deref()?).ok()?.to_ord(); + // Same small-pull gate as the time axis, with per-day rows + // capped by the query's own intraday window (the regular + // session when it carries none). if !chain_cross_product(q) { if let Some(ivl) = q.interval.as_deref().and_then(interval_ms) { + let window_ms = match ( + q.start_time.as_deref().and_then(parse_ms_of_day), + q.end_time.as_deref().and_then(parse_ms_of_day), + ) { + (Some(s), Some(e)) if e > s => e - s, + _ => RTH_WINDOW_MS, + }; let days = u64::try_from(end_ord - start_ord + 1).unwrap_or(u64::MAX); let per_day = u64::try_from(window_ms / ivl.max(1) + 1).unwrap_or(0); - if days.saturating_mul(per_day) < SHARD_MIN_EST_ROWS { - return Ok(None); + if days.saturating_mul(per_day) < SHARD_MIN_GRID_ROWS { + return None; } } } - let bins = usize::try_from(end_ord - start_ord + 1).unwrap_or(1); - let probe = run_probe( - &d, - date_probe_params(desc.family, q, start_date, end_date), - ProbeBinning::DateBins { start_ord }, - bins, - ) - .await?; - // Per-day row estimates additionally cap at the bar grid the - // query's own window and interval allow (one probe row is one - // day, so `window_ms` is the per-row span). - if estimate_rows(desc, q.interval.as_deref(), &probe, window_ms) < SHARD_MIN_EST_ROWS { - return Ok(None); - } - let cuts = equal_work_cuts(&probe.bins, width); - if cuts.is_empty() { - return Ok(None); - } - Ok(Some(ShardPlan { - bands: date_bands(start_ord, end_ord, &cuts), - })) + // Equal-day-count bands, at most one band per day. + let n = (end_ord - start_ord + 1).min(width); + (n >= 2).then(|| ShardPlan { + bands: date_bands(start_ord, end_ord, n), + }) } } } @@ -1223,23 +786,18 @@ impl MarketDataClient { /// plan stays available with `bulk_fetch = Off`. `endpoint` is the /// builder method name (for example `"option_history_quote"`). /// - /// Returns `Ok(None)` when the query should stay on a single stream: - /// the endpoint is not a bulk history endpoint, the shape offers no - /// cut axis, the density probe sizes the pull under the fan-out - /// break-even, or the probe's histogram offers no cut that leaves - /// work on both sides (for example a one-bar window). - /// - /// # Errors - /// - /// Returns an error when the density probe itself fails (network, - /// authentication, or decode failure). The automatic path treats the - /// same failure as "do not shard" instead. - pub async fn bulk_fetch_plan( - &self, - endpoint: &str, - query: &ShardQuery, - ) -> Result, Error> { - plan_query(self, endpoint, query).await + /// Pure computation on the request shape — no request is issued. + /// Returns `None` when the query should stay on a single stream: the + /// endpoint is not a bulk history endpoint, the shape offers no cut + /// axis, or the pull is provably too small (or its window too + /// narrow) for a fan-out to help. + #[must_use] + pub fn bulk_fetch_plan(&self, endpoint: &str, query: &ShardQuery) -> Option { + let width = shard_width( + self.config().market_data.shard_concurrency, + self.pool_size(), + ); + plan_query(endpoint, query, width) } } @@ -1886,113 +1444,138 @@ mod tests { } } - // ── cut placement ── - - #[test] - fn cuts_on_empty_histogram_yield_no_split() { - assert!(equal_work_cuts(&[], 8).is_empty()); - assert!(equal_work_cuts(&[0, 0, 0, 0], 8).is_empty()); + // ── plan construction ── + + /// Full-chain single-day tick query over the regular session — the + /// headline sharding shape. + fn chain_day_query() -> ShardQuery { + ShardQuery { + symbol: Some("SPXW".into()), + expiration: Some("*".into()), + strike: Some("*".into()), + right: Some("both".into()), + date: Some("20260710".into()), + start_time: Some("09:30:00".into()), + end_time: Some("16:00:00".into()), + interval: Some("tick".into()), + ..ShardQuery::default() + } } #[test] - fn cuts_with_n_of_one_yield_no_split() { - assert!(equal_work_cuts(&[5, 5, 5, 5], 1).is_empty()); - assert!(equal_work_cuts(&[5, 5, 5, 5], 0).is_empty()); + fn plan_cuts_a_chain_day_into_equal_time_bands() { + let plan = plan_query("option_history_quote", &chain_day_query(), 8) + .expect("full-day chain must shard"); + assert_eq!(plan.bands.len(), 8); + let windows: Vec<(i64, i64)> = plan.bands.iter().map(band_window_ms).collect(); + assert_eq!(windows[0].0, parse_ms_of_day("09:30:00").unwrap()); + assert_eq!( + windows.last().unwrap().1, + parse_ms_of_day("16:00:00").unwrap() + ); + for w in windows.windows(2) { + assert_eq!(w[1].0, w[0].1 + 1, "bands must abut at +1 ms"); + } } #[test] - fn uniform_histogram_cuts_into_equal_bands() { - // 8 uniform bins, 4 shards → boundaries every 2 bins. - let cuts = equal_work_cuts(&[10; 8], 4); - assert_eq!(cuts, vec![2, 4, 6]); + fn narrow_window_reduces_band_count_or_declines() { + // Too narrow for two minimum-width bands: single stream. + let mut query = chain_day_query(); + query.end_time = Some("09:36:00".into()); + assert!(plan_query("option_history_quote", &query, 8).is_none()); + // Wide enough for exactly two: two bands, not the full width. + let mut query = chain_day_query(); + query.end_time = Some("09:41:00".into()); + let plan = plan_query("option_history_quote", &query, 8).expect("two bands fit"); + assert_eq!(plan.bands.len(), 2); } #[test] - fn skewed_histogram_balances_cumulative_work() { - // The measured intraday shape: a heavy open. Equal-TIME bands - // would put ~10x the work into the first band; equal-WORK cuts - // must land most boundaries inside the heavy region. - let hist = [100, 80, 40, 10, 5, 5, 5, 5]; - let cuts = equal_work_cuts(&hist, 4); - assert!(cuts.windows(2).all(|w| w[0] < w[1]), "strictly increasing"); - assert!(cuts.len() <= 3); - // No band's work may exceed ~2 target shares (a bin is atomic, so - // perfect balance is impossible, but gross skew must be gone). - let total: u64 = hist.iter().sum(); - let per = total / 4; - let mut edges = vec![0usize]; - edges.extend(&cuts); - edges.push(hist.len()); - for w in edges.windows(2) { - let band: u64 = hist[w[0]..w[1]].iter().sum(); - assert!( - band <= per * 2, - "band {w:?} carries {band} of {total} (target {per})" - ); - } + fn width_under_two_declines() { + assert!(plan_query("option_history_quote", &chain_day_query(), 1).is_none()); + assert!(plan_query("option_history_quote", &chain_day_query(), 0).is_none()); } #[test] - fn all_work_in_one_bin_yields_no_cuts() { - // A bin is the atomic unit: with every event in one bin, any - // boundary would leave a zero-work band on one side — a shard - // the server answers with NotFound — so no cut is usable and - // the planner falls back to a single stream. - assert!(equal_work_cuts(&[0, 0, 0, 1_000, 0, 0], 8).is_empty()); + fn unlisted_endpoint_declines() { + assert!(plan_query("option_history_eod", &chain_day_query(), 8).is_none()); + assert!(plan_query("stock_snapshot_quote", &chain_day_query(), 8).is_none()); } #[test] - fn sparse_probe_yields_only_viable_bands_or_none() { - // The live short-window shape: a 1-minute window probes to a - // ~1-bar histogram, which cannot honor a requested N of 6. One - // non-empty bin admits no cut → single-stream fallback - // (`plan_query` declines on empty cuts). - assert!(equal_work_cuts(&[1_000, 0], 6).is_empty()); - assert!(equal_work_cuts(&[1_000], 6).is_empty()); - // Two non-empty bins admit exactly one cut — two viable bands, - // not six, and none empty. - assert_eq!(equal_work_cuts(&[10, 10, 0, 0], 6), vec![1]); + fn bounded_interval_gates_concrete_contracts_not_chains() { + // A concrete contract at a bounded bar interval is capped at the + // bar grid — provably small over one session — so it stays on + // the single stream. + let mut concrete = chain_day_query(); + concrete.expiration = Some("20260710".into()); + concrete.strike = Some("6000".into()); + concrete.right = Some("call".into()); + concrete.interval = Some("1s".into()); + assert!(plan_query("option_history_quote", &concrete, 8).is_none()); + // The same contract at tick interval has no grid ceiling. + concrete.interval = Some("tick".into()); + assert!(plan_query("option_history_quote", &concrete, 8).is_some()); + // A chain cross-product at the same bounded interval is unbounded + // per grid slot and still shards. + let mut chain = chain_day_query(); + chain.interval = Some("1s".into()); + assert!(plan_query("option_history_quote", &chain, 8).is_some()); } #[test] - fn every_emitted_band_carries_probe_work() { - // A zero-work band is a shard the server answers with NotFound; - // no histogram shape may plan one. Trailing zeros were the live - // failure; leading and interior zeros pin the rest. - for (hist, n) in [ - (&[100u64, 80, 40, 10, 5, 5, 0, 0][..], 4), - (&[5, 0, 5, 0, 0][..], 4), - (&[1, 1, 1, 0][..], 8), - (&[0, 7, 0, 7, 0][..], 3), - ] { - let cuts = equal_work_cuts(hist, n); - let mut edges = vec![0usize]; - edges.extend(&cuts); - edges.push(hist.len()); - for w in edges.windows(2) { - let band: u64 = hist[w[0]..w[1]].iter().sum(); - assert!( - band > 0, - "zero-work band {w:?} for hist {hist:?}, n={n} (cuts {cuts:?})" - ); - } + fn multi_day_range_splits_into_equal_date_bands() { + let query = ShardQuery { + symbol: Some("AAPL".into()), + start_date: Some("20240101".into()), + end_date: Some("20240131".into()), + interval: Some("tick".into()), + ..ShardQuery::default() + }; + let plan = plan_query("stock_history_trade", &query, 4).expect("31 days must shard"); + assert_eq!(plan.bands.len(), 4); + // Bands are contiguous, cover the range exactly once, and their + // day counts differ by at most one. + let ords: Vec<(i64, i64)> = plan + .bands + .iter() + .map(|b| match b { + ShardBand::Date { + start_date, + end_date, + } => ( + Ymd::from_yyyymmdd(start_date).unwrap().to_ord(), + Ymd::from_yyyymmdd(end_date).unwrap().to_ord(), + ), + ShardBand::Time { .. } => panic!("expected date bands"), + }) + .collect(); + assert_eq!(ords[0].0, Ymd::from_yyyymmdd("20240101").unwrap().to_ord()); + assert_eq!( + ords.last().unwrap().1, + Ymd::from_yyyymmdd("20240131").unwrap().to_ord() + ); + for w in ords.windows(2) { + assert_eq!(w[1].0, w[0].1 + 1, "date bands must be contiguous"); } + let days: Vec = ords.iter().map(|(s, e)| e - s + 1).collect(); + let (min, max) = (days.iter().min().unwrap(), days.iter().max().unwrap()); + assert!(max - min <= 1, "unequal band day counts: {days:?}"); + assert_eq!(days.iter().sum::(), 31); } #[test] - fn more_shards_than_bins_caps_at_bin_count() { - let cuts = equal_work_cuts(&[7, 7, 7], 8); - assert!(cuts.len() <= 2, "at most bins-1 boundaries, got {cuts:?}"); - assert!(cuts.windows(2).all(|w| w[0] < w[1])); - assert!(cuts.iter().all(|c| (1..=2).contains(c))); - } - - #[test] - fn tiny_totals_terminate_and_stay_bounded() { - // total < n exercises the per == 0 → max(1) guard. - let cuts = equal_work_cuts(&[1, 1, 1], 8); - assert!(cuts.len() <= 2); - assert!(cuts.windows(2).all(|w| w[0] < w[1])); + fn date_bands_cap_at_one_band_per_day() { + let query = ShardQuery { + symbol: Some("AAPL".into()), + start_date: Some("20240101".into()), + end_date: Some("20240102".into()), + interval: Some("tick".into()), + ..ShardQuery::default() + }; + let plan = plan_query("stock_history_trade", &query, 8).expect("two days shard"); + assert_eq!(plan.bands.len(), 2); } // ── axis selection ── @@ -2060,13 +1643,9 @@ mod tests { // ── band construction ── - #[test] - fn time_bands_partition_the_window_exactly() { - let start = parse_ms_of_day("09:30:00").unwrap(); - let end = parse_ms_of_day("16:00:00").unwrap(); - let bands = time_bands(start, end, &[60, 120, 240]); - assert_eq!(bands.len(), 4); - let parse = |b: &ShardBand| match b { + /// Parsed inclusive `[start_ms, end_ms]` window of a time band. + fn band_window_ms(b: &ShardBand) -> (i64, i64) { + match b { ShardBand::Time { start_time, end_time, @@ -2075,52 +1654,60 @@ mod tests { parse_ms_of_day(end_time).unwrap(), ), ShardBand::Date { .. } => panic!("expected time bands"), - }; + } + } + + #[test] + fn time_bands_partition_the_window_exactly() { + let start = parse_ms_of_day("09:30:00").unwrap(); + let end = parse_ms_of_day("16:00:00").unwrap(); + let bands = time_bands(start, end, 4); + assert_eq!(bands.len(), 4); // First band starts at the query start, last ends at the query // end, and adjacent inclusive bands abut at exactly +1 ms so every // tick lands in one band. - assert_eq!(parse(&bands[0]).0, start); - assert_eq!(parse(bands.last().unwrap()).1, end); + assert_eq!(band_window_ms(&bands[0]).0, start); + assert_eq!(band_window_ms(bands.last().unwrap()).1, end); for w in bands.windows(2) { - assert_eq!(parse(&w[1]).0, parse(&w[0]).1 + 1); + assert_eq!(band_window_ms(&w[1]).0, band_window_ms(&w[0]).1 + 1); } + // Equal wall-clock spans, within the 1 ms division remainder. + let spans: Vec = bands + .iter() + .map(|b| { + let (s, e) = band_window_ms(b); + e - s + 1 + }) + .collect(); + let (min, max) = (spans.iter().min().unwrap(), spans.iter().max().unwrap()); + assert!(max - min <= 1, "unequal band spans: {spans:?}"); } #[test] fn time_bands_format_wire_canonical_times() { let start = parse_ms_of_day("09:30:00").unwrap(); let end = parse_ms_of_day("16:00:00").unwrap(); - let bands = time_bands(start, end, &[60]); + let bands = time_bands(start, end, 2); assert_eq!( bands, vec![ ShardBand::Time { start_time: "09:30:00.000".into(), - end_time: "10:29:59.999".into(), + end_time: "12:44:59.999".into(), }, ShardBand::Time { - start_time: "10:30:00.000".into(), + start_time: "12:45:00.000".into(), end_time: "16:00:00.000".into(), }, ] ); } - #[test] - fn time_bands_drop_out_of_window_cuts() { - let start = parse_ms_of_day("09:30:00").unwrap(); - let end = parse_ms_of_day("09:35:00").unwrap(); - // Boundary 400 minutes in — far past the window end — must not - // create an inverted or empty band. - let bands = time_bands(start, end, &[2, 400]); - assert_eq!(bands.len(), 2); - } - #[test] fn date_bands_partition_the_range_exactly() { let s = Ymd::from_yyyymmdd("20240101").unwrap().to_ord(); let e = Ymd::from_yyyymmdd("20240131").unwrap().to_ord(); - let bands = date_bands(s, e, &[10, 20]); + let bands = date_bands(s, e, 3); assert_eq!( bands, vec![ @@ -2157,32 +1744,6 @@ mod tests { assert_eq!(parse_ms_of_day("09:-5:00"), None); } - #[test] - fn trade_quote_endpoints_are_trade_anchored_weight() { - // trade_quote pairs one NBBO with each trade — ~1 row per trade, so it - // estimates at weight 1, not the pure-quote-stream weight 15. - assert_eq!( - descriptor("option_history_trade_quote") - .unwrap() - .rows_per_event, - 1 - ); - assert_eq!( - descriptor("stock_history_trade_quote") - .unwrap() - .rows_per_event, - 1 - ); - assert_eq!( - descriptor("option_history_quote").unwrap().rows_per_event, - 15 - ); - assert_eq!( - descriptor("stock_history_quote").unwrap().rows_per_event, - 15 - ); - } - #[test] fn chain_key_ranks_match_the_wire_comparator() { let key = ChainKey::new; @@ -2220,38 +1781,7 @@ mod tests { assert_eq!(interval_ms("9223372036854775h"), None); } - // ── size guard ── - - #[test] - fn estimate_scales_events_by_endpoint_weight() { - let desc = descriptor("option_history_quote").unwrap(); - let mut probe = ProbeSummary::new(4); - probe.events = 1_000_000; - probe.rows = 100_000; - // Tick interval: 15 rows per trade. - assert_eq!( - estimate_rows(desc, Some("tick"), &probe, PROBE_BIN_MS), - 15_000_000 - ); - // A bounded 1s interval caps at the bar grid: one probe minute can - // expand into at most 60 one-second rows. - assert_eq!( - estimate_rows(desc, Some("1s"), &probe, PROBE_BIN_MS), - 6_000_000 - ); - } - - #[test] - fn trade_endpoints_estimate_one_row_per_event() { - let desc = descriptor("option_history_trade").unwrap(); - let mut probe = ProbeSummary::new(4); - probe.events = 1_000_000; - probe.rows = 100_000; - assert_eq!( - estimate_rows(desc, Some("tick"), &probe, PROBE_BIN_MS), - 1_000_000 - ); - } + // ── plan gates ── #[test] fn chain_detection_separates_wildcards_from_concrete_contracts() { @@ -2290,21 +1820,23 @@ mod tests { } #[test] - fn descriptor_covers_history_families_only() { - assert!(descriptor("option_history_quote").is_some()); - assert!(descriptor("stock_history_trade").is_some()); - assert!(descriptor("index_history_price").is_some()); - // Daily-only history (EOD / open interest / greeks-EOD) never - // shards — its per-day probe IS the pull — so it carries no - // descriptor and stays on the single stream, like snapshots, - // lists, and at-time queries. - assert!(descriptor("option_history_eod").is_none()); - assert!(descriptor("option_history_open_interest").is_none()); - assert!(descriptor("stock_history_eod").is_none()); - assert!(descriptor("index_history_eod").is_none()); - assert!(descriptor("stock_snapshot_quote").is_none()); - assert!(descriptor("option_list_contracts").is_none()); - assert!(descriptor("stock_at_time_trade").is_none()); + fn shardable_endpoints_cover_history_families_only() { + assert!(is_shardable_history_endpoint("option_history_quote")); + assert!(is_shardable_history_endpoint("stock_history_trade")); + assert!(is_shardable_history_endpoint("index_history_price")); + // Daily-only history (EOD / open interest / greeks-EOD) is + // bounded at one row per day, so it is not shardable and + // stays on the single stream, like snapshots, lists, and + // at-time queries. + assert!(!is_shardable_history_endpoint("option_history_eod")); + assert!(!is_shardable_history_endpoint( + "option_history_open_interest" + )); + assert!(!is_shardable_history_endpoint("stock_history_eod")); + assert!(!is_shardable_history_endpoint("index_history_eod")); + assert!(!is_shardable_history_endpoint("stock_snapshot_quote")); + assert!(!is_shardable_history_endpoint("option_list_contracts")); + assert!(!is_shardable_history_endpoint("stock_at_time_trade")); } // ── ordered merge ── diff --git a/thetadatadx-rs/src/mdds/stream.rs b/thetadatadx-rs/src/mdds/stream.rs index 9b2cf141..f6e2558c 100644 --- a/thetadatadx-rs/src/mdds/stream.rs +++ b/thetadatadx-rs/src/mdds/stream.rs @@ -724,39 +724,6 @@ impl TypedCollect { } } -/// Drain a response stream chunk-at-a-time, handing each decoded chunk — -/// with the first chunk's headers backfilled onto headers-only chunks — -/// to `f`. Peak memory stays bounded by one chunk. Used by the bulk-fetch -/// density probe, whose fold only ever needs one chunk of bars at a time. -/// -/// # Errors -/// -/// Propagates decode / decompress / header-drift errors and whatever `f` -/// returns. -pub(crate) async fn fold_stream_chunks( - mut stream: ServerStreaming, - mut f: F, -) -> Result<(), Error> -where - F: FnMut(&proto::DataTable) -> Result<(), Error>, -{ - let mut saved_headers: Option> = None; - let mut chunk_index: usize = 0; - let max_message_size = stream.max_message_size(); - while let Some(response) = stream.next().await { - let mut table = - decode_chunk_checked(response?, max_message_size, &mut saved_headers, chunk_index)?; - if table.headers.is_empty() { - if let Some(h) = saved_headers.as_ref() { - table.headers.clone_from(h); - } - } - f(&table)?; - chunk_index += 1; - } - Ok(()) -} - /// Decode one streamed `ResponseData` and apply the first-chunk header /// contract shared by [`MarketDataClient::for_each_chunk`] and /// [`MarketDataClient::for_each_chunk_async`]: record the first non-empty diff --git a/thetadatadx-ts/index.d.ts b/thetadatadx-ts/index.d.ts index 0e5911a6..89b78b20 100644 --- a/thetadatadx-ts/index.d.ts +++ b/thetadatadx-ts/index.d.ts @@ -531,9 +531,10 @@ export declare class Config { get marketDataConnectionWindowSizeKb(): bigint /** * Automatic bulk-fetch sharding policy for history pulls (buffered and - * chunk-streaming alike). Accepts `"auto"` (default: large history pulls - * are split into balanced concurrent sub-requests across the account's - * concurrent-request budget — buffered pulls are merged back into exactly + * chunk-streaming alike). Accepts `"auto"` (default: a large history + * pull's requested time or date range is split into equal concurrent + * bands across the account's concurrent-request budget — buffered pulls + * are merged back into exactly * the rows of the single-stream response, single-contract, stock, and * index pulls in the exact single-stream order and option-chain pulls in * a deterministic canonical order grouped by expiration, strike, and diff --git a/thetadatadx-ts/src/_generated/config_accessors.rs b/thetadatadx-ts/src/_generated/config_accessors.rs index df01ed8c..7fc5b636 100644 --- a/thetadatadx-ts/src/_generated/config_accessors.rs +++ b/thetadatadx-ts/src/_generated/config_accessors.rs @@ -790,9 +790,10 @@ impl Config { } /// Automatic bulk-fetch sharding policy for history pulls (buffered and - /// chunk-streaming alike). Accepts `"auto"` (default: large history pulls - /// are split into balanced concurrent sub-requests across the account's - /// concurrent-request budget — buffered pulls are merged back into exactly + /// chunk-streaming alike). Accepts `"auto"` (default: a large history + /// pull's requested time or date range is split into equal concurrent + /// bands across the account's concurrent-request budget — buffered pulls + /// are merged back into exactly /// the rows of the single-stream response, single-contract, stock, and /// index pulls in the exact single-stream order and option-chain pulls in /// a deterministic canonical order grouped by expiration, strike, and From ddc0c52adbe3935d47730855d4e0f967c1a971bc Mon Sep 17 00:00:00 2001 From: TalesOfThales <101120927+userFRM@users.noreply.github.com> Date: Sun, 19 Jul 2026 15:06:43 +0200 Subject: [PATCH 2/6] feat(mdds): bound the blast radius of a shard band failure; unify streaming chunk delivery A terminal band failure on a chunk-streaming sharded pull no longer cancels the sibling bands: they drain to completion, and the call returns the new Error::PartialShardFetch naming the failed band window(s) so the caller can re-pull exactly the missing slices instead of restarting the whole pull. A pull where no chunk reached the handler still fails wholesale with the underlying error, a band that fails before delivering anything still retries transparently, and the call deadline still cancels every band at once. The buffered path's per-band from-scratch re-fetch (attempt-local collection, dedup-free replay) is documented and pinned by tests. The new variant maps to StreamError / THETADATADX_ERR_STREAM across the Python, TypeScript, and C surfaces. The four dedicated *_stream builders now drain through MarketDataClient::deliver_chunk_slices, the same primitive the generated .stream(handler) methods use, so both public surfaces of one endpoint agree on the no-replay delivered gating (marked only when rows reach the handler, keeping a recoverable transient after empty keepalives retryable) and stop at the first decode failure; the divergent for_each_chunk_body template is deleted. The planner now logs at debug why a query did not shard (unlisted endpoint, no cut axis, width under two, malformed window, provably small grid, narrow window), and each band future runs inside a shard_band tracing span carrying its window plus a per-band completion line with rows and duration, so concurrent bands stay distinguishable in the log stream. The shardable-endpoint set is now a const roster asserted equal to the registry's intraday-history endpoints (history* subcategory carrying start_time/end_time filters), so a new intraday history endpoint cannot silently never-shard. A new integration suite drives sharded pulls end-to-end through a per-band scripted mock, buffered and streaming: fan-out shape, band-order merge, NotFound folding, buffered band re-fetch, pre-delivery streaming retry, and the streaming partial-fetch error. --- CHANGELOG.md | 10 + docs-site/docs/changelog.md | 10 + thetadatadx-ffi/src/error.rs | 12 +- thetadatadx-py/src/errors.rs | 21 +- thetadatadx-rs/Cargo.toml | 11 + .../endpoints/render/build_out.rs | 1 - .../build_support/endpoints/render/mdds.rs | 13 +- .../mdds/for_each_chunk_body.rs.tmpl | 34 - .../mdds/stream_method_header.rs.tmpl | 17 +- thetadatadx-rs/src/error.rs | 30 + thetadatadx-rs/src/mdds/macros.rs | 105 ++- thetadatadx-rs/src/mdds/shard.rs | 664 ++++++++++++++---- thetadatadx-rs/tests/sharded_fanout.rs | 577 +++++++++++++++ thetadatadx-ts/src/lib.rs | 13 +- 14 files changed, 1292 insertions(+), 226 deletions(-) delete mode 100644 thetadatadx-rs/build_support/endpoints/render/templates/mdds/for_each_chunk_body.rs.tmpl create mode 100644 thetadatadx-rs/tests/sharded_fanout.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 68b80fbe..624321a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. +- **A failed band no longer takes down a large sharded pull.** A buffered `.await` band that dies mid-collection re-fetches from scratch within the standard retry budget — its rows had not reached the caller, so the replay is invisible and duplicate-free — and only a band that spends its whole budget fails the query. On a chunk-streaming `.stream` / `.stream_async` pull, a band that fails terminally no longer cancels its siblings: the surviving bands drain to completion (every one of their chunks reaches the handler), and the call returns the new `Error::PartialShardFetch` naming the failed band window(s) — the start/end of each lost date or time band — so you can re-pull exactly those slices instead of restarting the whole pull. A streaming band that fails before delivering any chunk still retries transparently, a pull where no chunk reached the handler at all still fails with the underlying error like a single stream, and the call deadline still cancels every band at once, so size `with_deadline` / `timeout_ms` to the whole pull. Mapped to `StreamError` / `THETADATADX_ERR_STREAM` on the Python, TypeScript, C++, and C surfaces. + +### Added + +- **Shard-decision logging.** The bulk-fetch planner now logs at `debug` why a pull did NOT shard (endpoint outside the shardable set, no cut axis in the request shape, fan-out width under two, an unparsable band window, a provably small bar grid, or a window too narrow for two bands) next to the existing "sharded" line, and every band's work is tagged with its window: retry warnings carry the band span, and each band emits a completion line with its row count and duration, so concurrent bands stay distinguishable in the log stream. + +### Fixed + +- **The dedicated `*_stream()` builders now share the `.stream(handler)` chunk-delivery path.** The four dedicated streaming builders (`stock_history_trade_stream`, `stock_history_quote_stream`, `option_history_trade_stream`, `option_history_quote_stream`) previously marked their no-replay guard on every parsed chunk — including empty keepalives — and kept draining the stream after a decode failure, while the `.stream(handler)` methods on the same endpoints marked the guard only once rows actually reached the handler and stopped at the first decode failure. Both surfaces now route through the same delivery primitive, so a transient error that arrives after only empty keepalive chunks retries instead of surfacing terminal, and a decode failure ends the drain immediately — identically on both. + ## [0.2.0] - 2026-07-16 ### Added diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index 68b80fbe..624321a8 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -13,6 +13,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. +- **A failed band no longer takes down a large sharded pull.** A buffered `.await` band that dies mid-collection re-fetches from scratch within the standard retry budget — its rows had not reached the caller, so the replay is invisible and duplicate-free — and only a band that spends its whole budget fails the query. On a chunk-streaming `.stream` / `.stream_async` pull, a band that fails terminally no longer cancels its siblings: the surviving bands drain to completion (every one of their chunks reaches the handler), and the call returns the new `Error::PartialShardFetch` naming the failed band window(s) — the start/end of each lost date or time band — so you can re-pull exactly those slices instead of restarting the whole pull. A streaming band that fails before delivering any chunk still retries transparently, a pull where no chunk reached the handler at all still fails with the underlying error like a single stream, and the call deadline still cancels every band at once, so size `with_deadline` / `timeout_ms` to the whole pull. Mapped to `StreamError` / `THETADATADX_ERR_STREAM` on the Python, TypeScript, C++, and C surfaces. + +### Added + +- **Shard-decision logging.** The bulk-fetch planner now logs at `debug` why a pull did NOT shard (endpoint outside the shardable set, no cut axis in the request shape, fan-out width under two, an unparsable band window, a provably small bar grid, or a window too narrow for two bands) next to the existing "sharded" line, and every band's work is tagged with its window: retry warnings carry the band span, and each band emits a completion line with its row count and duration, so concurrent bands stay distinguishable in the log stream. + +### Fixed + +- **The dedicated `*_stream()` builders now share the `.stream(handler)` chunk-delivery path.** The four dedicated streaming builders (`stock_history_trade_stream`, `stock_history_quote_stream`, `option_history_trade_stream`, `option_history_quote_stream`) previously marked their no-replay guard on every parsed chunk — including empty keepalives — and kept draining the stream after a decode failure, while the `.stream(handler)` methods on the same endpoints marked the guard only once rows actually reached the handler and stopped at the first decode failure. Both surfaces now route through the same delivery primitive, so a transient error that arrives after only empty keepalive chunks retries instead of surfacing terminal, and a decode failure ends the drain immediately — identically on both. + ## [0.2.0] - 2026-07-16 ### Added diff --git a/thetadatadx-ffi/src/error.rs b/thetadatadx-ffi/src/error.rs index 92037cbd..53888421 100644 --- a/thetadatadx-ffi/src/error.rs +++ b/thetadatadx-ffi/src/error.rs @@ -184,7 +184,9 @@ pub(crate) fn error_code_for(err: &thetadatadx::Error) -> i32 { } _ => THETADATADX_ERR_STREAM, }, - Error::FlatFilesUnavailable(_) | Error::PartialReconnect { .. } => THETADATADX_ERR_STREAM, + Error::FlatFilesUnavailable(_) + | Error::PartialReconnect { .. } + | Error::PartialShardFetch { .. } => THETADATADX_ERR_STREAM, _ => THETADATADX_ERR_OTHER, } } @@ -547,6 +549,14 @@ mod tests { ); } + #[test] + fn partial_shard_fetch_routes_to_stream() { + assert_eq!( + error_code_for(&thetadatadx::Error::PartialShardFetch { failed: Vec::new() }), + THETADATADX_ERR_STREAM + ); + } + #[test] fn fpss_kinds_route_to_expected_codes() { assert_eq!( diff --git a/thetadatadx-py/src/errors.rs b/thetadatadx-py/src/errors.rs index 5524b83a..b74ec259 100644 --- a/thetadatadx-py/src/errors.rs +++ b/thetadatadx-py/src/errors.rs @@ -309,12 +309,14 @@ pub fn to_py_err(e: thetadatadx::Error) -> PyErr { StreamErrorKind::ProtocolError => StreamError::new_err(e.to_string()), _ => StreamError::new_err(e.to_string()), }, - // FlatFiles availability + partial-reconnect failures are - // streaming-surface faults; route them to `StreamError` so a - // `except StreamError` clause behaves identically to the C++ - // and C ABI mapping (both pin these to the stream discriminant). + // FlatFiles availability, partial-reconnect, and partial + // shard-fetch failures are streaming-surface faults; route them + // to `StreamError` so a `except StreamError` clause behaves + // identically to the C++ and C ABI mapping (both pin these to + // the stream discriminant). thetadatadx::Error::FlatFilesUnavailable(_) - | thetadatadx::Error::PartialReconnect { .. } => StreamError::new_err(e.to_string()), + | thetadatadx::Error::PartialReconnect { .. } + | thetadatadx::Error::PartialShardFetch { .. } => StreamError::new_err(e.to_string()), // Catch-all for future `#[non_exhaustive]` variants added on the // Rust side. We keep the build green and route to the root class // so the caller still sees a branded exception rather than an @@ -587,6 +589,15 @@ mod tests { }); } + #[test] + fn partial_shard_fetch_maps_to_stream_error() { + Python::initialize(); + Python::attach(|py| { + let err = to_py_err(thetadatadx::Error::PartialShardFetch { failed: Vec::new() }); + assert_exception_class(py, &err, "StreamError"); + }); + } + #[test] fn exception_hierarchy_roots_at_theta_data_error() { // Every branded exception should `isinstance(ThetaDataError)`. diff --git a/thetadatadx-rs/Cargo.toml b/thetadatadx-rs/Cargo.toml index ab24ae49..6ec48bd5 100644 --- a/thetadatadx-rs/Cargo.toml +++ b/thetadatadx-rs/Cargo.toml @@ -424,6 +424,17 @@ name = "grpc_mock_server" path = "tests/grpc_mock_server.rs" required-features = ["__test-helpers"] +# Drives sharded bulk-fetch pulls (buffered + streaming) end-to-end +# through the generated `stock_history_trade` builder against a +# per-band scripted mock: fan-out shape, band merge, empty-band +# folding, buffered band re-fetch, and the streaming partial-fetch +# error. Gated on `__test-helpers` for the test-only client +# constructor and the request-proto re-export. +[[test]] +name = "sharded_fanout" +path = "tests/sharded_fanout.rs" +required-features = ["__test-helpers"] + # Pins endpoint-to-parser routing for the seven `option_history_*` / # `index_at_time_price` endpoints. Gated on `__test-helpers` for # `MarketDataClient::for_endpoint_routing_test` and the `grpc::{Channel, ChannelPool}` diff --git a/thetadatadx-rs/build_support/endpoints/render/build_out.rs b/thetadatadx-rs/build_support/endpoints/render/build_out.rs index c2ebb756..c866f593 100644 --- a/thetadatadx-rs/build_support/endpoints/render/build_out.rs +++ b/thetadatadx-rs/build_support/endpoints/render/build_out.rs @@ -44,7 +44,6 @@ pub fn generate_all() -> Result<(), Box> { const BUILD_OUT_TEMPLATES: &[&str] = &[ "mdds/stream_method_header.rs.tmpl", "mdds/stub_call_error_arm.rs.tmpl", - "mdds/for_each_chunk_body.rs.tmpl", "mdds/metrics_result_block.rs.tmpl", "build_out/invoke_generated_endpoint_preamble.rs.tmpl", "build_out/invoke_generated_endpoint_stream_preamble.rs.tmpl", diff --git a/thetadatadx-rs/build_support/endpoints/render/mdds.rs b/thetadatadx-rs/build_support/endpoints/render/mdds.rs index 98f33698..dc8d8e9a 100644 --- a/thetadatadx-rs/build_support/endpoints/render/mdds.rs +++ b/thetadatadx-rs/build_support/endpoints/render/mdds.rs @@ -392,12 +392,17 @@ pub(super) fn generate_mdds_streaming_endpoint(out: &mut String, endpoint: &Gene ) .unwrap(); // Per-attempt stream body shared by both arms: open the stream via - // the generated stub, then drain chunk-by-chunk through the parser - // into `handler_mutex` — byte-for-byte the pre-shard delivery path. + // the generated stub, then drain chunk-by-chunk through + // `deliver_chunk_slices` — the same delivery primitive the + // `parsed_endpoint!` stream arms use, so the dedicated `*_stream` + // builders and the `.stream(handler)` methods of one endpoint agree + // on the no-resume `delivered` gating (non-empty chunks only) and + // on breaking the drain at the first decode failure. let stub_call = include_str!("templates/mdds/stub_call_error_arm.rs.tmpl") .replace("__GRPC_NAME__", &endpoint.grpc_name); - let chunk_body = include_str!("templates/mdds/for_each_chunk_body.rs.tmpl") - .replace("__PARSER_NAME__", &parser_name); + let chunk_body = format!( + " client.deliver_chunk_slices(stream, {parser_name}, handler_mutex, delivered).await\n" + ); let field_idents = endpoint .fields .iter() diff --git a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/for_each_chunk_body.rs.tmpl b/thetadatadx-rs/build_support/endpoints/render/templates/mdds/for_each_chunk_body.rs.tmpl deleted file mode 100644 index 8d01d403..00000000 --- a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/for_each_chunk_body.rs.tmpl +++ /dev/null @@ -1,34 +0,0 @@ - // Strict decode: a type mismatch inside a chunk - // sets `decode_error` and short-circuits further - // handler calls; the error is returned after - // `for_each_chunk` completes (it eats closure - // output). Decode failures are terminal — the - // retry classifier does not retry `Error::Decode` - // / `Error::Decompress`. - let mut decode_error: Option = None; - let drain_result = client.for_each_chunk(stream, |_headers, rows| { - if decode_error.is_some() { - return; - } - let table = proto::DataTable { - headers: _headers.to_vec(), - data_table: rows.to_vec(), - }; - match __PARSER_NAME__(&table) { - Ok(ticks) => { - // Single call chain at a time in practice; - // a poisoned lock only surfaces if - // `for_each_chunk` panicked mid-callback, - // already a hard error path. - if let Ok(mut h) = handler_mutex.lock() { - (*h)(&ticks); - } - delivered.store(true, std::sync::atomic::Ordering::Relaxed); - } - Err(e) => decode_error = Some(Error::from(e)), - } - }).await; - drain_result.and_then(|()| match decode_error { - Some(e) => Err(e), - None => Ok(()), - }) diff --git a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl b/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl index 893c4d3b..2cd1aaa3 100644 --- a/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl +++ b/thetadatadx-rs/build_support/endpoints/render/templates/mdds/stream_method_header.rs.tmpl @@ -56,11 +56,18 @@ /// single-contract / stock pull is internally time-ascending; /// bands of an option-chain pull each carry the server's own /// per-band enumeration. The retry / no-replay guard above applies - /// per band. A band error fails the whole call and cancels the - /// sibling streams (chunks already delivered stay delivered, like - /// any mid-stream error), and dropping the future — the deadline - /// path — cancels every band. For the single stream's exact chunk - /// order, use the buffered sibling builder (which merges) or set + /// per band. A band that fails terminally does NOT cancel its + /// siblings: they drain to completion, and the call then returns + /// [`Error::PartialShardFetch`](crate::Error::PartialShardFetch) + /// naming the failed band window(s) so you can re-pull exactly + /// those slices — unless no chunk reached `handler` at all, in + /// which case the underlying error surfaces unchanged like a + /// single-stream failure. Dropping the future — the deadline path — + /// still cancels every band; on a very large pull, size + /// `with_deadline` (or the configured `request_timeout_secs`) to + /// the whole pull, since it also bounds the sibling drain after a + /// band failure. For the single stream's exact chunk order, use the + /// buffered sibling builder (which merges) or set /// [`bulk_fetch = Off`](crate::config::BulkFetchPolicy::Off). /// Small pulls never shard. /// diff --git a/thetadatadx-rs/src/error.rs b/thetadatadx-rs/src/error.rs index 120676c4..184b5d78 100644 --- a/thetadatadx-rs/src/error.rs +++ b/thetadatadx-rs/src/error.rs @@ -527,6 +527,36 @@ pub enum Error { crate::fpss::protocol::Contract, )>, }, + + /// A sharded chunk-streaming history pull (`.stream*` under + /// `bulk_fetch = "auto"`) completed with one or more bands lost: + /// every surviving band delivered its chunks to the handler in + /// full, and each listed [`crate::ShardBand`] names a window whose + /// data is missing (in band order). Re-issue the same call narrowed + /// to each listed window to fetch the gap. + /// + /// A failed band may have delivered a chunk prefix before failing — + /// MDDS has no resume token, so the SDK never replays a + /// mid-delivered band (that would hand the handler duplicate rows). + /// A re-pull of a listed window therefore re-delivers any prefix + /// the failed band already handed over; rows carry their timestamps + /// and the window is named exactly, so the caller can drop what it + /// already holds for that window before (or while) re-pulling. + /// Each band's underlying error is logged at `warn` with the band + /// window before this error is returned. + /// + /// Only the streaming path reports partial fetches: the buffered + /// `.await` path re-fetches a failed band transparently (nothing + /// has reached the caller) and fails wholesale if the band's retry + /// budget spends out. A streaming pull where NO chunk reached the + /// handler also fails wholesale with the underlying error, exactly + /// like a single stream. + #[error("partial shard fetch: {} band window(s) failed; re-pull the listed windows", .failed.len())] + PartialShardFetch { + /// Band windows whose data did not (fully) arrive, in band + /// order. + failed: Vec, + }, } impl Error { diff --git a/thetadatadx-rs/src/mdds/macros.rs b/thetadatadx-rs/src/mdds/macros.rs index 94734828..b8b68cf6 100644 --- a/thetadatadx-rs/src/mdds/macros.rs +++ b/thetadatadx-rs/src/mdds/macros.rs @@ -807,8 +807,13 @@ macro_rules! shard_apply_field { /// deadlock-free contract as the buffered fan-out), runs the standard /// per-shard [`run_streaming_retry_loop`] with a per-shard `delivered` /// no-resume guard, and forwards its chunks to the shared handler -/// through `$attempt`. `join_streaming_shards` drives the bands -/// concurrently and applies the empty-band (`NotFound`) folding. +/// through `$attempt`. Each band future resolves to +/// `(delivered, outcome)` — the driver reads the delivered flag on +/// failures too — and runs inside its band's `tracing` span, so retry +/// warnings and the per-band completion line stay distinguishable +/// across concurrent bands. `join_streaming_shards` drives the bands +/// concurrently, folds empty (`NotFound`) bands, and reports terminal +/// band failures after the siblings drain (see its docs). /// /// `$attempt` is the per-attempt stream body, written by the calling arm /// with the injected bindings `$snap` (the session snapshot), `$banded` @@ -832,27 +837,38 @@ macro_rules! sharded_stream_fanout { #[allow(unused_mut)] // Reason: bands only override fields the endpoint has. let mut banded = $params.clone(); $(shard_apply_field!(banded, band, $field);)* - shard_futures.push(Box::pin(async move { - let _permit = $client.request_semaphore.acquire().await - .map_err(|_| Error::config_internal("request semaphore closed"))?; - // Per-shard no-resume guard: a shard that has handed a - // non-empty chunk to the handler must not replay from - // chunk zero, while a sibling's delivery leaves this - // shard's pre-first-chunk retry budget intact. + let band_span = $crate::mdds::shard::band_span(band); + shard_futures.push(Box::pin(tracing::Instrument::instrument(async move { + let band_started = std::time::Instant::now(); let shard_delivered = std::sync::atomic::AtomicBool::new(false); - let $delivered = &shard_delivered; - let $banded = &banded; - $crate::mdds::macros::run_streaming_retry_loop( - $client.session(), - policy, - stringify!($name), - $delivered, - move |$snap| $attempt, - ).await?; - Ok(shard_delivered.load(std::sync::atomic::Ordering::Relaxed)) - })); + let outcome = async { + let _permit = $client.request_semaphore.acquire().await + .map_err(|_| Error::config_internal("request semaphore closed"))?; + // Per-shard no-resume guard: a shard that has handed a + // non-empty chunk to the handler must not replay from + // chunk zero, while a sibling's delivery leaves this + // shard's pre-first-chunk retry budget intact. + let $delivered = &shard_delivered; + let $banded = &banded; + $crate::mdds::macros::run_streaming_retry_loop( + $client.session(), + policy, + stringify!($name), + $delivered, + move |$snap| $attempt, + ).await + }.await; + let delivered = shard_delivered.load(std::sync::atomic::Ordering::Relaxed); + tracing::debug!( + delivered, + ok = outcome.is_ok(), + elapsed_ms = band_started.elapsed().as_millis() as u64, + "shard band finished" + ); + (delivered, outcome) + }, band_span))); } - $crate::mdds::shard::join_streaming_shards(shard_futures).await + $crate::mdds::shard::join_streaming_shards(&$plan.bands, shard_futures).await }}; } @@ -978,12 +994,20 @@ macro_rules! parsed_endpoint { /// / index pull is internally time-ascending; bands of an /// option-chain pull each carry the server's own per-band /// enumeration. The retry / no-replay guard above applies - /// per band. A band error fails the whole call and cancels - /// the sibling streams (chunks already delivered stay - /// delivered, like any mid-stream error), and dropping the - /// future — the deadline path — cancels every band. For - /// the single stream's exact chunk order, use the buffered - /// `.await` (which merges) or set + /// per band. A band that fails terminally does NOT cancel + /// its siblings: they drain to completion, and the call + /// then returns + /// [`Error::PartialShardFetch`](crate::Error::PartialShardFetch) + /// naming the failed band window(s) so you can re-pull + /// exactly those slices — unless no chunk reached `handler` + /// at all, in which case the underlying error surfaces + /// unchanged like a single-stream failure. Dropping the + /// future — the deadline path — still cancels every band; + /// on a very large pull, size `with_deadline` (or the + /// configured `request_timeout_secs`) to the whole pull, + /// since it also bounds the sibling drain after a band + /// failure. For the single stream's exact chunk order, use + /// the buffered `.await` (which merges) or set /// [`bulk_fetch = Off`](crate::config::BulkFetchPolicy::Off). /// Small pulls and non-history endpoints never shard. /// @@ -1469,12 +1493,19 @@ macro_rules! parsed_endpoint { let mut banded = params.clone(); $(shard_apply_field!(banded, band, $field);)* let dispatch = client.shard_dispatch(); - tasks.push($crate::mdds::shard::spawn_shard(async move { + // The band span tags every event of + // this band's task — including the + // retry-sleep warnings — with the + // band window, so concurrent bands + // stay distinguishable in the logs. + let band_span = $crate::mdds::shard::band_span(band); + tasks.push($crate::mdds::shard::spawn_shard(tracing::Instrument::instrument(async move { + let band_started = std::time::Instant::now(); let _permit = dispatch.semaphore.acquire().await .map_err(|_| Error::config_internal("request semaphore closed"))?; let dispatch = &dispatch; let banded = &banded; - $crate::mdds::macros::run_unary_retry_loop( + let typed_band = $crate::mdds::macros::run_unary_retry_loop( &dispatch.session, &dispatch.retry, stringify!($name), @@ -1499,11 +1530,21 @@ macro_rules! parsed_endpoint { // proto table. The accumulator // lives inside this attempt // closure, so a replayed attempt - // starts from an empty band. + // — a band re-fetched from + // scratch after a mid-collect + // transient — starts from an + // empty band, which is what + // makes the replay dedup-free. $crate::mdds::stream::collect_stream_typed(stream, $parser).await }, - ).await - })); + ).await?; + tracing::debug!( + rows = typed_band.rows.len(), + elapsed_ms = band_started.elapsed().as_millis() as u64, + "shard band finished" + ); + Ok(typed_band) + }, band_span))); } // Join in band order — an empty band // (NotFound) folds to zero rows — then diff --git a/thetadatadx-rs/src/mdds/shard.rs b/thetadatadx-rs/src/mdds/shard.rs index 6ab10ec9..c663ca77 100644 --- a/thetadatadx-rs/src/mdds/shard.rs +++ b/thetadatadx-rs/src/mdds/shard.rs @@ -42,6 +42,24 @@ //! becomes `join_streaming_shards`, which drives the per-band streams //! concurrently and forwards chunks instead of joining tables. //! +//! # Band failure — blast radius +//! +//! A transient band failure recovers per band, inside the band's own +//! retry loop, before either join ever sees it: the buffered path +//! re-fetches the failed band from scratch (its partial collection is +//! attempt-local, nothing reached the caller, so the replay is +//! dedup-free), and a streaming band retries the same way while it has +//! not yet delivered a chunk. Only a band that exhausts its budget (or +//! fails non-retryably) is terminal. On the buffered path that fails +//! the whole query — a merged result missing a band cannot be +//! represented. On the streaming path the siblings DRAIN TO COMPLETION +//! instead of being cancelled — cancelling would truncate them at +//! arbitrary chunk boundaries, leaving ragged half-delivered windows, +//! while draining leaves exactly the failed bands' windows as the known +//! gaps — and the call returns [`crate::Error::PartialShardFetch`] +//! naming those windows so the caller can re-pull precisely the missing +//! slices (see `join_streaming_shards`). +//! //! # Semaphore safety //! //! The joining driver never holds a request permit: each shard acquires @@ -383,37 +401,84 @@ pub(crate) fn set_wire_str(field: &mut T, v: &str) { // ─── Shardable endpoints ───────────────────────────────────────────────── -/// Whether the generated endpoint method named `endpoint` may shard. +/// The generated endpoint methods that may shard. /// /// Only intraday (tick / bar / greeks) bulk history endpoints qualify: /// snapshots, lists, at-time queries, and the daily-only EOD / /// open-interest / greeks-EOD families return bounded responses that a /// fan-out cannot help. Endpoints not listed here always run on the /// single-stream path. +/// +/// The defining property is machine-checkable against the endpoint +/// registry: an endpoint belongs here exactly when its registry entry +/// has a `history*` subcategory AND carries the intraday +/// `start_time` / `end_time` window filters (the daily-only families — +/// EOD, open interest, greeks-EOD — have no intraday window to band). +/// `shardable_set_matches_registry_intraday_history` in this module's +/// tests enforces that equality, so a new intraday history endpoint +/// added to `endpoint_surface.toml` cannot silently never-shard. +const SHARDABLE_HISTORY_ENDPOINTS: &[&str] = &[ + "option_history_trade", + "option_history_ohlc", + "option_history_trade_quote", + "option_history_quote", + "option_history_trade_greeks_all", + "option_history_trade_greeks_first_order", + "option_history_trade_greeks_second_order", + "option_history_trade_greeks_third_order", + "option_history_trade_greeks_implied_volatility", + "option_history_greeks_all", + "option_history_greeks_first_order", + "option_history_greeks_second_order", + "option_history_greeks_third_order", + "option_history_greeks_implied_volatility", + "stock_history_trade", + "stock_history_trade_quote", + "stock_history_ohlc", + "stock_history_quote", + "index_history_price", + "index_history_ohlc", +]; + +/// Whether the generated endpoint method named `endpoint` may shard. fn is_shardable_history_endpoint(endpoint: &str) -> bool { - matches!( - endpoint, - "option_history_trade" - | "option_history_ohlc" - | "option_history_trade_quote" - | "option_history_quote" - | "option_history_trade_greeks_all" - | "option_history_trade_greeks_first_order" - | "option_history_trade_greeks_second_order" - | "option_history_trade_greeks_third_order" - | "option_history_trade_greeks_implied_volatility" - | "option_history_greeks_all" - | "option_history_greeks_first_order" - | "option_history_greeks_second_order" - | "option_history_greeks_third_order" - | "option_history_greeks_implied_volatility" - | "stock_history_trade" - | "stock_history_trade_quote" - | "stock_history_ohlc" - | "stock_history_quote" - | "index_history_price" - | "index_history_ohlc" - ) + SHARDABLE_HISTORY_ENDPOINTS.contains(&endpoint) +} + +/// Why a query did not shard. Logged at `debug` by [`auto_plan`] so an +/// operator wondering why a pull ran on a single stream can see the +/// gate that declined it; never part of the public surface +/// ([`MarketDataClient::bulk_fetch_plan`] keeps its `Option` return). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ShardDecline { + /// Endpoint is outside [`SHARDABLE_HISTORY_ENDPOINTS`]. + UnlistedEndpoint, + /// The request shape offers no cut axis (no multi-day range and no + /// single-day intraday window). + NoAxis, + /// Resolved fan-out width `< 2` — nothing to fan out across. + WidthUnderTwo, + /// An axis was selected but its window fields do not parse (or the + /// window is inverted), so no bands can be cut from it. + MalformedWindow, + /// Bounded-interval single-contract pull whose bar grid provably + /// stays under [`SHARD_MIN_GRID_ROWS`]. + SmallGrid, + /// Window too narrow for two bands of [`MIN_SHARD_BAND_MS`]. + NarrowWindow, +} + +impl ShardDecline { + fn as_str(self) -> &'static str { + match self { + Self::UnlistedEndpoint => "endpoint is not a shardable history endpoint", + Self::NoAxis => "request shape has no cut axis", + Self::WidthUnderTwo => "fan-out width under two", + Self::MalformedWindow => "band window does not parse", + Self::SmallGrid => "bar grid provably small", + Self::NarrowWindow => "window too narrow for two bands", + } + } } // ─── Pure planning math ────────────────────────────────────────────────── @@ -673,21 +738,30 @@ pub(crate) fn auto_plan( client.config().market_data.shard_concurrency, client.pool_size(), ); - let plan = plan_query(endpoint, q, width); - if let Some(p) = plan.as_ref() { - let axis = match p.bands.first() { - Some(ShardBand::Time { .. }) => "time", - Some(ShardBand::Date { .. }) => "date", - None => "none", - }; - tracing::debug!( - endpoint, - axis, - shards = p.bands.len(), - "bulk fetch sharded across the request pool" - ); + match plan_query(endpoint, q, width) { + Ok(p) => { + let axis = match p.bands.first() { + Some(ShardBand::Time { .. }) => "time", + Some(ShardBand::Date { .. }) => "date", + None => "none", + }; + tracing::debug!( + endpoint, + axis, + shards = p.bands.len(), + "bulk fetch sharded across the request pool" + ); + Some(p) + } + Err(decline) => { + tracing::debug!( + endpoint, + reason = decline.as_str(), + "bulk fetch not sharded — running on a single stream" + ); + None + } } - plan } /// Build the shard plan for `endpoint` and `q` at fan-out width `width`, @@ -701,26 +775,31 @@ pub(crate) fn auto_plan( /// dominates the residual row imbalance between bands. /// /// Shared by [`MarketDataClient::bulk_fetch_plan`] (manual mode) and -/// [`auto_plan`]. Returns `None` when the query should stay on a single -/// stream: an endpoint outside the shardable set, a shape with no cut -/// axis, `width < 2`, a window too narrow for two bands of +/// [`auto_plan`]. Declines — the query should stay on a single stream — +/// carry the gate that fired ([`ShardDecline`], logged by `auto_plan`): +/// an endpoint outside the shardable set, a shape with no cut axis, +/// `width < 2`, a window too narrow for two bands of /// [`MIN_SHARD_BAND_MS`], or a bounded-interval single-contract pull /// whose bar grid provably stays under [`SHARD_MIN_GRID_ROWS`]. -fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Option { +fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Result { if !is_shardable_history_endpoint(endpoint) { - return None; + return Err(ShardDecline::UnlistedEndpoint); } - let axis = select_axis(q)?; + let axis = select_axis(q).ok_or(ShardDecline::NoAxis)?; if width < 2 { - return None; + return Err(ShardDecline::WidthUnderTwo); } let width = i64::try_from(width).unwrap_or(i64::MAX); + // `select_axis` guarantees the axis fields are present; from here a + // missing or unparsable field is a malformed window, not a missing + // axis. + let window = ShardDecline::MalformedWindow; match axis { ShardAxis::Time => { - let start_ms = parse_ms_of_day(q.start_time.as_deref()?)?; - let end_ms = parse_ms_of_day(q.end_time.as_deref()?)?; + let start_ms = parse_ms_of_day(q.start_time.as_deref().ok_or(window)?).ok_or(window)?; + let end_ms = parse_ms_of_day(q.end_time.as_deref().ok_or(window)?).ok_or(window)?; if end_ms <= start_ms { - return None; + return Err(window); } // Request-shape small-pull gate: a single-contract (or stock // / index) query at a bounded bar interval cannot exceed one @@ -732,7 +811,7 @@ fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Option let grid_rows = u64::try_from((end_ms - start_ms) / ivl.max(1) + 1).unwrap_or(0); if grid_rows < SHARD_MIN_GRID_ROWS { - return None; + return Err(ShardDecline::SmallGrid); } } } @@ -740,13 +819,20 @@ fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Option // `MIN_SHARD_BAND_MS`; a window too narrow for two such // bands stays on the single stream. let n = ((end_ms - start_ms + 1) / MIN_SHARD_BAND_MS).min(width); - (n >= 2).then(|| ShardPlan { + if n < 2 { + return Err(ShardDecline::NarrowWindow); + } + Ok(ShardPlan { bands: time_bands(start_ms, end_ms, n), }) } ShardAxis::Date => { - let start_ord = Ymd::from_yyyymmdd(q.start_date.as_deref()?).ok()?.to_ord(); - let end_ord = Ymd::from_yyyymmdd(q.end_date.as_deref()?).ok()?.to_ord(); + let start_ord = Ymd::from_yyyymmdd(q.start_date.as_deref().ok_or(window)?) + .map_err(|_| window)? + .to_ord(); + let end_ord = Ymd::from_yyyymmdd(q.end_date.as_deref().ok_or(window)?) + .map_err(|_| window)? + .to_ord(); // Same small-pull gate as the time axis, with per-day rows // capped by the query's own intraday window (the regular // session when it carries none). @@ -762,13 +848,18 @@ fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Option let days = u64::try_from(end_ord - start_ord + 1).unwrap_or(u64::MAX); let per_day = u64::try_from(window_ms / ivl.max(1) + 1).unwrap_or(0); if days.saturating_mul(per_day) < SHARD_MIN_GRID_ROWS { - return None; + return Err(ShardDecline::SmallGrid); } } } // Equal-day-count bands, at most one band per day. + // `select_axis` only picks the date axis for `span >= 2`, so + // with `width >= 2` (checked above) two bands always fit. let n = (end_ord - start_ord + 1).min(width); - (n >= 2).then(|| ShardPlan { + if n < 2 { + return Err(ShardDecline::NarrowWindow); + } + Ok(ShardPlan { bands: date_bands(start_ord, end_ord, n), }) } @@ -797,12 +888,30 @@ impl MarketDataClient { self.config().market_data.shard_concurrency, self.pool_size(), ); - plan_query(endpoint, query, width) + plan_query(endpoint, query, width).ok() } } // ─── Concurrent driver ─────────────────────────────────────────────────── +/// `tracing` span for one shard band's work. The endpoint macros +/// instrument each band future with it, so every event inside the band +/// — the retry-sleep warnings from `sleep_for_retry`, the per-band +/// completion line, chunk-level debug — carries the band's window and +/// concurrent bands stay distinguishable in the log stream. +pub(crate) fn band_span(band: &ShardBand) -> tracing::Span { + match band { + ShardBand::Date { + start_date, + end_date, + } => tracing::debug_span!("shard_band", band_start = %start_date, band_end = %end_date), + ShardBand::Time { + start_time, + end_time, + } => tracing::debug_span!("shard_band", band_start = %start_time, band_end = %end_time), + } +} + /// A spawned shard request whose task is aborted when the handle drops. /// /// The buffered call's deadline works by dropping the in-flight future @@ -840,10 +949,20 @@ where /// of failing the pull: the bands partition the query, and the union can /// have data even when one band is empty. Only when EVERY shard reports /// `NotFound` is the first such error propagated, matching what a single -/// stream returns for a genuinely empty query. Any other shard error (or -/// panic) fails the whole logical query — the same contract as a -/// single-stream error — and dropping the remaining [`ShardTask`] guards -/// aborts their in-flight requests. +/// stream returns for a genuinely empty query. +/// +/// A band that fails mid-collection recovers transparently BEFORE it +/// reaches this join: each band runs inside its own +/// `run_unary_retry_loop`, whose attempt closure owns the whole band +/// fetch (open + `collect_stream_typed`), so a transient error after +/// partial collection discards the attempt-local buffer and re-fetches +/// the band from scratch, up to `RetryPolicy::max_attempts`. Nothing is +/// surfaced to the caller until the merge, which is what makes the +/// replay dedup-free. Only a band that still fails after its budget (or +/// fails terminally, or panics) fails the whole logical query — the +/// same contract as a single-stream error, since a buffered result +/// missing a band cannot be represented — and dropping the remaining +/// [`ShardTask`] guards aborts their in-flight requests. pub(crate) async fn join_shards( tasks: Vec>>, ) -> Result>, Error> { @@ -881,8 +1000,10 @@ pub(crate) async fn join_shards( // ─── Concurrent streaming driver ───────────────────────────────────────── /// One in-line per-band streaming shard, driven by -/// [`join_streaming_shards`]. Resolves to whether the band delivered any -/// non-empty chunk to the shared handler. +/// [`join_streaming_shards`]. Resolves to `(delivered, outcome)`: +/// whether the band handed any non-empty chunk to the shared handler +/// (reported on failure too — the driver needs it to tell a partial +/// fetch from a wholesale one), and the band's terminal outcome. /// /// Streaming shards are plain futures rather than spawned [`ShardTask`]s /// because they call the user's chunk handler directly: the `.stream*` @@ -893,9 +1014,10 @@ pub(crate) async fn join_shards( /// band's `ServerStreaming` (RST_STREAM) and releases its semaphore /// permit. pub(crate) type ShardStreamFuture<'a> = - std::pin::Pin> + Send + 'a>>; + std::pin::Pin)> + Send + 'a>>; -/// Drive per-band streaming shards concurrently to completion. +/// Drive per-band streaming shards concurrently, each to its own +/// terminal state. `shards[i]` must be the future for `bands[i]`. /// /// All bands are polled on the caller's task, so chunks reach the shared /// handler in arrival order across bands; the caller (the fan-out arm in @@ -909,58 +1031,99 @@ pub(crate) type ShardStreamFuture<'a> = /// gRPC `NotFound`, which folds to an empty contribution; only when /// every band is empty — NotFound-folded or completed without /// delivering a chunk — is the first `NotFound` propagated, matching -/// what a single stream returns for a genuinely empty query. The fold -/// relies on MDDS issuing `NotFound` only as a pre-stream verdict: a band -/// that delivered chunks and then terminated `NotFound` would be folded -/// rather than surfaced, but the server never returns `NotFound` after -/// data. Any other band error fails the whole logical stream as soon as -/// it lands -/// (chunks other bands already delivered stay delivered, like any -/// mid-stream error), and returning drops the remaining band futures, -/// aborting their in-flight requests. +/// what a single stream returns for a genuinely empty query. Only a +/// band that delivered nothing folds: MDDS issues `NotFound` as a +/// pre-stream verdict, and were a band ever to deliver chunks and then +/// terminate `NotFound`, it would surface as a failed band rather than +/// silently folding its delivered rows away. +/// +/// # Band failure bounds the blast radius +/// +/// A band that fails terminally (its per-band retry budget spent, or a +/// non-retryable error) does NOT cancel its siblings: the remaining +/// bands DRAIN TO COMPLETION, delivering their chunks as normal, and +/// the failure is reported once every band has reached its own terminal +/// state. Draining is deliberate — cancelling siblings would truncate +/// them at arbitrary chunk boundaries, leaving ragged half-delivered +/// windows no caller could reason about, while draining leaves exactly +/// the failed bands' windows as the known gaps. The wall-clock cost of +/// draining is bounded by the call's deadline +/// (`run_with_optional_deadline` wraps the whole fan-out), and dropping +/// the join future — the deadline path — still cancels every band at +/// once. /// /// # Errors /// -/// Returns the first non-`NotFound` band error unchanged, or the first -/// `NotFound` when every band was empty. +/// - Some band failed and any band delivered rows: +/// [`Error::PartialShardFetch`] naming the failed bands' windows (in +/// band order), so the caller can re-pull exactly those slices. Each +/// band's underlying error is logged at `warn` inside its band span. +/// - Some band failed and NO chunk was delivered by any band: the first +/// failed band's error unchanged — the pull failed wholesale, exactly +/// as a single stream would surface it. +/// - No band failed and every band was empty: the first `NotFound`. pub(crate) async fn join_streaming_shards( - mut shards: Vec>, + bands: &[ShardBand], + shards: Vec>, ) -> Result<(), Error> { + debug_assert_eq!(bands.len(), shards.len()); + let mut shards: Vec<(usize, ShardStreamFuture<'_>)> = shards.into_iter().enumerate().collect(); let mut first_not_found: Option = None; let mut any_rows = false; + let mut failed: Vec = Vec::new(); + let mut first_failure: Option = None; std::future::poll_fn(|cx| { let mut i = 0; while i < shards.len() { - match shards[i].as_mut().poll(cx) { - std::task::Poll::Ready(Ok(delivered)) => { + let (band_index, fut) = &mut shards[i]; + match fut.as_mut().poll(cx) { + std::task::Poll::Ready((delivered, outcome)) => { any_rows |= delivered; - drop(shards.swap_remove(i)); - } - std::task::Poll::Ready(Err(err)) => { - if matches!( - err, - Error::Grpc { - kind: GrpcStatusKind::NotFound, - .. + if let Err(err) = outcome { + let not_found = matches!( + err, + Error::Grpc { + kind: GrpcStatusKind::NotFound, + .. + } + ); + if not_found && !delivered { + tracing::debug!("empty shard band (NotFound) folded to zero chunks"); + first_not_found.get_or_insert(err); + } else { + tracing::warn!( + band = ?bands[*band_index], + error = %err, + "shard band failed terminally; sibling bands drain to completion" + ); + failed.push(*band_index); + first_failure.get_or_insert(err); } - ) { - tracing::debug!("empty shard band (NotFound) folded to zero chunks"); - first_not_found.get_or_insert(err); - drop(shards.swap_remove(i)); - } else { - return std::task::Poll::Ready(Err(err)); } + drop(shards.swap_remove(i)); } std::task::Poll::Pending => i += 1, } } if shards.is_empty() { - std::task::Poll::Ready(Ok(())) + std::task::Poll::Ready(()) } else { std::task::Poll::Pending } }) - .await?; + .await; + if let Some(err) = first_failure { + // Wholesale failure — the handler saw nothing — surfaces the + // underlying error unchanged, like a single stream; a genuinely + // partial fetch names the lost windows instead. + if !any_rows { + return Err(err); + } + failed.sort_unstable(); + return Err(Error::PartialShardFetch { + failed: failed.into_iter().map(|i| bands[i].clone()).collect(), + }); + } match first_not_found { // Every band was empty — NotFound-folded, or completed without a // delivered chunk — so the whole query is empty and the caller @@ -1483,7 +1646,10 @@ mod tests { // Too narrow for two minimum-width bands: single stream. let mut query = chain_day_query(); query.end_time = Some("09:36:00".into()); - assert!(plan_query("option_history_quote", &query, 8).is_none()); + assert_eq!( + plan_query("option_history_quote", &query, 8), + Err(ShardDecline::NarrowWindow) + ); // Wide enough for exactly two: two bands, not the full width. let mut query = chain_day_query(); query.end_time = Some("09:41:00".into()); @@ -1493,14 +1659,46 @@ mod tests { #[test] fn width_under_two_declines() { - assert!(plan_query("option_history_quote", &chain_day_query(), 1).is_none()); - assert!(plan_query("option_history_quote", &chain_day_query(), 0).is_none()); + assert_eq!( + plan_query("option_history_quote", &chain_day_query(), 1), + Err(ShardDecline::WidthUnderTwo) + ); + assert_eq!( + plan_query("option_history_quote", &chain_day_query(), 0), + Err(ShardDecline::WidthUnderTwo) + ); } #[test] fn unlisted_endpoint_declines() { - assert!(plan_query("option_history_eod", &chain_day_query(), 8).is_none()); - assert!(plan_query("stock_snapshot_quote", &chain_day_query(), 8).is_none()); + assert_eq!( + plan_query("option_history_eod", &chain_day_query(), 8), + Err(ShardDecline::UnlistedEndpoint) + ); + assert_eq!( + plan_query("stock_snapshot_quote", &chain_day_query(), 8), + Err(ShardDecline::UnlistedEndpoint) + ); + } + + #[test] + fn malformed_window_declines() { + // An axis is selected (both time fields present) but the window + // does not parse / is inverted: the malformed-window gate names + // the reason instead of silently reporting "no axis". + let mut query = chain_day_query(); + query.start_time = Some("garbage".into()); + assert_eq!( + plan_query("option_history_quote", &query, 8), + Err(ShardDecline::MalformedWindow) + ); + let mut query = chain_day_query(); + query.start_time = Some("16:00:00".into()); + query.end_time = Some("09:30:00".into()); + assert_eq!( + plan_query("option_history_quote", &query, 8), + Err(ShardDecline::MalformedWindow) + ); } #[test] @@ -1513,15 +1711,46 @@ mod tests { concrete.strike = Some("6000".into()); concrete.right = Some("call".into()); concrete.interval = Some("1s".into()); - assert!(plan_query("option_history_quote", &concrete, 8).is_none()); + assert_eq!( + plan_query("option_history_quote", &concrete, 8), + Err(ShardDecline::SmallGrid) + ); // The same contract at tick interval has no grid ceiling. concrete.interval = Some("tick".into()); - assert!(plan_query("option_history_quote", &concrete, 8).is_some()); + assert!(plan_query("option_history_quote", &concrete, 8).is_ok()); // A chain cross-product at the same bounded interval is unbounded // per grid slot and still shards. let mut chain = chain_day_query(); chain.interval = Some("1s".into()); - assert!(plan_query("option_history_quote", &chain, 8).is_some()); + assert!(plan_query("option_history_quote", &chain, 8).is_ok()); + } + + #[test] + fn date_axis_bounded_interval_gates_small_grids() { + // Date-axis twin of the time-axis grid gate: a stock query at a + // bounded bar interval over a multi-day range is capped at one + // row per grid slot per day (the regular session when the query + // carries no window), so a provably-small pull declines. + let mut query = ShardQuery { + symbol: Some("AAPL".into()), + start_date: Some("20240101".into()), + end_date: Some("20240131".into()), + interval: Some("1m".into()), + ..ShardQuery::default() + }; + assert_eq!( + plan_query("stock_history_ohlc", &query, 8), + Err(ShardDecline::SmallGrid) + ); + // The same range at tick interval has no grid ceiling and shards. + query.interval = Some("tick".into()); + assert!(plan_query("stock_history_ohlc", &query, 8).is_ok()); + // A bounded interval fine enough that the day-count × per-day + // grid clears SHARD_MIN_GRID_ROWS shards too: 60 days × 10ms + // over the assumed regular session is ~140 M grid slots. + query.interval = Some("10ms".into()); + query.end_date = Some("20240229".into()); + assert!(plan_query("stock_history_ohlc", &query, 8).is_ok()); } #[test] @@ -1819,6 +2048,46 @@ mod tests { assert_eq!(shard_width(None, 0), 1); } + /// Machine tie between [`SHARDABLE_HISTORY_ENDPOINTS`] and the + /// endpoint registry generated from `endpoint_surface.toml`: the + /// shardable set must equal, exactly, the registry's intraday + /// history endpoints — subcategory `history*` AND the + /// `start_time` / `end_time` intraday window filters. That pair of + /// properties is what separates bandable tick / bar / greeks + /// history from the bounded-per-day EOD / open-interest / + /// greeks-EOD families (which carry a date range but no intraday + /// window) and from snapshots / lists / at-time queries (not + /// `history*`). A new intraday history endpoint added to the + /// registry fails this test until it is added to the shardable + /// set, so it can never silently never-shard; a stale entry fails + /// it from the other direction. + /// + /// Gated on `__internal` because the registry table only exists + /// under that feature; the CI test job enables it (via + /// `__test-helpers`), so the tie is enforced on every CI run. + #[cfg(feature = "__internal")] + #[test] + fn shardable_set_matches_registry_intraday_history() { + let mut derived: Vec<&str> = crate::mdds::registry::ENDPOINTS + .iter() + .filter(|e| { + e.subcategory.starts_with("history") + && e.params.iter().any(|p| p.name == "start_time") + && e.params.iter().any(|p| p.name == "end_time") + }) + .map(|e| e.name) + .collect(); + derived.sort_unstable(); + let mut listed = SHARDABLE_HISTORY_ENDPOINTS.to_vec(); + listed.sort_unstable(); + assert_eq!( + listed, derived, + "SHARDABLE_HISTORY_ENDPOINTS must equal the registry's \ + intraday history set (subcategory `history*` with \ + start_time/end_time filters)" + ); + } + #[test] fn shardable_endpoints_cover_history_families_only() { assert!(is_shardable_history_endpoint("option_history_quote")); @@ -2377,7 +2646,32 @@ mod tests { fn pending_band(aborted: Arc) -> ShardStreamFuture<'static> { Box::pin(async move { let _guard = AbortFlag(aborted); - std::future::pending().await + std::future::pending::<(bool, Result<(), Error>)>().await + }) + } + + /// Positional stand-in bands for the driver tests: `bands[i]` is the + /// window of `shards[i]`, one synthetic day per band. + fn test_bands(n: usize) -> Vec { + (0..n) + .map(|i| ShardBand::Date { + start_date: format!("2024010{}", i + 1), + end_date: format!("2024010{}", i + 1), + }) + .collect() + } + + /// A band future that resolves an error after yielding once. + fn failing_band(err: Error) -> ShardStreamFuture<'static> { + failing_band_delivered(err, false) + } + + /// [`failing_band`] with an explicit delivered flag — a band that + /// handed chunks to the handler before failing. + fn failing_band_delivered(err: Error, delivered: bool) -> ShardStreamFuture<'static> { + Box::pin(async move { + tokio::task::yield_now().await; + (delivered, Err(err)) }) } @@ -2390,10 +2684,15 @@ mod tests { chunks: Vec>, ) -> ShardStreamFuture<'static> { Box::pin(async move { - let _permit = semaphore - .acquire() - .await - .map_err(|_| Error::config_internal("request semaphore closed"))?; + let _permit = match semaphore.acquire().await { + Ok(permit) => permit, + Err(_) => { + return ( + false, + Err(Error::config_internal("request semaphore closed")), + ) + } + }; let mut delivered = false; for chunk in chunks { tokio::task::yield_now().await; @@ -2401,7 +2700,7 @@ mod tests { delivered |= !chunk.is_empty(); h.extend(chunk); } - Ok(delivered) + (delivered, Ok(())) }) } @@ -2425,7 +2724,9 @@ mod tests { vec![vec![5], vec![6, 7]], ), ]; - join_streaming_shards(shards).await.expect("all bands ok"); + join_streaming_shards(&test_bands(3), shards) + .await + .expect("all bands ok"); let mut rows = std::mem::take(&mut *handler.lock().unwrap()); rows.sort_unstable(); assert_eq!(rows, vec![1, 2, 3, 4, 5, 6, 7]); @@ -2447,30 +2748,62 @@ mod tests { ) }) .collect(); - join_streaming_shards(shards) + join_streaming_shards(&test_bands(3), shards) .await .expect("bands serialize"); assert_eq!(handler.lock().unwrap().len(), 3); } #[tokio::test] - async fn streaming_join_first_error_aborts_sibling_bands() { - // A real band error fails the whole logical stream as soon as it - // lands; the still-pending sibling must be dropped (its stream - // aborted) rather than awaited. - let aborted = Arc::new(AtomicBool::new(false)); + async fn streaming_join_band_failure_drains_siblings_then_reports_partial() { + // A terminal band failure must NOT cancel the siblings: they + // drain to completion (every one of their chunks reaches the + // handler), and the error then names exactly the failed band's + // window so the caller can re-pull that slice. + let semaphore = Arc::new(tokio::sync::Semaphore::new(4)); + let handler = Arc::new(Mutex::new(Vec::new())); + let bands = test_bands(2); let shards = vec![ - pending_band(Arc::clone(&aborted)), - Box::pin(async move { - tokio::task::yield_now().await; - Err(Error::Grpc { - kind: GrpcStatusKind::PermissionDenied, - message: String::new(), - retry_after: None, - }) - }) as ShardStreamFuture<'static>, + chunk_band( + Arc::clone(&semaphore), + Arc::clone(&handler), + vec![vec![1, 2], vec![3], vec![4, 5]], + ), + failing_band(Error::Grpc { + kind: GrpcStatusKind::PermissionDenied, + message: String::new(), + retry_after: None, + }), ]; - let err = join_streaming_shards(shards).await.unwrap_err(); + let err = join_streaming_shards(&bands, shards).await.unwrap_err(); + let Error::PartialShardFetch { failed } = err else { + panic!("expected PartialShardFetch, got {err:?}"); + }; + assert_eq!(failed, vec![bands[1].clone()], "failed window named"); + let mut rows = std::mem::take(&mut *handler.lock().unwrap()); + rows.sort_unstable(); + assert_eq!( + rows, + vec![1, 2, 3, 4, 5], + "the surviving band must drain every chunk despite the sibling failure" + ); + } + + #[tokio::test] + async fn streaming_join_wholesale_failure_surfaces_underlying_error() { + // When NO chunk reached the handler, the pull failed wholesale: + // surface the failed band's own error unchanged (like a single + // stream), not a "partial" fetch that delivered nothing. + let bands = test_bands(2); + let shards = vec![ + Box::pin(async move { (false, Ok(())) }) as ShardStreamFuture<'static>, + failing_band(Error::Grpc { + kind: GrpcStatusKind::PermissionDenied, + message: String::new(), + retry_after: None, + }), + ]; + let err = join_streaming_shards(&bands, shards).await.unwrap_err(); assert!( matches!( err, @@ -2479,14 +2812,63 @@ mod tests { .. } ), - "expected PermissionDenied, got {err:?}" - ); - assert!( - aborted.load(Ordering::Relaxed), - "sibling band must be dropped (aborted) on the first error" + "expected the underlying PermissionDenied, got {err:?}" ); } + #[tokio::test] + async fn streaming_join_partial_counts_the_failed_bands_own_delivery() { + // A band that delivered a chunk prefix and then failed is itself + // the proof the fetch was partial: even with every sibling + // empty, the caller holds rows and must learn the failed + // window rather than receive a bare error. + let bands = test_bands(2); + let shards = vec![ + Box::pin(async move { (false, Ok(())) }) as ShardStreamFuture<'static>, + failing_band_delivered( + Error::Grpc { + kind: GrpcStatusKind::Unavailable, + message: String::new(), + retry_after: None, + }, + true, + ), + ]; + let err = join_streaming_shards(&bands, shards).await.unwrap_err(); + let Error::PartialShardFetch { failed } = err else { + panic!("expected PartialShardFetch, got {err:?}"); + }; + assert_eq!(failed, vec![bands[1].clone()]); + } + + #[tokio::test] + async fn streaming_join_names_every_failed_band_in_band_order() { + // Multiple failed bands: all their windows are listed, in band + // order regardless of completion order. + let semaphore = Arc::new(tokio::sync::Semaphore::new(4)); + let handler = Arc::new(Mutex::new(Vec::new())); + let bands = test_bands(3); + let unavailable = || Error::Grpc { + kind: GrpcStatusKind::Unavailable, + message: String::new(), + retry_after: None, + }; + let shards = vec![ + failing_band(unavailable()), + chunk_band( + Arc::clone(&semaphore), + Arc::clone(&handler), + vec![vec![1], vec![2]], + ), + failing_band(unavailable()), + ]; + let err = join_streaming_shards(&bands, shards).await.unwrap_err(); + let Error::PartialShardFetch { failed } = err else { + panic!("expected PartialShardFetch, got {err:?}"); + }; + assert_eq!(failed, vec![bands[0].clone(), bands[2].clone()]); + } + #[tokio::test] async fn streaming_join_folds_not_found_band_to_zero_chunks() { // Bands partition the query; one band can be empty (NotFound) @@ -2500,9 +2882,9 @@ mod tests { Arc::clone(&handler), vec![vec![1, 2]], ), - Box::pin(async move { Err(not_found()) }) as ShardStreamFuture<'static>, + Box::pin(async move { (false, Err(not_found())) }) as ShardStreamFuture<'static>, ]; - join_streaming_shards(shards) + join_streaming_shards(&test_bands(2), shards) .await .expect("sibling band has data"); assert_eq!(*handler.lock().unwrap(), vec![1, 2]); @@ -2514,10 +2896,12 @@ mod tests { // delivering a chunk — means the whole query is empty: surface // the same NotFound a single stream returns for it. let shards = vec![ - Box::pin(async move { Err(not_found()) }) as ShardStreamFuture<'static>, - Box::pin(async move { Ok(false) }) as ShardStreamFuture<'static>, + Box::pin(async move { (false, Err(not_found())) }) as ShardStreamFuture<'static>, + Box::pin(async move { (false, Ok(())) }) as ShardStreamFuture<'static>, ]; - let err = join_streaming_shards(shards).await.unwrap_err(); + let err = join_streaming_shards(&test_bands(2), shards) + .await + .unwrap_err(); assert!( matches!( err, @@ -2534,11 +2918,15 @@ mod tests { async fn dropping_the_streaming_join_aborts_in_flight_bands() { // The deadline contract: `run_with_optional_deadline` cancels by // dropping the in-flight future, so dropping the join future must - // drop (abort) every band stream. + // drop (abort) every band stream. This is also what bounds the + // sibling drain after a band failure — the deadline still cuts + // everything off at once. let aborted = Arc::new(AtomicBool::new(false)); - let mut fut = Box::pin(join_streaming_shards(vec![pending_band(Arc::clone( - &aborted, - ))])); + let bands = test_bands(1); + let mut fut = Box::pin(join_streaming_shards( + &bands, + vec![pending_band(Arc::clone(&aborted))], + )); let waker = std::task::Waker::noop(); let mut cx = std::task::Context::from_waker(waker); assert!(std::future::Future::poll(fut.as_mut(), &mut cx).is_pending()); diff --git a/thetadatadx-rs/tests/sharded_fanout.rs b/thetadatadx-rs/tests/sharded_fanout.rs new file mode 100644 index 00000000..12cfa783 --- /dev/null +++ b/thetadatadx-rs/tests/sharded_fanout.rs @@ -0,0 +1,577 @@ +//! End-to-end sharded bulk-fetch tests against an in-process gRPC mock. +//! +//! The unit tests in `mdds/shard.rs` pin the planner math and the join +//! drivers in isolation; these tests drive the REAL generated builders +//! (`stock_history_trade`, buffered `.await` and `.stream(handler)`) +//! through `auto_plan` → band fan-out → per-band retry → merge/forward, +//! over the wire against a mock server, so a drift anywhere in that +//! pipeline — the projection macros, the band overrides, the fan-out +//! arms, the join semantics — fails here. +//! +//! The mock accepts any number of connections and streams. Each stream +//! decodes the inbound `StockHistoryTradeRequest`, keys the band by its +//! `start_date` override, and answers from a per-band script (rows, a +//! pre-stream `NotFound`, or rows-then-transient-error), recording every +//! request so the tests can assert the fan-out shape itself. + +#![cfg(feature = "__test-helpers")] + +use std::collections::HashMap; +use std::net::SocketAddr; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use bytes::{BufMut, Bytes, BytesMut}; +use http::{HeaderMap, HeaderName, HeaderValue, Response, StatusCode}; +use prost::Message; +use tokio::net::TcpListener; +use tokio::sync::Semaphore; + +use thetadatadx::grpc::{Channel, ChannelPool}; +use thetadatadx::mdds::MarketDataClient; +use thetadatadx::wire::{ + data_value, CompressionAlgo, CompressionDescription, DataTable, DataValue, DataValueList, + ResponseData, +}; +use thetadatadx::{DirectConfig, Error, RetryPolicy, ShardBand}; + +/// Tag-compatible mirror of the `StockHistoryTradeRequest` / +/// `StockHistoryTradeRequestQuery` wire pair, trimmed to the one field +/// the mock dispatches on. Protobuf decodes by field tag and skips +/// unknown fields, so this reads any band request the client encodes +/// without re-exporting the crate-internal request protos. +#[derive(Clone, PartialEq, prost::Message)] +struct BandRequestProbe { + #[prost(message, optional, tag = "2")] + params: Option, +} + +#[derive(Clone, PartialEq, prost::Message)] +struct BandParamsProbe { + /// `start_date` — tag 6 on `StockHistoryTradeRequestQuery`. + #[prost(string, optional, tag = "6")] + start_date: Option, +} + +// ─── Per-band scripting ────────────────────────────────────────────────── + +/// One trade row: `(ms_of_day, price, date)`, all sent as `Number` +/// cells (`row_price_f64` accepts `Number` for `price`). +type Row = (i64, i64, i64); + +/// One attempt's scripted reply for a band. +#[derive(Clone)] +struct BandResponse { + /// `DataTable` chunks to stream, one `ResponseData` per entry. + chunks: Vec>, + /// Trailing `grpc-status`. With no chunks and a non-zero status the + /// mock answers trailers-only — the pre-stream verdict shape MDDS + /// uses for `NotFound`. + status: u32, +} + +impl BandResponse { + fn ok(chunks: Vec>) -> Self { + Self { chunks, status: 0 } + } + + fn not_found() -> Self { + Self { + chunks: Vec::new(), + status: 5, + } + } + + /// Stream `chunks`, then fail the stream with `Unavailable` — the + /// mid-band transport-error shape. + fn unavailable_after(chunks: Vec>) -> Self { + Self { chunks, status: 14 } + } +} + +/// Scripted replies per band, keyed by the band's `start_date` +/// override. Attempt `n` (0-based) answers with `responses[n]`, +/// clamped to the last entry, so "fail once then succeed" and "always +/// fail" are both one vector. +struct BandScript { + responses: Vec, + attempts: AtomicUsize, +} + +struct MockScript { + bands: HashMap, + /// Every inbound request's band key, in arrival order. + requests: Mutex>, +} + +impl MockScript { + fn new(bands: Vec<(&str, Vec)>) -> Arc { + Arc::new(Self { + bands: bands + .into_iter() + .map(|(key, responses)| { + ( + key.to_string(), + BandScript { + responses, + attempts: AtomicUsize::new(0), + }, + ) + }) + .collect(), + requests: Mutex::new(Vec::new()), + }) + } + + fn attempts(&self, band: &str) -> usize { + self.bands[band].attempts.load(Ordering::Relaxed) + } + + fn seen_bands(&self) -> Vec { + let mut seen = self.requests.lock().unwrap().clone(); + seen.sort(); + seen + } +} + +// ─── Mock server ───────────────────────────────────────────────────────── + +/// Multi-connection, multi-stream gRPC mock: every accepted stream is +/// answered from the shared [`MockScript`]. The sibling +/// `grpc_mock_server.rs` mock serves one fixed reply to every stream; +/// the shard tests need per-band, per-attempt replies, so this mock +/// dispatches on the decoded request instead. +struct BandMockServer { + addr: SocketAddr, + task: tokio::task::JoinHandle<()>, +} + +impl Drop for BandMockServer { + fn drop(&mut self) { + self.task.abort(); + } +} + +async fn spawn_band_mock(script: Arc) -> BandMockServer { + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind ephemeral port"); + let addr = listener.local_addr().expect("local addr"); + let task = tokio::spawn(async move { + loop { + let Ok((socket, _)) = listener.accept().await else { + return; + }; + let _ = socket.set_nodelay(true); + let script = Arc::clone(&script); + tokio::spawn(async move { + let Ok(mut connection) = h2::server::handshake(socket).await else { + return; + }; + while let Some(Ok((request, respond))) = connection.accept().await { + let script = Arc::clone(&script); + tokio::spawn(async move { + let _ = handle_stream(request, respond, &script).await; + }); + } + }); + } + }); + BandMockServer { addr, task } +} + +async fn handle_stream( + request: http::Request, + mut respond: h2::server::SendResponse, + script: &MockScript, +) -> Result<(), Box> { + // Drain the framed request body and decode the band key. + let mut body = request.into_body(); + let mut buf: Vec = Vec::new(); + while let Some(chunk) = body.data().await { + let chunk = chunk?; + let _ = body.flow_control().release_capacity(chunk.len()); + buf.extend_from_slice(&chunk); + } + // gRPC frame: 1-byte compressed flag + 4-byte length + payload. + let payload = buf.get(5..).unwrap_or_default(); + let decoded = BandRequestProbe::decode(payload)?; + let band_key = decoded + .params + .and_then(|p| p.start_date) + .unwrap_or_default(); + script.requests.lock().unwrap().push(band_key.clone()); + + let band = script + .bands + .get(&band_key) + .unwrap_or_else(|| panic!("request for unscripted band {band_key:?}")); + let attempt = band.attempts.fetch_add(1, Ordering::Relaxed); + let reply = band.responses[attempt.min(band.responses.len() - 1)].clone(); + + if reply.chunks.is_empty() && reply.status != 0 { + // Trailers-only response: `grpc-status` on the HEADERS frame + // with END_STREAM — the pre-stream error verdict. + let mut response = Response::new(()); + *response.status_mut() = StatusCode::OK; + response.headers_mut().insert( + http::header::CONTENT_TYPE, + HeaderValue::from_static("application/grpc+proto"), + ); + response.headers_mut().insert( + HeaderName::from_static("grpc-status"), + HeaderValue::from_str(&reply.status.to_string()).expect("numeric ASCII"), + ); + if reply.status == 5 { + response.headers_mut().insert( + HeaderName::from_static("grpc-message"), + HeaderValue::from_static("No data found for your request"), + ); + } + respond.send_response(response, true)?; + return Ok(()); + } + + let mut response = Response::new(()); + *response.status_mut() = StatusCode::OK; + response.headers_mut().insert( + http::header::CONTENT_TYPE, + HeaderValue::from_static("application/grpc+proto"), + ); + let mut send_stream = respond.send_response(response, false)?; + for rows in &reply.chunks { + send_stream.send_data(frame(&trade_chunk(rows)), false)?; + } + let mut trailers = HeaderMap::new(); + trailers.insert( + HeaderName::from_static("grpc-status"), + HeaderValue::from_str(&reply.status.to_string()).expect("numeric ASCII"), + ); + send_stream.send_trailers(trailers)?; + Ok(()) +} + +/// Compose a length-prefixed gRPC frame from a protobuf message. +fn frame(msg: &M) -> Bytes { + let payload = msg.encode_to_vec(); + let mut buf = BytesMut::with_capacity(5 + payload.len()); + buf.put_u8(0); + buf.put_u32(u32::try_from(payload.len()).unwrap()); + buf.extend_from_slice(&payload); + buf.freeze() +} + +/// Build one `ResponseData` chunk carrying trade rows under the +/// v3-canonical headers `parse_trade_ticks` reads. +fn trade_chunk(rows: &[Row]) -> ResponseData { + let number = |n: i64| DataValue { + data_type: Some(data_value::DataType::Number(n)), + }; + let table = DataTable { + headers: vec![ + "ms_of_day".to_string(), + "price".to_string(), + "date".to_string(), + ], + data_table: rows + .iter() + .map(|&(ms, price, date)| DataValueList { + values: vec![number(ms), number(price), number(date)], + }) + .collect(), + }; + ResponseData { + compression_description: Some(CompressionDescription { + algo: CompressionAlgo::None as i32, + level: 0, + }), + original_size: 0, + compressed_data: table.encode_to_vec(), + } +} + +// ─── Client wiring ─────────────────────────────────────────────────────── + +/// `MarketDataClient` with a pool of `width` channels to the mock — +/// `auto_plan` sizes the fan-out from the pool, so `width = 2` yields a +/// two-band plan for a two-day range. Retry backoff is shrunk to keep +/// the recovery tests fast; the budget (3 attempts) stays real. +async fn client_for_mock(mock: &BandMockServer, width: usize) -> MarketDataClient { + let mut channels = Vec::with_capacity(width); + for _ in 0..width { + channels.push( + Channel::connect_h2c("127.0.0.1", mock.addr.port()) + .await + .expect("h2c connect to mock"), + ); + } + let pool = ChannelPool::from_channels(channels); + let mut cfg = DirectConfig::production(); + let mut retry = RetryPolicy::default(); + retry.initial_delay = Duration::from_millis(1); + retry.max_delay = Duration::from_millis(5); + retry.max_attempts = 3; + retry.jitter = false; + cfg.retry = retry; + let sem = Arc::new(Semaphore::new(width)); + MarketDataClient::for_endpoint_routing_test(cfg, pool, sem) +} + +/// The two-day query every test issues: `auto_plan` cuts it into one +/// date band per day at pool width 2. +const DAY1: &str = "20240101"; +const DAY2: &str = "20240102"; + +fn day1_rows() -> Vec { + vec![(34_200_000, 101, 20_240_101), (34_200_500, 102, 20_240_101)] +} + +fn day2_rows() -> Vec { + vec![(34_200_250, 201, 20_240_102), (34_201_000, 202, 20_240_102)] +} + +fn prices(ticks: &[thetadatadx::TradeTick]) -> Vec { + ticks.iter().map(|t| t.price as i64).collect() +} + +// ─── Buffered `.await` path ────────────────────────────────────────────── + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn buffered_sharded_pull_fans_out_and_merges_in_band_order() { + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + (DAY2, vec![BandResponse::ok(vec![day2_rows()])]), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let ticks = client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .await + .expect("sharded buffered pull"); + + // The fan-out issued exactly one request per band, one band per day. + assert_eq!( + script.seen_bands(), + vec![DAY1.to_string(), DAY2.to_string()] + ); + // The merge concatenates the bands in band order — the single-stream + // row order for a stock pull (bands partition the date axis). + assert_eq!(prices(&ticks), vec![101, 102, 201, 202]); + assert_eq!(ticks[0].date, 20_240_101); + assert_eq!(ticks[3].date, 20_240_102); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn buffered_sharded_pull_folds_an_empty_band() { + // One band empty (pre-stream NotFound): it contributes zero rows and + // the union — the sibling band's rows — comes back clean. + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + (DAY2, vec![BandResponse::not_found()]), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let ticks = client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .await + .expect("empty band folds, sibling has data"); + + assert_eq!(prices(&ticks), vec![101, 102]); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn buffered_sharded_pull_refetches_a_failed_band_from_scratch() { + // Band 2 delivers a partial chunk and then dies with a transient + // error; its rows never reached the caller, so the band re-fetches + // from scratch (attempt-local buffer discarded) and the merged + // result carries the full band exactly once — no loss, no + // duplicates, no visible hiccup. + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + ( + DAY2, + vec![ + BandResponse::unavailable_after(vec![vec![(34_200_250, 201, 20_240_102)]]), + BandResponse::ok(vec![day2_rows()]), + ], + ), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let ticks = client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .await + .expect("failed band re-fetches within its retry budget"); + + assert_eq!( + script.attempts(DAY2), + 2, + "the failed band must be re-fetched once" + ); + assert_eq!( + prices(&ticks), + vec![101, 102, 201, 202], + "partial first attempt discarded — full band exactly once" + ); +} + +// ─── Streaming `.stream(handler)` path ─────────────────────────────────── + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn streaming_sharded_pull_forwards_every_bands_chunks() { + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + (DAY2, vec![BandResponse::ok(vec![day2_rows()])]), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let sink: Arc>> = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&sink); + client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .stream(move |ticks| { + seen.lock().unwrap().extend(prices(ticks)); + }) + .await + .expect("sharded streaming pull"); + + assert_eq!( + script.seen_bands(), + vec![DAY1.to_string(), DAY2.to_string()] + ); + // Chunks interleave across bands in arrival order; the union must be + // every row exactly once. + let mut rows = std::mem::take(&mut *sink.lock().unwrap()); + rows.sort_unstable(); + assert_eq!(rows, vec![101, 102, 201, 202]); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn streaming_sharded_pull_reports_the_failed_band_window() { + // Band 2 hands the handler a chunk and then fails terminally (a + // mid-delivery transient cannot replay — no resume token). The + // sibling band must drain to completion, and the error must name + // band 2's exact window so the caller can re-pull that slice. + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + ( + DAY2, + vec![BandResponse::unavailable_after(vec![vec![( + 34_200_250, 201, 20_240_102, + )]])], + ), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let sink: Arc>> = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&sink); + let err = client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .stream(move |ticks| { + seen.lock().unwrap().extend(prices(ticks)); + }) + .await + .expect_err("a mid-delivery band failure must surface"); + + let Error::PartialShardFetch { failed } = err else { + panic!("expected PartialShardFetch, got {err:?}"); + }; + assert_eq!( + failed, + vec![ShardBand::Date { + start_date: DAY2.to_string(), + end_date: DAY2.to_string(), + }], + "the failed band's exact window is named" + ); + assert_eq!( + script.attempts(DAY2), + 1, + "a mid-delivery transient must not replay the band" + ); + let mut rows = std::mem::take(&mut *sink.lock().unwrap()); + rows.sort_unstable(); + assert_eq!( + rows, + vec![101, 102, 201], + "the surviving band drains fully; the failed band's delivered prefix stays delivered" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn streaming_sharded_pull_retries_a_band_that_fails_before_delivery() { + // A band that fails BEFORE any chunk reaches the handler keeps its + // transparent retry: attempt 1 dies pre-delivery, attempt 2 serves + // the band, and the caller sees one clean stream. + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + ( + DAY2, + vec![ + BandResponse::unavailable_after(vec![]), + BandResponse::ok(vec![day2_rows()]), + ], + ), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let sink: Arc>> = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&sink); + client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .stream(move |ticks| { + seen.lock().unwrap().extend(prices(ticks)); + }) + .await + .expect("pre-delivery band failure retries transparently"); + + assert_eq!(script.attempts(DAY2), 2); + let mut rows = std::mem::take(&mut *sink.lock().unwrap()); + rows.sort_unstable(); + assert_eq!(rows, vec![101, 102, 201, 202]); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn streaming_sharded_pull_folds_an_empty_band() { + let script = MockScript::new(vec![ + (DAY1, vec![BandResponse::ok(vec![day1_rows()])]), + (DAY2, vec![BandResponse::not_found()]), + ]); + let mock = spawn_band_mock(Arc::clone(&script)).await; + let client = client_for_mock(&mock, 2).await; + + let sink: Arc>> = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&sink); + client + .stock_history_trade("AAPL") + .start_date(DAY1) + .end_date(DAY2) + .stream(move |ticks| { + seen.lock().unwrap().extend(prices(ticks)); + }) + .await + .expect("empty band folds, sibling has data"); + + let mut rows = std::mem::take(&mut *sink.lock().unwrap()); + rows.sort_unstable(); + assert_eq!(rows, vec![101, 102]); +} diff --git a/thetadatadx-ts/src/lib.rs b/thetadatadx-ts/src/lib.rs index 2856fe6f..d21c70e3 100644 --- a/thetadatadx-ts/src/lib.rs +++ b/thetadatadx-ts/src/lib.rs @@ -630,13 +630,14 @@ fn leaf_class_for(e: &thetadatadx::Error) -> &'static str { StreamErrorKind::ConnectionRefused | StreamErrorKind::Disconnected => "NetworkError", _ => "StreamError", }, - // FlatFiles availability + partial-reconnect failures are - // streaming-surface faults; pin them to `StreamError` so a - // `catch (e instanceof StreamError)` clause behaves identically - // to the C++ and C ABI mapping (both route these to the stream - // discriminant). + // FlatFiles availability, partial-reconnect, and partial + // shard-fetch failures are streaming-surface faults; pin them to + // `StreamError` so a `catch (e instanceof StreamError)` clause + // behaves identically to the C++ and C ABI mapping (both route + // these to the stream discriminant). thetadatadx::Error::FlatFilesUnavailable(_) - | thetadatadx::Error::PartialReconnect { .. } => "StreamError", + | thetadatadx::Error::PartialReconnect { .. } + | thetadatadx::Error::PartialShardFetch { .. } => "StreamError", _ => "ThetaDataError", } } From cb8e9464d2b4d07390c1f0d6b5680f578298b708 Mon Sep 17 00:00:00 2001 From: TalesOfThales <101120927+userFRM@users.noreply.github.com> Date: Mon, 20 Jul 2026 13:39:19 +0200 Subject: [PATCH 3/6] feat(mdds): shard whole-market at_time along the date axis Whole-market as-of queries (stock/index at_time) return one row per day and are dominated by per-day server compute, not transfer. Banding the requested date range runs those per-day lookups concurrently over the tier's channel pool, so a wide as-of pull drops from a single stream to a fraction of its wall time (a one-year stock pull measured ~188s to ~18s at Pro width) with byte-identical, date-ordered output. Add stock_at_time_trade, stock_at_time_quote and index_at_time_price to the shardable set, reusing the existing date-axis planner and ordered merge with no new axis. The set stays machine-tied to the registry: the tie now derives intraday history (history* with start_time/end_time) plus whole-market at_time (at_time subcategory, category not option). Single-contract option at_time carries a contract identity and is sparse over a wide range, so it stays single-stream. The server computes at_time per day independently, with no last-tick carry across days (a pre-open time_of_day yields NotFound, not the prior close), so a date band seam cannot diverge from the single stream; verified against single-stream output value-for-value. Multi-day at_time .stream() now fans out, so its chunks arrive interleaved across bands rather than strictly date-ascending, matching every other sharded history stream. Tests pin the plan (whole-market at_time to N date bands; option at_time and daily-only families decline) and the uncatchable residual (a concrete-contract tick over-shard is invisible to request shape). --- CHANGELOG.md | 2 + thetadatadx-rs/src/mdds/shard.rs | 167 ++++++++++++++++++++++++++----- 2 files changed, 143 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 624321a8..552cfea9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Shard-decision logging.** The bulk-fetch planner now logs at `debug` why a pull did NOT shard (endpoint outside the shardable set, no cut axis in the request shape, fan-out width under two, an unparsable band window, a provably small bar grid, or a window too narrow for two bands) next to the existing "sharded" line, and every band's work is tagged with its window: retry warnings carry the band span, and each band emits a completion line with its row count and duration, so concurrent bands stay distinguishable in the log stream. +- **Whole-market `at_time` history now shards along the date axis.** `stock_at_time_trade`, `stock_at_time_quote`, and `index_at_time_price` return one row per day and are dominated by per-day server compute rather than transfer, so a multi-day date range now splits into equal concurrent bands like the other bandable history endpoints — cutting a wide as-of pull to a fraction of its single-stream wall time. Buffered output is byte-identical to the single stream and date-ordered; the server computes each day independently (no last-tick carry across days), so band seams never diverge. A multi-day `.stream()` now fans out, delivering each band's chunks as they arrive and so interleaved across bands in arrival order rather than strictly date-ascending — use the buffered path, or `bulk_fetch = "off"`, when strict date order matters. Single-contract option `at_time` carries a per-contract identity and is sparse over a wide range, so it stays a single stream. + ### Fixed - **The dedicated `*_stream()` builders now share the `.stream(handler)` chunk-delivery path.** The four dedicated streaming builders (`stock_history_trade_stream`, `stock_history_quote_stream`, `option_history_trade_stream`, `option_history_quote_stream`) previously marked their no-replay guard on every parsed chunk — including empty keepalives — and kept draining the stream after a decode failure, while the `.stream(handler)` methods on the same endpoints marked the guard only once rows actually reached the handler and stopped at the first decode failure. Both surfaces now route through the same delivery primitive, so a transient error that arrives after only empty keepalive chunks retries instead of surfacing terminal, and a decode failure ends the drain immediately — identically on both. diff --git a/thetadatadx-rs/src/mdds/shard.rs b/thetadatadx-rs/src/mdds/shard.rs index c663ca77..f45f506f 100644 --- a/thetadatadx-rs/src/mdds/shard.rs +++ b/thetadatadx-rs/src/mdds/shard.rs @@ -410,13 +410,21 @@ pub(crate) fn set_wire_str(field: &mut T, v: &str) { /// single-stream path. /// /// The defining property is machine-checkable against the endpoint -/// registry: an endpoint belongs here exactly when its registry entry -/// has a `history*` subcategory AND carries the intraday -/// `start_time` / `end_time` window filters (the daily-only families — -/// EOD, open interest, greeks-EOD — have no intraday window to band). -/// `shardable_set_matches_registry_intraday_history` in this module's -/// tests enforces that equality, so a new intraday history endpoint -/// added to `endpoint_surface.toml` cannot silently never-shard. +/// registry. An endpoint belongs here when either: +/// - its registry entry has a `history*` subcategory AND carries the +/// intraday `start_time` / `end_time` window filters (bands split the +/// window); or +/// - its registry entry has an `at_time` subcategory over a date range +/// with no per-contract identity — a whole-market as-of series whose +/// per-day server compute parallelizes across date bands (bands split +/// the range). Single-contract option at_time carries a contract +/// identity, is sparse over a wide range, and is excluded. +/// +/// The daily-only families — EOD, open interest, greeks-EOD — have no +/// intraday window to band and return <= 1 row/day, so they are absent. +/// `shardable_set_matches_registry_bandable` in this module's tests +/// enforces the equality, so a new bandable endpoint added to +/// `endpoint_surface.toml` cannot silently never-shard. const SHARDABLE_HISTORY_ENDPOINTS: &[&str] = &[ "option_history_trade", "option_history_ohlc", @@ -438,6 +446,11 @@ const SHARDABLE_HISTORY_ENDPOINTS: &[&str] = &[ "stock_history_quote", "index_history_price", "index_history_ohlc", + // Whole-market as-of over a date range: per-day compute parallelizes + // across date bands (measured ~10x on a one-year stock at_time pull). + "stock_at_time_trade", + "stock_at_time_quote", + "index_at_time_price", ]; /// Whether the generated endpoint method named `endpoint` may shard. @@ -1725,6 +1738,101 @@ mod tests { assert!(plan_query("option_history_quote", &chain, 8).is_ok()); } + #[test] + fn concrete_contract_tick_shards_eight_ways_the_uncatchable_residual() { + // One concrete contract, full session, trade-anchored (no `interval` + // field, so `interval_ms` never bounds a grid): 8 equal time bands + // regardless of how many rows the contract actually traded. Correct + // for a liquid contract — the same shape on a dense NBBO returns + // hundreds of thousands of rows and wins — and the irreducible + // over-shard for an illiquid one that traded a handful of times. The + // realized count is absent from the request shape, so nothing short + // of the removed sizing probe separates them. Pinned so a future gate + // change cannot quietly decline the liquid winner while chasing the + // illiquid loser. + let concrete = ShardQuery { + symbol: Some("SPXW".into()), + expiration: Some("20260717".into()), + strike: Some("6000".into()), + right: Some("call".into()), + date: Some("20260717".into()), + start_time: Some("09:30:00".into()), + end_time: Some("16:00:00".into()), + interval: None, + ..ShardQuery::default() + }; + let plan = plan_query("option_history_trade", &concrete, 8) + .expect("concrete-contract tick pull shards on the time axis"); + assert_eq!(plan.bands.len(), 8); + } + + #[test] + fn daily_only_families_decline_first_class_over_a_wide_range() { + // eod / open_interest return <= 1 row/day/contract and carry no + // intraday window, so they are absent from the allowlist and decline + // via UnlistedEndpoint before an axis is ever selected — even over a + // wide, date-shardable-LOOKING one-year range. The allowlist is + // machine-tied to the registry by + // `shardable_set_matches_registry_bandable`; this pins the + // consequence at the plan level. + let stock_eod = ShardQuery { + symbol: Some("AAPL".into()), + start_date: Some("20250101".into()), + end_date: Some("20251231".into()), + ..ShardQuery::default() + }; + assert_eq!( + plan_query("stock_history_eod", &stock_eod, 8), + Err(ShardDecline::UnlistedEndpoint) + ); + let open_interest = ShardQuery { + symbol: Some("SPXW".into()), + expiration: Some("20260117".into()), + strike: Some("6000".into()), + right: Some("call".into()), + start_date: Some("20250101".into()), + end_date: Some("20251231".into()), + ..ShardQuery::default() + }; + assert_eq!( + plan_query("option_history_open_interest", &open_interest, 8), + Err(ShardDecline::UnlistedEndpoint) + ); + } + + #[test] + fn whole_market_at_time_shards_by_date_but_option_at_time_does_not() { + // Whole-market as-of over a date range: no per-contract identity, + // 1 row/day always present, and the per-day server compute + // parallelizes across date bands (measured ~10x on a one-year stock + // pull). No interval, so the date axis bands the range. + let stock_at_time = ShardQuery { + symbol: Some("AAPL".into()), + start_date: Some("20250101".into()), + end_date: Some("20251231".into()), + ..ShardQuery::default() + }; + let plan = plan_query("stock_at_time_trade", &stock_at_time, 8) + .expect("whole-market at_time shards on the date axis"); + assert_eq!(plan.bands.len(), 8); + // Single-contract option at_time is sparse over a wide range (the + // contract lives only near expiry), so it is excluded and stays on + // the single stream. + let option_at_time = ShardQuery { + symbol: Some("SPXW".into()), + expiration: Some("20260117".into()), + strike: Some("6000".into()), + right: Some("call".into()), + start_date: Some("20250101".into()), + end_date: Some("20251231".into()), + ..ShardQuery::default() + }; + assert_eq!( + plan_query("option_at_time_trade", &option_at_time, 8), + Err(ShardDecline::UnlistedEndpoint) + ); + } + #[test] fn date_axis_bounded_interval_gates_small_grids() { // Date-axis twin of the time-axis grid gate: a stock query at a @@ -2050,30 +2158,34 @@ mod tests { /// Machine tie between [`SHARDABLE_HISTORY_ENDPOINTS`] and the /// endpoint registry generated from `endpoint_surface.toml`: the - /// shardable set must equal, exactly, the registry's intraday - /// history endpoints — subcategory `history*` AND the - /// `start_time` / `end_time` intraday window filters. That pair of - /// properties is what separates bandable tick / bar / greeks - /// history from the bounded-per-day EOD / open-interest / - /// greeks-EOD families (which carry a date range but no intraday - /// window) and from snapshots / lists / at-time queries (not - /// `history*`). A new intraday history endpoint added to the - /// registry fails this test until it is added to the shardable - /// set, so it can never silently never-shard; a stale entry fails - /// it from the other direction. + /// shardable set must equal, exactly, the registry's bandable + /// endpoints. Two families qualify: + /// - intraday `history*` endpoints carrying a `start_time` / + /// `end_time` window — bands split the window; + /// - whole-market `at_time` endpoints (category not `option`, so no + /// per-contract identity) — bands split the date range. + /// + /// Excluded: the bounded-per-day EOD / open-interest / greeks-EOD + /// families (a date range but no intraday window, <= 1 row/day), + /// single-contract option `at_time` (sparse over a wide range), and + /// snapshots / lists. A new bandable endpoint added to the registry + /// fails this test until it is added to the shardable set, so it can + /// never silently never-shard; a stale entry fails it from the other + /// direction. /// /// Gated on `__internal` because the registry table only exists /// under that feature; the CI test job enables it (via /// `__test-helpers`), so the tie is enforced on every CI run. #[cfg(feature = "__internal")] #[test] - fn shardable_set_matches_registry_intraday_history() { + fn shardable_set_matches_registry_bandable() { let mut derived: Vec<&str> = crate::mdds::registry::ENDPOINTS .iter() .filter(|e| { - e.subcategory.starts_with("history") + (e.subcategory.starts_with("history") && e.params.iter().any(|p| p.name == "start_time") - && e.params.iter().any(|p| p.name == "end_time") + && e.params.iter().any(|p| p.name == "end_time")) + || (e.subcategory == "at_time" && e.category != "option") }) .map(|e| e.name) .collect(); @@ -2083,8 +2195,8 @@ mod tests { assert_eq!( listed, derived, "SHARDABLE_HISTORY_ENDPOINTS must equal the registry's \ - intraday history set (subcategory `history*` with \ - start_time/end_time filters)" + bandable set (intraday `history*` with start_time/end_time, \ + plus non-option `at_time` over a date range)" ); } @@ -2095,8 +2207,7 @@ mod tests { assert!(is_shardable_history_endpoint("index_history_price")); // Daily-only history (EOD / open interest / greeks-EOD) is // bounded at one row per day, so it is not shardable and - // stays on the single stream, like snapshots, lists, and - // at-time queries. + // stays on the single stream, like snapshots and lists. assert!(!is_shardable_history_endpoint("option_history_eod")); assert!(!is_shardable_history_endpoint( "option_history_open_interest" @@ -2105,7 +2216,11 @@ mod tests { assert!(!is_shardable_history_endpoint("index_history_eod")); assert!(!is_shardable_history_endpoint("stock_snapshot_quote")); assert!(!is_shardable_history_endpoint("option_list_contracts")); - assert!(!is_shardable_history_endpoint("stock_at_time_trade")); + // Whole-market at_time bands its date range; single-contract + // option at_time is sparse and stays on the single stream. + assert!(is_shardable_history_endpoint("stock_at_time_trade")); + assert!(is_shardable_history_endpoint("index_at_time_price")); + assert!(!is_shardable_history_endpoint("option_at_time_trade")); } // ── ordered merge ── From 61ee296a6c31634c7f88a0b8a4fca17942ca49ea Mon Sep 17 00:00:00 2001 From: TalesOfThales <101120927+userFRM@users.noreply.github.com> Date: Mon, 20 Jul 2026 15:44:24 +0200 Subject: [PATCH 4/6] feat(mdds): shard option chains by call/put, not by time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An option-chain pull (strike wildcard) is a contract cross-product, and its cost is the server enumerating those contracts, not transfer. Slicing the requested window by time made every band re-enumerate the whole chain; splitting by right (call / put) instead makes each band assemble half the contracts. A both-rights chain now fans out into a call half and a put half, and the tier's remaining lanes slice each half by time (at Pro width 8, call x 4 time bands + put x 4), so the fan-out fills the tier's concurrency with work that divides the bottleneck rather than multiplying it. The plan emits all-calls-then-all-puts, each half in time order. The ordered merge groups rows by (expiration, strike, right) and keeps band order within a contract, so it interleaves call before put within each strike and stays time-ascending — the exact single-stream canonical order, independent of the right x time split. Buffered output is therefore unchanged. A single-right chain (a strike wildcard already pinned to call or put) has nothing to divide on this axis and keeps the equal time split; a chain over a window too narrow for a time split still fans out into a plain call/put pair. ShardBand::Time carries an optional right override applied per band through shard_apply_field, so both the buffered and streaming fan-outs pick it up. Tests pin the plan (both-rights chain -> call/put x time bands; single-right chain -> time-only; narrow concrete contract -> decline) and the merge (call/put x time bands reassemble into canonical contract order). --- CHANGELOG.md | 2 + thetadatadx-rs/src/mdds/macros.rs | 16 ++ thetadatadx-rs/src/mdds/shard.rs | 253 +++++++++++++++++++++++++++--- 3 files changed, 247 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 552cfea9..bbf9e6f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Option-chain pulls now shard by `right` (call / put), not by time.** A chain (`strike = "*"`) is a contract cross-product whose cost is the server assembling its contracts, so slicing the requested window by time — the previous behavior — made every band re-enumerate the whole chain. A both-rights chain now splits into a call half and a put half, each band assembling half the contracts, and the tier's remaining lanes slice each half by time (at Pro width 8, call × 4 time bands + put × 4). Rows come back in the same canonical order as before (grouped by expiration, strike, right; call before put; time-ascending within a contract), so buffered output is unchanged — only the fan-out is smarter. A chain already pinned to one right (a strike wildcard with `right = "call"` / `"put"`) has nothing to divide on this axis and keeps the equal time split. + - **Streaming history builders (`*_stream()`) now shard concurrently like the buffered path, and shard sizing cuts the requested time or date range into equal concurrent bands from the request shape.** Under `bulk_fetch = "auto"` (the default), a large chunk-streaming history pull fans out across the same equal bands as its buffered sibling: every band streams as its own concurrent request, and each band's chunks reach the handler as they arrive — every chunk exactly once, with chunks from different bands interleaved in arrival order (use the buffered builder, or `bulk_fetch = "off"`, when the single stream's exact order matters). The split is decided from the request shape alone — a multi-day range cuts into equal date bands, a single day with an intraday window cuts into equal time bands — with no density-probe request and no per-endpoint tuning; provably small pulls still run as a single stream. - **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. diff --git a/thetadatadx-rs/src/mdds/macros.rs b/thetadatadx-rs/src/mdds/macros.rs index b8b68cf6..ed003736 100644 --- a/thetadatadx-rs/src/mdds/macros.rs +++ b/thetadatadx-rs/src/mdds/macros.rs @@ -794,6 +794,22 @@ macro_rules! shard_apply_field { $crate::mdds::shard::set_wire_str(&mut $params.end_date, end_date); } }; + // The option `right` lives inside `contract_spec` (the top-level + // `right` slot is the always-empty legacy field), so a chain band's + // call/put override rewrites `contract_spec.right`, mirroring how + // `shard_read_field!` reads it. Endpoints without a `contract_spec` + // never emit this arm; a Time band carrying no `right` leaves it as + // issued. + ($params:ident, $band:ident, contract_spec) => { + if let $crate::mdds::shard::ShardBand::Time { + right: Some(right), .. + } = $band + { + if let Some(cs) = $params.contract_spec.as_mut() { + $crate::mdds::shard::set_wire_str(&mut cs.right, right); + } + } + }; ($params:ident, $band:ident, $other:ident) => {}; } diff --git a/thetadatadx-rs/src/mdds/shard.rs b/thetadatadx-rs/src/mdds/shard.rs index f45f506f..8087d429 100644 --- a/thetadatadx-rs/src/mdds/shard.rs +++ b/thetadatadx-rs/src/mdds/shard.rs @@ -292,6 +292,12 @@ pub enum ShardBand { start_time: String, /// Inclusive band end (`HH:MM:SS.mmm`). end_time: String, + /// Contract-side override for an option-chain fan-out: `Some("call")` + /// / `Some("put")` restricts the band to half the chain so the + /// server assembles half the contracts per band (the chain's real + /// cost). `None` for every non-chain band, which leaves the + /// query's own `right` untouched. + right: Option, }, } @@ -613,6 +619,15 @@ fn chain_cross_product(q: &ShardQuery) -> bool { || q.right.as_deref() != Some("call") && q.right.as_deref() != Some("put")) } +/// Whether a chain query spans both option rights, so splitting it into a +/// `call` band and a `put` band actually halves the contract set. A chain +/// already pinned to one right (`right = "call"` / `"put"` with a strike +/// wildcard, or a multi-expiration single-right pull) has nothing to split +/// on this axis and takes the time split instead. +fn splittable_by_right(q: &ShardQuery) -> bool { + chain_cross_product(q) && matches!(q.right.as_deref(), None | Some("both")) +} + /// Pick the shard axis for a query shape, if any. /// /// Priority mirrors the balance the axes deliver: a multi-day range cuts @@ -649,12 +664,13 @@ fn select_axis(q: &ShardQuery) -> Option { /// stream's row count exactly on full-day chain pulls. The first band /// starts at the query's own `start_time` and the last ends at its /// `end_time`; band durations differ by at most 1 ms. -fn time_bands(start_ms: i64, end_ms: i64, n: i64) -> Vec { +fn time_bands(start_ms: i64, end_ms: i64, n: i64, right: Option<&str>) -> Vec { let window = end_ms - start_ms + 1; (0..n) .map(|k| ShardBand::Time { start_time: format_hms(start_ms + window * k / n), end_time: format_hms(start_ms + window * (k + 1) / n - 1), + right: right.map(str::to_owned), }) .collect() } @@ -828,6 +844,23 @@ fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Result Result { @@ -884,9 +917,12 @@ impl MarketDataClient { /// one history query, without running the pull. /// /// Manual-mode entry point: apply each returned [`ShardBand`] to a - /// clone of the same builder call (band fields override, everything - /// else stays as issued) and run the sub-requests under your own - /// concurrency. Ignores the configured [`BulkFetchPolicy`], so the + /// clone of the same builder call — its time or date window via the + /// matching setter, and, for a chain band that carries one, its + /// `right` (call / put) via `.right()`; everything else stays as + /// issued — and run the sub-requests under your own concurrency. + /// Applying only the window of a chain band leaves every band pulling + /// both rights, doubling the result. Ignores the configured [`BulkFetchPolicy`], so the /// plan stays available with `bulk_fetch = Off`. `endpoint` is the /// builder method name (for example `"option_history_quote"`). /// @@ -921,7 +957,13 @@ pub(crate) fn band_span(band: &ShardBand) -> tracing::Span { ShardBand::Time { start_time, end_time, - } => tracing::debug_span!("shard_band", band_start = %start_time, band_end = %end_time), + right, + } => tracing::debug_span!( + "shard_band", + band_start = %start_time, + band_end = %end_time, + band_right = ?right, + ), } } @@ -1638,36 +1680,158 @@ mod tests { } } + /// The `right` override a band carries (`None` for a plain time band). + fn band_right(b: &ShardBand) -> Option<&str> { + match b { + ShardBand::Time { right, .. } => right.as_deref(), + ShardBand::Date { .. } => None, + } + } + #[test] - fn plan_cuts_a_chain_day_into_equal_time_bands() { + fn plan_cuts_a_chain_day_into_call_put_time_bands() { + // A both-rights chain's cost is contract assembly, so it splits by + // `right` first — halving the contracts per band — then spends the + // tier's remaining lanes on time bands per half. At width 8 that is + // call x 4 time bands followed by put x 4, all-calls-then-all-puts + // so the ordered merge restores the single-stream chain order. let plan = plan_query("option_history_quote", &chain_day_query(), 8) .expect("full-day chain must shard"); assert_eq!(plan.bands.len(), 8); - let windows: Vec<(i64, i64)> = plan.bands.iter().map(band_window_ms).collect(); - assert_eq!(windows[0].0, parse_ms_of_day("09:30:00").unwrap()); assert_eq!( - windows.last().unwrap().1, - parse_ms_of_day("16:00:00").unwrap() + plan.bands.iter().map(band_right).collect::>(), + vec![ + Some("call"), + Some("call"), + Some("call"), + Some("call"), + Some("put"), + Some("put"), + Some("put"), + Some("put"), + ] ); - for w in windows.windows(2) { - assert_eq!(w[1].0, w[0].1 + 1, "bands must abut at +1 ms"); + // Each half's four time bands quarter the session, abutting at + // +1 ms, first band at the query start and last at its end. + for half in [&plan.bands[0..4], &plan.bands[4..8]] { + let w: Vec<(i64, i64)> = half.iter().map(band_window_ms).collect(); + assert_eq!(w[0].0, parse_ms_of_day("09:30:00").unwrap()); + assert_eq!(w.last().unwrap().1, parse_ms_of_day("16:00:00").unwrap()); + for pair in w.windows(2) { + assert_eq!(pair[1].0, pair[0].1 + 1, "bands abut within a right"); + } } } #[test] - fn narrow_window_reduces_band_count_or_declines() { - // Too narrow for two minimum-width bands: single stream. - let mut query = chain_day_query(); - query.end_time = Some("09:36:00".into()); + fn narrow_window_declines_concrete_but_still_splits_a_chain_by_right() { + // A concrete contract too narrow for two time bands stays on the + // single stream — there is nothing to split. + let mut concrete = chain_day_query(); + concrete.expiration = Some("20260710".into()); + concrete.strike = Some("6000".into()); + concrete.right = Some("call".into()); + concrete.end_time = Some("09:36:00".into()); assert_eq!( - plan_query("option_history_quote", &query, 8), + plan_query("option_history_quote", &concrete, 8), Err(ShardDecline::NarrowWindow) ); - // Wide enough for exactly two: two bands, not the full width. - let mut query = chain_day_query(); - query.end_time = Some("09:41:00".into()); - let plan = plan_query("option_history_quote", &query, 8).expect("two bands fit"); + // A both-rights chain over the same narrow window still has + // contracts to divide, so it fans out into one call band and one + // put band (no time split fits, so one full-window band each). + let mut chain = chain_day_query(); + chain.end_time = Some("09:36:00".into()); + let plan = plan_query("option_history_quote", &chain, 8).expect("chain splits by right"); assert_eq!(plan.bands.len(), 2); + assert_eq!( + plan.bands.iter().map(band_right).collect::>(), + vec![Some("call"), Some("put")] + ); + } + + #[test] + fn single_right_chain_takes_the_time_split_not_the_right_split() { + // A strike wildcard already pinned to calls has nothing to divide + // by right, so it falls back to equal time bands carrying no right + // override. + let mut query = chain_day_query(); + query.right = Some("call".into()); + let plan = plan_query("option_history_quote", &query, 8).expect("shards on time"); + assert_eq!(plan.bands.len(), 8); + assert!(plan.bands.iter().all(|b| band_right(b).is_none())); + } + + #[test] + fn chain_band_override_rewrites_contract_spec_right_on_the_wire() { + // Regression for the seam the plan/merge tests skip: a call/put + // override must actually reach the wire. `right` lives inside + // `contract_spec` (the top-level `right` slot is the empty legacy + // field), so `shard_apply_field!` has to rewrite + // `contract_spec.right`. An arm keyed on a non-existent top-level + // `right` field never expands, leaving the call and put bands + // byte-identical and doubling every row of a both-rights chain. + use crate::proto; + fn query() -> proto::OptionHistoryQuoteRequestQuery { + proto::OptionHistoryQuoteRequestQuery { + contract_spec: Some(proto::ContractSpec { + symbol: "SPXW".into(), + expiration: "20260717".into(), + strike: Some("*".into()), + right: Some("both".into()), + }), + start_time: Some("09:30:00.000".into()), + end_time: Some("16:00:00.000".into()), + ..Default::default() + } + } + + // Bands are applied by reference, exactly as the fan-out does. + let call_band = ShardBand::Time { + start_time: "09:30:00.000".into(), + end_time: "12:44:59.999".into(), + right: Some("call".into()), + }; + let plain_band = ShardBand::Time { + start_time: "12:45:00.000".into(), + end_time: "16:00:00.000".into(), + right: None, + }; + let date_band = ShardBand::Date { + start_date: "20260101".into(), + end_date: "20260131".into(), + }; + + // A call-carrying time band rewrites right -> "call" AND the window. + let mut q = query(); + let band = &call_band; + shard_apply_field!(q, band, contract_spec); + shard_apply_field!(q, band, start_time); + shard_apply_field!(q, band, end_time); + assert_eq!( + q.contract_spec.as_ref().unwrap().right.as_deref(), + Some("call"), + "call band must rewrite contract_spec.right on the wire" + ); + assert_eq!(q.start_time.as_deref(), Some("09:30:00.000")); + assert_eq!(q.end_time.as_deref(), Some("12:44:59.999")); + + // A time band with no right override leaves it as issued. + let mut q2 = query(); + let band = &plain_band; + shard_apply_field!(q2, band, contract_spec); + assert_eq!( + q2.contract_spec.as_ref().unwrap().right.as_deref(), + Some("both") + ); + + // A date band never carries a right and never touches contract_spec. + let mut q3 = query(); + let band = &date_band; + shard_apply_field!(q3, band, contract_spec); + assert_eq!( + q3.contract_spec.as_ref().unwrap().right.as_deref(), + Some("both") + ); } #[test] @@ -1986,6 +2150,7 @@ mod tests { ShardBand::Time { start_time, end_time, + .. } => ( parse_ms_of_day(start_time).unwrap(), parse_ms_of_day(end_time).unwrap(), @@ -1998,7 +2163,7 @@ mod tests { fn time_bands_partition_the_window_exactly() { let start = parse_ms_of_day("09:30:00").unwrap(); let end = parse_ms_of_day("16:00:00").unwrap(); - let bands = time_bands(start, end, 4); + let bands = time_bands(start, end, 4, None); assert_eq!(bands.len(), 4); // First band starts at the query start, last ends at the query // end, and adjacent inclusive bands abut at exactly +1 ms so every @@ -2024,17 +2189,19 @@ mod tests { fn time_bands_format_wire_canonical_times() { let start = parse_ms_of_day("09:30:00").unwrap(); let end = parse_ms_of_day("16:00:00").unwrap(); - let bands = time_bands(start, end, 2); + let bands = time_bands(start, end, 2, None); assert_eq!( bands, vec![ ShardBand::Time { start_time: "09:30:00.000".into(), end_time: "12:44:59.999".into(), + right: None, }, ShardBand::Time { start_time: "12:45:00.000".into(), end_time: "16:00:00.000".into(), + right: None, }, ] ); @@ -2361,6 +2528,44 @@ mod tests { assert_eq!(merged.as_slice(), expected.as_slice()); } + #[test] + fn merge_restores_canonical_order_from_call_put_time_bands() { + // The chain fan-out emits all-calls-then-all-puts, each right + // sliced into time bands. With two strikes and two time bands per + // right, the merge must interleave C before P within each strike + // and keep time order within each contract — the exact + // single-stream canonical order, independent of the right x time + // band split. + let call_t1 = chain_band(vec![ + tick(20260710, 100.0, 'C', 1000), + tick(20260710, 200.0, 'C', 1200), + ]); + let call_t2 = chain_band(vec![ + tick(20260710, 100.0, 'C', 5000), + tick(20260710, 200.0, 'C', 5200), + ]); + let put_t1 = chain_band(vec![ + tick(20260710, 100.0, 'P', 1100), + tick(20260710, 200.0, 'P', 1300), + ]); + let put_t2 = chain_band(vec![ + tick(20260710, 100.0, 'P', 5100), + tick(20260710, 200.0, 'P', 5300), + ]); + let merged = merge_typed_in_order(vec![call_t1, call_t2, put_t1, put_t2]).unwrap(); + let expected = [ + tick(20260710, 100.0, 'C', 1000), + tick(20260710, 100.0, 'C', 5000), + tick(20260710, 100.0, 'P', 1100), + tick(20260710, 100.0, 'P', 5100), + tick(20260710, 200.0, 'C', 1200), + tick(20260710, 200.0, 'C', 5200), + tick(20260710, 200.0, 'P', 1300), + tick(20260710, 200.0, 'P', 5300), + ]; + assert_eq!(merged.as_slice(), expected.as_slice()); + } + #[test] fn merge_orders_expirations_before_strikes() { // Expiration is the leading key column: a lower expiration's From 4aaf2327e68b4237a25bc78514d1685cec1e71e7 Mon Sep 17 00:00:00 2001 From: TalesOfThales <101120927+userFRM@users.noreply.github.com> Date: Tue, 28 Jul 2026 14:13:12 +0200 Subject: [PATCH 5/6] feat(mdds): date-shard `*` option chains for at_time pulls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `*` option chain at_time — every strike as-of a time, each day — is a dense per-day-compute pull that bands across a date range exactly like the whole-market stock and index at_time already do, but it was excluded along with the concrete single-contract case. Admit option_at_time_trade and option_at_time_quote to the shardable set and decline only the concrete contract (sparse, ~1 row/day) at runtime, so a chain shards and a single contract still runs on one stream. Reuses the existing date-band and ChainKey merge that already serve multi-day history chains, so no new machinery is added. Date bands reproduce the single-stream row set, including at a carry-sensitive as-of time: an option's prior-day carry resolves from the calendar day before the pull, not from a band's start date, so band seams never diverge. --- CHANGELOG.md | 2 +- thetadatadx-rs/src/mdds/shard.rs | 66 +++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bbf9e6f4..ab6e3550 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Shard-decision logging.** The bulk-fetch planner now logs at `debug` why a pull did NOT shard (endpoint outside the shardable set, no cut axis in the request shape, fan-out width under two, an unparsable band window, a provably small bar grid, or a window too narrow for two bands) next to the existing "sharded" line, and every band's work is tagged with its window: retry warnings carry the band span, and each band emits a completion line with its row count and duration, so concurrent bands stay distinguishable in the log stream. -- **Whole-market `at_time` history now shards along the date axis.** `stock_at_time_trade`, `stock_at_time_quote`, and `index_at_time_price` return one row per day and are dominated by per-day server compute rather than transfer, so a multi-day date range now splits into equal concurrent bands like the other bandable history endpoints — cutting a wide as-of pull to a fraction of its single-stream wall time. Buffered output is byte-identical to the single stream and date-ordered; the server computes each day independently (no last-tick carry across days), so band seams never diverge. A multi-day `.stream()` now fans out, delivering each band's chunks as they arrive and so interleaved across bands in arrival order rather than strictly date-ascending — use the buffered path, or `bulk_fetch = "off"`, when strict date order matters. Single-contract option `at_time` carries a per-contract identity and is sparse over a wide range, so it stays a single stream. +- **Whole-market `at_time` history now shards along the date axis.** `stock_at_time_trade`, `stock_at_time_quote`, and `index_at_time_price` return one row per day and are dominated by per-day server compute rather than transfer, so a multi-day date range now splits into equal concurrent bands like the other bandable history endpoints — cutting a wide as-of pull to a fraction of its single-stream wall time. Buffered output is byte-identical to the single stream and date-ordered; the server computes each day independently (no last-tick carry across days), so band seams never diverge. A multi-day `.stream()` now fans out, delivering each band's chunks as they arrive and so interleaved across bands in arrival order rather than strictly date-ascending — use the buffered path, or `bulk_fetch = "off"`, when strict date order matters. A `*` option-chain `at_time` (`option_at_time_trade` / `option_at_time_quote`) shards the same way — every strike as-of a time each day is the same dense per-day compute — and reproduces the single-stream row set even at a carry-sensitive as-of time, because an option's prior-day carry resolves from the calendar day before the pull, not from a band's start date, so band seams never diverge. A concrete single-contract option `at_time` is sparse over a wide range and stays a single stream. ### Fixed diff --git a/thetadatadx-rs/src/mdds/shard.rs b/thetadatadx-rs/src/mdds/shard.rs index 8087d429..26784fd8 100644 --- a/thetadatadx-rs/src/mdds/shard.rs +++ b/thetadatadx-rs/src/mdds/shard.rs @@ -452,11 +452,16 @@ const SHARDABLE_HISTORY_ENDPOINTS: &[&str] = &[ "stock_history_quote", "index_history_price", "index_history_ohlc", - // Whole-market as-of over a date range: per-day compute parallelizes - // across date bands (measured ~10x on a one-year stock at_time pull). + // As-of over a date range: per-day compute parallelizes across date + // bands (measured ~10x on a one-year stock at_time pull). Whole-market + // stock / index carry no contract identity; the option families band + // the same way for a `*` chain, while a concrete single-contract option + // at_time is sparse and declined at runtime in `plan_query`. "stock_at_time_trade", "stock_at_time_quote", "index_at_time_price", + "option_at_time_trade", + "option_at_time_quote", ]; /// Whether the generated endpoint method named `endpoint` may shard. @@ -814,6 +819,16 @@ fn plan_query(endpoint: &str, q: &ShardQuery, width: usize) -> Result Date: Tue, 28 Jul 2026 14:26:25 +0200 Subject: [PATCH 6/6] docs: sync docs-site changelog mirror with CHANGELOG --- docs-site/docs/changelog.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index 624321a8..ab6e3550 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Option-chain pulls now shard by `right` (call / put), not by time.** A chain (`strike = "*"`) is a contract cross-product whose cost is the server assembling its contracts, so slicing the requested window by time — the previous behavior — made every band re-enumerate the whole chain. A both-rights chain now splits into a call half and a put half, each band assembling half the contracts, and the tier's remaining lanes slice each half by time (at Pro width 8, call × 4 time bands + put × 4). Rows come back in the same canonical order as before (grouped by expiration, strike, right; call before put; time-ascending within a contract), so buffered output is unchanged — only the fan-out is smarter. A chain already pinned to one right (a strike wildcard with `right = "call"` / `"put"`) has nothing to divide on this axis and keeps the equal time split. + - **Streaming history builders (`*_stream()`) now shard concurrently like the buffered path, and shard sizing cuts the requested time or date range into equal concurrent bands from the request shape.** Under `bulk_fetch = "auto"` (the default), a large chunk-streaming history pull fans out across the same equal bands as its buffered sibling: every band streams as its own concurrent request, and each band's chunks reach the handler as they arrive — every chunk exactly once, with chunks from different bands interleaved in arrival order (use the buffered builder, or `bulk_fetch = "off"`, when the single stream's exact order matters). The split is decided from the request shape alone — a multi-day range cuts into equal date bands, a single day with an intraday window cuts into equal time bands — with no density-probe request and no per-endpoint tuning; provably small pulls still run as a single stream. - **`MarketDataClient::bulk_fetch_plan` is now `fn(&self, endpoint, query) -> Option` (was `async fn -> Result, Error>`).** The plan is pure computation on the request shape, so there is nothing to await and no error to surface; call sites drop the `.await` and the `Result` handling. This is a breaking change to the Rust API. @@ -19,6 +21,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Shard-decision logging.** The bulk-fetch planner now logs at `debug` why a pull did NOT shard (endpoint outside the shardable set, no cut axis in the request shape, fan-out width under two, an unparsable band window, a provably small bar grid, or a window too narrow for two bands) next to the existing "sharded" line, and every band's work is tagged with its window: retry warnings carry the band span, and each band emits a completion line with its row count and duration, so concurrent bands stay distinguishable in the log stream. +- **Whole-market `at_time` history now shards along the date axis.** `stock_at_time_trade`, `stock_at_time_quote`, and `index_at_time_price` return one row per day and are dominated by per-day server compute rather than transfer, so a multi-day date range now splits into equal concurrent bands like the other bandable history endpoints — cutting a wide as-of pull to a fraction of its single-stream wall time. Buffered output is byte-identical to the single stream and date-ordered; the server computes each day independently (no last-tick carry across days), so band seams never diverge. A multi-day `.stream()` now fans out, delivering each band's chunks as they arrive and so interleaved across bands in arrival order rather than strictly date-ascending — use the buffered path, or `bulk_fetch = "off"`, when strict date order matters. A `*` option-chain `at_time` (`option_at_time_trade` / `option_at_time_quote`) shards the same way — every strike as-of a time each day is the same dense per-day compute — and reproduces the single-stream row set even at a carry-sensitive as-of time, because an option's prior-day carry resolves from the calendar day before the pull, not from a band's start date, so band seams never diverge. A concrete single-contract option `at_time` is sparse over a wide range and stays a single stream. + ### Fixed - **The dedicated `*_stream()` builders now share the `.stream(handler)` chunk-delivery path.** The four dedicated streaming builders (`stock_history_trade_stream`, `stock_history_quote_stream`, `option_history_trade_stream`, `option_history_quote_stream`) previously marked their no-replay guard on every parsed chunk — including empty keepalives — and kept draining the stream after a decode failure, while the `.stream(handler)` methods on the same endpoints marked the guard only once rows actually reached the handler and stopped at the first decode failure. Both surfaces now route through the same delivery primitive, so a transient error that arrives after only empty keepalive chunks retries instead of surfacing terminal, and a decode failure ends the drain immediately — identically on both.