Raw column output (Arrow) for market data, without value conversion

[userFRM]

Add a raw output mode to the market-data client so callers can stream the wire columns as Arrow, unconverted, and do their own value handling: cents instead of dollars, custom precision, or their own types. It is additive with no breaking change, and it reuses the existing endpoints, parameters, and automatic sharding.

Problem

Every market-data method converts values before handing them back. Prices arrive on the wire as an integer plus a scale and get turned into dollar floats; timestamps become datetimes. That is the right default for most callers, but it costs precision and control for anyone building their own storage or pipeline.

Two concrete cases from users backfilling databases:

• They want cents, not dollars. The float conversion is lossy and they would rather keep the integer value and its scale and convert on their own terms.
• They want to mix and match: dollar strikes but cent quotes, or cents everywhere, or their own fixed-point type. Typed rows do not let them choose.

Today there is no way to get the raw response. You take the typed rows or nothing, and for high-volume backfills that also means paying for a conversion you are about to undo.

Proposed API

Add raw output terminals next to the existing ones on every builder. Same endpoint, same parameters, same automatic sharding. Only the output shape changes.

# typed, today
rows = md.option_history_quote(
    symbol="SPXW", expiration="20260717", strike="*", right="both",
    date="20260717", interval="tick",
)
md.option_history_quote_builder(...).stream(handler)      # handler(list[QuoteTick])

# raw, proposed
table = md.option_history_quote_builder(...).arrow()      # buffered  -> pyarrow.Table
md.option_history_quote_builder(...).stream_raw(handler)  # streamed  -> handler(pyarrow.RecordBatch)

The handler receives an Arrow RecordBatch with the raw columns off the wire: prices as an integer value column plus a scale column, times as integer milliseconds, strikes as integers, right as a string, and so on. You convert per column, however you want:

def handler(batch):
    price_value = batch.column("price_value")   # keep cents at the wire scale
    price_scale = batch.column("price_scale")
    # or dollars, your precision, your call:  value / 10**scale
    ...

Semantics: .arrow() buffers the whole result into one pyarrow.Table; .stream_raw(handler) delivers RecordBatch chunks as they arrive, interleaved across shards in arrival order, the same contract as the typed .stream(). Null cells map to Arrow nulls. The method names are a proposal and open to a different spelling (for example .raw() and .stream_arrow()).

Why Arrow, and why on the builders

• Arrow is columnar and cross-language. The same raw stream works in Python (pyarrow), TypeScript, and C++ with no custom glue.
• It is additive. Typed methods are untouched; the new terminals sit beside .list() and .stream(), so nothing breaks.
• The endpoint, the parameters, and the automatic fan-out are all reused. Raw is an output format, not a new endpoint and not a second client, so it inherits sharding for free. Raw plus sharded is exactly what bulk backfills want.
• One code path across every endpoint. The raw table is generic (headers plus typed columns), so a single mapping covers all methods rather than one per endpoint.

Column semantics

Raw columns mirror the wire, unconverted:

• price fields: an integer *_value column plus a *_scale column (dollars is value / 10**scale; cents is value at the wire scale)
• time fields: integer milliseconds
• strike: integer
• right, exchange, conditions: as sent

A short table in the docs mapping each raw column to its typed equivalent would cover the conversion people need.

Alternatives considered

• A format="arrow" argument on the typed methods. Rejected: the return type would depend on a runtime string, which is awkward to type in Python, TypeScript, and C++. A separate terminal keeps the return type static and discoverable.
• A separate raw client class. Rejected: it duplicates auth, connection, and config for what is only a different output format, and it would not share the fan-out.
• Returning the compressed payload bytes for the caller to parse. Rejected as the default: too low-level for most, and Arrow already exposes the raw values in a form every binding reads natively. The escape hatch below covers anyone who genuinely wants that.

Optional: a lower-level escape hatch

For callers who want to bypass the typed builders entirely and hit an endpoint by name with a parameter bag:

c.raw.stream("option_history_quote", {"symbol": "SPXW", ...}, handler)  # raw columns, you build the request

This is the rawest option, but it drops the builder ergonomics and the automatic sharding, so it is lower priority. The builder terminals above cover the common case.

Impact

• Lossless values for anyone who needs cents or a custom fixed-point type.
• Per-column conversion, decided by the caller instead of the SDK.
• Faster bulk backfills: no typed conversion to pay for and then undo, and the raw stream still shards.
• No breaking change. Everything is additive.