fix(bind): honour the arrow.json tag through dictionary encoding; test null-in-dictionary-values - #308
Merged
Merged
Conversation
…t null-in-dictionary-values Two follow-ups from reviewing fornwall/arrow-adbc#34 (dictionary bind support in the C SQLite/PostgreSQL drivers) against this driver's delegate-to-the-plain-path dictionary binding: - `is_json_field` now looks through `Dictionary(_, storage)` when checking for utf8-family storage: the Arrow spec allows an extension array to be dictionary-encoded, so a JSON-tagged `Dictionary(Int32, Utf8)` column previously bound as plain STRING (which Spanner refuses to coerce into a JSON column) and ingest create modes made a STRING(MAX) column instead of JSON. Present cells get the JSON param type via the existing plain-path delegation; `null_dictionary_value` now takes the field so a null cell keeps the type too (the plain path's typed-null rule). - Regression test for a non-null key pointing at a null entry inside the dictionary values array — the second way a dictionary cell can be null. Handled correctly today only as a side effect of the delegation design; that exact case stayed latent for years in the ADBC postgres driver's already-supported string dictionaries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BgsLAmXUPFThybc8nfF4Eh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups from reviewing fornwall/arrow-adbc#34 (dictionary bind support in the C SQLite/PostgreSQL drivers) against this driver. Our
bind.rsalready uses the delegate-to-the-plain-path design that PR retrofits — the key selects the dictionary value, which re-enters the samecell_valuemapping — so its whitelist-drift bugs can't occur here by construction. The comparison surfaced one genuine gap and one untested-but-correct edge case:JSON through dictionary encoding
is_json_fieldrequired the field's own data type to be utf8-family, but the Arrow spec allows an extension array to be dictionary-encoded (the field's storage type is thendictionary<indices, utf8>). So a JSON-taggedDictionary(Int32, Utf8)column — e.g. a pandas categorical of JSON strings — silently drifted from its plain form:STRING, which Spanner refuses to coerce into aJSONcolumn, andSTRING(MAX)column instead ofJSON.is_json_fieldnow looks throughDictionary(_, storage)for the storage check. Present cells then get the explicitJSONparam type for free via the existing plain-path delegation;null_dictionary_valuenow takes the field so a null cell keeps the type too (the plain path's typed-null rule). The tag stays ignored on non-string dictionary values (Dictionary(_, Int64)), matching the plain-storage rule.Regression test: non-null key → null dictionary value
A dictionary cell can be null two ways: a null key, or a valid key selecting a null entry inside the values array. All existing dictionary tests only exercised the first. The second is handled correctly today, but only as a side effect of the delegation design (the delegated scalar binder's
is_nullcheck fires on the values array) — and it is exactly the case that stayed latent for years in the ADBC postgres driver's already-supported string dictionaries (insufficient data left in message, found in fornwall/arrow-adbc#34). The new test locks it in so a future restructuring of the dictionary arm fails the suite instead of reintroducing that bug.No wire-format or option changes;
cargo test,clippy -D warningsandfmt --checkare clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01BgsLAmXUPFThybc8nfF4Eh