fix(qdf): read a columnar frame into a slice whose element has its own codec

[alex60217101990]

A wire written by the reflect encoder could not be decoded into a generated type from sixteen elements up, where the encoder switches to the hybrid columnar container. Neither side was doing anything unusual: the producer is a library user with no generated code, the consumer has only the generated type.

n=16  balanced     root=0xf7 -> qdf: type mismatch on decode
n=16  compression  root=0xbd -> qdf: type mismatch on decode
n=512 balanced     root=0xf7 -> qdf: type mismatch on decode

The cause was not what was recorded

The standing note blamed the generated decoder's baked column split. That is wrong for this case — the refusal happens before any generated code runs.

decodeSlice dispatches a columnar frame only when the slice descriptor carries a plan, and for an element with a codec of its own that plan is nil. Two separate mechanisms produce the nil: a type implementing both Marshaler and Unmarshaler (every generated type) returns early from fillDesc with no fields, and buildColumnarPlan yields nil for a fieldless descriptor; a type implementing only one is caught by an explicit guard. The failing test takes the first path.

The fix

structuralColumnarPlan builds a plan from a synthetic descriptor. descBuild cannot supply one — it consults typeCache first and returns the same fieldless descriptor. The synthetic one is never published and the shared descriptor is never modified.

That restriction is load-bearing, not tidiness: giving the shared descriptor fields would also give it a columnar plan, and that plan feeds the slice encoder. Decode-only would not survive it.

Reading such a frame structurally is faithful rather than a guess. A hand-written codec writes its own format and never emits a columnar frame, so meeting one is evidence the structural encoder wrote it.

Cost

Whether a type can reach the branch is decided once when the decoder closure is built, next to the existing elemDynamic / elemPF / elemHasMap hoists. The plan is built at most once per type, only after such a frame has actually been seen, and lives in the closure rather than in typeDesc — which grows by nothing.

A measurement whose sign depends on the build

Wire is 54,141 B columnar against 246,278 B row-major — 4.5x smaller — either way. Time is not so simple:

build	columnar	row-major
default (scalar)	194.6 µs	134.5 µs	columnar loses by 45%
-tags qdf_simd	118.8 µs	133.5 µs	columnar wins by 11%

Nearly all of the columnar decode is bit-unpacking an alphabet-packed string column — readStringColumnAlpha is 13.4% of a profiled run, bitUnpackU64LEFast 9.6% of it — and internal/bitpack already carries hand-written amd64/arm64 assembly for that, behind a tag that is off by default. Enabling it is -38.96% on the columnar arm (p=0.000, n=10) with row-major unmoved, which is the control: the vector path touches only the unpacking the columnar form uses.

An earlier revision of this description said flatly that the fallback is "7.4% slower". That was the scalar build, quoted without saying so. Both numbers now travel with their build, in the spec and in the benchmark comment.

None of it changes whether to do the work: the alternative to a slower decode here is no decode at all.

Whether qdf_simd should be the default is a separate question and is not decided here — on amd64 it also needs GOEXPERIMENT=simd. Root, bitpack and qdf-bench all pass under the tag, and CI already benchmarks with it.

Verification

Encoding is pinned byte-for-byte: re-encoding after a fallback decode must produce identical bytes, and a generated type must still refuse the columnar container its plain twin takes. That test counts the rows where the plain twin really did produce a columnar frame and fails if none did.

Both new behaviours were verified by breaking them — with the branch disabled all eight rows of the failure matrix return; with the refusal cache removed the allocation assertion fires.

Four modules green, race clean, both phases of the codegen CI job run locally. FuzzDecoder_NeverPanics 9,960,994 execs and FuzzRoundTrip_StringSlice 6,161,337 execs, no crashers. Columnar 38, ColumnIndex 1, Query 25, Select 12, Pushdown 1, Predicate 5, Batch 53, Skip 33, Hybrid 12 tests passing, none failing — counted, so an empty match cannot pass for coverage.

Two gaps were closed with evidence rather than reasoning: a self-referential element type does not exhaust the stack (the fresh buildCtx bypasses descBuild's cycle registration, so this needed testing), and a slice of pointers is never written columnar by reflect, so the Kind()==Struct predicate leaves no gap.

Bench. Decode geomean -0.01%, memory unchanged, n=10 interleaved from a worktree at base. The noise floor was measured rather than assumed: the same binary against itself reports -5.66% (p=0.045) and -3.79% (p=0.005), geomean -3.28%, so this harness fabricates significant-looking differences of three to six percent and every movement on this branch is smaller than that.

Included, found on the way

A flaky test on main, root-caused. TestWideScratchSurvivesLargeColumns failed about one full-suite run in three and blamed the widening scratch. A memory profile of a failing run puts widenI64 at 2.6% of allocation and resetForReuse at all the rest: it was measuring the output-buffer hint, which a pooled encoder pre-allocates from the previous message, so a hint left by a neighbouring test dominated the figure. The tell is counter-intuitive — the verdict inverted with the collector, GOGC=off failing 5/5 and GOGC=1 passing 3/3. It now asserts what putEnc promises directly on the scratch, both arms verified by breaking the matching half. Twelve full runs clean.

Dead code. typeDesc.scopeToken / scopeFields were written for every Marshaler type and read nowhere, left behind when the field-scope binding moved into EncodeQDF. typeDesc 160 -> 144 bytes.

One inefficiency in this branch's own code. A refused element was re-walked on every decode: 17 allocations per decode, now under 8.

Unblocks

Columnar encoding for a slice of a generated type — the +62.6% wire gap at 512 elements — which needed this first and is a separate change with a much wider blast radius.

Left alone, worth knowing

The buffer hint does not shrink for a smaller message while the encoder stays pooled, so one large message can size the next small one until GC drops the encoder. Deliberate, and untouched here.