ChatDemo iOS: maxResponseTokens clamped to 896 for dynamic-KV pipelined bundles (upstream on-device compiler bug at KV seq >= 2048)

[john-rocky]

What

Examples/ChatDemo clamps maxResponseTokens to 896 on iOS (ChatModel.swift) for pipelined chat models. This is a deliberate workaround, not a product decision — remove it when the upstream fix lands.

Why

The iOS on-device compiler miscompiles the pipelined engine's specialization when the bound KV cache state's seq dim is ≥2048: output is corrupt from token 1 at impossible speed (~10× the bandwidth floor). Capacity ≤1024 is clean (token-exact vs the fp32 oracle); macOS is clean at every size. A full cache evict does not help, and .fixedSize (4096 upfront) rides the broken shape on every turn — so the only safe envelope today is keeping history + prompt + maxTokens ≤ 1024 per turn.

Upstream report with the full bisect matrix: apple/coreai-models#124

Device-bisected 2026-07-24 on nanbeige4.2-3b (iPhone 17 Pro, iOS 27 beta 24A5380h). The bug is shape-triggered, so it affects any dynamically-sized-KV bundle driven past capacity 1024 on iOS — bench-sized runs (g≤256) never see it.

Affected

• ChatDemo iOS with any dynamic-KV pipelined catalog model and a long response budget
• Not affected: macOS (all sizes), fixed-seqDim exports, generation within capacity 1024

Exit criteria

1. Upstream fix confirmed (iOS: on-device compiled pipelined decode bundle produces corrupt output when the KV cache state is bound at seq >= 2048 (<=1024 correct; macOS correct at all sizes) apple/coreai-models#124)
2. PipelinedBench regression trial at g≥1024 passes token-exact on device
3. Remove the clamp in ChatModel.swift and the note in ModelRuntime.swift

[john-rocky]

Mitigation shipped:

• Engine guard: coreai-models 0.2.1-zoo (john-rocky/coreai-models@bd8dcf7) caps a pipelined turn's KV pre-grow at capacity 1024 on iOS for dynamically-sized caches, throwing contextLengthExceeded when the budget is exhausted.
• Kit: dependency bumped + ChatDemo's iOS maxResponseTokens=896 clamp kept as belt-and-suspenders (0becad3).
• Device-verified: a maxTokens=2048 request on nanbeige4.2-3b (iPhone 17 Pro) now generates coherently inside the cap at honest speed (5.25 tok/s, TTFT 5.7 s) — previously corrupt from token 1 at ~65 tok/s.

Exit criteria unchanged: upstream compiler fix (apple/coreai-models#124) → PipelinedBench regression at g≥1024 → remove both guards.