PERF: recover read_csv string-column speed after GH-66415

[jbrockmendel]

Follow-up to GH-66415. That PR fixed read_csv's c engine truncating strings at an embedded NUL by replacing strlen with _token_len in _string_pyarrow_utf8 pass 1. _token_len reads word_starts — a second per-token metadata array, streamed alongside the words array that coliter_next_with_idx already reads — which cost up to 12% on string columns.

_token_len_words takes the boundary from words instead. The tokenizer maintains words[i] == stream + word_starts[i] at every mutation site, so the arithmetic is identical; what changes is that the loop touches one metadata array instead of two.

No whatsnew: _string_pyarrow_utf8 arrived in 3.1.0-dev via GH-65283, so neither the regression nor this fix has shipped.

Benchmarks

Full 123-case shared read_csv suite, M3 Pro, branch 3faea86 vs a baseline worktree pinned at 9828540. Adjacent-paired, adaptive 8–40 rounds to a ±0.75% target, 99% CIs, 1% effect floor. A separate worktree pinned at the same commit provides the null control column. Both build artifacts hash-verified unchanged across the run; 0 cases skipped.

Median 1.0010. 18 cases ≥3% faster, 17 more 1–3% faster, no regression ≥3%.

case	speedup	99% CI	n	null
str @6	1.192x	[1.184, 1.200]	30	0.999
str dtype_backend=pyarrow @12	1.169x	[1.160, 1.177]	33	0.999
str 6-char high-card [par]	1.164x	[1.157, 1.171]	30	1.002
str @12	1.164x	[1.156, 1.171]	40	1.002
str low_memory [par]	1.123x	[1.115, 1.131]	35	0.999
str dtype_backend=pyarrow	1.121x	[1.113, 1.129]	29	1.005
high-card strings @12	1.114x	[1.103, 1.125]	40	1.002
str [par]	1.113x	[1.105, 1.121]	29	0.997
quoted @6	1.105x	[1.097, 1.112]	30	1.000
high-card strings @6	1.098x	[1.090, 1.106]	26	0.998
high-card pyarrow [par]	1.085x	[1.077, 1.093]	35	1.003
high-card strings [par]	1.071x	[1.064, 1.077]	40	1.002
quoted [par]	1.064x	[1.056, 1.072]	11	0.996
datetime-as-str	1.054x	[1.047, 1.060]	30	1.000
escaped quotes @6	1.038x	[1.031, 1.045]	12	1.001
parse_dates	1.037x	[1.029, 1.045]	39	1.005
non-ASCII UTF-8 [par]	1.036x	[1.028, 1.044]	40	1.007
str @2	1.034x	[1.027, 1.042]	33	0.998

Two small regressions

Both clear the significance bar, and the null control reproduces neither, so neither is machine noise.

case	speedup	99% CI	n	null
int64, 3 mixed-magnitude cols [par]	0.990x	[0.982, 0.997]	33	0.998
high-card strings @2	0.988x	[0.981, 0.996]	40	0.999

int_mixed_mag is three pure-integer columns, so _string_pyarrow_utf8 never runs for it and the changed code cannot be executing. The plausible mechanism is code layout: this PR adds a function to parsers.pyx and shifts everything after it. A base-vs-base null control cannot detect that, since identical source produces identical layout — so the null clearing this case is weaker evidence than it looks.

high-card strings @2 is a genuine narrow regression: the same fixture wins at 1, 6 and 12 threads and under [par] (1.022x / 1.098x / 1.114x / 1.071x), and loses only at 2. Unexplained.

One further case, medium (~6 MB) @2 at 1.016x, is classified an artifact — the null control returns 1.013x on identical code, so it is that fixture's own instability rather than an effect of the branch.

Variants tried and dropped

Noted so they are not re-proposed. A hybrid helper using strlen under a 32-byte cap plus a one-compare embedded-NUL verify cost 11.6% on plain str and 10% on str @2: token length does not separate the cases it was meant to separate, since str is 4–7 byte tokens and str_nonascii 16–19, both under the cap. A prefetch variant, testing whether strlen won by warming the lines pass 2 copies, came out statistically indistinguishable from this version across all 19 string cases.