BUG: length-aware string hashtable keys, fixing embedded-NUL collapse in unique/factorize/groupby

[jbrockmendel]

• closes BUG: DataFrame.drop_duplicates confuses NULL bytes #34551
• closes BUG: read_csv c engine collapses distinct fields sharing a NUL-terminated prefix with dtype=object or dtype='category' #66525
• Tests added and passing
• All code checks passed
• Added an entry in the latest doc/source/whatsnew/v3.1.0.rst file

pandas keys several khash tables on kh_cstr_t, a bare const char *, hashed with __ac_X31_hash_string (which stops at the first NUL) and compared with strcmp. Any value containing an embedded NUL is therefore truncated for both hashing and comparison, so distinct values sharing a prefix up to their first NUL collapse into one.

arr = np.array(["", "\x00"], dtype=object)
pd.unique(arr)        # before: ['']            after: ['', '\x00']
pd.factorize(arr)[0]  # before: [0, 0]          after: [0, 1]

df = pd.DataFrame({"c": pd.Series(np.array(["", "\x00", "", "\x00"], dtype=object))})
df.groupby("c").ngroups   # before: 1           after: 2

Keyed on an explicit {const char *ptr; size_t len} pair instead. The hash is X31 over exactly len bytes -- bit-identical to __ac_X31_hash_string for NUL-free input, so bucket distribution is unchanged -- and equality is a length check plus a byte comparison. PyUnicode_AsUTF8AndSize and _token_len supply the length, so no path pays an extra strlen.

Fixes unique, factorize, Series.unique, Index.unique and groupby on object dtype, plus read_csv's dtype=object interning and dtype="category" conversion (both of which silently boxed two distinct fields to one value).

Note on the issue's snippet

GH-34551 is titled for DataFrame.drop_duplicates, but that method routes through PyObjectHashTable and already returns the right answer -- the issue's own snippet passes on main. The confusion it reports is real, but it lives in unique/factorize/groupby. Worth knowing before comparing this diff against the issue title.

Performance

The byte-by-byte equality is deliberate and load-bearing: memcmp with a runtime length does not inline, and one libc call per token cost -9.2% on a bool column, where a hashset lookup hits on nearly every token. Full 123-case read_csv benchmark against this branch's merge-base: median 1.0081, 55 significant wins, 2 regressions (bool @12 0.974, mixed @12 0.978 -- 12-thread only; the same fixtures are +7.6% and +2.4% serial). Long keys were the obvious risk for a byte loop and did not materialise (long fields (~150B) +3.6%), so no length threshold is needed.

Scope

One of three PRs splitting the embedded-NUL work apart so the piece needing a semantics discussion doesn't block the rest:

• BUG: read_csv c engine truncated fixed-width, converter and column-name values at an embedded NUL #66511 -- fixed-width / converters / column names. Independent of this PR.
• this PR -- the hashtable key type. The na_values/true_values/false_values hashsets change key type here but still derive their length with strlen, so NA handling is unchanged by this PR.
• BUG: read_csv c engine matched na_values only up to an embedded NUL #66513 -- gives those hashsets an explicit length, which changes user-visible NA semantics and dtype inference. Stacked on this PR.

Draft until the set can be reviewed together.