diff --git a/CHANGELOG.md b/CHANGELOG.md index 3158ee5..22e24fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 see `docs/format/06-known-limitations.md` #9). Resolves Sigilweaver/OpenSZRaw#9. +### Documentation + +- Further PDA/chromatogram payload investigation + (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): seven same-day + sessions of clean-room analysis. Confirmed findings: 2 of the 4 + varying `PDA 3D Raw Data/CheckSum` fields are exact stream byte sizes + (correcting an earlier "flat vs. real flag" reading), and the + "split" envelope form's two regions are an exact + 256-wavelength-channel/remainder split (also explaining why "split" + vs. "symmetric" form correlates with wavelength count). Ruled out, in + each case run through actual value decoding, a physical-plausibility + check, and randomized controls rather than reported on walk-rate + alone: a 19-polynomial CRC-16 sweep and several count/derived-size + candidates for the remaining `CheckSum` fields; a fixed-width fp16 + array; a block-floating-point/adaptive-scale hypothesis family; a + region-tail marker-bit signal that passed two randomized controls but + traced to a compensating-error artifact; two more dramatic-looking + signals surfaced by re-running sweeps with corrected per-region + target counts; an anti-mode-collapse cost term added to the joint + temporal+magnitude decoder; from a deliberately manual (not + automated-sweep) byte-reading pass, a "leading byte of a 3-byte + token" hypothesis that traced to the same low-value-diversity metric + artifact on three independent files; and, cross-referencing the + PSI-MS/mzML open spec directly, two MS-Numpress-inspired + nibble-granular varint encodings and literal zlib/DEFLATE framing of + the payload - the one nonzero hit rate found (a nibble scheme on one + file) was disqualified by a shuffled-byte control and cross-file + testing, and DEFLATE's small hit rate proved statistically + indistinguishable from random-byte and shuffled-byte controls. Along + the way, quantified a ~48% false-positive base rate for this + document's zero-leftover acceptance test and fixed a real gap in the + physical-plausibility check itself (mode-dominated/low-diversity + decodes can look deceptively "smooth" under mean relative step + alone). The per-point payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-7 for + full detail. + ## [0.1.0] - 2026-07-18 ### Added diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 1307ce7..f8362ba 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -2,4 +2,5 @@ Thank you to everyone who has contributed to OpenSZRaw. -No external contributions yet - this list will grow as people help out. +- [@Nabejo](https://github.com/Nabejo) - PDA/chromatogram payload + investigation (Sigilweaver/OpenSZRaw#2). diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index b66894f..3c7d328 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -47,7 +47,28 @@ is a scannable summary, not a substitute for the detailed sections below - Real-mode body length is **tightly centered near `3 * npts` bytes** for the symmetric (`MTBLS432`) form - close to, but not exactly, a uniform 3-bytes-per-value scheme. See "width-table retry on bimodal - data". + data". The split (`PXD025121`/`MSV000084197`) form centers lower, near + `1.88 * npts` (range `[1.65, 2.08] * npts`, 100% of one file's + real-mode segments checked) - a distinct, form-specific constant, not + the same `3 * npts` figure. See the 2026-07-20 session. +- **`CheckSum` stream, 2 of 4 varying fields fully identified**: offset + 56 (`u32` LE) is the exact byte size of `PDA 3D Raw Data/3D Raw Data`; + offset 88 (`u32` LE) is the exact byte size of `PDA 3D Raw Data/Max + Plot` - both zero-mismatch across 77 files. This corrects an earlier + reading of offset 58 as a "flat vs. real" flag byte (it's actually + part of the offset-56 size field). Offsets 48 and 80 are confirmed + genuinely content-dependent but remain unidentified after a sweep of + 19 CRC-16 polynomials, Fletcher/Adler/plain-sum checksums, and 5 + candidate byte ranges. See the 2026-07-20 session. +- **The "split" form's two regions are an exact 256-channel/remainder + wavelength split**: region `A` always holds the first 256 wavelength + channels, region `tail` holds the remaining `npts - 256`, confirmed + with zero exceptions across every flat segment in all 4 locally + available split-form files (2 distinct `npts` values). This also + fully explains why "split" vs. "symmetric" envelope form correlates + with wavelength count: every corpus file with `npts <= 256` uses + symmetric, every file with `npts > 256` uses split. See 2026-07-20 + session 3. - Width selection is very likely **driven by actual per-wavelength signal magnitude**, not any of: an external per-run lookup table (no config/instrument stream correlates with modal length - "external- @@ -62,7 +83,45 @@ is a scannable summary, not a substitute for the detailed sections below from-scratch joint-decode DP ("joint temporal+magnitude decoder") - but no working decoder has been built from it yet; the joint-DP's scoring function is provably not selective enough (width agreement - isn't monotonic with solution cost). + isn't monotonic with solution cost). **Caveat added 2026-07-20 session + 5**: the joint-DP finding's effect size was originally based on a + single true-pair-vs-random-pair anecdote (`1.8x` cost ratio); a + 20-pair re-test on region `tail` reproduces the *direction* but at a + much smaller, less certain effect size (`~1.13x` mean ratio, true + pair cheaper in only 12/20 individual pairs) - cite this as a weak, + aggregate-level signal, not a strong or reliable pair-by-pair one. +- **The region-A-isolated per-byte-position entropy analysis (2026-07-20 + session 5) reproduces the earlier whole-stream entropy session's + numbers almost exactly** (marginal/conditional entropy within 0.01 + bits) and finds no periodic marker at either 2- or 3-byte spacing + across region `A`'s full true 256-channel span - confirming the + earlier entropy work was already correctly scoped and the + low-entropy-near-`0x40`-at-residue-1 pattern really is a low-variance + early-wavelength artifact, not a token-boundary marker. +- **Methodological: a physical-plausibility (temporal-smoothness) check + must also test each channel's mode fraction, not just mean + relative step.** A decode that is a frozen, repeated value most of + the time with rare large jumps trivially minimizes mean relative step + while being the opposite of real chromatography; 2026-07-20 session 4 + found this the hard way (a 67.8%-clean, apparently-smooth single-file + `MTBLS432` result turned out to be up to 96% mode-dominated per + channel) and revised the check accordingly. +- **Methodological, extending the mode-fraction lesson: even without + outright mode-domination, low value diversity alone mechanically + produces a good smoothness score, independent of decode correctness.** + 2026-07-20 session 6 found a `0.840` correlation (68 channels, one + file) between a channel's smoothness score and how many distinct + values it takes - genuinely variable (more analytically real) + channels score *worse*, and channels that barely vary score well + regardless of whether the decode is right. A physical-plausibility + check should be read skeptically for low-diversity channels + specifically, not just mode-dominated ones. +- **The flat-to-real transition has no visible byte-level structure at + all when the transition segment's actual bytes are read by eye** + (2026-07-20 session 6, two files checked): no leading or trailing + quiet region, no length-prefix-looking field, zero bytes scattered + not clustered - direct, by-eye confirmation of the already-established + "hard, instantaneous cliff" aggregate-statistics finding. **Ruled out** (see "This session's additional ruled-out hypotheses" and "further ruled-out hypotheses" for full detail): standard unsigned @@ -74,15 +133,56 @@ prefixes; magnitude-threshold split (whole-body and region-A-isolated); leading bitmap/nibble width-selector tables (1/2/4-bit codes, both pooled-file and per-file least-squares); the TTFL MS RLE scheme applied directly; an external per-run gain/width table stored elsewhere in the -file; `Max Plot` as an independently-decodable crib. +file; `Max Plot` as an independently-decodable crib; a fixed-width fp16 +(binary16) array (body length essentially never equals `2 * npts`); the +19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum, and an +Internet-checksum ones'-complement variant) against `CheckSum` offsets +48 and 80, and those two fields as any of several plain counts/derived +sizes (see 2026-07-20 sessions 1 and 2); the block-floating-point/ +adaptive-scale hypothesis family in four concrete forms - explicit +header-as-threshold, classic fixed-width-per-segment, marker-bit escape +with an algebraically-derived baseline width (a real single-file signal +that fails cross-file generalization), and a leading-bitmap popcount +validator (see 2026-07-20 session 2); the region-`tail`-isolated +marker-bit escape rule, despite passing two randomized-control checks +(uniform-random bytes and same-multiset-shuffled bytes), traced via a +physical-plausibility (temporal-smoothness) check to a compensating- +error artifact affecting all but one of 65 channels, not a real +per-channel decode (see 2026-07-20 session 3); the corrected-target- +count magnitude-threshold and continuation-bit sweeps against region +`A` (target 256) and region `tail` (target `npts - 256`) alone - region +`tail`'s sharp-cliff threshold=32 hit reproduces session 3's exact +channel-0/1 artifact under a different rule, and a fresh `MTBLS432` +sweep's single-file 67.8%-clean/"66-of-68-channels-smooth" result +failed cross-file generalization and was shown to be a smoothness- +metric artifact (mode-dominated decoded values, not real drift) rather +than a genuine decode (see 2026-07-20 session 4); a fixed +bit-identical-value penalty term added to the joint temporal+magnitude +DP's cost function, intended to discourage mode-collapsed solutions - +did not improve (and on a 20-pair sample, worsened) the correlation +between solution cost and width agreement, so does not fix the +DP's previously-diagnosed selectivity weakness (see 2026-07-20 session +5); mod-2 and (beyond the already-explained early-channel effect) +mod-3 periodicity in region `A`'s per-byte-position entropy (see +2026-07-20 session 5); the leading-byte-of-a-3-byte-token hypothesis +for `MTBLS432`, checked via exact-`3*npts`-length segments across three +files - the channels that looked smooth by eye in each case turned out +to have near-zero value diversity (as low as 2 distinct values across a +6-segment run), with a `0.49`-`0.84` smoothness-vs-diversity correlation +confirming the pattern is a metric artifact, not a decode (see +2026-07-20 session 6); two MS-Numpress-inspired nibble-granular varint +schemes (a generic per-nibble continuation flag, and the length-prefixed +scheme closer to Numpress's own published `encodeInt`) swept across three +independent files and four region targets - 0% clean everywhere except +one outlier that a shuffled-byte control and cross-file check both +disqualified as search-space noise; and zlib/DEFLATE framing for the +whole payload or body, ruled out via a randomized control after an +initial small hit rate turned out statistically indistinguishable from +both a random-byte and a shuffled-byte control (see 2026-07-20 session 7). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric interpretation) - the core unsolved problem. -- The functional meaning of the "split" form's two declared-length - regions (`A` bytes then `tail` bytes) - confirmed to exist, never - explained; the "region-boundary vocabulary check" found no byte-value - difference between them. - Whether `LSS Raw Data` (the issue's literally-named path) uses the *same* encoding as `PDA 3D Raw Data` at all - untested, since every locally available file has it empty. See "Further avenues" at the end @@ -91,6 +191,20 @@ file; `Max Plot` as an independently-decodable crib. a different/simpler-looking encoding, out of this issue's named scope) - not decoded, not attempted beyond the initial characterization. See "LC Raw Data..." near the end of this document. +- `CheckSum` offsets 48 and 80 (confirmed content-dependent, not + identified as any standard CRC/checksum algorithm nor as a plain + count/derived size - see 2026-07-20 sessions 1 and 2). +- Whether fp16 (binary16) numeric interpretation or spectral-domain + (wavelength-to-wavelength) delta coding is the right *value* + interpretation for a token, once a token-boundary rule is found - + both remain plausible as validators, neither has been testable yet + since no token-boundary rule has been found to validate. See + 2026-07-20 session 1. +- Whether a more elaborate adaptive-scale mechanism than the four forms + ruled out in 2026-07-20 session 2 exists (e.g. a per-segment gain + header using a non-linear transform, or region-local rather than + whole-body marker placement) - untested, but the simplest and most + natural versions of the idea do not fit. ## Segment Header @@ -642,6 +756,22 @@ byte at `CheckSum` offset 58 is `0x09` in all 9 real-signal files and says whether real data exists at all, not anything about its encoding width, so it doesn't help decode the payload itself. +**Correction (2026-07-20 session)**: offset 58 is not a dedicated flag +byte. It is the high 16 bits of a little-endian `u32` field starting at +offset 56, and that `u32` is the exact byte length of the +`PDA 3D Raw Data/3D Raw Data` stream itself (e.g. `0x0009_1A64` = +`596580`, the `12_65` file's exact stream size, decomposing into the +`0x1A64`/`6756` seen at offset 56-57 and the `0x09` seen at offset 58). +The apparent "flat vs. real flag" reading of the lone offset-58 byte was +a coincidence of magnitude: flat-only files' `3D Raw Data` streams +happen to stay small enough that byte 58 (bits 16-23 of the size) reads +`0x03` in this corpus's sample, and real-signal files' larger streams +happen to read `0x09` - not a purpose-built boolean. See the 2026-07-20 +session below for the full re-characterization, including the +analogous offset-88 field (`Max Plot` stream size) and a from-scratch +CRC/checksum sweep against the two still-unidentified offset-48 and +offset-80 fields. + **Verdict: this thread is closed.** Since the method configuration, instrument parameters, and wavelength table are provably identical byte-for-byte across files whose "real mode" encoding width differs, the @@ -1167,6 +1297,1168 @@ unbounded-beam "exact" DP with backtracking, and the true-vs-control comparison script) built directly on `iter_segments` from `pda_varint_bruteforce.py`. +## 2026-07-20 session: full `CheckSum` field identification (2 of 4), a CRC/checksum sweep that rules out the standard algorithms for the other 2, and a feasibility note on the fp16/spectral-delta avenues + +Picked up two of the "further avenues" this document's previous session +left open: fully characterizing the 112-byte `CheckSum` stream (avenue +3), and trying IEEE-754 half-precision floats and spectral-domain +(wavelength-to-wavelength) delta encoding (avenues 1 and 2). Net result: +one genuine structural correction with full corpus verification, a +thorough negative result on the two still-unidentified `CheckSum` +fields, and an analytical finding that the fp16/spectral-delta ideas - +as pure *framing* hypotheses - don't actually open new search space +beyond what earlier sessions' continuation-bit and magnitude-threshold +sweeps already covered exhaustively. **The per-value payload grammar +remains undecoded.** + +- **`CheckSum` offsets 56 and 88 fully identified: exact stream byte + sizes, not checksums.** Read as little-endian `u32` (not `u16`) at + each offset, offset 56 equals the exact byte length of + `PDA 3D Raw Data/3D Raw Data` and offset 88 equals the exact byte + length of `PDA 3D Raw Data/Max Plot`, both with **zero mismatches + across all 77 locally available files with a real `CheckSum` stream + and both sub-streams present** (45 `MTBLS432` files - both the 9 + real-signal and the fully-flat ones, 31 `PXD025121` files, and + `MSV000084197/20190607_NM16.lcd`; checked with `olefile`'s own + `get_size()` against each file's independently-read stream length, no + parsing of the payload itself required). This directly corrects the + external-table-hunt session's reading of offset 58 as a "flat vs. + real" flag byte - see the correction note inserted in that section + above. The two `u32` fields' high words (bytes 50-51 and 82-83) were + also confirmed `0x0000` in every file checked, consistent with them + being plain sizes rather than some other 32-bit quantity that happens + to coincide with a stream length. +- **Offsets 48 and 80 remain unidentified, but are now more precisely + characterized as genuinely data-dependent (not a per-session token or + random field), and a real checksum/CRC sweep has been run against + them and come up empty.** First, a cheap cross-file check + distinguishes "depends on file content" from "random/session-specific": + among the 15 `MTBLS432` files sampled, 5 fully-flat files that all + share the same segment count (`nseg=2674`) - and therefore, per the + cross-file pass's byte-identity finding two sessions ago, byte-identical + `3D Raw Data` stream content - all report the **exact same** offset-48 + value (`30350`) and offset-80 value (`28638`); a sixth flat file with a + different segment count (`nseg=2673`) reports different values + (`12815`/`57449`) for both. This rules out "opaque per-run token" (it + would not repeat exactly across independent acquisitions) and confirms + these fields are a deterministic function of the stream's actual byte + content, exactly what the stream's name (`CheckSum`) implies - but a + wide, explicit sweep against both fields still found no match: + + - **Algorithms tried**: 19 CRC-16 polynomials (including CCITT, + IBM/ANSI, XMODEM, standard reciprocal/reversed variants, and several + less common published polynomials) crossed with `{reflect-in, + reflect-out} x {0x0000, 0xFFFF, 0x1D0F, 0x800D, 0xB2AA, and two + data-derived seeds (stream length, segment count)} x {xorout 0x0000, + 0xFFFF}` - 19 x 2 x 2 x 7 x 2 = 1064 parameter combinations per + byte-range tried, implemented with a from-scratch table-based CRC-16 + (no vendor or third-party crypto library beyond Python's own + `zlib`/`crcmod`, both open-source, general-purpose, format-agnostic + checksum implementations, consistent with `CONTRIBUTING.md`'s + "independent open-source parsers used purely as format checkers" + allowance - `crcmod` was used only to spot-check the from-scratch + implementation's own predefined-name variants agree, not as a source + of Shimadzu-specific knowledge). Also tried: `zlib.crc32` (both + 16-bit halves), `zlib.adler32` (low 16 bits), Fletcher-16, Fletcher-32 + (low 16 bits), and plain 16-bit additive/XOR byte sums. + - **Byte ranges tried**: the full `PDA 3D Raw Data/3D Raw Data` stream + bytes (segment headers and payloads together), payload-only + (24-byte segment headers stripped), body-only (the split/symmetric + envelope's 4/8-byte header-and-footer further stripped), the 24-byte + segment headers alone concatenated, and (for the offset-80 field) + the `PDA 3D Raw Data/Max Plot` stream bytes directly - the natural + candidate given offset 80 sits in the same header-then-size + structural slot, immediately before offset 88's confirmed Max Plot + size, that offset 48 occupies relative to offset 56's confirmed `3D + Raw Data` size. + - **Result: no match, on any combination, against any of the four + files checked simultaneously** (`12_65`, `1_63`, `34_73`, and + `20_66` for the initial pass; `12_65`, `1_63`, `34_73`, and `21_29` + for the widened rerun). A simple direct-sum-of-real-segment-bytes + check (mod 65536) was also tried and did not match either offset. + - This is a real negative result (a specific, bounded, reproducible + search that came up empty), not proof no algorithm exists - Shimadzu + could use a nonstandard polynomial, a different data range than the + five tried, or a non-CRC construction entirely. But it means the + "brute-force small CRC polynomials against known bytes" plan from + the previous session's "further avenues" note does not immediately + pay off with the standard polynomial set, so a future session + revisiting this should either widen the byte-range search further + (e.g. per-segment individual checksums rather than one whole-stream + value) or treat this thread as lower-priority than the per-value + grammar itself. + +- **fp16 (binary16) as a fixed-width array: ruled out immediately by + body length, as expected.** If real-mode bodies were a plain array of + 2-byte half-floats, body length would equal exactly `2 * npts` on + every real segment. Checked directly: in `MSV000084197`'s split-form + stream (`npts=321`, 3502 segments), body length equals `2 * npts` + (642 bytes) in exactly **1 of 3498 real-mode segments** (segment index + 92, in the middle of a smoothly-growing run of neighboring lengths + 620/637/652/**642**/622/618/614 - a coincidental crossing, not a + population), and in `MTBLS432`'s symmetric-form stream (`npts=68`), + `2 * npts` (136 bytes) occurs in **0 of 2662** real-mode segments. + This is the same kind of hard ruling-out the document's existing + "byte-level volatility" and length-histogram findings already + established for a generic fixed-width hypothesis, now specifically + against fp16's 2-byte width. +- **A precise, previously-only-anecdotal characterization: the + split-form's real-mode body length centers near `1.88 * npts`, not + `3 * npts`.** The document's existing "tightly centered near `3 * + npts`" finding was established only for the symmetric (`MTBLS432`) + form; the split form's centering had only been described anecdotally + via one early-buildup example (segments 4-9 of `MSV000084197`, lengths + 530-571 for `npts=321`, i.e. roughly 1.65-1.78x). Measured precisely + across all 3498 real-mode segments of that same stream: mean real + body length is **604.8 bytes = 1.884 * npts**, median **609 bytes = + 1.897 * npts**, and the full observed range is **530 to 668 bytes + (1.651x to 2.081x npts)** - i.e. **100% of this file's real-mode + segments fall in a `[1.6, 2.2] * npts` band**, a tighter and + differently-centered analogue of the symmetric form's `3 * npts` + finding, not the same constant. This is consistent with (not + contradictory to) the existing magnitude-threshold sweep's + best-performing configuration on this same file being a *2-byte* wide + token with a 1-byte cheap case (`threshold=0x1f`/`0x20`), since a mix + of mostly-2-byte with some 1-byte tokens naturally averages below 2.0 + and above 1.0 per value - it does not, by itself, favor fp16 over the + already-tried magnitude-threshold framing. +- **Why neither fp16-with-escape nor spectral-domain delta actually + reopens new search space, analytically** (no new code needed for this + part - a scoping finding, recorded so a future session doesn't spend a + session re-deriving it): both ideas change how a *token's bytes* are + turned into a *number*, not which bytes belong to which token. The + already-exhausted continuation-bit sweep (all 8 bit positions x both + polarities x 0-11 byte header/footer skip) and magnitude-threshold + sweep (all 256 threshold values x 2-4 byte wide-token width) already + covered every possible token-boundary assignment those searches were + capable of expressing - swapping the *numeric interpretation* of a + found 2-byte or 4-byte token from "raw magnitude-threshold scalar" to + "IEEE-754 binary16" or "delta from the previous wavelength's value" + changes nothing about which byte offsets get grouped into which + token, so it cannot by itself turn a boundary search that already + plateaued at ~6% clean segments into a working decode. Both ideas + remain genuinely useful, but only as **validators/tie-breakers** for + candidate token-boundary assignments a future width-selection + breakthrough would produce (does the resulting number look like a + plausible fp16 absorbance value; does the resulting per-wavelength + sequence look smoother when spectral-differenced) - not as + independent framing strategies in their own right. This reframes two + of the previous session's four "further avenues" from "untried + decode strategies" to "untried validators," which is a more accurate + scoping for whoever picks this up next. +- **Not pursued this session, for budget reasons, still open**: avenue + 4 (aligning the flat-to-real transition to elapsed retention time + rather than segment count) was not attempted - there is no existing + per-segment retention-time decode for the PDA stream in this + codebase to cross-reference against (unlike TTFL's RT index), so + establishing one would be its own sub-investigation before the + time-alignment question could even be asked. Avenue 5 (physical + plausibility/peak-shape as a soft validator) was not attempted either, + for the same reason avenue 1's fp16 analysis above landed on: it + requires a candidate token-boundary assignment to validate, and none + of the framing searches to date (including this session's) have + produced one to test it against. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), built on a from-scratch `olefile`-based +`iter_segments`/`body_of` pair mirroring the shape of +`pda_varint_bruteforce.py` from earlier sessions, plus a from-scratch +table-based CRC-16 implementation (`crc_fast.py`-equivalent) and +`crcmod`/`zlib` for cross-checking standard-named variants. + +## 2026-07-20 session 2: block-floating-point/adaptive-scale hypothesis family tested and ruled out; CheckSum offsets 48/80 tested against count-based candidates; a quantified false-positive base rate + +Direct follow-up to session 1 above, prompted by a specific architectural +observation: every hypothesis ruled out so far assumed a single *global* +rule (fixed threshold, fixed escape byte, fixed continuation-bit +position) applied uniformly across an entire file, but the fact that +real-mode body length tracks each segment's own signal magnitude +continuously (not a fixed multiple of `npts`) is exactly the signature +of a *per-segment adaptive* quantizer - block floating point (a shared +per-block scale/exponent with narrower per-value mantissas), as used in +some real instrument firmware, being the leading concrete example. This +session tried four concrete variants of that family against the payload, +plus a fresh pass at the two still-unidentified `CheckSum` fields as +plain counts/derived sizes rather than checksums. **No decode was +found; the per-value payload grammar remains undecoded.** One negative +result is worth flagging as a genuine methodological contribution in its +own right: a directly quantified false-positive base rate for this +document's own "exact npts count, zero leftover" acceptance criterion, +which puts a hard number on a pattern this document has flagged +qualitatively many times before (Max Plot's 38 spuriously-successful +configs, the width-table retry's 60%->49% sample-size collapse, etc.). + +- **(a) Explicit per-segment header as a threshold source: ruled out, + weakly.** Hypothesized the body's first 1-2 bytes are a small + scale/gain field, separate from the first value token, and that a + *global* (same for every segment/file) formula converts that field + into a per-segment magnitude threshold for an otherwise-unchanged + 2-tier walk (byte-under-threshold = 1-byte token, else a wide token). + Tried header lengths 0/1/2 bytes x four threshold-derivation formulas + (`header_byte_0` directly, `header_byte_1`, low byte of a 2-byte + header read as `u16`, `255 - header_byte_0`) x wide-token widths + 2/3/4, against `MTBLS432/..._12_65...lcd`'s 2662 real-mode segments. + Best combination: **21/2662 (0.8%) clean** - weaker than the + document's existing fixed-global-threshold sweep, not better. An + explicit small header directly usable as a raw threshold value is not + the mechanism (though this doesn't rule out a header that needs a + more complex, not-yet-guessed transform). +- **(b) Classic fixed-width-per-block (true block floating point: + every value in one segment shares the same width): ruled out.** + If each segment's body were `H` header bytes plus `npts` values of a + single *per-segment* (but not per-value) width `W`, then `(body_len - + H)` should divide evenly by `npts` for the great majority of real + segments, for some small `H`. Checked `H` from 0 to 5 bytes against + the same 2662-segment sample: the best case (`H=0`) has only + **70/2662 (2.6%)** of segments dividing evenly into an integer + quotient in the plausible 1-6 byte range (all landing on quotient + `3`, consistent with the existing `3 * npts` centering finding + already documented, but far short of "most segments"), and every + nonzero `H` tried does no better. Per-value width genuinely varies + *within* a segment, not just *between* segments - ruling out a pure + shared-exponent/fixed-mantissa-width model. +- **(c) Marker-bit escape with an algebraically-derived baseline width: + a real within-file signal that does not survive cross-file testing.** + The most promising variant tried: for each segment, `base_w = body_len + // npts` and `n_wide = body_len - npts * base_w` are *exactly* + computable in advance from body length alone (no fitting) under a + "every token is `base_w` or `base_w + 1` bytes" model - so the only + free choice is which bit, in which byte of each token's `base_w`-byte + prefix, signals "this token is the wider one." Swept all 8 bit + positions x 2 polarities x {marker in the token's first byte, marker + in its last byte} (32 combinations) against 4 confirmed real-signal + `MTBLS432` files (`..._12_65`, `..._34_73`, `..._26_68`, `..._21_29` + - all `npts=68`, same instrument/method). Individual files show + real-looking signal - `bit=5, polarity=set, marker=first-byte` reaches + **868/2663 (32.6%)** clean on `..._34_73` and **1100/2663 (41.3%)** on + `..._26_68` - with successes spread across a representative range of + body lengths (207-220 bytes out of an overall 202-219 range on + `..._34_73`, not clustered at a degenerate edge case) - but **the same + configuration gets only 1.1% on `..._12_65` and 8.5% on `..._21_29`**, + and no single configuration reaches even 15% on all four files + simultaneously: the best worst-case (minimum-across-all-four-files) + rate across the full 32-combination sweep is **1.2%**. Since all four + files share the same instrument, method, and `npts`, a real shared + encoding rule should generalize across them the way the envelope and + `u32[2]` facts already do - this doesn't, so it's ruled out as the + real rule despite looking compelling on any single file in isolation. +- **(d) Leading bitmap whose popcount should equal `n_wide`: ruled + out.** A cheaper-to-test alternative to (c): instead of an inline + per-token marker bit, hypothesized a leading `ceil(npts/8)`-byte + (9 bytes for `npts=68`) bitmap at the start of the body (after 0-3 + header bytes) whose set-bit count equals the algebraically-required + `n_wide` for a `base_w=2` model. Checked header offsets 0-3 against + both `..._12_65` (2566/2662 segments applicable to the `base_w=2` + assumption) and `..._34_73` (only 28/2663 applicable, since that + file's real-mode segments center higher, nearer `3 * npts`): **zero + matches in every case**. Ruled out. +- **A quantified false-positive base rate for this document's own + "exact count, zero leftover" acceptance criterion** (a methodological + finding, not a decode attempt): to sanity-check how much weight a + single segment's "a threshold that decodes it cleanly" result should + carry, checked - for a random sample of 400 real-mode segments from + `..._12_65` - what fraction admit *at least one* successful + `(wide_width, threshold)` combination out of the 3 x 256 = 768 tried + (the same free 2-tier walk as (a) above, but with no header at all). + **191/400 (47.75%) of segments admit at least one spuriously "clean" + decode**, averaging **18.8 different working threshold values per + successful segment**, and - most tellingly - **body-length statistics + for segments that admit a lucky threshold are statistically + indistinguishable from those that don't** (mean 202.6 bytes for + "admits a threshold" vs. 202.8 bytes for "does not," essentially + identical). This means, for this document's typical `npts` scale + (dozens to low hundreds of tokens per segment), "there exists a + threshold under which this one segment decodes cleanly" is close to a + coin flip with no relationship to the segment's actual content - + putting a concrete number on why this document has repeatedly found + (Max Plot's 38-configuration false-positive set; the width-table + retry's 60%->49% sample-size collapse; this session's own item (c) + above) that single-file or small-sample "clean decode" rates cannot be + trusted without an explicit cross-file or cross-segment generalization + check, and why every genuinely-confirmed fact in this document (the + envelope, `u32[2]`, the two new `CheckSum` size fields) was checked + against every segment of multiple independent files before being + written down as confirmed. +- **`CheckSum` offsets 48 and 80 tested against plain counts and derived + sizes, not just checksum algorithms: no match found.** Before + resuming the checksum-algorithm search from session 1, tested whether + offset 48 (and separately offset 80) is simply a count or a + differently-scoped size rather than a hash, using the two files with + precisely known values (`..._1_63`/`..._20_66`/`..._30_71`/`..._3_64`/ + `..._44_76`, all `nseg=2674`, offset 48 = `30350`; `..._37_74`, + `nseg=2673`, offset 48 = `12815`). First verified the assumption + the candidate-count hypotheses depend on - that every segment always + declares exactly `npts` points (i.e. total declared point count really + is `nseg * npts`) - directly against `u32[2]` across both files' full + segment lists: true in both, zero exceptions. Checked offset 48/80 + against: `nseg` (2674/2673), `npts` (68), `nseg * npts` (181832/181764), + total stream bytes (already confirmed as offset 56, not 48), total + payload bytes with segment headers stripped (192528/192456), total + body bytes with the envelope also stripped (181832/181764, matching + `nseg * npts` exactly since these are fully-flat files - a useful + independent confirmation that flat-mode body length truly is exactly + `npts` with no other overhead, but not a match for offset 48/80 + either), total 24-byte segment-header bytes (64176/64152), and the + `Status` stream's own varying fields (offset 4 confirms `nseg` + exactly, as already suspected, but offset 20 - `6784`/`6144` - turned + out to depend only on `nseg` too, identical between `..._1_63` and two + *different-content* real-signal files sharing the same `nseg`, so it + cannot be the source of offset 48/80's real-vs-real variation either). + **None matched.** Also tried an Internet-style (RFC 1071) 16-bit + ones'-complement running-sum checksum - initially looked promising + (one accidental exact match against `..._1_63`'s `Max Plot` stream + for offset 80), but this did not reproduce against `..._12_65` or + `..._34_73` with the same formula and is discarded as coincidental (a + flat file's checksum has limited entropy to begin with, making a + 1-in-3-file accidental match unsurprising). This strengthens, rather + than replaces, session 1's conclusion: offsets 48 and 80 are + genuinely content-dependent (not simple counts, not opaque + per-session tokens) but their exact algorithm remains unidentified + after both a broad standard-checksum sweep (session 1) and this + session's count/derived-size and Internet-checksum checks. + +**Verdict**: the block-floating-point/adaptive-scale hypothesis family, +in the four concrete forms tried, is ruled out - most instructively via +(c), which demonstrates a real within-file statistical signal that +fails a proper cross-file generalization check, and the quantified +false-positive base-rate finding explains *why* that kind of signal is +expected to appear by chance at this token-count scale. This does not +close the door on every possible adaptive-scale mechanism (a genuine +per-segment gain header using a transform more complex than the ones +tried in (a), or a marker mechanism at a byte offset within `A`/`tail` +subregions rather than uniformly across the concatenated body, remain +untested), but the most natural, simplest versions of the idea do not +fit. `CheckSum` offsets 48/80 remain the least-understood pair of +fields in this document; a future session should treat them as lower +priority than the per-value grammar itself unless a stronger structural +clue emerges (per session 1's recommendation, unchanged by this +session). + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), extending session 1's `common.py` +(`iter_segments`/`body_of`) with new `walk_fixed2tier` (2-tier +magnitude-threshold walker with a skippable header) and `walk_bfp` +(marker-bit block-floating-point walker) helper functions. + +## 2026-07-20 session 3: the "split" form's two regions are a real, exact 256-channel/remainder wavelength split (resolved); a promising marker-bit signal traced to a compensating-error artifact, not a decode + +Direct follow-up to session 2's region-local suggestion. While setting +up a per-region rerun of session 2's marker-bit test, this session first +had to establish how many of a stream's `npts` wavelength channels +actually live in region `A` versus region `tail` - previously confirmed +to exist (session 1's envelope work) but explicitly flagged as +"never explained" (session 1's factsheet). That turned out to have a +clean, exact answer, which is this session's first real result. The +second half of the session used that answer to retry session 2's +marker-bit hypothesis region-by-region, found a statistically genuine +(not-by-chance) signal in region `tail` - but then, applying the +"physical plausibility as validator" idea from this document's own +"further avenues" list for the first time, showed that signal does not +correspond to a correct per-channel decode. **The per-value payload +grammar remains undecoded**, but one previously-open structural +question is now closed, and a methodologically important cautionary +result is recorded precisely. + +- **Resolved: region `A` always holds exactly the first 256 wavelength + channels; region `tail` holds the remaining `npts - 256`.** Checked + directly against every flat/baseline segment (not just the first one) + in all four locally available split-form files: `MSV000084197/ + 20190607_NM16.lcd` (`npts=321`) and `PXD025121/1.lcd`, `/10.lcd`, + `/11.lcd` (`npts=327` each). At baseline, every value is exactly 1 + byte (session 1's already-confirmed flat-mode fact), so a flat + segment's region-`A`/`tail` *byte* lengths are also directly its + region *value counts* - and in **every flat segment checked across + all four files, `len(A) == 256` exactly, with zero exceptions** + (`321 - 256 = 65` values in `tail` for `MSV000084197`; `327 - 256 = + 71` for the three `PXD025121` files). This resolves session 1's + "genuinely open" question about what the two declared-length regions + mean: they are not an arbitrary or data-dependent split, or a + "coarse array + exception list" (already ruled out in session 1 by + the vocabulary check) - they are a fixed partition of the wavelength + axis at channel index 256, extremely plausibly reflecting a + 256-entry (`2^8`, i.e. one-byte-addressable) hardware buffer or + register width in the PDA detector's acquisition electronics. This + also gives a complete, satisfying answer to session 1's other open + question - why "split" vs. "symmetric" envelope form correlates with + wavelength count: **every corpus file with `npts <= 256` uses the + symmetric (single-region) form, and every file with `npts > 256` + uses the split (two-region) form**, consistent with a design where a + wavelength count that fits inside one 256-slot buffer needs no split, + and one that doesn't gets divided into "the first 256" plus "the + overflow." (This corpus has no file with `npts` near exactly 256 to + test the boundary itself, so the precise cutoff semantics - e.g. + whether `npts == 256` exactly would still split - remain unconfirmed, + but the pattern is exact and exceptionless for every `npts` value + actually present locally: `68`, `321`, `327`.) +- **Retrying session 2's marker-bit escape hypothesis per-region (now + that the true per-region value counts are known) finds a real, + well-controlled signal in region `tail` alone.** Using `n_A = 256` + and `n_tail = npts - 256` (exact, not estimated) to compute each + region's own `base_w`/`n_wide` independently, and sweeping the same + 32 marker-bit configurations from session 2 separately against each + region: **region `A` shows no comparable signal** (best config only + 29/3498 = 0.8% clean on `MSV000084197`, no better than session 2's + whole-body attempts). **Region `tail` shows something real**: with + `bit_pos=5, polarity=set-means-wide, marker-in-first-byte`, clean + (exact `n_tail`-token, zero-leftover) walks succeed on **2479/3498 + (70.9%) of `MSV000084197`'s real-mode segments**, and the *same* + configuration, unchanged, succeeds on **2040/6187 (33.0%)**, + **2129/6179 (34.5%)**, and **2384/6179 (38.6%)** of the three + `PXD025121` files respectively - a real cross-file signal, unlike + every marker-bit config tried against the symmetric-form `MTBLS432` + files in session 2. Two independent randomized controls confirm this + is not a base-rate artifact of the search space (the concern + session 2's false-positive-rate finding raised): (1) replacing each + segment's real tail bytes with **uniformly random bytes of the same + length** gives **0/3498 (0.0%)** clean on `MSV000084197` and + 1.4%-5.2% on the three `PXD025121` files (vs. 33-39% real); (2) the + stricter control - **shuffling each segment's own real bytes** (same + exact byte-value multiset, scrambled order, controlling for the + payload's well-documented non-uniform leader-byte distribution, not + just for byte value frequency) - gives **7/3498 (0.2%)** on + `MSV000084197`, vs. 70.9% for the real, unshuffled order. Real byte + *order*, not just byte *content*, matters to this rule succeeding - + genuine evidence of positional structure in region `tail`'s bytes. +- **But decoding actual values under this rule and checking them for + physical plausibility - the "further avenues" list's avenue 5, + attempted here for the first time - shows the walk is not finding + real per-channel boundaries.** Decoded all `n_tail` channel values + (raw little-endian integer per token) for every segment that walked + cleanly under the rule above, then measured each channel's + segment-to-segment smoothness (mean relative step size between + consecutive successfully-decoded segments; lower means smoother, + more chromatogram-like). **Channel index 0 of the tail region is + dramatically smooth (mean relative step 0.024)** - but **every other + channel (indices 1 through 64) ranges from 0.25 to 1.13**, + statistically indistinguishable from a temporally-shuffled-order + control's aggregate score (0.96) and from pure noise. Only one + channel out of 65 looks like real data; the other 64 do not. Traced + this to a concrete mechanism, not just a vague "it's noisy": in every + one of the **3445 segments** where the marker fires "wide" for + channel 0 (base 1 byte extended to 2), **the second byte of that + "2-byte" token is exactly `0x00`, 3445/3445 times, no exceptions**. + This is much more consistent with channel 0's real width being a + *plain 1 byte* (the marker misfiring on a data byte that happens to + have bit 5 set, which is common given the payload's already-documented + `0x20`/`0x3f`-leader-byte concentration) and the walk's "second byte" + actually being **channel 1's own real first byte** - which happens to + be `0x00` essentially every time, i.e. wavelength channel 257 in this + file is very plausibly a genuinely dead/unmonitored edge channel that + reads exactly zero. The walk still reaches the correct *total* token + count and byte length despite this misclassification because the + very next channel's true value is trivially small - a directly + observed, concrete instance of the **compensating-error phenomenon** + session 2's false-positive-rate finding predicted abstractly: passing + the "exact count, zero leftover" test (even one that also beats two + different randomized controls) does not guarantee the token + boundaries themselves are correct, only that *some* combination of + possibly-wrong boundary choices happened to sum to the right total. +- **Follow-up: confirmed unconditionally, cross-file, with no + marker-bit assumption at all.** Dropped the marker-bit framing + entirely and checked directly: across all four split-form files + (`MSV000084197` plus all three `PXD025121` files, **15,043 real-mode + segments total**), region `tail`'s raw byte at position 1 is exactly + `0x00` in **100.00% of segments, zero exceptions in any file**, and + the raw byte at position 0 stays in a narrow, smoothly-varying range + in every file (`87`-`141` across the four files' means of `107.3`- + `120.6`) fully consistent with the already-observed channel-0 + temporal smoothness. This is strong, clean, marker-bit-independent + evidence that region `tail` position 0 is genuinely a plain 1-byte + token and position 1 is a hard-wired/dead channel reading a constant + zero - reproducible across every file checked, not a `MSV000084197` + quirk. Byte position 2 onward, by contrast, is dominated by the same + `0x20`/`0x3f`/`0x3e`-ish leader-byte values already documented + throughout this document for the undecoded payload in general (checked + via a byte-value histogram at position 2: the four most common values, + accounting for the large majority of segments, are `0x3c`-`0x3f` + and `0x20`-`0x21`) - i.e. the genuine open grammar problem picks back + up at channel index 2, unchanged from the rest of this document; only + the first two channels of region `tail` are special-cased edge + channels, not a hint about the general per-value rule. +- **Verdict**: region `tail`'s bytes do carry genuine, non-random, + order-dependent structure that this specific marker-bit rule + partially captures (the real-vs-shuffled-byte comparison is + decisive on that point) - but the rule itself is not the correct + per-channel grammar, as the physical-plausibility check demonstrates + concretely rather than just abstractly. This is a case where a + hypothesis survives a strong statistical control yet still fails a + physical-plausibility check, which is exactly why this document's + "further avenues" list included the physical-plausibility validator + in the first place - and it is the first time that validator has + actually been applied to a candidate decode, rather than remaining a + suggestion with nothing yet to validate. A future session picking + this up should not restart from session 2's whole-body marker-bit + framing; the more promising, narrower thread is region `tail` + specifically, and the concrete clue that channel 257 (tail-region + index 1) reads as a near-constant `0x00` across thousands of segments + in this file is a specific, reproducible, low-effort thing to verify + and build on directly (e.g. check whether it holds in the + `PXD025121` files too, and whether channel 0 is genuinely 1 byte wide + in the great majority of segments once decoded without the spurious + marker). + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), extending session 2's `walk_bfp` with a +`split_regions` helper (extracts `A`/`tail` sub-bodies from a "split" +form payload using the already-confirmed `A`/`tail` length prefix/ +suffix fields) and a `decode_bfp` variant that returns actual decoded +values (not just walk success) for the physical-plausibility check. + +## 2026-07-20 session 4: re-running the threshold/continuation-bit sweeps with the corrected per-region target counts finds two more dramatic-looking cliffs, both root-caused to artifacts - plus a methodological correction to the physical-plausibility check itself + +Direct follow-up to session 3, prompted by a specific observation: +session 1's original "per-region parsing instead of whole-body parsing" +sweep (in the "further ruled-out hypotheses" section, well before +session 3 established the 256-channel region boundary) isolated region +`A` but still required it to decode to exactly `npts` tokens - an +internally inconsistent test, since region `A` is now known to hold +only the first 256 of `npts` channels, not all of them. This session +re-ran the magnitude-threshold and continuation-bit sweeps against both +regions with the corrected target counts (`256` for region `A`, +`npts - 256` for region `tail`), plus a fresh sweep against `MTBLS432` +(no region split to get wrong in the first place, since `68 <= 256`). +**No working decode was found.** Two sweeps produced dramatic, +cliff-shaped signals that initially looked like real hits - both were +run all the way through to actual value decoding and a +physical-plausibility check (per this document's own established +practice) and both turned out to be artifacts, for two different +reasons. The second of those reasons is a genuine methodological +correction to the physical-plausibility check itself, worth any future +session internalizing before trusting a "smooth" result. + +- **Region `A` (target=256): no signal, at either sweep.** Magnitude + threshold: best `239/3498 (6.8%)` at `threshold=31, wide_w=2` - + barely above the levels already documented as noise elsewhere in this + document. Continuation-bit: best `76/3498 (2.2%)`. Physical + plausibility of the threshold sweep's best candidate: only `2/256` + channels smooth by the (subsequently-revised, see below) smoothness + metric - consistent with edge-channel artifacts, not a real decode. + Region `A`, even at its correct 256-channel target, remains as + undecoded as the rest of this document's whole-body attempts. +- **Region `tail` (target = `npts - 256`): a dramatic, razor-sharp cliff + - traced to the same channel-0/1 artifact session 3 already + diagnosed, not new structure.** The magnitude-threshold sweep found + something that initially looked like a breakthrough: `threshold=32, + wide_w=2` decodes **2518/3498 (72.0%)** of `MSV000084197`'s region + `tail` cleanly, and the *exact same* configuration reaches **33.4%, + 35.3%, and 40.2%** on the three `PXD025121` files - a real cross-file + signal, confirmed against both a uniform-random-byte control + (0.1%-9.4% across the four files) and a same-multiset-shuffle control + (4.7%-14.8%), both decisively beaten. Critically, the sweep shows a + **razor-sharp cliff, not a gradual peak**: `71.9%` at `threshold=31` + down to essentially `0%` at `threshold=33` for the whole-tail walk (a + drop of two full orders of magnitude over a threshold change of `1`), + which mechanically traces to `0x20` (decimal `32`) being the single + most common byte value in this payload (already documented + extensively elsewhere in this document) - flipping the threshold from + `32` to `33` reclassifies every occurrence of that one very common + byte value from "wide" to "narrow" at once, which is enough on its own + to explain a sharp transition without it necessarily marking a true + semantic boundary. Decoding actual values and checking channel-by- + channel temporal smoothness reproduced **exactly** session 3's + finding: channel 0 alone is smooth (mean relative step `0.024`, all + four files), every other channel is noise-like (`0.6`-`1.1`). Directly + confirmed this is the *same* artifact, not a coincidentally similar + one: `byte>=32` and `bit5-set` (session 3's marker rule) agree on + `82.4%` of individual bytes checked, channel 0 is classified "wide" + in **100%** of segments under this rule too, and its supposed second + byte is `0x00` in 100% of those cases - the identical compensating- + error mechanism session 3 root-caused, now shown to reproduce under a + completely different-looking rule (a plain magnitude threshold, not a + specific bit position), which is itself informative: **the artifact + is robust to how the "wide" decision is framed, because it isn't + really about a decision rule at all - it's a structural fact about + channel 1 always being a hard-zero edge channel, which any rule that + classifies channel 0 as "wide" will trip over the same way.** + Stripping the confirmed 2-byte fixed prefix (channel 0 = 1 byte, + channel 1 = constant `0x00`) and re-sweeping the remaining `n_tail - + 2` channels with the corrected target found only a smooth, gradual + peak (not a cliff) topping out at **976/3498 (27.9%)** at + `threshold=27` - and decoding that candidate showed **0 of 63** + channels smooth, with per-channel mode-fractions of only `0.7%-1.9%` + (i.e. genuinely close to uniformly random, the opposite of + mode-dominated). Channels 2 and beyond in region `tail` remain + undecoded. +- **`MTBLS432` (symmetric form, no region-boundary confound): a + single-file result that looked like a genuine decode under this + document's existing smoothness metric, and was not - which surfaced a + real flaw in the metric itself.** A fresh magnitude-threshold sweep + against `..._12_65...lcd`'s 2662 real-mode segments found + `threshold=1, wide_w=3` (i.e. "a literal `0x00` byte is a 1-byte + zero, anything else starts a 3-byte token") decodes **1805/2662 + (67.8%)** cleanly - and, alarmingly, the existing mean-relative-step + smoothness check reported **66 of 68 channels "smooth"** (values + `0.03`-`0.36`, only two channels above the `0.3` cutoff this document + has been using). Before accepting this, cross-file testing (the same + discipline session 2/3 already established) was run first: the exact + same configuration reaches only **14/2663 (0.5%)** clean on + `..._34_73...lcd`, **612/2663 (23.0%)** on `..._26_68...lcd`, and + **363/2662 (13.6%)** on `..._21_29...lcd` - all four files sharing + the same instrument, method, and `npts=68` - an immediate + disqualifying sign on its own. But the more important finding came + from asking *why* `..._12_65...lcd` alone looked so clean: **the + mean-relative-step metric is fooled by highly repetitive (mode- + dominated) decoded sequences.** Checking the value distribution per + channel (not just the step-to-step differences) shows every one of + the 68 channels has a single dominant repeated value accounting for + **41%-96% of its 1805 decoded segments** (most channels 80%-95%), with + the *non-repeated* values landing all over an essentially uniform + 24-bit range (min/max spanning `~10^4` to `~1.67*10^7`, no + concentration) rather than smoothly drifting from the dominant value. + A signal that is "the same fixed number 90% of the time, then jumps + to an unrelated large random-looking number 10% of the time" trivially + minimizes a *mean* relative-step statistic (nearly all steps are + exactly zero) while being the *opposite* of physically plausible + chromatography, which should show continuous, non-repeating drift as + a peak rises and falls. This is consistent with, and mechanically + explained by, an already-documented fact: `MTBLS432` real-mode body + length is tightly centered near `3 * npts` bytes (the "width-table + retry" session, well before this one), so a rule that treats nearly + every byte as the start of a 3-byte token will often land on the + *correct total length* by roughly chunking the body into thirds, + without the chunk boundaries corresponding to genuine per-channel + values - and apparently tends to re-derive the *same* misaligned chunk + boundary (hence the same wrong "value") on the majority of segments + in this one file specifically, which is plausible given how much of + this file's own body-length distribution is already known to cluster + tightly (`194` bytes in `1767/2674` segments per the factsheet). A + continuation-bit sweep against the same file found nothing better + (`34/2662`, `1.3%`). **This route is closed for `MTBLS432` as tried.** +- **Methodological correction, worth recording precisely so it doesn't + recur**: this document's physical-plausibility check (introduced in + session 3) computed only the mean relative step between consecutive + decoded values per channel. This session shows that statistic alone + is insufficient - it can be minimized by a decode that is mostly a + frozen, repeated (mode-dominated) value with rare large excursions, + which is not real physical smoothness. **Any future + physical-plausibility check should also report each channel's mode + fraction** (the proportion of decoded values equal to that channel's + single most common value) **and treat a high mode fraction (this + session saw up to 96%) as disqualifying, not confirming** - genuine + chromatographic data should show low repetition and continuous drift, + not a dominant constant. Both of this session's sweeps were re-checked + against this corrected two-part test (low mean relative step *and* low + mode fraction); neither survived it. + +**Verdict**: the corrected per-region target counts did not produce a +working decode, and the two most promising-looking candidates this +session found were both run through to real value decoding and a +(now-corrected) physical-plausibility check rather than being reported +as hits on the strength of a clean walk rate or a sharp threshold cliff +alone - both failed once actually inspected, for concretely identified, +different reasons. The per-value payload grammar for both the split and +symmetric envelope forms remains undecoded. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), building `region_sweep.py` (generalized magnitude- +threshold and continuation-bit walkers/sweepers parameterized by target +token count and optional header skip, reusable against any region) and +`decode_threshold.py` on top of session 3's `regions.py` helper. + +## 2026-07-20 session 5: region-A-isolated entropy re-check (reproduces, doesn't extend, the earlier finding); a corrected-target joint DP with an anti-mode-collapse penalty term, tested more rigorously than the original single-pair anecdote + +Two follow-ups, both aimed at leads that fall directly out of the +256-channel region boundary (session 3) and the mode-fraction lesson +(session 4). **Neither produced a working decode.** Both are honest, +fully-executed negative results with genuine new information in them +(a reproduced-not-contradicted entropy characterization, and a more +rigorous re-test of a claim this document had previously only supported +with a single-pair anecdote), not abandoned attempts. + +### Region-A-isolated entropy analysis: reproduces the earlier whole-stream numbers almost exactly, confirms no periodic marker + +The 2026-07-19 per-byte-position entropy session ran before the +256-channel region boundary was known, but - important scoping fact +worth stating precisely - that session's "region A" and "region tail" +labels were *already* byte-accurate: the `A`/`tail` byte ranges come +from the envelope's own length-prefix fields (confirmed since session +1), which session 3 never touched. What session 3 corrected was the +*target token count* used by the threshold/varint decode sweeps, not +which bytes belong to region `A`. So re-running the entropy analysis +with the "correct" region boundary was a real thing to check (the +earlier session's byte extraction could in principle still have been +subtly wrong), but was not guaranteed to change anything - and it +didn't: + +- **Marginal and conditional entropy reproduce the 2026-07-19 figures + to three decimal places.** Region `A`: `H(byte) = 6.281` bits, + `H(next|current) = 5.409` bits (13.9% reduction) - versus the + earlier session's `6.279` / `5.408` (13.8%). Region `tail`: `H(byte) + = 5.793`, `H(next|current) = 4.696` (18.9% reduction) - versus the + earlier session's `5.783` / `4.697` (18.8%). Region `A`'s + individual-vs-concatenated compression ratio (`93.3%` vs `77.9%`) + also matches the earlier session's numbers closely. This is a + genuine confirmation, not a redundant no-op: it directly verifies the + entropy session's region extraction was already correct, so no + periodicity or marker was being smeared out or hidden by a region + misalignment that has now been fixed - the entropy characterization + itself needed no correction. +- **New check: region `tail` conditional entropy with the two known + edge channels (position 0 and 1, identified in session 3) excluded**, + to see whether they were inflating the "genuine structure" finding. + Channels 2 onward: `H(byte) = 5.753`, `H(next|current) = 4.726` + (17.9% reduction) - only marginally lower than the full-region figure + (18.9%). The edge channels contribute a small amount to the + conditional-entropy reduction (unsurprising, since channel 1 being a + hard-wired constant given channel 0 is trivially predictable) but + most of the "genuine local structure, not noise" finding survives + their removal - real structure exists in the still-undecoded + channels 2+ too, consistent with session 4's finding that those + channels are not simply noise-passing-as-signal, just not yet + decoded. +- **No new periodicity found at region `A`'s true 256-channel scale.** + Checked mod-3 (the residue class the original entropy session flagged + and then explained away as a low-variance-early-channel artifact, not + a token marker) across the *entire* region `A` span (not just the + first 40 bytes): the low-entropy-at-residue-1 pattern is strong only + for roughly the first 60 bytes (entropy `2.1`-`3.7` bits, one byte + value `0x40` accounting for up to 75% of occurrences) and fades to a + flat `~5.3`-`5.7` bits at all three residues by position `~120` and + beyond - reproducing, not contradicting, the earlier session's + "low-variance early wavelengths, not a 3-byte-token marker" + conclusion, now checked against the true 420-536-byte region `A` span + rather than an approximate one. Also checked mod-2 (motivated by + region `A`'s own average width, `1.89`-`1.91` bytes/channel, being + closer to 2 than 3): no distinction between residues at any position + range checked (`[0,40)`, `[40,120)`, `[120,250)`, `[250,420)`) - both + residues track each other closely everywhere, ruling out a period-2 + marker just as cleanly as period-3 was already ruled out. + +**Verdict**: this was a legitimate check to run (the region boundary +*could* have changed the entropy picture), but it didn't - the earlier +session's entropy characterization holds up exactly, and no +periodicity at either of the two most plausible token widths (2 or 3 +bytes, matching region `A`'s and region `tail`'s own observed average +widths) was found anywhere in region `A`'s true span. If a marker byte +exists, it is not visible as a fixed-position, fixed-periodicity +entropy dip - consistent with (not a new contradiction of) the +transition-segment-comparison session's earlier "no marker byte found" +conclusion for a different part of this problem. + +### Joint temporal+magnitude DP, re-scoped to region-correct target counts, with an anti-mode-collapse cost term + +The 2026-07-19 joint-DP session built and ran its dynamic program +before the 256-channel boundary was known, walking the *combined* +`A`+`tail` body as one `npts`-token problem. Rebuilt the DP from +scratch this session (the original implementation was not saved, +consistent with that session's own "ad hoc, not saved" note) with two +corrections: region `A` and region `tail` scored as independent +sub-problems with their own correct target counts (`256` and `npts - +256` respectively), and an optional cost term that adds a fixed penalty +whenever the two segments' decoded values for a channel come out +*bit-identical*, directly targeting the mode-collapse/frozen-channel +failure mode session 4 diagnosed (a real decode should show continuous +drift, not exact repeats). + +- **Region `tail` (small enough to run at scale - `nT=65`, ~2-3 + seconds/pair): the true-vs-random-pair effect reproduces in + aggregate, but is markedly weaker and less certain than the original + session's single-pair anecdote implied.** The 2026-07-19 session's + entire evidentiary basis for "true neighbors are more self-consistent + than random pairs" was **one** true pair versus **one** random pair + (segment 4-vs-5, cost `2.3789`, versus segment 4-vs-1000, cost + `4.3407`, a `1.8x` difference). This session re-ran the comparison + across **20 true-neighbor pairs and 20 matched random-pair controls** + (same anchor segment, a random partner at least 50 segments away), + with `exact_match_penalty=0` (i.e. the original, unmodified cost + function, applied for the first time to region `tail` alone at its + correct target count). The aggregate direction replicates - mean + cost/channel `0.0113` (true) vs `0.0128` (random), true lower as + expected - but the *effect size* is much smaller than the original + `1.8x` anecdote (`~1.13x` here), and a **per-pair sign test shows + true-neighbor cost was lower in only 12 of 20 pairs (60%)** - not + distinguishable from chance at this sample size. This is a real, + useful correction to how confidently this document should state the + original finding: the *direction* of the temporal-correlation claim + holds up under a more rigorous multi-pair re-test, but the + *magnitude* the original single anecdote suggested was likely + optimistic, and a future session citing this finding should say + "weak, aggregate-level signal, not reliable pair-by-pair" rather than + repeating the original `1.8x` figure as if it were typical. +- **The anti-mode-collapse penalty term did not improve, and on this + sample appeared to worsen, the DP's selectivity - a negative result + for the specific fix tried, not a validation of it.** The original + session's diagnosed weakness was that width agreement does not track + solution cost monotonically. As a proxy for "does the objective + reward genuinely-correct-looking solutions," computed the correlation + between per-pair cost/channel and per-pair width agreement across the + same 20 true-neighbor pairs: with no penalty (`exact_match_penalty = + 0`), correlation is `-0.135` (weakly in the hoped-for direction - + lower cost mildly associated with higher agreement); with the + penalty active (`exact_match_penalty = 1.0`), correlation flips to + `+0.338` (moderately in the *wrong* direction - lower cost associated + with *lower* agreement). This is the opposite of what the penalty + term was designed to achieve. It is a small sample (20 pairs, one + file, one penalty value) and should not be read as definitively + ruling out every possible anti-repeat cost term, but the specific, + concretely-implemented version tried this session - a fixed penalty + for bit-identical decoded values between two segments - does not fix + the selectivity problem and should not be assumed to without + evidence. **This route, as implemented, is closed.** +- **Region `A` (target=256): computationally intractable for the exact + DP as implemented, confirming and sharpening the original session's + own noted limitation.** The original whole-body DP already reported + needing 100-120 seconds per pair at `npts=321` on the combined + `A`+`tail` body. Region `A` alone, at its corrected target of 256 + channels, is worse: a single true-neighbor pair (body lengths 441 and + 443 bytes) did not converge within a 3,000,000-state budget in 50 + seconds of search - state space growth outpaced the feasibility + pruning that kept the original whole-body DP tractable (peaking at + "only" 59,220 states there). This is plausibly because region `A`'s + own average width (`~1.7`-`1.9` bytes/channel, closer to a boundary + between width-1 and width-2 tokens) leaves the width-1-vs-2 choice + ambiguous at many more positions than the combined-body case did, + which is exactly the kind of denser branching that defeats this + pruning strategy. Not pursued further at larger state budgets this + session, given the already-substantial time cost for a single pair + with no sign of convergence; a genuinely faster (vectorized, or + non-Python) implementation - which the original session's own + recommendation already anticipated needing for a longer chain - would + need to come first before region `A` specifically is tractable for + this method at all. + +**Verdict**: both corrections from this session's two leads were +implemented and tested honestly, neither produced a working decode, and +the more rigorous multi-pair re-test of the original DP's headline +claim found the underlying temporal-correlation signal real but weaker +than previously documented - a useful correction for any future session +citing that finding, even though it doesn't change this document's +overall undecoded status. + +**What's left, in the requester's own words, if a future session wants +to keep going here**: (1) a faster (non-Python, e.g. Rust or a +vectorized numpy formulation) joint-DP implementation, to make region +`A`'s larger state space and a longer multi-segment chain (beyond +pairs) actually tractable - both were already recommended by the +2026-07-19 session and remain the single most concrete unblocked next +step; (2) a different, non-fixed-penalty formulation of the +anti-mode-collapse idea (e.g. penalizing a *run* of 3+ consecutive +identical values specifically, rather than any single bit-identical +pair, since real adjacent samples of a slowly-varying signal can +legitimately coincide occasionally) - this session tested only the +simplest version and found it unhelpful, not every version; (3) fully +manual, by-hand byte inspection of a handful of segments (not a +parameterized sweep or DP), which no session including this one has +actually done - every attempt to date has been an automated search over +some hypothesis family, and the corpus of things automated search +finds unpromising is now large enough that manual inspection may turn +up structure a parameterized search wasn't shaped to find. + +Scripts: `regions.py` (session 3, reused), `joint_dp.py` (new this +session - a from-scratch reimplementation of the 2026-07-19 joint DP, +since the original was not saved, extended with the optional +`exact_match_penalty` term and generalized to take an arbitrary target +token count so it works against an isolated region rather than only a +combined `npts`-token body). + +## 2026-07-20 session 6: manual byte-level reading (no sweep, no hypothesis-first search) - a genuine new inspection method, a strong but ultimately artifact-explained "leading byte" lead, and an eye-verified confirmation of the hard-cliff transition + +Every session to date has been hypothesis-first: guess a scheme, sweep +parameters, check zero-leftover, then (since session 3) check physical +plausibility. This session deliberately inverted that order - close, +manual, hypothesis-light reading of actual bytes, reaching for a +parametrized test only once something specific in the raw data +suggested one. **No new testable hypothesis survived scrutiny, and the +per-value payload grammar remains undecoded** - but the manual read +surfaced a genuinely new observational technique (inspecting segments +whose body length exactly equals `3 * npts`, which requires no decode +algorithm at all to align to a token grid) and one quantified, honest +account of *why* a promising-looking signal from that technique +dissolves under scrutiny, distinct from - but related to - session 4's +mode-fraction lesson. + +### Byte-diffing two adjacent real-mode `MTBLS432` segments by hand + +Started with `..._12_65...lcd` (symmetric form, `npts=68`, no +region-A/tail split to worry about), segments 30 and 31 (lengths 201 +and 203, a quiet/plateau part of the run away from both the transition +and any sharp peak). A naive fixed-offset byte-by-byte diff (printed in +8-byte rows with a running differing-byte count) reproduced the +already-documented "80-92% of bytes differ" finding almost exactly - 7 +or 8 of every 8 bytes differ at nearly every row, which is expected +given genuinely variable-per-value width means a fixed byte offset +essentially never lines up with the same channel in both segments once +any earlier channel's width has diverged. + +- **A closer, signed-delta-by-position read (not an aggregate percentage) + found something real: a cluster of small deltas at byte positions + that are consistently multiples of 3, and only when the byte value at + that position falls in the `0x40`-`0x5f` range** - the same + dominant-leader-byte range this document has flagged since its very + first PDP-endian-float observation. Of 68 real channels' worth of + byte positions checked by hand (0 through 202), roughly a dozen + positions showed `|delta| <= 4` while their neighbors showed deltas + in the hundreds - and every one of those small-delta positions landed + on a position `≡ 0 (mod 3)`, consistent with (not a new contradiction + of) this document's already-established `~3 * npts` centering for + this envelope form. Tried decoding 4-byte windows starting at these + positions as `float32` under all four `{LE, BE} x {plain, + PDP-word-swapped}` byte orderings already explored in earlier + sessions: none gave consistently plausible small-magnitude values + across even the pairs of positions checked, so a straightforward + "this byte starts a 4-byte float" reading does not explain the small + deltas on its own. +- **A new, genuinely useful observational technique: segments whose + body length is exactly `3 * npts` need no decode algorithm to align + to a token grid at all.** If a segment's real-mode body is exactly + 204 bytes for `npts=68`, and the true encoding really does put most + values at 3 bytes each, then position `channel * 3` is very likely + each channel's true leading byte, checkable by direct indexing with + no width-inference, no threshold, no walk. `..._12_65...lcd` has 70 + such segments (out of 2662 real-mode segments), including one run of + **4 temporally consecutive** exact-204 segments (indices 133-136). + Reading the leading byte of each channel's 3-byte group across this + run by eye: channels 0-8 and 14 are strikingly smooth across segments + 133, 134, and 135 (e.g. channel 4 reads `93, 93, 93` exactly; channel + 5 reads `64, 64, 64`; channel 0 reads `71, 69, 68`, a plausible small + drift) - but segment 136 shows a **simultaneous jump across every + channel at once** (channel 4 jumps to `225`, channel 5 to `198`, + etc.), which reads as segment 136 not actually being a uniform + 3-bytes-per-value body despite its total length coincidentally + matching `3 * npts` (consistent with this document's own + already-established fact that only a mix of widths, not a strict + per-value 3-byte rule, explains most 3-multiple lengths). +- **Extending this check to all 70 exact-204 segments (not just the one + lucky run) shows the initial "smooth!" impression does not hold up, + and traces to a specific, quantifiable artifact.** Only 15 of the 70 + exact-204 segments are temporally adjacent to another exact-204 + segment (15 total adjacent pairs, mostly isolated pairs plus the one + run of 4) - using all 15 pairs, mean per-channel relative step + (session 3/4's smoothness metric) is `0.516`, and only **15 of 68 + channels** fall under the `0.3` "plausible" threshold used elsewhere + in this document. Checking *which* 15: they are exactly the channels + at the extreme low-index (`0`-`8`) and high-index (`63`-`67`) ends of + the wavelength range - i.e. exactly the "low-variance early + wavelength" channels this document's 2026-07-19 entropy session and + this session's own region-A-isolated re-check (earlier in this same + session set) already identified as inherently low-dynamic-range, not + evidence of a correct decode. Quantified directly: **correlation + between a channel's smoothness score and its number of distinct + leading-byte values across the 70 exact-204 segments is `r = 0.840`** + - channels that are more genuinely variable (higher value diversity, + presumably more analytically active/real) systematically score + *worse* on the smoothness metric, and channels that barely vary at + all trivially score well. This is a distinct but closely related + trap to session 4's mode-fraction lesson: session 4 showed a + literally mode-*dominated* (one repeated value 80-96% of the time) + decode can look smooth; this session shows that even without hitting + that extreme, a channel's *low value diversity alone* - independent + of whether the decode is correct - mechanically produces a good + smoothness score. **A physical-plausibility check should be read + skeptically for any channel with few distinct decoded values, not + just ones with one dominant repeated value.** +- **Cross-checked against the general (non-exact-204-only) walk from + session 4** (`threshold=1, wide_w=3`, which parses 67.8% of this + file's real segments): extracting only the leading byte of each wide + token (rather than the full 3-byte integer session 4 used) does + *not* fix that walk's mode-collapse problem - mean per-channel mode + fraction is still `0.887`, indistinguishable from session 4's + original finding. This confirms session 4's root cause was correctly + diagnosed as being about the walk's *token boundaries* themselves + (which segments outside the lucky exact-204 case do not reliably + find), not about which bytes of a correctly-bounded token get turned + into a value. + +### Hand-inspecting the flat-to-real transition segment + +Read the exact first-real-mode segment body (immediately following the +last all-zero baseline segment) byte-by-byte, by eye, against the +all-zero segment before it, for two files: `..._12_65...lcd` (first +real segment, 200 bytes, following 11 segments of pure `0x00`) and +`..._13_27...lcd` (first real segment, 192 bytes, the file among the 9 +confirmed real-signal `MTBLS432` files with the most zero bytes in its +transition segment, on the theory that more zeros might mean a gentler, +more legible onset). Also checked zero-byte counts in all 8 other +confirmed real-signal files' first-real segments for comparison (range: +1 to 7 zero bytes out of 192-204). + +- **No file shows a "mostly still zero, one or two channels just + starting to show signal" onset.** Every first-real segment checked is + immediately as densely populated with the same `0x40`-`0x5f`/leader- + byte-dominated content as any other real-mode segment deeper into the + run - reading `..._13_27...lcd`'s first-real segment byte-by-byte + (its 7 zero bytes, the most of any file checked, are scattered at + positions `0, 49, 54, 80, 81, 100, 167` - not clustered at the start + or in any other single contiguous run) shows no leading quiet region, + no trailing quiet region, and nothing that reads as a length-prefix + or marker field distinct from the rest of the body. +- This is a genuine, eye-verified **confirmation**, not a new finding + by itself: the 2026-07-19 transition-segment-comparison session + already established via aggregate statistics that the flat-to-real + switch is an instantaneous cliff with "zero nonzero bytes" in the + segments immediately before it and no shared marker at the transition + segment itself. This session read the *actual bytes* of the + transition segment directly (not just its length or a checksum) for + the first time, across two files, and found the same thing a human + reading the raw hex would conclude: there is no visible structural + seam at the transition, only genuinely dense, high-entropy real data + starting immediately in every channel at once. + +**Verdict**: this session's inverted, manual-first approach found one +real, reusable technique (exact-`3*npts`-length segments as an +algorithm-free way to inspect a token grid) and used it honestly enough +to catch its own initially-promising result as an artifact, rather than +reporting the encouraging-looking first four-segment run as a hit. The +transition-segment read adds direct, by-eye confirmation to an +already-well-supported fact rather than contradicting or extending it. +Neither line of inquiry produced a specific byte pattern, marker, or +periodicity that suggests a new parametrized hypothesis worth building +and sweeping. Per this round's own instructions, this is being recorded +as a legitimate close to this manual-inspection lead for now rather than +forced further: the payload's per-value grammar remains undecoded after +six same-day sessions of systematic and, this round, manual +investigation, and this document's own historical pattern (this +session's finding included) is that promising-looking signals at this +scale of investigation keep resolving to already-known artifacts +(edge-channel low variance, value-diversity-driven metric bias, +compensating token-boundary errors) rather than to the real grammar. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), reusing `common.py`'s `iter_segments`/`body_of`; +no new reusable helper modules were warranted for a manual-reading +session. + +## 2026-07-20 session 7: cross-referencing the PSI-MS/mzML open spec - MS-Numpress-style nibble varints and zlib/DEFLATE framing, both ruled out with randomized controls + +Prompted specifically by the suggestion to cross-reference the mzML/ +PSI-MS open spec for an analogous UV/PDA encoding (rather than continuing +to sweep parameters of schemes already tried), this session tested two +concrete, public, non-Shimadzu ideas that come directly from that world: +**MS-Numpress**, the published PSI-MS-affiliated lossy compression scheme +for mzML `` content (Teleman et al., "Numerical +compression schemes for proteomics mass spectrometry data", Mol Cell +Proteomics 2014 - an open, community-standard algorithm with a public +BSD-licensed reference implementation, not vendor material), and whether +the payload is simply a standard **zlib/DEFLATE** stream, which is the +actual real-world mechanism mzML uses to compress its own binary data +arrays. **Neither fits; both are ruled out, with randomized controls for +the one place a nonzero signal briefly appeared.** The per-value payload +grammar remains undecoded. + +- **Why this is a materially different search space from every prior + continuation-bit/threshold sweep, not a repeat of one**: every + varint-style framing this document has tried before (the original + LEB128 attempt, the "8 bit positions x both polarities x 0-11 byte + header/footer skip" continuation-bit sweep, the magnitude-threshold + sweep, the region-local re-runs) operated at **byte** granularity - + a token always starts and ends on a byte boundary. MS-Numpress's + actual integer encoding operates at **nibble** (half-byte) granularity: + each residual is packed into a run of 4-bit nibbles with no requirement + that a token start on a byte boundary, which can natively produce + non-integer average bytes/value (e.g. this document's own observed + ~1.88x and ~3x `npts` centering figures) without needing a threshold or + escape-byte overlay on top. This session implemented two variants of + that idea and swept both against real segments from three independent + files (`MTBLS432/..._12_65...lcd`, `npts=68`, symmetric form, whole-body + target 68; `MSV000084197/20190607_NM16.lcd`, `npts=321`, split form, + region-A target 256 and region-tail target 65; `PXD025121/1.lcd` and + two sibling files, `npts=327`, split form, region-tail target 71), + reusing the already-established envelope/region extraction + (`docs/format/04` sessions 1 and 3) rather than re-deriving it. +- **Variant 1 - generic per-nibble continuation flag** (each 4-bit nibble + carries 3 payload bits plus 1 "more nibbles follow" flag, the nibble- + granular analogue of the already-tried byte-level continuation-bit + sweep): swept nibble order (high-nibble-first vs low-nibble-first) + x continuation-bit position within the nibble (top bit vs bottom bit) + x polarity (flag-set-means-more vs flag-clear-means-more) x value bit + accumulation order (MSB-first vs LSB-first) x header skip (0-4 bytes), + 80 combinations per target. Result: **flat 0% clean** on + `MTBLS432`'s whole-body target (0/2662) and `MSV000084197`'s region-A + target (0/3498); **effectively 0%** on `MSV000084197`'s region-tail + (1/3498, 0.03%). One file showed a real-looking outlier - + `PXD025121/1.lcd`'s region tail reached **166/6187 (2.68%)** at the + best configuration - but this does not survive scrutiny: a + same-multiset **shuffled-byte control** (same bytes, scrambled order) + retains **132/6187 (2.13%)**, i.e. **80% of the apparent signal + survives destroying byte order entirely**, meaning this is almost + entirely a byte-*value*-distribution artifact, not the order-dependent + structure a real token grammar would produce (contrast this document's + genuine positive controls elsewhere, e.g. session 3's region-`tail` + marker-bit finding, where the shuffled control collapsed to 0.2% against + a 70.9% real rate - a qualitatively different, decisive gap this + session's result does not show). The same configuration also fails + cross-file generalization outright, dropping to **62/6179 (1.00%)** on + `PXD025121/10.lcd` and **20/6179 (0.32%)** on `/11.lcd` - both files + sharing the same instrument, method, and `npts` as `/1.lcd`. This is + the same class of small-sample search-space noise this document's own + 2026-07-20 session 2 already quantified (a ~48% false-positive base + rate for a much smaller, single-threshold search); an 80-way sweep over + 6187 segments producing one 2.68%-hitting outlier that halves under a + shuffle control and collapses under cross-file testing is fully + consistent with chance, not a real per-nibble continuation scheme. +- **Variant 2 - length-prefixed nibbles, a closer structural match to + MS-Numpress's actual published `encodeInt`** (one 4-bit "length" + nibble stating how many raw data nibbles immediately follow, rather + than a per-nibble continuation flag - this is the real shape of + Numpress's own integer encoding, which exploits sign-extension to keep + small residuals to very few nibbles): swept header skip (0-4 bytes) x + nibble order x length semantics (`L` vs `L+1` data nibbles) x value bit + order x maximum token length (8 nibbles, matching Numpress's own 32-bit + cap, or 15, the full nibble range), 80 combinations per target, against + the same three files/four targets. Result: **0% clean, on every single + target checked, with zero exceptions** - not even the small, noisy, + non-generalizing signal Variant 1 produced. This is the most decisive + negative result of any per-value framing hypothesis this document has + tried: the existing byte-level continuation-bit sweep's best-ever + result was 88/3502 (2.5%, see the "further ruled-out hypotheses" + section); this nibble-length-prefix scheme does not reach that even + once, across 4 targets x 80 configurations x up to 6187 segments each. +- **zlib/DEFLATE framing** (does the payload, or its body, simply contain + a standard compressed stream - the actual mechanism mzML itself uses + for binary data arrays, as opposed to a Numpress-style custom integer + code): tried `zlib.decompress` at `wbits` 15 (zlib-wrapped), -15 (raw + DEFLATE, no header/checksum), 31 (gzip), and 47 (auto-detect + zlib/gzip) against every real-mode payload and body in + `MTBLS432/..._12_65...lcd` (2662 segments). **21/2662 (0.79%)** + "succeed" (all at `wbits=-15`, raw DEFLATE, on the whole payload - never + on the body alone, and never at any of the header'd variants). This + looked like it might be worth chasing until checked against controls, + exactly per this document's established discipline: a **length-matched + random-byte control** succeeds at **15/2662 (0.56%)**, and a + **same-multiset shuffled-byte control** at **14/2662 (0.53%)** - both + statistically indistinguishable from the real rate. This is expected + once the actual "decompressed" output is inspected: every successful + case yields a **1-byte output**, confirming these are not real + compressed streams but instances of raw DEFLATE's well-known weak + self-framing (no header magic, no checksum - a short byte run can + coincidentally contain a valid Huffman end-of-block marker and "parse," + producing a near-empty result, regardless of whether the input was ever + compressed data at all). **zlib/DEFLATE is ruled out** as the payload's + encoding, whole-payload or body-only, for either envelope form. + +**Verdict**: cross-referencing the PSI-MS/mzML ecosystem's own published +compression schemes was a genuinely new angle (nothing prior in this +document tried nibble-granular tokenization or literal DEFLATE framing), +and it produced two clean, thoroughly-controlled negative results rather +than another ambiguous partial signal - in the nibble-continuation case, +one candidate outlier was found and then properly disqualified by the +same shuffle-control-plus-cross-file-check discipline this document has +used throughout, rather than reported as a lead. This does not mean +mzML/PSI-MS ideas are exhausted as a source of inspiration (Numpress's +*specific* bit-packing scheme is ruled out, not the general idea that a +delta/prediction-based numeric transform might still explain the +*values* once a correct token-boundary rule is found by some other +means - see the existing fp16/spectral-delta "validator, not framing +strategy" scoping note earlier in this document, which applies here too), +but it closes off the two most concrete, literal readings of "does this +look like an mzML-world encoding" that this session could test. + +Scripts (new this session, under `re/src/analysis/`, gitignored per this +repo's `re/` sandbox convention): `common.py` (re-derived `iter_segments`/ +`body_of`/region-split/flat-detection helpers, since no prior session's +scripts were preserved), `nibble_varint.py` (Variant 1: generic per-nibble +continuation-flag decoder and sweep), `nibble_lenprefix.py` (Variant 2: +length-prefixed nibble decoder and sweep), `run_nibble_sweep.py` and +`run_lenprefix_sweep.py` (drivers tying the above to real corpus files). +The zlib/DEFLATE check was ad hoc (`python3 -c`/heredoc, not saved as a +standalone script, since it needed no reusable abstraction beyond the +standard library's own `zlib` module). + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -1198,11 +2490,69 @@ scope (`PDA 3D Raw Data` / `LSS Raw Data`): ## Further avenues for a future session -Beyond the single recommendation in the 2026-07-19 closing summary -(sharpen the joint temporal+magnitude decoder's scoring function), these -are additional, not-yet-tried directions - none executed this round, so -treat them as leads, not findings: - +The 2026-07-19 closing summary's single recommendation (sharpen the +joint temporal+magnitude decoder's scoring function) was attempted in +2026-07-20 session 5, with a negative result for the specific +anti-mode-collapse penalty term tried - see that session for detail and +for three concretely scoped follow-ups it left open (a faster/non-Python +DP implementation, a run-length-based rather than single-pair-based +anti-collapse penalty, and fully manual byte inspection). The third of +those was attempted in session 6 - general manual byte-level reading +(not targeted at region `tail` specifically) of `MTBLS432` real-mode +segments and the flat-to-real transition segment, with no new testable +hypothesis surfacing, though one reusable inspection technique +(exact-`3*npts`-length segments as an algorithm-free token-grid anchor) +came out of it. The region-`tail`-specific manual read described in the +next bullet remains untried. The items below are additional, +not-yet-tried directions - none executed this round, so treat them as +leads, not findings: + +- **(New, from 2026-07-20 session 3; followed up in session 4 with a + negative result) Push the region-`tail` walk past channel index 2.** + Session 3 confirmed, unconditionally and across all four split-form + files (15,043 segments, zero exceptions), that region `tail` position + 0 is a plain 1-byte token in a narrow smoothly-varying range and + position 1 is a hard-wired constant `0x00` - real, reproducible, + edge-channel behavior. Session 4 followed up automatically (stripping + the 2-byte prefix, re-running the magnitude-threshold and + continuation-bit sweeps against the remaining `n_tail - 2` channels + with the correspondingly corrected target count): found only a weak, + gradual (not sharp-cliff) peak topping out at 27.9% clean, and the + decoded values under that candidate are indistinguishable from random + noise (0 of 63 channels physically plausible, per-channel mode + fractions of only 0.7%-1.9%). The automated sweep infrastructure does + not find anything here. What remains genuinely untried is fully + manual, by-hand byte inspection of a handful of individual segments' + channel-2-onward bytes (not a parameterized sweep) - channel index 2 + onward reverts to the same `0x20`/`0x3f`-leader-byte-dominated bytes + as the rest of this document's undecoded payload, so this is really a + narrower, smaller restatement of the core open problem, not a + shortcut around it, but the much smaller channel count (63-69 instead + of 256-327) may still make manual inspection more tractable than it + is against a full segment. +- **(New, from 2026-07-20 session 6) Re-run the exact-`3*npts`-length + byte-diff technique against longer exact-`3*npts` runs: tried this + session, same artifact confirmed on two more files, this specific + avenue is now closed too.** Session 6's manual read of + `..._12_65...lcd`'s exact-204-byte segments found a handful of + channels that looked smooth, but all of them turned out to be the + same already-known low-variance edge channels (indices `0`-`8` and + `63`-`67`), with a `0.840` smoothness-vs-value-diversity correlation + across that file's 4-segment run. To check this wasn't an artifact of + that one short run, this session went on to find and check the two + longest same-length consecutive runs across all 9 confirmed + real-signal `MTBLS432` files: `..._26_68...lcd` (6 consecutive + exact-204-byte segments, indices `2036`-`2041`) and `..._10_26...lcd` + (another 6-long run, indices `1293`-`1298`). Both reproduce the same + pattern - `..._26_68...lcd`'s apparently-smooth *interior* channels + (e.g. channels 9-11, values like `[84, 83, 84, 83, 84, 84]`) turned + out to have only **2 distinct values** across the 6-segment run, and + the smoothness-vs-diversity correlation is `0.718` (`..._26_68`) and + `0.489` (`..._10_26`) - both substantial, both in the same + artifact-confirming direction as the original file. This rules out + "the 4-segment run was just too short to see real interior-channel + structure" as an explanation; longer runs on two independent files + show the identical low-diversity artifact, not new signal. - **Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` stream instead of (or before) `PDA 3D Raw Data`.** It's populated, structurally simpler (one giant segment per channel, not thousands of @@ -1222,46 +2572,65 @@ treat them as leads, not findings: MassIVE, searching for older instrument models or explicitly UV/RID-detector method descriptions) before assuming `PDA 3D Raw Data` is a valid stand-in for the named stream. -- **Test spectral-domain (wavelength-to-wavelength) delta encoding, not - just temporal (segment-to-segment) delta encoding.** Everything tried - so far assumes each segment's `npts` tokens are independent absolute - values, or deltas against the *same channel in the previous segment*. - UV absorbance spectra are also smooth *across wavelength* within a - single timepoint - worth testing whether a token's value is better - modeled as a delta from the *previous wavelength's* decoded value in - the *same* segment (first-differencing within one spectrum), which is - a distinct hypothesis from anything in the "ruled out" list above. -- **Try IEEE-754 half-precision (binary16) floats directly**, not just - the word-swapped "PDP-endian" full float32 interpretation already - explored. Absorbance units typically span a small range (0-3 AU), - which fp16 represents naturally in 2 bytes - worth checking whether a - mixed fp16/fp32 scheme (small values as 2-byte half floats, an escape - to 4-byte full float32 for outliers) fits better than the arbitrary - magnitude-agnostic proxy the joint-decoder pass used, and ties - naturally into the already-observed 2-vs-3-vs-4-byte width mix. -- **Characterize the `CheckSum` stream (112 bytes) more fully.** The - external-table-hunt pass found byte offset 58 cleanly flags real-vs- - flat mode (`0x09` vs `0x03`) but only checked correlation against - modal *length* - the rest of the stream's 112 bytes were never - systematically characterized. If any field there is a genuine - checksum/CRC of the segment payload (plausible, given the stream's - name), brute-forcing small CRC polynomials against known segment - bytes would give a powerful, cheap, objective validator for candidate - decodes - far stronger than "does the byte count come out exact," - which today's sessions repeatedly found admits multiple false - positives. -- **Check whether the flat-to-real transition aligns with a fixed - elapsed *time* rather than a fixed segment *count*.** The transition- - segment pass found the index varies (11 or 12) across files with the - same instrument/method - worth checking the actual retention-time - value at the transition point (cross-referencing whatever RT stream - the reader already decodes elsewhere in this repo, e.g. the TTFL - retention-time index) instead of segment index, in case it's a fixed - elapsed-time firmware constant (e.g. "N seconds of lamp warm-up/ - blanking before real acquisition starts") rather than a coincidence of - segment count. -- **Use physical plausibility (peak shape) as a soft validator, not just - exact byte-count matching.** The joint-decoder pass's core problem was +- **(Revisited 2026-07-20, still open) Test spectral-domain + (wavelength-to-wavelength) delta encoding, not just temporal + (segment-to-segment) delta encoding.** The 2026-07-20 session found + this idea, taken as a pure token-boundary/framing hypothesis, does not + actually expand the search space beyond what the continuation-bit and + magnitude-threshold sweeps already covered exhaustively - delta-vs- + absolute is a reinterpretation of an already-parsed token's *value*, + not a new rule for *which bytes form a token*. It remains open and + useful, but only as a **validator** for a candidate token-boundary + assignment a future width-selection breakthrough would produce, not + as an independent decode strategy in its own right. See the + 2026-07-20 session for the full reasoning. +- **(Revisited 2026-07-20, still open) Try IEEE-754 half-precision + (binary16) floats directly**, not just the word-swapped "PDP-endian" + full float32 interpretation already explored. A fixed-width fp16 + array is now ruled out directly (real-mode body length equals `2 * + npts` in only 1 of 3498 real-mode segments checked in one file, 0 of + 2662 in another). Like spectral-domain delta above, fp16 numeric + interpretation only matters once a token-boundary rule exists to + apply it to - it doesn't independently help find that rule. See the + 2026-07-20 session. +- **(Attempted 2026-07-20, partially resolved) Characterize the + `CheckSum` stream (112 bytes) more fully.** Two of the four varying + fields are now fully identified: offset 56 and offset 88 (each a + little-endian `u32`, not `u16`) are the exact byte sizes of the `3D + Raw Data` and `Max Plot` streams respectively, verified with zero + mismatches across 77 files - this also corrects the previous session's + reading of offset 58 as a boolean "flat vs. real" flag (it's actually + the high word of the offset-56 size field). The other two fields + (offsets 48 and 80) were confirmed genuinely content-dependent (not + a per-session token) but a sweep of 19 CRC-16 polynomials plus + Fletcher/Adler/plain-sum checksums against 5 candidate byte ranges + found no match - the "brute-force small CRC polynomials" plan this + bullet originally proposed has now been tried against the standard + polynomial set and come up empty; a future attempt should widen the + byte-range search (e.g. per-segment checksums) rather than repeat the + same polynomial sweep. See the 2026-07-20 session for full detail. +- **(Not attempted 2026-07-20, still open) Check whether the + flat-to-real transition aligns with a fixed elapsed *time* rather than + a fixed segment *count*.** The 2026-07-20 session did not attempt this + - there is no existing per-segment retention-time decode for the PDA + stream in this codebase to cross-reference against (unlike TTFL's RT + index), so establishing one is itself a prerequisite sub-investigation. + The transition-segment pass found the index varies (11 or 12) across + files with the same instrument/method - worth checking the actual + retention-time value at the transition point (cross-referencing + whatever RT stream the reader already decodes elsewhere in this repo, + e.g. the TTFL retention-time index) instead of segment index, in case + it's a fixed elapsed-time firmware constant (e.g. "N seconds of lamp + warm-up/blanking before real acquisition starts") rather than a + coincidence of segment count. +- **(Not attempted 2026-07-20, still open) Use physical plausibility + (peak shape) as a soft validator, not just exact byte-count + matching.** The 2026-07-20 session did not attempt this either, for + the same reason as the fp16/spectral-delta ideas above: it requires a + candidate token-boundary assignment to validate against, and none of + the framing searches to date (including this session's CRC sweep and + fp16/spectral-delta scoping analysis) produced one. The joint-decoder + pass's core problem was a degenerate scoring function admitting many equally-cheap but structurally different alignments. Plotting a candidate decode's values over time per channel and checking for smooth, single-peaked @@ -1269,3 +2638,54 @@ treat them as leads, not findings: checking zero-leftover exactness - could help disambiguate between otherwise-tied candidate parses, the same way a human would sanity- check a chromatogram by eye. +- **(New, from 2026-07-20 session 2) Region-local rather than + whole-body marker placement for the block-floating-point/marker-bit + family.** Session 2's item (c) tested a per-token marker bit against + the whole concatenated body (the `A`+`tail` regions treated as one + token stream, per session 1's "no vocabulary difference between + regions" finding). It did not try treating `A` and `tail` as two + *independently scaled* blocks, each with its own `base_w`/`n_wide` + derived from its own sub-length rather than the combined body length - + a real block-floating-point scheme could plausibly reset its + effective scale at the region boundary even if the byte-level + vocabulary looks similar on both sides (vocabulary similarity says + nothing about whether the *width-selection arithmetic* is shared). + This is a cheap variant of an already-built test (`walk_bfp` from + session 2) - re-run it with `A` and `tail` split before computing + `base_w`/`n_wide` for each, rather than merged. +- **(New, from 2026-07-20 session 2) A non-linear or table-driven + transform from an explicit per-segment header to a threshold.** + Session 2's item (a) only tried using a candidate header byte + *directly* as a threshold (or its bitwise complement). A real gain + header more plausibly indexes a small lookup table (as in ADPCM step + tables) rather than being usable as a raw magnitude cutoff - this + wasn't tried since it requires guessing the table itself, which is a + much larger search than session 2's budget allowed, but is a more + faithful test of the "explicit per-segment scale field" idea than + what was actually tried. +- **(Attempted 2026-07-20 session 7, ruled out) Cross-reference the + PSI-MS/mzML open spec's own compression schemes for the token-boundary + rule.** Two concrete readings were tried and ruled out this session: + MS-Numpress-style nibble-granular varint encoding (both a generic + per-nibble continuation flag and the length-prefixed scheme closer to + Numpress's own published algorithm), and literal zlib/DEFLATE framing + of the payload or body. See the 2026-07-20 session 7 section above for + full detail, including the shuffled-byte and cross-file controls that + disqualified the one place either search produced a nonzero hit rate. + This closes the two most literal readings of the idea; what remains + untried from this angle is applying MS-Numpress's *value* transforms + (double-delta linear prediction, or the "Slof" log-fixed-point + transform) as validators against a candidate token-boundary assignment + a future width-selection breakthrough would produce - the same + "validator, not framing strategy" caveat already noted above for + fp16 and spectral-domain delta, since neither transform changes which + bytes form a token, only how a found token's bytes become a number. +- **(New, from 2026-07-20 session 7) Manual, by-hand nibble-level + inspection of region `tail`'s channel-2-onward bytes was not attempted + this session.** Session 7's nibble-granular sweeps were parameterized + searches, in the same automated-sweep style this document's own session + 6 found unproductive for the byte-granular case. The manual read that + session recommends as the next step for region `tail` channels 2+ (see + that bullet above) has not yet been tried at nibble granularity either, + and remains a genuinely untried combination of "manual" and + "nibble-aware" that no session, including this one, has done. diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index dc8ddc1..91b5b77 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -286,3 +286,44 @@ that body remains undecoded despite a wide sweep of variable-length integer and escape-byte hypotheses (Sigilweaver/OpenSZRaw#2). This is UV detector / chromatogram data, not core MS spectra, so it does not block MS-level format parity. + +Seven same-day (2026-07-20) sessions of further clean-room analysis +narrowed the problem considerably without decoding it. Confirmed: 2 of +the 4 varying fields in the 112-byte `PDA 3D Raw Data/CheckSum` stream +are exact `u32` byte sizes of the `3D Raw Data` and `Max Plot` streams +(not a flat/real flag as previously read), and the "split" envelope +form's two declared-length regions are an exact +256-wavelength-channel/remainder split, which also explains why +"split" vs. "symmetric" envelope form correlates with wavelength +count. Ruled out, each time via actual value decoding plus a +physical-plausibility check and randomized controls rather than on +walk-rate alone: 19 CRC-16 polynomials and several count/derived-size +candidates for the remaining `CheckSum` fields; a fixed-width fp16 +array; a block-floating-point/adaptive-scale hypothesis family +(including a candidate that passed two randomized-control checks but +proved to be a compensating-error artifact); two more signals surfaced +by re-running the decode sweeps with corrected per-region target +counts; an anti-mode-collapse cost term added to the joint +temporal+magnitude decoder (the underlying temporal-correlation signal +replicates in a more rigorous multi-pair test but is markedly weaker +than the original single-pair anecdote suggested); and, from a +deliberately manual byte-reading pass (rather than another automated +sweep), a "leading byte of a 3-byte token" hypothesis that traced to a +low-value-diversity metric artifact confirmed on three independent +files. Two genuine process improvements came out of this: a directly +quantified ~48% false-positive base rate for this document's +zero-leftover acceptance test, and a fix to the physical-plausibility +check itself (mode-dominated or low-diversity decodes can look +deceptively "smooth" under mean relative step alone). A seventh session +cross-referenced the PSI-MS/mzML open spec directly (per a specific ask +to look there rather than sweep more parameters of already-tried +byte-granular schemes): two MS-Numpress-inspired nibble-granular varint +encodings and literal zlib/DEFLATE framing of the payload, all ruled out +- one nibble scheme's only nonzero hit rate was disqualified by a +shuffled-byte control (80% of it survived byte-order scrambling) and +cross-file testing (collapsed on two sibling files), and the DEFLATE +framing's small hit rate proved statistically indistinguishable from +both a random-byte and a shuffled-byte control. See +`docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-7 for +full detail. None of this decodes the per-value payload; that grammar +is still open.