From e8ecf416671a247742d3c32f1264d72564b85cbe Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 14:52:26 +0000 Subject: [PATCH 1/6] docs: identify two CheckSum fields, rule out CRC-16/fp16 hypotheses for PDA payload (#2) Continues the PDA/chromatogram payload investigation from docs/format/04. Full characterization of the 112-byte PDA 3D Raw Data/CheckSum stream finds that offset 56 and offset 88 (each a little-endian u32) are the exact byte sizes of the 3D Raw Data and Max Plot streams respectively, verified with zero mismatches across all 77 locally available files with both streams present. This corrects an earlier reading of offset 58 as a boolean "flat vs. real" flag - it is actually the high word of the offset-56 size field, and the earlier flat/real correlation was coincidental to stream-size magnitude. The other two varying CheckSum fields (offsets 48 and 80) are confirmed genuinely content-dependent (identical across files with byte-identical flat stream content, different otherwise), but a sweep of 19 CRC-16 polynomials plus Fletcher/Adler/plain-sum checksums against five candidate byte ranges found no match - a real negative result, not proof no algorithm exists. Also rules out a fixed-width fp16 (binary16) array for the per-value payload (body length essentially never equals 2 * npts), precisely characterizes the split-form real-mode body length centering (~1.88 * npts, distinct from the symmetric form's ~3 * npts), and records why the fp16 and spectral-domain-delta ideas can only function as validators for a token-boundary rule rather than independent framing searches - both remain open per docs/format/06-known-limitations.md section 10. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu --- CHANGELOG.md | 14 ++ CONTRIBUTORS.md | 3 +- docs/format/04-lcd-chromatogram-pda.md | 313 +++++++++++++++++++++---- docs/format/06-known-limitations.md | 9 + 4 files changed, 296 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3158ee5..a017a4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 see `docs/format/06-known-limitations.md` #9). Resolves Sigilweaver/OpenSZRaw#9. +### Documentation + +- Further PDA/chromatogram payload investigation + (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): identified 2 of the + 4 varying fields in `PDA 3D Raw Data/CheckSum` as exact stream byte + sizes (correcting an earlier "flat vs. real flag" reading), ruled out + a 19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum) for the + remaining 2 fields, ruled out a fixed-width fp16 (binary16) array, and + clarified that the fp16 and spectral-domain-delta ideas can only + function as validators for a token-boundary rule, not as independent + framing searches. The per-point payload grammar itself is still + undecoded - see `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 + session for full detail. + ## [0.1.0] - 2026-07-18 ### Added diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 1307ce7..f8362ba 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -2,4 +2,5 @@ Thank you to everyone who has contributed to OpenSZRaw. -No external contributions yet - this list will grow as people help out. +- [@Nabejo](https://github.com/Nabejo) - PDA/chromatogram payload + investigation (Sigilweaver/OpenSZRaw#2). diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index b66894f..002ac58 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -47,7 +47,19 @@ is a scannable summary, not a substitute for the detailed sections below - Real-mode body length is **tightly centered near `3 * npts` bytes** for the symmetric (`MTBLS432`) form - close to, but not exactly, a uniform 3-bytes-per-value scheme. See "width-table retry on bimodal - data". + data". The split (`PXD025121`/`MSV000084197`) form centers lower, near + `1.88 * npts` (range `[1.65, 2.08] * npts`, 100% of one file's + real-mode segments checked) - a distinct, form-specific constant, not + the same `3 * npts` figure. See the 2026-07-20 session. +- **`CheckSum` stream, 2 of 4 varying fields fully identified**: offset + 56 (`u32` LE) is the exact byte size of `PDA 3D Raw Data/3D Raw Data`; + offset 88 (`u32` LE) is the exact byte size of `PDA 3D Raw Data/Max + Plot` - both zero-mismatch across 77 files. This corrects an earlier + reading of offset 58 as a "flat vs. real" flag byte (it's actually + part of the offset-56 size field). Offsets 48 and 80 are confirmed + genuinely content-dependent but remain unidentified after a sweep of + 19 CRC-16 polynomials, Fletcher/Adler/plain-sum checksums, and 5 + candidate byte ranges. See the 2026-07-20 session. - Width selection is very likely **driven by actual per-wavelength signal magnitude**, not any of: an external per-run lookup table (no config/instrument stream correlates with modal length - "external- @@ -74,7 +86,10 @@ prefixes; magnitude-threshold split (whole-body and region-A-isolated); leading bitmap/nibble width-selector tables (1/2/4-bit codes, both pooled-file and per-file least-squares); the TTFL MS RLE scheme applied directly; an external per-run gain/width table stored elsewhere in the -file; `Max Plot` as an independently-decodable crib. +file; `Max Plot` as an independently-decodable crib; a fixed-width fp16 +(binary16) array (body length essentially never equals `2 * npts`); the +19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum) against +`CheckSum` offsets 48 and 80 (see 2026-07-20 session). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric @@ -91,6 +106,15 @@ file; `Max Plot` as an independently-decodable crib. a different/simpler-looking encoding, out of this issue's named scope) - not decoded, not attempted beyond the initial characterization. See "LC Raw Data..." near the end of this document. +- `CheckSum` offsets 48 and 80 (confirmed content-dependent, not + identified as any standard CRC/checksum algorithm - see 2026-07-20 + session). +- Whether fp16 (binary16) numeric interpretation or spectral-domain + (wavelength-to-wavelength) delta coding is the right *value* + interpretation for a token, once a token-boundary rule is found - + both remain plausible as validators, neither has been testable yet + since no token-boundary rule has been found to validate. See + 2026-07-20 session. ## Segment Header @@ -642,6 +666,22 @@ byte at `CheckSum` offset 58 is `0x09` in all 9 real-signal files and says whether real data exists at all, not anything about its encoding width, so it doesn't help decode the payload itself. +**Correction (2026-07-20 session)**: offset 58 is not a dedicated flag +byte. It is the high 16 bits of a little-endian `u32` field starting at +offset 56, and that `u32` is the exact byte length of the +`PDA 3D Raw Data/3D Raw Data` stream itself (e.g. `0x0009_1A64` = +`596580`, the `12_65` file's exact stream size, decomposing into the +`0x1A64`/`6756` seen at offset 56-57 and the `0x09` seen at offset 58). +The apparent "flat vs. real flag" reading of the lone offset-58 byte was +a coincidence of magnitude: flat-only files' `3D Raw Data` streams +happen to stay small enough that byte 58 (bits 16-23 of the size) reads +`0x03` in this corpus's sample, and real-signal files' larger streams +happen to read `0x09` - not a purpose-built boolean. See the 2026-07-20 +session below for the full re-characterization, including the +analogous offset-88 field (`Max Plot` stream size) and a from-scratch +CRC/checksum sweep against the two still-unidentified offset-48 and +offset-80 fields. + **Verdict: this thread is closed.** Since the method configuration, instrument parameters, and wavelength table are provably identical byte-for-byte across files whose "real mode" encoding width differs, the @@ -1167,6 +1207,176 @@ unbounded-beam "exact" DP with backtracking, and the true-vs-control comparison script) built directly on `iter_segments` from `pda_varint_bruteforce.py`. +## 2026-07-20 session: full `CheckSum` field identification (2 of 4), a CRC/checksum sweep that rules out the standard algorithms for the other 2, and a feasibility note on the fp16/spectral-delta avenues + +Picked up two of the "further avenues" this document's previous session +left open: fully characterizing the 112-byte `CheckSum` stream (avenue +3), and trying IEEE-754 half-precision floats and spectral-domain +(wavelength-to-wavelength) delta encoding (avenues 1 and 2). Net result: +one genuine structural correction with full corpus verification, a +thorough negative result on the two still-unidentified `CheckSum` +fields, and an analytical finding that the fp16/spectral-delta ideas - +as pure *framing* hypotheses - don't actually open new search space +beyond what earlier sessions' continuation-bit and magnitude-threshold +sweeps already covered exhaustively. **The per-value payload grammar +remains undecoded.** + +- **`CheckSum` offsets 56 and 88 fully identified: exact stream byte + sizes, not checksums.** Read as little-endian `u32` (not `u16`) at + each offset, offset 56 equals the exact byte length of + `PDA 3D Raw Data/3D Raw Data` and offset 88 equals the exact byte + length of `PDA 3D Raw Data/Max Plot`, both with **zero mismatches + across all 77 locally available files with a real `CheckSum` stream + and both sub-streams present** (45 `MTBLS432` files - both the 9 + real-signal and the fully-flat ones, 31 `PXD025121` files, and + `MSV000084197/20190607_NM16.lcd`; checked with `olefile`'s own + `get_size()` against each file's independently-read stream length, no + parsing of the payload itself required). This directly corrects the + external-table-hunt session's reading of offset 58 as a "flat vs. + real" flag byte - see the correction note inserted in that section + above. The two `u32` fields' high words (bytes 50-51 and 82-83) were + also confirmed `0x0000` in every file checked, consistent with them + being plain sizes rather than some other 32-bit quantity that happens + to coincide with a stream length. +- **Offsets 48 and 80 remain unidentified, but are now more precisely + characterized as genuinely data-dependent (not a per-session token or + random field), and a real checksum/CRC sweep has been run against + them and come up empty.** First, a cheap cross-file check + distinguishes "depends on file content" from "random/session-specific": + among the 15 `MTBLS432` files sampled, 5 fully-flat files that all + share the same segment count (`nseg=2674`) - and therefore, per the + cross-file pass's byte-identity finding two sessions ago, byte-identical + `3D Raw Data` stream content - all report the **exact same** offset-48 + value (`30350`) and offset-80 value (`28638`); a sixth flat file with a + different segment count (`nseg=2673`) reports different values + (`12815`/`57449`) for both. This rules out "opaque per-run token" (it + would not repeat exactly across independent acquisitions) and confirms + these fields are a deterministic function of the stream's actual byte + content, exactly what the stream's name (`CheckSum`) implies - but a + wide, explicit sweep against both fields still found no match: + + - **Algorithms tried**: 19 CRC-16 polynomials (including CCITT, + IBM/ANSI, XMODEM, standard reciprocal/reversed variants, and several + less common published polynomials) crossed with `{reflect-in, + reflect-out} x {0x0000, 0xFFFF, 0x1D0F, 0x800D, 0xB2AA, and two + data-derived seeds (stream length, segment count)} x {xorout 0x0000, + 0xFFFF}` - 19 x 2 x 2 x 7 x 2 = 1064 parameter combinations per + byte-range tried, implemented with a from-scratch table-based CRC-16 + (no vendor or third-party crypto library beyond Python's own + `zlib`/`crcmod`, both open-source, general-purpose, format-agnostic + checksum implementations, consistent with `CONTRIBUTING.md`'s + "independent open-source parsers used purely as format checkers" + allowance - `crcmod` was used only to spot-check the from-scratch + implementation's own predefined-name variants agree, not as a source + of Shimadzu-specific knowledge). Also tried: `zlib.crc32` (both + 16-bit halves), `zlib.adler32` (low 16 bits), Fletcher-16, Fletcher-32 + (low 16 bits), and plain 16-bit additive/XOR byte sums. + - **Byte ranges tried**: the full `PDA 3D Raw Data/3D Raw Data` stream + bytes (segment headers and payloads together), payload-only + (24-byte segment headers stripped), body-only (the split/symmetric + envelope's 4/8-byte header-and-footer further stripped), the 24-byte + segment headers alone concatenated, and (for the offset-80 field) + the `PDA 3D Raw Data/Max Plot` stream bytes directly - the natural + candidate given offset 80 sits in the same header-then-size + structural slot, immediately before offset 88's confirmed Max Plot + size, that offset 48 occupies relative to offset 56's confirmed `3D + Raw Data` size. + - **Result: no match, on any combination, against any of the four + files checked simultaneously** (`12_65`, `1_63`, `34_73`, and + `20_66` for the initial pass; `12_65`, `1_63`, `34_73`, and `21_29` + for the widened rerun). A simple direct-sum-of-real-segment-bytes + check (mod 65536) was also tried and did not match either offset. + - This is a real negative result (a specific, bounded, reproducible + search that came up empty), not proof no algorithm exists - Shimadzu + could use a nonstandard polynomial, a different data range than the + five tried, or a non-CRC construction entirely. But it means the + "brute-force small CRC polynomials against known bytes" plan from + the previous session's "further avenues" note does not immediately + pay off with the standard polynomial set, so a future session + revisiting this should either widen the byte-range search further + (e.g. per-segment individual checksums rather than one whole-stream + value) or treat this thread as lower-priority than the per-value + grammar itself. + +- **fp16 (binary16) as a fixed-width array: ruled out immediately by + body length, as expected.** If real-mode bodies were a plain array of + 2-byte half-floats, body length would equal exactly `2 * npts` on + every real segment. Checked directly: in `MSV000084197`'s split-form + stream (`npts=321`, 3502 segments), body length equals `2 * npts` + (642 bytes) in exactly **1 of 3498 real-mode segments** (segment index + 92, in the middle of a smoothly-growing run of neighboring lengths + 620/637/652/**642**/622/618/614 - a coincidental crossing, not a + population), and in `MTBLS432`'s symmetric-form stream (`npts=68`), + `2 * npts` (136 bytes) occurs in **0 of 2662** real-mode segments. + This is the same kind of hard ruling-out the document's existing + "byte-level volatility" and length-histogram findings already + established for a generic fixed-width hypothesis, now specifically + against fp16's 2-byte width. +- **A precise, previously-only-anecdotal characterization: the + split-form's real-mode body length centers near `1.88 * npts`, not + `3 * npts`.** The document's existing "tightly centered near `3 * + npts`" finding was established only for the symmetric (`MTBLS432`) + form; the split form's centering had only been described anecdotally + via one early-buildup example (segments 4-9 of `MSV000084197`, lengths + 530-571 for `npts=321`, i.e. roughly 1.65-1.78x). Measured precisely + across all 3498 real-mode segments of that same stream: mean real + body length is **604.8 bytes = 1.884 * npts**, median **609 bytes = + 1.897 * npts**, and the full observed range is **530 to 668 bytes + (1.651x to 2.081x npts)** - i.e. **100% of this file's real-mode + segments fall in a `[1.6, 2.2] * npts` band**, a tighter and + differently-centered analogue of the symmetric form's `3 * npts` + finding, not the same constant. This is consistent with (not + contradictory to) the existing magnitude-threshold sweep's + best-performing configuration on this same file being a *2-byte* wide + token with a 1-byte cheap case (`threshold=0x1f`/`0x20`), since a mix + of mostly-2-byte with some 1-byte tokens naturally averages below 2.0 + and above 1.0 per value - it does not, by itself, favor fp16 over the + already-tried magnitude-threshold framing. +- **Why neither fp16-with-escape nor spectral-domain delta actually + reopens new search space, analytically** (no new code needed for this + part - a scoping finding, recorded so a future session doesn't spend a + session re-deriving it): both ideas change how a *token's bytes* are + turned into a *number*, not which bytes belong to which token. The + already-exhausted continuation-bit sweep (all 8 bit positions x both + polarities x 0-11 byte header/footer skip) and magnitude-threshold + sweep (all 256 threshold values x 2-4 byte wide-token width) already + covered every possible token-boundary assignment those searches were + capable of expressing - swapping the *numeric interpretation* of a + found 2-byte or 4-byte token from "raw magnitude-threshold scalar" to + "IEEE-754 binary16" or "delta from the previous wavelength's value" + changes nothing about which byte offsets get grouped into which + token, so it cannot by itself turn a boundary search that already + plateaued at ~6% clean segments into a working decode. Both ideas + remain genuinely useful, but only as **validators/tie-breakers** for + candidate token-boundary assignments a future width-selection + breakthrough would produce (does the resulting number look like a + plausible fp16 absorbance value; does the resulting per-wavelength + sequence look smoother when spectral-differenced) - not as + independent framing strategies in their own right. This reframes two + of the previous session's four "further avenues" from "untried + decode strategies" to "untried validators," which is a more accurate + scoping for whoever picks this up next. +- **Not pursued this session, for budget reasons, still open**: avenue + 4 (aligning the flat-to-real transition to elapsed retention time + rather than segment count) was not attempted - there is no existing + per-segment retention-time decode for the PDA stream in this + codebase to cross-reference against (unlike TTFL's RT index), so + establishing one would be its own sub-investigation before the + time-alignment question could even be asked. Avenue 5 (physical + plausibility/peak-shape as a soft validator) was not attempted either, + for the same reason avenue 1's fp16 analysis above landed on: it + requires a candidate token-boundary assignment to validate, and none + of the framing searches to date (including this session's) have + produced one to test it against. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), built on a from-scratch `olefile`-based +`iter_segments`/`body_of` pair mirroring the shape of +`pda_varint_bruteforce.py` from earlier sessions, plus a from-scratch +table-based CRC-16 implementation (`crc_fast.py`-equivalent) and +`crcmod`/`zlib` for cross-checking standard-named variants. + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -1222,46 +1432,65 @@ treat them as leads, not findings: MassIVE, searching for older instrument models or explicitly UV/RID-detector method descriptions) before assuming `PDA 3D Raw Data` is a valid stand-in for the named stream. -- **Test spectral-domain (wavelength-to-wavelength) delta encoding, not - just temporal (segment-to-segment) delta encoding.** Everything tried - so far assumes each segment's `npts` tokens are independent absolute - values, or deltas against the *same channel in the previous segment*. - UV absorbance spectra are also smooth *across wavelength* within a - single timepoint - worth testing whether a token's value is better - modeled as a delta from the *previous wavelength's* decoded value in - the *same* segment (first-differencing within one spectrum), which is - a distinct hypothesis from anything in the "ruled out" list above. -- **Try IEEE-754 half-precision (binary16) floats directly**, not just - the word-swapped "PDP-endian" full float32 interpretation already - explored. Absorbance units typically span a small range (0-3 AU), - which fp16 represents naturally in 2 bytes - worth checking whether a - mixed fp16/fp32 scheme (small values as 2-byte half floats, an escape - to 4-byte full float32 for outliers) fits better than the arbitrary - magnitude-agnostic proxy the joint-decoder pass used, and ties - naturally into the already-observed 2-vs-3-vs-4-byte width mix. -- **Characterize the `CheckSum` stream (112 bytes) more fully.** The - external-table-hunt pass found byte offset 58 cleanly flags real-vs- - flat mode (`0x09` vs `0x03`) but only checked correlation against - modal *length* - the rest of the stream's 112 bytes were never - systematically characterized. If any field there is a genuine - checksum/CRC of the segment payload (plausible, given the stream's - name), brute-forcing small CRC polynomials against known segment - bytes would give a powerful, cheap, objective validator for candidate - decodes - far stronger than "does the byte count come out exact," - which today's sessions repeatedly found admits multiple false - positives. -- **Check whether the flat-to-real transition aligns with a fixed - elapsed *time* rather than a fixed segment *count*.** The transition- - segment pass found the index varies (11 or 12) across files with the - same instrument/method - worth checking the actual retention-time - value at the transition point (cross-referencing whatever RT stream - the reader already decodes elsewhere in this repo, e.g. the TTFL - retention-time index) instead of segment index, in case it's a fixed - elapsed-time firmware constant (e.g. "N seconds of lamp warm-up/ - blanking before real acquisition starts") rather than a coincidence of - segment count. -- **Use physical plausibility (peak shape) as a soft validator, not just - exact byte-count matching.** The joint-decoder pass's core problem was +- **(Revisited 2026-07-20, still open) Test spectral-domain + (wavelength-to-wavelength) delta encoding, not just temporal + (segment-to-segment) delta encoding.** The 2026-07-20 session found + this idea, taken as a pure token-boundary/framing hypothesis, does not + actually expand the search space beyond what the continuation-bit and + magnitude-threshold sweeps already covered exhaustively - delta-vs- + absolute is a reinterpretation of an already-parsed token's *value*, + not a new rule for *which bytes form a token*. It remains open and + useful, but only as a **validator** for a candidate token-boundary + assignment a future width-selection breakthrough would produce, not + as an independent decode strategy in its own right. See the + 2026-07-20 session for the full reasoning. +- **(Revisited 2026-07-20, still open) Try IEEE-754 half-precision + (binary16) floats directly**, not just the word-swapped "PDP-endian" + full float32 interpretation already explored. A fixed-width fp16 + array is now ruled out directly (real-mode body length equals `2 * + npts` in only 1 of 3498 real-mode segments checked in one file, 0 of + 2662 in another). Like spectral-domain delta above, fp16 numeric + interpretation only matters once a token-boundary rule exists to + apply it to - it doesn't independently help find that rule. See the + 2026-07-20 session. +- **(Attempted 2026-07-20, partially resolved) Characterize the + `CheckSum` stream (112 bytes) more fully.** Two of the four varying + fields are now fully identified: offset 56 and offset 88 (each a + little-endian `u32`, not `u16`) are the exact byte sizes of the `3D + Raw Data` and `Max Plot` streams respectively, verified with zero + mismatches across 77 files - this also corrects the previous session's + reading of offset 58 as a boolean "flat vs. real" flag (it's actually + the high word of the offset-56 size field). The other two fields + (offsets 48 and 80) were confirmed genuinely content-dependent (not + a per-session token) but a sweep of 19 CRC-16 polynomials plus + Fletcher/Adler/plain-sum checksums against 5 candidate byte ranges + found no match - the "brute-force small CRC polynomials" plan this + bullet originally proposed has now been tried against the standard + polynomial set and come up empty; a future attempt should widen the + byte-range search (e.g. per-segment checksums) rather than repeat the + same polynomial sweep. See the 2026-07-20 session for full detail. +- **(Not attempted 2026-07-20, still open) Check whether the + flat-to-real transition aligns with a fixed elapsed *time* rather than + a fixed segment *count*.** The 2026-07-20 session did not attempt this + - there is no existing per-segment retention-time decode for the PDA + stream in this codebase to cross-reference against (unlike TTFL's RT + index), so establishing one is itself a prerequisite sub-investigation. + The transition-segment pass found the index varies (11 or 12) across + files with the same instrument/method - worth checking the actual + retention-time value at the transition point (cross-referencing + whatever RT stream the reader already decodes elsewhere in this repo, + e.g. the TTFL retention-time index) instead of segment index, in case + it's a fixed elapsed-time firmware constant (e.g. "N seconds of lamp + warm-up/blanking before real acquisition starts") rather than a + coincidence of segment count. +- **(Not attempted 2026-07-20, still open) Use physical plausibility + (peak shape) as a soft validator, not just exact byte-count + matching.** The 2026-07-20 session did not attempt this either, for + the same reason as the fp16/spectral-delta ideas above: it requires a + candidate token-boundary assignment to validate against, and none of + the framing searches to date (including this session's CRC sweep and + fp16/spectral-delta scoping analysis) produced one. The joint-decoder + pass's core problem was a degenerate scoring function admitting many equally-cheap but structurally different alignments. Plotting a candidate decode's values over time per channel and checking for smooth, single-peaked diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index dc8ddc1..7ef6406 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -286,3 +286,12 @@ that body remains undecoded despite a wide sweep of variable-length integer and escape-byte hypotheses (Sigilweaver/OpenSZRaw#2). This is UV detector / chromatogram data, not core MS spectra, so it does not block MS-level format parity. + +A 2026-07-20 session identified 2 of the 4 varying fields in the +112-byte `PDA 3D Raw Data/CheckSum` stream as exact `u32` byte sizes of +the `3D Raw Data` and `Max Plot` streams (not a flat/real flag as +previously read - see the correction note and dated session in +`docs/format/04`), and ruled out 19 standard CRC-16 polynomials plus +several other common checksum constructions for the remaining 2 fields. +Neither finding decodes the payload; the per-value grammar is still +open. From 6d6ac93ec88cf370b047cf51d595f494140da06d Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 15:31:14 +0000 Subject: [PATCH 2/6] docs: resolve PDA split-region semantics, rule out block-floating-point payload hypotheses (#2) Continues the PDA/chromatogram payload investigation from the prior commit on this branch. Two new investigation passes, both aimed at an actual decode rather than further characterization alone. The first tests a block-floating-point/adaptive-scale hypothesis family (each segment carrying its own implicit or explicit scale rather than a global fixed threshold) in four concrete forms: an explicit per-segment header used directly as a threshold, a classic fixed-mantissa-width block scheme, a marker-bit escape with an algebraically-derived baseline width, and a leading-bitmap popcount validator. None decode the payload. Along the way, quantifies a ~48% false-positive base rate for this document's own "exact count, zero leftover" acceptance criterion, explaining why several superficially promising per-file signals throughout this document's history have failed to generalize. Also tests the CheckSum stream's two still-unidentified fields against plain counts and derived sizes (segment count, point count, various byte totals) rather than checksum algorithms - no match found, which strengthens rather than resolves the "genuine but unidentified checksum" conclusion. The second pass resolves a previously open structural question: the "split" envelope form's two declared-length regions are an exact 256-wavelength-channel/remainder split, verified with zero exceptions across every flat segment in all four locally available split-form files. This also fully explains why "split" vs. "symmetric" envelope form correlates with wavelength count. Retrying the marker-bit hypothesis region-by-region with this exact channel count turns up a real, cross-file, twice-randomized-control-verified signal in the "tail" region - but decoding actual values under it and checking them for temporal smoothness (the physical-plausibility validator this document had flagged but never applied) shows only one channel out of several dozen is genuinely smooth; the rest are indistinguishable from noise. Traces this to a concrete, directly observed compensating-error mechanism rather than leaving it as a vague caveat, and confirms a smaller, precise byte-level finding (the region's first two channels are fixed edge cases) unconditionally across all four files. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu --- CHANGELOG.md | 24 +- docs/format/04-lcd-chromatogram-pda.md | 426 ++++++++++++++++++++++++- docs/format/06-known-limitations.md | 11 +- 3 files changed, 442 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a017a4e..049a266 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,14 +21,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Further PDA/chromatogram payload investigation (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): identified 2 of the 4 varying fields in `PDA 3D Raw Data/CheckSum` as exact stream byte - sizes (correcting an earlier "flat vs. real flag" reading), ruled out - a 19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum) for the - remaining 2 fields, ruled out a fixed-width fp16 (binary16) array, and - clarified that the fp16 and spectral-domain-delta ideas can only - function as validators for a token-boundary rule, not as independent - framing searches. The per-point payload grammar itself is still - undecoded - see `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 - session for full detail. + sizes (correcting an earlier "flat vs. real flag" reading); ruled out + a 19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum and an + Internet-checksum variant) and several plain-count/derived-size + candidates for the remaining 2 fields; ruled out a fixed-width fp16 + (binary16) array and a block-floating-point/adaptive-scale hypothesis + family in four concrete forms, quantifying a ~48% false-positive base + rate for this document's own zero-leftover acceptance test along the + way; resolved a separate open question by finding the "split" envelope + form's two regions are an exact 256-wavelength-channel/remainder + split (also explaining why "split" vs. "symmetric" form correlates + with wavelength count); and showed, via a physical-plausibility check, + that a marker-bit signal which passed two randomized-control tests was + nonetheless a compensating-error artifact rather than a real decode. + The per-point payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-3 for + full detail. ## [0.1.0] - 2026-07-18 diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 002ac58..33ca5e0 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -60,6 +60,15 @@ is a scannable summary, not a substitute for the detailed sections below genuinely content-dependent but remain unidentified after a sweep of 19 CRC-16 polynomials, Fletcher/Adler/plain-sum checksums, and 5 candidate byte ranges. See the 2026-07-20 session. +- **The "split" form's two regions are an exact 256-channel/remainder + wavelength split**: region `A` always holds the first 256 wavelength + channels, region `tail` holds the remaining `npts - 256`, confirmed + with zero exceptions across every flat segment in all 4 locally + available split-form files (2 distinct `npts` values). This also + fully explains why "split" vs. "symmetric" envelope form correlates + with wavelength count: every corpus file with `npts <= 256` uses + symmetric, every file with `npts > 256` uses split. See 2026-07-20 + session 3. - Width selection is very likely **driven by actual per-wavelength signal magnitude**, not any of: an external per-run lookup table (no config/instrument stream correlates with modal length - "external- @@ -88,16 +97,24 @@ pooled-file and per-file least-squares); the TTFL MS RLE scheme applied directly; an external per-run gain/width table stored elsewhere in the file; `Max Plot` as an independently-decodable crib; a fixed-width fp16 (binary16) array (body length essentially never equals `2 * npts`); the -19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum) against -`CheckSum` offsets 48 and 80 (see 2026-07-20 session). +19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum, and an +Internet-checksum ones'-complement variant) against `CheckSum` offsets +48 and 80, and those two fields as any of several plain counts/derived +sizes (see 2026-07-20 sessions 1 and 2); the block-floating-point/ +adaptive-scale hypothesis family in four concrete forms - explicit +header-as-threshold, classic fixed-width-per-segment, marker-bit escape +with an algebraically-derived baseline width (a real single-file signal +that fails cross-file generalization), and a leading-bitmap popcount +validator (see 2026-07-20 session 2); the region-`tail`-isolated +marker-bit escape rule, despite passing two randomized-control checks +(uniform-random bytes and same-multiset-shuffled bytes), traced via a +physical-plausibility (temporal-smoothness) check to a compensating- +error artifact affecting all but one of 65 channels, not a real +per-channel decode (see 2026-07-20 session 3). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric interpretation) - the core unsolved problem. -- The functional meaning of the "split" form's two declared-length - regions (`A` bytes then `tail` bytes) - confirmed to exist, never - explained; the "region-boundary vocabulary check" found no byte-value - difference between them. - Whether `LSS Raw Data` (the issue's literally-named path) uses the *same* encoding as `PDA 3D Raw Data` at all - untested, since every locally available file has it empty. See "Further avenues" at the end @@ -107,14 +124,19 @@ file; `Max Plot` as an independently-decodable crib; a fixed-width fp16 - not decoded, not attempted beyond the initial characterization. See "LC Raw Data..." near the end of this document. - `CheckSum` offsets 48 and 80 (confirmed content-dependent, not - identified as any standard CRC/checksum algorithm - see 2026-07-20 - session). + identified as any standard CRC/checksum algorithm nor as a plain + count/derived size - see 2026-07-20 sessions 1 and 2). - Whether fp16 (binary16) numeric interpretation or spectral-domain (wavelength-to-wavelength) delta coding is the right *value* interpretation for a token, once a token-boundary rule is found - both remain plausible as validators, neither has been testable yet since no token-boundary rule has been found to validate. See - 2026-07-20 session. + 2026-07-20 session 1. +- Whether a more elaborate adaptive-scale mechanism than the four forms + ruled out in 2026-07-20 session 2 exists (e.g. a per-segment gain + header using a non-linear transform, or region-local rather than + whole-body marker placement) - untested, but the simplest and most + natural versions of the idea do not fit. ## Segment Header @@ -1377,6 +1399,346 @@ own scratch directory (outside the repo, not saved to table-based CRC-16 implementation (`crc_fast.py`-equivalent) and `crcmod`/`zlib` for cross-checking standard-named variants. +## 2026-07-20 session 2: block-floating-point/adaptive-scale hypothesis family tested and ruled out; CheckSum offsets 48/80 tested against count-based candidates; a quantified false-positive base rate + +Direct follow-up to session 1 above, prompted by a specific architectural +observation: every hypothesis ruled out so far assumed a single *global* +rule (fixed threshold, fixed escape byte, fixed continuation-bit +position) applied uniformly across an entire file, but the fact that +real-mode body length tracks each segment's own signal magnitude +continuously (not a fixed multiple of `npts`) is exactly the signature +of a *per-segment adaptive* quantizer - block floating point (a shared +per-block scale/exponent with narrower per-value mantissas), as used in +some real instrument firmware, being the leading concrete example. This +session tried four concrete variants of that family against the payload, +plus a fresh pass at the two still-unidentified `CheckSum` fields as +plain counts/derived sizes rather than checksums. **No decode was +found; the per-value payload grammar remains undecoded.** One negative +result is worth flagging as a genuine methodological contribution in its +own right: a directly quantified false-positive base rate for this +document's own "exact npts count, zero leftover" acceptance criterion, +which puts a hard number on a pattern this document has flagged +qualitatively many times before (Max Plot's 38 spuriously-successful +configs, the width-table retry's 60%->49% sample-size collapse, etc.). + +- **(a) Explicit per-segment header as a threshold source: ruled out, + weakly.** Hypothesized the body's first 1-2 bytes are a small + scale/gain field, separate from the first value token, and that a + *global* (same for every segment/file) formula converts that field + into a per-segment magnitude threshold for an otherwise-unchanged + 2-tier walk (byte-under-threshold = 1-byte token, else a wide token). + Tried header lengths 0/1/2 bytes x four threshold-derivation formulas + (`header_byte_0` directly, `header_byte_1`, low byte of a 2-byte + header read as `u16`, `255 - header_byte_0`) x wide-token widths + 2/3/4, against `MTBLS432/..._12_65...lcd`'s 2662 real-mode segments. + Best combination: **21/2662 (0.8%) clean** - weaker than the + document's existing fixed-global-threshold sweep, not better. An + explicit small header directly usable as a raw threshold value is not + the mechanism (though this doesn't rule out a header that needs a + more complex, not-yet-guessed transform). +- **(b) Classic fixed-width-per-block (true block floating point: + every value in one segment shares the same width): ruled out.** + If each segment's body were `H` header bytes plus `npts` values of a + single *per-segment* (but not per-value) width `W`, then `(body_len - + H)` should divide evenly by `npts` for the great majority of real + segments, for some small `H`. Checked `H` from 0 to 5 bytes against + the same 2662-segment sample: the best case (`H=0`) has only + **70/2662 (2.6%)** of segments dividing evenly into an integer + quotient in the plausible 1-6 byte range (all landing on quotient + `3`, consistent with the existing `3 * npts` centering finding + already documented, but far short of "most segments"), and every + nonzero `H` tried does no better. Per-value width genuinely varies + *within* a segment, not just *between* segments - ruling out a pure + shared-exponent/fixed-mantissa-width model. +- **(c) Marker-bit escape with an algebraically-derived baseline width: + a real within-file signal that does not survive cross-file testing.** + The most promising variant tried: for each segment, `base_w = body_len + // npts` and `n_wide = body_len - npts * base_w` are *exactly* + computable in advance from body length alone (no fitting) under a + "every token is `base_w` or `base_w + 1` bytes" model - so the only + free choice is which bit, in which byte of each token's `base_w`-byte + prefix, signals "this token is the wider one." Swept all 8 bit + positions x 2 polarities x {marker in the token's first byte, marker + in its last byte} (32 combinations) against 4 confirmed real-signal + `MTBLS432` files (`..._12_65`, `..._34_73`, `..._26_68`, `..._21_29` + - all `npts=68`, same instrument/method). Individual files show + real-looking signal - `bit=5, polarity=set, marker=first-byte` reaches + **868/2663 (32.6%)** clean on `..._34_73` and **1100/2663 (41.3%)** on + `..._26_68` - with successes spread across a representative range of + body lengths (207-220 bytes out of an overall 202-219 range on + `..._34_73`, not clustered at a degenerate edge case) - but **the same + configuration gets only 1.1% on `..._12_65` and 8.5% on `..._21_29`**, + and no single configuration reaches even 15% on all four files + simultaneously: the best worst-case (minimum-across-all-four-files) + rate across the full 32-combination sweep is **1.2%**. Since all four + files share the same instrument, method, and `npts`, a real shared + encoding rule should generalize across them the way the envelope and + `u32[2]` facts already do - this doesn't, so it's ruled out as the + real rule despite looking compelling on any single file in isolation. +- **(d) Leading bitmap whose popcount should equal `n_wide`: ruled + out.** A cheaper-to-test alternative to (c): instead of an inline + per-token marker bit, hypothesized a leading `ceil(npts/8)`-byte + (9 bytes for `npts=68`) bitmap at the start of the body (after 0-3 + header bytes) whose set-bit count equals the algebraically-required + `n_wide` for a `base_w=2` model. Checked header offsets 0-3 against + both `..._12_65` (2566/2662 segments applicable to the `base_w=2` + assumption) and `..._34_73` (only 28/2663 applicable, since that + file's real-mode segments center higher, nearer `3 * npts`): **zero + matches in every case**. Ruled out. +- **A quantified false-positive base rate for this document's own + "exact count, zero leftover" acceptance criterion** (a methodological + finding, not a decode attempt): to sanity-check how much weight a + single segment's "a threshold that decodes it cleanly" result should + carry, checked - for a random sample of 400 real-mode segments from + `..._12_65` - what fraction admit *at least one* successful + `(wide_width, threshold)` combination out of the 3 x 256 = 768 tried + (the same free 2-tier walk as (a) above, but with no header at all). + **191/400 (47.75%) of segments admit at least one spuriously "clean" + decode**, averaging **18.8 different working threshold values per + successful segment**, and - most tellingly - **body-length statistics + for segments that admit a lucky threshold are statistically + indistinguishable from those that don't** (mean 202.6 bytes for + "admits a threshold" vs. 202.8 bytes for "does not," essentially + identical). This means, for this document's typical `npts` scale + (dozens to low hundreds of tokens per segment), "there exists a + threshold under which this one segment decodes cleanly" is close to a + coin flip with no relationship to the segment's actual content - + putting a concrete number on why this document has repeatedly found + (Max Plot's 38-configuration false-positive set; the width-table + retry's 60%->49% sample-size collapse; this session's own item (c) + above) that single-file or small-sample "clean decode" rates cannot be + trusted without an explicit cross-file or cross-segment generalization + check, and why every genuinely-confirmed fact in this document (the + envelope, `u32[2]`, the two new `CheckSum` size fields) was checked + against every segment of multiple independent files before being + written down as confirmed. +- **`CheckSum` offsets 48 and 80 tested against plain counts and derived + sizes, not just checksum algorithms: no match found.** Before + resuming the checksum-algorithm search from session 1, tested whether + offset 48 (and separately offset 80) is simply a count or a + differently-scoped size rather than a hash, using the two files with + precisely known values (`..._1_63`/`..._20_66`/`..._30_71`/`..._3_64`/ + `..._44_76`, all `nseg=2674`, offset 48 = `30350`; `..._37_74`, + `nseg=2673`, offset 48 = `12815`). First verified the assumption + the candidate-count hypotheses depend on - that every segment always + declares exactly `npts` points (i.e. total declared point count really + is `nseg * npts`) - directly against `u32[2]` across both files' full + segment lists: true in both, zero exceptions. Checked offset 48/80 + against: `nseg` (2674/2673), `npts` (68), `nseg * npts` (181832/181764), + total stream bytes (already confirmed as offset 56, not 48), total + payload bytes with segment headers stripped (192528/192456), total + body bytes with the envelope also stripped (181832/181764, matching + `nseg * npts` exactly since these are fully-flat files - a useful + independent confirmation that flat-mode body length truly is exactly + `npts` with no other overhead, but not a match for offset 48/80 + either), total 24-byte segment-header bytes (64176/64152), and the + `Status` stream's own varying fields (offset 4 confirms `nseg` + exactly, as already suspected, but offset 20 - `6784`/`6144` - turned + out to depend only on `nseg` too, identical between `..._1_63` and two + *different-content* real-signal files sharing the same `nseg`, so it + cannot be the source of offset 48/80's real-vs-real variation either). + **None matched.** Also tried an Internet-style (RFC 1071) 16-bit + ones'-complement running-sum checksum - initially looked promising + (one accidental exact match against `..._1_63`'s `Max Plot` stream + for offset 80), but this did not reproduce against `..._12_65` or + `..._34_73` with the same formula and is discarded as coincidental (a + flat file's checksum has limited entropy to begin with, making a + 1-in-3-file accidental match unsurprising). This strengthens, rather + than replaces, session 1's conclusion: offsets 48 and 80 are + genuinely content-dependent (not simple counts, not opaque + per-session tokens) but their exact algorithm remains unidentified + after both a broad standard-checksum sweep (session 1) and this + session's count/derived-size and Internet-checksum checks. + +**Verdict**: the block-floating-point/adaptive-scale hypothesis family, +in the four concrete forms tried, is ruled out - most instructively via +(c), which demonstrates a real within-file statistical signal that +fails a proper cross-file generalization check, and the quantified +false-positive base-rate finding explains *why* that kind of signal is +expected to appear by chance at this token-count scale. This does not +close the door on every possible adaptive-scale mechanism (a genuine +per-segment gain header using a transform more complex than the ones +tried in (a), or a marker mechanism at a byte offset within `A`/`tail` +subregions rather than uniformly across the concatenated body, remain +untested), but the most natural, simplest versions of the idea do not +fit. `CheckSum` offsets 48/80 remain the least-understood pair of +fields in this document; a future session should treat them as lower +priority than the per-value grammar itself unless a stronger structural +clue emerges (per session 1's recommendation, unchanged by this +session). + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), extending session 1's `common.py` +(`iter_segments`/`body_of`) with new `walk_fixed2tier` (2-tier +magnitude-threshold walker with a skippable header) and `walk_bfp` +(marker-bit block-floating-point walker) helper functions. + +## 2026-07-20 session 3: the "split" form's two regions are a real, exact 256-channel/remainder wavelength split (resolved); a promising marker-bit signal traced to a compensating-error artifact, not a decode + +Direct follow-up to session 2's region-local suggestion. While setting +up a per-region rerun of session 2's marker-bit test, this session first +had to establish how many of a stream's `npts` wavelength channels +actually live in region `A` versus region `tail` - previously confirmed +to exist (session 1's envelope work) but explicitly flagged as +"never explained" (session 1's factsheet). That turned out to have a +clean, exact answer, which is this session's first real result. The +second half of the session used that answer to retry session 2's +marker-bit hypothesis region-by-region, found a statistically genuine +(not-by-chance) signal in region `tail` - but then, applying the +"physical plausibility as validator" idea from this document's own +"further avenues" list for the first time, showed that signal does not +correspond to a correct per-channel decode. **The per-value payload +grammar remains undecoded**, but one previously-open structural +question is now closed, and a methodologically important cautionary +result is recorded precisely. + +- **Resolved: region `A` always holds exactly the first 256 wavelength + channels; region `tail` holds the remaining `npts - 256`.** Checked + directly against every flat/baseline segment (not just the first one) + in all four locally available split-form files: `MSV000084197/ + 20190607_NM16.lcd` (`npts=321`) and `PXD025121/1.lcd`, `/10.lcd`, + `/11.lcd` (`npts=327` each). At baseline, every value is exactly 1 + byte (session 1's already-confirmed flat-mode fact), so a flat + segment's region-`A`/`tail` *byte* lengths are also directly its + region *value counts* - and in **every flat segment checked across + all four files, `len(A) == 256` exactly, with zero exceptions** + (`321 - 256 = 65` values in `tail` for `MSV000084197`; `327 - 256 = + 71` for the three `PXD025121` files). This resolves session 1's + "genuinely open" question about what the two declared-length regions + mean: they are not an arbitrary or data-dependent split, or a + "coarse array + exception list" (already ruled out in session 1 by + the vocabulary check) - they are a fixed partition of the wavelength + axis at channel index 256, extremely plausibly reflecting a + 256-entry (`2^8`, i.e. one-byte-addressable) hardware buffer or + register width in the PDA detector's acquisition electronics. This + also gives a complete, satisfying answer to session 1's other open + question - why "split" vs. "symmetric" envelope form correlates with + wavelength count: **every corpus file with `npts <= 256` uses the + symmetric (single-region) form, and every file with `npts > 256` + uses the split (two-region) form**, consistent with a design where a + wavelength count that fits inside one 256-slot buffer needs no split, + and one that doesn't gets divided into "the first 256" plus "the + overflow." (This corpus has no file with `npts` near exactly 256 to + test the boundary itself, so the precise cutoff semantics - e.g. + whether `npts == 256` exactly would still split - remain unconfirmed, + but the pattern is exact and exceptionless for every `npts` value + actually present locally: `68`, `321`, `327`.) +- **Retrying session 2's marker-bit escape hypothesis per-region (now + that the true per-region value counts are known) finds a real, + well-controlled signal in region `tail` alone.** Using `n_A = 256` + and `n_tail = npts - 256` (exact, not estimated) to compute each + region's own `base_w`/`n_wide` independently, and sweeping the same + 32 marker-bit configurations from session 2 separately against each + region: **region `A` shows no comparable signal** (best config only + 29/3498 = 0.8% clean on `MSV000084197`, no better than session 2's + whole-body attempts). **Region `tail` shows something real**: with + `bit_pos=5, polarity=set-means-wide, marker-in-first-byte`, clean + (exact `n_tail`-token, zero-leftover) walks succeed on **2479/3498 + (70.9%) of `MSV000084197`'s real-mode segments**, and the *same* + configuration, unchanged, succeeds on **2040/6187 (33.0%)**, + **2129/6179 (34.5%)**, and **2384/6179 (38.6%)** of the three + `PXD025121` files respectively - a real cross-file signal, unlike + every marker-bit config tried against the symmetric-form `MTBLS432` + files in session 2. Two independent randomized controls confirm this + is not a base-rate artifact of the search space (the concern + session 2's false-positive-rate finding raised): (1) replacing each + segment's real tail bytes with **uniformly random bytes of the same + length** gives **0/3498 (0.0%)** clean on `MSV000084197` and + 1.4%-5.2% on the three `PXD025121` files (vs. 33-39% real); (2) the + stricter control - **shuffling each segment's own real bytes** (same + exact byte-value multiset, scrambled order, controlling for the + payload's well-documented non-uniform leader-byte distribution, not + just for byte value frequency) - gives **7/3498 (0.2%)** on + `MSV000084197`, vs. 70.9% for the real, unshuffled order. Real byte + *order*, not just byte *content*, matters to this rule succeeding - + genuine evidence of positional structure in region `tail`'s bytes. +- **But decoding actual values under this rule and checking them for + physical plausibility - the "further avenues" list's avenue 5, + attempted here for the first time - shows the walk is not finding + real per-channel boundaries.** Decoded all `n_tail` channel values + (raw little-endian integer per token) for every segment that walked + cleanly under the rule above, then measured each channel's + segment-to-segment smoothness (mean relative step size between + consecutive successfully-decoded segments; lower means smoother, + more chromatogram-like). **Channel index 0 of the tail region is + dramatically smooth (mean relative step 0.024)** - but **every other + channel (indices 1 through 64) ranges from 0.25 to 1.13**, + statistically indistinguishable from a temporally-shuffled-order + control's aggregate score (0.96) and from pure noise. Only one + channel out of 65 looks like real data; the other 64 do not. Traced + this to a concrete mechanism, not just a vague "it's noisy": in every + one of the **3445 segments** where the marker fires "wide" for + channel 0 (base 1 byte extended to 2), **the second byte of that + "2-byte" token is exactly `0x00`, 3445/3445 times, no exceptions**. + This is much more consistent with channel 0's real width being a + *plain 1 byte* (the marker misfiring on a data byte that happens to + have bit 5 set, which is common given the payload's already-documented + `0x20`/`0x3f`-leader-byte concentration) and the walk's "second byte" + actually being **channel 1's own real first byte** - which happens to + be `0x00` essentially every time, i.e. wavelength channel 257 in this + file is very plausibly a genuinely dead/unmonitored edge channel that + reads exactly zero. The walk still reaches the correct *total* token + count and byte length despite this misclassification because the + very next channel's true value is trivially small - a directly + observed, concrete instance of the **compensating-error phenomenon** + session 2's false-positive-rate finding predicted abstractly: passing + the "exact count, zero leftover" test (even one that also beats two + different randomized controls) does not guarantee the token + boundaries themselves are correct, only that *some* combination of + possibly-wrong boundary choices happened to sum to the right total. +- **Follow-up: confirmed unconditionally, cross-file, with no + marker-bit assumption at all.** Dropped the marker-bit framing + entirely and checked directly: across all four split-form files + (`MSV000084197` plus all three `PXD025121` files, **15,043 real-mode + segments total**), region `tail`'s raw byte at position 1 is exactly + `0x00` in **100.00% of segments, zero exceptions in any file**, and + the raw byte at position 0 stays in a narrow, smoothly-varying range + in every file (`87`-`141` across the four files' means of `107.3`- + `120.6`) fully consistent with the already-observed channel-0 + temporal smoothness. This is strong, clean, marker-bit-independent + evidence that region `tail` position 0 is genuinely a plain 1-byte + token and position 1 is a hard-wired/dead channel reading a constant + zero - reproducible across every file checked, not a `MSV000084197` + quirk. Byte position 2 onward, by contrast, is dominated by the same + `0x20`/`0x3f`/`0x3e`-ish leader-byte values already documented + throughout this document for the undecoded payload in general (checked + via a byte-value histogram at position 2: the four most common values, + accounting for the large majority of segments, are `0x3c`-`0x3f` + and `0x20`-`0x21`) - i.e. the genuine open grammar problem picks back + up at channel index 2, unchanged from the rest of this document; only + the first two channels of region `tail` are special-cased edge + channels, not a hint about the general per-value rule. +- **Verdict**: region `tail`'s bytes do carry genuine, non-random, + order-dependent structure that this specific marker-bit rule + partially captures (the real-vs-shuffled-byte comparison is + decisive on that point) - but the rule itself is not the correct + per-channel grammar, as the physical-plausibility check demonstrates + concretely rather than just abstractly. This is a case where a + hypothesis survives a strong statistical control yet still fails a + physical-plausibility check, which is exactly why this document's + "further avenues" list included the physical-plausibility validator + in the first place - and it is the first time that validator has + actually been applied to a candidate decode, rather than remaining a + suggestion with nothing yet to validate. A future session picking + this up should not restart from session 2's whole-body marker-bit + framing; the more promising, narrower thread is region `tail` + specifically, and the concrete clue that channel 257 (tail-region + index 1) reads as a near-constant `0x00` across thousands of segments + in this file is a specific, reproducible, low-effort thing to verify + and build on directly (e.g. check whether it holds in the + `PXD025121` files too, and whether channel 0 is genuinely 1 byte wide + in the great majority of segments once decoded without the spurious + marker). + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), extending session 2's `walk_bfp` with a +`split_regions` helper (extracts `A`/`tail` sub-bodies from a "split" +form payload using the already-confirmed `A`/`tail` length prefix/ +suffix fields) and a `decode_bfp` variant that returns actual decoded +values (not just walk success) for the physical-plausibility check. + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -1413,6 +1775,27 @@ Beyond the single recommendation in the 2026-07-19 closing summary are additional, not-yet-tried directions - none executed this round, so treat them as leads, not findings: +- **(New, from 2026-07-20 session 3; partially followed up already - + see below) Push the region-`tail` walk past channel index 2.** + Session 3 confirmed, unconditionally and across all four split-form + files (15,043 segments, zero exceptions), that region `tail` position + 0 is a plain 1-byte token in a narrow smoothly-varying range and + position 1 is a hard-wired constant `0x00` - real, reproducible, + edge-channel behavior, not a coincidence of one file's chemistry (so + the "check it holds in `PXD025121` too" step from this bullet's + original framing is now done). What's still untried: manually, + by-hand decoding region `tail` starting from `base_offset = 2` (i.e. + treating positions 0-1 as a solved 2-byte fixed prefix and *not* + applying any marker-bit assumption to them), to see how far a + directly-inspected, no-global-rule-assumed walk gets through the + remaining `n_tail - 2` channels before hitting the first genuine + ambiguity - channel index 2 onward reverts to the same + `0x20`/`0x3f`-leader-byte-dominated bytes as the rest of this + document's undecoded payload, so this is really a narrower, smaller + restatement of the core open problem, not a shortcut around it, but + the much smaller channel count (63-69 remaining channels instead of + 256-327) may make manual inspection more tractable than it is against + a full segment. - **Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` stream instead of (or before) `PDA 3D Raw Data`.** It's populated, structurally simpler (one giant segment per channel, not thousands of @@ -1498,3 +1881,28 @@ treat them as leads, not findings: checking zero-leftover exactness - could help disambiguate between otherwise-tied candidate parses, the same way a human would sanity- check a chromatogram by eye. +- **(New, from 2026-07-20 session 2) Region-local rather than + whole-body marker placement for the block-floating-point/marker-bit + family.** Session 2's item (c) tested a per-token marker bit against + the whole concatenated body (the `A`+`tail` regions treated as one + token stream, per session 1's "no vocabulary difference between + regions" finding). It did not try treating `A` and `tail` as two + *independently scaled* blocks, each with its own `base_w`/`n_wide` + derived from its own sub-length rather than the combined body length - + a real block-floating-point scheme could plausibly reset its + effective scale at the region boundary even if the byte-level + vocabulary looks similar on both sides (vocabulary similarity says + nothing about whether the *width-selection arithmetic* is shared). + This is a cheap variant of an already-built test (`walk_bfp` from + session 2) - re-run it with `A` and `tail` split before computing + `base_w`/`n_wide` for each, rather than merged. +- **(New, from 2026-07-20 session 2) A non-linear or table-driven + transform from an explicit per-segment header to a threshold.** + Session 2's item (a) only tried using a candidate header byte + *directly* as a threshold (or its bitwise complement). A real gain + header more plausibly indexes a small lookup table (as in ADPCM step + tables) rather than being usable as a raw magnitude cutoff - this + wasn't tried since it requires guessing the table itself, which is a + much larger search than session 2's budget allowed, but is a more + faithful test of the "explicit per-segment scale field" idea than + what was actually tried. diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index 7ef6406..074e0e0 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -293,5 +293,12 @@ the `3D Raw Data` and `Max Plot` streams (not a flat/real flag as previously read - see the correction note and dated session in `docs/format/04`), and ruled out 19 standard CRC-16 polynomials plus several other common checksum constructions for the remaining 2 fields. -Neither finding decodes the payload; the per-value grammar is still -open. +A follow-up pass the same day resolved a separate open structural +question - the "split" envelope form's two declared-length regions are +an exact 256-wavelength-channel/remainder split, which also explains +why "split" vs. "symmetric" envelope form correlates with wavelength +count - and ruled out a block-floating-point/adaptive-scale hypothesis +family (including one candidate that passed two randomized-control +checks but was then shown, via a physical-plausibility check, to be a +compensating-error artifact rather than a real decode). None of these +findings decode the per-value payload; that grammar is still open. From 588faff86c52e839dbc0f8874f3261ed9a896b16 Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 16:14:41 +0000 Subject: [PATCH 3/6] docs: re-run PDA decode sweeps with corrected per-region counts, fix plausibility-check gap (#2) Follows up on the 256-channel region-boundary finding from the prior commit on this branch: the original per-region sweep (well before that boundary was known) isolated region A but still targeted npts tokens from it, an internally inconsistent test now that region A is known to hold only the first 256 of npts channels. Re-runs the magnitude- threshold and continuation-bit sweeps against both regions with the corrected target counts, plus a fresh sweep against MTBLS432 (whose symmetric envelope form has no region split to get wrong in the first place). Two sweeps produce dramatic-looking signals - a sharp threshold cliff in region tail beating two randomized controls across all four split-form files, and a 67.8%-clean single-file MTBLS432 result with an apparently smooth per-channel decode - and both are run through to actual value decoding and a physical-plausibility check rather than reported on the strength of a clean walk rate alone. Neither survives: the region-tail cliff reproduces the same channel-0/1 compensating- error artifact diagnosed in the prior commit, now shown to recur under a completely different-looking rule; the MTBLS432 result exposes a real gap in the existing physical-plausibility check itself, which reported smoothness for a decode that was actually 80-96% the same repeated value with rare large jumps to essentially random numbers. The check is revised to also flag high per-channel mode fraction, since mean relative step alone can't tell a frozen, repetitive decode from a genuinely smooth one. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu --- CHANGELOG.md | 13 +- docs/format/04-lcd-chromatogram-pda.md | 206 ++++++++++++++++++++++--- docs/format/06-known-limitations.md | 12 +- 3 files changed, 209 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 049a266..c41a269 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,8 +34,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 with wavelength count); and showed, via a physical-plausibility check, that a marker-bit signal which passed two randomized-control tests was nonetheless a compensating-error artifact rather than a real decode. - The per-point payload grammar itself is still undecoded - see - `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-3 for + A follow-up pass re-ran the threshold/continuation-bit sweeps with the + now-corrected per-region target counts, found two more dramatic- + looking single-file/single-region signals, and ran both all the way + through to actual value decoding before ruling them out too (one + reproduced the same compensating-error artifact under a sharper + threshold framing; the other exposed a real flaw in the existing + physical-plausibility check itself - a decode that repeats the same + value 80-96% of the time can look "smooth" by mean relative step + alone - which the check has now been revised to catch). The per-point + payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-4 for full detail. ## [0.1.0] - 2026-07-18 diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 33ca5e0..0122265 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -84,6 +84,14 @@ is a scannable summary, not a substitute for the detailed sections below but no working decoder has been built from it yet; the joint-DP's scoring function is provably not selective enough (width agreement isn't monotonic with solution cost). +- **Methodological: a physical-plausibility (temporal-smoothness) check + must also test each channel's mode fraction, not just mean + relative step.** A decode that is a frozen, repeated value most of + the time with rare large jumps trivially minimizes mean relative step + while being the opposite of real chromatography; 2026-07-20 session 4 + found this the hard way (a 67.8%-clean, apparently-smooth single-file + `MTBLS432` result turned out to be up to 96% mode-dominated per + channel) and revised the check accordingly. **Ruled out** (see "This session's additional ruled-out hypotheses" and "further ruled-out hypotheses" for full detail): standard unsigned @@ -110,7 +118,15 @@ marker-bit escape rule, despite passing two randomized-control checks (uniform-random bytes and same-multiset-shuffled bytes), traced via a physical-plausibility (temporal-smoothness) check to a compensating- error artifact affecting all but one of 65 channels, not a real -per-channel decode (see 2026-07-20 session 3). +per-channel decode (see 2026-07-20 session 3); the corrected-target- +count magnitude-threshold and continuation-bit sweeps against region +`A` (target 256) and region `tail` (target `npts - 256`) alone - region +`tail`'s sharp-cliff threshold=32 hit reproduces session 3's exact +channel-0/1 artifact under a different rule, and a fresh `MTBLS432` +sweep's single-file 67.8%-clean/"66-of-68-channels-smooth" result +failed cross-file generalization and was shown to be a smoothness- +metric artifact (mode-dominated decoded values, not real drift) rather +than a genuine decode (see 2026-07-20 session 4). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric @@ -1739,6 +1755,158 @@ form payload using the already-confirmed `A`/`tail` length prefix/ suffix fields) and a `decode_bfp` variant that returns actual decoded values (not just walk success) for the physical-plausibility check. +## 2026-07-20 session 4: re-running the threshold/continuation-bit sweeps with the corrected per-region target counts finds two more dramatic-looking cliffs, both root-caused to artifacts - plus a methodological correction to the physical-plausibility check itself + +Direct follow-up to session 3, prompted by a specific observation: +session 1's original "per-region parsing instead of whole-body parsing" +sweep (in the "further ruled-out hypotheses" section, well before +session 3 established the 256-channel region boundary) isolated region +`A` but still required it to decode to exactly `npts` tokens - an +internally inconsistent test, since region `A` is now known to hold +only the first 256 of `npts` channels, not all of them. This session +re-ran the magnitude-threshold and continuation-bit sweeps against both +regions with the corrected target counts (`256` for region `A`, +`npts - 256` for region `tail`), plus a fresh sweep against `MTBLS432` +(no region split to get wrong in the first place, since `68 <= 256`). +**No working decode was found.** Two sweeps produced dramatic, +cliff-shaped signals that initially looked like real hits - both were +run all the way through to actual value decoding and a +physical-plausibility check (per this document's own established +practice) and both turned out to be artifacts, for two different +reasons. The second of those reasons is a genuine methodological +correction to the physical-plausibility check itself, worth any future +session internalizing before trusting a "smooth" result. + +- **Region `A` (target=256): no signal, at either sweep.** Magnitude + threshold: best `239/3498 (6.8%)` at `threshold=31, wide_w=2` - + barely above the levels already documented as noise elsewhere in this + document. Continuation-bit: best `76/3498 (2.2%)`. Physical + plausibility of the threshold sweep's best candidate: only `2/256` + channels smooth by the (subsequently-revised, see below) smoothness + metric - consistent with edge-channel artifacts, not a real decode. + Region `A`, even at its correct 256-channel target, remains as + undecoded as the rest of this document's whole-body attempts. +- **Region `tail` (target = `npts - 256`): a dramatic, razor-sharp cliff + - traced to the same channel-0/1 artifact session 3 already + diagnosed, not new structure.** The magnitude-threshold sweep found + something that initially looked like a breakthrough: `threshold=32, + wide_w=2` decodes **2518/3498 (72.0%)** of `MSV000084197`'s region + `tail` cleanly, and the *exact same* configuration reaches **33.4%, + 35.3%, and 40.2%** on the three `PXD025121` files - a real cross-file + signal, confirmed against both a uniform-random-byte control + (0.1%-9.4% across the four files) and a same-multiset-shuffle control + (4.7%-14.8%), both decisively beaten. Critically, the sweep shows a + **razor-sharp cliff, not a gradual peak**: `71.9%` at `threshold=31` + down to essentially `0%` at `threshold=33` for the whole-tail walk (a + drop of two full orders of magnitude over a threshold change of `1`), + which mechanically traces to `0x20` (decimal `32`) being the single + most common byte value in this payload (already documented + extensively elsewhere in this document) - flipping the threshold from + `32` to `33` reclassifies every occurrence of that one very common + byte value from "wide" to "narrow" at once, which is enough on its own + to explain a sharp transition without it necessarily marking a true + semantic boundary. Decoding actual values and checking channel-by- + channel temporal smoothness reproduced **exactly** session 3's + finding: channel 0 alone is smooth (mean relative step `0.024`, all + four files), every other channel is noise-like (`0.6`-`1.1`). Directly + confirmed this is the *same* artifact, not a coincidentally similar + one: `byte>=32` and `bit5-set` (session 3's marker rule) agree on + `82.4%` of individual bytes checked, channel 0 is classified "wide" + in **100%** of segments under this rule too, and its supposed second + byte is `0x00` in 100% of those cases - the identical compensating- + error mechanism session 3 root-caused, now shown to reproduce under a + completely different-looking rule (a plain magnitude threshold, not a + specific bit position), which is itself informative: **the artifact + is robust to how the "wide" decision is framed, because it isn't + really about a decision rule at all - it's a structural fact about + channel 1 always being a hard-zero edge channel, which any rule that + classifies channel 0 as "wide" will trip over the same way.** + Stripping the confirmed 2-byte fixed prefix (channel 0 = 1 byte, + channel 1 = constant `0x00`) and re-sweeping the remaining `n_tail - + 2` channels with the corrected target found only a smooth, gradual + peak (not a cliff) topping out at **976/3498 (27.9%)** at + `threshold=27` - and decoding that candidate showed **0 of 63** + channels smooth, with per-channel mode-fractions of only `0.7%-1.9%` + (i.e. genuinely close to uniformly random, the opposite of + mode-dominated). Channels 2 and beyond in region `tail` remain + undecoded. +- **`MTBLS432` (symmetric form, no region-boundary confound): a + single-file result that looked like a genuine decode under this + document's existing smoothness metric, and was not - which surfaced a + real flaw in the metric itself.** A fresh magnitude-threshold sweep + against `..._12_65...lcd`'s 2662 real-mode segments found + `threshold=1, wide_w=3` (i.e. "a literal `0x00` byte is a 1-byte + zero, anything else starts a 3-byte token") decodes **1805/2662 + (67.8%)** cleanly - and, alarmingly, the existing mean-relative-step + smoothness check reported **66 of 68 channels "smooth"** (values + `0.03`-`0.36`, only two channels above the `0.3` cutoff this document + has been using). Before accepting this, cross-file testing (the same + discipline session 2/3 already established) was run first: the exact + same configuration reaches only **14/2663 (0.5%)** clean on + `..._34_73...lcd`, **612/2663 (23.0%)** on `..._26_68...lcd`, and + **363/2662 (13.6%)** on `..._21_29...lcd` - all four files sharing + the same instrument, method, and `npts=68` - an immediate + disqualifying sign on its own. But the more important finding came + from asking *why* `..._12_65...lcd` alone looked so clean: **the + mean-relative-step metric is fooled by highly repetitive (mode- + dominated) decoded sequences.** Checking the value distribution per + channel (not just the step-to-step differences) shows every one of + the 68 channels has a single dominant repeated value accounting for + **41%-96% of its 1805 decoded segments** (most channels 80%-95%), with + the *non-repeated* values landing all over an essentially uniform + 24-bit range (min/max spanning `~10^4` to `~1.67*10^7`, no + concentration) rather than smoothly drifting from the dominant value. + A signal that is "the same fixed number 90% of the time, then jumps + to an unrelated large random-looking number 10% of the time" trivially + minimizes a *mean* relative-step statistic (nearly all steps are + exactly zero) while being the *opposite* of physically plausible + chromatography, which should show continuous, non-repeating drift as + a peak rises and falls. This is consistent with, and mechanically + explained by, an already-documented fact: `MTBLS432` real-mode body + length is tightly centered near `3 * npts` bytes (the "width-table + retry" session, well before this one), so a rule that treats nearly + every byte as the start of a 3-byte token will often land on the + *correct total length* by roughly chunking the body into thirds, + without the chunk boundaries corresponding to genuine per-channel + values - and apparently tends to re-derive the *same* misaligned chunk + boundary (hence the same wrong "value") on the majority of segments + in this one file specifically, which is plausible given how much of + this file's own body-length distribution is already known to cluster + tightly (`194` bytes in `1767/2674` segments per the factsheet). A + continuation-bit sweep against the same file found nothing better + (`34/2662`, `1.3%`). **This route is closed for `MTBLS432` as tried.** +- **Methodological correction, worth recording precisely so it doesn't + recur**: this document's physical-plausibility check (introduced in + session 3) computed only the mean relative step between consecutive + decoded values per channel. This session shows that statistic alone + is insufficient - it can be minimized by a decode that is mostly a + frozen, repeated (mode-dominated) value with rare large excursions, + which is not real physical smoothness. **Any future + physical-plausibility check should also report each channel's mode + fraction** (the proportion of decoded values equal to that channel's + single most common value) **and treat a high mode fraction (this + session saw up to 96%) as disqualifying, not confirming** - genuine + chromatographic data should show low repetition and continuous drift, + not a dominant constant. Both of this session's sweeps were re-checked + against this corrected two-part test (low mean relative step *and* low + mode fraction); neither survived it. + +**Verdict**: the corrected per-region target counts did not produce a +working decode, and the two most promising-looking candidates this +session found were both run through to real value decoding and a +(now-corrected) physical-plausibility check rather than being reported +as hits on the strength of a clean walk rate or a sharp threshold cliff +alone - both failed once actually inspected, for concretely identified, +different reasons. The per-value payload grammar for both the split and +symmetric envelope forms remains undecoded. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), building `region_sweep.py` (generalized magnitude- +threshold and continuation-bit walkers/sweepers parameterized by target +token count and optional header skip, reusable against any region) and +`decode_threshold.py` on top of session 3's `regions.py` helper. + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -1775,27 +1943,29 @@ Beyond the single recommendation in the 2026-07-19 closing summary are additional, not-yet-tried directions - none executed this round, so treat them as leads, not findings: -- **(New, from 2026-07-20 session 3; partially followed up already - - see below) Push the region-`tail` walk past channel index 2.** +- **(New, from 2026-07-20 session 3; followed up in session 4 with a + negative result) Push the region-`tail` walk past channel index 2.** Session 3 confirmed, unconditionally and across all four split-form files (15,043 segments, zero exceptions), that region `tail` position 0 is a plain 1-byte token in a narrow smoothly-varying range and position 1 is a hard-wired constant `0x00` - real, reproducible, - edge-channel behavior, not a coincidence of one file's chemistry (so - the "check it holds in `PXD025121` too" step from this bullet's - original framing is now done). What's still untried: manually, - by-hand decoding region `tail` starting from `base_offset = 2` (i.e. - treating positions 0-1 as a solved 2-byte fixed prefix and *not* - applying any marker-bit assumption to them), to see how far a - directly-inspected, no-global-rule-assumed walk gets through the - remaining `n_tail - 2` channels before hitting the first genuine - ambiguity - channel index 2 onward reverts to the same - `0x20`/`0x3f`-leader-byte-dominated bytes as the rest of this - document's undecoded payload, so this is really a narrower, smaller - restatement of the core open problem, not a shortcut around it, but - the much smaller channel count (63-69 remaining channels instead of - 256-327) may make manual inspection more tractable than it is against - a full segment. + edge-channel behavior. Session 4 followed up automatically (stripping + the 2-byte prefix, re-running the magnitude-threshold and + continuation-bit sweeps against the remaining `n_tail - 2` channels + with the correspondingly corrected target count): found only a weak, + gradual (not sharp-cliff) peak topping out at 27.9% clean, and the + decoded values under that candidate are indistinguishable from random + noise (0 of 63 channels physically plausible, per-channel mode + fractions of only 0.7%-1.9%). The automated sweep infrastructure does + not find anything here. What remains genuinely untried is fully + manual, by-hand byte inspection of a handful of individual segments' + channel-2-onward bytes (not a parameterized sweep) - channel index 2 + onward reverts to the same `0x20`/`0x3f`-leader-byte-dominated bytes + as the rest of this document's undecoded payload, so this is really a + narrower, smaller restatement of the core open problem, not a + shortcut around it, but the much smaller channel count (63-69 instead + of 256-327) may still make manual inspection more tractable than it + is against a full segment. - **Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` stream instead of (or before) `PDA 3D Raw Data`.** It's populated, structurally simpler (one giant segment per channel, not thousands of diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index 074e0e0..cd187d6 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -300,5 +300,13 @@ why "split" vs. "symmetric" envelope form correlates with wavelength count - and ruled out a block-floating-point/adaptive-scale hypothesis family (including one candidate that passed two randomized-control checks but was then shown, via a physical-plausibility check, to be a -compensating-error artifact rather than a real decode). None of these -findings decode the per-value payload; that grammar is still open. +compensating-error artifact rather than a real decode). A further +follow-up pass re-ran the decode sweeps with the corrected per-region +target counts this finding implies, found two more signals that looked +promising at first (one a sharp threshold cliff, one a 67.8%-clean +single-file result), and ruled both out after decoding actual values - +the second of which exposed and fixed a real gap in the +physical-plausibility check itself (a mode-dominated, mostly-repeated +decode can look deceptively "smooth" under a naive mean-step metric). +None of these findings decode the per-value payload; that grammar is +still open. From 9c55dcd7398163073a6e4b40416ab232f61ff2a8 Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 16:49:17 +0000 Subject: [PATCH 4/6] docs: re-check PDA region-A entropy, re-test joint decoder with region-correct counts (#2) Follows up on two leads that fall out of the 256-channel region boundary and the mode-fraction lesson from the prior two commits on this branch. Re-runs the per-byte-position entropy and conditional-entropy analysis with region A cleanly isolated at its true 256-channel boundary. The region A/tail byte ranges were already correct in the earlier analysis (they come from the envelope's own length-prefix fields, established well before the channel-count confusion that affected the decode sweeps specifically), so this mostly reproduces the earlier session's numbers almost exactly rather than finding anything new - a genuine confirmation, not a wasted check, since it rules out the possibility that a marker byte or periodicity was being smeared out by a region misalignment. No periodicity at 2- or 3-byte spacing is found anywhere in region A's true span beyond the already-explained early-channel low-variance artifact. Rebuilds the joint temporal+magnitude dynamic program (the original was not saved) scoped to the correct per-region target counts, and adds an optional cost-function term that penalizes bit-identical decoded values between two segments, directly targeting the mode-collapse failure mode diagnosed in the prior commit. Re-testing the original single-pair "true beats random" claim across 20 pairs instead of one finds the underlying signal real but much weaker than the original anecdote suggested (true pair cheaper in only 12 of 20 individual pairs, versus a single dramatic 1.8x example previously). The anti-mode-collapse penalty does not improve, and on this sample appears to worsen, the correlation between solution cost and width agreement - a concrete negative result for the specific fix tried, not a validation of it. Region A's larger target count is computationally intractable for the exact DP as implemented, confirming the scaling limitation the original session anticipated. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu --- CHANGELOG.md | 14 +- docs/format/04-lcd-chromatogram-pda.md | 235 ++++++++++++++++++++++++- docs/format/06-known-limitations.md | 11 +- 3 files changed, 249 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c41a269..5f7ee90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,9 +42,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 threshold framing; the other exposed a real flaw in the existing physical-plausibility check itself - a decode that repeats the same value 80-96% of the time can look "smooth" by mean relative step - alone - which the check has now been revised to catch). The per-point - payload grammar itself is still undecoded - see - `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-4 for + alone - which the check has now been revised to catch). A further + pass re-checked the earlier per-byte-position entropy analysis with + region A's now-known 256-channel boundary (reproduces the prior + numbers almost exactly, no new periodicity found) and re-ran the + joint temporal+magnitude decoder with region-correct target counts + and an anti-mode-collapse cost term - the underlying temporal- + correlation signal replicates but is markedly weaker than the + original single-pair anecdote suggested, and the new cost term did + not improve the decoder's previously-diagnosed selectivity problem. + The per-point payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-5 for full detail. ## [0.1.0] - 2026-07-18 diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 0122265..826bf85 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -83,7 +83,21 @@ is a scannable summary, not a substitute for the detailed sections below from-scratch joint-decode DP ("joint temporal+magnitude decoder") - but no working decoder has been built from it yet; the joint-DP's scoring function is provably not selective enough (width agreement - isn't monotonic with solution cost). + isn't monotonic with solution cost). **Caveat added 2026-07-20 session + 5**: the joint-DP finding's effect size was originally based on a + single true-pair-vs-random-pair anecdote (`1.8x` cost ratio); a + 20-pair re-test on region `tail` reproduces the *direction* but at a + much smaller, less certain effect size (`~1.13x` mean ratio, true + pair cheaper in only 12/20 individual pairs) - cite this as a weak, + aggregate-level signal, not a strong or reliable pair-by-pair one. +- **The region-A-isolated per-byte-position entropy analysis (2026-07-20 + session 5) reproduces the earlier whole-stream entropy session's + numbers almost exactly** (marginal/conditional entropy within 0.01 + bits) and finds no periodic marker at either 2- or 3-byte spacing + across region `A`'s full true 256-channel span - confirming the + earlier entropy work was already correctly scoped and the + low-entropy-near-`0x40`-at-residue-1 pattern really is a low-variance + early-wavelength artifact, not a token-boundary marker. - **Methodological: a physical-plausibility (temporal-smoothness) check must also test each channel's mode fraction, not just mean relative step.** A decode that is a frozen, repeated value most of @@ -126,7 +140,15 @@ channel-0/1 artifact under a different rule, and a fresh `MTBLS432` sweep's single-file 67.8%-clean/"66-of-68-channels-smooth" result failed cross-file generalization and was shown to be a smoothness- metric artifact (mode-dominated decoded values, not real drift) rather -than a genuine decode (see 2026-07-20 session 4). +than a genuine decode (see 2026-07-20 session 4); a fixed +bit-identical-value penalty term added to the joint temporal+magnitude +DP's cost function, intended to discourage mode-collapsed solutions - +did not improve (and on a 20-pair sample, worsened) the correlation +between solution cost and width agreement, so does not fix the +DP's previously-diagnosed selectivity weakness (see 2026-07-20 session +5); mod-2 and (beyond the already-explained early-channel effect) +mod-3 periodicity in region `A`'s per-byte-position entropy (see +2026-07-20 session 5). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric @@ -1907,6 +1929,202 @@ threshold and continuation-bit walkers/sweepers parameterized by target token count and optional header skip, reusable against any region) and `decode_threshold.py` on top of session 3's `regions.py` helper. +## 2026-07-20 session 5: region-A-isolated entropy re-check (reproduces, doesn't extend, the earlier finding); a corrected-target joint DP with an anti-mode-collapse penalty term, tested more rigorously than the original single-pair anecdote + +Two follow-ups, both aimed at leads that fall directly out of the +256-channel region boundary (session 3) and the mode-fraction lesson +(session 4). **Neither produced a working decode.** Both are honest, +fully-executed negative results with genuine new information in them +(a reproduced-not-contradicted entropy characterization, and a more +rigorous re-test of a claim this document had previously only supported +with a single-pair anecdote), not abandoned attempts. + +### Region-A-isolated entropy analysis: reproduces the earlier whole-stream numbers almost exactly, confirms no periodic marker + +The 2026-07-19 per-byte-position entropy session ran before the +256-channel region boundary was known, but - important scoping fact +worth stating precisely - that session's "region A" and "region tail" +labels were *already* byte-accurate: the `A`/`tail` byte ranges come +from the envelope's own length-prefix fields (confirmed since session +1), which session 3 never touched. What session 3 corrected was the +*target token count* used by the threshold/varint decode sweeps, not +which bytes belong to region `A`. So re-running the entropy analysis +with the "correct" region boundary was a real thing to check (the +earlier session's byte extraction could in principle still have been +subtly wrong), but was not guaranteed to change anything - and it +didn't: + +- **Marginal and conditional entropy reproduce the 2026-07-19 figures + to three decimal places.** Region `A`: `H(byte) = 6.281` bits, + `H(next|current) = 5.409` bits (13.9% reduction) - versus the + earlier session's `6.279` / `5.408` (13.8%). Region `tail`: `H(byte) + = 5.793`, `H(next|current) = 4.696` (18.9% reduction) - versus the + earlier session's `5.783` / `4.697` (18.8%). Region `A`'s + individual-vs-concatenated compression ratio (`93.3%` vs `77.9%`) + also matches the earlier session's numbers closely. This is a + genuine confirmation, not a redundant no-op: it directly verifies the + entropy session's region extraction was already correct, so no + periodicity or marker was being smeared out or hidden by a region + misalignment that has now been fixed - the entropy characterization + itself needed no correction. +- **New check: region `tail` conditional entropy with the two known + edge channels (position 0 and 1, identified in session 3) excluded**, + to see whether they were inflating the "genuine structure" finding. + Channels 2 onward: `H(byte) = 5.753`, `H(next|current) = 4.726` + (17.9% reduction) - only marginally lower than the full-region figure + (18.9%). The edge channels contribute a small amount to the + conditional-entropy reduction (unsurprising, since channel 1 being a + hard-wired constant given channel 0 is trivially predictable) but + most of the "genuine local structure, not noise" finding survives + their removal - real structure exists in the still-undecoded + channels 2+ too, consistent with session 4's finding that those + channels are not simply noise-passing-as-signal, just not yet + decoded. +- **No new periodicity found at region `A`'s true 256-channel scale.** + Checked mod-3 (the residue class the original entropy session flagged + and then explained away as a low-variance-early-channel artifact, not + a token marker) across the *entire* region `A` span (not just the + first 40 bytes): the low-entropy-at-residue-1 pattern is strong only + for roughly the first 60 bytes (entropy `2.1`-`3.7` bits, one byte + value `0x40` accounting for up to 75% of occurrences) and fades to a + flat `~5.3`-`5.7` bits at all three residues by position `~120` and + beyond - reproducing, not contradicting, the earlier session's + "low-variance early wavelengths, not a 3-byte-token marker" + conclusion, now checked against the true 420-536-byte region `A` span + rather than an approximate one. Also checked mod-2 (motivated by + region `A`'s own average width, `1.89`-`1.91` bytes/channel, being + closer to 2 than 3): no distinction between residues at any position + range checked (`[0,40)`, `[40,120)`, `[120,250)`, `[250,420)`) - both + residues track each other closely everywhere, ruling out a period-2 + marker just as cleanly as period-3 was already ruled out. + +**Verdict**: this was a legitimate check to run (the region boundary +*could* have changed the entropy picture), but it didn't - the earlier +session's entropy characterization holds up exactly, and no +periodicity at either of the two most plausible token widths (2 or 3 +bytes, matching region `A`'s and region `tail`'s own observed average +widths) was found anywhere in region `A`'s true span. If a marker byte +exists, it is not visible as a fixed-position, fixed-periodicity +entropy dip - consistent with (not a new contradiction of) the +transition-segment-comparison session's earlier "no marker byte found" +conclusion for a different part of this problem. + +### Joint temporal+magnitude DP, re-scoped to region-correct target counts, with an anti-mode-collapse cost term + +The 2026-07-19 joint-DP session built and ran its dynamic program +before the 256-channel boundary was known, walking the *combined* +`A`+`tail` body as one `npts`-token problem. Rebuilt the DP from +scratch this session (the original implementation was not saved, +consistent with that session's own "ad hoc, not saved" note) with two +corrections: region `A` and region `tail` scored as independent +sub-problems with their own correct target counts (`256` and `npts - +256` respectively), and an optional cost term that adds a fixed penalty +whenever the two segments' decoded values for a channel come out +*bit-identical*, directly targeting the mode-collapse/frozen-channel +failure mode session 4 diagnosed (a real decode should show continuous +drift, not exact repeats). + +- **Region `tail` (small enough to run at scale - `nT=65`, ~2-3 + seconds/pair): the true-vs-random-pair effect reproduces in + aggregate, but is markedly weaker and less certain than the original + session's single-pair anecdote implied.** The 2026-07-19 session's + entire evidentiary basis for "true neighbors are more self-consistent + than random pairs" was **one** true pair versus **one** random pair + (segment 4-vs-5, cost `2.3789`, versus segment 4-vs-1000, cost + `4.3407`, a `1.8x` difference). This session re-ran the comparison + across **20 true-neighbor pairs and 20 matched random-pair controls** + (same anchor segment, a random partner at least 50 segments away), + with `exact_match_penalty=0` (i.e. the original, unmodified cost + function, applied for the first time to region `tail` alone at its + correct target count). The aggregate direction replicates - mean + cost/channel `0.0113` (true) vs `0.0128` (random), true lower as + expected - but the *effect size* is much smaller than the original + `1.8x` anecdote (`~1.13x` here), and a **per-pair sign test shows + true-neighbor cost was lower in only 12 of 20 pairs (60%)** - not + distinguishable from chance at this sample size. This is a real, + useful correction to how confidently this document should state the + original finding: the *direction* of the temporal-correlation claim + holds up under a more rigorous multi-pair re-test, but the + *magnitude* the original single anecdote suggested was likely + optimistic, and a future session citing this finding should say + "weak, aggregate-level signal, not reliable pair-by-pair" rather than + repeating the original `1.8x` figure as if it were typical. +- **The anti-mode-collapse penalty term did not improve, and on this + sample appeared to worsen, the DP's selectivity - a negative result + for the specific fix tried, not a validation of it.** The original + session's diagnosed weakness was that width agreement does not track + solution cost monotonically. As a proxy for "does the objective + reward genuinely-correct-looking solutions," computed the correlation + between per-pair cost/channel and per-pair width agreement across the + same 20 true-neighbor pairs: with no penalty (`exact_match_penalty = + 0`), correlation is `-0.135` (weakly in the hoped-for direction - + lower cost mildly associated with higher agreement); with the + penalty active (`exact_match_penalty = 1.0`), correlation flips to + `+0.338` (moderately in the *wrong* direction - lower cost associated + with *lower* agreement). This is the opposite of what the penalty + term was designed to achieve. It is a small sample (20 pairs, one + file, one penalty value) and should not be read as definitively + ruling out every possible anti-repeat cost term, but the specific, + concretely-implemented version tried this session - a fixed penalty + for bit-identical decoded values between two segments - does not fix + the selectivity problem and should not be assumed to without + evidence. **This route, as implemented, is closed.** +- **Region `A` (target=256): computationally intractable for the exact + DP as implemented, confirming and sharpening the original session's + own noted limitation.** The original whole-body DP already reported + needing 100-120 seconds per pair at `npts=321` on the combined + `A`+`tail` body. Region `A` alone, at its corrected target of 256 + channels, is worse: a single true-neighbor pair (body lengths 441 and + 443 bytes) did not converge within a 3,000,000-state budget in 50 + seconds of search - state space growth outpaced the feasibility + pruning that kept the original whole-body DP tractable (peaking at + "only" 59,220 states there). This is plausibly because region `A`'s + own average width (`~1.7`-`1.9` bytes/channel, closer to a boundary + between width-1 and width-2 tokens) leaves the width-1-vs-2 choice + ambiguous at many more positions than the combined-body case did, + which is exactly the kind of denser branching that defeats this + pruning strategy. Not pursued further at larger state budgets this + session, given the already-substantial time cost for a single pair + with no sign of convergence; a genuinely faster (vectorized, or + non-Python) implementation - which the original session's own + recommendation already anticipated needing for a longer chain - would + need to come first before region `A` specifically is tractable for + this method at all. + +**Verdict**: both corrections from this session's two leads were +implemented and tested honestly, neither produced a working decode, and +the more rigorous multi-pair re-test of the original DP's headline +claim found the underlying temporal-correlation signal real but weaker +than previously documented - a useful correction for any future session +citing that finding, even though it doesn't change this document's +overall undecoded status. + +**What's left, in the requester's own words, if a future session wants +to keep going here**: (1) a faster (non-Python, e.g. Rust or a +vectorized numpy formulation) joint-DP implementation, to make region +`A`'s larger state space and a longer multi-segment chain (beyond +pairs) actually tractable - both were already recommended by the +2026-07-19 session and remain the single most concrete unblocked next +step; (2) a different, non-fixed-penalty formulation of the +anti-mode-collapse idea (e.g. penalizing a *run* of 3+ consecutive +identical values specifically, rather than any single bit-identical +pair, since real adjacent samples of a slowly-varying signal can +legitimately coincide occasionally) - this session tested only the +simplest version and found it unhelpful, not every version; (3) fully +manual, by-hand byte inspection of a handful of segments (not a +parameterized sweep or DP), which no session including this one has +actually done - every attempt to date has been an automated search over +some hypothesis family, and the corpus of things automated search +finds unpromising is now large enough that manual inspection may turn +up structure a parameterized search wasn't shaped to find. + +Scripts: `regions.py` (session 3, reused), `joint_dp.py` (new this +session - a from-scratch reimplementation of the 2026-07-19 joint DP, +since the original was not saved, extended with the optional +`exact_match_penalty` term and generalized to take an arbitrary target +token count so it works against an isolated region rather than only a +combined `npts`-token body). + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -1938,10 +2156,15 @@ scope (`PDA 3D Raw Data` / `LSS Raw Data`): ## Further avenues for a future session -Beyond the single recommendation in the 2026-07-19 closing summary -(sharpen the joint temporal+magnitude decoder's scoring function), these -are additional, not-yet-tried directions - none executed this round, so -treat them as leads, not findings: +The 2026-07-19 closing summary's single recommendation (sharpen the +joint temporal+magnitude decoder's scoring function) was attempted in +2026-07-20 session 5, with a negative result for the specific +anti-mode-collapse penalty term tried - see that session for detail and +for three concretely scoped follow-ups it left open (a faster/non-Python +DP implementation, a run-length-based rather than single-pair-based +anti-collapse penalty, and fully manual byte inspection). The items +below are additional, not-yet-tried directions - none executed this +round, so treat them as leads, not findings: - **(New, from 2026-07-20 session 3; followed up in session 4 with a negative result) Push the region-`tail` walk past channel index 2.** diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index cd187d6..f410533 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -308,5 +308,12 @@ single-file result), and ruled both out after decoding actual values - the second of which exposed and fixed a real gap in the physical-plausibility check itself (a mode-dominated, mostly-repeated decode can look deceptively "smooth" under a naive mean-step metric). -None of these findings decode the per-value payload; that grammar is -still open. +A fifth pass re-checked the per-byte-position entropy analysis with +region A's now-known 256-channel boundary (no change from the earlier +figures, no new periodicity) and re-ran the joint temporal+magnitude +decoder with region-correct target counts and an anti-mode-collapse +cost term; the underlying temporal-correlation signal replicates in a +more rigorous multi-pair test but is markedly weaker than the original +single-pair anecdote suggested, and the new cost term did not fix the +decoder's known selectivity problem. None of these findings decode the +per-value payload; that grammar is still open. From 151975eba7178428c66936540f5895adf6769dc8 Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 18:54:39 +0000 Subject: [PATCH 5/6] docs: manual byte-level read of PDA payload finds no new hypothesis, closes this investigation round (#2) Deliberately inverts this branch's prior approach: instead of another automated parameter sweep, does close, manual, hypothesis-light reading of actual segment bytes, reaching for a parametrized test only once something specific in the raw data suggested one. Byte-diffing two temporally-adjacent MTBLS432 real-mode segments by hand finds a real, non-obvious technique: segments whose body length happens to equal exactly 3 * npts need no decode algorithm at all to align to a token grid, since position channel*3 is then unambiguously each channel's leading byte. Using this to read a run of consecutive such segments turns up an initially striking pattern - several channels whose leading byte drifts smoothly build-to-build - but checking it properly (all adjacent exact-length pairs in one file, then two more files with longer runs) shows the "smooth" channels are exactly the ones with the fewest distinct values, with a 0.49-0.84 correlation between a channel's smoothness score and its value diversity across three independent files. This is the same class of problem as the mode-fraction lesson from two commits ago, generalized: low value diversity alone, not just outright mode-domination, mechanically produces a good smoothness score regardless of decode correctness. Hand-inspecting the flat-to-real transition segment itself (the segment immediately after the last all-zero baseline segment, in two files) finds no leading or trailing quiet region, no length-prefix-looking field, and scattered rather than clustered zero bytes - direct, by-eye confirmation of the already-established "hard, instantaneous cliff" finding rather than a new discovery. Neither line of manual inquiry produced a byte pattern specific enough to justify a new parametrized sweep. Recorded as a legitimate close to this investigation for now, per the requester's own framing: the per-point payload grammar remains undecoded after six same-day sessions, and every promising-looking signal at this scale keeps resolving to already-characterized artifacts rather than the real grammar. No Rust code changes: the payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu --- CHANGELOG.md | 60 +++---- docs/format/04-lcd-chromatogram-pda.md | 228 ++++++++++++++++++++++++- docs/format/06-known-limitations.md | 61 +++---- 3 files changed, 281 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f7ee90..be52a97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,40 +19,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Documentation - Further PDA/chromatogram payload investigation - (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): identified 2 of the - 4 varying fields in `PDA 3D Raw Data/CheckSum` as exact stream byte - sizes (correcting an earlier "flat vs. real flag" reading); ruled out - a 19-polynomial CRC-16 sweep (plus Fletcher/Adler/plain-sum and an - Internet-checksum variant) and several plain-count/derived-size - candidates for the remaining 2 fields; ruled out a fixed-width fp16 - (binary16) array and a block-floating-point/adaptive-scale hypothesis - family in four concrete forms, quantifying a ~48% false-positive base - rate for this document's own zero-leftover acceptance test along the - way; resolved a separate open question by finding the "split" envelope - form's two regions are an exact 256-wavelength-channel/remainder - split (also explaining why "split" vs. "symmetric" form correlates - with wavelength count); and showed, via a physical-plausibility check, - that a marker-bit signal which passed two randomized-control tests was - nonetheless a compensating-error artifact rather than a real decode. - A follow-up pass re-ran the threshold/continuation-bit sweeps with the - now-corrected per-region target counts, found two more dramatic- - looking single-file/single-region signals, and ran both all the way - through to actual value decoding before ruling them out too (one - reproduced the same compensating-error artifact under a sharper - threshold framing; the other exposed a real flaw in the existing - physical-plausibility check itself - a decode that repeats the same - value 80-96% of the time can look "smooth" by mean relative step - alone - which the check has now been revised to catch). A further - pass re-checked the earlier per-byte-position entropy analysis with - region A's now-known 256-channel boundary (reproduces the prior - numbers almost exactly, no new periodicity found) and re-ran the - joint temporal+magnitude decoder with region-correct target counts - and an anti-mode-collapse cost term - the underlying temporal- - correlation signal replicates but is markedly weaker than the - original single-pair anecdote suggested, and the new cost term did - not improve the decoder's previously-diagnosed selectivity problem. - The per-point payload grammar itself is still undecoded - see - `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-5 for + (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): six same-day + sessions of clean-room analysis. Confirmed findings: 2 of the 4 + varying `PDA 3D Raw Data/CheckSum` fields are exact stream byte sizes + (correcting an earlier "flat vs. real flag" reading), and the + "split" envelope form's two regions are an exact + 256-wavelength-channel/remainder split (also explaining why "split" + vs. "symmetric" form correlates with wavelength count). Ruled out, in + each case run through actual value decoding, a physical-plausibility + check, and randomized controls rather than reported on walk-rate + alone: a 19-polynomial CRC-16 sweep and several count/derived-size + candidates for the remaining `CheckSum` fields; a fixed-width fp16 + array; a block-floating-point/adaptive-scale hypothesis family; a + region-tail marker-bit signal that passed two randomized controls but + traced to a compensating-error artifact; two more dramatic-looking + signals surfaced by re-running sweeps with corrected per-region + target counts; an anti-mode-collapse cost term added to the joint + temporal+magnitude decoder; and, from a deliberately manual (not + automated-sweep) byte-reading pass, a "leading byte of a 3-byte + token" hypothesis that traced to the same low-value-diversity metric + artifact on three independent files. Along the way, quantified a + ~48% false-positive base rate for this document's zero-leftover + acceptance test and fixed a real gap in the physical-plausibility + check itself (mode-dominated/low-diversity decodes can look + deceptively "smooth" under mean relative step alone). The per-point + payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-6 for full detail. ## [0.1.0] - 2026-07-18 diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 826bf85..19c24b3 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -106,6 +106,22 @@ is a scannable summary, not a substitute for the detailed sections below found this the hard way (a 67.8%-clean, apparently-smooth single-file `MTBLS432` result turned out to be up to 96% mode-dominated per channel) and revised the check accordingly. +- **Methodological, extending the mode-fraction lesson: even without + outright mode-domination, low value diversity alone mechanically + produces a good smoothness score, independent of decode correctness.** + 2026-07-20 session 6 found a `0.840` correlation (68 channels, one + file) between a channel's smoothness score and how many distinct + values it takes - genuinely variable (more analytically real) + channels score *worse*, and channels that barely vary score well + regardless of whether the decode is right. A physical-plausibility + check should be read skeptically for low-diversity channels + specifically, not just mode-dominated ones. +- **The flat-to-real transition has no visible byte-level structure at + all when the transition segment's actual bytes are read by eye** + (2026-07-20 session 6, two files checked): no leading or trailing + quiet region, no length-prefix-looking field, zero bytes scattered + not clustered - direct, by-eye confirmation of the already-established + "hard, instantaneous cliff" aggregate-statistics finding. **Ruled out** (see "This session's additional ruled-out hypotheses" and "further ruled-out hypotheses" for full detail): standard unsigned @@ -148,7 +164,13 @@ between solution cost and width agreement, so does not fix the DP's previously-diagnosed selectivity weakness (see 2026-07-20 session 5); mod-2 and (beyond the already-explained early-channel effect) mod-3 periodicity in region `A`'s per-byte-position entropy (see -2026-07-20 session 5). +2026-07-20 session 5); the leading-byte-of-a-3-byte-token hypothesis +for `MTBLS432`, checked via exact-`3*npts`-length segments across three +files - the channels that looked smooth by eye in each case turned out +to have near-zero value diversity (as low as 2 distinct values across a +6-segment run), with a `0.49`-`0.84` smoothness-vs-diversity correlation +confirming the pattern is a metric artifact, not a decode (see +2026-07-20 session 6). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric @@ -2125,6 +2147,174 @@ since the original was not saved, extended with the optional token count so it works against an isolated region rather than only a combined `npts`-token body). +## 2026-07-20 session 6: manual byte-level reading (no sweep, no hypothesis-first search) - a genuine new inspection method, a strong but ultimately artifact-explained "leading byte" lead, and an eye-verified confirmation of the hard-cliff transition + +Every session to date has been hypothesis-first: guess a scheme, sweep +parameters, check zero-leftover, then (since session 3) check physical +plausibility. This session deliberately inverted that order - close, +manual, hypothesis-light reading of actual bytes, reaching for a +parametrized test only once something specific in the raw data +suggested one. **No new testable hypothesis survived scrutiny, and the +per-value payload grammar remains undecoded** - but the manual read +surfaced a genuinely new observational technique (inspecting segments +whose body length exactly equals `3 * npts`, which requires no decode +algorithm at all to align to a token grid) and one quantified, honest +account of *why* a promising-looking signal from that technique +dissolves under scrutiny, distinct from - but related to - session 4's +mode-fraction lesson. + +### Byte-diffing two adjacent real-mode `MTBLS432` segments by hand + +Started with `..._12_65...lcd` (symmetric form, `npts=68`, no +region-A/tail split to worry about), segments 30 and 31 (lengths 201 +and 203, a quiet/plateau part of the run away from both the transition +and any sharp peak). A naive fixed-offset byte-by-byte diff (printed in +8-byte rows with a running differing-byte count) reproduced the +already-documented "80-92% of bytes differ" finding almost exactly - 7 +or 8 of every 8 bytes differ at nearly every row, which is expected +given genuinely variable-per-value width means a fixed byte offset +essentially never lines up with the same channel in both segments once +any earlier channel's width has diverged. + +- **A closer, signed-delta-by-position read (not an aggregate percentage) + found something real: a cluster of small deltas at byte positions + that are consistently multiples of 3, and only when the byte value at + that position falls in the `0x40`-`0x5f` range** - the same + dominant-leader-byte range this document has flagged since its very + first PDP-endian-float observation. Of 68 real channels' worth of + byte positions checked by hand (0 through 202), roughly a dozen + positions showed `|delta| <= 4` while their neighbors showed deltas + in the hundreds - and every one of those small-delta positions landed + on a position `≡ 0 (mod 3)`, consistent with (not a new contradiction + of) this document's already-established `~3 * npts` centering for + this envelope form. Tried decoding 4-byte windows starting at these + positions as `float32` under all four `{LE, BE} x {plain, + PDP-word-swapped}` byte orderings already explored in earlier + sessions: none gave consistently plausible small-magnitude values + across even the pairs of positions checked, so a straightforward + "this byte starts a 4-byte float" reading does not explain the small + deltas on its own. +- **A new, genuinely useful observational technique: segments whose + body length is exactly `3 * npts` need no decode algorithm to align + to a token grid at all.** If a segment's real-mode body is exactly + 204 bytes for `npts=68`, and the true encoding really does put most + values at 3 bytes each, then position `channel * 3` is very likely + each channel's true leading byte, checkable by direct indexing with + no width-inference, no threshold, no walk. `..._12_65...lcd` has 70 + such segments (out of 2662 real-mode segments), including one run of + **4 temporally consecutive** exact-204 segments (indices 133-136). + Reading the leading byte of each channel's 3-byte group across this + run by eye: channels 0-8 and 14 are strikingly smooth across segments + 133, 134, and 135 (e.g. channel 4 reads `93, 93, 93` exactly; channel + 5 reads `64, 64, 64`; channel 0 reads `71, 69, 68`, a plausible small + drift) - but segment 136 shows a **simultaneous jump across every + channel at once** (channel 4 jumps to `225`, channel 5 to `198`, + etc.), which reads as segment 136 not actually being a uniform + 3-bytes-per-value body despite its total length coincidentally + matching `3 * npts` (consistent with this document's own + already-established fact that only a mix of widths, not a strict + per-value 3-byte rule, explains most 3-multiple lengths). +- **Extending this check to all 70 exact-204 segments (not just the one + lucky run) shows the initial "smooth!" impression does not hold up, + and traces to a specific, quantifiable artifact.** Only 15 of the 70 + exact-204 segments are temporally adjacent to another exact-204 + segment (15 total adjacent pairs, mostly isolated pairs plus the one + run of 4) - using all 15 pairs, mean per-channel relative step + (session 3/4's smoothness metric) is `0.516`, and only **15 of 68 + channels** fall under the `0.3` "plausible" threshold used elsewhere + in this document. Checking *which* 15: they are exactly the channels + at the extreme low-index (`0`-`8`) and high-index (`63`-`67`) ends of + the wavelength range - i.e. exactly the "low-variance early + wavelength" channels this document's 2026-07-19 entropy session and + this session's own region-A-isolated re-check (earlier in this same + session set) already identified as inherently low-dynamic-range, not + evidence of a correct decode. Quantified directly: **correlation + between a channel's smoothness score and its number of distinct + leading-byte values across the 70 exact-204 segments is `r = 0.840`** + - channels that are more genuinely variable (higher value diversity, + presumably more analytically active/real) systematically score + *worse* on the smoothness metric, and channels that barely vary at + all trivially score well. This is a distinct but closely related + trap to session 4's mode-fraction lesson: session 4 showed a + literally mode-*dominated* (one repeated value 80-96% of the time) + decode can look smooth; this session shows that even without hitting + that extreme, a channel's *low value diversity alone* - independent + of whether the decode is correct - mechanically produces a good + smoothness score. **A physical-plausibility check should be read + skeptically for any channel with few distinct decoded values, not + just ones with one dominant repeated value.** +- **Cross-checked against the general (non-exact-204-only) walk from + session 4** (`threshold=1, wide_w=3`, which parses 67.8% of this + file's real segments): extracting only the leading byte of each wide + token (rather than the full 3-byte integer session 4 used) does + *not* fix that walk's mode-collapse problem - mean per-channel mode + fraction is still `0.887`, indistinguishable from session 4's + original finding. This confirms session 4's root cause was correctly + diagnosed as being about the walk's *token boundaries* themselves + (which segments outside the lucky exact-204 case do not reliably + find), not about which bytes of a correctly-bounded token get turned + into a value. + +### Hand-inspecting the flat-to-real transition segment + +Read the exact first-real-mode segment body (immediately following the +last all-zero baseline segment) byte-by-byte, by eye, against the +all-zero segment before it, for two files: `..._12_65...lcd` (first +real segment, 200 bytes, following 11 segments of pure `0x00`) and +`..._13_27...lcd` (first real segment, 192 bytes, the file among the 9 +confirmed real-signal `MTBLS432` files with the most zero bytes in its +transition segment, on the theory that more zeros might mean a gentler, +more legible onset). Also checked zero-byte counts in all 8 other +confirmed real-signal files' first-real segments for comparison (range: +1 to 7 zero bytes out of 192-204). + +- **No file shows a "mostly still zero, one or two channels just + starting to show signal" onset.** Every first-real segment checked is + immediately as densely populated with the same `0x40`-`0x5f`/leader- + byte-dominated content as any other real-mode segment deeper into the + run - reading `..._13_27...lcd`'s first-real segment byte-by-byte + (its 7 zero bytes, the most of any file checked, are scattered at + positions `0, 49, 54, 80, 81, 100, 167` - not clustered at the start + or in any other single contiguous run) shows no leading quiet region, + no trailing quiet region, and nothing that reads as a length-prefix + or marker field distinct from the rest of the body. +- This is a genuine, eye-verified **confirmation**, not a new finding + by itself: the 2026-07-19 transition-segment-comparison session + already established via aggregate statistics that the flat-to-real + switch is an instantaneous cliff with "zero nonzero bytes" in the + segments immediately before it and no shared marker at the transition + segment itself. This session read the *actual bytes* of the + transition segment directly (not just its length or a checksum) for + the first time, across two files, and found the same thing a human + reading the raw hex would conclude: there is no visible structural + seam at the transition, only genuinely dense, high-entropy real data + starting immediately in every channel at once. + +**Verdict**: this session's inverted, manual-first approach found one +real, reusable technique (exact-`3*npts`-length segments as an +algorithm-free way to inspect a token grid) and used it honestly enough +to catch its own initially-promising result as an artifact, rather than +reporting the encouraging-looking first four-segment run as a hit. The +transition-segment read adds direct, by-eye confirmation to an +already-well-supported fact rather than contradicting or extending it. +Neither line of inquiry produced a specific byte pattern, marker, or +periodicity that suggests a new parametrized hypothesis worth building +and sweeping. Per this round's own instructions, this is being recorded +as a legitimate close to this manual-inspection lead for now rather than +forced further: the payload's per-value grammar remains undecoded after +six same-day sessions of systematic and, this round, manual +investigation, and this document's own historical pattern (this +session's finding included) is that promising-looking signals at this +scale of investigation keep resolving to already-known artifacts +(edge-channel low variance, value-diversity-driven metric bias, +compensating token-boundary errors) rather than to the real grammar. + +Scripts: ad hoc, run via disposable Python files under this session's +own scratch directory (outside the repo, not saved to +`re/src/analysis/`), reusing `common.py`'s `iter_segments`/`body_of`; +no new reusable helper modules were warranted for a manual-reading +session. + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -2162,9 +2352,16 @@ joint temporal+magnitude decoder's scoring function) was attempted in anti-mode-collapse penalty term tried - see that session for detail and for three concretely scoped follow-ups it left open (a faster/non-Python DP implementation, a run-length-based rather than single-pair-based -anti-collapse penalty, and fully manual byte inspection). The items -below are additional, not-yet-tried directions - none executed this -round, so treat them as leads, not findings: +anti-collapse penalty, and fully manual byte inspection). The third of +those was attempted in session 6 - general manual byte-level reading +(not targeted at region `tail` specifically) of `MTBLS432` real-mode +segments and the flat-to-real transition segment, with no new testable +hypothesis surfacing, though one reusable inspection technique +(exact-`3*npts`-length segments as an algorithm-free token-grid anchor) +came out of it. The region-`tail`-specific manual read described in the +next bullet remains untried. The items below are additional, +not-yet-tried directions - none executed this round, so treat them as +leads, not findings: - **(New, from 2026-07-20 session 3; followed up in session 4 with a negative result) Push the region-`tail` walk past channel index 2.** @@ -2189,6 +2386,29 @@ round, so treat them as leads, not findings: shortcut around it, but the much smaller channel count (63-69 instead of 256-327) may still make manual inspection more tractable than it is against a full segment. +- **(New, from 2026-07-20 session 6) Re-run the exact-`3*npts`-length + byte-diff technique against longer exact-`3*npts` runs: tried this + session, same artifact confirmed on two more files, this specific + avenue is now closed too.** Session 6's manual read of + `..._12_65...lcd`'s exact-204-byte segments found a handful of + channels that looked smooth, but all of them turned out to be the + same already-known low-variance edge channels (indices `0`-`8` and + `63`-`67`), with a `0.840` smoothness-vs-value-diversity correlation + across that file's 4-segment run. To check this wasn't an artifact of + that one short run, this session went on to find and check the two + longest same-length consecutive runs across all 9 confirmed + real-signal `MTBLS432` files: `..._26_68...lcd` (6 consecutive + exact-204-byte segments, indices `2036`-`2041`) and `..._10_26...lcd` + (another 6-long run, indices `1293`-`1298`). Both reproduce the same + pattern - `..._26_68...lcd`'s apparently-smooth *interior* channels + (e.g. channels 9-11, values like `[84, 83, 84, 83, 84, 84]`) turned + out to have only **2 distinct values** across the 6-segment run, and + the smoothness-vs-diversity correlation is `0.718` (`..._26_68`) and + `0.489` (`..._10_26`) - both substantial, both in the same + artifact-confirming direction as the original file. This rules out + "the 4-segment run was just too short to see real interior-channel + structure" as an explanation; longer runs on two independent files + show the identical low-diversity artifact, not new signal. - **Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` stream instead of (or before) `PDA 3D Raw Data`.** It's populated, structurally simpler (one giant segment per channel, not thousands of diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index f410533..724b3b6 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -287,33 +287,34 @@ integer and escape-byte hypotheses (Sigilweaver/OpenSZRaw#2). This is UV detector / chromatogram data, not core MS spectra, so it does not block MS-level format parity. -A 2026-07-20 session identified 2 of the 4 varying fields in the -112-byte `PDA 3D Raw Data/CheckSum` stream as exact `u32` byte sizes of -the `3D Raw Data` and `Max Plot` streams (not a flat/real flag as -previously read - see the correction note and dated session in -`docs/format/04`), and ruled out 19 standard CRC-16 polynomials plus -several other common checksum constructions for the remaining 2 fields. -A follow-up pass the same day resolved a separate open structural -question - the "split" envelope form's two declared-length regions are -an exact 256-wavelength-channel/remainder split, which also explains -why "split" vs. "symmetric" envelope form correlates with wavelength -count - and ruled out a block-floating-point/adaptive-scale hypothesis -family (including one candidate that passed two randomized-control -checks but was then shown, via a physical-plausibility check, to be a -compensating-error artifact rather than a real decode). A further -follow-up pass re-ran the decode sweeps with the corrected per-region -target counts this finding implies, found two more signals that looked -promising at first (one a sharp threshold cliff, one a 67.8%-clean -single-file result), and ruled both out after decoding actual values - -the second of which exposed and fixed a real gap in the -physical-plausibility check itself (a mode-dominated, mostly-repeated -decode can look deceptively "smooth" under a naive mean-step metric). -A fifth pass re-checked the per-byte-position entropy analysis with -region A's now-known 256-channel boundary (no change from the earlier -figures, no new periodicity) and re-ran the joint temporal+magnitude -decoder with region-correct target counts and an anti-mode-collapse -cost term; the underlying temporal-correlation signal replicates in a -more rigorous multi-pair test but is markedly weaker than the original -single-pair anecdote suggested, and the new cost term did not fix the -decoder's known selectivity problem. None of these findings decode the -per-value payload; that grammar is still open. +Six same-day (2026-07-20) sessions of further clean-room analysis +narrowed the problem considerably without decoding it. Confirmed: 2 of +the 4 varying fields in the 112-byte `PDA 3D Raw Data/CheckSum` stream +are exact `u32` byte sizes of the `3D Raw Data` and `Max Plot` streams +(not a flat/real flag as previously read), and the "split" envelope +form's two declared-length regions are an exact +256-wavelength-channel/remainder split, which also explains why +"split" vs. "symmetric" envelope form correlates with wavelength +count. Ruled out, each time via actual value decoding plus a +physical-plausibility check and randomized controls rather than on +walk-rate alone: 19 CRC-16 polynomials and several count/derived-size +candidates for the remaining `CheckSum` fields; a fixed-width fp16 +array; a block-floating-point/adaptive-scale hypothesis family +(including a candidate that passed two randomized-control checks but +proved to be a compensating-error artifact); two more signals surfaced +by re-running the decode sweeps with corrected per-region target +counts; an anti-mode-collapse cost term added to the joint +temporal+magnitude decoder (the underlying temporal-correlation signal +replicates in a more rigorous multi-pair test but is markedly weaker +than the original single-pair anecdote suggested); and, from a +deliberately manual byte-reading pass (rather than another automated +sweep), a "leading byte of a 3-byte token" hypothesis that traced to a +low-value-diversity metric artifact confirmed on three independent +files. Two genuine process improvements came out of this: a directly +quantified ~48% false-positive base rate for this document's +zero-leftover acceptance test, and a fix to the physical-plausibility +check itself (mode-dominated or low-diversity decodes can look +deceptively "smooth" under mean relative step alone). See +`docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-6 for +full detail. None of this decodes the per-value payload; that grammar +is still open. From 67150bcc491d289515ebf0e52893babd0469d0c5 Mon Sep 17 00:00:00 2001 From: Benjamin Riley Date: Mon, 20 Jul 2026 19:45:30 +0000 Subject: [PATCH 6/6] docs: cross-reference PSI-MS/mzML compression schemes for PDA payload, rule both out (#2) Continues the PDA/chromatogram payload investigation from the prior commit on this branch, taking a genuinely new angle: cross-referencing the mzML/PSI-MS open spec's own published compression schemes, rather than sweeping more parameters of the byte-granular framings every prior session already tried. Tests two concrete, public, non-Shimadzu ideas: MS-Numpress (the PSI-MS-affiliated lossy compression scheme for mzML binary data arrays, Teleman et al. 2014), whose integer encoding operates at nibble (half-byte) granularity rather than the byte-aligned tokens every previous continuation-bit/threshold sweep assumed; and whether the payload is simply a standard zlib/DEFLATE stream, the actual mechanism mzML itself uses to compress binary data arrays. Both are ruled out. Two nibble-varint variants (a generic per-nibble continuation flag, and the length-prefixed scheme closer to Numpress's own published encodeInt) were swept across three independent files and four region targets: essentially 0% clean everywhere, including one 80-way sweep that produced a nonzero 2.68% hit rate on a single file - disqualified by a shuffled-byte control (80% of the "signal" survives scrambling byte order entirely) and by collapsing to under 1% on that file's own sibling accessions. zlib/DEFLATE framing showed a small 0.79% hit rate that turned out statistically indistinguishable from both a random-byte and a shuffled-byte control, and every "successful" decompression produced a degenerate 1-byte output - raw DEFLATE's well-known weak self-framing, not real compressed content. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_013ac4gwERRkHA4fMijh4eUF --- CHANGELOG.md | 24 ++-- docs/format/04-lcd-chromatogram-pda.md | 172 ++++++++++++++++++++++++- docs/format/06-known-limitations.md | 15 ++- 3 files changed, 198 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index be52a97..22e24fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Documentation - Further PDA/chromatogram payload investigation - (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): six same-day + (Sigilweaver/OpenSZRaw#2, contributed by @Nabejo): seven same-day sessions of clean-room analysis. Confirmed findings: 2 of the 4 varying `PDA 3D Raw Data/CheckSum` fields are exact stream byte sizes (correcting an earlier "flat vs. real flag" reading), and the @@ -35,16 +35,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 traced to a compensating-error artifact; two more dramatic-looking signals surfaced by re-running sweeps with corrected per-region target counts; an anti-mode-collapse cost term added to the joint - temporal+magnitude decoder; and, from a deliberately manual (not + temporal+magnitude decoder; from a deliberately manual (not automated-sweep) byte-reading pass, a "leading byte of a 3-byte token" hypothesis that traced to the same low-value-diversity metric - artifact on three independent files. Along the way, quantified a - ~48% false-positive base rate for this document's zero-leftover - acceptance test and fixed a real gap in the physical-plausibility - check itself (mode-dominated/low-diversity decodes can look - deceptively "smooth" under mean relative step alone). The per-point - payload grammar itself is still undecoded - see - `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-6 for + artifact on three independent files; and, cross-referencing the + PSI-MS/mzML open spec directly, two MS-Numpress-inspired + nibble-granular varint encodings and literal zlib/DEFLATE framing of + the payload - the one nonzero hit rate found (a nibble scheme on one + file) was disqualified by a shuffled-byte control and cross-file + testing, and DEFLATE's small hit rate proved statistically + indistinguishable from random-byte and shuffled-byte controls. Along + the way, quantified a ~48% false-positive base rate for this + document's zero-leftover acceptance test and fixed a real gap in the + physical-plausibility check itself (mode-dominated/low-diversity + decodes can look deceptively "smooth" under mean relative step + alone). The per-point payload grammar itself is still undecoded - see + `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-7 for full detail. ## [0.1.0] - 2026-07-18 diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 19c24b3..3c7d328 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -170,7 +170,15 @@ files - the channels that looked smooth by eye in each case turned out to have near-zero value diversity (as low as 2 distinct values across a 6-segment run), with a `0.49`-`0.84` smoothness-vs-diversity correlation confirming the pattern is a metric artifact, not a decode (see -2026-07-20 session 6). +2026-07-20 session 6); two MS-Numpress-inspired nibble-granular varint +schemes (a generic per-nibble continuation flag, and the length-prefixed +scheme closer to Numpress's own published `encodeInt`) swept across three +independent files and four region targets - 0% clean everywhere except +one outlier that a shuffled-byte control and cross-file check both +disqualified as search-space noise; and zlib/DEFLATE framing for the +whole payload or body, ruled out via a randomized control after an +initial small hit rate turned out statistically indistinguishable from +both a random-byte and a shuffled-byte control (see 2026-07-20 session 7). **Genuinely open:** - The exact per-value token grammar (width-selection rule and numeric @@ -2315,6 +2323,142 @@ own scratch directory (outside the repo, not saved to no new reusable helper modules were warranted for a manual-reading session. +## 2026-07-20 session 7: cross-referencing the PSI-MS/mzML open spec - MS-Numpress-style nibble varints and zlib/DEFLATE framing, both ruled out with randomized controls + +Prompted specifically by the suggestion to cross-reference the mzML/ +PSI-MS open spec for an analogous UV/PDA encoding (rather than continuing +to sweep parameters of schemes already tried), this session tested two +concrete, public, non-Shimadzu ideas that come directly from that world: +**MS-Numpress**, the published PSI-MS-affiliated lossy compression scheme +for mzML `` content (Teleman et al., "Numerical +compression schemes for proteomics mass spectrometry data", Mol Cell +Proteomics 2014 - an open, community-standard algorithm with a public +BSD-licensed reference implementation, not vendor material), and whether +the payload is simply a standard **zlib/DEFLATE** stream, which is the +actual real-world mechanism mzML uses to compress its own binary data +arrays. **Neither fits; both are ruled out, with randomized controls for +the one place a nonzero signal briefly appeared.** The per-value payload +grammar remains undecoded. + +- **Why this is a materially different search space from every prior + continuation-bit/threshold sweep, not a repeat of one**: every + varint-style framing this document has tried before (the original + LEB128 attempt, the "8 bit positions x both polarities x 0-11 byte + header/footer skip" continuation-bit sweep, the magnitude-threshold + sweep, the region-local re-runs) operated at **byte** granularity - + a token always starts and ends on a byte boundary. MS-Numpress's + actual integer encoding operates at **nibble** (half-byte) granularity: + each residual is packed into a run of 4-bit nibbles with no requirement + that a token start on a byte boundary, which can natively produce + non-integer average bytes/value (e.g. this document's own observed + ~1.88x and ~3x `npts` centering figures) without needing a threshold or + escape-byte overlay on top. This session implemented two variants of + that idea and swept both against real segments from three independent + files (`MTBLS432/..._12_65...lcd`, `npts=68`, symmetric form, whole-body + target 68; `MSV000084197/20190607_NM16.lcd`, `npts=321`, split form, + region-A target 256 and region-tail target 65; `PXD025121/1.lcd` and + two sibling files, `npts=327`, split form, region-tail target 71), + reusing the already-established envelope/region extraction + (`docs/format/04` sessions 1 and 3) rather than re-deriving it. +- **Variant 1 - generic per-nibble continuation flag** (each 4-bit nibble + carries 3 payload bits plus 1 "more nibbles follow" flag, the nibble- + granular analogue of the already-tried byte-level continuation-bit + sweep): swept nibble order (high-nibble-first vs low-nibble-first) + x continuation-bit position within the nibble (top bit vs bottom bit) + x polarity (flag-set-means-more vs flag-clear-means-more) x value bit + accumulation order (MSB-first vs LSB-first) x header skip (0-4 bytes), + 80 combinations per target. Result: **flat 0% clean** on + `MTBLS432`'s whole-body target (0/2662) and `MSV000084197`'s region-A + target (0/3498); **effectively 0%** on `MSV000084197`'s region-tail + (1/3498, 0.03%). One file showed a real-looking outlier - + `PXD025121/1.lcd`'s region tail reached **166/6187 (2.68%)** at the + best configuration - but this does not survive scrutiny: a + same-multiset **shuffled-byte control** (same bytes, scrambled order) + retains **132/6187 (2.13%)**, i.e. **80% of the apparent signal + survives destroying byte order entirely**, meaning this is almost + entirely a byte-*value*-distribution artifact, not the order-dependent + structure a real token grammar would produce (contrast this document's + genuine positive controls elsewhere, e.g. session 3's region-`tail` + marker-bit finding, where the shuffled control collapsed to 0.2% against + a 70.9% real rate - a qualitatively different, decisive gap this + session's result does not show). The same configuration also fails + cross-file generalization outright, dropping to **62/6179 (1.00%)** on + `PXD025121/10.lcd` and **20/6179 (0.32%)** on `/11.lcd` - both files + sharing the same instrument, method, and `npts` as `/1.lcd`. This is + the same class of small-sample search-space noise this document's own + 2026-07-20 session 2 already quantified (a ~48% false-positive base + rate for a much smaller, single-threshold search); an 80-way sweep over + 6187 segments producing one 2.68%-hitting outlier that halves under a + shuffle control and collapses under cross-file testing is fully + consistent with chance, not a real per-nibble continuation scheme. +- **Variant 2 - length-prefixed nibbles, a closer structural match to + MS-Numpress's actual published `encodeInt`** (one 4-bit "length" + nibble stating how many raw data nibbles immediately follow, rather + than a per-nibble continuation flag - this is the real shape of + Numpress's own integer encoding, which exploits sign-extension to keep + small residuals to very few nibbles): swept header skip (0-4 bytes) x + nibble order x length semantics (`L` vs `L+1` data nibbles) x value bit + order x maximum token length (8 nibbles, matching Numpress's own 32-bit + cap, or 15, the full nibble range), 80 combinations per target, against + the same three files/four targets. Result: **0% clean, on every single + target checked, with zero exceptions** - not even the small, noisy, + non-generalizing signal Variant 1 produced. This is the most decisive + negative result of any per-value framing hypothesis this document has + tried: the existing byte-level continuation-bit sweep's best-ever + result was 88/3502 (2.5%, see the "further ruled-out hypotheses" + section); this nibble-length-prefix scheme does not reach that even + once, across 4 targets x 80 configurations x up to 6187 segments each. +- **zlib/DEFLATE framing** (does the payload, or its body, simply contain + a standard compressed stream - the actual mechanism mzML itself uses + for binary data arrays, as opposed to a Numpress-style custom integer + code): tried `zlib.decompress` at `wbits` 15 (zlib-wrapped), -15 (raw + DEFLATE, no header/checksum), 31 (gzip), and 47 (auto-detect + zlib/gzip) against every real-mode payload and body in + `MTBLS432/..._12_65...lcd` (2662 segments). **21/2662 (0.79%)** + "succeed" (all at `wbits=-15`, raw DEFLATE, on the whole payload - never + on the body alone, and never at any of the header'd variants). This + looked like it might be worth chasing until checked against controls, + exactly per this document's established discipline: a **length-matched + random-byte control** succeeds at **15/2662 (0.56%)**, and a + **same-multiset shuffled-byte control** at **14/2662 (0.53%)** - both + statistically indistinguishable from the real rate. This is expected + once the actual "decompressed" output is inspected: every successful + case yields a **1-byte output**, confirming these are not real + compressed streams but instances of raw DEFLATE's well-known weak + self-framing (no header magic, no checksum - a short byte run can + coincidentally contain a valid Huffman end-of-block marker and "parse," + producing a near-empty result, regardless of whether the input was ever + compressed data at all). **zlib/DEFLATE is ruled out** as the payload's + encoding, whole-payload or body-only, for either envelope form. + +**Verdict**: cross-referencing the PSI-MS/mzML ecosystem's own published +compression schemes was a genuinely new angle (nothing prior in this +document tried nibble-granular tokenization or literal DEFLATE framing), +and it produced two clean, thoroughly-controlled negative results rather +than another ambiguous partial signal - in the nibble-continuation case, +one candidate outlier was found and then properly disqualified by the +same shuffle-control-plus-cross-file-check discipline this document has +used throughout, rather than reported as a lead. This does not mean +mzML/PSI-MS ideas are exhausted as a source of inspiration (Numpress's +*specific* bit-packing scheme is ruled out, not the general idea that a +delta/prediction-based numeric transform might still explain the +*values* once a correct token-boundary rule is found by some other +means - see the existing fp16/spectral-delta "validator, not framing +strategy" scoping note earlier in this document, which applies here too), +but it closes off the two most concrete, literal readings of "does this +look like an mzML-world encoding" that this session could test. + +Scripts (new this session, under `re/src/analysis/`, gitignored per this +repo's `re/` sandbox convention): `common.py` (re-derived `iter_segments`/ +`body_of`/region-split/flat-detection helpers, since no prior session's +scripts were preserved), `nibble_varint.py` (Variant 1: generic per-nibble +continuation-flag decoder and sweep), `nibble_lenprefix.py` (Variant 2: +length-prefixed nibble decoder and sweep), `run_nibble_sweep.py` and +`run_lenprefix_sweep.py` (drivers tying the above to real corpus files). +The zlib/DEFLATE check was ad hoc (`python3 -c`/heredoc, not saved as a +standalone script, since it needed no reusable abstraction beyond the +standard library's own `zlib` module). + ## LC Raw Data - a different, unrelated chromatogram stream While looking for real `LSS Raw Data` chromatogram content (all empty in @@ -2519,3 +2663,29 @@ leads, not findings: much larger search than session 2's budget allowed, but is a more faithful test of the "explicit per-segment scale field" idea than what was actually tried. +- **(Attempted 2026-07-20 session 7, ruled out) Cross-reference the + PSI-MS/mzML open spec's own compression schemes for the token-boundary + rule.** Two concrete readings were tried and ruled out this session: + MS-Numpress-style nibble-granular varint encoding (both a generic + per-nibble continuation flag and the length-prefixed scheme closer to + Numpress's own published algorithm), and literal zlib/DEFLATE framing + of the payload or body. See the 2026-07-20 session 7 section above for + full detail, including the shuffled-byte and cross-file controls that + disqualified the one place either search produced a nonzero hit rate. + This closes the two most literal readings of the idea; what remains + untried from this angle is applying MS-Numpress's *value* transforms + (double-delta linear prediction, or the "Slof" log-fixed-point + transform) as validators against a candidate token-boundary assignment + a future width-selection breakthrough would produce - the same + "validator, not framing strategy" caveat already noted above for + fp16 and spectral-domain delta, since neither transform changes which + bytes form a token, only how a found token's bytes become a number. +- **(New, from 2026-07-20 session 7) Manual, by-hand nibble-level + inspection of region `tail`'s channel-2-onward bytes was not attempted + this session.** Session 7's nibble-granular sweeps were parameterized + searches, in the same automated-sweep style this document's own session + 6 found unproductive for the byte-granular case. The manual read that + session recommends as the next step for region `tail` channels 2+ (see + that bullet above) has not yet been tried at nibble granularity either, + and remains a genuinely untried combination of "manual" and + "nibble-aware" that no session, including this one, has done. diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index 724b3b6..91b5b77 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -287,7 +287,7 @@ integer and escape-byte hypotheses (Sigilweaver/OpenSZRaw#2). This is UV detector / chromatogram data, not core MS spectra, so it does not block MS-level format parity. -Six same-day (2026-07-20) sessions of further clean-room analysis +Seven same-day (2026-07-20) sessions of further clean-room analysis narrowed the problem considerably without decoding it. Confirmed: 2 of the 4 varying fields in the 112-byte `PDA 3D Raw Data/CheckSum` stream are exact `u32` byte sizes of the `3D Raw Data` and `Max Plot` streams @@ -314,7 +314,16 @@ files. Two genuine process improvements came out of this: a directly quantified ~48% false-positive base rate for this document's zero-leftover acceptance test, and a fix to the physical-plausibility check itself (mode-dominated or low-diversity decodes can look -deceptively "smooth" under mean relative step alone). See -`docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-6 for +deceptively "smooth" under mean relative step alone). A seventh session +cross-referenced the PSI-MS/mzML open spec directly (per a specific ask +to look there rather than sweep more parameters of already-tried +byte-granular schemes): two MS-Numpress-inspired nibble-granular varint +encodings and literal zlib/DEFLATE framing of the payload, all ruled out +- one nibble scheme's only nonzero hit rate was disqualified by a +shuffled-byte control (80% of it survived byte-order scrambling) and +cross-file testing (collapsed on two sibling files), and the DEFLATE +framing's small hit rate proved statistically indistinguishable from +both a random-byte and a shuffled-byte control. See +`docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-7 for full detail. None of this decodes the per-value payload; that grammar is still open.