docs: extensive PDA payload investigation, split-region semantics resolved (#2)#19
Merged
Merged
Conversation
…or PDA payload (Sigilweaver#2) Continues the PDA/chromatogram payload investigation from docs/format/04. Full characterization of the 112-byte PDA 3D Raw Data/CheckSum stream finds that offset 56 and offset 88 (each a little-endian u32) are the exact byte sizes of the 3D Raw Data and Max Plot streams respectively, verified with zero mismatches across all 77 locally available files with both streams present. This corrects an earlier reading of offset 58 as a boolean "flat vs. real" flag - it is actually the high word of the offset-56 size field, and the earlier flat/real correlation was coincidental to stream-size magnitude. The other two varying CheckSum fields (offsets 48 and 80) are confirmed genuinely content-dependent (identical across files with byte-identical flat stream content, different otherwise), but a sweep of 19 CRC-16 polynomials plus Fletcher/Adler/plain-sum checksums against five candidate byte ranges found no match - a real negative result, not proof no algorithm exists. Also rules out a fixed-width fp16 (binary16) array for the per-value payload (body length essentially never equals 2 * npts), precisely characterizes the split-form real-mode body length centering (~1.88 * npts, distinct from the symmetric form's ~3 * npts), and records why the fp16 and spectral-domain-delta ideas can only function as validators for a token-boundary rule rather than independent framing searches - both remain open per docs/format/06-known-limitations.md section 10. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu
…nt payload hypotheses (Sigilweaver#2) Continues the PDA/chromatogram payload investigation from the prior commit on this branch. Two new investigation passes, both aimed at an actual decode rather than further characterization alone. The first tests a block-floating-point/adaptive-scale hypothesis family (each segment carrying its own implicit or explicit scale rather than a global fixed threshold) in four concrete forms: an explicit per-segment header used directly as a threshold, a classic fixed-mantissa-width block scheme, a marker-bit escape with an algebraically-derived baseline width, and a leading-bitmap popcount validator. None decode the payload. Along the way, quantifies a ~48% false-positive base rate for this document's own "exact count, zero leftover" acceptance criterion, explaining why several superficially promising per-file signals throughout this document's history have failed to generalize. Also tests the CheckSum stream's two still-unidentified fields against plain counts and derived sizes (segment count, point count, various byte totals) rather than checksum algorithms - no match found, which strengthens rather than resolves the "genuine but unidentified checksum" conclusion. The second pass resolves a previously open structural question: the "split" envelope form's two declared-length regions are an exact 256-wavelength-channel/remainder split, verified with zero exceptions across every flat segment in all four locally available split-form files. This also fully explains why "split" vs. "symmetric" envelope form correlates with wavelength count. Retrying the marker-bit hypothesis region-by-region with this exact channel count turns up a real, cross-file, twice-randomized-control-verified signal in the "tail" region - but decoding actual values under it and checking them for temporal smoothness (the physical-plausibility validator this document had flagged but never applied) shows only one channel out of several dozen is genuinely smooth; the rest are indistinguishable from noise. Traces this to a concrete, directly observed compensating-error mechanism rather than leaving it as a vague caveat, and confirms a smaller, precise byte-level finding (the region's first two channels are fixed edge cases) unconditionally across all four files. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu
…plausibility-check gap (Sigilweaver#2) Follows up on the 256-channel region-boundary finding from the prior commit on this branch: the original per-region sweep (well before that boundary was known) isolated region A but still targeted npts tokens from it, an internally inconsistent test now that region A is known to hold only the first 256 of npts channels. Re-runs the magnitude- threshold and continuation-bit sweeps against both regions with the corrected target counts, plus a fresh sweep against MTBLS432 (whose symmetric envelope form has no region split to get wrong in the first place). Two sweeps produce dramatic-looking signals - a sharp threshold cliff in region tail beating two randomized controls across all four split-form files, and a 67.8%-clean single-file MTBLS432 result with an apparently smooth per-channel decode - and both are run through to actual value decoding and a physical-plausibility check rather than reported on the strength of a clean walk rate alone. Neither survives: the region-tail cliff reproduces the same channel-0/1 compensating- error artifact diagnosed in the prior commit, now shown to recur under a completely different-looking rule; the MTBLS432 result exposes a real gap in the existing physical-plausibility check itself, which reported smoothness for a decode that was actually 80-96% the same repeated value with rare large jumps to essentially random numbers. The check is revised to also flag high per-channel mode fraction, since mean relative step alone can't tell a frozen, repetitive decode from a genuinely smooth one. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu
…n-correct counts (Sigilweaver#2) Follows up on two leads that fall out of the 256-channel region boundary and the mode-fraction lesson from the prior two commits on this branch. Re-runs the per-byte-position entropy and conditional-entropy analysis with region A cleanly isolated at its true 256-channel boundary. The region A/tail byte ranges were already correct in the earlier analysis (they come from the envelope's own length-prefix fields, established well before the channel-count confusion that affected the decode sweeps specifically), so this mostly reproduces the earlier session's numbers almost exactly rather than finding anything new - a genuine confirmation, not a wasted check, since it rules out the possibility that a marker byte or periodicity was being smeared out by a region misalignment. No periodicity at 2- or 3-byte spacing is found anywhere in region A's true span beyond the already-explained early-channel low-variance artifact. Rebuilds the joint temporal+magnitude dynamic program (the original was not saved) scoped to the correct per-region target counts, and adds an optional cost-function term that penalizes bit-identical decoded values between two segments, directly targeting the mode-collapse failure mode diagnosed in the prior commit. Re-testing the original single-pair "true beats random" claim across 20 pairs instead of one finds the underlying signal real but much weaker than the original anecdote suggested (true pair cheaper in only 12 of 20 individual pairs, versus a single dramatic 1.8x example previously). The anti-mode-collapse penalty does not improve, and on this sample appears to worsen, the correlation between solution cost and width agreement - a concrete negative result for the specific fix tried, not a validation of it. Region A's larger target count is computationally intractable for the exact DP as implemented, confirming the scaling limitation the original session anticipated. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu
…closes this investigation round (Sigilweaver#2) Deliberately inverts this branch's prior approach: instead of another automated parameter sweep, does close, manual, hypothesis-light reading of actual segment bytes, reaching for a parametrized test only once something specific in the raw data suggested one. Byte-diffing two temporally-adjacent MTBLS432 real-mode segments by hand finds a real, non-obvious technique: segments whose body length happens to equal exactly 3 * npts need no decode algorithm at all to align to a token grid, since position channel*3 is then unambiguously each channel's leading byte. Using this to read a run of consecutive such segments turns up an initially striking pattern - several channels whose leading byte drifts smoothly build-to-build - but checking it properly (all adjacent exact-length pairs in one file, then two more files with longer runs) shows the "smooth" channels are exactly the ones with the fewest distinct values, with a 0.49-0.84 correlation between a channel's smoothness score and its value diversity across three independent files. This is the same class of problem as the mode-fraction lesson from two commits ago, generalized: low value diversity alone, not just outright mode-domination, mechanically produces a good smoothness score regardless of decode correctness. Hand-inspecting the flat-to-real transition segment itself (the segment immediately after the last all-zero baseline segment, in two files) finds no leading or trailing quiet region, no length-prefix-looking field, and scattered rather than clustered zero bytes - direct, by-eye confirmation of the already-established "hard, instantaneous cliff" finding rather than a new discovery. Neither line of manual inquiry produced a byte pattern specific enough to justify a new parametrized sweep. Recorded as a legitimate close to this investigation for now, per the requester's own framing: the per-point payload grammar remains undecoded after six same-day sessions, and every promising-looking signal at this scale keeps resolving to already-characterized artifacts rather than the real grammar. No Rust code changes: the payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_011JNZ3Q26Pg9qzn2LEiUuTu
Nabejo
changed the title
…, rule both out (Sigilweaver#2) Continues the PDA/chromatogram payload investigation from the prior commit on this branch, taking a genuinely new angle: cross-referencing the mzML/PSI-MS open spec's own published compression schemes, rather than sweeping more parameters of the byte-granular framings every prior session already tried. Tests two concrete, public, non-Shimadzu ideas: MS-Numpress (the PSI-MS-affiliated lossy compression scheme for mzML binary data arrays, Teleman et al. 2014), whose integer encoding operates at nibble (half-byte) granularity rather than the byte-aligned tokens every previous continuation-bit/threshold sweep assumed; and whether the payload is simply a standard zlib/DEFLATE stream, the actual mechanism mzML itself uses to compress binary data arrays. Both are ruled out. Two nibble-varint variants (a generic per-nibble continuation flag, and the length-prefixed scheme closer to Numpress's own published encodeInt) were swept across three independent files and four region targets: essentially 0% clean everywhere, including one 80-way sweep that produced a nonzero 2.68% hit rate on a single file - disqualified by a shuffled-byte control (80% of the "signal" survives scrambling byte order entirely) and by collapsing to under 1% on that file's own sibling accessions. zlib/DEFLATE framing showed a small 0.79% hit rate that turned out statistically indistinguishable from both a random-byte and a shuffled-byte control, and every "successful" decompression produced a degenerate 1-byte output - raw DEFLATE's well-known weak self-framing, not real compressed content. No Rust code changes: the per-point payload grammar itself is still undecoded, so nothing is wired into crates/openszraw, consistent with this project's clean-room "don't guess" policy. Claude-Session: https://claude.ai/code/session_013ac4gwERRkHA4fMijh4eUF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continues the PDA/chromatogram payload investigation tracked in
#2: six same-day sessions of clean-room analysis.
Still no decode of the per-value payload grammar - consistent with
this project's own precedent (#7) that an honest "still open, here's
what's ruled out" write-up is a valid, mergeable outcome - but this
includes a resolved structural question and several methodological
improvements to how this document validates candidate decodes.
Confirmed findings:
PDA 3D Raw Data/CheckSumfields are exactstream byte sizes, correcting an earlier "flat vs. real flag"
reading.
256-wavelength-channel/remainder split, also explaining why "split"
vs. "symmetric" form correlates with wavelength count.
Ruled out (each run through actual value decoding, a
physical-plausibility check, and randomized controls, not just
walk-rate/zero-leftover matching):
candidates for the remaining
CheckSumfields.but traced to a compensating-error artifact.
per-region target counts.
temporal+magnitude decoder (region A's DP also confirmed
computationally intractable at the current approach).
manual (non-automated-sweep) byte-reading pass, which traced to the
same low-value-diversity artifact on three independent files.
Methodology improvements: quantified a ~48% false-positive base
rate for this document's zero-leftover acceptance test, and fixed a
real gap in the physical-plausibility check (mode-dominated/
low-diversity decodes can look deceptively "smooth" under mean
relative step alone).
Contributed by @Nabejo.
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspaceundecoded, so nothing is wired into
crates/openszraw.