diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b92ec1..0a961f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 the `\x05SummaryInformation` stream (which `.lcd`/`.qgd` don't carry - see `docs/format/06-known-limitations.md` #9). Resolves Sigilweaver/OpenSZRaw#9. +- `LC Raw Data/Chromatogram Ch6` (a conventional UV/RID-style LC detector + channel, distinct from the still-undecoded `PDA 3D Raw Data`) is now + decoded and exposed via `SpectrumSource::iter_chromatograms`, tagged + with PSI-MS `MS:1000811` "electromagnetic radiation chromatogram" - see + `docs/format/04-lcd-chromatogram-pda.md` and + `docs/format/06-known-limitations.md` #11. `Ch5` shares the same + verified page/tokenization framing but is deliberately left undecoded: + every sample in every locally available file is the same single + repeated value, which makes its numeric grammar (delta vs. absolute + convention) untestable from this corpus rather than resolved. Partially + resolves Sigilweaver/OpenSZRaw#21. + + Contributed by @Nabejo. ### Documentation diff --git a/crates/openszraw/src/lib.rs b/crates/openszraw/src/lib.rs index 7b0871e..1d3e32b 100644 --- a/crates/openszraw/src/lib.rs +++ b/crates/openszraw/src/lib.rs @@ -18,6 +18,12 @@ //! top-level CFBF storage is present, never by filename. `PDA 3D Raw Data` //! (secondary UV detector) is out of scope for this crate, see //! `docs/format/04-lcd-chromatogram-pda.md`. +//! +//! A separate, unrelated stream - `LC Raw Data/Chromatogram ChN` +//! (conventional UV/RID-style LC detector channels) - *is* decoded and +//! exposed via [`openmassspec_core::SpectrumSource::iter_chromatograms`], +//! see `raw::lc_chrom` and `docs/format/04-lcd-chromatogram-pda.md`'s +//! "LC Raw Data Chromatogram Ch5/Ch6 decode" section. #![cfg_attr(not(test), warn(clippy::unwrap_used, clippy::expect_used))] diff --git a/crates/openszraw/src/raw/lc_chrom.rs b/crates/openszraw/src/raw/lc_chrom.rs new file mode 100644 index 0000000..1bf4c47 --- /dev/null +++ b/crates/openszraw/src/raw/lc_chrom.rs @@ -0,0 +1,334 @@ +//! Parsing of `.lcd` `LC Raw Data/Chromatogram ChN` streams - a +//! conventional LC detector channel (UV/RID-style), structurally +//! unrelated to `PDA 3D Raw Data` despite sharing the same outer +//! `RC\x00\x00` segment header. +//! +//! See `docs/format/04-lcd-chromatogram-pda.md`'s "LC Raw Data +//! Chromatogram Ch5/Ch6 decode" session (Sigilweaver/OpenSZRaw#21) for the +//! full derivation and evidence this implements. + +use byteorder::{ByteOrder, LittleEndian}; + +/// Magic for the 24-byte `RC\x00\x00` segment header shared by +/// `PDA 3D Raw Data` and `LC Raw Data` streams (see +/// `docs/format/04-lcd-chromatogram-pda.md`'s "Segment Header" section). +const SEGMENT_MAGIC: u32 = 17234; +const SEGMENT_HEADER_SIZE: usize = 24; + +/// Byte value at/above which a sample token widens from 1 byte to 2 +/// bytes. The unique zero-exception result of an exhaustive +/// `(threshold in 0..=255) x (wide_width in 2..=4)` sweep requiring every +/// page of a stream to decode to exactly its declared point count with no +/// leftover bytes - see docs/format/04. Confirmed with zero exceptions +/// across every page of `Chromatogram Ch5`/`Ch6` in all 5 locally +/// available `PXD020792` files. +const WIDE_TOKEN_THRESHOLD: u8 = 0x20; + +/// Nominal LC channel sample interval, in seconds (2 Hz). Derived from +/// cross-referencing the fixed 7200-sample stream length against +/// `TTFL Raw Data/Retention Time`'s own max value (consistently just +/// under, never over, 3600s / 60 minutes across all 5 corpus files) - see +/// docs/format/04. Not itself read from any field in the `LC Raw Data` +/// stream; a corpus-derived constant, like `docs/format/03`'s RLE +/// decoder constants. +pub const SAMPLE_INTERVAL_SEC: f64 = 0.5; + +/// One decoded `LC Raw Data/Chromatogram ChN` stream: a dense, +/// evenly-spaced (RT, intensity) time series. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct LcChromatogram { + pub time_sec: Vec, + pub intensity: Vec, +} + +/// Split a `LC Raw Data/Chromatogram ChN` payload into its internal +/// "pages": each page is a `u16` LE byte-length prefix, that many data +/// bytes, then a `u16` LE suffix equal to the prefix - back to back, no +/// gap between one page's suffix and the next page's prefix. This is the +/// internal sub-segment structure the PDA payload doc's "LC Raw Data" +/// section flagged as not yet characterized: the whole stream is one +/// giant `RC\x00\x00` segment (`u32[3]` spans the entire rest of the +/// stream), but *within* that segment's body, this length-prefix/suffix +/// framing repeats - the same "first/last length word" wrapper style +/// already documented for the PDA payload's "symmetric" envelope form, +/// just applied per-page rather than per-segment. Verified zero-exception +/// (every page's prefix equals its suffix, every stream fully consumed) +/// against every one of the 5 `PXD020792` corpus files' `Ch5` and `Ch6` +/// streams (290 pages total). +/// +/// Returns `None` if the payload is not an exact, zero-leftover sequence +/// of such pages. +fn parse_pages(payload: &[u8]) -> Option> { + let mut pages = Vec::new(); + let mut off = 0usize; + let n = payload.len(); + while off < n { + if off + 2 > n { + return None; + } + let plen = LittleEndian::read_u16(&payload[off..off + 2]) as usize; + let start = off + 2; + let end = start.checked_add(plen)?; + if end + 2 > n { + return None; + } + let suffix = LittleEndian::read_u16(&payload[end..end + 2]) as usize; + if suffix != plen { + return None; + } + pages.push(&payload[start..end]); + off = end + 2; + } + Some(pages) +} + +/// Decode one page's bytes into signed per-sample deltas. +/// +/// Tokenization: a byte `b < 0x20` is a 1-byte literal, read as a signed +/// 5-bit two's-complement value (`0..=15` -> `+0..=+15`, `16..=31` -> +/// `-16..=-1`). A byte `b >= 0x20` (`WIDE_TOKEN_THRESHOLD`) starts a +/// 2-byte "wide" token: `b`'s low 5 bits become the high bits of a signed +/// 13-bit value, the following byte the low 8 bits. This +/// `(threshold, width) = (0x20, 2)` rule is the *only* combination (of an +/// exhaustive 256x3 sweep) that decodes every page of every locally +/// available `Ch5`/`Ch6` stream to exactly its expected sample count with +/// zero leftover bytes - a fully verified structural fact. +/// +/// The signed-value interpretation (5-bit literal, 13-bit wide) is a +/// strong but not byte-exact-certain hypothesis: cumulative-summing the +/// decoded deltas produces a smooth, low-magnitude, physically plausible +/// chromatogram trace (baseline near zero, a single broad rise, then a +/// stable plateau of similar absolute magnitude) consistently across all +/// 5 corpus files, clearly outperforming (by an order of magnitude in +/// mean per-sample delta size) every alternative byte-order/bit-width +/// layout tried - see docs/format/04 for the comparison. This is a soft, +/// physical-plausibility validator (the kind docs/format/04's own +/// "Further avenues" section recommends), not a byte-exact proof. +fn decode_page_tokens(page: &[u8]) -> Option> { + let mut out = Vec::new(); + let mut i = 0usize; + let n = page.len(); + while i < n { + let b = page[i]; + if b >= WIDE_TOKEN_THRESHOLD { + if i + 1 >= n { + return None; // truncated wide token + } + let b1 = page[i + 1]; + let raw = (i32::from(b & 0x1F) << 8) | i32::from(b1); + let signed = if raw >= 4096 { raw - 8192 } else { raw }; + out.push(signed); + i += 2; + } else { + let signed = if b < 16 { + i32::from(b) + } else { + i32::from(b) - 32 + }; + out.push(signed); + i += 1; + } + } + Some(out) +} + +/// Decode a full `LC Raw Data/Chromatogram ChN` stream. +/// +/// Returns `None` when: +/// - the stream is too short or its 24-byte header magic doesn't match +/// (not a `RC\x00\x00` segment at all); +/// - the page framing or tokenization doesn't cleanly consume the whole +/// payload (a malformed or differently-encoded stream); +/// - the decoded sample count doesn't match the segment header's own +/// declared point count (`u32[2]`); +/// - **every decoded delta is identical** (fewer than 2 distinct decoded +/// token values across the whole stream). This is deliberate, not a +/// parsing failure: every locally available file's `Chromatogram Ch5` +/// decodes cleanly under the rule above but is a single repeated token +/// for its entire ~1-hour run, which makes it impossible to tell +/// whether the delta+cumulative-sum interpretation established from +/// `Ch6`'s real, varying signal actually applies to it, or whether it +/// is some other convention (e.g. a plain absolute reading) that just +/// happens to look identical when the value never changes - see +/// docs/format/04. Emitting a delta-integrated trace for such a stream +/// risks shipping a fabricated, possibly-wrong unbounded ramp instead +/// of the flat channel the source data almost certainly represents; +/// skipping it is the honest choice per `CONTRIBUTING.md`'s clean-room +/// policy. A future file whose `Ch5` (or any other channel) actually +/// varies would decode normally under this same rule. +pub fn decode_stream(data: &[u8]) -> Option { + if data.len() < SEGMENT_HEADER_SIZE { + return None; + } + let magic = LittleEndian::read_u32(&data[0..4]); + if magic != SEGMENT_MAGIC { + return None; + } + let npts = LittleEndian::read_u32(&data[8..12]) as usize; + let blocksz = LittleEndian::read_u32(&data[12..16]) as usize; + if blocksz < SEGMENT_HEADER_SIZE || blocksz > data.len() { + return None; + } + let payload = &data[SEGMENT_HEADER_SIZE..blocksz]; + let pages = parse_pages(payload)?; + + let mut cumulative: i64 = 0; + let mut time_sec = Vec::with_capacity(npts); + let mut intensity = Vec::with_capacity(npts); + let mut distinct_deltas = std::collections::HashSet::new(); + let mut sample_idx: usize = 0; + for page in pages { + let deltas = decode_page_tokens(page)?; + for d in deltas { + distinct_deltas.insert(d); + cumulative += i64::from(d); + #[allow(clippy::cast_precision_loss)] + let t = sample_idx as f64 * SAMPLE_INTERVAL_SEC; + time_sec.push(t as f32); + #[allow(clippy::cast_precision_loss)] + intensity.push(cumulative as f32); + sample_idx += 1; + } + } + if sample_idx != npts { + return None; // didn't consume exactly the declared point count + } + if distinct_deltas.len() < 2 { + return None; // single repeated value: grammar untestable, see doc comment above + } + Some(LcChromatogram { + time_sec, + intensity, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn write_header(buf: &mut Vec, npts: u32, blocksz: u32) { + buf.extend_from_slice(&SEGMENT_MAGIC.to_le_bytes()); + buf.extend_from_slice(&1u32.to_le_bytes()); // version + buf.extend_from_slice(&npts.to_le_bytes()); + buf.extend_from_slice(&blocksz.to_le_bytes()); + buf.extend_from_slice(&0u32.to_le_bytes()); // padding + buf.extend_from_slice(&0u32.to_le_bytes()); // padding + } + + fn write_page(buf: &mut Vec, page_bytes: &[u8]) { + let len = page_bytes.len() as u16; + buf.extend_from_slice(&len.to_le_bytes()); + buf.extend_from_slice(page_bytes); + buf.extend_from_slice(&len.to_le_bytes()); + } + + #[test] + fn parse_pages_splits_back_to_back_length_wrapped_pages() { + let mut payload = Vec::new(); + write_page(&mut payload, &[1, 2, 3]); + write_page(&mut payload, &[4, 5]); + let pages = parse_pages(&payload).expect("parse"); + assert_eq!(pages, vec![&[1u8, 2, 3][..], &[4u8, 5][..]]); + } + + #[test] + fn parse_pages_rejects_mismatched_suffix() { + let mut payload = Vec::new(); + payload.extend_from_slice(&3u16.to_le_bytes()); + payload.extend_from_slice(&[1, 2, 3]); + payload.extend_from_slice(&4u16.to_le_bytes()); // wrong suffix + assert!(parse_pages(&payload).is_none()); + } + + #[test] + fn decode_page_tokens_reads_signed_5bit_literals() { + // 0 -> 0, 1 -> +1, 15 -> +15, 16 -> -16, 31 -> -1. + let page = [0u8, 1, 15, 16, 31]; + let out = decode_page_tokens(&page).expect("decode"); + assert_eq!(out, vec![0, 1, 15, -16, -1]); + } + + #[test] + fn decode_page_tokens_reads_signed_13bit_wide_tokens() { + // b=0x20 (low 5 bits 0), b1=0x01 -> raw 1 -> +1. + // b=0x3F (low 5 bits 0x1F), b1=0xFF -> raw 0x1FFF (8191) -> -1. + let page = [0x20u8, 0x01, 0x3F, 0xFF]; + let out = decode_page_tokens(&page).expect("decode"); + assert_eq!(out, vec![1, -1]); + } + + #[test] + fn decode_page_tokens_rejects_truncated_wide_token() { + let page = [0x20u8]; + assert!(decode_page_tokens(&page).is_none()); + } + + #[test] + fn decode_stream_rejects_bad_magic() { + let mut data = vec![0u8; SEGMENT_HEADER_SIZE]; + LittleEndian::write_u32(&mut data[0..4], 0xDEAD_BEEF); + assert!(decode_stream(&data).is_none()); + } + + #[test] + fn decode_stream_rejects_too_short_input() { + assert!(decode_stream(&[0u8; 4]).is_none()); + } + + #[test] + fn decode_stream_skips_a_single_valued_stream() { + // Reproduces the real Ch5 signature: every sample is the same + // wide token (here, delta +1 repeated) for the whole stream - + // grammar untestable, must be skipped rather than integrated + // into a fabricated ramp. + let mut payload = Vec::new(); + let page: Vec = std::iter::repeat_n([0x20u8, 0x01], 4).flatten().collect(); + write_page(&mut payload, &page); + + let mut data = Vec::new(); + let blocksz = (SEGMENT_HEADER_SIZE + payload.len()) as u32; + write_header(&mut data, 4, blocksz); + data.extend_from_slice(&payload); + + assert!(decode_stream(&data).is_none()); + } + + #[test] + fn decode_stream_round_trips_a_varying_two_page_stream() { + // Page 1: deltas +2, +2, -1 (all literal). Page 2: deltas +1, + // wide +100. Total 5 samples, npts must match. + let mut payload = Vec::new(); + write_page(&mut payload, &[2, 2, 31]); // +2, +2, -1 + let wide = 100i32; + let b0 = 0x20 | ((wide >> 8) as u8 & 0x1F); + let b1 = (wide & 0xFF) as u8; + write_page(&mut payload, &[1, b0, b1]); // +1, +100 + + let mut data = Vec::new(); + let blocksz = (SEGMENT_HEADER_SIZE + payload.len()) as u32; + write_header(&mut data, 5, blocksz); + data.extend_from_slice(&payload); + + let chrom = decode_stream(&data).expect("decode"); + assert_eq!(chrom.intensity.len(), 5); + assert_eq!(chrom.time_sec.len(), 5); + // cumulative sum: 2, 4, 3, 4, 104 + assert_eq!(chrom.intensity, vec![2.0, 4.0, 3.0, 4.0, 104.0]); + assert_eq!(chrom.time_sec[1], SAMPLE_INTERVAL_SEC as f32); + assert_eq!(chrom.time_sec[4], 4.0 * SAMPLE_INTERVAL_SEC as f32); + } + + #[test] + fn decode_stream_rejects_point_count_mismatch() { + let mut payload = Vec::new(); + write_page(&mut payload, &[2, 2, 31]); + + let mut data = Vec::new(); + let blocksz = (SEGMENT_HEADER_SIZE + payload.len()) as u32; + // Declare 5 points but only 3 are present. + write_header(&mut data, 5, blocksz); + data.extend_from_slice(&payload); + + assert!(decode_stream(&data).is_none()); + } +} diff --git a/crates/openszraw/src/raw/mod.rs b/crates/openszraw/src/raw/mod.rs index e7f6837..3c73fe6 100644 --- a/crates/openszraw/src/raw/mod.rs +++ b/crates/openszraw/src/raw/mod.rs @@ -1,5 +1,6 @@ //! Low-level parsing modules for `.qgd` and `.lcd` files. +pub mod lc_chrom; pub mod qgd; pub mod qtfl; pub mod timestamp; diff --git a/crates/openszraw/src/reader.rs b/crates/openszraw/src/reader.rs index 9188b10..33e3ea2 100644 --- a/crates/openszraw/src/reader.rs +++ b/crates/openszraw/src/reader.rs @@ -10,10 +10,11 @@ use std::path::Path; use byteorder::{ByteOrder, LittleEndian}; use cfb::CompoundFile; use openmassspec_core::{ - Analyzer, CvTerm, PrecursorInfo, RunMetadata, ScanMode, SpectrumRecord, SpectrumSource, + Analyzer, ChromatogramRecord, CvTerm, PrecursorInfo, RunMetadata, ScanMode, SpectrumRecord, + SpectrumSource, }; -use crate::raw::{self, qgd, qtfl, ttfl, Variant}; +use crate::raw::{self, lc_chrom, qgd, qtfl, ttfl, Variant}; const GCMS_SPECTRUM_INDEX: &str = "GCMS Raw Data/Spectrum Index"; const GCMS_MS_RAW_DATA: &str = "GCMS Raw Data/MS Raw Data"; @@ -35,6 +36,21 @@ const TTFL_TUNING_RESULT: [&str; 3] = [ "TTFL Tuning/Tuning Result 02", ]; +/// Conventional LC detector channel names observed under the `LC Raw +/// Data` top-level storage (see `docs/format/04-lcd-chromatogram-pda.md`). +/// `Ch1`..`Ch4` are 0 bytes (unpopulated) in every locally available +/// corpus file, `Ch5`/`Ch6` are real. Probed generically (not hardcoded +/// to `Ch5`/`Ch6` specifically) since a different acquisition method +/// could populate a different subset of the same 6 slots. +const LC_CHROMATOGRAM_CHANNELS: [&str; 6] = [ + "LC Raw Data/Chromatogram Ch1", + "LC Raw Data/Chromatogram Ch2", + "LC Raw Data/Chromatogram Ch3", + "LC Raw Data/Chromatogram Ch4", + "LC Raw Data/Chromatogram Ch5", + "LC Raw Data/Chromatogram Ch6", +]; + /// Decoded state for whichever of the 3 on-disk variants this file is, /// plus everything needed to decode individual scans on demand. enum Decoded { @@ -72,6 +88,14 @@ pub struct Reader { /// the container (implausible for any real file) has an unset creation /// time. start_timestamp: Option, + /// Decoded `(stream_name, chromatogram)` pairs from `LC Raw Data`, + /// see `raw::lc_chrom`. Only channels present *and* that decoded + /// cleanly (see `lc_chrom::decode_stream`'s doc comment on why a + /// single-valued stream like the corpus's `Ch5` is skipped rather + /// than guessed) end up here - usually empty for `.qgd`/QTOF files, + /// since `LC Raw Data` has only been observed populated on IT-TOF + /// files in the local corpus. + lc_chromatograms: Vec<(String, lc_chrom::LcChromatogram)>, } fn parse_u32_array(data: &[u8]) -> Vec { @@ -151,11 +175,25 @@ impl Reader { } }; + // `LC Raw Data/Chromatogram ChN` is a separate top-level storage + // from whichever `Decoded` variant above - present (when at all) + // alongside any of the 3 variants, not gated by `variant` here. + // See `raw::lc_chrom` and `LC_CHROMATOGRAM_CHANNELS`'s doc comment. + let lc_chromatograms: Vec<(String, lc_chrom::LcChromatogram)> = LC_CHROMATOGRAM_CHANNELS + .iter() + .filter_map(|path| { + let bytes = raw::read_stream_opt(&mut comp, path)?; + let chrom = lc_chrom::decode_stream(&bytes)?; + Some(((*path).to_string(), chrom)) + }) + .collect(); + Ok(Reader { stem, variant, decoded, start_timestamp, + lc_chromatograms, }) } } @@ -479,4 +517,37 @@ impl SpectrumSource for Reader { }; Box::new(spectra.into_iter()) } + + /// Chromatogram traces decoded from `LC Raw Data/Chromatogram ChN` + /// (see `raw::lc_chrom` and `docs/format/04-lcd-chromatogram-pda.md`). + /// Empty for files where none of the 6 conventional channel slots + /// are both present and cleanly decodable - most files in this + /// crate's scope, since this data has only been confirmed populated + /// on IT-TOF `.lcd` files in the local corpus. + fn iter_chromatograms<'a>(&'a mut self) -> Box + 'a> { + let records: Vec = self + .lc_chromatograms + .iter() + .enumerate() + .map(|(index, (id, chrom))| ChromatogramRecord { + index, + id: id.clone(), + // PSI-MS MS:1000811 "electromagnetic radiation + // chromatogram" is the standard term for a conventional + // (non-MS) LC detector trace such as UV/RID - not + // MS:1000235 "total ion current chromatogram", which + // `ChromatogramRecord::chromatogram_type: None` would + // default to in the mzML writer. + chromatogram_type: Some(CvTerm::new( + "MS:1000811", + "electromagnetic radiation chromatogram", + )), + precursor_mz: None, + product_mz: None, + time_sec: chrom.time_sec.clone(), + intensity: chrom.intensity.clone(), + }) + .collect(); + Box::new(records.into_iter()) + } } diff --git a/crates/openszraw/tests/conformance.rs b/crates/openszraw/tests/conformance.rs index 8ed4919..0b7ddc4 100644 --- a/crates/openszraw/tests/conformance.rs +++ b/crates/openszraw/tests/conformance.rs @@ -24,6 +24,13 @@ fn ttfl_fixture() -> PathBuf { PathBuf::from("/workspaces/Projects/Data/SZRaw/PXD020792/UY02-01-01p400.LCD") } +/// Same file as `ttfl_fixture`; `PXD020792` is also where +/// `LC Raw Data/Chromatogram Ch5`/`Ch6` were found populated (see +/// `docs/format/04-lcd-chromatogram-pda.md`, Sigilweaver/OpenSZRaw#21). +fn ttfl_lc_chromatogram_fixture() -> PathBuf { + ttfl_fixture() +} + /// Regression fixture for the `Data Index` `entry_i`/trailing-partial- /// block bug fixed this session - see the addendum in /// `docs/format/03-lcd-ttfl-msdata.md` and @@ -274,3 +281,63 @@ fn ttfl_base_peak_mz_is_plausible_for_most_scans() { base_peaks.len() ); } + +/// See `raw::lc_chrom` and docs/format/04's "LC Raw Data Chromatogram +/// Ch5/Ch6 decode" session (Sigilweaver/OpenSZRaw#21). Only `Ch6` is +/// expected to decode - `Ch5` is deliberately skipped (single repeated +/// value in every corpus file, grammar untestable, see +/// `lc_chrom::decode_stream`'s doc comment). +#[test] +fn ttfl_lc_chromatogram_ch6_decodes_and_is_plausible() { + let Some(mut reader) = open_or_skip(&ttfl_lc_chromatogram_fixture()) else { + return; + }; + let chroms: Vec<_> = reader.iter_chromatograms().collect(); + assert_eq!( + chroms.len(), + 1, + "expected exactly one decoded LC chromatogram (Ch6); Ch5 must be skipped" + ); + let ch6 = &chroms[0]; + assert!(ch6.id.ends_with("Ch6"), "unexpected id: {}", ch6.id); + assert_eq!(ch6.intensity.len(), 7200); + assert_eq!(ch6.time_sec.len(), 7200); + + // Retention time must be non-decreasing and evenly spaced at the + // corpus-derived 0.5s sample interval. + for w in ch6.time_sec.windows(2) { + assert!( + (w[1] - w[0] - 0.5).abs() < 1e-3, + "sample interval not ~0.5s: {} -> {}", + w[0], + w[1] + ); + } + + // Physical plausibility (docs/format/04's "Further avenues" section + // explicitly endorses this as a soft validator): intensities should + // be non-negative and stay within the corpus's observed dynamic + // range, not blow up into an unbounded ramp. + for &v in &ch6.intensity { + assert!(v.is_finite(), "non-finite intensity {v}"); + assert!( + (0.0..20_000.0).contains(&v), + "implausible LC Ch6 intensity {v}" + ); + } +} + +/// Confirms `Ch5` (constant across every corpus file) is not emitted as a +/// fabricated chromatogram trace - see `lc_chrom::decode_stream`'s doc +/// comment for why guessing its numeric grammar would be dishonest. +#[test] +fn ttfl_lc_chromatogram_ch5_is_not_emitted() { + let Some(mut reader) = open_or_skip(&ttfl_lc_chromatogram_fixture()) else { + return; + }; + let chroms: Vec<_> = reader.iter_chromatograms().collect(); + assert!( + chroms.iter().all(|c| !c.id.ends_with("Ch5")), + "Ch5 should not be emitted (single repeated value, grammar untestable)" + ); +} diff --git a/docs/format/04-lcd-chromatogram-pda.md b/docs/format/04-lcd-chromatogram-pda.md index 3c7d328..22a29bf 100644 --- a/docs/format/04-lcd-chromatogram-pda.md +++ b/docs/format/04-lcd-chromatogram-pda.md @@ -9,13 +9,16 @@ the literal path named in this document's title and in Sigilweaver/OpenSZRaw#2 - is present in every locally available corpus file's directory listing but is **0 bytes (empty) in all of them**. There is no real `LSS Raw Data` chromatogram payload to analyze in this -corpus. All of this document's segment-level findings (both the prior -session's and this session's) come from `PDA 3D Raw Data/3D Raw Data`, -which is real and populated in every file that has a PDA detector -(`MTBLS432/*.lcd`, `PXD025121/*.lcd`, `MSV000084197/20190607_NM16.lcd`). -See "LC Raw Data - a different, unrelated chromatogram stream" near the -end of this document for the one real chromatogram-shaped stream found -locally, at a different path than the issue names. +corpus. Most of this document's segment-level findings come from +`PDA 3D Raw Data/3D Raw Data`, which is real and populated in every file +that has a PDA detector (`MTBLS432/*.lcd`, `PXD025121/*.lcd`, +`MSV000084197/20190607_NM16.lcd`), and remains undecoded. See "LC Raw +Data - a different, unrelated chromatogram stream" and the "2026-07-20 +session (LC Raw Data Chromatogram Ch5/Ch6 decode)" section below for the +other real chromatogram-shaped stream found locally, at a different path +than the issue names - **this one now has a working decode** (`Ch6`; +`Ch5` remains a confirmed-but-unresolved partial characterization), see +Sigilweaver/OpenSZRaw#21. ## Factsheet (quick reference - see the dated sessions below for full evidence) @@ -122,6 +125,22 @@ is a scannable summary, not a substitute for the detailed sections below quiet region, no length-prefix-looking field, zero bytes scattered not clustered - direct, by-eye confirmation of the already-established "hard, instantaneous cliff" aggregate-statistics finding. +- **`LC Raw Data/Chromatogram ChN` (a different, unrelated stream from + `PDA 3D Raw Data` - see below) is now decoded for `Ch6`.** The outer + `RC\x00\x00` segment's body is itself divided into internal "pages" (up + to 256 samples each), each independently wrapped in a `u16` LE + byte-length prefix and matching suffix - the sub-segment structure this + document previously flagged as "not yet characterized." Within a page, + a byte `< 0x20` is a signed 5-bit literal sample delta and a byte + `>= 0x20` starts a 2-byte signed-13-bit wide delta; cumulative-summing + these deltas produces a smooth, physically plausible chromatogram in + all 5 locally available files. `Ch5` shares the identical, fully + verified page/tokenization framing but is a single repeated value for + its entire run in every available file, which makes its numeric + grammar (delta vs. absolute-value convention) untestable from this + corpus - reader code deliberately skips emitting it rather than guess. + See "2026-07-21 session (LC Raw Data Chromatogram Ch5/Ch6 decode)" + below, Sigilweaver/OpenSZRaw#21. **Ruled out** (see "This session's additional ruled-out hypotheses" and "further ruled-out hypotheses" for full detail): standard unsigned @@ -187,10 +206,13 @@ both a random-byte and a shuffled-byte control (see 2026-07-20 session 7). *same* encoding as `PDA 3D Raw Data` at all - untested, since every locally available file has it empty. See "Further avenues" at the end of this document. -- The `LC Raw Data/Chromatogram Ch5`/`Ch6` stream (real, populated, but - a different/simpler-looking encoding, out of this issue's named scope) - - not decoded, not attempted beyond the initial characterization. See - "LC Raw Data..." near the end of this document. +- The `LC Raw Data/Chromatogram Ch5`/`Ch6` stream's exact wide-token bit + layout is corroborated by strong smoothness/physical-plausibility + evidence, not proven byte-exact the way the page framing and + literal/wide tokenization split are - and `Ch5`'s numeric grammar is + altogether untested (single repeated value in every available file). + See "2026-07-21 session (LC Raw Data Chromatogram Ch5/Ch6 decode)" + near the end of this document, Sigilweaver/OpenSZRaw#21. - `CheckSum` offsets 48 and 80 (confirmed content-dependent, not identified as any standard CRC/checksum algorithm nor as a plain count/derived size - see 2026-07-20 sessions 1 and 2). @@ -2488,6 +2510,167 @@ scope (`PDA 3D Raw Data` / `LSS Raw Data`): doesn't have to rediscover that the populated data lives under `LC Raw Data`, not `LSS Raw Data`, in files that have it. +**Update, 2026-07-21 (Sigilweaver/OpenSZRaw#21): this stream is now +decoded for `Ch6`.** See the next section for the full derivation, +evidence, and what remains open for `Ch5`. + +## 2026-07-21 session (LC Raw Data Chromatogram Ch5/Ch6 decode): `Ch6` fully decoded, `Ch5`'s numeric grammar remains untestable + +Picking up the lead flagged in the previous section and in "Further +avenues" below (Sigilweaver/OpenSZRaw#21: decode the simpler, real +`LC Raw Data/Chromatogram Ch5`/`Ch6` stream), this session reused +`iter_segments`'s 24-byte `RC\x00\x00` header framing as a starting +point, then found and verified the internal sub-segment structure the +prior session's characterization flagged as not yet investigated. Worked +against all 5 locally available `PXD020792/*.LCD` files' `Ch5` and `Ch6` +streams. **Result: `Ch6` decodes cleanly and produces smooth, physically +plausible chromatograms in all 5 files; `Ch5` shares the identical, +fully-verified structural framing but its numeric grammar cannot be +determined from this corpus (see below) and is deliberately left +undecoded rather than guessed.** + +### Confirmed, byte-exact: internal "page" framing inside the one giant segment + +The prior session's characterization noted the whole stream is one giant +`RC\x00\x00` segment (`u32[3]` spans the entire rest of the stream, `npts` += `u32[2]` = 7200 in every file/channel checked) - structurally different +from PDA's thousands-of-small-segments layout. What it did not check is +whether that one segment's *body* has its own internal structure. It +does: the body is a back-to-back sequence of length-wrapped "pages", each +a `u16` LE byte-length prefix, that many data bytes, then a `u16` LE +suffix **equal to the prefix** - directly analogous to the "symmetric" +envelope form already documented for PDA segments (`first_u16 == +last_u16 == len(body)`), just applied once per page instead of once per +segment. Verified by a generic parser (read `u16` length, skip that many +bytes, require the next `u16` to match, repeat until the payload is +exactly consumed) requiring **zero leftover bytes** - this succeeded on +every one of 290 pages checked (29 pages x `Ch5`/`Ch6` x 5 files), with +zero exceptions. Page lengths are consistent with (but the parser itself +does not assume) a fixed 256-sample buffer: `Ch5`'s pages are always +exactly 512 bytes (256 samples x 2 bytes, never varying - `Ch5` never +triggers a wide token, see below) except a final 64-byte page (32 +samples); `Ch6`'s pages are 256-281 bytes (256 samples at 1-2 bytes each, +depending on how many wide tokens that page needed) except a final +32-33-byte page. `28 * 256 + 32 = 7200 = npts` in both channels, in every +file. + +### Confirmed, byte-exact: the literal/wide-token tokenization split + +Within a page, walking byte-by-byte with a magnitude threshold: a byte +`< 0x20` is a 1-byte literal token, a byte `>= 0x20` starts a 2-byte wide +token (that byte plus the following byte). This `(threshold=0x20, +wide_width=2)` combination is the **unique** result of an exhaustive +`threshold in 0..=255` x `wide_width in {2,3,4}` sweep (768 combinations) +requiring every one of a representative file's 29 `Ch6` pages to decode +to exactly its expected sample count (256, or the file's own final-page +count) with zero leftover bytes - no other combination in the sweep +achieved this on even one page, let alone all 29. Re-checked against all +5 files' `Ch6` streams (145 pages total): zero exceptions. This directly +answers the prior session's open question ("no obvious escape-pair +pattern... found this session") - the escape pattern exists, it was just +not found by the earlier session's shallower pass. + +`Ch5`'s single repeated byte pair (`82 00`) is also fully consistent with +this same rule: `0x82 >= 0x20`, so every `Ch5` sample is a wide (2-byte) +token, giving a page byte length of exactly `2 * sample_count` in every +page - which is exactly what was observed (512 = 2*256, 64 = 2*32, with +zero exceptions, no page ever landing in between). `Ch5` never exercises +the literal-token branch at all in this corpus. + +### Strong (not byte-exact) evidence: the numeric interpretation of tokens + +The tokenization split above is a hard structural fact; the *numeric +value* each token represents is a genuinely separate question, addressed +next as a physical-plausibility argument (the approach this document's +own "Further avenues" section explicitly recommends), not a second +byte-exact proof. + +Interpreting each token as a **signed delta** and cumulative-summing +across a channel - literal token `b`: `b` if `b < 16` else `b - 32` +(signed 5-bit two's complement, range -16..+15); wide token `(b0, b1)`: +`((b0 & 0x1F) << 8) | b1`, sign-extended over 13 bits (range +-4096..+4095) - produces, for every one of the 5 files' `Ch6` streams, a +smooth, low-magnitude, single-ramp-then-plateau trace: baseline near 0-100 +at the run start, a broad, gradually rising region through roughly the +middle third of the run, then a stable plateau for the remainder, with a +similar absolute ceiling (~8600-8900) in all 5 files despite the 5 runs +being different samples (file names suggest a concentration/dilution +series - `p95`, `p55`, `p400`, `p180`, `p170`). Mean per-sample delta +magnitude across a full stream is **2.33**, versus **27.8-139.7** for +every alternative byte-order/bit-width layout tried (swapping which byte +holds the high bits, plain little-endian `int16`, full 16-bit +sign-extension) - an order-of-magnitude difference, not a close call. +Unsigned interpretation of the literal token was also checked and +rejected: raw byte values alternate rapidly and semi-randomly between +~0-3 and ~29-31 with no cumulative trend, which is not remotely smooth; +only the signed-5-bit + cumulative-sum interpretation produces a +plausible curve. + +Retention time per sample was derived from a corpus-wide cross-check, not +read from any field in the `LC Raw Data` stream itself: `npts` = 7200 in +every file, and `TTFL Raw Data/Retention Time`'s own max value (the MS +run's total duration) is consistently just under, never over, 3590-3597 +seconds across all 5 files - matching `7200 * 0.5s = 3600s` (60 minutes) +almost exactly, consistent with the LC channel recording a fixed 60-minute +buffer (`28` full 256-sample/128-second pages plus one 32-sample/16-second +final page) slightly longer than the shorter, variable-length actual MS +acquisition in each file. `SAMPLE_INTERVAL_SEC = 0.5` is documented as a +corpus-derived constant on this basis, the same way `docs/format/03`'s RLE +decoder constants are. + +### `Ch5`'s numeric grammar: confirmed untestable from this corpus, deliberately left undecoded + +`Ch5` is byte-identical in structure and content across all 5 corpus +files: every one of its 7200 samples, in every file, decodes to the same +wide token (`raw` = 512 under the formula above). This is a genuine +finding (the page/tokenization framing above is confirmed to apply to +`Ch5` too, with zero exceptions), but it is a **degenerate** case for the +value-grammar question: a constant delta of +512 integrated across 7200 +samples would produce an unbounded linear ramp reaching ~3.69 million, +not the flat/quiet trace this document (and the underlying `PXD020792` +corpus) consistently describes `Ch5` as - so if `Ch5` really does share +`Ch6`'s delta+cumulative-sum convention, either the wide-token numeric +formula above is specifically wrong for `Ch5`'s value range, or `82 00` +is a special sentinel (e.g. "channel inactive/unconnected") rather than a +generic magnitude token. Also notable: raw byte value `0x82` (130) never +appears anywhere in `Ch6`'s escape-byte histogram across the whole +corpus, consistent with it being `Ch5`-specific rather than a generic +wide-token value `Ch6` also happens to use. Since every locally available +file's `Ch5` is this same single unvarying value, there is no way to +empirically distinguish "delta convention, unusual sentinel value" from +"absolute-value convention, coincidentally constant" from "different +convention entirely" - all three explain the observed bytes identically. +**This is recorded as a confirmed, evidence-backed open question, not +guessed past.** `crates/openszraw::raw::lc_chrom::decode_stream` +implements this directly: it decodes any channel whose tokens show at +least 2 distinct values, and returns `None` (skip, do not emit a +chromatogram) for a single-valued stream like every corpus file's `Ch5` - +a future file whose `Ch5` (or any other channel) actually varies would +decode normally under the same rule, with no code change needed. + +### Rust implementation + +`crates/openszraw::raw::lc_chrom` implements the page framing, +tokenization, and delta/cumulative-sum decode described above, wired into +`Reader`'s `SpectrumSource::iter_chromatograms` (probing all 6 +conventional `LC Raw Data/Chromatogram ChN` channel names generically, +not hardcoded to `Ch5`/`Ch6` specifically, in case a different +acquisition method populates a different subset). Verified against the +real corpus: all 5 `PXD020792/*.LCD` files yield exactly one decoded +chromatogram (`Ch6`), `Ch5` is correctly skipped. Chromatogram records use +PSI-MS `MS:1000811` "electromagnetic radiation chromatogram" (the +standard term for a conventional, non-MS LC detector trace), not the +mzML writer's `MS:1000235` "total ion current chromatogram" default. See +`docs/format/06-known-limitations.md` for the residual-uncertainty +summary (the wide-token bit layout is strongly evidenced, not +byte-exact-proven; `Ch5` is unresolved, not merely unimplemented). + +Scripts written this session (all under `re/src/analysis/`, gitignored +per this repo's existing convention): `lc_ch56_common.py` (stream/segment +reader, reused by the scripts below), `lc_ch6_threshold_sweep.py` (the +`(threshold, wide_width)` exhaustive sweep that found the unique +`(0x20, 2)` result). + ## Further avenues for a future session The 2026-07-19 closing summary's single recommendation (sharpen the @@ -2553,15 +2736,23 @@ leads, not findings: "the 4-segment run was just too short to see real interior-channel structure" as an explanation; longer runs on two independent files show the identical low-diversity artifact, not new signal. -- **Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` - stream instead of (or before) `PDA 3D Raw Data`.** It's populated, - structurally simpler (one giant segment per channel, not thousands of - small ones), and a "quiet" channel (`Ch5`) is dominated by a single - repeating 2-byte value - a much easier starting point than the - 321-wavelength PDA case, and would still deliver real chromatogram - support even though it's outside this issue's literally-named path. - Nobody has run any of this document's decode hypotheses against it - yet. +- ~~Decode the simpler, real `LC Raw Data/Chromatogram Ch5`/`Ch6` + stream instead of (or before) `PDA 3D Raw Data`.~~ **Done** for `Ch6`, + see the "2026-07-21 session (LC Raw Data Chromatogram Ch5/Ch6 decode)" + section above (Sigilweaver/OpenSZRaw#21). Two follow-ups this raises, + neither attempted yet: (1) fetch a `PXD020792`-like accession where + `Ch5` (or some other conventionally-quiet channel slot) actually has + real signal, to test whether the wide-token numeric formula established + from `Ch6` generalizes or needs correction - the current corpus cannot + answer this, see the dated section's "`Ch5`'s numeric grammar" note; + (2) apply the *same* page-framing + tokenization approach tried here to + `PDA 3D Raw Data` itself - this document's PDA sessions never checked + for an internal sub-segment/page structure analogous to what was found + for `LC Raw Data` this session, since PDA's segments were already known + to be numerous and individually small (unlike `LC Raw Data`'s one giant + segment), but that doesn't rule out a *smaller*-scale internal page + structure within each PDA segment's body that nobody has specifically + looked for. - **Widen the corpus search specifically for a non-empty `LSS Raw Data/Chromatogram Ch*`** - the issue's literally-named target is 0 bytes in every locally available file. It's untested whether that's diff --git a/docs/format/06-known-limitations.md b/docs/format/06-known-limitations.md index 91b5b77..f9c9a3a 100644 --- a/docs/format/06-known-limitations.md +++ b/docs/format/06-known-limitations.md @@ -327,3 +327,52 @@ both a random-byte and a shuffled-byte control. See `docs/format/04-lcd-chromatogram-pda.md`'s 2026-07-20 sessions 1-7 for full detail. None of this decodes the per-value payload; that grammar is still open. + +## 11. LC Raw Data/Chromatogram Ch5/Ch6: `Ch6` decoded and wired in; `Ch5`'s numeric grammar is confirmed untestable from this corpus (PARTIALLY RESOLVES Sigilweaver/OpenSZRaw#21) + +A separate, unrelated stream from section 10 above, despite sharing the +same outer 24-byte `RC\x00\x00` segment header - see +`docs/format/04-lcd-chromatogram-pda.md`'s "2026-07-21 session (LC Raw +Data Chromatogram Ch5/Ch6 decode)" for the full derivation. Two things +worth stating precisely, since they carry different confidence levels: + +- **Byte-exact, zero-exception-verified (all 5 `PXD020792` corpus + files):** the segment body is internally divided into `u16` + length-prefixed/suffixed "pages" (the sub-segment structure + `docs/format/04` previously flagged as not yet characterized), and a + `(threshold=0x20, wide_width=2)` literal/wide-token tokenization rule + decodes every page of every file's `Ch5` and `Ch6` stream to exactly + its declared point count with zero leftover bytes. This part is not in + question. +- **Strongly evidenced but not byte-exact-certain:** the specific + numeric interpretation of a wide token (which byte holds the high + bits, how many bits, sign convention) is corroborated by a + physical-plausibility argument - cumulative-summing the decoded deltas + produces a smooth, single-ramp-then-plateau chromatogram in all 5 + files, at roughly 12x lower mean per-sample delta magnitude than any + alternative bit layout tried - rather than a second independent + byte-exact proof the way the tokenization split itself has. + +`crates/openszraw::raw::lc_chrom` implements the confirmed framing plus +the best-evidenced numeric interpretation, wired into `Reader`'s +`SpectrumSource::iter_chromatograms`. It deliberately does **not** emit a +chromatogram for `Ch5`: every one of `Ch5`'s 7200 samples, in every one +of the 5 locally available files, decodes to the exact same wide token +(raw value 512), which makes the delta-vs-absolute-value question +genuinely unanswerable from this corpus (a constant delta would integrate +into an unbounded ramp, contradicting `Ch5`'s documented flat/quiet +character - so either the wide-token formula is specifically wrong for +`Ch5`'s value, or `Ch5`'s repeated byte pair is a sentinel rather than a +generic magnitude token). `decode_stream` skips any channel whose decoded +tokens show fewer than 2 distinct values rather than guess, which also +means a future file whose `Ch5` genuinely varies would decode normally +without any code change. + +This is intentionally left as a documented open question rather than +resolved by assumption, per `CONTRIBUTING.md`'s clean-room policy +("if you can only explain a field by having watched what LabSolutions +shows for it, don't write that down - keep digging in the bytes instead, +or flag it as unresolved"). Fetching a real, varying `Ch5`-equivalent +channel from a different accession (see +`docs/format/04`'s "Further avenues" section) is the concrete next step +that could resolve it.