The current corpus (Data/SZRaw/, ~960 MB across 94 files, 6 accessions - see CORPUS.md) is far smaller than the scale this project's sibling readers typically acquire before their format work is considered well-covered (OpenARaw: 129.5 GB logical; OpenTFRaw: ~124 GB across 283 files). Nathan's usual practice is around 100 GB per format, though that's a rough guideline from past projects, not a hard floor or ceiling - the real goal is several genuine samples per acquisition mode/instrument family, not a byte-count target.
Why this matters beyond just "more data"
Some of the current format conclusions rest on very thin sample counts:
- QTOF (
.lcd QTFL) has exactly one file in the whole corpus (MSV000084197). All of docs/format/05-qtfl-centroid.md's CONFIRMED claims (intensity byte-width variance, BPI-vs-max(intensity) consistency, MS2 event-ID cycle structure) are verified against that single file. A second real QTOF source is the single highest-value acquisition target - it would either corroborate or break assumptions baked in from n=1.
- No QQQ-specific sample (e.g. LCMS-8060/8050 triple-quad) is confirmed anywhere in the corpus - only IT-TOF, QTOF, and PRIDE-unspecified/older instrument contexts so far.
.qgd GC-MS currently has two observed size-classes (~12.5 MB MRM-style runs vs ~42 MB full-scan-profile runs) from only 2 PRIDE accessions (PXD034978, PXD019638) - more GC-MS instrument generations and acquisition methods would help confirm those aren't coincidental groupings.- MTBLS432 has 93 real IT-TOF
.lcd files available and only 15 are fetched locally - the cheapest possible expansion (python -m analysis.external fetch MTBLS432 --max-files N, no new source-hunting needed) and a good first step.
Known corpus-sourcing gotchas from prior sessions (don't rediscover these)
- PRIDE's generic "Shimadzu instrument model" CV term badly undercounts real hits - some submitters tag the specific instrument model instead (e.g. "LCMS-IT-TOF", "GCMS-QP2010SE"), and PXD019638 is mistagged "quadrupole" with zero Shimadzu-identifying metadata at all, only discoverable by searching the
.qgd file extension directly. Always search file-extension keywords alongside instrument-name keywords, and verify by fetching the actual file list per candidate project, not just trusting the instruments metadata field.
- MassIVE has no working full-manifest listing API found so far - its dataset browser renders the file tree client-side via JS.
MSV000084197 (the one QTOF file) was found via manual web research, not a script. Worth another pass to see if a real API exists, since MassIVE is the only lead so far for more QTOF samples.
- Outbound FTP (port 21) is dead in this sandbox specifically (PRIDE's FTP control connection stalls mid-transfer, massive.ucsd.edu:21 never completes the handshake). Every fetch script already defaults to HTTPS for exactly this reason - don't spend time re-debugging FTP here, though it might work fine in a different environment.
Fetch tooling (already exists, gitignored, local-only)
python -m analysis.pride search <query> # find Shimadzu PRIDE projects
python -m analysis.pride fetch <accession> # download .lcd/.qgd files
python -m analysis.pride catalog # rebuild Data/SZRaw/index.csv
python -m analysis.external list # known MassIVE/MetaboLights datasets
python -m analysis.external fetch <accession> [--max-files N]
Note: Data/SZRaw/index.csv's recorded file sizes are currently stale relative to what's actually on disk (catalog sums to ~1.7 GiB against ~960 MB of real files) - rerun python -m analysis.pride catalog as part of this work rather than trusting the existing index blindly.
Done means
Meaningfully more instances per instrument family/acquisition mode, especially a second real QTOF source and at least one confirmed QQQ sample if one can be found. Update CORPUS.md's accession table and size figures, and flag in docs/format/05-qtfl-centroid.md if any new QTOF sample contradicts a claim currently backed by only the single existing file.
The current corpus (
Data/SZRaw/, ~960 MB across 94 files, 6 accessions - see CORPUS.md) is far smaller than the scale this project's sibling readers typically acquire before their format work is considered well-covered (OpenARaw: 129.5 GB logical; OpenTFRaw: ~124 GB across 283 files). Nathan's usual practice is around 100 GB per format, though that's a rough guideline from past projects, not a hard floor or ceiling - the real goal is several genuine samples per acquisition mode/instrument family, not a byte-count target.Why this matters beyond just "more data"
Some of the current format conclusions rest on very thin sample counts:
.lcdQTFL) has exactly one file in the whole corpus (MSV000084197). All ofdocs/format/05-qtfl-centroid.md's CONFIRMED claims (intensity byte-width variance, BPI-vs-max(intensity) consistency, MS2 event-ID cycle structure) are verified against that single file. A second real QTOF source is the single highest-value acquisition target - it would either corroborate or break assumptions baked in from n=1..qgdGC-MS currently has two observed size-classes (~12.5 MB MRM-style runs vs ~42 MB full-scan-profile runs) from only 2 PRIDE accessions (PXD034978, PXD019638) - more GC-MS instrument generations and acquisition methods would help confirm those aren't coincidental groupings..lcdfiles available and only 15 are fetched locally - the cheapest possible expansion (python -m analysis.external fetch MTBLS432 --max-files N, no new source-hunting needed) and a good first step.Known corpus-sourcing gotchas from prior sessions (don't rediscover these)
.qgdfile extension directly. Always search file-extension keywords alongside instrument-name keywords, and verify by fetching the actual file list per candidate project, not just trusting theinstrumentsmetadata field.MSV000084197(the one QTOF file) was found via manual web research, not a script. Worth another pass to see if a real API exists, since MassIVE is the only lead so far for more QTOF samples.Fetch tooling (already exists, gitignored, local-only)
Note:
Data/SZRaw/index.csv's recorded file sizes are currently stale relative to what's actually on disk (catalog sums to ~1.7 GiB against ~960 MB of real files) - rerunpython -m analysis.pride catalogas part of this work rather than trusting the existing index blindly.Done means
Meaningfully more instances per instrument family/acquisition mode, especially a second real QTOF source and at least one confirmed QQQ sample if one can be found. Update CORPUS.md's accession table and size figures, and flag in
docs/format/05-qtfl-centroid.mdif any new QTOF sample contradicts a claim currently backed by only the single existing file.