feat(parallelsearch): per-ID PEP for transient databases + family-aware organism detection by trishorts · Pull Request #10 · nbollis/MetaMorpheus

trishorts · 2026-06-10T12:37:37Z

Summary

Two analysis-layer improvements over the spectral-library search, plus tests.

PEP for transient databases. Train one PEP model on the base (human) search and reuse it to score every transient database ID — the transient DBs are far too small to train their own. Each match gets a model PEP and a PEP_QValue mapped onto a background PEP → PEP_QValue curve snapshotted from the decoy-rich base search. Per-database output is filtered to confident IDs (PEP_QValue < 0.05, with a QValue fallback when PEP is unavailable), and the summary reports confident counts at both 1% and 5% PEP_QValue.

Family-aware organism detection. Select detected organisms by independent evidence families (≥4 of 7 significant AND combined q ≤ threshold) computed directly from the statistical test results, instead of the old test-ratio gate that required ≥50% of all tests.

Benefits

Every reported peptide carries a calibrated confidence even though its database is tiny; output is limited to confident IDs and the dual 1%/5% columns show both operating points. On the SARS spike-in, SARS-CoV-2 peptides land at PEP_QValue ≈ 0.0002.
The organism call is now actually emitted: SARS-CoV-2 is the rank-1 detection (7/7 families, combined q = 4.7e-300), where the old gate selected nothing.

Rationale

A transient DB of a few hundred peptides cannot support its own target/decoy PEP q-value — but PEP is a per-peptide, organism-agnostic property, so a model trained on the large decoy-rich human background transfers cleanly. PEP_QValue is assigned strictly by PEP, not borrowed from the score-based QValue (the two rank matches differently). For detection, the test-ratio gate required passing ≥50% of all 78 tests, but many sub-tests structurally cannot fire for a small genome, so even a perfect spike-in (31/78) never cleared the bar; family-level evidence with a combined-q cutoff is the principled criterion.

Tests

27 unit tests — PEP curve build + lower-bound lookup, train→assign (incl. the missing-FdrInfo regression that left every transient PEP at the sentinel), the confidence gate, dual 1%/5% counts, the metrics round-trip, and the family-detection predicate — plus an end-to-end RunTask that trains PEP on bundled data and exercises the per-database assignment, the confident-output filter, the statistics, and the family writer on a live pipeline.

Stacked on the .msl search PR. Until that PR merges into ManySearchTask, this PR's diff also shows the search layer; once it merges, this collapses to just the PEP + family changes. Please review/merge the search PR first.

🤖 Generated with Claude Code

…rned mass+RT pre-filter Use mzLib .msl spectral libraries as the transient-search peptide source instead of re-digesting FASTAs. Adds a lean (fragment-less) reader that fragments on the fly, a candidate pre-filter that learns the precursor-mass tolerance and an RT (iRT->observed) calibration from the confident base PSMs and indexes the library by mass and retention time, a single merged "index of indexes" so thousands of databases load as one file, a load-ahead producer/consumer pipeline, and a CPU fragment-matching abstraction (ISpectralScorer / CpuSpectralScorer / struct-of-arrays ScoringBatch). On the SARS spike-in (12,973 virus DBs, 3 raws): ~40% faster wall-clock end-to-end (42 vs 73 min), ~5x less search-engine CPU, the merged load collapses 12,973 file-opens (~558 s) to a single ~23 s read, and zero specificity cost (every confident MSL peptide is also found by the exhaustive FASTA search). Bumps the mzLib dependency to 9.9.23 for the .msl reader types. Tests: 7 unit tests for the candidate filter (mass/RT window, ppm + 0.01 Da margin, iRT==0 mass-only), the merged "db|accession" parse, and the lower-bound helper; both read loops call the shared predicate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nbollis · 2026-06-10T19:27:36Z

Best approach would be to break out a TransientPEPAnalysisEngine : PEPAnalysisEngine so that we leave the old unchanged and have our derived class to the special work of caching the results from baseline. This pattern is used throughout the parallel search.

May require moving some things from private to protected

Done in 797ff65. Extracted TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch. It owns the train-once base model, the reuse / apply-to-transient path, and the background PEP -> PEP_QValue curve. PepAnalysisEngine is back to just the standard cross-validated workflow — the only seam moved private -> protected was _randomSeed. Mirrors TransientProteinParsimonyEngine / TransientProteinScoringAndFdrEngine as you suggested.

nbollis · 2026-06-10T19:29:32Z

+        /// <summary>Charge of the matched experimental envelope.</summary>
+        public readonly int ExperimentalCharge;
+
+        public FragmentMatch(int localProductIndex, double experimentalMz, double experimentalIntensity, int experimentalCharge)


Why this class instead of MatchedFragmentIon? Feels like unnecessary object churn.

This one belongs to the stacked .msl search PR rather than this PEP/family PR — the scoring layer only shows up in this diff because this PR is stacked on the search PR, and it collapses out once that merges into ManySearchTask. So I'm tracking it there, not here.

Agreed on the direction: keep MatchedFragmentIon as the real domain type at the engine boundary, and contain the flat FragmentMatch / batching representation as a scorer-internal implementation detail rather than a public engine API. I'll fold that into the search PR.

nbollis · 2026-06-10T19:37:37Z

+        // MERGED-INDEX mode: one .msl file holds many databases (entries tagged "db|accession"). The number
+        // of databases to search comes from inside the file, not the (count==1) file list, so parallelism
+        // must not be capped by the file count.
+        bool mergedMode = Environment.GetEnvironmentVariable("MM_PARALLELSEARCH_MERGED") == "1"


Relying on an environmental variable is tricky. Could we instead add a parameter to ParallelSearchParameters like GenerateCompositeOrganismLibrary or something like that to indicate we will spend the extra time before hand building a library to speed up searches later?

Done in 797ff65. Replaced MM_PARALLELSEARCH_MERGED with a real ParallelSearchParameters.UseMergedTransientLibrary flag, so merged-mode is on the parameter / TOML / CLI surface and reproducible instead of process state. The merged-mode branch now derives from the parameter, and a default-value test was added.

(There is still one perf knob read from the environment — MM_PARALLELSEARCH_PRODUCERS, the merged-shard producer DOP. Happy to promote that to a parameter too if you'd like it explicit; I left it as a tuning-only env var for now.)

nbollis · 2026-06-10T20:28:48Z

Issue: this PR is expanding PepAnalysisEngine with transient-search-specific behavior like baseline model caching and out-of-sample transient rescoring. That mixes the generic PEP engine with ParallelSearch-only concerns.

Original design intent: ParallelSearch already tends to preserve the baseline engine behavior and add transient-specific wrappers where needed, e.g. TransientProteinParsimonyEngine and TransientProteinScoringAndFdrEngine.

Proposed change: introduce TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch, keep PepAnalysisEngine focused on the standard workflow, and move only the required seams from private to protected to support the derived implementation.

Done in 797ff65. Extracted TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch. It owns the train-once base model, the reuse / apply-to-transient path, and the background PEP -> PEP_QValue curve. PepAnalysisEngine is back to just the standard cross-validated workflow — the only seam moved private -> protected was _randomSeed. Mirrors TransientProteinParsimonyEngine / TransientProteinScoringAndFdrEngine as you suggested.

nbollis · 2026-06-10T20:28:49Z

Issue: the task now owns long-lived PEP model state and directly coordinates baseline training plus per-database transient assignment. That makes ParallelSearchTask responsible for ML lifecycle details.

Original design intent: EngineLayer/ParallelSearch/FdrAlignment/ already gives us a narrow reuse seam for "build baseline state once, then apply it to transient results many times", but those services are intentionally small and composable.

Proposed change: I would not move all PEP behavior into FdrAlignment, since that layer is too narrow for training and predictor ownership. Instead, create a TransientPepAnalysisEngine that owns the model/predictor lifecycle, and optionally compose it with a small baseline PEP alignment helper for the passive PEP -> PEP_QValue lookup step.

Addressed in 797ff65. The model/predictor lifecycle now lives in TransientPepAnalysisEngine (new derived class under EngineLayer/ParallelSearch). The task no longer owns any ML state — it constructs the engine, calls TrainSingleModelAndAssignBasePep() once on the base search, then AssignPepFromTrainedModel(...) per transient database. The passive PEP -> PEP_QValue background lookup is held inside that engine too.

nbollis · 2026-06-10T20:28:50Z

+        // MERGED-INDEX mode: one .msl file holds many databases (entries tagged "db|accession"). The number
+        // of databases to search comes from inside the file, not the (count==1) file list, so parallelism
+        // must not be capped by the file count.
+        bool mergedMode = Environment.GetEnvironmentVariable("MM_PARALLELSEARCH_MERGED") == "1"


Issue: MM_PARALLELSEARCH_MERGED introduces a hidden execution mode that bypasses the normal parameter/TOML/GUI/CLI surface.

Original design intent: ParallelSearchParameters is already the canonical home for behavior switches like WriteTransientResultsOnly, CompressTransientSearchOutputs, and UseFamilyAwareRanking, so task behavior is explicit and reproducible.

Proposed change: replace the environment-variable gate with a real parameter on ParallelSearchParameters, something like UseMergedTransientLibrary or BuildCompositeOrganismLibrary, and derive merged-mode behavior from that instead of from process state.

Done in 797ff65. Replaced MM_PARALLELSEARCH_MERGED with a real ParallelSearchParameters.UseMergedTransientLibrary flag, so merged-mode is on the parameter / TOML / CLI surface and reproducible instead of process state. The merged-mode branch now derives from the parameter, and a default-value test was added.

(There is still one perf knob read from the environment — MM_PARALLELSEARCH_PRODUCERS, the merged-shard producer DOP. Happy to promote that to a parameter too if you'd like it explicit; I left it as a tuning-only env var for now.)

nbollis · 2026-06-10T20:28:51Z

Issue: this PR adds a lot of non-orchestration logic directly into ParallelSearchTask: PEP training, MSL calibration, merged-mode branching, confidence gating, detected-organism policy, and expanded writer coordination.

Original design intent: the task should primarily coordinate the run, manage progress/output, and compose reusable engines/services. The repo already follows that pattern in several places.

Proposed change: keep orchestration in the task, but extract the new behavior into focused collaborators: a transient PEP engine/service, an MSL calibration or load-ahead coordinator, a confidence/output policy, and a detected-organism selection policy.

Partially addressed in 797ff65, with the rest planned as focused follow-ups:

PEP — extracted to TransientPepAnalysisEngine (see the PEP threads); the task no longer owns ML lifecycle.

Confidence / output policy — already a separate concern: the membership filter (FilterToTransientDatabaseOnly) is distinct from the shared row-level confidence helper (IsConfidentMatch / FilterToConfident). See the FilterToTransientDatabaseOnly thread.

Detected-organism selection — SelectDatabasesForWritingByFamily. Its core predicate QualifiesAsDetectedOrganism is already a pure, unit-tested internal static; I'd like to lift the orchestration glue into a DetectedOrganismSelector policy in a follow-up so we can agree the seam first.

MSL calibration / load-ahead coordinator — this lives in the underlying .msl search PR (it's only visible here because this PR is stacked on it), so I'll extract it there rather than in this PEP/family PR.

Flagging the last two so we settle the boundaries before I move the code.

nbollis · 2026-06-10T20:28:52Z

+        /// <summary>Charge of the matched experimental envelope.</summary>
+        public readonly int ExperimentalCharge;
+
+        public FragmentMatch(int localProductIndex, double experimentalMz, double experimentalIntensity, int experimentalCharge)


Issue: the scorer API now introduces FragmentMatch, then TransientClassicSearchEngine immediately reconstructs MatchedFragmentIon from it. This is probably not heap churn because FragmentMatch is a reusable struct buffer, but it is still abstraction churn: the real domain type at the engine boundary is still MatchedFragmentIon.

Original design intent: the transient search path historically stayed close to the real search objects used by downstream scoring and PSM creation.

Proposed change: if batching is needed for performance, keep the intermediate flat match representation internal to the scorer/batch implementation. I would avoid making FragmentMatch part of the public engine boundary until there is a real second backend that needs that abstraction.

This one belongs to the stacked .msl search PR rather than this PEP/family PR — the scoring layer only shows up in this diff because this PR is stacked on the search PR, and it collapses out once that merges into ManySearchTask. So I'm tracking it there, not here.

Agreed on the direction: keep MatchedFragmentIon as the real domain type at the engine boundary, and contain the flat FragmentMatch / batching representation as a scorer-internal implementation detail rather than a public engine API. I'll fold that into the search PR.

nbollis · 2026-06-10T20:28:53Z

Issue: a large portion of the original search loop was rewritten to support batching, flattened scoring data, scorer providers, and sinks. Even if the performance motivation is valid, the amount of structural change makes the transient search engine much farther from the original simple wrapper.

Original design intent: TransientClassicSearchEngine started as a more memory-efficient adaptation of ClassicSearchEngine: share baseline PSMs, clone only modified PSMs, and keep the search semantics close to the classic implementation.

Proposed change: keep the optimization work, but contain it more aggressively: hide batching/helper types as internal implementation details, preserve direct use of MatchedFragmentIon at the engine boundary, and extract only the hot-path optimization rather than reshaping the whole engine API.

This one belongs to the stacked .msl search PR rather than this PEP/family PR — the scoring layer only shows up in this diff because this PR is stacked on the search PR, and it collapses out once that merges into ManySearchTask. So I'm tracking it there, not here.

Agreed on the direction: keep MatchedFragmentIon as the real domain type at the engine boundary, and contain the flat FragmentMatch / batching representation as a scorer-internal implementation detail rather than a public engine API. I'll fold that into the search PR.

nbollis · 2026-06-10T20:34:45Z

        string dbName = context.DatabaseName;
        string outputFolder = context.OutputFolder;
        List<string> nestedIds = context.NestedIds;
        bool writeAllResults = !ParallelSearchParameters.WriteTransientResultsOnly;


I think we can reuse the existing transient-only filter path here rather than introducing a separate writer-side confidence path, but I would avoid baking a confidenceOnly boolean directly into FilterToTransientDatabaseOnly(...).\n\nIssue: that helper's original responsibility is database-membership filtering, not confidence policy. A hardcoded confidence switch would blur those concerns and make the helper less reusable.\n\nOriginal design intent: FilterToTransientDatabaseOnly(...) answers "does this match belong to the transient DB set?" and is a good seam to preserve.\n\nProposed change: keep it as the transient-membership filter, but make it composable with an optional predicate or additionalFilter. Then the transient writer can reuse the existing path with something like p => IsConfident(p, peptideLevel: false) while preserving the original responsibility of the method. For the AllPsms / AllPeptides outputs, I would still use a small lower-level shared filter helper rather than forcing everything through the transient-only method.

Agreed — and this is already how the current code is structured, so there's no confidenceOnly boolean baked into the membership filter. FilterToTransientDatabaseOnly(...) stays a pure "does this match belong to the transient DB set?" filter. Confidence is applied separately by a shared row-level helper (IsConfidentMatch / FilterToConfident) that the per-database writer composes on top, and the AllPsms / AllPeptides outputs go through that same lower-level helper rather than being forced through the transient-only path. That matches the seam you proposed. (Confirmed in 797ff65, where the confidence predicate also changed — see the "confident by either metric" thread.)

nbollis · 2026-06-10T20:39:14Z

+    /// QValue being at or below <paramref name="qThreshold"/>. A null FdrInfo is never confident.
+    /// </summary>
+    internal static bool IsConfidentMatch(EngineLayer.FdrAnalysis.FdrInfo info, bool pepActive, double pepQThreshold, double qThreshold)
+    {


For writing, I think the threshold should be confident by either metric instead of by PEP alone if we run PEP (which will be the new default) These written files are for the user to investigate any potential (reasonable) hit to the database and thus the few edge case PSMs/peptides should be written as well.

Done in 797ff65. When a PEP model is active (the new default), IsConfidentMatch now treats a match as confident by either metric — PEP_QValue < 5% OR the score-based QValue <= 5% — so the few reasonable edge-case PSMs/peptides that only one metric flags are still written for the user to investigate. When PEP is unavailable it falls back to the score-based q-value alone. ConfidenceFilterTests were updated to cover the either-metric cases.

nbollis

Add the optimization attempts markdown to the wiki

nbollis

Add the optimization attempts markdown to the wiki

nbollis · 2026-06-10T20:55:58Z

The purpose of the major scoring refactor was to convert the transient search inner loop into a backend-swappable, data-oriented scoring pipeline, with GPU as the motivating future backend and .msl/reuse/caching as the practical immediate benefit.
But if GPU is already known to be slower, then I would question whether the full abstraction is still warranted. In that world, the best version is probably:

keep the .msl and scan-caching wins,

keep only the CPU improvements that benchmark positively,

and remove or shrink the GPU-oriented abstraction layer unless there is a concrete next backend.

Agreed, and this belongs to the stacked .msl search PR (it only appears in this diff because this PR is stacked on it). I'll handle it there: since GPU benchmarks slower, I'll shrink the backend abstraction down to what actually pays off — the .msl index + scan-caching wins and the CPU improvements that benchmark positively — rather than keeping a speculative GPU seam with no concrete second backend. Tracking on the search PR.

…d dedicated seams - Extract TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch so the generic PEP engine keeps only its standard cross-validated workflow; the train-once / apply-to-transient model reuse and the background PEP->PEP_QValue curve move to the derived class. Only the _randomSeed seam moves private->protected (mirrors TransientProteinParsimony/ ScoringAndFdr engines). - Replace the MM_PARALLELSEARCH_MERGED environment variable with a real ParallelSearchParameters.UseMergedTransientLibrary flag so merged-mode is on the parameter/TOML/CLI surface and reproducible instead of process state. - Per-database output now counts a match confident by EITHER its PEP_QValue or the score-based q-value when PEP is active, so the few reasonable edge-case hits a single metric flags are still written for the user to investigate. The membership filter (FilterToTransientDatabaseOnly) stays separate from the shared row-level confidence filter. Addresses nbollis review on nbollis#10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

trishorts · 2026-06-15T16:10:55Z

Thanks for the review. Pushed 797ff65 addressing the architecture feedback on this PR's own surface:

TransientPepAnalysisEngine : PepAnalysisEngine extracted under EngineLayer/ParallelSearch — the base PEP engine keeps only its standard cross-validated workflow; the train-once / reuse-across-transient-DBs model and the background PEP -> PEP_QValue curve move to the derived class (only _randomSeed went private -> protected).
MM_PARALLELSEARCH_MERGED env var -> ParallelSearchParameters.UseMergedTransientLibrary, so merged-mode is on the parameter/TOML/CLI surface.
Per-database output is now confident by either PEP_QValue or score q-value when PEP is active, so reasonable edge-case hits are still written. The membership filter stays separate from the shared confidence filter.

Deferred, as noted in the threads:

Detected-organism selection -> DetectedOrganismSelector policy — proposed as a follow-up so we can agree the seam first (the core QualifiesAsDetectedOrganism predicate is already a pure, tested static).
Scoring abstraction (FragmentMatch / ISpectralScorer / GPU backend) and the MSL calibration coordinator — these belong to the stacked .msl search PR (only visible here because this PR is stacked on it); I'll handle them there, including shrinking the GPU-oriented abstraction since GPU benchmarks slower.

On the optimization-attempts writeup for the wiki — will do as a separate wiki edit (not part of the code diff).

The .msl dependency bump stripped the byte-order mark from six project files as unrelated churn; restore it so each csproj diff is just the mzLib version line. Addresses manual review (C5) on nbollis#9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ch faults If a consumer (search) threw, Parallel.ForEach propagated out of the loop and the producer's GetResult() was skipped: the load task could stay blocked on a full bounded queue until disposal, and its exception went unobserved. Add a CancellationTokenSource the consumer cancels on fault so the producer's Add unwinds promptly, and always observe the producer task in finally. Also replace the MM_PARALLELSEARCH_MERGED environment variable with a real ParallelSearchParameters.UseMergedTransientLibrary flag (parameter/TOML/CLI surface, reproducible) and document where the multi-writer bookkeeping is synchronized (AppendToFile / MarkDatabaseCompleted / UpdateProgress locks). Addresses manual review (C1, A4, C2) on nbollis#9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…d dedicated seams - Extract TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch so the generic PEP engine keeps only its standard cross-validated workflow; the train-once / apply-to-transient model reuse and the background PEP->PEP_QValue curve move to the derived class. Only the _randomSeed seam moves private->protected (mirrors TransientProteinParsimony/ ScoringAndFdr engines). - Replace the MM_PARALLELSEARCH_MERGED environment variable with a real ParallelSearchParameters.UseMergedTransientLibrary flag so merged-mode is on the parameter/TOML/CLI surface and reproducible instead of process state. - Per-database output now counts a match confident by EITHER its PEP_QValue or the score-based q-value when PEP is active, so the few reasonable edge-case hits a single metric flags are still written for the user to investigate. The membership filter (FilterToTransientDatabaseOnly) stays separate from the shared row-level confidence filter. Addresses nbollis review on nbollis#10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tion The ISpectralScorer / SpectralScorerProvider / ScoringBatch / FragmentMatch / IScoringSink / SpectralScoringData layer existed to host a GPU fragment-matching backend. GPU benchmarks slower and there is no concrete second backend, so the abstraction is pure surface area. Collapse the transient search inner loop back onto the shared MatchFragmentIons / CalculatePeptideScore so MatchedFragmentIon is again the type at the engine boundary; the .msl precomputed-fragments path and the shared-base-PSM / copy-on-write optimizations are unchanged. Behavior matches the original pre-abstraction engine (same closest-envelope match, 1 + intensity/TIC scoring, count-based ScoreCutoff). Addresses manual review (A1-A3) on nbollis#9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…re organism detection Two analysis-layer improvements over the spectral-library search, plus tests. PEP: train one model on the base (human) search and reuse it to score every transient database ID -- the transient DBs are far too small to train their own. Each match gets a model PEP and a PEP_QValue mapped onto a background PEP->PEP_QValue curve snapshotted from the decoy-rich base search. Per-database output is filtered to confident IDs (PEP_QValue < 0.05, QValue fallback when PEP is unavailable) and the summary reports confident counts at both 1% and 5% PEP_QValue. PEP is a per-peptide, organism-agnostic property, so the human-trained model transfers cleanly; PEP_QValue is assigned strictly by PEP (not the score-based QValue, which ranks matches differently). Family-aware detection: select detected organisms by independent evidence families (>=4 of 7 significant AND combined q <= threshold) computed directly from the test results, instead of the old test-ratio gate that required >=50% of ALL tests -- unreachable for a small genome, so it selected nothing. On the SARS spike-in, SARS-CoV-2 is now the rank-1 detection (7/7 families, q=4.7e-300). Tests: 27 unit tests (PEP curve/lookup, train->assign incl. the missing-FdrInfo regression, confidence gate, dual 1%/5% counts, metrics round-trip, the family-detection predicate) plus an end-to-end RunTask that trains PEP and exercises the per-database assignment, output filter, statistics, and the family writer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…-shard scaling) The merged ManySearch retains every database's results in memory for the end-of-run cross-organism statistics, so the heap grows with shard count. Workstation GC collects that large heap serially, making each successive shard index load progressively slower (measured 22s -> 106s over 10 shards). Server GC's parallel collection keeps loads flat (~22s), ~4x faster total load at 10 shards and the difference between ~1h and ~4.7h at the full 160-shard production scale. Output identifications are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ter) The merged producer loaded+filtered shards one at a time (~72s/shard: LoadIndexOnly + single-threaded candidate filter over ~18M entries + GetEntry fetch), making it the search wall bottleneck (~3.2h at 160 shards; the 31-way consumer search keeps up easily). Run the producer K-way (MM_PARALLELSEARCH_PRODUCERS, default 4) — each shard has its own index/file handle so loads are independent; GetEntry within a shard stays single-threaded. Bounded so it doesn't oversubscribe the consumer search; the loadedQueue still caps memory. Measured 10-shard x 3-file: transient wall 919s -> 621s (~1.5x; disk-bound on the concurrent fragment fetches, not CPU). Results byte-identical (BasePSMs 4728, 1083 summary rows). Extrapolates ~3.2h -> ~2.1h at 160 shards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…d dedicated seams - Extract TransientPepAnalysisEngine : PepAnalysisEngine under EngineLayer/ParallelSearch so the generic PEP engine keeps only its standard cross-validated workflow; the train-once / apply-to-transient model reuse and the background PEP->PEP_QValue curve move to the derived class. Only the _randomSeed seam moves private->protected (mirrors TransientProteinParsimony/ ScoringAndFdr engines). - Replace the MM_PARALLELSEARCH_MERGED environment variable with a real ParallelSearchParameters.UseMergedTransientLibrary flag so merged-mode is on the parameter/TOML/CLI surface and reproducible instead of process state. - Per-database output now counts a match confident by EITHER its PEP_QValue or the score-based q-value when PEP is active, so the few reasonable edge-case hits a single metric flags are still written for the user to investigate. The membership filter (FilterToTransientDatabaseOnly) stays separate from the shared row-level confidence filter. Addresses nbollis review on nbollis#10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nbollis reviewed Jun 10, 2026

View reviewed changes

trishorts and others added 2 commits June 15, 2026 12:31

trishorts force-pushed the ms-analysis branch from f8f946a to 766c8f1 Compare June 15, 2026 17:38

trishorts and others added 5 commits June 15, 2026 13:03

trishorts force-pushed the ms-analysis branch from 766c8f1 to 797ff65 Compare June 15, 2026 18:11

nbollis mentioned this pull request Jun 17, 2026

feat(parallelsearch): per-ID PEP for transient databases #11

Open

Conversation

trishorts commented Jun 10, 2026

Summary

Benefits

Rationale

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbollis left a comment

Choose a reason for hiding this comment

Uh oh!

nbollis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trishorts commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts Jun 15, 2026 •

edited

Loading

trishorts commented Jun 15, 2026 •

edited

Loading