Compare-vcfs and cleanup - #301
Merged
Merged
Conversation
Compares a downstream variant caller's VCF against a NEAT-simulated truth VCF and attributes each false negative to the simulator's configuration. Variant equivalence is delegated to hap.py; NEAT contributes the FN attribution against the mutation/target BED and simulated-contig set recorded in a new simulation_summary.json artifact. Outputs in --output-dir: - comparison_summary.json: counts, precision/recall/F1, FN reasons - comparison_summary.txt: human-readable rollup - FN_with_reasons.vcf: hap.py FNs with NEAT_REASON INFO tag - fn_attribution.png: optional bar chart (--plot) The read simulator now emits simulation_summary.json alongside its other outputs (config echo + delivered counts) as a prerequisite for compare-vcfs. hap.py is an optional external dependency (conda bioconda::hap.py); compare-vcfs exits with a clear install hint when not found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit found five untested paths in the compare-vcfs work:
- CLI layer: subcommand registration + argument routing
- --plot flag: runner integration (on/off)
- Stale BED path in simulation_summary: undefined failure mode
- Chrom-prefix mismatch ('1' vs 'chr1'): documents a user gotcha
- Multi-FN scaling through real hap.py
Adds 9 tests across 4 files (+1 new test_cli.py). Full suite: 727 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a mutation_bed or target_bed uses different chrom names than the reference (e.g., '1' vs 'chr1', 'MT' vs 'chrM'), NEAT previously produced silent garbage — every FN tagged 'outside_mutation_bed' even though they were actually inside. Changes: - new neat/common/chrom_names.py with prefix/mitochondrial heuristics and a user-supplied alias-file parser - simulation_summary.json now records delivered.reference_contigs (full FASTA contig set, separate from contigs_simulated) - compare_vcfs_runner runs detect_chrom_naming_mismatches before attribution; emits WARNING logs AND a 'warnings' array in comparison_summary.json with suggested aliases - new --chrom-aliases TSV flag applies user-supplied mappings to BED chrom names at load time - README + ChangeLog updated; mismatch behavior documented as non- normalizing by default NEAT does not auto-normalize: silent renaming masks real bugs. Detection + warnings + opt-in normalization is the policy. 34 new tests (chrom_names utility, schema, attribution detection, runner integration). Full suite: 761 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Semantic fix: when chrom-name mismatches make a BED unusable, the runner now skips attribution against that BED entirely (via new skip_beds param on attribute_fns) instead of tagging every FN as 'outside_<bed>'. The warning in comparison_summary.json explains the cause; the fn_attribution counts now reflect what NEAT actually checked. Test additions (7): - render_summary_txt with/without warnings section (rendering was untested) - detect_chrom_naming_mismatches skips empty BEDs (no spurious warning) - detect_chrom_naming_mismatches still warns when user aliases don't resolve the mismatch (suggested mapping comes from raw BED chroms, not user input) - attribute_fns skip_beds suppresses outside_<bed> tags - attribute_fns skip_beds doesn't affect outside_simulated_contigs - Real hap.py end-to-end with mismatched mutation_bed (warning fires, outside_mutation_bed absent, --chrom-aliases silences warning) Existing test updated: - test_runner_emits_chrom_mismatch_warning_into_json_report now also asserts the semantic guarantee that outside_mutation_bed is absent. Full suite: 768 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compare vcfs
reuse_splits was broken (raised unconditionally regardless of whether the splits dir existed). cleanup_splits existed only to support it. Splits always go in the TemporaryDirectory that auto-cleans on exit. Both keys added to DEPRECATED_KEYS so old configs warn instead of silently no-oping. README examples, test fixtures, and test_options updated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reuse_splits was broken (raised unconditionally regardless of whether the splits dir existed). cleanup_splits existed only to support it. Splits always go in the TemporaryDirectory that auto-cleans on exit. Both keys added to DEPRECATED_KEYS so old configs warn instead of silently no-oping. README examples, test fixtures, and test_options updated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop cleanup_splits and reuse_splits; bump to v4.5.1
Replaces the NEAT 2.1 validateFQ/validateBam scripts (Python 2, broken read-name assumption, no gzip support, BAM tool was a raw hex dump not a validator) with recommended standard tools: FastQC/fastp for FASTQ, samtools quickcheck/flagstat and Picard ValidateSamFile for BAM, and bcftools stats for VCF. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Version headers: README title/intro/ToC updated from v4.4 → v4.5 - read_len default: corrected from 101 → 151 (matches from_cli defs) - coverage type: corrected from "float or integer" → "integer only" - threads: removed duplicate row from config table - off_target_scalar: removed from template_neat_config.yml (key has never existed in options.py defs; silently ignored since at least v4.3) - Added three undocumented config keys to README table: quality_offset, overwrite_output, no_coverage_bias - Added neat bacterial-wrapper section to Utilities + ToC entry Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Docs validation section
This was
linked to
issues
May 20, 2026
Closed
Closed
TraditionalQualityModel now accepts an optional quality_bins parameter that snaps sampled scores to the nearest discrete bin using vectorized np.searchsorted, matching how NovaSeq/NextSeq 2000 instruments emit only a small set of Phred values (e.g. Q2/Q12/Q23/Q37). The runner passes allowed_bins through so both Markov and traditional models produce realistically binned output when bins are specified. - Add quality_bins to TraditionalQualityModel.__init__ and get_quality_scores() - Pass allowed_bins to TraditionalQualityModel in model_qual_score runner - Expose _down_bin_quality as public down_bin_quality in markov_utils - Add QUALITY_PRESETS dict in neat/quality_score_modeling/presets.py (novaseq, nextseq2000, nextseq500) - Add --quality-preset CLI flag to neat model-qual-score; implies --markov - Fix stale -Q help text in model_sequencing_error CLI - Add TraditionalQualityModel binning tests + preset data/CLI tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a "Binned quality scoring" subsection to neat model-qual-score showing how to use --quality-preset (novaseq, nextseq2000, nextseq500) and explicit -Q bin lists for instruments that emit discrete Phred levels. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Feature/binned quality scoring
# Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds compare VCFs tool, to compensate for NEAT 2.1's similar tool. This one uses hap.py under the hood and is very fast.
Reenabled binned scoring, added tests to ensure that works properly. User will have to input the list of quality score bins for NEAT to recognize that the data should be binned.