Conversation
jgslunde
commented
Jun 15, 2026
jgslunde
commented
Contributor
- Component separation now has Sampling Groups, where you can specify which components and bands go into a sampling step.
- Added some abstraction to the way the TOD is accessed, through the TODView class, which interfaces with both TODSamples and DetectorTOD.
- Some general refactoring.
- Various bugfixes.
…models into classes. Also added an SO-SAT reader
…QU-sensitivity can be adjusted per-detector.
…etectors are discarded on read-in).
… masking calculations
…ed data, combining information from both TODSamples and Detector_TOD
…d to the TOD upon reads.
…g any compsep-ranks), and refactored some stuff
…d, to make wrong-detector-indexing bugs (due to missing detectors on different ranks) less likely.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR performs a broad refactor across TOD I/O, noise sampling, CompSep orchestration, and chain output, while adding SO-specific readers and expanding test coverage. The changes introduce new abstractions (e.g., TODView, PixelPointing, NoisePSD) and new configuration patterns (CompSep sampling groups and nested gain-calibration blocks).
Changes:
- Add CompSep sampling groups with execution-view band IDs (
*_I/*_QU) and update TOD↔CompSep communication accordingly. - Refactor TOD data access and storage (new pointing model classes, jump corrections, diagnostic spectra written to chain, richer
TODSampleslayout). - Generalize correlated-noise sampling around a
NoisePSDinterface and add response-scaled mapmaking kernels (C++ + ctypes), plus extensive new tests.
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tod_power_spectra.py | Adds tests for binned TOD power spectra and TOD diagnostic routing. |
| tests/test_noise_sampling.py | Expands tests for NoisePSD(1/f), correlated-noise sampling, and apply_N_inv. |
| tests/test_math_utils.py | Removes legacy math utils tests that referenced old paths. |
| tests/test_huffman_decode.py | Adds roundtrip + dtype validation tests for Huffman decode bindings. |
| tests/test_gain_calibration.py | Adds tests for gain-calibration target resolution, TODView.get_calib_tod, rel-gain solve, and downsampling. |
| tests/test_compsep_framework.py | Adds tests for sampling-group logic, execution-band IDs, and CompSep send/receive selection. |
| tests/test_components.py | Adds tests around component list construction/splitting, init-from-chain/map behavior, and unit handling. |
| src/lib_cpp/utils_pymod.cc | Extends Huffman decode to more integer dtypes + validates out dtype matches symb. |
| src/lib_cpp/ctypes/mapmaker.cpp | Adds response-scaled IQU accumulators and weight accumulators. |
| src/commander4/utils/math_operations.py | Tightens complist_dot consistency checks and updates FFT helper signatures. |
| src/commander4/utils/mapmaker.py | Wires new response-scaled mapmaking entry points and Python reference paths. |
| src/commander4/utils/CG_mapmaker.py | Aligns LHS traversal with accepted detector-scans; fixes ntod derivation. |
| src/commander4/tod_reader.py | Registers SO LAT/SAT TOD readers. |
| src/commander4/solvers/perpix_compsep_solver.py | Minor cleanup in matrix build; fixes log message to use npol. |
| src/commander4/solvers/CG_compsep_solver.py | Removes unused imports/comments; standardizes logging fields. |
| src/commander4/sky_models/sky_model.py | Adds build_initial_sky_model and supports both split and joined component lists in realization. |
| src/commander4/simulations/inplace_litebird_sim.py | Updates component instantiation and resolves scalar nu_ref for in-place sim usage. |
| src/commander4/parse_params.py | Ensures parameter_file_as_string preserves YAML key order (sort_keys=False). |
| src/commander4/output/write_chains_files.py | Makes compsep chain writer accept CompList, enforces unique shortnames, and stores richer metadata. |
| src/commander4/output/log.py | Adds prefix-stripping + color console formatter and updates formatting defaults. |
| src/commander4/noise_sampling/sample_ncorr.py | Introduces higher-level sample_correlated_noise API + MPI-reduced diagnostics logging. |
| src/commander4/noise_sampling/noise_sampling.py | Removes old PSD param sampling implementation (moved into noise_psd.py). |
| src/commander4/noise_sampling/noise_psd.py | Introduces NoisePSD interface updates and implements param sampling for NoisePSDOof. |
| src/commander4/mpi_management.py | Makes CompSep optional when no ranks allocated; validates sampling-group config vs rank allocation. |
| src/commander4/experiments/SO/tod_reader_SO_SAT.py | Adds SO SAT reader with processing mask handling, pointing, compression support, and stats. |
| src/commander4/experiments/planck/tod_reader_planck.py | Refactors Planck reader to new DetectorTOD/PixelPointing and per-sample masking semantics. |
| src/commander4/experiments/litebird/tod_reader_litebird_sim.py | Refactors LiteBIRD sim reader to new DetectorTOD/PixelPointing. |
| src/commander4/experiments/litebird/tod_reader_litebird_sim_spawndetectors.py | Refactors spawned-detector sim reader to new DetectorTOD/PixelPointing. |
| src/commander4/experiments/akari/tod_reader_akari.py | Refactors Akari reader to new DetectorTOD/PixelPointing and adds noise model wiring. |
| src/commander4/data_models/TOD_samples.py | Major expansion: present/accept masks, power spectra diagnostics, optional ragged ncorr storage, jump catalog, detector-name persistence. |
| src/commander4/data_models/scan_TOD.py | Cleans outdated docstring fields. |
| src/commander4/data_models/pointing.py | Adds pointing abstraction (ScanBoresightPointing, DetectorBoresightPointing, PixelPointing). |
| src/commander4/data_models/jump_corrections.py | Adds jump detection/correction structures and HDF5 pack/unpack. |
| src/commander4/data_models/detector_TOD.py | Refactors detector TOD container around pointing objects, packed masks, and optional compressed streams. |
| src/commander4/data_models/detector_group_TOD.py | Adds detector-scan iterator and improves apply_N_inv (DC projection + mirrored FFT + white fast path). |
| src/commander4/compsep_processing.py | Implements sampling-group driven CompSep loop and execution-view band matching. |
| src/commander4/communication.py | Switches CompSep exchange to pass SkyModel objects and supports TOD-only initial sky build. |
| src/commander4/cli.py | Uses CompSep allocation (not perform_compsep) to determine initial-sky exchange behavior. |
| pyproject.toml | Adds new dependencies (numpy-quaternion, qpoint). |
| params/Planck_TODs/param_PlanckTODs_perpix.yml | Updates outputs and configures sampling_groups_compsep. |
| params/Planck_TODs/param_PlanckTODs_CG.yml | Updates outputs and configures sampling_groups_compsep + noise model config. |
| params/param_AkariTODs_CG.yml | Migrates to nested gain/noise blocks and adds new correlated-noise config options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+181
to
+183
| # C++ decoder accepts only int64 for the tree. | ||
| self.huffman_tree = huffman_tree.astype(np.int64, copy=False) | ||
| self.huffman_symbols = huffman_symbols |
|
|
||
| pix = pix[:self.ntod] | ||
| if target_nside != self.data_nside: | ||
| nthreads = int(os.environ["OMP_NUM_THREADS"]) |
Comment on lines
+196
to
+198
| if "gain" in my_band.detectors[experiment_data.scans[0].detectors[0].name]: | ||
| for iscan, det in experiment_data.iter_detector_scans(): | ||
| all_det_gains[iscan, det.det_idx_fullband] = my_band[det.name].gain |
Comment on lines
1
to
+3
| import logging | ||
| import logging.config | ||
| import sys |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.