feat: native XYZTS (tsheader_procP3) support in XYTSFile#368
Conversation
Agent-Logs-Url: https://github.com/ucgmsim/qcore/sessions/6829334f-dcbc-47d2-9f97-d63ebff1bb3a Co-authored-by: lispandfound <12835929+lispandfound@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ucgmsim/qcore/sessions/6829334f-dcbc-47d2-9f97-d63ebff1bb3a Co-authored-by: lispandfound <12835929+lispandfound@users.noreply.github.com>
…leanup) Agent-Logs-Url: https://github.com/ucgmsim/qcore/sessions/6829334f-dcbc-47d2-9f97-d63ebff1bb3a Co-authored-by: lispandfound <12835929+lispandfound@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ucgmsim/qcore/sessions/6829334f-dcbc-47d2-9f97-d63ebff1bb3a Co-authored-by: lispandfound <12835929+lispandfound@users.noreply.github.com>
There was a problem hiding this comment.
You've accidentally committed in the egg-info directory
There was a problem hiding this comment.
Removed in 2da41e0 — added qcore_utils.egg-info/ to .gitignore and unstaged all five files.
| nz = np.fromfile(xytf, dtype=">i4", count=7)[-1] | ||
| if nz == 0x00000001: | ||
| # --------------------------------------------------------------- | ||
| # Step 1 – determine endianness and whether this is a proc-local |
There was a problem hiding this comment.
Step 1, step 2 nonsense not required.
There was a problem hiding this comment.
Byte layout of header is helpful. everything else is noise.
There was a problem hiding this comment.
Done in 2da41e0 — replaced the "Step 1/2/3/4" banner comments with a single block that shows the byte layout of both header variants (standard XYTS and proc-local XYZTS). All other prose removed.
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # XYZTS-specific tests (use synthetic data; no network required) |
There was a problem hiding this comment.
This header can be deleted.
| # Common dimensions used across XYZTS tests | ||
| _LOCAL_NX, _LOCAL_NY, _LOCAL_NZ = 4, 5, 3 | ||
| _GLOBAL_NX, _GLOBAL_NY = 20, 20 | ||
| _NT = 6 | ||
|
|
There was a problem hiding this comment.
rather than fixing these a composite hypothesis strategy to generate them is better.
| @pytest.fixture(scope="module") | ||
| def rng() -> np.random.Generator: | ||
| """Fixed-seed random number generator for reproducible tests.""" | ||
| return np.random.default_rng(42) |
There was a problem hiding this comment.
hypothesis does this better.
There was a problem hiding this comment.
Make use of hypothesis to automatically generate payloads and then parse them. See the test_timeseries.py code for inspiration
There was a problem hiding this comment.
Done in 2da41e0. Replaced the rng-fixture + @parametrize tests with a @st.composite strategy xyzts_file_data that generates endianness, ncomp, all dimensions, and the full payload via hypothesis.extra.numpy.arrays. Four @given tests now cover round-trip parsing, tslice_get guard, pgv guard, and meta_only — closely following the test_timeseries.py pattern.
…xyts.py comments, remove egg-info Agent-Logs-Url: https://github.com/ucgmsim/qcore/sessions/83689858-8f50-44f9-83e1-452c23694707 Co-authored-by: lispandfound <12835929+lispandfound@users.noreply.github.com>
Summary
Generalises the existing
XYTSFileparser inqcore/xyts.pyto natively support EMOD3D v3.0.13 volumetric xyzts proc-local files (tsheader_procP3, 72-byte header) without requiring any user flag.Background
In EMOD3D ≥ v3.0.13, volumetric output (
ts_xyz) writes one proc-local file per MPI rank. Each file uses thetsheader_procP3header (72 bytes) that storesloc_nx / loc_ny / loc_nzand then appendsnt × ncomp × local_nz × local_ny × local_nxcontiguousfloat32values.ncompis 3, 6, or 9 depending onts_xyz_ncomp. Filenames follow the*_xyzts-*pattern.The previous reader required
proc_local_file=Trueand only handledlocal_nz == 1surface slices.Changes
qcore/xyts.py_MAX_GRID_DIMconstantXYTSFile.ncomp__init__– Step 1 (detection)local_nx/local_nyplausibility against_MAX_GRID_DIM__init__– Step 3 (ncomp)ncompfrom file size; validate result ∈ {3, 6, 9} with a descriptiveValueError__init__– Step 4 (memmap)(nt, ncomp, local_nz, local_ny, local_nx)for XYZTS; 4-D (backwards-compatible) for proc-local XYTS withlocal_nz == 1tslice_getValueErrorwith guidance for XYZTS files (local_nz > 1)pgvValueErrorfor XYZTS filesproc_local_fileflaglocal_nz == 1)qcore/test/test_xyts/test_xyts.pyTen new tests using fully synthetic binary data (no network required):
test_xyzts_auto_detection– parametrised over endianness and ncomp ∈ {3, 6, 9}test_xyzts_data_shape– 5-D memmap shape check (both endians)test_xyzts_payload_values– round-trip value accuracytest_xyzts_tslice_get_raises–ValueErrorguardtest_xyzts_pgv_raises–ValueErrorguardtest_xyzts_meta_only–meta_only=Trueleavesdata=NoneTesting