feat(reporting): emit whole-month expected-count tables and plots - #187
Merged
Conversation
The report keeps quoting the canonical 6-monthly ages_query table; this
adds a finer companion for readers who need monthly resolution. Every
fit now writes, per outcome:
posterior_summary_monthly[_<outcome>].csv
expected_counts_by_month[_<outcome>].{png,svg,csv}
The table has the canonical table's columns and bucket thresholds, one
row per whole month, plus n_obs (observed administrations that month)
and a grid_age_months / grid_offset_months provenance pair. The figure
draws expected counts with 50% and 89% bands and monthly markers, with
the individual-child predictive interval as a distinct dashed outline —
the two are routinely conflated and quoting Ey where Y belongs badly
understates the range of individual children.
Derived from the *plot* grid rather than by adding query ages. That
matters: ages_query is part of the model graph (the GP is evaluated
there), so extending it would change every model's fingerprint and the
query_id dimension the report and comparisons consume. n_plot = 500
gives a step of about 0.2 months, so each whole month reads the nearest
grid point, within a few days; the offset is recorded per row and a
grid too coarse for whole months raises rather than mislabelling ages.
Coverage is therefore the observed age range — the DS models' 90-month
query row, past the oldest observation, has no monthly counterpart by
design, because this table does not manufacture extrapolations.
posterior_summary_table and the new monthly_summary_table now share one
row builder and one COUNT_BUCKET_THRESHOLDS constant, so the canonical
and monthly tables cannot drift in either thresholds or column names. A
test asserts the monthly rows equal the canonical rows at shared ages —
the check that this is a refinement of the reported table rather than a
different quantity.
Wired for all fifteen models: the univariate path via common (VG01-04,
and VG11/VG12 through common_univariate_re), the bivariate outcome
helper (VG05, VG07-VG10, VG13, VG16), the trivariate helper (VG14), and
the joint engine (VG15). VG15 draws no predictive counts on the plot
grid, so predictive draws are optional throughout: its monthly tables
carry the expected count only, matching its query-age tables, with the
Y_* and bucket columns absent rather than zero-filled. Giving VG15
predictive counts means adding model nodes and is a separate decision.
Twelve tests, including a wiring suite that calls the emitter for real
against each engine's own sample class. A wrong attribute name in a call
site raises only when a fit runs, which is invisible to a test that
exercises the helper alone — that failure mode already got through once
in this codebase (definition.max_age_months in the VG15 sensitivity).
Documented in docs/models/README.md. No traces exist locally, so these
artefacts first appear with the refit in #186.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds whole-month expected-count reporting artefacts (tables + plots) as post-processing derived from each model’s plot grid, keeping the canonical 6‑monthly ages_query report outputs unchanged while providing finer-grained companions for interpretation and diagnostics.
Changes:
- Refactors posterior summary row construction into a shared builder and introduces
monthly_summary_table()for whole-month outputs derived fromX_plot. - Adds a new plotting helper to render expected counts by month (with optional predictive interval overlay) and wires monthly emission into each model engine path.
- Expands test coverage with unit tests for monthly table behaviour and a wiring suite that invokes each engine’s emission call sites against real sample classes.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_posterior_analysis.py | Adds unit tests for whole-month summary table semantics, provenance, bucket monotonicity, and VG15-style “no predictive draws” behaviour. |
| tests/test_monthly_summary_wiring.py | New wiring tests to ensure engine-specific call sites and sample-field expectations survive real invocations and emit the expected artefacts. |
| src/vocab_growth/posterior_analysis.py | Introduces shared COUNT_BUCKET_THRESHOLDS + summary_row() and adds monthly_summary_table() derived from plot-grid snapping. |
| src/vocab_growth/plotting.py | Adds plot_expected_counts_by_month() to plot whole-month expected trajectories and optionally overlay predictive intervals; writes PNG/SVG/CSV sidecars. |
| src/vocab_growth/models/common.py | Wires monthly emission into the univariate pipeline via new emit_monthly_summary() helper. |
| src/vocab_growth/models/common_bivariate.py | Extends bivariate outcome plotting helper to accept p_plot and emit monthly tables/plots per outcome. |
| src/vocab_growth/models/common_trivariate.py | Extends trivariate outcome plotting helper to accept p_plot and emit monthly tables/plots per outcome. |
| src/vocab_growth/models/common_joint_modality.py | Emits whole-month companions for VG15 expected-count-only outcomes (no plot-grid predictive draws). |
| docs/models/README.md | Documents the relationship between canonical query-age tables and the new whole-month companion outputs, including provenance and coverage constraints. |
…ct two claims Real dev fits of VG05, VG14 and VG15 — the three engine paths CI does not exercise, since CI fits only VG01 — confirmed the wiring works in all four engines, and found one bug and two wrong statements. The bug: the joint engine passed its whole analysis frame's ages for every outcome, so n_obs came back as 1115 for understood, spoken and signed alike — the frame's row count, not an observation count. The three outcomes have materially different coverage there, so it overstated all of them. Now filtered by the outcome column, and verified against VG15's own fit manifest, which the corrected output matches exactly: understood 671, spoken 947, signed 241. (Those last two are below the raw pool's 1114 and 637 because the engine's signing-source harmonisation drops rows — n_obs counts what the model fitted, which is the useful quantity.) Two regression tests: one on the per-outcome contract, one asserting the joint block filters rather than handing over the frame. Two documentation corrections, both from reading the fitted output rather than reasoning about it: - Coverage was documented backwards. The monthly table is *wider* than the canonical ages, not narrower: the DS observed range is 8-115 months, giving 108 rows that contain every canonical 12-90 age. The claim that the 90-month row has no monthly counterpart was wrong. What is true, and more useful, is that 33 of those 108 months contain no observation at all — so n_obs matters more than first credited, and a zero row is the model interpolating, not evidence. - The nearest-point snap makes a monthly figure differ from the canonical figure at the same nominal age, which was unquantified. Measured on VG05 understood: 1.3% at 12 months, 1.2% at 18, under 0.4% from 24 on — largest where growth is steepest. Documented with the numbers and the rule that the canonical table is the one to quote when the two must agree. Also recorded: the monthly tables use one schema across all fifteen models with the outcome in the filename, whereas VG15's canonical tables prefix by outcome (Ey_u_median), so joining them needs the rename. That inconsistency is pre-existing, not introduced here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ring claim Addresses the review comment on PR #187, which read the boundary behaviour as inconsistent with the documented snapping: with a grid starting at 8.1, month 8 is 0.1 months away — inside MAX_MONTH_SNAP_OFFSET — yet ceil() drops it. The behaviour is intentional and stays. The plot grid spans exactly the observed age range, so a month below X_plot.min() is below every observed age: reporting it would extrapolate, and its value would be the trajectory at 8.1 wearing an "8" label. Widening the coverage rule to "nearest point within the snapping bound" would reintroduce both faults. What was missing is that the rule was nowhere stated — the docstring gave the coverage principle and the snapping rule separately and left their interaction at the boundary to inference. Now stated explicitly in the docstring and at the computation, with the reason. Two tests pin it: one with a fractional span (8.1-89.9) asserting months 8 and 90 are absent and every reported month is bracketed by real grid points, and one with an integral span asserting the real case loses nothing and its endpoints carry zero snap error. Recorded ages are whole months in both pools (DS 8-115, TD 8-25), so no month is dropped today; the tests keep a future fractional-age source from acquiring a silently extrapolated boundary row. Separately, the docstring still carried the coverage claim corrected in docs/models/README.md two commits ago — that the DS 90-month query row has no monthly counterpart because it sits beyond the oldest observation. It does have one; the pool reaches 115 months. Corrected in the place a developer actually reads, with the true figures and the note that coverage is wider than the canonical ages, not narrower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Drafted by an LLM-based AI tool (Claude Code/Fable 5).
Adds whole-month expected-count tables and plots alongside the existing 6-monthly ones. The report is unchanged and still quotes the canonical
ages_querytable — these are finer-grained companions, not a change to what is published.Every fit now writes, per outcome:
posterior_summary_monthly[_<outcome>].csvP(Y<=k)bucket thresholds, plusn_obsand thegrid_age_months/grid_offset_monthsprovenance pairexpected_counts_by_month[_<outcome>].{png,svg,csv}Why it reads the plot grid rather than adding query ages
The obvious implementation — put monthly ages in
ages_query— is the wrong one here.ages_queryis part of the model graph: the standardised query grid is stacked intoX_all_zand the GP is evaluated at it. Extending it would change every model's fingerprint, grow thequery_iddimension that the report and the comparison pipeline consume, and force the report to filter rows everywhere to keep its 6-monthly view.Deriving from the plot grid avoids all of that and is pure post-processing of a fitted trace.
n_plot = 500gives a step of about 0.2 months, so each whole month reads its nearest grid point — within a few days. Two honest consequences, both handled explicitly rather than hidden:grid_age_monthsandgrid_offset_months, and a grid too coarse for whole-month resolution raises with the offending offset instead of emitting rows whose stated age is wrong. A test drives that path with a deliberately coarse grid.n_obscounts the observed administrations in each whole month, so a row interpolated between sparse ages is visible as such rather than looking equally well-supported.The two estimands stay separated
posterior_summary_tableand the newmonthly_summary_tablenow share one row builder and oneCOUNT_BUCKET_THRESHOLDSconstant, so the canonical and monthly tables cannot drift in thresholds or column names. A test asserts the monthly rows equal the canonical rows at shared ages — the check that this is a refinement of the reported table rather than a different quantity.The shared builder also documents the distinction that motivated this PR's companion discussion:
Ey_*is the expected count, a credible interval on the mean trajectory carrying parameter uncertainty only, which narrows toward zero width as more children are observed;Y_*andP(Y<=k)are the posterior predictive count for an individual child, carrying between-child and occasion-level dispersion too, and converging on the real population spread. Quoting the first where the second belongs understates the range of individual children badly, so the figure draws them in visually distinct styles and the docstrings say which is which.Coverage, including the one model that differs
Wired for all fifteen models: the univariate path via
common(VG01–VG04, and VG11/VG12 throughcommon_univariate_re), the bivariate outcome helper (VG05, VG07–VG10, VG13, VG16), the trivariate helper (VG14), and the joint engine (VG15).VG15 draws no predictive counts on the plot grid, so predictive draws are optional throughout: its monthly tables carry the expected count only — matching what its query-age tables already report, which likewise have no
Y_*or bucket columns — with the predictive columns absent rather than zero-filled, so a missing estimand cannot be mistaken for a computed one. Giving VG15 predictive counts and buckets means adding nodes to its model graph; that is a separate decision, not smuggled in here.Tests
Twelve new tests (338 collected, up from 326), including a wiring suite that calls the emitter for real against each engine's own sample class. A wrong attribute name in a call site raises only when a fit actually runs, which is invisible to a test that exercises the shared helper alone — that exact failure mode already got through review once in this repository (
definition.max_age_monthsin the VG15 sensitivity), so the guard is deliberate.ruff,cspellandprettierclean. Documented indocs/models/README.mdunder Reporting ages: 6-monthly tables, whole-month companions.No traces exist locally, so these artefacts first appear with the full refit — noted on #186 so the run expects them.
Refs #186.