Skip to content

viz: scatter, histogram, stacked bar + analyse: filter_rows#25

Merged
moonlight-lupin merged 2 commits into
mainfrom
feat/viz-charts-and-filter-rows
Jul 18, 2026
Merged

viz: scatter, histogram, stacked bar + analyse: filter_rows#25
moonlight-lupin merged 2 commits into
mainfrom
feat/viz-charts-and-filter-rows

Conversation

@moonlight-lupin

Copy link
Copy Markdown
Owner

Summary

Three chart types and a filtering primitive — the gaps that forced agents to hand-roll the same code every run — plus the docs-accuracy fix that reviewing #24 surfaced.

Charts (viz.py, HTML only — the xlsx path refuses them with the existing message)

scatter_chart(x, y, …, trend_line=False) Correlation / outlier spotting. Both axes float via _nice_ticks (matching line_chart). The OLS overlay spans only the observed x-range and carries slope / Pearson r / n in its tooltip.
histogram(values, bins=10, …) Bin count (equal-width) or explicit edges like [0,30,60,90,365]. Y-axis forced to 0; bars touch.
stacked_bar(data, …) Takes a pivot() result directly, or {segment: [(cat, value)]}, or {categories, series}. Legend with swatches.

Three judgment calls worth reviewing:

  • histogram edges are half-open [lo, hi) except the last, which includes its upper bound — otherwise the maximum observation silently vanishes.
  • scatter_chart draws no trend line when x has zero variance. There is no defensible slope for a vertical cloud, and drawing one would invent a relationship the data doesn't contain.
  • stacked_bar stacks negatives below the zero line. Folding a credit note into the positive stack would inflate the very bar it reduces.

All three reuse the engine's parse_number when reachable ('15%' → 0.15, '1.2m' → 1200000, '(500)' → -500) rather than inventing a second numeric dialect — otherwise the chart and the working paper could show different numbers for the same cell. Non-numeric inputs are skipped and reported, never plotted at the origin where they'd read as a real zero.

filter_rows(header, rows, filters) (analyse.py)

12 operators (== != > >= < <= in not_in between not_between contains is_empty not_empty), ANDed, returning (rows, report) with in/out totals and per-filter removed counts.

Two comparison rules exist because the alternative is a plausible wrong answer, not an error — both found by testing, not by reading:

  1. Dates compare as dates. parse_number('15/02/2026') returns Decimal('15022026') — it strips the separators. A number-first coercion therefore sorts 15 Feb after 1 Mar. Values that look like dates are now tested as dates first.
  2. A type mismatch is incomparable, not a string compare. The string fallback made 'n/a' > 1000 true. Such rows are excluded and counted in report["incomparable"].

between is inclusive of both ends (finance quotes ranges that way). Unknown columns and operators raise — a typo must not look like "no results".

Docs accuracy

The README's "no network calls / no cloud OCR / no credentials" and DATA-HANDLING.md's "no external APIs, no third-party uploads" predated the opt-in vision image extract shipped in #19, which calls a user-configured vision endpoint and needs a key. All three claims now carve out that one exception explicitly. SKILL.md and COMPATIBILITY.md already disclosed it correctly — it was the top-level pitch that overstated, which is the wrong way round.

Verification

Wired through _viz_block and schemas/dashboard-spec.schema.json (new block types, their properties, and per-type required-property rules; additionalProperties:false still holds).

Five regression tests, all mutation-checked — each was confirmed to fail when its guard is narrowed. One of them initially escaped: the stacked-negatives test only asserted relative ordering, which holds even when both segments stack upward. It now measures against the zero line.

Gates: data-lint OK · 101 pytest · 68 standalone · 19 runtime · 12 schemas · 6/6 golden plans · quickstart smoke.

🤖 Generated with Claude Code

moonlight-lupin and others added 2 commits July 19, 2026 00:08
Three chart types and a filtering primitive, closing gaps that forced agents to
hand-roll the same code every run. Plus a docs-accuracy fix that PR #24 surfaced.

viz.py (HTML only; the xlsx path refuses them with the existing clear message):
- scatter_chart(x, y, ..., trend_line=False) - both axes float via _nice_ticks.
  The OLS overlay spans only the observed x-range and carries slope / Pearson r /
  n in its tooltip; a cloud with no x-variance gets no line at all rather than an
  invented relationship.
- histogram(values, bins=10, ...) - bin count or explicit edges. Half-open
  [lo,hi) except the last bin, which includes its upper bound so the maximum is
  never silently dropped. Y-axis forced to zero (a floated frequency axis
  misstates every bar); bars touch. Unparseable values and values outside the
  caller's own edges are reported separately - they mean different things.
- stacked_bar(data, ...) - takes pivot() output directly. Negative segments stack
  BELOW the zero line, so a credit note reads as the reduction it is instead of
  inflating the bar it belongs to.

All three reuse the engine's parse_number when reachable ('15%' -> 0.15, '(500)'
-> -500) rather than inventing a second numeric dialect that would put a
different number on the chart than in the working paper.

analyse.py:
- filter_rows(header, rows, filters) - 12 operators, ANDed, returning
  (rows, report) with in/out and per-filter removed counts.

  Two comparison rules exist because the alternative is a plausible WRONG answer,
  not an error:
  * Dates compare as dates. parse_number('15/02/2026') returns 15022026 (it
    strips the separators), so a number-first coercion sorts 15 Feb after 1 Mar.
  * A type mismatch is incomparable, not a string compare - the fallback would
    make 'n/a' > 1000 true. Those rows are excluded AND counted.
  Unknown columns/operators raise; a typo must not look like "no results".

Docs: the README's "no network calls / no cloud OCR / no credentials" and
DATA-HANDLING's "no external APIs" predated the opt-in vision image extract
shipped in #19, which calls a user-configured endpoint. All three now carve out
that one exception. SKILL.md and COMPATIBILITY.md already disclosed it correctly.

Wired through _viz_block and dashboard-spec.schema.json. Five regression tests,
all mutation-checked - each fails when its guard is narrowed.

Gates: data-lint OK, 101 pytest, 68 standalone, 19 runtime, 12 schemas,
6/6 golden plans, quickstart smoke.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rief

The brief arrived after the first pass. Reconciling against it found three
places where my API would have shipped wrong, plus several smaller deviations.

Breaking (relative to the previous commit on this branch, unmerged):

- stacked_bar: the brief's PRIMARY input shape, {category: [v1, v2, ...]},
  raised TypeError - I had only implemented the segment-first
  {segment: [(cat, value)]} form. _norm_stacked now normalises five shapes:
  pivot() output, {category: [values]}, [(category, [values])],
  {categories, series}, and the segment-first form. Positional shapes have no
  segment names, so segments are numbered rather than left blank.
- scatter_chart: added the missing x_label / y_label axis annotations, and
  renamed unit/x_unit -> unit_y/unit_x per the brief. Now also accepts
  [(label, value)] pairs via _norm_pairs, and trims to the shorter side on a
  length mismatch (counting the unpaired tail).
- filter_rows: filter specs take "col", "values" for membership, and "lo"/"hi"
  for ranges; the report key is n_dropped. "column" and a two-item "value"
  remain accepted so nothing already written breaks.

Smaller alignments: histogram requires >= 2 parseable values (one point has no
distribution to show), bars carry rx="1", the tooltip reads "{n} values in
[lo, hi)" with an honest closing bracket on the last bin, and ~8 edge labels.
stacked_bar uses the brief's 6px gap and padding, with a "cat . segment: value"
tooltip. scatter's trend line spans the plot area as specified - the axes are
floated to the data, so the edges sit a tick outside the observed range.

The two comparison rules found earlier still hold and are unchanged: dates
coerce before numbers (parse_number strips date separators), and a type
mismatch is incomparable rather than a string compare.

Added: cookbook snippets for all three charts in references/blocks.md (required
by the brief and missed first time), two acceptance-shape regression tests
pinning the documented call signatures, and SKILL.md/schema/runtime updates for
the renamed parameters.

Verified every acceptance checkbox in the brief literally - 41 checks, all pass.
The five-mutant suite still catches all five.

Gates: data-lint OK, 103 pytest, 70 standalone, 19 runtime, 12 schemas,
6/6 golden plans, quickstart smoke.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@moonlight-lupin
moonlight-lupin merged commit f2aa82a into main Jul 18, 2026
3 checks passed
@moonlight-lupin
moonlight-lupin deleted the feat/viz-charts-and-filter-rows branch July 19, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant