perf(ch): serve lastSeen query from signal_latest projection#297
Merged
Conversation
The lastSeen UNION ALL branch aggregated max(timestamp) with no GROUP BY, which the prod projection matcher does not serve -- every signalsLatest/signalsSnapshot selecting lastSeen scanned the subject's full signal history (p50 ~2900ms vs ~380ms for the name-filtered branches, worse for high-history vehicles). Wrap the aggregate so the inner query groups by name at the signal_latest_by_subject_source_name projection grain; the outer query re-aggregates to a single row. Result shape and semantics unchanged. Refs #284
master install.sh currently downloads the .sbom.json asset in place of the tarball, so checksum verification fails for every version and CI breaks on install-tools. Fetch install.sh from the release tag and pin the version instead of tracking latest/master.
zer0stars
force-pushed
the
fix/lastseen-projection-scan
branch
from
262cc37 to
0718635
Compare
zer0stars
added a commit
that referenced
this pull request
Jun 11, 2026
…les (#298) * chore: bump model-garage for signal_latest + summary table migrations Pinned to the feat/signal-latest-table branch head (pre-v1.0.13 pseudo-version); re-pin to the v1.0.13 tag once released. * perf(ch): serve latest-state and summary queries from precomputed tables signalsLatest/signalsSnapshot/lastSeen/availableSignals now read the precomputed signal_latest table (model-garage 00010) instead of fanning out across every part of the raw signal table. Location fields read kind=1 rows, which pre-filter (0,0) fixes, so plain argMax replaces argMaxIf. Supersedes the projection-matching shapes from #284/#297. dataSummary per-name count/first-seen/last-seen now read the AggregatingMergeTree summary tables (model-garage 00011) instead of aggregating the raw tables. Counts are over raw inserts, matching the approximation of the previous non-FINAL count(*) reads. * style(ch): alias latest location column value_location for UNION consistency Scanning is positional so agg_location worked, but the non-location branch of the latest UNION ALL aliases its zero tuple value_location; matching aliases keep the result shape self-describing. * chore: pin model-garage to v1.0.13 release tag
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
signalsLatest/signalsSnapshotwithlastSeenselected runs a three-branch UNION ALL. Branches 1–2 (name IN (...) GROUP BY name) are served by thesignal_latest_by_subject_source_nameprojection. Branch 3 (lastSeen) aggregatesmax(timestamp)with no GROUP BY — the projection matcher in prod does not serve it, so it scans the subject's entire signal partition. Cost scales with vehicle history, not with the request:Follow-up to #284, which fixed branches 1–2 but left the lastSeen branch flat.
Fix
Reshape
getLastSeenQueryto aggregate at the projection grain:Inner query matches the projection's
GROUP BY subject, source, namedimensions (subject pinned, re-aggregated across source); outer collapses to one row. Response shape and lastSeen semantics (max over all names) unchanged.Testing
TestGetLastSeenQuery— generated SQL shape, nil-args, source-filter placementTestLatestQueriesUseProjection— all three latest-query branches run underforce_optimize_projection = 1against the migrated test container (errors if any branch can't be projection-served)internal/service/chsuite + e2eTestSignalsLatest/TestApproximateLocationgreen; lint cleanDeploy verification
EXPLAIN indexes=1on the lastSeen branch in prod should showReadFromMergeTree (signal_latest_by_subject_source_name)and granule counts in the single digits