Hide events based on new tombstone rows#4
Closed
elffjs wants to merge 1 commit into
Closed
Conversation
These events of type dimo.tombstone refer to files with the new voids_id column. By default, we hide voided files in the results.
zer0stars
added a commit
that referenced
this pull request
Jun 20, 2026
#3 events: dedup the decoded-event read path. lakeEventsDeduped collapses at-rest duplicates (QUALIFY ROW_NUMBER over subject,timestamp,name,source — ClickHouse's event ReplacingMergeTree key) so two cloud_event_ids that decode to the same logical event count once, not twice. The materializer's INSERT anti-join keeps them (its key includes cloud_event_id) and the signal read path already deduped; the event read path did not, over-counting events vs CH. #4 fetch: drop the 400-day lookback floor for subject-scoped fetches. It is a DoS guard for subject-less, id-less scans only — ClickHouse imposes no floor when given no `after`, and a subject prunes to one vehicle's files via the (subject, time) sort, so latestCloudEvent / cloudEvents now reach arbitrarily old events. A dormant vehicle whose newest event predates the window no longer wrongly looks empty. The guard stays for genuinely unbounded searches. #8 materializer: hard-error when MATERIALIZER_SHARD_COUNT>1 on the DuckLake path. Sharding is not honored there (the global ingest_progress cursor allows exactly one logical processor); refuse the config rather than silently run every replica over the full stream and roll back the CAS losers. #10 aggregations: implement inPolygon / inCircle location filters in pure SQL (haversine great-circle distance; an even-odd ray cast unrolled over the request's vertices) with no spatial-extension dependency, replacing the "not supported on duckdb backend" rejection. Mirrors ClickHouse geoDistance / pointInPolygon.
zer0stars
added a commit
that referenced
this pull request
Jun 23, 2026
Code-grounded, prioritized analysis of where the DuckLake query layer can exceed the CH-parity implementation (CH limits: pre-materialized MVs, fetch-all-to-Go detector loops, approximate functions). Covers: #0 ignition ongoing-trips (shipped), #1 exact aggregations (shipped+pinned), #2 ASOF trip enrichment (location gap-fill schema-free; distance/avg-speed via the existing Signals channel under no-schema), #3 SQL-native idling/refuel/recharge, #4 spatial (needs LOAD spatial first), #5-7 secondary. Verified by the round-3 E2E reviews (spatial-not-loaded + the schema caveats corrected).
Member
|
Closing as superseded — tombstone voiding is already on `main`, reimplemented natively in the DuckLake layer.
This PR targets the removed ClickHouse `eventrepo`, so rebasing would resurrect that stack. Functionality verified by #8: `TestLakeEventService_VoidingTombstoneBeforeEvent` (tombstone ingested before its target — order-independent) plus existing `TestLakeEventService_VoidingExcludes` (list / latest / summaries all exclude voided + tombstone). Scope is identical to this PR: voiding hides raw CloudEvents (`cloudEvents`/`latestCloudEvent`); decoded `signals`/`events` are not voided (materializer keeps `_ = voidsID`). A separate propagation pass would be needed if VSS-level voiding is ever wanted. |
zer0stars
added a commit
that referenced
this pull request
Jun 25, 2026
Lock in the two behaviors ported from the legacy ClickHouse eventrepo (superseded PRs #3, #4), now served natively from the DuckLake layer: - PresignBlobURL: a blob key becomes a presigned S3 GET URL for the configured blob bucket; nil presigner, empty bucket, and presigner failure all error loudly rather than panic or silently inline. Complements TestLakeEventService_BlobIndexKey (blob-key routing into the resolver's HasPrefix(key, BlobKeyPrefix) presign branch). - Voiding under reverse ingest order: a tombstone whose timestamp precedes the event it voids still hides that event (the anti-join keys on voids_id == id, not on time). Complements TestLakeEventService_VoidingExcludes (tombstone-after).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These events of type dimo.tombstone refer to files with the new voids_id column. By default, we hide voided files in the results.
Port of DIMO-Network/fetch-api#80.