Ingester: account for duplicate samples silently dropped at TSDB commit - #16347
Draft
dmatth1 wants to merge 2 commits into
Draft
Ingester: account for duplicate samples silently dropped at TSDB commit#16347dmatth1 wants to merge 2 commits into
dmatth1 wants to merge 2 commits into
Conversation
|
|
dmatth1
force-pushed
the
dan/ingester-duplicate-sample-stats
branch
from
August 11, 2026 17:03
1270ab5 to
3da564f
Compare
dmatth1
force-pushed
the
dan/ingester-duplicate-sample-stats
branch
from
August 11, 2026 19:12
3da564f to
cdc565f
Compare
Contributor
|
💻 Deploy preview available (Mimir): |
Brings in the CommitStats appender capability and the prometheus_tsdb_head_duplicate_samples_dropped_total metric from grafana/mimir-prometheus#1259. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dan Mattheiss <dan.mattheiss@grafana.com>
The TSDB head appender silently drops samples whose timestamp collides with an already-stored sample: exact duplicates (same value, e.g. client retries, Kafka replay), and same-timestamp conflicts that are only detectable at commit time (duplicates within one batch, or races with concurrent appends). Append returns no error for these, so the ingester counted them as ingested and recorded no discard, making ingestion impossible to reconcile with what senders report. After a successful commit, read tsdb.DiscardedSampleStats from the appender and reclassify the drops: - cortex_discarded_samples_total gains reason same-value-for-timestamp for exact duplicates; commit-time conflicts count into the existing new-value-for-timestamp reason. - cortex_ingester_ingested_samples_total no longer counts the dropped samples (it now reflects post-commit truth). - The drops are attributed per series in cost attribution. - Dropped duplicates still refresh the TSDB last-update timestamp, so duplicate-only traffic does not idle-close the tenant TSDB. The stats capability is part of extendedAppender, so a future appender wrapper that fails to forward it breaks loudly on the first push instead of silently zeroing the accounting. Co-authored-by: Yuri Nikolic <durica.nikolic@grafana.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dan Mattheiss <dan.mattheiss@grafana.com>
dmatth1
force-pushed
the
dan/ingester-duplicate-sample-stats
branch
from
August 11, 2026 19:35
cdc565f to
945b18f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
The TSDB head appender silently drops samples whose timestamp collides with an already-stored sample: exact duplicates (client retries, duplicate shippers, Kafka replay after an ingester restart) and same-timestamp conflicts that are only detectable at commit time.
Appendreturns no error for these, so the ingester counted them as ingested and recorded no discard — making ingestion impossible to reconcile with what senders report.After a successful commit, the ingester now reads
tsdb.CommitStatsfrom the appender (added in grafana/mimir-prometheus#1259) and reclassifies the drops:cortex_discarded_samples_totalgains reasonsame-value-for-timestamp(exact duplicates); commit-time conflicts count into the existingnew-value-for-timestamp.cortex_ingester_ingested_samples_totalno longer counts the dropped samples (it now reflects what was actually stored).The stats capability is part of
extendedAppender, so an appender wrapper that fails to forward it breaks loudly on the first push instead of silently zeroing the accounting.Notes for reviewers:
vendor/commit is mechanical — review the TSDB side in tsdb: report CommitStats including duplicate samples dropped at commit time mimir-prometheus#1259. It currently pins that PR's branch (62700c08baf5) and will be dropped (or re-pinned to forkmain) once Revise mimirtool docs #1259 merges. Do not merge before tsdb: report CommitStats including duplicate samples dropped at commit time mimir-prometheus#1259.cortex_ingester_ingested_samples_totalchanges meaning from Append-time counting to post-commit truth (see CHANGELOG[CHANGE]); the only mixin consumer is the ingest-activity guard in theMimirIngesterHasNotShippedBlocks*alerts, which this makes marginally more accurate.prometheus_tsdb_out_of_order_samples_total(vendored change).Validated end-to-end on a local ingest-storage cluster and on a classic-architecture dev cell: per-scenario discard accounting, first-value-wins storage, identical counts across zones, and the Kafka-replay restart path.
Which issue(s) this PR fixes or relates to
Fixes #15552
Checklist
pkg/ingester/ingester_push_duplicate_test.go— scenarios by @duricanikolic — plus TSDB-level coverage in tsdb: report CommitStats including duplicate samples dropped at commit time mimir-prometheus#1259).CHANGELOG.mdupdated.Co-authored with @duricanikolic (design + prototype + tests).
🤖 Generated with Claude Code