Skip to content

fix(incremental): bound boundary-dedup state growth (#4030)#4131

Open
AndreaBozzo wants to merge 1 commit into
dlt-hub:develfrom
AndreaBozzo:fix/4030-unique-hashes-state-bloat
Open

fix(incremental): bound boundary-dedup state growth (#4030)#4131
AndreaBozzo wants to merge 1 commit into
dlt-hub:develfrom
AndreaBozzo:fix/4030-unique-hashes-state-bloat

Conversation

@AndreaBozzo

@AndreaBozzo AndreaBozzo commented Jun 28, 2026

Copy link
Copy Markdown

Fixes #4030.

Problem

When many records share the boundary (maximum) cursor value, dlt stores one deduplication hash per record in unique_hashes and re-writes the full list into _dlt_pipeline_state on every run, with no upper bound. A backfill that stamps every row with the same timestamp, or a low-resolution cursor (a date without a time), produces hundreds of thousands of hashes — tens of MB written to the state table on each run, growing the destination's pipeline-state table at run_count * state_size.

Why not just drop the hashes

The hashes are load-bearing. The closed range start re-fetches boundary rows on the next run, and unique_hashes is how dlt filters the ones already emitted. Dropping them (e.g. the issue's suggestion to skip them when primary_key is set) re-appends every boundary row on the next run — i.e. it reintroduces duplicate data. I confirmed this with a repro: a 3-run pipeline stays at exactly 1000 rows because of the hashes; without them it would grow to 2000/3000.

So this PR keeps the dedup semantics intact and instead makes the cost visible and boundable:

  • Opt-in hard cap. New Incremental.duplicate_cursor_error_threshold (ClassVar, None/disabled by default) raises the new terminal IncrementalCursorThresholdExceeded once the boundary-hash count crosses it, so state can't silently balloon. Fails fast on the batch that crosses the bound.
  • Actionable warning. The existing duplicate_cursor_warning_threshold warning now names the concrete mitigations: use a higher-resolution cursor, or switch to merge_key + range_start="open" to disable boundary deduplication, or set the error threshold to fail instead of accumulating.
  • Docs. New "Growing pipeline state from boundary deduplication" section in the incremental troubleshooting page.

Default behavior is unchanged — the cap is off unless explicitly set.

Tests

  • test_duplicate_cursor_error_threshold: default keeps all hashes and never raises; with the threshold set, crossing it raises IncrementalCursorThresholdExceeded (terminal, wrapped in PipelineStepFailed).
  • test_duplicate_cursor_warning_threshold: the warning fires once and includes the range_start='open' / duplicate_cursor_error_threshold guidance.

When many records share the boundary (max) cursor value, dlt stores one
deduplication hash per record in `unique_hashes` and re-writes the full
list to `_dlt_pipeline_state` on every run, with no upper bound — bloating
the pipeline-state table to tens/hundreds of MB on backfills or
low-resolution cursors.

The hashes are load-bearing (dropping them re-emits boundary rows as
duplicates on the next run), so rather than change dedup semantics this:

- adds an opt-in `Incremental.duplicate_cursor_error_threshold` (ClassVar,
  disabled by default) that raises `IncrementalCursorThresholdExceeded`
  once the boundary-hash count crosses it, so state can't silently balloon
- rewrites the existing warning to be actionable (higher-resolution cursor,
  or `merge_key` + `range_start="open"` to disable boundary dedup)
- documents the behavior and mitigations in incremental troubleshooting

Default behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tens of MB of duplicated data written to pipeline state on every run due to boundary-row retention in unique_hashes

1 participant