Skip to content

fix: preserve incremental step order on clone#4118

Open
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:codex/preserve-insert-at-clone-order
Open

fix: preserve incremental step order on clone#4118
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:codex/preserve-insert-at-clone-order

Conversation

@fengjikui

Copy link
Copy Markdown

Summary

Refs #3838.

This preserves the explicit pipe position of an existing incremental step when a resource is cloned and its hints/config wrappers are ejected and injected again. The issue was that hint-based incremental replacement during clone removed the existing incremental transform and re-added it through the default affinity-based append path, which could move user-inserted steps such as MetricsItem before incremental filtering.

The fix records the current incremental step index before replacement/ejection and passes that index back into set_incremental when the incremental is reinserted. Normal first-time incremental insertion still uses the existing affinity behavior.

Validation

  • uv run --with duckdb --with pyarrow --with pydantic pytest tests/extract/test_incremental.py::test_clone_preserves_inserted_step_after_hints_incremental tests/extract/test_incremental.py::test_pydantic_columns_validator tests/extract/test_extract.py::test_add_metrics tests/extract/test_extract.py::test_custom_metrics_preserved_when_all_items_filtered -q -> 6 passed in 1.37s
  • uv run ruff check dlt/extract/resource.py tests/extract/test_incremental.py -> All checks passed!
  • python3 -m py_compile dlt/extract/resource.py tests/extract/test_incremental.py -> passed

I also ran the issue-style pipeline.run reproduction locally after the fix and observed row_count: 2, so the metrics step now sees only rows that survive incremental filtering.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant