Skip to content

fix: start extract resource timing before first item#4121

Open
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:fix/3518-restapi-progress-timing
Open

fix: start extract resource timing before first item#4121
fengjikui wants to merge 1 commit into
dlt-hub:develfrom
fengjikui:fix/3518-restapi-progress-timing

Conversation

@fengjikui

Copy link
Copy Markdown

Fixes #3518.

Summary

  • initialize static extract resource counters when their source pipe starts, before the first item is pulled
  • keep dynamic table-name resources on the existing item-driven timing path because their table name is not known before data arrives
  • add a regression test that simulates a resource waiting before yielding its first item and verifies the logged resource time includes that wait

Validation

  • uv run black dlt/extract/pipe_iterator.py dlt/extract/extract.py tests/extract/test_extract.py
  • git diff --check
  • uv run pytest tests/common/runtime/test_collector.py tests/extract/test_extract_pipe.py tests/extract/test_extract.py::test_resource_log_time_includes_wait_before_first_item -q
  • uv run pytest tests/extract/test_extract.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RestAPI incorrectly reports resource runtime for non-paginated requests

1 participant