Add serialized Dag version diff engine#1591
Closed
ephraimbuddy wants to merge 1 commit into
Closed
Conversation
Dag authors and operators need a reliable way to understand how two stored Dag versions differ without downloading serialized payloads or manually comparing implementation details. This adds the shared comparison engine that later API and CLI surfaces build on. Historical serialized Dags can use different schema shapes, so comparisons normalize task, schedule, dependency, and metadata state before calculating changes. Malformed legacy payloads and missing historical data degrade to unavailable results instead of failing the whole request.
Author
|
Superseded — opened against apache/airflow instead: apache#69864 Drafted-by: Claude Code (Opus 4.8) (no human review before posting) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the shared engine that computes an observed-state diff between two stored serialized Dag versions. Dag authors and operators need a reliable way to understand how two stored versions differ without downloading serialized payloads or manually comparing implementation details.
This is the foundation of a three-part stack; the REST API endpoint and the
airflow dags versions diffCLI command follow in separate PRs that build on this engine. It intentionally ships with no runtime caller yet — it is fully exercised by unit tests.What it does
build_serialized_dag_diff— pure, deterministic, bounded diff over two serialized payloads. Normalizes task/schedule/dependency/metadata state, converts legacy schema shapes (v1/v2 → v3), keys order-insensitive collections, categorizes each change (task, dependency, schedule, param, asset, callback, deadline, metadata, provenance) and its impact, and emits digests plus optional raw values.get_dag_version_diff— DB-facing wrapper that loads twoDagVersions, assembles provenance, and classifies optional source/values data as available, redacted, or unavailable.unavailableresult instead of failing the whole request.Testing
@pytest.mark.db_testclass coveringget_dag_version_diffend-to-end: observed-state diff, not-found handling, version-number validation, and the values/source status plumbing (available, redacted, unavailable-when-code-missing).No newsfragment: the engine has no user-facing surface on its own; user-facing notes ship with the API and CLI PRs.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines
Drafted-by: Claude Code (Opus 4.8) (no human review before posting)