Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-66 Refactor DagRun to DagVersion association #46565

Closed
1 task done
pierrejeambrun opened this issue Feb 7, 2025 · 0 comments · Fixed by #46626
Closed
1 task done

AIP-66 Refactor DagRun to DagVersion association #46565

pierrejeambrun opened this issue Feb 7, 2025 · 0 comments · Fixed by #46626
Assignees
@ephraimbuddy
Labels
AIP-66: DAG Bundle/Manifest area:core kind:meta High-level information important to the community
Milestone
Airflow 3.0.0

Comments

@pierrejeambrun
Copy link
Member

pierrejeambrun commented Feb 7, 2025 •
edited
Loading

Context

A DagRun can have multiple DagVersions associated. The reason for that is because the DagVersion can change in between tasks and even in between task tries.

To retrieve the DagVersions associated to a particular DagRun, we should explore the TI (current version) and TIH (history version for tries) and aggregate those.

Ideally this should be accessible through a property / association_proxy / relationship on the DagRun ORM instance so we can easily do DagRun.dag_versions to retrieve all of the related ORM DagVersion objects. (In addition we should be able to specify if we want to eagerload or lazyload that attribute).

The direct link between DagRun and DagVersion (via the FK) needs to be deleted.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.
@pierrejeambrun pierrejeambrun added AIP-66: DAG Bundle/Manifest kind:meta High-level information important to the community labels Feb 7, 2025
@pierrejeambrun pierrejeambrun added this to the Airflow 3.0.0 milestone Feb 7, 2025
@pierrejeambrun pierrejeambrun mentioned this issue Feb 7, 2025
Open
@dosubot dosubot bot added the area:core label Feb 7, 2025
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 10, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
15e10c0 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 10, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
f03b619 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
@ephraimbuddy ephraimbuddy mentioned this issue Feb 10, 2025
Merged
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 11, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
b17cfcf 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 11, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
e027ca2 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 12, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
b88bc1d 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 13, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
6c9a510 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 13, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
11a03f4 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 14, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
f006478 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 14, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
51dc3a9 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 14, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
b7014b9 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 15, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
167f567 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 17, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
97e9889 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 17, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
b10786a 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 17, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
d7dd1e1 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 18, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
2924ae5 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
ephraimbuddy added a commit to astronomer/airflow that referenced this issue Feb 18, 2025
@ephraimbuddy
AIP-66 Refactor DagRun to DagVersion association
8fb6277 
This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565
@github-project-automation github-project-automation bot moved this from In Progress to Done in AIP-66: DAG Bundles & Parsing Feb 18, 2025
ntr pushed a commit to ntr/airflow that referenced this issue Feb 20, 2025
@ephraimbuddy @ntr
AIP-66 Refactor DagRun to DagVersion association (apache#46626)
ba268b0 
* AIP-66 Refactor DagRun to DagVersion association

This removes DagRun.dag_version association and replaces it with dag_versions
property on DagRun that collects all the dag_version_ids associated with the
task instances of the DagRun.

closes: apache#46565

* fixup! AIP-66 Refactor DagRun to DagVersion association

* Update dagrun.dag_version method

* fixup! Update dagrun.dag_version method

* fixup! fixup! Update dagrun.dag_version method

* add more test

* Update .pre-commit-config.yaml

* fix test

* Use association proxy for ti.dag_versions and tih.dag_versions

* remove pre-commit changes

* loop over dagrun.task_instances and update the dagversion

* fixup! loop over dagrun.task_instances and update the dagversion

* deduplicate dag_versions using dict.fromkeys

* fix test failures

* fixup! fix test failures

* fixup! fixup! fix test failures

* fixup! fixup! fixup! fix test failures

* Test that verify_integrity is not called for versioned bundle

* order dagrun.version_number correctly

* join load task_instances when examining running dagruns
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ephraimbuddy ephraimbuddy

Labels
AIP-66: DAG Bundle/Manifest area:core kind:meta High-level information important to the community
Projects
AIP-66: DAG Bundles & Parsing
Status: Done
Milestone
Airflow 3.0.0
Development

Successfully merging a pull request may close this issue.

AIP-66 Refactor DagRun to DagVersion association astronomer/airflow
2 participants
@ephraimbuddy @pierrejeambrun