Skip to content

Commit

Permalink
Fix bug where big query operator creates duplicate job versions (Marq…
Browse files Browse the repository at this point in the history 
…uezProject#111)

Bump python circleci

    Remove empty dataset on failure, clean up bq extractor

    Remove output schema for failing tasks, add tests, refactor airflow to share logic

    Fix rebase err

    Fix bug where big query operator creates duplicate job versions

    Moves the big query operator to emit run and job information after the run has completed. Also emits actual task started date as defined by the task instance.

    Fix bug where all runs fail if DAG fails

    Reports task state information for each run instead of dag state.

    Also: reworks the handle_callback logic to remove superfluous call to report_task for operators with extract_on_complete (BigQuery).
    Signed-off-by: henneberger <[email protected]>

Signed-off-by: henneberger <[email protected]>
  • Loading branch information
@henneberger
henneberger authored Dec 4, 2020
1 parent 9575a17 commit 778fe8c
Show file tree
Hide file tree
Showing 5 changed files with 561 additions and 296 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 2

defaults: &defaults
docker:
- image: circleci/python:3.6
- image: circleci/python:3.8.5
working_directory: ~/marquez_airflow


Expand Down
Loading

0 comments on commit 778fe8c

Please sign in to comment.