Add DAG for exporting retool data #550

amishas157 · 2024-12-06T23:03:30Z

PR Checklist

PR Structure

This PR has reasonably narrow scope (if not, break it down into smaller PRs).
This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
This PR's title starts with the jira ticket associated with the PR.

Thoroughness

This PR adds tests for the most critical parts of the new functionality or fixes.
I've updated the README with the added features, breaking changes, new instructions on how to use the repository.

What

This PR adds DAG to fetch custom data from retool API. It triggers command in stellar-internal , pull data created/updated within given time range.
Existing batch_id is deleted and then re-created(This is to keep idempotent results while retries)

Why

To support new data ingestion

Known limitations

Not a limitation, but open to renaming the dag if it too generic. Probable options:

external_data_dag_retool
export_retool
external_data_dag_daily (This can be used to import any external data which runs at daily cadence)

Testing

Tested the airflow in test-hubble and records look loaded in BQ table

udpate image udpate image udpate image udpate image

Add alias

update xcom value update set xcom input simplify rename

update image

update move update update

amishas157 · 2024-12-16T22:25:33Z

airflow_variables_dev.json

@@ -315,6 +315,7 @@
  "schema_filepath": "/home/airflow/gcs/dags/schemas/",
  "sentry_dsn": "https://[email protected]/6190849",
  "sentry_environment": "development",
+  "stellar_etl_internal_image_name": "amishastellar/stellar-etl-internal:cd53bcf70",


This is temporary. Once we have build setup for stellar-etl, will switch to stellar workspace.

amishas157 · 2024-12-16T22:26:22Z

schemas/retool_entity_data_schema.json

@@ -0,0 +1,163 @@
+[


Took this JSON schema from bigquery after creating the table

dags/external_data_dag.py

chowbao · 2024-12-16T22:35:15Z

dags/external_data_dag.py

+dag = DAG(
+    "external_data_dag",
+    default_args=get_default_dag_args(),
+    start_date=datetime(2024, 12, 5, 14, 30),


I recommend updating this start date. Generally because this is set to run "0 22 * * *" you don't want to set the hour::minutes part of the datetime

Done in 9a4e60f

dags/external_data_dag.py

chowbao · 2024-12-16T22:44:06Z

dags/external_data_dag.py

+)
+
+
+def get_insert_to_bq_task(


Is this function different than https://github.com/stellar/stellar-etl-airflow/blob/master/dags/stellar_etl_airflow/build_bq_insert_job_task.py or anything else in ./dags/stellar_etl_ariflow/build_*?

If it's different it might be worthwhile to separate this out into either utils.py or another *.py file

I think, it is different since it is combination of delete and insert. Whereas the one in build_bq_insert_job_task is just insertion

Ah that's fine then. Yeah a nit to move to a separate helper file. Also recommend renaming the function to be more descriptive of it deleting and inserting instead of just get_insert_to_bq_task. Maybe just add delete to the function name

So I moved it to dags/stellar_etl_airflow/build_del_ins_operator.py in 9a4e60f

Also renamed the method to create_export_del_insert_operator. Essentially create_export_del_insert_operator is specific type of create_del_ins_task

dags/external_data_dag.py

chowbao · 2024-12-16T22:48:32Z

dags/stellar_etl_airflow/build_internal_export_task.py

+    image = "{{ var.value.stellar_etl_internal_image_name }}"
+
+    output_filepath = ""
+    if use_gcs:


nit: technically the gcs if/case could be split out into a separate function or interface so that it's easy to add support for other cloud storage systems. BUT I don't think we'll ever need support outside of GCS for this so it's fine to leave this as is

Agree. I think in general we could try to write the operators as classes, which can help with overriding.. Right now, it is more functional.

Honestly one day we'll probably refactor all of stellar-etl-airflow

chowbao

LGTM left some minor comments

Add DAG for exporting retool data

30ee494

amishas157 force-pushed the patch/fetch-retool-data branch 3 times, most recently from bbcf345 to c1aa6df Compare December 6, 2024 23:28

udpate image

ef7e871

udpate image udpate image udpate image udpate image

amishas157 force-pushed the patch/fetch-retool-data branch from c1aa6df to ef7e871 Compare December 6, 2024 23:32

amishas157 added 3 commits December 10, 2024 17:08

Pass api key

47ca1d2

send cloud flags

3e56ccb

Add task to write to bq

1102681

Add alias

amishas157 force-pushed the patch/fetch-retool-data branch from 4f4b420 to 1102681 Compare December 11, 2024 22:49

rename

718238e

amishas157 force-pushed the patch/fetch-retool-data branch 3 times, most recently from 64605ef to b95a518 Compare December 11, 2024 23:48

Add batch_run_date

c632660

update xcom value update set xcom input simplify rename

amishas157 force-pushed the patch/fetch-retool-data branch from b95a518 to c632660 Compare December 12, 2024 00:03

amishas157 added 7 commits December 11, 2024 18:14

update suffix

fe6647d

update image

6378f51

Add metadata fields

a2ec946

Change schedule and update start/end time

e15108f

Add macro

52ca776

Refactor into methods

3d3fc6a

moar refactor

7ee9e0d

update image

amishas157 force-pushed the patch/fetch-retool-data branch from 4090b59 to 7ee9e0d Compare December 13, 2024 21:46

Merge branch 'master' into patch/fetch-retool-data

ab8be7a

amishas157 force-pushed the patch/fetch-retool-data branch from 75c4c6e to 2461095 Compare December 13, 2024 22:37

update env_vars early on

06884ed

amishas157 force-pushed the patch/fetch-retool-data branch 3 times, most recently from f0e9186 to d09b178 Compare December 16, 2024 21:40

amishas157 force-pushed the patch/fetch-retool-data branch from d09b178 to 1576b35 Compare December 16, 2024 22:00

Extract secret from gcs

5920807

update move update update

amishas157 force-pushed the patch/fetch-retool-data branch from 1576b35 to 5920807 Compare December 16, 2024 22:07

Remove extranneous vars

9497ebb

amishas157 commented Dec 16, 2024

View reviewed changes

amishas157 marked this pull request as ready for review December 16, 2024 22:27

amishas157 requested a review from a team as a code owner December 16, 2024 22:27

amishas157 commented Dec 16, 2024

View reviewed changes

dags/external_data_dag.py Show resolved Hide resolved

chowbao reviewed Dec 16, 2024

View reviewed changes

dags/external_data_dag.py Show resolved Hide resolved

chowbao reviewed Dec 16, 2024

View reviewed changes

dags/external_data_dag.py Show resolved Hide resolved

chowbao reviewed Dec 16, 2024

View reviewed changes

chowbao approved these changes Dec 16, 2024

View reviewed changes

Move generic method to utils and update date

9a4e60f

amishas157 merged commit afc5bb1 into master Dec 16, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DAG for exporting retool data #550

Add DAG for exporting retool data #550

amishas157 commented Dec 6, 2024 •

edited

Loading

amishas157 Dec 16, 2024

amishas157 Dec 16, 2024

chowbao Dec 16, 2024

amishas157 Dec 16, 2024

chowbao Dec 16, 2024

amishas157 Dec 16, 2024

chowbao Dec 16, 2024

amishas157 Dec 16, 2024

chowbao Dec 16, 2024

amishas157 Dec 16, 2024

chowbao Dec 16, 2024

amishas157 Dec 16, 2024

chowbao left a comment

		)


		def get_insert_to_bq_task(

Add DAG for exporting retool data #550

Add DAG for exporting retool data #550

Conversation

amishas157 commented Dec 6, 2024 • edited Loading

PR Structure

Thoroughness

What

Why

Known limitations

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chowbao left a comment

Choose a reason for hiding this comment

amishas157 commented Dec 6, 2024 •

edited

Loading