This is a data engineering project that extracts data from yfinace api , converts the extracted json to avro and dumps it in Google Cloud Storage. After that it transfers the data to GCP BigQuery. For data orchestration we use airflow.
This project is heavily influenced by this article
For future I may use the alpaca api instead of yfinace api to adjust the usage of realtime data and for shorter time frames (seconds, minutes etc.) that are not offered with yfinance api