CodeDay DC - A Brief Introduction to Data Engineering

Hello! This repository was created as part of a presentation I made at CodeDay DC on 2022-11-19. Here's a link to that presentation.

The other parts of this repository include the notebook that I used to demonstrate the different parts of a data pipeline (get_data_and_plot.ipynb), a version of the pipeline working just on my laptop (fetch_parking_tickets.py), and a version of the pipeline working with Prefect (fetch_parking_tickets_with_prefect.py).

If you have any questions or would just like to connect, feel free to reach me on LinkedIn.

Environment Setup

Conda

For OSX, use Homebrew's distribution of Miniconda:

brew install miniconda

Installing & Activating This Environment

conda env create -f environment.yml
conda activate code_day_dc