Chi Hack Night Ghost Buses Breakout Group

This is the repo for the Chi Hack Night Ghost Buses breakout group! As of September 2024, the Mansueto Institute for Urban Innovation is taking over maintenance of the data and data pipelines originally created for this project. Archived real-time bus location data dating back to May 2022 will remain publicly available through them. This repo is no longer actively maintained. It is preserved in its state as of September 2024 for reference.

AWS architecture

Updated May 19, 2022

Currently, the AWS setup is based on this post with some references to the CPS Covid Dashboard back-end.

TODO: Describe how to make updates to the code in AWS

Getting started

Set up a virtual environment with python3 -m venv .venv and activate it with source .venv/bin/activate (.venv\Scripts\activate in Windows). Then run pip install -r requirements.txt to install the required packages.

To set environment variables, create a .env file in the top-level directory and add the following contents:

BUCKET_PUBLIC='chn-ghost-buses-public'
BUCKET_PRIVATE='chn-ghost-buses-private'

Note that there may be permissions issues when running rt_daily_aggregations.py with BUCKET_PUBLIC as the BUCKET_NAME.

You may also need to set the PROJECT_NAME environment variable:

PROJECT_NAME=chn-ghost-buses

Updating data

The update_data script reads realtime and schedule data, combines it, and produces output files used by the frontend. The common use case will be to process data that has arrived since the previous frontend data deploy. To do this, check out the ghost-buses-frontend repository alongside this backend repository. Then run the following:

python3 -m update_data --update ../ghost-buses-frontend/src/Routes/schedule_vs_realtime_all_day_types_routes.json

To calculate all data from the start of the project, run the script with no arguments:

python3 -m update_data

In either case, output files will be written under the data_output/scratch directory with names indicating the date ranges that they cover. You can then deploy by copying the following output files:

• schedule_vs_realtime_all_day_types_overall_<date>_to_<date>.json to ghost-buses-frontend/src/Routes/schedule_vs_realtime_all_day_types_routes.json
• frontend_data_<date>_to_<date>_wk.json to ghost-buses-frontend/src/Routes/data.json

Pre-commit hooks

Pre-commit hooks are defined in .pre-commit-config.yaml, based on this post. The hooks will enforce style with the linter black, check for commited credentials and extraneous debugger breakpoints, and sort imports, among other things.

To get the latest versions of dependencies, update the .pre-commit-config.yaml file with pre-commit autoupdate. To run the hooks against all files without committing, use pre-commit run --all-files. To have the pre-commit hooks run when executing git commit, use pre-commit install.