the_gladiators
- Nikita Mandlik ([email protected])
- Brinda Eshwar ([email protected])
- Kanishk Bharampurkar ([email protected])
- Harshad Bhandwaldar ([email protected])
The data collected using the Twitter API and the RedditAPI, will be used to study and develop insights about the public opinion on the economic crisis, inflation, and the influence of upcoming-recession updates. The data will help to predict the impact of the upcoming recession on the community. Also, to analyze the insights generated from the public opinionated datasets (Reddit and Twitter) and news articles (collected from NewYork Times API).
- Twitter API
- Reddit API
- New York Times API
- Scrapper: app folder has twitter, reddit and ny times api to scrape data and store it in database.
- UI: Using fastapi data can be visualize. The api function only has functionality to display the database data.
Building env: Installing poetry and setting up virtiual environment.
$ sh build.sh
Running application (run each scrapper independently):
$ python3 /app/twitter/scrapper_twitter.py
$ python3 /app/reddit/scrapper_reddit.py
$ python3 /app/nyt/scrapper_nyt.py
Running UI:
cd ui/
$ uvicorn main:app --reload
Go to http://localhost:8000/docs to open UI.
API keys and database credentials changed from config.py located at root of each app.
| Name | Requirement |
|---|---|
| Memory | 8Gb |
| OS | Linux |
[1] Twitter API Documentation. https://developer.twitter.com/en/docs/twitter-api
[2] Reddit API Documentation. https://www.reddit.com/dev/api/
[3] NYTimes API Documentation. https://developer.nytimes.com/apis
[4] Article Search API Documentation. https://developer.nytimes.com/docs/articlesearch-product/1/overview
[5] Docker Documentation. https://docs.docker.com/get-started/overview/
[6] Poetry Documentation. https://python-poetry.org/docs/
[7] MySQL Documentation. https://dev.mysql.com/doc/