Skip to content

Latest commit

 

History

History
186 lines (123 loc) · 5.71 KB

README.md

File metadata and controls

186 lines (123 loc) · 5.71 KB

Python

Version Management

Preferred Tool: pyenv

How to Install: brew install pyenv

Installing Python

pyenv install 3.7 2.7 && pyenv global 3.7 2.7

Dependency Management

Project (soon to be obsolete, thinking of upgrading to pipx + poetry)

Preferred Tool: pipenv

How to Install: pip install pipenv (so that I can change pipenv versions easily)

Notes:

  • For packages that is installed globally, use pip
  • For packages that are installed local to a project, use pipenv

Debugging failed installations in Pipenv

https://pipenv.readthedocs.io/en/latest/diagnose/

pipenv --rm # Deletes the virtualenv
pipenv lock --clear # Removes cache
pipenv clean # Clean unused packages

rm -rf ~/Library/Caches/pipenv
rm -rf ~/.cache/pipenv
rm Pipfile.lock

Global

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda

# activate base
source <path to conda>/bin/activate
conda init [zsh]

IDE

Preferred Tool: PyCharm

How to Install:

Version Install Command
Community Edition brew cask install pycharm-ce
Ultimate brew cask install pycharm

https://www.jetbrains.com/pycharm/download/#section=mac

Exploration / Interactive Tools

Tool Install Command
IPython pipenv install ipython
Jupyter Lab pipenv install jupyterlab
Jupyter Notebook pipenv install jupyter

Data Science Tools

Usage Tools
Maths and statistics , , statsmodels
Visualizations matplotlib, seaborn, plotly
Machine Learning Models scikit-learn
Neural Network tensorflow, keras, pytorch
Interactive Visualizations Dash-plotly

Common Libraries

Maths and Statistics

Package How to install
numpy pipenv install numpy
scipy pipenv install scipy
statsmodels pipenv install statsmodels

Data Processing and Wrangling

Package How to install
pandas pipenv install pandas
pyspark See below

Visualizations

Package How to install
matplotlib pipenv install matplotlib
seaborn pipenv install seaborn
plotly pipenv install plotly

Machine Learning

Package How to install
scikit-learn pipenv install scikit-learn

Neural Network

Package How to install
tensorflow pipenv install tensorflow
keras pipenv install keras
pytorch pipenv install torch torchvision

Interactive Dashboard and Visualizations

Package How to install
Dash Plotly pipenv install dash dash-html-components dash-core-components dash-table dash-daq

pyspark

Ensure that python and spark is installed

Documentations:

pyspark explorations

To explore and run pyspark, we have two options:

Exploration with Apache Zeppelin
  1. Install Apache Zeppelin: brew install apache-zeppelin
  2. Run Zeppelin: zeppelin-daemon.sh start
Exploration with Jupyter Notebook
  1. Ensure you've installed jupyter / jupyterlab
  2. Follow this instruction to setup pyspark with jupyter

This article compare and contrast the benefits and disadvantages of using either Apache Zeppelin or Jupyter Notebook

Testing

Preferred Tool: pytest

How to Install: pipenv install --dev pytest

Server-side Development

Framework and Tools

Tool Usage
Marshmallow a Marshalling library for converting complex datatypes to and from native Python datatypes
webargs Library for parsing and validating HTTP request arguments, used in conjunction with Marshmallow

Flask

Tool Usage
Flask-RESTPlus Flask extension for building REST APIs
flasgger Integrate OpenAPI specs and Swagger UI for Flask APIs
pytest-flask Testing flask apps

References:

Documentation Resources
Flask Testing http://flask.pocoo.org/docs/1.0/testing/#the-testing-skeleton
Creating flask extensions - useful for modularizing the APIs and testing http://flask.pocoo.org/docs/1.0/extensiondev/
Scaling applications: using API namespaces or blueprints https://flask-restplus.readthedocs.io/en/stable/scaling.html

FastAPI

https://github.com/tiangolo/fastapi

WSGI HTTP Server

Preferred Tool: https://gunicorn.org/