python3 readiness

list of frameworks

proxy

import os

proxy = 'http://<user>:<pass>@<proxy>:<port>'
os.environ['http_proxy'] = proxy 
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy

package manager easy_install

using it from cmd

%PYTHON%/Scripts/easy_install.exe <package name>

using easy_install from script

from setuptools.command import easy_install
# install package
easy_install.main( ["termcolor"] )
# update package
easy_install.main( ["-U","termcolor"] )

package manager pip

find package in browser

find package by name

pip search {search key}

install pip

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

issue:

ImportError: cannot import name 'sysconfig'

solution

# pip install
# sudo apt install python3-distutils
sudo apt install python3-pip

upgrade pip

pip3 install --upgrade pip
pip install -U pip

debian

apt-get install -y --no-install-recommends python3-pip

pip install with proxy, pip install proxy, pip proxy, pip via proxy

pip install --proxy=http://proxy.muc:8080

pip install with specific proxy

pip install --index-url http://cc-artifactory.mynetwork.net my_own_package

install package into home of current user ( do not use for virtual environment )

pip install --user .

pip install from package install from zip

pip3 install --user ~/Downloads/PyGUI-2.5.4.tar.gz

setup.py

install

python setup.py install

uninstall

python setup.py install --record list_of_files.txt
cat list_of_files.txt | xargs sudo rm -rf

list of packages

pip list

list of all installed libraries, installed modules

pip freeze

import sys
sys.modules

list of all folders with source code, installed packages

import sys
print(sys.prefix)

for easy_install, pip

import site
print(site.getsitepackages())

using pip from interpreter ( install wheels package)

import pip
pip.__version__
pip.main(["install", "wheels"])
#pip.main("install", "wheels")

fix for version 9 and 10

error message: AttributeError: 'module' object has no attribute 'main'

solution:

try:
    from pip import main as pipmain
except:
    from pip._internal import main as pipmain
pipmain(["install", "wheels"]);

path to external artifacts, external index, pip configuration

cat /etc/pip.conf

[global]
index-url = https://cc-artifactory.myserver.net/artifactory/api/pypi/adp-pypi-virtual/simple

install certain version of artifact

pip install tornado==2.1.1

create virtual environment, dedicated env

pip install virtualenv

# create 
#python3 -m virtualenv venv
virtualenv venv

# activate
source venv/bin/activate

# commands will be executed into virtual env
pip install wheel 
python3 

# exit from virtual environment
deactivate

install list of artifacts

pip install -r requirements.txt
# sometimes after installation not working
# 'pip list' & 'pip freeze' are not consistent
cat requirements.txt | xargs -I {} ./pip3 install {}

where requirements.txt is:

-r requirements-base.txt
docker==3.7.0
enum34==1.1.6
flask-restful==0.3.7

install artifact from git

pip install git+https://github.com/django-extensions/django-extensions
pip install git+https://github.com/django-extensions/django-extensions.git
pip install -e git+https://github.com/django-extensions/django-extensions.git#egg=django-extensions
pip install https://github.com/django/django/archive/stable/1.7.x.zip
pip install git+ssh://git@github.com/myuser/foo.git@my_version

install package to specific folder

pip install --target=/home/user/my/python/packages package_name
export PYTHONPATH=$PYTHONPATH:"/home/ubuntu/.local/lib/python3.8/site-packages/gunicorn"

load package from specific folder inline

import sys
sys.path.append("/path/to/your/my_extra_component")
import extra_component

update package

pip install tornado --update

remove package, uninstall package

pip uninstall {package name}

print dependency tree

pip install pipdeptree
pipdeptree

for transitive dependency:

constraints.txt file should be considered pip install -c constraints.txt
poetry ( under the hood uses pip )

import package by string name

target = __import__("data-migration")

import package in protected block

try:
    import json
except ImportError:
    import simplejson as json

script execution ModuleNotFoundError

when you execute your own project locally

ModuleNotFoundError: No module named 'list_comparator'

echo $PYTHONPATH
export PYTHONPATH=$PYTHONPATH:"/home/projects/wondersign/integration-prototype"

/home/projects/integration-prototype/list-comparator/venv/bin/python /home/projects/integration-prototype/list-comparator/list_comparator/data_api/tools/data_api_reader.py

create executable environment

pex --python=python3 flask requests tornado -o samplepkg.pex

path to python, obtain python interpreter path

import sys
sys.executable

also check

sys.modules

execute command inline, base64 example

python -c "import base64;print(base64.encodestring('hello'.encode()));"

execute string as commands

a=10
eval(" print(a)")

find vs index ( string find, string index )

find - return -1 if not found
index - throw exception if not found

user libraries can be placed into the folder

python -m site --user-site

help

help contextlib
pydoc contextlib

special methods

__call__  <class instance>()

special file name folder execution main execute main

mkdir ./my_folder
touch ./my_folder/__main__.py
python3 ./my_folder

build rpm: python setup.py bdist_rpm

	rpm -ba --define _topdir ...
	-ba: unknown option
	error: command 'rpm' failed with exit status 1

need to execute: yum install rpm-build

build package usind pants with special folder

# BUILD PEX
PANTS_PATH="../../../.."
PANTS_OUTPUT=$CURRENT_PATH/dist
$PANTS_PATH/pants --pants-distdir=$PANTS_OUTPUT binary src/path/to/package:package-name

(debug with cli)[https://docs.python.org/3/library/pdb.html]

import pdb
# ...
pdb.set_trace()
# breakpoint()

for ipdb:

# python3 -m pip install --user ipdb
__import__('ipdb').set_trace(context=21)

execute python app in debug

python3 -m pdb myscript.py
python3 -m pdb myscript.pex arg1 arg2 
pdb myscript.py

debug commands

b my_python_file.py:65
cont
step
next
p x+y
until 99999
continues
return

install packages

selenium, virtualdisplay

sudo pip install selenium
sudo pip install xvfbwrapper
sudo pip install pyvirtualdisplay
sudo apt-get install xvfb
wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux64.tar.gz
chmod +x geckodriver
sudo cp geckodriver /usr/local/bin/

virtual environment automation, tox tool

tox.ini

commands = pex . -c download_symbolic_link_creation.py --disable-cache -i {env:PIP_INDEX_URL} -r requirements.txt -o {env:PEX_OUTPUT_PATH} --python-shebang="/usr/bin/env python3.8"

setup.py

from setuptools import setup

setup(
    name='download_python_app',
    version='0.0.1',
    scripts=['download_python_app/download_symbolic_link_creation.py']
)

IDEA

standard modules not found

Have you set up a python interpreter facet?
Open Project Structure CTRL+ALT+SHIFT+S

Project settings -> Facets -> expand Python click on child -> Python Interpreter

Then:
Project settings -> Modules -> Expand module -> Python -> Dependencies -> select Python module SDK

sql, sqlgenerator, codegenerator, sqlalchemy

pip3 install sqlacodegen
sqlacodegen sqlite:///db.sqlite > generated-code.txt
sqlacodegen postgresql:///some_local_db
sqlacodegen mysql+oursql://user:password@localhost/dbname

# Flask approach
pip3 install flask-sqlacodegen
flask-sqlacodegen --flask sqlite:///db.sqlite > generated-code.txt
flask-sqlacodegen --flask mysql+pymysql://admin:admin@localhost:3310/masterdb > generated-code.txt

sql client, console mysql client, mysql console, db cli

commands

apt-get install mycli
pip install -U mycli
mycli --user my_user --password my_password --host my_host.com --port 3310 --database my_database --execute 'show tables'

# for activating multiline mode 'F3'

Code Style code Formatter

# installation
sudo apt-get remove black && python3 -m pip install black==20.8b1
# execution
black --line-length 100
python -m black {file of directory}

black with IDEA, black with PyCharm

install FileWatcher plugin
Tool to Run on Changes: Program: ~/.locl/bin/black
Tool to Run on Changes: Arguments: $FilePath$

Test coverage

# cli tool
pip3 install tox 
# plugin for PyCharm: pytest-cov

run tests

python -m unittest airflow_shopify.shopify.test_shopify_common

Alembic

migration

# step #1 
alembic init datastorage_mysql

# step #2 
# check your "sqlalchemy.url" in "ini file"
# check your "target_metadata" in "env.py"
# target_metadata = Base.metadata # from database.models import Base

# step #3
# generate current difference between model and 
export PYTHONPATH=$PYTHONPATH:$(pwd)/src
alembic --config alembic-local.ini revision --autogenerate -m "baseline" 2>out.txt

alembic --config alembic-local.ini history

alembic --config alembic-local.ini upgrade head

reset alembic

delete from alembic_version where 1=1;

rm datastorage_mysql/versions/*

supplementary steps

# step set marker in DB
# create table if not exists alembic_version
alembic --config alembic-local.ini current

# upgrade to specific revision
alembic --config alembic-local.ini upgrade cfaf8359a319 --sql

GUI

PySimpleGUI

 sudo apt-get install python3-tk
 pip3 install PySimpleGUI

PyQT5
```
 pip install pyqt5
```
Python Tkinter
PySide 2

Kivy

 pip install docutils pygments pypiwin32 kivy.deps.sd12 kivy.deps.glew
 pip install kivy

wxPython
```
 pip install wxPython
```

Web frameworks lightweight

bottle
CherryPy
Falcon
FastAPI
Pyramid
Sanic
Tornado
Streamlit
viola

Database

PostgreSQL

pip3 install psycopg2-binary
# need to execute: yum install rpm-build
pip3 install -U pip

Files

python.md

Latest commit

History

python.md

File metadata and controls

python3 readiness

list of frameworks

proxy

package manager easy_install

using it from cmd

using easy_install from script

package manager pip

find package in browser

find package by name

install pip

upgrade pip

pip install with proxy, pip install proxy, pip proxy, pip via proxy

pip install with specific proxy

install package into home of current user ( do not use for virtual environment )

pip install from package install from zip

setup.py

install

uninstall

list of packages

list of all installed libraries, installed modules

list of all folders with source code, installed packages

using pip from interpreter ( install wheels package)

path to external artifacts, external index, pip configuration

install certain version of artifact

create virtual environment, dedicated env

install list of artifacts

install artifact from git

install package to specific folder

load package from specific folder inline

update package

remove package, uninstall package

print dependency tree

import package by string name

import package in protected block

script execution ModuleNotFoundError

create executable environment

path to python, obtain python interpreter path

execute command inline, base64 example

execute string as commands

find vs index ( string find, string index )

user libraries can be placed into the folder

help

special methods

special file name folder execution main execute main

build rpm: python setup.py bdist_rpm

build package usind pants with special folder

(debug with cli)[https://docs.python.org/3/library/pdb.html]

install packages

selenium, virtualdisplay

virtual environment automation, tox tool

IDEA

standard modules not found

sql, sqlgenerator, codegenerator, sqlalchemy

sql client, console mysql client, mysql console, db cli

Code Style code Formatter

Test coverage

run tests

Alembic

migration

reset alembic

supplementary steps

GUI

Web frameworks lightweight

Database

PostgreSQL