Profile and Benchmark Python Code

The purpose of this part is to have a ready made vademecum to profile Python code, with no need to google stuff every time.

Normal Scripts (no Jupyter)

Time

Tools to measure time spent inside functions, with the procedures to use them.

• Snakeviz
  • generate a .prof file with the command

   python -m cProfile -o p.prof code_profiling_main.py --dim 10000 --loops 4

  • in a local machine with access to a browser, simply run snakeviz p.prof
  • if we're working remotely, use the following procedure:
    • user@remote: snakeviz --server p.prof
    • user@local: ssh -Y -N -f -L localhost:8080:localhost:8080 user@remote
    • in the local browser, paste the link printed by snakeviz on the remote cluster (e.g. http://127.0.0.1:8080/snakeviz/%2Fwork%2Fgallim%2Fdevel%2FUsefulHEPScripts%2Fprofiling%2Fp.prof)
• line_profiler
  • add the decorator @profile to the functions we want to profile
  • run kernprof -l -v code_profiling_main.py --dim 1000 --loops 4

Memory Usage

• memory_profiler
  • decorate functions with @profile
  • run python -m memory_profiler code_profiling_main.py --dim 1000 --loops 4 (see here for details )
• scalene
  • references: github repo and tutorial
  • scalene --outfile output.txt code_profiling_main.py --dim 1000 --loops 4

IPython/Jupyter

See here for IPython magic commands and here for Snakeviz.