Skip to content

omarathon/compression-geospatial

Repository files navigation

Compressive Streaming for Geospatial Pipelines

Beating main-memory bandwidths in geospatial pipelines with fast in-memory compression

MRes Project

Author: Omar Tanner (omsst2)

Report: here

Files

codecs/*: core codecs implementing the codecs/generic_codecs.h inteface

Main programs:

  • test_comp.cpp: test codecs
  • bench_comp.cpp: benchmark codecs
  • bench_pipeline.cpp: benchmark geospatial pipelines

Additional files:

  • codec_collection.h: bundled codecs
  • test_remappings.cpp: verifies Morton remappings
  • util.h, transformations.h, remappings.h: CPP utilities
  • py/*: Python utilities
  • sh/*: Shell performance-monitoring utilities

Setup (Basic)

We assume a Linux environment.

  1. install these packages with apt-get (you might need more, debug appropriately)
    1. g++, g++-11, libgdal-dev, python3-gdal, liblz4-dev, libzstd-dev, zlib1g-dev, liblzma-dev
  2. obtain the submodules in external and build them
  3. make

Setup (Fusing Summing into Decompression)

  1. re-build external/FastPFor and external/simdcomp from these forks:
    1. FastPFor: https://github.com/omarathon/FastPFor
    2. simdcomp: https://github.com/omarathon/simdcomp
  2. replace codecs/custom_vec_logic_codecs.h with agg/custom_vec_logic_codecs.h (the new file contains the modification to the custom_rle_vecavx512 codec which fuses summing into decompression)
  3. replace bench_pipeline.cpp with agg/bench_pipeline.cpp
  4. make clean && make

Setup (Running on HPC)

  1. source hpc/modules.sh
  2. replace Makefile with hpc/Makefile (the new Makefile contains compiler modifications for the HPC)
  3. make

Miscellanea

Licence:

  • MIT for all files in codecs, except the TurboPFor wrapper (codecs/turbopfor_codecs.h), LZ4 wrapper (codecs/lz4_codecs.h) and 2ibench wrapper (codecs/2ibench_codecs.h) which are GPL.
  • GPL for everything else.

Full repo/data/report on request

About

Compressive streaming for geospatial pipelines.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published