Chia Proof of Space

With this fork of chiapos I try to add some threads to improve plot creation time.

Important: the main changes are done for enabled bitmap plotting.

For now sorting is done with threads. Here it is an example of how it improved with thread when no IO limits:

size: 0.11GiB, threads: 2, times:  read:1.112s, total: 1.998s 
size: 0.141GiB, threads: 3, times:  read:0.494s, total: 1.033s 
size: 0.141GiB, threads: 4, times:  read:0.346s, total: 0.758s 
size: 0.141GiB, threads: 5, times:  read:0.330s, total: 0.690s 
size: 0.141GiB, threads: 6, times:  read:0.261s, total: 0.575s 
size: 0.141GiB, threads: 7, times:  read:0.239s, total: 0.536s 
size: 0.141GiB, threads: 8, times:  read:0.269s, total: 0.512s 
size: 0.141GiB, threads: 9, times:  read:0.240s, total: 0.504s 
size: 0.141GiB, threads: 10, times: read:0.203s, total: 0.463s

In case of enough RAM next bucket presorted in parallel.

Threads added to phase2 and computational stage 2 of phase 3.

Bitmap of table 7 is calculated on write of the table and rewriting of table 7 is done on read of it in phase 3, than no need of table 7 stage in phase 2.

In addition added IO compaction that can lead to faster creation with HDD plotting or prolonge SSD life with less writtings. But compaction do some more work than on slow CPUs or in systems where IO is not a botleneck creation will be slower. To use all implemented compactions buckets number should be at least 256. The IO compaction allows less ram to plot on ramdisk. By my measures to create k32 it needed around 170GiB space in temp directory when using compaction with 256 (or more) buckets.

Free buffer can be used as write caching that allows to write less. For my measures for k32 each 1GiB of ram up to 22Gib saves around 16GiB of writtings and 25GiB buffers saves ~45% of writtings, 46GiB buffer saves ~63% of writtings and 110GiB buffer saves ~83% of writtings ( all this for 256 bukets and compaction enabled ).

Known issue with using buffer as cache is a memory fragmentation.
It is almost fixed when creating one plot at a time, but if python 
calls the plotter many times the memory usage can be increased 
drastically (2-3 times) for second and more plots.
This is the reason it is disabled by default.
It is possible to enable cache also by prefixing temp dir with :CACHE:
i.e. providing "-t :CACHE:/my/temp/dir" parameter

I tested ram plotting on amazon R6i.8xlarge instance that is 256GiB RAM and 30 vcores, the time of creation k32 plot is 33 minutes.

How to use? Get it from github

git clone https://github.com/vladimirttt123/chiaposht.git

Compile as original one. Then you have ProofOfSpace that not very usefull, but you can replace original chiapos with this one in installed chia. In order to do so you need replace file chiapos.cpython-310-x86_64-linux-gnu.so (the name depends on your python version) in chia installation sub dir venv/lib/python3.10/site-packages in addtion add there 2 more files: libfse.a and libuint128.a. Now when you run plots as regular your will run by this plotter.

Chia's proof of space is written in C++. Includes a plotter, prover, and verifier. It exclusively runs on 64 bit architectures. Read the Proof of Space document to learn about what proof of space is and how it works.

C++ Usage Instructions

Compile

# Requires cmake 3.14+

mkdir -p build && cd build
cmake ../
cmake --build . -- -j 6

Static Compilation With glibc

Statically compile ProofOfSpace

mkdir -p build && cd build
cmake -DBUILD_PROOF_OF_SPACE_STATICALLY=ON ../
cmake --build . -- -j 6

Run tests

./RunTests

CLI usage

./ProofOfSpace -k 25 -f "plot.dat" -m "0x1234" create
./ProofOfSpace -k 25 -f "final-plot.dat" -m "0x4567" -t TMPDIR -2 SECOND_TMPDIR create
./ProofOfSpace -f "plot.dat" prove <32 byte hex challenge>
./ProofOfSpace -k 25 verify <hex proof> <32 byte hex challenge>
./ProofOfSpace -f "plot.dat" check <iterations>

Benchmark

time ./ProofOfSpace -k 25 create

Hellman Attacks usage

There is an experimental implementation which implements some of the Hellman Attacks that can provide significant space savings for the final file.

./HellmanAttacks -k 18 -f "plot.dat" -m "0x1234" create
./HellmanAttacks -f "plot.dat" check <iterations>

Python

Finally, python bindings are provided in the python-bindings directory.

Install

python3 -m venv .venv
. .venv/bin/activate
pip3 install .

Run python tests

Testings uses pytest. Linting uses flake8 and mypy.

py.test ./tests -s -v

ci Building

The primary build process for this repository is to use GitHub Actions to build binary wheels for MacOS, Linux (x64 and aarch64), and Windows and publish them with a source wheel on PyPi. See .github/workflows/build.yml. CMake uses FetchContent to download pybind11. Building is then managed by cibuildwheel. Further installation is then available via pip install chiapos e.g.

Contributing and workflow

Contributions are welcome and more details are available in chia-blockchain's CONTRIBUTING.md.

The main branch is usually the currently released latest version on PyPI. Note that at times chiapos will be ahead of the release version that chia-blockchain requires in it's main/release version in preparation for a new chia-blockchain release. Please branch or fork main and then create a pull request to the main branch. Linear merging is enforced on main and merging requires a completed review. PRs will kick off a GitHub actions ci build and analysis of chiapos at lgtm.com. Please make sure your build is passing and that it does not increase alerts at lgtm.