Skip to content

Commit

Permalink
Update TensorFlow pipeline.
Browse files Browse the repository at this point in the history
* Refactor the training input pipeline.

1. Introduce a binary format (v2).
2. Support for loading v2 format off disk (much faster to parse)
3. Use the v2 format for the shuffle buffer (shrinks it by ~4x)
4. Amend test to be full end-to-end test, including worker interfaces.
5. Add benchmark for the TensorFlow frontend.
6. Change worker->main process communication to be fixed size binary format
instead of protobuf.

Net result is that the pipeline now saturates at about 45,000 moves/sec
into TensorFlow (aka ~88 batches/sec with n=512)

Pull request leela-zero#917.
  • Loading branch information
ywrt authored and gcp committed Feb 28, 2018
1 parent f871c2e commit 800990c
Showing 1 changed file with 320 additions and 102 deletions.
Loading

0 comments on commit 800990c

Please sign in to comment.