Initial commit

narwhal/.gitignore

@@ -0,0 +1,25 @@
+*.json
+.DS_Store
+.storage_*
+.*storage*
+.db_test*
+.venv/
+.idea/*
+rust/.idea/*
+
+target/
+!/**/src/**/target/
+.test_*
+
+# Generated by Cargo
+# will have compiled files and executables
+/target/
+
+# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
+# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
+Cargo.lock
+workspace.xml
+
+# These are backup files generated by rustfmt
+**/*.rs.bk
+scripts/build_and_fab

narwhal/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing to Narwhal and Tusk
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to Narwhal and Tusk, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.

narwhal/README.md

@@ -0,0 +1,74 @@
+# Narwhal and Tusk
+
+[![rustc](https://img.shields.io/badge/rustc-1.48+-blue?style=flat-square&logo=rust)](https://www.rust-lang.org)
+[![license](https://img.shields.io/badge/license-Apache-blue.svg?style=flat-square)](LICENSE)
+
+This repo contains a prototype of Narwhal and Tusk. It supplements the paper [Narwhal and Tusk: A DAG-based Mempool and Efficient BFT Consensus](https://arxiv.org/pdf/2105.11827.pdf).
+
+## Overview
+We propose separating the task of transaction dissemination from transaction ordering, to enable high-performance
+Byzantine fault-tolerant consensus in a permissioned setting. To this end, we design and evaluate a mempool protocol,
+Narwhal, specializing in high-throughput reliable dissemination and storage of causal histories of transactions. Narwhal
+tolerates an asynchronous network and maintains its performance despite failures. We demonstrate that composing
+Narwhal with a partially synchronous consensus protocol (HotStuff) yields significantly better throughput even in the
+presence of faults. However, loss of liveness during view-changes can result in high latency. To achieve overall good
+performance when faults occur we propose Tusk, a zero-message overhead asynchronous consensus protocol embedded within Narwhal. We demonstrate its high performance under a variety of configurations and faults. Further, Narwhal is designed to easily scale-out using multiple workers at each validator, and we demonstrate that there is no foreseeable limit to the throughput we can achieve for consensus,
+with a few seconds latency.
+
+As a summary of results, on a Wide Area Network (WAN), Hotstuff over Narwhal achieves 170,000 tx/sec with a 2.5-sec
+latency instead of 1,800 tx/sec with 1-sec latency of Hotstuff. 
+Additional workers increase throughput linearly to 600,000
+tx/sec without any latency increase. Tusk achieves 140,000
+tx/sec with 4 seconds latency or 20x better than the state-of-the-art asynchronous protocol. Under faults, both Narwhal
+based protocols maintain high throughput, but the HotStuff
+variant suffers from slightly higher latency.
+
+## Getting Started
+The core protocols are written in Rust, but all benchmarking scripts are written in Python and run with [Fabric](http://www.fabfile.org/).
+To deploy and benchmark a testbed of 4 nodes on your local machine, clone the repo and compile it in release mode:
+```
+$ git clone https://github.com/facebookresearch/narwhal.git
+$ cd rust
+$ cargo build --release
+```
+Then install the Python dependencies:
+```
+$ cd ../scripts
+$ pip install -r requirements.txt
+```
+You also need to install Clang (required by rocksdb) and [tmux](https://linuxize.com/post/getting-started-with-tmux/#installing-tmux) (which runs all nodes and clients in the background). Finally, run a local benchmark using fabric:
+```
+$ fab local
+```
+This command may take a long time the first time you run it (compiling rust code in `release` mode may be slow) and you can customize a number of benchmark parameters in `fabfile.py`. When the benchmark terminates, it displays a summary of the execution similarly to the one below.
+```
+-----------------------------------------
+ SUMMARY:
+-----------------------------------------
+ Committee size: 4 nodes
+ Number of workers: 1 worker(s) per node
+ Faults: 0 nodes
+ Transaction size: 512 B
+ Max batch size: 1,000 txs
+ Transaction rate: 60,000 tx/s
+
+ Dag Results:
+ + Total certified bytes: 799,468,544 B
+ + Execution time: 29,646 ms
+ + Estimated BPS: 26,967,619 B/s
+ + Estimated TPS: 52,671 txs/s
+ + Block Latency: 6 ms
+ + Client Latency: 93 ms
+
+ Consensus Results:
+ + Total committed bytes: 786,986,496 B
+ + Execution time: 29,542 ms
+ + Estimated BPS: 26,639,130 B/s
+ + Estimated TPS: 52,030 txs/s
+ + Block Latency: 395 ms
+ + Client Latency: 482 ms
+-----------------------------------------
+```
+
+## License
+This software is licensed as [Apache 2.0](LICENSE).

narwhal/scripts/.gitignore

@@ -0,0 +1,138 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/