TT-NN is python & C++ Neural Network OP library.
| Model | Batch | End-to-end throughput | Device throughput [1] | Target |
|---|---|---|---|---|
| ResNet-50 (fps) | 20 | 2,070 | 7,200 | 10,000 |
| BERT-Large (sen/s) | 12 | 362 | 406 | 410 |
| Falcon7B-decode (t/s) | 32 | 135 | 135 | 140 |
| U-Net | coming soon | |||
| T5 small | coming soon | |||
| Bloom | coming soon |
[1] - Throughput on device ignores the overhead of host runtime, which is being currently optimized.
| Model | Batch | End-to-end throughput | Device throughput [1] | Target |
|---|---|---|---|---|
| Falcon-7B-decode (t/s/u) | 32 | 6.6 | 11.6 | 14 |
| Mistral-7B-decode (t/s/u) | 32 | 3.3 | 12.6 | 14 |
| Mamba-2.8B-decode (t/s/u) | 32 | coming soon | 17 | |
| Stable Diffusion 1.4 512x512 | 1 | coming soon |
| Model | Batch | Throughput |
|---|---|---|
| Falcon40B | coming soon | |
| LLaMA-2-70B | coming soon | |
| Mixtral7Bx8 | coming soon |
import ttnn
import torch
with ttnn.manage_device(device_id=0) as device:
a = torch.ones((5, 7))
b = torch.ones((1, 7))
a = ttnn.from_torch(a, device=device, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT)
b = ttnn.from_torch(b, device=device, dtype=ttnn.bfloat16, layout=ttnn.TILE_LAYOUT)
output = a + b
output = ttnn.to_torch(output)
print(output)
TT-Metalium is our low-level programming model, enabling kernel development for Tenstorrent hardware.
- Installing
- Getting started
- Documentation
- Troubleshooting and debugging tips
- Contributing
- Communication
Table of Contents generated with markdown-toc.
Note: Currently, all features are only fully tested on Grayskull E150 accelerators. We are currently working on functionality for other Tenstorrent architectures.
To find through all necessary instructions for setting up your Tenstorrent accelerator and this software, please refer to our full installation instructions.
You should look ahead to Getting started to further use this project.
If you just came reading from building from source, you can read ahead to running an example.
Otherwise, you must set up the necessary environment variables to use this project every time:
export ARCH_NAME=<arch name>
export TT_METAL_HOME=<appropriate value based on installation method above>
where <arch name> is your target, which could be:
grayskullwormhole_b0
etc...
If you're setting up an environment from source, you must further set up and activate the environment with:
export PYTHONPATH=<this repo dir>
export TT_METAL_ENV=dev
source build/python_env/bin/activate
After installing, please refer to our Getting Started page in our documentation.
Note that example programs are only available through source installation at this time.
Please refer to our documentation:
In addition to our documentation above, you can check out relevant sections in the contribution standards if you ever need hardware troubleshooting help or debugging tips.
We are excited to move our development to the public, open-source domain. However, we are not adequately staffed to review contributions in an expedient and manageable time frame at this time. In the meantime, please review the contributor's guide for more information about contribution standards.
If you would like to contribute, your submissions must pass post-commit regressions. If you would like more information on running tests locally and CI, please refer to the relevant section in the the contributor's guide and read it in its entirety.
Announcements from the Tenstorrent team regarding this project will be in the discussions page.
We also have a Discord channel that you can join. You may discuss with other members of the community and developers there. You may use this invite link. If you would like to formally propose a new feature, report a bug, or have issues with permissions, please file through GitHub issues.