Skip to content

Commit

Permalink
[Example] Update README of transformers for multi-gpu support (dmlc#1435
Browse files Browse the repository at this point in the history
)

* Update README.md

* Update README.md
  • Loading branch information
mufeili authored Apr 9, 2020
1 parent e317f71 commit 88c3448
Showing 1 changed file with 4 additions and 12 deletions.
16 changes: 4 additions & 12 deletions examples/pytorch/transformer/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Transformer in DGL
In this example we implement the [Transformer](https://arxiv.org/pdf/1706.03762.pdf) and [Universal Transformer](https://arxiv.org/abs/1807.03819) with ACT in DGL.
In this example we implement the [Transformer](https://arxiv.org/pdf/1706.03762.pdf) with ACT in DGL.

The folder contains training module and inferencing module (beam decoder) for Transformer and training module for Universal Transformer
The folder contains training module and inferencing module (beam decoder) for Transformer.

## Dependencies

Expand All @@ -18,6 +18,8 @@ The folder contains training module and inferencing module (beam decoder) for Tr
python3 translation_train.py [--gpus id1,id2,...] [--N #layers] [--dataset DATASET] [--batch BATCHSIZE] [--universal]
```
By specifying multiple gpu ids separated by comma, we will employ multi-gpu training with multiprocessing.
- For evaluating BLEU score on test set(by enabling `--print` to see translated text):
```
Expand All @@ -28,19 +30,9 @@ Available datasets: `copy`, `sort`, `wmt14`, `multi30k`(default).
## Test Results
### Transformer
- Multi30k: we achieve BLEU score 35.41 with default setting on Multi30k dataset, without using pre-trained embeddings. (if we set the number of layers to 2, the BLEU score could reach 36.45).
- WMT14: work in progress
### Universal Transformer
- work in progress
## Notes
- Currently we do not support Multi-GPU training(this will be fixed soon), you should only specify only one gpu\_id when running the training script.
## Reference
- [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
Expand Down

0 comments on commit 88c3448

Please sign in to comment.