Skip to content

Commit

Permalink
update path-related notes in submission guideline
Browse files Browse the repository at this point in the history
  • Loading branch information
dophist committed Aug 11, 2021
1 parent 74f7e72 commit a5f6039
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions HOW_TO_SUBMIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,19 +161,19 @@ then inside SBI code, SBI can always use `./assets/asr.{mdl,cfg}` to locate ASR
### 1.7 Debugging and validating your submission model dir
Once you have prepared submission model dir following above specification, you need to validate everything works as expected. Here is what you can do:

1. make sure you can find a `MINI` test set in your local testset-zoo, i.e. `{leaderboard_git_repo}/datasets/MINI`
1. make sure you can find a `MINI` test set in your local testset-zoo, i.e. `<leaderboard_git_repo>/datasets/MINI`
2. move prepared submission model to your local model-zoo
```
mv {prepared_submission_model_dir} {leaderboard_git_repo}/models/{model_id_that_you_chose}
mv <prepared_submission_model_dir> <leaderboard_git_repo>/models/{model_id_that_you_chose}
```
3. create a benchmark request(as described later in this guideline) under `{leaderboard_git_repo}/requests/mini_debug.yaml`
3. create a benchmark request(as described later in this guideline) under `<leaderboard_git_repo>/requests/mini_debug.yaml`
4. run a MINI benchmark:
```
cd {leaderboard_git_repo}
cd <leaderboard_git_repo>
./leaderboard_runner requests/mini_debug.yaml
```
5. check `{leaderboard_git_repo}/results/{CER,CHECK}`
5. check `<leaderboard_git_repo>/results/{CER,CHECK}`
This basically invokes a benchmarking pipeline locally on your machine, testing your model accuracy against MINI test set. If you can pass this validation, then congratulations, you have successfully made your ASR system reproducible, it's now safe to share and publish.
Expand All @@ -188,7 +188,7 @@ utils/install_oss.sh
2.2 Move prepared submission model dir into local model zoo
```
mv prepared_model_dir leaderboard/models/<model_id>
mv <prepared_submission_model_dir> <leaderboard_git_repo>/models/<model_id>
```
**`model_id`** is a unique identifier, used to refer to this model in future benchmarks.
Expand Down

0 comments on commit a5f6039

Please sign in to comment.