SpeechColab ASR leaderboard

1. Overview

"If you can’t measure it, you can’t improve it." -- Peter Drucker

SpeechIO leaderboard serves as an ASR benchmarking platform by providing 3 components:

1. TestSet Zoo: A collection of test sets covering wide range of speech recognition tasks & scenarios

2. Model Zoo: A collection of models including commercial APIs & open-sourced models

3. Benchmarking Pipeline: a simple & well-specified pipeline to take care of data preparation / recognition / post processing / error rate evaluation.

People should be able to easily benchmark, reproduce, examine ASR systems from each other

---

2. TestSet Zoo: datasets/*

Academic Test Sets (EN & ZH)

已公开 UNLOCKED	编号 TEST_SET_ID	说明 DESCRIPTION	语言 LANGUAGE
✓	AISHELL1_TEST	test set of AISHELL-1	zh
✓	AISHELL2_IOS_TEST	test set of AISHELL-2 (iOS channel)	zh
✓	AISHELL2_ANDROID_TEST	test set of AISHELL-2 (Android channel)	zh
✓	AISHELL2_MIC_TEST	test set of AISHELL-2 (Microphone channel)	zh
✓	LIBRISPEECH_TEST_CLEAN	"test_clean" set of LibriSpeech	en
✓	LIBRISPEECH_TEST_OTHER	"test_other" set of LibriSpeech	en
✓	GIGASPEECH_V1.0.0_DEV	dev set of GigaSpeech	en
✓	GIGASPEECH_V1.0.0_TEST	test set of GigaSpeech	en

SpeechIO Test Sets (ZH)

SpeechIO test sets are carefully curated by SpeechIO authors, crawled from publicly available sources (Youtube, TV programs, Podcast etc), covering various well-known scenarios and topics, transcribed by payed professional annotators.

已公开 UNLOCKED	编号 TEST_SET_ID	名称 NAME	场景 SCENARIO	内容领域 TOPIC	时长 HOURS	难度(1-5) DIFFICULTY
✓	SPEECHIO_ASR_ZH00000	调试集 for debugging	视频会议、论坛演讲 conference & speech	经济、货币、金融 economy, currency, finance	1.0	★★☆
✓	SPEECHIO_ASR_ZH00001	新闻联播	新闻播报 TV News	时政 news & politics	9	★
✓	SPEECHIO_ASR_ZH00002	鲁豫有约	访谈电视节目 TV interview	名人工作/生活 celebrity & film & music & daily	3	★★☆
✓	SPEECHIO_ASR_ZH00003	天下足球	专题电视节目 TV program	足球 Sports & Football & Worldcup	2.7	★★☆
✓	SPEECHIO_ASR_ZH00004	罗振宇跨年演讲	会场演讲 Stadium Public Speech	社会、人文、商业 Society & Culture & Business Trend	2.7	★★
✓	SPEECHIO_ASR_ZH00005	李永乐讲堂	在线教育 Online Education	科普 Popular Science	4.4	★★★
✓	SPEECHIO_ASR_ZH00006	王者荣耀 张大仙 & 骚白	直播 Live Broadcasting	游戏 Game	1.6	★★★☆
✓	SPEECHIO_ASR_ZH00007	直播带货 李佳琪 & 薇娅	直播 Live Broadcasting	电商、美妆 Makeup & Online shopping/advertising	0.9	★★★★☆
✓	SPEECHIO_ASR_ZH00008	老罗语录	线下培训 Offline lecture	段子、做人 Life & Purpose & Ethics	1.3	★★★★☆
✓	SPEECHIO_ASR_ZH00009	故事FM	播客 Podcast	人生故事、见闻 Ordinary Life Story Telling	4.5	★★☆
✓	SPEECHIO_ASR_ZH00010	创业内幕	播客 Podcast	创业、产品、投资 Startup & Enterprenuer & Product & Investment	4.2	★★☆
✓	SPEECHIO_ASR_ZH00011	罗翔刑法法考	在线教育 Online Education	法律 法考 Law & Lawyer Qualification Exams	3.4	★★☆
✓	SPEECHIO_ASR_ZH00012	张雪峰考研	在线教育 Online Education	考研 高校报考 University & Graduate School Entrance Exams	3.4	★★★☆
✓	SPEECHIO_ASR_ZH00013	谷阿莫 牛叔说电影	短视频 VLog	电影剪辑 Movie Cuts	1.8	★★★
✓	SPEECHIO_ASR_ZH00014	贫穷料理 琼斯爱生活	短视频 VLog	美食、烹饪 Food & Cooking & Gourmet	1	★★★☆
✓	SPEECHIO_ASR_ZH00015	单田芳 白眉大侠	评书 Traditional Podcast	江湖、武侠 Kongfu Fiction	2.2	★★☆
✗	SPEECHIO_ASR_ZH00016	德云社演出	剧场相声 Theater Crosstalk Show	包袱段子 Funny Stories	1	★★★
✗	SPEECHIO_ASR_ZH00017	吐槽大会	脱口秀电视节目 Standup Comedy	明星糗事 Celebrity Jokes	1.8	★★☆
✗	SPEECHIO_ASR_ZH00018	小猪佩奇 熊出没	少儿动画 Children Cartoon	童话故事、日常 Fairy Tale	0.9	★☆
✗	SPEECHIO_ASR_ZH00019	CCTV5 NBA 转播	体育赛事解说 Sports Game Live	篮球、NBA NBA Game	0.7	★★★
✗	SPEECHIO_ASR_ZH00020	篮球人物	纪录片 Documentary	篮球明星、成长 NBA Super Stars' Life & History	2.2	★★
✗	SPEECHIO_ASR_ZH00021	汽车之家评测	短视频 VLog	汽车测评 Car benchmarks, Road driving test	1.7	★★★☆
✗	SPEECHIO_ASR_ZH00022	小艾大叔 豪宅带看	短视频 VLog	房地产、豪宅 Realestate, Mansion tour	1.7	★★★
✗	SPEECHIO_ASR_ZH00023	无聊开箱 Zealer评测	短视频 VLog	产品开箱评测 Unboxing	2	★★★
✗	SPEECHIO_ASR_ZH00024	付老师种植技术	短视频 VLog	农业、种植 Agriculture, Planting	2.7	★★★☆
✗	SPEECHIO_ASR_ZH00025	石国鹏讲历史	线下培训 Offline lecture	历史，古希腊哲学 History, Greek philosophy	1.3	★★☆
✗	SPEECHIO_ASR_ZH00026	张震鬼故事	广播节目 Broadcasting Program	鬼故事 Horror Stories	2.4	★★★
✗	SPEECHIO_ASR_ZH00027	华语辩论世界杯	辩论赛 Debates Contest	兴趣、技能、成长 Hobby, Skill, Growth	1.4	★★★
✗	SPEECHIO_ASR_ZH00028	时政现场同传	同声传译 Simultaneous Translation	时政、社会公共治理 News & Events on Public Governance	2.1	★★★☆
✗	SPEECHIO_ASR_ZH00029	港台明星访谈 周杰伦,曾志伟 张家辉,陈小春 周星驰	口音(港台) HongKong/Taiwan Accents	娱乐、生活、演艺 Entertainment, Acting, Musics	1.5	★★★☆
✗	SPEECHIO_ASR_ZH00030	世界青年说	口音(老外) Foreigner Accents	异国文化比较 Cultural Difference	2	★★★☆

How to get a test set

To download an unlocked test set to your local test set zoo:

ops/pull dataset <TEST_SET_ID>

---

3. Model Zoo: models/*

There are two types of models:

• Cloud Models represent ASR systems backed with commercial cloud-services (e.g. Google Cloud, Azure)
• Local Models represent local ASR systems (e.g. open-source toolkits + pretrained models)

EN Cloud Models

编号 MODEL_ID	类型 TYPE	厂商 PROVIDER	简介 DESCRIPTION	链接 URL
aliyun_api_en	Cloud	阿里巴巴 Alibaba	阿里云 - 一句话识别	link
amazon_api_en	Cloud	亚马逊 Amazon	亚马逊云服务平台	link
baidu_api_en	Cloud	百度 Baidu	百度智能云	link
google_api_en	Cloud	谷歌 Google	谷歌云	link
microsoft_sdk_en	Cloud	微软 Microsoft	Azure	link
tencent_api_en	Cloud	腾讯 Tencent	腾讯云	link

EN Local Models

编号 MODEL_ID	类型 TYPE	作者 AUTHOR	简介 DESCRIPTION
vosk_model_en	Local	alphacephei	ASR solution from link
vosk_model_en_large	Local	alphacephei	ASR solution(large model) from link
deepspeech_model_en	Local	deepspeech	DeepSpeech pretrained Model link
coqui_model_en	Local	coqui	ASR solution from link
NeMo_conformer_en	Local	NeMo	Conformer pretrained model from NVidia's NeMo project link

ZH Cloud Models

编号 MODEL_ID	类型 TYPE	厂商 PROVIDER	简介 DESCRIPTION	链接 URL
aispeech_api_zh	Cloud	思必驰 AISpeech	思必驰开放平台	link
aliyun_api_zh	Cloud	阿里巴巴 Alibaba	阿里云 - 一句话识别	link
aliyun_ftasr_api_zh	Cloud	阿里巴巴 Alibaba	阿里云 - 文件识别(非流式)	link
baidu_pro_api_zh	Cloud	百度 Baidu	百度智能云 (极速版)	link
bilibili_api_zh	Cloud	哔哩哔哩 bilibili	哔哩哔哩AI开放平台	not available yet
hiasr_api_zh	Cloud	喜马拉雅 ximalaya	喜马拉雅AI开放平台 (转写,非流式)	not available yet
iflytek_lfasr_api_zh	Cloud	讯飞 IFlyTek	讯飞开放平台 (转写,非流式)	link
microsoft_sdk_zh	Cloud	微软 Microsoft	Azure	link
tencent_api_zh	Cloud	腾讯 Tencent	腾讯云	link
yitu_api_zh	Cloud	依图 YituTech	依图语音开放平台	link

ZH Local Models

编号 MODEL_ID	类型 TYPE	作者 AUTHOR	简介 DESCRIPTION
speechio_kaldi_multicn	Local	Xingyu NA(那兴宇)	Kaldi multi_cn recipe
wenet_multi_cn	Local	Binbin Zhang(张彬彬)@wenet-e2e	WeNet multi_cn recipe
vosk_model_cn	Local	alphacephei	Chinese engine of Vosk
wenet_wenetspeech	Local	Binbin Zhang(张彬彬)@wenet-e2e	WeNet wenetspeech recipe

How to get a model

• Cloud Models are normally light-weight client programs, small enough to store within this github repo
• Local Models normally contain large assets so they are stored in cloud, you can download a model via:

ops/pull model <MODEL_ID>

---

4. Benchmarking Pipeline

How to submit

To submit a new model, follow this pipeline specification. Existing models are good references as well.

How to run

With downloaded models & test sets, you can trigger the benchmarking pipeline on your local machine via:

ops/leaderboard_runner requests/request.yaml

where request.yaml specifies a <MODEL_ID> and a list of <TEST_SET_ID> (refer to examples in above specification)

---

5. Latest Leaderboard Results

Ranking on unlocked SpeechIO test sets (ZH00001 ~ ZH00015)

Rank 排名	Model 模型	CER 字错误率	Date 时间
1	hiasr_api_zh(*)	2.16 %	2022.08
2	yitu_api_zh	2.62 %	2022.08
3	tencent_api_zh	2.95%	2022.08
4	bilibili_api_zh(*)	2.95%	2022.08
5	aliyun_api_zh	3.02%	2022.08
6	microsoft_sdk_zh	3.03%	2022.08
7	aispeech_api_zh	3.39%	2022.08
8	iflytek_lfasr_api_zh	3.66%	2022.08
9	baidu_pro_api_zh	6.64%	2022.08

Ranking on all SpeechIO test sets

Rank 排名	Model 模型	CER 字错误率	Date 时间
1	hiasr_api_zh(*)	2.61 %	2022.08
2	yitu_api_zh	2.80 %	2022.08
3	bilibili_api_zh(*)	3.30 %	2022.08
4	tencent_api_zh	3.31%	2022.08
5	microsoft_sdk_zh	3.47%	2022.08
6	aispeech_api_zh	3.63%	2022.08
7	aliyun_api_zh	3.78%	2022.08
8	iflytek_lfasr_api_zh	4.01%	2022.08
9	baidu_pro_api_zh	7.38%	2022.08

note: models with (*) marker can be found in model zoo, but not universally available to public yet.

Details

---

Contacts

Email: [email protected]