Deep Xi: A Deep Learning Approach to A Priori SNR Estimation. Used for Speech Enhancement and Robust ASR.

Deep Xi (where the Greek letter 'xi' or ξ is pronounced /zaɪ/) is a deep learning approach to a priori SNR estimation that was proposed in [1] and is implemented in TensorFlow. Some of its use cases include:

• Minimum mean-square error (MMSE) approaches to speech enhancement like the MMSE short-time spectral amplitude (MMSE-STSA) estimator, the MMSE log-spectral amplitude (MMSE-LSA) estimator, and the Wiener filter (WF) approach.
• Estimate the ideal ratio mask (IRM) and the ideal binary mask (IBM).
• A front-end for robust ASR, as shown in Figure 1.

Figure 1: Deep Xi used as a front-end for robust ASR. The back-end (Deep Speech) is available here. The noisy speech magnitude spectrogram, as shown in (a), is a mixture of clean speech with voice babble noise at an SNR level of -5 dB, and is the input to Deep Xi. Deep Xi estimates the a priori SNR, as shown in (b). The a priori SNR estimate is used to compute an MMSE approach gain function, which is multiplied elementwise with the noisy speech magnitude spectrum to produce the clean speech magnitude spectrum estimate, as shown in (c). MFCCs are computed from the estimated clean speech magnitude spectrogram, producing the estimated clean speech cepstrogram, as shown in (d). The back-end system, Deep Speech, computes the hypothesis transcript, from the estimated clean speech cepstrogram, as shown in (e).

Current Models

The ResLSTM and ResBLSTM networks used for Deep Xi in [1] (Deep Xi - ResLSTM and Deep Xi - ResBSLTM) have been replaced with a residual network (ResNet) that employs causal dilated convolutional units, otherwise known as a temporal convolutional network (TCN). Deep Xi - ResNet can be seen in Figure 2. It comprises of 2 million parameters.

Figure 2: TCN a priori SNR estimator. (NOTE: the filter size for the first conv unit should be 64).

Availability

A trained network, version 3a, can be found in the ./model directory.

Installation

Prerequisites for GPU usage:

• CUDA 10.0
• cuDNN (>= 7.4.1)

To install:

1. git clone https://github.com/anicolson/DeepXi.git
2. virtualenv --system-site-packages -p python3 ~/venv/DeepXi
3. source ~/venv/DeepXi/bin/activate
4. pip install --upgrade tensorflow-gpu
5. cd DeepXi
6. pip install -r requirements.txt

If a GPU is not going to be used, replace step 4 with: pip install --upgrade tensorflow

How to Use the Deep Xi Scripts

Inference:

python3 deepxi.py --infer 1 --out_type y --gain mmse-lsa --gpu 0

y for --out_type specifies enhanced speech .wav output. mmse-lsa specifies the used gain function (others include mmse-stsa, wf, irm, ibm, srwf, cwf).

Training:

python3 deepxi.py --train 1 --verbose 1 --gpu 0

Retraining:

python3 deepxi.py --train 1 --cont 1 --epoch 175 --verbose 1 --gpu 0

Other options can be found in the deepxi.py script.

Datasets Used For Training

The .wav files used for training are single-channel, with a sampling frequency of 16 kHz.

The following speech datasets were used:

• The train-clean-100 set from Librispeech corpus, which can be found here.
• The CSTR VCTK corpus, which can be found here.
• The si and sx training sets from the TIMIT corpus, which can be found here (not open source).

The following noise datasets were used:

• The QUT-NOISE dataset, which can be found here.
• The Nonspeech dataset, which can be found here.
• The Environemental Background Noise dataset, which can be found here.
• The noise set from the MUSAN corpus, which can be found here.
• Multiple packs from the FreeSound website, which can be found here

Citation

If you find this repository helpful in your resarch or work, please use the following citation:

@article{NICOLSON201944,
title = "Deep learning for minimum mean-square error approaches to speech enhancement",
journal = "Speech Communication",
volume = "111",
pages = "44 - 55",
year = "2019",
issn = "0167-6393",
doi = "https://doi.org/10.1016/j.specom.2019.06.002",
url = "http://www.sciencedirect.com/science/article/pii/S0167639318304308",
author = "Aaron Nicolson and Kuldip K. Paliwal"
}

References

Please cite the following when using Deep Xi:

[1] A. Nicolson, K. K. Paliwal, Deep learning for minimum mean-square error approaches to speech enhancement, Speech Communication 111 (2019) 44 - 55, https://doi.org/10.1016/j.specom.2019.06.002.