scholarly journals Adversarial joint training with self-attention mechanism for robust end-to-end speech recognition

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Lujun Li ◽  
Yikai Kang ◽  
Yuchen Shi ◽  
Ludwig Kürzinger ◽  
Tobias Watzel ◽  
...  
Keyword(s):  
Speech Recognition ◽  
Speech Enhancement ◽  
Input Sequence ◽  
Attention Mechanism ◽  
The Self ◽  
Generative Adversarial Networks ◽  
Noise Robustness ◽  
Front End ◽  
Joint Training ◽  
End To End

AbstractLately, the self-attention mechanism has marked a new milestone in the field of automatic speech recognition (ASR). Nevertheless, its performance is susceptible to environmental intrusions as the system predicts the next output symbol depending on the full input sequence and the previous predictions. A popular solution for this problem is adding an independent speech enhancement module as the front-end. Nonetheless, due to being trained separately from the ASR module, the independent enhancement front-end falls into the sub-optimum easily. Besides, the handcrafted loss function of the enhancement module tends to introduce unseen distortions, which even degrade the ASR performance. Inspired by the extensive applications of the generative adversarial networks (GANs) in speech enhancement and ASR tasks, we propose an adversarial joint training framework with the self-attention mechanism to boost the noise robustness of the ASR system. Generally, it consists of a self-attention speech enhancement GAN and a self-attention end-to-end ASR model. There are two advantages which are worth noting in this proposed framework. One is that it benefits from the advancement of both self-attention mechanism and GANs, while the other is that the discriminator of GAN plays the role of the global discriminant network in the stage of the adversarial joint training, which guides the enhancement front-end to capture more compatible structures for the subsequent ASR module and thereby offsets the limitation of the separate training and handcrafted loss functions. With the adversarial joint optimization, the proposed framework is expected to learn more robust representations suitable for the ASR task. We execute systematic experiments on the corpus AISHELL-1, and the experimental results show that on the artificial noisy test set, the proposed framework achieves the relative improvements of 66% compared to the ASR model trained by clean data solely, 35.1% compared to the speech enhancement and ASR scheme without joint training, and 5.3% compared to multi-condition training.

scholarly journals Light-weight Self-Attention Augmented Generative Adversarial Networks for Speech Enhancement

Electronics ◽  
2021 ◽  
Vol 10 (13) ◽  
pp. 1586
Author(s):  
Lujun Li ◽  
Zhenxing Lu ◽  
Tobias Watzel ◽  
Ludwig Kürzinger ◽  
Gerhard Rigoll
Keyword(s):  
Neural Networks ◽  
Performance Improvement ◽  
Long Range ◽  
Speech Enhancement ◽  
Evaluation Criteria ◽  
Input Sequence ◽  
The Self ◽  
Generative Adversarial Networks ◽  
Computationally Efficient ◽  
Adversarial Networks

Generative adversarial networks (GANs) have shown their superiority for speech enhancement. Nevertheless, most previous attempts had convolutional layers as the backbone, which may obscure long-range dependencies across an input sequence due to the convolution operator’s local receptive field. One popular solution is substituting recurrent neural networks (RNNs) for convolutional neural networks, but RNNs are computationally inefficient, caused by the unparallelization of their temporal iterations. To circumvent this limitation, we propose an end-to-end system for speech enhancement by applying the self-attention mechanism to GANs. We aim to achieve a system that is flexible in modeling both long-range and local interactions and can be computationally efficient at the same time. Our work is implemented in three phases: firstly, we apply the stand-alone self-attention layer in speech enhancement GANs. Secondly, we employ locality modeling on the stand-alone self-attention layer. Lastly, we investigate the functionality of the self-attention augmented convolutional speech enhancement GANs. Systematic experiment results indicate that equipped with the stand-alone self-attention layer, the system outperforms baseline systems across classic evaluation criteria with up to 95 % fewer parameters. Moreover, locality modeling can be a parameter-free approach for further performance improvement, and self-attention augmentation also overtakes all baseline systems with acceptably increased parameters.

scholarly journals Gated Recurrent Fusion With Joint Training Framework for Robust End-to-End Speech Recognition

2021 ◽  
Vol 29 ◽  
pp. 198-209
Author(s):  
Cunhang Fan ◽  
Jiangyan Yi ◽  
Jianhua Tao ◽  
Zhengkun Tian ◽  
Bin Liu ◽  
...  
Keyword(s):  
Speech Recognition ◽  
Joint Training ◽  
End To End

Speech Enhancement Using End-to-End Speech Recognition Objectives

2019 ◽  
Author(s):  
Aswin Shanmugam Subramanian ◽  
Xiaofei Wang ◽  
Murali Karthick Baskar ◽  
Shinji Watanabe ◽  
Toru Taniguchi ◽  
...  
Keyword(s):  
Speech Recognition ◽  
Speech Enhancement ◽  
End To End

scholarly journals Improving Hybrid CTC/Attention Architecture with Time-Restricted Self-Attention CTC for End-to-End Speech Recognition

2019 ◽  
Vol 9 (21) ◽  
pp. 4639 ◽  
Author(s):  
Long Wu ◽  
Ta Li ◽  
Li Wang ◽  
Yonghong Yan
Keyword(s):  
Speech Recognition ◽  
Proper Time ◽  
Window Size ◽  
Attention Mechanism ◽  
Wall Street ◽  
Word Error Rate ◽  
Perceptual Evaluation ◽  
Left And Right ◽  
End To End ◽  
Multiparty Interaction

As demonstrated in hybrid connectionist temporal classification (CTC)/Attention architecture, joint training with a CTC objective is very effective to solve the misalignment problem existing in the attention-based end-to-end automatic speech recognition (ASR) framework. However, the CTC output relies only on the current input, which leads to the hard alignment issue. To address this problem, this paper proposes the time-restricted attention CTC/Attention architecture, which integrates an attention mechanism with the CTC branch. “Time-restricted” means that the attention mechanism is conducted on a limited window of frames to the left and right. In this study, we first explore time-restricted location-aware attention CTC/Attention, establishing the proper time-restricted attention window size. Inspired by the success of self-attention in machine translation, we further introduce the time-restricted self-attention CTC/Attention that can better model the long-range dependencies among the frames. Experiments with wall street journal (WSJ), augmented multiparty interaction (AMI), and switchboard (SWBD) tasks demonstrate the effectiveness of the proposed time-restricted self-attention CTC/Attention. Finally, to explore the robustness of this method to noise and reverberation, we join a train neural beamformer frontend with the time-restricted attention CTC/Attention ASR backend in the CHIME-4 dataset. The reduction of word error rate (WER) and the increase of perceptual evaluation of speech quality (PESQ) approve the effectiveness of this framework.

scholarly journals A new joint CTC-attention-based speech recognition model with multi-level multi-head attention

2019 ◽  
Vol 2019 (1) ◽  
Author(s):  
Chu-Xiong Qin ◽  
Wen-Lin Zhang ◽  
Dan Qu
Keyword(s):  
Speech Recognition ◽  
Nonnegative Matrix Factorization ◽  
State Of The Art ◽  
Nonnegative Matrix ◽  
Attention Mechanism ◽  
Word Error Rate ◽  
Absolute Value ◽  
Multi Level ◽  
End To End ◽  
High Level

Abstract A method called joint connectionist temporal classification (CTC)-attention-based speech recognition has recently received increasing focus and has achieved impressive performance. A hybrid end-to-end architecture that adds an extra CTC loss to the attention-based model could force extra restrictions on alignments. To explore better the end-to-end models, we propose improvements to the feature extraction and attention mechanism. First, we introduce a joint model trained with nonnegative matrix factorization (NMF)-based high-level features. Then, we put forward a hybrid attention mechanism by incorporating multi-head attentions and calculating attention scores over multi-level outputs. Experiments on TIMIT indicate that the new method achieves state-of-the-art performance with our best model. Experiments on WSJ show that our method exhibits a word error rate (WER) that is only 0.2% worse in absolute value than the best referenced method, which is trained on a much larger dataset, and it beats all present end-to-end methods. Further experiments on LibriSpeech show that our method is also comparable to the state-of-the-art end-to-end system in WER.

Sign in / Sign up

Export Citation Format

Share Document