scholarly journals A Fast Learning Method for Multilayer Perceptrons in Automatic Speech Recognition Systems

2015 ◽  
Vol 2015 ◽  
pp. 1-7 ◽  
Author(s):  
Chenghao Cai ◽  
Yanyan Xu ◽  
Dengfeng Ke ◽  
Kaile Su
Keyword(s):  
Speech Recognition ◽  
Training Data ◽  
Cross Entropy ◽  
Multilayer Perceptrons ◽  
Learning Method ◽  
Continuous Speech Recognition ◽  
Dynamic Learning ◽  
Entropy Loss ◽  
Fast Learning ◽  
The Cross

We propose a fast learning method for multilayer perceptrons (MLPs) on large vocabulary continuous speech recognition (LVCSR) tasks. A preadjusting strategy based on separation of training data and dynamic learning-rate with a cosine function is used to increase the accuracy of a stochastic initial MLP. Weight matrices of the preadjusted MLP are restructured by a method based on singular value decomposition (SVD), reducing the dimensionality of the MLP. A back propagation (BP) algorithm that fits the unfolded weight matrices is used to train the restructured MLP, reducing the time complexity of the learning process. Experimental results indicate that on LVCSR tasks, in comparison with the conventional learning method, this fast learning method can achieve a speedup of around 2.0 times with improvement on both the cross entropy loss and the frame accuracy. Moreover, it can achieve a speedup of approximately 3.5 times with only a little loss of the cross entropy loss and the frame accuracy. Since this method consumes less time and space than the conventional method, it is more suitable for robots which have limitations on hardware.

scholarly journals A Russian Keyword Spotting System Based on Large Vocabulary Continuous Speech Recognition and Linguistic Knowledge

2016 ◽  
Vol 2016 ◽  
pp. 1-9 ◽  
Author(s):  
Valentin Smirnov ◽  
Dmitry Ignatov ◽  
Michael Gusev ◽  
Mais Farkhadov ◽  
Natalia Rumyantseva ◽  
...  
Keyword(s):  
Speech Recognition ◽  
Real Life ◽  
Training Data ◽  
Linguistic Knowledge ◽  
Keyword Spotting ◽  
Continuous Speech ◽  
Continuous Speech Recognition ◽  
Large Vocabulary ◽  
Effective Combination ◽  
Industrial Use

The paper describes the key concepts of a word spotting system for Russian based on large vocabulary continuous speech recognition. Key algorithms and system settings are described, including the pronunciation variation algorithm, and the experimental results on the real-life telecom data are provided. The description of system architecture and the user interface is provided. The system is based on CMU Sphinx open-source speech recognition platform and on the linguistic models and algorithms developed by Speech Drive LLC. The effective combination of baseline statistic methods, real-world training data, and the intensive use of linguistic knowledge led to a quality result applicable to industrial use.

scholarly journals An Effective Learning Method for Automatic Speech Recognition in Korean CI Patients’ Speech

Electronics ◽  
2021 ◽  
Vol 10 (7) ◽  
pp. 807
Author(s):  
Jiho Jeong ◽  
S. I. M. M. Raton Mondol ◽  
Yeon Wook Kim ◽  
Sangmin Lee
Keyword(s):  
Speech Recognition ◽  
Cochlear Implant ◽  
Automatic Speech Recognition ◽  
Error Rate ◽  
Training Data ◽  
Learning Method ◽  
Test Dataset ◽  
Privacy Concerns ◽  
Effective Learning ◽  
Speech Test

The automatic speech recognition (ASR) model usually requires a large amount of training data to provide better results compared with the ASR models trained with a small amount of training data. It is difficult to apply the ASR model to non-standard speech such as that of cochlear implant (CI) patients, owing to privacy concerns or difficulty of access. In this paper, an effective finetuning and augmentation ASR model is proposed. Experiments compare the character error rate (CER) after training the ASR model with the basic and the proposed method. The proposed method achieved a CER of 36.03% on the CI patient’s speech test dataset using only 2 h and 30 min of training data, which is a 62% improvement over the basic method.

scholarly journals Personal Interest Attention Graph Neural Networks for Session-Based Recommendation

Entropy ◽  
2021 ◽  
Vol 23 (11) ◽  
pp. 1500
Author(s):  
Xiangde Zhang ◽  
Yuan Zhou ◽  
Jianping Wang ◽  
Xiaojun Lu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Objective Function ◽  
Cross Entropy ◽  
Personal Interest ◽  
Entropy Loss ◽  
Convolutional Networks ◽  
The Cross ◽  
Graph Neural Networks

Session-based recommendations aim to predict a user’s next click based on the user’s current and historical sessions, which can be applied to shopping websites and APPs. Existing session-based recommendation methods cannot accurately capture the complex transitions between items. In addition, some approaches compress sessions into a fixed representation vector without taking into account the user’s interest preferences at the current moment, thus limiting the accuracy of recommendations. Considering the diversity of items and users’ interests, a personalized interest attention graph neural network (PIA-GNN) is proposed for session-based recommendation. This approach utilizes personalized graph convolutional networks (PGNN) to capture complex transitions between items, invoking an interest-aware mechanism to activate users’ interest in different items adaptively. In addition, a self-attention layer is used to capture long-term dependencies between items when capturing users’ long-term preferences. In this paper, the cross-entropy loss is used as the objective function to train our model. We conduct rich experiments on two real datasets, and the results show that PIA-GNN outperforms existing personalized session-aware recommendation methods.

scholarly journals Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis

2020 ◽  
pp. 25-55
Author(s):  
Gabriel Zaid ◽  
Lilian Bossuet ◽  
François Dassance ◽  
Amaury Habrard ◽  
Alexandre Venelli
Keyword(s):  
Deep Learning ◽  
Success Rate ◽  
Loss Function ◽  
Estimation Error ◽  
Approximation Error ◽  
Cross Entropy ◽  
Side Channel ◽  
Entropy Loss ◽  
The Cross ◽  
Ranking Loss

The side-channel community recently investigated a new approach, based on deep learning, to significantly improve profiled attacks against embedded systems. Compared to template attacks, deep learning techniques can deal with protected implementations, such as masking or desynchronization, without substantial preprocessing. However, important issues are still open. One challenging problem is to adapt the methods classically used in the machine learning field (e.g. loss function, performance metrics) to the specific side-channel context in order to obtain optimal results. We propose a new loss function derived from the learning to rank approach that helps preventing approximation and estimation errors, induced by the classical cross-entropy loss. We theoretically demonstrate that this new function, called Ranking Loss (RkL), maximizes the success rate by minimizing the ranking error of the secret key in comparison with all other hypotheses. The resulting model converges towards the optimal distinguisher when considering the mutual information between the secret and the leakage. Consequently, the approximation error is prevented. Furthermore, the estimation error, induced by the cross-entropy, is reduced by up to 23%. When the ranking loss is used, the convergence towards the best solution is up to 23% faster than a model using the cross-entropy loss function. We validate our theoretical propositions on public datasets.

scholarly journals Domain-Adversarial Based Model with Phonological Knowledge for Cross-Lingual Speech Recognition

Electronics ◽  
2021 ◽  
Vol 10 (24) ◽  
pp. 3172
Author(s):  
Qingran Zhan ◽  
Xiang Xie ◽  
Chenguang Hu ◽  
Juan Zuluaga-Gomez ◽  
Jing Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Speech Recognition ◽  
Training Data ◽  
Target Language ◽  
Learning Method ◽  
Acoustic Features ◽  
Adversarial Learning ◽  
Phonological Knowledge ◽  
Cross Lingual

Phonological-based features (articulatory features, AFs) describe the movements of the vocal organ which are shared across languages. This paper investigates a domain-adversarial neural network (DANN) to extract reliable AFs, and different multi-stream techniques are used for cross-lingual speech recognition. First, a novel universal phonological attributes definition is proposed for Mandarin, English, German and French. Then a DANN-based AFs detector is trained using source languages (English, German and French). When doing the cross-lingual speech recognition, the AFs detectors are used to transfer the phonological knowledge from source languages (English, German and French) to the target language (Mandarin). Two multi-stream approaches are introduced to fuse the acoustic features and cross-lingual AFs. In addition, the monolingual AFs system (i.e., the AFs are directly extracted from the target language) is also investigated. Experiments show that the performance of the AFs detector can be improved by using convolutional neural networks (CNN) with a domain-adversarial learning method. The multi-head attention (MHA) based multi-stream can reach the best performance compared to the baseline, cross-lingual adaptation approach, and other approaches. More specifically, the MHA-mode with cross-lingual AFs yields significant improvements over monolingual AFs with the restriction of training data size and, which can be easily extended to other low-resource languages.

scholarly journals Cross Entropy of Neural Language Models at Infinity—A New Bound of the Entropy Rate

Entropy ◽  
2018 ◽  
Vol 20 (11) ◽  
pp. 839 ◽  
Author(s):  
Shuntaro Takahashi ◽  
Kumiko Tanaka-Ishii
Keyword(s):  
Natural Language ◽  
State Of The Art ◽  
Training Data ◽  
Language Models ◽  
Cross Entropy ◽  
Entropy Rate ◽  
Natural Language Text ◽  
Power Law Decay ◽  
The Cross ◽  
Two Parameters

Neural language models have drawn a lot of attention for their strong ability to predict natural language text. In this paper, we estimate the entropy rate of natural language with state-of-the-art neural language models. To obtain the estimate, we consider the cross entropy, a measure of the prediction accuracy of neural language models, under the theoretically ideal conditions that they are trained with an infinitely large dataset and receive an infinitely long context for prediction. We empirically verify that the effects of the two parameters, the training data size and context length, on the cross entropy consistently obey a power-law decay with a positive constant for two different state-of-the-art neural language models with different language datasets. Based on the verification, we obtained 1.12 bits per character for English by extrapolating the two parameters to infinity. This result suggests that the upper bound of the entropy rate of natural language is potentially smaller than the previously reported values.

scholarly journals Can Cross Entropy Loss Be Robust to Label Noise?

2020 ◽  
Author(s):  
Lei Feng ◽  
Senlin Shu ◽  
Zhuoyi Lin ◽  
Fengmao Lv ◽  
Li Li ◽  
...  
Keyword(s):  
Mean Squared Error ◽  
Absolute Error ◽  
Training Data ◽  
Cross Entropy ◽  
Loss Functions ◽  
Label Noise ◽  
Entropy Loss ◽  
Squared Error ◽  
Great Performance ◽  
Benchmark Datasets

Trained with the standard cross entropy loss, deep neural networks can achieve great performance on correctly labeled data. However, if the training data is corrupted with label noise, deep models tend to overfit the noisy labels, thereby achieving poor generation performance. To remedy this issue, several loss functions have been proposed and demonstrated to be robust to label noise. Although most of the robust loss functions stem from Categorical Cross Entropy (CCE) loss, they fail to embody the intrinsic relationships between CCE and other loss functions. In this paper, we propose a general framework dubbed Taylor cross entropy loss to train deep models in the presence of label noise. Specifically, our framework enables to weight the extent of fitting the training labels by controlling the order of Taylor Series for CCE, hence it can be robust to label noise. In addition, our framework clearly reveals the intrinsic relationships between CCE and other loss functions, such as Mean Absolute Error (MAE) and Mean Squared Error (MSE). Moreover, we present a detailed theoretical analysis to certify the robustness of this framework. Extensive experimental results on benchmark datasets demonstrate that our proposed approach significantly outperforms the state-of-the-art counterparts.

Sign in / Sign up

Export Citation Format

Share Document