scholarly journals A Generalized Language Model in Tensor Space

2019 ◽  
Vol 33 ◽  
pp. 7450-7458 ◽  
Author(s):  
Lipeng Zhang ◽  
Peng Zhang ◽  
Xindian Ma ◽  
Shuqin Gu ◽  
Zhan Su ◽  
...  
Keyword(s):  
Language Model ◽  
Tensor Decomposition ◽  
Expressive Power ◽  
Semantic Space ◽  
Language Models ◽  
Tensor Representation ◽  
Order Tensor ◽  
Tensor Space ◽  
Recursive Calculation ◽  
N Gram

In the literature, tensors have been effectively used for capturing the context information in language models. However, the existing methods usually adopt relatively-low order tensors, which have limited expressive power in modeling language. Developing a higher-order tensor representation is challenging, in terms of deriving an effective solution and showing its generality. In this paper, we propose a language model named Tensor Space Language Model (TSLM), by utilizing tensor networks and tensor decomposition. In TSLM, we build a high-dimensional semantic space constructed by the tensor product of word vectors. Theoretically, we prove that such tensor representation is a generalization of the n-gram language model. We further show that this high-order tensor representation can be decomposed to a recursive calculation of conditional probability for language modeling. The experimental results on Penn Tree Bank (PTB) dataset and WikiText benchmark demonstrate the effectiveness of TSLM.

INTEGRATION OF n-GRAM LANGUAGE MODELS IN MULTIPLE CLASSIFIER SYSTEMS FOR OFFLINE HANDWRITTEN TEXT LINE RECOGNITION

2008 ◽  
Vol 22 (07) ◽  
pp. 1301-1321 ◽  
Author(s):  
ROMAN BERTOLAMI ◽  
HORST BUNKE
Keyword(s):  
Language Model ◽  
Language Models ◽  
Combination Method ◽  
Text Line ◽  
Multiple Classifier Systems ◽  
Classifier Systems ◽  
Handwritten Text ◽  
Handwritten Text Recognition ◽  
Multiple Classifier ◽  
N Gram

Current multiple classifier systems for unconstrained handwritten text recognition do not provide a straightforward way to utilize language model information. In this paper, we describe a generic method to integrate a statistical n-gram language model into the combination of multiple offline handwritten text line recognizers. The proposed method first builds a word transition network and then rescores this network with an n-gram language model. Experimental evaluation conducted on a large dataset of offline handwritten text lines shows that the proposed approach improves the recognition accuracy over a reference system as well as over the original combination method that does not include a language model.

Half-Context Language Models

2011 ◽  
Vol 37 (4) ◽  
pp. 843-865
Author(s):  
Hinrich Schütze ◽  
Michael Walsh
Keyword(s):  
Clustering Algorithm ◽  
Language Model ◽  
Model Performance ◽  
Language Models ◽  
Fine Grained ◽  
Specific Analysis ◽  
Distributional Information ◽  
N Gram ◽  
Context Specific ◽  
Context Models

This article investigates the effects of different degrees of contextual granularity on language model performance. It presents a new language model that combines clustering and half-contextualization, a novel representation of contexts. Half-contextualization is based on the half-context hypothesis that states that the distributional characteristics of a word or bigram are best represented by treating its context distribution to the left and right separately and that only directionally relevant distributional information should be used. Clustering is achieved using a new clustering algorithm for class-based language models that compares favorably to the exchange algorithm. When interpolated with a Kneser-Ney model, half-context models are shown to have better perplexity than commonly used interpolated n-gram models and traditional class-based approaches. A novel, fine-grained, context-specific analysis highlights those contexts in which the model performs well and those which are better treated by existing non-class-based models.

scholarly journals A General Technique to Train Language Models on Language Models

2005 ◽  
Vol 31 (2) ◽  
pp. 173-185 ◽  
Author(s):  
Mark-Jan Nederhof
Keyword(s):  
Second Language ◽  
Finite Automaton ◽  
Language Model ◽  
Language Models ◽  
General Technique ◽  
Context Free Grammar ◽  
N Gram ◽  
Context Free ◽  
Probabilistic Context

We show that under certain conditions, a language model can be trained on the basis of a second language model. The main instance of the technique trains a finite automaton on the basis of a probabilistic context-free grammar, such that the Kullback-Leibler distance between grammar and trained automaton is provably minimal. This is a substantial generalization of an existing algorithm to train an n-gram model on the basis of a probabilistic context-free grammar.

A Comparison of Phrase Based and Word based Language Model for Punjabi

2017 ◽  
Vol 7 (7) ◽  
pp. 444
Author(s):  
Umrinderpal Singh
Keyword(s):  
Machine Translation ◽  
Great Improvement ◽  
Language Model ◽  
Language Models ◽  
Translation System ◽  
Information Base ◽  
Model Yield ◽  
Machine Translation System ◽  
N Gram

A language model provides connection to the decoding process to determine a precise word from several available options in the information base or phrase table. The language model can be generated using n-gram approach. Various language models and smoothing procedures are there to determine this model, like unigram, bigram, trigram, interpolation, backoff language model etc. We have done some experiments with different language models where we have used phrases in place of words as the smallest unit. Experiments have shown that phrase based language model yield more accurate results as compared to simple word based mode. We have also done some experiments with machine translation system where we have used phrase based language model rather than word based model and system yield great improvement.

scholarly journals Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation

2015 ◽  
Vol 3 ◽  
pp. 169-182 ◽  
Author(s):  
Rico Sennrich
Keyword(s):  
Statistical Machine Translation ◽  
Language Model ◽  
Language Models ◽  
Translation Quality ◽  
N Gram ◽  
Evaluation Metric ◽  
Log Linear ◽  
Dependency Structures ◽  
Free Word

The role of language models in SMT is to promote fluent translation output, but traditional n-gram language models are unable to capture fluency phenomena between distant words, such as some morphological agreement phenomena, subcategorisation, and syntactic collocations with string-level gaps. Syntactic language models have the potential to fill this modelling gap. We propose a language model for dependency structures that is relational rather than configurational and thus particularly suited for languages with a (relatively) free word order. It is trainable with Neural Networks, and not only improves over standard n-gram language models, but also outperforms related syntactic language models. We empirically demonstrate its effectiveness in terms of perplexity and as a feature function in string-to-tree SMT from English to German and Russian. We also show that using a syntactic evaluation metric to tune the log-linear parameters of an SMT system further increases translation quality when coupled with a syntactic language model.

scholarly journals LanguageCrawl: a generic tool for building language models upon common Crawl

2021 ◽  
Author(s):  
Szymon Roziewski ◽  
Marek Kozłowski
Keyword(s):  
Language Processing ◽  
Deep Neural Networks ◽  
Language Model ◽  
Language Models ◽  
Unstructured Data ◽  
Web Pages ◽  
Data Intensive ◽  
The Common ◽  
Internet Community ◽  
N Gram

AbstractThe exponential growth of the internet community has resulted in the production of a vast amount of unstructured data, including web pages, blogs and social media. Such a volume consisting of hundreds of billions of words is unlikely to be analyzed by humans. In this work we introduce the tool LanguageCrawl, which allows Natural Language Processing (NLP) researchers to easily build web-scale corpora using the Common Crawl Archive—an open repository of web crawl information, which contains petabytes of data. We present three use cases in the course of this work: filtering of Polish websites, the construction of n-gram corpora and the training of a continuous skipgram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and n-gram ranks. This paper focuses particularly on high computing efficiency by applying highly concurrent multitasking. Our tool utilizes effective libraries and design. LanguageCrawl has been made publicly available to enrich the current set of NLP resources. We strongly believe that our work will facilitate further NLP research, especially in under-resourced languages, in which the lack of appropriately-sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks.

scholarly journals Dynamic out-of-vocabulary word registration to language model for speech recognition

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Norihide Kitaoka ◽  
Bohan Chen ◽  
Yuya Obashi
Keyword(s):  
Speech Recognition ◽  
Language Model ◽  
Language Models ◽  
Natural Occurrence ◽  
Registration Method ◽  
Acoustic Models ◽  
Part Of Speech ◽  
Vocabulary Word ◽  
N Gram ◽  
The Impact

AbstractWe propose a method of dynamically registering out-of-vocabulary (OOV) words by assigning the pronunciations of these words to pre-inserted OOV tokens, editing the pronunciations of the tokens. To do this, we add OOV tokens to an additional, partial copy of our corpus, either randomly or to part-of-speech (POS) tags in the selected utterances, when training the language model (LM) for speech recognition. This results in an LM containing OOV tokens, to which we can assign pronunciations. We also investigate the impact of acoustic complexity and the “natural” occurrence frequency of OOV words on the recognition of registered OOV words. The proposed OOV word registration method is evaluated using two modern automatic speech recognition (ASR) systems, Julius and Kaldi, using DNN-HMM acoustic models and N-gram language models (plus an additional evaluation using RNN re-scoring with Kaldi). Our experimental results show that when using the proposed OOV registration method, modern ASR systems can recognize OOV words without re-training the language model, that the acoustic complexity of OOV words affects OOV recognition, and that differences between the “natural” and the assigned occurrence frequencies of OOV words have little impact on the final recognition results.

scholarly journals OxLM: A Neural Language Modelling Framework for Machine Translation

2014 ◽  
Vol 102 (1) ◽  
pp. 81-92 ◽  
Author(s):  
Baltescu Paul ◽  
Blunsom Phil ◽  
Hoang Hieu
Keyword(s):  
Machine Translation ◽  
Language Model ◽  
Computational Cost ◽  
Training Data ◽  
Language Models ◽  
Training Algorithm ◽  
Beam Search ◽  
Modelling Framework ◽  
Language Modelling ◽  
N Gram

Abstract This paper presents an open source implementation1 of a neural language model for machine translation. Neural language models deal with the problem of data sparsity by learning distributed representations for words in a continuous vector space. The language modelling probabilities are estimated by projecting a word's context in the same space as the word representations and by assigning probabilities proportional to the distance between the words and the context's projection. Neural language models are notoriously slow to train and test. Our framework is designed with scalability in mind and provides two optional techniques for reducing the computational cost: the so-called class decomposition trick and a training algorithm based on noise contrastive estimation. Our models may be extended to incorporate direct n-gram features to learn weights for every n-gram in the training data. Our framework comes with wrappers for the cdec and Moses translation toolkits, allowing our language models to be incorporated as normalized features in their decoders (inside the beam search).

scholarly journals Semantic programming by example with pre-trained models

2021 ◽  
Vol 5 (OOPSLA) ◽  
pp. 1-25
Author(s):  
Gust Verbruggen ◽  
Vu Le ◽  
Sumit Gulwani
Keyword(s):  
Syntactic Structure ◽  
Language Model ◽  
Expressive Power ◽  
Language Models ◽  
Inductive Synthesis ◽  
Inductive Programming ◽  
The Given ◽  
Learning Language ◽  
Semantic Operators

The ability to learn programs from few examples is a powerful technology with disruptive applications in many domains, as it allows users to automate repetitive tasks in an intuitive way. Existing frameworks on inductive synthesis only perform syntactic manipulations, where they rely on the syntactic structure of the given examples and not their meaning. Any semantic manipulations, such as transforming dates, have to be manually encoded by the designer of the inductive programming framework. Recent advances in large language models have shown these models to be very adept at performing semantic transformations of its input by simply providing a few examples of the task at hand. When it comes to syntactic transformations, however, these models are limited in their expressive power. In this paper, we propose a novel framework for integrating inductive synthesis with few-shot learning language models to combine the strength of these two popular technologies. In particular, the inductive synthesis is tasked with breaking down the problem in smaller subproblems, among which those that cannot be solved syntactically are passed to the language model. We formalize three semantic operators that can be integrated with inductive synthesizers. To minimize invoking expensive semantic operators during learning, we introduce a novel deferred query execution algorithm that considers the operators to be oracles during learning. We evaluate our approach in the domain of string transformations: the combination methodology can automate tasks that cannot be handled using either technologies by themselves. Finally, we demonstrate the generality of our approach via a case study in the domain of string profiling.

scholarly journals Character n-Gram Embeddings to Improve RNN Language Models

2019 ◽  
Vol 33 ◽  
pp. 5074-5082 ◽  
Author(s):  
Sho Takase ◽  
Jun Suzuki ◽  
Masaaki Nagata
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Recurrent Neural Network ◽  
Language Model ◽  
Language Modeling ◽  
Word Embedding ◽  
Experimental Results ◽  
Language Models ◽  
Word Embeddings ◽  
N Gram

This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character ngram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks

Sign in / Sign up

Export Citation Format

Share Document