scholarly journals OxLM: A Neural Language Modelling Framework for Machine Translation

2014 ◽  
Vol 102 (1) ◽  
pp. 81-92 ◽  
Author(s):  
Baltescu Paul ◽  
Blunsom Phil ◽  
Hoang Hieu
Keyword(s):  
Machine Translation ◽  
Language Model ◽  
Computational Cost ◽  
Training Data ◽  
Language Models ◽  
Training Algorithm ◽  
Beam Search ◽  
Modelling Framework ◽  
Language Modelling ◽  
N Gram

Abstract This paper presents an open source implementation1 of a neural language model for machine translation. Neural language models deal with the problem of data sparsity by learning distributed representations for words in a continuous vector space. The language modelling probabilities are estimated by projecting a word's context in the same space as the word representations and by assigning probabilities proportional to the distance between the words and the context's projection. Neural language models are notoriously slow to train and test. Our framework is designed with scalability in mind and provides two optional techniques for reducing the computational cost: the so-called class decomposition trick and a training algorithm based on noise contrastive estimation. Our models may be extended to incorporate direct n-gram features to learn weights for every n-gram in the training data. Our framework comes with wrappers for the cdec and Moses translation toolkits, allowing our language models to be incorporated as normalized features in their decoders (inside the beam search).

A Comparison of Phrase Based and Word based Language Model for Punjabi

2017 ◽  
Vol 7 (7) ◽  
pp. 444
Author(s):  
Umrinderpal Singh
Keyword(s):  
Machine Translation ◽  
Great Improvement ◽  
Language Model ◽  
Language Models ◽  
Translation System ◽  
Information Base ◽  
Model Yield ◽  
Machine Translation System ◽  
N Gram

A language model provides connection to the decoding process to determine a precise word from several available options in the information base or phrase table. The language model can be generated using n-gram approach. Various language models and smoothing procedures are there to determine this model, like unigram, bigram, trigram, interpolation, backoff language model etc. We have done some experiments with different language models where we have used phrases in place of words as the smallest unit. Experiments have shown that phrase based language model yield more accurate results as compared to simple word based mode. We have also done some experiments with machine translation system where we have used phrase based language model rather than word based model and system yield great improvement.

scholarly journals CloudLM: a Cloud-based Language Model for Machine Translation

2016 ◽  
Vol 105 (1) ◽  
pp. 51-61 ◽  
Author(s):  
Jorge Ferrández-Tordera ◽  
Sergio Ortiz-Rojas ◽  
Antonio Toral
Keyword(s):  
Big Data ◽  
Natural Language Processing ◽  
Machine Translation ◽  
Language Processing ◽  
State Of The Art ◽  
Language Model ◽  
Essential Element ◽  
Language Models ◽  
Language Modelling ◽  
Statistical Approaches

Abstract Language models (LMs) are an essential element in statistical approaches to natural language processing for tasks such as speech recognition and machine translation (MT). The advent of big data leads to the availability of massive amounts of data to build LMs, and in fact, for the most prominent languages, using current techniques and hardware, it is not feasible to train LMs with all the data available nowadays. At the same time, it has been shown that the more data is used for a LM the better the performance, e.g. for MT, without any indication yet of reaching a plateau. This paper presents CloudLM, an open-source cloud-based LM intended for MT, which allows to query distributed LMs. CloudLM relies on Apache Solr and provides the functionality of state-of-the-art language modelling (it builds upon KenLM), while allowing to query massive LMs (as the use of local memory is drastically reduced), at the expense of slower decoding speed.

scholarly journals Character n-Gram Embeddings to Improve RNN Language Models

2019 ◽  
Vol 33 ◽  
pp. 5074-5082 ◽  
Author(s):  
Sho Takase ◽  
Jun Suzuki ◽  
Masaaki Nagata
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Recurrent Neural Network ◽  
Language Model ◽  
Language Modeling ◽  
Word Embedding ◽  
Experimental Results ◽  
Language Models ◽  
Word Embeddings ◽  
N Gram

This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character ngram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks

scholarly journals An exploratory research on grammar checking of Bangla sentences using statistical language models

2020 ◽  
Vol 10 (3) ◽  
pp. 3244
Author(s):  
M. D. Riazur Rahman ◽  
M. D. Tarek Habib ◽  
M. D. Sadekur Rahman ◽  
Gazi Zahirul Islam ◽  
M. D. Abbas Ali Khan
Keyword(s):  
Language Processing ◽  
Language Model ◽  
Language Models ◽  
Exploratory Research ◽  
Smoothing Technique ◽  
Comparative Performance ◽  
Statistical Language Models ◽  
Language Modelling ◽  
N Gram ◽  
Improved Technique

N-gram based language models are very popular and extensively used statistical methods for solving various natural language processing problems including grammar checking. Smoothing is one of the most effective techniques used in building a language model to deal with data sparsity problem. Kneser-Ney is one of the most prominently used and successful smoothing technique for language modelling. In our previous work, we presented a Witten-Bell smoothing based language modelling technique for checking grammatical correctness of Bangla sentences which showed promising results outperforming previous methods. In this work, we proposed an improved method using Kneser-Ney smoothing based n-gram language model for grammar checking and performed a comparative performance analysis between Kneser-Ney and Witten-Bell smoothing techniques for the same purpose. We also provided an improved technique for calculating the optimum threshold which further enhanced the the results. Our experimental results show that, Kneser-Ney outperforms Witten-Bell as a smoothing technique when used with n-gram LMs for checking grammatical correctness of Bangla sentences.

scholarly journals Efficient Embedded Decoding of Neural Network Language Models in a Machine Translation System

2018 ◽  
Vol 28 (09) ◽  
pp. 1850007
Author(s):  
Francisco Zamora-Martinez ◽  
Maria Jose Castro-Bleda
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Language Processing ◽  
Traditional Approach ◽  
Computational Cost ◽  
Integrated Approach ◽  
Language Models ◽  
Translation System ◽  
Neural Net ◽  
Network Language

Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on [Formula: see text]-best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and [Formula: see text]-gram-based systems, showing that the integrated approach seems more promising for [Formula: see text]-gram-based systems, even with nonfull-quality NNLMs.

INTEGRATION OF n-GRAM LANGUAGE MODELS IN MULTIPLE CLASSIFIER SYSTEMS FOR OFFLINE HANDWRITTEN TEXT LINE RECOGNITION

2008 ◽  
Vol 22 (07) ◽  
pp. 1301-1321 ◽  
Author(s):  
ROMAN BERTOLAMI ◽  
HORST BUNKE
Keyword(s):  
Language Model ◽  
Language Models ◽  
Combination Method ◽  
Text Line ◽  
Multiple Classifier Systems ◽  
Classifier Systems ◽  
Handwritten Text ◽  
Handwritten Text Recognition ◽  
Multiple Classifier ◽  
N Gram

Current multiple classifier systems for unconstrained handwritten text recognition do not provide a straightforward way to utilize language model information. In this paper, we describe a generic method to integrate a statistical n-gram language model into the combination of multiple offline handwritten text line recognizers. The proposed method first builds a word transition network and then rescores this network with an n-gram language model. Experimental evaluation conducted on a large dataset of offline handwritten text lines shows that the proposed approach improves the recognition accuracy over a reference system as well as over the original combination method that does not include a language model.

scholarly journals Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees

2016 ◽  
Vol 4 ◽  
pp. 477-490 ◽  
Author(s):  
Ehsan Shareghi ◽  
Matthias Petri ◽  
Gholamreza Haffari ◽  
Trevor Cohn
Keyword(s):  
Infinite Order ◽  
State Of The Art ◽  
Language Model ◽  
Language Models ◽  
Memory Usage ◽  
Suffix Trees ◽  
Construction Time ◽  
Modest Increase ◽  
Order Language ◽  
Language Modelling

Efficient methods for storing and querying are critical for scaling high-order m-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500×, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).

scholarly journals Pushdown Automata in Statistical Machine Translation

2014 ◽  
Vol 40 (3) ◽  
pp. 687-723 ◽  
Author(s):  
Cyril Allauzen ◽  
Bill Byrne ◽  
Adrià de Gispert ◽  
Gonzalo Iglesias ◽  
Michael Riley
Keyword(s):  
Machine Translation ◽  
Large Scale ◽  
Complexity Analysis ◽  
Statistical Machine Translation ◽  
Language Model ◽  
General Purpose ◽  
Language Models ◽  
Experimental Conditions ◽  
Context Free ◽  
Pushdown Automata

This article describes the use of pushdown automata (PDA) in the context of statistical machine translation and alignment under a synchronous context-free grammar. We use PDAs to compactly represent the space of candidate translations generated by the grammar when applied to an input sentence. General-purpose PDA algorithms for replacement, composition, shortest path, and expansion are presented. We describe HiPDT, a hierarchical phrase-based decoder using the PDA representation and these algorithms. We contrast the complexity of this decoder with a decoder based on a finite state automata representation, showing that PDAs provide a more suitable framework to achieve exact decoding for larger synchronous context-free grammars and smaller language models. We assess this experimentally on a large-scale Chinese-to-English alignment and translation task. In translation, we propose a two-pass decoding strategy involving a weaker language model in the first-pass to address the results of PDA complexity analysis. We study in depth the experimental conditions and tradeoffs in which HiPDT can achieve state-of-the-art performance for large-scale SMT.

scholarly journals Injecting Event Knowledge into Pre-Trained Language Models for Event Extraction

2020 ◽  
Author(s):  
Zining Yang ◽  
Siyu Zhan ◽  
Mengshu Hou ◽  
Xiaoyang Zeng ◽  
Hao Zhu
Keyword(s):  
Language Model ◽  
Empirical Evaluation ◽  
Event Extraction ◽  
Training Data ◽  
Language Models ◽  
Extraction System ◽  
Training Dataset ◽  
Great Success ◽  
Event Knowledge ◽  
Event Trigger

The recent pre-trained language model has made great success in many NLP tasks. In this paper, we propose an event extraction system based on the novel pre-trained language model BERT to extract both event trigger and argument. As a deep-learningbased method, the size of the training dataset has a crucial impact on performance. To address the lacking training data problem for event extraction, we further train the pretrained language model with a carefully constructed in-domain corpus to inject event knowledge to our event extraction system with minimal efforts. Empirical evaluation on the ACE2005 dataset shows that injecting event knowledge can significantly improve the performance of event extraction.

scholarly journals Analysis and Evaluation of Language Models for Word Sense Disambiguation

2021 ◽  
pp. 1-55
Author(s):  
Daniel Loureiro ◽  
Kiamehr Rezaee ◽  
Mohammad Taher Pilehvar ◽  
Jose Camacho-Collados
Keyword(s):  
Feature Extraction ◽  
Word Sense Disambiguation ◽  
Language Model ◽  
Training Data ◽  
Fine Tuning ◽  
Language Models ◽  
Coarse Grained ◽  
Word Sense ◽  
Sense Disambiguation ◽  
High Level

Abstract Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, the simple feature extraction strategy of averaging contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements obtained by increasing the size of this training data.

Sign in / Sign up

Export Citation Format

Share Document