Scope and Challenges of Language Modelling - An Interrogative Survey on Context and Embeddings

2019 ◽  
pp. 131-145
Author(s):  
Matthias Nitsche ◽  
Marina Tropmann-Frick
Keyword(s):  
Language Modelling

Out of Set Language Modelling in Hierarchical Language Identification

2016 ◽  
Author(s):  
Saad Irtza ◽  
Vidhyasaharan Sethu ◽  
Sarith Fernando ◽  
Eliathamby Ambikairajah ◽  
Haizhou Li
Keyword(s):  
Language Identification ◽  
Language Modelling

scholarly journals Supervised and unsupervised language modelling in Chest X-Ray radiological reports

PLoS ONE ◽  
2020 ◽  
Vol 15 (3) ◽  
pp. e0229963 ◽  
Author(s):  
Ignat Drozdov ◽  
Daniel Forbes ◽  
Benjamin Szubert ◽  
Mark Hall ◽  
Chris Carlin ◽  
...  
Keyword(s):  
X Ray ◽  
Chest X Ray ◽  
Language Modelling

Natural statistics in language Modelling

1998 ◽  
Vol 5 (3) ◽  
pp. 246-255 ◽  
Author(s):  
Royal Skousen
Keyword(s):  
Language Modelling ◽  
Natural Statistics

scholarly journals Language Modelling for a Low-Resource Language in Sarawak, Malaysia

2019 ◽  
pp. 147-158 ◽  
Author(s):  
Sarah Samson Juan ◽  
Muhamad Fikri Che Ismail ◽  
Hamimah Ujir ◽  
Irwandi Hipiny
Keyword(s):  
Low Resource ◽  
Language Modelling

scholarly journals Cross-domain paraphrasing for improving language modelling using out-of-domain data

2013 ◽  
Author(s):  
X. Liu ◽  
M. J. F. Gales ◽  
P. C. Woodland
Keyword(s):  
Cross Domain ◽  
Language Modelling

scholarly journals Towards Fully 8-bit Integer Inference for the Transformer Model

2020 ◽  
Author(s):  
Ye Lin ◽  
Yanyang Li ◽  
Tengbo Liu ◽  
Tong Xiao ◽  
Tongran Liu ◽  
...  
Keyword(s):  
Deep Neural Networks ◽  
Floating Point ◽  
Memory Footprint ◽  
Great Progress ◽  
Comparable Performance ◽  
Complex Models ◽  
Language Modelling ◽  
Transformer Model ◽  
And Storage ◽  
Modelling Task

8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e.g., Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En<->Ro, WMT14 En<->De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.

Sign in / Sign up

Export Citation Format

Share Document