GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor

2021 ◽  
Author(s):  
Ellen Jiang ◽  
Edwin Toh ◽  
Alejandra Molina ◽  
Aaron Donsbach ◽  
Carrie J Cai ◽  
...  
Keyword(s):  
Language Models ◽  
Generative Language

scholarly journals Generating Sentences by Editing Prototypes

2018 ◽  
Vol 6 ◽  
pp. 437-450 ◽  
Author(s):  
Kelvin Guu ◽  
Tatsunori B. Hashimoto ◽  
Yonatan Oren ◽  
Percy Liang
Keyword(s):  
Language Model ◽  
Language Modeling ◽  
Language Models ◽  
Training Corpus ◽  
Human Evaluation ◽  
Sentence Level ◽  
Sentence Similarity ◽  
Traditional Language ◽  
Generative Language

We propose a new generative language model for sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional language models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perplexity on language modeling and generates higher quality outputs according to human evaluation. Furthermore, the model gives rise to a latent edit vector that captures interpretable semantics such as sentence similarity and sentence-level analogies.

scholarly journals Language and cognition in bilingual production: the real work still lies ahead

2016 ◽  
Vol 19 (5) ◽  
pp. 895-896 ◽  
Author(s):  
ANTONELLA SORACE
Keyword(s):  
Symbolic Computation ◽  
Mental Representations ◽  
Language Models ◽  
The Real ◽  
Code Mixing ◽  
Real Work ◽  
Generative Language ◽  
Language And Cognition

Goldrick, Putnam and Schwarz (Goldrick, Putnam & Schwarz) argue that code-mixing in bilingual production involves not only combining forms from both languages but also – crucially – integrating grammatical principles with gradient mental representations. They further propose an analysis of a particular case of intrasentential code mixing – doubling constructions – framed within the formalism of Gradient Symbolic Computation. This formalism, in their view, is better suited to accounting for code mixing than other generative language models because it allows the weighting of constraints both in the choice of particular structures within a single language and in blends of structures in code-mixed productions.

scholarly journals Thinking ahead: prediction in context as a keystone of language in humans and machines

2020 ◽  
Author(s):  
Ariel Goldstein ◽  
Zaid Zada ◽  
Eliav Buchnik ◽  
Mariano Schain ◽  
Amy Price ◽  
...  
Keyword(s):  
Real Life ◽  
Language Models ◽  
Language Faculty ◽  
Computational Framework ◽  
New Family ◽  
Neural Representations ◽  
Semantics And Pragmatics ◽  
Generative Language ◽  
Specific Meaning ◽  
Context Specific

Departing from classical rule-based linguistic models, advances in deep learning have led to the development of a new family of self-supervised deep language models (DLMs). These models are trained using a simple self-supervised autoregressive objective, which aims to predict the next word in the context of preceding words in real-life corpora. After training, autoregressive DLMs are able to generate new 'context-aware' sentences with appropriate syntax and convincing semantics and pragmatics. Here we provide empirical evidence for the deep connection between autoregressive DLMs and the human language faculty using a 30-min spoken narrative and electrocorticographic (ECoG) recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Next, we leverage DLM embeddings to demonstrate that many electrodes spontaneously predict the meaning of upcoming words, even hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that deep language models provide an important step toward creating a biologically feasible computational framework for generative language.

scholarly journals Incorporating Stylistic Lexical Preferences in Generative Language Models

2020 ◽  
Author(s):  
Hrituraj Singh ◽  
Gaurav Verma ◽  
Balaji Vasan Srinivasan
Keyword(s):  
Language Models ◽  
Generative Language

Adolescent Language: Models, Assessment, and Links to Reading

2019 ◽  
Author(s):  
Amanda Goodwin ◽  
Yaacov Petscher ◽  
Jamie Tock
Keyword(s):  
Reading Comprehension ◽  
Bifactor Model ◽  
Language Models ◽  
Multiple Group ◽  
Global Factor ◽  
Eighth Grade Students ◽  
Key Aspects ◽  
Future Work ◽  
The Relationship ◽  
Best Fit

Various models have highlighted the complexity of language. Building on foundational ideas regarding three key aspects of language, our study contributes to the literature by 1) exploring broader conceptions of morphology, vocabulary, and syntax, 2) operationalizing this theoretical model into a gamified, standardized, computer-adaptive assessment of language for fifth to eighth grade students entitled Monster, PI, and 3) uncovering further evidence regarding the relationship between language and standardized reading comprehension via this assessment. Multiple-group item response theory (IRT) across grades show that morphology was best fit by a bifactor model of task specific factors along with a global factor related to each skill. Vocabulary was best fit by a bifactor model that identifies performance overall and on specific words. Syntax, though, was best fit by a unidimensional model. Next, Monster, PI produced reliable scores suggesting language can be assessed efficiently and precisely for students via this model. Lastly, performance on Monster, PI explained more than 50% of variance in standardized reading, suggesting operationalizing language via Monster, PI can provide meaningful understandings of the relationship between language and reading comprehension. Specifically, considering just a subset of a construct, like identification of units of meaning, explained significantly less variance in reading comprehension. This highlights the importance of considering these broader constructs. Implications indicate that future work should consider a model of language where component areas are considered broadly and contributions to reading comprehension are explored via general performance on components as well as skill level performance.

Sign in / Sign up

Export Citation Format

Share Document