scholarly journals Which Sentence Embeddings and Which Layers Encode Syntactic Structure?

2020 ◽  
Author(s):  
M. Alex Kelly ◽  
Yang Xu ◽  
Jesús Calvillo ◽  
David Reitter
Keyword(s):  
Syntactic Structure ◽  
Dimensional Space ◽  
Language Model ◽  
Empirical Support ◽  
Language Models ◽  
Double Object ◽  
Different Types ◽  
Prepositional Object ◽  
High Degree ◽  
Shed Light

Recent models of language have eliminated syntactic-semantic dividing lines. We explore the psycholinguistic implications of this development by comparing different types of sentence embeddings in their ability to encode syntactic constructions. Our study uses contrasting sentence structures known to cause syntactic priming effects, that is, the tendency in humans to re- peat sentence structures after recent exposure. We compare how syntactic alternatives are captured by sentence embed- dings produced by a neural language model (BERT) or by the composition of word embeddings (BEAGLE, HHM, GloVe). Dative double object vs. prepositional object and active vs. passive sentences are separable in the high-dimensional space of the sentence embeddings and can be classified with a high degree of accuracy. The results lend empirical support to the modern, computational, integrated accounts of semantics and syntax, and they shed light on the information stored at different layers in deep language models such as BERT.

Processing verb argument structure across languages: Evidence for shared representations in the bilingual lexicon

2007 ◽  
Vol 28 (4) ◽  
pp. 627-660 ◽  
Author(s):  
ANGELIKI SALAMOURA ◽  
JOHN N. WILLIAMS
Keyword(s):  
Argument Structure ◽  
Syntactic Structure ◽  
Double Object ◽  
Thematic Role ◽  
Advanced Learners ◽  
Shared Representations ◽  
Bilingual Lexicon ◽  
Verb Argument Structure ◽  
Bilingual Literature ◽  
Prepositional Object

Although the organization of first language (L1) and second language (L2) lexicosemantic information has been extensively studied in the bilingual literature, little evidence exists concerning how syntactic information associated with words is represented across languages. The present study examines the shared or independent nature of the representation of verb argument structure in the bilingual mental lexicon and the contribution of constituent order and thematic role information in these representations. In three production tasks, Greek (L1) advanced learners of English (L2) generated an L1 prime structure (Experiment 1: prepositional object [PO] and double object [DO] structures; Experiment 2: PO, DO, and intransitive structures; Experiment 3: PO, DO, locative, and “provide (someone) with (something)” structures) before completing an L2 target structure (PO or DO only). Experiment 1 showed L1-to-L2 syntactic priming; participants tended to reuse L1 structure when producing L2 utterances. Experiments 2 and 3 showed that this tendency was contingent on the combination of both syntactic structure and thematic roles up to the first postverbal argument. Based on these findings, we outline a model of shared representations of syntactic and thematic information for L1 and L2 verbs in the bilingual lexicon.

scholarly journals Low-Rank RNN Adaptation for Context-Aware Language Modeling

2018 ◽  
Vol 6 ◽  
pp. 497-510 ◽  
Author(s):  
Aaron Jaech ◽  
Mari Ostendorf
Keyword(s):  
Language Model ◽  
Language Modeling ◽  
Language Models ◽  
Low Rank ◽  
Model Parameters ◽  
Context Aware ◽  
Context Vector ◽  
Additional Input ◽  
Different Types ◽  
Powerful Mechanism

A context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for using context to adapt an RNN by letting the context vector control a low-rank transformation of the recurrent layer weight matrix. Experiments show that allowing a greater fraction of the model parameters to be adjusted has benefits in terms of perplexity and classification for several different types of context.

scholarly journals Semantic programming by example with pre-trained models

2021 ◽  
Vol 5 (OOPSLA) ◽  
pp. 1-25
Author(s):  
Gust Verbruggen ◽  
Vu Le ◽  
Sumit Gulwani
Keyword(s):  
Syntactic Structure ◽  
Language Model ◽  
Expressive Power ◽  
Language Models ◽  
Inductive Synthesis ◽  
Inductive Programming ◽  
The Given ◽  
Learning Language ◽  
Semantic Operators

The ability to learn programs from few examples is a powerful technology with disruptive applications in many domains, as it allows users to automate repetitive tasks in an intuitive way. Existing frameworks on inductive synthesis only perform syntactic manipulations, where they rely on the syntactic structure of the given examples and not their meaning. Any semantic manipulations, such as transforming dates, have to be manually encoded by the designer of the inductive programming framework. Recent advances in large language models have shown these models to be very adept at performing semantic transformations of its input by simply providing a few examples of the task at hand. When it comes to syntactic transformations, however, these models are limited in their expressive power. In this paper, we propose a novel framework for integrating inductive synthesis with few-shot learning language models to combine the strength of these two popular technologies. In particular, the inductive synthesis is tasked with breaking down the problem in smaller subproblems, among which those that cannot be solved syntactically are passed to the language model. We formalize three semantic operators that can be integrated with inductive synthesizers. To minimize invoking expensive semantic operators during learning, we introduce a novel deferred query execution algorithm that considers the operators to be oracles during learning. We evaluate our approach in the domain of string transformations: the combination methodology can automate tasks that cannot be handled using either technologies by themselves. Finally, we demonstrate the generality of our approach via a case study in the domain of string profiling.

scholarly journals A Closer Look at the Cellular and Molecular Components of the Deep/Muscular Fasciae

2021 ◽  
Vol 22 (3) ◽  
pp. 1411
Author(s):  
Caterina Fede ◽  
Carmelo Pirri ◽  
Chenglei Fan ◽  
Lucia Petrelli ◽  
Diego Guidolin ◽  
...  
Keyword(s):  
Extracellular Matrix ◽  
Connective Tissue ◽  
Treatment Strategies ◽  
Clear Understanding ◽  
Pathological Characteristics ◽  
Appropriate Treatment ◽  
Different Types ◽  
Molecular Components ◽  
Entire Network ◽  
Shed Light

The fascia can be defined as a dynamic highly complex connective tissue network composed of different types of cells embedded in the extracellular matrix and nervous fibers: each component plays a specific role in the fascial system changing and responding to stimuli in different ways. This review intends to discuss the various components of the fascia and their specific roles; this will be carried out in the effort to shed light on the mechanisms by which they affect the entire network and all body systems. A clear understanding of fascial anatomy from a microscopic viewpoint can further elucidate its physiological and pathological characteristics and facilitate the identification of appropriate treatment strategies.

Syncretism of plural forms in Spanish Dialects

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
María Mare
Keyword(s):  
Syntactic Structure ◽  
Person Plural ◽  
Person Features ◽  
Third Person ◽  
Lexical Items ◽  
First Person Plural ◽  
The Moment ◽  
The Third Person ◽  
Shed Light ◽  
Spanish Dialects

Abstract One of the main discussions about the interaction between morphology and syntax revolves around the richness or poverty of features and wherever this richness/poverty is found either in the syntactic structure or the lexical items. A phenomenon subject to this debate has been syncretism, especially in theories that assume late insertion such as Distributed Morphology. This paper delves into the syncretism observed between the first person plural and the third person in the clitic domain in some Spanish dialects. Our analysis will lead to a revision of the distribution of person features and their relationship with plural number, while at the same time it will shed light on other morphological alternations displayed in Spanish dialects; that is, subject-verb unagreement and mesoclisis in imperatives. In order to explain the behavior of the data under discussion, I propose that lexical items are specified for all the relevant features at the moment of insertion, although the values of these features can be neutralized. I argue that the distribution proposed allows for some fundamental generalizations about the vocabulary inventories in Spanish varieties, and shows that the variation pattern exhibits an *ABA effect, i.e., only contiguous cells in a paradigm are syncretic.

scholarly journals Whole-Body Regeneration in Sponges: Diversity, Fine Mechanisms, and Future Prospects

Genes ◽  
2021 ◽  
Vol 12 (4) ◽  
pp. 506
Author(s):  
Alexander Ereskovsky ◽  
Ilya E. Borisenko ◽  
Fyodor V. Bolshakov ◽  
Andrey I. Lavrov
Keyword(s):  
Single Species ◽  
Whole Body ◽  
Phylogenetic Position ◽  
Main Body ◽  
Regenerative Processes ◽  
Cellular Mechanisms ◽  
Different Types ◽  
Dissociated Cells ◽  
Shed Light ◽  
Regeneration Processes

While virtually all animals show certain abilities for regeneration after an injury, these abilities vary greatly among metazoans. Porifera (Sponges) is basal metazoans characterized by a wide variety of different regenerative processes, including whole-body regeneration (WBR). Considering phylogenetic position and unique body organization, sponges are highly promising models, as they can shed light on the origin and early evolution of regeneration in general and WBR in particular. The present review summarizes available data on the morphogenetic and cellular mechanisms accompanying different types of WBR in sponges. Sponges show a high diversity of WBR, which principally could be divided into (1) WBR from a body fragment and (2) WBR by aggregation of dissociated cells. Sponges belonging to different phylogenetic clades and even to different species and/or differing in the anatomical structure undergo different morphogeneses after similar operations. A common characteristic feature of WBR in sponges is the instability of the main body axis: a change of the organism polarity is described during all types of WBR. The cellular mechanisms of WBR are different across sponge classes, while cell dedifferentiations and transdifferentiations are involved in regeneration processes in all sponges. Data considering molecular regulation of WBR in sponges are extremely scarce. However, the possibility to achieve various types of WBR ensured by common morphogenetic and cellular basis in a single species makes sponges highly accessible for future comprehensive physiological, biochemical, and molecular studies of regeneration processes.

scholarly journals Development of Speech Recognition Systems in Emergency Call Centers

Symmetry ◽  
2021 ◽  
Vol 13 (4) ◽  
pp. 634
Author(s):  
Alakbar Valizada ◽  
Natavan Akhundova ◽  
Samir Rustamov
Keyword(s):  
Speech Recognition ◽  
Markov Model ◽  
Hidden Markov ◽  
Call Centers ◽  
The Other ◽  
Language Models ◽  
Emergency Call ◽  
Acoustic Model ◽  
Different Types ◽  
Recognition Systems

In this paper, various methodologies of acoustic and language models, as well as labeling methods for automatic speech recognition for spoken dialogues in emergency call centers were investigated and comparatively analyzed. Because of the fact that dialogue speech in call centers has specific context and noisy, emotional environments, available speech recognition systems show poor performance. Therefore, in order to accurately recognize dialogue speeches, the main modules of speech recognition systems—language models and acoustic training methodologies—as well as symmetric data labeling approaches have been investigated and analyzed. To find an effective acoustic model for dialogue data, different types of Gaussian Mixture Model/Hidden Markov Model (GMM/HMM) and Deep Neural Network/Hidden Markov Model (DNN/HMM) methodologies were trained and compared. Additionally, effective language models for dialogue systems were defined based on extrinsic and intrinsic methods. Lastly, our suggested data labeling approaches with spelling correction are compared with common labeling methods resulting in outperforming the other methods with a notable percentage. Based on the results of the experiments, we determined that DNN/HMM for an acoustic model, trigram with Kneser–Ney discounting for a language model and using spelling correction before training data for a labeling method are effective configurations for dialogue speech recognition in emergency call centers. It should be noted that this research was conducted with two different types of datasets collected from emergency calls: the Dialogue dataset (27 h), which encapsulates call agents’ speech, and the Summary dataset (53 h), which contains voiced summaries of those dialogues describing emergency cases. Even though the speech taken from the emergency call center is in the Azerbaijani language, which belongs to the Turkic group of languages, our approaches are not tightly connected to specific language features. Hence, it is anticipated that suggested approaches can be applied to the other languages of the same group.

scholarly journals Application of Biomimetics to Architectural and Urban Design: A Review across Scales

2020 ◽  
Vol 12 (23) ◽  
pp. 9813 ◽  
Author(s):  
Yuta Uchiyama ◽  
Eduardo Blanco ◽  
Ryo Kohsaka
Keyword(s):  
Urban Design ◽  
Architectural Design ◽  
Evolutionary Process ◽  
Future Research ◽  
Conceptual Level ◽  
Sustainable Society ◽  
Types Of Knowledge ◽  
The Social ◽  
Different Types ◽  
Shed Light

Application of biomimetics has expanded progressively to other fields in recent years, including urban and architectural design, scaling up from materials to a larger scale. Besides its contribution to design and functionality through a long evolutionary process, the philosophy of biomimetics contributes to a sustainable society at the conceptual level. The aim of this review is to shed light on trends in the application of biomimetics to architectural and urban design, in order to identify potential issues and successes resulting from implementation. In the application of biomimetics to architectural design, parts of individual “organisms”, including their form and surface structure, are frequently mimicked, whereas in urban design, on a larger scale, biomimetics is applied to mimic whole ecosystems. The overall trends of the reviewed research indicate future research necessity in the field of on biomimetic application in architectural and urban design, including Biophilia and Material. As for the scale of the applications, the urban-scale research is limited and it is a promising research which can facilitate the social implementation of biomimetics. As for facilitating methods of applications, it is instrumental to utilize different types of knowledge, such as traditional knowledge, and providing scientific clarification of functions and systems based on reviews. Thus, interdisciplinary research is required additionally to reach such goals.

scholarly journals Disclosure of Children's Positive Serostatus to Family and Nonfamily Members: Informal Caregivers in Togo, West Africa

2011 ◽  
Vol 2011 ◽  
pp. 1-8 ◽  
Author(s):  
Ami R. Moore ◽  
David Williamson
Keyword(s):  
Financial Support ◽  
Qualitative Data ◽  
Informal Caregivers ◽  
Structural Constraints ◽  
Spiritual Support ◽  
Regression Analyses ◽  
Number Of Children ◽  
Different Types ◽  
Shed Light ◽  
Specific Reason

This study examined the structural constraints to disclosure of children's positive serostatus among informal caregivers to family and nonfamily members in Togo. It drew on two data sources, one qualitative and the other quantitative. Qualitative data showed that caregivers cautiously disclosed child's positive serostatus for fear of being stigmatized and discriminated against as well as to protect the children from being stigmatized. Binary regression analyses revealed that different factors influenced reasons for disclosure of a child's serostatus. For instance, while caregivers' serostatus and number of children significantly influenced disclosure for financial support, disclosure of a child's serostatus for spiritual support was strongly affected by education and religion. These results shed light on factors and reasons for disclosure among caregivers. This knowledge is important because different types of programs and advice should be given to caregivers with specific reason(s) for disclosure instead of creating a “one-size-fits all” program for all caregivers.

INTEGRATION OF n-GRAM LANGUAGE MODELS IN MULTIPLE CLASSIFIER SYSTEMS FOR OFFLINE HANDWRITTEN TEXT LINE RECOGNITION

2008 ◽  
Vol 22 (07) ◽  
pp. 1301-1321 ◽  
Author(s):  
ROMAN BERTOLAMI ◽  
HORST BUNKE
Keyword(s):  
Language Model ◽  
Language Models ◽  
Combination Method ◽  
Text Line ◽  
Multiple Classifier Systems ◽  
Classifier Systems ◽  
Handwritten Text ◽  
Handwritten Text Recognition ◽  
Multiple Classifier ◽  
N Gram

Current multiple classifier systems for unconstrained handwritten text recognition do not provide a straightforward way to utilize language model information. In this paper, we describe a generic method to integrate a statistical n-gram language model into the combination of multiple offline handwritten text line recognizers. The proposed method first builds a word transition network and then rescores this network with an n-gram language model. Experimental evaluation conducted on a large dataset of offline handwritten text lines shows that the proposed approach improves the recognition accuracy over a reference system as well as over the original combination method that does not include a language model.

Sign in / Sign up

Export Citation Format

Share Document