Non-movement-based approaches

2020 ◽  
pp. 156-202
Author(s):  
Rui P. Chaves ◽  
Michael T. Putnam
Keyword(s):  
Language Processing ◽  
Point Of View ◽  
Language Models ◽  
Sentence Structure ◽  
Human Language ◽  
Unbounded Dependency

This chapter compares movement-based conceptions of grammar and of unbounded dependency constructions with their construction- and non-movement-based antithesis. In particular, the focus of this chapter is on how unification and construction-based grammar provides not only a better handle on the phenomena than the MP from a linguistic perspective, but also from a psycholinguistic point of view. The flexibility of non-movement-based accounts allows a much wider and much more complex array of unbounded dependency patterns because it rejects the basic idea that extracted phrases start out as being embedded in sentence structure, and instead views the propagation of all information in sentence structure as a local and distributed (featural) process. The grammatical theory discussed in this chapter is also more consistent with extant models of human language processing than the MP, and demonstrably allows for efficient incremental and probabilistic language models of both comprehension and production.

scholarly journals Text: An R-package for Analyzing and Visualizing Human Language Using Natural Language Processing and Deep Learning

2021 ◽  
Author(s):  
Oscar Nils Erik Kjell ◽  
H. Andrew Schwartz ◽  
Salvatore Giorgi
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Rating Scale ◽  
State Of The Art ◽  
R Package ◽  
Language Models ◽  
Categorical Variables ◽  
Human Language

The language that individuals use for expressing themselves contains rich psychological information. Recent significant advances in Natural Language Processing (NLP) and Deep Learning (DL), namely transformers, have resulted in large performance gains in tasks related to understanding natural language such as machine translation. However, these state-of-the-art methods have not yet been made easily accessible for psychology researchers, nor designed to be optimal for human-level analyses. This tutorial introduces text (www.r-text.org), a new R-package for analyzing and visualizing human language using transformers, the latest techniques from NLP and DL. Text is both a modular solution for accessing state-of-the-art language models and an end-to-end solution catered for human-level analyses. Hence, text provides user-friendly functions tailored to test hypotheses in social sciences for both relatively small and large datasets. This tutorial describes useful methods for analyzing text, providing functions with reliable defaults that can be used off-the-shelf as well as providing a framework for the advanced users to build on for novel techniques and analysis pipelines. The reader learns about six methods: 1) textEmbed: to transform text to traditional or modern transformer-based word embeddings (i.e., numeric representations of words); 2) textTrain: to examine the relationships between text and numeric/categorical variables; 3) textSimilarity and 4) textSimilarityTest: to computing semantic similarity scores between texts and significance test the difference in meaning between two sets of texts; and 5) textProjection and 6) textProjectionPlot: to examine and visualize text within the embedding space according to latent or specified construct dimensions (e.g., low to high rating scale scores).

Language change as a scientific construct of a probabilistically organized information system

Glottotheory ◽  
2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Eike U. Decker
Keyword(s):  
Information System ◽  
Probability Distribution ◽  
Language Processing ◽  
Language Change ◽  
Probability Distributions ◽  
Point Of View ◽  
Theoretical Point ◽  
Human Language ◽  
Individual Level ◽  
Collective Levels

AbstractThe present paper discusses language change from an information and systems theoretical point of view, taking on a diachronic perspective. It is argued that human language has to be regarded as a probabilistically organized information system in which synchronizations of linguistic systems of individuals create unstable (dynamic, ever-changing) collective levels (“language systems”). Therefore, probabilistic organization of language processing on an individual level leads – via bottom-up structure – to probabilistic organization of language systems as a whole. If we thus regard linguistic objects like e.g. a Saussurean sign as generally unstable and defined by probability distributions even from a synchronic point of view, we must understand language change (diachronic developments) as probabilistic as well. Therefore, language change in its “classical sense” (a change in linguistic objects) has to be reinterpreted as a change in probability distribution. Nevertheless, the term language change and its meaning then still lack exactness regarding some details; so we have to use this term carefully and be aware of its weaknesses. With a close look at language as an information system with both a synchronic as well as a diachronic dimension, we finally have to admit that language change is a scientific construct serving as a – sometimes quite useful – simplification within the linguistic field.

scholarly journals Quasi-compositional mapping from form to meaning: a neural network-based approach to capturing neural responses during human language comprehension

2019 ◽  
Vol 375 (1791) ◽  
pp. 20190313 ◽  
Author(s):  
Milena Rabovsky ◽  
James L. McClelland
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Natural Language ◽  
Language Processing ◽  
Sentence Comprehension ◽  
Language Models ◽  
Human Language ◽  
Controlled Attention ◽  
Comprehension Process ◽  
N400 Component

We argue that natural language can be usefully described as quasi-compositional and we suggest that deep learning-based neural language models bear long-term promise to capture how language conveys meaning. We also note that a successful account of human language processing should explain both the outcome of the comprehension process and the continuous internal processes underlying this performance. These points motivate our discussion of a neural network model of sentence comprehension, the Sentence Gestalt model, which we have used to account for the N400 component of the event-related brain potential (ERP), which tracks meaning processing as it happens in real time. The model, which shares features with recent deep learning-based language models, simulates N400 amplitude as the automatic update of a probabilistic representation of the situation or event described by the sentence, corresponding to a temporal difference learning signal at the level of meaning. We suggest that this process happens relatively automatically, and that sometimes a more-controlled attention-dependent process is necessary for successful comprehension, which may be reflected in the subsequent P600 ERP component. We relate this account to current deep learning models as well as classic linguistic theory, and use it to illustrate a domain general perspective on some specific linguistic operations postulated based on compositional analyses of natural language. This article is part of the theme issue ‘Towards mechanistic models of meaning composition’.

scholarly journals A Neurolinguistic Model of Grammatical Construction Processing

2006 ◽  
Vol 18 (12) ◽  
pp. 2088-2107 ◽  
Author(s):  
Peter Ford Dominey ◽  
Michel Hoen ◽  
Toshio Inui
Keyword(s):  
Language Processing ◽  
Theoretical Approach ◽  
Construction Grammar ◽  
Sentence Structure ◽  
Human Language ◽  
Functional Description ◽  
Compositional Structure ◽  
Neurophysiological Studies ◽  
Grammatical Construction ◽  
Simulation Results

One of the functions of everyday human language is to communicate meaning. Thus, when one hears or reads the sentence, “John gave a book to Mary,” some aspect of an event concerning the transfer of possession of a book from John to Mary is (hopefully) transmitted. One theoretical approach to language referred to as construction grammar emphasizes this link between sentence structure and meaning in the form of grammatical constructions. The objective of the current research is to (1) outline a functional description of grammatical construction processing based on principles of psycholinguistics, (2) develop a model of how these functions can be implemented in human neurophysiology, and then (3) demonstrate the feasibility of the resulting model in processing languages of typologically diverse natures, that is, English, French, and Japanese. In this context, particular interest will be directed toward the processing of novel compositional structure of relative phrases. The simulation results are discussed in the context of recent neurophysiological studies of language processing.

scholarly journals Fine-tuning of deep language models as a computational framework of modeling listeners' perspective during language comprehension

2021 ◽  
Author(s):  
Refael Tikochinski ◽  
Ariel Goldstein ◽  
Yaara Yeshurun ◽  
Uri Hasson ◽  
Roi Reichart
Keyword(s):  
Cognitive Processing ◽  
Language Processing ◽  
Language Comprehension ◽  
Fine Tuning ◽  
Language Models ◽  
Human Language ◽  
Neural Responses ◽  
Brain Areas ◽  
Computational Framework ◽  
Different Levels

Computational Deep Language Models (DLMs) have been shown to be effective in predicting neural responses during natural language processing. This study introduces a novel computational framework, based on the concept of fine-tuning (Hinton, 2007), for modeling differences in interpretation of narratives based on the listeners' perspective (i.e. their prior knowledge, thoughts, and beliefs). We draw on an fMRI experiment conducted by Yeshurun et al. (2017), in which two groups of listeners were listening to the same narrative but with two different perspectives (cheating versus paranoia). We collected a dedicated dataset of ~3000 stories, and used it to create two modified (fine-tuned) versions of a pre-trained DLM, each representing the perspective of a different group of listeners. Information extracted from each of the two fine-tuned models was better fitted with neural responses of the corresponding group of listeners. Furthermore, we show that the degree of difference between the listeners' interpretation of the story - as measured both neurally and behaviorally - can be approximated using the distances between the representations of the story extracted from these two fine-tuned models. These models-brain associations were expressed in many language-related brain areas, as well as in several higher-order areas related to the default-mode and the mentalizing networks, therefore implying that computational fine-tuning reliably captures relevant aspects of human language comprehension across different levels of cognitive processing.

In Search of a Human Language Processing System

1985 ◽  
Vol 30 (7) ◽  
pp. 529-531
Author(s):  
Patrick Carroll
Keyword(s):  
Language Processing ◽  
Processing System ◽  
Human Language

scholarly journals Hierarchical Phoneme Classification for Improved Speech Recognition

2021 ◽  
Vol 11 (1) ◽  
pp. 428
Author(s):  
Donghoon Oh ◽  
Jeong-Sik Park ◽  
Ji-Hwan Kim ◽  
Gil-Jin Jang
Keyword(s):  
Speech Recognition ◽  
Language Processing ◽  
Confusion Matrix ◽  
Critical Factor ◽  
Recognition System ◽  
Classification Performance ◽  
Language Models ◽  
Successful Implementation ◽  
Phoneme Classification ◽  
Improved Performance

Speech recognition consists of converting input sound into a sequence of phonemes, then finding text for the input using language models. Therefore, phoneme classification performance is a critical factor for the successful implementation of a speech recognition system. However, correctly distinguishing phonemes with similar characteristics is still a challenging problem even for state-of-the-art classification methods, and the classification errors are hard to be recovered in the subsequent language processing steps. This paper proposes a hierarchical phoneme clustering method to exploit more suitable recognition models to different phonemes. The phonemes of the TIMIT database are carefully analyzed using a confusion matrix from a baseline speech recognition model. Using automatic phoneme clustering results, a set of phoneme classification models optimized for the generated phoneme groups is constructed and integrated into a hierarchical phoneme classification method. According to the results of a number of phoneme classification experiments, the proposed hierarchical phoneme group models improved performance over the baseline by 3%, 2.1%, 6.0%, and 2.2% for fricative, affricate, stop, and nasal sounds, respectively. The average accuracy was 69.5% and 71.7% for the baseline and proposed hierarchical models, showing a 2.2% overall improvement.

Towards corpus and model: Hierarchical structured-attention-based features for Indonesian named entity recognition

2021 ◽  
pp. 1-12
Author(s):  
Yingwen Fu ◽  
Nankai Lin ◽  
Xiaotian Lin ◽  
Shengyi Jiang
Keyword(s):  
Language Processing ◽  
State Of The Art ◽  
Named Entity Recognition ◽  
Entity Recognition ◽  
Language Models ◽  
Neural Models ◽  
Performance Models ◽  
Named Entity ◽  
High Resource ◽  
Benchmark Datasets

Named entity recognition (NER) is fundamental to natural language processing (NLP). Most state-of-the-art researches on NER are based on pre-trained language models (PLMs) or classic neural models. However, these researches are mainly oriented to high-resource languages such as English. While for Indonesian, related resources (both in dataset and technology) are not yet well-developed. Besides, affix is an important word composition for Indonesian language, indicating the essentiality of character and token features for token-wise Indonesian NLP tasks. However, features extracted by currently top-performance models are insufficient. Aiming at Indonesian NER task, in this paper, we build an Indonesian NER dataset (IDNER) comprising over 50 thousand sentences (over 670 thousand tokens) to alleviate the shortage of labeled resources in Indonesian. Furthermore, we construct a hierarchical structured-attention-based model (HSA) for Indonesian NER to extract sequence features from different perspectives. Specifically, we use an enhanced convolutional structure as well as an enhanced attention structure to extract deeper features from characters and tokens. Experimental results show that HSA establishes competitive performance on IDNER and three benchmark datasets.

scholarly journals Efficient Embedded Decoding of Neural Network Language Models in a Machine Translation System

2018 ◽  
Vol 28 (09) ◽  
pp. 1850007
Author(s):  
Francisco Zamora-Martinez ◽  
Maria Jose Castro-Bleda
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Language Processing ◽  
Traditional Approach ◽  
Computational Cost ◽  
Integrated Approach ◽  
Language Models ◽  
Translation System ◽  
Neural Net ◽  
Network Language

Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on [Formula: see text]-best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and [Formula: see text]-gram-based systems, showing that the integrated approach seems more promising for [Formula: see text]-gram-based systems, even with nonfull-quality NNLMs.

Sign in / Sign up

Export Citation Format

Share Document