scholarly journals Incorporating Concreteness in Multi-Modal Language Models with Curriculum Learning

2021 ◽  
Vol 11 (17) ◽  
pp. 8241
Author(s):  
Erhan Sezerer ◽  
Selma Tekir
Keyword(s):  
Visual Information ◽  
Language Modeling ◽  
Language Models ◽  
Abstract Concepts ◽  
Modal Language ◽  
Training Approach ◽  
Proposed Model ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Modeling Representation

Over the last few years, there has been an increase in the studies that consider experiential (visual) information by building multi-modal language models and representations. It is shown by several studies that language acquisition in humans starts with learning concrete concepts through images and then continues with learning abstract ideas through the text. In this work, the curriculum learning method is used to teach the model concrete/abstract concepts through images and their corresponding captions to accomplish multi-modal language modeling/representation. We use the BERT and Resnet-152 models on each modality and combine them using attentive pooling to perform pre-training on the newly constructed dataset, which is collected from the Wikimedia Commons based on concrete/abstract words. To show the performance of the proposed model, downstream tasks and ablation studies are performed. The contribution of this work is two-fold: A new dataset is constructed from Wikimedia Commons based on concrete/abstract words, and a new multi-modal pre-training approach based on curriculum learning is proposed. The results show that the proposed multi-modal pre-training approach contributes to the success of the model.

scholarly journals Constructing Semantic Models From Words, Images, and Emojis

2020 ◽  
Author(s):  
Armand Stefan Rotaru ◽  
Gabriella Vigliocco
Keyword(s):  
Visual Information ◽  
Semantic Processing ◽  
Empirical Work ◽  
Abstract Concepts ◽  
Visual Model ◽  
Semantic Data ◽  
Text Corpora ◽  
Abstract Words ◽  
Best Fit ◽  
Better Than

A number of recent models of semantics combine linguistic information, derived from text corpora, and visual information, derived from image collections, demonstrating that the resulting multimodal models are better than either of their unimodal counterparts, in accounting for behavioural data. Empirical work on semantic processing has shown that emotion also plays an important role especially for abstract concepts, however, models integrating emotion along with linguistic and visual information are lacking. Here, we first improve on visual and affective representations, derived from state-of-the-art existing models, by choosing models that best fit available human semantic data and extending the number of concepts they cover. Crucially then, we assess whether adding affective representations (obtained from a neural network model designed to predict emojis from co-occurring text) improves the model’s ability to fit semantic similarity/relatedness judgements from a purely linguistic and linguistic-visual model. We find that, given specific weights assigned to the models, adding both visual and affective representations improve performance, with visual representations providing an improvement especially for more concrete words, and affective representations improving especially the fit for more abstract words.

EXPRESS: Articulatory suppression delays processing of abstract words: the role of inner speech

2021 ◽  
pp. 174702182110536
Author(s):  
Chiara Fini ◽  
Gian Daniele Zannino ◽  
Matteo Orsoni ◽  
Giovanni Augusto Carlesimo ◽  
Mariagrazia Benassi ◽  
...  
Keyword(s):  
Articulatory Suppression ◽  
Judgment Task ◽  
Inner Speech ◽  
Dominant Hand ◽  
Conceptual Representation ◽  
Abstract Concepts ◽  
Single Object ◽  
Abstract Words ◽  
Concrete Concepts

Compared to concrete concepts, like “book”, abstract concepts expressed by words like “justice” are more detached from sensorial experiences, even though they are also grounded in sensorial modalities. Abstract concepts lack a single object as referent and are characterized by higher variability both within and across participants. According to the Word as Social Tool (WAT) proposal, owing to their complexity, abstract concepts need to be processed with the help of inner language. Inner language can namely help participants to re-explain to themselves the meaning of the word, to keep information active in working memory, and to prepare themselves to ask information from more competent people. While previous studies have demonstrated that the mouth is involved during abstract concepts’ processing, both the functional role and the mechanisms underlying this involvement still need to be clarified. We report an experiment in which participants were required to evaluate whether 78 words were abstract or concrete by pressing two different pedals. During the judgment task, they were submitted, in different blocks, to a baseline, an articulatory suppression, and a manipulation condition. In the last two conditions, they had to repeat a syllable continually and to manipulate a softball with their dominant hand. Results showed that articulatory suppression slowed down the processing of abstract more than that of concrete words. Overall results confirm the WAT proposal’s hypothesis that abstract concepts processing involves the mouth motor system and specifically inner speech. We discuss the implications for current theories of conceptual representation.

scholarly journals Cortical Representations of Concrete and Abstract Concepts in Language Combine Visual and Linguistic Representations

2021 ◽  
Author(s):  
Jerry Tang ◽  
Amanda LeBel ◽  
Alexander G Huth
Keyword(s):  
Visual Cortex ◽  
Visual Information ◽  
Semantic Representations ◽  
Abstract Concepts ◽  
Linguistic Processing ◽  
Linguistic Information ◽  
Linguistic Representations ◽  
Cortical Regions ◽  
Concrete Concepts ◽  
Perceptual Systems

The human semantic system stores knowledge acquired through both perception and language. To study how semantic representations in cortex integrate perceptual and linguistic information, we created semantic word embedding spaces that combine models of visual and linguistic processing. We then used these visually-grounded semantic spaces to fit voxelwise encoding models to fMRI data collected while subjects listened to hours of narrative stories. We found that cortical regions near the visual system represent concepts by combining visual and linguistic information, while regions near the language system represent concepts using mostly linguistic information. Assessing individual representations near visual cortex, we found that more concrete concepts contain more visual information, while even abstract concepts contain some amount of visual information from associated concrete concepts. Finally we found that these visual grounding effects are localized near visual cortex, suggesting that semantic representations specifically reflect the modality of adjacent perceptual systems. Our results provide a computational account of how visual and linguistic information are combined to represent concrete and abstract concepts across cortex.

scholarly journals Abstract, emotional and concrete concepts and the activation of mouth-hand effectors.

2018 ◽  
Author(s):  
Claudia Mazzuca ◽  
Luisa Lugli ◽  
Roberto Nicoletti ◽  
Anna M Borghi
Keyword(s):  
Lexical Decision ◽  
Recognition Task ◽  
Decision Task ◽  
Abstract Concepts ◽  
Emotional Words ◽  
Processing Level ◽  
Concreteness Effect ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Emotional Concepts

According to embodied and grounded theories concepts are grounded in sensorimotor systems. The majority of evidence supporting these views concerns concepts referring to objects or actions, while evidence on abstract concepts is more scarce. Explaining how abstract concepts, as “freedom”, are represented, would however be pivotal for grounded theories. According to some recent proposals, abstract concepts are grounded evoking both sensorimotor and linguistic experience, thus activating the mouth motor system more than concrete concepts. Two experiments are reported, aimed at verifying whether abstract, concrete and emotional words activate the mouth and hand effectors. In both experiments participants performed first a lexical decision, then a recognition task. In Experiment 1 participants responded by pressing a button either with the mouth or with the hand, in Experiment 2 responses were given with the foot, while a button held either in the mouth or in the hand was used to respond to catch-trials. Abstract words were slower to process in both tasks (concreteness effect). Across the tasks and experiments, emotional concepts had instead a fluctuating pattern, different from those of both concrete and abstract concepts, suggesting that they cannot be considered as a subset of abstract concepts. The interaction between kind of concept (abstract, concrete and emotional) and effector (mouth, hand) was not significant in the lexical decision task, likely because it emerged only with tasks implying a deeper processing level. It reached significance, instead, in the accuracy analyses of the recognition tasks. In both experiments abstract concepts yielded less errors in the mouth than in the hand condition, supporting our main prediction. Emotional concepts had instead a more variable pattern. Overall, our findings indicate that different kinds of concepts differently activate the mouth and hand effectors, but they also suggests that concepts activate effectors in a flexible and task-dependent way.

scholarly journals Visual and Affective Grounding in Language and Mind

2018 ◽  
Author(s):  
Simon De Deyne ◽  
Danielle Navarro ◽  
Guillem Collell ◽  
Amy Perfors
Keyword(s):  
Mental Representations ◽  
Word Meaning ◽  
Language Models ◽  
Abstract Concepts ◽  
Text Corpora ◽  
Semantic Models ◽  
Abstract Words ◽  
Distributional Semantic Models ◽  
Affective Information ◽  
New Evidence

One of the main limitations in natural language-based approaches to meaning is that they are not grounded. In this study, we evaluate how well different kinds of models account for people’s representations of both concrete and abstract concepts. The models are both unimodal (language-based only) models and multimodal distributional semantic models (which additionallyincorporate perceptual and/or affective information). The language-based models include both external (based on text corpora) and internal (derived from word associations) language. We present two new studies and a re-analysis of a series of previous studies demonstrating that the unimodal performance is substantially higher for internal models, especially when comparisons at the basiclevel are considered. For multimodal models, our findings suggest that additional visual and affective features lead to only slightly more accurate mental representations of word meaning than what is already encoded in internal language models; however, for abstract concepts, visual andaffective features improve the predictions of external text-based models. Our work presents new evidence that the grounding problem includes abstract words as well and is therefore more widespread than previously suggested. Implications for both embodied and distributional views arediscussed.

scholarly journals From Affordances to Abstract Words: The Flexibility of Sensorimotor Grounding

2021 ◽  
Vol 11 (10) ◽  
pp. 1304
Author(s):  
Claudia Mazzuca ◽  
Chiara Fini ◽  
Arthur Henri Michalland ◽  
Ilenia Falcinelli ◽  
Federico Da Rold ◽  
...  
Keyword(s):  
Critical Role ◽  
Future Research ◽  
Sensorimotor System ◽  
Abstract Concepts ◽  
Word Use ◽  
Linguistic System ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Sensorimotor Mechanisms ◽  
Object Interactions

The sensorimotor system plays a critical role in several cognitive processes. Here, we review recent studies documenting this interplay at different levels. First, we concentrate on studies that have shown how the sensorimotor system is flexibly involved in interactions with objects. We report evidence demonstrating how social context and situations influence affordance activation, and then focus on tactile and kinesthetic components in body–object interactions. Then, we turn to word use, and review studies that have shown that not only concrete words, but also abstract words are grounded in the sensorimotor system. We report evidence that abstract concepts activate the mouth effector more than concrete concepts, and discuss this effect in light of studies on adults, children, and infants. Finally, we pinpoint possible sensorimotor mechanisms at play in the acquisition and use of abstract concepts. Overall, we show that the involvement of the sensorimotor system is flexibly modulated by context, and that its role can be integrated and flanked by that of other systems such as the linguistic system. We suggest that to unravel the role of the sensorimotor system in cognition, future research should fully explore the complexity of this intricate, and sometimes slippery, relation.

scholarly journals Modelling concrete and abstract concepts using brain-constrained deep neural networks

2021 ◽  
Author(s):  
Malte R. Henningsen-Schomers ◽  
Friedemann Pulvermüller
Keyword(s):  
Current Knowledge ◽  
Semantic Representation ◽  
Family Resemblance ◽  
Semantic Feature ◽  
Abstract Concepts ◽  
Strongly Connected ◽  
Feature Sharing ◽  
Abstract Words ◽  
Feature Overlap ◽  
Concrete Concepts

AbstractA neurobiologically constrained deep neural network mimicking cortical areas relevant for sensorimotor, linguistic and conceptual processing was used to investigate the putative biological mechanisms underlying conceptual category formation and semantic feature extraction. Networks were trained to learn neural patterns representing specific objects and actions relevant to semantically ‘ground’ concrete and abstract concepts. Grounding sets consisted of three grounding patterns with neurons representing specific perceptual or action-related features; neurons were either unique to one pattern or shared between patterns of the same set. Concrete categories were modelled as pattern triplets overlapping in their ‘shared neurons’, thus implementing semantic feature sharing of all instances of a category. In contrast, abstract concepts had partially shared feature neurons common to only pairs of category instances, thus, exhibiting family resemblance, but lacking full feature overlap. Stimulation with concrete and abstract conceptual patterns and biologically realistic unsupervised learning caused formation of strongly connected cell assemblies (CAs) specific to individual grounding patterns, whose neurons were spread out across all areas of the deep network. After learning, the shared neurons of the instances of concrete concepts were more prominent in central areas when compared with peripheral sensorimotor ones, whereas for abstract concepts the converse pattern of results was observed, with central areas exhibiting relatively fewer neurons shared between pairs of category members. We interpret these results in light of the current knowledge about the relative difficulty children show when learning abstract words. Implications for future neurocomputational modelling experiments as well as neurobiological theories of semantic representation are discussed.

scholarly journals Learning and Using Abstract Words: Evidence from Clinical Populations

2017 ◽  
Vol 2017 ◽  
pp. 1-8 ◽  
Author(s):  
Maria Luisa Lorusso ◽  
Michele Burigo ◽  
Alessandro Tavano ◽  
Anna Milani ◽  
Sara Martelli ◽  
...  
Keyword(s):  
Down Syndrome ◽  
Williams Syndrome ◽  
Cognitive Profile ◽  
Typically Developing ◽  
Abstract Concepts ◽  
Visual Spatial ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Concrete Words

It has been shown that abstract concepts are more difficult to process and are acquired later than concrete concepts. We analysed the percentage of concrete words in the narrative lexicon of individuals with Williams Syndrome (WS) as compared to individuals with Down Syndrome (DS) and typically developing (TD) peers. The cognitive profile of WS is characterized by visual-spatial difficulties, while DS presents with predominant impairments in linguistic abilities. We predicted that if linguistic abilities are crucial to the development and use of an abstract vocabulary, DS participants should display a higher concreteness index than both Williams Syndrome and typically developing individuals. Results confirm this prediction, thus supporting the hypothesis of a crucial role of linguistic processes in abstract language acquisition. Correlation analyses suggest that a maturational link exists between the level of abstractness in narrative production and syntactic comprehension.

scholarly journals Abstract, emotional and concrete concepts and the activation of mouth-hand effectors.

2018 ◽  
Author(s):  
Claudia Mazzuca ◽  
Luisa Lugli ◽  
Roberto Nicoletti ◽  
Anna M Borghi
Keyword(s):  
Lexical Decision ◽  
Recognition Task ◽  
Decision Task ◽  
Abstract Concepts ◽  
Emotional Words ◽  
Processing Level ◽  
Concreteness Effect ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Emotional Concepts

According to embodied and grounded theories concepts are grounded in sensorimotor systems. The majority of evidence supporting these views concerns concepts referring to objects or actions, while evidence on abstract concepts is more scarce. Explaining how abstract concepts, as “freedom”, are represented, would however be pivotal for grounded theories. According to some recent proposals, abstract concepts are grounded evoking both sensorimotor and linguistic experience, thus activating the mouth motor system more than concrete concepts. Two experiments are reported, aimed at verifying whether abstract, concrete and emotional words activate the mouth and hand effectors. In both experiments participants performed first a lexical decision, then a recognition task. In Experiment 1 participants responded by pressing a button either with the mouth or with the hand, in Experiment 2 responses were given with the foot, while a button held either in the mouth or in the hand was used to respond to catch-trials. Abstract words were slower to process in both tasks (concreteness effect). Across the tasks and experiments, emotional concepts had instead a fluctuating pattern, different from those of both concrete and abstract concepts, suggesting that they cannot be considered as a subset of abstract concepts. The interaction between kind of concept (abstract, concrete and emotional) and effector (mouth, hand) was not significant in the lexical decision task, likely because it emerged only with tasks implying a deeper processing level. It reached significance, instead, in the accuracy analyses of the recognition tasks. In both experiments abstract concepts yielded less errors in the mouth than in the hand condition, supporting our main prediction. Emotional concepts had instead a more variable pattern. Overall, our findings indicate that different kinds of concepts differently activate the mouth and hand effectors, but they also suggests that concepts activate effectors in a flexible and task-dependent way.

scholarly journals Abstract, emotional and concrete concepts and the activation of mouth-hand effectors

PeerJ ◽  
2018 ◽  
Vol 6 ◽  
pp. e5987 ◽  
Author(s):  
Claudia Mazzuca ◽  
Luisa Lugli ◽  
Mariagrazia Benassi ◽  
Roberto Nicoletti ◽  
Anna M. Borghi
Keyword(s):  
Lexical Decision ◽  
Recognition Task ◽  
Decision Task ◽  
Abstract Concepts ◽  
Emotional Words ◽  
Processing Level ◽  
Concreteness Effect ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Emotional Concepts

According to embodied and grounded theories, concepts are grounded in sensorimotor systems. The majority of evidence supporting these views concerns concepts referring to objects or actions, while evidence on abstract concepts is more scarce. Explaining how abstract concepts such as “freedom” are represented would thus be pivotal for grounded theories. According to some recent proposals, abstract concepts are grounded in both sensorimotor and linguistic experience, thus they activate the mouth motor system more than concrete concepts. Two experiments are reported, aimed at verifying whether abstract, concrete and emotional words activate the mouth and the hand effectors. In both experiments participants performed first a lexical decision, then a recognition task. In Experiment 1 participants responded by pressing a button either with the mouth or with the hand, in Experiment 2 responses were given with the foot, while a button held either in the mouth or in the hand was used to respond to catch-trials. Abstract words were slower to process in both tasks (concreteness effect). Across the tasks and experiments, emotional concepts had instead a fluctuating pattern, different from those of both concrete and abstract concepts, suggesting that they cannot be considered as a subset of abstract concepts. The interaction between type of concept (abstract, concrete and emotional) and effector (mouth, hand) was not significant in the lexical decision task, likely because it emerged only with tasks implying a deeper processing level. It reached significance, instead, in the recognition tasks. In both experiments abstract concepts were facilitated in the mouth condition compared to the hand condition, supporting our main prediction. Emotional concepts instead had a more variable pattern. Overall, our findings indicate that various kinds of concepts differently activate the mouth and hand effectors, but they also suggest that concepts activate effectors in a flexible and task-dependent way.

Sign in / Sign up

Export Citation Format

Share Document