scholarly journals Constructing Semantic Models From Words, Images, and Emojis

2020 ◽  
Author(s):  
Armand Stefan Rotaru ◽  
Gabriella Vigliocco
Keyword(s):  
Visual Information ◽  
Semantic Processing ◽  
Empirical Work ◽  
Abstract Concepts ◽  
Visual Model ◽  
Semantic Data ◽  
Text Corpora ◽  
Abstract Words ◽  
Best Fit ◽  
Better Than

A number of recent models of semantics combine linguistic information, derived from text corpora, and visual information, derived from image collections, demonstrating that the resulting multimodal models are better than either of their unimodal counterparts, in accounting for behavioural data. Empirical work on semantic processing has shown that emotion also plays an important role especially for abstract concepts, however, models integrating emotion along with linguistic and visual information are lacking. Here, we first improve on visual and affective representations, derived from state-of-the-art existing models, by choosing models that best fit available human semantic data and extending the number of concepts they cover. Crucially then, we assess whether adding affective representations (obtained from a neural network model designed to predict emojis from co-occurring text) improves the model’s ability to fit semantic similarity/relatedness judgements from a purely linguistic and linguistic-visual model. We find that, given specific weights assigned to the models, adding both visual and affective representations improve performance, with visual representations providing an improvement especially for more concrete words, and affective representations improving especially the fit for more abstract words.

scholarly journals The role of emotional valence in learning novel abstract concepts

2020 ◽  
Author(s):  
Gabriella Vigliocco ◽  
Marta Ponari ◽  
Courtenay Norbury
Keyword(s):  
Teaching Strategy ◽  
Age Of Acquisition ◽  
Emotional Valence ◽  
The Other ◽  
Abstract Concepts ◽  
Emotional Information ◽  
Emotional Words ◽  
Abstract Words ◽  
Better Than

A recent study by Ponari et al. (2017), showed that emotional valence (i.e., whether a word evokes positive, negative or no affect) predicts age-of-acquisition ratings, and that up to the age of 8-9, children know abstract emotional words better than neutral ones. On the basis of these findings, emotional valence has been argued to provide a bootstrapping mechanism for the acquisition of abstract concepts. However, no previous work has directly assessed whether words’ valence, or valence of the context in which words are used, facilitates learning of unknown abstract words. Here, we investigate whether valence supports acquisition of novel abstract concepts. Seven to 10 years old children were taught novel abstract words and concepts (words typically learnt at an older age and that children did not know); words were either valenced (positive or negative) or neutral. We also manipulated the context in which words were presented: for one group of children, the teaching strategy emphasised emotional information; for the other, it emphasised encyclopaedic, non-emotional information. Abstract words with emotional valence were learnt better than neutral abstract words by children up to the age of 8-9, replicating previous findings; no effect of teaching strategy was found. These results indicate that emotional valence supports abstract concepts acquisition, and further suggest that it is the valence information intrinsic to the word’s meaning to have a role, rather than the valence of the context in which the word is learnt.

scholarly journals Visual and Affective Grounding in Language and Mind

2018 ◽  
Author(s):  
Simon De Deyne ◽  
Danielle Navarro ◽  
Guillem Collell ◽  
Amy Perfors
Keyword(s):  
Mental Representations ◽  
Word Meaning ◽  
Language Models ◽  
Abstract Concepts ◽  
Text Corpora ◽  
Semantic Models ◽  
Abstract Words ◽  
Distributional Semantic Models ◽  
Affective Information ◽  
New Evidence

One of the main limitations in natural language-based approaches to meaning is that they are not grounded. In this study, we evaluate how well different kinds of models account for people’s representations of both concrete and abstract concepts. The models are both unimodal (language-based only) models and multimodal distributional semantic models (which additionallyincorporate perceptual and/or affective information). The language-based models include both external (based on text corpora) and internal (derived from word associations) language. We present two new studies and a re-analysis of a series of previous studies demonstrating that the unimodal performance is substantially higher for internal models, especially when comparisons at the basiclevel are considered. For multimodal models, our findings suggest that additional visual and affective features lead to only slightly more accurate mental representations of word meaning than what is already encoded in internal language models; however, for abstract concepts, visual andaffective features improve the predictions of external text-based models. Our work presents new evidence that the grounding problem includes abstract words as well and is therefore more widespread than previously suggested. Implications for both embodied and distributional views arediscussed.

scholarly journals Images of the unseen: extrapolating visual representations for abstract and concrete words in a data-driven computational model

2020 ◽  
Author(s):  
Fritz Günther ◽  
Marco Alessandro Petilli ◽  
Alessandra Vergallito ◽  
Marco Marelli
Keyword(s):  
Computational Model ◽  
Perceptual Experience ◽  
Learning Experience ◽  
Mapping Function ◽  
Grounded Cognition ◽  
Data Driven ◽  
Abstract Concepts ◽  
Text Corpora ◽  
Abstract Words ◽  
Concrete Words

AbstractTheories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants’ judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don’t have.

scholarly journals Incorporating Concreteness in Multi-Modal Language Models with Curriculum Learning

2021 ◽  
Vol 11 (17) ◽  
pp. 8241
Author(s):  
Erhan Sezerer ◽  
Selma Tekir
Keyword(s):  
Visual Information ◽  
Language Modeling ◽  
Language Models ◽  
Abstract Concepts ◽  
Modal Language ◽  
Training Approach ◽  
Proposed Model ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Modeling Representation

Over the last few years, there has been an increase in the studies that consider experiential (visual) information by building multi-modal language models and representations. It is shown by several studies that language acquisition in humans starts with learning concrete concepts through images and then continues with learning abstract ideas through the text. In this work, the curriculum learning method is used to teach the model concrete/abstract concepts through images and their corresponding captions to accomplish multi-modal language modeling/representation. We use the BERT and Resnet-152 models on each modality and combine them using attentive pooling to perform pre-training on the newly constructed dataset, which is collected from the Wikimedia Commons based on concrete/abstract words. To show the performance of the proposed model, downstream tasks and ablation studies are performed. The contribution of this work is two-fold: A new dataset is constructed from Wikimedia Commons based on concrete/abstract words, and a new multi-modal pre-training approach based on curriculum learning is proposed. The results show that the proposed multi-modal pre-training approach contributes to the success of the model.

scholarly journals Images of the unseen: Extrapolating visual representations for abstract and concrete words in a data-driven computational model

2020 ◽  
Author(s):  
Fritz Guenther ◽  
Marco Alessandro Petilli ◽  
Alessandra Vergallito ◽  
Marco Marelli
Keyword(s):  
Computational Model ◽  
Perceptual Experience ◽  
Learning Experience ◽  
Mapping Function ◽  
Grounded Cognition ◽  
Data Driven ◽  
Abstract Concepts ◽  
Text Corpora ◽  
Abstract Words ◽  
Concrete Words

Theories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants' judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don't have.

Distinct Brain Systems for Processing Concrete and Abstract Concepts

2005 ◽  
Vol 17 (6) ◽  
pp. 905-917 ◽  
Author(s):  
J. R. Binder ◽  
C. F. Westbury ◽  
K. A. McKiernan ◽  
E. T. Possing ◽  
D. A. Medler
Keyword(s):  
Conceptual Knowledge ◽  
Word Processing ◽  
Semantic Processing ◽  
Verbal Working Memory ◽  
Concrete Word ◽  
Angular Gyrus ◽  
Abstract Concepts ◽  
Central Interest ◽  
Abstract Words ◽  
Concrete Words

Behavioral and neurophysiological effects of word imageability and concreteness remain a topic of central interest in cognitive neuroscience and could provide essential clues for understanding how the brain processes conceptual knowledge. We examined these effects using event-related functional magnetic resonance imaging while participants identified concrete and abstract words. Relative to nonwords, concrete and abstract words both activated a left-lateralized network of multimodal association areas previously linked with verbal semantic processing. Areas in the left lateral temporal lobe were equally activated by both word types, whereas bilateral regions including the angular gyrus and the dorsal prefrontal cortex were more strongly engaged by concrete words. Relative to concrete words, abstract words activated left inferior frontal regions previously linked with phonological and verbal working memory processes. The results show overlapping but partly distinct neural systems for processing concrete and abstract concepts, with greater involvement of bilateral association areas during concrete word processing, and processing of abstract concepts almost exclusively by the left hemisphere.

scholarly journals That means something to me: the effect of linguistic and emotional experience on the acquisition and processing of novel abstract concepts

2021 ◽  
Author(s):  
Linda Espey ◽  
Marta Ghio ◽  
Christian Bellebaum ◽  
Laura Bechtold
Keyword(s):  
Semantic Processing ◽  
Emotional Experience ◽  
Future Research ◽  
Imagery Task ◽  
The Novel ◽  
Abstract Concepts ◽  
Emotional Information ◽  
Semantic Judgment ◽  
Electrophysiological Measurements ◽  
Abstract Words

This study aimed to investigate the acquisition and representation of novel abstract concepts grounded in linguistic and emotional experience. In five linguistic training sessions, participants learned emotional and neutral abstract concepts and either engaged in explicit mental imagery (n = 32) or lexico-semantic processing (n = 34) with the linguistic material. After training, a high lexical decision and semantic judgment accuracy showed that participants successfully acquired the novel concepts. A feature production task showed that emotional concepts were generally enriched by a surplus of (emotion) features, and neutral concepts by relatively more cognition features. More features led to a higher LDT accuracy but not to faster reactions, which cannot be fully explained by noise due to the online assessment as, descriptively, more features slowed down reactions in participants, who did the imagery task and accelerated them for those, who did the rephrasing task. Our findings support the notion that linguistic and emotional information are crucial for grounding abstract concepts, and that this grounding does not rely on explicit imagery. Future research might combine the linguistic training paradigm with controlled reaction time and electrophysiological measurements to further corroborate the experiential grounding of abstract words.

scholarly journals Emotion and concreteness effects when learning novel concepts in the native language

2021 ◽  
Vol 42 (2) ◽  
pp. 177-191
Author(s):  
Marc Guasch ◽  
Pilar Ferré
Keyword(s):  
Native Language ◽  
Recognition Task ◽  
Intentional Learning ◽  
The Novel ◽  
Abstract Concepts ◽  
Word Forms ◽  
Abstract Words ◽  
Concrete Concepts ◽  
Concrete Words ◽  
Better Than

Abstract The aim of the present study was to test the proposal of Kousta et al. (2011), according to which abstract words are more affectively loaded than concrete words. To this end, we focused on the acquisition of novel concepts by means of an intentional learning experiment in which participants had to learn a set of 40 novel concepts in Spanish (definitions) associated with novel word forms (pseudowords). Concreteness (concrete vs. abstract concepts) and emotionality (neutral vs. negative concepts) were orthogonally manipulated. Acquisition was assessed through a recognition task in which participants were asked to match the novel word forms with their definitions. Results showed that concrete concepts were acquired better than abstract concepts. Importantly, the concreteness advantage disappeared when the content of the concept was negative. Hence, emotional (negative) content facilitated the acquisition of abstract concepts, but not of concrete concepts, giving support to the proposal of Kousta et al. (2011).

THE CONCEPTUALIZATION OF EMOTIONS AND AXIOLOGICAL CONCEPTS IN SLAVONIC PROVERBS AND IDIOMS: FROM CONSCIENCE TO ENVY

2019 ◽  
Vol 2019 (29) ◽  
pp. 248-271
Author(s):  
Oleh Tyshchenko
Keyword(s):  
Cultural Anthropology ◽  
Legal Aspect ◽  
English Text ◽  
Folk Culture ◽  
Cultural Concepts ◽  
Text Corpora ◽  
Psychic Disorders ◽  
Body And Soul ◽  
Germanic Languages ◽  
Better Than

The presented research reveals imagery-metaphoric and phraseological objectivities of the conceptual spheres Soul, Consciousness, Envy, Jealousy and Greed in Polish, Russian, Ukrainian, Czech and Slovak languages and conceptual picture of the world (first of all in proverbs and sayings, idioms, imagery means of secondary nomination both in standard language and its regional or dialectal variants) according to the indication of holistic characteristic and semantic intersection of these concepts. It describes the spheres of their typological coincidence and differences from the point of imagery motivation. It defines the symbolic functions of these ethno cultural concepts (object sphere) with respect to the specificity of manifestation of Envy in archaic texts, believes, in the language of traditional folk culture and archaic expressions with religious sense that reach Christian ideology, ideas of moral purity and dirt, Body and Soul. It has been defined the collocations with the components envy and jealousy in some thesauri and dictionaries in terms of the specificity of interlingual equivalence and expressions of envy and similar negative emotions and their functioning in the Ukrainian and English text corpora. The analysis demonstrated that practically in all compared languages and linguistic cultures Envy is associated with greed and jealousy, psychic disorders with a corresponding complex of feelings, expressed by metaphoric predicates of destruction and remorse that encode the moral and legal aspect of conscience (conscience is a judge, witness and executioner). Metaphor of Envy containing nominations of colours differ in the Slavonic and Germanic languages whereas those denoting spatial, gustatory, odour, acoustic and parametrical meaning are similar. Many imagery contexts of Envy correlate with such conceptual oppositions as richness and poverty, light and darkness; success is associated with the frames “foreign is better than domestic” where Envy encodes the meaning of encroachment upon another's property, “envy is better than sympathy”, “envy dominates where there are richness, success, welfare, happiness” which confirms the ideas of representatives in the field of psychoanalysis, cultural anthropology and sociology. In some languages the motives of black magic, evil eye (in Polish, Ukrainian and Russian) are rooted in the sphere of folk believes and invocations, as well as cultural anthroponyms.

scholarly journals Functional Neuroanatomy of Contextual Acquisition of Concrete and Abstract Words

2009 ◽  
Vol 21 (11) ◽  
pp. 2154-2171 ◽  
Author(s):  
Anna Mestres-Missé ◽  
Thomas F. Münte ◽  
Antoni Rodriguez-Fornells
Keyword(s):  
Native Speakers ◽  
Brain Regions ◽  
Functional Neuroanatomy ◽  
Linguistic Context ◽  
Abstract Concepts ◽  
New Words ◽  
Simulation Paradigm ◽  
Abstract Words ◽  
The Brain ◽  
Concrete Words

The meaning of a novel word can be acquired by extracting it from linguistic context. Here we simulated word learning of new words associated to concrete and abstract concepts in a variant of the human simulation paradigm that provided linguistic context information in order to characterize the brain systems involved. Native speakers of Spanish read pairs of sentences in order to derive the meaning of a new word that appeared in the terminal position of the sentences. fMRI revealed that learning the meaning associated to concrete and abstract new words was qualitatively different and recruited similar brain regions as the processing of real concrete and abstract words. In particular, learning of new concrete words selectively boosted the activation of the ventral anterior fusiform gyrus, a region driven by imageability, which has previously been implicated in the processing of concrete words.

Sign in / Sign up

Export Citation Format

Share Document