Multimodal Data Evaluation for Classification Problems

Social media data is currently the main input to a wide variety of research works in many knowledge fields. This kind of data is generally multimodal, i.e., it contains different modalities of information such as text, images, video or audio, mainly. To deal with multimodal data to tackle a specific task could be very difficult. One of the main challenges is to find useful representations of the data, capable of capturing the subtle information that the users who generate that information provided, or even the way they use it. In this paper, we analysed the usage of two modalities of data, images, and text, both in a separate way and by combining them to address two classification problems: meme's classification and user profiling. For images, we use a textual semantic representation by using a pre-trained model of image captioning. Later, a text classifier based on optimal lexical representations was used to build a classification model. Interesting findings were found in the usage of these two modalities of data, and the pros and cons of using them to solve the two classification problems are also discussed.

Fuzzy Lexical Representations in Adult Second Language Speakers

We propose the fuzzy lexical representations (FLRs) hypothesis that regards fuzziness as a core property of nonnative (L2) lexical representations (LRs). Fuzziness refers to imprecise encoding at different levels of LRs and interacts with input frequency during lexical processing and learning in adult L2 speakers. The FLR hypothesis primarily focuses on the encoding of spoken L2 words. We discuss the causes of fuzzy encoding of phonological form and meaning as well as fuzzy form-meaning mappings and the consequences of fuzzy encoding for word storage and retrieval. A central factor contributing to the fuzziness of L2 LRs is the fact that the L2 lexicon is acquired when the L1 lexicon is already in place. There are two immediate consequences of such sequential learning. First, L2 phonological categorization difficulties lead to fuzzy phonological form encoding. Second, the acquisition of L2 word forms subsequently to their meanings, which had already been acquired together with the L1 word forms, leads to weak L2 form-meaning mappings. The FLR hypothesis accounts for a range of phenomena observed in L2 lexical processing, including lexical confusions, slow lexical access, retrieval of incorrect lexical entries, weak lexical competition, reliance on sublexical rather than lexical heuristics in word recognition, the precedence of word form over meaning, and the prominence of detailed, even if imprecisely encoded, information about LRs in episodic memory. The main claim of the FLR hypothesis – that the quality of lexical encoding is a product of a complex interplay between fuzziness and input frequency – can contribute to increasing the efficiency of the existing models of LRs and lexical access.

The effect of lexical content on sentence production in nonfluent aphasia

<p>Individuals with nonfluent aphasia are able to produce many words in isolation, but have great difficulty producing sentences. Most research to date has compared accuracy across different types of sentence structures, focussing on grammatical aspects that may be compromised in nonfluent aphasia. However, based on the premise that lexical elements activate their associated grammatical frames as well as vice versa, lexical content may also be of vital importance. For example, rapid access to lexical elements – particularly ones appearing early in the sentence - may be crucial, especially if the sentence plan is weakly activated or rapidly decaying. The current study investigated the effect of different aspects of lexical content on nonfluent aphasic sentence production. Five participants with nonfluent aphasia, four participants with fluent aphasia and eight controls completed two picture description tasks eliciting subject-verb-object sentences (e.g., the dog is chasing the fox). Based on existing evidence suggesting that common words are accessed more rapidly than rarer ones, Experiment 1 manipulated the frequency of sentence nouns, thereby varying their speed of lexical retrieval by varying the frequency of sentence nouns. Nonfluent participants' accuracy was consistently higher for sentences commencing with a high frequency subject noun, even when errors on those nouns were themselves excluded. This was not the case for the fluent participants. Experiment 2 manipulated the semantic relationship between subject and object nouns. Previous research suggests that phrases containing related words may be challenging for individuals with nonfluent aphasia, possibly because lexical representations are inadequately tied to appropriate structural representations. The nonfluent participants produced sentences less accurately when they contained related lexical items, even when those items were in different noun phrases. The fluent participants exhibited the opposite trend. Finally, the relationship between the patterns observed in Experiment 1 and 2 and lesion location in the aphasic participants was explored by analysing magnetic resonance scans. We discuss the implications of our findings for theoretical accounts of sentence production more generally, and of nonfluent aphasia in particular. More precisely, we propose that individuals with nonfluent aphasia are disproportionately reliant on activated lexical representations to drive the sentence generation process, an idea we call the Content Drives Structure (COST) hypothesis.</p>

The effect of lexical content on sentence production in nonfluent aphasia

<p>Individuals with nonfluent aphasia are able to produce many words in isolation, but have great difficulty producing sentences. Most research to date has compared accuracy across different types of sentence structures, focussing on grammatical aspects that may be compromised in nonfluent aphasia. However, based on the premise that lexical elements activate their associated grammatical frames as well as vice versa, lexical content may also be of vital importance. For example, rapid access to lexical elements – particularly ones appearing early in the sentence - may be crucial, especially if the sentence plan is weakly activated or rapidly decaying. The current study investigated the effect of different aspects of lexical content on nonfluent aphasic sentence production. Five participants with nonfluent aphasia, four participants with fluent aphasia and eight controls completed two picture description tasks eliciting subject-verb-object sentences (e.g., the dog is chasing the fox). Based on existing evidence suggesting that common words are accessed more rapidly than rarer ones, Experiment 1 manipulated the frequency of sentence nouns, thereby varying their speed of lexical retrieval by varying the frequency of sentence nouns. Nonfluent participants' accuracy was consistently higher for sentences commencing with a high frequency subject noun, even when errors on those nouns were themselves excluded. This was not the case for the fluent participants. Experiment 2 manipulated the semantic relationship between subject and object nouns. Previous research suggests that phrases containing related words may be challenging for individuals with nonfluent aphasia, possibly because lexical representations are inadequately tied to appropriate structural representations. The nonfluent participants produced sentences less accurately when they contained related lexical items, even when those items were in different noun phrases. The fluent participants exhibited the opposite trend. Finally, the relationship between the patterns observed in Experiment 1 and 2 and lesion location in the aphasic participants was explored by analysing magnetic resonance scans. We discuss the implications of our findings for theoretical accounts of sentence production more generally, and of nonfluent aphasia in particular. More precisely, we propose that individuals with nonfluent aphasia are disproportionately reliant on activated lexical representations to drive the sentence generation process, an idea we call the Content Drives Structure (COST) hypothesis.</p>

Cross-Dialectal Novel Word Learning and Borrowing

The objective of this paper was to study the cognitive processes underlying cross-dialectal novel word borrowing and loanword establishment in a Standard-Chinese-to-Shanghainese (SC-SH) auditory lexical learning and borrowing experiment. To investigate these underlying cognitive processes, SC-SH bi-dialectals were compared with SC monolectals as well as bi-dialectals of SC and other Chinese dialects (OD) to investigate the influence of short-term and long-term linguistic experience. Both comprehension and production borrowings were tested. This study found that early and proficient bi-dialectism, even if it is not directly related to the recipient dialect of lexical borrowing, has a protective effect on the ability of cross-dialectal lexical borrowing in early adulthood. Bi-dialectals tend to add separate lexical representations for incidentally encountered dialectal variants, while monolectals tend to assimilate dialectal variants to standard forms. Bi-dialectals, but not monolectals, use etymologically related morphemes between the source and recipient dialects to create nonce-borrowing compounds. Dialectal variability facilitates lexical borrowing via enriching instead of increasing the short-term lexical experience of learners. The long-term bi-dialectal experience of individuals, as well as their short-term exposure to each specific loanword, may collectively shape the route of lexical evolution of co-evolving linguistic varieties.

Lexical and syntactic influences on syntactic encoding in written language production by deaf speakers

It remains unclear whether deaf and hearing speakers differ in the processes and representations underlying written language production. Using the structural priming paradigm, this study investigated syntactic and lexical influences on syntactic encoding in writing by deaf speakers of Chinese in comparison with hearing controls. Experiment 1 showed that deaf speakers tended to re-use a prior syntactic structure in written sentence production (i.e., structural priming) to the same extent as hearing speakers did; in addition, such a tendency was enhanced when the target sentence repeated the verb from the prime sentence (i.e., lexical boost) in both deaf and hearing speakers to the same extent. These results suggest that deaf and hearing speakers are similarly affected by syntactic and lexical factors in syntactic encoding in writing. Experiment 2 revealed comparable boosts in structural priming between prime-target pairs with homographic homophone verbs and prime-target pairs with heterographic homophone verbs in hearing speakers, but a boost for prime-target pairs with homographic homophone verbs but not those with heterographic homophone verbs in deaf speakers. These results suggest that while syntactic encoding in writing is influenced by lemma associations developed for homophones as a result of phonological identity in hearing speakers, it is influenced by lemma associations developed for homographs as a result of orthographic identity in deaf speakers. In all, syntactic encoding in writing seems to employ the same syntactic and lexical representations in hearing and deaf speakers, though lexical representations are shaped more by orthography than phonology in deaf speakers.

Does learning to read change the perception of speech? Evidence from perceptual recalibration of speech sounds in adult illiterates, semi-literates, and literates of Tamil

How do different levels of representation interact in the mind? Key evidence for answering this question comes from experimental work that investigates the influence of knowledge of written language on spoken language processing. Here we tested whether learning orthographic representations (through reading) influences pre-lexical phonological representations in spoken-word recognition using a perceptual learning paradigm. Perceptual learning is well suited to reveal differences in pre-lexical representations that might be caused by learning to read because it requires the functional use of pre-lexical representations in order to generalize a learning experience. In a large-scale behavioural study in Chennai, India, 97 native speakers of Tamil with varying reading experience (from completely illiterate to highly literate) participated. In marked contrast to their performance in other cognitive tasks, even completely illiterate participants showed a perceptual learning effect that was not moderated by reading experience. This finding suggests that pre-lexical phonological representations are not substantially changed by learning to read and thus poses important constraints for the debate about the degree of interactivity between different levels of representations during human information processing.

Vocabulary Size Is a Key Factor in Predicting Second Language Lexical Encoding Accuracy

This study investigates the relationship between the accuracy of second language lexical representations and perception, phonological short-term memory, inhibitory control, attention control, and second language vocabulary size. English-speaking learners of Spanish were tested on their lexical encoding of the Spanish /ɾ-r/, /ɾ-d/, /r-d/, and /f-p/ contrasts through a lexical decision task. Perception ability was measured with an oddity task, phonological short-term memory with a serial non-word recognition task, attention control with a flanker task, inhibitory control with a retrieval-induced inhibition task, and vocabulary size with the X_Lex vocabulary test. Results revealed that differences in perception performance, inhibitory control, and attention control were not related to differences in lexical encoding accuracy. Phonological short-term memory was a significant factor, but only for the /r-ɾ/ contrast. This suggests that when representations contain sounds that are differentiated along a dimension not used in the native language, learners with higher phonological short-term memory have an advantage because they are better able to hold the relevant phonetic details in memory long enough to be transferred to long-term representations. Second language vocabulary size predicted lexical encoding across three of the four contrasts, such that a larger vocabulary predicted greater accuracy. This is likely because the acquisition of more phonologically similar words forces learners’ phonological systems to create more detailed representations in order for such words to be differentiated. Overall, this study suggests that vocabulary size in the second language is the most important factor in the accuracy of lexical representations.