SwitchNet: Learning to switch for word-level language identification in code-mixed social media text

2021 ◽  
pp. 1-23
Author(s):  
Neelakshi Sarma ◽  
Ranbir Sanasam Singh ◽  
Diganta Goswami
Keyword(s):  
Social Media ◽  
Contextual Information ◽  
Language Identification ◽  
Local Context ◽  
Important Indicator ◽  
External Resources ◽  
Switching Mechanism ◽  
Word Level ◽  
Multiple Languages ◽  
Language Classification

Abstract Word-level language identification is an essential prerequisite for extracting useful information from code-mixed social media content. Previous studies in word-level language identification show two important observations. First, the local context is an important indicator of the language of a word when a word is valid in multiple languages. Second, considering the word in isolation from its context leads to more effective language classification when a word is borrowed or embedded into sentences of other languages. In this paper, we propose a framework for language identification that makes use of a dynamic switching mechanism for effective language classification of both words that are borrowed or embedded from other languages as well as words that are valid in multiple languages. For a given input, the proposed switching mechanism makes a dynamic decision to bias its prediction either towards the prediction obtained by the contextual information or that obtained by the word in isolation. In contrast to existing studies that rely upon large amounts of annotated data for robust performance in a multilingual environment, the proposed approach uses minimal annotated resources and no external resources, making it easily extendible to newer languages. Evaluation over a corpus of transliterated Facebook comments shows that the proposed approach outperforms its baseline counterparts: classification based on the contextual information, classification based on the word in isolation, as well as an ensemble of the two classifiers.

scholarly journals Deep Learning-Based Language Identification in English-Hindi-Bengali Code-Mixed Social Media Corpora

2019 ◽  
Vol 28 (3) ◽  
pp. 399-408 ◽  
Author(s):  
Anupam Jamatia ◽  
Amitava Das ◽  
Björn Gambäck
Keyword(s):  
Social Media ◽  
Deep Learning ◽  
Conditional Random Fields ◽  
Short Term Memory ◽  
Language Identification ◽  
Code Mixing ◽  
Word Level ◽  
Social Media Text ◽  
Feature Based ◽  
Bidirectional Lstm

Abstract This article addresses language identification at the word level in Indian social media corpora taken from Facebook, Twitter and WhatsApp posts that exhibit code-mixing between English-Hindi, English-Bengali, as well as a blend of both language pairs. Code-mixing is a fusion of multiple languages previously mainly associated with spoken language, but which social media users also deploy when communicating in ways that tend to be rather casual. The coarse nature of code-mixed social media text makes language identification challenging. Here, the performance of deep learning on this task is compared to feature-based learning, with two Recursive Neural Network techniques, Long Short Term Memory (LSTM) and bidirectional LSTM, being contrasted to a Conditional Random Fields (CRF) classifier. The results show the deep learners outscoring the CRF, with the bidirectional LSTM demonstrating the best language identification performance.

scholarly journals DAViS: a unified solution for data collection, analyzation, and visualization in real-time stock market prediction

2021 ◽  
Vol 7 (1) ◽  
Author(s):  
Suppawong Tuarob ◽  
Poom Wettayakorn ◽  
Ponpat Phetchai ◽  
Siripong Traivijitkhun ◽  
Sunghoon Lim ◽  
...  
Keyword(s):  
Machine Learning ◽  
Social Media ◽  
Data Collection ◽  
Stock Market ◽  
Real Time ◽  
Stock Prices ◽  
Contextual Information ◽  
Online News ◽  
Stock Prediction ◽  
Stock Market Prediction

AbstractThe explosion of online information with the recent advent of digital technology in information processing, information storing, information sharing, natural language processing, and text mining techniques has enabled stock investors to uncover market movement and volatility from heterogeneous content. For example, a typical stock market investor reads the news, explores market sentiment, and analyzes technical details in order to make a sound decision prior to purchasing or selling a particular company’s stock. However, capturing a dynamic stock market trend is challenging owing to high fluctuation and the non-stationary nature of the stock market. Although existing studies have attempted to enhance stock prediction, few have provided a complete decision-support system for investors to retrieve real-time data from multiple sources and extract insightful information for sound decision-making. To address the above challenge, we propose a unified solution for data collection, analysis, and visualization in real-time stock market prediction to retrieve and process relevant financial data from news articles, social media, and company technical information. We aim to provide not only useful information for stock investors but also meaningful visualization that enables investors to effectively interpret storyline events affecting stock prices. Specifically, we utilize an ensemble stacking of diversified machine-learning-based estimators and innovative contextual feature engineering to predict the next day’s stock prices. Experiment results show that our proposed stock forecasting method outperforms a traditional baseline with an average mean absolute percentage error of 0.93. Our findings confirm that leveraging an ensemble scheme of machine learning methods with contextual information improves stock prediction performance. Finally, our study could be further extended to a wide variety of innovative financial applications that seek to incorporate external insight from contextual information such as large-scale online news articles and social media data.

scholarly journals A Hybrid Approach for Transliterated Word-Level Language Identification

2015 ◽  
Author(s):  
Somnath Banerjee ◽  
Alapan Kuila ◽  
Aniruddha Roy ◽  
Sudip Kumar Naskar ◽  
Paolo Rosso ◽  
...  
Keyword(s):  
Hybrid Approach ◽  
Language Identification ◽  
Word Level

scholarly journals Context modeling based on feature models expressed as views on ontologies via mappings

2015 ◽  
Vol 12 (3) ◽  
pp. 961-977 ◽  
Author(s):  
Sinisa Neskovic ◽  
Rade Matic
Keyword(s):  
Contextual Information ◽  
Product Line ◽  
Feature Model ◽  
Local Context ◽  
Context Modeling ◽  
Feature Models ◽  
Context Model ◽  
Global Context ◽  
Software Product ◽  
Global And Local

This paper presents an approach for context modeling in complex self adapted systems consisting of many independent context-aware applications. The contextual information used for adaptation of all system applications is described by an ontology treated as a global context model. A local context model tailored to the specific needs of a particular application is defined as a view over the global context in the form of a feature model. Feature models and their configurations derived from the global context state are then used by a specific dynamic software product line in order to adapt applications at runtime. The main focus of the paper is on the realization of mappings between global and local contexts. The paper describes an overall model architecture and provides corresponding metamodels as well as rules for a mapping between feature models and ontologies.

Attention Based Context Boosted Cyberbullying Detection in Social Media

2021 ◽  
Vol 2 (4) ◽  
pp. 418-433
Author(s):  
Nabi Rezvani ◽  
Amin Beheshti
Keyword(s):  
Machine Learning ◽  
Social Media ◽  
Contextual Information ◽  
Language Model ◽  
Multiple Sources ◽  
Learner Model ◽  
Deep Architecture ◽  
Efficient Machine ◽  
Cyberbullying Detection ◽  
Textual Content

Cyberbullying detection is a rising research topic due to its paramount impact on social media users, especially youngsters and adolescents. While there has been an enormous amount of progress in utilising efficient machine learning and NLP techniques for tackling this task, recent methods have not fully addressed contextualizing the textual content to the highest possible extent. The textual content of social media posts and comments is normally long, noisy and mixed with lots of irrelevant tokens and characters, and therefore utilizing an attention-based approach that can focus on more relevant parts of the text can be quite pertinent. Moreover, social media information is normally multi-modal in nature and may contain various metadata and contextual information that can contribute to enhancing the Cyberbullying prediction system. In this research, we propose a novel machine learning method that, (i) fine tunes a variant of BERT, a deep attention-based language model, which is capable of detecting patterns in long and noisy bodies of text; (ii)~extracts contextual information from multiple sources including metadata information, images and even external knowledge sources and uses these features to complement the learner model; and (iii) efficiently combines textual and contextual features using boosting and a wide-and-deep architecture. We compare our proposed method with state-of-the-art methods and highlight how our approach significantly outperforming the quality of results compared to those methods in most cases.

Sign in / Sign up

Export Citation Format

Share Document