scholarly journals Twitter data analysis using hadoop ecosystems and apache zeppelin

2019 ◽  
Vol 16 (3) ◽  
pp. 1490
Author(s):  
Stanly Wilson ◽  
Sivakumar R
Keyword(s):  
Streaming Data ◽  
Data Streaming ◽  
Text Data ◽  
Twitter Data ◽  
The People ◽  
Textual Data ◽  
Hadoop Distributed File System ◽  
Twitter Data Analysis ◽  
Apache Pig ◽  
Small Industries

The day-to-day life of the people doesn't depend only on what they think, but it is affected and influenced by what others think. The advertisements and campaigns of the favourite celebrities and mesmerizing personalities influence the way people think and see the world. People get the news and information at lightning speed than ever before. The growth of textual data on the internet is very fast. People express themselves in various ways on the web every minute. They make use of various platforms to share their views and opinions. A huge amount of data is being generated at every moment on this process. Being one of the most important and well-known social media of the present time, millions of tweets are posted on Twitter every day. These tweets are a source of very important information and it can be made use for business, small industries, creating government policies, and various studies can be performed by using it. This paper focuses on the location from where the tweets are posted and the language in which the tweets are written. These details can be effectively analysed by using Hadoop. Hadoop is a tool that is used to analyze distributed big data, streaming data, timestamp data and text data. With the help of Apache Flume, the tweets can be collected from Twitter and then sink in the HDFS (Hadoop Distributed File System). These raw data then analyzed using Apache Pig and the information available can be made use for social and commercial purposes. The result will be visualized using Apache Zeppelin.

scholarly journals An Approach To Twitter Sentiment Analysis Over Hadoop

2018 ◽  
Vol 7 (4.5) ◽  
pp. 374
Author(s):  
Yazala Ritika Siril Paul ◽  
Dilipkumar A. Borikar
Keyword(s):  
Sentiment Analysis ◽  
Opinion Mining ◽  
Emotional State ◽  
Streaming Data ◽  
Data Streaming ◽  
Text Data ◽  
Twitter Data ◽  
The People ◽  
Data Platform ◽  
Apache Hive

Sentiment analysis is the process of identifying people’s attitude and emotional state from the language they use via any social websites or other sources. The main aim is to identify a set of potential features in the review and extract the opinion expressions of those features by making full use of their associations. The Twitter has now become a routine for the people around the world to post thousands of reactions and opinions on every topic, every second of every single day. It’s like one big psychological database that’s constantly being updated and which can be used to analyze the sentiments of the people. Hadoop is one of the best options available for twitter data sentiment analysis and which also works for the distributed big data, streaming data, text data etc.  This paper provides an efficient mechanism to perform sentiment analysis/ opinion mining on Twitter data over Hortonworks Data platform, which provides Hadoop on Windows, with the assistance of Apache Flume, Apache HDFS and Apache Hive. 

scholarly journals Designing Framework for Real Time Twitter Data Analytics using Apache Flume and Pig

2020 ◽  
Vol 8 (6) ◽  
pp. 4474-4477
Keyword(s):  
Real Time ◽  
Data Analytics ◽  
Social Issues ◽  
Analytical Framework ◽  
Streaming Data ◽  
Data Streaming ◽  
Text Data ◽  
Breaking News ◽  
Unstructured Text ◽  
Twitter Data

In the world of technology, people prefer social media to express themselves. Record says Twitter has more than 321 million active users with 100 million users posting approximately 340 million tweets a day. Twitter is the largest source of breaking news on social issues specially election-related where people can express their views also suggest their opinion. Twitter is generating unlimited unstructured text data. Hadoop is one of the finest tools accessible for analyzing twitter data because it supports processing of distributed big data, streaming data, time stamped data, text data etc. Whereas Apache Flume is used to extract real time twitter data into HDFS. This study attempts to establish an analytical framework to derive and interpret structured as well as unstructured Twitter data. The proposed framework comprises of real time twitter data insertion, its processing, and data visualization utilizing Apache Flume and pig. In this project we fetch positive and negative tweets on election data from twitter and analyzing the party status and the probability to win the election.

Twitter Data Analysis by Live Streaming

2019 ◽  
Vol 16 (8) ◽  
pp. 3178-3182
Author(s):  
Sugnik Roy Chowdhury
Keyword(s):  
Streaming Data ◽  
Pictorial Representation ◽  
Live Streaming ◽  
Time Data ◽  
The Public ◽  
Twitter Data ◽  
Real Time Data ◽  
Pros And Cons ◽  
Twitter Data Analysis ◽  
Day By Day

Streaming now a days have been of great use when comes to Social Media. Streaming of data have made it easy for Companies to understand the pros and cons of their product. Streaming acts as a survey now a days which a few years ago were done by a team of individual using pen and papers. In order to collect and process the streaming data from various streaming sites to produce an analytical report that helps to get a clear pictorial representation of events, the assets of streaming process generates a huge volume of real time data mainly referred to as “Big Data.” In order to aggregate, store and analyses the streaming data that are being generated Day-By-Day we get into the concept of Hadoop and Flume Technologies, API that helps to collect data from Twitter/other streaming sites by using “#” tag/Keywords. Tweets by the News channel and retweets by the public are being collected.

Twitter Data Analysis

2018 ◽  
pp. 124-151 ◽  
Author(s):  
Chitrakala S
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Information Diffusion ◽  
Big Data Analytics ◽  
Influence Maximization ◽  
Streaming Data ◽  
Hybrid Analysis ◽  
Topic Analysis ◽  
Twitter Data ◽  
Twitter Data Analysis

Analyzing Social network data using Big Data Tools and techniques promises to provide information that could be of use in recommendation systems, personalized service and many other applications. A few of the analytics that do this include sentiment analysis, trending topic analysis, topic modeling, information diffusion modeling, provenance determination and social influence study. Twitter Data Analysis involves analyzing data specifically obtained from Twitter, both tweets and the topology. There are three major classifications on the type of analysis being performed such as Content based, Network based and Hybrid analysis. Trending Topic Analysis in the context of Content based static data analysis and Influence Maximization in the context of Hybrid analysis on data streams using the power of Big Data Analytics are discussed. A novel solution to Trending Topic analysis to generate topic evolved, conflict-free sequential sub summaries and influence maximization to handle streaming data are explained with experimental results.

scholarly journals Deep learning framework for handling concept drift and class imbalanced complex decision-making on streaming data

2021 ◽  
Author(s):  
S. Priya ◽  
R. Annie Uthra
Keyword(s):  
Decision Making ◽  
Deep Learning ◽  
Concept Drift ◽  
Class Imbalance ◽  
Streaming Data ◽  
Superior Performance ◽  
Data Streaming ◽  
Minority Class ◽  
Concept Drift Detection

AbstractIn present times, data science become popular to support and improve decision-making process. Due to the accessibility of a wide application perspective of data streaming, class imbalance and concept drifting become crucial learning problems. The advent of deep learning (DL) models finds useful for the classification of concept drift in data streaming applications. This paper presents an effective class imbalance with concept drift detection (CIDD) using Adadelta optimizer-based deep neural networks (ADODNN), named CIDD-ADODNN model for the classification of highly imbalanced streaming data. The presented model involves four processes namely preprocessing, class imbalance handling, concept drift detection, and classification. The proposed model uses adaptive synthetic (ADASYN) technique for handling class imbalance data, which utilizes a weighted distribution for diverse minority class examples based on the level of difficulty in learning. Next, a drift detection technique called adaptive sliding window (ADWIN) is employed to detect the existence of the concept drift. Besides, ADODNN model is utilized for the classification processes. For increasing the classifier performance of the DNN model, ADO-based hyperparameter tuning process takes place to determine the optimal parameters of the DNN model. The performance of the presented model is evaluated using three streaming datasets namely intrusion detection (NSL KDDCup) dataset, Spam dataset, and Chess dataset. A detailed comparative results analysis takes place and the simulation results verified the superior performance of the presented model by obtaining a maximum accuracy of 0.9592, 0.9320, and 0.7646 on the applied KDDCup, Spam, and Chess dataset, respectively.

scholarly journals Features of Big Text Data Visualization for Managerial Decision Making

2019 ◽  
Author(s):  
Елена Макарова ◽  
Elena Makarova ◽  
Дмитрий Лагерев ◽  
Dmitriy Lagerev ◽  
Федор Лозбинев ◽  
...  
Keyword(s):  
Decision Making ◽  
Vector Representation ◽  
Managerial Decision ◽  
Text Data ◽  
Managerial Decision Making ◽  
Human Control ◽  
Textual Data ◽  
Interactive Visualizations ◽  
N Gram ◽  
Text Data Analysis

This paper describes text data analysis in the course of managerial decision making. The process of collecting textual data for further analysis as well as the use of visualization in human control over the correctness of data collection is considered in depth. An algorithm modification for creating an "n-gram cloud" visualization is proposed, which can help to make visualization accessible to people with visual impairments. Also, a method of visualization of n-gram vector representation models (word embedding) is proposed. On the basis of the conducted research, a part of a software package was implemented, which is responsible for creating interactive visualizations in a browser and interoperating with them.

Using Computational Techniques to Fill the Gap between Qualitative Data Analysis and Text Analytics

KWALON ◽  
2010 ◽  
Vol 15 (3) ◽  
Author(s):  
Curtis Atkisson ◽  
Colin Monaghan ◽  
Edward Brent
Keyword(s):  
Data Analysis ◽  
Qualitative Data ◽  
Computational Techniques ◽  
Text Analytics ◽  
Qualitative Data Analysis ◽  
Text Data ◽  
Textual Data ◽  
Data Flows ◽  
Mass Digitization ◽  
Analysis System

The recent mass digitization of text data has led to a need to efficiently and effectively deal with the mountain of textual data that is generated. Digitized text is increasingly in the form of digitized data flows (Brent, 2008). Digitized data flows are non-static streams of generated content – including twitter, electronic news, etc. An oft-cited statistic is that currently 85% of all business data is in the form of text (cited in Hotho, Nürnberger & Paass, 2005). This mountain of data leads us to the question whether the labor-intensive traditional qualitative data analysis techniques are best suited for this large amount of data. Other techniques for dealing with large amounts of data may also be found wanting because those techniques remove the researcher from an immersion in the data. Both dealing with large amounts of data and allowing immersion in data are clearly desired features of any text analysis system.

Sign in / Sign up

Export Citation Format

Share Document