scholarly journals Development of a national-scale real-time Twitter data mining pipeline for social geodata on the potential impacts of flooding on communities

2019 ◽  
Author(s):  
Luke Barker ◽  
CJA MacLeod
Keyword(s):  
Data Mining ◽  
At Risk ◽  
Real Time ◽  
Language Processing ◽  
Environmental Data ◽  
Data Sources ◽  
Extreme Weather Events ◽  
Modular Architecture ◽  
National Scale ◽  
Twitter Data

Social media, particularly Twitter, is increasingly used to improve resilience during extreme weather events/emergency management situations, including floods: by communicating potential risks and their impacts, and informing agencies and responders. In this paper, we developed a prototype national-scale Twitter data mining pipeline for improved stakeholder situational awareness during flooding events across Great Britain, by retrieving relevant social geodata, grounded in environmental data sources (flood warnings and river levels). With potential users we identified and addressed three research questions to develop this application, whose components constitute a modular architecture for real-time dashboards. First, polling national flood warning and river level Web data sources to obtain at-risk locations. Secondly, real-time retrieval of geotagged tweets, proximate to at-risk areas. Thirdly, filtering flood-relevant tweets with natural language processing and machine learning libraries, using word embeddings of tweets. We demonstrated the national-scale social geodata pipeline using over 420,000 georeferenced tweets obtained between 20 and 29th June 2016.

scholarly journals An Ontology-based Visual Analytics for Apple Variety Testing

2021 ◽  
Author(s):  
Ekaterina Chuprikova ◽  
Abraham Mejia Aguilar ◽  
Roberto Monsorno
Keyword(s):  
Data Mining ◽  
Data Analysis ◽  
Data Integration ◽  
Visual Analytics ◽  
Agricultural Sector ◽  
Environmental Data ◽  
Data Sources ◽  
Apple Variety ◽  
Testing Program ◽  
Variety Testing

<p>Increasing agricultural production challenges, such as climate change, environmental concerns, energy demands, and growing expectations from consumers triggered the necessity for innovation using data-driven approaches such as visual analytics. Although the visual analytics concept was introduced more than a decade ago, the latest developments in the data mining capacities made it possible to fully exploit the potential of this approach and gain insights into high complexity datasets (multi-source, multi-scale, and different stages). The current study focuses on developing prototypical visual analytics for an apple variety testing program in South Tyrol, Italy. Thus, the work aims (1) to establish a visual analytics interface enabled to integrate and harmonize information about apple variety testing and its interaction with climate by designing a semantic model; and (2) to create a single visual analytics user interface that can turn the data into knowledge for domain experts. </p><p>This study extends the visual analytics approach with a structural way of data organization (ontologies), data mining, and visualization techniques to retrieve knowledge from an extensive collection of apple variety testing program and environmental data. The prototype stands on three main components: ontology, data analysis, and data visualization. Ontologies provide a representation of expert knowledge and create standard concepts for data integration, opening the possibility to share the knowledge using a unified terminology and allowing for inference. Building upon relevant semantic models (e.g., agri-food experiment ontology, plant trait ontology, GeoSPARQL), we propose to extend them based on the apple variety testing and climate data. Data integration and harmonization through developing an ontology-based model provides a framework for integrating relevant concepts and relationships between them, data sources from different repositories, and defining a precise specification for the knowledge retrieval. Besides, as the variety testing is performed on different locations, the geospatial component can enrich the analysis with spatial properties. Furthermore, the visual narratives designed within this study will give a better-integrated view of data entities' relations and the meaningful patterns and clustering based on semantic concepts.</p><p>Therefore, the proposed approach is designed to improve decision-making about variety management through an interactive visual analytics system that can answer "what" and "why" about fruit-growing activities. Thus, the prototype has the potential to go beyond the traditional ways of organizing data by creating an advanced information system enabled to manage heterogeneous data sources and to provide a framework for more collaborative scientific data analysis. This study unites various interdisciplinary aspects and, in particular: Big Data analytics in the agricultural sector and visual methods; thus, the findings will contribute to the EU priority program in digital transformation in the European agricultural sector.</p><p>This project has received funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 894215.</p>

Semantic Integration and Knowledge Discovery for Environmental Research

2008 ◽  
pp. 213-235
Author(s):  
Zhiyuan Chen ◽  
Aryya Gangopadhyay ◽  
George Karabatis ◽  
Michael McGuire ◽  
Claire Welty
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
Semantic Network ◽  
Semantic Integration ◽  
Environmental Data ◽  
Data Sources ◽  
Environmental Research ◽  
Related Data ◽  
Use Of Data ◽  
Data Semantics

Environmental research and knowledge discovery both require extensive use of data stored in various sources and created in different ways for diverse purposes. We describe a new metadata approach to elicit semantic information from environmental data and implement semantics-based techniques to assist users in integrating, navigating, and mining multiple environmental data sources. Our system contains specifications of various environmental data sources and the relationships that are formed among them. User requests are augmented with semantically related data sources and automatically presented as a visual semantic network. In addition, we present a methodology for data navigation and pattern discovery using multi-resolution browsing and data mining. The data semantics are captured and utilized in terms of their patterns and trends at multiple levels of resolution. We present the efficacy of our methodology through experimental results.

scholarly journals Review On: Detection of Spam Comments Using NLP Algorithm

2018 ◽  
Vol 7 (01) ◽  
pp. 23386-23489
Author(s):  
Miss Rohini D.Warkar ◽  
Mr.I.R. Shaikh
Keyword(s):  
Data Mining ◽  
Social Media ◽  
Natural Language Processing ◽  
Natural Language ◽  
Social Networking ◽  
Real Time ◽  
Social Services ◽  
Language Processing ◽  
Social Networking Sites ◽  
Online Media

Detecting trending topics is perfect to summarize information getting from social media. To extract what topic is becoming hot on online media is one of the challenges. As we considering social media so social services are opportunity for spamming which greatly affect on value of real time search. Therefore the next task is to control spamming from social networking sites. For completing these challenges different concepts of data mining will be used. For now whatever work has been done is narrated below like spam control using natural language processing for preprocessing and clustering. One account has been created for making it real.

Data Mining

2012 ◽  
pp. 75-94 ◽  
Author(s):  
Martin Atzmueller
Keyword(s):  
Data Mining ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Text Classification ◽  
Pattern Mining ◽  
Data Sources ◽  
Future Research ◽  
Research Directions ◽  
Future Research Directions

Data Mining provides approaches for the identification and discovery of non-trivial patterns and models hidden in large collections of data. In the applied natural language processing domain, data mining usually requires preprocessed data that has been extracted from textual documents. Additionally, this data is often integrated with other data sources. This chapter provides an overview on data mining focusing on approaches for pattern mining, cluster analysis, and predictive model construction. For those, we discuss exemplary techniques that are especially useful in the applied natural language processing context. Additionally, we describe how the presented data mining approaches are connected to text mining, text classification, and clustering, and discuss interesting problems and future research directions.

scholarly journals Sentiment Trend Analysis of Big Data

2019 ◽  
Vol 8 (10) ◽  
pp. 59-64
Keyword(s):  
Decision Making ◽  
Social Media ◽  
Big Data ◽  
Sentiment Analysis ◽  
Real Time ◽  
Language Processing ◽  
Opinion Mining ◽  
Data Sources ◽  
Product Reviews

Various fields like Text Mining, Linguistics, Decision Making and Natural Language Processing together form the basis for Opinion Mining or Sentiment Analysis. People share their feelings, observations and thoughts on social media, which has emerged as a powerful tool for rapidly growing enormous repository of real time discussions and thoughts shared by people. In this paper, we aim to decipher the current popular opinions or emotions from various sources, hence, contributing to sentiment analysis domain. Text from social media, blogs and product reviews are classified according to the sentiment they project. We re-examine the traditional processes of sentiment extraction, to incorporate the increase in complexity and number of the data sources and relevant topics, while re-populating the meaning of sentiment. Working across and within numerous streams of social media, expression of sentiment and classification of polarity is re-examined, thereby redefining and enhancing the realm of sentiment. Numerous social media streams are analyzed to build datasets that are topical for each stream and are later polarized according to their sentiment expression. In conclusion, defining a sentiment and developing tools for its analysis in real time of human idea exchange is the motive.

scholarly journals On the forecastability of food insecurity

2021 ◽  
Author(s):  
Pietro Foini ◽  
Michele Tizzoni ◽  
Daniela Paolotti ◽  
Elisa Omodei
Keyword(s):  
At Risk ◽  
Food Insecurity ◽  
Real Time ◽  
Food Consumption ◽  
Boosted Regression Trees ◽  
Extreme Weather Events ◽  
Forecasting Model ◽  
Time Data ◽  
Real Time Data ◽  
The Future

Food insecurity, defined as the lack of physical or economic access to safe, nutritious and sufficient food, remains one of the main challenges included in the 2030 Agenda for Sustainable Development. Near real-time data on the food insecurity situation collected by international organizations such as the World Food Programme can be crucial to monitor and forecast time trends of insufficient food consumption levels in countries at risk. Here, using food consumption observations in combination with secondary data on conflict, extreme weather events and economic shocks, we build a forecasting model based on gradient boosted regression trees to create predictions on the evolution of insufficient food consumption trends up to 30 days in to the future in 6 countries (Burkina Faso, Cameroon, Mali, Nigeria, Syria and Yemen). Results show that the number of available historical observations is a key element for the forecasting model performance. Among the 6 countries studied in this work, for those with the longest food insecurity time series, the proposed forecasting model makes it possible to forecast the prevalence of people with insufficient food consumption up to 30 days into the future with higher accuracy than a naive approach based on the last measured prevalence only. The framework developed in this work could provide decision makers with a tool to assess how the food insecurity situation will evolve in the near future in countries at risk. Results clearly point to the added value of continuous near real-time data collection at sub-national level.

scholarly journals Understanding Concerns, Sentiments, and Disparities Among Population Groups During the COVID-19 Pandemic Via Twitter Data Mining: Large-scale Cross-sectional Study (Preprint)

2020 ◽  
Author(s):  
Chunyan Zhang ◽  
Songhua Xu ◽  
Zongfang Li ◽  
Shunxu Hu
Keyword(s):  
Public Health ◽  
Data Mining ◽  
Language Processing ◽  
Large Scale ◽  
Cross Sectional Study ◽  
Health Economy ◽  
Sectional Study ◽  
Cross Sectional ◽  
Population Groups ◽  
Twitter Data

BACKGROUND Since the beginning of the COVID-19 pandemic in late 2019, its far-reaching impacts have been witnessed globally across all aspects of human life, such as health, economy, politics, and education. Such widely penetrating impacts cast significant and profound burdens on all population groups, incurring varied concerns and sentiments among them. OBJECTIVE This study aims to identify the concerns, sentiments, and disparities of various population groups during the COVID-19 pandemic through a cross-sectional study conducted via large-scale Twitter data mining infoveillance. METHODS This study consisted of three steps: first, tweets posted during the pandemic were collected and preprocessed on a large scale; second, the key population attributes, concerns, sentiments, and emotions were extracted via a collection of natural language processing procedures; third, multiple analyses were conducted to reveal concerns, sentiments, and disparities among population groups during the pandemic. Overall, this study implemented a quick, effective, and economical approach for analyzing population-level disparities during a public health event. The source code developed in this study was released for free public use at GitHub. RESULTS A total of 1,015,655 original English tweets posted from August 7 to 12, 2020, were acquired and analyzed to obtain the following results. Organizations were significantly more concerned about COVID-19 (odds ratio [OR] 3.48, 95% CI 3.39-3.58) and expressed more fear and depression emotions than individuals. Females were less concerned about COVID-19 (OR 0.73, 95% CI 0.71-0.75) and expressed less fear and depression emotions than males. Among all age groups (ie, ≤18, 19-29, 30-39, and ≥40 years of age), the attention ORs of COVID-19 fear and depression increased significantly with age. It is worth noting that not all females paid less attention to COVID-19 than males. In the age group of 40 years or older, females were more concerned than males, especially regarding the economic and education topics. In addition, males 40 years or older and 18 years or younger were the least positive. Lastly, in all sentiment analyses, the sentiment polarities regarding political topics were always the lowest among the five topics of concern across all population groups. CONCLUSIONS Through large-scale Twitter data mining, this study revealed that meaningful differences regarding concerns and sentiments about COVID-19-related topics existed among population groups during the study period. Therefore, specialized and varied attention and support are needed for different population groups. In addition, the efficient analysis method implemented by our publicly released code can be utilized to dynamically track the evolution of each population group during the pandemic or any other major event for better informed public health research and interventions.

scholarly journals Automatic identification of risk factors for SARS-CoV-2 positivity and severe clinical outcomes of COVID-19 using Data Mining and Natural Language Processing

2021 ◽  
Author(s):  
Verena Schoening ◽  
Evangelia Liakoni ◽  
Juergen Drewe ◽  
Felix Hammann
Keyword(s):  
Risk Factors ◽  
Data Mining ◽  
Natural Language Processing ◽  
Natural Language ◽  
Real Time ◽  
Clinical Outcomes ◽  
Language Processing ◽  
Structured Data ◽  
Diabetic Patients ◽  
Free Text

Objectives: Several risk factors have been identified for severe clinical outcomes of COVID-19 caused by SARS-CoV-2. Some can be found in structured data of patients' Electronic Health Records. Others are included as unstructured free-text, and thus cannot be easily detected automatically. We propose an automated real-time detection of risk factors using a combination of data mining and Natural Language Processing (NLP). Material and methods: Patients were categorized as negative or positive for SARS-CoV-2, and according to disease severity (severe or non-severe COVID-19). Comorbidities were identified in the unstructured free-text using NLP. Further risk factors were taken from the structured data. Results: 6250 patients were analysed (5664 negative and 586 positive; 461 non-severe and 125 severe). Using NLP, comorbidities, i.e. cardiovascular and pulmonary conditions, diabetes, dementia and cancer, were automatically detected (error rate ≤2%). Old age, male sex, higher BMI, arterial hypertension, chronic heart failure, coronary heart disease, COPD, diabetes, insulin only treatment of diabetic patients, reduced kidney and liver function were risk factors for severe COVID-19. Interestingly, the proportion of diabetic patients using metformin but not insulin was significantly higher in the non-severe COVID-19 cohort (p<0.05). Discussion and conclusion: Our findings were in line with previously reported risk factors for severe COVID-19. NLP in combination with other data mining approaches appears to be a suitable tool for the automated real-time detection of risk factors, which can be a time saving support for risk assessment and triage, especially in patients with long medical histories and multiple comorbidities.

Semantic Integration and Knowledge Discovery for Environmental Research

2009 ◽  
pp. 39-61
Author(s):  
Zhiyuan Chen
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
Semantic Network ◽  
Semantic Integration ◽  
Environmental Data ◽  
Data Sources ◽  
Environmental Research ◽  
Related Data ◽  
Use Of Data ◽  
Data Semantics

Environmental research and knowledge discovery both require extensive use of data stored in various sources and created in different ways for diverse purposes. We describe a new metadata approach to elicit semantic information from environmental data and implement semantics-based techniques to assist users in integrating, navigating, and mining multiple environmental data sources. Our system contains specifications of various environmental data sources and the relationships that are formed among them. User requests are augmented with semantically related data sources and automatically presented as a visual semantic network. In addition, we present a methodology for data navigation and pattern discovery using multi-resolution browsing and data mining. The data semantics are captured and utilized in terms of their patterns and trends at multiple levels of resolution. We present the efficacy of our methodology through experimental results.

Sign in / Sign up

Export Citation Format

Share Document