Real-Time Big Data Analysis Using Web Scraping in Apache Spark Environment: Case Study—Mobile Data Analysis from Flipkart

Author(s):  
Pushpita Ganguly ◽  
Giriraj Parihar ◽  
M. Sivagami
2017 ◽  
Vol 13 (7) ◽  
pp. 155014771772181 ◽  
Author(s):  
Seok-Woo Jang ◽  
Gye-Young Kim

This article proposes an intelligent monitoring system for semiconductor manufacturing equipment, which determines spec-in or spec-out for a wafer in process, using Internet of Things–based big data analysis. The proposed system consists of three phases: initialization, learning, and prediction in real time. The initialization sets the weights and the effective steps for all parameters of equipment to be monitored. The learning performs a clustering to assign similar patterns to the same class. The patterns consist of a multiple time-series produced by semiconductor manufacturing equipment and an after clean inspection measured by the corresponding tester. We modify the Line, Buzo, and Gray algorithm for classifying the time-series patterns. The modified Line, Buzo, and Gray algorithm outputs a reference model for every cluster. The prediction compares a time-series entered in real time with the reference model using statistical dynamic time warping to find the best matched pattern and then calculates a predicted after clean inspection by combining the measured after clean inspection, the dissimilarity, and the weights. Finally, it determines spec-in or spec-out for the wafer. We will present experimental results that show how the proposed system is applied on the data acquired from semiconductor etching equipment.


Web Services ◽  
2019 ◽  
pp. 1301-1329
Author(s):  
Suren Behari ◽  
Aileen Cater-Steel ◽  
Jeffrey Soar

The chapter discusses how Financial Services organizations can take advantage of Big Data analysis for disruptive innovation through examination of a case study in the financial services industry. Popular tools for Big Data Analysis are discussed and the challenges of big data are explored as well as how these challenges can be met. The work of Hayes-Roth in Valued Information at the Right Time (VIRT) and how it applies to the case study is examined. Boyd's model of Observe, Orient, Decide, and Act (OODA) is explained in relation to disruptive innovation in financial services. Future trends in big data analysis in the financial services domain are explored.


2020 ◽  
Author(s):  
Pingyu Fan ◽  
Kwok Pan Chun ◽  
Ana Mijic ◽  
Daphne Ngar-Yin Mah

<p>Digital water and energy maps allow fast information retrieval, big data analysis and resources demand prediction for real time responses in 5-G networks. A regulatory systems framework is needed to enable and promote integrated actions grounded on map-based feedback information, to facilitate resources movements and knowledge transfer for water and energy security. At the same time, the proposed regulatory system needs to safeguard national security and personal privacy when general public and the private sectors have access to big databases.</p><p>The Guangdong-Hong Kong-Macao Greater Bay Area (GBA) in China is an initiative on regional economic development involving nine mainland cities and two Special Administrative Regions (SARs). As central policies cannot be efficiently executed in the whole regions, institutional fragmentation could be a prominent barrier to achieve regional water and energy optimum rather than individual city maxima for the water and energy nexus.</p><p>In this study, we propose a systems regulatory framework that integrates natural, urban and social systems across multiple scales in which the relevant laws, policies, decisions and actions are supported by digital maps. On a planning scale, our new regulatory system based on spatial map information promotes optimum uses of natural capitals and ecosystem services (ES). For linking different urban spatial processes on different scales, satellite images and Local Climate Zone (LCZ) maps are used to describe natural environment and urban characteristics from 200km to 10km resolutions for supporting land-use planning laws and estimating regional development carrying capacity to mitigate water and energy insecurity.</p><p>On an operational scale, smart meters and remote sensor systems provide real time water and energy information from a fast developing 5-G network for the proposed digital maps. Forecasted energy and water demands from the digital maps can be used for regional or local environment regulation reinforcement. Proposed spatial maps also improve transboundary collaboration by providing visualisation of legal targets and emission limits. Through digital maps, key agencies and sectors will have a capacity to share transboundary knowledge, information and responsibility, to foster smooth system flows in terms of culture, economy, policy and technology, by active participations and decentralized actions.</p><p>On an evaluation scale, open map information increases the transparency of legal targets and pollution limits. By rapid information retrieval and big data analysis from digital maps, regulators can assess the performance of water and energy security practices.</p><p>In summary, the proposed framework based on LCZ maps for the GBA can be applied to other rapidly developing regions with emerging 5-G networks. The integrated regulatory framework also guides water and energy security practices and transfer central policies to local actions by rapid information retrieval, big data analysis and prediction of demand for real time responses based on digital water and energy maps.</p><p></p><p></p><p></p><p></p><p></p>


2016 ◽  
Vol 11 (2) ◽  
pp. 164-174 ◽  
Author(s):  
Shunichi Koshimura ◽  

A project titled “Establishing the advanced disaster reduction management system by fusion of real-time disaster simulation and big data assimilation,” was launched as Core Research for Evolutional Science and Technology (CREST) by the Japan Science and Technology Agency (JST). Intended to save as many lives as possible in future national crises involving earthquake and tsunami disasters, the project works on a disaster mitigation system of the big data era, based on cooperation of large-scale, high-resolution, real-time numerical simulations and assimilation of real-time observation data. The world’s most advanced specialists in disaster simulation, disaster management, mathematical science, and information science work together to create the world’s first analysis platform for real-time simulation and big data that effectively processes, analyzes, and assimilates data obtained through various observations. Based on quantitative data, the platform designs proactive measures and supports disaster operations immediately after disaster occurrence. The project was launched in 2014 and is working on the following issues at present.Sophistication and fusion of simulations and damage prediction models using observational big data: Development of a real-time simulation core system that predicts the time evolution of disaster effect by assimilating of location information, fire information, and building collapse information which are obtained from mobile terminals, satellite images, aerial images, and other new observation data in addition to sensing data obtained by the undersea high-density seismic observation network.Latent structure analysis and major disaster scenario creation based on a huge amount of simulation results: Development of an analysis and extraction method for the latent structure of a huge amount of disaster scenarios generated by simulation, and creation of severe scenarios with minimum “unexpectedness” by controlling disaster scenario explosion (an explosive increase in the number of predicted scenarios).Establishment of an earthquake and tsunami disaster mitigation big data analysis platform: Development of an earthquake and tsunami disaster mitigation big data analysis platform that realizes analyses of a huge number of disaster scenarios and increases in speed of data assimilation, and clarifies the requirements for operation of the platform as a disaster mitigation system.The project was launched in 2014 as a 5-year project. It consists of element technology development and system fusion, feasibility study as a next-generation disaster mitigation system (validation with/without introduction of the developed real-time simulation and big data analysis platform) in the affected areas of the Great East Japan Earthquake, and test operations in affected areas of the Tokyo metropolitan earthquake and the Nankai Trough earthquake.


Sign in / Sign up

Export Citation Format

Share Document