Design of big data processing system for spacecraft testing experiment

Semiconductor production efficiency is closely related to the defect rate in the production process. The temperature and humidity control in the production line are very important because these affect the defect rate. So many smart factory of semiconductor production uses sensor. It is installed in the semiconductor process, which send huge amounts of data per second to a central server to carry out temperature and humidity control in each production line. However, big data processing systems that analyze and process large-scale data are subject to frequent delays in processing, and transmitted data are lost owing to bottlenecks and insufficient memory caused by traffic concentrated in the central server. In this paper, we propose a real-time big data processing system to improve semiconductor production efficiency. The proposed system consists of a production line collection system, task processing system and data storage system, and improves the productivity of the semiconductor manufacturing process by reducing data processing delays as well as data loss and discarded data.

Download Full-text

Design of Big Data Processing System Architecture Based on Hadoop under the Cloud Computing

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.556-562.6302 ◽

2014 ◽

Vol 556-562 ◽

pp. 6302-6306 ◽

Cited By ~ 3

Author(s):

Chun Mei Duan

Keyword(s):

Cloud Computing ◽

Big Data ◽

Data Processing ◽

Data Storage ◽

System Architecture ◽

Processing System ◽

System Stability ◽

Security Model ◽

Data Processing System ◽

Big Data Processing

In allusion to limitations of traditional data processing technology in big data processing, big data processing system architecture based on hadoop is designed, using the characteristics of quantification, unstructured and dynamic of cloud computing.It uses HDFS be responsible for big data storage, and uses MapReduce be responsible for big data calculation and uses Hbase as unstructured data storage database, at the same time a system of storage and cloud computing security model are designed, in order to implement efficient storage, management, and retrieval of data,thus it can save construction cost, and guarantee system stability, reliability and security.

Download Full-text

High-Performance Geospatial Big Data Processing System Based on MapReduce

ISPRS International Journal of Geo-Information ◽

10.3390/ijgi7100399 ◽

2018 ◽

Vol 7 (10) ◽

pp. 399 ◽

Cited By ~ 10

Author(s):

Junghee Jo ◽

Kang-Woo Lee

Keyword(s):

Big Data ◽

Spatial Analysis ◽

Data Processing ◽

Spatial Data ◽

High Performance ◽

Rapid Development ◽

Processing System ◽

Data Processing System ◽

Seamless Integration ◽

Big Data Processing

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

Download Full-text