A Consistent Approach to Building Secure Big Data Processing and Storage Systems

AbstractFor even small astronomy projects, the petabyte scale is now upon us. The Asteroid Terrestrial-impact Last Alert System (Tonry 2011) will survey the entire visible sky from Hawaii multiple times per night to search for near-Earth asteroids on impact trajectories. While the ATLAS optical system is modest by modern astronomical standards — two 0.5 m F/2.0 telescopes — each night the ATLAS system will measure nearly 109 astronomical sources to a photometric accuracy of <5%, totaling 1012 individual observations over its initial 3-year mission. This ever-growing dataset must be searched in real-time for moving objects and transients then archived for further analysis, and alerts for newly discovered near-Earth asteroids (NEAs) disseminated within tens of minutes from detection. ATLAS's all-sky coverage ensures it will discover many ‘rifle shot’ near-misses moving rapidly on the sky as they shoot past the Earth, so the system will need software to automatically detect highly-trailed sources and discriminate them from the thousands of low-Earth orbit (LEO) and geosynchronous orbit (GEO) satellites ATLAS will see each night. Additional interrogation will identify interesting phenomena from millions of transient sources per night beyond the solar system. The data processing and storage requirements for ATLAS demand a ‘big data’ approach typical of commercial internet enterprises. We describe our experience in deploying a nimble, scalable and reliable data processing infrastructure, and suggest ATLAS as steppingstone to data processing capability needed as we enter the era of LSST.

Download Full-text

Data Prefetching and Eviction Mechanisms of In-Memory Storage Systems Based on Scheduling for Big Data Processing

IEEE Transactions on Parallel and Distributed Systems ◽

10.1109/tpds.2019.2892957 ◽

2019 ◽

Vol 30 (8) ◽

pp. 1738-1752 ◽

Cited By ~ 1

Author(s):

Chien-Hung Chen ◽

Ting-Yuan Hsia ◽

Yennun Huang ◽

Sy-Yen Kuo

Keyword(s):

Big Data ◽

Data Processing ◽

Storage Systems ◽

Memory Storage ◽

Data Prefetching ◽

Big Data Processing

Download Full-text

Open Source Software (OSS) for Big Data

Research Anthology on Usage and Development of Open Source Software ◽

10.4018/978-1-7998-9158-1.ch042 ◽

2021 ◽

pp. 858-875

Author(s):

Richard S. Segall

Keyword(s):

Big Data ◽

Data Processing ◽

Open Source ◽

Open Source Software ◽

Statistical Software ◽

Big Data Processing ◽

Open Source Data ◽

Source Data ◽

And Storage ◽

Aggregation Analysis

This chapter discusses Open Source Software and associated technologies for the processing of Big Data. This includes discussions of Hadoop-related projects, the current top open source data tools and frameworks such as SMACK that is acronym for open source technologies Spark, Mesos, Akka, Cassandra, and Kafka that together compose the ingestion, aggregation, analysis, and storage layers for Big Data processing. Tabular summaries and categories for 38 Open Source Statistical Software (OSSS) are provided that include for each listing of features and URLs for free downloads. The current challenges of Big Data and Open Source Software are also discussed.

Download Full-text

Open Source Software (OSS) for Big Data

Open Source Software for Statistical Analysis of Big Data - Advances in Computer and Electrical Engineering ◽

10.4018/978-1-7998-2768-9.ch002 ◽

2020 ◽

pp. 50-72

Author(s):

Richard S. Segall

Keyword(s):

Big Data ◽

Data Processing ◽

Open Source ◽

Open Source Software ◽

Statistical Software ◽

Big Data Processing ◽

Open Source Data ◽

Source Data ◽

And Storage ◽

Aggregation Analysis

This chapter discusses Open Source Software and associated technologies for the processing of Big Data. This includes discussions of Hadoop-related projects, the current top open source data tools and frameworks such as SMACK that is acronym for open source technologies Spark, Mesos, Akka, Cassandra, and Kafka that together compose the ingestion, aggregation, analysis, and storage layers for Big Data processing. Tabular summaries and categories for 38 Open Source Statistical Software (OSSS) are provided that include for each listing of features and URLs for free downloads. The current challenges of Big Data and Open Source Software are also discussed.

Download Full-text

Limitations and Perspectives of Optically Switched Interconnects for Large-scale Data Processing and Storage Systems

MRS Proceedings ◽

10.1557/opl.2012.1409 ◽

2012 ◽

Vol 1438 ◽

Cited By ~ 1

Author(s):

Slavisa Aleksic ◽

Gerhard Schmid ◽

Naida Fehratovic

Keyword(s):

Power Consumption ◽

Data Processing ◽

Large Scale ◽

Storage Systems ◽

Reducing Power ◽

Large Scale Data ◽

Large Scale Data Processing ◽

Processing And Storage ◽

And Storage ◽

Scale Data

ABSTRACTThe ever-growing Internet data traffic leads to a continuously increasing demand in both capacity and performance of large-scale Information and Communication (ICT) systems such as high-capacity routers and switches, large data centers, and supercomputers. Complex and spatially distributed multirack systems comprising a large number of data processing and storage modules with high-speed interfaces have already become reality. A consequence of this trend is that internal interconnection systems also become large and complex. Interconnection distances, total required number of cables, and power consumption increase rapidly with the increase in capacity, which can cause limitations in scalability of the whole system. This paper addresses requirements and limitations of intrasystem interconnects for application in large-scale data processing and storage systems. Various point-to-point and optically switched interconnection options are reviewed with regard to their potential to achieve large scalability while reducing power consumption.

Download Full-text