scholarly journals Cloak-Reduce Load Balancing Strategy for Mapreduce

2021 ◽  
Vol 13 (4) ◽  
Author(s):  
Mamadou Diarra ◽  
Telesphore Tiendrebeogo
Keyword(s):  
Load Balancing ◽  
Response Time ◽  
Large Scale ◽  
Distributed Processing ◽  
Parallel Applications ◽  
Design Load ◽  
Load Regulation ◽  
New Processing ◽  
Processing And Storage ◽  
And Storage

The advent of Big Data has seen the emergence of new processing and storage challenges. These challenges are often solved by distributed processing. Distributed systems are inherently dynamic and unstable, so it is realistic to expect that some resources will fail during use. Load balancing and task scheduling is an important step in determining the performance of parallel applications. Hence the need to design load balancing algorithms adapted to grid computing. In this paper, we propose a dynamic and hierarchical load balancing strategy at two levels: Intrascheduler load balancing, in order to avoid the use of the large-scale communication network, and interscheduler load balancing, for a load regulation of our whole system. The strategy allows improving the average response time of CLOAK-Reduce application tasks with minimal communication. We first focus on the three performance indicators, namely response time, process latency and running time of MapReduce tasks.

Periodic hierarchical load balancing for large supercomputers

2011 ◽  
Vol 25 (4) ◽  
pp. 371-385 ◽  
Author(s):  
Gengbin Zheng ◽  
Abhinav Bhatelé ◽  
Esteban Meneses ◽  
Laxmikant V. Kalé
Keyword(s):  
Load Balancing ◽  
Large Scale ◽  
Parallel Machines ◽  
National Laboratory ◽  
Argonne National Laboratory ◽  
Parallel Applications ◽  
Scientific Application ◽  
Computing Center ◽  
Blue Gene ◽  
Advanced Computing

Large parallel machines with hundreds of thousands of processors are becoming more prevalent. Ensuring good load balance is critical for scaling certain classes of parallel applications on even thousands of processors. Centralized load balancing algorithms suffer from scalability problems, especially on machines with a relatively small amount of memory. Fully distributed load balancing algorithms, on the other hand, tend to take longer to arrive at good solutions. In this paper, we present an automatic dynamic hierarchical load balancing method that overcomes the scalability challenges of centralized schemes and longer running times of traditional distributed schemes. Our solution overcomes these issues by creating multiple levels of load balancing domains which form a tree. This hierarchical method is demonstrated within a measurement-based load balancing framework in Charm++. We discuss techniques to deal with scalability challenges of load balancing at very large scale. We present performance data of the hierarchical load balancing method on up to 16,384 cores of Ranger (at the Texas Advanced Computing Center) and 65,536 cores of Intrepid (the Blue Gene/P at Argonne National Laboratory) for a synthetic benchmark. We also demonstrate the successful deployment of the method in a scientific application, NAMD, with results on Intrepid.

Keysystems in large systems implementing distributed data processing and storage technologies

2021 ◽  
Author(s):  
V.G. Belenkov ◽  
V.I. Korolev ◽  
V.I. Budzko ◽  
D.A. Melnikov
Keyword(s):  
Data Processing ◽  
Distributed Processing ◽  
Specific Work ◽  
Distributed Data ◽  
Distributed Data Processing ◽  
Technical Specifications ◽  
The Creation ◽  
Processing And Storage ◽  
Storage Technologies ◽  
And Storage

The article discusses the features of the use of the cryptographic information protection means (CIPM)in the environment of distributed processing and storage of data of large information and telecommunication systems (LITS).A brief characteristic is given of the properties of the cryptographic protection control subsystem - the key system (CS). A description is given of symmetric and asymmetric cryptographic systems, required to describe the problem of using KS in LITS.Functional and structural models of the use of KS and CIPM in LITS, are described. Generalized information about the features of using KS in LITS is given. The obtained results form the basis for further work on the development of the architecture and principles of KS construction in LITS that implement distributed data processing and storage technologies. They can be used both as a methodological guide, and when carrying out specific work on the creation and development of systems that implement these technologies, as well as when forming technical specifications for the implementation of work on the creation of such systems.

scholarly journals A Clustered Approach for Load Balancing in Distributed Systems

2021 ◽  
Vol 23 (06) ◽  
pp. 448-463
Author(s):  
Mrs. Geetmala ◽  
◽  
Dr. Neelendra Badal ◽  
Dr. Shri Om Mishra ◽  
◽  
...  
Keyword(s):  
Distributed Systems ◽  
Load Balancing ◽  
Distributed Processing ◽  
Processing System ◽  
Holistic View ◽  
Workload Balancing ◽  
Single System Image ◽  
Computational Paradigm ◽  
Load Balancing Algorithm ◽  
And Storage

Distributed systems are increasingly becoming the dominant and rapidly expanding computational paradigm of tomorrow. A cluster is really a form of parallel or distributed processing system that consists of a set of intertwined stand-alone machines that function together like truly coherent computing and storage resources with a single system image (SSI) which means that perhaps the clusters are viewed as a single platform by the consumers. Global resource management, on the other hand, poses several concerns due to the sheer complexity and range of tools, as well as the need for user accountability. The possible advantages of load balancing in addressing the occasional congestion faced by some nodes when everyone else is idle or congested are widely agreed on a level of performance. This is also widely acknowledged that neither specific load balancing algorithm can adequately address evolving device characteristics and complex capacity management in a distributed ecosystem. To have a systematic approach and also in distributed systems, a proposed approach is created for a holistic view of element load balancing and also the qualities features of load balancing algorithms. The nomenclature has been expanded. In order for adaptive algorithms to understand the problem and manner of prefixing resilience along with different components in distributed systems, they must first recognize the concerns. In addition, a proposed approach is specified. The much more effective load balancing techniques and the modeling hypotheses used in prior load balancing experiments are established through a study of related research. We consider the most appropriate load balancing algorithm and optimum metrics for parameter estimation of the algorithm as a consequence of and output of this assessment for a range of formulations of resulting goals, distributed system features, and workload balancing framework.

scholarly journals Stability of 15 Estrogens and Estrogen Metabolites in Urine Samples under Processing and Storage Conditions Typically Used in Epidemiologic Studies

2010 ◽  
Vol 25 (4) ◽  
pp. 185-194 ◽  
Author(s):  
Barbara J. Fuhrman ◽  
Xia Xu ◽  
Roni T. Falk ◽  
Susan E. Hankinson ◽  
Timothy D. Veenstra ◽  
...  
Keyword(s):  
Ascorbic Acid ◽  
Large Scale ◽  
Storage Conditions ◽  
Epidemiologic Studies ◽  
Individual Variability ◽  
Estrogen Metabolism ◽  
Estrogen Metabolites ◽  
Time Points ◽  
Processing And Storage ◽  
And Storage

Background In preparation for large-scale epidemiologic studies of the role of estrogen metabolism in the etiology of breast and other cancers, we examined the stability of estrogens and estrogen metabolites (EM) in urine during processing and storage protocols. Methods Fifteen EM were measured using liquid chromatography–tandem mass spectrometry (LC-MS/MS) in first morning urines from 3 premenopausal women. Linear regression was used to model log EM concentrations for each woman, with and without adding ascorbic acid (0.1% w/v), during storage at 4°C (7–8 time points, up to 48 hours), during long-term storage at –80°C (10 time points, up to 1 year), and by freeze-thaw cycles (up to 3). Results Without ascorbic acid, concentrations (pmol/mL) of nearly all EM changed <1% per 24 hours of storage at 4°C, and <1% during storage at –80°C for 1 year; similarly, thawing and refreezing samples 3 times was not consistently associated with losses for any EM. Ascorbic acid had no clear beneficial effect on EM stability in these experiments. Conclusions Given the large inter-individual variability in urinary EM concentrations, changes of the magnitude observed here are unlikely to cause substantial misclassification. Furthermore, processing and storage conditions studied here are adequate for use in epidemiologic studies.

scholarly journals The Zoltan and Isorropia Parallel Toolkits for Combinatorial Scientific Computing: Partitioning, Ordering and Coloring

2012 ◽  
Vol 20 (2) ◽  
pp. 129-150 ◽  
Author(s):  
Erik G. Boman ◽  
Ümit V. Çatalyürek ◽  
Cédric Chevalier ◽  
Karen D. Devine
Keyword(s):  
Load Balancing ◽  
Graph Coloring ◽  
User Interfaces ◽  
Large Scale ◽  
Scientific Computing ◽  
Combinatorial Problems ◽  
Parallel Applications ◽  
Matrix Algorithms ◽  
Scientific Simulations ◽  
Matrix Ordering

Partitioning and load balancing are important problems in scientific computing that can be modeled as combinatorial problems using graphs or hypergraphs. The Zoltan toolkit was developed primarily for partitioning and load balancing to support dynamic parallel applications, but has expanded to support other problems in combinatorial scientific computing, including matrix ordering and graph coloring. Zoltan is based on abstract user interfaces and uses callback functions. To simplify the use and integration of Zoltan with other matrix-based frameworks, such as the ones in Trilinos, we developed Isorropia as a Trilinos package, which supports most of Zoltan's features via a matrix-based interface. In addition to providing an easy-to-use matrix-based interface to Zoltan, Isorropia also serves as a platform for additional matrix algorithms. In this paper, we give an overview of the Zoltan and Isorropia toolkits, their design, capabilities and use. We also show how Zoltan and Isorropia enable large-scale, parallel scientific simulations, and describe current and future development in the next-generation package Zoltan2.

scholarly journals Review of Research on Physical and Mechanical Properties and Engineering Application of Metal Tailings

2022 ◽  
Vol 2148 (1) ◽  
pp. 012059
Author(s):  
Shengfu Wang ◽  
Lechen Yan ◽  
Kaixi Xue ◽  
Liang Lv ◽  
Dongjie Zhang ◽  
...  
Keyword(s):  
Mechanical Properties ◽  
Large Scale ◽  
Building Materials ◽  
Engineering Application ◽  
Physical And Mechanical Properties ◽  
Future Research ◽  
Tailing Dams ◽  
The Stability ◽  
Processing And Storage ◽  
And Storage

Abstract Processing and storage requirements for metal residues are becoming stricter to achieve the carbon neutralization target. The physical and mechanical properties of tailings affect the stability of tailing dams. Metal tailings can be used as secondary resources, and it is easy to pollute the environment under poorly managed conditions. Therefore, it is necessary and urgent to reuse these deposits such as iron tailings, copper tailings, zinc tailings et al. This article discusses the current research on the mechanical properties of metal tailings and its engineering application. Based on previous research, it is pointed out that there still needs more attention on the mechanical properties of metal tailing sands, especially under different conditions like dry-wet, freeze-thaw, dynamic loads and large-scale application. In the future, research on the filling of metal tailings as roadbed and new building materials will be one of the directions to solve the problem of tailing pond accumulation.

scholarly journals Standardization developments for large scale biobanks in smoking related diseases - a model system for blood sample processing and storage

2013 ◽  
Vol 1 (1) ◽  
pp. 14 ◽  
Author(s):  
Johan Malm ◽  
Thomas E Fehniger ◽  
Pia Danmyr ◽  
Ákos Végvári ◽  
Charlotte Welinder ◽  
...  
Keyword(s):  
Blood Sample ◽  
Large Scale ◽  
Model System ◽  
Sample Processing ◽  
Processing And Storage ◽  
And Storage

Limitations and Perspectives of Optically Switched Interconnects for Large-scale Data Processing and Storage Systems

2012 ◽  
Vol 1438 ◽  
Author(s):  
Slavisa Aleksic ◽  
Gerhard Schmid ◽  
Naida Fehratovic
Keyword(s):  
Power Consumption ◽  
Data Processing ◽  
Large Scale ◽  
Storage Systems ◽  
Reducing Power ◽  
Large Scale Data ◽  
Large Scale Data Processing ◽  
Processing And Storage ◽  
And Storage ◽  
Scale Data

ABSTRACTThe ever-growing Internet data traffic leads to a continuously increasing demand in both capacity and performance of large-scale Information and Communication (ICT) systems such as high-capacity routers and switches, large data centers, and supercomputers. Complex and spatially distributed multirack systems comprising a large number of data processing and storage modules with high-speed interfaces have already become reality. A consequence of this trend is that internal interconnection systems also become large and complex. Interconnection distances, total required number of cables, and power consumption increase rapidly with the increase in capacity, which can cause limitations in scalability of the whole system. This paper addresses requirements and limitations of intrasystem interconnects for application in large-scale data processing and storage systems. Various point-to-point and optically switched interconnection options are reviewed with regard to their potential to achieve large scalability while reducing power consumption.

Sign in / Sign up

Export Citation Format

Share Document