SW-LZMA: Parallel Implementation of LZMA Based on SW26010 Many-Core Processor

With the development of high-performance computing and big data applications, the scale of data transmitted, stored, and processed by high-performance computing cluster systems is increasing explosively. Efficient compression of large-scale data and reducing the space required for data storage and transmission is one of the keys to improving the performance of high-performance computing cluster systems. In this paper, we present SW-LZMA, a parallel design and optimization of LZMA based on the Sunway 26010 heterogeneous many-core processor. Combined with the characteristics of SW26010 processors, we analyse the storage space requirements, memory access characteristics, and hotspot functions of the LZMA algorithm and implement the thread-level parallelism of the LZMA algorithm based on Athread interface. Furthermore, we make a fine-grained layout of LDM address space to achieve DMA double buffer cyclic sliding window algorithm, which optimizes the performance of SW-LZMA. The experimental results show that compared with the serial baseline implementation of LZMA, the parallel LZMA algorithm obtains a maximum speedup ratio of 4.1 times using the Silesia corpus benchmark, while on the large-scale data set, speedup is 5.3 times.

Joint use of OpenMP and MPI technologies on the nodes of the computational cluster

This work is devoted to the problem of implementing an efficient parallel program that solves the asigned task using the maximum available amount of computing cluster resources in order to obtain the corresponding gain in performance with respect to the sequential version of the algorithm. The main objective of the work was to study the possibilities of joint use of the parallelization technologies OpenMP and MPI, considering the characteristics and features of the problems being solved, to increase the performance of executing parallel algorithms and programs on a computing cluster. This article provides a brief overview of approaches to calculating the sequential programs complexity functions. To determine the parallel programs complexity, an approach based on operational analysis was used. The features of the sequential programs parallelization technologies OpenMP and MPI are described. The main software and hardware factors affecting the execution speed of parallel programs on the nodes of a computing cluster are presented. The main attention in this paper is paid to the study of the impact on performance of computational and exchange operations number ratio in programs. To implement the research, parallel OpenMP and MPI testing programs were developed, in which the total number of operations and the correlation between computational and exchange operations are set. A computing cluster consisting of several nodes was used as a hardware and software platform. Experimental studies have made it possible to confirm the effectiveness of the hybrid model of a parallel program in multi-node systems with heterogeneous memory using OpenMP in shared memory subsystems, and MPI in a distributed memory subsystems

Orquestração de Aplicações de Computação de Alta Performance em Ambientes Cloud Conteinerizados

Este trabalho explora a orquestração da plataforma HPCC Systems (High Performance Computing Cluster) em ambientes cloud conteinerizados, com a ferramenta de orquestração Kubernetes. O objetivo do trabalho é avaliar as características, benefícios e desafios da implantação da plataforma HPCC Systems nesse paradigma através de diferentes provedores de cloud pública, especificamente Amazon Web Service (AWS) e Microsoft Azure. Os resultados preliminares sugerem que o paradigma de orquestração traz diversos benefícios para a plataforma em questão, mas suposições estritas sobre persistência de armazenamento de dados e recursos compartilhados específicos ao host, entre outras condições, geram desafios ao tentar levar a tecnologia a este ambiente.

Simple and Scalable Streaming: The GRETA Data Pipeline

The Gamma Ray Energy Tracking Array (GRETA) is a state of the art gamma-ray spectrometer being built at Lawrence Berkeley National Laboratory to be first sited at the Facility for Rare Isotope Beams (FRIB) at Michigan State University. A key design requirement for the spectrometer is to perform gamma-ray tracking in near real time. To meet this requirement we have used an inline, streaming approach to signal processing in the GRETA data acquisition system, using a GPU-equipped computing cluster. The data stream will reach 480 thousand events per second at an aggregate data rate of 4 gigabytes per second at full design capacity. We have been able to simplify the architecture of the streaming system greatly by interfacing the FPGA-based detector electronics with the computing cluster using standard network technology. A set of highperformance software components to implement queuing, flow control, event processing and event building have been developed, all in a streaming environment which matches detector performance. Prototypes of all high-performance components have been completed and meet design specifications.