Job scheduling for data-parallel frameworks with hybrid electrical/optical datacenter networks

Author(s):  
Zhuozhao Li ◽  
Haiying Shen
2010 ◽  
Vol 14 (2) ◽  
pp. 183-197 ◽  
Author(s):  
Hyuck Han ◽  
Hyungsoo Jung ◽  
Hyeonsang Eom ◽  
Heon Y. Yeom

2012 ◽  
Vol 239-240 ◽  
pp. 599-602
Author(s):  
Xing Wen Cai ◽  
Jian Hu ◽  
Zi Yang Li ◽  
Bo Zhu

Parallel computing technology has been widely used to process massive remote sensing data high efficiently. In order to simplify the development of remote sensing data parallel processing system and consider about the characteristics of remote sensing data pre-processing, this paper designs a cluster-based universal parallel processing framework. The framework encapsulates parallel job scheduling and management, adapts the strategy of components development, provides the simple interface for the users to develop new functionalities by adding new data-processing components into the framework. Basing on Message Passing Interface (MPI), the framework is implemented. Experiments, such as adding remote sensing data extracting, radiometric correction and geometric correction into the framework, show that the framework performed well in computing efficiency and speedup rate.


AIAA Journal ◽  
1998 ◽  
Vol 36 ◽  
pp. 1603-1609 ◽  
Author(s):  
Michael J. Wright ◽  
Graham V. Candler ◽  
Deepak Bose

2010 ◽  
Vol 30 (8) ◽  
pp. 2066-2069
Author(s):  
Yong-cai TAO ◽  
Lei SHI

2020 ◽  
Vol 15 ◽  
Author(s):  
Weiwen Zhang ◽  
Long Wang ◽  
Theint Theint Aye ◽  
Juniarto Samsudin ◽  
Yongqing Zhu

Background: Genotype imputation as a service is developed to enable researchers to estimate genotypes on haplotyped data without performing whole genome sequencing. However, genotype imputation is computation intensive and thus it remains a challenge to satisfy the high performance requirement of genome wide association study (GWAS). Objective: In this paper, we propose a high performance computing solution for genotype imputation on supercomputers to enhance its execution performance. Method: We design and implement a multi-level parallelization that includes job level, process level and thread level parallelization, enabled by job scheduling management, message passing interface (MPI) and OpenMP, respectively. It involves job distribution, chunk partition and execution, parallelized iteration for imputation and data concatenation. Due to the design of multi-level parallelization, we can exploit the multi-machine/multi-core architecture to improve the performance of genotype imputation. Results: Experiment results show that our proposed method can outperform the Hadoop-based implementation of genotype imputation. Moreover, we conduct the experiments on supercomputers to evaluate the performance of the proposed method. The evaluation shows that it can significantly shorten the execution time, thus improving the performance for genotype imputation. Conclusion: The proposed multi-level parallelization, when deployed as an imputation as a service, will facilitate bioinformatics researchers in Singapore to conduct genotype imputation and enhance the association study.


Sign in / Sign up

Export Citation Format

Share Document