Using Managed High Performance Computing Systems for High-Throughput Computing

2016 ◽  
pp. 61-79 ◽  
Author(s):  
Lucas A. Wilson
Keyword(s):  
High Performance Computing ◽  
High Throughput ◽  
High Performance ◽  
Computing Systems ◽  
High Throughput Computing ◽  
Performance Computing

scholarly journals Running IceCube GPU simulations on Titan

2019 ◽  
Vol 214 ◽  
pp. 03024
Author(s):  
Vladimir Brik ◽  
David Schultz ◽  
Gonzalo Merino
Keyword(s):  
High Performance Computing ◽  
High Throughput ◽  
High Performance ◽  
Management Systems ◽  
Workload Management ◽  
High Throughput Computing ◽  
Large Numbers ◽  
Mpi Applications ◽  
Performance Computing

Here we report IceCube’s first experiences of running GPU simulations on the Titan supercomputer. This undertaking was non-trivial because Titan is designed for High Performance Computing (HPC) workloads, whereas IceCube’s workloads fall under the High Throughput Computing (HTC) category. In particular: (i) Titan’s design, policies, and tools are geared heavily toward large MPI applications, while IceCube’s workloads consist of large numbers of relatively small independent jobs, (ii) Titan compute nodes run Cray Linux, which is not directly compatible with IceCube software, and (iii) Titan compute nodes cannot access outside networks, making it impossible to access IceCube’s CVMFS repositories and workload management systems. This report examines our experience of packaging our application in Singularity containers and using HTCondor as the second-level scheduler on the Titan supercomputer.

scholarly journals Cosmos : A Unified Accounting System both for the HTCondor and Slurm Clusters at IHEP

2020 ◽  
Vol 245 ◽  
pp. 07060
Author(s):  
Ran Du ◽  
Jingyan Shi ◽  
Xiaowei Jiang ◽  
Jiaheng Zou
Keyword(s):  
High Performance Computing ◽  
Virtual Machine ◽  
High Throughput ◽  
High Performance ◽  
Accounting System ◽  
High Throughput Computing ◽  
Parallel Jobs ◽  
The Status ◽  
Set Up ◽  
Performance Computing

HTCondor was adopted to manage the High Throughput Computing (HTC) cluster at IHEP in 2016. In 2017 a Slurm cluster was set up to run High Performance Computing (HPC) jobs. To provide accounting services for these two clusters, we implemented a unified accounting system named Cosmos. Multiple workloads bring different accounting requirements. Briefly speaking, there are four types of jobs to account. First of all, 30 million single-core jobs run in the HTCondor cluster every year. Secondly, Virtual Machine (VM) jobs run in the legacy HTCondor VM cluster. Thirdly, parallel jobs run in the Slurm cluster, and some of these jobs are run on the GPU worker nodes to accelerate computing. Lastly, some selected HTC jobs are migrated from the HTCondor cluster to the Slurm cluster for research purposes. To satisfy all the mentioned requirements, Cosmos is implemented with four layers: acquisition, integration, statistics and presentation. Details about the issues and solutions of each layer will be presented in the paper. Cosmos has run in production for two years, and the status shows that it is a well-functioning system, also meets the requirements of the HTCondor and Slurm clusters.

Sign in / Sign up

Export Citation Format

Share Document