HPC Cloud Architecture to Reduce HPC Workflow Complexity in Containerized Environments

The complexity of high-performance computing (HPC) workflows is an important issue in the provision of HPC cloud services in most national supercomputing centers. This complexity problem is especially critical because it affects HPC resource scalability, management efficiency, and convenience of use. To solve this problem, while exploiting the advantage of bare-metal-level high performance, container-based cloud solutions have been developed. However, various problems still exist, such as an isolated environment between HPC and the cloud, security issues, and workload management issues. We propose an architecture that reduces this complexity by using Docker and Singularity, which are the container platforms most often used in the HPC cloud field. This HPC cloud architecture integrates both image management and job management, which are the two main elements of HPC cloud workflows. To evaluate the serviceability and performance of the proposed architecture, we developed and implemented a platform in an HPC cluster experiment. Experimental results indicated that the proposed HPC cloud architecture can reduce complexity to provide supercomputing resource scalability, high performance, user convenience, various HPC applications, and management efficiency.

Reaching new peaks for the future of the CMS HTCondor Global Pool

The CMS experiment at CERN employs a distributed computing infrastructure to satisfy its data processing and simulation needs. The CMS Submission Infrastructure team manages a dynamic HTCondor pool, aggregating mainly Grid clusters worldwide, but also HPC, Cloud and opportunistic resources. This CMS Global Pool, which currently involves over 70 computing sites worldwide and peaks at 350k CPU cores, is employed to successfully manage the simultaneous execution of up to 150k tasks. While the present infrastructure is sufficient to harness the current computing power scales, CMS latest estimates predict a noticeable expansion in the amount of CPU that will be required in order to cope with the massive data increase of the High-Luminosity LHC (HL-LHC) era, planned to start in 2027. This contribution presents the latest results of the CMS Submission Infrastructure team in exploring and expanding the scalability reach of our Global Pool, in order to preventively detect and overcome any barriers in relation to the HL-LHC goals, while maintaining high effciency in our workload scheduling and resource utilization.

Challenges and Opportunities in High Performance Cloud Computing

The scientist, engineers, and researchers highly need the high-performance computing (HPC) services for executing the energy, engineering, environmental sciences, weather, and life science simulations. The virtual machine (VM) or docker-enabled HPC Cloud service provides the advantages of consolidation and support for multiple users in public cloud environment. Adding the hypervisor on the top of bare metal hardware brings few challenges like the overhead of computation due to virtualization, especially in HPC environment. This chapter discusses the challenges, solutions, and opportunities due to input-output, VMM overheads, interconnection overheads, VM migration problems, and scalability problems in HPC Cloud. This chapter portrays HPC Cloud as highly complex distributed environment consisting of the heterogeneous types of architectures consisting of the different processor architectures, inter-connectivity techniques, the problems of the shared memory, distributed memory, and hybrid architectures in distributed computing like resilience, scalability, check-pointing, and fault tolerance.

EXPERIENCE OF FRC CSC RAS IN PROVIDING HPC CLOUD SERVICES FOR MATERIALS SCIENCE

The article presents the results of scientific research of the Federal Research Center "Computer Science and Control" of the Russian Academy of Sciences in field of providing of cloud services for material science.

Sosialisasi Implementasi Prototype Portal Manajemen Sumber Daya High Performance Computing (HPC): Simulasi Dinamika Molekular

Masalah pertama yang dihadapi terkait kegiatan ini adalah Penggunaan sumber daya High Performance Computing (HPC) membutuhkan fasilitas superkomputer yang sangat mahal, baik pengadaan maupun perawatannya. Sehingga fasilitas HPC tersebut hanya dimiliki institusi tertentu yang memiliki sumber pendanaan cukup besar. Terutama di Indonesia, mungkin hanya segelintir lembaga pendidikan dan penelitian yang mampu memilikinya. Hal ini mengakibatkan, pemanfaatan HPC untuk penelitian menjadi terbatas, karena sangat sedikit sekali aktivitas penelitian yang memiliki akses ke fasilitas HPC tertentu. Sehingga hal ini menjadi suatu hambatan tersendiri, terutama untuk kasus penelitian yang menuntut sumber daya komputasi besar. Masalah kedua yakni para peneliti yang umumnya berasal dari berbagai macam disiplin ilmu pengetahuan sering tidak memiliki kemampuan tentang bagaimana menggunakan infrastruktur HPC tersebut. Umumnya, pengguna HPC cloud akan diberikan beberapa server virtual, kemudian server virtual tersebut harus disiapkan secara mandiri sesuai kebutuhan aplikasinya. Setup tersebut berkaitan dengan instalasi Sistem operasi, midleware, aplikasi, serta beberapa konfigurasi yang tidak sederhana. (Rajan et all, 2011) Sehingga, peneliti tersebut harus bertambah pekerjaan dan waktu tambahan untuk mempelajari suatu kemampuan lain yang cukup rumit di luar esensi penelitian itu sendiri agar mampu menggunakan cloud IAAS tersebut Untuk mengatasi masalah masalah pertama tersebut, muncul satu alternatif solusi, yaitu dengan penggunaan layanan cloud Infrastruktur-as-a-Service (IAAS), di mana layanan cloud tersebut menyediakan infrastruktur HPC. Layanan infrastruktur tersebut meliputi prosesor, memory, storage, jaringan internet, listrik serta perawatan. Saat ini banyak bermunculan vendor IAAS, seperti Amazon EC2 (Elastic Computing Cloud for Computing Service), S3 (Simple Storage Service), Microsoft Azure (PAAS), Google AppEngine, dan lainnya. Penulis telah mengembangkan prototype portal Sumber Daya HPC untuk simulasi dinamika molekuler sebagai output dari kegiatan penelitian beberapa tahun belakangan ini. Dalam kegiatan ini, dilakukan ujicoba implementasi prototype tersebut kepada usernya yakni para peneliti baik dosen dan mahasiswa. Sosialisasi pengenalan dan ujicoba prototype tersebut telah dilakukan kepada beberapa rekan dosen, peneliti dan mahasiswa di Universitas Padjajadan dan Institute Teknologi Bandung. Berdasarkan hasil kuesioner kegiatan sosialisasi ini, seluruh peserta merasa puas dengan kegiatan sosialisasi ini dan menganggap prototype tersebut dapat membantu memperbaiki kondisi mereka. Sistem yang diperkenalkan ini juga dianggap sesuai oleh seluruh peserta untuk mengangkat potensi bidang mereka (farmasi/kimia). Sebagian besar peserta juga merasa puas dengan acara yang diselenggarakan ini dan merasa cukup mampu untuk memanfaatkan sistem ini secara mandiri tanpa bantuan/pendampingan dari tim UI.