scholarly journals Skip high-volume data transfer and access free computing resources for your CMIP6 multi-model analyses

2021 ◽  
Author(s):  
Maria Moreno de Castro ◽  
Marco Kulüke ◽  
Fabian Wachsmann ◽  
Regina Kwee-Hinzmann ◽  
Stephan Kindermann ◽  
...  
Keyword(s):  
High Performance ◽  
Process Model ◽  
Impact Analysis ◽  
Data Transfer ◽  
Computing Time ◽  
High Volume ◽  
System Modelling ◽  
Earth System ◽  
Volume Data ◽  
Climate Data

<p>Tired of downloading tons of model results? Is your internet connection flakey? Are you about to overload your computer’s memory with the constant increase of data volume and you need more computing resources? You can request free of charge computing time at one of the supercomputers of the Infrastructure of the European Network of Earth System modelling (IS-ENES)<sup>1</sup>, the European part of Earth System Grid Federation (ESGF)<sup>2</sup>, which also hosts and maintains more than 6 Petabytes of CMIP6 and CORDEX data.</p><p>Thanks to this new EU Comission funded service, you can run your own scripts in your favorite programming language and straightforward pre- and post-process model data. There is no need for heavy data transfer, just load with one line of code the data slice you need because your script will directly access the data pool. Therefore, days-lasting calculations will be done in seconds. You can test the service, we very easily provide pre-access activities.</p><p>In this session we will run Jupyter notebooks directly on the German Climate Computing Center (DKRZ)<sup>3</sup>, one of the ENES high performance computers and a ESGF data center, showing how to load, filter, concatenate, take means, and plot several CMIP6 models to compare their results, use some CMIP6 models to calculate some climate indexes for any location and period, and evaluate model skills with observational data. We will use Climate Data Operators (cdo)<sup>4</sup> and Python packages for Big Data manipulation, as Intake<sup>5</sup>, to easily extract the data from the huge catalog, and Xarray<sup>6</sup>, to easily read NetDCF files and scale to parallel computing. We are continuously creating more use cases for multi-model evaluation, mechanisms of variability, and impact analysis, visit the demos, find more information, and apply here: https://portal.enes.org/data/data-metadata-service/analysis-platforms.<br><br>[1] https://is.enes.org/<br>[2] https://esgf.llnl.gov/<br>[3] https://www.dkrz.de/<br>[4] https://code.mpimet.mpg.de/projects/cdo/<br>[5] https://intake.readthedocs.io/en/latest/<br>[6] http://xarray.pydata.org/en/stable/</p>

scholarly journals Broadening access to supercomputers for CMIP6 and CORDEX multimodel comparisons

2020 ◽  
Author(s):  
Stephan Kindermann ◽  
Maria Moreno
Keyword(s):  
High Performance ◽  
Data Science ◽  
High Volume ◽  
System Modelling ◽  
Earth System ◽  
Volume Data ◽  
Server Side ◽  
Network Bandwidth ◽  
Data Collections ◽  
Client Side

<p>We will present a new service designed to assist the users of model data in running their analyses in world-class supercomputers. The increase of data volumes and model complexities can be challenging for data users with limited access to high performance computers or low network bandwidth. To avoid heavy data transfers, strong memory requirements, and slow sequential processing, the data science community is rapidly moving from classical client-side to new server-side frameworks. Three simple steps enable server-side users to compute in parallel and near the data: (1) discover the data you are interested in, (2) perform your analyses and visualizations in the supercomputer, and (3) download the outcome. A server-side service is especially beneficial for exploiting the high-volume data collections produced in the framework of internationally coordinated model intercomparison projects like CMIP5/6 and CORDEX and disseminated via the  Earth System Grid Federation (ESGF) infrastructure. To facilitate the adoption of server-side capabilities by the ESGF users, the infrastructure project of the European Network for Earth System Modelling (IS-ENES3) is now opening its high performance resources and data pools at the CMCC (Italy), JASMIN (UK), IPSL (France), and DKRZ (Germany) supercomputing centers. The data pools allow access to results from several models on the same site and the data and resources are locally maintained by the hosts. Besides, our server-side framework not only speeds the workload but also reduces the errors in file format conversions and standardizations and software dependencies and upgrade. The service is founded by the EU Commission and it is free of charge. Find more information here: https://portal.enes.org/data/data-metadata-service/analysis-platforms. Demos and tutorials have been created by a dedicated user support team. We will present several use cases showing how easy and flexible it is to use our analysis platforms for multimodel comparisons of CMIP5/6 and CORDEX data. </p>

Boosting climate change research with direct access to high performance computers

2020 ◽  
Author(s):  
Maria Moreno de Castro ◽  
Stephan Kindermann ◽  
Sandro Fiore ◽  
Paola Nassisi ◽  
Guillaume Levavasseur ◽  
...  
Keyword(s):  
Climate Change ◽  
High Performance Computing ◽  
High Performance ◽  
System Modelling ◽  
Earth System ◽  
Climate Data ◽  
Climate Change Research ◽  
Detection And Attribution ◽  
Wide Range ◽  
Performance Computing

<p>Earth System observational and model data volumes are constantly increasing and it can be challenging to discover, download, and analyze data if scientists do not have the required computing and storage resources at hand. This is especially the case for detection and attribution studies in the field of climate change research since we need to perform multi-source and cross-disciplinary comparisons for datasets of high-spatial and large temporal coverage. Researchers and end-users are therefore looking for access to cloud solutions and high performance compute facilities. The Earth System Grid Federation (ESGF, https://esgf.llnl.gov/) maintains a global system of federated data centers that allow access to the largest archive of model climate data world-wide. ESGF portals provide free access to the output of the data contributing to the next assessment report of the Intergovernmental Panel on Climate Change through the Coupled Model Intercomparison Project. In order to support users to directly access to high performance computing facilities to perform analyses such as detection and attribution of climate change and its impacts, the EU Commission funded a new service within the infrastructure of the European Network for Earth System Modelling (ENES, https://portal.enes.org/data/data-metadata-service/analysis-platforms). This new service is designed to reduce data transfer issues, speed up the computational analysis, provide storage, and ensure the resources access and maintenance. Furthermore, the service is free of charge, only requires a lightweight application. We will present a demo on how flexible it is to calculate climate indices from different ESGF datasets covering a wide range of temporal and spatial scales using cdo (Climate Data Operators, https://code.mpimet.mpg.de/projects/cdo/) and Jupyter notebooks running directly on the ENES partners: the DKRZ (Germany), JASMIN (UK), CMCC(Italy), and IPSL (France) high performance computing centers.</p>

REAL-TIME HIGH-VOLUME DATA TRANSFER AND PROCESSING FOR e-VLBI

2010 ◽  
pp. 81-90
Author(s):  
YASUHIRO KOYAMA ◽  
TETSURO KONDO ◽  
MORITAKA KIMURA ◽  
MASAKI HIRABARU ◽  
HIROSHI TAKEUCHI
Keyword(s):  
Real Time ◽  
Data Transfer ◽  
High Volume ◽  
Volume Data

scholarly journals ESM-Tools Version 4.0: A modular infrastructure for stand-alone and coupled Earth System Modelling (ESM)

2020 ◽  
Author(s):  
Dirk Barbi ◽  
Nadine Wieters ◽  
Paul Gierz ◽  
Fatemeh Chegini ◽  
Sara Khosravi ◽  
...  
Keyword(s):  
Land Surface ◽  
High Performance ◽  
Surface Model ◽  
System Modelling ◽  
Climate Modelling ◽  
Earth System ◽  
Coupled Models ◽  
Wide Range ◽  
Runtime Infrastructure ◽  
Earth System Modelling

Abstract. Earth system and climate modelling involves the simulation of processes on a wide range of scales and within and across various components of the Earth system. In practice, component models are often developed independently by different research groups and then combined using a dedicated coupling software. This procedure not only leads to a strongly growing number of available versions of model components and coupled setups but also to model- and system-dependent ways of obtaining and operating them. Therefore, implementing these Earth System Models (ESMs) can be challenging and extremely time-consuming, especially for less experienced modellers, or scientists aiming to use different ESMs as in the case of inter-comparison projects. To assist researchers and modellers by reducing avoidable complexity, we developed the ESM-Tools software, which provides a standard way for downloading, configuring, compiling, running and monitoring different models - coupled ESMs and stand-alone models alike - on a variety of High-Performance Computing (HPC) systems. (The ESM-Tools are equally applicable and helpful for stand-alone as for coupled models. In fact, the ESM-Tools are used as standard compile and runtime infrastructure for FESOM2, and currently also applied for ECHAM and ICON standalone simulations. As coupled ESMs are technically the more challenging tasks, we will focus on coupled setups, always implying that stand-alone models can benefit in the same way.) With the ESM-Tools, the user is only required to provide a short script consisting of only the experiment specific definitions, while the software executes all the phases of a simulation in the correct order. The software, which is well documented and easy to install and use, currently supports four ocean models, three atmosphere models, two biogeochemistry models, an ice sheet model, an isostatic adjustment model, a hydrology model and a land-surface model. ESM-Tools has been entirely re-coded in a high-level programming language (Python) and provides researchers with an even more user-friendly interface for Earth system modelling lately. The ESM-Tools were developed within the framework of the project Advanced Earth System Model Capacity, supported by the Helmholtz Association.

scholarly journals Performance gains in an ESM using parallel ad-hoc file systems

2020 ◽  
Author(s):  
Stefan Versick ◽  
Ole Kirner ◽  
Jörg Meyer ◽  
Holger Obermaier ◽  
Mehmet Soysal
Keyword(s):  
High Performance ◽  
Ad Hoc ◽  
File System ◽  
File Systems ◽  
System Modelling ◽  
Earth System ◽  
Post Processing ◽  
In Situ Data ◽  
The Impact

<p>Earth System Models (ESM) got much more demanding over the last years. Modelled processes got more complex and more and more processes are considered in models. In addition resolutions of the models got higher to improve weather and climate forecasts. This requires faster high performance computers (HPC) and better I/O performance.</p><p>Within our Pilot Lab Exascale Earth System Modelling (PL-EESM) we do performance analysis of the ESM EMAC using a standard Lustre file system for output and compare it to the performance using a parallel ad-hoc overlay file system. We will show the impact for two scenarios: one for todays standard amount of output and one with artificial heavy output simulating future ESMs.</p><p>An ad-hoc file system is a private parallel file system which is created on-demand for an HPC job using the node-local storage devices, in our case solid-state-disks (SSD). It only exists during the runtime of the job. Therefore output data have to be moved to a permanent file system before the job has finished. Quasi in-situ data analysis and post-processing allows to gain performance as it might result in a decreased amount of data which you have to store - saving disk space and time during the transfer of data to permanent storage. We will show first tests for quasi in-situ post-processing.</p>

Spin-Based Fully Nonvolatile Full-Adder Circuit for Computing in Memory

SPIN ◽  
2019 ◽  
Vol 09 (01) ◽  
pp. 1950007 ◽  
Author(s):  
Abdolah Amirany ◽  
Ramin Rajaei
Keyword(s):  
High Performance ◽  
Soft Errors ◽  
Magnetic Tunnel Junction ◽  
High Volume ◽  
Full Adder ◽  
Cmos Technology ◽  
Logic Circuits ◽  
Volume Data ◽  
Single Node ◽  
Radiation Induced

As CMOS technology scales down toward below 2-digit nanometer dimensions, exponentially increasing leakage power, vulnerability to radiation induced soft errors have become a major problem in today’s logic circuits. Emerging spin-based logic circuits and architectures based on nonvolatile magnetic tunnel junction (MTJ) cells show a great potential to overcome the aforementioned issues. However, radiation induced soft errors are still a problem in MTJ-based circuits as they need sequential peripheral CMOS circuits for sensing the MTJs. This paper proposes a novel nonvolatile and low-cost radiation hardened magnetic full adder (MFA). In comparison with the previous designs, the proposed MFA is capable of tolerating particle strikes regardless of the amount of charge induced to a single node and even multiple nodes. Besides, the proposed MFA offers low power operation, low area and high performance as compared with previous counterparts. One of the most important features suggested by the proposed MFA circuit is full nonvolatility. Nonvolatile logic circuits remove the cost of high volume data transactions between memory and logic and also facilitate power gating in logic-in-memory architectures.

scholarly journals ESM-Tools version 5.0: a modular infrastructure for stand-alone and coupled Earth system modelling (ESM)

2021 ◽  
Vol 14 (6) ◽  
pp. 4051-4067
Author(s):  
Dirk Barbi ◽  
Nadine Wieters ◽  
Paul Gierz ◽  
Miguel Andrés-Martínez ◽  
Deniz Ural ◽  
...  
Keyword(s):  
Land Surface ◽  
High Performance ◽  
Land Surface Model ◽  
Management Tool ◽  
Surface Model ◽  
System Modelling ◽  
Climate Modelling ◽  
Earth System ◽  
Wide Range ◽  
Earth System Modelling

Abstract. Earth system and climate modelling involves the simulation of processes on a wide range of scales and within and across various compartments of the Earth system. In practice, component models are often developed independently by different research groups, adapted by others to their special interests and then combined using a dedicated coupling software. This procedure not only leads to a strongly growing number of available versions of model components and coupled setups but also to model- and high-performance computing (HPC)-system-dependent ways of obtaining, configuring, building and operating them. Therefore, implementing these Earth system models (ESMs) can be challenging and extremely time consuming, especially for less experienced modellers or scientists aiming to use different ESMs as in the case of intercomparison projects. To assist researchers and modellers by reducing avoidable complexity, we developed the ESM-Tools software, which provides a standard way for downloading, configuring, compiling, running and monitoring different models on a variety of HPC systems. It should be noted that ESM-Tools is not a coupling software itself but a workflow and infrastructure management tool to provide access to increase usability of already existing components and coupled setups. As coupled ESMs are technically the more challenging tasks, we will focus on coupled setups, always implying that stand-alone models can benefit in the same way. With ESM-Tools, the user is only required to provide a short script consisting of only the experiment-specific definitions, while the software executes all the phases of a simulation in the correct order. The software, which is well documented and easy to install and use, currently supports four ocean models, three atmosphere models, two biogeochemistry models, an ice sheet model, an isostatic adjustment model, a hydrology model and a land-surface model. Compared to previous versions, ESM-Tools has lately been entirely recoded in a high-level programming language (Python) and provides researchers with an even more user-friendly interface for Earth system modelling. ESM-Tools was developed within the framework of the Advanced Earth System Model Capacity project, supported by the Helmholtz Association.

scholarly journals On-Chip Power Minimization Using Serialization-Widening with Frequent Value Encoding

VLSI Design ◽  
2014 ◽  
Vol 2014 ◽  
pp. 1-14 ◽  
Author(s):  
Khader Mohammad ◽  
Ahsan Kabeer ◽  
Tarek Taha
Keyword(s):  
Power Consumption ◽  
High Performance ◽  
Chip Multiprocessors ◽  
Data Transfer ◽  
High Volume ◽  
Power Minimization ◽  
Processor Core ◽  
L2 Cache ◽  
Data Bus ◽  
On Chip

In chip-multiprocessors (CMP) architecture, the L2 cache is shared by the L1 cache of each processor core, resulting in a high volume of diverse data transfer through the L1-L2 cache bus. High-performance CMP and SoC systems have a significant amount of data transfer between the on-chip L2 cache and the L3 cache of off-chip memory through the power expensive off-chip memory bus. This paper addresses the problem of the high-power consumption of the on-chip data buses, exploring a framework for memory data bus power consumption minimization approach. A comprehensive analysis of the existing bus power minimization approaches is provided based on the performance, power, and area overhead consideration. A novel approaches for reducing the power consumption for the on-chip bus is introduced. In particular, a serialization-widening (SW) of data bus with frequent value encoding (FVE), called the SWE approach, is proposed as the best power savings approach for the on-chip cache data bus. The experimental results show that the SWE approach with FVE can achieve approximately 54% power savings over the conventional bus for multicore applications using a 64-bit wide data bus in 45 nm technology.

scholarly journals Monoclonal Antibody-Based Immunotherapy and Its Role in the Development of Cardiac Toxicity

Cancers ◽  
2020 ◽  
Vol 13 (1) ◽  
pp. 86
Author(s):  
Mohit Kumar ◽  
Chellappagounder Thangavel ◽  
Richard C. Becker ◽  
Sakthivel Sadayappan
Keyword(s):  
Cancer Patients ◽  
Immune Modulation ◽  
Checkpoint Inhibitors ◽  
Survival Rates ◽  
Induced Pluripotent Stem Cell ◽  
High Volume ◽  
Oncolytic Viruses ◽  
Model Systems ◽  
Volume Data ◽  
Late Cardiotoxicity

Immunotherapy is one of the most effective therapeutic options for cancer patients. Five specific classes of immunotherapies, which includes cell-based chimeric antigenic receptor T-cells, checkpoint inhibitors, cancer vaccines, antibody-based targeted therapies, and oncolytic viruses. Immunotherapies can improve survival rates among cancer patients. At the same time, however, they can cause inflammation and promote adverse cardiac immune modulation and cardiac failure among some cancer patients as late as five to ten years following immunotherapy. In this review, we discuss cardiotoxicity associated with immunotherapy. We also propose using human-induced pluripotent stem cell-derived cardiomyocytes/ cardiac-stromal progenitor cells and cardiac organoid cultures as innovative experimental model systems to (1) mimic clinical treatment, resulting in reproducible data, and (2) promote the identification of immunotherapy-induced biomarkers of both early and late cardiotoxicity. Finally, we introduce the integration of omics-derived high-volume data and cardiac biology as a pathway toward the discovery of new and efficient non-toxic immunotherapy.

scholarly journals Compiler-directed scratchpad memory data transfer optimization for multithreaded applications on a heterogeneous many-core architecture

2021 ◽  
Author(s):  
Xiaohan Tao ◽  
Jianmin Pang ◽  
Jinlong Xu ◽  
Yu Zhu
Keyword(s):  
Energy Consumption ◽  
High Performance ◽  
Scientific Computing ◽  
Data Transfer ◽  
Performance Model ◽  
Experimental Result ◽  
Transfer Model ◽  
Scratchpad Memory ◽  
On Chip ◽  
Many Core

AbstractThe heterogeneous many-core architecture plays an important role in the fields of high-performance computing and scientific computing. It uses accelerator cores with on-chip memories to improve performance and reduce energy consumption. Scratchpad memory (SPM) is a kind of fast on-chip memory with lower energy consumption compared with a hardware cache. However, data transfer between SPM and off-chip memory can be managed only by a programmer or compiler. In this paper, we propose a compiler-directed multithreaded SPM data transfer model (MSDTM) to optimize the process of data transfer in a heterogeneous many-core architecture. We use compile-time analysis to classify data accesses, check dependences and determine the allocation of data transfer operations. We further present the data transfer performance model to derive the optimal granularity of data transfer and select the most profitable data transfer strategy. We implement the proposed MSDTM on the GCC complier and evaluate it on Sunway TaihuLight with selected test cases from benchmarks and scientific computing applications. The experimental result shows that the proposed MSDTM improves the application execution time by 5.49$$\times$$ × and achieves an energy saving of 5.16$$\times$$ × on average.

Sign in / Sign up

Export Citation Format

Share Document