scholarly journals Broadening access to supercomputers for CMIP6 and CORDEX multimodel comparisons

2020 ◽  
Author(s):  
Stephan Kindermann ◽  
Maria Moreno
Keyword(s):  
High Performance ◽  
Data Science ◽  
High Volume ◽  
System Modelling ◽  
Earth System ◽  
Volume Data ◽  
Server Side ◽  
Network Bandwidth ◽  
Data Collections ◽  
Client Side

<p>We will present a new service designed to assist the users of model data in running their analyses in world-class supercomputers. The increase of data volumes and model complexities can be challenging for data users with limited access to high performance computers or low network bandwidth. To avoid heavy data transfers, strong memory requirements, and slow sequential processing, the data science community is rapidly moving from classical client-side to new server-side frameworks. Three simple steps enable server-side users to compute in parallel and near the data: (1) discover the data you are interested in, (2) perform your analyses and visualizations in the supercomputer, and (3) download the outcome. A server-side service is especially beneficial for exploiting the high-volume data collections produced in the framework of internationally coordinated model intercomparison projects like CMIP5/6 and CORDEX and disseminated via the  Earth System Grid Federation (ESGF) infrastructure. To facilitate the adoption of server-side capabilities by the ESGF users, the infrastructure project of the European Network for Earth System Modelling (IS-ENES3) is now opening its high performance resources and data pools at the CMCC (Italy), JASMIN (UK), IPSL (France), and DKRZ (Germany) supercomputing centers. The data pools allow access to results from several models on the same site and the data and resources are locally maintained by the hosts. Besides, our server-side framework not only speeds the workload but also reduces the errors in file format conversions and standardizations and software dependencies and upgrade. The service is founded by the EU Commission and it is free of charge. Find more information here: https://portal.enes.org/data/data-metadata-service/analysis-platforms. Demos and tutorials have been created by a dedicated user support team. We will present several use cases showing how easy and flexible it is to use our analysis platforms for multimodel comparisons of CMIP5/6 and CORDEX data. </p>

scholarly journals Skip high-volume data transfer and access free computing resources for your CMIP6 multi-model analyses

2021 ◽  
Author(s):  
Maria Moreno de Castro ◽  
Marco Kulüke ◽  
Fabian Wachsmann ◽  
Regina Kwee-Hinzmann ◽  
Stephan Kindermann ◽  
...  
Keyword(s):  
High Performance ◽  
Process Model ◽  
Impact Analysis ◽  
Data Transfer ◽  
Computing Time ◽  
High Volume ◽  
System Modelling ◽  
Earth System ◽  
Volume Data ◽  
Climate Data

<p>Tired of downloading tons of model results? Is your internet connection flakey? Are you about to overload your computer’s memory with the constant increase of data volume and you need more computing resources? You can request free of charge computing time at one of the supercomputers of the Infrastructure of the European Network of Earth System modelling (IS-ENES)<sup>1</sup>, the European part of Earth System Grid Federation (ESGF)<sup>2</sup>, which also hosts and maintains more than 6 Petabytes of CMIP6 and CORDEX data.</p><p>Thanks to this new EU Comission funded service, you can run your own scripts in your favorite programming language and straightforward pre- and post-process model data. There is no need for heavy data transfer, just load with one line of code the data slice you need because your script will directly access the data pool. Therefore, days-lasting calculations will be done in seconds. You can test the service, we very easily provide pre-access activities.</p><p>In this session we will run Jupyter notebooks directly on the German Climate Computing Center (DKRZ)<sup>3</sup>, one of the ENES high performance computers and a ESGF data center, showing how to load, filter, concatenate, take means, and plot several CMIP6 models to compare their results, use some CMIP6 models to calculate some climate indexes for any location and period, and evaluate model skills with observational data. We will use Climate Data Operators (cdo)<sup>4</sup> and Python packages for Big Data manipulation, as Intake<sup>5</sup>, to easily extract the data from the huge catalog, and Xarray<sup>6</sup>, to easily read NetDCF files and scale to parallel computing. We are continuously creating more use cases for multi-model evaluation, mechanisms of variability, and impact analysis, visit the demos, find more information, and apply here: https://portal.enes.org/data/data-metadata-service/analysis-platforms.<br><br>[1] https://is.enes.org/<br>[2] https://esgf.llnl.gov/<br>[3] https://www.dkrz.de/<br>[4] https://code.mpimet.mpg.de/projects/cdo/<br>[5] https://intake.readthedocs.io/en/latest/<br>[6] http://xarray.pydata.org/en/stable/</p>

scholarly journals A JavaScript API for the Ice Sheet System Model: towards on online interactive model for the Cryosphere Community

2016 ◽  
Author(s):  
Eric Larour ◽  
Daniel Cheng ◽  
Gilberto Perez ◽  
Justin Quinn ◽  
Mathieu Morlighem ◽  
...  
Keyword(s):  
High Performance ◽  
Ice Sheet ◽  
System Model ◽  
Earth System ◽  
Efficient Manner ◽  
Post Process ◽  
Software Model ◽  
Science Community ◽  
Deep Integration ◽  
Client Side

Abstract. Earth System Models (ESMs) are becoming increasingly complex, requiring extensive knowledge and experience to deploy and use in an efficient manner. They run on high-performance architectures that are significantly different from the everyday environments that scientists use to pre and post-process results (i.e. MATLAB, Python). This results in models that are hard to use for non specialists, and that are increasingly specific in their application. It also makes them relatively inaccessible to the wider science community, not to mention to the general public. Here, we present a new software/model paradigm that attempts to bridge the gap between the science community and the complexity of ESMs, by developing a new JavaScript Application Program Interface (API) for the Ice Sheet System Model (ISSM). The aforementioned API allows Cryosphere Scientists to run ISSM on the client-side of a webpage, within the JavaScript environment. When combined with a Web server running ISSM (using a Python API), it enables the serving of ISSM computations in an easy and straightforward way. The deep integration and similarities between all the APIs in ISSM (MATLAB, Python, and now JavaScript) significantly shortens and simplifies the turnaround of state-of-the-art science runs and their use by the larger community. We demonstrate our approach via a new Virtual Earth System Laboratory (VESL) Web site.

HTTPV

2014 ◽  
pp. 84-95
Author(s):  
Subrata Acharya
Keyword(s):  
Common Sense ◽  
Content Distribution ◽  
High Volume ◽  
Distribution Networks ◽  
Content Distribution Networks ◽  
Web Browser ◽  
Server Side ◽  
Integrity Protection ◽  
In Transit ◽  
Client Side

There is a need to be able to verify plaintext HTTP content transfers. Common sense dictates authentication and sensitive content should always be protected by SSL/HTTPS, but there is still great exploitation potential in the modification of static content in transit. Pre-computed signatures and client-side verification offers integrity protection of HTTP content in applications where SSL is not feasible. In this chapter, the authors demonstrate a mechanism by which a Web browser or other HTTP client can verify that content transmitted over an untrusted channel has not been modified. Verifiable HTTP is not intended to replace SSL. Rather, it is intended to be used in applications where SSL is not feasible, specifically, when serving high-volume static content and/or content from non-secure sources such as Content Distribution Networks. Finally, the authors find content verification is effective with server-side overhead similar to SSL. With future optimization such as native browser support, content verification could achieve comparable client-side efficiency.

scholarly journals ESM-Tools Version 4.0: A modular infrastructure for stand-alone and coupled Earth System Modelling (ESM)

2020 ◽  
Author(s):  
Dirk Barbi ◽  
Nadine Wieters ◽  
Paul Gierz ◽  
Fatemeh Chegini ◽  
Sara Khosravi ◽  
...  
Keyword(s):  
Land Surface ◽  
High Performance ◽  
Surface Model ◽  
System Modelling ◽  
Climate Modelling ◽  
Earth System ◽  
Coupled Models ◽  
Wide Range ◽  
Runtime Infrastructure ◽  
Earth System Modelling

Abstract. Earth system and climate modelling involves the simulation of processes on a wide range of scales and within and across various components of the Earth system. In practice, component models are often developed independently by different research groups and then combined using a dedicated coupling software. This procedure not only leads to a strongly growing number of available versions of model components and coupled setups but also to model- and system-dependent ways of obtaining and operating them. Therefore, implementing these Earth System Models (ESMs) can be challenging and extremely time-consuming, especially for less experienced modellers, or scientists aiming to use different ESMs as in the case of inter-comparison projects. To assist researchers and modellers by reducing avoidable complexity, we developed the ESM-Tools software, which provides a standard way for downloading, configuring, compiling, running and monitoring different models - coupled ESMs and stand-alone models alike - on a variety of High-Performance Computing (HPC) systems. (The ESM-Tools are equally applicable and helpful for stand-alone as for coupled models. In fact, the ESM-Tools are used as standard compile and runtime infrastructure for FESOM2, and currently also applied for ECHAM and ICON standalone simulations. As coupled ESMs are technically the more challenging tasks, we will focus on coupled setups, always implying that stand-alone models can benefit in the same way.) With the ESM-Tools, the user is only required to provide a short script consisting of only the experiment specific definitions, while the software executes all the phases of a simulation in the correct order. The software, which is well documented and easy to install and use, currently supports four ocean models, three atmosphere models, two biogeochemistry models, an ice sheet model, an isostatic adjustment model, a hydrology model and a land-surface model. ESM-Tools has been entirely re-coded in a high-level programming language (Python) and provides researchers with an even more user-friendly interface for Earth system modelling lately. The ESM-Tools were developed within the framework of the project Advanced Earth System Model Capacity, supported by the Helmholtz Association.

scholarly journals Performance gains in an ESM using parallel ad-hoc file systems

2020 ◽  
Author(s):  
Stefan Versick ◽  
Ole Kirner ◽  
Jörg Meyer ◽  
Holger Obermaier ◽  
Mehmet Soysal
Keyword(s):  
High Performance ◽  
Ad Hoc ◽  
File System ◽  
File Systems ◽  
System Modelling ◽  
Earth System ◽  
Post Processing ◽  
In Situ Data ◽  
The Impact

<p>Earth System Models (ESM) got much more demanding over the last years. Modelled processes got more complex and more and more processes are considered in models. In addition resolutions of the models got higher to improve weather and climate forecasts. This requires faster high performance computers (HPC) and better I/O performance.</p><p>Within our Pilot Lab Exascale Earth System Modelling (PL-EESM) we do performance analysis of the ESM EMAC using a standard Lustre file system for output and compare it to the performance using a parallel ad-hoc overlay file system. We will show the impact for two scenarios: one for todays standard amount of output and one with artificial heavy output simulating future ESMs.</p><p>An ad-hoc file system is a private parallel file system which is created on-demand for an HPC job using the node-local storage devices, in our case solid-state-disks (SSD). It only exists during the runtime of the job. Therefore output data have to be moved to a permanent file system before the job has finished. Quasi in-situ data analysis and post-processing allows to gain performance as it might result in a decreased amount of data which you have to store - saving disk space and time during the transfer of data to permanent storage. We will show first tests for quasi in-situ post-processing.</p>

Technical Considerations for Building a Landslide Tracker Mobile App

2021 ◽  
Author(s):  
Ramesh Guntha ◽  
Maneesha Vinodini Ramesh
Keyword(s):  
High Performance ◽  
Monsoon Season ◽  
Performance Testing ◽  
Mobile App ◽  
Mobile Databases ◽  
Synchronization Mechanism ◽  
Server Side ◽  
Repeated Calls ◽  
The World ◽  
Client Side

<p>Substantially complete landslide inventories aid the accurate landslide modelling of a region’s susceptibility and landslide forecasting. Recording of landslides soon after they have occurred is important as their presence can be quickly erased (e.g., the landslide removed by people or through erosion/vegetation). In this paper, we present the technical software considerations that went into building a Landslide Tracker app to aid in the collection of landslide information by non-technical local citizens, trained volunteers, and experts to create more complete inventories on a real-time basis through the model of crowdsourcing. The tracked landslide information is available for anyone across the world to view. This app is available on Google Play Store for free, and at http://landslides.amrita.edu, with software conceived and developed by Amrita University in the context of the UK NERC/FCDO funded LANDSLIP research project (http://www.landslip.org/).</p><p>The three technical themes we discuss in this paper are the following: (i) security, (ii) performance, and (iii) network resilience. (i) Security considerations include authentication, authorization, and client/server-side enforcement. Authentication allows only the registered users to record and view the landslides, whereas authorization protects the data from illegal access. For example, landslides created by one user are not editable by others, and no user should be able to delete landslides. This validation is enforced at the client-side (mobile and web apps) and also at the server-side software to prevent unintentional and intentional illegal access. (ii) Performance considerations include designing high-performance data structures, mobile databases, client-side caching, server-side caching, cache synchronization, and push-notifications. The database is designed to ensure the best performance without sacrificing data integrity. Then the read-heavy data is cached in memory to get this data with very low latency. Similarly, the data, once fetched, is cached in memory on the app so that it can be re-used without making repeated calls to the server every time when the user visits a screen.  The data persists in the mobile database so the app can load faster while reopening. A cache-synchronization mechanism is implemented to prevent the caches' data from becoming stale as new data comes into the database. The synchronization mechanism consists of push-notifications and incremental data pulls. (iii) Network resiliency considerations are achieved with the help of local storage on the app. This allows recording the landslides even when there is no internet connection. The app automatically pushes the updates to the server as soon as the connectivity resumes. We have observed over 300% reduction in time taken to load 2000 landslides, between the no-cache mode to cache mode during the performance testing. </p><p>The Landslide tracker app was released during the 2020 monsoon season and more than 250 landslides were recorded through the app across India and the world.</p>

Spin-Based Fully Nonvolatile Full-Adder Circuit for Computing in Memory

SPIN ◽  
2019 ◽  
Vol 09 (01) ◽  
pp. 1950007 ◽  
Author(s):  
Abdolah Amirany ◽  
Ramin Rajaei
Keyword(s):  
High Performance ◽  
Soft Errors ◽  
Magnetic Tunnel Junction ◽  
High Volume ◽  
Full Adder ◽  
Cmos Technology ◽  
Logic Circuits ◽  
Volume Data ◽  
Single Node ◽  
Radiation Induced

As CMOS technology scales down toward below 2-digit nanometer dimensions, exponentially increasing leakage power, vulnerability to radiation induced soft errors have become a major problem in today’s logic circuits. Emerging spin-based logic circuits and architectures based on nonvolatile magnetic tunnel junction (MTJ) cells show a great potential to overcome the aforementioned issues. However, radiation induced soft errors are still a problem in MTJ-based circuits as they need sequential peripheral CMOS circuits for sensing the MTJs. This paper proposes a novel nonvolatile and low-cost radiation hardened magnetic full adder (MFA). In comparison with the previous designs, the proposed MFA is capable of tolerating particle strikes regardless of the amount of charge induced to a single node and even multiple nodes. Besides, the proposed MFA offers low power operation, low area and high performance as compared with previous counterparts. One of the most important features suggested by the proposed MFA circuit is full nonvolatility. Nonvolatile logic circuits remove the cost of high volume data transactions between memory and logic and also facilitate power gating in logic-in-memory architectures.

scholarly journals ESM-Tools version 5.0: a modular infrastructure for stand-alone and coupled Earth system modelling (ESM)

2021 ◽  
Vol 14 (6) ◽  
pp. 4051-4067
Author(s):  
Dirk Barbi ◽  
Nadine Wieters ◽  
Paul Gierz ◽  
Miguel Andrés-Martínez ◽  
Deniz Ural ◽  
...  
Keyword(s):  
Land Surface ◽  
High Performance ◽  
Land Surface Model ◽  
Management Tool ◽  
Surface Model ◽  
System Modelling ◽  
Climate Modelling ◽  
Earth System ◽  
Wide Range ◽  
Earth System Modelling

Abstract. Earth system and climate modelling involves the simulation of processes on a wide range of scales and within and across various compartments of the Earth system. In practice, component models are often developed independently by different research groups, adapted by others to their special interests and then combined using a dedicated coupling software. This procedure not only leads to a strongly growing number of available versions of model components and coupled setups but also to model- and high-performance computing (HPC)-system-dependent ways of obtaining, configuring, building and operating them. Therefore, implementing these Earth system models (ESMs) can be challenging and extremely time consuming, especially for less experienced modellers or scientists aiming to use different ESMs as in the case of intercomparison projects. To assist researchers and modellers by reducing avoidable complexity, we developed the ESM-Tools software, which provides a standard way for downloading, configuring, compiling, running and monitoring different models on a variety of HPC systems. It should be noted that ESM-Tools is not a coupling software itself but a workflow and infrastructure management tool to provide access to increase usability of already existing components and coupled setups. As coupled ESMs are technically the more challenging tasks, we will focus on coupled setups, always implying that stand-alone models can benefit in the same way. With ESM-Tools, the user is only required to provide a short script consisting of only the experiment-specific definitions, while the software executes all the phases of a simulation in the correct order. The software, which is well documented and easy to install and use, currently supports four ocean models, three atmosphere models, two biogeochemistry models, an ice sheet model, an isostatic adjustment model, a hydrology model and a land-surface model. Compared to previous versions, ESM-Tools has lately been entirely recoded in a high-level programming language (Python) and provides researchers with an even more user-friendly interface for Earth system modelling. ESM-Tools was developed within the framework of the Advanced Earth System Model Capacity project, supported by the Helmholtz Association.

scholarly journals A JavaScript API for the Ice Sheet System Model (ISSM) 4.11: towards an online interactive model for the cryosphere community

2017 ◽  
Vol 10 (12) ◽  
pp. 4393-4403 ◽  
Author(s):  
Eric Larour ◽  
Daniel Cheng ◽  
Gilberto Perez ◽  
Justin Quinn ◽  
Mathieu Morlighem ◽  
...  
Keyword(s):  
High Performance ◽  
Ice Sheet ◽  
System Model ◽  
Earth System ◽  
Efficient Manner ◽  
Post Process ◽  
Software Model ◽  
Science Community ◽  
Deep Integration ◽  
Client Side

Abstract. Earth system models (ESMs) are becoming increasingly complex, requiring extensive knowledge and experience to deploy and use in an efficient manner. They run on high-performance architectures that are significantly different from the everyday environments that scientists use to pre- and post-process results (i.e., MATLAB, Python). This results in models that are hard to use for non-specialists and are increasingly specific in their application. It also makes them relatively inaccessible to the wider science community, not to mention to the general public. Here, we present a new software/model paradigm that attempts to bridge the gap between the science community and the complexity of ESMs by developing a new JavaScript application program interface (API) for the Ice Sheet System Model (ISSM). The aforementioned API allows cryosphere scientists to run ISSM on the client side of a web page within the JavaScript environment. When combined with a web server running ISSM (using a Python API), it enables the serving of ISSM computations in an easy and straightforward way. The deep integration and similarities between all the APIs in ISSM (MATLAB, Python, and now JavaScript) significantly shortens and simplifies the turnaround of state-of-the-art science runs and their use by the larger community. We demonstrate our approach via a new Virtual Earth System Laboratory (VESL) website (http://vesl.jpl.nasa.gov, VESL(2017)).

scholarly journals Experiential Learning Of Networking Technologies: Evolution Of Socket Programming – Part I

2019 ◽  
Author(s):  
Ram P Rustagi ◽  
Viraj Kumar
Keyword(s):  
High Performance ◽  
Web Server ◽  
Transport Layer ◽  
The Internet ◽  
Communication Performance ◽  
Best Effort ◽  
Connection Management ◽  
Server Side ◽  
Client Side ◽  
Tcp Connections

In the 21st century, the internet has become essential part of everyday tasks including banking, interacting with government services, education, entertainment, text/voice/video communication, etc. Individuals access the internet using client-side applications such as a browser or an app on their mobile phone or laptop/desktop. This client-side application communicates with a server-side application, typically running on a web server, which in turn may interact with other business applications. The underlying protocol is typically HTTP [1] running on top of the TCP/IP protocol [2][3]. A typical web server supports a large number (hundreds or thousands) of concurrent TCP connections. The most commonly deployed web servers in use today are Apache server [4], Nginx [5], or Microsoft Internet Information Server (IIS)[6]. Nginx is mostly used on Linux and IIS runs only on Windows OS. In contrast, Apache web server (which is almost as old as the web itself) is supported on all platforms (Linux, Windows, MacOS, etc.). In its initial release in 1995 (version 1.3), Apache server could serve only a few concurrent clients, but its current release (2.4.41) can support a huge number of concurrent clients. In this article (as well as Part II that will follow), we will present a simplified view of this evolution that nevertheless explains how current web manage such high levels of concurrency. To do so, we will delve into socket programming, which is at the heart of managing TCP connections, and we will examine the key role that it plays in delivering high performance. We have studied both transport layers protocol i.e., TCP [2] and UDP [7], in detail in the last few articles, and we have developed a basic understanding of the working of the transport layer. This is a communication-enabling layer used by applications to exchange application-level data. Simple working of applications using TCP (providing reliable delivery) and UDP (providing best effort delivery) socket programming are provided in [8]. In this article, however, we will discuss increasingly complex levels of socket programming, from simple socket connections to complex connection management that are necessary to attain high TCP performance. We will focus on TCP Socket programming only. UDP socket programming is simply a best effort delivery and socket implementation support does not impact the application communication performance.

Sign in / Sign up

Export Citation Format

Share Document