The Strong Scaling Advantage of FPGAs in HPC for N-body Simulations

N-body methods are one of the essential algorithmic building blocks of high-performance and parallel computing. Previous research has shown promising performance for implementing n-body simulations with pairwise force calculations on FPGAs. However, to avoid challenges with accumulation and memory access patterns, the presented designs calculate each pair of forces twice, along with both force sums of the involved particles. Also, they require large problem instances with hundreds of thousands of particles to reach their respective peak performance, limiting the applicability for strong scaling scenarios. This work addresses both issues by presenting a novel FPGA design that uses each calculated force twice and overlaps data transfers and computations in a way that allows to reach peak performance even for small problem instances, outperforming previous single precision results even in double precision, and scaling linearly over multiple interconnected FPGAs. For a comparison across architectures, we provide an equally optimized CPU reference, which for large problems actually achieves higher peak performance per device, however, given the strong scaling advantages of the FPGA design, in parallel setups with few thousand particles per device, the FPGA platform achieves highest performance and power efficiency.

Highly scalable numerical simulation of coupled reaction–Diffusion systems with moving interfaces

A combination of reaction–diffusion models with moving-boundary problems yields a system in which the diffusion (spreading and penetration) and reaction (transformation) evolve the system’s state and geometry over time. These systems can be used in a wide range of engineering applications. In this study, as an example of such a system, the degradation of metallic materials is investigated. A mathematical model is constructed of the diffusion-reaction processes and the movement of corrosion front of a magnesium block floating in a chemical solution. The corresponding parallelized computational model is implemented using the finite element method, and the weak and strong-scaling behaviors of the model are evaluated to analyze the performance and efficiency of the employed high-performance computing techniques.

Open and strong-scaling tools for atom-probe crystallography: high-throughput methods for indexing crystal structure and orientation

Volumetric crystal structure indexing and orientation mapping are key data processing steps for virtually any quantitative study of spatial correlations between the local chemical composition features and the microstructure of a material. For electron and X-ray diffraction methods it is possible to develop indexing tools which compare measured and analytically computed patterns to decode the structure and relative orientation within local regions of interest. Consequently, a number of numerically efficient and automated software tools exist to solve the above characterization tasks. For atom-probe tomography (APT) experiments, however, the strategy of making comparisons between measured and analytically computed patterns is less robust because many APT data sets contain substantial noise. Given that sufficiently general predictive models for such noise remain elusive, crystallography tools for APT face several limitations: their robustness to noise is limited, and therefore so too is their capability to identify and distinguish different crystal structures and orientations. In addition, the tools are sequential and demand substantial manual interaction. In combination, this makes robust uncertainty quantification with automated high-throughput studies of the latent crystallographic information a difficult task with APT data. To improve the situation, the existing methods are reviewed and how they link to the methods currently used by the electron and X-ray diffraction communities is discussed. As a result of this, some of the APT methods are modified to yield more robust descriptors of the atomic arrangement. Also reported is how this enables the development of an open-source software tool for strong scaling and automated identification of a crystal structure, and the mapping of crystal orientation in nanocrystalline APT data sets with multiple phases.

AI-driven multiscale simulations illuminate mechanisms of SARS-CoV-2 spike dynamics

We develop a generalizable AI-driven workflow that leverages heterogeneous HPC resources to explore the time-dependent dynamics of molecular systems. We use this workflow to investigate the mechanisms of infectivity of the SARS-CoV-2 spike protein, the main viral infection machinery. Our workflow enables more efficient investigation of spike dynamics in a variety of complex environments, including within a complete SARS-CoV-2 viral envelope simulation, which contains 305 million atoms and shows strong scaling on ORNL Summit using NAMD. We present several novel scientific discoveries, including the elucidation of the spike’s full glycan shield, the role of spike glycans in modulating the infectivity of the virus, and the characterization of the flexible interactions between the spike and the human ACE2 receptor. We also demonstrate how AI can accelerate conformational sampling across different systems and pave the way for the future application of such methods to additional studies in SARS-CoV-2 and other molecular systems.

Critical domain: Universality, ε-expansion

In Chapter 15, the scaling behaviour of correlation functions at criticality, T = Tc, has been derived. This chapter is devoted to the critical domain, where the correlation length is large with respect to the microscopic scale, but finite. In dimensions d < 4, above Tc, the property of strong scaling is derived: in the critical domain above Tc, all correlation functions, after rescaling, can be expressed in terms of universal correlation functions, in which the scale of distance is provided by the correlation length. However, because the correlation length is singular at Tc, in this formalism, the critical temperature cannot be crossed. Alternatively, one can expand correlation functions in formal power series of the deviation (T −Tc) from the critical temperature, in presence of a magnetic field. The sum of the expansion satisfies renormalization group (RG) equations also valid for T < Tc and in a magnetic field, from which follow scaling properties in the whole critical domain. The universal two-point function can be expanded when T approaches Tc, using the short-distance expansion (SDE). A few terms of the ϵ expansion (ϵ is the deviation from dimension 4) of a few universal quantities are reported. Calculations at fixed dimension and summation of perturbative expansions are described. The conformal bootstrap based on the SDE and conformal invariance at the infrared (IR) fixed point provides an alternative method to determine critical exponents.

Next-Generation Time Integration targeting Weather and Climate Simulations

&lt;p&gt;Running simulations on high-performance computers faces new challenges due to e.g. the stagnating or even decreasing per-core speed. This poses new restrictions and therefore challenges on solving PDEs within a particular time frame in the strong scaling case. Here, disruptive mathematical reformulations, which e.g. exploit additional degrees of parallelism also along the time dimension, gained increasing interest over the last two decades.&lt;/p&gt;&lt;p&gt;This talk will cover various examples of our current research on (parallel-in-)time integration methods in the context of weather and climate simulations such as rational approximation of exponential integrators, multi-level time integration of spectral deferred correction (PFASST) as well as other methods.&lt;/p&gt;&lt;p&gt;These methods are realized and studied with numerics similar to the ones used by the European Centre for Medium-Range Weather Forecasts (ECMWF). Our results motivate further investigation for operational weather/climate systems in order to cope with the hardware imposed restrictions of future super computer architectures.&lt;/p&gt;&lt;p&gt;I gratefully acknowledge contributions and more from Jed Brown, Francois Hamon, Terry S. Haut, Richard Loft, Michael L. Minion, Pedro S. Peixoto, Nathana&amp;#235;l Schaeffer, Raphael Schilling&lt;/p&gt;

On strong-scaling and open-source tools for analyzing atom probe tomography data

The development of strong-scaling computational tools for high-throughput methods with an open-source code and transparent metadata standards has successfully transformed many computational materials science communities. While such tools are mature already in the condensed-matter physics community, the situation is still very different for many experimentalists. Atom probe tomography (APT) is one example. This microscopy and microanalysis technique has matured into a versatile nano-analytical characterization tool with applications that range from materials science to geology and possibly beyond. Here, data science tools are required for extracting chemo-structural spatial correlations from the reconstructed point cloud. For APT and other high-end analysis techniques, post-processing is mostly executed with proprietary software tools, which are opaque in their execution and have often limited performance. Software development by members of the scientific community has improved the situation but compared to the sophistication in the field of computational materials science several gaps remain. This is particularly the case for open-source tools that support scientific computing hardware, tools which enable high-throughput workflows, and open well-documented metadata standards to align experimental research better with the fair data stewardship principles. To this end, we introduce paraprobe, an open-source tool for scientific computing and high-throughput studying of point cloud data, here exemplified with APT. We show how to quantify uncertainties while applying several computational geometry, spatial statistics, and clustering tasks for post-processing APT datasets as large as two billion ions. These tools work well in concert with Python and HDF5 to enable several orders of magnitude performance gain, automation, and reproducibility.

AI-Driven Multiscale Simulations Illuminate Mechanisms of SARS-CoV-2 Spike Dynamics

ABSTRACTWe develop a generalizable AI-driven workflow that leverages heterogeneous HPC resources to explore the time-dependent dynamics of molecular systems. We use this workflow to investigate the mechanisms of infectivity of the SARS-CoV-2 spike protein, the main viral infection machinery. Our workflow enables more efficient investigation of spike dynamics in a variety of complex environments, including within a complete SARS-CoV-2 viral envelope simulation, which contains 305 million atoms and shows strong scaling on ORNL Summit using NAMD. We present several novel scientific discoveries, including the elucidation of the spike’s full glycan shield, the role of spike glycans in modulating the infectivity of the virus, and the characterization of the flexible interactions between the spike and the human ACE2 receptor. We also demonstrate how AI can accelerate conformational sampling across different systems and pave the way for the future application of such methods to additional studies in SARS-CoV-2 and other molecular systems.ACM Reference FormatLorenzo Casalino1†, Abigail Dommer1†, Zied Gaieb1†, Emilia P. Barros1, Terra Sztain1, Surl-Hee Ahn1, Anda Trifan2,3, Alexander Brace2, Anthony Bogetti4, Heng Ma2, Hyungro Lee5, Matteo Turilli5, Syma Khalid6, Lillian Chong4, Carlos Simmerling7, David J. Hardy3, Julio D. C. Maia3, James C. Phillips3, Thorsten Kurth8, Abraham Stern8, Lei Huang9, John McCalpin9, Mahidhar Tatineni10, Tom Gibbs8, John E. Stone3, Shantenu Jha5, Arvind Ramanathan2∗, Rommie E. Amaro1∗. 2020. AI-Driven Multiscale Simulations Illuminate Mechanisms of SARS-CoV-2 Spike Dynamics. In Supercomputing ’20: International Conference for High Performance Computing, Networking, Storage, and Analysis. ACM, New York, NY, USA, 14 pages. https://doi.org/finalDOI

Parallelizing a serial code: open–source module, EZ Parallel 1.0, and geophysics examples

Parallel computing can offer substantial speedup of numerical simulations in comparison to serial computing, as parallel computing uses many processors simultaneously rather than a single processor. However, it typically also requires substantial time and effort to convert a serial code into a parallel code. Here, a new module is developed to reduce the time and effort required to parallelize a serial code. The tested version of the module is written in the Fortran programming language,while the framework could also be extended to other languages (C++, Python, Julia, etc.). The Message Passing Interface is used to allow for either shared-memory or distributed-memory computer architectures. The software is designed for solving partial differential equations on a rectangular two-dimensional or three-dimensional domain, using finite difference, finite volume, pseudo-spectral, or other similar numerical methods. Examples are provided for two idealized models of atmospheric and oceanic fluid dynamics: the two-level quasi-geostrophic equations, and the stochastic heat equation as a model for turbulent advection–diffusion of either water vapor and clouds or sea surface height variability. In tests of the parallelized code, the strong scaling efficiency for the finite difference code is seen to be roughly 80 % to 90 %, which is achieved by adding roughly only 10 new lines to the serial code. Therefore, EZ Parallel provides great benefits with minimal additional effort.