scholarly journals In situ visualization of large-scale turbulence simulations in Nek5000 with ParaView Catalyst

2021 ◽  
Author(s):  
Marco Atzori ◽  
Wiebke Köpp ◽  
Steven W. D. Chien ◽  
Daniele Massaro ◽  
Fermín Mallor ◽  
...  
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Data Sets ◽  
High Fidelity Simulation ◽  
Computing Systems ◽  
Performance Simulation ◽  
Image Composition ◽  
In Situ Visualization ◽  
Scale Turbulence

AbstractIn situ visualization on high-performance computing systems allows us to analyze simulation results that would otherwise be impossible, given the size of the simulation data sets and offline post-processing execution time. We develop an in situ adaptor for Paraview Catalyst and Nek5000, a massively parallel Fortran and C code for computational fluid dynamics. We perform a strong scalability test up to 2048 cores on KTH’s Beskow Cray XC40 supercomputer and assess in situ visualization’s impact on the Nek5000 performance. In our study case, a high-fidelity simulation of turbulent flow, we observe that in situ operations significantly limit the strong scalability of the code, reducing the relative parallel efficiency to only $$\approx 21\%$$ ≈ 21 % on 2048 cores (the relative efficiency of Nek5000 without in situ operations is $$\approx 99\%$$ ≈ 99 % ). Through profiling with Arm MAP, we identified a bottleneck in the image composition step (that uses the Radix-kr algorithm) where a majority of the time is spent on MPI communication. We also identified an imbalance of in situ processing time between rank 0 and all other ranks. In our case, better scaling and load-balancing in the parallel image composition would considerably improve the performance of Nek5000 with in situ capabilities. In general, the result of this study highlights the technical challenges posed by the integration of high-performance simulation codes and data-analysis libraries and their practical use in complex cases, even when efficient algorithms already exist for a certain application scenario.

scholarly journals High Performance Computational Analysis of Large-scale Proteome Data Sets to Assess Incremental Contribution to Coverage of the Human Genome

2013 ◽  
Vol 12 (6) ◽  
pp. 2858-2868 ◽  
Author(s):  
Nadin Neuhauser ◽  
Nagarjuna Nagaraj ◽  
Peter McHardy ◽  
Sara Zanivan ◽  
Richard Scheltema ◽  
...  
Keyword(s):  
Human Genome ◽  
High Performance ◽  
Large Scale ◽  
Computational Analysis ◽  
Data Sets

scholarly journals AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

2020 ◽  
Vol 496 (1) ◽  
pp. 629-637
Author(s):  
Ce Yu ◽  
Kun Li ◽  
Shanjiang Tang ◽  
Chao Sun ◽  
Bin Ma ◽  
...  
Keyword(s):  
Time Series ◽  
High Performance ◽  
Large Scale ◽  
Extrasolar Planets ◽  
Time Series Data ◽  
Series Data ◽  
Data Sets ◽  
Observation Data ◽  
Data Volume ◽  
And Performance

ABSTRACT Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analysing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FITS) files or data bases, match each item to determine which object it belongs to, and finally produce time series data sets. To support the high-performance parallel processing of large-scale data sets, AstroCatR uses the extract-transform-load (ETL) pre-processing module to create sky zone files and balance the workload. The matching module uses the overlapped indexing method and an in-memory reference table to improve accuracy and performance. The output of AstroCatR can be stored in CSV files or be transformed other into formats as needed. Simultaneously, the module-based software architecture ensures the flexibility and scalability of AstroCatR. We evaluated AstroCatR with actual observation data from The three Antarctic Survey Telescopes (AST3). The experiments demonstrate that AstroCatR can efficiently and flexibly reconstruct all time series data by setting relevant parameters and configuration files. Furthermore, the tool is approximately 3× faster than methods using relational data base management systems at matching massive catalogues.

scholarly journals PFASST-ER: combining the parallel full approximation scheme in space and time with parallelization across the method

2020 ◽  
Vol 23 (1-4) ◽  
Author(s):  
Ruth Schöbel ◽  
Robert Speck
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Approximation Scheme ◽  
Reaction Diffusion ◽  
Multiple Time ◽  
Time Step ◽  
Computing Systems ◽  
Space And Time ◽  
Quasi Newton ◽  
Spectral Deferred Correction

AbstractTo extend prevailing scaling limits when solving time-dependent partial differential equations, the parallel full approximation scheme in space and time (PFASST) has been shown to be a promising parallel-in-time integrator. Similar to space–time multigrid, PFASST is able to compute multiple time-steps simultaneously and is therefore in particular suitable for large-scale applications on high performance computing systems. In this work we couple PFASST with a parallel spectral deferred correction (SDC) method, forming an unprecedented doubly time-parallel integrator. While PFASST provides global, large-scale “parallelization across the step”, the inner parallel SDC method allows integrating each individual time-step “parallel across the method” using a diagonalized local Quasi-Newton solver. This new method, which we call “PFASST with Enhanced concuRrency” (PFASST-ER), therefore exposes even more temporal concurrency. For two challenging nonlinear reaction-diffusion problems, we show that PFASST-ER works more efficiently than the classical variants of PFASST and can use more processors than time-steps.

Study on High-Performance Simulation Computer for Large-Scale System of Systems Simulation

2014 ◽  
Vol 548-549 ◽  
pp. 1311-1318
Author(s):  
Zhi Qiang Zhao ◽  
Jia Xin Hao
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Engineering Application ◽  
System Of Systems ◽  
Simulation Framework ◽  
Performance Simulation ◽  
Overall Performance ◽  
Systems Simulation ◽  
Key Techniques ◽  
Equipment System

The high-performance parallel computing (HPPC) has a better overall performance and higher productivity, for a generical large-scale army equipment system of systems (AESoS) simulation, and the runtime efficiency can be multiplied several tenfold to several hundredfold. The requirement analysis of simulation framework of AESoS based on HPPC was proposed. After the simulation framework of AESoS based on HPPC and its key techniques were discussed, the simulation framework of AESoS Based HPPC was designed. it is of great significance to offer certain references for the engineering application in the simulation fields of AESoS based on HPPC.

In Situ and In Transit Computing for Large Scale Geoscientific Simulation

2020 ◽  
Author(s):  
Sebastian Friedemann ◽  
Bruno Raffin ◽  
Basile Hector ◽  
Jean-Martial Cohard
Keyword(s):  
Data Assimilation ◽  
High Performance ◽  
Large Scale ◽  
Transfer Model ◽  
Simulation Data ◽  
Computing Platform ◽  
Continental Scale ◽  
Physically Based ◽  
In Transit

<p>In situ and in transit computing is an effective way to place postprocessing and preprocessing tasks for large scale simulations on the high performance computing platform. The resulting proximity between the execution of preprocessing, simulation and postprocessing permits to lower I/O by bypassing slow and energy inefficient persistent storages. This permits to scale workflows consisting of heterogeneous components such as simulation, data analysis and visualization, to modern massively parallel high performance platforms. Reordering the workflow components gives a manifold of new advanced data processing possibilities for research. Thus in situ and in transit computing are vital for advances in the domain of geoscientific simulation which relies on the increasing amount of sensor and simulation data available.</p><p>In this talk, different in situ and in transit workflows, especially those that are useful in the field of geoscientific simulation, are discussed. Furthermore our experiences augmenting ParFlow-CLM, a physically based, state-of-the-art, fully coupled water transfer model for the critical zone, with FlowVR, an in situ framework with a strict component paradigm, are presented.<br>This allows shadowed in situ file writing, in situ online steering and in situ visualization.</p><p>In situ frameworks further can be coupled to data assimilation tools.<br>In the on going EoCoE-II we propose to embed data assimilation codes into an in transit computing environment. This is expected to enable ensemble based data assimilation on continental scale hydrological simulations with multiple thousands of ensemble members.</p>

In Situ Visualization for Large-Scale Combustion Simulations

2010 ◽  
Vol 30 (3) ◽  
pp. 45-57 ◽  
Author(s):  
Hongfeng Yu ◽  
Chaoli Wang ◽  
Ray W Grout ◽  
Jacqueline H Chen ◽  
Kwan-Liu Ma
Keyword(s):  
Large Scale ◽  
In Situ Visualization

Permeability characterization of the Soultz and Ogachi large‐scale reservoir using induced microseismicity

Geophysics ◽  
2002 ◽  
Vol 67 (1) ◽  
pp. 204-211 ◽  
Author(s):  
Pascal Audigane ◽  
Jean‐Jacques Royer ◽  
Hideshi Kaieda
Keyword(s):  
Large Scale ◽  
Pressure Oscillation ◽  
Data Sets ◽  
Active Volume ◽  
Common Procedure ◽  
Order Of Magnitude ◽  
Different Depths ◽  
The Difference

Hydraulic fracturing is a common procedure to increase the permeability of a reservoir. It consists in injecting high‐pressure fluid into pilot boreholes. These hydraulic tests induce locally seismic emission (microseismicity) from which large‐scale permeability estimates can be derived assuming a diffusion‐like process of the pore pressure into the surrounding stimulated rocks. Such a procedure is applied on six data sets collected in the vicinity of two geothermal sites at Soultz (France) and Ogachi (Japan). The results show that the method is adequate to estimate large‐scale permeability tensors at different depths in the reservoir. Such an approach provides permeability of the medium before fracturing compatible with in situ measurements. Using a line source formulation of the diffusion equation rather than a classical point source approach, improvements are proposed for accounting in situation where the injection is performed on a well section. This technique applied to successive fluid‐injection tests indicates an increase in permeability by an order of magnitude. The underestimates observed in some cases are attributed to the difference of scale at which the permeability is estimated (some 1 km3 corresponding to the seismic active volume of rock compared to a few meters around the well for the pumping or pressure oscillation tests). One advantage of the proposed method is that it provides permeability tensor estimates at the reservoir scale.

scholarly journals MS-PyCloud: An open-source, cloud computing-based pipeline for LC-MS/MS data analysis

2018 ◽  
Author(s):  
Li Chen ◽  
Bai Zhang ◽  
Michael Schnaubelt ◽  
Punit Shah ◽  
Paul Aiyetan ◽  
...  
Keyword(s):  
Cloud Computing ◽  
Data Analysis ◽  
Open Source ◽  
High Performance ◽  
Large Scale ◽  
Rapid Development ◽  
Data File ◽  
Data Sets ◽  
Proteomics Data ◽  
Amazon Web Services

ABSTRACTRapid development and wide adoption of mass spectrometry-based proteomics technologies have empowered scientists to study proteins and their modifications in complex samples on a large scale. This progress has also created unprecedented challenges for individual labs to store, manage and analyze proteomics data, both in the cost for proprietary software and high-performance computing, and the long processing time that discourages on-the-fly changes of data processing settings required in explorative and discovery analysis. We developed an open-source, cloud computing-based pipeline, MS-PyCloud, with graphical user interface (GUI) support, for LC-MS/MS data analysis. The major components of this pipeline include data file integrity validation, MS/MS database search for spectral assignment, false discovery rate estimation, protein inference, determination of protein post-translation modifications, and quantitation of specific (modified) peptides and proteins. To ensure the transparency and reproducibility of data analysis, MS-PyCloud includes open source software tools with comprehensive testing and versioning for spectrum assignments. Leveraging public cloud computing infrastructure via Amazon Web Services (AWS), MS-PyCloud scales seamlessly based on analysis demand to achieve fast and efficient performance. Application of the pipeline to the analysis of large-scale iTRAQ/TMT LC-MS/MS data sets demonstrated the effectiveness and high performance of MS-PyCloud. The software can be downloaded at: https://bitbucket.org/mschnau/ms-pycloud/downloads/

Sign in / Sign up

Export Citation Format

Share Document