scholarly journals D-EE: Distributed software for visualizing intrinsic structure of large-scale single-cell data

GigaScience ◽  
2020 ◽  
Vol 9 (11) ◽  
Author(s):  
Shaokun An ◽  
Jizu Huang ◽  
Lin Wan
Keyword(s):  
Time Series ◽  
Dimensionality Reduction ◽  
Single Cell ◽  
High Performance ◽  
Large Scale ◽  
Distributed Storage ◽  
Distributed Computation ◽  
Low Dimensional ◽  
Cell Data ◽  
Performance Computing

Abstract Background Dimensionality reduction and visualization play vital roles in single-cell RNA sequencing (scRNA-seq) data analysis. While they have been extensively studied, state-of-the-art dimensionality reduction algorithms are often unable to preserve the global structures underlying data. Elastic embedding (EE), a nonlinear dimensionality reduction method, has shown promise in revealing low-dimensional intrinsic local and global data structure. However, the current implementation of the EE algorithm lacks scalability to large-scale scRNA-seq data. Results We present a distributed optimization implementation of the EE algorithm, termed distributed elastic embedding (D-EE). D-EE reveals the low-dimensional intrinsic structures of data with accuracy equal to that of elastic embedding, and it is scalable to large-scale scRNA-seq data. It leverages distributed storage and distributed computation, achieving memory efficiency and high-performance computing simultaneously. In addition, an extended version of D-EE, termed distributed optimization implementation of time-series elastic embedding (D-TSEE), enables the user to visualize large-scale time-series scRNA-seq data by incorporating experimentally temporal information. Results with large-scale scRNA-seq data indicate that D-TSEE can uncover oscillatory gene expression patterns by using experimentally temporal information. Conclusions D-EE is a distributed dimensionality reduction and visualization tool. Its distributed storage and distributed computation technique allow us to efficiently analyze large-scale single-cell data at the cost of constant time speedup. The source code for D-EE algorithm based on C and MPI tailored to a high-performance computing cluster is available at https://github.com/ShaokunAn/D-EE.

scholarly journals Providing large-scale disk storage at CERN

2019 ◽  
Vol 214 ◽  
pp. 04033
Author(s):  
Hervé Rousseau ◽  
Belinda Chan Kwok Cheong ◽  
Cristian Contescu ◽  
Xavier Espinal Curull ◽  
Jan Iven ◽  
...  
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Distributed Storage ◽  
Storage System ◽  
Primary Data ◽  
Disk Storage ◽  
The Road ◽  
Ongoing Work ◽  
Software Distribution ◽  
Microsoft Office

The CERN IT Storage group operates multiple distributed storage systems and is responsible for the support of the infrastructure to accommodate all CERN storage requirements, from the physics data generated by LHC and non-LHC experiments to the personnel users' files. EOS is now the key component of the CERN Storage strategy. It allows to operate at high incoming throughput for experiment data-taking while running concurrent complex production work-loads. This high-performance distributed storage provides now more than 250PB of raw disks and it is the key component behind the success of CERNBox, the CERN cloud synchronisation service which allows syncing and sharing files on all major mobile and desktop platforms to provide offline availability to any data stored in the EOS infrastructure. CERNBox recorded an exponential growth in the last couple of year in terms of files and data stored thanks to its increasing popularity inside CERN users community and thanks to its integration with a multitude of other CERN services (Batch, SWAN, Microsoft Office). In parallel CASTOR is being simplified and transitioning from an HSM into an archival system, focusing mainly in the long-term data recording of the primary data from the detectors, preparing the road to the next-generation tape archival system, CTA. The storage services at CERN cover as well the needs of the rest of our community: Ceph as data back-end for the CERN OpenStack infrastructure, NFS services and S3 functionality; AFS for legacy home directory filesystem services and its ongoing phase-out and CVMFS for software distribution. In this paper we will summarise our experience in supporting all our distributed storage system and the ongoing work in evolving our infrastructure, testing very-dense storage building block (nodes with more than 1PB of raw space) for the challenges waiting ahead.

scholarly journals Fast Playback Framework for Analysis of Ground-Based Doppler Radar Observations Using MapReduce Technology

2016 ◽  
Vol 33 (4) ◽  
pp. 621-634 ◽  
Author(s):  
Jingyin Tang ◽  
Corene J. Matyas
Keyword(s):  
Spatial Analysis ◽  
High Performance ◽  
Large Scale ◽  
Doppler Radar ◽  
Weather Events ◽  
Data Architecture ◽  
Research Grade ◽  
High Performance Computing Cluster ◽  
Time Systems ◽  
Performance Computing

AbstractThe creation of a 3D mosaic is often the first step when using the high-spatial- and temporal-resolution data produced by ground-based radars. Efficient yet accurate methods are needed to mosaic data from dozens of radar to better understand the precipitation processes in synoptic-scale systems such as tropical cyclones. Research-grade radar mosaic methods of analyzing historical weather events should utilize data from both sides of a moving temporal window and process them in a flexible data architecture that is not available in most stand-alone software tools or real-time systems. Thus, these historical analyses require a different strategy for optimizing flexibility and scalability by removing time constraints from the design. This paper presents a MapReduce-based playback framework using Apache Spark’s computational engine to interpolate large volumes of radar reflectivity and velocity data onto 3D grids. Designed as being friendly to use on a high-performance computing cluster, these methods may also be executed on a low-end configured machine. A protocol is designed to enable interoperability with GIS and spatial analysis functions in this framework. Open-source software is utilized to enhance radar usability in the nonspecialist community. Case studies during a tropical cyclone landfall shows this framework’s capability of efficiently creating a large-scale high-resolution 3D radar mosaic with the integration of GIS functions for spatial analysis.

scholarly journals High-Performance Computing Framework Based on Distributed Systems for Large-Scale Neurophysiological Data

2021 ◽  
Author(s):  
Mohsen Hadianpour ◽  
Ehsan Rezayat ◽  
Mohammad-Reza Dehaqani
Keyword(s):  
High Performance Computing ◽  
High Performance ◽  
Large Scale ◽  
Electrophysiological Recording ◽  
Neural Data ◽  
Data Framework ◽  
Neurophysiological Data ◽  
Computing Framework ◽  
Performance Computing ◽  
Neuroscience Community

Abstract Due to the significantly drastic progress and improvement in neurophysiological recording technologies, neuroscientists have faced various complexities dealing with unstructured large-scale neural data. In the neuroscience community, these complexities could create serious bottlenecks in storing, sharing, and processing neural datasets. In this article, we developed a distributed high-performance computing (HPC) framework called `Big neuronal data framework' (BNDF), to overcome these complexities. BNDF is based on open-source big data frameworks, Hadoop and Spark providing a flexible and scalable structure. We examined BNDF on three different large-scale electrophysiological recording datasets from nonhuman primate’s brains. Our results exhibited faster runtimes with scalability due to the distributed nature of BNDF. We compared BNDF results to a widely used platform like MATLAB in an equitable computational resource. Compared with other similar methods, using BNDF provides more than five times faster performance in spike sorting as a usual neuroscience application.

scholarly journals Taxonomic assignment for large-scale metagenomic data on high-perfomance systems

2017 ◽  
Vol 33 (2) ◽  
pp. 119-130
Author(s):  
Vinh Van Le ◽  
Hoai Van Tran ◽  
Hieu Ngoc Duong ◽  
Giang Xuan Bui ◽  
Lang Van Tran
Keyword(s):  
High Performance Computing ◽  
Assignment Problem ◽  
High Performance ◽  
Large Scale ◽  
Computing System ◽  
Metagenomic Data ◽  
Taxonomic Assignment ◽  
High Performance Computing System ◽  
Powerful Approach ◽  
Performance Computing

Metagenomics is a powerful approach to study environment samples which do not require the isolation and cultivation of individual organisms. One of the essential tasks in a metagenomic project is to identify the origin of reads, referred to as taxonomic assignment. Due to the fact that each metagenomic project has to analyze large-scale datasets, the metatenomic assignment is very much computation intensive. This study proposes a parallel algorithm for the taxonomic assignment problem, called SeMetaPL, which aims to deal with the computational challenge. The proposed algorithm is evaluated with both simulated and real datasets on a high performance computing system. Experimental results demonstrate that the algorithm is able to achieve good performance and utilize resources of the system efficiently. The software implementing the algorithm and all test datasets can be downloaded at http://it.hcmute.edu.vn/bioinfo/metapro/SeMetaPL.html.

scholarly journals EpiScanpy: integrated single-cell epigenomic analysis

2019 ◽  
Author(s):  
Anna Danese ◽  
Maria L. Richter ◽  
David S. Fischer ◽  
Fabian J. Theis ◽  
Maria Colomé-Tatché
Keyword(s):  
Dna Methylation ◽  
Single Cell ◽  
Large Scale ◽  
Feature Space ◽  
Rna Seq ◽  
Computational Framework ◽  
Learning Techniques ◽  
Multiple Feature ◽  
The Many ◽  
Cell Data

ABSTRACTEpigenetic single-cell measurements reveal a layer of regulatory information not accessible to single-cell transcriptomics, however single-cell-omics analysis tools mainly focus on gene expression data. To address this issue, we present epiScanpy, a computational framework for the analysis of single-cell DNA methylation and single-cell ATAC-seq data. EpiScanpy makes the many existing RNA-seq workflows from scanpy available to large-scale single-cell data from other -omics modalities. We introduce and compare multiple feature space constructions for epigenetic data and show the feasibility of common clustering, dimension reduction and trajectory learning techniques. We benchmark epiScanpy by interrogating different single-cell brain mouse atlases of DNA methylation, ATAC-seq and transcriptomics. We find that differentially methylated and differentially open markers between cell clusters enrich transcriptome-based cell type labels by orthogonal epigenetic information.

A computational prototyping environment interface for DoD CREATE™-AV Helios simulations

2021 ◽  
Author(s):  
Robert Haehnel ◽  
Scott Christensen ◽  
J. Whitlow ◽  
Andrew Bauer ◽  
Ari Meyer ◽  
...  
Keyword(s):  
Time Series ◽  
Modeling And Simulation ◽  
High Performance Computing ◽  
Department Of Defense ◽  
High Performance ◽  
Data Input ◽  
Web Based ◽  
Simulation Tools ◽  
Parametric Studies ◽  
Performance Computing

Computational Prototyping Environment (CPE) is a web-based portal designed to simplify running Department of Defense (DoD) modeling and simulation tools on the DoD Supercomputing Resource Center’s (DSRC) High Performance Computing (HPC) systems. The first of these tools to be deployed in the CPE is an application (app) to conduct parametric studies and view results using the CREATE-AV Helios CFD software. Initial capability includes hover (collective sweep) and forward flight (speed sweep) performance calculations. The CPE Helios app allows for job submission to a DSRC’s HPC system and for the viewing of results created by Helios, i.e., time series and volumetric data. Example data input and results viewing are presented. Planned future functionality is also outlined.

Cloud Computing for Scientific Simulation and High Performance Computing

2013 ◽  
pp. 51-70
Author(s):  
Adrian Jackson ◽  
Michèle Weiland
Keyword(s):  
Cloud Computing ◽  
High Performance Computing ◽  
High Performance ◽  
Large Scale ◽  
Parallel Programs ◽  
Small Scale ◽  
Cloud Infrastructure ◽  
Scientific Simulations ◽  
Cloud Infrastructures ◽  
Performance Computing

This chapter describes experiences using Cloud infrastructures for scientific computing, both for serial and parallel computing. Amazon’s High Performance Computing (HPC) Cloud computing resources were compared to traditional HPC resources to quantify performance as well as assessing the complexity and cost of using the Cloud. Furthermore, a shared Cloud infrastructure is compared to standard desktop resources for scientific simulations. Whilst this is only a small scale evaluation these Cloud offerings, it does allow some conclusions to be drawn, particularly that the Cloud can currently not match the parallel performance of dedicated HPC machines for large scale parallel programs but can match the serial performance of standard computing resources for serial and small scale parallel programs. Also, the shared Cloud infrastructure cannot match dedicated computing resources for low level benchmarks, although for an actual scientific code, performance is comparable.

Green Computing

2014 ◽  
pp. 248-260
Keyword(s):  
Climate Change ◽  
Cloud Computing ◽  
High Performance Computing ◽  
High Performance ◽  
Large Scale ◽  
Green Computing ◽  
Research Topic ◽  
The Other ◽  
Cloud Infrastructures ◽  
Performance Computing

Green computing is a contemporary research topic to address climate and energy challenges. In this chapter, the authors envision the duality of green computing with technological trends in other fields of computing such as High Performance Computing (HPC) and cloud computing on one hand and economy and business on the other hand. For instance, in order to provide electricity for large-scale cloud infrastructures and to reach exascale computing, we need huge amounts of energy. Thus, green computing is a challenge for the future of cloud computing and HPC. Alternatively, clouds and HPC provide solutions for green computing and climate change. In this chapter, the authors discuss this proposition by looking at the technology in detail.

Sign in / Sign up

Export Citation Format

Share Document