scholarly journals MG-RAST version 4—lessons learned from a decade of low-budget ultra-high-throughput metagenome analysis

2017 ◽  
Vol 20 (4) ◽  
pp. 1151-1159 ◽  
Author(s):  
Folker Meyer ◽  
Saurabh Bagchi ◽  
Somali Chaterji ◽  
Wolfgang Gerlach ◽  
Ananth Grama ◽  
...  
Keyword(s):  
Performance Optimization ◽  
High Performance ◽  
Lessons Learned ◽  
Added Value ◽  
Data Sets ◽  
Real World Data ◽  
Metagenome Analysis ◽  
Trade Offs ◽  
And Performance ◽  
The Right

Abstract As technologies change, MG-RAST is adapting. Newly available software is being included to improve accuracy and performance. As a computational service constantly running large volume scientific workflows, MG-RAST is the right location to perform benchmarking and implement algorithmic or platform improvements, in many cases involving trade-offs between specificity, sensitivity and run-time cost. The work in [Glass EM, Dribinsky Y, Yilmaz P, et al. ISME J 2014;8:1–3] is an example; we use existing well-studied data sets as gold standards representing different environments and different technologies to evaluate any changes to the pipeline. Currently, we use well-understood data sets in MG-RAST as platform for benchmarking. The use of artificial data sets for pipeline performance optimization has not added value, as these data sets are not presenting the same challenges as real-world data sets. In addition, the MG-RAST team welcomes suggestions for improvements of the workflow. We are currently working on versions 4.02 and 4.1, both of which contain significant input from the community and our partners that will enable double barcoding, stronger inferences supported by longer-read technologies, and will increase throughput while maintaining sensitivity by using Diamond and SortMeRNA. On the technical platform side, the MG-RAST team intends to support the Common Workflow Language as a standard to specify bioinformatics workflows, both to facilitate development and efficient high-performance implementation of the community’s data analysis tasks.

Innovative Devops for Artificial Intelligence

2019 ◽  
Vol 19 (1) ◽  
pp. 58-63 ◽  
Author(s):  
R. Ciucu ◽  
F.C. Adochiei ◽  
Ioana-Raluca Adochiei ◽  
F. Argatu ◽  
G.C. Seriţan ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Performance Optimization ◽  
High Performance ◽  
Computational Models ◽  
Data Sets ◽  
Production Environments ◽  
Model Training ◽  
And Performance ◽  
Set Up ◽  
Computational Resources

AbstractDeveloping Artificial Intelligence is a labor intensive task. It implies both storage and computational resources. In this paper, we present a state-of-the-art service based infrastructure for deploying, managing and serving computational models alongside their respective data-sets and virtual environments. Our architecture uses key-based values to store specific graphs and datasets into memory for fast deployment and model training, furthermore leveraging the need for manual data reduction in the drafting and retraining stages. To develop the platform, we used clustering and orchestration to set up services and containers that allow deployment within seconds. In this article, we cover high performance computing concepts such as swarming, GPU resource management for model implementation in production environments with emphasis on standardized development to reduce integration tasks and performance optimization.

Integration, Development and Performance of the 500 TFLOPS Heterogeneous Cluster (Condor)

2012 ◽  
Author(s):  
Mark Barnell ◽  
Qing Wu ◽  
Richard Linderman
Keyword(s):  
Performance Optimization ◽  
High Performance ◽  
Large Scale ◽  
Situational Awareness ◽  
General Purpose ◽  
Lessons Learned ◽  
Heterogeneous Cluster ◽  
Graphic Processing Units ◽  
Intelligence Models ◽  
And Performance

The Air Force Research Laboratory Information Directorate Advanced Computing Division (AFRL/RIT) High Performance Computing Affiliated Resource Center (HPC-ARC) is the host to a very large scale interactive computing cluster consisting of about 1800 nodes. Condor, the largest interactive Cell cluster in the world, consists of integrated heterogeneous processors of IBM Cell Broadband Engine (Cell BE) multicore CPUs, NVIDIA General Purpose Graphic Processing Units (GPGPUs) and Intel x86 server nodes in a 10Gb Ethernet Star Hub network and 20Gb/s Infiniband Mesh, with a combined capability of 500 trillion floating operations per second (TFLOPS). Applications developed and running on CONDOR include large-scale computational intelligence models, video synthetic aperture radar (SAR) back-projection, Space Situational Awareness (SSA), video target tracking, linear algebra and others. This presentation will discuss the design and integration of the system. It will also show progress on performance optimization efforts and lessons learned on algorithm scalability on a heterogeneous architecture.

High-Performance Image Filters via Sparse Approximations

2020 ◽  
Vol 3 (2) ◽  
pp. 1-19
Author(s):  
Kersten Schuster ◽  
Philip Trettner ◽  
Leif Kobbelt
Keyword(s):  
High Performance ◽  
Hardware Acceleration ◽  
Optimization Method ◽  
Translation Invariant ◽  
Approximation Quality ◽  
Trade Offs ◽  
Sparse Approximations ◽  
Image Filters ◽  
Good Trade ◽  
And Performance

We present a numerical optimization method to find highly efficient (sparse) approximations for convolutional image filters. Using a modified parallel tempering approach, we solve a constrained optimization that maximizes approximation quality while strictly staying within a user-prescribed performance budget. The results are multi-pass filters where each pass computes a weighted sum of bilinearly interpolated sparse image samples, exploiting hardware acceleration on the GPU. We systematically decompose the target filter into a series of sparse convolutions, trying to find good trade-offs between approximation quality and performance. Since our sparse filters are linear and translation-invariant, they do not exhibit the aliasing and temporal coherence issues that often appear in filters working on image pyramids. We show several applications, ranging from simple Gaussian or box blurs to the emulation of sophisticated Bokeh effects with user-provided masks. Our filters achieve high performance as well as high quality, often providing significant speed-up at acceptable quality even for separable filters. The optimized filters can be baked into shaders and used as a drop-in replacement for filtering tasks in image processing or rendering pipelines.

scholarly journals Task-based programming in COMPSs to converge from HPC to big data

2017 ◽  
Vol 32 (1) ◽  
pp. 45-60 ◽  
Author(s):  
Javier Conejero ◽  
Sandra Corella ◽  
Rosa M Badia ◽  
Jesus Labarta
Keyword(s):  
Big Data ◽  
High Performance ◽  
Programming Model ◽  
Good Alternative ◽  
Programming Models ◽  
Suitable Model ◽  
Advantages And Disadvantages ◽  
Big Data Applications ◽  
And Performance ◽  
The Right

Task-based programming has proven to be a suitable model for high-performance computing (HPC) applications. Different implementations have been good demonstrators of this fact and have promoted the acceptance of task-based programming in the OpenMP standard. Furthermore, in recent years, Apache Spark has gained wide popularity in business and research environments as a programming model for addressing emerging big data problems. COMP Superscalar (COMPSs) is a task-based environment that tackles distributed computing (including Clouds) and is a good alternative for a task-based programming model for big data applications. This article describes why we consider that task-based programming models are a good approach for big data applications. The article includes a comparison of Spark and COMPSs in terms of architecture, programming model, and performance. It focuses on the differences that both frameworks have in structural terms, on their programmability interface, and in terms of their efficiency by means of three widely known benchmarking kernels: Wordcount, Kmeans, and Terasort. These kernels enable the evaluation of the more important functionalities of both programming models and analyze different work flows and conditions. The main results achieved from this comparison are (1) COMPSs is able to extract the inherent parallelism from the user code with minimal coding effort as opposed to Spark, which requires the existing algorithms to be adapted and rewritten by explicitly using their predefined functions, (2) it is an improvement in terms of performance when compared with Spark, and (3) COMPSs has shown to scale better than Spark in most cases. Finally, we discuss the advantages and disadvantages of both frameworks, highlighting the differences that make them unique, thereby helping to choose the right framework for each particular objective.

scholarly journals Sustaining open innovation through a “Center of Excellence”

2019 ◽  
Vol 47 (3) ◽  
pp. 19-26 ◽  
Author(s):  
Elizabeth E. Richard ◽  
Jeffrey R. Davis ◽  
Jin H. Paik ◽  
Karim R. Lakhani
Keyword(s):  
Open Innovation ◽  
Business Strategy ◽  
Pilot Scale ◽  
Lessons Learned ◽  
Cultural Issues ◽  
Content Type ◽  
Center Of Excellence ◽  
And Performance ◽  
Time Required ◽  
The Right

Purpose This paper presents NASA’s experience using a Center of Excellence (CoE) to scale and sustain an open innovation program as an effective problem-solving tool and includes strategic management recommendations for other organizations based on lessons learned. Design/methodology/approach This paper defines four phases of implementing an open innovation program: Learn, Pilot, Scale and Sustain. It provides guidance on the time required for each phase and recommendations for how to utilize a CoE to succeed. Recommendations are based upon the experience of NASA’s Human Health and Performance Directorate, and experience at the Laboratory for Innovation Science at Harvard running hundreds of challenges with research and development organizations. Findings Lessons learned include the importance of grounding innovation initiatives in the business strategy, assessing the portfolio of work to select problems most amenable to solving via crowdsourcing methodology, framing problems that external parties can solve, thinking strategically about early wins, selecting the right platforms, developing criteria for evaluation, and advancing a culture of innovation. Establishing a CoE provides an effective infrastructure to address both technical and cultural issues. Originality/value The NASA experience spanned more than seven years from initial learnings about open innovation concepts to the successful scaling and sustaining of an open innovation program; this paper provides recommendations on how to decrease this timeline to three years.

scholarly journals Malicious web domain identification using online credibility and performance data by considering the class imbalance issue

2019 ◽  
Vol 119 (3) ◽  
pp. 676-696 ◽  
Author(s):  
Zhongyi Hu ◽  
Raymond Chiong ◽  
Ilung Pranata ◽  
Yukun Bao ◽  
Yuqing Lin
Keyword(s):  
Machine Learning ◽  
Class Imbalance ◽  
Performance Data ◽  
Machine Learning Techniques ◽  
Data Sets ◽  
Real World Data ◽  
Content Type ◽  
Domain Identification ◽  
Learning Techniques ◽  
And Performance

Purpose Malicious web domain identification is of significant importance to the security protection of internet users. With online credibility and performance data, the purpose of this paper to investigate the use of machine learning techniques for malicious web domain identification by considering the class imbalance issue (i.e. there are more benign web domains than malicious ones). Design/methodology/approach The authors propose an integrated resampling approach to handle class imbalance by combining the synthetic minority oversampling technique (SMOTE) and particle swarm optimisation (PSO), a population-based meta-heuristic algorithm. The authors use the SMOTE for oversampling and PSO for undersampling. Findings By applying eight well-known machine learning classifiers, the proposed integrated resampling approach is comprehensively examined using several imbalanced web domain data sets with different imbalance ratios. Compared to five other well-known resampling approaches, experimental results confirm that the proposed approach is highly effective. Practical implications This study not only inspires the practical use of online credibility and performance data for identifying malicious web domains but also provides an effective resampling approach for handling the class imbalance issue in the area of malicious web domain identification. Originality/value Online credibility and performance data are applied to build malicious web domain identification models using machine learning techniques. An integrated resampling approach is proposed to address the class imbalance issue. The performance of the proposed approach is confirmed based on real-world data sets with different imbalance ratios.

scholarly journals AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

2020 ◽  
Vol 496 (1) ◽  
pp. 629-637
Author(s):  
Ce Yu ◽  
Kun Li ◽  
Shanjiang Tang ◽  
Chao Sun ◽  
Bin Ma ◽  
...  
Keyword(s):  
Time Series ◽  
High Performance ◽  
Large Scale ◽  
Extrasolar Planets ◽  
Time Series Data ◽  
Series Data ◽  
Data Sets ◽  
Observation Data ◽  
Data Volume ◽  
And Performance

ABSTRACT Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analysing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FITS) files or data bases, match each item to determine which object it belongs to, and finally produce time series data sets. To support the high-performance parallel processing of large-scale data sets, AstroCatR uses the extract-transform-load (ETL) pre-processing module to create sky zone files and balance the workload. The matching module uses the overlapped indexing method and an in-memory reference table to improve accuracy and performance. The output of AstroCatR can be stored in CSV files or be transformed other into formats as needed. Simultaneously, the module-based software architecture ensures the flexibility and scalability of AstroCatR. We evaluated AstroCatR with actual observation data from The three Antarctic Survey Telescopes (AST3). The experiments demonstrate that AstroCatR can efficiently and flexibly reconstruct all time series data by setting relevant parameters and configuration files. Furthermore, the tool is approximately 3× faster than methods using relational data base management systems at matching massive catalogues.

Sleep – A Game Changer in the Athletic World?

2018 ◽  
Vol 66 (4) ◽  
Keyword(s):  
Performance Optimization ◽  
High Performance ◽  
Recovery Process ◽  
Poor Sleep ◽  
Recovery Method ◽  
Lack Of Sleep ◽  
And Performance ◽  
The Individual ◽  
Traditional Approaches ◽  
And Control

The restorative qualities of sleep are fundamentally the basis of the individual athlete’s ability to recover and perform, and to optimally be able to challenge and control the effects of exercise regimes in high performance sport. Research consistently shows that a large percentage of the population fails to obtain the recommended 7–9 hours of sleep per night [17]. Moreover, recent years’ research has found that athletes have a high prevalence of poor sleep quality [6]. Given its implications on the recovery process, sleep affects the quality of the athlete’s training and outcome of competitions. Although an increasing number of recovery aids (such as cold baths, anti-inflammatory agents, high protein intake etc.) are available, recent years research show the important and irreplaceable role of sleep and that no recovery method can compensate for the lack of sleep. Every facet of an athlete’s life has the capacity to either create or take out energy, contribute to the overall stress level and subsequently the level of both recovery and performance. While traditional approaches to performance optimization focus simply on the physical stressors, this overview will highlight the benefits and the basic principles of sleep, its relation to recovery and performance, and provide input and reflect on what to consider when working with development and maintenance of athletic performance.

Material-structure-performance integrated laser-metal additive manufacturing

Science ◽  
2021 ◽  
Vol 372 (6545) ◽  
pp. eabg1487
Author(s):  
Dongdong Gu ◽  
Xinyu Shi ◽  
Reinhart Poprawe ◽  
David L. Bourell ◽  
Rossitza Setchi ◽  
...  
Keyword(s):  
Additive Manufacturing ◽  
High Performance ◽  
Material Structure ◽  
Metal Additive Manufacturing ◽  
Material Development ◽  
Design And Manufacturing ◽  
And Performance ◽  
The Right ◽  
And Function ◽  
Metal Additive

Laser-metal additive manufacturing capabilities have advanced from single-material printing to multimaterial/multifunctional design and manufacturing. Material-structure-performance integrated additive manufacturing (MSPI-AM) represents a path toward the integral manufacturing of end-use components with innovative structures and multimaterial layouts to meet the increasing demand from industries such as aviation, aerospace, automobile manufacturing, and energy production. We highlight two methodological ideas for MSPI-AM—“the right materials printed in the right positions” and “unique structures printed for unique functions”—to realize major improvements in performance and function. We establish how cross-scale mechanisms to coordinate nano/microscale material development, mesoscale process monitoring, and macroscale structure and performance control can be used proactively to achieve high performance with multifunctionality. MSPI-AM exemplifies the revolution of design and manufacturing strategies for AM and its technological enhancement and sustainable development.

Integrated Cloud Computing Environment for Upstream Geoscience Workflows

2021 ◽  
Author(s):  
Murtadha Al-Habib ◽  
Yasser Al-Ghamdi
Keyword(s):  
High Performance ◽  
Large Scale ◽  
End Users ◽  
Data Sets ◽  
Production Environment ◽  
Remote Visualization ◽  
Test Environment ◽  
Petroleum Resources ◽  
Customized Production ◽  
And Performance

Abstract Extensive computing resources are required to leverage todays advanced geoscience workflows that are used to explore and characterize giant petroleum resources. In these cases, high-performance workstations are often unable to adequately handle the scale of computing required. The workflows typically utilize complex and massive data sets, which require advanced computing resources to store, process, manage, and visualize various forms of the data throughout the various lifecycles. This work describes a large-scale geoscience end-to-end interpretation platform customized to run on a cluster-based remote visualization environment. A team of computing infrastructure and geoscience workflow experts was established to collaborate on the deployment, which was broken down into separate phases. Initially, an evaluation and analysis phase was conducted to analyze computing requirements and assess potential solutions. A testing environment was then designed, implemented and benchmarked. The third phase used the test environment to determine the scale of infrastructure required for the production environment. Finally, the full-scale customized production environment was deployed for end users. During testing phase, aspects such as connectivity, stability, interactivity, functionality, and performance were investigated using the largest available geoscience datasets. Multiple computing configurations were benchmarked until optimal performance was achieved, under applicable corporate information security guidelines. It was observed that the customized production environment was able to execute workflows that were unable to run on local user workstations. For example, while conducting connectivity, stability and interactivity benchmarking, the test environment was operated for extended periods to ensure stability for workflows that require multiple days to run. To estimate the scale of the required production environment, varying categories of users’ portfolio were determined based on data type, scale and workflow. Continuous monitoring of system resources and utilization enabled continuous improvements to the final solution. The utilization of a fit-for-purpose, customized remote visualization solution may reduce or ultimately eliminate the need to deploy high-end workstations to all end users. Rather, a shared, scalable and reliable cluster-based solution can serve a much larger user community in a highly performant manner.

Sign in / Sign up

Export Citation Format

Share Document