scholarly journals Parallel Coordinates Visualization in the ELK Stack

2020 ◽  
pp. paper10-1-paper10-12
Author(s):  
Timofei Galkin ◽  
Maria Grigorieva
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
Visual Exploration ◽  
Visual Data ◽  
Parallel Coordinates ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
Multiple Parameters ◽  
Interactive Analysis ◽  
Visualization Tools

Modern large-scale distributed computing systems, processing large volumes of data, require mature monitoring systems able to control and track in re-sources, networks, computing tasks, queues and other components. In recent years, the ELK stack has become very popular for the monitoring of computing environment, largely due to the efficiency and flexibility of the Elastic Search storage and wide variety of Kibana visualization tools. The analysis of computing infrastructure metadata often requires the visual exploration of multiple parameters simultaneously on one graphical image. Stacked bar charts, heat maps, radar charts are widely used for the multivariate visual data analysis, but these methods have limitations on the number of parameters. In this research the authors propose to enhance the capacity of Kibana, adding Parallel Coordinates diagram - one of the most powerful method for visual interactive analysis of high-dimensional data. It allows to compare many variables together and observe correlations between them. This work describes the development process of Parallel Coordinates as a Kibana plugin, and demonstrates an example of visual data analysis based on the Nginx logs metadata.

scholarly journals Cytosplore-Transcriptomics: a scalable inter-active framework for single-cell RNA sequencing data analysis

2020 ◽  
Author(s):  
Tamim Abdelaal ◽  
Jeroen Eggermont ◽  
Thomas Höllt ◽  
Ahmed Mahfouz ◽  
Marcel J.T. Reinders ◽  
...  
Keyword(s):  
Data Analysis ◽  
Single Cell ◽  
Rna Sequencing ◽  
Visual Exploration ◽  
Sequencing Data ◽  
Interactive Analysis ◽  
Levels Of Detail ◽  
Single Cell Rna Sequencing ◽  
Downstream Analysis ◽  
Low Dimensional

SummaryThe ever-increasing number of analyzed cells in Single-cell RNA sequencing (scRNA-seq) experiments imposes several challenges on the data analysis. Current analysis methods lack scalability to large datasets hampering interactive visual exploration of the data. We present Cytosplore-Transcriptomics, a framework to analyze scRNA-seq data, including data preprocessing, visualization and downstream analysis. At its core, it uses a hierarchical, manifold preserving representation of the data that allows the inspection and annotation of scRNA-seq data at different levels of detail. Consequently, Cytosplore-Transcriptomics provides interactive analysis of the data using low-dimensional visualizations that scales to millions of cells.AvailabilityCytosplore-Transcriptomics can be freely downloaded from [email protected]

scholarly journals OLIVER: A Tool for Visual Data Analysis on Longitudinal Plant Phenomics Data

2018 ◽  
Author(s):  
Oliver L Tessmer ◽  
David M Kramer ◽  
Jin Chen
Keyword(s):  
Data Analysis ◽  
Large Scale ◽  
Unmet Need ◽  
Biological Data ◽  
Statistical Hypothesis ◽  
Plant Phenotyping ◽  
Plant Responses ◽  
Visual Data ◽  
Statistical Hypothesis Testing ◽  
Plant Phenomics

AbstractThere is a critical unmet need for new tools to analyze and understand “big data” in the biological sciences where breakthroughs come from connecting massive genomics data with complex phenomics data. By integrating instant data visualization and statistical hypothesis testing, we have developed a new tool called OLIVER for phenomics visual data analysis with a unique function that any user adjustment will trigger real-time display updates for any affected elements in the workspace. By visualizing and analyzing omics data with OLIVER, biomedical researchers can quickly generate hypotheses and then test their thoughts within the same tool, leading to efficient knowledge discovery from complex, multi-dimensional biological data. The practice of OLIVER on multiple plant phenotyping experiments has shown that OLIVER can facilitate scientific discoveries. In the use case of OLIVER for large-scale plant phenotyping, a quick visualization identified emergent phenotypes that are highly transient and heterogeneous. The unique circular heat map with false-color plant images also indicates that such emergent phenotypes appear in different leaves under different conditions, suggesting that such previously unseen processes are critical for plant responses to dynamic environments.

Interactive 3-D visual analysis of ERA 5 data: improving diagnostic indices for Marine Cold Air Outbreaks

2021 ◽  
Author(s):  
Marcel Meyer ◽  
Iuliia Polkova ◽  
Marc Rautenhaus
Keyword(s):  
Data Analysis ◽  
Open Source ◽  
Visual Analysis ◽  
Extreme Weather ◽  
Visual Exploration ◽  
Extreme Weather Events ◽  
Visual Data ◽  
Cold Air ◽  
Diagnostic Indices ◽  
Weather Events

<p>We present the application of interactive 3-D visual analysis techniques using the open-source meteorological visualization framework Met.3D <strong>[1]</strong> for investigating ERA5 reanalysis data. Our focus lies on inspecting atmospheric conditions favoring the development of extreme weather events in the Arctic. Marine Cold Air Outbreaks (MCAOs) and Polar Lows (PLs) are analyzed with the aim of improving diagnostic indices for capturing extreme weather events in seasonal and climatological assessments. We adopt an integrated workflow starting with the interactive visual exploration of single MCAO and PL events, using an extended version of Met.3D, followed by the design and testing of new diagnostic indices in a climatological assessment. Our interactive visual exploration provides insights into the complex 3-D shape and dynamics of MCAOs and PLs. For instance, we reveal a slow wind eye of a PL that extends from the surface up into the stratosphere. Motivated by the interactive visual analysis of single cases of MCAOs, we design new diagnostic indices, which address shortcomings of previously used indices, by capturing the vertical extent of the lower-level static instability induced by MCAOs. The new indices are tested by comparison with observed PLs in the Barents and the Nordic Seas (as reported in the STARS data set). Results show that the new MCAO index introduced here has an important advantage compared with previously used MCAO indices: it is more successful in indicating the times and locations of PLs. We thus propose the new index for further analyses in seasonal climate predictions and climatological studies. The methods for interactive 3-D visual data analysis presented here are made freely available for public use as part of the open-source tool Met.3D. We thereby provide a generic tool that can be used for investigating atmospheric processes in ERA5 data by means of interactive 3-D visual data analysis. Met.3D can be used, for example, during an initial explorative phase of scientific workflows, as a complement to standard 2-D plots, and for detailed meteorological case-analyses in 3-D.</p><div><br><div> <p>[1] http://met3d.wavestoweather.de, https://collaboration.cen.uni-hamburg.de/display/Met3D/</p> </div> </div>

Exploiting P2P and Grid Computing Technologies for Resource Sharing to Support High Performance Distributed System

2010 ◽  
pp. 450-475
Author(s):  
Liangxiu Han
Keyword(s):  
Resource Sharing ◽  
High Performance ◽  
Large Scale ◽  
Fault Tolerant ◽  
Resource Discovery ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
Efficient Resource ◽  
Service Oriented ◽  
Important Design

This chapter identifies challenges and requirements for resource sharing to support high performance distributed Service-Oriented Computing (SOC) systems. The chapter draws attention to two popular and important design paradigms: Grid and Peer-to-Peer (P2P) computing systems, which are evolving as two practical solutions to supporting wide-area resource sharing over the Internet. As a fundamental task of resource sharing, the efficient resource discovery is playing an important role in the context of the SOC setting. The chapter presents the resource discovery in Grid and P2P environments through an overview of related systems, both historical and emerging. The chapter then discusses the exploitation of both technologies for facilitating the resource discovery within large-scale distributed computing systems in a flexible, scalable, fault-tolerant, interoperable and security fashion.

scholarly journals Designing Fault Tolerance Strategy by Iterative Redundancy for Component-Based Distributed Computing Systems

2014 ◽  
Vol 2014 ◽  
pp. 1-11
Author(s):  
Hui Wang ◽  
Yun Wang
Keyword(s):  
Fault Tolerance ◽  
Distributed Computing ◽  
Large Scale ◽  
Critical Issue ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
Tolerance Strategy ◽  
Use Of Resources ◽  
Distributed Components ◽  
Modular Redundancy

Reliability is a critical issue for component-based distributed computing systems, some distributed software allows the existence of large numbers of potentially faulty components on an open network. Faults are inevitable in this large-scale, complex, distributed components setting, which may include a lot of untrustworthy parts. How to provide highly reliable component-based distributed systems is a challenging problem and a critical research. Generally, redundancy and replication are utilized to realize the goal of fault tolerance. In this paper, we propose a CFI (critical fault iterative) redundancy technique, by which the efficiency can be guaranteed to make use of resources (e.g., computation and storage) and to create fault-tolerance applications. When operating in an environment with unknown components’ reliability, CFI redundancy is more efficient and adaptive than other techniques (e.g., K-Modular Redundancy and N-Version Programming). In the CFI strategy of redundancy, the function invocation relationships and invocation frequencies are employed to rank the functions’ importance and identify the most vulnerable function implemented via functionally equivalent components. A tradeoff has to be made between efficiency and reliability. In this paper, a formal theoretical analysis and an experimental analysis are presented. Compared with the existing methods, the reliability of components-based distributed system can be greatly improved by tolerating a small part of significant components.

REQUIREMENTS FOR AN EVENT-BASED SIMULATION PACKAGE FOR GRID SYSTEMS

2007 ◽  
Vol 08 (02) ◽  
pp. 163-178 ◽  
Author(s):  
FATOS XHAFA ◽  
JAVIER CARRETERO ◽  
LEONARD BAROLLI ◽  
ARJAN DURRESI
Keyword(s):  
Distributed Systems ◽  
Distributed Computing ◽  
Large Scale ◽  
Experimental Studies ◽  
Distributed Applications ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
Grid Systems ◽  
Recent Developments ◽  
Simulation Package

In this paper we present a study on the requirements for the design and implementation of simulation packages for Grid systems. Grids are emerging as new distributed computing systems whose main objective is to manage and allocate geographically distributed computing resources to applications and users in an efficient and transparent manner. Grid systems are at present very difficult and complex to use for experimental studies of large-scale distributed applications. Although the field of simulation of distributed computing systems is mature, recent developments in large-scale distributed systems are raising needs not present in the simulation of the traditional distributed systems. Motivated by this, we present in this work a set of basic requirements that any simulation package for Grid computing should offer. This set of functionalities is obtained after a careful review of most important existing Grid simulation packages and includes new requirements not considered in such simulation packages. Based on the identified set of requirements, a Grid simulator is developed and exemplified for the Grid scheduling problem.

Sign in / Sign up

Export Citation Format

Share Document