Accelerating In-Transit Co-Processing for Scientific Simulations Using Region-Based Data-Driven Analysis

Increasing processing capabilities and input/output constraints of supercomputers have increased the use of co-processing approaches, i.e., visualizing and analyzing data sets of simulations on the fly. We present a method that evaluates the importance of different regions of simulation data and a data-driven approach that uses the proposed method to accelerate in-transit co-processing of large-scale simulations. We use the importance metrics to simultaneously employ multiple compression methods on different data regions to accelerate the in-transit co-processing. Our approach strives to adaptively compress data on the fly and uses load balancing to counteract memory imbalances. We demonstrate the method’s efficiency through a fluid mechanics application, a Richtmyer–Meshkov instability simulation, showing how to accelerate the in-transit co-processing of simulations. The results show that the proposed method expeditiously can identify regions of interest, even when using multiple metrics. Our approach achieved a speedup of 1.29× in a lossless scenario. The data decompression time was sped up by 2× compared to using a single compression method uniformly.

Download Full-text

A Method for Identifying Environmental Stimuli and Genes Responsible for Genotype-by-Environment Interactions From a Large-Scale Multi-Environment Data Set

Frontiers in Genetics ◽

10.3389/fgene.2021.803636 ◽

2021 ◽

Vol 12 ◽

Author(s):

Akio Onogi ◽

Daisuke Sekine ◽

Akito Kaga ◽

Satoshi Nakano ◽

Tetsuya Yamada ◽

...

Keyword(s):

Large Scale ◽

Genetic Correlations ◽

Data Driven ◽

Data Sets ◽

Data Set ◽

Environmental Stimuli ◽

Genotype By Environment ◽

Genome Wide ◽

Sowing Dates ◽

Data Driven Approach

It has not been fully understood in real fields what environment stimuli cause the genotype-by-environment (G × E) interactions, when they occur, and what genes react to them. Large-scale multi-environment data sets are attractive data sources for these purposes because they potentially experienced various environmental conditions. Here we developed a data-driven approach termed Environmental Covariate Search Affecting Genetic Correlations (ECGC) to identify environmental stimuli and genes responsible for the G × E interactions from large-scale multi-environment data sets. ECGC was applied to a soybean (Glycine max) data set that consisted of 25,158 records collected at 52 environments. ECGC illustrated what meteorological factors shaped the G × E interactions in six traits including yield, flowering time, and protein content and when these factors were involved in the interactions. For example, it illustrated the relevance of precipitation around sowing dates and hours of sunshine just before maturity to the interactions observed for yield. Moreover, genome-wide association mapping on the sensitivities to the identified stimuli discovered candidate and known genes responsible for the G × E interactions. Our results demonstrate the capability of data-driven approaches to bring novel insights on the G × E interactions observed in fields.

Download Full-text

A method for identifying environmental stimuli and genes responsible for genotype-by-environment interactions from a large-scale multi-environment data set

10.1101/2021.10.25.465681 ◽

2021 ◽

Author(s):

Akio Onogi ◽

Daisuke Sekine ◽

Akito Kaga ◽

Satoshi Nakano ◽

Tetsuya Yamada ◽

...

Keyword(s):

Large Scale ◽

Genetic Correlations ◽

Data Driven ◽

Data Sets ◽

Data Set ◽

Environmental Stimuli ◽

Genotype By Environment ◽

Genome Wide ◽

Sowing Dates ◽

Data Driven Approach

It has not been fully understood in real fields what environment stimuli cause the genotype-by-environment (G x E) interactions, when they occur, and what genes react to them. Large-scale multi-environment data sets are attractive data sources for these purposes because they potentially experienced various environmental conditions. Here we developed a data-driven approach termed Environmental Covariate Search Affecting Genetic Correlations (ECGC) to identify environmental stimuli and genes responsible for the G x E interactions from large-scale multi-environment data sets. ECGC was applied to a soybean (Glycine max) data set that consisted of 25,158 records collected at 52 environments. ECGC illustrated what meteorological factors shaped the G x E interactions in six traits including yield, flowering time, and protein content and when they were involved. For example, it illustrated the relevance of precipitation around sowing dates and hours of sunshine just before maturity to the interactions observed for yield. Moreover, genome-wide association mapping on the sensitivities to the identified stimuli discovered candidate and known genes responsible for the G x E interactions. Our results demonstrate the capability of data-driven approaches to bring novel insights on the G x E interactions observed in fields.

Download Full-text

Improving the management of type 2 diabetes through large-scale general practice: the role of a data-driven and technology-enabled education programme

BMJ Open Quality ◽

10.1136/bmjoq-2020-001087 ◽

2021 ◽

Vol 10 (1) ◽

pp. e001087

Author(s):

Tarek F Radwan ◽

Yvette Agyako ◽

Alireza Ettefaghian ◽

Tahira Kamran ◽

Omar Din ◽

...

Keyword(s):

Type 2 Diabetes ◽

Primary Care ◽

Large Scale ◽

Education Programme ◽

Educational Programme ◽

Data Driven ◽

Treatment Targets ◽

Care Processes ◽

Data Driven Approach

A quality improvement (QI) scheme was launched in 2017, covering a large group of 25 general practices working with a deprived registered population. The aim was to improve the measurable quality of care in a population where type 2 diabetes (T2D) care had previously proved challenging. A complex set of QI interventions were co-designed by a team of primary care clinicians and educationalists and managers. These interventions included organisation-wide goal setting, using a data-driven approach, ensuring staff engagement, implementing an educational programme for pharmacists, facilitating web-based QI learning at-scale and using methods which ensured sustainability. This programme was used to optimise the management of T2D through improving the eight care processes and three treatment targets which form part of the annual national diabetes audit for patients with T2D. With the implemented improvement interventions, there was significant improvement in all care processes and all treatment targets for patients with diabetes. Achievement of all the eight care processes improved by 46.0% (p<0.001) while achievement of all three treatment targets improved by 13.5% (p<0.001). The QI programme provides an example of a data-driven large-scale multicomponent intervention delivered in primary care in ethnically diverse and socially deprived areas.

Download Full-text

RAFFI: Accurate and fast familial relationship inference in large scale biobank studies using RaPID

PLoS Genetics ◽

10.1371/journal.pgen.1009315 ◽

2021 ◽

Vol 17 (1) ◽

pp. e1009315

Author(s):

Ardalan Naseri ◽

Junjie Shi ◽

Xihong Lin ◽

Shaojie Zhang ◽

Degui Zhi

Keyword(s):

Large Scale ◽

Association Studies ◽

Scale Up ◽

Data Driven ◽

Genome Wide Association Studies ◽

Inference Method ◽

Genome Wide ◽

Familial Relationship ◽

Kinship Coefficients ◽

Data Driven Approach

Inference of relationships from whole-genome genetic data of a cohort is a crucial prerequisite for genome-wide association studies. Typically, relationships are inferred by computing the kinship coefficients (ϕ) and the genome-wide probability of zero IBD sharing (π0) among all pairs of individuals. Current leading methods are based on pairwise comparisons, which may not scale up to very large cohorts (e.g., sample size >1 million). Here, we propose an efficient relationship inference method, RAFFI. RAFFI leverages the efficient RaPID method to call IBD segments first, then estimate the ϕ and π0 from detected IBD segments. This inference is achieved by a data-driven approach that adjusts the estimation based on phasing quality and genotyping quality. Using simulations, we showed that RAFFI is robust against phasing/genotyping errors, admix events, and varying marker densities, and achieves higher accuracy compared to KING, the current leading method, especially for more distant relatives. When applied to the phased UK Biobank data with ~500K individuals, RAFFI is approximately 18 times faster than KING. We expect RAFFI will offer fast and accurate relatedness inference for even larger cohorts.

Download Full-text

Data-Driven Approach for Metabolite Relationship Recovery in Biological1H NMR Data Sets Using Iterative Statistical Total Correlation Spectroscopy

Analytical Chemistry ◽

10.1021/ac102870u ◽

2011 ◽

Vol 83 (6) ◽

pp. 2075-2082 ◽

Cited By ~ 34

Author(s):

Caroline J. Sands ◽

Muireann Coen ◽

Timothy M. D. Ebbels ◽

Elaine Holmes ◽

John C. Lindon ◽

...

Keyword(s):

Correlation Spectroscopy ◽

Data Driven ◽

Data Sets ◽

Nmr Data ◽

Total Correlation ◽

Data Driven Approach

Download Full-text

A robust data-driven approach identifies four personality types across four large data sets

Nature Human Behaviour ◽

10.1038/s41562-018-0419-z ◽

2018 ◽

Vol 2 (10) ◽

pp. 735-742 ◽

Cited By ~ 41

Author(s):

Martin Gerlach ◽

Beatrice Farb ◽

William Revelle ◽

Luís A. Nunes Amaral

Keyword(s):

Large Data ◽

Personality Types ◽

Large Data Sets ◽

Data Driven ◽

Data Sets ◽

Data Driven Approach

Download Full-text

Outlying Sequence Detection in Large Data Sets: A data-driven approach

IEEE Signal Processing Magazine ◽

10.1109/msp.2014.2329428 ◽

2014 ◽

Vol 31 (5) ◽

pp. 44-56 ◽

Cited By ~ 26

Author(s):

Ali Tajer ◽

Venugopal V. Veeravalli ◽

H. Vincent Poor

Keyword(s):

Large Data ◽

Large Data Sets ◽

Data Driven ◽

Data Sets ◽

Sequence Detection ◽

Data Driven Approach

Download Full-text

Inferring dynamic topology for decoding spatiotemporal structures in complex heterogeneous networks

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1721286115 ◽

2018 ◽

Vol 115 (37) ◽

pp. 9300-9305 ◽

Cited By ~ 10

Author(s):

Shuo Wang ◽

Erik D. Herzog ◽

István Z. Kiss ◽

William J. Schwartz ◽

Guy Bloch ◽

...

Keyword(s):

Network Topology ◽

Large Scale ◽

Data Driven ◽

Electrical Networks ◽

Chemical Oscillators ◽

Complex Interactions ◽

Dynamic Topology ◽

Estimation Problems ◽

Data Driven Approach ◽

Fast Chemical

Extracting complex interactions (i.e., dynamic topologies) has been an essential, but difficult, step toward understanding large, complex, and diverse systems including biological, financial, and electrical networks. However, reliable and efficient methods for the recovery or estimation of network topology remain a challenge due to the tremendous scale of emerging systems (e.g., brain and social networks) and the inherent nonlinearity within and between individual units. We develop a unified, data-driven approach to efficiently infer connections of networks (ICON). We apply ICON to determine topology of networks of oscillators with different periodicities, degree nodes, coupling functions, and time scales, arising in silico, and in electrochemistry, neuronal networks, and groups of mice. This method enables the formulation of these large-scale, nonlinear estimation problems as a linear inverse problem that can be solved using parallel computing. Working with data from networks, ICON is robust and versatile enough to reliably reveal full and partial resonance among fast chemical oscillators, coherent circadian rhythms among hundreds of cells, and functional connectivity mediating social synchronization of circadian rhythmicity among mice over weeks.

Download Full-text

Screening performance of abbreviated versions of the UPSIT smell test

10.1101/443127 ◽

2018 ◽

Author(s):

Theresita Joseph ◽

Stephen D. Auger ◽

Luisa Peress ◽

Daniel Rack ◽

Jack Cuzick ◽

...

Keyword(s):

Large Scale ◽

Effective Means ◽

Cost Effective ◽

Data Driven ◽

Screening Performance ◽

Smell Test ◽

Identification Test ◽

Data Driven Approach ◽

Smell Identification ◽

Smell Tests

ABSTRACTBackgroundHyposmia features in several neurodegenerative conditions, including Parkinson’s disease (PD). The University of Pennsylvania Smell Identification Test (UPSIT) is a widely used screening tool for detecting hyposmia, but is time-consuming and expensive when used on a large scale.MethodsWe assessed shorter subsets of UPSIT items for their ability to detect hyposmia in 891 healthy participants from the PREDICT-PD study. Established shorter tests included Versions A and B of both the 4-item Pocket Smell Test (PST) and 12-item Brief Smell Identification Test (BSIT). Using a data-driven approach, we evaluated screening performances of 23,231,378 combinations of 1-7 smell items from the full UPSIT.ResultsPST Versions A and B achieved sensitivity/specificity of 76.8%/64.9% and 86.6%/45.9% respectively, whilst BSIT Versions A and B achieved 83.1%/79.5% and 96.5%/51.8% for detecting hyposmia defined by the longer UPSIT. From the data-driven analysis, two optimised sets of 7 smells surpassed the screening performance of the 12 item BSITs (with validation sensitivity/specificities of 88.2%/85.4% and 100%/53.5%). A set of 4 smells (Menthol, Clove, Gingerbread and Orange) had higher sensitivity for hyposmia than PST-A, -B and even BSIT-A (with validation sensitivity 91.2%). The same 4 smells also featured amongst those most commonly misidentified by 44 individuals with PD compared to 891 PREDICT-PD controls and a screening test using these 4 smells would have identified all hyposmic patients with PD.ConclusionUsing abbreviated smell tests could provide a cost-effective means of screening for hyposmia in large cohorts, allowing more targeted administration of the UPSIT or similar smell tests.

Download Full-text

A Data-Driven Approach to Product Usage Context Identification From Online Customer Reviews

Journal of Mechanical Design ◽

10.1115/1.4044523 ◽

2019 ◽

Vol 141 (12) ◽

Author(s):

Dedy Suryadi ◽

Harrison M. Kim

Keyword(s):

Sentiment Analysis ◽

Language Processing ◽

Consumer Satisfaction ◽

Data Driven ◽

Data Sets ◽

Product Usage ◽

Customer Reviews ◽

Online Customer Reviews ◽

Data Driven Approach ◽

Usage Context

Abstract This paper proposes a data-driven methodology to automatically identify product usage contexts from online customer reviews. Product usage context is one of the factors that affect product design, consumer behavior, and consumer satisfaction. The previous works identify the usage contexts using the survey-based method or subjectively determine them. The proposed methodology, on the other hand, uses machine learning and Natural Language Processing tools to identify and cluster usage contexts from a large volume of customer reviews. Furthermore, aspect sentiment analysis is applied to capture the sentiment toward a particular usage context in a sentence. The methodology is implemented to two data sets of products, i.e., laptop and tablet. The result shows that the methodology is able to capture relevant product usage contexts and cluster bigrams that refer to similar usage context. The aspect sentiment analysis enables the observation of a product’s position with respect to its competitors for a particular usage context. For a product designer, the observation may indicate a requirement to improve the product. It may also indicate a possible market opportunity in a usage context in which most of the current products are perceived negatively by customers. Finally, it is shown that overall rating might not be a strong indicator for representing customer sentiment toward a particular usage context, due to the moderate linear correlation for most of the usage contexts in the case study.

Download Full-text