scholarly journals Fast ordered sampling of DNA sequence variants

2017 ◽  
Author(s):  
Anthony J. Greenberg
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Hard Drives ◽  
Genome Wide ◽  
On Line ◽  
Dna Sequence Variants ◽  
Simple Sampling ◽  
Representative Samples ◽  
Memory Efficient

AbstractExplosive growth in the amount of genomic data is matched by increasing power of consumer-grade computers. Even applications that require powerful servers can be quickly tested on desktop or laptop machines if we can generate representative samples from large data sets. I describe a fast and memory-efficient implementation of an on-line sampling method developed for tape drives 30 years ago. Focusing on genotype files, I test the performance of this technique on modern solid-state and spinning hard drives, and show that it performs well compared to a simple sampling scheme. I illustrate its utility by developing a method to quickly estimate genome-wide patterns of linkage disequilibrium (LD) decay with distance. I provide open-source software that samples loci from several variant format files, a separate program that performs LD decay estimates, and a C++ library that lets developers incorporate these methods into their own projects.

scholarly journals GenomegaMap: Within-Species Genome-Wide dN/dS Estimation from over 10,000 Genomes

2020 ◽  
Vol 37 (8) ◽  
pp. 2450-2460 ◽  
Author(s):  
Daniel J Wilson ◽  
Derrick W Crook ◽  
Timothy E A Peto ◽  
A Sarah Walker ◽  
Sarah J Hoosdally ◽  
...  
Keyword(s):  
Protein A ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Protein Coding ◽  
Dead Box ◽  
Coalescent Simulations ◽  
Genome Wide ◽  
Signature Of Selection ◽  
Signatures Of Selection

Abstract The dN/dS ratio provides evidence of adaptation or functional constraint in protein-coding genes by quantifying the relative excess or deficit of amino acid-replacing versus silent nucleotide variation. Inexpensive sequencing promises a better understanding of parameters, such as dN/dS, but analyzing very large data sets poses a major statistical challenge. Here, I introduce genomegaMap for estimating within-species genome-wide variation in dN/dS, and I apply it to 3,979 genes across 10,209 tuberculosis genomes to characterize the selection pressures shaping this global pathogen. GenomegaMap is a phylogeny-free method that addresses two major problems with existing approaches: 1) It is fast no matter how large the sample size and 2) it is robust to recombination, which causes phylogenetic methods to report artefactual signals of adaptation. GenomegaMap uses population genetics theory to approximate the distribution of allele frequencies under general, parent-dependent mutation models. Coalescent simulations show that substitution parameters are well estimated even when genomegaMap’s simplifying assumption of independence among sites is violated. I demonstrate the ability of genomegaMap to detect genuine signatures of selection at antimicrobial resistance-conferring substitutions in Mycobacterium tuberculosis and describe a novel signature of selection in the cold-shock DEAD-box protein A gene deaD/csdA. The genomegaMap approach helps accelerate the exploitation of big data for gaining new insights into evolution within species.

Visual Communication in Times of Crisis: The Fukushima Nuclear Accident

Leonardo ◽  
2012 ◽  
Vol 45 (2) ◽  
pp. 113-118 ◽  
Author(s):  
Rama C. Hoetzlein
Keyword(s):  
News Media ◽  
Visual Communication ◽  
Large Data ◽  
Nuclear Accident ◽  
Large Data Sets ◽  
Fukushima Nuclear Accident ◽  
Data Sets ◽  
On Line ◽  
The Impact ◽  
Information Aesthetics

This paper follows the development of visual communication through information visualization in the wake of the Fukushima nuclear accident in Japan. While information aesthetics are often applied to large data sets retrospectively, the author developed new works concurrently with an ongoing crisis to examine the impact and social aspects of visual communication while events continued to unfold. The resulting work, Fukushima Nuclear Accident—Radiation Comparison Map, is a reflection of rapidly acquired data, collaborative on-line analysis and reflective criticism of contemporary news media, resolved into a coherent picture through the participation of an on-line community.

Integrating Process and Structure in the Concept of Youth: A Case for Secondary Analysis

1988 ◽  
Vol 36 (4) ◽  
pp. 706-732 ◽  
Author(s):  
Gill Jones
Keyword(s):  
Young People ◽  
Social Groups ◽  
Secondary Analysis ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Pass Through ◽  
Integrating Process ◽  
Large Representative ◽  
Representative Samples

During its eighty-year history, the sociology of youth has provided a succession of partial explanations of the circumstances and responses of the young. The paper critically reviews this history and defines the challenge currently faced by sociologists: to develop a conceptual framework for understanding both the transitions young people pass through as they become adult, and the differential experiences of young people from different social groups. It is argued that such a framework requires the integration of the concepts of process and structure Secondary analysis of existing large data sets is suggested as an available means of meeting the challenge, since such data sets can provide both large representative samples which allow inter-group comparisons, and wide-ranging longitudinal data which allows the study of processes

On-line learning for very large data sets

2005 ◽  
Vol 21 (2) ◽  
pp. 137-151 ◽  
Author(s):  
L�on Bottou ◽  
Yann Le Cun
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
On Line ◽  
On Line Learning

scholarly journals rehh 2.0: a reimplementation of the R package rehh to detect positive selection from haplotype structure

2016 ◽  
Author(s):  
Mathieu Gautier ◽  
Alexander Klassmann ◽  
Renaud Vitalis
Keyword(s):  
Positive Selection ◽  
Computation Time ◽  
Large Data ◽  
R Package ◽  
Large Data Sets ◽  
Data Sets ◽  
Genome Wide ◽  
Haplotype Data ◽  
Order Of Magnitude ◽  
Genomic Regions

AbstractIdentifying genomic regions with unusually high local haplotype homozygosity represents a powerful strategy to characterize candidate genes responding to natural or artificial positive selection. To that end, statistics measuring the extent of haplotype homozygosity within (e.g., EHH, iHS) and between (Rsb or XP-EHH) populations have been proposed in the literature. The rehh package for R was previously developed to facilitate genome-wide scans of selection, based on the analysis of long-range haplotypes. However, its performance wasn’t sufficient to cope with the growing size of available data sets. Here we propose a major upgrade of the rehh package, which includes an improved processing of the input files, a faster algorithm to enumerate haplotypes, as well as multi-threading. As illustrated with the analysis of large human haplotype data sets, these improvements decrease the computation time by more than an order of magnitude. This new version of rehh will thus allow performing iHS-, Rsb- or XP-EHH-based scans on large data sets. The package rehh 2.0 is available from the CRAN repository (http://cran.r-project.org/web/packages/rehh/index.html) together with help files and a detailed manual.

An example of spectrum imaging used for comparison of EELS quantitative analysis techniques on Al-Li

1991 ◽  
Vol 49 ◽  
pp. 726-727
Author(s):  
John A. Hunt
Keyword(s):  
Quantitative Analysis ◽  
Large Data ◽  
Difference Spectrum ◽  
Large Data Sets ◽  
Foil Thickness ◽  
Data Sets ◽  
Analysis Techniques ◽  
Spectrum Imaging ◽  
Normal Spectrum ◽  
Electron Energy Loss

Spectrum-imaging is a useful technique for comparing different processing methods on very large data sets which are identical for each method. This paper is concerned with comparing methods of electron energy-loss spectroscopy (EELS) quantitative analysis on the Al-Li system. The spectrum-image analyzed here was obtained from an Al-10at%Li foil aged to produce δ' precipitates that can span the foil thickness. Two 1024 channel EELS spectra offset in energy by 1 eV were recorded and stored at each pixel in the 80x80 spectrum-image (25 Mbytes). An energy range of 39-89eV (20 channels/eV) are represented. During processing the spectra are either subtracted to create an artifact corrected difference spectrum, or the energy offset is numerically removed and the spectra are added to create a normal spectrum. The spectrum-images are processed into 2D floating-point images using methods and software described in [1].

Cluster analysis for large data sets: applications to individual aerosol particles from the mid-pacific

1992 ◽  
Vol 50 (2) ◽  
pp. 1488-1489
Author(s):  
Thomas W. Shattuck ◽  
James R. Anderson ◽  
Neil W. Tindale ◽  
Peter R. Buseck
Keyword(s):  
Cluster Analysis ◽  
Chemical Reactivity ◽  
Large Data ◽  
Large Data Sets ◽  
Particle Analysis ◽  
Data Sets ◽  
Halogen Chemistry ◽  
Complete Study ◽  
Components Analysis ◽  
Automated Scanning

Individual particle analysis involves the study of tens of thousands of particles using automated scanning electron microscopy and elemental analysis by energy-dispersive, x-ray emission spectroscopy (EDS). EDS produces large data sets that must be analyzed using multi-variate statistical techniques. A complete study uses cluster analysis, discriminant analysis, and factor or principal components analysis (PCA). The three techniques are used in the study of particles sampled during the FeLine cruise to the mid-Pacific ocean in the summer of 1990. The mid-Pacific aerosol provides information on long range particle transport, iron deposition, sea salt ageing, and halogen chemistry.Aerosol particle data sets suffer from a number of difficulties for pattern recognition using cluster analysis. There is a great disparity in the number of observations per cluster and the range of the variables in each cluster. The variables are not normally distributed, they are subject to considerable experimental error, and many values are zero, because of finite detection limits. Many of the clusters show considerable overlap, because of natural variability, agglomeration, and chemical reactivity.

Sign in / Sign up

Export Citation Format

Share Document