scholarly journals Bpop: an efficient program for estimating base population allele frequencies in single and multiple group structured populations

2020 ◽  
Vol 29 (3) ◽  
Author(s):  
Ismo Stranden ◽  
Esa A. Mäntysaari
Keyword(s):  
Computing Time ◽  
Large Data ◽  
Generalized Least Squares ◽  
Structured Populations ◽  
Allele Frequencies ◽  
Data Sets ◽  
Dense Matrix ◽  
Base Population ◽  
Multiple Group ◽  
Efficient Program

Base population allele frequencies (AF) should be used in genomic evaluations. A program named Bpop was implemented to estimate base population AF using a generalized least squares (GLS) method when the base population individuals can be assigned to groups. The required dense matrix products involving (A22 )-1v were implemented efficiently using sparse submatrices of A-1, where A and A22 are pedigree relationship matrices for all and genotyped animals, respectively. Three approaches were implemented: iteration on pedigree (IOP), iteration in memory (IM), and direct inversion by sparsity preserving Cholesky decomposition (CHM). The test data had 1.5 million animals genotyped using 50240 markers. Total computing time (the product (A22)-11) was 53 min (1.2 min) by IOP, 51 min (0.3 min) by IM, and 56 min (4.6 min) by CHM. Peak computer core memory use was 0.67 GB by IOP, 0.80 GB by IM, and 7.53 GB by CHM. Thus, the IOP and IM approaches can be recommended for large data sets because of their low memory use and computing time.

scholarly journals Tucker Tensor Analysis of Matérn Functions in Spatial Statistics

2019 ◽  
Vol 19 (1) ◽  
pp. 101-122 ◽  
Author(s):  
Alexander Litvinenko ◽  
David Keyes ◽  
Venera Khoromskaia ◽  
Boris N. Khoromskij ◽  
Hermann G. Matthies
Keyword(s):  
Covariance Matrix ◽  
Computing Time ◽  
Large Data ◽  
Three Dimensions ◽  
Low Rank ◽  
Tensor Rank ◽  
Data Sets ◽  
Storage Cost ◽  
Covariance Functions ◽  
Rank Tensor

AbstractIn this work, we describe advanced numerical tools for working with multivariate functions and for the analysis of large data sets. These tools will drastically reduce the required computing time and the storage cost, and, therefore, will allow us to consider much larger data sets or finer meshes. Covariance matrices are crucial in spatio-temporal statistical tasks, but are often very expensive to compute and store, especially in three dimensions. Therefore, we approximate covariance functions by cheap surrogates in a low-rank tensor format. We apply the Tucker and canonical tensor decompositions to a family of Matérn- and Slater-type functions with varying parameters and demonstrate numerically that their approximations exhibit exponentially fast convergence. We prove the exponential convergence of the Tucker and canonical approximations in tensor rank parameters. Several statistical operations are performed in this low-rank tensor format, including evaluating the conditional covariance matrix, spatially averaged estimation variance, computing a quadratic form, determinant, trace, loglikelihood, inverse, and Cholesky decomposition of a large covariance matrix. Low-rank tensor approximations reduce the computing and storage costs essentially. For example, the storage cost is reduced from an exponential{\mathcal{O}(n^{d})}to a linear scaling{\mathcal{O}(drn)}, wheredis the spatial dimension,nis the number of mesh points in one direction, andris the tensor rank. Prerequisites for applicability of the proposed techniques are the assumptions that the data, locations, and measurements lie on a tensor (axes-parallel) grid and that the covariance function depends on a distance,{\|x-y\|}.

G-TDSCOR: one-phonon TDS corrections for measured integrated Bragg intensities

1983 ◽  
Vol 16 (1) ◽  
pp. 154-156 ◽  
Author(s):  
M. Sakata ◽  
A. W. Stevenson ◽  
J. Harada
Keyword(s):  
Reciprocal Lattice ◽  
Computing Time ◽  
Large Data ◽  
Data Sets ◽  
Reciprocal Lattice Point ◽  
Acta Cryst ◽  
The One ◽  
Comparison Of The Results ◽  
Integrated Intensities ◽  
Volume Approximation

A computer program for calculating the one-phonon thermal diffuse scattering (TDS) contribution to observed integrated intensities of Bragg reflections from single crystals has been written. The program is based on a general formula [Harada & Sakata (1974). Acta Cryst. A30, 77–82; Sakata & Harada (1976). Acta Cryst. A35, 426–433] which is applicable to any crystal system if elastic constants are available. The volume integral with respect to the wavevector, over the region swept out around the reciprocal-lattice point by the counter in the course of a measurement, has been simplified by use of the spherical volume approximation (SVA). Use of the SVA greatly reduces computing time for the case of large data sets. Comparison of the results with those obtained without using the SVA is given and the limitations of the SVA are pointed out.

An example of spectrum imaging used for comparison of EELS quantitative analysis techniques on Al-Li

1991 ◽  
Vol 49 ◽  
pp. 726-727
Author(s):  
John A. Hunt
Keyword(s):  
Quantitative Analysis ◽  
Large Data ◽  
Difference Spectrum ◽  
Large Data Sets ◽  
Foil Thickness ◽  
Data Sets ◽  
Analysis Techniques ◽  
Spectrum Imaging ◽  
Normal Spectrum ◽  
Electron Energy Loss

Spectrum-imaging is a useful technique for comparing different processing methods on very large data sets which are identical for each method. This paper is concerned with comparing methods of electron energy-loss spectroscopy (EELS) quantitative analysis on the Al-Li system. The spectrum-image analyzed here was obtained from an Al-10at%Li foil aged to produce δ' precipitates that can span the foil thickness. Two 1024 channel EELS spectra offset in energy by 1 eV were recorded and stored at each pixel in the 80x80 spectrum-image (25 Mbytes). An energy range of 39-89eV (20 channels/eV) are represented. During processing the spectra are either subtracted to create an artifact corrected difference spectrum, or the energy offset is numerically removed and the spectra are added to create a normal spectrum. The spectrum-images are processed into 2D floating-point images using methods and software described in [1].

Cluster analysis for large data sets: applications to individual aerosol particles from the mid-pacific

1992 ◽  
Vol 50 (2) ◽  
pp. 1488-1489
Author(s):  
Thomas W. Shattuck ◽  
James R. Anderson ◽  
Neil W. Tindale ◽  
Peter R. Buseck
Keyword(s):  
Cluster Analysis ◽  
Chemical Reactivity ◽  
Large Data ◽  
Large Data Sets ◽  
Particle Analysis ◽  
Data Sets ◽  
Halogen Chemistry ◽  
Complete Study ◽  
Components Analysis ◽  
Automated Scanning

Individual particle analysis involves the study of tens of thousands of particles using automated scanning electron microscopy and elemental analysis by energy-dispersive, x-ray emission spectroscopy (EDS). EDS produces large data sets that must be analyzed using multi-variate statistical techniques. A complete study uses cluster analysis, discriminant analysis, and factor or principal components analysis (PCA). The three techniques are used in the study of particles sampled during the FeLine cruise to the mid-Pacific ocean in the summer of 1990. The mid-Pacific aerosol provides information on long range particle transport, iron deposition, sea salt ageing, and halogen chemistry.Aerosol particle data sets suffer from a number of difficulties for pattern recognition using cluster analysis. There is a great disparity in the number of observations per cluster and the range of the variables in each cluster. The variables are not normally distributed, they are subject to considerable experimental error, and many values are zero, because of finite detection limits. Many of the clusters show considerable overlap, because of natural variability, agglomeration, and chemical reactivity.

NVESTIGATION OF THE EFFICIENCY OF DISTRIBUTED INFORMATION SYSTEMS BASED ON THE PROCESSING OF LARGE AMOUNTS OF DATA

2019 ◽  
Author(s):  
Mykhajlo Klymash ◽  
Olena Hordiichuk — Bublivska ◽  
Ihor Tchaikovskyi ◽  
Oksana Urikova
Keyword(s):  
Distributed Systems ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Data Decomposition ◽  
Distributed Information ◽  
Software Model ◽  
Computing Performance ◽  
Mapreduce Model ◽  
Singular Data

In this article investigated the features of processing large arrays of information for distributed systems. A method of singular data decomposition is used to reduce the amount of data processed, eliminating redundancy. Dependencies of com­putational efficiency on distributed systems were obtained using the MPI messa­ging protocol and MapReduce node interaction software model. Were analyzed the effici­ency of the application of each technology for the processing of different sizes of data: Non — distributed systems are inefficient for large volumes of information due to low computing performance. It is proposed to use distributed systems that use the method of singular data decomposition, which will reduce the amount of information processed. The study of systems using the MPI protocol and MapReduce model obtained the dependence of the duration calculations time on the number of processes, which testify to the expediency of using distributed computing when processing large data sets. It is also found that distributed systems using MapReduce model work much more efficiently than MPI, especially with large amounts of data. MPI makes it possible to perform calculations more efficiently for small amounts of information. When increased the data sets, advisable to use the Map Reduce model.

Using Large Data Sets and Statistical Tools to Define “Failure” and Demonstrate the Safety of DPR

2018 ◽  
Vol 2018 (6) ◽  
pp. 38-39
Author(s):  
Austa Parker ◽  
Yan Qu ◽  
David Hokanson ◽  
Jeff Soller ◽  
Eric Dickenson ◽  
...  
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Statistical Tools

scholarly journals Online Judging Platform Utilizing Dynamic Plagiarism Detection Facilities

Computers ◽  
2021 ◽  
Vol 10 (4) ◽  
pp. 47
Author(s):  
Fariha Iffath ◽  
A. S. M. Kayes ◽  
Md. Tahsin Rahman ◽  
Jannatul Ferdows ◽  
Mohammad Shamsul Arefin ◽  
...  
Keyword(s):  
Source Code ◽  
Large Data ◽  
Large Data Sets ◽  
Detection Technique ◽  
Data Sets ◽  
Plagiarism Detection ◽  
Source Codes ◽  
Efficient Detection ◽  
Mathematical Problems ◽  
Automatic Scoring

A programming contest generally involves the host presenting a set of logical and mathematical problems to the contestants. The contestants are required to write computer programs that are capable of solving these problems. An online judge system is used to automate the judging procedure of the programs that are submitted by the users. Online judges are systems designed for the reliable evaluation of the source codes submitted by the users. Traditional online judging platforms are not ideally suitable for programming labs, as they do not support partial scoring and efficient detection of plagiarized codes. When considering this fact, in this paper, we present an online judging framework that is capable of automatic scoring of codes by detecting plagiarized contents and the level of accuracy of codes efficiently. Our system performs the detection of plagiarism by detecting fingerprints of programs and using the fingerprints to compare them instead of using the whole file. We used winnowing to select fingerprints among k-gram hash values of a source code, which was generated by the Rabin–Karp Algorithm. The proposed system is compared with the existing online judging platforms to show the superiority in terms of time efficiency, correctness, and feature availability. In addition, we evaluated our system by using large data sets and comparing the run time with MOSS, which is the widely used plagiarism detection technique.

Correlations of random classifiers on large data sets

2021 ◽  
Author(s):  
Věra Kůrková ◽  
Marcello Sanguineti
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets

scholarly journals Galaxy spin direction distribution in HST and SDSS show similar large-scale asymmetry

2020 ◽  
Vol 37 ◽  
Author(s):  
Lior Shamir
Keyword(s):  
Large Scale ◽  
Spiral Galaxies ◽  
Hubble Space Telescope ◽  
Gravitational Interaction ◽  
Large Data ◽  
Sloan Digital Sky Survey ◽  
Data Sets ◽  
Dipole Axis ◽  
Data Set ◽  
The Asymmetry

Abstract Several recent observations using large data sets of galaxies showed non-random distribution of the spin directions of spiral galaxies, even when the galaxies are too far from each other to have gravitational interaction. Here, a data set of $\sim8.7\cdot10^3$ spiral galaxies imaged by Hubble Space Telescope (HST) is used to test and profile a possible asymmetry between galaxy spin directions. The asymmetry between galaxies with opposite spin directions is compared to the asymmetry of galaxies from the Sloan Digital Sky Survey. The two data sets contain different galaxies at different redshift ranges, and each data set was annotated using a different annotation method. The results show that both data sets show a similar asymmetry in the COSMOS field, which is covered by both telescopes. Fitting the asymmetry of the galaxies to cosine dependence shows a dipole axis with probabilities of $\sim2.8\sigma$ and $\sim7.38\sigma$ in HST and SDSS, respectively. The most likely dipole axis identified in the HST galaxies is at $(\alpha=78^{\rm o},\delta=47^{\rm o})$ and is well within the $1\sigma$ error range compared to the location of the most likely dipole axis in the SDSS galaxies with $z>0.15$ , identified at $(\alpha=71^{\rm o},\delta=61^{\rm o})$ .

Sign in / Sign up

Export Citation Format

Share Document