scholarly journals Improvements of the Kohonen R package for application of self-organising maps on large data sets

2018 ◽  
Author(s):  
J. Kruisselbrink
Keyword(s):  
Large Data ◽  
R Package ◽  
Large Data Sets ◽  
Data Sets

scholarly journals Taxa: An R package implementing data standards and methods for taxonomic data

F1000Research ◽  
2018 ◽  
Vol 7 ◽  
pp. 272
Author(s):  
Zachary S.L. Foster ◽  
Scott Chamberlain ◽  
Niklaus J. Grünwald
Keyword(s):  
Large Data ◽  
R Package ◽  
Large Data Sets ◽  
Specific Information ◽  
Data Sets ◽  
Widespread Application ◽  
Taxonomic Information ◽  
Hierarchical Nature ◽  
Application Specific ◽  
Associated Data

The taxa R package provides a set of tools for defining and manipulating taxonomic data. The recent and widespread application of DNA sequencing to community composition studies is making large data sets with taxonomic information commonplace. However, compared to typical tabular data, this information is encoded in many different ways and the hierarchical nature of taxonomic classifications makes it difficult to work with. There are many R packages that use taxonomic data to varying degrees but there is currently no cross-package standard for how this information is encoded and manipulated. We developed the R package taxa to provide a robust and flexible solution to storing and manipulating taxonomic data in R and any application-specific information associated with it. Taxa provides parsers that can read common sources of taxonomic information (taxon IDs, sequence IDs, taxon names, and classifications) from nearly any format while preserving associated data. Once parsed, the taxonomic data and any associated data can be manipulated using a cohesive set of functions modeled after the popular R package dplyr. These functions take into account the hierarchical nature of taxa and can modify the taxonomy or associated data in such a way that both are kept in sync. Taxa is currently being used by the metacoder and taxize packages, which provide broadly useful functionality that we hope will speed adoption by users and developers.

scholarly journals plantR: An R package and workflow for managing species records from biological collections

2021 ◽  
Author(s):  
Renato Augusto Ferreira Lima ◽  
Andrea Sanchez-Tapia ◽  
Sara R. Mortara ◽  
Hans Steege ◽  
Marinez F. Siqueira
Keyword(s):  
Biodiversity Conservation ◽  
Large Data ◽  
Data Retrieval ◽  
R Package ◽  
Large Data Sets ◽  
Data Sets ◽  
Data Repositories ◽  
Biological Collections ◽  
Data Editing ◽  
User Friendly

Species records from biological collections are becoming increasingly available online. This unprecedented availability of records has largely supported recent studies in taxonomy, bio-geography, macro-ecology, and biodiversity conservation. Biological collections vary in their documentation and notation standards, which have changed through time. For different reasons, neither collections nor data repositories perform the editing, formatting and standardization of the data, leaving these tasks to the final users of the species records (e.g. taxonomists, ecologists and conservationists). These tasks are challenging, particularly when working with millions of records from hundreds of biological collections. To help collection curators and final users to perform those tasks, we introduce plantR an open-source package that provides a comprehensive toolbox to manage species records from biological collections. The package is accompanied by the proposal of a reproducible workflow to manage this type of data in taxonomy, ecology and biodiversity conservation. It is implemented in R and designed to handle relatively large data sets as fast as possible. Initially designed to handle plant species records, many of the plantR features also apply to other groups of organisms, given that the data structure is similar. The plantR workflow includes tools to (1) download records from different data repositories, (2) standardize typical fields associated with species records, (3) validate the locality, geographical coordinates, taxonomic nomenclature and species identifications, including the retrieval of duplicates across collections, and (4) summarize and export records, including the construction of species checklists with vouchers. Other R packages provide tools to tackle some of the workflow steps described above. But in addition to the new features and resources related to the data editing and validation, the greatest strength of plantR is to provide a comprehensive and user-friendly workflow in one single environment, performing all tasks from data retrieval to export. Thus, plantR can help researchers to better assess data quality and avoid data leakage in a wide variety of studies using species records.

scholarly journals Taxa: An R package implementing data standards and methods for taxonomic data

F1000Research ◽  
2018 ◽  
Vol 7 ◽  
pp. 272 ◽  
Author(s):  
Zachary S.L. Foster ◽  
Scott Chamberlain ◽  
Niklaus J. Grünwald
Keyword(s):  
Large Data ◽  
R Package ◽  
Large Data Sets ◽  
Specific Information ◽  
Data Sets ◽  
Widespread Application ◽  
Taxonomic Information ◽  
Hierarchical Nature ◽  
Application Specific ◽  
Associated Data

The taxa R package provides a set of tools for defining and manipulating taxonomic data. The recent and widespread application of DNA sequencing to community composition studies is making large data sets with taxonomic information commonplace. However, compared to typical tabular data, this information is encoded in many different ways and the hierarchical nature of taxonomic classifications makes it difficult to work with. There are many R packages that use taxonomic data to varying degrees but there is currently no cross-package standard for how this information is encoded and manipulated. We developed the R package taxa to provide a robust and flexible solution to storing and manipulating taxonomic data in R and any application-specific information associated with it. Taxa provides parsers that can read common sources of taxonomic information (taxon IDs, sequence IDs, taxon names, and classifications) from nearly any format while preserving associated data. Once parsed, the taxonomic data and any associated data can be manipulated using a cohesive set of functions modeled after the popular R package dplyr. These functions take into account the hierarchical nature of taxa and can modify the taxonomy or associated data in such a way that both are kept in sync. Taxa is currently being used by the metacoder and taxize packages, which provide broadly useful functionality that we hope will speed adoption by users and developers.

scholarly journals rehh 2.0: a reimplementation of the R package rehh to detect positive selection from haplotype structure

2016 ◽  
Author(s):  
Mathieu Gautier ◽  
Alexander Klassmann ◽  
Renaud Vitalis
Keyword(s):  
Positive Selection ◽  
Computation Time ◽  
Large Data ◽  
R Package ◽  
Large Data Sets ◽  
Data Sets ◽  
Genome Wide ◽  
Haplotype Data ◽  
Order Of Magnitude ◽  
Genomic Regions

AbstractIdentifying genomic regions with unusually high local haplotype homozygosity represents a powerful strategy to characterize candidate genes responding to natural or artificial positive selection. To that end, statistics measuring the extent of haplotype homozygosity within (e.g., EHH, iHS) and between (Rsb or XP-EHH) populations have been proposed in the literature. The rehh package for R was previously developed to facilitate genome-wide scans of selection, based on the analysis of long-range haplotypes. However, its performance wasn’t sufficient to cope with the growing size of available data sets. Here we propose a major upgrade of the rehh package, which includes an improved processing of the input files, a faster algorithm to enumerate haplotypes, as well as multi-threading. As illustrated with the analysis of large human haplotype data sets, these improvements decrease the computation time by more than an order of magnitude. This new version of rehh will thus allow performing iHS-, Rsb- or XP-EHH-based scans on large data sets. The package rehh 2.0 is available from the CRAN repository (http://cran.r-project.org/web/packages/rehh/index.html) together with help files and a detailed manual.

An example of spectrum imaging used for comparison of EELS quantitative analysis techniques on Al-Li

1991 ◽  
Vol 49 ◽  
pp. 726-727
Author(s):  
John A. Hunt
Keyword(s):  
Quantitative Analysis ◽  
Large Data ◽  
Difference Spectrum ◽  
Large Data Sets ◽  
Foil Thickness ◽  
Data Sets ◽  
Analysis Techniques ◽  
Spectrum Imaging ◽  
Normal Spectrum ◽  
Electron Energy Loss

Spectrum-imaging is a useful technique for comparing different processing methods on very large data sets which are identical for each method. This paper is concerned with comparing methods of electron energy-loss spectroscopy (EELS) quantitative analysis on the Al-Li system. The spectrum-image analyzed here was obtained from an Al-10at%Li foil aged to produce δ' precipitates that can span the foil thickness. Two 1024 channel EELS spectra offset in energy by 1 eV were recorded and stored at each pixel in the 80x80 spectrum-image (25 Mbytes). An energy range of 39-89eV (20 channels/eV) are represented. During processing the spectra are either subtracted to create an artifact corrected difference spectrum, or the energy offset is numerically removed and the spectra are added to create a normal spectrum. The spectrum-images are processed into 2D floating-point images using methods and software described in [1].

Cluster analysis for large data sets: applications to individual aerosol particles from the mid-pacific

1992 ◽  
Vol 50 (2) ◽  
pp. 1488-1489
Author(s):  
Thomas W. Shattuck ◽  
James R. Anderson ◽  
Neil W. Tindale ◽  
Peter R. Buseck
Keyword(s):  
Cluster Analysis ◽  
Chemical Reactivity ◽  
Large Data ◽  
Large Data Sets ◽  
Particle Analysis ◽  
Data Sets ◽  
Halogen Chemistry ◽  
Complete Study ◽  
Components Analysis ◽  
Automated Scanning

Individual particle analysis involves the study of tens of thousands of particles using automated scanning electron microscopy and elemental analysis by energy-dispersive, x-ray emission spectroscopy (EDS). EDS produces large data sets that must be analyzed using multi-variate statistical techniques. A complete study uses cluster analysis, discriminant analysis, and factor or principal components analysis (PCA). The three techniques are used in the study of particles sampled during the FeLine cruise to the mid-Pacific ocean in the summer of 1990. The mid-Pacific aerosol provides information on long range particle transport, iron deposition, sea salt ageing, and halogen chemistry.Aerosol particle data sets suffer from a number of difficulties for pattern recognition using cluster analysis. There is a great disparity in the number of observations per cluster and the range of the variables in each cluster. The variables are not normally distributed, they are subject to considerable experimental error, and many values are zero, because of finite detection limits. Many of the clusters show considerable overlap, because of natural variability, agglomeration, and chemical reactivity.

NVESTIGATION OF THE EFFICIENCY OF DISTRIBUTED INFORMATION SYSTEMS BASED ON THE PROCESSING OF LARGE AMOUNTS OF DATA

2019 ◽  
Author(s):  
Mykhajlo Klymash ◽  
Olena Hordiichuk — Bublivska ◽  
Ihor Tchaikovskyi ◽  
Oksana Urikova
Keyword(s):  
Distributed Systems ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Data Decomposition ◽  
Distributed Information ◽  
Software Model ◽  
Computing Performance ◽  
Mapreduce Model ◽  
Singular Data

In this article investigated the features of processing large arrays of information for distributed systems. A method of singular data decomposition is used to reduce the amount of data processed, eliminating redundancy. Dependencies of com­putational efficiency on distributed systems were obtained using the MPI messa­ging protocol and MapReduce node interaction software model. Were analyzed the effici­ency of the application of each technology for the processing of different sizes of data: Non — distributed systems are inefficient for large volumes of information due to low computing performance. It is proposed to use distributed systems that use the method of singular data decomposition, which will reduce the amount of information processed. The study of systems using the MPI protocol and MapReduce model obtained the dependence of the duration calculations time on the number of processes, which testify to the expediency of using distributed computing when processing large data sets. It is also found that distributed systems using MapReduce model work much more efficiently than MPI, especially with large amounts of data. MPI makes it possible to perform calculations more efficiently for small amounts of information. When increased the data sets, advisable to use the Map Reduce model.

Using Large Data Sets and Statistical Tools to Define “Failure” and Demonstrate the Safety of DPR

2018 ◽  
Vol 2018 (6) ◽  
pp. 38-39
Author(s):  
Austa Parker ◽  
Yan Qu ◽  
David Hokanson ◽  
Jeff Soller ◽  
Eric Dickenson ◽  
...  
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Statistical Tools

scholarly journals Online Judging Platform Utilizing Dynamic Plagiarism Detection Facilities

Computers ◽  
2021 ◽  
Vol 10 (4) ◽  
pp. 47
Author(s):  
Fariha Iffath ◽  
A. S. M. Kayes ◽  
Md. Tahsin Rahman ◽  
Jannatul Ferdows ◽  
Mohammad Shamsul Arefin ◽  
...  
Keyword(s):  
Source Code ◽  
Large Data ◽  
Large Data Sets ◽  
Detection Technique ◽  
Data Sets ◽  
Plagiarism Detection ◽  
Source Codes ◽  
Efficient Detection ◽  
Mathematical Problems ◽  
Automatic Scoring

A programming contest generally involves the host presenting a set of logical and mathematical problems to the contestants. The contestants are required to write computer programs that are capable of solving these problems. An online judge system is used to automate the judging procedure of the programs that are submitted by the users. Online judges are systems designed for the reliable evaluation of the source codes submitted by the users. Traditional online judging platforms are not ideally suitable for programming labs, as they do not support partial scoring and efficient detection of plagiarized codes. When considering this fact, in this paper, we present an online judging framework that is capable of automatic scoring of codes by detecting plagiarized contents and the level of accuracy of codes efficiently. Our system performs the detection of plagiarism by detecting fingerprints of programs and using the fingerprints to compare them instead of using the whole file. We used winnowing to select fingerprints among k-gram hash values of a source code, which was generated by the Rabin–Karp Algorithm. The proposed system is compared with the existing online judging platforms to show the superiority in terms of time efficiency, correctness, and feature availability. In addition, we evaluated our system by using large data sets and comparing the run time with MOSS, which is the widely used plagiarism detection technique.

Sign in / Sign up

Export Citation Format

Share Document