ei.Datasets: Real Data Sets for Assessing Ecological Inference Algorithms

2021 ◽  
pp. 089443932110408
Author(s):  
Jose M. Pavía
Keyword(s):  
Simulated Data ◽  
Ground Truth ◽  
Real Data ◽  
R Package ◽  
Data Sets ◽  
Ecological Inference ◽  
Inference Models ◽  
Individual Level ◽  
Inference Algorithms ◽  
Cross Classification

Ecological inference models aim to infer individual-level relationships using aggregate data. They are routinely used to estimate voter transitions between elections, disclose split-ticket voting behaviors, or infer racial voting patterns in U.S. elections. A large number of procedures have been proposed in the literature to solve these problems; therefore, an assessment and comparison of them are overdue. The secret ballot however makes this a difficult endeavor since real individual data are usually not accessible. The most recent work on ecological inference has assessed methods using a very small number of data sets with ground truth, combined with artificial, simulated data. This article dramatically increases the number of real instances by presenting a unique database (available in the R package ei.Datasets) composed of data from more than 550 elections where the true inner-cell values of the global cross-classification tables are known. The article describes how the data sets are organized, details the data curation and data wrangling processes performed, and analyses the main features characterizing the different data sets.

scholarly journals Evaluating the reproducibility of single-cell gene regulatory network inference algorithms

2020 ◽  
Author(s):  
Yoonjee Kang ◽  
Denis Thieffry ◽  
Laura Cantini
Keyword(s):  
Single Cell ◽  
Network Inference ◽  
Simulated Data ◽  
Ground Truth ◽  
Real Data ◽  
Gene Regulatory Network Inference ◽  
Sequencing Platform ◽  
Cell Network ◽  
Inference Algorithms ◽  
Inference Methods

AbstractNetworks are powerful tools to represent and investigate biological systems. The development of algorithms inferring regulatory interactions from functional genomics data has been an active area of research. With the advent of single-cell RNA-seq data (scRNA-seq), numerous methods specifically designed to take advantage of single-cell datasets have been proposed. However, published benchmarks on single-cell network inference are mostly based on simulated data. Once applied to real data, these benchmarks take into account only a small set of genes and only compare the inferred networks with an imposed ground-truth.Here, we benchmark four single-cell network inference methods based on their reproducibility, i.e. their ability to infer similar networks when applied to two independent datasets for the same biological condition. We tested each of these methods on real data from three biological conditions: human retina, T-cells in colorectal cancer, and human hematopoiesis.GENIE3 results to be the most reproducible algorithm, independently from the single-cell sequencing platform, the cell type annotation system, the number of cells constituting the dataset, or the thresholding applied to the links of the inferred networks. In order to ensure the reproducibility and ease extensions of this benchmark study, we implemented all the analyses in scNET, a Jupyter notebook available at https://github.com/ComputationalSystemsBiology/scNET.

scholarly journals Ecological Regression with Partial Identification

2019 ◽  
Vol 28 (1) ◽  
pp. 65-86
Author(s):  
Wenxin Jiang ◽  
Gary King ◽  
Allen Schmaltz ◽  
Martin A. Tanner
Keyword(s):  
Conceptual Framework ◽  
Ground Truth ◽  
Real Data ◽  
Contextual Effects ◽  
Partial Identification ◽  
Aggregate Data ◽  
Data Sets ◽  
Ecological Inference ◽  
Point Estimates ◽  
Ecological Regression

Ecological inference (EI) is the process of learning about individual behavior from aggregate data. We relax assumptions by allowing for “linear contextual effects,” which previous works have regarded as plausible but avoided due to nonidentification, a problem we sidestep by deriving bounds instead of point estimates. In this way, we offer a conceptual framework to improve on the Duncan–Davis bound, derived more than 65 years ago. To study the effectiveness of our approach, we collect and analyze 8,430 $2\times 2$ EI datasets with known ground truth from several sources—thus bringing considerably more data to bear on the problem than the existing dozen or so datasets available in the literature for evaluating EI estimators. For the 88% of real data sets in our collection that fit a proposed rule, our approach reduces the width of the Duncan–Davis bound, on average, by about 44%, while still capturing the true district-level parameter about 99% of the time. The remaining 12% revert to the Duncan–Davis bound.

scholarly journals Algorithmic Jingle Jungle: A Comparison of Implementations of Principal Axis Factoring and Promax Rotation in R and SPSS

2020 ◽  
Author(s):  
Silvia Grieder ◽  
Markus D. Steiner
Keyword(s):  
Data Structures ◽  
Principal Axis ◽  
Model Averaging ◽  
Simulated Data ◽  
Real Data ◽  
R Package ◽  
Data Sets ◽  
Potential Alternative ◽  
Principal Axis Factoring ◽  
Promax Rotation

A statistical procedure is assumed to produce comparable results across programs. Using the case of an exploratory factor analysis procedure—principal axis factoring (PAF) and promax rotation—we show that this assumption is not always justified. Procedures with equal names are sometimes implemented differently across programs: a jingle fallacy. Focusing on two popular statistical analysis programs, we indeed discovered a jingle jungle for the above procedure: Both PAF and promax rotation are implemented differently in the psych R package and in SPSS. Based on analyses with 230 real and 216,000 simulated data sets implementing 108 different data structures, we show that these differences in implementations can result in fairly different factor solutions for a variety of different data structures. Differences in the solutions for real data sets ranged from negligible to very large, with 38% displaying at least one different indicator-to-factor correspondence. A simulation study revealed systematic differences in accuracies between different implementations, and large variation between data structures, with small numbers of indicators per factor, high factor intercorrelations, and weak factors resulting in the lowest accuracies. Moreover, although there was no single combination of settings that was superior for all data structures, we identified implementations of PAF and promax that maximize performance on average. We recommend researchers to use these implementations as best way through the jungle, discuss model averaging as a potential alternative, and highlight the importance of adhering to best practices of scale construction.

scholarly journals Algorithmic jingle jungle: A comparison of implementations of principal axis factoring and promax rotation in R and SPSS

2021 ◽  
Author(s):  
Silvia Grieder ◽  
Markus D. Steiner
Keyword(s):  
Data Structures ◽  
Principal Axis ◽  
Model Averaging ◽  
Simulated Data ◽  
Real Data ◽  
R Package ◽  
Data Sets ◽  
Potential Alternative ◽  
Principal Axis Factoring ◽  
Promax Rotation

AbstractA statistical procedure is assumed to produce comparable results across programs. Using the case of an exploratory factor analysis procedure—principal axis factoring (PAF) and promax rotation—we show that this assumption is not always justified. Procedures with equal names are sometimes implemented differently across programs: a jingle fallacy. Focusing on two popular statistical analysis programs, we indeed discovered a jingle jungle for the above procedure: Both PAF and promax rotation are implemented differently in the psych R package and in SPSS. Based on analyses with 247 real and 216,000 simulated data sets implementing 108 different data structures, we show that these differences in implementations can result in fairly different factor solutions for a variety of different data structures. Differences in the solutions for real data sets ranged from negligible to very large, with 42% displaying at least one different indicator-to-factor correspondence. A simulation study revealed systematic differences in accuracies between different implementations, and large variation between data structures, with small numbers of indicators per factor, high factor intercorrelations, and weak factors resulting in the lowest accuracies. Moreover, although there was no single combination of settings that was superior for all data structures, we identified implementations of PAF and promax that maximize performance on average. We recommend researchers to use these implementations as best way through the jungle, discuss model averaging as a potential alternative, and highlight the importance of adhering to best practices of scale construction.

scholarly journals Super RaSE: Super Random Subspace Ensemble Classification

2021 ◽  
Vol 14 (12) ◽  
pp. 612
Author(s):  
Jianan Zhu ◽  
Yang Feng
Keyword(s):  
Simulated Data ◽  
Real Data ◽  
Classification Problem ◽  
R Package ◽  
Ensemble Classification ◽  
Data Sets ◽  
Random Subspace ◽  
Base Classifier ◽  
Wide Range ◽  
Random Subspace Ensemble

We propose a new ensemble classification algorithm, named super random subspace ensemble (Super RaSE), to tackle the sparse classification problem. The proposed algorithm is motivated by the random subspace ensemble algorithm (RaSE). The RaSE method was shown to be a flexible framework that can be coupled with any existing base classification. However, the success of RaSE largely depends on the proper choice of the base classifier, which is unfortunately unknown to us. In this work, we show that Super RaSE avoids the need to choose a base classifier by randomly sampling a collection of classifiers together with the subspace. As a result, Super RaSE is more flexible and robust than RaSE. In addition to the vanilla Super RaSE, we also develop the iterative Super RaSE, which adaptively changes the base classifier distribution as well as the subspace distribution. We show that the Super RaSE algorithm and its iterative version perform competitively for a wide range of simulated data sets and two real data examples. The new Super RaSE algorithm and its iterative version are implemented in a new version of the R package RaSEn.

scholarly journals Evaluating the Reproducibility of Single-Cell Gene Regulatory Network Inference Algorithms

2021 ◽  
Vol 12 ◽  
Author(s):  
Yoonjee Kang ◽  
Denis Thieffry ◽  
Laura Cantini
Keyword(s):  
Single Cell ◽  
Network Inference ◽  
Simulated Data ◽  
Ground Truth ◽  
Real Data ◽  
Biological Interactions ◽  
Gene Regulatory Network Inference ◽  
Sequencing Platform ◽  
Cell Network ◽  
Inference Algorithms

Networks are powerful tools to represent and investigate biological systems. The development of algorithms inferring regulatory interactions from functional genomics data has been an active area of research. With the advent of single-cell RNA-seq data (scRNA-seq), numerous methods specifically designed to take advantage of single-cell datasets have been proposed. However, published benchmarks on single-cell network inference are mostly based on simulated data. Once applied to real data, these benchmarks take into account only a small set of genes and only compare the inferred networks with an imposed ground-truth. Here, we benchmark six single-cell network inference methods based on their reproducibility, i.e., their ability to infer similar networks when applied to two independent datasets for the same biological condition. We tested each of these methods on real data from three biological conditions: human retina, T-cells in colorectal cancer, and human hematopoiesis. Once taking into account networks with up to 100,000 links, GENIE3 results to be the most reproducible algorithm and, together with GRNBoost2, show higher intersection with ground-truth biological interactions. These results are independent from the single-cell sequencing platform, the cell type annotation system and the number of cells constituting the dataset. Finally, GRNBoost2 and CLR show more reproducible performance once a more stringent thresholding is applied to the networks (1,000–100 links). In order to ensure the reproducibility and ease extensions of this benchmark study, we implemented all the analyses in scNET, a Jupyter notebook available at https://github.com/ComputationalSystemsBiology/scNET.

Estimating Candidate Support in Voting Rights Act Cases: Comparing Iterative EI and EI-R×C Methods

2019 ◽  
pp. 004912411985239
Author(s):  
Matt Barreto ◽  
Loren Collingwood ◽  
Sergio Garcia-Rios ◽  
Kassra AR Oskooii
Keyword(s):  
Voting Behavior ◽  
Simulated Data ◽  
Voting Rights ◽  
Data Sets ◽  
Voting Rights Act ◽  
Ecological Inference ◽  
Individual Level ◽  
Level Data ◽  
Candidate Support ◽  
Simulated Data Sets

Scholars and legal practitioners of voting rights are concerned with estimating individual-level voting behavior from aggregate-level data. The most commonly used technique, King’s ecological inference (EI), has been questioned for inflexibility in multiethnic settings or with multiple candidates. One method for estimating vote support for multiple candidates in the same election is called ecological inference: row by columns (R×C). While some simulations suggest that R×C may produce more precise estimates than the iterative EI technique, there has not been a comprehensive side-by-side comparison of the two methods using real election data that analysts and legal practitioners often rely upon in courts. We fill this void by comparing iterative EI and R×C models with a new statistical package—eiCompare—in a variety of R×C combinations including 2 candidates and 2 groups, 3 candidates and 3 groups, and up to 12 candidates and three groups and multiple candidates and four groups. Additionally, we examine the two methods with 500 simulated data sets that differ in combinations of heterogeneity, polarization, and correlation. Finally, we introduce a new model congruence score to aid scholars and voting rights analysts in the substantive interpretation of the estimates. Across all of our analyses, we find that both methods produce substantively similar results. This suggests that iterative EI and R×C can be used interchangeably when assessing precinct-level voting patterns in Voting Rights Act cases and that neither method produces bias in favor or against finding racially polarized voting patterns.

A Growth Model for Multilevel Ordinal Data

2005 ◽  
Vol 30 (4) ◽  
pp. 369-396 ◽  
Author(s):  
Eisuke Segawa
Keyword(s):  
Latent Variable ◽  
Ordinal Data ◽  
Linear Models ◽  
Growth Models ◽  
Simulated Data ◽  
Real Data ◽  
Analytic Structure ◽  
Data Sets ◽  
Data Set ◽  
Time Points

Multi-indicator growth models were formulated as special three-level hierarchical generalized linear models to analyze growth of a trait latent variable measured by ordinal items. Items are nested within a time-point, and time-points are nested within subject. These models are special because they include factor analytic structure. This model can analyze not only data with item- and time-level missing observations, but also data with time points freely specified over subjects. Furthermore, features useful for longitudinal analyses, “autoregressive error degree one” structure for the trait residuals and estimated time-scores, were included. The approach is Bayesian with Markov Chain and Monte Carlo, and the model is implemented in WinBUGS. They are illustrated with two simulated data sets and one real data set with planned missing items within a scale.

scholarly journals A Framework for the Objective Assessment of Registration Accuracy

2014 ◽  
Vol 2014 ◽  
pp. 1-11 ◽  
Author(s):  
Francesca Pizzorni Ferrarese ◽  
Flavio Simonetti ◽  
Roberto Israel Foroni ◽  
Gloria Menegaz
Keyword(s):  
Accuracy Assessment ◽  
Objective Assessment ◽  
Synthetic Data ◽  
Simulated Data ◽  
Ground Truth ◽  
Real Data ◽  
Magnetic Resonance Images ◽  
Good Prediction ◽  
Registration Accuracy ◽  
Affine Registration

Validation and accuracy assessment are the main bottlenecks preventing the adoption of image processing algorithms in the clinical practice. In the classical approach, a posteriori analysis is performed through objective metrics. In this work, a different approach based on Petri nets is proposed. The basic idea consists in predicting the accuracy of a given pipeline based on the identification and characterization of the sources of inaccuracy. The concept is demonstrated on a case study: intrasubject rigid and affine registration of magnetic resonance images. Both synthetic and real data are considered. While synthetic data allow the benchmarking of the performance with respect to the ground truth, real data enable to assess the robustness of the methodology in real contexts as well as to determine the suitability of the use of synthetic data in the training phase. Results revealed a higher correlation and a lower dispersion among the metrics for simulated data, while the opposite trend was observed for pathologic ones. Results show that the proposed model not only provides a good prediction performance but also leads to the optimization of the end-to-end chain in terms of accuracy and robustness, setting the ground for its generalization to different and more complex scenarios.

scholarly journals GRAPHICAL APPROACH TO AGE-PERIOD-COHORT ANALYSIS

2021 ◽  
Vol 20 (2) ◽  
pp. 69-77
Author(s):  
Adam Sagan
Keyword(s):  
Cohort Studies ◽  
Longitudinal Design ◽  
Cohort Analysis ◽  
Simulated Data ◽  
Real Data ◽  
R Package ◽  
Cohort Effects ◽  
Graphical Approach ◽  
Age Cohort ◽  
Accelerated Longitudinal Design

The paper presents the graphical approach to decomposition of APC effect in cohort studies (mainly applied to demographic phenomena) using multilevel or accelerated longitudinal design. The aim of the paper is to present and visualize the pure age, period and cohort effects based on simulated data with an increment of five for each successive age, period and cohort variation. In cohort analysis on real data all of the effects are usually interrelated. The analysis shows basic patterns of two-variate APC decomposition (age within period, age within cohort, cohort within period, period within age, cohort within age, period within cohort) and reveals the trajectory of curves for each of the pure effects. The APC plots are developed using apc library of R package.

Sign in / Sign up

Export Citation Format

Share Document