Comparing the MCMC Efficiency of JAGS and Stan for the Multi-Level Intercept-Only Model in the Covariance- and Mean-Based and Classic Parametrization

Bayesian MCMC is a widely used model estimation technique, and software from the BUGS family, such as JAGS, have been popular for over two decades. Recently, Stan entered the market with promises of higher efficiency fueled by advanced and more sophisticated algorithms. With this study, we want to contribute empirical results to the discussion about the sampling efficiency of JAGS and Stan. We conducted three simulation studies in which we varied the number of warmup iterations, the prior informativeness, and sample sizes and employed the multi-level intercept-only model in the covariance- and mean-based and in the classic parametrization. The target outcome was MCMC efficiency measured as effective sample size per second (ESS/s). Based on our specific (and limited) study setup, we found that (1) MCMC efficiency is much higher for the covariance- and mean-based parametrization than for the classic parametrization, (2) Stan clearly outperforms JAGS when the covariance- and mean-based parametrization is used, and that (3) JAGS clearly outperforms Stan when the classic parametrization is used.

Download Full-text

Approaches for Structural Investigations of Binary Data Using Confirmatory Factor Models

International Journal of Statistics and Probability ◽

10.5539/ijsp.v7n6p68 ◽

2018 ◽

Vol 7 (6) ◽

pp. 68

Author(s):

Karl Schweizer ◽

Siegbert Reiß ◽

Stefan Troche

Keyword(s):

Sample Size ◽

Binary Data ◽

Factor Models ◽

The Other ◽

Simulation Studies ◽

Sample Sizes ◽

Structural Investigations ◽

Ml Estimation ◽

Confirmatory Factor ◽

The Relationship

An investigation of the suitability of threshold-based and threshold-free approaches for structural investigations of binary data is reported. Both approaches implicitly establish a relationship between binary data following the binomial distribution on one hand and continuous random variables assuming a normal distribution on the other hand. In two simulation studies we investigated: whether the fit results confirm the establishment of such a relationship, whether the differences between correct and incorrect models are retained and to what degree the sample size influences the results. Both approaches proved to establish the relationship. Using the threshold-free approach it was achieved by customary ML estimation whereas robust ML estimation was necessary in the threshold-based approach. Discrimination between correct and incorrect models was observed for both approaches. Larger CFI differences were found for the threshold-free approach than for the threshold-based approach. Dependency on sample size characterized the threshold-based approach but not the threshold-free approach. The threshold-based approach tended to perform better in large sample sizes, while the threshold-free approach performed better in smaller sample sizes.

Download Full-text

Modified Toxicity Probability Interval Design: A Safer and More Reliable Method Than the 3 + 3 Design for Practical Phase I Trials

Journal of Clinical Oncology ◽

10.1200/jco.2012.45.7903 ◽

2013 ◽

Vol 31 (14) ◽

pp. 1785-1791 ◽

Cited By ~ 74

Author(s):

Yuan Ji ◽

Sue-Jane Wang

Keyword(s):

Phase I ◽

Sample Size ◽

Dose Escalation ◽

Free Software ◽

Phase I Trials ◽

Simulation Studies ◽

Matched Sample ◽

Sample Sizes ◽

Probability Interval ◽

Model Based

The 3 + 3 design is the most common choice among clinicians for phase I dose-escalation oncology trials. In recent reviews, more than 95% of phase I trials have been based on the 3 + 3 design. Given that it is intuitive and its implementation does not require a computer program, clinicians can conduct 3 + 3 dose escalations in practice with virtually no logistic cost, and trial protocols based on the 3 + 3 design pass institutional review board and biostatistics reviews quickly. However, the performance of the 3 + 3 design has rarely been compared with model-based designs in simulation studies with matched sample sizes. In the vast majority of statistical literature, the 3 + 3 design has been shown to be inferior in identifying true maximum-tolerated doses (MTDs), although the sample size required by the 3 + 3 design is often orders-of-magnitude smaller than model-based designs. In this article, through comparative simulation studies with matched sample sizes, we demonstrate that the 3 + 3 design has higher risks of exposing patients to toxic doses above the MTD than the modified toxicity probability interval (mTPI) design, a newly developed adaptive method. In addition, compared with the mTPI design, the 3 + 3 design does not yield higher probabilities in identifying the correct MTD, even when the sample size is matched. Given that the mTPI design is equally transparent, costless to implement with free software, and more flexible in practical situations, we highly encourage its adoption in early dose-escalation studies whenever the 3 + 3 design is also considered. We provide free software to allow direct comparisons of the 3 + 3 design with other model-based designs in simulation studies with matched sample sizes.

Download Full-text

Mechanical Reliability Confidence Limits

Journal of Mechanical Design ◽

10.1115/1.3453977 ◽

1978 ◽

Vol 100 (4) ◽

pp. 607-612 ◽

Cited By ~ 5

Author(s):

D. Kececioglu ◽

G. Lamarre

Keyword(s):

Sample Size ◽

Confidence Limit ◽

Effective Sample Size ◽

Confidence Limits ◽

Mechanical Reliability ◽

Sample Sizes ◽

Confidence Levels ◽

Standard Deviations ◽

Strength Distributions

Charts are presented relating the lower one-sided confidence limit on the reliability, RL1, to the effective sample size, ne, calculated from the sample sizes used to estimate the failure governing stress and strength distributions, or f(s) and f(S) respectively, and a factor K which is a function of the estimated means and standard deviations of f(s) and f(S). These graphs cover an ne range of 5 to 1000, confidence levels of 0.80, 0.90, 0.95, and 0.99, and lower one-sided limits on the reliability of 0.85 to 0.9145. The equations used to develop these charts are derived and two examples of their applications are given.

Download Full-text

Evaluation of satellite collar sample size requirements for mitigation of low-level military jet disturbance of the George River caribou herd

Rangifer ◽

10.7557/2.23.5.1713 ◽

2003 ◽

Vol 23 (5) ◽

pp. 297 ◽

Cited By ~ 1

Author(s):

Robert D. Otto ◽

Neal P.P. Simon ◽

Serge Couturier ◽

Isabelle Schmelzer

Keyword(s):

Sample Size ◽

Statistical Power ◽

Rangifer Tarandus ◽

Radio Telemetry ◽

A Priori ◽

Effective Sample Size ◽

Sample Sizes ◽

Low Level ◽

Department Of National Defence ◽

George River

Wildlife radio-telemetry and tracking projects often determine a priori required sample sizes by statistical means or default to the maximum number that can be maintained within a limited budget. After initiation of such projects, little attention is focussed on effective sample size requirements, resulting in lack of statistical power. The Department of National Defence operates a base in Labrador, Canada for low level jet fighter training activities, and maintain a sample of satellite collars on the George River caribou (Rangifer tarandus caribou) herd of the region for spatial avoidance mitiga¬tion purposes. We analysed existing location data, in conjunction with knowledge of life history, to develop estimates of satellite collar sample sizes required to ensure adequate mitigation of GRCH. We chose three levels of probability in each of six annual caribou seasons. Estimated number of collars required ranged from 15 to 52, 23 to 68, and 36 to 184 for 50%, 75%, and 90% probability levels, respectively, depending on season. Estimates can be used to make more informed decisions about mitigation of GRCH, and, generally, our approach provides a means to adaptively assess radio collar sam¬ple sizes for ongoing studies.

Download Full-text

The Implications of Alternative Allocation Criteria in Adaptive Design for Panel Surveys

Journal of Official Statistics ◽

10.1515/jos-2017-0036 ◽

2017 ◽

Vol 33 (3) ◽

pp. 781-799 ◽

Cited By ~ 3

Author(s):

Olena Kaminska ◽

Peter Lynn

Keyword(s):

Data Collection ◽

Sample Size ◽

Mixed Mode ◽

Response Rate ◽

Single Mode ◽

Quality Measure ◽

Alternative Methods ◽

Effective Sample Size ◽

Random Allocation ◽

Sample Sizes

AbstractAdaptive survey designs can be used to allocate sample elements to alternative data collection protocols in order to achieve a desired balance between some quality measure and survey costs. We compare four alternative methods for allocating sample elements to one of two data collection protocols. The methods differ in terms of the quality measure that they aim to optimize: response rate, R-indicator, coefficient of variation of the participation propensities, or effective sample size. Costs are also compared for a range of sample sizes. The data collection protocols considered are CAPI single-mode and web-CAPI sequential mixed-mode. We use data from a large experiment with random allocation to one of these two protocols. For each allocation method we predict outcomes in terms of several quality measures and costs. Although allocating the whole sample to single-mode CAPI produces a higher response rate than allocating the whole sample to the mixed-mode protocol, we find that two of the targeted allocations achieve a better response rate than single-mode CAPI at a lower cost. We also find that all four of the targeted designs out-perform both single-protocol designs in terms of representativity and effective sample size. For all but the smallest sample sizes, the adaptive designs bring cost savings relative to CAPI-only, though these are fairly modest in magnitude.

Download Full-text

A Pooled Analysis of Pharmacokinetic Variability Information for Common Probe Substrates Used in Drug-Drug Interaction Studies

Pharmacology ◽

10.1159/000485516 ◽

2018 ◽

Vol 101 (3-4) ◽

pp. 170-175

Author(s):

Chunsheng He ◽

Amber Griffies ◽

Xuan Liu ◽

Robert Adamczyk ◽

Shu-Pang Huang

Keyword(s):

Drug Interaction ◽

Sample Size ◽

Area Under The Curve ◽

Pooled Analysis ◽

Small Sample ◽

Effective Sample Size ◽

Maximum Plasma ◽

Sample Sizes ◽

Drug Drug Interaction ◽

Small Sample Sizes

Sample size estimates for drug-drug interaction (DDI) studies are often based on variability information from the literature or from historical studies, but small sample sizes in these sources may limit the precision of the estimates obtained. This project aimed to create an intra-subject variability library of the pharmacokinetic (PK) exposure parameters, area under the curve, and maximum plasma concentration, for probes commonly used in DDI studies. Data from 66 individual DDI studies in healthy subjects relating to 18 common probe substrates were pooled to increase the effective sample size for the identified probes by 1.5- to 9-fold, with corresponding improvements in precision of the intra-subject PK variability estimates in this library. These improved variability estimates will allow better assessment of the sample sizes needed for DDI studies in future.

Download Full-text

A comprehensive evaluation of methods for Mendelian randomization using realistic simulations and an analysis of 38 biomarkers for risk of type 2 diabetes

International Journal of Epidemiology ◽

10.1093/ije/dyaa262 ◽

2021 ◽

Author(s):

Guanghao Qi ◽

Nilanjan Chatterjee

Keyword(s):

Type 2 Diabetes ◽

Mendelian Randomization ◽

Association Studies ◽

Real Data ◽

Causal Effects ◽

Type I ◽

Genome Wide Association Studies ◽

Simulation Studies ◽

Sample Sizes

Abstract Background Previous studies have often evaluated methods for Mendelian randomization (MR) analysis based on simulations that do not adequately reflect the data-generating mechanisms in genome-wide association studies (GWAS) and there are often discrepancies in the performance of MR methods in simulations and real data sets. Methods We use a simulation framework that generates data on full GWAS for two traits under a realistic model for effect-size distribution coherent with the heritability, co-heritability and polygenicity typically observed for complex traits. We further use recent data generated from GWAS of 38 biomarkers in the UK Biobank and performed down sampling to investigate trends in estimates of causal effects of these biomarkers on the risk of type 2 diabetes (T2D). Results Simulation studies show that weighted mode and MRMix are the only two methods that maintain the correct type I error rate in a diverse set of scenarios. Between the two methods, MRMix tends to be more powerful for larger GWAS whereas the opposite is true for smaller sample sizes. Among the other methods, random-effect IVW (inverse-variance weighted method), MR-Robust and MR-RAPS (robust adjust profile score) tend to perform best in maintaining a low mean-squared error when the InSIDE assumption is satisfied, but can produce large bias when InSIDE is violated. In real-data analysis, some biomarkers showed major heterogeneity in estimates of their causal effects on the risk of T2D across the different methods and estimates from many methods trended in one direction with increasing sample size with patterns similar to those observed in simulation studies. Conclusion The relative performance of different MR methods depends heavily on the sample sizes of the underlying GWAS, the proportion of valid instruments and the validity of the InSIDE assumption. Down-sampling analysis can be used in large GWAS for the possible detection of bias in the MR methods.

Download Full-text

Effects of Training Set Size on Supervised Machine-Learning Land-Cover Classification of Large-Area High-Resolution Remotely Sensed Data

Remote Sensing ◽

10.3390/rs13030368 ◽

2021 ◽

Vol 13 (3) ◽

pp. 368

Author(s):

Christopher A. Ramezan ◽

Timothy A. Warner ◽

Aaron E. Maxwell ◽

Bradley S. Price

Keyword(s):

Machine Learning ◽

Sample Size ◽

Remotely Sensed ◽

Training Data ◽

Supervised Machine Learning ◽

Sample Sizes ◽

Remotely Sensed Data ◽

Large Area ◽

Training Set ◽

Set Size

The size of the training data set is a major determinant of classification accuracy. Nevertheless, the collection of a large training data set for supervised classifiers can be a challenge, especially for studies covering a large area, which may be typical of many real-world applied projects. This work investigates how variations in training set size, ranging from a large sample size (n = 10,000) to a very small sample size (n = 40), affect the performance of six supervised machine-learning algorithms applied to classify large-area high-spatial-resolution (HR) (1–5 m) remotely sensed data within the context of a geographic object-based image analysis (GEOBIA) approach. GEOBIA, in which adjacent similar pixels are grouped into image-objects that form the unit of the classification, offers the potential benefit of allowing multiple additional variables, such as measures of object geometry and texture, thus increasing the dimensionality of the classification input data. The six supervised machine-learning algorithms are support vector machines (SVM), random forests (RF), k-nearest neighbors (k-NN), single-layer perceptron neural networks (NEU), learning vector quantization (LVQ), and gradient-boosted trees (GBM). RF, the algorithm with the highest overall accuracy, was notable for its negligible decrease in overall accuracy, 1.0%, when training sample size decreased from 10,000 to 315 samples. GBM provided similar overall accuracy to RF; however, the algorithm was very expensive in terms of training time and computational resources, especially with large training sets. In contrast to RF and GBM, NEU, and SVM were particularly sensitive to decreasing sample size, with NEU classifications generally producing overall accuracies that were on average slightly higher than SVM classifications for larger sample sizes, but lower than SVM for the smallest sample sizes. NEU however required a longer processing time. The k-NN classifier saw less of a drop in overall accuracy than NEU and SVM as training set size decreased; however, the overall accuracies of k-NN were typically less than RF, NEU, and SVM classifiers. LVQ generally had the lowest overall accuracy of all six methods, but was relatively insensitive to sample size, down to the smallest sample sizes. Overall, due to its relatively high accuracy with small training sample sets, and minimal variations in overall accuracy between very large and small sample sets, as well as relatively short processing time, RF was a good classifier for large-area land-cover classifications of HR remotely sensed data, especially when training data are scarce. However, as performance of different supervised classifiers varies in response to training set size, investigating multiple classification algorithms is recommended to achieve optimal accuracy for a project.

Download Full-text