Clarifying Agreement Calculations and Analysis for End-User Elicitation Studies

We clarify fundamental aspects of end-user elicitation, enabling such studies to be run and analyzed with confidence, correctness, and scientific rigor. To this end, our contributions are multifold. We introduce a formal model of end-user elicitation in HCI and identify three types of agreement analysis: expert , codebook , and computer . We show that agreement is a mathematical tolerance relation generating a tolerance space over the set of elicited proposals. We review current measures of agreement and show that all can be computed from an agreement graph . In response to recent criticisms, we show that chance agreement represents an issue solely for inter-rater reliability studies and not for end-user elicitation, where it is opposed by chance disagreement . We conduct extensive simulations of 16 statistical tests for agreement rates, and report Type I errors and power. Based on our findings, we provide recommendations for practitioners and introduce a five-level hierarchy for elicitation studies.

Statistical tests for intra-tumour clonal co-occurrence and exclusivity

Tumour progression is an evolutionary process in which different clones evolve over time, leading to intra-tumour heterogeneity. Interactions between clones can affect tumour evolution and hence disease progression and treatment outcome. Intra-tumoural pairs of mutations that are overrepresented in a co-occurring or clonally exclusive fashion over a cohort of patient samples may be suggestive of a synergistic effect between the different clones carrying these mutations. We therefore developed a novel statistical testing framework, called GeneAccord, to identify such gene pairs that are altered in distinct subclones of the same tumour. We analysed our framework for calibration and power. By comparing its performance to baseline methods, we demonstrate that to control type I errors, it is essential to account for the evolutionary dependencies among clones. In applying GeneAccord to the single-cell sequencing of a cohort of 123 acute myeloid leukaemia patients, we find 1 clonally co-occurring and 8 clonally exclusive gene pairs. The clonally exclusive pairs mostly involve genes of the key signalling pathways.

Forecasting the Risks of an Organization Operating Natural Gas Vehicles Using a Scoring Model of Logistic Regression in the Presence of Expert Restrictions

The paper proposes a method and describes a mathematical model for express analysis of the attractiveness of the operation of vehicles running on natural gas for a motor transport company. The proposed solution is based on a logistic regression scoring model used by banks to assess the creditworthiness of a borrower. To improve the quality of the results, the model is extended with a set of expert restrictions formulated in the form of rules. During the analysis, signs were identified that require quantization, since individual intervals of values ??turned out to be associated with risk in different ways. The developed mathematical model is implemented in the form of software in a high-level programming language, the information of the model is stored in a database management system and is integrated with an information system for supporting management decisions when operating vehicles on natural gas. The developed athematical model was tested on a test training sample. The test results showed a satisfactory accuracy of the proposed model at the level of 77 % without the use of expert restrictions and 79 % with their use. At the same time, the share of Type II errors was 2.7 %, and Type I errors were 7.2 %, which indicates that the model is quite conservative, and a relatively high proportion of vehicles that meet the requirements were rejected.

When Enough Is Really Enough? On the Minimum Number of Landslides to Build Reliable Susceptibility Models

Mapping existing landslides is a fundamental prerequisite to build any reliable susceptibility model. From a series of landslide presence/absence conditions and associated landscape characteristics, a binary classifier learns how to distinguish potentially stable and unstable slopes. In data rich areas where landslide inventories are available, addressing the collection of these can already be a challenging task. However, in data scarce contexts, where geoscientists do not get access to pre-existing inventories, the only solution is to map landslides from scratch. This operation can be extremely time-consuming if manually performed or prone to type I errors if done automatically. This is even more exacerbated if done over large geographic regions. In this manuscript we examine the issue of mapping requirements for west Tajikistan where no complete landslide inventory is available. The key question is: How many landslides should be required to develop reliable landslide susceptibility models based on statistical modeling? In fact, for such a wide and extremely complex territory, the collection of an inventory that is sufficiently detailed requires a large investment in time and human resources. However, at which point of the mapping procedure, would the resulting susceptibility model produce significantly better results as compared to a model built with less information? We addressed this question by implementing a binomial Generalized Additive Model trained and validated with different landslide proportions and measured the induced variability in the resulting susceptibility model. The results of this study are very site-specific but we proposed a very functional protocol to investigate a problem which is underestimated in the literature.

Statistical Modelling with Birnbaum-Saunders Distribution

In this article, it is assumed that the distribution of the error terms is the Birnbaum-Saunders distribution in the process of one-way ANOVA. The Birnbaum-Saunders distribution has been widely used in reliability analysis especially in fatigue-life models. In reliability analysis, nonnormal distribution is much more common than the normal distribution. We obtain the estimation of the parameters og interest by maximum likelihood method. We also propose new test statistics based on these estimators . The efficiencies of the maximum likelihood estimators and the Type I errors obtained by using the proposed estimators are compared with normal theory via Monte Carlo simulation study. At the end of the study, the real life example is given just for the illustration of the method.

Controlling for human population stratification in rare variant association studies

Population stratification is a confounder of genetic association studies. In analyses of rare variants, corrections based on principal components (PCs) and linear mixed models (LMMs) yield conflicting conclusions. Studies evaluating these approaches generally focused on limited types of structure and large sample sizes. We investigated the properties of several correction methods through a large simulation study using real exome data, and several within- and between-continent stratification scenarios. We considered different sample sizes, with situations including as few as 50 cases, to account for the analysis of rare disorders. Large samples showed that accounting for stratification was more difficult with a continental than with a worldwide structure. When considering a sample of 50 cases, an inflation of type-I-errors was observed with PCs for small numbers of controls (≤ 100), and with LMMs for large numbers of controls (≥ 1000). We also tested a novel local permutation method (LocPerm), which maintained a correct type-I-error in all situations. Powers were equivalent for all approaches pointing out that the key issue is to properly control type-I-errors. Finally, we found that power of analyses including small numbers of cases can be increased, by adding a large panel of external controls, provided an appropriate stratification correction was used.

Update: Gender differences in CABG outcomes—Have we bridged the gap?

Background Appreciation of unique presentation, patterns and underlying pathophysiology of coronary artery disease in women has driven gender based risk stratification and risk reduction efforts over the last decade. Data regarding whether these advances have resulted in unequivocal improvements in outcomes of CABG in women is conflicting. The objective of our study was to assess gender differences in post-operative outcomes following CABG. Methods Retrospective analyses of institutional data housed in the Society of Thoracic Surgeons (STS) database for patients undergoing CABG between 2002 and 2020 were conducted. Multivariable regression analysis was conducted to investigate gender differences in post-operative outcomes. P-values were adjusted using Bonferroni correction to reduce type-I errors. Results Our final cohort of 6,250 patients had fewer women than men (1,339 vs. 4,911). more women were diabetic (52.0% vs. 41.2%, p<0.001) and hypertensive (89.1% vs. 84.0%, p<0.001). Women had higher adjusted odds of developing ventilator dependence >48 hours (OR: 1.65 [1.21, 2.45], p = 0.002) and cardiac readmissions (OR: 1.56 [1.27, 2.30], p = 0.003). After adjustment for comorbidity burden, mortality rates in women were comparable to those of age-matched men. Conclusion The findings of our study indicate that despite apparent reduction of differences in mortality, the burden of postoperative morbidity is still high among women.

Computer-generated facial areas of interest in eye tracking research: A simulation study

Advances in eye tracking technology have enabled the development of interactive experimental setups to study social attention. Since these setups differ substantially from the eye tracker manufacturer’s test conditions, validation is essential with regard to data quality and other factors potentially threatening data validity. In this study, we evaluated the impact of data accuracy and areas of interest (AOIs) size on the classification of simulated gaze data. We defined AOIs of different sizes using the Limited-Radius Voronoi-Tessellation (LRVT) method, and simulated gaze data for facial target points with varying data accuracy. As hypothesized, we found that data accuracy and AOI size had strong effects on gaze classification. In addition, these effects were not independent and differed for falsely classified gaze inside AOIs (Type I errors) and falsely classified gaze outside the predefined AOIs (Type II errors). The results indicate that smaller AOIs generally minimize false classifications as long as data accuracy is good enough. For studies with lower data accuracy, Type II errors can still be compensated to some extent by using larger AOIs, but at the cost of an increased probability of Type I errors. Proper estimation of data accuracy is therefore essential for making informed decisions regarding the size of AOIs.

Perhaps Psychology’s Replication Crisis is a Theoretical Crisis that is Only Masquerading as a Statistical One

The “replication crisis” may well be the single most important challenge facing empirical psychological research today. It appears that highly trained scientists, often without understanding the potentially dire long-term implications, have been mishandling standard statistical procedures in the service of attaining statistical “significance.” Exacerbating the problem, most academic journals do not publish research that has not produced a “significant” result. This toxic combination has resulted in journals apparently publishing many Type I errors and declining to publish many true failures to reject H0. In response, there has been an urgent call from some psychologists that studies be registered in advance so that their rationales, hypotheses, variables, sample sizes, and statistical analyses are recorded in advance, leaving less room for post hoc manipulation. In this chapter, I argue that this “open science” approach, though laudable, will prove insufficient because the null hypothesis significance test (NHST) is a poor criterion for scientific truth, even when it is handled correctly. The root of the problem is that, whatever statistical problems psychology may have, the discipline never developed the theoretical maturity required. For decades it has been satisfied testing weak theories that predict, at best, only the direction of the effect, rather than the size of effect. Indeed, uncritical acceptance of NHST by the discipline may have served to stunt psychology’s theoretical growth by giving researchers a way of building a successful career without having to develop models that make precise predictions. Improving our statistical “hygiene” would be a good thing, to be sure, but it is unlikely to resolve psychology’s growing credibility problem until our theoretical practices mature considerably.

The Optimal Machine Learning-Based Missing Data Imputation for the Cox Proportional Hazard Model

An adequate imputation of missing data would significantly preserve the statistical power and avoid erroneous conclusions. In the era of big data, machine learning is a great tool to infer the missing values. The root means square error (RMSE) and the proportion of falsely classified entries (PFC) are two standard statistics to evaluate imputation accuracy. However, the Cox proportional hazards model using various types requires deliberate study, and the validity under different missing mechanisms is unknown. In this research, we propose supervised and unsupervised imputations and examine four machine learning-based imputation strategies. We conducted a simulation study under various scenarios with several parameters, such as sample size, missing rate, and different missing mechanisms. The results revealed the type-I errors according to different imputation techniques in the survival data. The simulation results show that the non-parametric “missForest” based on the unsupervised imputation is the only robust method without inflated type-I errors under all missing mechanisms. In contrast, other methods are not valid to test when the missing pattern is informative. Statistical analysis, which is improperly conducted, with missing data may lead to erroneous conclusions. This research provides a clear guideline for a valid survival analysis using the Cox proportional hazard model with machine learning-based imputations.