scholarly journals Relative model fit does not predict topological accuracy in single-gene protein phylogenetics

2019 ◽  
Author(s):  
Stephanie J. Spielman
Keyword(s):  
Model Selection ◽  
Best Practice ◽  
Goodness Of Fit ◽  
Single Gene ◽  
Phylogenetic Reconstruction ◽  
Evolutionary Model ◽  
Specific Model ◽  
Model Fit ◽  
Real Sequence ◽  
Inference Models

AbstractIt is regarded as best practice in phylogenetic reconstruction to perform relative model selection to determine an appropriate evolutionary model for the data. This procedure ranks a set of candidate models according to their goodness-of-fit to the data, commonly using an information theoretic criterion. Users then specify the best-ranking model for inference. While it is often assumed that better-fitting models translate to increase accuracy, recent studies have shown that the specific model employed may not substantially affect inferences. We examine whether there is a systematic relationship between relative model fit and topological inference accuracy in protein phylogenetics, using simulations and real sequences. Simulations employed site-heterogeneous mechanistic codon models that are distinct from protein-level phylogenetic inference models. This strategy allows us to investigate how protein models performs when they are mis-specified to the data, as will be the case for any real sequence analysis. We broadly find that phylogenies inferred across models with vastly different fits to the data produce highly consistent topologies. We additionally find that all models infer similar proportions of false positive splits, raising the possibility that all available models of protein evolution are similarly misspecified. Moreover, we find that the parameter-rich GTR model, whose amino-acid exchangeabilities are free parameters, performs similarly to models with fixed exchangeabilities, although the inference precision associated with GTR models was not examined. We conclude that, while relative model selection may not hinder phylogenetic analysis on protein data, it may not offer specific predictable improvements and is not a reliable proxy for accuracy.

scholarly journals Relative Model Fit Does Not Predict Topological Accuracy in Single-Gene Protein Phylogenetics

2020 ◽  
Vol 37 (7) ◽  
pp. 2110-2123 ◽  
Author(s):  
Stephanie J Spielman
Keyword(s):  
Model Selection ◽  
Best Practice ◽  
Goodness Of Fit ◽  
Single Gene ◽  
Phylogenetic Reconstruction ◽  
Evolutionary Model ◽  
Specific Model ◽  
Model Fit ◽  
Real Sequence ◽  
Inference Models

Abstract It is regarded as best practice in phylogenetic reconstruction to perform relative model selection to determine an appropriate evolutionary model for the data. This procedure ranks a set of candidate models according to their goodness of fit to the data, commonly using an information theoretic criterion. Users then specify the best-ranking model for inference. Although it is often assumed that better-fitting models translate to increase accuracy, recent studies have shown that the specific model employed may not substantially affect inferences. We examine whether there is a systematic relationship between relative model fit and topological inference accuracy in protein phylogenetics, using simulations and real sequences. Simulations employed site-heterogeneous mechanistic codon models that are distinct from protein-level phylogenetic inference models, allowing us to investigate how protein models performs when they are misspecified to the data, as will be the case for any real sequence analysis. We broadly find that phylogenies inferred across models with vastly different fits to the data produce highly consistent topologies. We additionally find that all models infer similar proportions of false-positive splits, raising the possibility that all available models of protein evolution are similarly misspecified. Moreover, we find that the parameter-rich GTR (general time reversible) model, whose amino acid exchangeabilities are free parameters, performs similarly to models with fixed exchangeabilities, although the inference precision associated with GTR models was not examined. We conclude that, although relative model selection may not hinder phylogenetic analysis on protein data, it may not offer specific predictable improvements and is not a reliable proxy for accuracy.

scholarly journals Relative model selection can be sensitive to multiple sequence alignment uncertainty

2021 ◽  
Author(s):  
Stephanie J Spielman ◽  
Molly Miraglia
Keyword(s):  
Model Selection ◽  
Phylogenetic Analyses ◽  
Phylogenetic Reconstruction ◽  
Selection Procedure ◽  
Evolutionary Model ◽  
Ancestral State ◽  
Sequence Alignments ◽  
Multiple Sequence ◽  
Multiple Sequence Alignments ◽  
Nucleotide Data

Multiple sequence alignments (MSAs) represent the fundamental unit of data inputted to most comparative sequence analyses. In phylogenetic analyses in particular, errors in MSA construction have the potential to induce further errors in downstream analyses such as phylogenetic reconstruction itself, ancestral state reconstruction, and divergence estimation. In addition to providing phylogenetic methods with an MSA to analyze, researchers must also specify a suitable evolutionary model for the given analysis. Most commonly, researchers apply relative model selection to select a model from candidate set and then provide both the MSA and the selected model as input to subsequent analyses. While the influence of MSA errors has been explored for most stages of phylogenetics pipelines, the potential effects of MSA uncertainty on the relative model selection procedure itself have not been explored. In this study, we assessed the consistency of relative model selection when presented with multiple perturbed versions of a given MSA. We find that while relative model selection is mostly robust to MSA uncertainty, in a substantial proportion of circumstances, relative model selection identifies distinct best-fitting models from different MSAs created from the same set of sequences. We find that this issue is more pervasive for nucleotide data compared to amino-acid data. However, we also find that it is challenging to predict whether relative model selection will be robust or sensitive to uncertainty in a given MSA. We find that that MSA uncertainty can affect virtually all steps of phylogenetic analysis pipelines to a greater extent than has previously been recognized, including relative model selection.

Distance-Based Statistical Inference

2020 ◽  
Vol 8 (1) ◽  
Author(s):  
Marianthi Markatou ◽  
Dimitrios Karlis ◽  
Yuxin Ding
Keyword(s):  
Model Selection ◽  
Goodness Of Fit ◽  
Scientific Literature ◽  
Scientific Work ◽  
Model Fit ◽  
Loss Functions ◽  
Annual Review ◽  
Publication Date ◽  
Model Assessment ◽  
Large Samples

Statistical distances, divergences, and similar quantities have an extensive history and play an important role in the statistical and related scientific literature. This role shows up in estimation, where we often use estimators based on minimizing a distance. Distances also play a prominent role in hypothesis testing and in model selection. We review the statistical properties of distances that are often used in scientific work, present their properties, and show how they compare to each other. We discuss an approximation framework for model-based inference using statistical distances. Emphasis is placed on identifying in what sense and which statistical distances can be interpreted as loss functions and used for model assessment. We review a special class of distances, the class of quadratic distances, connect it with the classical goodness-of-fit paradigm, and demonstrate its use in the problem of assessing model fit. These methods can be used in analyzing very large samples. Expected final online publication date for the Annual Review of Statistics, Volume 8 is March 8, 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

scholarly journals Neglecting model selection alters phylogenetic inference

2019 ◽  
Author(s):  
Michael Gerth
Keyword(s):  
Model Selection ◽  
Molecular Phylogenetics ◽  
Negative Impact ◽  
Model Fitting ◽  
Phylogenetic Reconstruction ◽  
Evolutionary Model ◽  
Sequence Alignments ◽  
Modern Biology ◽  
Wide Range ◽  
History Of

ABSTRACTMolecular phylogenetics is a standard tool in modern biology that informs the evolutionary history of genes, organisms, and traits, and as such is important in a wide range of disciplines from medicine to palaeontology. Maximum likelihood phylogenetic reconstruction involves assumptions about the evolutionary processes that underlie the dataset to be analysed. These assumptions must be specified in forms of an evolutionary model, and a number of criteria may be used to identify the best-fitting from a plethora of available models of DNA evolution. Using many empirical and simulated nucleotide sequence alignments, Abadi et al.1 have recently found that phylogenetic inferences using best models identified by six different model selection criteria are, on average, very similar to each other. They further claimed that using the model GTR+I+G4 without prior model-fitting results in similarly accurate phylogenetic estimates, and consequently that skipping model selection entirely has no negative impact on many phylogenetic applications. Focussing on this claim, I here revisit and re-analyse some of the data put forward by Abadi et al. I argue that while the presented analyses are sound, the results are misrepresented and in fact - in line with previous work - demonstrate that model selection consistently leads to different phylogenetic estimates compared with using fixed models.

scholarly journals Optimizing Consistency and Coverage in Configurational Causal Modeling

2021 ◽  
pp. 004912412199555
Author(s):  
Michael Baumgartner ◽  
Mathias Ambühl
Keyword(s):  
Model Selection ◽  
Causal Inference ◽  
Comparative Methods ◽  
Causal Models ◽  
Causal Modeling ◽  
Model Fit

Consistency and coverage are two core parameters of model fit used by configurational comparative methods (CCMs) of causal inference. Among causal models that perform equally well in other respects (e.g., robustness or compliance with background theories), those with higher consistency and coverage are typically considered preferable. Finding the optimally obtainable consistency and coverage scores for data [Formula: see text], so far, is a matter of repeatedly applying CCMs to [Formula: see text] while varying threshold settings. This article introduces a procedure called ConCovOpt that calculates, prior to actual CCM analyses, the consistency and coverage scores that can optimally be obtained by models inferred from [Formula: see text]. Moreover, we show how models reaching optimal scores can be methodically built in case of crisp-set and multi-value data. ConCovOpt is a tool, not for blindly maximizing model fit, but for rendering transparent the space of viable models at optimal fit scores in order to facilitate informed model selection—which, as we demonstrate by various data examples, may have substantive modeling implications.

scholarly journals Computational Modeling of Music Cognition: A Case Study on Model Selection

2006 ◽  
Vol 23 (5) ◽  
pp. 365-376 ◽  
Author(s):  
Henkjan Honing
Keyword(s):  
Model Selection ◽  
Strong Evidence ◽  
Goodness Of Fit ◽  
Computational Models ◽  
Recent Literature ◽  
Music Cognition ◽  
Kinematic Model ◽  
Music Performance ◽  
Limited Range

While the most common way of evaluating a computational model is to see whether it shows a good fit with the empirical data, recent literature on theory testing and model selection criticizes the assumption that this is actually strong evidence for the validity of a model. This article presents a case study from music cognition (modeling the ritardandi in music performance) and compares two families of computational models (kinematic and perceptual) using three different model selection criteria: goodness-of-fit, model simplicity, and the degree of surprise in the predictions. In the light of what counts as strong evidence for a model’s validity—namely that it makes limited range, nonsmooth, and relatively surprising predictions—the perception-based model is preferred over the kinematic model.

Edinburgh Handedness Inventory Scores: Caucasian and American Indian College Students

1994 ◽  
Vol 78 (2) ◽  
pp. 675-680 ◽  
Author(s):  
Sherry M. Dingman ◽  
Mary A. Mroczka
Keyword(s):  
College Students ◽  
American Indian ◽  
American Indians ◽  
Goodness Of Fit ◽  
Single Gene ◽  
Empirical Support ◽  
Racial Groups ◽  
Frequency Distributions ◽  
Chi Squared ◽  
American Indian College Students

Laterality Quotients for 80 American Indian college students were less right-biased than those for 80 Caucasian college students on the Edinburgh Handedness Inventory. Oldfield's 1971 empirically derived deciles for the Edinburgh Handedness Inventory were used to assign decile levels to the data. Deciles were then used to assign data to one of three proposed handedness phenotype classifications. Pheno-type classifications were based on Annett's 1985 proposed distribution for a single gene theorized to underlie human handedness. Chi-squared goodness-of-fit analysis showed that the data for Caucasian college students did not differ significantly from what would be anticipated by Annett's model, but American Indians differed significantly. Results provide empirical support for the hypothesis that frequency distributions for Annett's hypothesized right-shift gene may differ across racial groups.

scholarly journals Influence of Hydrological Model Selection on Simulation of Moderate and Extreme Flow Events: A Case Study of the Blue Nile Basin

2016 ◽  
Vol 2016 ◽  
pp. 1-28 ◽  
Author(s):  
Charles Onyutha
Keyword(s):  
Model Selection ◽  
Extreme Events ◽  
Goodness Of Fit ◽  
Model Performance ◽  
Flow Conditions ◽  
Nile Basin ◽  
Blue Nile ◽  
Low Flows ◽  
Blue Nile Basin ◽  
Extreme Flow

Five hydrological models were applied based on data from the Blue Nile Basin. Optimal parameters of each model were obtained by automatic calibration. Model performance was tested under both moderate and extreme flow conditions. Extreme events for the model performance evaluation were extracted based on seven criteria. Apart from graphical techniques, there were nine statistical “goodness-of-fit” metrics used to judge the model performance. It was found that whereas the influence of model selection may be minimal in the simulation of normal flow events, it can lead to large under- and/or overestimations of extreme events. Besides, the selection of the best model for extreme events may be influenced by the choice of the statistical “goodness-of-fit” measures as well as the criteria for extraction of high and low flows. It was noted that the use of overall water-balance-based objective function not only is suitable for moderate flow conditions but also influences the models to perform better for high flows than low flows. Thus, the choice of a particular model is recommended to be made on a case by case basis with respect to the objectives of the modeling as well as the results from evaluation of the intermodel differences.

scholarly journals A Revised Model of Trust in Internet-Based Health Information and Advice: Cross-Sectional Questionnaire Study

10.2196/11125 ◽  
2019 ◽  
Vol 21 (11) ◽  
pp. e11125
Author(s):  
Elizabeth Sillence ◽  
John Matthew Blythe ◽  
Pam Briggs ◽  
Mark Moss
Keyword(s):  
Goodness Of Fit ◽  
Online Survey ◽  
Patient Experiences ◽  
Model Fit ◽  
Key Factors ◽  
The United Kingdom ◽  
Personal Experiences ◽  
And Coping ◽  
Fit Index ◽  
Significant Direct Effect

Background The internet continues to offer new forms of support for health decision making. Government, charity, and commercial websites increasingly offer a platform for shared personal health experiences, and these are just some of the opportunities that have arisen in a largely unregulated arena. Understanding how people trust and act on this information has always been an important issue and remains so, particularly as the design practices of health websites continue to evolve and raise further concerns regarding their trustworthiness. Objective The aim of this study was to identify the key factors influencing US and UK citizens’ trust and intention to act on advice found on health websites and to understand the role of patient experiences. Methods A total of 1123 users took part in an online survey (625 from the United States and 498 from the United Kingdom). They were asked to recall their previous visit to a health website. The online survey consisted of an updated general Web trust questionnaire to account for personal experiences plus questions assessing key factors associated with trust in health websites (information corroboration and coping perception) and intention to act. We performed principal component analysis (PCA), then explored the relationship between the factor structure and outcomes by testing the fit to the sampled data using structural equation modeling (SEM). We also explored the model fit across US and UK populations. Results PCA of the general Web trust questionnaire revealed 4 trust factors: (1) personal experiences, (2) credibility and impartiality, (3) privacy, and (4) familiarity. In the final SEM model, trust was found to have a significant direct effect on intention to act (beta=.59; P<.001), and of the trust factors, only credibility and impartiality had a significant direct effect on trust (beta=.79; P<.001). The impact of personal experiences on trust was mediated through information corroboration (beta=.06; P=.04). Variables specific to electronic health (eHealth; information corroboration and coping) were found to substantially improve the model fit, and differences in information corroboration were found between US and UK samples. The final model accounting for all factors achieved a good fit (goodness-of-fit index [0.95], adjusted goodness-of-fit index [0.93], root mean square error of approximation [0.50], and comparative fit index [0.98]) and explained 65% of the variance in trust and 41% of the variance in intention to act. Conclusions Credibility and impartiality continue to be key predictors of trust in eHealth websites. Websites with patient experiences can positively influence trust but only if users first corroborate the information through other sources. The need for corroboration was weaker in the United Kingdom, where website familiarity reduced the need to check information elsewhere. These findings are discussed in relation to existing trust models, patient experiences, and health literacy.

scholarly journals Model Selection in Peptide-receptor Radionuclide Therapy for an Accurate Determination of Time Integrated Activity Coefficients

2021 ◽  
Vol 2019 (1) ◽  
pp. 012079
Author(s):  
N Atikah ◽  
A Riana ◽  
A Dwi ◽  
Z Anwari ◽  
Misrawati ◽  
...  
Keyword(s):  
Model Selection ◽  
Activity Coefficients ◽  
Radionuclide Therapy ◽  
Goodness Of Fit ◽  
Accurate Determination ◽  
Peptide Receptor Radionuclide Therapy ◽  
Information Criterion ◽  
Selection Function ◽  
Pharmacokinetic Data ◽  
Peptide Receptor

Abstract Calculation of accurate time-integrated activity coefficients (TIACs) is desirable in nuclear medicine dosimetry. The accuracy of the calculated TIACs is highly dependent on the fit function. However, systematic studies of determining a good function for peptide-receptor radionuclide therapy (PRRT) in different patients have not been reported in the literature. The aim of this study was to individually determine the best function for the calculation of TIACs in tumor and kidneys using a model selection based on the goodness of fit criteria and Corrected Akaike Information Criterion (AICc). The data used in this study was pharmacokinetic data of 111In-DOTATATE in tumor and kidneys obtained from 4 PRRT patients. Eleven functions with various parameterizations were formulated to describe the biokinetic data of 111In-DOTATATE in tumor and kidneys. The model selection was performed by choosing the best function from the function with sufficient goodness of fit based on the smallest AICc. Based on the results of model selection, function A 1 -(λ 1+λphys )t was selected as the best function for all tumor and kidneys in patients with meningioma tumors. By using this function, the calculated of TIACs could be more accurate for future patients with meningioma tumor.

Sign in / Sign up

Export Citation Format

Share Document