scholarly journals Assessing the Global and Local Uncertainty of Scientific Evidence in the Presence of Model Misspecification

2021 ◽  
Vol 9 ◽  
Author(s):  
Mark L. Taper ◽  
Subhash R. Lele ◽  
José M. Ponciano ◽  
Brian Dennis ◽  
Christopher L. Jerde
Keyword(s):  
Reference Model ◽  
Scientific Evidence ◽  
Model Misspecification ◽  
Parametric Bootstrap ◽  
Identification Problem ◽  
Structural Equations ◽  
Sampling Distribution ◽  
Standard Theory ◽  
Strength Of Evidence ◽  
Global And Local

Scientists need to compare the support for models based on observed phenomena. The main goal of the evidential paradigm is to quantify the strength of evidence in the data for a reference model relative to an alternative model. This is done via an evidence function, such as ΔSIC, an estimator of the sample size scaled difference of divergences between the generating mechanism and the competing models. To use evidence, either for decision making or as a guide to the accumulation of knowledge, an understanding of the uncertainty in the evidence is needed. This uncertainty is well characterized by the standard statistical theory of estimation. Unfortunately, the standard theory breaks down if the models are misspecified, as is commonly the case in scientific studies. We develop non-parametric bootstrap methodologies for estimating the sampling distribution of the evidence estimator under model misspecification. This sampling distribution allows us to determine how secure we are in our evidential statement. We characterize this uncertainty in the strength of evidence with two different types of confidence intervals, which we term “global” and “local.” We discuss how evidence uncertainty can be used to improve scientific inference and illustrate this with a reanalysis of the model identification problem in a prominent landscape ecology study using structural equations.

scholarly journals Assessing Model Fit in Latent Class Analysis When Asymptotics Do Not Hold

Methodology ◽  
2015 ◽  
Vol 11 (2) ◽  
pp. 65-79 ◽  
Author(s):  
Geert H. van Kollenburg ◽  
Joris Mulder ◽  
Jeroen K. Vermunt
Keyword(s):  
Sample Size ◽  
Goodness Of Fit ◽  
Latent Class ◽  
Parametric Bootstrap ◽  
Error Rates ◽  
Asymptotic Distributions ◽  
Type I ◽  
Test Statistics ◽  
P Values ◽  
Global And Local

The application of latent class (LC) analysis involves evaluating the LC model using goodness-of-fit statistics. To assess the misfit of a specified model, say with the Pearson chi-squared statistic, a p-value can be obtained using an asymptotic reference distribution. However, asymptotic p-values are not valid when the sample size is not large and/or the analyzed contingency table is sparse. Another problem is that for various other conceivable global and local fit measures, asymptotic distributions are not readily available. An alternative way to obtain the p-value for the statistic of interest is by constructing its empirical reference distribution using resampling techniques such as the parametric bootstrap or the posterior predictive check (PPC). In the current paper, we show how to apply the parametric bootstrap and two versions of the PPC to obtain empirical p-values for a number of commonly used global and local fit statistics within the context of LC analysis. The main difference between the PPC using test statistics and the parametric bootstrap is that the former takes into account parameter uncertainty. The PPC using discrepancies has the advantage that it is computationally much less intensive than the other two resampling methods. In a Monte Carlo study we evaluated Type I error rates and power of these resampling methods when used for global and local goodness-of-fit testing in LC analysis. Results show that both the bootstrap and the PPC using test statistics are generally good alternatives to asymptotic p-values and can also be used when (asymptotic) distributions are not known. Nominal Type I error rates were not met when sample size was small and the contingency table has many cells. Overall the PPC using test statistics was somewhat more conservative than the parametric bootstrap. We have also replicated previous research suggesting that the Pearson χ2 statistic should in many cases be preferred over the likelihood-ratio G2 statistic. Power to reject a model for which the number of LCs was one less than in the population was very high, unless sample size was small. When the contingency tables are very sparse, the total bivariate residual (TBVR) statistic, which is based on bivariate relationships, still had very high power, signifying its usefulness in assessing model fit.

Late lessons from early warnings: towards precaution and realism in research and policy

2005 ◽  
Vol 52 (6) ◽  
pp. 25-34 ◽  
Author(s):  
D. Gee ◽  
M.P. Krayer von Krauss
Keyword(s):  
Precautionary Principle ◽  
Scientific Evidence ◽  
False Positives ◽  
False Negatives ◽  
Strength Of Evidence ◽  
The Precautionary Principle ◽  
Value Judgements ◽  
Bradford Hill Criteria ◽  
The Difference ◽  
Early Warnings

This paper focuses on the evidentiary aspects of the precautionary principle. Three points are highlighted: (i) the difference between association and causation; (ii) how the strength of scientific evidence can be considered; and (iii) the reasons why regulatory regimes tend to err in the direction of false negatives rather than false positives. The point is made that because obtaining evidence of causation can take many decades of research, the precautionary principle can be invoked to justify action when evidence of causation is not available, but there is good scientific evidence of an association between exposures and impacts. It is argued that the appropriate level of proof is context dependent, as “appropriateness” is based on value judgements about the acceptability of the costs, about the distribution of the costs, and about the consequences of being wrong. A complementary approach to evaluating the strength of scientific evidence is to focus on the level of uncertainty. If decision makers are made aware of the limitations of the knowledge base, they can compensate by adopting measures aimed at providing early warnings of un-anticipated effects and mitigating their impacts. The point is made that it is often disregarded that the Bradford Hill criteria for evaluating evidence are asymmetrical, in that the applicability of a criterion increases the strength of evidence on the presence of an effect, but the inapplicability of a criterion does not increase the strength of evidence on the absence of an effect. The paper discusses the reason why there are so many examples of regulatory “false negatives” as opposed to “false positives”. Two main reasons are put forward: (i) the methodological bias within the health and environmental sciences; and (ii) the dominance within decision-making of short term economic and political interests. Sixteen features of methods and culture in the environmental and health sciences are presented. Of these, only three features tend to generate “false positives”. It is concluded that although the different features of scientific methods and culture produce robust science, they can lead to poor regulatory decisions on hazard prevention.

Non-Standard Parametric Statistical Inference

2017 ◽  
Author(s):  
Russell Cheng
Keyword(s):  
Statistical Inference ◽  
Statistical Methods ◽  
Model Building ◽  
Parametric Bootstrap ◽  
Basic Knowledge ◽  
Standard Theory ◽  
Regularity Conditions ◽  
Parameter Estimates ◽  
Bootstrap Resampling ◽  
Distributional Properties

This book discusses the fitting of parametric statistical models to data samples. Emphasis is placed on (i) how to recognize situations where the problem is non-standard, when parameter estimates behave unusually, and (ii) the use of parametric bootstrap resampling methods in analysing such problems. Simple and practical model building is an underlying theme. A frequentist viewpoint based on likelihood is adopted, for which there is a well-established and very practical theory. The standard situation is where certain widely applicable regularity conditions hold. However, there are many apparently innocuous situations where standard theory breaks down, sometimes spectacularly. Most of the departures from regularity are described geometrically in the book, with mathematical detail only sufficient to clarify the non-standard nature of a problem and to allow formulation of practical solutions. The book is intended for anyone with a basic knowledge of statistical methods typically covered in a university statistical inference course who wishes to understand or study how standard methodology might fail. Simple, easy-to-understand statistical methods are presented which overcome these difficulties, and illustrated by detailed examples drawn from real applications. Parametric bootstrap resampling is used throughout for analysing the properties of fitted models, illustrating its ease of implementation even in non-standard situations. Distributional properties are obtained numerically for estimators or statistics not previously considered in the literature because their theoretical distributional properties are too hard to obtain theoretically. Bootstrap results are presented mainly graphically in the book, providing easy-to-understand demonstration of the sampling behaviour of estimators.

scholarly journals Modeling the Influences of Social Mobility Net of Origin and Destination Based on the Front-Door Criterion: A Simulation Study

Methodology ◽  
2021 ◽  
Vol 17 (2) ◽  
pp. 111-126
Author(s):  
Anning Hu
Keyword(s):  
Social Mobility ◽  
Reference Model ◽  
Additive Model ◽  
Identification Problem ◽  
Social Scientists ◽  
Acyclic Graphs ◽  
Diamond Model ◽  
Independent Mechanism ◽  
Identification Approach ◽  
Intermediate Variables

The consequences of social mobility have been a persistent theme on the research agenda of social scientists, but the estimation of the net mobility effect controlling for both social origin and destination confronts with the identification problem. This research 1) highlights the mechanical identification approaches deployed by the conventional methods—the square additive model, the diamond model, and the diagonal reference model; 2) draws on the directional acyclic graphs to present an identification framework that is based on the intermediate variables; and 3) elaborates the specific identification strategies in typical research scenarios: independent mechanism, joint mechanism, partial mechanism, and intermediate confounded mechanism. The results of the Monte Carlo simulations suggest that the mechanism-based identification approach helps to obtain an unbiased estimate of the net mobility effect.

scholarly journals Evaluation of OPC-UA communication in an autonomous advanced manufacturing cell implementation

2020 ◽  
Vol 27 (4) ◽  
Author(s):  
Rodrigo Filev Maia ◽  
Ângelo Jorge Bálsamo ◽  
Guilherme Alberto Wachs Lopes ◽  
Alexandre Augusto Massote ◽  
Fábio Lima
Keyword(s):  
Control Systems ◽  
Reference Model ◽  
Global Markets ◽  
Advanced Manufacturing ◽  
Data Transport ◽  
Manufacturing Cell ◽  
Open Platform ◽  
Communication Framework ◽  
Local Companies ◽  
Global And Local

Abstract: The recent development of advanced manufacturing concepts brings new challenges for developing countries in order to prepare their workforce and industries for local and global markets. To explore and disseminate advanced manufacturing concepts, the first autonomous advanced manufacturing cell developed in Brazil as a result of a partnership between global and local companies and academia was designed and evaluated. This paper aims to describe the advanced manufacturing cell, the equipment composing the infrastructure, and the network topology and equipment that could deal with the heterogeneity of equipment and protocol. Moreover, the behavior of the Open Platform Communications Unified Architecture (OPC UA) communication framework (and related protocols) as the integration element between the equipment composing the cell was evaluated. The results indicated that the OPC UA communication framework promotes a low traffic overhead and can also be used to carry data from the traditional protocols in their fields without restrictions that could hinder production. The paper also discusses how the exchange of data occurred between the equipment used in the production and the control systems via the protocol studied. The OPC UA communication framework protocols presented a messaging structure and data transport characteristics that satisfy the integration needs between equipment of several manufacturers that composed an autonomous cell of advanced manufacturing. This paper also presents some key challenges to integrating all equipment not addressed by the OPC UA reference model.

scholarly journals A common dynamic prior for time in duration discrimination

2021 ◽  
Author(s):  
Joost de Jong ◽  
Elkan G. Akyürek ◽  
Hedderik van Rijn
Keyword(s):  
Context Effects ◽  
Reference Model ◽  
Time Estimation ◽  
Duration Discrimination ◽  
Internal Reference ◽  
Standard Duration ◽  
Global And Local ◽  
Estimation Of Time ◽  
Relative Weighting ◽  
Perceived Duration

AbstractEstimation of time depends heavily on both global and local statistical context. Durations that are short relative to the global distribution are systematically overestimated; durations that are locally preceded by long durations are also overestimated. Context effects are prominent in duration discrimination tasks, where a standard duration and a comparison duration are presented on each trial. In this study, we compare and test two models that posit a dynamically updating internal reference that biases time estimation on global and local scales in duration discrimination tasks. The internal reference model suggests that the internal reference operates during postperceptual stages and only interacts with the first presented duration. In contrast, a Bayesian account of time estimation implies that any perceived duration updates the internal reference and therefore interacts with both the first and second presented duration. We implemented both models and tested their predictions in a duration discrimination task where the standard duration varied from trial to trial. Our results are in line with a Bayesian perspective on time estimation. First, the standard systematically biased estimation of the comparison, such that shorter standards increased the likelihood of reporting that the comparison was shorter. Second, both the previous standard and comparison systematically biased time estimation of subsequent trials in the same direction. Third, more precise observers showed smaller biases. In sum, our findings suggest a common dynamic prior for time that is updated by each perceived duration and where the relative weighting of old and new observations is determined by their relative precision.

scholarly journals The ethical-political dimension of social and community praxis: The case of Ecuador´s early response to covid-19

2021 ◽  
Author(s):  
Manuel Capella
Keyword(s):  
Social Impact ◽  
Local Community ◽  
Scientific Evidence ◽  
Early Response ◽  
Democratic Legitimacy ◽  
Political Dimension ◽  
Community Resistance ◽  
The Government ◽  
Global And Local

In Ecuador, the painful impact of the covid-19 pandemic elicited early responses by the government, and by local communities. This critical, positioned and exploratory case study analyses such responses, underscoring the fundamental ethical-political dimension of any academic and professional praxis aimed at the construction of healthier societies worldwide. While critical traditions are familiar with this stance, the inequalities and ideological mechanisms made visible by covid-19 responses may enable the wider community of researchers and practitioners to join ongoing collective ethical-political efforts. Findings from Ecuador underline the potentially harmful role of neoliberalism, and issues of democratic legitimacy; significant problems before and during the pandemic shock; and official discourses which blame communities for their own suffering and death. Neutrality and depoliticized notions of scientific evidence are notoriously insufficient in these scenarios. We need to engage more deeply with diverse forms of global and local community resistance, in times of covid-19, and beyond. Please refer to the Supplementary Material section to find this article’s Community and Social Impact Statement

scholarly journals Assessing the Lognormal Distribution Assumption For the Crude Odds Ratio: Implications For Point and Interval Estimation

2020 ◽  
Author(s):  
David Douglas Newstein
Keyword(s):  
Confidence Intervals ◽  
Standard Method ◽  
Lognormal Distribution ◽  
Odds Ratio ◽  
Estimation Method ◽  
Interval Estimation ◽  
Parametric Bootstrap ◽  
Sampling Distribution ◽  
Estimation Methods ◽  
Crude Odds Ratio

Abstract Background: The assumption that the sampling distribution of the crude Odds Ratio (ORcrude) is a lognormal distribution with parameters mu and sigma leads to the incorrect conclusion that the expectation of the log of ORcrude is equal to the parameter mu. Here, the standard method of point and interval estimation (I) is compared with a modified method utilizing ORstar where ln(ORstar) = ln(ORcrude )– sigma **2/2. Methods: Confidence intervals are obtained utilizing ln(ORstar) by both parametric bootstrap simulations with a percentile derived confidence interval (II), and a simple calculation done by replacing ln(ORcrude) with ln(ORstar) in the standard formula (III) as well as a method proposed by Barendregt (IV), who also noted the bias present in estimating ORtrue by ORcrude. Simulations are conducted for a “protective” exposure (ORtrue < 1) as well as for a “harmful” exposure (ORtrue >1). Results: In simulations the estimation methods (II and III) exhibited the highest level of statistical conclusion validity for their confidence intervals as indicated by one minus the coverage probability being close to alpha. Also, as demonstrated by the MC simulations, these two methods exhibited the least biased point estimates and the narrowest confidence intervals of the four estimation approaches. Conclusions: Monte Carlo simulations prove useful in validating the inferential procedures used in data analysis. In the case of the odds ratio, the standard method of point and interval estimation is based on the assumption that the crude odds ratio has a sampling distribution that is lognormal. Utilizing this assumption, as well as the formula for the expectation of this distribution function, an alternative estimation method was obtained for ORtrue (but different from a method from the earlier report (Barendregt)), that yielded point and interval estimates that MC simulations indicate are the most statistically valid.

Sign in / Sign up

Export Citation Format

Share Document