scholarly journals How to embrace variation and accept uncertainty in linguistic and psycholinguistic data analysis

Linguistics ◽  
2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Shravan Vasishth ◽  
Andrew Gelman
Keyword(s):  
Data Analysis ◽  
Statistical Power ◽  
Statistical Significance ◽  
Statistical Significance Testing ◽  
Binary Decision ◽  
Alternative Approaches ◽  
Almost All ◽  
Quantifying Uncertainty ◽  
Data Analytic ◽  
Size Estimates

Abstract The use of statistical inference in linguistics and related areas like psychology typically involves a binary decision: either reject or accept some null hypothesis using statistical significance testing. When statistical power is low, this frequentist data-analytic approach breaks down: null results are uninformative, and effect size estimates associated with significant results are overestimated. Using an example from psycholinguistics, several alternative approaches are demonstrated for reporting inconsistencies between the data and a theoretical prediction. The key here is to focus on committing to a falsifiable prediction, on quantifying uncertainty statistically, and learning to accept the fact that – in almost all practical data analysis situations – we can only draw uncertain conclusions from data, regardless of whether we manage to obtain statistical significance or not. A focus on uncertainty quantification is likely to lead to fewer excessively bold claims that, on closer investigation, may turn out to be not supported by the data.

scholarly journals How to embrace variation and accept uncertainty in linguistic and psycholinguistic data analysis

2019 ◽  
Author(s):  
Shravan Vasishth ◽  
andrew gelman
Keyword(s):  
Data Analysis ◽  
Statistical Power ◽  
Statistical Significance ◽  
Statistical Significance Testing ◽  
Binary Decision ◽  
Null Results ◽  
Alternative Approaches ◽  
Almost All ◽  
Quantifying Uncertainty ◽  
Data Analytic

The use of statistical inference in linguistics and related areas like psychology typically involves a binary decision: either reject or accept some null hypothesis using statistical significance testing. When statistical power is low, this frequentist data-analytic approach breaks down: null results are uninformative, and significant results are driven by Type M error. Using an example from psycholinguistics, several alternative approaches are demonstrated for reporting inconsistencies between the data and a theoretical prediction. The key here is to focus on committing to a falsifiable prediction, on quantifying uncertainty statistically, and to learn to accept the fact that---in almost all practical data analysis situations---we can only draw uncertain conclusions from data, regardless of whether we manage to obtain statistical significance or not. A focus on uncertainty quantification is likely to lead to fewer excessively bold claims that, on closer investigation, may turn out to be not supported by the data.

Abstract 162: The Challenges of Decision-making Regarding Futility in a Randomized Trial: The IMS III Experience

Stroke ◽  
2013 ◽  
Vol 44 (suppl_1) ◽  
Author(s):  
Sharon D Yeatts ◽  
Renee’ H Martin ◽  
Lydia D Foster ◽  
Robert F Woolson ◽  
Joseph P Broderick ◽  
...  
Keyword(s):  
Decision Making ◽  
Ischemic Stroke ◽  
Data Analysis ◽  
Interim Analysis ◽  
Statistical Power ◽  
Statistical Significance ◽  
Randomized Study ◽  
Conditional Power ◽  
Utmost Care ◽  
Iv Tpa

INTRODUCTION: IMS 3 Trial is a multicenter randomized study to compare the effectiveness of the bridging (IV tPA + intra-arterial) therapy to the IV tPA only treatment in patients with acute ischemic stroke presenting within 3 hours of symptom onset. The trial intended to enroll 900 subjects to ensure adequate statistical power to detect an absolute 10% difference in the % of subjects with good outcome, defined as mRS score of 0-2 at 3 months. In April 2012, after 656 subjects were randomized, further enrollment was terminated by the NINDS upon recommendation by the DSMB. The decision was based on the pre-specified criterion for futility using conditional power of <20%. We describe the process in which the IMS 3 Trial was deemed futile by the DSMB. METHOD: Conditional power was defined as the likelihood of finding statistical significance at the end of the study given the accumulated data to date and with the assumption that a minimum hypothesized difference of 10% truly exists between the two groups. Hence, it is akin to ordinary statistical power. RESULTS: The futility boundary was crossed at the Trial’s 4 th interim analysis. The details of the evolution of the study that led to the futility determination are presented, including the interaction between the unblinded study statisticians and the DSMB in the complex deliberation of the data analysis results. CONCLUSION: Even in spite of pre-specifying the parameters of interim analysis, interim looks at the data pose challenges in the interpretation and decision-making. It therefore underscores the importance of taking utmost care in the design of the study, including specification of objective stopping guidelines.

scholarly journals Cumulative areawise testing in wavelet analysis and its application to geophysical time series

2016 ◽  
Vol 23 (1) ◽  
pp. 45-57 ◽  
Author(s):  
Justin A. Schulte
Keyword(s):  
Wavelet Analysis ◽  
Statistical Power ◽  
Signal To Noise Ratio ◽  
Statistical Significance ◽  
Scientific Investigation ◽  
Statistical Significance Testing ◽  
Significance Level ◽  
Significance Levels ◽  
Geophysical Time Series ◽  
Selection Of

Abstract. Statistical significance testing in wavelet analysis was improved through the development of a cumulative areawise test. The test was developed to eliminate the selection of two significance levels that an existing geometric test requires for implementation. The selection of two significance levels was found to make the test sensitive to the chosen pointwise significance level, which may preclude further scientific investigation. A set of experiments determined that the cumulative areawise test has greater statistical power than the geometric test in most cases, especially when the signal-to-noise ratio is high. The number of false positives identified by the tests was found to be similar if the respective significance levels were set to 0.05.

Statistical Methods in Education and Psychology: A Critical Perspective

1996 ◽  
Vol 40 (2) ◽  
pp. 190-209 ◽  
Author(s):  
Brian D. Haig
Keyword(s):  
Statistical Methods ◽  
Statistical Significance ◽  
Psychological Research ◽  
Critical Perspective ◽  
Statistical Significance Testing ◽  
Resampling Methods ◽  
Exploratory Data ◽  
Modelling Methods ◽  
Data Analytic ◽  
Realist Conception

This paper offers a sympathetic, but critical, perspective on selected statistical methods relevant to both educational and psychological research. I argue that the classical statistical procedures used in these fields should, where appropriate, be deployed in the service of a more liberal realist conception of research. In this regard, it is claimed that the principal function of statistical methods is to help us detect robust empirical phenomena. With this in mind, I suggest that exploratory factor analysis is a quasi-statistical method that serves an important role in the generation of new theory. A sceptical attitude to the commonly used methods of statistical significance testing is encouraged, together with the suggestion that Bayesian methods can serve a legitimate role in scientific inference. In addition, the regular use of exploratory data analytic methods is urged in conjunction with computer intensive resampling methods. Critical, but constructive, suggestions are made about the recently developed meta-analytic and causal modelling methods.

scholarly journals Graphical Data Analysis

2021 ◽  
Vol 2020 (1) ◽  
pp. 1
Author(s):  
Paul Collett
Keyword(s):  
Data Analysis ◽  
Quantitative Data ◽  
Quantitative Research ◽  
Statistical Significance ◽  
Academic Disciplines ◽  
Ongoing Debate ◽  
Statistical Significance Testing ◽  
Graphical Data ◽  
Quantitative Data Analysis

Limitations in statistical significance testing are an issue of ongoing debate in many academic disciplines. Moving away from a reliance on their use in quantitative research is seen as an important step towards improving the quality of such research. One area that can help here is graphical data analysis, both as an exploratory and explanatory tool. This paper presents an overview of graphical techniques for quantitative data analysis. After outlining the rationale for the use of graphical data analysis, consideration of the appropriate types of graphs to use is provided. A number of useful graphs, created using the R statistical package, are introduced, along with a link to the full code to reproduce the examples. Suggestions are presented for how graphical techniques can help with both the exploration and conformation stages in research. 統計的有意差検定の限界は、多くの学術分野で継続的に議論されている問題である。量的研究における統計的有意差検定への使用依存から脱却することは、研究の質を向上させるための重要なステップであると考えられる。ここで役立つのが、探索的・説明的なツールとしてのグラフデータ分析である。本稿は、定量的なデータ分析のためのグラフ技術の概要を説明する。グラフィカルなデータ分析を使用する理由を説明した後、使用するグラフの適切な種類について検討する。R統計パッケージを用いて作成された便利なグラフの数々を紹介し、例題を再現するための完全なコードへのリンクも掲載する。本稿は、研究における探索と適合の両方の段階で、グラフ技術がどのように役立つかを提案するものである。

scholarly journals Truth and transparency in expertise research

2019 ◽  
Author(s):  
Rachel A Searston ◽  
Matthew B Thompson ◽  
Samuel Gebhard Robson ◽  
Brooklyn Corbett ◽  
Gianni Ribeiro ◽  
...  
Keyword(s):  
Data Analysis ◽  
Statistical Power ◽  
Degrees Of Freedom ◽  
Open Data ◽  
Reform Movement ◽  
Open Communication ◽  
The Public ◽  
Research Areas ◽  
Data Analytic ◽  
Open Materials

Across research areas, general issues of low statistical power, publication bias, undisclosed flexibility in data analysis, and researcher degrees of freedom, can be recipes for irreproducibility. To address the problem, a reform movement known as the ‘credibility revolution’ emphasises the need for greater transparency in how research is conducted. In this article, we describe a general approach to creating a culture of openness—tailored for expertise researchers—and describe how and why practices such as ‘preregistration,’ ‘open notebooks,’ ‘open data,’ ‘open materials,’ and ‘open communication,’ might be applied to research on experts. We argue that adopting these practices helps to connect end-users with the entire research lifecycle, and helps to reconnect researchers with the process of gaining knowledge. By sharing notes about our predictions and plans along the way, we are forced to confront their merits. By documenting design and data analytic decisions ahead of time, and by sharing data and materials, we make errors and insights more discoverable. And by inviting research partners, expert practitioners, and the public into the lab, we stand the best chance of successfully translating research into practice.

scholarly journals Cumulative areawise testing in wavelet analysis and its application to geophysical time series

2015 ◽  
Vol 2 (4) ◽  
pp. 1227-1273 ◽  
Author(s):  
J. A. Schulte
Keyword(s):  
Time Series ◽  
Wavelet Analysis ◽  
Statistical Power ◽  
Signal To Noise Ratio ◽  
Statistical Significance ◽  
Testing Procedure ◽  
Statistical Significance Testing ◽  
Significance Level ◽  
Significance Levels ◽  
Selection Of

Abstract. Statistical significance testing in wavelet analysis was improved through the development of a cumulative areawise test. The test was developed to eliminate the selection of two significance levels that an existing geometric test requires for implementation. The selection of two significance levels was found to make the test sensitive to the chosen pointwise significance level, which may preclude further scientific investigation. A set of experiments determined that the cumulative areawise test has greater statistical power than the geometric test in most cases, especially when the signal-to-noise ratio is high. The number of false positives identified by the tests was found to be similar if the respective significance levels were set to 0.05. The new testing procedure was applied to the time series of the Atlantic Multi-decadal Oscillation (AMO), North Atlantic Oscillation (NAO), Pacific Decadal Oscillation (PDO), and Niño 3.4 index. The testing procedure determined that the NAO, PDO, and AMO are consistent with red-noise processes, whereas significant power was found in the 2–7 year period band for the Niño 3.4 index.

Relationship of Workload and Double Role With Work Fatigue on Batik Craftsmen

2019 ◽  
Vol 4 (2) ◽  
pp. 293
Author(s):  
Herdianti Herdianti ◽  
Tatik Maryana
Keyword(s):  
Data Analysis ◽  
Quantitative Research ◽  
Sampling Technique ◽  
Work Load ◽  
Dual Role ◽  
P Value ◽  
Chi Square ◽  
Cross Sectional ◽  
Chi Square Test ◽  
Almost All

<p><em><em>Background: In Batik Mawar, almost all work is done manually using the hands and upper arms on a continuous basis combined with the rigor of work and the use of traditional tools. The work has a heavy workload because all the work process is done by the same craftsman causing fatigue besides that the worker also have double role. The purpose of this study is to determine the relationship between workload and dual role with feelings of fatigue on craftsmen batik roses.Method: This research is Quantitative research with Cross Sectional research design. The population in this study are all artisans in Batik Mawar. Sampling in this study using total sampling technique with the number of research samples as many as 40 respondents. Data analysis used by Univariat and Bivariat.Result: Result of data analysis using Chi-Square test for work load got value p-Value = 0,001. The result of data analysis using Chi-square test for double role got p-value = 0,031. Thus it is concluded that there is a meaningful relationship between workload and dual role with feeling tired. We recommend that craftsmen wash clothes 2 times a day, cook ready meals, other than together in completing the work at home</em></em></p><p><em><br /></em></p><p><em>Di Batik Mawar, hampir semua pekerjaan dikerjakan secara manual menggunakan tangan dan lengan atas secara berkesinambungan yang dikombinasi dengan ketelitian kerja dan penggunaan alat-alat tradisional. Pekerjaan mempunyai beban kerja yang berat dikarenakan semua proses kerja dilakukan oleh pengrajin yang sama sehingga menimbulkan kelelahan</em><em> disamping itu pekerjanya juga memiliki peran ganda</em><em>.</em><em> Tujuan penelitian ini adalah untuk mengetahui hubungan beban kerja dan peran ganda dengan perasaan lelah pada pengrajin batik mawar.Metode: </em><em>Penelitian ini merupakan penelitian Kuantitatif dengan desain penelitian Cross Sectional. Populasi dalam penelitian ini adalah semua pengrajin di Batik Mawar. Pengambilan sampel pada penelitian ini menggunakan teknik total sampling dengan jumlah sampel penelitian sebanyak 40 responden. Analisis data yang digunakan Univariat dan Bivariat.</em><em>Hasil: </em><em>Hasil analisis data yang menggunakan uji Chi-Square untuk beban kerja didapatkan nilai p-Value = 0,001. Hasil analisis data yang menggunakan uji Chi-square untuk peran ganda didapatkan nilai p-value= 0,031. Dengan demikian  disimpulkan bahwa ada hubungan yang bermakna antara beban kerja dan peran ganda dengan perasaan lelah.Sebaiknya pengrajin mencuci pakaian 2 kali sehari, memasak makanan siap saji, selain itu dengan cara bersama-sama dalam menyelesaikan pekerjaan dirumah.</em><em></em></p><strong><em></em></strong>

scholarly journals Bayesian modeling and significant features exploration in wavelet power spectra

2007 ◽  
Vol 14 (1) ◽  
pp. 79-88 ◽  
Author(s):  
D. V. Divine ◽  
F. Godtliebsen
Keyword(s):  
Data Analysis ◽  
Traditional Approach ◽  
Statistical Significance ◽  
Power Spectra ◽  
Simulated Data ◽  
Scale Space ◽  
Independent Random Variables ◽  
Discrete Wavelet ◽  
Wavelet Coefficients ◽  
Testing Procedures

Abstract. This study proposes and justifies a Bayesian approach to modeling wavelet coefficients and finding statistically significant features in wavelet power spectra. The approach utilizes ideas elaborated in scale-space smoothing methods and wavelet data analysis. We treat each scale of the discrete wavelet decomposition as a sequence of independent random variables and then apply Bayes' rule for constructing the posterior distribution of the smoothed wavelet coefficients. Samples drawn from the posterior are subsequently used for finding the estimate of the true wavelet spectrum at each scale. The method offers two different significance testing procedures for wavelet spectra. A traditional approach assesses the statistical significance against a red noise background. The second procedure tests for homoscedasticity of the wavelet power assessing whether the spectrum derivative significantly differs from zero at each particular point of the spectrum. Case studies with simulated data and climatic time-series prove the method to be a potentially useful tool in data analysis.

Sign in / Sign up

Export Citation Format

Share Document