Calculating the Expected Value of Sample Information in Practice: Considerations from 3 Case Studies

Background. Investing efficiently in future research to improve policy decisions is an important goal. Expected value of sample information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Recently, several more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared, and therefore their comparative performance across different examples has not been explored. Methods. We compared 4 EVSI methods using 3 previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared. Results. In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. Conclusions. As all the evaluated methods gave estimates similar to those given by traditional Monte Carlo, we suggest that EVSI can now be efficiently computed with confidence in realistic examples. No systematically superior EVSI computation method exists as the properties of the different methods depend on the underlying health economic model, data generation process, and user expertise.

Download Full-text

Estimating the Expected Value of Sample Information across Different Sample Sizes Using Moment Matching and Nonlinear Regression

Medical Decision Making ◽

10.1177/0272989x19837983 ◽

2019 ◽

Vol 39 (4) ◽

pp. 347-359 ◽

Cited By ~ 2

Author(s):

Anna Heath ◽

Ioanna Manolopoulou ◽

Gianluca Baio

Keyword(s):

Health Economic ◽

Nonlinear Regression ◽

Economic Model ◽

Trial Design ◽

Expected Value ◽

Moment Matching ◽

Sample Sizes ◽

Sample Information ◽

Health Economic Model ◽

The Cost

Background. The expected value of sample information (EVSI) determines the economic value of any future study with a specific design aimed at reducing uncertainty about the parameters underlying a health economic model. This has potential as a tool for trial design; the cost and value of different designs could be compared to find the trial with the greatest net benefit. However, despite recent developments, EVSI analysis can be slow, especially when optimizing over a large number of different designs. Methods. This article develops a method to reduce the computation time required to calculate the EVSI across different sample sizes. Our method extends the moment-matching approach to EVSI estimation to optimize over different sample sizes for the underlying trial while retaining a similar computational cost to a single EVSI estimate. This extension calculates the posterior variance of the net monetary benefit across alternative sample sizes and then uses Bayesian nonlinear regression to estimate the EVSI across these sample sizes. Results. A health economic model developed to assess the cost-effectiveness of interventions for chronic pain demonstrates that this EVSI calculation method is fast and accurate for realistic models. This example also highlights how different trial designs can be compared using the EVSI. Conclusion. The proposed estimation method is fast and accurate when calculating the EVSI across different sample sizes. This will allow researchers to realize the potential of using the EVSI to determine an economically optimal trial design for reducing uncertainty in health economic models. Limitations. Our method involves rerunning the health economic model, which can be more computationally expensive than some recent alternatives, especially in complex models.

Download Full-text

A General Monte Carlo Method for Sample Size Analysis in the Context of Network Models

10.31234/osf.io/j5v7u ◽

2021 ◽

Author(s):

Mihai Alexandru Constantin ◽

Noémi Katalin Schuurman ◽

Jeroen Vermunt

Keyword(s):

Monte Carlo ◽

Sample Size ◽

Graphical Model ◽

Network Models ◽

Performance Measure ◽

Monte Carlo Algorithm ◽

Size Analysis ◽

Sample Sizes ◽

Cross Sectional ◽

Target Value

We introduce a general method for sample size computations in the context of cross-sectional network models. The method takes the form of an automated Monte Carlo algorithm, designed to find an optimal sample size while iteratively concentrating the computations on the sample sizes that seem most relevant. The method requires three inputs: 1) a hypothesized network structure or desired characteristics of that structure, 2) an estimation performance measure and its corresponding target value (e.g., a sensitivity of 0.6), and 3) a statistic and its corresponding target value that determine how the target value for the performance measure be reached (e.g., reaching a sensitivity of 0.6 with a probability of 0.8). The method consists of a Monte Carlo simulation step for computing the performance measure and the statistic for several sample sizes selected from an initial candidate sample size range, a curve-fitting step for interpolating the statistic across the entire candidate range, and a stratified bootstrapping step to quantify the uncertainty around the recommendation provided. We evaluated the performance of the method for the Gaussian Graphical Model, but it can easily extend to other models. It displayed good performance, with the sample size recommendations provided being, on average, at most 1.14 sample sizes away from the truth, with a highest standard deviation of 26.25 sample sizes. The method is implemented in the form of an R package called powerly, available on GitHub and CRAN.

Download Full-text

Developing a Health Economic Model for Asians with Type 2 Diabetes Based on the Japan Diabetes Complications Study and the Japanese Elderly Diabetes Intervention Trial

Diabetes ◽

10.2337/db18-2319-pub ◽

2018 ◽

Vol 67 (Supplement 1) ◽

pp. 2319-PUB

Author(s):

SHIRO TANAKA ◽

JAKOB LANGER ◽

TIM MORTON ◽

LARS WILKINSON ◽

SACHIKO TANAKA ◽

...

Keyword(s):

Type 2 Diabetes ◽

Health Economic ◽

Economic Model ◽

Diabetes Complications ◽

Intervention Trial ◽

Health Economic Model ◽

Japanese Elderly ◽

Diabetes Intervention

Download Full-text

A health economic model to estimate the costs and benefits of an mRNA vs DNA high-risk HPV assay in a theoretical HPV primary screening algorithm in Ontario, Canada

Preventive Medicine Reports ◽

10.1016/j.pmedr.2021.101448 ◽

2021 ◽

pp. 101448

Author(s):

Georgie Weston ◽

Caroline Dombrowski ◽

Marc Steben ◽

Catherine Popadiuk ◽

James Bentley ◽

...

Keyword(s):

Health Economic ◽

High Risk ◽

Economic Model ◽

Costs And Benefits ◽

Primary Screening ◽

Screening Algorithm ◽

Health Economic Model ◽

High Risk Hpv

Download Full-text

Clinical and Economic Outcomes of Memantine Used in Moderate or Severe Dementia Patients in China: Results from a Health Economic Model

Value in Health ◽

10.1016/j.jval.2014.08.028 ◽

2014 ◽

Vol 17 (7) ◽

pp. A722

Author(s):

L Ge ◽

E Clay ◽

J Yan ◽

M Toumi ◽

D Milea

Keyword(s):

Health Economic ◽

Economic Model ◽

Economic Outcomes ◽

Severe Dementia ◽

Dementia Patients ◽

Health Economic Model

Download Full-text

Statistical Power in Content Analysis Designs: How Effect Size, Sample Size and Coding Accuracy Jointly Affect Hypothesis Testing ‐ A Monte Carlo Simulation Approach.

Computational Communication Research ◽

10.5117/ccr2021.1.003.geis ◽

2021 ◽

Vol 3 (1) ◽

pp. 61-89

Author(s):

Stefan Geiß

Keyword(s):

Monte Carlo Simulation ◽

Monte Carlo ◽

Content Analysis ◽

Sample Size ◽

Effect Size ◽

Statistical Power ◽

Effect Sizes ◽

Sample Sizes ◽

Expected Effect ◽

Sample Size Effect

Abstract This study uses Monte Carlo simulation techniques to estimate the minimum required levels of intercoder reliability in content analysis data for testing correlational hypotheses, depending on sample size, effect size and coder behavior under uncertainty. The ensuing procedure is analogous to power calculations for experimental designs. In most widespread sample size/effect size settings, the rule-of-thumb that chance-adjusted agreement should be ≥.80 or ≥.667 corresponds to the simulation results, resulting in acceptable α and β error rates. However, this simulation allows making precise power calculations that can consider the specifics of each study’s context, moving beyond one-size-fits-all recommendations. Studies with low sample sizes and/or low expected effect sizes may need coder agreement above .800 to test a hypothesis with sufficient statistical power. In studies with high sample sizes and/or high expected effect sizes, coder agreement below .667 may suffice. Such calculations can help in both evaluating and in designing studies. Particularly in pre-registered research, higher sample sizes may be used to compensate for low expected effect sizes and/or borderline coding reliability (e.g. when constructs are hard to measure). I supply equations, easy-to-use tables and R functions to facilitate use of this framework, along with example code as online appendix.

Download Full-text