Efficient Monte Carlo Estimation of the Expected Value of Sample Information Using Moment Matching

Background. The expected value of sample information (EVSI) determines the economic value of any future study with a specific design aimed at reducing uncertainty about the parameters underlying a health economic model. This has potential as a tool for trial design; the cost and value of different designs could be compared to find the trial with the greatest net benefit. However, despite recent developments, EVSI analysis can be slow, especially when optimizing over a large number of different designs. Methods. This article develops a method to reduce the computation time required to calculate the EVSI across different sample sizes. Our method extends the moment-matching approach to EVSI estimation to optimize over different sample sizes for the underlying trial while retaining a similar computational cost to a single EVSI estimate. This extension calculates the posterior variance of the net monetary benefit across alternative sample sizes and then uses Bayesian nonlinear regression to estimate the EVSI across these sample sizes. Results. A health economic model developed to assess the cost-effectiveness of interventions for chronic pain demonstrates that this EVSI calculation method is fast and accurate for realistic models. This example also highlights how different trial designs can be compared using the EVSI. Conclusion. The proposed estimation method is fast and accurate when calculating the EVSI across different sample sizes. This will allow researchers to realize the potential of using the EVSI to determine an economically optimal trial design for reducing uncertainty in health economic models. Limitations. Our method involves rerunning the health economic model, which can be more computationally expensive than some recent alternatives, especially in complex models.

Download Full-text

Estimating Expected Value of Sample Information for Incomplete Data Models Using Bayesian Approximation

Medical Decision Making ◽

10.1177/0272989x11399920 ◽

2011 ◽

Vol 31 (6) ◽

pp. 839-852 ◽

Cited By ~ 11

Author(s):

Samer A. Kharroubi ◽

Alan Brennan ◽

Mark Strong

Keyword(s):

Monte Carlo ◽

Incomplete Data ◽

Bayesian Updating ◽

Expected Value ◽

Data Models ◽

Mcmc Methods ◽

Approximation Formula ◽

Posterior Density ◽

Sample Information ◽

Bayesian Approximation

Expected value of sample information (EVSI) involves simulating data collection, Bayesian updating, and reexamining decisions. Bayesian updating in incomplete data models typically requires Markov chain Monte Carlo (MCMC). This article describes a revision to a form of Bayesian Laplace approximation for EVSI computation to support decisions in incomplete data models. The authors develop the approximation, setting out the mathematics for the likelihood and log posterior density function, which are necessary for the method. They compare the accuracy of EVSI estimates in a case study cost-effectiveness model using first- and second-order versions of their approximation formula and traditional Monte Carlo. Computational efficiency gains depend on the complexity of the net benefit functions, the number of inner-level Monte Carlo samples used, and the requirement or otherwise for MCMC methods to produce the posterior distributions. This methodology provides a new and valuable approach for EVSI computation in health economic decision models and potential wider benefits in many fields requiring Bayesian approximation.

Download Full-text

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

Medical Decision Making ◽

10.1177/0272989x211073098 ◽

2022 ◽

pp. 0272989X2110730

Author(s):

Anna Heath

Keyword(s):

Computational Cost ◽

Expected Value ◽

Efficient Estimation ◽

Estimation Methods ◽

Computation Method ◽

Moment Matching ◽

Strength Of Evidence ◽

Treatment Implementation ◽

Sample Information ◽

The Moment

Background The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge. Methods Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples. Results The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice. Conclusions This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis. Highlights Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study. Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment. The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI. Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Download Full-text

Calculating the Expected Value of Sample Information in Practice: Considerations from 3 Case Studies

Medical Decision Making ◽

10.1177/0272989x20912402 ◽

2020 ◽

Vol 40 (3) ◽

pp. 314-326 ◽

Cited By ~ 1

Author(s):

Anna Heath ◽

Natalia Kunst ◽

Christopher Jackson ◽

Mark Strong ◽

Fernando Alarid-Escudero ◽

...

Keyword(s):

Health Economic ◽

Monte Carlo ◽

Sample Size ◽

Economic Model ◽

Expected Value ◽

Monte Carlo Algorithm ◽

Approximation Methods ◽

Sample Sizes ◽

Sample Information ◽

Health Economic Model

Background. Investing efficiently in future research to improve policy decisions is an important goal. Expected value of sample information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Recently, several more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared, and therefore their comparative performance across different examples has not been explored. Methods. We compared 4 EVSI methods using 3 previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared. Results. In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. Conclusions. As all the evaluated methods gave estimates similar to those given by traditional Monte Carlo, we suggest that EVSI can now be efficiently computed with confidence in realistic examples. No systematically superior EVSI computation method exists as the properties of the different methods depend on the underlying health economic model, data generation process, and user expertise.

Download Full-text