scholarly journals Optimal switching problems under partial information

2015 ◽  
Vol 21 (2) ◽  
Author(s):  
Kai Li ◽  
Kaj Nyström ◽  
Marcus Olofsson
Keyword(s):  
Monte Carlo ◽  
Dynamic Programming ◽  
Value Function ◽  
Partial Information ◽  
Convergence Result ◽  
Optimal Switching ◽  
Filtering Theory ◽  
Stochastic Filtering ◽  
Observation Process ◽  
The Value Function

AbstractIn this paper, we formulate and study an optimal switching problem under partial information. In our model, the agent/manager/investor attempts to maximize the expected reward by switching between different states/investments. However, he is not fully aware of his environment and only an observation process, which contains partial information about the environment/underlying, is accessible. It is based on the partial information carried by this observation process that all decisions must be made. We propose a probabilistic numerical algorithm, based on dynamic programming, regression Monte Carlo methods, and stochastic filtering theory, to compute the value function. In this paper, the approximation of the value function and the corresponding convergence result are obtained when the underlying and observation processes satisfy the linear Kalman–Bucy setting. A numerical example is included to show some specific features of partial information.

Adaptive Learning of Drug Quality and Optimization of Patient Recruitment for Clinical Trials with Dropouts

2021 ◽  
Author(s):  
Zhili Tian ◽  
Weidong Han ◽  
Warren B. Powell
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Dynamic Programming ◽  
Value Function ◽  
Standard Treatment ◽  
Treatment Effectiveness ◽  
Interim Analyses ◽  
Clinical Trial Program ◽  
Patient Enrollment ◽  
The Value Function

Problem definition: Clinical trials are crucial to new drug development. This study investigates optimal patient enrollment in clinical trials with interim analyses, which are analyses of treatment responses from patients at intermediate points. Our model considers uncertainties in patient enrollment and drug treatment effectiveness. We consider the benefits of completing a trial early and the cost of accelerating a trial by maximizing the net present value of drug cumulative profit. Academic/practical relevance: Clinical trials frequently account for the largest cost in drug development, and patient enrollment is an important problem in trial management. Our study develops a dynamic program, accurately capturing the dynamics of the problem, to optimize patient enrollment while learning the treatment effectiveness of an investigated drug. Methodology: The model explicitly captures both the physical state (enrolled patients) and belief states about the effectiveness of the investigated drug and a standard treatment drug. Using Bayesian updates and dynamic programming, we establish monotonicity of the value function in state variables and characterize an optimal enrollment policy. We also introduce, for the first time, the use of backward approximate dynamic programming (ADP) for this problem class. We illustrate the findings using a clinical trial program from a leading firm. Our study performs sensitivity analyses of the input parameters on the optimal enrollment policy. Results: The value function is monotonic in cumulative patient enrollment and the average responses of treatment for the investigated drug and standard treatment drug. The optimal enrollment policy is nondecreasing in the average response from patients using the investigated drug and is nonincreasing in cumulative patient enrollment in periods between two successive interim analyses. The forward ADP algorithm (or backward ADP algorithm) exploiting the monotonicity of the value function reduced the run time from 1.5 months using the exact method to a day (or 20 minutes) within 4% of the exact method. Through an application to a leading firm’s clinical trial program, the study demonstrates that the firm can have a sizable gain of drug profit following the optimal policy that our model provides. Managerial implications: We developed a new model for improving the management of clinical trials. Our study provides insights of an optimal policy and insights into the sensitivity of value function to the dropout rate and prior probability distribution. A firm can have a sizable gain in the drug’s profit by managing its trials using the optimal policies and the properties of value function. We illustrated that firms can use the ADP algorithms to develop their patient enrollment strategies.

A uniform convergence theorem for the numerical solving of the nonlinear filtering problem

1998 ◽  
Vol 35 (04) ◽  
pp. 873-884 ◽  
Author(s):  
P. Del Moral
Keyword(s):  
Monte Carlo ◽  
Uniform Convergence ◽  
Convergence Theorem ◽  
Nonlinear Filtering ◽  
Partial Information ◽  
Particle System ◽  
Convergence Result ◽  
Recursive Solution ◽  
Ergodic Condition ◽  
Filtering Problem

The filtering problem concerns the estimation of a stochastic process X from its noisy partial information Y. With the notable exception of the linear-Gaussian situation, general optimal filters have no finitely recursive solution. The aim of this work is the design of a Monte Carlo particle system approach to solve discrete time and nonlinear filtering problems. The main result is a uniform convergence theorem. We introduce a concept of regularity and we give a simple ergodic condition on the signal semigroup for the Monte Carlo particle filter to converge in law and uniformly with respect to time to the optimal filter, yielding what seems to be the first uniform convergence result for a particle approximation of the nonlinear filtering equation.

Dynamic programming and feedback analysis of the two dimensional tidal dynamics system

2020 ◽  
Vol 26 ◽  
pp. 109
Author(s):  
Manil T. Mohan
Keyword(s):  
Optimal Control ◽  
Dynamic Programming ◽  
Jacobi Equation ◽  
Value Function ◽  
Two Dimensional ◽  
Tidal Dynamics ◽  
Infinite Dimensional ◽  
Bellman Principle ◽  
The Value Function ◽  
Hamilton Jacobi Equation

In this work, we consider the controlled two dimensional tidal dynamics equations in bounded domains. A distributed optimal control problem is formulated as the minimization of a suitable cost functional subject to the controlled 2D tidal dynamics equations. The existence of an optimal control is shown and the dynamic programming method for the optimal control of 2D tidal dynamics system is also described. We show that the feedback control can be obtained from the solution of an infinite dimensional Hamilton-Jacobi equation. The non-differentiability and lack of smoothness of the value function forced us to use the method of viscosity solutions to obtain a solution of the infinite dimensional Hamilton-Jacobi equation. The Bellman principle of optimality for the value function is also obtained. We show that a viscosity solution to the Hamilton-Jacobi equation can be used to derive the Pontryagin maximum principle, which give us the first order necessary conditions of optimality. Finally, we characterize the optimal control using the adjoint variable.

scholarly journals A fully backward representation of semilinear PDEs applied to the control of thermostatic loads in power systems

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Lucas Izydorczyk ◽  
Nadia Oudjane ◽  
Francesco Russo
Keyword(s):  
Monte Carlo ◽  
Power Systems ◽  
Control Problem ◽  
Stochastic Control ◽  
Computational Efficiency ◽  
Value Function ◽  
Semilinear Pdes ◽  
Areas Of Interest ◽  
The Value Function

Abstract We propose a fully backward representation of semilinear PDEs with application to stochastic control. Based on this, we develop a fully backward Monte-Carlo scheme allowing to generate the regression grid, backwardly in time, as the value function is computed. This offers two key advantages in terms of computational efficiency and memory. First, the grid is generated adaptively in the areas of interest, and second, there is no need to store the entire grid. The performances of this technique are compared in simulations to the traditional Monte-Carlo forward-backward approach on a control problem of thermostatic loads.

scholarly journals An evolutionary monotone follower problem in [0, 1]

1989 ◽  
Vol 31 (1) ◽  
pp. 97-107 ◽  
Author(s):  
Min Sun
Keyword(s):  
Dynamic Programming ◽  
Value Function ◽  
Diffusion Processes ◽  
Control Action ◽  
Integral Type ◽  
Cost Functional ◽  
Controlled Diffusion ◽  
The Cost ◽  
The Value Function ◽  
Controlled Diffusion Processes

AbstractWe consider in this article an evolutionary monotone follower problem in [0,1]. State processes under consideration are controlled diffusion processes , solutions of dyx(t) = g(yx(t), t)dt + σu(yx(t), t) dwt + dυt with yx(0) = x ∈[0, 1], where the control processes υt are increasing, positive, and adapted. The cost functional is of integral type, with certain explicit cost of control action including the cost of jumps. We shall present some analytic results of the value function, mainly its characterisation, by standard dynamic programming arguments.

scholarly journals An Annuitization Problem in the Tax-Deferred Annuity Model

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Yanan Li
Keyword(s):  
Dynamic Programming ◽  
Value Function ◽  
Hjb Equation ◽  
Time Dependent ◽  
Dynamic Programming Principle ◽  
Value Functions ◽  
Deferred Annuity ◽  
Consumption Strategies ◽  
The Value Function ◽  
Dependent Mortality

This paper examines the optimal annuitization, investment, and consumption strategies of an individual facing a time-dependent mortality rate in the tax-deferred annuity model and considers both the case when the rate of buying annuities is unrestricted and the case when it is restricted. At the beginning, by using the dynamic programming principle, we obtain the corresponding HJB equation. Since the existence of the tax and the time-dependence of the value function make the corresponding HJB equation hard to solve, firstly, we analyze the problem in a simpler case and use some numerical methods to get the solution and some of its useful properties. Then, by using the obtained properties and Kuhn–Tucker conditions, we discuss the problem in general cases and get the value functions and the optimal annuitization strategies, respectively.

scholarly journals Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation

2020 ◽  
Vol 26 ◽  
pp. 25
Author(s):  
Alessandro Calvia
Keyword(s):  
Optimal Control ◽  
Control Problem ◽  
Infinite Horizon ◽  
Value Functions ◽  
Partial Observation ◽  
Discounted Cost ◽  
Stochastic Filtering ◽  
Observation Process ◽  
The Value Function ◽  
Complete Observation

We consider an infinite horizon optimal control problem for a pure jump Markov process X, taking values in a complete and separable metric space I, with noise-free partial observation. The observation process is defined as Yt = h(Xt), t ≥ 0, where h is a given map defined on I. The observation is noise-free in the sense that the only source of randomness is the process X itself. The aim is to minimize a discounted cost functional. In the first part of the paper we write down an explicit filtering equation and characterize the filtering process as a Piecewise Deterministic Process. In the second part, after transforming the original control problem with partial observation into one with complete observation (the separated problem) using filtering equations, we prove the equivalence of the original and separated problems through an explicit formula linking their respective value functions. The value function of the separated problem is also characterized as the unique fixed point of a suitably defined contraction mapping.

Sign in / Sign up

Export Citation Format

Share Document