scholarly journals Efficient Actor-Critic Algorithm with Hierarchical Model Learning and Planning

2016 ◽  
Vol 2016 ◽  
pp. 1-15 ◽  
Author(s):  
Shan Zhong ◽  
Quan Liu ◽  
QiMing Fu
Keyword(s):  
Convergence Rate ◽  
Hierarchical Model ◽  
Function Approximation ◽  
Local Linear Regression ◽  
Benchmark Problems ◽  
Time Step ◽  
Model Learning ◽  
Linear Function Approximation ◽  
Efficient Learning ◽  
The Value Function

To improve the convergence rate and the sample efficiency, two efficient learning methods AC-HMLP and RAC-HMLP (AC-HMLP withl2-regularization) are proposed by combining actor-critic algorithm with hierarchical model learning and planning. The hierarchical models consisting of the local and the global models, which are learned at the same time during learning of the value function and the policy, are approximated by local linear regression (LLR) and linear function approximation (LFA), respectively. Both the local model and the global model are applied to generate samples for planning; the former is used only if the state-prediction error does not surpass the threshold at each time step, while the latter is utilized at the end of each episode. The purpose of taking both models is to improve the sample efficiency and accelerate the convergence rate of the whole algorithm through fully utilizing the local and global information. Experimentally, AC-HMLP and RAC-HMLP are compared with three representative algorithms on two Reinforcement Learning (RL) benchmark problems. The results demonstrate that they perform best in terms of convergence rate and sample efficiency.

scholarly journals Conditions on Features for Temporal Difference-Like Methods to Converge

2019 ◽  
Author(s):  
Marcus Hutter ◽  
Samuel Yang-Zhao ◽  
Sultan Javed Majeed
Keyword(s):  
Reinforcement Learning ◽  
Function Approximation ◽  
Bellman Equation ◽  
Complete Characterization ◽  
Value Functions ◽  
Approximation Space ◽  
State Aggregation ◽  
Linear Function Approximation ◽  
The Value Function

The convergence of many reinforcement learning (RL) algorithms with linear function approximation has been investigated extensively but most proofs assume that these methods converge to a unique solution. In this paper, we provide a complete characterization of non-uniqueness issues for a large class of reinforcement learning algorithms, simultaneously unifying many counter-examples to convergence in a theoretical framework. We achieve this by proving a new condition on features that can determine whether the convergence assumptions are valid or non-uniqueness holds. We consider a general class of RL methods, which we call natural algorithms, whose solutions are characterized as the fixed point of a projected Bellman equation. Our main result proves that natural algorithms converge to the correct solution if and only if all the value functions in the approximation space satisfy a certain shape. This implies that natural algorithms are, in general, inherently prone to converge to the wrong solution for most feature choices even if the value function can be represented exactly. Given our results, we show that state aggregation-based features are a safe choice for natural algorithms and also provide a condition for finding convergent algorithms under other feature constructions.

scholarly journals Studying Inertia Effects in Open Channel Flow Using Saint-Venant Equations

Water ◽  
2018 ◽  
Vol 10 (11) ◽  
pp. 1652
Author(s):  
Dong-Sin Shih ◽  
Gour-Tsyh Yeh
Keyword(s):  
Stokes Equations ◽  
Field Application ◽  
Benchmark Problems ◽  
Algebraic Equations ◽  
Step Size ◽  
Time Step ◽  
Cross Sectional ◽  
Inertia Effects ◽  
Time Step Size ◽  
Saint Venant Equations

One-dimensional (1D) Saint-Venant equations, which originated from the Navier–Stokes equations, are usually applied to express the transient stream flow. The governing equation is based on the mass continuity and momentum equivalence. Its momentum equation, partially comprising the inertia, pressure, gravity, and friction-induced momentum loss terms, can be expressed as kinematic wave (KIW), diffusion wave (DIW), and fully dynamic wave (DYW) flow. In this study, the method of characteristics (MOCs) is used for solving the diagonalized Saint-Venant equations. A computer model, CAMP1DF, including KIW, DIW, and DYW approximations, is developed. Benchmark problems from MacDonald et al. (1997) are examined to study the accuracy of the CAMP1DF model. The simulations revealed that CAMP1DF can simulate almost identical results that are valid for various fluvial conditions. The proposed scheme that not only allows a large time step size but also solves half of the simultaneous algebraic equations. Simulations of accuracy and efficiency are both improved. Based on the physical relevance, the simulations clearly showed that the DYW approximation has the best performance, whereas the KIW approximation results in the largest errors. Moreover, the field non-prismatic case of the Zhuoshui River in central Taiwan is studied. The simulations indicate that the DYW approach does not ensure achievement of a better simulation result than the other two approximations. The investigated cross-sectional geometries play an important role in stream routing. Because of the consideration of the acceleration terms, the simulated hydrograph of a DYW reveals more physical characteristics, particularly regarding the raising and recession of limbs. Note that the KIW does not require assignment of a downstream boundary condition, making it more convenient for field application.

scholarly journals KERNEL CONVERGENCE ESTIMATES FOR DIFFUSIONS WITH CONTINUOUS COEFFICIENTS

2011 ◽  
Vol 14 (07) ◽  
pp. 979-1004
Author(s):  
CLAUDIO ALBANESE
Keyword(s):  
Convergence Rate ◽  
Convergence Rates ◽  
Fourier Transforms ◽  
Uniform Continuity ◽  
Time Step ◽  
Uniform Bounds ◽  
Kernel Convergence ◽  
Discretization Schemes ◽  
A New Technique ◽  
Derivatives Of

Bidirectional valuation models are based on numerical methods to obtain kernels of parabolic equations. Here we address the problem of robustness of kernel calculations vis a vis floating point errors from a theoretical standpoint. We are interested in kernels of one-dimensional diffusion equations with continuous coefficients as evaluated by means of explicit discretization schemes of uniform step h > 0 in the limit as h → 0. We consider both semidiscrete triangulations with continuous time and explicit Euler schemes with time step so small that the Courant condition is satisfied. We find uniform bounds for the convergence rate as a function of the degree of smoothness. We conjecture these bounds are indeed sharp. The bounds also apply to the time derivatives of the kernel and its first two space derivatives. The proof is constructive and is based on a new technique of path conditioning for Markov chains and a renormalization group argument. We make the simplifying assumption of time-independence and use longitudinal Fourier transforms in the time direction. Convergence rates depend on the degree of smoothness and Hölder differentiability of the coefficients. We find that the fastest convergence rate is of order O(h2) and is achieved if the coefficients have a bounded second derivative. Otherwise, explicit schemes still converge for any degree of Hölder differentiability except that the convergence rate is slower. Hölder continuity itself is not strictly necessary and can be relaxed by an hypothesis of uniform continuity.

scholarly journals Local Discontinuous Galerkin Method for Nonlinear Time-Space Fractional Subdiffusion/Superdiffusion Equations

2020 ◽  
Vol 2020 ◽  
pp. 1-21
Author(s):  
Meilan Qiu ◽  
Dewang Li ◽  
Yanyun Wu
Keyword(s):  
Convergence Rate ◽  
Fractional Derivative ◽  
Discontinuous Galerkin ◽  
Numerical Schemes ◽  
Step Size ◽  
Time Step ◽  
Local Discontinuous Galerkin ◽  
Time Space ◽  
Backward Euler ◽  
Heterogeneous Rocks

Fractional partial differential equations with time-space fractional derivatives describe some important physical phenomena. For example, the subdiffusion equation (time order 0<α<1) is more suitable to describe the phenomena of charge carrier transport in amorphous semiconductors, nuclear magnetic resonance (NMR) diffusometry in percolative, Rouse, or reptation dynamics in polymeric systems, the diffusion of a scalar tracer in an array of convection rolls, or the dynamics of a bead in a polymeric network, and so on. However, the superdiffusion case (1<α<2) is more accurate to depict the special domains of rotating flows, collective slip diffusion on solid surfaces, layered velocity fields, Richardson turbulent diffusion, bulk-surface exchange controlled dynamics in porous glasses, the transport in micelle systems and heterogeneous rocks, quantum optics, single molecule spectroscopy, the transport in turbulent plasma, bacterial motion, and even for the flight of an albatross (for more physical applications of fractional sub-super diffusion equations, one can see Metzler and Klafter in 2000). In this work, we establish two fully discrete numerical schemes for solving a class of nonlinear time-space fractional subdiffusion/superdiffusion equations by using backward Euler difference 1<α<2 or second-order central difference 1<α<2/local discontinuous Galerkin finite element mixed method. By introducing the mathematical induction method, we show the concrete analysis for the stability and the convergence rate under the L2 norm of the two LDG schemes. In the end, we adopt several numerical experiments to validate the proposed model and demonstrate the features of the two numerical schemes, such as the optimal convergence rate in space direction is close to Ohk+1. The convergence rate in time direction can arrive at Oτ2−α when the fractional derivative is 0<α<1. If the fractional derivative parameter is 1<α<2 and we choose the relationship as h=C′τ (h denotes the space step size, C′ is a constant, and τ is the time step size), then the time convergence rate can reach to Oτ3−α. The experiment results illustrate that the proposed method is effective in solving nonlinear time-space fractional subdiffusion/superdiffusion equations.

scholarly journals Performance Analyses of IDEAL Algorithm on Highly Skewed Grid System

2014 ◽  
Vol 6 ◽  
pp. 813510
Author(s):  
Dongliang Sun ◽  
Jinliang Xu ◽  
Peng Ding
Keyword(s):  
Convergence Rate ◽  
Curvilinear Coordinates ◽  
Grid System ◽  
Time Step ◽  
Fine Grid ◽  
Simpler Algorithm ◽  
Inclined Cavity ◽  
Transfer Problems ◽  
Ideal Algorithm ◽  
The Ideal

IDEAL is an efficient segregated algorithm for the fluid flow and heat transfer problems. This algorithm has now been extended to the 3D nonorthogonal curvilinear coordinates. Highly skewed grids in the nonorthogonal curvilinear coordinates can decrease the convergence rate and deteriorate the calculating stability. In this study, the feasibility of the IDEAL algorithm on highly skewed grid system is analyzed by investigating the lid-driven flow in the inclined cavity. It can be concluded that the IDEAL algorithm is more robust and more efficient than the traditional SIMPLER algorithm, especially for the highly skewed and fine grid system. For example, at θ = 5° and grid number = 70 × 70 × 70, the convergence rate of the IDEAL algorithm is 6.3 times faster than that of the SIMPLER algorithm, and the IDEAL algorithm can converge almost at any time step multiple.

Sign in / Sign up

Export Citation Format

Share Document