Reinforcement Learning From Pixels: Waterflooding Optimization

The application of Artificial Intelligence (AI) methods in the petroleum industry gain traction in recent years. In this paper, Deep Reinforcement Learning (RL) is used to maximize the Net Present Value (NPV) of waterflooding by changing the water injection rate. This research is the first step towards showing that the use of pixel information for reinforcement learning provides many advantages, such as a fundamental understanding of reservoir physics by controlling changes in pressure and saturation without directly accounting for the reservoir petrophysical properties and wells. The optimization routine based on RL by pixel data is tested on the 2D model, which is a vertical section of the SPE 10 model. It has been shown that RL can optimize waterflooding in a 2D compressible reservoir with the 2-phase flow (oil-water). The proposed optimization method is an iterative process. In the first few thousands of updates, NPV remains in the baseline since it takes more time to converge from raw pixel data than to use classical well production/injection rate information. RL optimization resulted in improving the NPV by 15 percent, where the optimum scenario shows less watercut values and more stable production in contrast to baseline optimization. Additionally, we evaluated the impact of selecting the different action set for optimization and examined two cases where water injection well can change injection pressure with a step of 200 psi and 600 psi. The results show that in the second case, RL optimization is exploiting the limitation of the reservoir simulation engine and tries to imitate a cycled injection regime, which results in a 7% higher NPV than the first case.