Reinforcement Learning Enables Field-Development Policy Optimization

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 201254, “Reinforcement Learning for Field-Development Policy Optimization,” by Giorgio De Paola, SPE, and Cristina Ibanez-Llano, Repsol, and Jesus Rios, IBM, et al., prepared for the 2020 SPE Annual Technical Conference and Exhibition, originally scheduled to be held in Denver, Colorado, 5–7 October. The paper has not been peer reviewed. A field-development plan consists of a sequence of decisions. Each action taken affects the reservoir and conditions any future decision. The presence of uncertainty associated with this process, however, is undeniable. The novelty of the approach proposed by the authors in the complete paper is the consideration of the sequential nature of the decisions through the framework of dynamic programming (DP) and reinforcement learning (RL). This methodology allows moving the focus from a static field-development plan optimization to a more-dynamic framework that the authors call field-development policy optimization. This synopsis focuses on the methodology, while the complete paper also contains a real-field case of application of the methodology. Methodology Deep RL (DRL). RL is considered an important learning paradigm in artificial intelligence (AI) but differs from supervised or unsupervised learning, the most commonly known types currently studied in the field of machine learning. During the last decade, RL has attracted greater attention because of success obtained in applications related to games and self-driving cars resulting from its combination with deep-learning architectures such as DRL, which has allowed RL to scale on to previously unsolvable problems and, therefore, solve much larger sequential decision problems. RL, also referred to as stochastic approximate dynamic programming, is a goal-directed sequential-learning-from-interaction paradigm. The learner or agent is not told what to do but instead has to learn which actions or decisions yield a maximum reward through interaction with an uncertain environment without losing too much reward along the way. This way of learning from interaction to achieve a goal must be achieved in balance with the exploration and exploitation of possible actions. Another key characteristic of this type of problem is its sequential nature, where the actions taken by the agent affect the environment itself and, therefore, the subsequent data it receives and the subsequent actions to be taken. Mathematically, such problems are formulated in the framework of the Markov decision process (MDP) that primarily arises in the field of optimal control. An RL problem consists of two principal parts: the agent, or decision-making engine, and the environment, the interactive world for an agent (in this case, the reservoir). Sequentially, at each timestep, the agent takes an action (e.g., changing control rates or deciding a well location) that makes the environment (reservoir) transition from one state to another. Next, the agent receives a reward (e.g., a cash flow) and an observation of the state of the environment (partial or total) before taking the next action. All relevant information informing the agent of the state of the system is assumed to be included in the last state observed by the agent (Markov property). If the agent observes the full environment state once it has acted, the MDP is said to be fully observable; otherwise, a partially observable Markov decision process (POMDP) results. The agent’s objective is to learn policy mapping from states (MDPs) or histories (POMDPs) to actions such that the agent’s cumulated (discounted) reward in the long run is maximized.

Understanding the Energy Behavior of Building Occupants Through the Chronology of Their Energy Interactions

The energy consumption of buildings has traditionally been driven by the consumption habits of building occupants. However, with the proliferation of smart building technologies and appliances, automated machine decisions are beginning to impart their influence on building energy behavior as well. This is giving rise to a disconnect between occupant energy behavior and the overall energy consumption of buildings. Consequently, researchers can no longer leverage building energy consumption as a proxy for understanding human energy behavior. This paper addresses this problem by exploiting the habitual and sequential nature of human energy consumption. By studying the chronology of human energy actions, the results of this work present a promising new approach for non-intrusively learning about human energy behavior directly from building energy demand data.

An Approach for a Next-Word Prediction for Ukrainian Language

Text generation, in particular, next-word prediction, is convenient for users because it helps to type without errors and faster. Therefore, a personalized text prediction system is a vital analysis topic for all languages, primarily for Ukrainian, because of limited support for the Ukrainian language tools. LSTM and Markov chains and their hybrid were chosen for next-word prediction. Their sequential nature (current output depends on previous) helps to successfully cope with the next-word prediction task. The Markov chains presented the fastest and adequate results. The hybrid model presents adequate results but it works slowly. Using the model, user can generate not only one word but also a few or a sentence or several sentences, unlike T9.

Is there any interaction between real earnings management and accrual-based earnings management?

This research aims to investigate whether firms employ real earnings management (REM) and accrual-based earnings management (AEM) as substitutes for each other when managing earnings to meet earnings benchmarks. It specifically looks at the sequential nature of both forms of earnings management. REM is proxied by an abnormal amount of operating cash developed by Dechow et al. (1998), while AEM is proxied by the discretionary accrual model by Dechow, Sloan, & Sweeney (1995). The data was obtained from the Economics and Business Data Center, Faculty of Economics and Business, Gadjah Mada University, focusing on manufacturing and mining companies during the period from 2005 to 2013, which resulted in 754 firm-years data. Using correlation tests and an empirical model developed by this research, which captures the interaction between REM and AEM, this research shows that firms use both forms of earnings management sequentially; managers more often engage in accrual-based earnings management if the earnings produced by real manipulations do not meet the earnings target. This finding is important as REM and AEM occur sequentially instead of simultaneously, and earnings performance is not only driven by accrual-based earnings management but also by real earnings management.

Culture, learning and rational decision-making: evidence from a TV show

This paper analyzes the French and the Vietnamese versions of the TV game show “The Price is Right”, using data from 130 episodes. We focus on the bidding game, covering 434 rounds and 1,736 bids. We document that players deviate signifi cantly from what is predicted by the model of rational expectations, especially in the French population. Moreover, Vietnamese fourth bidders are found to win more frequently than their French counterparts in spite of using strategic bids less often. We attribute these results to cultural reasons. Contestants from the collectivistic, uncertainty-tolerant culture (i.e., Vietnam) are more reluctant to engage in strategic bidding than individuals from the individualistic, uncertainty-avoidant culture (i.e., France). However, Vietnamese contestants pay more attention to the estimates of the previous players and thus make a better use of the informational advantage inherent to the sequential nature of the game. Overall, our evidence suggests that culture is an important omitted variable in studies that examine cross-country differences in decision-making.

Entity–Relation Extraction—A Novel and Lightweight Method Based on a Gate Linear Mechanism

Entity–relation extraction has attracted considerable attention in recent years as a fundamental task in natural language processing. The goal of entity–relation extraction is to discover the relation structures of entities from a natural language sentence. Most existing models approach this task using recurrent neural nets (RNNs); however, given the sequential nature of RNNs, the states cannot be computed in parallel, which slows the machine comprehension. In this paper, we propose a new end-to-end model based on dilated convolutional units and the gate linear mechanism as an alternative to those recurrent models. We find that relation extraction becomes more difficult as the sentence length increases. In this paper, we introduce dynamic convolutions based on lightweight convolutions to process long sequences, which thus reduces the number of parameters to a low level. Another challenge in relation extraction is relation spans potentially overlapping in a sentence, representing a bottleneck for the detection of multiple relational triplets. To alleviate this problem, we design an entirely new prediction scheme to extract relational pairs and additionally boost performance. We conduct experiments on two widely used datasets, and the results show that our model outperforms the baselines by a large margin.

k-SDPP: Fixed-Size Video Summarization via Sequential Determinantal Point Processes

With the explosive growth of video data, video summarization which converts long-time videos to key frame sequences has become an important task in information retrieval and machine learning. Determinantal point processes (DPPs) which are elegant probabilistic models have been successfully applied to video summarization. However, existing DPP-based video summarization methods suffer from poor efficiency of outputting a specified size summary or neglecting inherent sequential nature of videos. In this paper, we propose a new model in the DPP lineage named k-SDPP in vein of sequential determinantal point processes but with fixed user specified size k. Our k-SDPP partitions sampled frames of a video into segments where each segment is with constant number of video frames. Moreover, an efficient branch and bound method (BB) considering sequential nature of the frames is provided to optimally select k frames delegating the summary from the divided segments. Experimental results show that our proposed BB method outperforms not only k-DPP and sequential DPP (seqDPP) but also the partition and Markovian assumption based methods.