A Deep Q-Network for the Beer Game: Deep Reinforcement Learning for Inventory Optimization

Problem definition: The beer game is widely used in supply chain management classes to demonstrate the bullwhip effect and the importance of supply chain coordination. The game is a decentralized, multiagent, cooperative problem that can be modeled as a serial supply chain network in which agents choose order quantities while cooperatively attempting to minimize the network’s total cost, although each agent only observes local information. Academic/practical relevance: Under some conditions, a base-stock replenishment policy is optimal. However, in a decentralized supply chain in which some agents act irrationally, there is no known optimal policy for an agent wishing to act optimally. Methodology: We propose a deep reinforcement learning (RL) algorithm to play the beer game. Our algorithm makes no assumptions about costs or other settings. As with any deep RL algorithm, training is computationally intensive, but once trained, the algorithm executes in real time. We propose a transfer-learning approach so that training performed for one agent can be adapted quickly for other agents and settings. Results: When playing with teammates who follow a base-stock policy, our algorithm obtains near-optimal order quantities. More important, it performs significantly better than a base-stock policy when other agents use a more realistic model of human ordering behavior. We observe similar results using a real-world data set. Sensitivity analysis shows that a trained model is robust to changes in the cost coefficients. Finally, applying transfer learning reduces the training time by one order of magnitude. Managerial implications: This paper shows how artificial intelligence can be applied to inventory optimization. Our approach can be extended to other supply chain optimization problems, especially those in which supply chain partners act in irrational or unpredictable ways. Our RL agent has been integrated into a new online beer game, which has been played more than 17,000 times by more than 4,000 people.

«THE BULLWHIP EFFECT» IN THE LOGISTIC ACTIVITY OF THE ENTERPRISE

This article is devoted to the development of international logistical activities, examination and prevention negative factors on any links of logistic chain. The important requirements to obtain maximum efficiency at all components of the logistic chain with minimal costs in each of its items, compliance with market interests and goals of each participant of logistic chain are considered. The simulated model has proved that the farther on logistic chain the information flow passes from consumer the more distorted information about the necessary stocks of products and real stocks in warehouse, and as a result – the extraordinary accumulation of mistakes in the forms of surplus of goods at all stages of its movement from producer to consumer. The similar phenomenon in logistics is called the effect of "The Bullwhip Effect". The method of practical analysis of the "bullwhip" effect was used in order to investigate resistance to this effect on the situation were simulated in «BEER GAME». The simulation method “BEER GAME” is the result of the work of the consulting system MA system, which works in the field of business logistics around the world. Four parts of the logistics chain took part in the simulation: a retailer, a distributor, a wholesaler and a manufacturer. They make a logistics chain 52 weeks long. The overall results of the game are presented in detail: the total cost of the simulated game, and manufacturer's game statistics, the statistics of game such as costs, orders and information of warehouse stock in terms of retailer, distributor and wholesaler. Furthermore, the article highlights other destabilizing effects that are closely related to the "bullwhip" effect, such as: the Forrester effect - a phenomenon in the sales channel, which predicts the inefficiency of the logistics chain; the Burbage effect - when the size of supplies can change in the direction of increase without managerial decisions; the Halligan effect - when there is a deviation from the existing supply and production plan; promotion effect - implicit price changes. On the basis of the conducted modeling and consideration of the reasons of occurrence of effect methods of stabilization of a logistic chain are offered.

Results of Beer Game Trials Played by Natural Resource Managers Versus Students: Does Age Influence Ordering Decisions?

Systems involving agriculture and natural resources (AGNR) management and representing integrations of biologic, geologic, socio-economic, and climatic characteristics are incredibly complex. AGNR managers purport using a systems-oriented mental model while many observed management and policy strategies remain linear or symptom-driven. To improve AGNR professionals’ systems thinking abilities, two programs, the King Ranch® Institute for Ranch Management at Texas A&M University-Kingsville (KRIRM) and the Honors College at South Dakota State University (SDSUHC), implemented the famous Production Distribution Simulation Game (a.k.a. the Beer Game) into their programs beginning in 2003 and 2011. A Beer Game database consisting of 10 years of trials or over 270 individual players was compared to seminal work in the literature as well as to one another. We found that AGNR managers and students performed worse than players in a seminal Beer Game study. More interestingly, we found that younger players adapted more readily to inventory surpluses by reducing the order rates and effective inventories significantly when compared to older players (p < 0.10 for retailer and distributors, and p < 0.05 for wholesales and factories). We substantiated our results to those in more recent studies of age-related decision-making and in the context of common learning disabilities. Lastly, we discuss some implications of such decision-making on 21st century AGNR problems and encourage AGNR disciplines to better integrate system dynamics-based education and collaboration in order to better prepare for such complex issues.

Minimizing the bullwhip effect in a supply chain: a simulation approach using the beer game

In this research, the aim is to find the most appropriate inventory management logic and set of rules along with the optimal decision values that will minimize the bullwhip effect in a supply chain, taking the beer game supply chain as a reference model. In order to achieve this, a simulation model of the beer game supply chain is developed along with an ordering strategy based on the Economic Order Quantity with additional rules, such as no backorder policy, vendor-managed inventory, and taking into consideration route deliveries, all of which are implemented in the ordering algorithm. In the literature, there is extensive research conducted on the causes of the bullwhip effect and in the presence of certain inventory management policies. However, these terms are rarely combined with simulation modeling to provide satisfactory proven results. In this article, our proposed ordering algorithm avoids the bullwhip effect to a very large extent. The results show that approximately half the cost is incurred compared to recent studies with the same settings.

SOFTWARE DEVELOPMENT FOR SUPPLY CHAIN INTERACTION MODELING

The study of complex logistic systems often requires specialized software that allows studying the characteristics and properties of the observed object using simulation methods. An example of such a system is the supply network, in the general case of an arbitrary structure. For linear supply chains known business game “Beer Game”, which allows to observe and explore the effects that occur in long supply chains. The aim of the work was to develop a model that is a generalization of the original “Beer Game” in the case of a supply network of arbitrary topology and the development of software that allows investigating the resulting model using simulation methods. In this paper the mathematical model of the supply chain is presented. A set of algorithms for estimating and forming the order volume by each network participant is proposed. The developed application that emulates the work of the supply chain is described. Various ways of creating a supply network topology have been proposed, and a description of the parameters for modeling the supply network has been given. The capabilities of the application are demonstrated when generating data and interpreting the obtained results in the form of tables, graphs, and animation. The ways of forming supply chains with different characteristics are studied. Experiments on the study of algorithms for estimating the volume of orders on various topologies of supply chains were carried out. Ways to assess the performance of the supply chain were proposed. The developed software can be used to justify the choice of strategy for managing supply chains in distribution logistics for industrial and commercial enterprises of various sizes, from large networks to retail. In addition, the developed software can be used in the educational process for students of economic specialties.