scholarly journals Robust Market Making via Adversarial Reinforcement Learning

2020 ◽  
Author(s):  
Thomas Spooner ◽  
Rahul Savani
Keyword(s):  
Reinforcement Learning ◽  
Nash Equilibria ◽  
Single Agent ◽  
Market Maker ◽  
Test Environment ◽  
Agent Model ◽  
Market Participants ◽  
Domain Specific ◽  
Special Cases ◽  
Zero Sum

We show that adversarial reinforcement learning (ARL) can be used to produce market marking agents that are robust to adversarial and adaptively-chosen market conditions. To apply ARL, we turn the well-studied single-agent model of Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a market maker and adversary. The adversary acts as a proxy for other market participants that would like to profit at the market maker's expense. We empirically compare two conventional single-agent RL agents with ARL, and show that our ARL approach leads to: 1) the emergence of risk-averse behaviour without constraints or domain-specific penalties; 2) significant improvements in performance across a set of standard metrics, evaluated with or without an adversary in the test environment, and; 3) improved robustness to model uncertainty. We empirically demonstrate that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.

scholarly journals Correlated Drug Action as a Baseline Additivity Model for Combination Cancer Therapy in Patient Cohorts and Cell Cultures

2021 ◽  
Author(s):  
Adith S Arun ◽  
Sung-Cheol Kim ◽  
Mehmet Eren Ahsen ◽  
Gustavo A Stolovitzky
Keyword(s):  
Cell Culture ◽  
Clinical Trials ◽  
Single Agent ◽  
Drug Action ◽  
Closed Form Expression ◽  
Cancer Therapies ◽  
Response Curves ◽  
Agent Model ◽  
Special Cases ◽  
Combination Cancer Therapy

Identifying and characterizing the effect of combination cancer therapies is of paramount importance in cancer research. The benefit of a combination can either be due to inherent heterogeneity in patient populations or because of molecular synergy between the compounds given in combination, usually studied in cell culture, or both. To shed light and help characterized combinations and their enhanced benefits over single therapies, we introduce Correlated Drug Action (CDA) as a baseline additivity model. We formulate the CDA model as a closed-form expression, which lends itself to be scalable and interpretable, both in the temporal domain (tCDA) to explain survival curves, and in the dose domain (dCDA), to explain dose-response curves. CDA can be used in clinical trials and cell culture experiments. At the level of clinical trials, we demonstrate tCDA's utility in explaining the benefit of clinical combinations, identifying non-additive combinations, and cases where biomarkers may be able to decouple the combination into monotherapies. At the level of cells in culture, dCDA naturally embodies null models such as Bliss additivity and the Highest Single Agent model as special cases, and can be extended to be sham combination compliant. We demonstrate the applicability of dCDA in assessing non-additive combinations and doses. Additionally, we introduce a new synergy metric, Excess over CDA (EOCDA), that incorporates elements of Bliss additivity and dose equivalence concepts in the same measure. CDA is a novel general framework for additivity at the cell line and patient population levels and provides a method to characterize and quantify the action of drug combinations.

scholarly journals Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling

Energies ◽  
2019 ◽  
Vol 13 (1) ◽  
pp. 123 ◽  
Author(s):  
Xiaohan Fang ◽  
Jinkuan Wang ◽  
Guanru Song ◽  
Yinghua Han ◽  
Qiang Zhao ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Single Agent ◽  
New Paradigm ◽  
Agent Based ◽  
Vehicle To Grid ◽  
Market Participants ◽  
Home Energy Management ◽  
Market Operation ◽  
Multi Agent ◽  
Home Energy Management System

Residential microgrid is widely considered as a new paradigm of the home energy management system. The complexity of Microgrid Energy Scheduling (MES) is increasing with the integration of Electric Vehicles (EVs) and Renewable Generations (RGs). Moreover, it is challenging to determine optimal scheduling strategies to guarantee the efficiency of the microgrid market and to balance all market participants’ benefits. In this paper, a Multi-Agent Reinforcement Learning (MARL) approach for residential MES is proposed to promote the autonomy and fairness of microgrid market operation. First, a multi-agent based residential microgrid model including Vehicle-to-Grid (V2G) and RGs is constructed and an auction-based microgrid market is built. Then, distinguish from Single-Agent Reinforcement Learning (SARL), MARL can achieve distributed autonomous learning for each agent and realize the equilibrium of all agents’ benefits, therefore, we formulate an equilibrium-based MARL framework according to each participant’ market orientation. Finally, to guarantee the fairness and privacy of the MARL process, we proposed an improved optimal Equilibrium Selection-MARL (ES-MARL) algorithm based on two mechanisms, private negotiation and maximum average reward. Simulation results demonstrate the overall performance and efficiency of proposed MARL are superior to that of SARL. Besides, it is verified that the improved ES-MARL can get higher average profit to balance all agents.

scholarly journals Multi-Agent Reinforcement Learning: A Review of Challenges and Applications

2021 ◽  
Vol 11 (11) ◽  
pp. 4948
Author(s):  
Lorenzo Canese ◽  
Gian Carlo Cardarilli ◽  
Luca Di Di Nunzio ◽  
Rocco Fazzolari ◽  
Daniele Giardino ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Mathematical Models ◽  
Learning Algorithms ◽  
Single Agent ◽  
Critical Issues ◽  
Multi Agent ◽  
Pros And Cons ◽  
Application Fields

In this review, we present an analysis of the most used multi-agent reinforcement learning algorithms. Starting with the single-agent reinforcement learning algorithms, we focus on the most critical issues that must be taken into account in their extension to multi-agent scenarios. The analyzed algorithms were grouped according to their features. We present a detailed taxonomy of the main multi-agent approaches proposed in the literature, focusing on their related mathematical models. For each algorithm, we describe the possible application fields, while pointing out its pros and cons. The described multi-agent algorithms are compared in terms of the most important characteristics for multi-agent reinforcement learning applications—namely, nonstationarity, scalability, and observability. We also describe the most common benchmark environments used to evaluate the performances of the considered methods.

scholarly journals A game strategy model in the digital curling system based on NFSP

2021 ◽  
Author(s):  
Yuntao Han ◽  
Qibin Zhou ◽  
Fuqing Duan
Keyword(s):  
Reinforcement Learning ◽  
Nash Equilibrium ◽  
Action Space ◽  
Learning Networks ◽  
Game Tree ◽  
Continuous Action ◽  
Extensive Game ◽  
Strategy Model ◽  
Zero Sum ◽  
Tree Searching

AbstractThe digital curling game is a two-player zero-sum extensive game in a continuous action space. There are some challenging problems that are still not solved well, such as the uncertainty of strategy, the large game tree searching, and the use of large amounts of supervised data, etc. In this work, we combine NFSP and KR-UCT for digital curling games, where NFSP uses two adversary learning networks and can automatically produce supervised data, and KR-UCT can be used for large game tree searching in continuous action space. We propose two reward mechanisms to make reinforcement learning converge quickly. Experimental results validate the proposed method, and show the strategy model can reach the Nash equilibrium.

scholarly journals On-Demand Channel Bonding in Heterogeneous WLANs: A Multi-Agent Deep Reinforcement Learning Approach

Sensors ◽  
2020 ◽  
Vol 20 (10) ◽  
pp. 2789 ◽  
Author(s):  
Hang Qi ◽  
Hao Huang ◽  
Zhiqun Hu ◽  
Xiangming Wen ◽  
Zhaoming Lu
Keyword(s):  
Reinforcement Learning ◽  
Transmission Rate ◽  
Single Agent ◽  
Time Of Day ◽  
Action Space ◽  
Traffic Load ◽  
Traffic Demand ◽  
Channel Bonding ◽  
On Demand ◽  
Multi Agent

In order to meet the ever-increasing traffic demand of Wireless Local Area Networks (WLANs), channel bonding is introduced in IEEE 802.11 standards. Although channel bonding effectively increases the transmission rate, the wider channel reduces the number of non-overlapping channels and is more susceptible to interference. Meanwhile, the traffic load differs from one access point (AP) to another and changes significantly depending on the time of day. Therefore, the primary channel and channel bonding bandwidth should be carefully selected to meet traffic demand and guarantee the performance gain. In this paper, we proposed an On-Demand Channel Bonding (O-DCB) algorithm based on Deep Reinforcement Learning (DRL) for heterogeneous WLANs to reduce transmission delay, where the APs have different channel bonding capabilities. In this problem, the state space is continuous and the action space is discrete. However, the size of action space increases exponentially with the number of APs by using single-agent DRL, which severely affects the learning rate. To accelerate learning, Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is used to train O-DCB. Real traffic traces collected from a campus WLAN are used to train and test O-DCB. Simulation results reveal that the proposed algorithm has good convergence and lower delay than other algorithms.

scholarly journals MARKET STATISTICS OF A PSYCHOLOGY-BASED HETEROGENEOUS AGENT MODEL

2008 ◽  
Vol 11 (07) ◽  
pp. 717-737 ◽  
Author(s):  
HARBIR LAMBA ◽  
TIM SEAMAN
Keyword(s):  
Heterogeneous Agent ◽  
Agent Based ◽  
Agent Model ◽  
Fundamental Parameters ◽  
Market Participants ◽  
Information Stream ◽  
Momentum Trading ◽  
Real Market ◽  
Available Information ◽  
Parameter Values

We continue an investigation into a class of agent-based market models that are motivated by a psychologically-plausible form of bounded rationality. Some of the agents in an otherwise efficient hypothetical market are endowed with differing tolerances to the tension caused by being in the minority. This herding tendency may be due to purely psychological effects, momentum-trading strategies, or the rational response to perverse marketplace incentives. The resulting model has the important properties of being both very simple and insensitive to its small number of fundamental parameters. While it is most certainly a caricature market, with only boundedly rational traders and the globally available information stream being modeled directly, other market participants and effects are indirectly replicated. We show that all of the most important "stylized facts" of real market statistics are reproduced by this model. Another useful aspect of the model is that, for certain parameter values, it reduces to a standard efficient-market system. This allows us to isolate and observe the effects of particular kinds of non-rationality. To this end, we consider the effects of different asymmetries in agent behavior and show that one in particular leads to skew statistics consistent with those seen in some real financial markets.

scholarly journals Intelligent Ramp Control for Incident Response Using Dyna-QArchitecture

2015 ◽  
Vol 2015 ◽  
pp. 1-16
Author(s):  
Chao Lu ◽  
Yanan Zhao ◽  
Jianwei Gong
Keyword(s):  
Reinforcement Learning ◽  
Travel Time ◽  
Single Agent ◽  
Superior Performance ◽  
Model Free ◽  
Road Users ◽  
Total Travel Time ◽  
The Uk ◽  
Traffic Operation ◽  
Ramp Control

Reinforcement learning (RL) has shown great potential for motorway ramp control, especially under the congestion caused by incidents. However, existing applications limited to single-agent tasks and based onQ-learning have inherent drawbacks for dealing with coordinated ramp control problems. For solving these problems, a Dyna-Qbased multiagent reinforcement learning (MARL) system named Dyna-MARL has been developed in this paper. Dyna-Qis an extension ofQ-learning, which combines model-free and model-based methods to obtain benefits from both sides. The performance of Dyna-MARL is tested in a simulated motorway segment in the UK with the real traffic data collected from AM peak hours. The test results compared with Isolated RL and noncontrolled situations show that Dyna-MARL can achieve a superior performance on improving the traffic operation with respect to increasing total throughput, reducing total travel time and CO2emission. Moreover, with a suitable coordination strategy, Dyna-MARL can maintain a highly equitable motorway system by balancing the travel time of road users from different on-ramps.

scholarly journals DDPG Agent to Swing Up and Balance Cart- Pole System

2021 ◽  
pp. 102-116
Author(s):  
Buvanesh Pandian V
Keyword(s):  
Reinforcement Learning ◽  
Supervised Learning ◽  
Real World ◽  
Learning Algorithm ◽  
Current Approach ◽  
Control Problems ◽  
Mathematical Framework ◽  
Test Environment ◽  
Continuous Action ◽  
Action Spaces

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

LEARNING TO COOPERATE IN SOLVING THE TRAVELING SALESMAN PROBLEM

2005 ◽  
Vol 15 (01n02) ◽  
pp. 151-162 ◽  
Author(s):  
DEHU QI ◽  
RON SUN
Keyword(s):  
Reinforcement Learning ◽  
Traveling Salesman Problem ◽  
Single Agent ◽  
Traveling Salesman ◽  
Cooperative Team ◽  
The Traveling Salesman Problem ◽  
Better Than

A cooperative team of agents may perform many tasks better than single agents. The question is how cooperation among self-interested agents should be achieved. It is important that, while we encourage cooperation among agents in a team, we maintain autonomy of individual agents as much as possible, so as to maintain flexibility and generality. This paper presents an approach based on bidding utilizing reinforcement values acquired through reinforcement learning. We tested and analyzed this approach and demonstrated that a team indeed performed better than the best single agent as well as the average of single agents.

Multi-agent reinforcement learning using ordinal action selection and approximate policy iteration

2016 ◽  
Vol 14 (06) ◽  
pp. 1650053
Author(s):  
Daxue Liu ◽  
Jun Wu ◽  
Xin Xu
Keyword(s):  
Reinforcement Learning ◽  
Single Agent ◽  
Action Selection ◽  
Policy Iteration ◽  
Common Interest ◽  
Policy Space ◽  
Markov Games ◽  
Approximate Policy Iteration ◽  
Multi Agent ◽  
Agent Coordination

Multi-agent reinforcement learning (MARL) provides a useful and flexible framework for multi-agent coordination in uncertain dynamic environments. However, the generalization ability and scalability of algorithms to large problem sizes, already problematic in single-agent RL, is an even more formidable obstacle in MARL applications. In this paper, a new MARL method based on ordinal action selection and approximate policy iteration called OAPI (Ordinal Approximate Policy Iteration), is presented to address the scalability issue of MARL algorithms in common-interest Markov Games. In OAPI, an ordinal action selection and learning strategy is integrated with distributed approximate policy iteration not only to simplify the policy space and eliminate the conflicts in multi-agent coordination, but also to realize the approximation of near-optimal policies for Markov Games with large state spaces. Based on the simplified policy space using ordinal action selection, the OAPI algorithm implements distributed approximate policy iteration utilizing online least-squares policy iteration (LSPI). This resulted in multi-agent coordination with good convergence properties with reduced computational complexity. The simulation results of a coordinated multi-robot navigation task illustrate the feasibility and effectiveness of the proposed approach.

Sign in / Sign up

Export Citation Format

Share Document