scholarly journals ME-MCTS: Online Generalization by Combining Multiple Value Estimators

2021 ◽  
Author(s):  
Hendrik Baier ◽  
Michael Kaisers
Keyword(s):  
Monte Carlo ◽  
Tree Search ◽  
Monte Carlo Tree Search ◽  
Board Games ◽  
Evaluation Functions ◽  
Recent Advances ◽  
Action Value ◽  
Multiple Value ◽  
Static Evaluation

This paper addresses the challenge of online generalization in tree search. We propose Multiple Estimator Monte Carlo Tree Search (ME-MCTS), with a two-fold contribution: first, we introduce a formalization of online generalization that can represent existing techniques such as "history heuristics", "RAVE", or "OMA" -- contextual action value estimators or abstractors that generalize across specific contexts. Second, we incorporate recent advances in estimator averaging that enable guiding search by combining the online action value estimates of any number of such abstractors or similar types of action value estimators. Unlike previous work, which usually proposed a single abstractor for either the selection or the rollout phase of MCTS simulations, our approach focuses on the combination of multiple estimators and applies them to all move choices in MCTS simulations. As the MCTS tree itself is just another value estimator -- unbiased, but without abstraction -- this blurs the traditional distinction between action choices inside and outside of the MCTS tree. Experiments with three abstractors in four board games show significant improvements of ME-MCTS over MCTS using only a single abstractor, both for MCTS with random rollouts as well as for MCTS with static evaluation functions. While we used deterministic, fully observable games, ME-MCTS naturally extends to more challenging settings.

Do evaluation functions really improve Monte-Carlo tree search?

ICGA Journal ◽  
2019 ◽  
Vol 40 (3) ◽  
pp. 294-304
Author(s):  
Kiminori Matsuzaki ◽  
Naoki Kitamura
Keyword(s):  
Monte Carlo ◽  
Tree Search ◽  
Monte Carlo Tree Search ◽  
Evaluation Functions

scholarly journals Three-Head Neural Network Architecture for Monte Carlo Tree Search

2018 ◽  
Author(s):  
Chao Gao ◽  
Martin Müller ◽  
Ryan Hayward
Keyword(s):  
Monte Carlo ◽  
Network Architecture ◽  
Data Augmentation ◽  
The State ◽  
Neural Nets ◽  
Tree Search ◽  
Neural Net ◽  
Neural Network Architecture ◽  
Monte Carlo Tree Search ◽  
Action Value

AlphaGo Zero pioneered the concept of two-head neural networks in Monte Carlo Tree Search (MCTS), where the policy output is used for prior action probability and the state-value estimate is used for leaf node evaluation. We propose a three-head neural net architecture with policy, state- and action-value outputs, which could lead to more efficient MCTS since neural leaf estimate can still be back-propagated in tree with delayed node expansion and evaluation. To effectively train the newly introduced action-value head on the same game dataset as for two-head nets, we exploit the optimal relations between parent and children nodes for data augmentation and regularization. In our experiments for the game of Hex, the action-value head learning achieves similar error as the state-value prediction of a two-head architecture. The resulting neural net models are then combined with the same Policy Value MCTS (PV-MCTS) implementation. We show that, due to more efficient use of neural net evaluations, PV-MCTS with three-head neural nets consistently performs better than the two-head ones, significantly outplaying the state-of-the-art player MoHex-CNN.

scholarly journals MCTS-Minimax Hybrids with State Evaluations

2018 ◽  
Vol 62 ◽  
pp. 193-231 ◽  
Author(s):  
Hendrik Baier ◽  
Mark H. M. Winands
Keyword(s):  
Monte Carlo ◽  
Domain Knowledge ◽  
New Technique ◽  
Tree Search ◽  
Evaluation Function ◽  
Heuristic Evaluation ◽  
Monte Carlo Tree Search ◽  
Evaluation Functions ◽  
Selective Search ◽  
Computing Node

Monte-Carlo Tree Search (MCTS) has been found to show weaker play than minimax-based search in some tactical game domains. This is partly due to its highly selective search and averaging value backups, which make it susceptible to traps. In order to combine the strategic strength of MCTS and the tactical strength of minimax, MCTS-minimax hybrids have been introduced, embedding shallow minimax searches into the MCTS framework. Their results have been promising even without making use of domain knowledge such as heuristic evaluation functions. This article continues this line of research for the case where evaluation functions are available. Three different approaches are considered, employing minimax with an evaluation function in the rollout phase of MCTS, as a replacement for the rollout phase, and as a node prior to bias move selection. The latter two approaches are newly proposed. Furthermore, all three hybrids are enhanced with the help of move ordering and k-best pruning for minimax. Results show that the use of enhanced minimax for computing node priors results in the strongest MCTS-minimax hybrid investigated in the three test domains of Othello, Breakthrough, and Catch the Lion. This hybrid, called MCTS-IP-M-k, also outperforms enhanced minimax as a standalone player in Breakthrough, demonstrating that at least in this domain, MCTS and minimax can be combined to an algorithm stronger than its parts. Using enhanced minimax for computing node priors is therefore a promising new technique for integrating domain knowledge into an MCTS framework.

scholarly journals On Monte Carlo Tree Search and Reinforcement Learning

2017 ◽  
Vol 60 ◽  
pp. 881-936 ◽  
Author(s):  
Tom Vodopivec ◽  
Spyridon Samothrakis ◽  
Branko Ster
Keyword(s):  
Monte Carlo ◽  
Reinforcement Learning ◽  
Video Game ◽  
Close Relation ◽  
Tree Search ◽  
Monte Carlo Tree Search ◽  
Computer Go ◽  
Board Games ◽  
Planning Methods ◽  
Unified View

Fuelled by successes in Computer Go, Monte Carlo tree search (MCTS) has achieved widespread adoption within the games community. Its links to traditional reinforcement learning (RL) methods have been outlined in the past; however, the use of RL techniques within tree search has not been thoroughly studied yet. In this paper we re-examine in depth this close relation between the two fields; our goal is to improve the cross-awareness between the two communities. We show that a straightforward adaptation of RL semantics within tree search can lead to a wealth of new algorithms, for which the traditional MCTS is only one of the variants. We confirm that planning methods inspired by RL in conjunction with online search demonstrate encouraging results on several classic board games and in arcade video game competitions, where our algorithm recently ranked first. Our study promotes a unified view of learning, planning, and search.

scholarly journals MCTS-Minimax Hybrids with State Evaluations (Extended Abstract)

2018 ◽  
Author(s):  
Hendrik Baier ◽  
Mark H. M. Winands
Keyword(s):  
Monte Carlo ◽  
Tree Search ◽  
Prior Work ◽  
Monte Carlo Tree Search ◽  
Evaluation Functions ◽  
State Evaluation ◽  
Computing Node

Monte-Carlo Tree Search (MCTS) has been found to show weaker play than minimax-based search in some tactical game domains. In order to combine the tactical strength of minimax and the strategic strength of MCTS, MCTS-minimax hybrids have been proposed in prior work. This article continues this line of research for the case where heuristic state evaluation functions are available. Three different approaches are considered, employing minimax in the rollout phase of MCTS, as a replacement for the rollout phase, and as a node prior to bias move selection. The latter two approaches are newly proposed. Results show that the use of enhanced minimax for computing node priors results in the strongest MCTS-minimax hybrid in the three test domains of Othello, Breakthrough, and Catch the Lion. This hybrid also outperforms enhanced minimax as a standalone player in Breakthrough, demonstrating that at least in this domain, MCTS and minimax can be combined to an algorithm stronger than its parts.

Sign in / Sign up

Export Citation Format

Share Document