scholarly journals Pitfalls and Best Practices in Algorithm Configuration

2019 ◽  
Vol 64 ◽  
pp. 861-893 ◽  
Author(s):  
Katharina Eggensperger ◽  
Marius Lindauer ◽  
Frank Hutter
Keyword(s):  
Best Practices ◽  
High Performance ◽  
State Of The Art ◽  
Parameter Tuning ◽  
Propositional Satisfiability ◽  
Ai Planning ◽  
Practical Applications ◽  
Automated Algorithm ◽  
Algorithm Configuration ◽  
Satisfiability Solving

Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as propositional satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning). Automated algorithm configuration methods have recently received much attention in the AI community since they replace tedious, irreproducible and error-prone manual parameter tuning and can lead to new state-of-the-art performance. However, practical applications of algorithm configuration are prone to several (often subtle) pitfalls in the experimental design that can render the procedure ineffective. We identify several common issues and propose best practices for avoiding them. As one possibility for automatically handling as many of these as possible, we also propose a tool called GenericWrapper4AC.

scholarly journals Automated Algorithm Selection: Survey and Perspectives

2019 ◽  
Vol 27 (1) ◽  
pp. 3-45 ◽  
Author(s):  
Pascal Kerschke ◽  
Holger H. Hoos ◽  
Frank Neumann ◽  
Heike Trautmann
Keyword(s):  
State Of The Art ◽  
Combinatorial Problems ◽  
The State ◽  
Problem Instance ◽  
Ai Planning ◽  
Algorithm Selection ◽  
Automated Algorithm ◽  
Continuous Problems ◽  
Algorithm Configuration ◽  
The One

It has long been observed that for practically any computational problem that has been intensely studied, different instances are best solved using different algorithms. This is particularly pronounced for computationally hard problems, where in most cases, no single algorithm defines the state of the art; instead, there is a set of algorithms with complementary strengths. This performance complementarity can be exploited in various ways, one of which is based on the idea of selecting, from a set of given algorithms, for each problem instance to be solved the one expected to perform best. The task of automatically selecting an algorithm from a given set is known as the per-instance algorithm selection problem and has been intensely studied over the past 15 years, leading to major improvements in the state of the art in solving a growing number of discrete combinatorial problems, including propositional satisfiability and AI planning. Per-instance algorithm selection also shows much promise for boosting performance in solving continuous and mixed discrete/continuous optimisation problems. This survey provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Different from earlier work, it covers applications to discrete and continuous problems, and discusses algorithm selection in context with conceptually related approaches, such as algorithm configuration, scheduling, or portfolio selection. Since informative and cheaply computable problem instance features provide the basis for effective per-instance algorithm selection systems, we also provide an overview of such features for discrete and continuous problems. Finally, we provide perspectives on future work in the area and discuss a number of open research challenges.

scholarly journals AutoFolio: An Automatically Configured Algorithm Selector

2015 ◽  
Vol 53 ◽  
pp. 745-778 ◽  
Author(s):  
Marius Lindauer ◽  
Holger H. Hoos ◽  
Frank Hutter ◽  
Torsten Schaub
Keyword(s):  
State Of The Art ◽  
Machine Learning Techniques ◽  
Problem Instance ◽  
Algorithm Selection ◽  
The Core ◽  
Automated Algorithm ◽  
Learning Techniques ◽  
Algorithm Configuration ◽  
Optimal Values ◽  
The One

Algorithm selection (AS) techniques -- which involve choosing from a set of algorithms the one expected to solve a given problem instance most efficiently -- have substantially improved the state of the art in solving many prominent AI problems, such as SAT, CSP, ASP, MAXSAT and QBF. Although several AS procedures have been introduced, not too surprisingly, none of them dominates all others across all AS scenarios. Furthermore, these procedures have parameters whose optimal values vary across AS scenarios. This holds specifically for the machine learning techniques that form the core of current AS procedures, and for their hyperparameters. Therefore, to successfully apply AS to new problems, algorithms and benchmark sets, two questions need to be answered: (i) how to select an AS approach and (ii) how to set its parameters effectively. We address both of these problems simultaneously by using automated algorithm configuration. Specifically, we demonstrate that we can automatically configure claspfolio 2, which implements a large variety of different AS approaches and their respective parameters in a single, highly-parameterized algorithm framework. Our approach, dubbed AutoFolio, allows researchers and practitioners across a broad range of applications to exploit the combined power of many different AS methods. We demonstrate AutoFolio can significantly improve the performance of claspfolio 2 on 8 out of the 13 scenarios from the Algorithm Selection Library, leads to new state-of-the-art algorithm selectors for 7 of these scenarios, and matches state-of-the-art performance (statistically) on all other scenarios. Compared to the best single algorithm for each AS scenario, AutoFolio achieves average speedup factors between 1.3 and 15.4.

scholarly journals AutoFolio: An Automatically Configured Algorithm Selector (Extended Abstract)

2017 ◽  
Author(s):  
Marius Lindauer ◽  
Frank Hutter ◽  
Holger H. Hoos ◽  
Torsten Schaub
Keyword(s):  
High Performance ◽  
State Of The Art ◽  
The State ◽  
Problem Instance ◽  
Algorithm Selection ◽  
Algorithm Configuration ◽  
Optimal Values ◽  
Art Performance ◽  
The One

Algorithm selection (AS) techniques -- which involve choosing from a set of algorithms the one expected to solve a given problem instance most efficiently -- have substantially improved the state of the art in solving many prominent AI problems, such as SAT, CSP, ASP, MAXSAT and QBF. Although several AS procedures have been introduced, not too surprisingly, none of them dominates all others across all AS scenarios. Furthermore, these procedures have parameters whose optimal values vary across AS scenarios. In this extended abstract of our 2015 JAIR article of the same title, we summarize AutoFolio, which uses an algorithm configuration procedure to automatically select an AS approach and optimize its parameters for a given AS scenario. AutoFolio allows researchers and practitioners across a broad range of applications to exploit the combined power of many different AS methods and to automatically construct high-performance algorithm selectors. We demonstrate that AutoFolio was able to produce new state-of-the-art algorithm selectors for 7 well-studied AS scenarios and matches state-of-the-art performance statistically on all other scenarios. Compared to the best single algorithm for each AS scenario, AutoFolio achieved average speedup factors between 1.3 and 15.4.

Chapter 12. Automated Configuration and Selection of SAT Solvers

2021 ◽  
Author(s):  
Holger H. Hoos ◽  
Frank Hutter ◽  
Kevin Leyton-Brown
Keyword(s):  
Research And Development ◽  
State Of The Art ◽  
Sat Solving ◽  
Sat Solvers ◽  
Effective Application ◽  
Automated Algorithm ◽  
Fertile Ground ◽  
Algorithm Configuration ◽  
Algorithmic Approaches ◽  
Selection Of

This chapter provides an introduction to the automated configuration and selection of SAT algorithms and gives an overview of the most prominent approaches. Since the early 2000s, these so-called meta-algorithmic approaches have played a major role in advancing the state of the art in SAT solving, giving rise to new ways of using and evaluating SAT solvers. At the same time, SAT has proven to be particularly fertile ground for research and development in the area of automated configuration and selection, and methods developed there have meanwhile achieved impact far beyond SAT, across a broad range of computationally challenging problems. Conceptually more complex approaches that go beyond “pure” algorithm configuration and selection are also discussed, along with some open challenges related to meta-algorithmic approaches, such as automated algorithm configuration and selection, to the tools based on these approaches, and to their effective application.

Performance of a Method for Formulating Geometrically Exact Complementarity Constraints in Multibody Dynamic Simulation

2014 ◽  
Vol 10 (1) ◽  
Author(s):  
Daniel Montrallo Flickinger ◽  
Jedediyah Williams ◽  
Jeffrey C. Trinkle
Keyword(s):  
Dynamic Simulation ◽  
Collision Detection ◽  
High Performance ◽  
Parameter Tuning ◽  
Problem Formulation ◽  
Rigid Bodies ◽  
Detection Scheme ◽  
Practical Applications ◽  
System States ◽  
And Performance

Contemporary problem formulation methods used in the dynamic simulation of rigid bodies suffer from problems in accuracy, performance, and robustness. Significant allowances for parameter tuning, coupled with careful implementation of a broad-phase collision detection scheme are required to make dynamic simulation useful for practical applications. A constraint formulation method is presented herein that is more robust, and not dependent on broad-phase collision detection or system tuning for its behavior. Several uncomplicated benchmark examples are presented to give an analysis and make a comparison of the new polyhedral exact geometry (PEG) method with the well-known Stewart–Trinkle method. The behavior and performance for the two methods are discussed. This includes specific cases where contemporary methods fail to match theorized and observed system states in simulation, and how they are ameliorated by the new method presented here. The goal of this work is to complete the groundwork for further research into high performance simulation.

scholarly journals Boosting the Performance of SLS and CDCL Solvers by Preprocessor Tuning

10.29007/28ww ◽  
2018 ◽  
Author(s):  
Adrian Balint ◽  
Norbert Manthey
Keyword(s):  
State Of The Art ◽  
Performance Gain ◽  
Sat Solvers ◽  
Automated Algorithm ◽  
Algorithm Configuration ◽  
Art Performance ◽  
Preprocessing Technique

Preprocessing techniques are crucial for SAT solvers when it comes to reaching state-of-the-art performance as it was shown by the results of the last SAT Competitions. Theusefulness of a preprocessing technique depends highly on its own parameters, on the in-stances on which it is applied and on the used solver. In this paper we first give an extendedanalysis of the performance gain reached by using different preprocessing techniques in-dividually in combination with CDCL solvers on application instances and SLS solverson crafted instances. Further, we provide an analysis of combinations of preprocessingtechniques by means of automated algorithm configuration, where we search for optimalpreprocessor configurations for different scenarios. Our results show that the performanceof CDCL and especially of SLS solvers can be further improved when using appropriatepreprocessor configurations. The solvers augmented with the best found preprocessingconfigurations outperform the original solvers on the instances from the SAT Challenge2012, achieving new state-of-the-art results.

scholarly journals Greybox Algorithm Configuration

2021 ◽  
Author(s):  
Marie Anastacio
Keyword(s):  
State Of The Art ◽  
Automated Algorithm ◽  
Wide Range ◽  
Algorithm Configuration ◽  
Improve Algorithm ◽  
The Difference ◽  
Problem Instances ◽  
The Right ◽  
Parameter Values ◽  
Configuration Problem

The performance of state-of-the-art algorithms is highly dependent on their parameter values, and choosing the right configuration can make the difference between solving a problem in a few minutes or hours. Automated algorithm configurators have shown their efficiency on a wide range of applications. However, they still encounter limitations when confronted to a large number of parameters to tune or long algorithm running time. We believe that there is untapped knowledge that can be gathered from the elements of the configuration problem, such as the default value in the configuration space, the source code of the algorithm, and the distribution of the problem instances at hand. We aim at utilising this knowledge to improve algorithm configurators.

scholarly journals Mid-infrared polarization-controlled broadband achromatic metadevice

2020 ◽  
Vol 6 (37) ◽  
pp. eabc0711 ◽  
Author(s):  
Kai Ou ◽  
Feilong Yu ◽  
Guanhai Li ◽  
Wenjuan Wang ◽  
Andrey E. Miroshnichenko ◽  
...  
Keyword(s):  
Electromagnetic Waves ◽  
High Performance ◽  
State Of The Art ◽  
Extinction Ratio ◽  
Chromatic Aberration ◽  
Cmos Technology ◽  
Practical Applications ◽  
Polarization Control ◽  
Large Size ◽  
Polarization Multiplexing

Metasurfaces provide a compact, flexible, and efficient platform to manipulate the electromagnetic waves. However, chromatic aberration imposes severe restrictions on their applications in broadband polarization control. Here, we propose a broadband achromatic methodology to implement polarization-controlled multifunctional metadevices in mid-wavelength infrared with birefringent meta-atoms. We demonstrate the generation of polarization-controlled and achromatically on-axis focused optical vortex beams with diffraction-limited focal spots and switchable topological charge (L∥ = 0 and L⊥ = 2). Besides, we further implement broadband achromatic polarization beamsplitter with high polarization isolation (extinction ratio up to 21). The adoption of all-silicon configuration not only facilitates the integration with CMOS technology but also endows the polarization multiplexing meta-atoms with broad phase dispersion coverage, ensuring the large size and high performance of the metadevices. Compared with the state-of-the-art chromatic aberration-restricted polarization-controlled metadevices, our work represents a substantial advance and a step toward practical applications.

Sign in / Sign up

Export Citation Format

Share Document