scholarly journals ARCH-COMP20 Repeatability Evaluation Report

10.29007/8dp4 ◽  
2020 ◽  
Author(s):  
Taylor T Johnson
Keyword(s):  
Performance Evaluation ◽  
Virtual Environments ◽  
Hybrid Systems ◽  
International Competition ◽  
Benchmark Problems ◽  
Fourth Edition ◽  
World Congress ◽  
Evaluation Report ◽  
Execution Platform ◽  
Evaluation Information

This report presents the results of the repeatability evaluation for the 4th International Competition on Verifying Continuous and Hybrid Systems (ARCH-COMP’20). The competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2020, affiliated with the IFAC World Congress. In its fourth edition, twenty-eight tools submitted artifacts through a Git repository for the repeatability evaluation, applied to solve benchmark problems for seven competition categories. The majority of participants adhered to the requirements for this year’s repeatability evaluation, namely to submit scripts to automatically install and execute tools in containerized virtual environments (specifically Dockerfiles to execute within Docker), and several categories used performance evaluation information from a common execution platform. The repeatability results represent a snapshot of the current landscape of tools and the types of benchmarks for which they are particularly suited and for which others may repeat their analyses. Due to the diversity of problems in verification of continuous and hybrid systems, as well as basing on standard practice in repeatability evaluations, we evaluate the tools with pass and/or failing being repeatable.

scholarly journals ARCH-COMP19 Repeatability Evaluation Report

10.29007/wbl3 ◽  
2019 ◽  
Author(s):  
Taylor T. Johnson
Keyword(s):  
Internet Of Things ◽  
Virtual Environments ◽  
Hybrid Systems ◽  
Cyber Physical Systems ◽  
International Competition ◽  
Benchmark Problems ◽  
Evaluation Report ◽  
Standard Practice ◽  
Physical Systems

This report presents the results of the repeatability evaluation for the 3rd International Competition on Verifying Continuous and Hybrid Systems (ARCH-COMP'19). The competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2019, affiliated with the Cyber-Physical Systems and Internet of Things (CPS-IoT Week'19). In its third edition, twenty-five tools submitted artifacts through a Git repository for the repeatability evaluation, applied to solve benchmark problems for eight competition categories. The majority of participants adhered to new requirements for this year's repeatability evaluation, namely to submit scripts to automatically install and execute tools in containerized virtual environments (specifically Dockerfiles to execute within Docker). The repeatability results represent a snapshot of the current landscape of tools and the types of benchmarks for which they are particularly suited and for which others may repeat their analyses. Due to the diversity of problems in verification of continuous and hybrid systems, as well as basing on standard practice in repeatability evaluations, we evaluate the tools with pass and/or failing being repeatable.

scholarly journals ARCH-COMP18 Repeatability Evaluation Report

10.29007/n9t3 ◽  
2018 ◽  
Author(s):  
Taylor T. Johnson
Keyword(s):  
Hybrid Systems ◽  
International Competition ◽  
Benchmark Problems ◽  
Evaluation Report ◽  
Standard Practice

This report presents the results of the repeatability evaluation for the 2nd International Competition on Verifying Continuous and Hybrid Systems (ARCH-COMP'18). The competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2018. In its second edition, twenty-five tools submitted artifacts for the repeatability evaluation and applied to solve benchmark problems for seven competition categories. The repeatability results represent a snapshot of the current landscape of tools and the types of benchmarks for which they are particularly suited and for which others may repeat their analyses. Due to the diversity of problems in verification of continuous and hybrid systems, as well as basing on standard practice in repeatability evaluations, we evaluate the tools with pass and/or failing being repeatable.

scholarly journals ARCH-COMP17 Repeatability Evaluation Report

10.29007/7hvk ◽  
2018 ◽  
Author(s):  
Taylor T. Johnson
Keyword(s):  
Formal Verification ◽  
Hybrid Systems ◽  
Benchmark Problems ◽  
Safety Verification ◽  
Evaluation Report ◽  
Standard Practice ◽  
Friendly Competition

This report presents the results of the repeatability evaluation for a friendly competition for formal verification of continuous and hybrid systems. The friendly competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2017. In its first edition, thirteen tools have been applied to solve benchmark problems for the six competition categories, of which, ten tools were evaluated and passed the repeatability evaluation. The repeatability results represent a snapshot of the current landscape of tools and the types of benchmarks for which they are particularly suited and for which others may repeat their analyses. Due to the diversity of problems in verification of continuous and hybrid systems, as well as basing on standard practice in repeatability evaluations, we evaluate the tools with pass and/or failing being repeatable. These re- sults probably provide the most complete assessment of tools for the safety verification of continuous and hybrid systems up to this date.

scholarly journals ARCH-COMP20 Category Report: Continuous and Hybrid Systems with Linear Continuous Dynamics

10.29007/7dt2 ◽  
2020 ◽  
Author(s):  
Matthias Althoff ◽  
Stanley Bak ◽  
Zongnan Bao ◽  
Marcelo Forets ◽  
Goran Frehse ◽  
...  
Keyword(s):  
Formal Verification ◽  
Hybrid Systems ◽  
Benchmark Problems ◽  
Fourth Edition ◽  
Safety Verification ◽  
Alphabetical Order ◽  
Friendly Competition

We present the results of the ARCH1 2020 friendly competition for formal verification of continuous and hybrid systems with linear continuous dynamics. In its fourth edition, eight tools have been applied to solve eight different benchmark problems in the category for linear continuous dynamics (in alphabetical order): CORA, C2E2, HyDRA, Hylaa, Hylaa-Continuous, JuliaReach, SpaceEx, and XSpeed. This report is a snapshot of the current landscape of tools and the types of benchmarks they are particularly suited for. Due to the diversity of problems, we are not ranking tools, yet the presented results provide one of the most complete assessments of tools for the safety verification of continuous and hybrid systems with linear continuous dynamics up to this date.

scholarly journals ARCH-COMP20 Category Report: Hybrid Systems with Piecewise Constant Dynamics and Bounded Model Checking

10.29007/bhwx ◽  
2020 ◽  
Author(s):  
Lei Bu ◽  
Alessandro Abate ◽  
Dieky Adzkiya ◽  
Muhammad Syifa'Ul Mufid ◽  
Rajarshi Ray ◽  
...  
Keyword(s):  
Model Checking ◽  
Formal Verification ◽  
Hybrid Systems ◽  
Bounded Model Checking ◽  
Benchmark Problems ◽  
Fourth Edition ◽  
Safety Verification ◽  
Piecewise Constant ◽  
Friendly Competition

This report presents the results of a friendly competition for formal verification of continuous and hybrid systems with piecewise constant dynamics. The friendly competition took place as part of the workshop Applied Verification for Continuous and Hybrid Systems (ARCH) in 2020. In this fourth edition, five tools have been applied to solve six different benchmark problems in the category for piecewise constant dynamics: BACH, PHAVerLite, PHAVer/SX, TROPICAL, and XSpeed. Compared to last year, we combine the HBMC and HPWC categories of ARCH-COMP 2019 to a new category PCDB (hybrid systems with Piecewise Constant bounds on the Dynamics (HPCD) and Bounded model checking (BMC) of HPCD systems). The result is a snapshot of the current landscape of tools and the types of benchmarks they are particularly suited for. Due to the diversity of problems, we are not ranking tools, yet the presented results probably provide the most complete assessment of tools for the safety verification of continuous and hybrid systems with piecewise constant dynamics up to this date.

Actor-Critic–Like Stochastic Adaptive Search for Continuous Simulation Optimization

2021 ◽  
Author(s):  
Qi Zhang ◽  
Jiaqiao Hu
Keyword(s):  
Performance Evaluation ◽  
Reinforcement Learning ◽  
Simulation Optimization ◽  
Search Algorithms ◽  
Benchmark Problems ◽  
Adaptive Search ◽  
Time Analysis ◽  
Simulation Experiments ◽  
Simulation Data ◽  
Continuous Simulation

Many systems arising in applications from engineering design, manufacturing, and healthcare require the use of simulation optimization (SO) techniques to improve their performance. In “Actor-Critic–Like Stochastic Adaptive Search for Continuous Simulation Optimization,” Q. Zhang and J. Hu propose a randomized approach that integrates ideas from actor-critic reinforcement learning within a class of adaptive search algorithms for solving SO problems. The approach fully retains the previous simulation data and incorporates them into an approximation architecture to exploit knowledge of the objective function in searching for improved solutions. The authors provide a finite-time analysis for the method when only a single simulation observation is collected at each iteration. The method works well on a diverse set of benchmark problems and has the potential to yield good performance for complex problems using expensive simulation experiments for performance evaluation.

Algorithms Performance Evaluation in Hybrid Systems

2016 ◽  
pp. 1169-1181
Author(s):  
Rafael Manochio ◽  
David Buzatto ◽  
Paulo Muniz de Ávila ◽  
Rodrigo Palucci Pantoni
Keyword(s):  
Performance Evaluation ◽  
Hybrid Systems
Sign in / Sign up

Export Citation Format

Share Document