Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

Reinforcement learning (RL) methods usually treat reward functions as black boxes. As such, these methods must extensively interact with the environment in order to discover rewards and optimal policies. In most RL applications, however, users have to program the reward function and, hence, there is the opportunity to make the reward function visible – to show the reward function’s code to the RL agent so it can exploit the function’s internal structure to learn optimal policies in a more sample efficient manner. In this paper, we show how to accomplish this idea in two steps. First, we propose reward machines, a type of finite state machine that supports the specification of reward functions while exposing reward function structure. We then describe different methodologies to exploit this structure to support learning, including automated reward shaping, task decomposition, and counterfactual reasoning with off-policy learning. Experiments on tabular and continuous domains, across different tasks and RL agents, show the benefits of exploiting reward structure with respect to sample efficiency and the quality of resultant policies. Finally, by virtue of being a form of finite state machine, reward machines have the expressive power of a regular language and as such support loops, sequences and conditionals, as well as the expression of temporally extended properties typical of linear temporal logic and non-Markovian reward specification.

Ontology-Based Modelling of State Machines for Production Robots in Smart Manufacturing Systems

The integration of modern IT technologies in production equipment does not only enable them to acquire information from different sources and provide it to others but also to make decisions depending on the situation. Due to the limited processing power of such equipment, usage of state machine to describe and program it is considered a promising direction. However, the necessity of intensive interaction of the equipment units causes problems related to interoperability, which are usually solved with the usage of ontologies. The objective of the presented research is to model state machines of production robots via ontologies. The results are demonstrated on the example of a fragment of an automated production line.

Software Architecture Design of Animation Studies Platform Using Structure-Behavior Coalescence Method

Animation education in the new media era is moving toward the goal of cultivating high-end talents. The development of an architecture-oriented animation studies platform provides guarantee for the training of talents in terms of teaching quality. This research uses the Internet as the medium and mobile phones and computer clients as the main technology platforms, starting from the software architecture and constructing the system model of the animation studies platform according to the Structure-Behavior Coalescence (SBC) method. The core theme of Model-Based Systems Engineering (MBSE) is a modeling language with model consistency of systems structure and systems behavior. This paper developed Structure-Behavior Coalescence State Machine (SBC-SM) as the formal language for the MBSE animation studies platform design model singularity. The model consistency will be fully guaranteed in the MBSE animation studies platform design when the SBC state machine approach is adopted. It not only improves the efficiency of platform development but also reduces the difficulty and risk of platform development.

APPLICATION OF SOME MODELS OF POWER SUPPLY SYSTEMS WITH AN EXAMPLE OF TAKING INTO ACCOUNT THEIR RELIABILITY FOR INDUSTRIAL ENTERPRISES IN PRACTICE

The publication presents the most common and promising models and schemes for replacing the power supply system of industrial enterprises, and an example of their use in practice for power supply to sewage treatment plants of a chemical enterprise. Various types of models are considered (in tabular form, in the form of an electrical replacement circuit or in the form of a finite-state machine), as well as a logical-probabilistic approach to the representation of the power supply system. The use of automation systems in power supply systems of industrial enterprises can reduce the total number of accidents, prevent their development, reduce the time of disconnections of electrical installations and downtime of mechanisms, and also allows to transfer to work without constant maintenance a significant number of electrical installations of substations, which, in turn, leads to a reduction in the number of service personnel, increase labor productivity and reduce maintenance costs. One of the main tasks on the way to automating the power supply system of enterprises is to present it in the form of a model. The list of replacement models and schemes that can be used to represent the power supply system of enterprises is quite large, and choosing the appropriate model, for that best meets the requirements of the final goal is quite a complex scientific task. The purpose of this article is to analyze the most common and promising models and schemes for replacing the power supply system and provide recommendations on their application in practice for power supply to sewage treatment plants of a chemical enterprise. Presenting the power supply system in the form of a table allows you to simplify and visually perform certification of electrical equipment, identify problem and weak points in the power supply system of industrial enterprises. Such a table proves that a large amount of electrical equipment at lower levels depends on a single electrical device at higher levels. The representation of the power supply system in the form of a three-phase symmetric replacement circuit can be used to calculate short-circuit currents in symmetric and unbalanced modes in the MatLab program and similar. Using the model of the power supply system in the form of a finite-state machine allows you to visually check the reliability of power supply to consumers, simplify the design of the power supply system of an industrial enterprise, select the state of switching devices in the power supply system and take into account the advantages and disadvantages of each state. It is advisable to use the methods discussed in this article to represent the power supply system (in tabular form, in the form of a finite-state machine or a three-phase symmetric replacement scheme) and their further application and use in automated power supply systems of industrial enterprises. The logical-probabilistic method allows us to consider the issue of power supply reliability based on the theory of logical algebra.

Aplikasi Perancang Abstraksi Verilog Mesin Keadaan Terbatas Otomatis

Saat ini, hampir semua perangkat elektronik menggunakan prosesor di dalamnya. Dalam sebuah prosesor, terdapat bagian control unit yang berfungsi mengatur operasi dari komponen-komponen di dalam prosesor. Control unit merupakan sebuah mesin keadaan terbatas atau disebut finite state machine (FSM). Rangkaian FSM dapat disintesis secara manual ataupun secara otomatis menggunakan bahasa abstraksi Verilog. Dalam penelitian ini, dibuat sebuah aplikasi yang dapat membantu pengguna merancang FSM dan selanjutnya menyimpannya dalam format Verilog. Aplikasi yang dibuat secara fungsional dapat berjalan dengan kesesuaian 100% dan mampu untuk membuat rancangan Verilog untuk FSM dengan berbagai model dan teknik pengkodean state. Simulasi modul Verilog yang dihasilkan juga sesuai dengan spesifikasi rangkaian FSM yang dirancang.

Lodestone: An Efficient Byzantine Fault-Tolerant Protocol in Consortium Blockchains

We present Lodestone, a chain-based Byzantine fault-tolerant (BFT) state machine replication (SMR) protocol under partial synchrony. Lodestone enables replicas to achieve consensus with two phases of voting and enjoys (1) optimistic responsiveness and (2) linear communication complexity on average. Similar to the state-of-the-art chain-based BFT protocols, Lodestone can be optimized with a pipelining idea elegantly. We implement pipelined Lodestone and deploy experiments to evaluate its performance. The evaluation results demonstrate that Lodestone has a lower latency than HotStuff under various workloads.

Augmenting Customer Support with an NLP-based Receptionist

In this paper, we show how a Portuguese BERT model can be combined with structured data in order to deploy a chatbot based on a finite state machine to create a conversational AI system that helps a real-estate company to predict its client's contact motivation. The model achieves human level results in a dataset that containts 235 unbalanced labels. Then, we also show its benefits considering the business impact comparing it against classical NLP methods.