Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

Reinforcement learning (RL) methods usually treat reward functions as black boxes. As such, these methods must extensively interact with the environment in order to discover rewards and optimal policies. In most RL applications, however, users have to program the reward function and, hence, there is the opportunity to make the reward function visible – to show the reward function’s code to the RL agent so it can exploit the function’s internal structure to learn optimal policies in a more sample efficient manner. In this paper, we show how to accomplish this idea in two steps. First, we propose reward machines, a type of finite state machine that supports the specification of reward functions while exposing reward function structure. We then describe different methodologies to exploit this structure to support learning, including automated reward shaping, task decomposition, and counterfactual reasoning with off-policy learning. Experiments on tabular and continuous domains, across different tasks and RL agents, show the benefits of exploiting reward structure with respect to sample efficiency and the quality of resultant policies. Finally, by virtue of being a form of finite state machine, reward machines have the expressive power of a regular language and as such support loops, sequences and conditionals, as well as the expression of temporally extended properties typical of linear temporal logic and non-Markovian reward specification.

ANALYSIS OF SIMPLE SENTENCE STRUCTURE IN ENGLISH: FUNCTIONAL SYNTAX APPROACH

Many approaches in syntax give a deeper chance to analyze the sentence beyond its formal structure even though those approaches focus on the issues of words category, function and role in a sentence construction. The approach of functional syntax emphasizes the sentence structure on the ‘role’ of each function in each category of words that construct the sentence. This paper analyzes simple sentence in English especially declarative sentence to describe the constituent units and the function structure of simple sentence with active voice. The analysis employs Morley’s Functional Syntax Approach to identify the function of each english active sentence element and describe the structure of a simple sentence in English. The results of analysis show that simple sentence can have a syntactic function structure ofS-P-O, S-P-C, or S-P-ACIRC with a functional role structure which is principally composed of semantic roles of the filler elements of each function in the form of participant-process-entity/thing; participant-process-quality or participant-process-feature/quality. The semantic role of the verbs in a sentence determine the elements of filler before and after the verb given in a sentence. Keywords: Functional syntax, verbs, semantic role, syntactic function.

Exploring the Effects of Individual Differences in Function Structure Modeling Behaviors

This paper presents an exploratory study conducted to understand the role of individual differences between designers in the function modeling process and with respect to final models. An input-process-output framework of function modeling is proposed to systematically approach this theory building and discovery research study. Four measures of individual differences are identified of interest. These include the systemizing quotient, goal orientation, risk propensity, and concept design thinking style. Each metric is composed of multiple items that can be assessed through survey instruments. A previously developed protocol study is used to capture function modeling behaviors and a final function structure model. Data collected from the survey instruments and protocol study is processed to generate input, process, and output measures. A regression-based analysis is used to identify correlations in three groups: input-process, input-output, and process-output. Potential correlations of interest are identified within each group. Implications of these correlations are discussed from a function structure modeling perspective and hypotheses for future research are identified based on the patterns observed in this study. Three testable hypotheses are proposed for future investigation: (1) Goal orientation has no effect on activity distribution in the function modeling process, (2) Thinking style has no effect on the function modeling process, and (3) Risk propensity has no effect on element distribution in the function modeling process. Finally, an anticipated experiment is outlined to investigate one of the potential relationships discovered in this study.

Utility Measurement in Integrative Negotiation

This paper develops an adjustment to utility measurement in integrative negotiation where the negotiation information context is incomplete. The developed function reveals not only win-win outcomes but also deceptive practices where negotiators accept a win-lose deal and then compensate their loss in a deceptive way and greedy practices where negotiators realize their strong competitive position and try to extremely maximize their gains. However, to realize the objective, the utility measurement function literature and theories are reviewed to determine the relevant function structure and the necessary attributes that reveal the desired outcome in an incomplete information context. After examination, relationship measurement is added to the function under two utilities: Decision Utility and Experienced Utility. The foundation of the utility measurement function contributes to revealing satisfying win-win outcomes in an incomplete information negotiation context. Therefore, it develops the negotiation field by designing win-win deals that are beneficial and satisfying in which the advantage is distributed between the negotiators.

Robust Normalized Subband Adaptive Filter Algorithm with a Sigmoid-Function-Based Step-Size Scaler and Its Convex Combination Version

In this paper, by inserting the logarithm cost function of the normalized subband adaptive filter algorithm with the step-size scaler (SSS-NSAF) into the sigmoid function structure, the proposed sigmoid-function-based SSS-NSAF algorithm yields improved robustness against impulsive interferences and lower steady-state error. In order to identify sparse impulse response further, a series of sparsity-aware algorithms, including the sigmoid L0 norm constraint SSS-NSAF (SL0-SSS-NSAF), sigmoid step-size scaler improved proportionate NSAF (S-SSS-IPNSAF), and sigmoid L0 norm constraint step-size scaler improved proportionate NSAF (SL0-SSS-IPNSAF), is derived by inserting the logarithm cost function into the sigmoid function structure as well as the L0 norm of the weight coefficient vector to act as a new cost function. Since the use of the fix step size in the proposed SL0-SSS-IPNSAF algorithm, it needs to make a trade-off between fast convergence rate and low steady-state error. Thus, the convex combination version of the SL0-SSS-IPNSAF (CSL0-SSS-IPNSAF) algorithm is proposed. Simulations in acoustic echo cancellation (AEC) scenario have justified the improved performance of these proposed algorithms in impulsive interference environments and even in the impulsive interference-free condition.

Conceptual Design of Electromechanical Actuation Systems for Large-Sized Directional Control Valves

Increasing performance in modern hydraulics is achieved by a close investigation of possible enhancements of its components. Prior research has pointed out that electromechanical actuators can form suitable alternatives to hydraulically piloted control systems. Since the requirements at these actuation systems depend on the operating conditions of the system, each actuator can be optimized to the respective hydraulic system. Considering that many different conceptual designs are suitable, the phase of conceptual design plays a decisive role during the design process. Therefore, this paper focuses on the process of developing new conceptual designs for electromechanical valve actuation systems using the method of function structures. Aiming to identify special design features, which need to be considered during the design process of electromechanical actuation systems, an exemplary actuator was designed based on the derived function structure. To highlight the potential of function structures for the development of new electromechanical valve actuation systems, two principal concepts, which allow the reduction of the necessary forces, have been developed by extending the function structure. These concepts have been experimentally investigated to identify their advantages and disadvantages.