Hierarchical Reinforcement Learning for Open-Domain Dialog

Open-domain dialog generation is a challenging problem; maximum likelihood training can lead to repetitive outputs, models have difficulty tracking long-term conversational goals, and training on standard movie or online datasets may lead to the generation of inappropriate, biased, or offensive text. Reinforcement Learning (RL) is a powerful framework that could potentially address these issues, for example by allowing a dialog model to optimize for reducing toxicity and repetitiveness. However, previous approaches which apply RL to open-domain dialog generation do so at the word level, making it difficult for the model to learn proper credit assignment for long-term conversational rewards. In this paper, we propose a novel approach to hierarchical reinforcement learning (HRL), VHRL, which uses policy gradients to tune the utterance-level embedding of a variational sequence model. This hierarchical approach provides greater flexibility for learning long-term, conversational rewards. We use self-play and RL to optimize for a set of human-centered conversation metrics, and show that our approach provides significant improvements – in terms of both human evaluation and automatic metrics – over state-of-the-art dialog models, including Transformers.

CLVSA: A Convolutional LSTM Based Variational Sequence-to-Sequence Model with Attention for Predicting Trends of Financial Markets

Financial markets are a complex dynamical system. The complexity comes from the interaction between a market and its participants, in other words, the integrated outcome of activities of the entire participants determines the markets trend, while the markets trend affects activities of participants. These interwoven interactions make financial markets keep evolving. Inspired by stochastic recurrent models that successfully capture variability observed in natural sequential data such as speech and video, we propose CLVSA, a hybrid model that consists of stochastic recurrent networks, the sequence-to-sequence architecture, the self- and inter-attention mechanism, and convolutional LSTM units to capture variationally underlying features in raw financial trading data. Our model outperforms basic models, such as convolutional neural network, vanilla LSTM network, and sequence-to-sequence model with attention, based on backtesting results of six futures from January 2010 to December 2017. Our experimental results show that, by introducing an approximate posterior, CLVSA takes advantage of an extra regularizer based on the Kullback-Leibler divergence to prevent itself from overfitting traps.

Variations by generalized symmetries of local Noether strong currents equivalent to global canonical Noether currents

We will pose the inverse problem question within the Krupka variational sequence framework. In particular, the interplay of inverse problems with symmetry and invariance properties will be exploited considering that the cohomology class of the variational Lie derivative of an equivalence class of forms, closed in the variational sequence, is trivial. We will focalize on the case of symmetries of globally defined field equations which are only locally variational and prove that variations of local Noether strong currents are variationally equivalent to global canonical Noether currents. Variations, taken to be generalized symmetries and also belonging to the kernel of the second variational derivative of the local problem, generate canonical Noether currents - associated with variations of local Lagrangians - which in particular turn out to be conserved along any section. We also characterize the variation of the canonical Noether currents associated with a local variational problem.

Variational derivatives in locally Lagrangian field theories and Noether–Bessel-Hagen currents

The variational Lie derivative of classes of forms in the Krupka’s variational sequence is defined as a variational Cartan formula at any degree, in particular for degrees lesser than the dimension of the basis manifold. As an example of application, we determine the condition for a Noether–Bessel-Hagen current, associated with a generalized symmetry, to be variationally equivalent to a Noether current for an invariant Lagrangian. We show that, if it exists, this Noether current is exact on-shell and generates a canonical conserved quantity.

On a bicomplex induced by the variational sequence

The construction of a finite-order bicomplex whose morphisms are the horizontal and vertical derivatives of differential forms on finite-order jet prolongations of fibered manifolds over one-dimensional bases is presented. In particular, relationship between the morphisms and classes entering the variational sequence and the associated finite-order bicomplex is studied. Properties of classes entering the infinite-order bicomplex, induced from the finite-order variational sequences by means of an infinite canonical construction, are formulated as a remark, insisting further research.

The Interior Euler-Lagrange Operator in Field Theory

The paper is devoted to the interior Euler-Lagrange operator in field theory, representing an important tool for constructing the variational sequence. We give a new invariant definition of this operator by means of a natural decomposition of spaces of differential forms, appearing in the sequence, which defines its basic properties. Our definition extends the well-known cases of the Euler-Lagrange class (Euler-Lagrange form) and the Helmholtz class (Helmholtz form). This linear operator has the property of a projector, and its kernel consists of contact forms. The result generalizes an analogous theorem valid for variational sequences over 1-dimensional manifolds and completes the known heuristic expressions by explicit characterizations and proofs.