scholarly journals Synthesizing Context-free Grammars from Recurrent Neural Networks

2021 ◽  
Author(s):  
Daniel M. Yellin ◽  
Gail Weiss
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Context Free ◽  
Context Free Grammars

scholarly journals Can Recurrent Neural Networks Learn Nested Recursion?

2018 ◽  
Vol 16 ◽  
Author(s):  
Jean-Philippe Bernardy
Keyword(s):  
Neural Networks ◽  
Upper Bound ◽  
Recurrent Neural Networks ◽  
Natural Languages ◽  
Relevant Today ◽  
Nested Recursion ◽  
The Cost ◽  
Context Free ◽  
Do So ◽  
Context Free Grammars

Context-free grammars (CFG) were one of the first formal tools used to model natural languages, and they remain relevant today as the basis of several frameworks. A key ingredient of CFG is the presence of nested recursion. In this paper, we investigate experimentally the capability of several recurrent neural networks (RNNs) to learn nested recursion. More precisely, we measure an upper bound of their capability to do so, by simplifying the task to learning a generalized Dyck language, namely one composed of matching parentheses of various kinds. To do so, we present the RNNs with a set of random strings having a given maximum nesting depth and test its ability to predict the kind of closing parenthesis when facing deeper nested strings. We report mixed results: when generalizing to deeper nesting levels, the accuracy of standard RNNs is significantly higher than random, but still far from perfect. Additionally, we propose some non-standard stack-based models which can approach perfect accuracy, at the cost of robustness.

scholarly journals Synthesizing Context-free Grammars from Recurrent Neural Networks

2021 ◽  
pp. 351-369
Author(s):  
Daniel M. Yellin ◽  
Gail Weiss
Keyword(s):  
Neural Network ◽  
Recurrent Neural Networks ◽  
Regular Language ◽  
Predictive Accuracy ◽  
Finite Automata ◽  
Deterministic Finite Automata ◽  
Rule Sets ◽  
Context Free ◽  
New Framework ◽  
Context Free Grammars

AbstractWe present an algorithm for extracting a subclass of the context free grammars (CFGs) from a trained recurrent neural network (RNN). We develop a new framework, pattern rule sets (PRSs), which describe sequences of deterministic finite automata (DFAs) that approximate a non-regular language. We present an algorithm for recovering the PRS behind a sequence of such automata, and apply it to the sequences of automata extracted from trained RNNs using the $$L^{*}$$ L ∗ algorithm. We then show how the PRS may converted into a CFG, enabling a familiar and useful presentation of the learned language.Extracting the learned language of an RNN is important to facilitate understanding of the RNN and to verify its correctness. Furthermore, the extracted CFG can augment the RNN in classifying correct sentences, as the RNN’s predictive accuracy decreases when the recursion depth and distance between matching delimiters of its input sequences increases.

scholarly journals Symbolic Priors for RNN-based Semantic Parsing

2017 ◽  
Author(s):  
Chunyang Xiao ◽  
Marc Dymetman ◽  
Claire Gardent
Keyword(s):  
Neural Networks ◽  
Prior Knowledge ◽  
Recurrent Neural Networks ◽  
Logical Form ◽  
Finite State Automata ◽  
Semantic Parsing ◽  
Context Free Grammar ◽  
Finite State ◽  
Intersection Algorithm ◽  
Context Free

Seq2seq models based on Recurrent Neural Networks (RNNs) have recently received a lot of attention in the domain of Semantic Parsing. While in principle they can be trained directly on pairs (natural language utterances, logical forms), their performance is limited by the amount of available data. To alleviate this problem, we propose to exploit various sources of prior knowledge: the well-formedness of the logical forms is modeled by a weighted context-free grammar; the likelihood that certain entities present in the input utterance are also present in the logical form is modeled by weighted finite-state automata. The grammar and automata are combined together through an efficient intersection algorithm to form a soft guide (“background”) to the RNN.We test our method on an extension of the Overnight dataset and show that it not only strongly improves over an RNN baseline, but also outperforms non-RNN models based on rich sets of hand-crafted features.

Sign in / Sign up

Export Citation Format

Share Document