Chapter 8. A Constraint-Based Approach to Learning and Reasoning

Neural-symbolic models bridge the gap between sub-symbolic and symbolic approaches, both of which have significant limitations. Sub-symbolic approaches, like neural networks, require a large amount of labeled data to be successful, whereas symbolic approaches, like logic reasoners, require a small amount of prior domain knowledge but do not easily scale to large collections of data. This chapter presents a general approach to integrate learning and reasoning that is based on the translation of the available prior knowledge into an undirected graphical model. Potentials on the graphical model are designed to accommodate dependencies among random variables by means of a set of trainable functions, like those computed by neural networks. The resulting neural-symbolic framework can effectively leverage the training data, when available, while exploiting high-level logic reasoning in a certain domain of discourse. Although exact inference is intractable within this model, different tractable models can be derived by making different assumptions. In particular, three models are presented in this chapter: Semantic-Based Regularization, Deep Logic Models and Relational Neural Machines. Semantic-Based Regularization is a scalable neural-symbolic model, that does not adapt the parameters of the reasoner, under the assumption that the provided prior knowledge is correct and must be exactly satisfied. Deep Logic Models preserve the scalability of Semantic-Based Regularization, while providing a flexible exploitation of logic knowledge by co-training the parameters of the reasoner during the learning procedure. Finally, Relational Neural Machines provide the fundamental advantages of perfectly replicating the effectiveness of training from supervised data of standard deep architectures, and of preserving the same generality and expressive power of Markov Logic Networks, when considering pure reasoning on symbolic data. The bonding between learning and reasoning is very general as any (deep) learner can be adopted, and any output structure expressed via First-Order Logic can be integrated. However, exact inference within a Relational Neural Machine is still intractable, and different factorizations are discussed to increase the scalability of the approach.

Efficient Decomposition of Bayesian Networks With Non-graded Variables

Elicitation, estimation and exact inference in Bayesian Networks (BNs) are often difficult because the dimension of each Conditional Probability Table (CPT) grows exponentially with the increase in the number of parent variables. The Noisy-MAX decomposition has been proposed to break down a large CPT into several smaller CPTs exploiting the assumption of causal independence, i.e., absence of causal interaction among parent variables. In this way, the number of conditional probabilities to be elicited or estimated and the computational burden of the joint tree algorithm for exact inference are reduced. Unfortunately, the Noisy-MAX decomposition is suited to graded variables only, i.e., ordinal variables with the lowest state as reference, but real-world applications of BNs may also involve a number of non-graded variables, like the ones with reference state in the middle of the sample space (double-graded variables) and with two or more unordered non-reference states (multi-valued nominal variables). In this paper, we propose the causal independence decomposition, which includes the Noisy-MAX and two generalizations suited to double-graded and multi-valued nominal variables. While the general definition of BN implicitly assumes the presence of all the possible causal interactions, our proposal is based on causal independence, and causal interaction is a feature that can be added upon need. The impact of our proposal is investigated on a published BN for the diagnosis of acute cardiopulmonary diseases.

PROBABILISTIC INFERENCE IN BAYESIAN INSURANCE NETWORK

In this paper, the technique to solve the prediction problem of reparation of the financial losses caused by a road traffic accident is solved. Exact inference is represented using the Sum-Product Variable Elimination algorithm, Sum-Product Variable Elimination algorithm for computing conditional probabilities, Max-Product Variable Elimination algorithm for MAP, Max-Sum-Product Variable Elimination algorithm for marginal MAP. Reasoning patterns are presented graphically and descriptively.

An EM Algorithm for Capsule Regression

We investigate a latent variable model for multinomial classification inspired by recent capsule architectures for visual object recognition (Sabour, Frosst, & Hinton, 2017 ). Capsule architectures use vectors of hidden unit activities to encode the pose of visual objects in an image, and they use the lengths of these vectors to encode the probabilities that objects are present. Probabilities from different capsules can also be propagated through deep multilayer networks to model the part-whole relationships of more complex objects. Notwithstanding the promise of these networks, there still remains much to understand about capsules as primitive computing elements in their own right. In this letter, we study the problem of capsule regression—a higher-dimensional analog of logistic, probit, and softmax regression in which class probabilities are derived from vectors of competing magnitude. To start, we propose a simple capsule architecture for multinomial classification: the architecture has one capsule per class, and each capsule uses a weight matrix to compute the vector of hidden unit activities for patterns it seeks to recognize. Next, we show how to model these hidden unit activities as latent variables, and we use a squashing nonlinearity to convert their magnitudes as vectors into normalized probabilities for multinomial classification. When different capsules compete to recognize the same pattern, the squashing nonlinearity induces nongaussian terms in the posterior distribution over their latent variables. Nevertheless, we show that exact inference remains tractable and use an expectation-maximization procedure to derive least-squares updates for each capsule's weight matrix. We also present experimental results to demonstrate how these ideas work in practice.

Exact Inference for an Exponential Parameter under Generalized Adaptive Progressive Hybrid Censored Competing Risks Data

It is known that the lifetimes of items may not be recorded exactly. In addition, it is known that more than one risk factor (RisF) may be present at the same time. In this paper, we discuss exact likelihood inference for competing risk model (CoRiM) with generalized adaptive progressive hybrid censored exponential data. We derive the conditional moment generating function (ConMGF) of the maximum likelihood estimators of scale parameters of exponential distribution (ExpD) and the resulting lower confidence bound under generalized adaptive progressive hybrid censoring scheme (GeAdPHCS). From the example data, it can be seen that the PDF of MLE is almost symmetrical.

Best-First Beam Search

Decoding for many NLP tasks requires an effective heuristic algorithm for approximating exact search because the problem of searching the full output space is often intractable, or impractical in many settings. The default algorithm for this job is beam search—a pruned version of breadth-first search. Quite surprisingly, beam search often returns better results than exact inference due to beneficial search bias for NLP tasks. In this work, we show that the standard implementation of beam search can be made up to 10x faster in practice. Our method assumes that the scoring function is monotonic in the sequence length, which allows us to safely prune hypotheses that cannot be in the final set of hypotheses early on. We devise effective monotonic approximations to popular nonmonontic scoring functions, including length normalization and mutual information decoding. Lastly, we propose a memory-reduced variant of best-first beam search, which has a similar beneficial search bias in terms of downstream performance, but runs in a fraction of the time.