An information theoretic method to resolve millisecond-scale spike timing precision in a comprehensive motor program

Sensory inputs in nervous systems are often encoded at the millisecond scale in a temporally precise code. There is now a growing appreciation for the prevalence of precise timing encoding in motor systems. Animals from moths to birds control motor outputs using precise spike timing, but we largely do not know at what scale timing matters in these circuits due to the difficulty of recording a complete set of spike-resolved motor signals and relatively few methods for assessing spike timing precision. We introduce a method to estimate spike timing precision in motor circuits using continuous MI estimation at increasing levels of added uniform noise. This method can assess spike timing precision at fine scales for encoding rich motor output variation. We demonstrate the advantages of this approach compared to a previously established discrete information theoretic method of assessing spike timing precision. We use this method to analyze a data set of simultaneous turning (yaw) torque output and EMG recordings from the 10 primary muscles of Manduca sexta as tethered moths visually tracked a robotic flower moving with a 1 Hz sinusoidal trajectory. We know that all 10 muscles in this motor program encode the majority of information about yaw torque in spike timings, but we do not know whether individual muscles receive information encoded at different levels of precision. Using the continuous MI method, we demonstrate that the scale of temporal precision in all motor units in this insect flight circuit is at the sub-millisecond or millisecond-scale, with variation in precision scale present between muscle types. This method can be applied broadly to estimate spike timing precision in sensory and motor circuits in both invertebrates and vertebrates.

Weighted approximate Bayesian computation via Sanov’s theorem

We consider the problem of sample degeneracy in Approximate Bayesian Computation. It arises when proposed values of the parameters, once given as input to the generative model, rarely lead to simulations resembling the observed data and are hence discarded. Such “poor” parameter proposals do not contribute at all to the representation of the parameter’s posterior distribution. This leads to a very large number of required simulations and/or a waste of computational resources, as well as to distortions in the computed posterior distribution. To mitigate this problem, we propose an algorithm, referred to as the Large Deviations Weighted Approximate Bayesian Computation algorithm, where, via Sanov’s Theorem, strictly positive weights are computed for all proposed parameters, thus avoiding the rejection step altogether. In order to derive a computable asymptotic approximation from Sanov’s result, we adopt the information theoretic “method of types” formulation of the method of Large Deviations, thus restricting our attention to models for i.i.d. discrete random variables. Finally, we experimentally evaluate our method through a proof-of-concept implementation.

Renyi entropy driven hierarchical graph clustering

This article explores a graph clustering method that is derived from an information theoretic method that clusters points in ${{\mathbb{R}}^{n}}$ relying on Renyi entropy, which involves computing the usual Euclidean distance between these points. Two view points are adopted: (1) the graph to be clustered is first embedded into ${\mathbb{R}}^{d}$ for some dimension d so as to minimize the distortion of the embedding, then the resulting points are clustered, and (2) the graph is clustered directly, using as distance the shortest path distance for undirected graphs, and a variation of the Jaccard distance for directed graphs. In both cases, a hierarchical approach is adopted, where both the initial clustering and the agglomeration steps are computed using Renyi entropy derived evaluation functions. Numerical examples are provided to support the study, showing the consistency of both approaches (evaluated in terms of F-scores).

Information-Theoretic Estimation of Econometric Functions

This chapter introduces an information-theoretic approach to specify econometric functions as an alternative to avoid parametric assumptions. We investigate the performances of the information-theoretic method in estimating the regression (conditional mean) and response (derivative) functions. We have demonstrated that they are easy to implement and are advantageous over parametric models and nonparametric kernel techniques. For the implementation of our estimation method, a recursive integration procedure is introduced. An empirical illustration is also presented to study the Canadian earning function. The asymptotic properties of the regression and derivative functions are established. In addition, a test for normality is also proposed.

Information-Theoretical Criteria for Characterizing the Earliness of Time-Series Data

Biomedical signals constitute time-series that sustain machine learning techniques to achieve classification. These signals are complex with measurements of several features over, eventually, an extended period. Characterizing whether the data can anticipate prediction is an essential task in time-series mining. The ability to obtain information in advance by having early knowledge about a specific event may be of great utility in many areas. Early classification arises as an extension of the time-series classification problem, given the need to obtain a reliable prediction as soon as possible. In this work, we propose an information-theoretic method, named Multivariate Correlations for Early Classification (MCEC), to characterize the early classification opportunity of a time-series. Experimental validation is performed on synthetic and benchmark data, confirming the ability of the MCEC algorithm to perform a trade-off between accuracy and earliness in a wide-spectrum of time-series data, such as those collected from sensors, images, spectrographs, and electrocardiograms.

Internal and collective interpretation for improving human interpretability of multi-layered neural networks

The present paper aims to propose a new type of information-theoretic method to interpret the inference mechanism of neural networks. We interpret the internal inference mechanism for itself without any external methods such as symbolic or fuzzy rules. In addition, we make interpretation processes as stable as possible. This means that we interpret the inference mechanism, considering all internal representations, created by those different conditions and patterns. To make the internal interpretation possible, we try to compress multi-layered neural networks into the simplest ones without hidden layers. Then, the natural information loss in the process of compression is complemented by the introduction of a mutual information augmentation component. The method was applied to two data sets, namely, the glass data set and the pregnancy data set. In both data sets, information augmentation and compression methods could improve generalization performance. In addition, compressed or collective weights from the multi-layered networks tended to produce weights, ironically, similar to the linear correlation coefficients between inputs and targets, while the conventional methods such as the logistic regression analysis failed to do so.

Supposed Maximum Mutual Information for Improving Generalization and Interpretation of Multi-Layered Neural Networks

The present paper1 aims to propose a new type of information-theoretic method to maximize mutual information between inputs and outputs. The importance of mutual information in neural networks is well known, but the actual implementation of mutual information maximization has been quite difficult to undertake. In addition, mutual information has not extensively been used in neural networks, meaning that its applicability is very limited. To overcome the shortcoming of mutual information maximization, we present it here in a very simplified manner by supposing that mutual information is already maximized before learning, or at least at the beginning of learning. The method was applied to three data sets (crab data set, wholesale data set, and human resources data set) and examined in terms of generalization performance and connection weights. The results showed that by disentangling connection weights, maximizing mutual information made it possible to explicitly interpret the relations between inputs and outputs.