scholarly journals An Algorithm for Computing Average Mutual Information using Probability Distribution Smoothing

2019 ◽  
Vol 7 (9) ◽  
pp. 16-23
Author(s):  
Amr M.S. Goneid
Keyword(s):  
Probability Distribution ◽  
Mutual Information ◽  
Multivariate Time Series ◽  
Systematic Errors ◽  
Density Estimator ◽  
Histogram Method ◽  
Average Mutual Information ◽  
Frequency Space ◽  
Smoothing Functions ◽  
Present Algorithm

There is continuing interest in using Average Mutual Information (AMI) to quantify the pair-wise distance between dataset profiles. Among several algorithms used to find a numerical estimation of AMI, the histogram method is the most common since it provides simplicity and least cost. However, this algorithm is known to underestimate the computed entropies and to overestimate the resulting AMI.  Kernel Density Estimator (KDE)-based algorithms advanced to alleviate such systematic errors rely on bin-level smoothing. In the present work, we propose an alternative algorithm that uses smoothing on the probability distribution level. We consider several smoothing functions, both in the probability space and in its frequency space. An experimental approach is used to investigate the effect of such modification on the computation of both the entropy and the AMI. Results show that, to a significant extent, the present method is able to remove systematic errors in computing entropy and AMI. It is also shown that the present algorithm leads to better reconstruction of multivariate time series when AMI is used in conjunction with their independent components.

Capacity Analysis for Correlated Multi-Hop MIMO Channels under Colored Noise

2015 ◽  
Vol 1 ◽  
pp. 41
Author(s):  
Nguyen N. Tran ◽  
Ha X. Nguyen
Keyword(s):  
Computational Complexity ◽  
Mutual Information ◽  
Closed Form Solution ◽  
Optimal Solution ◽  
Form Solution ◽  
Maximization Problem ◽  
Capacity Analysis ◽  
Mimo Channels ◽  
Average Mutual Information ◽  
Channel Output

A capacity analysis for generally correlated wireless multi-hop multi-input multi-output (MIMO) channels is presented in this paper. The channel at each hop is spatially correlated, the source symbols are mutually correlated, and the additive Gaussian noises are colored. First, by invoking Karush-Kuhn-Tucker condition for the optimality of convex programming, we derive the optimal source symbol covariance for the maximum mutual information between the channel input and the channel output when having the full knowledge of channel at the transmitter. Secondly, we formulate the average mutual information maximization problem when having only the channel statistics at the transmitter. Since this problem is almost impossible to be solved analytically, the numerical interior-point-method is employed to obtain the optimal solution. Furthermore, to reduce the computational complexity, an asymptotic closed-form solution is derived by maximizing an upper bound of the objective function. Simulation results show that the average mutual information obtained by the asymptotic design is very closed to that obtained by the optimal design, while saving a huge computational complexity.

Input Vector Identification and System Model Construction by Average Mutual Information

2000 ◽  
Author(s):  
Paul B. Deignan ◽  
Peter H. Meckl ◽  
Matthew A. Franchek ◽  
Salim A. Jaliwala ◽  
George G. Zhu
Keyword(s):  
Mutual Information ◽  
System Model ◽  
Input Output ◽  
Output Data ◽  
Average Mutual Information ◽  
Virtual Sensors ◽  
Input Signals ◽  
Model Independent ◽  
Intelligent Model ◽  
Selection Of

Abstract A methodology for the intelligent, model-independent selection of an appropriate set of input signals for the system identification of an unknown process is demonstrated. In modeling this process, it is shown that the terms of a simple nonlinear polynomial model may also be determined through the analysis of the average mutual information between inputs and the output. Average mutual information can be thought of as a nonlinear correlation coefficient and can be calculated from input/output data alone. The methodology described here is especially applicable to the development of virtual sensors.

scholarly journals Estimating time delays for constructing dynamical networks

2014 ◽  
Vol 21 (5) ◽  
pp. 929-937 ◽  
Author(s):  
E. A. Martin ◽  
J. Davidsen
Keyword(s):  
Random Walk ◽  
Time Delays ◽  
Large Time ◽  
Multivariate Time Series ◽  
Time Lag ◽  
Time Lags ◽  
Climate Data ◽  
Dynamical Networks ◽  
Frequency Space ◽  
Cross Correlations

Abstract. Dynamical networks – networks inferred from multivariate time series – have been widely applied to climate data and beyond, resulting in new insights into the underlying dynamics. However, these inferred networks can suffer from biases that need to be accounted for to properly interpret the results. Here, we report on a previously unrecognized bias in the estimate of time delays between nodes in dynamical networks inferred from cross-correlations, a method often used. This bias results in the maximum correlation occurring disproportionately often at large time lags. This is of particular concern in dynamical networks where the large number of possible links necessitates finding the correct time lag in an automated way. We show that this bias can arise due to the similarity of the estimator to a random walk, and are able to map them to each other explicitly for some cases. For the random walk there is an analytical solution for the bias that is closely related to the famous Lévy arcsine distribution, which provides an upper bound in many other cases. Finally, we show that estimating the cross-correlation in frequency space effectively eliminates this bias. Reanalysing large lag links (from a climate network) with this method results in a distribution peaked near zero instead, as well as additional peaks at the originally assigned lag. Links that are reassigned smaller time lags tend to have a smaller distance between them, which indicates that the new time delays are physically reasonable.

scholarly journals Quantifying Total Influence between Variables with Information Theoretic and Machine Learning Techniques

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 141 ◽  
Author(s):  
Andrea Murari ◽  
Riccardo Rossi ◽  
Michele Lungaroni ◽  
Pasquale Gaudio ◽  
Michela Gelfusa
Keyword(s):  
Probability Distribution ◽  
Mutual Information ◽  
Information Quality ◽  
Pearson Correlation ◽  
Accurate Determination ◽  
Distribution Functions ◽  
Accurate Estimation ◽  
Computational Tools ◽  
Probability Distribution Functions ◽  
Total Influence

The increasingly sophisticated investigations of complex systems require more robust estimates of the correlations between the measured quantities. The traditional Pearson correlation coefficient is easy to calculate but sensitive only to linear correlations. The total influence between quantities is, therefore, often expressed in terms of the mutual information, which also takes into account the nonlinear effects but is not normalized. To compare data from different experiments, the information quality ratio is, therefore, in many cases, of easier interpretation. On the other hand, both mutual information and information quality ratio are always positive and, therefore, cannot provide information about the sign of the influence between quantities. Moreover, they require an accurate determination of the probability distribution functions of the variables involved. As the quality and amount of data available are not always sufficient to grant an accurate estimation of the probability distribution functions, it has been investigated whether neural computational tools can help and complement the aforementioned indicators. Specific encoders and autoencoders have been developed for the task of determining the total correlation between quantities related by a functional dependence, including information about the sign of their mutual influence. Both their accuracy and computational efficiencies have been addressed in detail, with extensive numerical tests using synthetic data. A careful analysis of the robustness against noise has also been performed. The neural computational tools typically outperform the traditional indicators in practically every respect.

Sign in / Sign up

Export Citation Format

Share Document