Tightening Mutual Information Based Bounds on Generalization Error

Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.

Download Full-text

The maximum mutual information without coding for binary quantum-state signals

Journal of Modern Optics ◽

10.1080/095003498151960 ◽

1998 ◽

Vol 45 (2) ◽

pp. 269-282 ◽

Cited By ~ 2

Author(s):

MASAO OSAKI, OSAMU HIROTA MASASHI BAN

Keyword(s):

Mutual Information ◽

Quantum State ◽

Maximum Mutual Information

Download Full-text

A mutual information‐based likelihood function for particle filter flood extent assimilation

Water Resources Research ◽

10.1029/2020wr027859 ◽

2021 ◽

Author(s):

Antara Dasgupta ◽

Renaud Hostache ◽

RAAJ Ramasankaran ◽

Guy J.‐P Schumann ◽

Stefania Grimaldi ◽

...

Keyword(s):

Mutual Information ◽

Particle Filter ◽

Likelihood Function

Download Full-text

Nonlinear Pattern Analysis of Ventricular Premature Beats by Mutual Information

Methods of Information in Medicine ◽

10.1055/s-0038-1636872 ◽

1997 ◽

Vol 36 (04/05) ◽

pp. 257-260 ◽

Cited By ~ 1

Author(s):

H. Saitoh ◽

T. Yokoshima ◽

H. Kishida ◽

H. Hayakawa ◽

R. J. Cohen ◽

...

Keyword(s):

Mutual Information ◽

Temporal Pattern ◽

Pattern Analysis ◽

Autonomic Activity ◽

Correlation Measure ◽

Ventricular Premature Beats ◽

Risk Of Mortality ◽

Premature Beats ◽

Information Values ◽

Nonlinear Pattern

Abstract:The frequency of ventricular premature beats (VPBs) has been related to the risk of mortality. However, little is known about the temporal pattern of occurrence of VPBs and its relationship to autonomic activity. Hence, we applied a general correlation measure, mutual information, to quantify how VPBs are generated over time. We also used mutual information to determine the correlation between VPB production and heart rate in order to evaluate effects of autonomic activity on VPB production. We examined twenty subjects with more than 3000 VPBs/day and simulated ran-( dom time series of VPB occurrence. We found that mutual information values could be used to characterize quantitatively the temporal patterns of VPB generation. Our data suggest that VPB production is not random and VPBs generated with a higher value of mutual information may be more greatly affected by autonomic activity.

Download Full-text

Data Screening Methods – Application to Differential Diagnosis in Pancreatic Pathology from Radiological Signs

Methods of Information in Medicine ◽

10.1055/s-0038-1636606 ◽

1978 ◽

Vol 17 (01) ◽

pp. 36-40 ◽

Cited By ~ 4

Author(s):

J.-P. Durbec ◽

Jaqueline Cornée ◽

P. Berthezene

Keyword(s):

Differential Diagnosis ◽

Mutual Information ◽

Data Processing ◽

Screening Methods ◽

Automatic Data ◽

Chronic Calcifying Pancreatitis ◽

Number Of Patients ◽

Calcifying Pancreatitis ◽

Pancreatic Pathology ◽

Data Screening

The practice of systematic examinations in hospitals and the increasing development of automatic data processing permits the storing of a great deal of information about a large number of patients belonging to different diagnosis groups.To predict or to characterize these diagnosis groups some descriptors are particularly useful, others carry no information. Data screening based on the properties of mutual information and on the log cross products ratios in contingency tables is developed. The most useful descriptors are selected. For each one the characterized groups are specified.This approach has been performed on a set of binary (presence—absence) radiological variables. Four diagnoses groups are concerned: cancer of pancreas, chronic calcifying pancreatitis, non-calcifying pancreatitis and probable pancreatitis. Only twenty of the three hundred and forty initial radiological variables are selected. The presence of each corresponding sign is associated with one or more diagnosis groups.

Download Full-text