scholarly journals Mutual Information, the Linear Prediction Model, and CELP Voice Codecs

Information ◽  
2019 ◽  
Vol 10 (5) ◽  
pp. 179 ◽  
Author(s):  
Jerry Gibson
Keyword(s):  
Mutual Information ◽  
Linear Prediction ◽  
Rate Distortion ◽  
Source Model ◽  
Rate Distortion Theory ◽  
Input Source ◽  
Distortion Measure ◽  
Adaptive Codebook ◽  
Log Ratio ◽  
Linear Prediction Model

We write the mutual information between an input speech utterance and its reconstruction by a code-excited linear prediction (CELP) codec in terms of the mutual information between the input speech and the contributions due to the short-term predictor, the adaptive codebook, and the fixed codebook. We then show that a recently introduced quantity, the log ratio of entropy powers, can be used to estimate these mutual informations in terms of bits/sample. A key result is that for many common distributions and for Gaussian autoregressive processes, the entropy powers in the ratio can be replaced by the corresponding minimum mean squared errors. We provide examples of estimating CELP codec performance using the new results and compare these to the performance of the adaptive multirate (AMR) codec and other CELP codecs. Similar to rate distortion theory, this method only needs the input source model and the appropriate distortion measure.

scholarly journals Mutual Information, the Linear Prediction Model, and CELP Voice Codecs

2019 ◽  
Author(s):  
Jerry Gibson
Keyword(s):  
Mutual Information ◽  
Linear Prediction ◽  
Rate Distortion ◽  
Source Model ◽  
Rate Distortion Theory ◽  
Input Source ◽  
Distortion Measure ◽  
Adaptive Codebook ◽  
Log Ratio ◽  
Linear Prediction Model

We write the mutual information between an input speech utterance and its reconstruction by a Code-Excited Linear Prediction (CELP) codec in terms of the mutual information between the input speech and the contributions due to the short term predictor, the adaptive codebook, and the fixed codebook. We then show that a recently introduced quantity, the log ratio of entropy powers, can be used to estimate these mutual informations in terms of bits/sample. A key result is that for many common distributions and for Gaussian autoregressive processes, the entropy powers in the ratio can be replaced by the corresponding minimum mean squared errors. We provide examples of estimating CELP codec performance using the new results and compare to the performance of the AMR codec and other CELP codecs. Similar to rate distortion theory, this method only needs the input source model and the appropriate distortion measure.

A Reward-Maximizing Spiking Neuron as a Bounded Rational Decision Maker

2015 ◽  
Vol 27 (8) ◽  
pp. 1686-1720 ◽  
Author(s):  
Felix Leibfried ◽  
Daniel A. Braun
Keyword(s):  
Mutual Information ◽  
Decision Maker ◽  
Sensory Input ◽  
Rate Distortion ◽  
Relevant Information ◽  
Motor Output ◽  
Spiking Neuron ◽  
Rational Decision ◽  
Rate Distortion Theory ◽  
The Many

Rate distortion theory describes how to communicate relevant information most efficiently over a channel with limited capacity. One of the many applications of rate distortion theory is bounded rational decision making, where decision makers are modeled as information channels that transform sensory input into motor output under the constraint that their channel capacity is limited. Such a bounded rational decision maker can be thought to optimize an objective function that trades off the decision maker’s utility or cumulative reward against the information processing cost measured by the mutual information between sensory input and motor output. In this study, we interpret a spiking neuron as a bounded rational decision maker that aims to maximize its expected reward under the computational constraint that the mutual information between the neuron’s input and output is upper bounded. This abstract computational constraint translates into a penalization of the deviation between the neuron’s instantaneous and average firing behavior. We derive a synaptic weight update rule for such a rate distortion optimizing neuron and show in simulations that the neuron efficiently extracts reward-relevant information from the input by trading off its synaptic strengths against the collected reward.

scholarly journals Entropy Power, Autoregressive Models, and Mutual Information

Entropy ◽  
2018 ◽  
Vol 20 (10) ◽  
pp. 750 ◽  
Author(s):  
Jerry Gibson
Keyword(s):  
Signal Processing ◽  
Mutual Information ◽  
Speech Processing ◽  
Linear Prediction ◽  
Seismic Signal ◽  
Autoregressive Processes ◽  
Differential Entropy ◽  
Squared Prediction Error ◽  
Seismic Signal Processing ◽  
Log Ratio

Autoregressive processes play a major role in speech processing (linear prediction), seismic signal processing, biological signal processing, and many other applications. We consider the quantity defined by Shannon in 1948, the entropy rate power, and show that the log ratio of entropy powers equals the difference in the differential entropy of the two processes. Furthermore, we use the log ratio of entropy powers to analyze the change in mutual information as the model order is increased for autoregressive processes. We examine when we can substitute the minimum mean squared prediction error for the entropy power in the log ratio of entropy powers, thus greatly simplifying the calculations to obtain the differential entropy and the change in mutual information and therefore increasing the utility of the approach. Applications to speech processing and coding are given and potential applications to seismic signal processing, EEG classification, and ECG classification are described.

scholarly journals Research a New Tradeoff Method Between Privacy and Utility in Differential privacy

2021 ◽  
Author(s):  
panjun sun
Keyword(s):  
Mutual Information ◽  
Privacy Protection ◽  
Differential Privacy ◽  
Rate Distortion ◽  
Information Privacy ◽  
Distortion Function ◽  
Approximate Algorithm ◽  
Data Utility ◽  
Rate Distortion Theory ◽  
Privacy Leakage

Abstract The solution of the contradiction between privacy protection and data utility is a research hotspot in the field of privacy protection. Aiming at the problem of tradeoff between privacy and utility in the scenario of differential privacy offline data release, the optimal differential privacy mechanism is studied by using the rate distortion theory. Firstly, based on Shannon communication theory, the noise channel model of differential privacy is abstracted, and the mutual information and the distortion function is used to measure the privacy and utility of data publishing, and the optimization model based on rate distortion theory is constructed. Secondly, considering the influence of associated auxiliary background knowledge on mutual information privacy leakage, a mutual information privacy measure based on joint events is proposed, and a minimum privacy leakage model is proposed by modifying the rate distortion function. Finally, aiming at the difficulty in solving the Lagrange multiplier method, an approximate algorithm for solving the mutual information privacy optimization channel mechanism is proposed based on the alternating iterative method. The effectiveness of the proposed iterative approximation method is verified by experimental simulation. At the same time, the experimental results show that the proposed method reduces the mutual information privacy leakage under the condition of limited distortion, and improves the data utility under the same privacy tolerance

scholarly journals A non-linear prediction model for ageing foetal and neonatal sheep reveals basic issues in early neolithic husbandry

2021 ◽  
pp. 105344
Author(s):  
Nadja Pöllath ◽  
Ricardo García-González ◽  
Sevag Kevork ◽  
Ursula Mutze ◽  
Michaela I. Zimmermann ◽  
...  
Keyword(s):  
Prediction Model ◽  
Linear Prediction ◽  
Early Neolithic ◽  
Non Linear ◽  
Linear Prediction Model

Multi-day flow forecasting using the Kalman filter

1991 ◽  
Vol 18 (2) ◽  
pp. 320-327 ◽  
Author(s):  
Murray A. Fitch ◽  
Edward A. McBean
Keyword(s):  
Kalman Filter ◽  
Prediction Model ◽  
Linear Prediction ◽  
Measured Data ◽  
System Model ◽  
Parameter Estimates ◽  
Model Parameters ◽  
Forecasting Performance ◽  
Optimal Forecasting ◽  
Linear Prediction Model

A model is developed for the prediction of river flows resulting from combined snowmelt and precipitation. The model employs a Kalman filter to reflect uncertainty both in the measured data and in the system model parameters. The forecasting algorithm is used to develop multi-day forecasts for the Sturgeon River, Ontario. The algorithm is shown to develop good 1-day and 2-day ahead forecasts, but the linear prediction model is found inadequate for longer-term forecasts. Good initial parameter estimates are shown to be essential for optimal forecasting performance. Key words: Kalman filter, streamflow forecast, multi-day, streamflow, Sturgeon River, MISP algorithm.

Sign in / Sign up

Export Citation Format

Share Document