Nonlinear and non-Gaussian signal processing

Linear, time-invariant (LTI) Gaussian DSP, has substantial mathematical conveniences that make it valuable in practical DSP applications and machine learning. When the signal really is generated by such an LTI-Gaussian model then this kind of processing is optimal from a statistical point of view. However, there are substantial limitations to the use of these techniques when we cannot guarantee that the assumptions of linearity, time-invariance and Gaussianity hold. In particular, signals that exhibit jumps or significant non-Gaussian outliers cause substantial adverse effects such as Gibb's phenomena in LTI filter outputs, and nonstationary signals cannot be compactly represented in the Fourier domain. In practice, many real signals show such phenomena to a greater or lesser degree, so it is important to have a `toolkit' of DSP methods that are effective in many situations. This chapter is dedicated to exploring the use of the statistical machine learning concepts in DSP.

Statistical machine learning

This chapter describes in detail how the main techniques of statistical machine learning can be constructed from the components described in earlier chapters. It presents these concepts in a way which demonstrates how these techniques can be viewed as special cases of a more general probabilistic model which we fit to some data.

Mathematical foundations

Statistical machine learning and signal processing are topics in applied mathematics, which are based upon many abstract mathematical concepts. Defining these concepts clearly is the most important first step in this book. The purpose of this chapter is to introduce these foundational mathematical concepts. It also justifies the statement that much of the art of statistical machine learning as applied to signal processing, lies in the choice of convenient mathematical models that happen to be useful in practice. Convenient in this context means that the algebraic consequences of the choice of mathematical modeling assumptions are in some sense manageable. The seeds of this manageability are the elementary mathematical concepts upon which the subject is built.

Random sampling

This chapter provides an overview of generating samples from random variables with a given (joint) distribution, and using these samples to find quantities of interest from digital signals. This task plays a fundamental role in many problems in statistical machine learning and signal processing. For example, effectively simulating the behaviour of the statistical model offers a viable alternative to optimization problems arising from some models for signals with large numbers of variables.

Optimization

Decision-making under uncertainty is a central topic of this book. A common scenario is the following: data is recorded from some (digital) sensor device, and we know (or assume) that there is some “underlying” signal contained in this data, which is obscured by noise. The goal is to extract this signal, but the noise causes this task to be impossible: we can never know the actual underlying signal. We must make mathematical assumptions that make this taskp possible at all. Uncertainty is formalized through the mathematical machinery of probability, and decisions are made that find the optimal choices under these assumptions. This chapter explores the main methods by which these optimal choices are made in DSP and machine learning.

Nonparametric Bayesian machine learning and signal processing

We have seen that stochastic processes play an important foundational role in a wide range of methods in DSP. For example, we treat a discrete-time signal as a Gaussian process, and thereby obtain many mathematically simplified algorithms, particularly based on the power spectral density. At the same time, in machine learning, it has generally been observed that nonparametric methods outperform parametric methods in terms of predictive accuracy since they can adapt to data with arbitrary complexity. However, these techniques are not Bayesian so we are unable to do important inferential procedures such as draw samples from the underlying probabilistic model or compute posterior confidence intervals. But, Bayesian models are often only mathematically tractable if parametric, with the corresponding loss of predictive accuracy. An alternative, discussed in this section, is to extend the mathematical tractability of stochastic processes to Bayesian methods. This leads to so-called Bayesian nonparametrics exemplified by techniques such as Gaussian process regression and Dirichlet process mixture modelling that have been shown to be extremely useful in practical DSP and machine learning applications.

Statistical modelling and inference

The modern view of statistical machine learning and signal processing is that the central task is one of finding good probabilistic models for the joint distribution over all the variables in the problem. We can then make `queries' of this model, also known as inferences, to determine optimal parameter values or signals. Hence, the importance of statistical methods to this book cannot be overstated. This chapter is an in-depth exploration of what this probabilistic modeling entails, the origins of the concepts involved, how to perform inferences and how to test the quality of a model produced this way.

Discrete signals: sampling, quantization and coding

Digital signal processing and machine learning require digital data which can be processed by algorithms on computer. However, most of the real-world signals that we observe are real numbers, occurring at real time values. This means that it is impossible in practice to store these signals on a computer and we must find some approximate signal representation which is amenable to finite, digital storage. This chapter describes the main methods which are used in practice to solve this representation problem.

Linear-Gaussian systems and signal processing

Linear systems theory, based on the mathematics of vector spaces, is the backbone of all “classical” DSP and a large part of statistical machine learning. The basic idea -- that linear algebra applied to a signal can of substantial practical value -- has counterparts in many areas of science and technology. In other areas of science and engineering, linear algebra is often justified by the fact that it is often an excellent model for real-world systems. For example, in acoustics the theory of (linear) wave propagation emerges from the concept of linearization of small pressure disturbances about the equilibrium pressure in classical fluid dynamics. Similarly, the theory of electromagnetic waves is also linear. Except when a signal emerges from a justifiably linear system, in DSP and machine learning we do not have any particular correspondence to reality to back up the choice of linearity. However, the mathematics of vector spaces, particularly when applied to systems which are time-invariant and jointly Gaussian, is highly tractable, elegant and immensely useful.

Probabilistic graphical models

Statistical machine learning and statistical DSP are built on the foundations of probability theory and random variables. Different techniques encode different dependency structure between these variables. This structure leads to specific algorithms for inference and estimation. Many common dependency structures emerge naturally in this way, as a result, there are many common patterns of inference and estimation that suggest general algorithms for this purpose. So, it becomes important to formalize these algorithms; this is the purpose of this chapter. These general algorithms can often lead to substantial computational savings over more brute-force approaches, another benefit that comes from studying the structure of these models in the abstract.