In two previous posts, I talked about linear systems and how we can easily characterize them by stimulating with specific classes of functions, such as impulses or sine functions. Here, I would like to talk about a third way of characterizing linear systems, that is commonly used: Responses to noise.

Let's start by how we can describe noise in the one dimensional case (the same concepts carry over to higher dimensions easily). Our noise would then be a sequence of random variables \(x=(x_t)\) (this notation means that \(x\) is a sequence and every element of \(x\) is labelled as \(x_t\)). In the simplest case, we would assume that all these random variables are stochastically independent and have the same distribution. Many papers refer to that as "iid" for "independent, identically distributed". Some engineering papers also talk about "white" noise, which sometimes requires that in addition all the random variables have mean 0 and variance 1. Often, we demand a noise sequence to be stationary, which means that the distribution of sequence element \(x_t\) is independent of the specific value of \(t\), but it could be dependent on other values in the sequence (e.g. \(x_{t-1}\) or \(x_{t+5}\)). We have so far focussed on time-invariant linear systems and stationarity is a noise characteristic that corresponds to time-invariance. The rest of this blog post will therefore always assume that the noise is stationary.

We can get a first idea of the characteristics of the noise sequence \(x\) by looking at its autocorrelation function. The autocorrelation function (of a stationary noise sequence) describes the correlation between values \(x_t\) and \(x_s\) from the noise sequence as a function of the distance \(t-s\) between them.

Example signals and autocorrelation functions

The figure shows two example signals with their corresponding autocorrelation functions. The top row shows a signal for which subsequent samples are uncorrelated—left side is the signal, right side is the autocorrelation function. Observe how the autocorrelation function is zero everywhere except for at lag \(t-s=0\). The lower row shows a signal with a clear correlation structure. We can see that neighbouring samples tend to be more similar than samples that are far apart. Looking at the autocorrelation structure, we can see even more: samples that are 4-5 steps apart tend to be negatively correlated and samples that are about 15 steps apart tend to be very slightly positively correlated! This is one of the strengths of the autocorrelation function. it allows us to quite quickly understand structure in random sequences.

An important insight is that the autocorrelation function is very similar to convolving the signal with its time reversed self! To see this, let's look at the autocorrelation function at an arbitrary lag \(t-s\). For a stationary signal, we can assume that the correlation between \(x_t\) and \(x_s\) is independent of the specific values of \(t\) and \(s\) and we can write the autocorrelation function as

$$ c_k = \frac{E[ (x_{t} - \mu)(x_{t-k}-\mu)]}{\sigma^2} = \frac{E(x_tx_{k-t}) - \mu^2}{\sigma^2}. $$

Here, we used \(E\) to denote expectation (practically, this is often replaced by the mean \(E_i(x_i)\mapsto\sum_i x_i\)). We note that \(E(x_tx_{k-t})\) is very similar to a convolution of \(x\) with a reversed version of \(x\), \(x_{reverse}=(x_{-t})\). Thus, if \(x\) has mean \(\mu = 0\) and standard deviation \(\sigma = 1\), the convolution of the signal with its time reversed self is exactly the same as the autocorrelation and \(\mu\neq 0\) or \(\sigma\neq 1\) result in constant offsets and scaling of the autocorrelation function.

However, we know what this convolution operation will do to the Fourier transform. Specifically, the Fourier transform of the autocorrelation is

$$ C_f = X_f \bar{X}_f = |X_f|^2, $$

Where \(\bar{X}\) is called the complex conjugate of \(X\) and corresponds to the Fourier transform of the time reversed version of \(X\).

We can use this insight in many ways to estimate the frequency response of a linear system. Here is the simplest strategy: Apply noise with known input power spectrum \(C\) to the system and measure the output power spectrum \(P\). Clearly, the output of the linear system will also be noise, but it will have a different power spectrum that will reflect how the linear system operated on the input noise. We can then estimate the frequency response of the linear spectrum as

$$ P_f = H_f^2 C_f \implies H_f = \sqrt{P_f/C_f}. $$

This strategy is illustrated in the following figure, using the noise from the first row of the previous figure as input and assuming that the second row showed the output (noise) of the linear system.

Example spectra and estimated frequency response

The left side shows the power spectrum of the of the input noise (blue) and the output noise (orange). The right side shows frequency response as estimated by the above equation as well as the true frequency response of the linear system that was used to generated the second noise signal form the first one. Given our very simple estimation technique, this is a fairly good estimate.