CM · Noise and information

Probability and Random Signals

A single noise sample cannot be known in advance. Yet collect many and the histogram always converges to the same bell shape. Learn that while each value is random the crowd is predictable, and that the distribution is pinned by just two numbers, mean and variance.

The more samples, the clearer the bell

Switch the sample count from few to many. With only a handful the bars are jagged and miss the bell curve; collect many and the histogram snaps onto the gold normal curve.

Sample count and convergence

N = 16 → μ = 0, σ² = 1

Rough estimate

One value is unknown

No one can say in advance what the next noise sample will be. Any attempt to pin noise down as a single value always fails. But random does not mean lawless. You cannot get the same value twice on demand, yet how often each value shows up is remarkably regular.

Pile them up and a bell appears

Gather a few hundred samples and stack them into bins by size, and a bell shape appears — high in the middle, low at the ends. The fewer the samples the rougher the bars; the more you collect the closer they come to a smooth curve. That curve is the normal, or Gaussian, distribution. Anything made of many small random fluctuations added together — like thermal noise — almost always wears this bell shape.

Observe∫ p(x) dx = 1

The total area of a density is 1.

Chooseμ = E[X] = ?

Noise has mean 0 — it leans to neither side.

Mean and variance, two numbers suffice

The bell is fixed completely by two numbers. The mean μ says where the bell is centered, the variance σ² how widely it spreads. Noise usually has mean 0, so it leans to neither side, and its variance equals the average power. The probability that a value lands in some range is the area under the curve above that range, and the total area under the curve is always 1. Knowing one sample tells you nothing, but knowing these two numbers puts the whole crowd in your hand.

Fill inσ² = E[(X ? μ)²]

Variance is the mean square of the deviation from the mean.

On your ownP_avg = ?

The average power is the variance σ².

Back to the first screen

With sixteen samples the bars were jagged and missed the gold bell, but at nine hundred the histogram fit the curve seamlessly — even though each individual sample was still unpredictable. The one thing to hear in probability is this: a single random value cannot be known, yet the crowd always forms the same bell, and that bell is held completely by two numbers, the mean that sets its center and the variance that sets its spread.

The values of a random signal are individually unpredictable but collectively follow a probability distribution. Thermal noise follows the bell-shaped normal (Gaussian) distribution, fixed completely by two numbers: the mean μ that sets the center (0 for noise) and the variance σ² that sets the spread (= average power). The probability of a range is the area under the density curve over it, and the total area is 1. This statistical description is what makes noise tractable.

In the next unit

Now noise can be measured as a distribution, its strength as a variance, and the ratio of signal to noise power, the SNR, is in hand too. Next is what binds them into one: Shannon’s channel capacity. Given only bandwidth and SNR, the absolute ceiling on the error-free bit rate of that channel is set at a stroke by C = B log₂(1 + SNR).