seegongsik
CM · Noise and information

Probability and Random Signals

No one can say in advance what value a single noise sample will take. Yet collect many of them into a histogram and they always gather into the same bell shape. This unit shows that individual values may be random while the crowd is predictable, and that its distribution is pinned down by two numbers, the mean and the variance.

The more samples, the clearer the bell

The blue bars are a histogram that stacks the samples by size, and the gold curve is the theoretical normal distribution. Switch the sample-count tab from few to many: the bars, ragged when there are only a handful, fit the gold curve more and more closely as samples pile up. The faint vertical line in the middle marks the mean.

Sample count and convergence
N = 16 → μ = 0, σ² = 1
Rough estimate

One value is unknown

No one can predict in advance what the next noise sample will be. Any attempt to pin noise down to a single value is bound to fail, yet being random does not mean being lawless. A given value never comes up twice, but how often each range of values appears is remarkably regular.

Pile them up and a bell appears

Collect a few hundred samples and stack them into bins by size, and a bell shape appears, tall in the middle and low at both ends. With few samples the bars are rough; as the count grows they approach a smooth curve. That curve is the normal, or Gaussian, distribution. A quantity built from many small random fluctuations added together, like thermal noise, almost always takes this bell shape, and the samples in the plot are themselves made by adding three evenly spread random numbers.

Observe∫ p(x) dx = 1
The total area under a probability density curve is 1.
Chooseμ = E[X] = ?
Noise has a mean of 0, so it leans to neither side.

Mean and variance, two numbers suffice

Two numbers fix the bell curve completely. The mean μ tells where the bell is centered, and the variance σ² tells how widely it spreads. Noise usually has a mean of 0, so it leans to neither side, and its variance equals its average power. The probability that a value lands in a given interval is the area under the curve over that interval, and the total area under the curve is always 1. Without knowing any single sample, knowing these two numbers puts the whole crowd in hand.

Fill inσ² = E[(X ? μ)²]
The variance is the mean of the squared deviation from the mean.
On your ownPavg = ?
Average power is exactly the variance σ².

Back to the first screen

With sixteen samples, the bars were ragged and missed the gold bell curve, but at nine hundred the histogram fit the curve with no gaps. All the while, each individual sample remained just as unpredictable. One thing to remember: a single random value cannot be known, yet the crowd always forms the same bell, and that bell is held completely by just two numbers, the mean that sets its center and the variance that sets its spread.

The values of a random signal cannot be predicted one at a time, but as a crowd they follow a probability distribution. Thermal noise follows the bell-shaped normal (Gaussian) distribution, fixed entirely by two numbers: the mean μ, which sets the center (0 for noise), and the variance σ² (= average power), which sets the spread. The probability of an interval is the area under the density curve over it, and the total area is 1. This statistical description is what lets noise be handled by calculation.

In the next unit

Noise can now be measured as a distribution and its strength as a variance. The ratio of signal power to noise power, the SNR, is already in hand. The next unit is Shannon's channel capacity, which ties all of this together. Given only the bandwidth and the SNR, the ceiling on the bit rate that can cross the channel without error is set at once by C = B log₂(1 + SNR).