expected value and variance explained
The last lesson said a distribution is the whole shape of how probability spreads. Carrying an entire distribution around is heavy, though. In practice we summarize a distribution with just two numbers: where it is centered, and how widely it spreads. These two are the expected value and the variance. The expected value E[X] is the center of the distribution, more precisely its balance point. Think of the distribution as weights on bars, and the expected value is the spot where a fulcrum would make those bars balance exactly. It marks, as a single point, "on average, values near here come up." The variance is the size of the spread: how far, on average, the values are scattered from that center. With the same center, values huddled together and values spread wide are completely different situations. In this lesson you handle the expected value as a balance point and the variance as an average of squared distances, and learn by feel what the two summarizing numbers mean.
First, feel the expected value on a balance beam. Four weights of different sizes sit on the gold beam, and the number on each one is its weight. The fulcrum where the beam balances exactly is the balance point. Drag a weight and the gold fulcrum below follows at once to restore the balance. You can see the fulcrum lean toward the heavy weights. Send a light weight as far as you like and the fulcrum barely moves, but move a heavy weight a little and the fulcrum follows a lot. That is exactly the expected value. In probability, the weight is the probability of a value and the position is the value itself. The more probable a value, the harder it pulls the average. So the expected value is the center measured with probability as weight, the single point at which the distribution balances.
How do we compute that balance point? It is a weighted average. Not the ordinary average that adds up the values and divides by how many there are, but one that multiplies each value by its probability and adds them: E[X] = Σ x·p(x). Take a die. A fair die gives 1 through 6, each with probability 16, so the average is 3.5. A loaded die whose weight leans to one side is another story. Drag a bar's height to raise the probability of a particular face; that face comes up more often, and the blue triangle below, the average E[X], is pulled toward it. The more probable a value, the more it contributes to the sum. Each value gets a say proportional to its probability, and they play tug-of-war over where the average sits. Translate the balance beam of the previous block into numbers and you get exactly this weighted sum: weight = probability, position = value, balance point = Σ x·p(x).
Once you know the center, measure the spread: the variance. The key question is how far, on average, the values lie from the mean. Simply averaging the distances will not do, because values above the mean give positive distances and values below give negative ones, and they cancel out to 0. So square the distances. Squaring removes the sign and makes everything positive. Drag the blue points apart and, for each one, a square is drawn whose side is the distance to the mean marked by the gold dashed line; the area of that square is the squared distance. The variance is the average of all these square areas. When the points huddle near the mean, the squares are small and the variance is small; when they scatter widely, the squares grow and the variance shoots up. Because the distances are squared, a far-off point is penalized much more heavily. That makes the variance a strict measure of spread, sensitive to extreme values.
The variance has one inconvenience. Since the distances were squared, the units get squared too. If height is measured in cm, the variance is in cm², a unit of area. A spread in height of 25 square centimeters does not mean much to anyone. So the square is undone with a square root, and that is the standard deviation. Standard deviation = √variance, and its unit returns to cm, the same as the original values. A spread in height of 5cm is instantly clear. Drag σ along the track below and you see that the standard deviation σ is the natural width of the bell curve, while the blue ruler above the curve marks the stretch of σ on each side of the center. σ is also the distance from the peak to where the curve's shoulder bends. With a small σ the curve is narrow and sharp; with a large σ it is wide and flat. In short, the variance σ² is a squared-unit quantity that is convenient to compute, and the standard deviation σ is the everyday width that lets you feel, in the original units, how scattered things really are.
Finally, see how these two numbers change when you tamper with the values. Changing X to aX+b is very common in unit conversion or calibration, as when converting Celsius to Fahrenheit or meters to feet. Move a and b with the two tracks below, and the transformed gold distribution is drawn next to the original gray dashed one. First, b adds the same amount to every value, so it slides the whole distribution sideways without changing its shape. The mean moves by b, but the spread does not change at all, because everything moved together and the distances between values stay the same. a, on the other hand, stretches or shrinks. The mean is multiplied by a and the width of the curve by a as well. So the standard deviation becomes |a| times as large, and the variance, a squared distance, becomes a² times as large. As formulas: E[aX+b]=aE[X]+b and Var(aX+b)=a²Var(X). Adding moves only the center and leaves the spread alone; multiplying stretches both center and spread, but the variance grows with the square. Thanks to this rule, whether you change units or standardize data, you can work out in your head how the statistics will follow along.