Assume the Null, Then Measure the Surprise
Run an experiment and a new drug looks effective, or a new algorithm looks faster, yet that alone is not enough to be sure. It could have come out that way by chance. So statistics works like a courtroom. First we set up a skeptical default, the null hypothesis H0, that says there is no effect. Then we ask: if H0 really were true, how rare would it be to see the data we just saw? If the probability of a result this extreme under H0 is very small, holding on to H0 becomes awkward, so we reject H0 and accept the alternative H1. The key is that the data does not prove what is true; it measures whether the result is too surprising to explain by chance. Earlier we saw how the sample mean scatters (the sampling distribution) and how that scatter pins down a range for the true value (the confidence interval). Now we use the same bell as a ruler for surprise.
The first step is to state the two hypotheses cleanly. The null H0 is always an equality, pinning down "no difference" (for example, mean = μ0). The alternative H1 is the claim we want to show. H1 can take three shapes: greater on one side, less on one side, or simply different (two-sided). Move the slider to push the center of H1 away from H0, and use the toggle to pick the direction. The shaded tail of the null bell is where a surprising result would fall: both tails for two-sided, a single tail for a one-sided test. Which side you suspect must be decided before looking at the data, to keep the test fair.
Once the hypotheses are set, we summarize the data into a single number and place it on the null bell. That number is the test statistic. Here we standardize the sample mean x̄: z = (x̄ − μ0) / SE. Its meaning is simple: how many standard errors the observed mean sits away from the null value μ0. SE is the width of the sampling distribution from the previous lesson, namely σ/√n. Move the slider to change the observed mean and z slides along the standard-normal bell. A z near 0 fits H0 well; a large z (like 2 or 3) lands in a rare spot far from the center. By converting data in any units into a single z, everything can be compared on the same standard bell.
Now we turn surprise into a number. The p-value is the probability, assuming H0 is true, of getting a result as extreme as the one observed or more so. In the picture it is the area in the tail of the null bell beyond the observed statistic. Push the observed value farther out with the slider and the tail thins and p shrinks. A small p means that in a world where H0 is true, data like this rarely occurs, which is why it is more surprising. One thing must be stressed: the p-value is not the probability that H0 is true. It is a conditional probability that assumes H0 is true in the first place. Confuse that direction and you read the statistics completely wrong.
How small is small enough? The line drawn in advance is the significance level α. Common choices are 0.10, 0.05, 0.01. α is the area of the rejection region in the tail of the null bell, extreme enough to abandon H0, and its edge is the critical value z*. The rule is simple: if the observed statistic crosses the critical value (that is, if p < α) we reject H0; if it does not, we fail to reject. Change α with the toggle and the critical line moves; slide the observed value across that line and watch the verdict flip. The smaller α is, the higher the bar for rejection, so H0 survives unless the data is quite surprising. So α is the knob that sets, in advance, the rate of false alarms we are willing to accept.
Two mistakes hide inside the verdict. A Type I error is rejecting H0 when H0 is actually true, a false alarm (false positive). Its probability is exactly α, the area beyond the critical value under the null bell. A Type II error is failing to abandon H0 when H1 is actually true, a miss (false negative). Its probability is β, the area inside the critical value under the alternative bell. Use the toggle to see the two errors in turn, and the slider to increase the true effect size. The larger the effect, the farther the H1 bell moves to the right, so β shrinks and the power (1 − β) grows. Power is the probability of correctly catching a real effect when there is one. Lowering α reduces false alarms but raises β and lets more misses through, a trade-off. That is why enlarging the sample to shrink SE is the honest way to reduce both errors at once.