hypothesis testing explained
Even when a new drug looks effective or a new algorithm looks faster, that alone cannot settle it. It might have come out that way by chance. So statistics moves like a court. First it sets the skeptical default that there is no effect, the null hypothesis H0. Then it asks how rare the data we saw would be if H0 were really true. If a result this large would be very unlikely even when H0 is true, it becomes awkward to keep H0, so we reject H0 and accept the alternative H1. The point is not that the data prove the truth. It measures whether the result is too surprising to explain by chance. The last lessons saw how a sample mean scatters and how that scatter builds a range for the true value. This time the same bell is used as a ruler of surprise. Surprise is the language of the decision. Chance can imitate an effect, so the same question of surprise still has to be asked before anyone trusts the result.
The first step is to set the two hypotheses clearly. The null H0 is always an equality, pinning down that there is no difference, for example mean = μ0. The alternative H1 is the claim we want to show. H1 can take three shapes, larger on one side, smaller on one side, or simply different. Move the center of H1 away from H0 and pick the direction. On the bell of H0, the shaded tail is where a surprising result would fall. Two sides mean both tails, and a one-sided test looks at one tail. Which side you suspect has to be fixed before you see the data, or it is not fair. Fix the direction first. The side is part of the claim.
Once the hypotheses are set, the data have to be summarized as one number and placed on the bell of H0. That number is the test statistic. Here the sample mean x̄ is standardized. z = (x̄ − μ0) / SE. The meaning is simple. How many standard errors SE is the observed mean from the reference μ0 of H0. That standard error is the width of the sampling distribution from the last lesson. Move the observed mean and z moves on the standard bell. When z is near 0 it fits H0, and when z is large, like 2 or 3, it goes to a rare place far from the center. Turn data of any unit into one z and they can all be compared on the same standard bell. Different units, the same bell.
Now turn the surprise into a number. The p-value is the probability of a result as extreme as the one observed, or more extreme, assuming H0 is true. In the picture it is the tail area under the bell of H0, outside the observed statistic. Push the observed value outward and the tail thins and p shrinks. A small p means that in a world where H0 is true, data like these hardly appear, and that is how surprising it is. One thing has to be pinned down. The p-value is not the probability that H0 is true. It is a conditional probability that assumes H0 is true. Mix up that direction and you misread statistics completely. It is a probability on top of an assumption. That assumption stays in place.
How small is small enough? The line fixed in advance is the significance level. People often use 0.10, 0.05, and 0.01. On the bell of H0 that level is the rejection region, the area of the tail extreme enough to discard H0, and the boundary value is the critical value z*. The rule is simple. If the observed statistic passes the critical value, that is if p < α, reject H0, and if it does not, you cannot reject H0. Change the level and the critical line moves, and the verdict flips as the observed value crosses that line. The smaller the level, the higher the threshold of rejection, and you cannot discard the null unless the result is quite surprising. So the level is the handle that fixes in advance the rate of false alarms we will accept. Fix the threshold first.
Two mistakes hide in the verdict. A type 1 error rejects even though H0 is in fact true, a false alarm, and its probability is the significance level, the area outside the critical value on the H0 bell. A type 2 error fails to discard H0 even though H1 is in fact true, a miss, and its probability is the area inside the critical value on the H1 bell. Alternate the two errors and grow the real effect and the current becomes clear. The larger the effect, the farther the H1 bell moves to the right, the miss shrinks, and the power (1 − β) grows. Power is the probability of catching an effect when it is really there. Shrink the false alarms and the misses grow. That trade is there. So growing the sample and shrinking the standard error is the straight way to shrink both errors at once. The sample presses both mistakes together. Both mistakes sit on the same verdict.