floating point and condition number
So far we have treated error as something that comes from the method. The step is too big, the degree is too low. Under every one of those methods lies a more basic limit. A computer cannot store a real number exactly. Not even 0.1. A 64-bit floating-point number remembers only about 15 to 16 digits and rounds the rest away. Usually this tiny error stays buried and invisible, but it jumps out when you subtract two nearly equal numbers, or stack a small error millions of times, or meet a badly conditioned problem whose answer is sensitive to the input. This lesson looks at the floor of numerical analysis, the way a computer handles numbers and the error that leaks from that. It is what the error floor (roundoff floor) really is, and where it comes from.
The gold ticks on the number line are the numbers a computer can store exactly. It looks odd at first. Near 0 the ticks are dense, and they thin out as you go right. Floating point writes a number the way scientific notation does, a few significant digits with an exponent attached. So every time the size doubles, the gap between neighboring representable values doubles too. That smallest gap is called an ulp(units in the last place). Move x and you see the ulp grow as the number gets large. Large numbers are stored coarsely and small numbers finely. Every rounding error starts here.
Measure how small that gap really is, near 1. Add ε = 2k to 1 while you lower k, and at some moment 1+ε becomes just 1. Once the added ε is smaller than half the gap above 1, it rounds to the nearest representable value, which is 1 itself. While it survives the bar is green, and when it vanishes the bar is red. The gold line is that half gap. This boundary is the machine epsilon, and in float64 it is 2⁻⁵² ≈ 2.2×10⁻¹⁶. That is about 15 to 16 significant digits. This is not an abstract remark. It is a real calculation. To a computer, 1 plus a very small number can just be 1. The first evidence that precision has a floor.
One error is on the order of 10⁻¹⁶, so it seems safe to ignore. But 0.1 does not come out exact in binary, so the moment it is stored it already carries a tiny error. Each time you add it, that error stacks a little too. Raise the number of additions with the slider and the error away from the true value n/10 grows steadily. What you cannot see once becomes clear after a few hundred times, and in a calculation that repeats billions of times, like a simulation, it can ruin the result. So a numerical programmer changes the order of addition, small first, or uses a compensated sum such as Kahan summation to keep this accumulation in hand.
The most frightening one stands apart. It is subtracting two nearly equal numbers. Compute (1−cos x)/x² at a small x. The true value should go to 0.5 as x gets small. But the gold curve, computed straight, collapses to 0 at some moment. When x is small, cos x is nearly 1, so 1−cos x loses almost all of its significant digits. The leading digits match and cancel, and only an untrustworthy tail remains. That is catastrophic cancellation. The blue curve computes the same value rewritten as 1−cos x = 2 sin²(x/2). It never subtracts, so it holds 0.5 all the way. The formulas are the same, and the order of the calculation saves or kills the precision.
The last one is not an error. It is the character of the problem itself. Solving a linear system is finding the intersection of two lines. When the gold line and the blue line cross cleanly, the intersection is sharp. When they are nearly parallel, nudging one line by a hair sends the intersection flying far away. That shift is drawn as a red line. Lower the slope with the slider until the two lines are parallel, and the same 10% nudge moves the solution more and more. That amplification is the condition number. A large condition number is a bad condition, where a small error in the input swells in the answer. The frightening part is that, unlike rounding error, a better computer cannot fix it. The problem itself is touchy. So good numerical analysis looks after both, a stable algorithm and a well-conditioned problem.