Projection Is the Nearest Shadow
Shine a light on a stick and it casts a shadow on the floor. That shadow is the single point on the floor nearest to the stick, and the line dropping down to it always meets the floor at a right angle. That is projection. Drop the shadow of not one point but dozens of data points the same way, and you get least squares: the line that passes nearest to every point, the regression line. Fitting a model to data in AI is, in the end, this one move of casting a shadow.
The blue point p is fixed. Use the slider to rotate the angle of a line through the origin. The gold point f on the line is p's shadow, the spot on the line nearest to p. The dashed red segment dropping from p to f always meets the line at a right angle, and that is the residual. When the line points exactly toward p the residual shrinks to zero and the shadow lands on p; the more the line is off from p, the longer the residual grows. Read the projection length p·u alongside the residual length and watch how the two trade off.
Now the line is fixed at 30 degrees and you drag the point p yourself. Wherever p goes, the shadow on the line (the projection foot) follows, and the dashed line dropping from p to it is always perpendicular to the line. That right angle is the whole point: when the residual is perpendicular to the line, its length is exactly the shortest distance from p to the line. Any other point on the line is farther away. Drag p far from the line and the distance readout grows; tuck it close and it nears zero. Nearest and perpendicular are two faces of the same fact.
Why is the shadow length p·u? Put the line on the x-axis (u=e1) and use the slider to change only the angle θ of a vector p whose length is fixed at 3. The x-axis shadow of p (gold) has length |p|cosθ, made by the dashed vertical drop. At θ=0, p lies along the line and the shadow is the full 3; as θ heads to 90 degrees, p is perpendicular to the line and the shadow vanishes to 0. This |p|cosθ is exactly the definition of the dot product p·u. The dot product is, in the end, a ruler measuring the length of the shadow one vector casts along another.
Now it is not one point but seven. Use the sliders to move the line's slope m and intercept b. The vertical dashed drop from each point to the line is that point's residual, and the sum of their squares is the SSE (sum of squared errors). The better the line threads through the middle of the points, the shorter the residuals and the smaller the SSE. Try to find the best by hand, then press the best-fit button: m and b jump to the exact values the least-squares formula gives, and the SSE drops to its lowest possible value. This is the regression line.
Finally, seen geometrically, least squares closes in one line. There is a blue target vector b, and all the model can build are points on the line drawn by the gold direction vector a. The nearest approximation is the point p where b projects onto that line, and the leftover residual r=b−p must be at a right angle to a. Flip the toggle and the perpendicular drop (dashed) appears, showing a·r near zero. This right-angle condition is exactly the normal equation aᵀ(b−Ax)=0. The residual perpendicularity from block one has become the equation itself here. Least squares is always the act of standing the residual perpendicular to the model space.