least squares projection explained
Picture the shadow that a point in the air casts on the floor when the sun is directly overhead. That shadow is the spot on the floor closest to the original point, and the line dropping from the point to its shadow always meets the floor at a right angle. That is projection. Cast the shadows of dozens of data points the same way all at once, rather than a single point, and you have least squares; the line that passes closest to all the points is the regression line. When AI fits a model to data, it is ultimately doing this one move: casting a shadow.
The blue point is fixed. Turn the angle of the line through the origin with the slider and watch what changes. The gold point on the line is the blue point's shadow, the spot on the line closest to it. The red dashed line dropping from the blue point to its shadow always makes a right angle with the line; that is the residual. When the line points exactly at the blue point, the residual shrinks to 0 and the shadow lands on the point; the more the line misses, the longer the residual. The gold segment from the origin to the shadow is the projection length, and below, the projection length p·u and the residual length are shown side by side, so you can see how the two fit together.
This time, fix the line at 30 degrees and drag the blue point yourself. Wherever you move it, the gold shadow on the line follows, and the red dashed line from the point to its shadow always meets the line at a right angle. That right angle is the key. When the residual is perpendicular to the line, its length is the shortest distance from the point to the line; any other point on the line is farther away. Drag the point far from the line and the distance value below grows; press it close to the line and it approaches 0. Closest and perpendicular are two faces of the same fact.
Why is the shadow length p·u? Take the line to be the x-axis (u=e1) and change only the angle θ of the blue vector, whose length is fixed at 3, with the slider. The vector's shadow on the x-axis has length |p|cosθ, and the red dashed line dropping vertically produces that gold shadow. At θ of 0 the vector lies along the line and the shadow is the full 3; as θ goes to 90 degrees the vector becomes perpendicular to the line and the shadow vanishes to 0. This |p|cosθ is exactly the definition of the dot product p·u. The dot product is, in the end, a ruler that measures the length of the shadow one vector casts along another's direction.
Now there are not one but seven points. Move the slope and the intercept of the line with the sliders: the red vertical dashed line from each point to the gold line is that point's residual, and the sum of their squares is the SSE (sum of squared errors). The better the line threads through the middle of the points, the shorter the residuals and the smaller the value below. After hunting for the best line by hand, press "Best fit" and the slope and intercept jump to the exact values from the least-squares formula, while the sum of squared errors drops to the smallest value possible. That is the regression line. Comparing how close your hand-found line came is good practice too.
Finally, seen geometrically, least squares closes in a single line. There is a blue target vector, and the only things the model can produce are points on the line traced by the gold direction vector. The closest approximation is the projection of the target onto that line, and the leftover residual r=b−p must be perpendicular to the model direction. Turn on the toggle and that perpendicular drop appears as a red dashed line, while the readout below shows that a·r is close to 0. This right-angle condition is exactly the normal equation aᵀ(b−Ax)=0. The orthogonal residual of the first block has turned directly into an equation here. Least squares always means standing the residual perpendicular to the model space.