A Quadratic Form Is a Bowl Shaped by Eigenvalues
A line was first order. Now step up one rung and look at a height that curves up over the plane. It's q(x) = xᵀAx, and writing out the symmetric matrix A = [a, b; b, c] gives q = a·x² + 2b·xy + c·y². Slice this surface at a constant height (a level set) and you get an ellipse or a hyperbola. The striking part: the shape is set entirely by the eigenvalues and eigenvectors of A. The eigenvectors point along the bowl's principal axes, and the eigenvalues set how steeply each axis curves. When both eigenvalues are positive it's a bowl with a single bottom; when their signs disagree it's a saddle. And the second derivatives (the Hessian) of any smooth function form exactly a quadratic form, so this one idea threads the second-derivative test, PCA, and convex optimization onto a single line.
Use the sliders to set the three entries a, b, c of the symmetric matrix A. The view draws the height q = a·x² + 2b·xy + c·y² as contour lines. When both eigenvalues are positive the contours are closed ellipses, so you see a bowl dipping toward the center (blue tint). When both are negative it's the same ellipses but the center is highest, a dome (red tint). When one is positive and one negative the contours open into a hyperbola, and you get a saddle that climbs one way and drops the other. Raise b and the contours tilt at an angle. The very same q stays one formula, yet changing just a, b, c moves it among bowl, dome, and saddle.
Now look at just one contour: the level set sliced at q = 1, that is xᵀAx = 1. Keep A near positive definite and this curve is a clean ellipse. But it does not lie at random; its two axes lie exactly along the eigenvectors of A. And each semi-axis has length 1/√λ. A large eigenvalue curves steeply that way so the ellipse is short there; a small eigenvalue is gentle so the ellipse is long. Move the sliders to change A and the eigenvectors (principal axes) and eigenvalues update together. This ellipse is exactly the covariance ellipse of PCA, and a blood relative of the ellipse you saw in SVD; the only difference is that there the semi-axis is √λ while here it is 1/√λ.
Now push the two eigenvalues across zero yourself. Move the sliders λ1 and λ2 and the widget renders a verdict at once. Both positive: positive definite, a bowl curving up every way, so the center is the unique minimum. Both negative: negative definite, an upside-down dome, so the center is a maximum. Opposite signs: indefinite, a saddle. One exactly zero: semidefinite, a trough that curves one way and stays flat the other. Positive shows green, negative shows red, and a small shape glyph tells you whether it is a bowl, dome, saddle, or trough. The point is that the two signs decide everything.
See why the b·xy cross term is a nuisance, and how it vanishes. Hold the two eigenvalues λ1, λ2 fixed and turn only the angle θ; that builds A = R Λ Rᵀ. When θ is not zero the cross term b is alive and the ellipse lies at a slant. Use the slider to rotate the ellipse. The instant the principal axes line up exactly with the x and y axes, b drops to 0, and in those coordinates q resolves cleanly into q = λ1·y1² + λ2·y2², a form with no cross term. That is diagonalization. In A = Q Λ Qᵀ, rotating coordinates by Q removes the mixing and leaves a pure sum where the two axes act independently. The current q expression for these coordinates is shown below.
Finally, see why all of this is the heart of optimization. Any smooth function, near a point, curves according to a symmetric matrix called the Hessian that collects its second derivatives. The Hessian is a quadratic form. So at a critical point where the gradient is zero: a positive definite Hessian makes it a local minimum (the bottom of a bowl), a negative definite one a local maximum (the top of a dome), and an indefinite one a saddle point. That is the second-derivative test. Move the sliders to change the Hessian and the contour landscape and the verdict change together. This is exactly why gradient descent rolls happily into positive definite bowls and gets stuck wandering at saddle points. Whether the curvature at a point is positive, negative, or mixed decides the answer.