quadratic forms and eigenvalues
A line was degree one. Now step up one and look at a height that bends up off the plane. q(x) = xᵀAx, and written out for the symmetric matrix A = [a, b; b, c] it is q = a·x² + 2b·xy + c·y². Cut this surface at a constant height and the level set is an ellipse or a hyperbola. What fixes that shape is entirely the eigenvalues and eigenvectors of that matrix. The eigenvectors point along the principal axes of the bowl, and the eigenvalues set how steeply each axis bends. If both eigenvalues are positive there is a bowl with one floor, and if the signs disagree it is a saddle. For any smooth function the Hessian of second derivatives is exactly a quadratic form, so this one thing threads the second-derivative test and principal components on one line, PCA. The sign of the bowl separates a minimum, a maximum, and a saddle.
Set the three values a, b, c of the symmetric matrix with the sliders. The screen draws the height q = a·x² + 2b·xy + c·y² as contours. If both eigenvalues are positive the contours become a closed ellipse, a blue bowl falling toward the middle. If both are negative it is the same ellipse but a red dome highest in the middle. If one is positive and one is negative the contours become an open hyperbola, a gold saddle that climbs one way and falls the other. If one eigenvalue is nearly gone it becomes a green trough. Raise b and the contours lean. Change only the three numbers a, b, c and the same q² formula walks among bowl, dome, and saddle. The color follows the sign as it is.
Now look at exactly one contour. The level set cut at height q = 1, that is xᵀAx = 1. Put the matrix near positive definite and this curve is a clean ellipse. The ellipse does not lie any old way. Its two axes lie exactly along the eigenvectors. And the length of each half-axis is 1/√λ. A large eigenvalue bends steeply that way and the ellipse is short. A small eigenvalue is gentle and the ellipse is long. Change the matrix with the slider and the principal axes, the eigenvectors, and the eigenvalues update together. This ellipse, with half-axes 1/√λ, is kin to the covariance ellipse of principal components. Length and direction come out together from one matrix.
Now push the two eigenvalues across 0 yourself. Move the sliders λ1 and λ2 and the widget gives the verdict at once. The first number is green and the second number is orange. Both positive is positive definite, a bowl painted green, and the middle is the only minimum. Both negative is negative definite, a red dome, and the middle is the maximum. Signs that disagree are indefinite, a gold saddle. If one of them is exactly 0 it is semidefinite, a trough that bends one way and stays flat the other. Two signs decide everything. The verdict text follows the color of that sign too.
See why the cross term b·xy is a nuisance, and how it disappears. Keep the two eigenvalues λ1, λ2 and turn only the angle, and A = R Λ Rᵀ is built. If the angle is not 0 the cross term is alive and the ellipse lies tilted. Turn the angle with the slider and rotate the ellipse, and the moment the principal axes line up exactly with the x, y axes the cross term falls to 0. In those coordinates it opens cleanly as q = λ1·y1² + λ2·y2², a form with no cross term. That is diagonalization. In A = Q Λ Qᵀ, turn the coordinates by the orthogonal matrix and the mixing disappears, and the two axes play separately as a pure sum. The q formula in the present coordinates shows below. Lining up the angle is the whole of erasing the cross term.
Last, see why all of this is the heart of optimization. Any smooth function, near one point, bends by the symmetric matrix of its second derivatives, the Hessian. The Hessian is exactly a quadratic form. So at a critical point where the slope is 0, if the Hessian is positive definite the point is the floor of a bowl, a local minimum. If it is negative definite it is the top of a dome, a local maximum. If it is indefinite it is a saddle point. That is the second-derivative test. Change the Hessian with the slider and the contour terrain and the verdict change together. This is why gradient descent rolls well into a positive-definite bowl and wanders at a saddle. Whether the curvature at one point is positive, negative, or mixed decides the answer. The sign of the curvature is the name of that point.