SVD singular value decomposition explained
However jumbled a matrix looks, it turns out to be just three clean moves. First rotate once (Vᵀ), then stretch separately along two axes (Σ), then rotate once more (U). Feed the whole unit circle into this matrix and out comes one neat ellipse. The lengths of that ellipse's semi-axes are exactly the singular values. And the single largest singular value accounts for almost all of the ellipse's shape. Image compression and PCA stand on this one fact. In the first figure, you step through the stages one by one and follow how the circle becomes an ellipse.
Step through the stages one at a time with the buttons. Stage 0 is just the unit circle, with the right singular vectors v1, v2 marked on it. Stage 1, Vᵀ, rotates the circle, but a circle is still a circle, so its shape does not change. Instead, v1, v2 come to rest exactly on the x, y axes. Stage 2, Σ, stretches separately along those two axes only, making a neat ellipse of σ1 along the x-axis and σ2 along the y-axis. Stage 3, U, rotates that ellipse once more into the final tilted ellipse: the image of the unit circle under the matrix. The two semi-axes of that ellipse are σ1·u1 and σ2·u2. Rotate, stretch, rotate: these three moves produce any matrix at all.
This time, remove all rotation and keep only the stretch (U = V = I). The sliders σ1 and σ2 set how far the dashed unit circle is pulled along the horizontal and vertical axes, and the two values are exactly the semi-axis lengths of the gold ellipse. A singular value is, in the end, how many times that axis is stretched. Raise σ1 and the ellipse grows wider; lower σ2 and it flattens vertically. Lower one slider close to 0 and the ellipse is squashed completely along that axis into a single segment. At that moment the rank of the matrix drops from 2 to 1, the ellipse turns red, and a warning appears.
Now take the three pieces in hand. Set the first rotation angle (V), the two stretch factors (σ1, σ2), and the last rotation angle (U) with the sliders, and together they make A = U Σ Vᵀ. The stage draws both how the grid deforms and which gold ellipse the unit circle goes to, while the readout below shows the four entries of that matrix. Turn the angles and factors this way and that, and you see that however arbitrary the matrix entries look, they always come from two clean rotations and one axis stretch. Any 2×2 matrix whatsoever cannot escape this decomposition: rotation → axis stretch → rotation.
This decomposition has two sets of directions: v1, v2 on the way in (the right singular vectors) and u1, u2 on the way out (the left singular vectors). Choose a pair with the toggle. Put v1 into the matrix and exactly σ1·u1 comes out; put in v2 and out comes σ2·u2. In other words, an input direction becomes an output direction, with its length multiplied by the singular value. The blue arrow on the dashed unit circle is the input v, the arrow on the gold ellipse is its output, and a faint dashed line connects the two. V directions go in, U directions come out, and the singular values set the size. That is SVD in one line.
Finally, check whether the shape of the ellipse really fits almost entirely into one large singular value. "Rank 2 (full)" on the toggle is the full A, the whole tilted ellipse. Switch to "Rank 1" and only the single piece σ1·u1·v1ᵀ remains: the ellipse collapses completely onto its longest axis and becomes a segment of length σ1. The small σ2 piece has been thrown away. Below, the energy fraction that the large piece captures, σ1²/(σ1²+σ2²), appears as a percentage. Usually this one piece alone preserves most of the shape. Keeping only the largest singular value still keeps almost all of the shape: that is the key idea behind compression and PCA.