seegongsik
Saved words
Engineering Mathematics

SVD Is Rotate, Stretch, Rotate

Any matrix is three steps, rotate then stretch along axes then rotate, and the unit circle becomes an ellipse whose radii are the singular values

However tangled a matrix looks, it's really just three clean moves: turn once (Vᵀ), stretch separately along two axes (Σ), then turn again (U). Feed the whole unit circle into it and out comes a tidy ellipse, and the lengths of that ellipse's semi-axes are exactly the singular values. The single largest singular value carries almost all of the ellipse's shape. Image compression and PCA stand on that one fact. In the first widget, step through the moves one at a time and watch a circle become an ellipse.

Step through the stages with the buttons. Stage 0 is just the unit circle, with the right singular vectors v1, v2 marked on it. Stage 1, Vᵀ, turns the circle, but a circle is a circle so the shape doesn't change; instead v1 and v2 land exactly on the x and y axes. Stage 2, Σ, stretches only along those two axes, making a tidy ellipse with semi-axis σ1 along x and σ2 along y. Stage 3, U, turns that ellipse once more into the tilted final shape, the image of A, whose two semi-axes are σ1·u1 and σ2·u2. Rotate, stretch, rotate: those three moves build any matrix at all.

Now strip away the rotations and keep only the stretch (U = V = I). The slider σ1 sets how far the unit circle is pulled along the x axis, σ2 along the y axis, and those two values become the semi-axis lengths of the ellipse. A singular value is just how many times that axis is stretched. Raise σ1 and the ellipse grows wide; lower σ2 and it flattens vertically. Slide one of them down toward 0 and the ellipse collapses entirely along that axis into a single line segment. At that moment the matrix's rank drops from 2 to 1, and a warning appears.

Now take the three pieces into your own hands. The sliders set the first rotation angle (V), the two stretch factors (σ1, σ2), and the last rotation angle (U), and together they are A = U Σ Vᵀ. The view shows both how the grid deforms and which ellipse the unit circle goes to, while the readout below displays the four entries of that A. Turn the angles and factors around and you'll see that however arbitrary the matrix numbers look, they always come from two clean rotations and one axis-stretch. No 2x2 matrix escapes this rotate, stretch, rotate decomposition.

This decomposition has two sets of directions: the inputs v1, v2 (the right singular vectors) and the outputs u1, u2 (the left singular vectors). Use the toggle to pick a pair. Put v1 through A and out comes exactly σ1·u1; put v2 through and out comes σ2·u2. So an input direction v turns into an output direction u, scaled by its singular value. The blue arrow on the unit circle is the input v; the arrow on the ellipse is its output. V directions go in, U directions come out, sized by the singular values. That is SVD in one sentence.

Finally, check whether the ellipse's shape really lives almost entirely in one big singular value. The toggle's rank-2 is the full A, the whole tilted ellipse. Switch to rank-1 and only the piece σ1·u1·v1ᵀ remains, so the ellipse settles completely onto its longest axis (the major axis), becoming a line segment of length σ1. You've thrown away the small σ2 piece. The view shows the energy that big piece captures, σ1²/(σ1²+σ2²), as a percent. Usually that one piece keeps most of the shape. Keeping only the largest singular value still preserves the shape almost completely, and that is the core idea behind compression and PCA.

In PracticeSVD splits every matrix into A = U Σ Vᵀ, that is rotate then stretch along axes then rotate. Feed in the unit circle and an ellipse comes out, whose semi-axis lengths are the singular values σ1 ≥ σ2. Σ is the pure stretch by σ along each axis, while U and V are pure rotations. The right singular vectors v are the input directions and the left singular vectors u are the output directions, so A·vi = σi·ui holds. When a σ is 0 that axis dies and the rank drops, and a low-rank approximation that keeps only the largest σ preserves most of the shape. Image compression, PCA, recommender systems, and denoising all flow from this one fact. Even arbitrary data is, in the end, rotate, stretch, rotate.
Engineering Mathematics
Was this helpful? Support seegongsik