inner product space explained
Length, angle, and a right angle feel too familiar to need a definition. In a vector space the three are not given for free. One device, the inner product <u,v>, makes all three at once. Length is the square root of the inner product of a vector with itself, angle is the inner product of two vectors divided by their lengths, and a right angle is where the inner product is 0. The dot product is only one example of this inner product. Change the inner product, for instance by weighting the axes differently, and the geometry on the same vectors bends. A function is a vector too. The inner product of two functions is the integral of their product, and the Fourier transform and kernels start there. The arrows on the screen draw that geometry. Change the inner product and the length and the angle of the same arrows change together. Change the weights and the circle is pressed into an ellipse.
Hold the green vector u horizontal and drag the blue vector v, and four numbers below follow together. The inner product <u,v>, the length of u, the length of v, and the angle between them. Stretch v long in the same direction as u and the inner product grows. Stand it at a right angle and the inner product becomes 0. Turn it the other way and the inner product goes negative. One inner product holds length and angle in one box. Length, angle, and whether the pair is perpendicular all unpack from this single quantity. Green and blue are the two vectors. Of the four numbers, the inner product is the one that gives birth to the other three.
Length first. The inner product of a vector with itself is <v,v>, and that is the square of the length. So the length ||v|| is the square root of <v,v>. Drag v and the radius out to the tip of the arrow is that length. The dashed unit circle is the set of vectors whose length is exactly 1. Place v on the unit circle and <v,v> is exactly 1. You do not define length separately. If you have an inner product, length comes with it. The inner product of a vector with itself is the square of its length, and that sentence is where every norm starts. The dashed circle is the standard for that length.
Now change the inner product itself. A weighted inner product is <u,v>w = w1·u1·v1 + w2·u2·v2, giving the two axes different weights. Move the sliders w1 and w2 and the set of vectors of length 1, the unit ball, stops being a circle and becomes an ellipse. The half-axes are 1/sqrt(w1) and 1/sqrt(w2), so raising the weight on an axis shortens that direction. Distance is priced more dearly that way. The vectors stay put and the standard of what counts as long changes, so the whole geometry bends. When the axes of a data set are measured in different units, like height and weight, the inner product in use is a weighted one. The long axis of the ellipse is the direction whose weight is light.
Orthogonality is defined by the inner product too. Two vectors are orthogonal when <u,v>w = 0. Fix a heavier weight on one axis and turn only the angle of v with the slider. The moment the weighted inner product crosses 0, v lights up. At that moment u and v on the screen do not look like 90 degrees. The weights have bent the space, so a right angle in that world misses the right angle of the eye. A right angle is not the 90 degrees you see. It is the fact that the inner product is 0. Which inner product you use decides what counts as orthogonal. The arrow that lights up is the right angle of that world.
Last, a function is a vector. The inner product of two functions is the integral of their product over an interval, <f,g> = ∫ f·g. Fix f = sin(x) and use the slider to change the integer frequency k in g = sin(kx). The area of the product f·g is shaded. When k is 1 a positive area remains. When k is an integer other than 1, the positive part and the negative part cancel and the integral is 0, which is orthogonality. Sine waves of different frequencies are perpendicular in the space of functions. The Fourier transform, which splits a signal by frequency, stands on that orthogonality. Different frequencies make the areas of the product cancel each other. When the shading splits above and below the baseline, the integral cancels.