what is the gradient
Imagine standing on a fog-bound mountainside. You cannot see an inch ahead, yet you can clearly feel the slope under your feet. That alone tells you which way climbs most steeply, and that direction is the gradient. In a scalar field, where each point carries one value such as height or temperature, it is the arrow pointing where the value grows fastest. It is easy to read only its direction, but its length means something too: the steeper the slope, the longer the arrow. Hold this one arrow and trail design, the direction heat flows, and the gradient descent that trains AI all line up on a single thread.
Drag the probe over the color-shaded height field and, wherever you place it, a white arrow follows it. That arrow points the steepest way up from that spot: the gradient. Closer to the peak at the upper right, the slope eases and the arrow shortens; on steep ground it grows long. Its length, the size of the slope, is written below as a number. So this arrow holds both which way climbs fastest and how steep the climb is. On a flat plain there is nowhere higher to go, so the arrow dwindles to almost 0.
The contour lines on a map join points of equal height, and the gradient meets them at exactly a right angle everywhere. The reason is simple. Walking along a contour, your height stays the same, so the climb is 0; the fastest way up has no choice but to cut straight across the line. The figure is a dome with its peak at the origin, so the contours are concentric circles. Move the probe around and check: the gradient, the gold arrow, and the tangent to the contour, the blue line, stay at 90 degrees everywhere. In this figure the arrow's length is fixed so that only its direction shows, but a real gradient grows longer where the contours crowd together, that is, where the slope is steep.
The gradient is a vector too, so it splits into a horizontal and a vertical component. The horizontal component is how fast the value rises as you go east (∂f/∂x), and the vertical component is how fast it rises as you go north (∂f/∂y). These two are the partial derivatives: slopes measured in one direction while everything else is held still for a moment. Drag the probe and the green east component and the orange north component combine into the white gradient arrow, while the equation below shows the two components' values in the same colors. The assembly we used in "A Vector Is a Movement," reading a vector as so much î plus so much ĵ, works here exactly the same way.
If the gradient points uphill, its opposite (−∇f) is the fastest way down. Drop a ball and at every moment it moves one step in the steepest downhill direction from where it stands, until it finally stops at the bottom of a valley. The terrain in the figure has two valleys, so drag the gold ball to different places and each one rolls into a valley near its starting point; the white line marks the path and the small circle marks where it stops. That is gradient descent. Training an AI model works the same way: on a landscape of error, it steps down along −∇(error) to find the valley where the error is smallest. It adjusts billions of numbers on exactly the same principle as rolling a ball down a mountain.
There are places where the gradient vanishes to 0: the summit of a peak, the bottom of a valley, and a saddle point. At such places there is no direction to climb or descend, so the arrow disappears entirely. The figure has a peak on the left and a valley on the right. Drag the probe to find where the arrow shrinks to 0, and the point grows larger in gold with a sign saying it is flat. These places matter because any problem of making something as large or as small as possible turns into the problem of finding the points where ∇f = 0. The cheapest design, the strongest structure, the smallest error: all of them work this way. At an optimum the terrain is always flat. From the next lesson on, we move from scalars to how a vector field itself changes from point to point.