Deep Black BoxOpen up the black box of AI one problem at a time, with your own hands. 21 problems in all.
Question 2
Walk down the landscape of the error
The same landscape seen from straight above — darker means less error. The × is the bottom of the valley.You will walk on it for real in the figure below
The same landscape seen from an angle — the green ridge is the floor of the valley, and the orange dot is where you are.
The height is exaggerated so that it can be seen
The data and the line — move a knob and this line moves
The error map (seen from straight above; orange is the path you walked)
The green ridge is the floor of the valley. Orange is the path you walked.
โpoints = error /
0 steps
0goal 1.5 ↑30
Press [One step] a few times and watch the dot move on the map.
The table shows you what is wrong
Right now a = 3.00 b = 20.00 / error
27.18
What you can read off this table
โ
[Measure] Raise one knob by 1 and look at the error
Raise b (height) by 1 (a stays where it is)
โ
Raise a (slope) by 1 (b stays where it is)
โ
โ
[Move] Once the measuring is done, rewrite a and b by what you measured, in the direction that lowers the error
Press it and both move by exactly what was just measured
Press a button.
Where this pays off
When training goes nowhere, the first thing to suspect is the step size (learning rate). Too large and it flies off; too small and it never arrives.
What to take away
This is gradient descent. Measure the slope, take a small step in that direction, repeat.
GPT-3 did it about 100,000 times.