Deep Black Box - Problem 15: just predicting the next word (Chapter 4)

Question 15

Just predicting the next word, and everything else comes along

What this problem is doing — the 5 words at the exit are the 50,000 pieces from Chapter 3, cut down to 5 (we did the cutting). The numbers show up in the bar chart below

Bring the error down to 0.20 or below.

① What comes after “The largest ocean on Earth is the”? ② What comes after “On the track, the train”?
The bars are the probabilities for the 5 words

score = a × place-like + b × motion-like probability = softmax(score) error = −log(p of the right word), averaged over the 2 sentences

a place-like0.00

b motion-like0.00

0 steps

–error

0goal 0.202.0

–

What is happening when you turn the knobs — move a knob and every number here moves

–

error = −log (the probability given to the right word) — the better it guesses, the closer to 0; the more it misses, the higher it jumps. The scale is 0 to 2.0, the same as the gauge above

The real thing has 12,288 numbers for a single word

Who decided these numbers — the two columns in the table above (place-like and motion-like) are ones I wrote by hand. The real thing looks like this

Why grammar, facts and summarizing come along — because they are needed in order to predict the next word

What to take away

This is pretraining.

One batch, and how many of them

	This problem	GPT-3
Knobs the machine decides	2 (a and b)	all 175 billion
Numbers a human wrote by hand	10 (5 words × 2 columns)	0
Words it practised on	2 sentences	300 billion words
Words at the exit	5 words	50,000 words
Repeats	0 times	93,750 times

The original paper

300B ÷ 3.2M = 93,750. On the order of 100,000 times. The numbers for the words (12,288 per word × 50,000 words) are inside that 175 billion too. The 12,288 per word is the d_model in Table 2.1.

Where this pays off

A plausible falsehood is not a bug that gets fixed later on. It comes out of the design itself — picking the word that is likely to come next. So build “the user checks it” into the process. Do not sit and wait for it as a defect that might go away.