Deep Black Box - Problem 16: it does not get smarter (Chapter 4)

Question 16

It does not get smarter. Its behaviour lines up with someone’s taste

Where you are right now — the band above shows the chapters so far and where this problem sits. Every step down gains something and loses something (it is not a simple stacking up). The lower section is what this problem is about — the method in a single OpenAI paper from March 2022 (“Training language models to follow instructions”, arXiv:2203.02155. What it tuned was GPT-3 — 2020, 175 billion knobs). The ①②③ in the lower section are steps inside this problem (Problem 16) only (nothing to do with Problems 15 and 17). They are the order in which a human works on the LLM that came out of Problem 15, and the only one you press is ②. How the result then gets used (steering its thinking and so on) sits outside this column and comes up in the last problem. Whether the companies of today do it this way is not something this paper says.

What is different between A and B — which one is “better” is not settled. The taste of whoever presses becomes the score, just as it is

In all three pairs, get the prediction for the side you picked to 80% or above.

Press whichever you think is better. There is no right answer

P = σ( r(picked) − r(not picked) ) error = mean of −log P over 3 pairs

σ is the sigmoid. The knobs are the 6 r values (rewards), and the machine is what turns them

What [10 steps] is doing — it is fixing how the scores get handed out, so that they come out the way you pressed. The same walk downhill as Problems 1 and 2; the only difference is how the error is measured — it measures whether a high score sits on the side the human pressed. What is being built here is the scoring. The LLM itself (③ in the diagram above) has not moved a single step on this screen.

0 steps

–% the lowest pair

0goal 80%100

–

There are two measures — the same thing is being measured two ways, and which one you use flips the answer. 1.3 billion and 175 billion are counts of knobs (parameters)

What “70% agreement” means — it is not a vote among 40 people. It counts whether two people shown the same question picked the same side

What to take away

This is RLHF — reinforcement learning from human feedback.

The spread of answer styles (schematic)

Gained	Usefulness. It wins 85% against the untuned model of the same size. 1.3 billion was preferred over 175 billion
Lost	Some abilities drop (SQuAD, DROP, HellaSwag, WMT15). Calibration breaks: 0.007 → 0.074

The original paper

Where this pays off

The one case where “whose taste it was tuned to” is published is the roughly 40 people in this paper. For the tools you use day to day, whose taste they were tuned to is normally not knowable. Read the quirks in the output as the quirks of the people who tuned that tool. Whether other companies work the same way is not something this paper says.