Deep Black BoxOpen up the black box of AI one problem at a time, with your own hands. 21 problems in all.
Question 17
Right or wrong cannot be gamed — but only where right and wrong can be judged
Where you are right now —
the band above shows the chapters so far and where this problem sits.
The lower section is what this problem is about — the method in a single DeepSeek paper from January 2025
(DeepSeek-R1, arXiv:2501.12948).
Two things differ from Problem 16 — not one human model answer is used.
And the grading is done by a machine, not a person
(it checks numbers, or runs the tests on a piece of code).
The โ and โ you are about to press stand in for that machine.
Whether the companies of today do it this way is not something
this paper says.
What ร (the advantage) does —
it measures whether you are above or below the mean.
Above, it gets pushed up (+); below, it gets held down (โ).
Dividing by the spread is what puts every problem on the same scale
(there are four, so divide by 4)
Mark the four answers โ or โ (press to flip)
ร = ( r โ mean ) รท std
r = 1 if right, 0 if wrong
+ gets pushed up, โ gets held down. std is the standard deviation, divided by 4
โ
What “right and wrong can be judged” means —
if the answer settles on one thing, a machine can grade it.
If it changes with the reader, it cannot be graded
Which of these can a machine judge right or wrong?
0/ 6 right
0goal 66
โ
Drawing once, and drawing many times —
the top two bars are how often it was right when asked once (on AIME 2024, a maths test.
Measured on the version that uses no model answers).
The bottom two are how often it was right at least once when asked many times, and they
barely change.
The bottom two are schematic, drawn to show that they do not change
(the paper gives no figures). The paper itself holds back, saying it may be down to their own method.
What to take away
This is RLVR — reinforcement learning with verifiable rewards.
pass@1 on AIME 2024 (%)
Gained
With no model answers,
rewarding only right/wrong and format: 15.6% โ 77.9%
Lost
Put model answers in between
and it drops 77.9 โ 59.0
pass@1 is the average accuracy over several attempts, not “right on the first try”
The original paper
Where this pays off
This way of training only works where a machine can judge right from wrong.
Arithmetic, code and checking numbers all improve.
Work whose answer changes with the reader (how good a piece of writing is, how sincere an apology is) does not improve here. If someone claims it did, ask first what they took as the right answer.
summary
Is this about building, or about using?
The building steps come in order and cannot be undone. Ways of using have no order and combine freely.
“AI drifts to the average” and “AI solves olympiad maths” are statements about different stages.
next
From here on, it is your turn to do the work
The next chapter alone sits outside this nesting. The tools change.