A Record of Thinking with AI: An AI Olympiad Medalist from Japan's Most Selective High School on Growing More AI Builders

The question “is it acceptable to have AI do Kaggle for you without understanding what is inside” was worked through by the moderator with an AI after the session ended. The conclusion was: no, it is not — but insisting that you understand everything is also wrong, because within understanding there are layers you may hand over to someone else, and layers you may not.

① What is inside the tools ── The calculations and the mathematics that an off-the-shelf tool performs internally. This may be handed over to AI. Even specialists do not read the inside in detail. It is the same as not needing engine theory to drive a car.

② Whether the procedure is sound ── How the data is prepared, whether answers have leaked in, how the scoring works, whether you have fitted too closely to your practice data. This is the vital point.

③ Whether the conclusion is sound ── Whether you may accept what came out as correct. This cannot be handed over. There is one reason: AI cannot set the standard of correctness by itself. That is why it can be wrong in a confident tone, and why it will sometimes run a test it wrote itself, pass it itself, and report back that there were no problems. The same holds in analysis: if two numbers move together, it will talk about one as the cause. Correlation and causation are different things, and separating them is a human job. On top of that, specialists disagree about it.

② Whether the procedure is sound — the reason this is the vital point is not that it is difficult. It is that when you get ② wrong, the result does not necessarily look wrong. And it does not necessarily look worse, either.

There is a record of this actually happening. Andrey Lukyanenko, a Kaggle Competition Master and a machine learning engineer at Meta, published a record of it in July 2026. He had an AI agent run an entire Kaggle competition. The moderator looked this up after the session.

① What is inside the tools ── the AI did all of this layer. The task was to look at health checkup records for 630,000 people and sort each one into “needs attention” or “no problem.” The AI explored the data, built about 140 columns of its own as predictive signals, trained three kinds of models in parallel, tuned how to blend them, and made the submission entirely on its own. The human wrote almost no code. Nobody looked at what was in the columns, or at what was being computed inside the models. And it worked.

② Whether the procedure is sound ── this is the layer where it stopped. The AI checks how well it is doing as it goes and improves from there. The way it was checking did not match this task. Scaled down to 100 people, it looks like this.

 Sort 100 people into "needs attention" and "no problem."
 The actual split is lopsided

    needs attention   94  ●●●●●●●●●●●●●●●
    no problem         6  ○

 The way the AI was counting
    "Out of 100 people, how many did I get right?"

 The way this task called for
    Look at "did you find the people who need attention" and
    "did you find the people who are fine" with equal weight

That difference pulls the AI’s judgment to one side.

 Say a borderline case comes up. Which do you answer?

   answer "needs attention"  → 94 of 100 need attention, so the odds of being right are high
   answer "no problem"       → the odds of being right are low
        │
        ▼
   Under the AI's way of counting, when in doubt it pays to answer "needs attention"
        │
        ▼
   That accumulates, and the answers drift toward "needs attention"
        │
        ▼
   The ability to find the 6 people who are fine never develops

The AI did not cut corners. It optimized correctly against the way of counting it was handed. Taken to the extreme, “everyone needs attention” scores 94 out of 100 under that way of counting. An answer that distinguishes nobody produces the highest number. And the machine issues no warning here. The number keeps going up.

③ Whether the conclusion is sound ── it missed this layer too. When the submitted result came back poor, the AI’s diagnosis was that getting more data was the only way forward. It proposed using unofficial data uploaded by other participants and argued confidently that this was the only way through the wall. The human stopped it.

What the human doubted was not the data but the way of counting. The fix was to go back to what he had decided for himself at the start: use the way of counting the task calls for, and rebuild the self-check around that same measure. What could be handed to the AI stopped at ①.

Lined up side by side, they look like this.

Where Kaggle moved closer to the real world is one thing only: the dirtiness. The hardest thing in the real world is deciding what should be solved in the first place, and on Kaggle the organizers do all of that for you — what to predict, what to measure it by, and even collecting the data.

And a single property — being closed — created its strength, its weakness, and its limit at the same time.

• Why it is excellent as training ── Results come back immediately, so you find out whether your own judgment was right
• Why AI overtakes it ── There is a scoring sheet, which puts it in exactly the shape machines are best at
• Its limit as proof of ability ── All it can prove is skill at the intermediate work. And that intermediate work is precisely where AI is now becoming good

The first of those involves a mechanism the UTokyo exam does not have. Until a competition ends, part of the answer key used for scoring is hidden.

If you keep tuning against the score that is visible throughout, that score climbs. Then at the end, a different score appears, computed on the data that had been hidden. Some people fall a long way in the rankings at that point. Matching the visible score only means matching the quirks of the data behind that score; it does not mean your ability to predict improved. This is called overfitting.

So Kaggle’s scoring is not only measuring whether your answer is right. It measures whether you trusted your own scoring at hand, or chased the number in front of you — a measurement of whether your own judgment was sound. “Scoring exists” is true of UTokyo mathematics as well, but hidden scoring exists only on Kaggle. The reverse also holds: Kaggle will not develop your ability to write a rigorous proof, because on Kaggle you never write one.

There are three things AI training aims at. They are added in order. ① Trace what it has seen (the material is text from the internet; the exercise is predicting the next word) / ② Lean toward what people prefer (the material is records of people being shown two options, A and B, and picking the one they thought was better; the exercise is making the chosen one score higher) / ③ Get right the things a machine can mark (the material is mathematics and code; the exercise is having a machine check the answer against the key).

There is a common misunderstanding about ①. AI is not taught to “prefer words you see often.” It is trained to assign correct probabilities to rare words too. The blandness of what comes out is not caused by this exercise. It happens because when the conditions in your question are thin, probability piles up on the phrasings you see most often — and because the generation step is biased toward picking the tall peaks (→ Column 4).

None of these three contains a measure for “is this a valuable kind of new?” ① has “does it match what I saw,” ② has “did people think it was good,” ③ has “does it match the answer key.” The only thing missing anywhere is a measure of newness.

At this point you would expect ② to cover it. “Which is better” cannot be marked by a machine, so they had people choose and turned that into a reward. In the same way, you could have people choose “which is newer.” In principle, you can. The reason it does not work lies elsewhere.

Genuine newness, at the moment it appears, is legible as valuable to only a few people. Score it by majority vote and it gets the lowest score. ① traces the average and ② leans toward the majority. Both push in the same direction.

That leaves ③, but ③ only works where an answer key can be written. A machine can mark “2 + 3 = 5”; it cannot mark “is this question interesting?” What is worth creating belongs to the latter. So it does not ride on ③ either. This is not a defect. It is structure.

Line up the cases where AI surpassed humans and the same thing becomes visible. In every one, a human made the scoring sheet.

What the AI did was raise its score inside that scoring sheet. AlphaZero played millions of games against itself and made its own practice partner. It looks as though it set its own problem. Even so, the rules and the definition of winning stayed where a human put them. That is the part it did not move.

The human work ended at building the problem and the scoring mechanism in advance.

These two hours were placed in exactly the opposite position. You cannot check the answer, there is no reward anywhere, and the selection is done by a human one item at a time. Open the tap and thousands of candidates come out. So the real nature of creation becomes less “producing from nothing” and more “picking the rare meaningful one out of an enormous pile of candidates”.

That said, “AI cannot create” goes too far. More precisely: it cannot aim at a valuable kind of new, and judge it, by itself. Put the aim and the judgment in from outside and it reaches genuine discoveries. AlphaFold is an example of reaching one that way.

The 9:18 p.m. output was obvious; the 9:34 p.m. output moved the room. In those sixteen minutes the AI did not change by a single byte. What changed were the conditions a human added just before.

There is a mechanical reason for this. AI is a machine that produces “given this context, which word comes next with what probability.” So when you ask carelessly, the conditions are thin and probability piles up on the phrasings most common in the world. A bland answer is not a malfunction. It is the design working correctly.

Two things get confused here.

• Lowering the temperature ── Temperature is a setting used when generating the answer. Lower it and it picks only the highest-probability words; raise it and it also reaches for low-probability ones. But it is only changing how it chooses within the same distribution. It is an operation that moves toward the bland middle
• Adding conditions ── This deforms the distribution itself and moves where the peaks are. A new peak rises somewhere that had been buried in the tail

Two conditions were added at 9:34 p.m.: Yamai’s direction to “break the stages down into a minimum,” and the constraint to “look up similar services first.”

But while conditions can move you off the commonplace answer, they do not decide which low-probability region holds the value. That has to come from outside.

There is one thing to take away. When you are not satisfied with an answer, suspect your own last set of conditions before you blame the AI.

At the end of the two hours, Kitamura said this about the method: “Everyone talks, you feed it to the AI, and have the AI think. That’s an approach that produces an answer under conditions rather than an average — I found that interesting.”

Being “a combination of things that already exist” does not mean it has no value. The reason is the number of combinations.

So newness is not decided on the side of the materials. It is decided by whether anyone had found that link yet.

Here are well-known cases of creating a valuable new business.

  【Hailing a taxi】        you phone for one and wait
  【GPS in a smartphone】    you know where you are right now
  【Idle time and cars】     sitting unused
              ↓
  Connect the three and you get Uber

  ※ All three existed before 2010.
    The mechanism did not exist because nobody had connected them.

Research has a case that has been known for a long time.

  【Research on fish oil】     lowers blood viscosity
  【Research on Raynaud's】     a disease of poor circulation in the fingers
              ↓
  What they had in common was "blood viscosity"

  ※ Both bodies of literature already existed.
    Nobody crossed this bridge because nobody read both.

What happened that day had the same shape.

  【Shogi】     has tsume-shogi, a single decomposed problem
  【Kaggle】    has no single decomposed problem
              ↓
  What they had in common was the form "a competition that decomposes the real game into stages"

  ※ Everyone knows both.
    Nobody had noticed because nobody had put them side by side.

This is where the reason AI is strong and the reason a human is needed part ways. AI can produce candidate links in bulk — more than a human can. But it cannot decide which of the candidates that come out has meaning. As in Column 3, it has no instrument for measuring that.

AlphaFold and AlphaTensor became superhuman less because they were clever than because they could be scored. Whether a predicted shape matches an experiment, or whether the number of multiplications is smaller, is something a machine can judge. The question in these two hours has no such thing. So candidates come out, but the choosing falls to a human.

There is more than one way to make something new. There is a second one.

  Adding (linking)
    connect two elements that already exist
    example  tsume-shogi × the stages of Kaggle ＝ Tsume-Kaggle
    ※ because the number of combinations is enormous, unfound ones always remain

  Subtracting (removing an assumption)
    take something believed to be a premise and turn it into an option
    example  discard the parallel postulate
    ※ cannot be produced by combination. it is an operation outside addition

Euclid set down “through a point not on a line there is exactly one parallel” as a postulate. For roughly two thousand years afterward, everyone assumed it must be provable from the other postulates. Around 1830, Lobachevsky and Bolyai stopped trying to prove it and discarded it. What followed was that a world with infinitely many parallels, and a world with none, could both be built without contradiction.

And what is subtracted looks, afterward, like something that was added. Non-Euclidean geometry can now be explained as “four postulates combined with the properties of a curved surface.” Before the discovery nobody held it as an element; after the discovery anyone can explain it as a combination. The line between inside and outside swaps depending on when you look.

What they did that day was the connecting-dots side. The side that removes something believed to be a premise, they did not do.