Deep Black Box - Problem 21: push it higher and it falls (Chapter 5)

Question 21

Push it higher and there is a place where it falls

From here we use a different method — the decision tree

A decision tree is a tree of questions answered yes or no. What you have been doing by hand (split on a column, then answer each band with the bigger side) is what the machine now repeats on its own.

This is a different mechanism from the gradient descent of Chapters 1 to 4. That one moved knobs a little at a time, heading the way the error goes down. A decision tree just lines up every candidate cut and picks the best one. Nothing moves a little at a time.
For data in the shape of a table, the decision tree is the standard choice. Deep learning is not the answer to everything.

What you did in Problem 19 — the same question for every group, added in turn

What a decision tree does — at each branch, it picks the question that works best right there, all over again

Here is what “works best” means. What the machine measures is not how many people it gets right. It is how much the uncertainty drops.

Putting a number on uncertainty

The closer a group is to fifty-fifty, the greater the uncertainty. If everyone is the same, it is 0.

–

p = the share of that group who lived, q = the share who died (p + q = 1). log₂ asks “two to what power gives this number”. You do not have to remember what is inside the formula. Fifty-fifty gives 1.00, and the more it leans one way the closer to 0 — that is all you need.

How much the uncertainty drops when you ask one question

Take the uncertainty on each side, average it weighted by how many people are in each, and subtract that from what it was before. That is the measure of how well a question works.

–

The uncertainty of a bigger group counts for more. Of the 891, 577 are male (64.8%), so their uncertainty is counted at 0.648 times its value.

All 15 columns measured (across all 891)

The yes side and the no side are how many people fall each way on that question. For the numeric columns (class, fare, age, family size) every candidate cut was tried and the one that drops it most was taken.

Here is the catch

Split on the top-ranked column and you do not get the most people right.

The machine is not looking at how many it gets right. It looks only at the drop in uncertainty.
The reason that is fine is that it does not stop after one question. Leave the groups with less uncertainty and the next question cuts them down efficiently. The 4 people it gives up on the first question are given up for the ones after it.

–

how many questions to allow1

training score (the 891) –%

unseen score (people never shown) –%

Move the slider to the right.

blue = training score (the 891 whose answers you know) / red = unseen score (people never once seen while building it)

Places an answer comes out of (leaves): 2 — the bar is all 891. The vertical lines are where the leaves divide

–

How the unseen score is measured — split the 891 into five, hide one, build on the rest, and score on the hidden part. Change which block is hidden, go round five times, and take the average

□ = the side hidden and scored on (over these five rounds all 891 get scored exactly once).
The five blocks are matched on survival rate (against 0.3838 for all 891, they are 0.3855 / 0.3820 / 0.3820 / 0.3820 / 0.3876).
Measure once and simply changing which 178 you hide moves it by 5.6 points. That is why we look at the average of five.

last

Changing the model will not get you there. You have to change the columns

The bars are the unseen score. The horizontal lines are the spread over the five rounds (standard deviation). Four builds that are nothing alike line up at the same height

For the top two (random forest with 300 trees, and a decision tree of depth 4), can you say there is a real difference?
Two conditions — (1) the range of the difference (mean ± spread) does not straddle 0, (2) all five rounds point the same way

Press one of the two.

Left = this problem (built by a person; it has a name, so you know what is being looked at) / right = an LLM (one word becomes 12,288 numbers, and those numbers are settled by learning too. Which number stands for what has no name)

Where this pays off

Watching the training score alone will not tell you. Always measure on data you have never seen.
And when you want the score to go up, building the columns you hand over works better than swapping the model.

Go to the Titanic on Kaggle → Go to Kaggle’s own tutorial page →

Every number here is measured from the public data (the Titanic on Kaggle). Problems 18 to 21 counted on the 891 in the training set. The unseen score is the average of five rounds, splitting the 891 into five.