Deep Black Box - Problem 10: go deeper and it can break (Chapter 2)

Question 10

Go deeper and it can break

Take the depth from 1 layer to 3, and watch what happens to the boundary, the colour of the ● and the number correct.

The boundary (a circle passed, a cross did not) and which ● are passing numbers on — a grey ● is returning 0 for all 40 and passing nothing on (the ReLU from Problem 9)

Depth 13 knobs

Skip a layer and add the number straight on

0 steps

Press [Walk down to 3,000 steps].

The trip out and the trip back, for a single ● — the trip back has a proper name: backpropagation. A ● returning 0 brings that trip to a halt

What one step is made of — the trip out (forward propagation) and the trip back (backpropagation), repeated thousands of times

What the skip path is doing — its proper name is a skip connection (also a residual connection). It skips one layer and adds the incoming number straight on

What to take away

Two parts hold a deep network steady — the skip path (skip connection, residual connection) and putting everything on 0 to 1, from Problem 7 (normalization).

Where this pays off

When training goes nowhere, suspect things in this order: (a) the size of the inputs (Problem 7), (b) the step size, (c) the bend, (d) whether there is a skip path.
If every ● has gone grey, it is (c) or (d).

How the terms line up (see the sources for more)

Term	On this screen
Fully connected layer	every line drawn from ● to ●
Activation function	the bend (ReLU, sigmoid)
Layer	the middle layer
Forward propagation	the trip out — handing numbers from input to output to produce a prediction
Backpropagation	the trip back — sharing the error at the output back towards the input, which gives each knob its own “which way, and how far”
Vanishing gradients	● returning nothing but 0, so the clue for moving the knobs never gets back upstream
Skip connection, residual connection	the skip path
Divergence (in gradient descent)	the numbers running away
Normalization (batch, layer)	putting everything on 0 to 1 (dividing by the top of the range)
Learning rate	the step size
Multilayer perceptron	a form with one layer or more
Simple perceptron	a form with no layer (that is Problem 8)