Deep Black BoxOpen up the black box of AI one problem at a time, with your own hands. 21 problems in all.
Question 13
Which words to look at
The four words in green carry a tag (in the real thing, every word does).
“It” puts out a query, and the query is compared against the tags.
Your target
Get the share going to coffee above one half
This is a value the problem chose.
The shares add up to 1, so once one is over a half,
that word is being looked at more than all the others put together.
Where the tag and the query come from
x the word’s vector
— ×W_q → q the query
— ×W_k → k the tag
— ×W_v → v the contents
q ๏ผ W_q ยท x
k ๏ผ W_k ยท x
v ๏ผ W_v ยท x
The real thing learns W_q, W_k and W_v from data, and the directions are the result of that.
There is no W in this course. The directions are numbers placed by hand. No hill gets walked down to solve this one (you turn the directions by hand).
The panel just below does show you the machine settling on a W, though.
Tags (black) and the query (red)
How well the directions line up (multiply and add = the dot product. The middle line is 0)
↓ the better a word lines up, the larger its share
Attention weights (the four add up to 1. The red line is 0.50)
↓ mix the contents by those weights, and you get the next word
What to take away
What you just did is self-attention.
Build a query, a tag and some contents for each word, compare the directions, and
mix the contents by weights that add up to 1 — that is the whole mechanism. Where “it” looks changes the next word. The four shares add up to 1.
Where this pays off
What “it” refers to is not settled inside an AI.
Which earlier word it looks at changes the word that comes out. So rather than leaving it at “it” or “this”,
naming the thing gives you steadier answers.