This video introduces how ChatGPT works as a language model and then dives into building a smaller, simpler version. It explains how to prepare text data by tokenizing it into numbers and splitting it into training and validation sets. The video then demonstrates a basic 'bigram' model that predicts the next character based on the current one and discusses how to train it and generate text.
Key Takeaways
1
ChatGPT is a probabilistic language model that predicts the next word in a sequence based on prior context, and it can generate multiple different answers for the same prompt.
2
The core of large language models like GPT is the Transformer architecture, which was introduced in the 'Attention Is All You Need' paper in 2017.
3
To simplify building a language model, the video uses a character-level model trained on a small dataset called 'tiny Shakespeare,' which is a collection of all of Shakespeare's works.
4
Tokenization is the process of converting raw text into a sequence of integers, and a character-level tokenizer assigns a unique integer to each character in the vocabulary.
5
Training a Transformer involves feeding it small, random chunks of data, where each chunk contains multiple examples of predicting the next character based on various lengths of preceding context.
6
The model processes data in batches, meaning multiple chunks of text are processed simultaneously for efficiency, though each chunk is treated independently.
7
The simplest language model, a 'bigram' model, predicts the next token based only on the identity of the current token, using an embedding table to convert input integers into 'logits' (scores for the next character).
8
The 'cross-entropy' loss function is used to measure the quality of the model's predictions by comparing the predicted 'logits' to the actual target characters.
9
To generate new text, the model takes a starting sequence, predicts the next token, samples from the probabilities, and then concatenates the sampled token to the sequence, repeating the process.
10
An optimizer, such as Adam, updates the model's parameters based on the calculated gradients to reduce the loss and improve the model's ability to predict text.
No matching segments found
0:00
hi everyone so by now you have probably
0:03
heard of chat GPT it has taken the world
0:05
and AI Community by storm and it is a
0:08
system that allows you to interact with
0:10
an AI and give it text based tasks so
0:13
for example we can ask chat GPT to write
0:15
us a small Hau about how important it is
0:17
that people understand Ai and then they
0:19
can use it to improve the world and make
0:20
it more prosperous so when we run this
0:24
AI knowledge brings prosperity for all
0:26
to see Embrace its
0:27
power okay not bad and so you could see
0:30
that chpt went from left to right and
0:32
generated all these words SE sort of
0:35
sequentially now I asked it already the
0:37
exact same prompt a little bit earlier
0:40
and it generated a slightly different
0:41
outcome ai's power to grow ignorance
0:44
holds us back learn Prosperity weights
0:47
so uh pretty good in both cases and
0:49
slightly different so you can see that
0:51
chat GPT is a probabilistic system and
0:53
for any one prompt it can give us
0:55
multiple answers sort of uh replying to
0:57
it now this is just one example of a
1:00
problem people have come up with many
1:01
many examples and there are entire
1:03
websites that index interactions with
1:06
chpt and so many of them are quite
1:09
humorous explain HTML to me like I'm a
1:11
dog uh write release notes for chess 2
1:15
write a note about Elon Musk buying a
1:16
Twitter and so on so as an example uh
1:21
please write a breaking news article
1:22
about a leaf falling from a
1:23
tree uh and a shocking turn of events a
1:27
leaf has fallen from a tree in the local
1:28
park Witnesses report that the leaf
1:30
which was previously attached to a
1:31
branch of a tree attached itself and
1:33
fell to the ground very dramatic so you
1:36
can see that this is a pretty remarkable
1:38
system and it is what we call a language
1:40
model uh because it um it models the
1:44
sequence of words or characters or
1:46
tokens more generally and it knows how
1:49
sort of words follow each other in
1:51
English language and so from its
1:53
perspective what it is doing is it is
1:56
completing the sequence so I give it the
1:58
start of a sequence and it completes the
2:00
sequence with the outcome and so it's a
2:03
language model in that sense now I would
2:05
like to focus on the under the hood of
2:08
um under the hood components of what
2:10
makes CH GPT work so what is the neural
2:12
network under the hood that models the
2:14
sequence of these words and that comes
2:18
from this paper called attention is all
2:19
you need in 2017 a landmark paper a
2:23
landmark paper in AI that produced and
2:26
proposed the Transformer
2:28
architecture so GPT is uh short for
2:31
generally generatively pre-trained
2:33
Transformer so Transformer is the neuron
2:36
nut that actually does all the heavy
2:37
lifting under the hood it comes from
2:39
this paper in 2017 now if you read this
2:42
paper this uh reads like a pretty random
2:45
machine translation paper and that's
2:46
because I think the authors didn't fully
2:48
anticipate the impact that the
2:49
Transformer would have on the field and
2:51
this architecture that they produced in
2:53
the context of machine translation in
2:55
their case actually ended up taking over
2:57
uh the rest of AI in the next 5 years
3:00
after and so this architecture with
3:03
minor changes was copy pasted into a
3:06
huge amount of applications in AI in
3:08
more recent years and that includes at
3:10
the core of chat GPT now we are not
3:14
going to what I'd like to do now is I'd
3:15
like to build out something like chat
3:18
GPT but uh we're not going to be able to
3:20
of course reproduce chat GPT this is a
3:22
very serious production grade system it
3:24
is trained on uh a good chunk of
3:27
internet and then there's a lot of uh
3:29
pre-training and fine-tuning stages to
3:31
it and so it's very complicated what I'd
3:33
like to focus on is just to train a
3:36
Transformer based language model and in
3:39
our case it's going to be a character
3:40
level language model I still think that
3:43
is uh very educational with respect to
3:45
how these systems work so I don't want
3:47
to train on the chunk of Internet we
3:49
need a smaller data set in this case I
3:51
propose that we work with uh my favorite
3:53
toy data set it's called tiny
3:55
Shakespeare and um what it is is
3:57
basically it's a concatenation of all of
3:59
the works of sh Shakespeare in my
4:01
understanding and so this is all of
4:03
Shakespeare in a single file uh this
4:05
file is about 1 megab and it's just all
4:08
of
4:09
Shakespeare and what we are going to do
4:11
now is we're going to basically model
4:13
how these characters uh follow each
4:14
other so for example given a chunk of
4:17
these characters like this uh given some
4:20
context of characters in the past the
4:23
Transformer neural network will look at
4:25
the characters that I've highlighted and
4:26
is going to predict that g is likely to
4:28
come next in the sequence and it's going
4:30
to do that because we're going to train
4:32
that Transformer on Shakespeare and it's
4:34
just going to try to produce uh
4:37
character sequences that look like this
4:39
and in that process is going to model
4:41
all the patterns inside this data so
4:44
once we've trained the system i' just
4:45
like to give you a preview we can
4:48
generate infinite Shakespeare and of
4:50
course it's a fake thing that looks kind
4:52
of like
4:53
Shakespeare
4:56
um apologies for there's some Jank that
4:59
I'm not able to resolve in in here but
5:03
um you can see how this is going
5:05
character by character and it's kind of
5:07
like predicting Shakespeare like
5:09
language so verily my Lord the sites
5:12
have left the again the king coming with
5:16
my curses with precious pale and then
5:19
tranos say something else Etc and this
5:21
is just coming out of the Transformer in
5:23
a very similar manner as it would come
5:25
out in chat GPT in our case character by
5:28
character in chat GPT uh it's coming out
5:31
on the token by token level and tokens
5:33
are these sort of like little subword
5:35
pieces so they're not Word level they're
5:37
kind of like word chunk
5:39
level um and now I've already written
5:43
this entire code uh to train these
5:45
Transformers um and it is in a GitHub
5:49
repository that you can find and it's
5:50
called nanog
5:52
GPT so nanog GPT is a repository that
5:54
you can find in my GitHub and it's a
5:57
repository for training Transformers um
5:59
on any given text and what I think is
6:02
interesting about it because there's
6:03
many ways to train Transformers but this
6:05
is a very simple implementation so it's
6:07
just two files of 300 lines of code each
6:10
one file defines the GPT model the
6:12
Transformer and one file trains it on
6:15
some given Text data set and here I'm
6:17
showing that if you train it on a open
6:19
web Text data set which is a fairly
6:20
large data set of web pages then I
6:22
reproduce the the performance of
6:25
gpt2 so gpt2 is an early version of open
6:29
AI GPT uh from 2017 if I recall
6:33
correctly and I've only so far
6:34
reproduced the the smallest 124 million
6:37
parameter model uh but basically this is
6:39
just proving that the codebase is
6:40
correctly arranged and I'm able to load
6:43
the uh neural network weights that openi
6:45
has released later so you can take a
6:48
look at the finished code here in N GPT
6:51
but what I would like to do in this
6:52
lecture is I would like to basically uh
6:55
write this repository from scratch so
6:57
we're going to begin with an empty file
6:59
and we're we're going to define a
7:01
Transformer piece by piece we're going
7:04
to train it on the tiny Shakespeare data
7:06
set and we'll see how we can then uh
7:08
generate infinite Shakespeare and of
7:11
course this can copy paste to any
7:12
arbitrary Text data set uh that you like
7:15
uh but my goal really here is to just
7:16
make you understand and appreciate uh
7:19
how under the hood chat GPT works and um
7:23
really all that's required is a
7:24
Proficiency in Python and uh some basic
7:27
understanding of um calculus and
7:29
statistics
7:30
and it would help if you also see my
7:32
previous videos on the same YouTube
7:34
channel in particular my make more
7:36
series where I um Define smaller and
7:40
simpler neural network language models
7:43
uh so multi perceptrons and so on it
7:45
really introduces the language modeling
7:47
framework and then uh here in this video
7:49
we're going to focus on the Transformer
7:51
neural network itself okay so I created
7:54
a new Google collab uh jup notebook here
7:57
and this will allow me to later easily
7:59
share this code that we're going to
8:00
develop together uh with you so you can
8:02
follow along so this will be in a video
8:04
description uh later now here I've just
8:07
done some preliminaries I downloaded the
8:09
data set the tiny Shakespeare data set
8:11
at this URL and you can see that it's
8:13
about a 1 Megabyte file then here I open
8:16
the input.txt file and just read in all
8:18
the text of the string and we see that
8:20
we are working with 1 million characters
8:22
roughly and the first 1,000 characters
8:25
if we just print them out are basically
8:27
what you would expect this is the first
8:28
1,000 characters of the tiny Shakespeare
8:31
data set roughly up to here so so far so
8:35
good next we're going to take this text
8:37
and the text is a sequence of characters
8:39
in Python so when I call the set
8:41
Constructor on it I'm just going to get
8:44
the set of all the characters that occur
8:47
in this text and then I call list on
8:50
that to create a list of those
8:52
characters instead of just a set so that
8:53
I have an ordering an arbitrary ordering
8:56
and then I sort that so basically we get
8:59
just all the characters that occur in
9:01
the entire data set and they're sorted
9:03
now the number of them is going to be
9:04
our vocabulary size these are the
9:06
possible elements of our sequences and
9:09
we see that when I print here the
9:11
characters there's 65 of them in total
9:14
there's a space character and then all
9:16
kinds of special characters and then U
9:19
capitals and lowercase letters so that's
9:22
our vocabulary and that's the sort of
9:23
like possible uh characters that the
9:26
model can see or emit okay so next we
9:29
will would like to develop some strategy
9:31
to tokenize the input text now when
9:35
people say tokenize they mean convert
9:37
the raw text as a string to some
9:39
sequence of integers According to some
9:42
uh notebook According to some vocabulary
9:44
of possible elements so as an example
9:47
here we are going to be building a
9:48
character level language model so we're
9:50
simply going to be translating
9:51
individual characters into integers so
9:54
let me show you uh a chunk of code that
9:55
sort of does that for us so we're
9:57
building both the encoder and the
9:59
decoder
10:00
and let me just talk through what's
10:01
happening
10:02
here when we encode an arbitrary text
10:05
like hi there we're going to receive a
10:08
list of integers that represents that
10:11
string so for example 46 47 Etc and then
10:15
we also have the reverse mapping so we
10:17
can take this list and decode it to get
10:20
back the exact same string so it's
10:23
really just like a translation to
10:24
integers and back for arbitrary string
10:27
and for us it is done on a character
10:29
level
10:30
now the way this was achieved is we just
10:32
iterate over all the characters here and
10:34
create a lookup table from the character
10:36
to the integer and vice versa and then
10:38
to encode some string we simply
10:41
translate all the characters
10:42
individually and to decode it back we
10:45
use the reverse mapping and concatenate
10:46
all of it now this is only one of many
10:49
possible encodings or many possible sort
10:51
of tokenizers and it's a very simple one
10:54
but there's many other schemas that
10:56
people have come up with in practice so
10:58
for example Google uses a sentence
11:00
piece uh so sentence piece will also
11:02
encode text into um integers but in a
11:06
different schema and using a different
11:09
vocabulary and sentence piece is a
11:11
subword uh sort of tokenizer and what
11:14
that means is that um you're not
11:16
encoding entire words but you're not
11:17
also encoding individual characters it's
11:20
it's a subword unit level and that's
11:23
usually what's adopted in practice for
11:25
example also openai has this Library
11:26
called tick token that uses a bite pair
11:29
encode
11:30
tokenizer um and that's what GPT uses
11:33
and you can also just encode words into
11:36
like hell world into a list of integers
11:39
so as an example I'm using the Tik token
11:41
Library here I'm getting the encoding
11:43
for gpt2 or that was used for gpt2
11:46
instead of just having 65 possible
11:48
characters or tokens they have 50,000
11:52
tokens and so when they encode the exact
11:54
same string High there we only get a
11:57
list of three integers but those
11:59
integers are not between 0 and 64 they
12:02
are between Z and 5,
12:06
5,256 so basically you can trade off the
12:09
code book size and the sequence lengths
12:12
so you can have very long sequences of
12:14
integers with very small vocabularies or
12:16
we can have short um sequences of
12:20
integers with very large vocabularies
12:23
and so typically people use in practice
12:26
these subword encodings but I'd like to
12:29
keep our token ier very simple so we're
12:31
using character level tokenizer and that
12:33
means that we have very small code books
12:35
we have very simple encode and decode
12:37
functions uh but we do get very long
12:40
sequences as a result but that's the
12:42
level at which we're going to stick with
12:44
this lecture because it's the simplest
12:45
thing okay so now that we have an
12:47
encoder and a decoder effectively a
12:49
tokenizer we can tokenize the entire
12:51
training set of Shakespeare so here's a
12:54
chunk of code that does that and I'm
12:56
going to start to use the pytorch
12:57
library and specifically the torch.
12:59
tensor from the pytorch library so we're
13:01
going to take all of the text in tiny
13:03
Shakespeare encode it and then wrap it
13:06
into a torch. tensor to get the data
13:08
tensor so here's what the data tensor
13:10
looks like when I look at just the first
13:12
1,000 characters or the 1,000 elements
13:14
of it so we see that we have a massive
13:17
sequence of integers and this sequence
13:19
of integers here is basically an
13:21
identical translation of the first
13:23
10,000 characters
13:24
here so I believe for example that zero
13:27
is a new line character and maybe one
13:29
one is a space not 100% sure but from
13:32
now on the entire data set of text is
13:34
re-represented as just it's just
13:36
stretched out as a single very large uh
13:38
sequence of
13:39
integers let me do one more thing before
13:42
we move on here I'd like to separate out
13:44
our data set into a train and a
13:46
validation split so in particular we're
13:48
going to take the first 90% of the data
13:51
set and consider that to be the training
13:53
data for the Transformer and we're going
13:55
to withhold the last 10% at the end of
13:57
it to be the validation data and this
13:59
will help us understand to what extent
14:01
our model is overfitting so we're going
14:03
to basically hide and keep the
14:05
validation data on the side because we
14:07
don't want just a perfect memorization
14:09
of this exact Shakespeare we want a
14:11
neural network that sort of creates
14:13
Shakespeare like uh text and so it
14:15
should be fairly likely for it to
14:17
produce the actual like stowed away uh
14:21
true Shakespeare text um and so we're
14:25
going to use this to uh get a sense of
14:26
the overfitting okay so now we would
14:28
like to start plugging these text
14:30
sequences or integer sequences into the
14:32
Transformer so that it can train and
14:34
learn those patterns now the important
14:37
thing to realize is we're never going to
14:39
actually feed entire text into a
14:41
Transformer all at once that would be
14:42
computationally very expensive and
14:44
prohibitive so when we actually train a
14:46
Transformer on a lot of these data sets
14:48
we only work with chunks of the data set
14:50
and when we train the Transformer we
14:52
basically sample random little chunks
14:54
out of the training set and train on
14:56
just chunks at a time and these chunks
14:58
have basically some kind of a length and
15:02
some maximum length now the maximum
15:04
length typically at least in the code I
15:06
usually write is called block size you
15:09
can you can uh find it under different
15:11
names like context length or something
15:13
like that let's start with the block
15:14
size of just eight and let me look at
15:16
the first train data characters the
15:19
first block size plus one characters
15:21
I'll explain why plus one in a
15:23
second so this is the first nine
15:25
characters in the sequence in the
15:28
training set now what I'd like to point
15:30
out is that when you sample a chunk of
15:32
data like this so say the these nine
15:34
characters out of the training set this
15:37
actually has multiple examples packed
15:39
into it and uh that's because all of
15:41
these characters follow each other and
15:44
so what this thing is going to say when
15:47
we plug it into a Transformer is we're
15:49
going to actually simultaneously train
15:51
it to make prediction at every one of
15:52
these
15:53
positions now in the in a chunk of nine
15:57
characters there's actually eight indiv
15:59
ual examples packed in there so there's
16:02
the example that when 18 when in the
16:05
context of 18 47 likely comes next in a
16:08
context of 18 and 47 56 comes next in a
16:12
context of 18 47 56 57 can come next and
16:17
so on so that's the eight individual
16:19
examples let me actually spell it out
16:21
with
16:22
code so here's a chunk of code to
16:24
illustrate X are the inputs to the
16:26
Transformer it will just be the first
16:28
block size characters y will be the uh
16:33
next block size characters so it's
16:35
offset by one and that's because y are
16:37
the targets for each position in the
16:40
input and then here I'm iterating over
16:43
all the block size of eight and the
16:45
context is always all the characters in
16:47
x uh up to T and including T and the
16:51
target is always the teth character but
16:54
in the targets array y so let me just
16:57
run
16:57
this and basically it spells out what I
17:00
said in words uh these are the eight
17:02
examples hidden in a chunk of nine
17:05
characters that we uh sampled from the
17:08
training set I want to mention one more
17:11
thing we train on all the eight examples
17:14
here with context between one all the
17:17
way up to context of block size and we
17:19
train on that not just for computational
17:21
reasons because we happen to have the
17:22
sequence already or something like that
17:24
it's not just done for efficiency it's
17:26
also done um to make the Transformer
17:29
Network be used to seeing contexts all
17:32
the way from as little as one all the
17:34
way to block size and we'd like the
17:37
transform to be used to seeing
17:38
everything in between and that's going
17:40
to be useful later during inference
17:42
because while we're sampling we can
17:44
start the sampling generation with as
17:46
little as one character of context and
17:48
the Transformer knows how to predict the
17:49
next character with all the way up to
17:51
just context of one and so then it can
17:54
predict everything up to block size and
17:56
after block size we have to start
17:57
truncating because the Transformer will
17:59
will never um receive more than block
18:01
size inputs when it's predicting the
18:03
next
18:04
character Okay so we've looked at the
18:06
time dimension of the tensors that are
18:08
going to be feeding into the Transformer
18:10
there's one more Dimension to care about
18:11
and that is the batch Dimension and so
18:13
as we're sampling these chunks of text
18:16
we're going to be actually every time
18:18
we're going to feed them into a
18:18
Transformer we're going to have many
18:20
batches of multiple chunks of text that
18:22
are all like stacked up in a single
18:24
tensor and that's just done for
18:25
efficiency just so that we can keep the
18:27
gpus busy uh because they are very good
18:29
at parallel processing of um of data and
18:33
so we just want to process multiple
18:35
chunks all at the same time but those
18:37
chunks are processed completely
18:38
independently they don't talk to each
18:40
other and so on so let me basically just
18:42
generalize this and introduce a batch
18:44
Dimension here's a chunk of
18:46
code let me just run it and then I'm
18:48
going to explain what it
18:51
does so here because we're going to
18:53
start sampling random locations in the
18:55
data set to pull chunks from I am
18:57
setting the seed so that um in the
19:00
random number generator so that the
19:01
numbers I see here are going to be the
19:03
same numbers you see later if you try to
19:05
reproduce this now the batch size here
19:07
is how many independent sequences we are
19:09
processing every forward backward pass
19:12
of the
19:13
Transformer the block size as I
19:14
explained is the maximum context length
19:17
to make those predictions so let's say B
19:19
size four block size eight and then
19:21
here's how we get batch for any
19:23
arbitrary split if the split is a
19:25
training split then we're going to look
19:27
at train data otherwise at valid data
19:30
that gives us the data array and then
19:34
when I Generate random positions to grab
19:36
a chunk out of I actually grab I
19:39
actually generate batch size number of
19:42
Random offsets so because this is four
19:45
we are ex is going to be a uh four
19:47
numbers that are randomly generated
19:49
between zero and Len of data minus block
19:52
size so it's just random offsets into
19:54
the training
19:55
set and then X's as I explained are the
19:58
first first block size characters
20:00
starting at I the Y's are the offset by
20:05
one of that so just add plus one and
20:09
then we're going to get those chunks for
20:10
every one of integers I INX and use a
20:14
torch. stack to take all those uh uh
20:18
one-dimensional tensors as we saw here
20:20
and we're going to um stack them up at
20:24
rows and so they all become a row in a
20:27
4x8 tensor
20:30
so here's where I'm printing then when I
20:32
sample a batch XB and YB the inputs to
20:35
the Transformer now are the input X is
20:39
the 4x8 tensor four uh rows of eight
20:44
columns and each one of these is a chunk
20:47
of the training
20:49
set and then the targets here are in the
20:52
associated array Y and they will come in
20:54
to the Transformer all the way at the
20:56
end uh to um create the loss function
20:59
uh so they will give us the correct
21:01
answer for every single position inside
21:04
X and then these are the four
21:06
independent
21:08
rows so spelled out as we did
21:11
before uh this 4x8 array contains a
21:15
total of 32 examples and they're
21:17
completely independent as far as the
21:19
Transformer is
21:21
concerned uh so when the input is 24 the
21:25
target is 43 or rather 43 here in the Y
21:28
array
21:29
when the input is 2443 the target is
21:32
58 uh when the input is 24 43 58 the
21:35
target is 5 Etc or like when it is a 52
21:38
581 the target is
21:41
58 right so you can sort of see this
21:44
spelled out these are the 32 independent
21:46
examples packed in to a single batch of
21:49
the input X and then the desired targets
21:52
are in y and so now this integer tensor
21:57
of um X is going to feed into the
22:01
Transformer and that Transformer is
22:03
going to simultaneously process all
22:04
these examples and then look up the
22:06
correct um integers to predict in every
22:09
one of these positions in the tensor y
22:12
okay so now that we have our batch of
22:14
input that we'd like to feed into a
22:15
Transformer let's start basically
22:17
feeding this into neural networks now
22:19
we're going to start off with the
22:20
simplest possible neural network which
22:22
in the case of language modeling in my
22:23
opinion is the Byram language model and
22:26
we've covered the Byram language model
22:27
in my make more series in a lot of depth
22:30
and so here I'm going to sort of go
22:32
faster and let's just Implement pytorch
22:33
module directly that implements the byr
22:36
language
22:37
model so I'm importing the pytorch um NN
22:41
module uh for
22:43
reproducibility and then here I'm
22:45
constructing a Byram language model
22:46
which is a subass of NN
22:48
module and then I'm calling it and I'm
22:51
passing it the inputs and the targets
22:54
and I'm just printing now when the
22:56
inputs on targets come here you see that
22:58
I'm just taking the index uh the inputs
23:01
X here which I rename to idx and I'm
23:03
just passing them into this token
23:04
embedding table so it's going on here is
23:07
that here in the Constructor we are
23:10
creating a token embedding table and it
23:12
is of size vocap size by vocap
23:15
size and we're using an. embedding which
23:18
is a very thin wrapper around basically
23:21
a tensor of shape voap size by vocab
23:23
size and what's happening here is that
23:26
when we pass idx here every single
23:28
integer in our input is going to refer
23:30
to this embedding table and it's going
23:32
to pluck out a row of that embedding
23:34
table corresponding to its index so 24
23:38
here will go into the embedding table
23:40
and we'll pluck out the 24th row and
23:42
then 43 will go here and pluck out the
23:44
43d row Etc and then pytorch is going to
23:47
arrange all of this into a batch by Time
23:50
by channel uh tensor in this case batch
23:54
is four time is eight and C which is the
23:58
channels is vocab size or 65 and so
24:01
we're just going to pluck out all those
24:02
rows arrange them in a b by T by C and
24:06
now we're going to interpret this as the
24:07
logits which are basically the scores
24:10
for the next character in the sequence
24:13
and so what's happening here is we are
24:15
predicting what comes next based on just
24:17
the individual identity of a single
24:19
token and you can do that because um I
24:22
mean currently the tokens are not
24:24
talking to each other and they're not
24:25
seeing any context except for they're
24:27
just seeing themselves so I'm a f I'm a
24:29
token number five and then I can
24:32
actually make pretty decent predictions
24:34
about what comes next just by knowing
24:35
that I'm token five because some
24:37
characters uh know um C follow other
24:40
characters in in typical scenarios so we
24:43
saw a lot of this in a lot more depth in
24:45
the make more series and here if I just
24:47
run this then we currently get the
24:49
predictions the scores the lits for
24:53
every one of the 4x8 positions now that
24:56
we've made predictions about what comes
24:57
next we'd like to evaluate the loss
24:59
function and so in make more series we
25:01
saw that a good way to measure a loss or
25:03
like a quality of the predictions is to
25:06
use the negative log likelihood loss
25:08
which is also implemented in pytorch
25:09
under the name cross entropy so what we'
25:12
like to do here is loss is the cross
25:15
entropy on the predictions and the
25:18
targets and so this measures the quality
25:20
of the logits with respect to the
25:22
Targets in other words we have the
25:24
identity of the next character so how
25:26
well are we predicting the next
25:28
character based on the lits and
25:30
intuitively the correct um the correct
25:34
dimension of low jits uh depending on
25:37
whatever the target is should have a
25:38
very high number and all the other
25:40
dimensions should be very low number
25:42
right now the issue is that this won't
25:44
actually this is what we want we want to
25:46
basically output the logits and the
25:50
loss this is what we want but
25:52
unfortunately uh this won't actually run
25:56
we get an error message but intuitively
25:58
we want to uh measure this now when we
26:01
go to the pytorch um cross entropy
26:05
documentation here um we're trying to
26:08
call the cross entropy in its functional
26:10
form uh so that means we don't have to
26:12
create like a module for it but here
26:15
when we go to the documentation you have
26:17
to look into the details of how pitor
26:19
expects these inputs and basically the
26:21
issue here is ptor expects if you have
26:24
multi-dimensional input which we do
26:26
because we have a b BYT by C tensor then
26:29
it actually really wants the channels to
26:31
be the second uh Dimension here so if
26:35
you um so basically it wants a b by C
26:39
BYT instead of a b by T by C and so it's
26:43
just the details of how P torch treats
26:46
um these kinds of inputs and so we don't
26:50
actually want to deal with that so what
26:51
we're going to do instead is we need to
26:53
basically reshape our logits so here's
26:55
what I like to do I like to take
26:57
basically give names to the dimensions
26:58
so lit. shape is B BYT by C and unpack
27:01
those numbers and then let's uh say that
27:04
logits equals lit. View and we want it
27:08
to be a b * c b * T by C so just a two-
27:11
dimensional
27:13
array right so we're going to take all
27:15
the we're going to take all of these um
27:19
positions here and we're going to uh
27:21
stretch them out in a onedimensional
27:23
sequence and uh preserve the channel
27:25
Dimension as the second
27:27
dimension so we're just kind of like
27:28
stretching out the array so it's two-
27:30
dimensional and in that case it's going
27:32
to better conform to what pytorch uh
27:34
sort of expects in its Dimensions now we
27:37
have to do the same to targets because
27:39
currently targets are um of shape B by T
27:45
and we want it to be just B * T so
27:47
onedimensional now alternatively you
27:50
could always still just do minus one
27:52
because pytor will guess what this
27:54
should be if you want to lay it out uh
27:55
but let me just be explicit and say p *
27:57
t once we've reshaped this it will match
28:00
the cross entropy case and then we
28:03
should be able to evaluate our
28:06
loss okay so that R now and we can do
28:10
loss and So currently we see that the
28:12
loss is
28:14
4.87 now because our uh we have 65
28:17
possible vocabulary elements we can
28:19
actually guess at what the loss should
28:21
be and in
28:22
particular we covered negative log
28:24
likelihood in a lot of detail we are
28:26
expecting log or lawn of um 1 over 65
28:32
and negative of that so we're expecting
28:34
the loss to be about 4.1 17 but we're
28:38
getting 4.87 and so that's telling us
28:40
that the initial predictions are not uh
28:42
super diffuse they've got a little bit
28:44
of entropy and so we're guessing wrong
28:47
uh so uh yes but actually we're I a we
28:51
are able to evaluate the loss okay so
28:53
now that we can evaluate the quality of
28:55
the model on some data we'd like to also
28:58
be able to generate from the model so
28:59
let's do the generation now I'm going to
29:02
go again a little bit faster here
29:03
because I covered all this already in
29:05
previous
29:06
videos
29:07
so here's a generate function for the
29:11
model so we take some uh we take the the
29:15
same kind of input idx here and
29:19
basically this is the current uh context
29:22
of some characters in a batch in some
29:25
batch so it's also B BYT and the job of
29:28
generate is to basically take this B BYT
29:31
and extend it to be B BYT + 1 plus 2
29:33
plus 3 and so it's just basically it
29:35
continues the generation in all the
29:36
batch dimensions in the time Dimension
29:39
So that's its job and it will do that
29:41
for Max new tokens so you can see here
29:44
on the bottom there's going to be some
29:45
stuff here but on the bottom whatever is
29:47
predicted is concatenated on top of the
29:50
previous idx along the First Dimension
29:53
which is the time Dimension to create a
29:55
b BYT + one so that becomes a new idx so
29:58
the job of generate is to take a b BYT
30:00
and make it a b BYT plus 1 plus 2 plus
30:03
three as many as we want Max new tokens
30:06
so this is the generation from the model
30:08
now inside the generation what what are
30:10
we doing we're taking the current
30:12
indices we're getting the predictions so
30:15
we get uh those are in the low jits and
30:18
then the loss here is going to be
30:20
ignored because um we're not we're not
30:22
using that and we have no targets that
30:24
are sort of ground truth targets that
30:26
we're going to be comparing with
30:29
then once we get the logits we are only
30:30
focusing on the last step so instead of
30:33
a b by T by C we're going to pluck out
30:36
the negative-1 the last element in the
30:39
time Dimension because those are the
30:41
predictions for what comes next so that
30:43
gives us the logits which we then
30:44
convert to probabilities via softmax and
30:48
then we use tor. multinomial to sample
30:49
from those probabilities and we ask
30:51
pytorch to give us one sample and so idx
30:55
next will become a b by one because in
30:58
each uh one of the batch Dimensions
31:00
we're going to have a single prediction
31:01
for what comes next so this num samples
31:04
equals one will make this be a
31:06
one and then we're going to take those
31:08
integers that come from the sampling
31:11
process according to the probability
31:12
distribution given here and those
31:14
integers got just concatenated on top of
31:16
the current sort of like running stream
31:18
of integers and this gives us a b BYT +
31:21
one and then we can return that now one
31:24
thing here is you see how I'm calling
31:27
self of idx which will end up going to
31:30
the forward function I'm not providing
31:32
any Targets So currently this would give
31:34
an error because targets is uh is uh
31:37
sort of like not given so targets has to
31:39
be optional so targets is none by
31:41
default and then if targets is none then
31:45
there's no loss to create so it's just
31:47
loss is none but else all of this
31:50
happens and we can create a loss so this
31:53
will make it so um if we have the
31:56
targets we provide them and get a loss
31:58
if we have no targets it will'll just
31:59
get the
32:00
loits so this here will generate from
32:03
the model um and let's take that for a
32:07
ride
32:09
now oops so I have another code chunk
32:12
here which will generate for the model
32:14
from the model and okay this is kind of
32:16
crazy so maybe let me let me break this
32:18
down so these are the idx
32:24
right I'm creating a batch will be just
32:27
one time will be just one so I'm
32:30
creating a little one by one tensor and
32:32
it's holding a zero and the D type the
32:35
data type is uh integer so zero is going
32:38
to be how we kick off the generation and
32:40
remember that zero is uh is the element
32:44
standing for a new line character so
32:46
it's kind of like a reasonable thing to
32:47
to feed in as the very first character
32:49
in a sequence to be the new
32:52
line um so it's going to be idx which
32:55
we're going to feed in here then we're
32:56
going to ask for 100 tokens
32:59
and then. generate will continue that
33:02
now because uh generate works on the
33:05
level of batches we we then have to
33:07
index into the zero throw to basically
33:10
unplug the um the single batch Dimension
33:14
that exists and then that gives us a um
33:19
time steps just a onedimensional array
33:21
of all the indices which we will convert
33:23
to simple python list from pytorch
33:26
tensor so that that can feed into our
33:29
decode function and uh convert those
33:32
integers into text so let me bring this
33:35
back and we're generating 100 tokens
33:37
let's
33:38
run and uh here's the generation that we
33:41
achieved so obviously it's garbage and
33:43
the reason it's garbage is because this
33:45
is a totally random model so next up
33:47
we're going to want to train this model
33:49
now one more thing I wanted to point out
33:51
here is this function is written to be
33:53
General but it's kind of like ridiculous
33:56
right now because
33:58
we're feeding in all this we're building
34:00
out this context and we're concatenating
34:02
it all and we're always feeding it all
34:05
into the model but that's kind of
34:08
ridiculous because this is just a simple
34:09
Byram model so to make for example this
34:11
prediction about K we only needed this W
34:14
but actually what we fed into the model
34:16
is we fed the entire sequence and then
34:18
we only looked at the very last piece
34:20
and predicted K so the only reason I'm
34:23
writing it in this way is because right
34:25
now this is a byr model but I'd like to
34:28
keep keep this function fixed and I'd
34:29
like it to work um later when our
34:33
characters actually um basically look
34:36
further in the history and so right now
34:38
the history is not used so this looks
34:39
silly uh but eventually the history will
34:42
be used and so that's why we want to uh
34:45
do it this way so just a quick comment
34:47
on that so now we see that this is um
34:50
random so let's train the model so it
34:52
becomes a bit less random okay let's Now
34:54
train the model so first what I'm going
34:56
to do is I'm going to create a pyour
34:58
optimization object so here we are using
35:01
the optimizer ATM W um now in a make
35:05
more series we've only ever use tastic
35:07
gradi in descent the simplest possible
35:08
Optimizer which you can get using the
35:10
SGD instead but I want to use Adam which
35:13
is a much more advanced and popular
35:14
Optimizer and it works extremely well
35:17
for uh typical good setting for the
35:19
learning rate is roughly 3 E4 uh but for
35:23
very very small networks like is the
35:24
case here you can get away with much
35:26
much higher learning rates R3 or even
35:28
higher probably but let me create the
35:31
optimizer object which will basically
35:33
take the gradients and uh update the
35:35
parameters using the
35:37
gradients and then here our batch size
35:40
up above was only four so let me
35:42
actually use something bigger let's say
35:43
32 and then for some number of steps um
35:47
we are sampling a new batch of data
35:49
we're evaluating the loss uh we're
35:51
zeroing out all the gradients from the
35:53
previous step getting the gradients for
35:55
all the parameters and then using those
35:57
gradients to up update our parameters so
35:59
typical training loop as we saw in the
36:01
make more series so let me now uh run
36:05
this for say 100 iterations and let's
36:07
see what kind of losses we're going to
36:10
get so we started around
36:13
4.7 and now we're getting to down to
36:15
like 4.6 4.5 Etc so the optimization is
36:19
definitely happening but um let's uh
36:22
sort of try to increase number of
36:24
iterations and only print at the
36:26
end because we probably want train for
36:29
longer okay so we're down to 3.6
36:35
roughly roughly down to
36:41
three this is the most janky
36:46
optimization okay it's working let's
36:48
just do
36:51
10,000 and then from here we want to
36:54
copy this and hopefully that we're going
36:57
to get something reason and of course
36:58
it's not going to be Shakespeare from a
37:00
byr model but at least we see that the
37:02
loss is improving and uh hopefully we're
37:05
expecting something a bit more
37:07
reasonable okay so we're down at about
37:09
2.5 is let's see what we get okay
37:13
dramatic improvements certainly on what
37:14
we had here so let me just increase the
37:18
number of tokens okay so we see that
37:20
we're starting to get something at least
37:22
like reasonable is
37:26
um certainly not shakes spear but uh the
37:29
model is making progress so that is the
37:31
simplest possible
37:33
model so now what I'd like to do
37:37
is obviously this is a very simple model
37:40
because the tokens are not talking to
37:41
each other so given the previous context
37:43
of whatever was generated we're only
37:45
looking at the very last character to
37:47
make the predictions about what comes
37:48
next so now these uh now these tokens
37:51
have to start talking to each other and
37:53
figuring out what is in the context so
37:55
that they can make better predictions
37:56
for what comes next and this is how
37:58
we're going to kick off the uh
38:00
Transformer okay so next I took the code
38:02
that we developed in this juper notebook
38:03
and I converted it to be a script and
38:06
I'm doing this because I just want to
38:08
simplify our intermediate work into just
38:10
the final product that we have at this
38:12
point so in the top here I put all the
38:15
hyp parameters that we to find I
38:17
introduced a few and I'm going to speak
38:18
to that in a little bit otherwise a lot
38:21
of this should be recognizable uh
38:23
reproducibility read data get the
38:26
encoder and the decoder create the train
38:28
into splits uh use the uh kind of like
38:31
data loader um that gets a batch of the
38:34
inputs and Targets this is new and I'll
38:37
talk about it in a second now this is
38:39
the Byram language model that we
38:41
developed and it can forward and give us
38:43
a logits and loss and it can
38:46
generate and then here we are creating
38:48
the optimizer and this is the training
38:51
Loop so everything here should look
38:53
pretty familiar now some of the small
38:55
things that I added number one I added
38:58
the ability to run on a GPU if you have
39:00
it so if you have a GPU then you can
39:03
this will use Cuda instead of just CPU
39:05
and everything will be a lot more faster
39:07
now when device becomes Cuda then we
39:10
need to make sure that when we load the
39:11
data we move it to
39:13
device when we create the model we want
39:16
to move uh the model parameters to
39:18
device so as an example here we have the
39:21
N an embedding table and it's got a
39:23
weight inside it which stores the uh
39:26
sort of lookup table so so that would be
39:28
moved to the GPU so that all the
39:30
calculations here happen on the GPU and
39:32
they can be a lot faster and then
39:34
finally here when I'm creating the
39:35
context that feeds in to generate I have
39:38
to make sure that I create it on the
39:39
device number two what I introduced is
39:43
uh the fact that here in the training
39:47
Loop here I was just printing the um l.
39:51
item inside the training Loop but this
39:53
is a very noisy measurement of the
39:55
current loss because every batch will be
39:57
more or less lucky and so what I want to
40:00
do usually um is uh I have an estimate
40:03
loss function and the estimate loss
40:06
basically then um goes up here and it
40:11
averages up the loss over multiple
40:13
batches so in particular we're going to
40:15
iterate eval iter times and we're going
40:18
to basically get our loss and then we're
40:20
going to get the average loss for both
40:22
splits and so this will be a lot less
40:24
noisy so here when we call the estimate
40:26
loss we're we're going to report the uh
40:28
pretty accurate train and validation
40:31
loss now when we come back up you'll
40:33
notice a few things here I'm setting the
40:35
model to evaluation phase and down here
40:38
I'm resetting it back to training phase
40:41
now right now for our model as is this
40:43
doesn't actually do anything because the
40:45
only thing inside this model is this uh
40:47
nn. embedding and um this this um
40:52
Network would behave both would behave
40:54
the same in both evaluation mode and
40:55
training mode we have no drop off layers
40:58
we have no batm layers Etc but it is a
41:00
good practice to Think Through what mode
41:02
your neural network is in because some
41:05
layers will have different Behavior Uh
41:07
at inference time or training time and
41:11
there's also this context manager torch
41:13
up nograd and this is just telling
41:15
pytorch that everything that happens
41:16
inside this function we will not call do
41:19
backward on and so pytorch can be a lot
41:22
more efficient with its memory use
41:23
because it doesn't have to store all the
41:25
intermediate variables uh because we're
41:27
never going to call backward and so it
41:29
can it can be a lot more memory
41:31
efficient in that way so also a good
41:33
practice to tpy torch when we don't
41:35
intend to do back
41:37
propagation so right now this script is
41:40
about 120 lines of code of and that's
41:43
kind of our starter code I'm calling it
41:46
b.p and I'm going to release it later
41:49
now running this
41:50
script gives us output in the terminal
41:53
and it looks something like this it
41:55
basically as I ran this code uh it was
41:58
giving me the train loss and Val loss
42:00
and we see that we convert to somewhere
42:01
around
42:02
2.5 with the pyr model and then here's
42:05
the sample that we produced at the
42:07
end and so we have everything packaged
42:10
up in the script and we're in a good
42:12
position now to iterate on this okay so
42:14
we are almost ready to start writing our
42:16
very first self attention block for
42:18
processing these uh tokens now before we
42:23
actually get there I want to get you
42:24
used to a mathematical trick that is
42:26
used in the self attention inside a
42:28
Transformer and is really just like at
42:30
the heart of an an efficient
42:32
implementation of self attention and so
42:35
I want to work with this toy example to
42:36
just get you used to this operation and
42:38
then it's going to make it much more
42:40
clear once we actually get to um to it
42:43
uh in the script
42:44
again so let's create a b BYT by C where
42:48
BT and C are just 48 and two in the toy
42:50
example and these are basically channels
42:53
and we have uh batches and we have the
42:56
time component and we have information
42:58
at each point in the sequence so
43:01
see now what we would like to do is we
43:03
would like these um tokens so we have up
43:06
to eight tokens here in a batch and
43:09
these eight tokens are currently not
43:10
talking to each other and we would like
43:12
them to talk to each other we'd like to
43:13
couple them and in particular we don't
43:17
we we want to couple them in a very
43:19
specific way so the token for example at
43:21
the fifth location it should not
43:24
communicate with tokens in the sixth
43:26
seventh and eighth location
43:28
because uh those are future tokens in
43:30
the sequence the token on the fifth
43:32
location should only talk to the one in
43:34
the fourth third second and first so
43:36
it's only so information only flows from
43:39
previous context to the current time
43:40
step and we cannot get any information
43:43
from the future because we are about to
43:44
try to predict the
43:46
future so what is the easiest way for
43:49
tokens to communicate okay the easiest
43:52
way I would say is okay if we're up to
43:55
if we're a fifth token and I'd like to
43:56
communicate with my past the simplest
43:58
way we can do that is to just do a
44:01
weight is to just do an average of all
44:04
the um of all the preceding elements so
44:06
for example if I'm the fif token I would
44:08
like to take the channels uh that make
44:11
up that are information at my step but
44:14
then also the channels from the fourth
44:16
step third step second step and the
44:17
first step I'd like to average those up
44:20
and then that would become sort of like
44:21
a feature Vector that summarizes me in
44:24
the context of my history now of course
44:26
just doing a sum or like an average is
44:29
an extremely weak form of interaction
44:30
like this communication is uh extremely
44:32
lossy we've lost a ton of information
44:34
about the spatial Arrangements of all
44:36
those tokens uh but that's okay for now
44:38
we'll see how we can bring that
44:39
information back later for now what we
44:42
would like to do is for every single
44:44
batch element independently for every
44:47
teeth token in that sequence we'd like
44:50
to now calculate the average of all the
44:53
vectors in all the previous tokens and
44:56
also at this token so let's write that
44:59
out um I have a small snippet here and
45:02
instead of just fumbling around let me
45:04
just copy paste it and talk to
45:06
it so in other words we're going to
45:08
create X and B is short for bag of words
45:12
because bag of words is um is kind of
45:15
like um a term that people use when you
45:18
are just averaging up things so this is
45:19
just a bag of words basically there's a
45:21
word stored on every one of these eight
45:24
locations and we're doing a bag of words
45:25
we're just averaging
45:27
so in the beginning we're going to say
45:29
that it's just initialized at Zero and
45:31
then I'm doing a for Loop here so we're
45:32
not being efficient yet that's coming
45:34
but for now we're just iterating over
45:36
all the batch Dimensions independently
45:38
iterating over time and then the
45:41
previous uh tokens are at this uh batch
45:45
Dimension and then everything up to and
45:48
including the teeth token okay so when
45:51
we slice out X in this way X prev
45:54
Becomes of shape um how many T elements
45:58
there were in the past and then of
46:00
course C so all the two-dimensional
46:03
information from these little tokens so
46:05
that's the previous uh sort of chunk of
46:08
um tokens from my current sequence and
46:12
then I'm just doing the average or the
46:14
mean over the zero Dimension so I'm
46:16
averaging out the time here and I'm just
46:19
going to get a little c one dimensional
46:21
Vector which I'm going to store in X bag
46:24
of words so I can run this and and uh
46:28
this is not going to be very informative
46:30
because let's see so this is X of Zer so
46:33
this is the zeroth batch element and
46:35
then expo at zero now you see how the at
46:40
the first location here you see that the
46:42
two are equal and that's because it's
46:45
we're just doing an average of this one
46:47
token but here this one is now an
46:50
average of these two and now this one is
46:53
an average of these
46:55
three and so on
46:58
so uh and this last one is the average
47:01
of all of these elements so vertical
47:03
average just averaging up all the tokens
47:05
now gives this outcome
47:07
here so this is all well and good uh but
47:10
this is very inefficient now the trick
47:13
is that we can be very very efficient
47:14
about doing this using matrix
47:16
multiplication so that's the
47:18
mathematical trick and let me show you
47:20
what I mean let's work with the toy
47:21
example here let me run it and I'll
47:24
explain I have a simple Matrix here that
47:28
is a 3X3 of all ones a matrix B of just
47:31
random numbers and it's a 3x2 and a
47:34
matrix C which will be 3x3 multip 3x2
47:37
which will give out a 3x2 so here we're
47:40
just using um matrix multiplication so a
47:44
multiply B gives us
47:46
C okay so how are these numbers in C um
47:51
achieved right so this number in the top
47:54
left is the first row of a dot product
47:58
with the First Column of B and since all
48:01
the the row of a right now is all just
48:03
ones then the do product here with with
48:06
this column of B is just going to do a
48:08
sum of these of this column so 2 + 6 + 6
48:12
is
48:13
14 the element here in the output of C
48:16
is also the first column here the first
48:18
row of a multiplied now with the second
48:20
column of B so 7 + 4 + 5 is 16 now you
48:25
see that there's repeating elements here
48:26
so this 14 again is because this row is
48:29
again all ones and it's multiplying the
48:30
First Column of B so we get 14 and this
48:33
one is and so on so this last number
48:36
here is the last row do product last
48:39
column now the trick here is uh the
48:42
following this is just a boring number
48:45
of um it's just a boring array of all
48:48
ones but torch has this function called
48:51
Trail which is short for a
48:54
triangular uh something like that and
48:57
you can wrap it in torch up once and it
48:58
will just return the lower triangular
49:00
portion of this
49:04
okay so now it will basically zero out
49:07
uh these guys here so we just get the
49:08
lower triangular part well what happens
49:10
if we do
49:14
that so now we'll have a like this and B
49:18
like this and now what are we getting
49:19
here in C well what is this number well
49:22
this is the first row times the First
49:25
Column and because this is zeros
49:29
uh these elements here are now ignored
49:31
so we just get a two and then this
49:33
number here is the first row times the
49:35
second column and because these are
49:37
zeros they get ignored and it's just
49:39
seven this seven multiplies this one but
49:43
look what happened here because this is
49:44
one and then zeros we what ended up
49:47
happening is we're just plucking out the
49:48
row of this row of B and that's what we
49:51
got now here we have one 1 Z so here 110
49:57
do product with these two columns will
49:59
now give us 2 + 6 which is 8 and 7 + 4
50:02
which is 11 and because this is 111 we
50:05
ended up with the addition of all of
50:08
them and so basically depending on how
50:10
many ones and zeros we have here we are
50:13
basically doing a sum currently of a
50:17
variable number of these rows and that
50:19
gets deposited into
50:21
C So currently we're doing sums because
50:23
these are ones but we can also do
50:25
average right and you can start to see
50:28
how we could do average uh of the rows
50:30
of B uh sort of in an incremental
50:33
fashion because we don't have to we can
50:35
basically normalize these rows so that
50:38
they sum to one and then we're going to
50:39
get an average so if we took a and then
50:42
we did aals
50:44
aide torch. sum in the um of a in the um
50:52
oneth Dimension and then let's keep them
50:55
as true so so therefore the broadcasting
50:58
will work out so if I rerun this you see
51:01
now that these rows now sum to one so
51:04
this row is one this row is 0. 5.5 Z and
51:07
here we get 1/3 and now when we do a
51:10
multiply B what are we getting here we
51:13
are just getting the first row first row
51:16
here now we are getting the average of
51:18
the first two
51:20
rows okay so 2 and six average is four
51:23
and four and seven average is
51:25
5.5 and on the bottom here we are now
51:28
getting the average of these three rows
51:31
so the average of all of elements of B
51:34
are now deposited here and so you can
51:37
see that by manipulating these uh
51:40
elements of this multiplying Matrix and
51:42
then multiplying it with any given
51:44
Matrix we can do these averages in this
51:47
incremental fashion because we just get
51:50
um and we can manipulate that based on
51:53
the elements of a okay so that's very
51:55
convenient so let's let's swing back up
51:57
here and see how we can vectorize this
51:59
and make it much more efficient using
52:01
what we've learned so in
52:04
particular we are going to produce an
52:06
array a but here I'm going to call it we
52:08
short for weights but this is our
52:12
a and this is how much of every row we
52:15
want to average up and it's going to be
52:17
an average because you can see that
52:18
these rows sum to
52:20
one so this is our a and then our B in
52:24
this example of course is X
52:28
so what's going to happen here now is
52:29
that we are going to have an expo
52:32
2 and this Expo 2 is going to be way
52:37
multiplying
52:38
RX so let's think this true way is T BYT
52:43
and this is Matrix multiplying in
52:45
pytorch a b by T by
52:47
C and it's giving us uh different what
52:51
shape so pytorch will come here and it
52:53
will see that these shapes are not the
52:55
same so it will create a batch Dimension
52:57
here and this is a batched matrix
53:00
multiply and so it will apply this
53:02
matrix multiplication in all the batch
53:04
elements um in parallel and individually
53:08
and then for each batch element there
53:10
will be a t BYT multiplying T by C
53:13
exactly as we had
53:16
below so this will now create B by T by
53:21
C and Expo 2 will now become identical
53:25
to Expo
53:28
so we can see that torch. all close of
53:32
xbo and xbo 2 should be true
53:36
now so this kind of like convinces us
53:39
that uh these are in fact um the same so
53:43
xbo and xbo 2 if I just print
53:47
them uh okay we're not going to be able
53:50
to okay we're not going to be able to
53:52
just stare it down but
53:54
um well let me try Expo basically just
53:57
at the zeroth element and Expo two at
53:58
the zeroth element so just the first
54:00
batch and we should see that this and
54:02
that should be identical which they
54:05
are right so what happened here the
54:07
trick is we were able to use batched
54:10
Matrix multiply to do this uh
54:13
aggregation really and it's a weighted
54:16
aggregation and the weights are
54:18
specified in this um T BYT array and
54:21
we're basically doing weighted sums and
54:24
uh these weighted sums are are U
54:26
according to uh the weights inside here
54:29
they take on sort of this triangular
54:31
form and so that means that a token at
54:34
the teth dimension will only get uh sort
54:36
of um information from the um tokens
54:40
perceiving it so that's exactly what we
54:41
want and finally I would like to rewrite
54:44
it in one more way and we're going to
54:46
see why that's useful so this is the
54:49
third version and it's also identical to
54:51
the first and second but let me talk
54:53
through it it uses
54:55
softmax so Trill here is this Matrix
55:00
lower triangular
55:02
ones way begins as all
55:05
zero okay so if I just print way in the
55:08
beginning it's all zero then I
55:12
used masked fill so what this is doing
55:15
is we. masked fill it's all zeros and
55:18
I'm saying for all the elements where
55:21
Trill is equal equal Z make them be
55:24
negative Infinity so all the elements
55:26
where Trill is zero will become negative
55:28
Infinity now so this is what we get and
55:32
then the final line here is
55:36
softmax so if I take a softmax along
55:39
every single so dim is negative one so
55:41
along every single row if I do softmax
55:44
what is that going to
55:47
do well softmax is um is also like a
55:52
normalization operation right and so
55:55
spoiler alert you get the exact same
55:58
Matrix let me bring back to
56:00
softmax and recall that in softmax we're
56:03
going to exponentiate every single one
56:05
of these and then we're going to divide
56:07
by the sum and so if we exponentiate
56:10
every single element here we're going to
56:11
get a one and here we're going to get uh
56:14
basically zero 0 z0 Z everywhere else
56:17
and then when we normalize we just get
56:19
one here we're going to get one one and
56:22
then zeros and then softmax will again
56:25
divide and this will give us 5.5 and so
56:27
on and so this is also the uh the same
56:31
way to produce uh this mask now the
56:34
reason that this is a bit more
56:35
interesting and the reason we're going
56:36
to end up using it in self
56:38
attention is that these weights here
56:42
begin uh with zero and you can think of
56:45
this as like an interaction strength or
56:47
like an affinity so basically it's
56:49
telling us how much of each uh token
56:52
from the past do we want to Aggregate
56:55
and average up
56:57
and then this line is saying tokens from
57:00
the past cannot communicate by setting
57:03
them to negative Infinity we're saying
57:04
that we will not aggregate anything from
57:06
those
57:07
tokens and so basically this then goes
57:10
through softmax and through the weighted
57:11
and this is the aggregation through
57:13
matrix
57:14
multiplication and so what this is now
57:16
is you can think of these as um these
57:19
zeros are currently just set by us to be
57:22
zero but a quick preview is that these
57:25
affinities between the tokens are not
57:27
going to be just constant at zero
57:29
they're going to be data dependent these
57:31
tokens are going to start looking at
57:33
each other and some tokens will find
57:35
other tokens more or less interesting
57:38
and depending on what their values are
57:40
they're going to find each other
57:41
interesting to different amounts and I'm
57:43
going to call those affinities I think
57:45
and then here we are saying the future
57:47
cannot communicate with the past we're
57:49
we're going to clamp them and then when
57:52
we normalize and sum we're going to
57:54
aggregate uh sort of their values
57:56
depending on how interesting they find
57:57
each other and so that's the preview for
57:59
self attention and basically long story
58:03
short from this entire section is that
58:05
you can do weighted aggregations of your
58:08
past
58:09
Elements by having by using matrix
58:12
multiplication of a lower triangular
58:15
fashion and then the elements here in
58:17
the lower triangular part are telling
58:19
you how much of each element uh fuses
58:22
into this position so we're going to use
58:24
this trick now to develop the self
58:26
attention block block so first let's get
58:28
some quick preliminaries out of the way
58:30
first the thing I'm kind of bothered by
58:32
is that you see how we're passing in
58:33
vocap size into the Constructor there's
58:35
no need to do that because vocap size is
58:37
already defined uh up top as a global
58:39
variable so there's no need to pass this
58:41
stuff
58:42
around next what I want to do is I don't
58:45
want to actually create I want to create
58:47
like a level of indirection here where
58:48
we don't directly go to the embedding
58:50
for the um logits but instead we go
58:53
through this intermediate phase because
58:54
we're going to start making that bigger
58:57
so let me introduce a new variable n
59:00
embed it shorted for number of embedding
59:03
Dimensions so
59:05
nbed here will be say 32 that was a
59:09
suggestion from GitHub co-pilot by the
59:11
way um it also suest 32 which is a good
59:14
number so this is an embedding table and
59:17
only 32 dimensional
59:19
embeddings so then here this is not
59:21
going to give us logits directly instead
59:24
this is going to give us token
59:25
embeddings that's I'm going to call it
59:27
and then to go from the token Tings to
59:29
the logits we're going to need a linear
59:31
layer so self. LM head let's call it
59:34
short for language modeling head is n
59:36
and linear from n ined up to vocap size
59:40
and then when we swing over here we're
59:41
actually going to get the loits by
59:44
exactly what the co-pilot says now we
59:46
have to be careful here because this C
59:48
and this C are not equal um this is nmed
59:52
C and this is vocap size so let's just
59:56
say that n ined is equal to
59:58
C and then this just creates one spous
1:00:01
layer of interaction through a linear
1:00:03
layer but uh this should basically
1:00:11
run so we see that this runs and uh this
1:00:15
currently looks kind of spous but uh
1:00:17
we're going to build on top of this now
1:00:19
next up so far we've taken these indices
1:00:22
and we've encoded them based on the
1:00:24
identity of the uh tokens in inside idx
1:00:28
the next thing that people very often do
1:00:30
is that we're not just encoding the
1:00:32
identity of these tokens but also their
1:00:34
position so we're going to have a second
1:00:36
position uh embedding table here so
1:00:38
self. position embedding table is an an
1:00:41
embedding of block size by an embed and
1:00:44
so each position from zero to block size
1:00:46
minus one will also get its own
1:00:48
embedding vector and then here first let
1:00:51
me decode B BYT from idx do
1:00:54
shape and then here we're also going to
1:00:57
have a pause embedding which is the
1:00:58
positional embedding and these are this
1:01:00
is to arrange so this will be basically
1:01:03
just integers from Z to T minus one and
1:01:06
all of those integers from 0 to T minus
1:01:08
one get embedded through the table to
1:01:10
create a t by
1:01:11
C and then here this gets renamed to
1:01:15
just say x and x will be the addition of
1:01:19
the token embeddings with the positional
1:01:21
embeddings and here the broadcasting
1:01:23
note will work out so B by T by C plus T
1:01:25
by C
1:01:26
this gets right aligned a new dimension
1:01:29
of one gets added and it gets
1:01:30
broadcasted across
1:01:32
batch so at this point x holds not just
1:01:35
the token identities but the positions
1:01:38
at which these tokens occur and this is
1:01:40
currently not that useful because of
1:01:42
course we just have a simple byr model
1:01:43
so it doesn't matter if you're in the
1:01:44
fifth position the second position or
1:01:46
wherever it's all translation invariant
1:01:48
at this stage uh so this information
1:01:50
currently wouldn't help uh but as we
1:01:53
work on the self attention block we'll
1:01:54
see that this starts to matter
1:02:00
okay so now we get the Crux of self
1:02:01
attention so this is probably the most
1:02:03
important part of this video to
1:02:05
understand we're going to implement a
1:02:07
small self attention for a single
1:02:09
individual head as they're called so we
1:02:11
start off with where we were so all of
1:02:14
this code is familiar so right now I'm
1:02:17
working with an example where I Chang
1:02:18
the number of channels from 2 to 32 so
1:02:20
we have a 4x8 arrangement of tokens and
1:02:24
each to and the information each token
1:02:27
is currently 32 dimensional but we just
1:02:29
are working with random
1:02:30
numbers now we saw here that the code as
1:02:34
we had it before does a uh simple weight
1:02:38
simple average of all the past tokens
1:02:41
and the current token so it's just the
1:02:43
previous information and current
1:02:44
information is just being mixed together
1:02:46
in an average and that's what this code
1:02:48
currently achieves and it Doo by
1:02:50
creating this lower triangular structure
1:02:52
which allows us to mask out this uh we
1:02:56
uh Matrix that we create so we mask it
1:02:59
out and then we normalize it and
1:03:01
currently when we initialize the
1:03:04
affinities between all the different
1:03:06
sort of tokens or nodes I'm going to use
1:03:09
those terms
1:03:10
interchangeably so when we initialize
1:03:12
the affinities between all the different
1:03:13
tokens to be zero then we see that way
1:03:16
gives us this um structure where every
1:03:18
single row has these um uniform numbers
1:03:22
and so that's what that's what then uh
1:03:25
in this Matrix multiply makes it so that
1:03:27
we're doing a simple
1:03:29
average now we don't actually want this
1:03:32
to be all uniform because different uh
1:03:36
tokens will find different other tokens
1:03:39
more or less interesting and we want
1:03:40
that to be data dependent so for example
1:03:42
if I'm a vowel then maybe I'm looking
1:03:45
for consonants in my past and maybe I
1:03:47
want to know what those consonants are
1:03:48
and I want that information to flow to
1:03:50
me and so I want to now gather
1:03:53
information from the past but I want to
1:03:55
do it in the data dependent way and this
1:03:57
is the problem that self attention
1:03:58
solves now the way self attention solves
1:04:01
this is the following every single node
1:04:04
or every single token at each position
1:04:07
will emit two vectors it will emit a
1:04:10
query and it will emit a
1:04:13
key now the query Vector roughly
1:04:15
speaking is what am I looking for and
1:04:19
the key Vector roughly speaking is what
1:04:21
do I
1:04:22
contain and then the way we get
1:04:24
affinities between these uh tokens now
1:04:27
in a sequence is we basically just do a
1:04:30
do product between the keys and the
1:04:32
queries so my query dot products with
1:04:35
all the keys of all the other tokens and
1:04:38
that dot product now becomes
1:04:42
wayy and so um if the key and the query
1:04:46
are sort of aligned they will interact
1:04:48
to a very high amount and then I will
1:04:50
get to learn more about that specific
1:04:53
token as opposed to any other token in
1:04:55
the sequence
1:04:56
so let's implement this
1:05:01
now we're going to implement a
1:05:04
single what's called head of self
1:05:07
attention so this is just one head
1:05:10
there's a hyper parameter involved with
1:05:11
these heads which is the head size and
1:05:14
then here I'm initializing linear
1:05:16
modules and I'm using bias equals false
1:05:18
so these are just going to apply a
1:05:19
matrix multiply with some fixed
1:05:22
weights and now let me produce a key and
1:05:26
q k and Q by forwarding these modules on
1:05:30
X so the size of this will now
1:05:33
become B by T by 16 because that is the
1:05:37
head size and the same here B by T by
1:05:45
16 so this being the head size so you
1:05:48
see here that when I forward this linear
1:05:50
on top of my X all the tokens in all the
1:05:53
positions in the B BYT Arrangement all
1:05:55
of them them in parallel and
1:05:57
independently produce a key and a query
1:06:00
so no communication has happened
1:06:02
yet but the communication comes now all
1:06:04
the queries will do product with all the
1:06:08
keys so basically what we want is we
1:06:10
want way now or the affinities between
1:06:12
these to be query multiplying key but we
1:06:17
have to be careful with uh we can't
1:06:18
Matrix multiply this we actually need to
1:06:20
transpose uh K but we have to be also
1:06:23
careful because these are when you have
1:06:26
The Bash Dimension so in particular we
1:06:28
want to transpose uh the last two
1:06:30
dimensions dimension1 and dimension -2
1:06:34
so
1:06:36
-21 and so this Matrix multiply now will
1:06:40
basically do the following B by T by
1:06:44
16 Matrix multiplies B by 16 by T to
1:06:49
give us B by T by
1:06:54
T right
1:06:56
so for every row of B we're now going to
1:06:59
have a t Square Matrix giving us the
1:07:01
affinities and these are now the way so
1:07:04
they're not zeros they are now coming
1:07:06
from this dot product between the keys
1:07:08
and the queries so this can now run I
1:07:11
can I can run this and the weighted
1:07:14
aggregation now is a function in a data
1:07:16
Bandon manner between the keys and
1:07:18
queries of these nodes so just
1:07:21
inspecting what happened
1:07:23
here the way takes on this form
1:07:26
and you see that before way was uh just
1:07:29
a constant so it was applied in the same
1:07:31
way to all the batch elements but now
1:07:33
every single batch elements will have
1:07:35
different sort of we because uh every
1:07:38
single batch element contains different
1:07:39
uh tokens at different positions and so
1:07:42
this is not data dependent so when we
1:07:45
look at just the zeroth uh Row for
1:07:47
example in the input these are the
1:07:50
weights that came out and so you can see
1:07:52
now that they're not just exactly
1:07:53
uniform um and in particular as an
1:07:56
example here for the last row this was
1:07:58
the eighth token and the eighth token
1:08:00
knows what content it has and it knows
1:08:02
at what position it's in and now the E
1:08:05
token based on that uh creates a query
1:08:09
hey I'm looking for this kind of stuff
1:08:11
um I'm a vowel I'm on the E position I'm
1:08:13
looking for any consonant at positions
1:08:15
up to four and then all the nodes get to
1:08:18
emit keys and maybe one of the channels
1:08:21
could be I am a I am a consonant and I
1:08:23
am in a position up to four and that
1:08:26
that key would have a high number in
1:08:28
that specific Channel and that's how the
1:08:30
query and the key when they do product
1:08:32
they can find each other and create a
1:08:33
high affinity and when they have a high
1:08:35
Affinity like say uh this token was
1:08:38
pretty interesting to uh to this eighth
1:08:41
token when they have a high Affinity
1:08:44
then through the softmax I will end up
1:08:46
aggregating a lot of its information
1:08:47
into my position and so I'll get to
1:08:50
learn a lot about
1:08:52
it now just this we're looking at way
1:08:56
after this has already happened um let
1:08:59
me erase this operation as well so let
1:09:01
me erase the masking and the softmax
1:09:03
just to show you the under the hood
1:09:05
internals and how that works so without
1:09:07
the masking in the softmax Whey comes
1:09:10
out like this right this is the outputs
1:09:12
of the do products um and these are the
1:09:14
raw outputs and they take on values from
1:09:16
negative you know two to positive two
1:09:19
Etc so that's the raw interactions and
1:09:22
raw affinities between all the nodes but
1:09:25
now if I'm going if I'm a fifth node I
1:09:27
will not want to aggregate anything from
1:09:29
the sixth node seventh node and the
1:09:30
eighth node so actually we use the upper
1:09:33
triangular masking so those are not
1:09:36
allowed to
1:09:37
communicate and now we actually want to
1:09:40
have a nice uh distribution uh so we
1:09:43
don't want to aggregate negative .11 of
1:09:45
this node that's crazy so instead we
1:09:47
exponentiate and normalize and now we
1:09:49
get a nice distribution that sums to one
1:09:52
and this is telling us now in the data
1:09:53
dependent manner how much of information
1:09:55
to aggregate from any of these tokens in
1:09:57
the
1:09:59
past so that's way and it's not zeros
1:10:02
anymore but but it's calculated in this
1:10:04
way now there's one more uh part to a
1:10:08
single self attention head and that is
1:10:11
that when we do the aggregation we don't
1:10:12
actually aggregate the tokens exactly we
1:10:15
aggregate we produce one more value here
1:10:17
and we call that the
1:10:20
value so in the same way that we
1:10:22
produced p and query we're also going to
1:10:24
create a value
1:10:26
and
1:10:27
then here we don't
1:10:30
aggregate X we calculate a v which is
1:10:34
just achieved by uh propagating this
1:10:37
linear on top of X again and then we
1:10:41
output way multiplied by V so V is the
1:10:44
elements that we aggregate or the the
1:10:46
vectors that we aggregate instead of the
1:10:48
raw
1:10:49
X and now of course uh this will make it
1:10:52
so that the output here of this single
1:10:53
head will be 16 dimensional because that
1:10:55
is the head
1:10:57
size so you can think of X as kind of
1:11:00
like private information to this token
1:11:02
if you if you think about it that way so
1:11:04
X is kind of private to this token so
1:11:06
I'm a fifth token at some and I have
1:11:08
some identity and uh my information is
1:11:11
kept in Vector X and now for the
1:11:14
purposes of the single head here's what
1:11:16
I'm interested in here's what I have and
1:11:20
if you find me interesting here's what I
1:11:22
will communicate to you and that's
1:11:24
stored in v and so V is the thing that
1:11:26
gets aggregated for the purposes of this
1:11:28
single head between the different
1:11:31
notes and that's uh basically the self
1:11:34
attention mechanism this is this is what
1:11:37
it does there are a few notes that I
1:11:39
would make like to make about attention
1:11:42
number one attention is a communication
1:11:44
mechanism you can really think about it
1:11:46
as a communication mechanism where you
1:11:48
have a number of nodes in a directed
1:11:50
graph where basically you have edges
1:11:52
pointed between noes like
1:11:54
this and what happens is every node has
1:11:57
some Vector of information and it gets
1:11:59
to aggregate information via a weighted
1:12:01
sum from all of the nodes that point to
1:12:04
it and this is done in a data dependent
1:12:06
manner so depending on whatever data is
1:12:08
actually stored that you should not at
1:12:09
any point in time now our graph doesn't
1:12:13
look like this our graph has a different
1:12:15
structure we have eight nodes because
1:12:17
the block size is eight and there's
1:12:19
always eight to
1:12:20
tokens and uh the first node is only
1:12:23
pointed to by itself the second node is
1:12:26
pointed to by the first node and itself
1:12:28
all the way up to the eighth node which
1:12:30
is pointed to by all the previous nodes
1:12:32
and itself and so that's the structure
1:12:35
that our directed graph has or happens
1:12:37
happens to have in Auto regressive sort
1:12:39
of scenario like language modeling but
1:12:41
in principle attention can be applied to
1:12:43
any arbitrary directed graph and it's
1:12:44
just a communication mechanism between
1:12:46
the nodes the second note is that notice
1:12:49
that there is no notion of space so
1:12:51
attention simply acts over like a set of
1:12:54
vectors in this graph and so by default
1:12:57
these nodes have no idea where they are
1:12:58
positioned in the space and that's why
1:13:00
we need to encode them positionally and
1:13:02
sort of give them some information that
1:13:04
is anchored to a specific position so
1:13:06
that they sort of know where they are
1:13:09
and this is different than for example
1:13:10
from convolution because if you're run
1:13:12
for example a convolution operation over
1:13:13
some input there's a very specific sort
1:13:16
of layout of the information in space
1:13:18
and the convolutional filters sort of
1:13:20
act in space and so it's it's not like
1:13:24
an attention in ATT ention is just a set
1:13:26
of vectors out there in space they
1:13:28
communicate and if you want them to have
1:13:30
a notion of space you need to
1:13:31
specifically add it which is what we've
1:13:33
done when we calculated the um relative
1:13:36
the positional encode encodings and
1:13:38
added that information to the vectors
1:13:40
the next thing that I hope is very clear
1:13:42
is that the elements across the batch
1:13:44
Dimension which are independent examples
1:13:46
never talk to each other they're always
1:13:47
processed independently and this is a
1:13:49
batched matrix multiply that applies
1:13:51
basically a matrix multiplication uh
1:13:53
kind of in parallel across the batch
1:13:54
dimension so maybe it would be more
1:13:56
accurate to say that in this analogy of
1:13:58
a directed graph we really have because
1:14:01
the back size is four we really have
1:14:03
four separate pools of eight nodes and
1:14:06
those eight nodes only talk to each
1:14:07
other but in total there's like 32 nodes
1:14:09
that are being processed uh but there's
1:14:11
um sort of four separate pools of eight
1:14:14
you can look at it that way the next
1:14:16
note is that here in the case of
1:14:18
language modeling uh we have this
1:14:20
specific uh structure of directed graph
1:14:23
where the future tokens will not
1:14:25
communicate to the Past tokens but this
1:14:28
doesn't necessarily have to be the
1:14:29
constraint in the general case and in
1:14:31
fact in many cases you may want to have
1:14:33
all of the uh noes talk to each other uh
1:14:36
fully so as an example if you're doing
1:14:38
sentiment analysis or something like
1:14:39
that with a Transformer you might have a
1:14:41
number of tokens and you may want to
1:14:43
have them all talk to each other fully
1:14:45
because later you are predicting for
1:14:47
example the sentiment of the sentence
1:14:49
and so it's okay for these NOS to talk
1:14:51
to each other and so in those cases you
1:14:54
will use an encoder block of self
1:14:56
attention and uh all it means that it's
1:14:58
an encoder block is that you will delete
1:15:01
this line of code allowing all the noes
1:15:03
to completely talk to each other what
1:15:05
we're implementing here is sometimes
1:15:06
called a decoder block and it's called a
1:15:10
decoder because it is sort of like a
1:15:13
decoding language and it's got this
1:15:15
autor regressive format where you have
1:15:17
to mask with the Triangular Matrix so
1:15:20
that uh nodes from the future never talk
1:15:22
to the Past because they would give away
1:15:24
the answer
1:15:26
and so basically in encoder blocks you
1:15:27
would delete this allow all the noes to
1:15:29
talk in decoder blocks this will always
1:15:31
be present so that you have this
1:15:33
triangular structure uh but both are
1:15:35
allowed and attention doesn't care
1:15:37
attention supports arbitrary
1:15:38
connectivity between nodes the next
1:15:40
thing I wanted to comment on is you keep
1:15:42
me you keep hearing me say attention
1:15:44
self attention Etc there's actually also
1:15:46
something called cross attention what is
1:15:47
the
1:15:48
difference
1:15:50
so basically the reason this attention
1:15:53
is self attention is because because the
1:15:56
keys queries and the values are all
1:15:58
coming from the same Source from X so
1:16:01
the same Source X produces Keys queries
1:16:03
and values so these nodes are self
1:16:06
attending but in principle attention is
1:16:09
much more General than that so for
1:16:10
example an encoder decoder Transformers
1:16:13
uh you can have a case where the queries
1:16:15
are produced from X but the keys and the
1:16:17
values come from a whole separate
1:16:19
external source and sometimes from uh
1:16:21
encoder blocks that encode some context
1:16:24
that we'd like to condition on
1:16:26
and so the keys and the values will
1:16:27
actually come from a whole separate
1:16:28
Source those are nodes on the side and
1:16:31
here we're just producing queries and
1:16:33
we're reading off information from the
1:16:34
side so cross attention is used when
1:16:37
there's a separate source of nodes we'd
1:16:41
like to pull information from into our
1:16:43
nodes and it's self attention if we just
1:16:45
have nodes that would like to look at
1:16:47
each other and talk to each other so
1:16:49
this attention here happens to be self
1:16:52
attention but in principle um attention
1:16:56
is a lot more General okay and the last
1:16:58
note at this stage is if we come to the
1:17:00
attention is all need paper here we've
1:17:02
already implemented attention so given
1:17:04
query key and value we've U multiplied
1:17:06
the query and a key we've soft maxed it
1:17:09
and then we are aggregating the values
1:17:12
there's one more thing that we're
1:17:13
missing here which is the dividing by
1:17:14
one / square root of the head size the
1:17:17
DK here is the head size why are they
1:17:19
doing this finds this important so they
1:17:21
call it the scaled attention and it's
1:17:24
kind of like an important normalization
1:17:26
to basically
1:17:27
have the problem is if you have unit gsh
1:17:30
and inputs so zero mean unit variance K
1:17:32
and Q are unit gashin then if you just
1:17:34
do we naively then you see that your we
1:17:37
actually will be uh the variance will be
1:17:39
on the order of head size which in our
1:17:41
case is 16 but if you multiply by one
1:17:44
over head size square root so this is
1:17:46
square root and this is one
1:17:47
over then the variance of we will be one
1:17:51
so it will be
1:17:52
preserved now why is this important
1:17:54
you'll not notice that way
1:17:56
here will feed into
1:17:59
softmax and so it's really important
1:18:01
especially at initialization that we be
1:18:03
fairly diffuse so in our case here we
1:18:07
sort of locked out here and we had a
1:18:10
fairly diffuse numbers here so um like
1:18:14
this now the problem is that because of
1:18:16
softmax if weight takes on very positive
1:18:18
and very negative numbers inside it
1:18:20
softmax will actually converge towards
1:18:23
one hot vectors and so I can illustrate
1:18:25
that here um say we are applying softmax
1:18:30
to a tensor of values that are very
1:18:31
close to zero then we're going to get a
1:18:33
diffuse thing out of
1:18:35
softmax but the moment I take the exact
1:18:37
same thing and I start sharpening it
1:18:39
making it bigger by multiplying these
1:18:40
numbers by eight for example you'll see
1:18:42
that the softmax will start to sharpen
1:18:44
and in fact it will sharpen towards the
1:18:46
max so it will sharpen towards whatever
1:18:48
number here is the highest and so um
1:18:51
basically we don't want these values to
1:18:53
be too extreme especially at
1:18:54
initialization otherwise softmax will be
1:18:56
way too peaky and um you're basically
1:18:59
aggregating um information from like a
1:19:01
single node every node just agregates
1:19:03
information from a single other node
1:19:05
that's not what we want especially at
1:19:06
initialization and so the scaling is
1:19:09
used just to control the variance at
1:19:11
initialization okay so having said all
1:19:13
that let's now take our self attention
1:19:15
knowledge and let's uh take it for a
1:19:17
spin so here in the code I created this
1:19:20
head module and it implements a single
1:19:22
head of self attention so you give it a
1:19:25
head size and then here it creates the
1:19:27
key query and the value linear layers
1:19:29
typically people don't use biases in
1:19:31
these uh so those are the linear
1:19:33
projections that we're going to apply to
1:19:35
all of our nodes now here I'm creating
1:19:38
this Trill variable Trill is not a
1:19:40
parameter of the module so in sort of
1:19:42
pytorch naming conventions uh this is
1:19:44
called a buffer it's not a parameter and
1:19:46
you have to call it you have to assign
1:19:47
it to the module using a register buffer
1:19:50
so that creates the trill uh the triang
1:19:52
lower triangular Matrix and we're given
1:19:55
the input X this should look very
1:19:56
familiar now we calculate the keys the
1:19:59
queries we C calculate the attention
1:20:01
scores inside way uh we normalize it so
1:20:04
we're using scaled attention here then
1:20:06
we make sure that uh future doesn't
1:20:08
communicate with the past so this makes
1:20:10
it a decoder block and then softmax and
1:20:13
then aggregate the value and
1:20:16
output then here in the language model
1:20:18
I'm creating a head in the Constructor
1:20:20
and I'm calling it self attention head
1:20:22
and the head size I'm going to keep as
1:20:25
the same and embed just for
1:20:28
now and then here once we've encoded the
1:20:31
information with the token embeddings
1:20:33
and the position embeddings we're simply
1:20:35
going to feed it into the self attention
1:20:36
head and then the output of that is
1:20:39
going to go into uh the decoder language
1:20:42
modeling head and create the logits so
1:20:45
this the sort of the simplest way to
1:20:46
plug in a self attention component uh
1:20:49
into our Network right now I had to make
1:20:52
one more change which is that here in
1:20:55
the generate uh we have to make sure
1:20:57
that our idx that we feed into the model
1:21:01
because now we're using positional
1:21:02
embeddings we can never have more than
1:21:04
block size coming in because if idx is
1:21:08
more than block size then our position
1:21:09
embedding table is going to run out of
1:21:11
scope because it only has embeddings for
1:21:13
up to block size and so therefore I
1:21:15
added some uh code here to crop the
1:21:17
context that we're going to feed into
1:21:20
self um so that uh we never pass in more
1:21:24
than block siiz elements
1:21:26
so those are the changes and let's Now
1:21:27
train the network okay so I also came up
1:21:29
to the script here and I decreased the
1:21:31
learning rate because uh the self
1:21:32
attention can't tolerate very very high
1:21:34
learning rates and then I also increased
1:21:37
number of iterations because the
1:21:38
learning rate is lower and then I
1:21:40
trained it and previously we were only
1:21:41
able to get to up to 2.5 and now we are
1:21:44
down to 2.4 so we definitely see a
1:21:46
little bit of an improvement from 2.5 to
1:21:48
2.4 roughly uh but the text is still not
1:21:51
amazing so clearly the self attention
1:21:54
head is doing some useful communication
1:21:57
but um we still have a long way to go
1:21:59
okay so now we've implemented the scale.
1:22:01
product attention now next up and the
1:22:03
attention is all you need paper there's
1:22:05
something called multi-head attention
1:22:07
and what is multi-head attention it's
1:22:09
just applying multiple attentions in
1:22:11
parallel and concatenating their results
1:22:14
so they have a little bit of diagram
1:22:16
here I don't know if this is super clear
1:22:18
it's really just multiple attentions in
1:22:21
parallel so let's Implement that fairly
1:22:24
straightforward
1:22:25
if we want a multi-head attention then
1:22:27
we want multiple heads of self attention
1:22:29
running in parallel so in pytorch we can
1:22:33
do this by simply creating multiple
1:22:35
heads so however heads how however many
1:22:38
heads you want and then what is the head
1:22:40
size of each and then we run all of them
1:22:43
in parallel into a list and simply
1:22:46
concatenate all of the outputs and we're
1:22:49
concatenating over the channel
1:22:51
Dimension so the way this looks now is
1:22:53
we don't have just a single ATT
1:22:56
that uh has a hit size of 32 because
1:22:59
remember n Ed is
1:23:01
32 instead of having one Communication
1:23:04
channel we now have four communication
1:23:06
channels in parallel and each one of
1:23:09
these communication channels typically
1:23:11
will be uh smaller uh correspondingly so
1:23:15
because we have four communication
1:23:16
channels we want eight dimensional self
1:23:18
attention and so from each Communication
1:23:20
channel we're going to together eight
1:23:22
dimensional vectors and then we have
1:23:24
four of them and that concatenates to
1:23:26
give us 32 which is the original and
1:23:28
embed and so this is kind of similar to
1:23:31
um if you're familiar with convolutions
1:23:32
this is kind of like a group convolution
1:23:34
uh because basically instead of having
1:23:36
one large convolution we do convolution
1:23:38
in groups and uh that's multi-headed
1:23:41
self
1:23:42
attention and so then here we just use
1:23:45
essay heads self attention heads instead
1:23:48
now I actually ran it and uh scrolling
1:23:51
down I ran the same thing and then we
1:23:54
now get this down to 2.28 roughly and
1:23:57
the output is still the generation is
1:23:59
still not amazing but clearly the
1:24:00
validation loss is improving because we
1:24:02
were at 2.4 just now and so it helps to
1:24:06
have multiple communication channels
1:24:07
because obviously these tokens have a
1:24:09
lot to talk about they want to find the
1:24:11
consonants the vowels they want to find
1:24:13
the vowels just from certain positions
1:24:15
uh they want to find any kinds of
1:24:17
different things and so it helps to
1:24:19
create multiple independent channels of
1:24:21
communication gather lots of different
1:24:23
types of data and then uh decode the
1:24:25
output now going back to the paper for a
1:24:27
second of course I didn't explain this
1:24:29
figure in full detail but we are
1:24:31
starting to see some components of what
1:24:32
we've already implemented we have the
1:24:34
positional encodings the token encodings
1:24:35
that add we have the masked multi-headed
1:24:38
attention implemented now here's another
1:24:41
multi-headed attention which is a cross
1:24:43
attention to an encoder which we haven't
1:24:45
we're not going to implement in this
1:24:46
case I'm going to come back to that
1:24:48
later but I want you to notice that
1:24:50
there's a feed forward part here and
1:24:52
then this is grouped into a block that
1:24:54
gets repeat it again and again now the
1:24:56
feedforward part here is just a simple
1:24:58
uh multi-layer perceptron
1:25:01
um so the multi-headed so here position
1:25:04
wise feed forward networks is just a
1:25:06
simple little MLP so I want to start
1:25:09
basically in a similar fashion also
1:25:11
adding computation into the network and
1:25:13
this computation is on a per node level
1:25:16
so I've already implemented it and you
1:25:19
can see the diff highlighted on the left
1:25:21
here when I've added or changed things
1:25:23
now before we had the self multi-headed
1:25:25
self attention that did the
1:25:26
communication but we went way too fast
1:25:29
to calculate the logits so the tokens
1:25:32
looked at each other but didn't really
1:25:33
have a lot of time to think on what they
1:25:35
found from the other tokens and so what
1:25:39
I've implemented here is a little feet
1:25:41
forward single layer and this little
1:25:43
layer is just a linear followed by a Rel
1:25:45
nonlinearity and that's that's it so
1:25:48
it's just a little layer and then I call
1:25:51
it feed
1:25:52
forward um and embed
1:25:55
and then this feed forward is just
1:25:56
called sequentially right after the self
1:25:58
attention so we self attend then we feed
1:26:01
forward and you'll notice that the feet
1:26:03
forward here when it's applying linear
1:26:05
this is on a per token level all the
1:26:07
tokens do this independently so the self
1:26:10
attention is the communication and then
1:26:12
once they've gathered all the data now
1:26:13
they need to think on that data
1:26:15
individually and so that's what feed
1:26:17
forward is doing and that's why I've
1:26:19
added it here now when I train this the
1:26:22
validation LW actually continues to go
1:26:23
down now to 2. 24 which is down from
1:26:27
2.28 uh the output still look kind of
1:26:29
terrible but at least we've improved the
1:26:31
situation and so as a preview we're
1:26:34
going to now start to intersperse the
1:26:37
communication with the computation and
1:26:40
that's also what the Transformer does
1:26:42
when it has blocks that communicate and
1:26:44
then compute and it groups them and
1:26:47
replicates them okay so let me show you
1:26:50
what we'd like to do we'd like to do
1:26:52
something like this we have a block and
1:26:54
this block is is basically this part
1:26:55
here except for the cross
1:26:58
attention now the block basically
1:26:59
intersperses communication and then
1:27:02
computation the computation the
1:27:04
communication is done using multi-headed
1:27:05
selfelf attention and then the
1:27:07
computation is done using a feed forward
1:27:09
Network on all the tokens
1:27:12
independently now what I've added here
1:27:14
also is you'll
1:27:16
notice this takes the number of
1:27:18
embeddings in the embedding Dimension
1:27:20
and number of heads that we would like
1:27:21
which is kind of like group size in
1:27:23
group convolution and and I'm saying
1:27:25
that number of heads we'd like is four
1:27:27
and so because this is 32 we calculate
1:27:29
that because this is 32 the number of
1:27:31
heads should be four um the head size
1:27:35
should be eight so that everything sort
1:27:36
of works out Channel wise um so this is
1:27:39
how the Transformer structures uh sort
1:27:41
of the uh the sizes typically so the
1:27:44
head size will become eight and then
1:27:46
this is how we want to intersperse them
1:27:48
and then here I'm trying to create
1:27:50
blocks which is just a sequential
1:27:52
application of block block block so that
1:27:54
we're interspersing communication feed
1:27:56
forward many many times and then finally
1:27:58
we decode now I actually tried to run
1:28:01
this and the problem is this doesn't
1:28:03
actually give a very good uh answer and
1:28:06
very good result and the reason for that
1:28:08
is we're start starting to actually get
1:28:09
like a pretty deep neural net and deep
1:28:11
neural Nets uh suffer from optimization
1:28:13
issues and I think that's what we're
1:28:15
kind of like slightly starting to run
1:28:16
into so we need one more idea that we
1:28:18
can borrow from the um Transformer paper
1:28:21
to resolve those difficulties now there
1:28:23
are two optimizations that dramatically
1:28:25
help with the depth of these networks
1:28:27
and make sure that the networks remain
1:28:29
optimizable let's talk about the first
1:28:31
one the first one in this diagram is you
1:28:33
see this Arrow here and then this arrow
1:28:37
and this Arrow those are skip
1:28:38
connections or sometimes called residual
1:28:40
connections they come from this paper uh
1:28:43
the presidual learning for image
1:28:45
recognition from about
1:28:47
2015 uh that introduced the concept now
1:28:51
these are basically what it means is you
1:28:53
transform data but then you have a skip
1:28:55
connection with addition from the
1:28:58
previous features now the way I like to
1:29:00
visualize it uh that I prefer is the
1:29:03
following here the computation happens
1:29:05
from the top to bottom and basically you
1:29:08
have this uh residual pathway and you
1:29:11
are free to Fork off from the residual
1:29:13
pathway perform some computation and
1:29:15
then project back to the residual
1:29:16
pathway via addition and so you go from
1:29:20
the the uh inputs to the targets only
1:29:23
via plus and plus plus and the reason
1:29:26
this is useful is because during back
1:29:27
propagation remember from our microG
1:29:30
grad video earlier addition distributes
1:29:32
gradients equally to both of its
1:29:34
branches that that fed as the input and
1:29:37
so the supervision or the gradients from
1:29:40
the loss basically hop through every
1:29:43
addition node all the way to the input
1:29:47
and then also Fork off into the residual
1:29:50
blocks but basically you have this
1:29:52
gradient Super Highway that goes
1:29:54
directly from the supervision all the
1:29:56
way to the input unimpeded and then
1:29:58
these viral blocks are usually
1:30:00
initialized in the beginning so they
1:30:01
contribute very very little if anything
1:30:03
to the residual pathway they they are
1:30:05
initialized that way so in the beginning
1:30:08
they are sort of almost kind of like not
1:30:09
there but then during the optimization
1:30:12
they come online over time and they uh
1:30:15
start to contribute but at least at the
1:30:17
initialization you can go from directly
1:30:19
supervision to the input gradient is
1:30:21
unimpeded and just flows and then the
1:30:23
blocks over time
1:30:25
kick in and so that dramatically helps
1:30:27
with the optimization so let's implement
1:30:29
this so coming back to our block here
1:30:31
basically what we want to do is we want
1:30:34
to do xal
1:30:35
X+ self attention and xal X+ self. feed
1:30:40
forward so this is X and then we Fork
1:30:43
off and do some communication and come
1:30:45
back and we Fork off and we do some
1:30:47
computation and come back so those are
1:30:49
residual connections and then swinging
1:30:52
back up here we also have to introd use
1:30:54
this projection so nn.
1:30:58
linear and uh this is going to be
1:31:01
from after we concatenate this this is
1:31:03
the prze and embed so this is the output
1:31:06
of the self tension itself but then we
1:31:08
actually want the uh to apply the
1:31:11
projection and that's the
1:31:13
result so the projection is just a
1:31:15
linear transformation of the outcome of
1:31:17
this
1:31:18
layer so that's the projection back into
1:31:20
the virual pathway and then here in a
1:31:22
feet forward it's going to be the same
1:31:24
same thing I could have a a self doot
1:31:26
projection here as well but let me just
1:31:28
simplify it and let me uh couple it
1:31:33
inside the same sequential container and
1:31:35
so this is the projection layer going
1:31:37
back into the residual
1:31:38
pathway and
1:31:40
so that's uh well that's it so now we
1:31:43
can train this so I implemented one more
1:31:45
small change when you look into the
1:31:47
paper again you see that the
1:31:50
dimensionality of input and output is
1:31:51
512 for them and they're saying that the
1:31:53
inner layer here in the feet forward has
1:31:55
dimensionality of 248 so there's a
1:31:58
multiplier of four and so the inner
1:32:00
layer of the feet forward Network should
1:32:03
be multiplied by four in terms of
1:32:04
Channel sizes so I came here and I
1:32:06
multiplied four times embed here for the
1:32:09
feed forward and then from four times
1:32:11
nmed coming back down to nmed when we go
1:32:13
back to the pro uh to the projection so
1:32:16
adding a bit of computation here and
1:32:17
growing that layer that is in the
1:32:19
residual block on the side of the
1:32:21
residual
1:32:22
pathway and then I train this and we
1:32:24
actually get down all the way to uh 2.08
1:32:27
validation loss and we also see that
1:32:29
network is starting to get big enough
1:32:31
that our train loss is getting ahead of
1:32:32
validation loss so we're starting to see
1:32:34
like a little bit of
1:32:35
overfitting and um our our
1:32:39
um uh Generations here are still not
1:32:41
amazing but at least you see that we can
1:32:43
see like is here this now grief syn like
1:32:47
this starts to almost look like English
1:32:49
so um yeah we're starting to really get
1:32:51
there okay and the second Innovation
1:32:53
that is very helpful for optimizing very
1:32:55
deep neural networks is right here so we
1:32:57
have this addition now that's the
1:32:58
residual part but this Norm is referring
1:33:00
to something called layer Norm so layer
1:33:03
Norm is implemented in pytorch it's a
1:33:05
paper that came out a while back here
1:33:09
um and layer Norm is very very similar
1:33:12
to bash Norm so remember back to our
1:33:15
make more series part three we
1:33:16
implemented bash
1:33:18
normalization and uh bash normalization
1:33:20
basically just made sure that um Across
1:33:23
The Bash dimension any individual neuron
1:33:26
had unit uh Gan um distribution so it
1:33:30
was zero mean and unit standard
1:33:32
deviation one standard deviation output
1:33:36
so what I did here is I'm copy pasting
1:33:38
the bashor 1D that we developed in our
1:33:39
make more series and see here we can
1:33:42
initialize for example this module and
1:33:45
we can have a batch of 32 100
1:33:47
dimensional vectors feeding through the
1:33:49
bachor layer so what this does is it
1:33:52
guarantees that when we look at just the
1:33:55
zeroth column it's a zero mean one
1:33:58
standard deviation so it's normalizing
1:34:01
every single column of this uh input now
1:34:04
the rows are not uh going to be
1:34:06
normalized by default because we're just
1:34:08
normalizing columns so let's now
1:34:10
Implement layer Norm uh it's very
1:34:12
complicated look we come here we change
1:34:16
this from zero to one so we don't
1:34:18
normalize The Columns we normalize the
1:34:21
rows and now we've implemented layer
1:34:23
Norm
1:34:25
so now the columns are not going to be
1:34:28
normalized um but the rows are going to
1:34:31
be normalized for every individual
1:34:33
example it's 100 dimensional Vector is
1:34:35
normalized uh in this way and because
1:34:38
our computation Now does not span across
1:34:40
examples we can delete all of this
1:34:43
buffers stuff uh because uh we can
1:34:46
always apply this operation and don't
1:34:49
need to maintain any running buffers so
1:34:51
we don't need the
1:34:52
buffers uh we
1:34:54
don't There's no distinction between
1:34:56
training and test
1:34:58
time uh and we don't need these running
1:35:01
buffers we do keep gamma and beta we
1:35:04
don't need the momentum we don't care if
1:35:06
it's training or not and this is now a
1:35:09
layer
1:35:10
norm and it normalizes the rows instead
1:35:13
of the columns and this here is
1:35:16
identical to basically this here so
1:35:20
let's now Implement layer Norm in our
1:35:22
Transformer before I incorporate the
1:35:23
layer Norm I just wanted to note that as
1:35:26
I said very few details about the
1:35:27
Transformer have changed in the last 5
1:35:29
years but this is actually something
1:35:30
that slightly departs from the original
1:35:32
paper you see that the ADD and Norm is
1:35:34
applied after the
1:35:37
transformation but um in now it is a bit
1:35:40
more uh basically common to apply the
1:35:42
layer Norm before the transformation so
1:35:45
there's a reshuffling of the layer Norms
1:35:47
uh so this is called the prorm
1:35:48
formulation and that's the one that
1:35:49
we're going to implement as well so
1:35:51
select deviation from the original paper
1:35:53
basically we need two layer Norms layer
1:35:55
Norm one is uh NN do layer norm and we
1:35:59
tell it how many um what is the
1:36:01
embedding Dimension and we need the
1:36:04
second layer norm and then here the
1:36:07
layer Norms are applied immediately on X
1:36:09
so self. layer Norm one applied on X and
1:36:13
self. layer Norm two applied on X before
1:36:16
it goes into self attention and feed
1:36:18
forward and uh the size of the layer
1:36:21
Norm here is an ed so 32 so when the
1:36:24
layer Norm is normalizing our features
1:36:26
it is uh the normalization here uh
1:36:30
happens the mean and the variance are
1:36:32
taken over 32 numbers so the batch and
1:36:35
the time act as batch Dimensions both of
1:36:38
them so this is kind of like a per token
1:36:41
um transformation that just normalizes
1:36:43
the features and makes them a unit mean
1:36:46
uh unit Gan at
1:36:48
initialization but of course because
1:36:50
these layer Norms inside it have these
1:36:52
gamma and beta training
1:36:54
parameters uh the layer Norm will U
1:36:57
eventually create outputs that might not
1:36:59
be unit gion but the optimization will
1:37:02
determine that so for now this is the uh
1:37:05
this is incorporating the layer norms
1:37:07
and let's train them on okay so I let it
1:37:09
run and we see that we get down to 2.06
1:37:12
which is better than the previous 2.08
1:37:14
so a slight Improvement by adding the
1:37:16
layer norms and I'd expect that they
1:37:18
help uh even more if we had bigger and
1:37:20
deeper Network one more thing I forgot
1:37:22
to add is that there should be a layer
1:37:24
Norm here also typically as at the end
1:37:27
of the Transformer and right before the
1:37:29
final uh linear layer that decodes into
1:37:32
vocabulary so I added that as well so at
1:37:35
this stage we actually have a pretty
1:37:36
complete uh Transformer according to the
1:37:38
original paper and it's a decoder only
1:37:40
Transformer I'll I'll talk about that in
1:37:42
a second uh but at this stage uh the
1:37:45
major pieces are in place so we can try
1:37:46
to scale this up and see how well we can
1:37:48
push this number now in order to scale
1:37:50
out the model I had to perform some
1:37:52
cosmetic changes here to make it nicer
1:37:55
so I introduced this variable called n
1:37:56
layer which just specifies how many
1:37:58
layers of the blocks we're going to have
1:38:01
I created a bunch of blocks and we have
1:38:03
a new variable number of heads as well I
1:38:06
pulled out the layer Norm here and uh so
1:38:08
this is identical now one thing that I
1:38:10
did briefly change is I added a Dropout
1:38:13
so Dropout is something that you can add
1:38:16
right before the residual connection
1:38:18
back right before the connection back
1:38:20
into the residual pathway so we can drop
1:38:22
out that as l layer here we can drop out
1:38:26
uh here at the end of the multi-headed
1:38:28
exension as well and we can also drop
1:38:30
out here uh when we calculate the um
1:38:34
basically affinities and after the
1:38:36
softmax we can drop out some of those so
1:38:38
we can randomly prevent some of the
1:38:40
nodes from
1:38:41
communicating and so Dropout uh comes
1:38:44
from this paper from 2014 or so and
1:38:49
basically it takes your neural
1:38:51
nut and it randomly every forward
1:38:53
backward pass shuts off some subset of
1:38:56
uh neurons so randomly drops them to
1:39:00
zero and trains without them and what
1:39:03
this does effectively is because the
1:39:05
mask of what's being dropped out is
1:39:07
changed every single forward backward
1:39:08
pass it ends up kind of uh training an
1:39:11
ensemble of sub networks and then at
1:39:14
test time everything is fully enabled
1:39:15
and kind of all of those sub networks
1:39:17
are merged into a single Ensemble if you
1:39:19
can if you want to think about it that
1:39:20
way so I would read the paper to get the
1:39:22
full detail for now we're just going to
1:39:24
stay on the level of this is a
1:39:26
regularization technique and I added it
1:39:29
because I'm about to scale up the model
1:39:30
quite a bit and I was concerned about
1:39:32
overfitting so now when we scroll up to
1:39:35
the top uh we'll see that I changed a
1:39:37
number of hyper parameters here about
1:39:39
our neural nut so I made the batch size
1:39:41
be much larger now it's 64 I changed the
1:39:44
block size to be 256 so previously it
1:39:46
was just eight eight characters of
1:39:48
context now it is 256 characters of
1:39:50
context to predict the 257th
1:39:54
uh I brought down the learning rate a
1:39:56
little bit because the neural net is now
1:39:57
much bigger so I brought down the
1:39:59
learning rate the embedding Dimension is
1:40:01
now 384 and there are six heads so 384
1:40:05
divide 6 means that every head is 64
1:40:08
dimensional as it as a standard and then
1:40:12
there's going to be six layers of that
1:40:14
and the Dropout will be at 02 so every
1:40:16
forward backward pass 20% of all of
1:40:18
these um intermediate calculations are
1:40:22
disabled and dropped to zero
1:40:24
and then I already trained this and I
1:40:26
ran it so uh drum roll how well does it
1:40:29
perform so let me just scroll up
1:40:32
here we get a validation loss of
1:40:35
1.48 which is actually quite a bit of an
1:40:37
improvement on what we had before which
1:40:39
I think was 2.07 so it went from 2.07
1:40:42
all the way down to 1.48 just by scaling
1:40:44
up this neural nut with the code that we
1:40:45
have and this of course ran for a lot
1:40:48
longer this maybe trained for I want to
1:40:50
say about 15 minutes on my a100 GPU so
1:40:53
that's a pretty a GPU and if you don't
1:40:55
have a GPU you're not going to be able
1:40:56
to reproduce this uh on a CPU this would
1:40:59
be um I would not run this on a CPU or
1:41:01
MacBook or something like that you'll
1:41:03
have to Brak down the number of uh
1:41:05
layers and the embedding Dimension and
1:41:07
so on uh but in about 15 minutes we can
1:41:10
get this kind of a result and um I'm
1:41:13
printing some of the Shakespeare here
1:41:15
but what I did also is I printed 10,000
1:41:17
characters so a lot more and I wrote
1:41:19
them to a file and so here we see some
1:41:21
of the outputs
1:41:24
so it's a lot more recognizable as the
1:41:27
input text file so the input text file
1:41:29
just for reference looked like this so
1:41:32
there's always like someone speaking in
1:41:34
this manner and uh our predictions now
1:41:38
take on that form except of course
1:41:40
they're they're nonsensical when you
1:41:42
actually read them
1:41:44
so it is every crimp tap be a house oh
1:41:48
those
1:41:49
prepation we give
1:41:51
heed um you know
1:41:56
Oho sent me you mighty
1:42:00
Lord anyway so you can read through this
1:42:02
um it's nonsensical of course but this
1:42:05
is just a Transformer trained on a
1:42:07
character level for 1 million characters
1:42:09
that come from Shakespeare so there's
1:42:11
sort of like blabbers on in Shakespeare
1:42:13
like manner but it doesn't of course
1:42:15
make sense at this scale uh but I think
1:42:18
I think still a pretty good
1:42:19
demonstration of what's
1:42:21
possible so now
1:42:24
I think uh that kind of like concludes
1:42:26
the programming section of this video we
1:42:29
basically kind of uh did a pretty good
1:42:30
job and um of implementing this
1:42:33
Transformer uh but the picture doesn't
1:42:35
exactly match up to what we've done so
1:42:37
what's going on with all these digital
1:42:39
Parts here so let me finish explaining
1:42:41
this architecture and why it looks so
1:42:43
funky basically what's happening here is
1:42:46
what we implemented here is a decoder
1:42:48
only Transformer so there's no component
1:42:51
here this part is called the encoder and
1:42:53
there's no cross attention block here
1:42:56
our block only has a self attention and
1:42:58
the feet forward so it is missing this
1:43:00
third in between piece here this piece
1:43:03
does cross attention so we don't have it
1:43:05
and we don't have the encoder we just
1:43:07
have the decoder and the reason we have
1:43:09
a decoder only uh is because we are just
1:43:12
uh generating text and it's
1:43:13
unconditioned on anything we're just
1:43:15
we're just blabbering on according to a
1:43:17
given data set what makes it a decoder
1:43:20
is that we are using the Triangular mask
1:43:22
in our uh trans former so it has this
1:43:24
Auto regressive property where we can
1:43:26
just uh go and sample from it so the
1:43:29
fact that it's using the Triangular
1:43:30
triangular mask to mask out the
1:43:32
attention makes it a decoder and it can
1:43:35
be used for language modeling now the
1:43:37
reason that the original paper had an
1:43:39
incoder decoder architecture is because
1:43:41
it is a machine translation paper so it
1:43:43
is concerned with a different setting in
1:43:46
particular it expects some uh tokens
1:43:50
that encode say for example French and
1:43:52
then it is expecting to decode the
1:43:54
translation in English so so you
1:43:57
typically these here are special tokens
1:43:59
so you are expected to read in this and
1:44:02
condition on it and then you start off
1:44:04
the generation with a special token
1:44:06
called start so this is a special new
1:44:08
token um that you introduce and always
1:44:11
place in the beginning and then the
1:44:13
network is expected to Output neural
1:44:16
networks are awesome and then a special
1:44:18
end token to finish the
1:44:20
generation so this part here will be
1:44:23
decoded exactly as we we've done it
1:44:26
neural networks are awesome will be
1:44:27
identical to what we did but unlike what
1:44:30
we did they wanton to condition the
1:44:33
generation on some additional
1:44:35
information and in that case this
1:44:37
additional information is the French
1:44:38
sentence that they should be
1:44:40
translating so what they do now is they
1:44:43
bring in the encoder now the encoder
1:44:46
reads this part here so we're only going
1:44:49
to take the part of French and we're
1:44:51
going to uh create tokens from it
1:44:53
exactly as we've seen in our video and
1:44:55
we're going to put a Transformer on it
1:44:57
but there's going to be no triangular
1:44:59
mask and so all the tokens are allowed
1:45:01
to talk to each other as much as they
1:45:02
want and they're just encoding
1:45:04
whatever's the content of this French uh
1:45:07
sentence once they've encoded it they
1:45:11
they basically come out in the top here
1:45:13
and then what happens here is in our
1:45:15
decoder which does the uh language
1:45:18
modeling there's an additional
1:45:20
connection here to the outputs of the
1:45:22
encoder
1:45:24
and that is brought in through a cross
1:45:26
attention so the queries are still
1:45:28
generated from X but now the keys and
1:45:30
the values are coming from the side the
1:45:33
keys and the values are coming from the
1:45:34
top generated by the nodes that came
1:45:37
outside of the de the encoder and those
1:45:40
tops the keys and the values there the
1:45:42
top of it feed in on a side into every
1:45:45
single block of the decoder and so
1:45:48
that's why there's an additional cross
1:45:49
attention and really what it's doing is
1:45:51
it's conditioning the decoding
1:45:54
not just on the past of this current
1:45:56
decoding but also on having seen the
1:46:00
full fully encoded French um prompt sort
1:46:04
of and so it's an encoder decoder model
1:46:07
which is why we have those two
1:46:07
Transformers an additional block and so
1:46:10
on so we did not do this because we have
1:46:12
no we have nothing to encode there's no
1:46:14
conditioning we just have a text file
1:46:15
and we just want to imitate it and
1:46:17
that's why we are using a decoder only
1:46:19
Transformer exactly as done in
1:46:22
GPT okay okay so now I wanted to do a
1:46:24
very brief walkthrough of nanog GPT
1:46:26
which you can find in my GitHub and uh
1:46:29
nanog GPT is basically two files of
1:46:30
Interest there's train.py and model.py
1:46:34
train.py is all the boilerplate code for
1:46:36
training the network it is basically all
1:46:38
the stuff that we had here it's the
1:46:40
training loop it's just that it's a lot
1:46:43
more complicated because we're saving
1:46:44
and loading checkpoints and pre-trained
1:46:46
weights and we are uh decaying the
1:46:48
learning rate and compiling the model
1:46:50
and using distributed training across
1:46:52
multiple nodes or GP use so the training
1:46:55
Pi gets a little bit more hairy
1:46:56
complicated uh there's more options Etc
1:47:00
but the model.py should look very very
1:47:02
um similar to what we've done here in
1:47:04
fact the model is is almost identical so
1:47:08
first here we have the causal self
1:47:10
attention block and all of this should
1:47:12
look very very recognizable to you we're
1:47:14
producing queries Keys values we're
1:47:16
doing Dot products we're masking
1:47:18
applying soft Maxs optionally dropping
1:47:21
out and here we are pulling the wi the
1:47:23
values what is different here is that in
1:47:26
our code I have separated out the
1:47:30
multi-headed detention into just a
1:47:32
single individual head and then here I
1:47:35
have multiple heads and I explicitly
1:47:36
concatenate them whereas here uh all of
1:47:39
it is implemented in a batched manner
1:47:41
inside a single causal self attention
1:47:44
and so we don't just have a b and a T
1:47:45
and A C Dimension we also end up with a
1:47:48
fourth dimension which is the heads and
1:47:50
so it just gets a lot more sort of hairy
1:47:52
because we have four dimensional array
1:47:54
um tensors now but it is um equivalent
1:47:57
mathematically so the exact same thing
1:47:59
is happening as what we have it's just
1:48:01
it's a bit more efficient because all
1:48:03
the heads are now treated as a batch
1:48:05
Dimension as
1:48:06
well then we have the multier perceptron
1:48:08
it's using the Galu nonlinearity which
1:48:11
is defined here except instead of Ru and
1:48:13
this is done just because opening I used
1:48:15
it and I want to be able to load their
1:48:17
checkpoints uh the blocks of the
1:48:19
Transformer are identical to communicate
1:48:21
in the compute phase as we saw and then
1:48:23
the GPT will be identical we have the
1:48:25
position encodings token encodings the
1:48:28
blocks the layer Norm at the end uh the
1:48:30
final linear layer and this should look
1:48:33
all very recognizable and there's a bit
1:48:36
more here because I'm loading
1:48:37
checkpoints and stuff like that I'm
1:48:39
separating out the parameters into those
1:48:41
that should be weight decayed and those
1:48:42
that
1:48:43
shouldn't um but the generate function
1:48:45
should also be very very similar so a
1:48:47
few details are different but you should
1:48:49
definitely be able to look at this uh
1:48:51
file and be able to understand little
1:48:53
the pieces now so let's now bring things
1:48:55
back to chat GPT what would it look like
1:48:58
if we wanted to train chat GPT ourselves
1:49:00
and how does it relate to what we
1:49:01
learned today well to train in chat GPT
1:49:04
there are roughly two stages first is
1:49:06
the pre-training stage and then the
1:49:07
fine-tuning stage in the pre-training
1:49:10
stage uh we are training on a large
1:49:12
chunk of internet and just trying to get
1:49:15
a first decoder only Transformer to
1:49:17
babble text so it's very very similar to
1:49:20
what we've done ourselves except we've
1:49:23
done like a tiny little baby
1:49:25
pre-training step um and so in our case
1:49:29
uh this is how you print a number of
1:49:31
parameters I printed it and it's about
1:49:33
10 million so this Transformer that I
1:49:35
created here to create little
1:49:37
Shakespeare um Transformer was about 10
1:49:40
million parameters our data set is
1:49:43
roughly 1 million uh characters so
1:49:45
roughly 1 million tokens but you have to
1:49:47
remember that opening I is different
1:49:49
vocabulary they're not on the Character
1:49:50
level they use these um subword chunks
1:49:54
of words and so they have a vocabulary
1:49:56
of 50,000 roughly elements and so their
1:49:58
sequences are a bit more condensed so
1:50:02
our data set the Shakespeare data set
1:50:03
would be probably around 300,000 uh
1:50:06
tokens in the open AI vocabulary roughly
1:50:09
so we trained about 10 million parameter
1:50:12
model on roughly 300,000 tokens now when
1:50:15
you go to the gpt3
1:50:17
paper and you look at the Transformers
1:50:20
that they trained they trained a number
1:50:22
of trans Transformers of different sizes
1:50:24
but the biggest Transformer here has 175
1:50:27
billion parameters uh so ours is again
1:50:29
10 million they used this number of
1:50:32
layers in the Transformer this is the
1:50:34
nmed this is the number of heads and
1:50:37
this is the head size and then this is
1:50:40
the batch size uh so ours was
1:50:43
65 and the learning rate is similar now
1:50:46
when they train this Transformer they
1:50:48
trained on 300 billion tokens so again
1:50:51
remember ours is about 300,000
1:50:53
so this is uh about a millionfold
1:50:56
increase and this number would not be
1:50:58
even that large by today's standards
1:50:59
you'd be going up uh 1 trillion and
1:51:02
above so they are training a
1:51:05
significantly larger
1:51:06
model on uh a good chunk of the internet
1:51:10
and that is the pre-training stage but
1:51:12
otherwise these hyper parameters should
1:51:14
be fairly recognizable to you and the
1:51:15
architecture is actually like nearly
1:51:17
identical to what we implemented
1:51:18
ourselves but of course it's a massive
1:51:21
infrastructure challenge to train this
1:51:23
you're talking about typically thousands
1:51:24
of gpus having to you know talk to each
1:51:27
other to train models of this size so
1:51:30
that's just a pre-training stage now
1:51:32
after you complete the pre-training
1:51:33
stage uh you don't get something that
1:51:36
responds to your questions with answers
1:51:38
and is not helpful and Etc you get a
1:51:41
document
1:51:42
completer right so it babbles but it
1:51:45
doesn't Babble Shakespeare it babbles
1:51:46
internet it will create arbitrary news
1:51:48
articles and documents and it will try
1:51:50
to complete documents because that's
1:51:52
what it's trained for it's trying to
1:51:53
complete the sequence so when you give
1:51:55
it a question it would just uh
1:51:56
potentially just give you more questions
1:51:59
it would follow with more questions it
1:52:00
will do whatever it looks like the some
1:52:03
close document would do in the training
1:52:05
data on the internet and so who knows
1:52:07
you're getting kind of like undefined
1:52:08
Behavior it might basically answer with
1:52:11
to questions with other questions it
1:52:13
might ignore your question it might just
1:52:15
try to complete some news article it's
1:52:17
totally unineed as we say so the second
1:52:21
fine-tuning stage is to actually align
1:52:22
it to be an assistant and uh this is the
1:52:25
second stage and so this chat GPT block
1:52:29
post from openi talks a little bit about
1:52:30
how the stage is achieved we basically
1:52:34
um there's roughly three steps to to
1:52:37
this stage uh so what they do here is
1:52:39
they start to collect training data that
1:52:41
looks specifically like what an
1:52:43
assistant would do so these are
1:52:45
documents that have to format where the
1:52:46
question is on top and then an answer is
1:52:48
below and they have a large number of
1:52:50
these but probably not on the order of
1:52:52
the internet uh this is probably on the
1:52:54
of maybe thousands of examples and so
1:52:58
they they then fine-tune the model to
1:53:01
basically only focus on documents that
1:53:03
look like that and so you're starting to
1:53:05
slowly align it so it's going to expect
1:53:07
a question at the top and it's going to
1:53:09
expect to complete the answer and uh
1:53:12
these very very large models are very
1:53:14
sample efficient during their
1:53:15
fine-tuning so this actually somehow
1:53:17
works but that's just step one that's
1:53:19
just fine tuning so then they actually
1:53:21
have more steps where okay the second
1:53:23
step is you let the model respond and
1:53:25
then different Raiders look at the
1:53:27
different responses and rank them for
1:53:29
their preference as to which one is
1:53:31
better than the other they use that to
1:53:33
train a reward model so they can predict
1:53:35
uh basically using a different network
1:53:37
how much of any candidate
1:53:40
response would be desirable and then
1:53:43
once they have a reward model they run
1:53:45
po which is a form of polic policy
1:53:48
gradient um reinforcement learning
1:53:50
Optimizer to uh fine-tune this sampling
1:53:53
policy uh so that the answers that the
1:53:56
GP chat GPT now generates are expected
1:53:59
to score a high reward according to the
1:54:02
reward model and so basically there's a
1:54:05
whole aligning stage here or fine-tuning
1:54:07
stage it's got multiple steps in between
1:54:09
there as well and it takes the model
1:54:12
from being a document completer to a
1:54:14
question answerer and that's like a
1:54:17
whole separate stage a lot of this data
1:54:19
is not available publicly it is internal
1:54:21
to open AI and uh it's much harder to
1:54:24
replicate this stage um and so that's
1:54:27
roughly what would give you a chat GPT
1:54:30
and nanog GPT focuses on the
1:54:31
pre-training stage okay and that's
1:54:33
everything that I wanted to cover today
1:54:35
so we trained to summarize a decoder
1:54:39
only Transformer following this famous
1:54:42
paper attention is all you need from
1:54:44
2017 and so that's basically a GPT we
1:54:47
trained it on Tiny Shakespeare and got
1:54:50
sensible results
1:54:53
all of the training code is
1:54:55
roughly 200 lines of code I will be
1:54:58
releasing this um code base so also it
1:55:02
comes with all the git log commits along
1:55:04
the way as we built it
1:55:06
up in addition to this code I'm going to
1:55:08
release the um notebook of course the
1:55:11
Google collab and I hope that gave you a
1:55:13
sense for how you can train um these
1:55:16
models like say gpt3 that will be um
1:55:19
architecturally basically identical to
1:55:21
what we have but they are somewhere
1:55:22
between 10,000 and 1 million times
1:55:24
bigger depending on how you count and so
1:55:28
uh that's all I have for now uh we did
1:55:31
not talk about any of the fine-tuning
1:55:32
stages that would typically go on top of
1:55:34
this so if you're interested in
1:55:35
something that's not just language
1:55:37
modeling but you actually want to you
1:55:38
know say perform tasks um or you want
1:55:41
them to be aligned in a specific way or
1:55:43
you want um to detect sentiment or
1:55:46
anything like that basically anytime you
1:55:47
don't want something that's just a
1:55:49
document completer you have to complete
1:55:51
further stages of fine tuning which did
1:55:53
not cover uh and that could be simple
1:55:55
supervised fine tuning or it can be
1:55:57
something more fancy like we see in chat
1:55:58
jpt where we actually train a reward
1:56:00
model and then do rounds of Po to uh
1:56:03
align it with respect to the reward
1:56:04
model so there's a lot more that can be
1:56:06
done on top of it I think for now we're
1:56:08
starting to get to about two hours Mark
1:56:10
uh so I'm going to um kind of finish
1:56:13
here uh I hope you enjoyed the lecture
1:56:16
uh and uh yeah go forth and transform
1:56:18
see you later
Start a conversation
Ask anything about Let's build GPT: from scratch, in code, spelled out.
100/100 free messages remaining
AI responses may be inaccurate. Verify important information.
Let's build GPT: from scratch, in code, spelled out.
Andrej Karpathy