This video provides an accessible introduction to large language models (LLMs) like ChatGPT. It explains how these models are built, starting with collecting and processing vast amounts of internet text data, converting it into tokens, and then training neural networks to predict the next token in a sequence. The video also covers the computational resources required for training, using GPUs and data centers, and demonstrates the training process by showing the improvement of a GPT-2 model over time.
Key Takeaways
1
The first step in building an LLM like ChatGPT is to download and process a massive amount of text data from the internet, often referred to as a dataset like FineWeb or Common Crawl.
2
Processing internet data involves several filtering stages, including removing unwanted URLs, extracting pure text from HTML, and filtering by language to ensure high-quality and diverse content.
3
Text data is converted into a one-dimensional sequence of unique symbols called tokens, using a process called tokenization, where common sequences of characters are grouped into new symbols to shorten the sequence length.
4
Neural networks are trained by feeding them sequences of tokens (context) and having them predict the next token, adjusting their internal parameters to increase the probability of correct predictions.
5
Inference is the process of generating new data from a trained LLM by feeding it a starting sequence and then sampling subsequent tokens based on the model's predicted probabilities, which results in unique, yet statistically similar, text.
6
GPT-2, a foundational LLM, had 1.6 billion parameters, a maximum context length of 1,024 tokens, and was trained on approximately 100 billion tokens, which are significantly smaller scales than modern LLMs like GPT-4.
7
The cost of training LLMs has decreased significantly due to improvements in data quality, faster hardware (GPUs), and optimized software for model execution.
8
During training, researchers monitor a 'loss' number, which indicates the model's performance; a decreasing loss signifies that the model is improving its ability to predict the next token.
9
Training large language models requires significant computational power, typically relying on powerful GPUs housed in cloud data centers, which can be rented for their immense processing capabilities.
10
The high demand for GPUs by tech companies to train LLMs has significantly driven up the market value of companies like Nvidia.
No matching segments found
0:01
hi everyone so I've wanted to make this
0:03
video for a while it is a comprehensive
0:05
but General audience introduction to
0:08
large language models like Chachi PT and
0:11
what I'm hoping to achieve in this video
0:13
is to give you kind of mental models for
0:15
thinking through what it is that this
0:17
tool is it is obviously magical and
0:20
amazing in some respects it's uh really
0:22
good at some things not very good at
0:24
other things and there's also a lot of
0:25
sharp edges to be aware of so what is
0:28
behind this text box you can put
0:30
anything in there and press enter but uh
0:32
what should we be putting there and what
0:35
are these words generated back how does
0:37
this work and what what are you talking
0:38
to exactly so I'm hoping to get at all
0:40
those topics in this video we're going
0:42
to go through the entire pipeline of how
0:44
this stuff is built but I'm going to
0:46
keep everything uh sort of accessible to
0:48
a general audience so let's take a look
0:51
at first how you build something like
0:52
chpt and along the way I'm going to talk
0:54
about um you know some of the sort of
0:57
cognitive psychological implications of
1:00
the tools okay so let's build Chachi PT
1:03
so there's going to be multiple stages
1:04
arranged sequentially the first stage is
1:07
called the pre-training stage and the
1:10
first step of the pre-training stage is
1:11
to download and process the internet now
1:13
to get a sense of what this roughly
1:15
looks like I recommend looking at this
1:17
URL here so um this company called
1:20
hugging face uh collected and created
1:23
and curated this data set called Fine
1:26
web and they go into a lot of detail on
1:29
this block post on how how they
1:30
constructed the fine web data set and
1:32
all of the major llm providers like open
1:34
AI anthropic and Google and so on will
1:36
have some equivalent internally of
1:39
something like the fine web data set so
1:41
roughly what are we trying to achieve
1:42
here we're trying to get ton of text
1:44
from the internet from publicly
1:46
available sources so we're trying to
1:48
have a huge quantity of very high
1:51
quality documents and we also want very
1:53
large diversity of documents because we
1:55
want to have a lot of knowledge inside
1:57
these models so we want large diversity
1:59
of high quality documents and we want
2:02
many many of them and achieving this is
2:04
uh quite complicated and as you can see
2:06
here takes multiple stages to do well so
2:09
let's take a look at what some of these
2:10
stages look like in a bit for now I'd
2:12
like to just like to note that for
2:13
example the fine web data set which is
2:15
fairly representative what you would see
2:17
in a production grade application
2:19
actually ends up being only about 44
2:20
terabyt of dis space um you can get a
2:24
USB stick for like a terabyte very
2:25
easily or I think this could fit on a
2:27
single hard drive almost today so this
2:29
is not a huge amount of data at the end
2:32
of the day even though the internet is
2:33
very very large we're working with text
2:35
and we're also filtering it aggressively
2:37
so we end up with about 44 terabytes in
2:39
this example so let's take a look at uh
2:42
kind of what this data looks like and
2:45
what some of these stages uh also are so
2:47
the starting point for a lot of these
2:48
efforts and something that contributes
2:50
most of the data by the end of it is
2:53
Data from common crawl so common craw is
2:56
an organization that has been basically
2:57
scouring the internet since 2007 so as
3:01
of 2024 for example common CW has
3:03
indexed 2.7 billion web
3:06
pages uh and uh they have all these
3:08
crawlers going around the internet and
3:10
what you end up doing basically is you
3:11
start with a few seed web pages and then
3:13
you follow all the links and you just
3:15
keep following links and you keep
3:16
indexing all the information and you end
3:17
up with a ton of data of the internet
3:19
over time so this is usually the
3:21
starting point for a lot of the uh for a
3:24
lot of these efforts now this common C
3:26
data is quite raw and is filtered in
3:28
many many different ways
3:30
so here they Pro they document this is
3:33
the same diagram they document a little
3:35
bit the kind of processing that happens
3:37
in these stages so the first thing here
3:40
is something called URL
3:41
filtering so what that is referring to
3:44
is that there's these block
3:47
lists of uh basically URLs that are or
3:50
domains that uh you don't want to be
3:53
getting data from so usually this
3:54
includes things like U malware websites
3:57
spam websites marketing websites uh
3:59
racist websites adult sites and things
4:01
like that so there's a ton of different
4:03
types of websites that are just
4:04
eliminated at this stage because we
4:06
don't want them in our data set um the
4:09
second part is text extraction you have
4:11
to remember that all these web pages
4:12
this is the raw HTML of these web pages
4:14
that are being saved by these crawlers
4:17
so when I go to inspect
4:19
here this is what the raw HTML actually
4:21
looks like you'll notice that it's got
4:23
all this markup uh like lists and stuff
4:27
like that and there's CSS and all this
4:29
kind of stuff so this is um computer
4:31
code almost for these web pages but what
4:34
we really want is we just want this text
4:36
right we just want the text of this web
4:37
page and we don't want the navigation
4:39
and things like that so there's a lot of
4:40
filtering and processing uh and heris
4:43
that go into uh adequately filtering for
4:45
just their uh good content of these web
4:48
pages the next stage here is language
4:51
filtering so for example fine web
4:54
filters uh using a language classifier
4:56
they try to guess what language every
4:59
single web page is in and then they only
5:00
keep web pages that have more than 65%
5:03
of English as an
5:04
example and so you can get a sense that
5:06
this is like a design decision that
5:07
different companies can uh can uh take
5:10
for themselves what fraction of all
5:13
different types of languages are we
5:14
going to include in our data set because
5:16
for example if we filter out all of the
5:18
Spanish as an example then you might
5:19
imagine that our model later will not be
5:21
very good at Spanish because it's just
5:23
never seen that much data of that
5:24
language and so different companies can
5:26
focus on multilingual performance to uh
5:29
to a different degree as an example so
5:31
fine web is quite focused on English and
5:33
so their language model if they end up
5:35
training one later will be very good at
5:37
English but not may be very good at
5:38
other
5:40
languages after language filtering
5:42
there's a few other filtering steps and
5:43
D duplication and things like that um
5:47
finishing with for example the pii
5:49
removal this is personally identifiable
5:52
information so as an example addresses
5:54
Social Security numbers and things like
5:56
that you would try to detect them and
5:58
you would try to filter out those kinds
5:59
of web pages from the the data set as
6:00
well so there's a lot of stages here and
6:03
I won't go into full detail but it is a
6:05
fairly extensive part of the
6:07
pre-processing and you end up with for
6:09
example the fine web data set so when
6:11
you click in on it uh you can see some
6:13
examples here of what this actually ends
6:14
up looking like and anyone can download
6:16
this on the huging phase web page and so
6:19
here are some examples of the final text
6:21
that ends up in the training set so this
6:24
is some article about tornadoes in
6:28
2012 um so there's some t tadoes in 2020
6:31
in 2012 and what
6:33
happened uh this next one is something
6:36
about did you know you have two little
6:39
yellow 9vt battery sized adrenal glands
6:41
in your body okay so this is some kind
6:44
of a odd medical
6:47
article so just think of these as
6:49
basically uh web pages on the internet
6:52
filtered just for the text in various
6:54
ways and now we have a ton of text 40
6:57
terabytes off it and that now is the
6:59
starting point for the next step of this
7:01
stage now I wanted to give you an
7:03
intuitive sense of where we are right
7:04
now so I took the first 200 web pages
7:07
here and remember we have tons of them
7:09
and I just take all that text and I just
7:11
put it all together concatenate it and
7:14
so this is what we end up with we just
7:15
get this just just raw text raw internet
7:19
text and there's a ton of it even in
7:21
these 200 web pages so I can continue
7:23
zooming out here and we just have this
7:25
like massive tapestry of Text data and
7:28
this text data has all these p patterns
7:30
and what we want to do now is we want to
7:32
start training neural networks on this
7:33
data so the neural networks can
7:35
internalize and model how this text
7:39
flows right so we just have this giant
7:43
texture of text and now we want to get
7:45
neural Nets that mimic it okay now
7:48
before we plug text into neural networks
7:51
we have to decide how we're going to
7:53
represent this text uh and how we're
7:55
going to feed it in now the way our
7:57
technology works for these neuron Lots
7:59
is that they expect
8:00
a one-dimensional sequence of symbols
8:03
and they want a finite set of symbols
8:06
that are possible and so we have to
8:08
decide what are the symbols and then we
8:10
have to represent our data as
8:12
one-dimensional sequence of those
8:14
symbols so right now what we have is a
8:16
onedimensional sequence of text it
8:19
starts here and it goes here and then it
8:21
comes here Etc so this is a
8:22
onedimensional sequence even though on
8:24
my monitor of course it's laid out in a
8:26
two-dimensional way but it goes from
8:28
left to right and top to bottom right so
8:30
it's a one-dimensional sequence of text
8:32
now this being computers of course
8:34
there's an underlying representation
8:35
here so if I do what's called utf8 uh
8:38
encode this text then I can get the raw
8:41
bits that correspond to this text in the
8:44
computer and that's what uh that looks
8:47
like this so it turns out that for
8:50
example this very first bar here is the
8:53
first uh eight bits as an
8:56
example so what is this thing right this
8:59
is um representation that we are looking
9:02
for uh in in a certain sense we have
9:05
exactly two possible symbols zero and
9:07
one and we have a very long sequence of
9:10
it right now as it turns out um this
9:14
sequence length is actually going to be
9:16
very finite and precious resource uh in
9:19
our neural network and we actually don't
9:21
want extremely long sequences of just
9:23
two symbols instead what we want is we
9:26
want to trade off uh this um symbol
9:30
size uh of this vocabulary as we call it
9:33
and the resulting sequence length so we
9:35
don't want just two symbols and
9:36
extremely long sequences we're going to
9:39
want more symbols and shorter sequences
9:42
okay so one naive way of compressing or
9:45
decreasing the length of our sequence
9:46
here is to basically uh consider some
9:50
group of consecutive bits for example
9:52
eight bits and group them into a single
9:55
what's called bite so because uh these
9:58
bits are either on or off if we take a
10:00
group of eight of them there turns out
10:02
to be only 256 possible combinations of
10:04
how these bits could be on or off and so
10:06
therefore we can re repesent this
10:08
sequence into a sequence of bytes
10:11
instead so this sequence of bytes will
10:14
be eight times shorter but now we have
10:16
256 possible symbols so every number
10:19
here goes from 0 to
10:21
255 now I really encourage you to think
10:23
of these not as numbers but as unique
10:25
IDs or like unique symbols so maybe it's
10:28
a bit more maybe it's better to actually
10:30
think of these to replace every one of
10:32
these with a unique Emoji you'd get
10:34
something like this so um we basically
10:37
have a sequence of emojis and there's
10:39
256 possible emojis you can think of it
10:41
that way now it turns out that in
10:45
production for state-of-the-art language
10:46
models uh you actually want to go even
10:48
Beyond this you want to continue to
10:50
shrink the length of the sequence uh
10:53
because again it is a precious resource
10:55
in return for more symbols in your
10:58
vocabulary and the way this is done is
11:00
done by running what's called The Bite
11:02
pair encoding algorithm and the way this
11:04
works is we're basically looking for
11:06
consecutive bytes or symbols that are
11:10
very common so for example turns out
11:14
that the sequence 116 followed by 32 is
11:17
quite common and occurs very frequently
11:19
so what we're going to do is we're going
11:20
to group uh this um pair into a new
11:24
symbol so we're going to Mint a symbol
11:27
with an ID 256 and we're going to
11:29
rewrite every single uh pair 11632 with
11:33
this new symbol and then can we can
11:35
iterate this algorithm as many times as
11:36
we wish and each time when we mint a new
11:39
symbol we're decreasing the length and
11:41
we're increasing the symbol size and in
11:44
practice it turns out that a pretty good
11:45
setting of um the basically the
11:48
vocabulary size turns out to be about
11:50
100,000 possible symbols so in
11:52
particular GPT 4 uses
11:55
100,
11:57
277 symbols
12:00
um and this process of converting from
12:04
raw text into these symbols or as we
12:07
call them tokens is the process called
12:10
tokenization so let's now take a look at
12:13
how gp4 performs tokenization conting
12:16
from text to tokens and from tokens back
12:18
to text and what this actually looks
12:20
like so one website I like to use to
12:22
explore these token representations is
12:25
called tick tokenizer and so come here
12:27
to the drop down and select CL 100 a
12:29
base which is the gp4 base model
12:32
tokenizer and here on the left you can
12:34
put in text and it shows you the
12:36
tokenization of that text so for example
12:41
heo space
12:44
world so hello world turns out to be
12:46
exactly two Tokens The Token hello which
12:49
is the token with ID
12:52
15339 and the token space
12:55
world that is the token 1
12:58
1917 so um hello space world now if I
13:02
was to join these two for example I'm
13:05
going to get again two tokens but it's
13:06
the token H followed by the token L
13:10
world without the
13:12
H um if I put in two Spa two spaces here
13:15
between hello and world it's again a
13:17
different uh tokenization there's a new
13:19
token 220
13:22
here okay so you can play with this and
13:24
see what happens here also keep in mind
13:27
this is not uh this is case sensitive so
13:29
if this is a capital H it is something
13:31
else or if it's uh hello world then
13:35
actually this ends up being three tokens
13:37
instead of just two
13:41
tokens yeah so you can play with this
13:43
and get an sort of like an intuitive
13:45
sense of uh what these tokens work like
13:47
we're actually going to loop around to
13:48
tokenization a bit later in the video
13:50
for now I just wanted to show you the
13:51
website and I wanted to uh show you that
13:54
this text basically at the end of the
13:56
day so for example if I take one line
13:58
here this is what GT4 will see it as so
14:01
this text will be a sequence of length
14:04
62 this is the sequence here and this is
14:08
how the chunks of text correspond to
14:12
these symbols and again there's 100,
14:17
27777 possible symbols and we now have
14:19
one-dimensional sequences of those
14:22
symbols so um yeah we're going to come
14:24
back to tokenization but that's uh for
14:26
now where we are okay so what I've done
14:28
now is I've taken this uh sequence of
14:30
text that we have here in the data set
14:32
and I have re-represented it using our
14:34
tokenizer into a sequence of tokens and
14:37
this is what that looks like now so for
14:40
example when we go back to the Fine web
14:42
data set they mentioned that not only is
14:44
this 44 terab of dis space but this is
14:46
about a 15 trillion token sequence of um
14:51
in this data set and so here these are
14:54
just some of the first uh one or two or
14:56
three or a few thousand here I think uh
14:59
tokens of this data set but there's 15
15:01
trillion here uh to keep in mind and
15:04
again keep in mind one more time that
15:06
all of these represent little text
15:07
chunks they're all just like atoms of
15:10
these sequences and the numbers here
15:12
don't make any sense they're just uh
15:13
they're just unique IDs okay so now we
15:17
get to the fun part which is the uh
15:20
neural network training and this is
15:21
where a lot of the heavy lifting happens
15:23
computationally when you're training
15:25
these neural networks so what we do here
15:28
in this this step is we want to model
15:31
the statistical relationships of how
15:32
these tokens follow each other in the
15:34
sequence so what we do is we come into
15:36
the data and we take Windows of tokens
15:40
so we take a window of tokens uh from
15:43
this data fairly
15:45
randomly and um the windows length can
15:49
range anywhere anywhere between uh zero
15:52
tokens actually all the way up to some
15:54
maximum size that we decide on uh so for
15:57
example in practice you could see a
15:59
token with Windows of say 8,000 tokens
16:01
now in principle we can use arbitrary
16:04
window lengths of tokens uh but uh
16:07
processing very long uh basically U
16:11
window sequences would just be very
16:13
computationally expensive so we just
16:15
kind of decide that say 8,000 is a good
16:17
number or 4,000 or 16,000 and we crop it
16:20
there now in this example I'm going to
16:22
be uh taking the first four tokens just
16:25
so everything fits nicely so these
16:28
tokens
16:30
we're going to take a window of four
16:32
tokens this bar view in and space single
16:37
which are these token
16:39
IDs and now what we're trying to do here
16:41
is we're trying to basically predict the
16:43
token that comes next in the sequence so
16:46
3962 comes next right so what we do now
16:49
here is that we call this the context
16:52
these four tokens are context and they
16:54
feed into a neural
16:56
network and this is the input to the
16:58
neural network
17:00
now I'm going to go into the detail of
17:02
what's inside this neural network in a
17:03
little bit for now it's important to
17:05
understand is the input and the output
17:06
of the neural net so the input are
17:09
sequences of tokens of variable length
17:12
anywhere between zero and some maximum
17:14
size like 8,000 the output now is a
17:18
prediction for what comes next so
17:21
because our vocabulary has
17:24
100277 possible tokens the neural
17:27
network is going to Output exactly that
17:28
many numbers
17:29
and all of those numbers correspond to
17:31
the probability of that token as coming
17:34
next in the sequence so it's making
17:36
guesses about what comes
17:37
next um in the beginning this neural
17:40
network is randomly initialized so um
17:43
and we're going to see in a little bit
17:44
what that means but it's a it's a it's a
17:47
random transformation so these
17:48
probabilities in the very beginning of
17:50
the training are also going to be kind
17:51
of random uh so here I have three
17:54
examples but keep in mind that there's
17:55
100,000 numbers here um so the
17:58
probability of this token space
18:00
Direction neural network is saying that
18:02
this is 4% likely right now 11799 is 2%
18:06
and then here the probility of 3962
18:08
which is post is 3% now of course we've
18:11
sampled this window from our data set so
18:14
we know what comes next we know and
18:16
that's the label we know that the
18:18
correct answer is that 3962 actually
18:20
comes next in the sequence so now what
18:23
we have is this mathematical process for
18:26
doing an update to the neural network we
18:28
have the way of tuning it and uh we're
18:31
going to go into a little bit of of
18:32
detail in a bit but basically we know
18:35
that this probability here of 3% we want
18:38
this probability to be higher and we
18:41
want the probabilities of all the other
18:42
tokens to be
18:44
lower and so we have a way of
18:46
mathematically calculating how to adjust
18:49
and update the neural network so that
18:52
the correct answer has a slightly higher
18:54
probability so if I do an update to the
18:56
neural network now the next time I Fe
18:59
this particular sequence of four tokens
19:01
into neural network the neural network
19:03
will be slightly adjusted now and it
19:04
will say Okay post is maybe 4% and case
19:07
now maybe is
19:09
1% and uh Direction could become 2% or
19:12
something like that and so we have a way
19:14
of nudging of slightly updating the
19:16
neuronet to um basically give a higher
19:20
probability to the correct token that
19:21
comes next in the sequence and now you
19:23
just have to remember that this process
19:26
happens not just for uh this um token
19:29
here where these four fed in and
19:31
predicted this one this process happens
19:34
at the same time for all of these tokens
19:36
in the entire data set and so in
19:38
practice we sample little windows little
19:40
batches of Windows and then at every
19:43
single one of these tokens we want to
19:45
adjust our neural network so that the
19:47
probability of that token becomes
19:48
slightly higher and this all happens in
19:50
parallel in large batches of these
19:52
tokens and this is the process of
19:54
training the neural network it's a
19:56
sequence of updating it so that it's
19:59
predictions match up the statistics of
20:01
what actually happens in your training
20:03
set and its probabilities become
20:05
consistent with the uh statistical
20:08
patterns of how these tokens follow each
20:10
other in the data so let's now briefly
20:12
get into the internals of these neural
20:13
networks just to give you a sense of
20:15
what's inside so neural network
20:17
internals so as I mentioned we have
20:20
these inputs uh that are sequences of
20:22
tokens in this case this is four input
20:25
tokens but this can be anywhere between
20:27
zero up to let's say 8,000 tokens in
20:30
principle this can be an infinite number
20:32
of tokens we just uh it would just be
20:34
too computationally expensive to process
20:36
an infinite number of tokens so we just
20:38
crop it at a certain length and that
20:39
becomes the maximum context length of
20:41
that uh
20:43
model now these inputs X are mixed up in
20:46
a giant mathematical expression together
20:49
with the parameters or the weights of
20:52
these neural networks so here I'm
20:54
showing six example parameters and their
20:57
setting but in practice these uh um
21:00
modern neural networks will have
21:02
billions of these uh parameters and in
21:05
the beginning these parameters are
21:06
completely randomly set now with a
21:09
random setting of parameters you might
21:11
expect that this uh this neural network
21:14
would make random predictions and it
21:15
does in the beginning it's totally
21:17
random predictions but it's through this
21:20
process of iteratively updating the
21:23
network uh as and we call that process
21:25
training a neural network so uh that the
21:28
setting of these parameters gets
21:29
adjusted such that the outputs of our
21:32
neural network becomes consistent with
21:34
the patterns seen in our training
21:36
set so think of these parameters as kind
21:39
of like knobs on a DJ set and as you're
21:41
twiddling these knobs you're getting
21:43
different uh predictions for every
21:45
possible uh token sequence input and
21:49
training in neural network just means
21:51
discovering a setting of parameters that
21:53
seems to be consistent with the
21:55
statistics of the training
21:56
set now let me just give you an example
21:59
what this giant mathematical expression
22:00
looks like just to give you a sense and
22:02
modern networks are massive expressions
22:04
with trillions of terms probably but let
22:06
me just show you a simple example here
22:09
it would look something like this I mean
22:10
these are the kinds of Expressions just
22:11
to show you that it's not very scary we
22:14
have inputs x uh like X1 x2 in this case
22:17
two example inputs and they get mixed up
22:20
with the weights of the network w0 W1 2
22:23
3 Etc and this mixing is simple things
22:27
like multiplication addition addition
22:30
exponentiation division Etc and it is
22:33
the subject of neural network
22:34
architecture research to design
22:37
effective mathematical Expressions uh
22:39
that have a lot of uh kind of convenient
22:41
characteristics they are expressive
22:43
they're optimizable they're paralyzable
22:45
Etc and so but uh at the end of the day
22:48
these are these are not complex
22:50
expressions and basically they mix up
22:52
the inputs with the parameters to make
22:54
predictions and we're optimizing uh the
22:58
parameters of this neural network so
23:00
that the predictions come out consistent
23:02
with the training set now I would like
23:04
to show you an actual production grade
23:06
example of what these neural networks
23:08
look like so for that I encourage you to
23:10
go to this website that has a very nice
23:12
visualization of one of these
23:14
networks so this is what you will find
23:16
on this website and this neural network
23:19
here that is used in production settings
23:22
has this special kind of structure this
23:24
network is called the Transformer and
23:27
this particular one as an example has 8
23:29
5,000 roughly
23:31
parameters now here on the top we take
23:33
the inputs which are the token
23:36
sequences and then information flows
23:40
through the neural network until the
23:42
output which here are the logit softmax
23:45
but these are the predictions for what
23:46
comes next what token comes
23:49
next and then here there's a sequence of
23:52
Transformations and all these
23:54
intermediate values that get produced
23:56
inside this mathematical expression s it
23:59
is sort of predicting what comes next so
24:01
as an example these tokens are embedded
24:05
into kind of like this distributed
24:06
representation as it's called so every
24:08
possible token has kind of like a vector
24:10
that represents it inside the neural
24:12
network so first we embed the tokens and
24:15
then those values uh kind of like flow
24:18
through this diagram and these are all
24:21
very simple mathematical Expressions
24:22
individually so we have layer norms and
24:24
Matrix multiplications and uh soft Maxes
24:27
and so on so here kind of like the
24:29
attention block of this Transformer and
24:32
then information kind of flows through
24:34
into the multi-layer perceptron block
24:35
and so on and all these numbers here
24:39
these are the intermediate values of the
24:40
expression and uh you can almost think
24:42
of these as kind of like the firing
24:45
rates of these synthetic neurons but I
24:48
would caution you to uh not um kind of
24:50
think of it too much like neurons
24:53
because these are extremely simple
24:54
neurons compared to the neurons you
24:55
would find in your brain your biological
24:57
neurons are very complex dynamical
24:59
processes that have memory and so on
25:01
there's no memory in this expression
25:03
it's a fixed mathematical expression
25:04
from input to Output with no memory it's
25:07
just a
25:07
stateless so these are very simple
25:09
neurons in comparison to biological
25:11
neurons but you can still kind of
25:12
loosely think of this as like a
25:14
synthetic piece of uh brain tissue if
25:16
you if you like uh to think about it
25:18
that way so information flows through
25:21
all these neurons fire until we get to
25:24
the predictions now I'm not actually
25:27
going to dwell too much on the precise
25:29
kind of like mathematical details of all
25:30
these Transformations honestly I don't
25:32
think it's that important to get into
25:34
what's really important to understand is
25:35
that this is a mathematical function it
25:38
is uh parameterized by some fixed set of
25:42
parameters like say 85,000 of them and
25:44
it is a way of transforming inputs into
25:46
outputs and as we twiddle the parameters
25:49
we are getting uh different kinds of
25:51
predictions and then we need to find a
25:53
good setting of these parameters so that
25:54
the predictions uh sort of match up with
25:57
the patterns seen in training set
25:59
so that's the Transformer okay so I've
26:02
shown you the internals of the neural
26:03
network and we talked a bit about the
26:05
process of training it I want to cover
26:07
one more major stage of working with
26:10
these networks and that is the stage
26:12
called inference so in inference what
26:14
we're doing is we're generating new data
26:16
from the model and so uh we want to
26:19
basically see what kind of patterns it
26:21
has internalized in the parameters of
26:23
its Network so to generate from the
26:27
model is relatively straightforward
26:29
we start with some tokens that are
26:31
basically your prefix like what you want
26:33
to start with so say we want to start
26:34
with the token 91 well we feed it into
26:37
the
26:38
network and remember that the network
26:40
gives us probabilities right it gives us
26:43
this probability Vector here so what we
26:45
can do now is we can basically flip a
26:47
biased coin so um we can sample uh
26:52
basically a token based on this
26:55
probability distribution so the tokens
26:57
that are given High probability by the
26:59
model are more likely to be sampled when
27:02
you flip this biased coin you can think
27:04
of it that way so we sample from the
27:06
distribution to get a single unique
27:08
token so for example token 860 comes
27:11
next uh so 860 in this case when we're
27:14
generating from model could come next
27:16
now 860 is a relatively likely token it
27:19
might not be the only possible token in
27:21
this case there could be many other
27:22
tokens that could have been sampled but
27:24
we could see that 86c is a relatively
27:25
likely token as an example and indeed in
27:28
our training examp example here 860 does
27:30
follow 91 so let's now say that we um
27:34
continue the process so after 91 there's
27:37
a60 we append it and we again ask what
27:39
is the third token let's sample and
27:42
let's just say that it's 287 exactly as
27:45
here let's do that again we come back in
27:48
now we have a sequence of three and we
27:50
ask what is the likely fourth token and
27:53
we sample from that and get this one and
27:56
now let's say we do it one more time we
27:58
take those four we sample and we get
28:00
this one and this
28:03
13659 uh this is not actually uh 3962 as
28:07
we had before so this token is the token
28:10
article uh instead so viewing a single
28:13
article and so in this case we didn't
28:15
exactly reproduce the sequence that we
28:17
saw here in the training data so keep in
28:20
mind that these systems are stochastic
28:23
they have um we're sampling and we're
28:26
flipping coins and sometimes we lock out
28:29
and we reproduce some like small chunk
28:31
of the text and training set but
28:33
sometimes we're uh we're getting a token
28:36
that was not verbatim part of any of the
28:38
documents in the training data so we're
28:40
going to get sort of like remixes of the
28:43
data that we saw in the training because
28:45
at every step of the way we can flip and
28:47
get a slightly different token and then
28:49
once that token makes it in if you
28:51
sample the next one and so on you very
28:53
quickly uh start to generate token
28:55
streams that are very different from the
28:57
token streams that UR
28:59
in the training documents so
29:01
statistically they will have similar
29:02
properties but um they are not identical
29:05
to your training data they're kind of
29:07
like inspired by the training data and
29:09
so in this case we got a slightly
29:11
different sequence and why would we get
29:13
article you might imagine that article
29:15
is a relatively likely token in the
29:17
context of bar viewing single Etc and
29:21
you can imagine that the word article
29:22
followed this context window somewhere
29:25
in the training documents uh to some
29:27
extent and we just happen to sample it
29:29
here at that stage so basically
29:31
inference is just uh predicting from
29:33
these distributions one at a time we
29:35
continue feeding back tokens and getting
29:37
the next one and we uh we're always
29:40
flipping these coins and depending on
29:42
how lucky or unlucky we get um we might
29:46
get very different kinds of patterns
29:47
depending on how we sample from these
29:49
probability distributions so that's
29:52
inference so in most common scenarios uh
29:55
basically downloading the internet and
29:57
tokenizing it is is a pre-processing
29:59
step you do that a single time and then
30:02
uh once you have your token sequence we
30:04
can start training networks and in
30:07
Practical cases you would try to train
30:09
many different networks of different
30:10
kinds of uh settings and different kinds
30:12
of arrangements and different kinds of
30:14
sizes and so you''ll be doing a lot of
30:15
neural network training and um then once
30:18
you have a neural network and you train
30:20
it and you have some specific set of
30:22
parameters that you're happy with um
30:24
then you can take the model and you can
30:26
do inference and you can actually uh
30:28
generate data from the model and when
30:30
you're on chat GPT and you're talking
30:32
with a model uh that model is trained
30:34
and has been trained by open aai many
30:36
months ago probably and they have a
30:39
specific set of Weights that work well
30:41
and when you're talking to the model all
30:43
of that is just inference there's no
30:45
more training those parameters are held
30:47
fixed and you're just talking to the
30:49
model sort of uh you're giving it some
30:52
of the tokens and it's kind of
30:53
completing token sequences and that's
30:55
what you're seeing uh generated when you
30:57
actually use the model on CH GPT so that
31:00
model then just does inference alone so
31:02
let's now look at an example of training
31:04
an inference that is kind of concrete
31:06
and gives you a sense of what this
31:07
actually looks like uh when these models
31:09
are trained now the example that I would
31:11
like to work with and that I'm
31:12
particularly fond of is that of opening
31:14
eyes gpt2 so GPT uh stands for
31:17
generatively pre-trained Transformer and
31:20
this is the second iteration of the GPT
31:22
series by open AI when you are talking
31:24
to chat GPT today the model that is
31:26
underlying all of the magic of that
31:28
interaction is GPT 4 so the fourth
31:30
iteration of that series now gpt2 was
31:33
published in 2019 by openi in this paper
31:36
that I have right here and the reason I
31:39
like gpt2 is that it is the first time
31:42
that a recognizably modern stack came
31:44
together so um all of the pieces of gpd2
31:49
are recognizable today by modern
31:51
standards it's just everything has
31:52
gotten bigger now I'm not going to be
31:54
able to go into the full details of this
31:56
paper of course because it is a
31:57
technical publication but some of the
32:00
details that I would like to highlight
32:01
are as follows gpt2 was a Transformer
32:04
neural network just like you were just
32:06
like the neural networks you would work
32:07
with today it was it had 1.6 billion
32:10
parameters right so these are the
32:12
parameters that we looked at here it
32:14
would have 1.6 billion of them today
32:17
modern Transformers would have a lot
32:18
closer to a trillion or several hundred
32:20
billion
32:22
probably the maximum context length here
32:25
was 1,24 tokens so it is when we are
32:29
sampling chunks of Windows of tokens
32:32
from the data set we're never taking
32:34
more than 1,24 tokens and so when you
32:37
are trying to predict the next token in
32:38
a sequence you will never have more than
32:40
1,24 tokens uh kind of in your context
32:43
in order to make that prediction now
32:46
this is also tiny by modern standards
32:47
today the token uh the context lengths
32:50
would be a lot closer to um couple
32:53
hundred thousand or maybe even a million
32:55
and so you have a lot more context a lot
32:57
more tokens in history history and you
32:59
can make a lot better prediction about
33:00
the next token in the sequence in that
33:02
way and finally gpt2 was trained on
33:05
approximately 100 billion tokens and
33:07
this is also fairly small by modern
33:09
standards as I mentioned the fine web
33:10
data set that we looked at here the fine
33:12
web data set has 15 trillion tokens uh
33:15
so 100 billion is is quite
33:17
small
33:19
now uh I actually tried to reproduce uh
33:21
gpt2 for fun as part of this project
33:24
called lm. C so you can see my rup of
33:27
doing that in this post on GitHub under
33:30
the lm. C repository so in particular
33:34
the cost of training gpd2 in 2019 what
33:37
was estimated to be approximately
33:39
$40,000 but today you can do
33:41
significantly better than that and in
33:43
particular here it took about one day
33:46
and about
33:47
$600 uh but this wasn't even trying too
33:50
hard I think you could really bring this
33:51
down to about $100 today now why is it
33:55
that the costs have come down so much
33:57
well number one these data sets have
33:59
gotten a lot better and the way we
34:01
filter them extract them and prepare
34:03
them has gotten a lot more refined and
34:06
so the data set is of just a lot higher
34:08
quality so that's one thing but really
34:10
the biggest difference is that our
34:12
computers have gotten much faster in
34:14
terms of the hardware and we're going to
34:15
look at that in a second and also the
34:17
software for uh running these models and
34:20
really squeezing out all all the speed
34:23
from the hardware as it is possible uh
34:26
that software has also gotten much
34:27
better as as everyone has focused on
34:29
these models and try to run them very
34:30
very
34:31
quickly now I'm not going to be able to
34:34
go into the full detail of this gpd2
34:37
reproduction and this is a long
34:38
technical post but I would like to still
34:40
give you an intuitive sense for what it
34:42
looks like to actually train one of
34:43
these models as a researcher like what
34:45
are you looking at and what does it look
34:46
like what does it feel like so let me
34:48
give you a sense of that a little bit
34:50
okay so this is what it looks like let
34:51
me slide this
34:53
over so what I'm doing here is I'm
34:56
training a gpt2 model right now
34:59
and um what's happening here is that
35:01
every single line here like this one is
35:05
one update to the model so remember how
35:09
here we are um basically making the
35:12
prediction better for every one of these
35:14
tokens and we are updating these weights
35:16
or parameters of the neural net so here
35:19
every single line is One update to the
35:21
neural network where we change its
35:23
parameters by a little bit so that it is
35:24
better at predicting next token and
35:26
sequence in particular every single line
35:29
here is improving the prediction on 1
35:32
million tokens in the training set so
35:36
we've basically taken 1 million tokens
35:39
out of this data set and we've tried to
35:42
improve the prediction of that token as
35:45
coming next in a sequence on all 1
35:47
million of them
35:49
simultaneously and at every single one
35:51
of these steps we are making an update
35:53
to the network for that now the number
35:55
to watch closely is this number called
35:58
loss and the loss is a single number
36:01
that is telling you how well your neural
36:03
network is performing right now and it
36:05
is created so that low loss is good so
36:09
you'll see that the loss is decreasing
36:11
as we make more updates to the neural
36:13
nut which corresponds to making better
36:14
predictions on the next token in a
36:16
sequence and so the loss is the number
36:19
that you are watching as a neural
36:21
network researcher and you are kind of
36:23
waiting you're twiddling your thumbs uh
36:24
you're drinking coffee and you're making
36:26
sure that this looks good so that with
36:29
every update your loss is improving and
36:31
the network is getting better at
36:33
prediction now here you see that we are
36:36
processing 1 million tokens per update
36:39
each update takes about 7 Seconds
36:41
roughly and here we are going to process
36:44
a total of 32,000 steps of
36:47
optimization so 32,000 steps with 1
36:50
million tokens each is about 33 billion
36:53
tokens that we are going to process and
36:55
we're currently only about 420 step 20
36:58
out of 32,000 so we are still only a bit
37:01
more than 1% done because I've only been
37:03
running this for 10 or 15 minutes or
37:05
something like
37:06
that now every 20 steps I have
37:09
configured this optimization to do
37:11
inference so what you're seeing here is
37:13
the model is predicting the next token
37:15
in a sequence and so you sort of start
37:18
it randomly and then you continue
37:20
plugging in the tokens so we're running
37:22
this inference step and this is the
37:24
model sort of predicting the next token
37:25
in the sequence and every time you see
37:26
something appear that's a new
37:29
token um so let's just look at this and
37:34
you can see that this is not yet very
37:36
coherent and keep in mind that this is
37:37
only 1% of the way through training and
37:40
so the model is not yet very good at
37:41
predicting the next token in the
37:42
sequence so what comes out is actually
37:45
kind of a little bit of gibberish right
37:47
but it still has a little bit of like
37:49
local coherence so since she is mine
37:52
it's a part of the information should
37:53
discuss my father great companions
37:56
Gordon showed me sitting over at and Etc
37:59
so I know it doesn't look very good but
38:01
let's actually scroll up and see what it
38:04
looked like when I started the
38:06
optimization so all the way here at
38:10
step
38:12
one so after 20 steps of optimization
38:15
you see that what we're getting here is
38:17
looks completely random and of course
38:19
that's because the model has only had 20
38:20
updates to its parameters and so it's
38:22
giving you random text because it's a
38:24
random Network and so you can see that
38:26
at least in comparison to this model is
38:28
starting to do much better and indeed if
38:30
we waited the entire 32,000 steps the
38:33
model will have improved the point that
38:34
it's actually uh generating fairly
38:36
coherent English uh and the tokens
38:39
stream correctly um and uh they they
38:42
kind of make up English a a lot
38:45
better
38:46
um so this has to run for about a day or
38:50
two more now and so uh at this stage we
38:52
just make sure that the loss is
38:54
decreasing everything is looking good um
38:56
and we just have to wait
38:58
and now um let me turn now to the um
39:02
story of the computation that's required
39:05
because of course I'm not running this
39:06
optimization on my laptop that would be
39:09
way too expensive uh because we have to
39:11
run this neural network and we have to
39:13
improve it and we have we need all this
39:15
data and so on so you can't run this too
39:17
well on your computer uh because the
39:19
network is just too large uh so all of
39:21
this is running on the computer that is
39:23
out there in the cloud and I want to
39:25
basically address the compute side of
39:27
the store of training these models and
39:29
what that looks like so let's take a
39:30
look okay so the computer that I'm
39:32
running this optimization on is this 8X
39:35
h100 node so there are eight h100s in a
39:40
single node or a single computer now I
39:43
am renting this computer and it is
39:44
somewhere in the cloud I'm not sure
39:46
where it is physically actually the
39:48
place I like to rent from is called
39:49
Lambda but there are many other
39:50
companies who provide this service so
39:52
when you scroll down you can see that uh
39:56
they have some on demand pricing for
39:58
um sort of computers that have these uh
40:02
h100s which are gpus and I'm going to
40:04
show you what they look like in a second
40:06
but on demand 8times Nvidia h100 uh
40:10
GPU this machine comes for $3 per GPU
40:14
per hour for example so you can rent
40:17
these and then you get a machine in a
40:18
cloud and you can uh go in and you can
40:20
train these
40:22
models and these uh gpus they look like
40:26
this so this is one h100 GPU uh this is
40:29
kind of what it looks like and you slot
40:30
this into your computer and gpus are
40:33
this uh perfect fit for training your
40:35
networks because they are very
40:37
computationally expensive but they
40:39
display a lot of parallelism in the
40:41
computation so you can have many
40:42
independent workers kind of um working
40:45
all at the same time in solving uh the
40:48
matrix multiplication that's under the
40:51
hood of training these neural
40:53
networks so this is just one of these
40:55
h100s but actually you would put them
40:57
you would put multiple of them together
40:59
so you could stack eight of them into a
41:00
single node and then you can stack
41:02
multiple nodes into an entire data
41:04
center or an entire system
41:07
so when we look at a data
41:12
center can't spell when we look at a
41:15
data center we start to see things that
41:16
look like this right so we have one GPU
41:18
goes to eight gpus goes to a single
41:20
system goes to many systems and so these
41:22
are the bigger data centers and there of
41:24
course would be much much more expensive
41:26
um and what's happening is that all the
41:29
big tech companies really desire these
41:31
gpus so they can train all these
41:33
language models because they are so
41:35
powerful and that has is fundamentally
41:37
what has driven the stock price of
41:39
Nvidia to be $3.4 trillion today as an
41:42
example and why Nvidia has kind of
41:45
exploded so this is the Gold Rush the
41:47
Gold Rush is getting the gpus getting
41:50
enough of them so they can all
41:52
collaborate to perform this optimization
41:55
and they're what are they all doing
41:57
they're all collaborating to predict the
41:59
next token on a data set like the fine
42:02
web data
42:03
set this is the computational workflow
42:05
that that basically is extremely
42:07
expensive the more gpus you have the
42:09
more tokens you can try to predict and
42:11
improve on and you're going to process
42:13
this data set faster and you can iterate
42:15
faster and get a bigger Network and
42:17
train a bigger Network and so on so this
42:19
is what all those machines are look like
42:21
are uh are doing and this is why all of
42:24
this is such a big deal and for example
42:27
this is a
42:29
article from like about a month ago or
42:30
so this is why it's a big deal that for
42:32
example Elon Musk is getting 100,000
42:35
gpus uh in a single Data Center and all
42:39
of these gpus are extremely expensive
42:41
are going to take a ton of power and all
42:42
of them are just trying to predict the
42:44
next token in the sequence and improve
42:46
the network uh by doing so and uh get
42:49
probably a lot more coherent text than
42:51
what we're seeing here a lot faster okay
42:53
so unfortunately I do not have a couple
42:55
10 or hundred million of dollars to
42:57
spend on training a really big model
42:59
like this but luckily we can turn to
43:01
some big tech companies who train these
43:04
models routinely and release some of
43:06
them once they are done training so
43:09
they've spent a huge amount of compute
43:10
to train this network and they release
43:12
the network at the end of the
43:14
optimization so it's very useful because
43:16
they've done a lot of compute for that
43:18
so there are many companies who train
43:19
these models routinely but actually not
43:21
many of them release uh these what's
43:23
called base models so the model that
43:26
comes out at the end here is is what's
43:27
called a base model what is a base model
43:30
it's a token simulator right it's an
43:32
internet text token simulator and so
43:36
that is not by itself useful yet because
43:38
what we want is what's called an
43:39
assistant we want to ask questions and
43:41
have it respond to answers these models
43:44
won't do that they just uh create sort
43:46
of remixes of the internet they dream
43:49
internet pages so the base models are
43:52
not very often released because they're
43:53
kind of just only a step one of a few
43:55
other steps that we still need to take
43:56
to get in system
43:58
however a few releases have been made so
44:01
as an example the gbt2 model released
44:04
the 1.6 billion sorry 1.5 billion model
44:08
back in 2019 and this gpt2 model is a
44:11
base model now what is a model release
44:14
what does it look like to release these
44:15
models so this is the gpt2 repository on
44:18
GitHub well you need two things
44:20
basically to release model number one we
44:23
need the um python code usually that
44:28
describes the sequence of operations in
44:30
detail that they make in their model so
44:35
um if you remember
44:37
back this
44:39
Transformer the sequence of steps that
44:41
are taken here in this neural network is
44:43
what is being described by this code so
44:46
this code is sort of implementing the
44:47
what's called forward pass of this
44:49
neural network so we need the specific
44:52
details of exactly how they wired up
44:54
that neural network so this is just
44:56
computer code and it's usually just a
44:57
couple hundred lines of code it's not
44:59
it's not that crazy and uh this is all
45:02
fairly understandable and usually fairly
45:03
standard what's not standard are the
45:05
parameters that's where the actual value
45:07
is what are the parameters of this
45:09
neural network because there's 1.6
45:11
billion of them and we need the correct
45:13
setting or a really good setting and so
45:16
that's why in addition to this source
45:18
code they release the parameters which
45:20
in this case is roughly 1.5 billion
45:23
parameters and these are just numbers so
45:25
it's one single list of 1.5 billion
45:27
numbers the precise and good setting of
45:30
all the knobs such that the tokens come
45:32
out
45:34
well so uh you need those two things to
45:37
get a base model
45:40
release
45:41
now gpt2 was released but that's
45:44
actually a fairly old model as I
45:45
mentioned so actually the model we're
45:46
going to turn to is called llama 3 and
45:49
that's the one that I would like to show
45:50
you next so llama 3 so gpt2 again was
45:54
1.6 billion parameters trained on 100
45:56
billion tokens Lama 3 is a much bigger
45:59
model and much more modern model it is
46:01
released and trained by meta and it is a
46:04
45 billion parameter model trained on 15
46:07
trillion tokens in very much the same
46:10
way just much much
46:11
bigger um and meta has also made a
46:15
release of llama 3 and that was part of
46:18
this
46:19
paper so with this paper that goes into
46:21
a lot of detail the biggest base model
46:23
that they released is the Lama 3.1 4.5
46:27
405 billion parameter model so this is
46:30
the base model and then in addition to
46:32
the base model you see here
46:33
foreshadowing for later sections of the
46:35
video they also released the instruct
46:37
model and the instruct means that this
46:39
is an assistant you can ask it questions
46:41
and it will give you answers we still
46:43
have yet to cover that part later for
46:45
now let's just look at this base model
46:47
this token simulator and let's play with
46:49
it and try to think about you know what
46:51
is this thing and how does it work and
46:54
um what do we get at the end of this
46:55
optimization if you let this run Until
46:57
the End uh for a very big neural network
47:00
on a lot of data so my favorite place to
47:02
interact with the base models is this um
47:05
company called hyperbolic which is
47:07
basically serving the base model of the
47:10
405b Llama 3.1 so when you go to the
47:13
website and I think you may have to
47:15
register and so on make sure that in the
47:17
models make sure that you are using
47:19
llama 3.1 405 billion base it must be
47:22
the base model and then here let's say
47:25
the max tokens is how many tokens we're
47:26
going to be gener rating so let's just
47:28
decrease this to be a bit less just so
47:30
we don't waste compute we just want the
47:32
next 128 tokens and leave the other
47:35
stuff alone I'm not going to go into the
47:36
full detail here um now fundamentally
47:39
what's going to happen here is identical
47:42
to what happens here during inference
47:43
for us so this is just going to continue
47:45
the token sequence of whatever you
47:47
prefix you're going to give it so I want
47:50
to first show you that this model here
47:52
is not yet an assistant so you can for
47:54
example ask it what is 2 plus 2 it's not
47:56
going to tell you oh it's four uh what
47:58
else can I help you with it's not going
48:00
to do that because what is 2 plus 2 is
48:03
going to be tokenized and then those
48:05
tokens just act as a prefix and then
48:08
what the model is going to do now is
48:09
just going to get the probability for
48:10
the next token and it's just a glorified
48:12
autocomplete it's a very very expensive
48:15
autocomplete of what comes next um
48:17
depending on the statistics of what it
48:19
saw in its training documents which are
48:21
basically web
48:22
pages so let's just uh hit enter to see
48:25
what tokens it comes up with as a
48:31
continuation okay so here it kind of
48:33
actually answered the question and
48:34
started to go off into some
48:35
philosophical territory uh let's try it
48:38
again so let me copy and paste and let's
48:40
try again from scratch what is 2 plus
48:46
two so okay so it just goes off again so
48:50
notice one more thing that I want to
48:51
stress is that the system uh I think
48:54
every time you put it in it just kind of
48:55
starts from scratch
48:58
so it doesn't uh the system here is
49:00
stochastic so for the same prefix of
49:02
tokens we're always getting a different
49:04
answer and the reason for that is that
49:07
we get this probity distribution and we
49:09
sample from it and we always get
49:10
different samples and we sort of always
49:12
go into a different territory uh
49:14
afterwards so here in this case um I
49:18
don't know what this is let's try one
49:19
more
49:23
time so it just continues on so it's
49:25
just doing the stuff that it's saw on
49:26
the internet right um and it's just kind
49:29
of like regurgitating those uh
49:31
statistical
49:33
patterns so first things it's not an
49:36
assistant yet it's a token autocomplete
49:39
and second it is a stochastic system now
49:42
the crucial thing is that even though
49:44
this model is not yet by itself very
49:46
useful for a lot of applications just
49:49
yet um it is still very useful because
49:53
in the task of predicting the next token
49:54
in the sequence the model has learned a
49:57
lot about the world and it has stored
49:59
all that knowledge in the parameters of
50:01
the network so remember that our text
50:04
looked like this right internet web
50:06
pages and now all of this is sort of
50:09
compressed in the weights of the network
50:11
so you can think of um these 405 billion
50:15
parameters is a kind of compression of
50:17
the internet you can think of the
50:19
45 billion parameters is kind of like a
50:22
zip file uh but it's not a loss less
50:25
compression it's a loss C compression
50:27
we're kind of like left with kind of a
50:29
gal of the internet and we can generate
50:31
from it right now we can elicit some of
50:34
this knowledge by prompting the base
50:36
model uh accordingly so for example
50:38
here's a prompt that might work to
50:40
elicit some of that knowledge that's
50:42
hiding in the parameters here's my top
50:44
10 list of the top landmarks to see in
50:46
the
50:48
pairs
50:50
um and I'm doing it this way because I'm
50:53
trying to Prime the model to now
50:54
continue this list so let's see if that
50:56
works when I press
50:57
enter okay so you see that it started a
51:01
list and it's now kind of giving me some
51:02
of those
51:03
landmarks and now notice that it's
51:05
trying to give a lot of information here
51:07
now you might not be able to actually
51:09
fully trust some of the information here
51:11
remember that this is all just a
51:12
recollection of some of the internet
51:14
documents and so the things that occur
51:17
very frequently in the internet data are
51:19
probably more likely to be remembered
51:21
correctly compared to things that happen
51:23
very infrequently so you can't fully
51:26
trust some of the things that and some
51:27
of the information that is here because
51:28
it's all just a vague recollection of
51:30
Internet documents because the
51:33
information is not stored explicitly in
51:35
any of the parameters it's all just the
51:37
recollection that said we did get
51:39
something that is probably approximately
51:40
correct and I don't actually have the
51:42
expertise to verify that this is roughly
51:44
correct but you see that we've elicited
51:46
a lot of the knowledge of the model and
51:49
this knowledge is not precise and exact
51:51
this knowledge is vague and
51:54
probabilistic and statistical and the
51:56
kinds of things that occur often are the
51:58
kinds of things that are more likely to
51:59
be remembered um in the model now I want
52:03
to show you a few more examples of this
52:04
model's Behavior the first thing I want
52:06
to show you is this example I went to
52:08
the Wikipedia page for zebra and let me
52:11
just copy paste the first uh even one
52:13
sentence
52:15
here and let me put it here now when I
52:18
click enter what kind of uh completion
52:20
are we going to get so let me just hit
52:23
enter there are three living species
52:27
etc etc what the model is producing here
52:30
is an exact regurgitation of this
52:32
Wikipedia entry it is reciting this
52:34
Wikipedia entry purely from memory and
52:36
this memory is stored in its parameters
52:39
and so it is possible that at some point
52:41
in these 512 tokens the model will uh
52:44
stray away from the Wikipedia entry but
52:46
you can see that it has huge chunks of
52:48
it memorized here uh let me see for
52:50
example if this sentence
52:52
occurs by now okay so this so we're
52:55
still on track let me check
52:59
here okay we're still on
53:01
track it will eventually uh stray
53:04
away okay so this thing is just recited
53:07
to a very large extent it will
53:09
eventually deviate uh because it won't
53:11
be able to remember exactly now the
53:13
reason that this happens is because
53:15
these models can be extremely good at
53:17
memorization and usually this is not
53:19
what you want in the final model and
53:20
this is something called regurgitation
53:22
and it's usually undesirable to site uh
53:25
things uh directly uh that you have
53:27
trained on now the reason that this
53:29
happens actually is because for a lot of
53:32
documents like for example Wikipedia
53:33
when these documents are deemed to be of
53:35
very high quality as a source like for
53:37
example Wikipedia it is very often uh
53:40
the case that when you train the model
53:42
you will preferentially sample from
53:44
those sources so basically the model has
53:46
probably done a few epochs on this data
53:49
meaning that it has seen this web page
53:51
like maybe probably 10 times or so and
53:53
it's a bit like you like when you read
53:54
some kind of a text many many times say
53:57
you read something a 100 times uh then
53:59
you'll be able to recite it and it's
54:00
very similar for this model if it sees
54:02
something way too often it's going to be
54:03
able to recite it later from memory
54:06
except these models can be a lot more
54:08
efficient um like per presentation than
54:11
human so probably it's only seen this
54:12
Wikipedia entry 10 times but basically
54:14
it has remembered this article exactly
54:16
in its parameters okay the next thing I
54:18
want to show you is something that the
54:19
model has definitely not seen during its
54:22
training so for example if we go to the
54:24
paper uh and then we navigate to the
54:27
pre-training data we'll see here that uh
54:31
the data set has a knowledge cut off
54:33
until the end of 2023 so it will not
54:36
have seen documents after this point and
54:39
certainly it has not seen anything about
54:40
the 2024 election and how it turned out
54:43
now if we Prime the model with the
54:46
tokens from the future it will continue
54:49
the token sequence and it will just take
54:51
its best guess according to the
54:52
knowledge that it has in its own
54:53
parameters so let's take a look at what
54:55
that could look like
54:57
so the Republican Party kit
55:00
Trump okay president of the United
55:02
States from
55:03
2017 and let's see what it says after
55:05
this point so for example the model will
55:07
have to guess at the running mate and
55:09
who it's against Etc so let's hit
55:12
enter so here thingss that Mike Pence
55:14
was the running mate instead of JD Vance
55:17
and the ticket was against Hillary
55:21
Clinton and Tim Kane so this is kind of
55:23
a interesting parallel universe
55:25
potentially of what could have happened
55:26
happened according to the LM let's get a
55:29
different sample so the identical prompt
55:31
and let's
55:33
resample so here the running mate was
55:35
Ronda santis and they ran against Joe
55:38
Biden and Camala Harris so this is again
55:41
a different parallel universe so the
55:43
model will take educated guesses and it
55:44
will continue the token sequence based
55:46
on this knowledge um and it will just
55:48
kind of like all of what we're seeing
55:50
here is what's called hallucination the
55:52
model is just taking its best guess uh
55:55
in a probalistic manner the next thing I
55:57
would like to show you is that even
55:58
though this is a base model and not yet
56:00
an assistant model it can still be
56:02
utilized in Practical applications if
56:04
you are clever with your prompt design
56:07
so here's something that we would call a
56:08
few shot
56:09
prompt so what it is here is that I have
56:12
10 words or 10 pairs and each pair is a
56:16
word of English column and then a the
56:20
translation in Korean and we have 10 of
56:23
them and what the model does here is at
56:25
the end we have teacher column and then
56:27
here's where we're going to do a
56:28
completion of say just five tokens and
56:32
these models have what we call in
56:33
context learning abilities and what
56:36
that's referring to is that as it is
56:37
reading this context it is learning sort
56:40
of in
56:42
place that there's some kind of a
56:44
algorithmic pattern going on in my data
56:46
and it knows to continue that pattern
56:49
and this is called kind of like Inc
56:50
context learning so it takes on the role
56:53
of a
56:55
translator and when we hit uh completion
56:58
we see that the teacher translation is
57:00
Sim which is correct um and so this is
57:03
how you can build apps by being clever
57:05
with your prompting even though we still
57:07
just have a base model for now and it
57:09
relies on what we call this um uh in
57:12
context learning ability and it is done
57:14
by constructing what's called a few shot
57:16
prompt okay and finally I want to show
57:18
you that there is a clever way to
57:19
actually instantiate a whole language
57:22
model assistant just by prompting and
57:25
the trick to it is that we're structure
57:27
a prompt to look like a web page that is
57:29
a conversation between a helpful AI
57:32
assistant and a human and then the model
57:34
will continue that conversation so
57:37
actually to write the prompt I turned to
57:39
chat gbt itself which is kind of meta
57:42
but I told it I want to create an llm
57:44
assistant but all I have is the base
57:45
model so can you please write my um uh
57:50
prompt and this is what it came up with
57:52
which is actually quite good so here's a
57:54
conversation between an AI assistant and
57:56
a human
57:57
the AI assistant is knowledgeable
57:58
helpful capable of answering wide
58:00
variety of questions Etc and then here
58:03
it's not enough to just give it a sort
58:05
of description it works much better if
58:08
you create this fot prompt so here's a
58:10
few terms of human assistant human
58:13
assistant and we have uh you know a few
58:16
turns of conversation and then here at
58:18
the end is we're going to be putting the
58:19
actual query that we like so let me copy
58:22
paste this into the base model prompt
58:25
and now let me do human column and this
58:29
is where we put our actual prompt why is
58:31
the sky
58:33
blue and uh let's uh
58:37
run assistant the sky appears blue due
58:40
to the phenomenon called R lights
58:42
scattering etc etc so you see that the
58:44
base model is just continuing the
58:45
sequence but because the sequence looks
58:48
like this conversation it takes on that
58:50
role but it is a little subtle because
58:52
here it just uh you know it ends the
58:54
assistant and then just you know
58:56
hallucinate Ates the next question by
58:57
the human Etc so it'll just continue
58:59
going on and on uh but you can see that
59:01
we have sort of accomplished the task
59:04
and if you just took this why is the sky
59:06
blue and if we just refresh this and put
59:09
it here then of course we don't expect
59:11
this to work with a base model right
59:12
we're just going to who knows what we're
59:14
going to get okay we're just going to
59:15
get more
59:16
questions okay so this is one way to
59:19
create an assistant even though you may
59:22
only have a base model okay so this is
59:24
the kind of brief summary of the things
59:26
we talked about over the last few
59:29
minutes now let me zoom out
59:33
here and this is kind of like what we've
59:35
talked about so far we wish to train LM
59:38
assistants like chpt we've discussed the
59:41
first stage of that which is the
59:42
pre-training stage and we saw that
59:44
really what it comes down to is we take
59:45
Internet documents we break them up into
59:47
these tokens these atoms of little text
59:49
chunks and then we predict token
59:51
sequences using neural networks the
59:54
output of this entire stage is this base
59:57
model it is the setting of The
59:58
parameters of this network and this base
1:00:02
model is basically an internet document
1:00:03
simulator on the token level so it can
1:00:06
just uh it can generate token sequences
1:00:08
that have the same kind of like
1:00:10
statistics as Internet documents and we
1:00:13
saw that we can use it in some
1:00:14
applications but we actually need to do
1:00:15
better we want an assistant we want to
1:00:17
be able to ask questions and we want the
1:00:19
model to give us answers and so we need
1:00:21
to now go into the second stage which is
1:00:24
called the post-training stage so we
1:00:26
take our base model our internet
1:00:28
document simulator and hand it off to
1:00:30
post training so we're now going to
1:00:32
discuss a few ways to do what's called
1:00:34
post training of these models these
1:00:36
stages in post training are going to be
1:00:38
computationally much less expensive most
1:00:40
of the computational work all of the
1:00:42
massive data centers um and all of the
1:00:45
sort of heavy compute and millions of
1:00:48
dollars are the pre-training stage but
1:00:51
now we go into the slightly cheaper but
1:00:53
still extremely important stage called
1:00:55
post trining where we turn this llm
1:00:57
model into an assistant so let's take a
1:00:59
look at how we can get our model to not
1:01:02
sample internet documents but to give
1:01:05
answers to questions so in other words
1:01:08
what we want to do is we want to start
1:01:09
thinking about conversations and these
1:01:11
are conversations that can be multi-turn
1:01:13
so so uh there can be multiple turns and
1:01:16
they are in the simplest case a
1:01:17
conversation between a human and an
1:01:19
assistant and so for example we can
1:01:21
imagine the conversation could look
1:01:22
something like this when a human says
1:01:24
what is 2 plus2 the assistant should re
1:01:26
respond with something like 2 plus 2 is
1:01:27
4 when a human follows up and says what
1:01:30
if it was star instead of a plus
1:01:32
assistant could respond with something
1:01:33
like
1:01:34
this um and similar here this is another
1:01:36
example showing that the assistant could
1:01:38
also have some kind of a personality
1:01:39
here uh that it's kind of like nice and
1:01:42
then here in the third example I'm
1:01:43
showing that when a human is asking for
1:01:45
something that we uh don't wish to help
1:01:47
with we can produce what's called
1:01:49
refusal we can say that we cannot help
1:01:51
with that so in other words what we want
1:01:53
to do now is we want to think through
1:01:55
how in a system should interact with the
1:01:57
human and we want to program the
1:01:59
assistant and Its Behavior in these
1:02:01
conversations now because this is neural
1:02:04
networks we're not going to be
1:02:05
programming these explicitly in code
1:02:08
we're not going to be able to program
1:02:09
the assistant in that way because this
1:02:10
is neural networks everything is done
1:02:12
through neural network training on data
1:02:14
sets and so because of that we are going
1:02:17
to be implicitly programming the
1:02:19
assistant by creating data sets of
1:02:22
conversations so these are three
1:02:23
independent examples of conversations in
1:02:25
a data dat set an actual data set and
1:02:28
I'm going to show you examples will be
1:02:29
much larger it could have hundreds of
1:02:31
thousands of conversations that are
1:02:32
multi- turn very long Etc and would
1:02:35
cover a diverse breath of topics but
1:02:37
here I'm only showing three examples but
1:02:40
the way this works basically is uh a
1:02:43
assistant is being programmed by example
1:02:46
and where is this data coming from like
1:02:47
2 * 2al 4 same as 2 plus 2 Etc where
1:02:50
does that come from this comes from
1:02:52
Human labelers so we will basically give
1:02:55
human labelers some conversational
1:02:56
context and we will ask them to um
1:02:59
basically give the ideal assistant
1:03:01
response in this situation and a human
1:03:04
will write out the ideal response for an
1:03:06
assistant in any situation and then
1:03:08
we're going to get the model to
1:03:10
basically train on this and to imitate
1:03:13
those kinds of
1:03:14
responses so the way this works then is
1:03:16
we are going to take our base model
1:03:18
which we produced in the preing stage
1:03:20
and this base model was trained on
1:03:22
internet documents we're now going to
1:03:24
take that data set of internet documents
1:03:25
and we're gonna throw it out and we're
1:03:27
going to substitute a new data set and
1:03:29
that's going to be a data set of
1:03:31
conversations and we're going to
1:03:32
continue training the model on these
1:03:34
conversations on this new data set of
1:03:36
conversations and what happens is that
1:03:38
the model will very rapidly adjust and
1:03:41
will sort of like learn the statistics
1:03:43
of how this assistant responds to human
1:03:46
queries and then later during inference
1:03:48
we'll be able to basically um Prime the
1:03:52
assistant and get the response and it
1:03:54
will be imitating what the humans will
1:03:56
human labelers would do in that
1:03:57
situation if that makes sense so we're
1:04:00
going to see examples of that and this
1:04:01
is going to become bit more concrete I
1:04:03
also wanted to mention that this
1:04:05
post-training stage we're going to
1:04:06
basically just continue training the
1:04:08
model but um the pre-training stage can
1:04:11
in practice take roughly three months of
1:04:13
training on many thousands of computers
1:04:15
the post-training stage will typically
1:04:17
be much shorter like 3 hours for example
1:04:20
um and that's because the data set of
1:04:22
conversations that we're going to create
1:04:23
here manually is much much smaller than
1:04:26
the data set of text on the internet and
1:04:29
so this training will be very short but
1:04:31
fundamentally we're just going to take
1:04:33
our base model we're going to continue
1:04:35
training using the exact same algorithm
1:04:37
the exact same everything except we're
1:04:39
swapping out the data set for
1:04:41
conversations so the questions now are
1:04:43
what are these conversations how do we
1:04:45
represent them how do we get the model
1:04:47
to see conversations instead of just raw
1:04:49
text and then what are the outcomes of
1:04:53
um this kind of training and what do you
1:04:55
get in a certain like psychological
1:04:57
sense uh when we talk about the model so
1:04:59
let's turn to those questions now so
1:05:01
let's start by talking about the
1:05:02
tokenization of conversations everything
1:05:05
in these models has to be turned into
1:05:07
tokens because everything is just about
1:05:09
token sequences so how do we turn
1:05:11
conversations into token sequences is
1:05:13
the question and so for that we need to
1:05:15
design some kind of ending coding and uh
1:05:17
this is kind of similar to maybe if
1:05:19
you're familiar you don't have to be
1:05:21
with for example the TCP IP packet in um
1:05:24
on the internet there are precise rules
1:05:25
and protocols for how you represent
1:05:27
information how everything is structured
1:05:29
together so that you have all this kind
1:05:30
of data laid out in a way that is
1:05:33
written out on a paper and that everyone
1:05:35
can agree on and so it's the same thing
1:05:37
now happening in llms we need some kind
1:05:38
of data structures and we need to have
1:05:40
some rules around how these data
1:05:42
structures like conversations get
1:05:44
encoded and decoded to and from tokens
1:05:47
and so I want to show you now how I
1:05:49
would
1:05:50
recreate uh this conversation in the
1:05:52
token space so if you go to Tech
1:05:54
tokenizer
1:05:56
I can take that conversation and this is
1:05:59
how it is represented in uh for the
1:06:01
language model so here we have we are
1:06:04
iterating a user and an assistant in
1:06:06
this two- turn
1:06:08
conversation and what you're seeing here
1:06:10
is it looks ugly but it's actually
1:06:12
relatively simple the way it gets turned
1:06:14
into a token sequence here at the end is
1:06:16
a little bit complicated but at the end
1:06:18
this conversation between a user and
1:06:20
assistant ends up being 49 tokens it is
1:06:23
a one-dimensional sequence of 49 tokens
1:06:25
and these are the tokens
1:06:26
okay and all the different llms will
1:06:30
have a slightly different format or
1:06:32
protocols and it's a little bit of a
1:06:33
wild west right now but for example GPT
1:06:36
40 does it in the following way you have
1:06:39
this special token called imore start
1:06:42
and this is short for IM imaginary
1:06:44
monologue uh the
1:06:46
start then you have to specify um I
1:06:50
don't actually know why it's called that
1:06:51
to be honest then you have to specify
1:06:53
whose turn it is so for example user
1:06:55
which is a token 4
1:06:57
28 then you have internal monologue
1:07:00
separator and then it's the exact
1:07:03
question so the tokens of the question
1:07:05
and then you have to close it so I am
1:07:07
end the end of the imaginary monologue
1:07:10
so
1:07:11
basically the question from a user of
1:07:13
what is 2 plus two ends up being the
1:07:16
token sequence of these tokens and now
1:07:19
the important thing to mention here is
1:07:21
that IM start this is not text right IM
1:07:24
start is a special token that gets added
1:07:27
it's a new token and um this token has
1:07:31
never been trained on so far it is a new
1:07:33
token that we create in a post-training
1:07:35
stage and we introduce and so these
1:07:38
special tokens like IM seep IM start Etc
1:07:40
are introduced and interspersed with
1:07:43
text so that they sort of um get the
1:07:45
model to learn that hey this is a the
1:07:47
start of a turn for who is it start of
1:07:50
the turn for the start of the turn is
1:07:52
for the user and then this is what the
1:07:54
user says and then the user ends and
1:07:57
then it's a new start of a turn and it
1:07:59
is by the assistant and then what does
1:08:01
the assistant say well these are the
1:08:03
tokens of what the assistant says Etc
1:08:06
and so this conversation is not turned
1:08:07
into the sequence of tokens the specific
1:08:10
details here are not actually that
1:08:11
important all I'm trying to show you in
1:08:13
concrete terms is that our conversations
1:08:16
which we think of as kind of like a
1:08:17
structured object end up being turned
1:08:19
via some encoding into onedimensional
1:08:22
sequences of tokens and so because this
1:08:25
is one dimensional sequence of tokens we
1:08:27
can apply all the stuff that we applied
1:08:29
before now it's just a sequence of
1:08:31
tokens and now we can train a language
1:08:33
model on it and so we're just predicting
1:08:35
the next token in a sequence uh just
1:08:37
like before and um we can represent and
1:08:40
train on conversations and then what
1:08:42
does it look like at test time during
1:08:44
inference so say we've trained a model
1:08:47
and we've trained a model on these kinds
1:08:49
of data sets of conversations and now we
1:08:51
want to
1:08:52
inference so during inference what does
1:08:55
this look like when you're on on chash
1:08:56
apt well you come to chash apt and you
1:08:59
have say like a dialogue with it and the
1:09:01
way this works is
1:09:03
basically um say that this was already
1:09:06
filled in so like what is 2 plus 2 2
1:09:08
plus 2 is four and now you issue what if
1:09:10
it was times I am end and what basically
1:09:14
ends up happening um on the servers of
1:09:16
open AI or something like that is they
1:09:18
put in I start assistant I amep and this
1:09:22
is where they end it right here so they
1:09:25
construct this context and now they
1:09:27
start sampling from the model so it's at
1:09:29
this stage that they will go to the
1:09:31
model and say okay what is a good for
1:09:32
sequence what is a good first token what
1:09:35
is a good second token what is a good
1:09:37
third token and this is where the LM
1:09:38
takes over and creates a response like
1:09:41
for example response that looks
1:09:44
something like this but it doesn't have
1:09:45
to be identical to this but it will have
1:09:47
the flavor of this if this kind of a
1:09:49
conversation was in the data set so um
1:09:52
that's roughly how the protocol Works
1:09:55
although the details of this protocol
1:09:57
are not important so again my goal is
1:09:59
that just to show you that everything
1:10:01
ends up being just a one-dimensional
1:10:02
token sequence so we can apply
1:10:04
everything we've already seen but we're
1:10:07
now training on conversations and we're
1:10:09
now uh basically generating
1:10:11
conversations as well okay so now I
1:10:13
would like to turn to what these data
1:10:14
sets look like in practice the first
1:10:16
paper that I would like to show you and
1:10:18
the first effort in this direction is
1:10:20
this paper from openai in 2022 and this
1:10:23
paper was called instruct GPT or the
1:10:26
technique that they developed and this
1:10:27
was the first time that opena has kind
1:10:29
of talked about how you can take
1:10:30
language models and fine-tune them on
1:10:33
conversations and so this paper has a
1:10:35
number of details that I would like to
1:10:36
take you through so the first stop I
1:10:38
would like to make is in section 3.4
1:10:40
where they talk about the human
1:10:42
contractors that they hired uh in this
1:10:44
case from upwork or through scale AI to
1:10:47
uh construct these conversations and so
1:10:50
there are human labelers involved whose
1:10:52
job it is professionally to create these
1:10:55
conversations and these labelers are
1:10:57
asked to come up with prompts and then
1:10:59
they are asked to also complete the
1:11:01
ideal assistant responses and so these
1:11:03
are the kinds of prompts that people
1:11:05
came up with so these are human labelers
1:11:07
so list five ideas for how to regain
1:11:09
enthusiasm for my career what are the
1:11:11
top 10 science fiction books I should
1:11:12
read next and there's many different
1:11:14
types of uh kind of prompts here so
1:11:17
translate this sentence from uh to
1:11:19
Spanish Etc and so there's many things
1:11:22
here that people came up with they first
1:11:24
come up with the prompt and then they
1:11:26
also uh answer that prompt and they give
1:11:28
the ideal assistant response now how do
1:11:30
they know what is the ideal assistant
1:11:32
response that they should write for
1:11:34
these prompts so when we scroll down a
1:11:36
little bit further we see that here we
1:11:38
have this excerpt of labeling
1:11:40
instructions uh that are given to the
1:11:41
human labelers so the company that is
1:11:44
developing the language model like for
1:11:45
example open AI writes up labeling
1:11:47
instructions for how the humans should
1:11:50
create ideal responses and so here for
1:11:53
example is an excerpt uh of these kinds
1:11:55
of labeling instruction instructions on
1:11:56
High level you're asking people to be
1:11:58
helpful truthful and harmless and you
1:12:00
can pause the video if you'd like to see
1:12:02
more here but on a high level basically
1:12:04
just just answer try to be helpful try
1:12:06
to be truthful and don't answer
1:12:08
questions that we don't want um kind of
1:12:10
the system to handle uh later in chat
1:12:13
gbt and so roughly speaking the company
1:12:17
comes up with the labeling instructions
1:12:18
usually they are not this short usually
1:12:20
there are hundreds of pages and people
1:12:22
have to study them professionally and
1:12:24
then they write out the ideal assistant
1:12:26
responses uh following those labeling
1:12:28
instructions so this is a very human
1:12:31
heavy process as it was described in
1:12:33
this paper now the data set for instruct
1:12:35
GPT was never actually released by openi
1:12:37
but we do have some open- Source um
1:12:40
reproductions that were're trying to
1:12:41
follow this kind of a setup and collect
1:12:43
their own data so one that I'm familiar
1:12:45
with for example is the effort of open
1:12:48
Assistant from a while back and this is
1:12:50
just one of I think many examples but I
1:12:52
just want to show you an example so
1:12:55
here's so these were people on the
1:12:56
internet that were asked to basically
1:12:58
create these conversations similar to
1:13:00
what um open I did with human labelers
1:13:03
and so here's an entry of a person who
1:13:05
came up with this BR can you write a
1:13:07
short introduction to the relevance of
1:13:09
the term
1:13:10
manop uh in economics please use
1:13:13
examples Etc and then the same person or
1:13:15
potentially a different person will
1:13:17
write up the response so here's the
1:13:19
assistant response to this and so then
1:13:22
the same person or different person will
1:13:24
actually write out this ideal
1:13:27
response and then this is an example of
1:13:29
maybe how the conversation could
1:13:31
continue now explain it to a dog and
1:13:33
then you can try to come up with a
1:13:35
slightly a simpler explanation or
1:13:37
something like that now this then
1:13:40
becomes the label and we end up training
1:13:42
on this so what happens during training
1:13:45
is that um of course we're not going to
1:13:48
have a full coverage of all the possible
1:13:51
questions that um the model will
1:13:54
encounter at test time during inference
1:13:56
we can't possibly cover all the possible
1:13:58
prompts that people are going to be
1:13:59
asking in the future but if we have a
1:14:02
like a data set of a few of these
1:14:04
examples then the model during training
1:14:07
will start to take on this Persona of
1:14:09
this helpful truthful harmless assistant
1:14:12
and it's all programmed by example and
1:14:15
so these are all examples of behavior
1:14:17
and if you have conversations of these
1:14:18
example behaviors and you have enough of
1:14:20
them like 100,00 and you train on it the
1:14:22
model sort of starts to understand the
1:14:24
statistical pattern and it kind of takes
1:14:26
on this personality of this
1:14:28
assistant now it's possible that when
1:14:30
you get the exact same question like
1:14:32
this at test time it's possible that the
1:14:36
answer will be recited as exactly what
1:14:39
was in the training set but more likely
1:14:41
than that is that the model will kind of
1:14:43
like do something of a similar Vibe um
1:14:46
and we will understand that this is the
1:14:47
kind of answer that you want um so
1:14:51
that's what we're doing we're
1:14:52
programming the system um by example and
1:14:56
the system adopts statistically this
1:14:58
Persona of this helpful truthful
1:15:01
harmless assistant which is kind of like
1:15:03
reflected in the labeling instructions
1:15:04
that the company creates now I want to
1:15:06
show you that the state-of-the-art has
1:15:08
kind of advanced in the last 2 or 3
1:15:10
years uh since the instr GPT paper so in
1:15:13
particular it's not very common for
1:15:15
humans to be doing all the heavy lifting
1:15:16
just by themselves anymore and that's
1:15:18
because we now have language models and
1:15:20
these language models are helping us
1:15:21
create these data sets and conversations
1:15:23
so it is very rare that the people will
1:15:25
like literally just write out the
1:15:26
response from scratch it is a lot more
1:15:29
likely that they will use an existing
1:15:30
llm to basically like uh come up with an
1:15:32
answer and then they will edit it or
1:15:34
things like that so there's many
1:15:36
different ways in which now llms have
1:15:38
started to kind of permeate this
1:15:40
posttraining Set uh stack and llms are
1:15:43
basically used pervasively to help
1:15:45
create these massive data sets of
1:15:47
conversations so I don't want to show
1:15:49
like Ultra chat is one um such example
1:15:52
of like a more modern data set of
1:15:54
conversations it is to a very large
1:15:56
extent synthetic but uh I believe
1:15:58
there's some human involvement I could
1:16:00
be wrong with that usually there will be
1:16:01
a little bit of human but there will be
1:16:03
a huge amount of synthetic help um and
1:16:06
this is all kind of like uh constructed
1:16:09
in different ways and Ultra chat is just
1:16:10
one example of many sft data sets that
1:16:12
currently exist and the only thing I
1:16:14
want to show you is that uh these data
1:16:16
sets have now millions of conversations
1:16:18
uh these conversations are mostly
1:16:20
synthetic but they're probably edited to
1:16:21
some extent by humans and they span a
1:16:24
huge diversity of sort of
1:16:27
um uh areas and so on so these are
1:16:32
fairly extensive artifacts by now and
1:16:34
there's all these like sft mixtures as
1:16:36
they're called so you have a mixture of
1:16:38
like lots of different types and sources
1:16:39
and it's partially synthetic partially
1:16:41
human and it's kind of like um gone in
1:16:44
that direction since uh but roughly
1:16:47
speaking we still have sft data sets
1:16:49
they're made up of conversations we're
1:16:51
training on them um just like we did
1:16:53
before and
1:16:55
uh I guess like the last thing to note
1:16:57
is that I want to dispel a little bit of
1:17:00
the magic of talking to an AI like when
1:17:02
you go to chat GPT and you give it a
1:17:04
question and then you hit enter uh what
1:17:08
is coming back is kind of like
1:17:10
statistically aligned with what's
1:17:12
happening in the training set and these
1:17:14
training sets I mean they really just
1:17:17
have a seed in humans following labeling
1:17:19
instructions so what are you actually
1:17:22
talking to in chat GPT or how should you
1:17:24
think about it well it's not coming from
1:17:26
some magical AI like roughly speaking
1:17:28
it's coming from something that is
1:17:30
statistically imitating human labelers
1:17:32
which comes from labeling instructions
1:17:34
written by these companies and so you're
1:17:36
kind of imitating this uh you're kind of
1:17:39
getting um it's almost as if you're
1:17:40
asking human labeler and imagine that
1:17:43
the answer that is given to you uh from
1:17:45
chbt is some kind of a simulation of a
1:17:48
human labeler uh and it's kind of like
1:17:51
asking what would a human labeler say in
1:17:53
this kind of a conversation
1:17:57
and uh it's not just like this human
1:17:59
labeler is not just like a random person
1:18:00
from the internet because these
1:18:02
companies actually hire experts so for
1:18:03
example when you are asking questions
1:18:05
about code and so on the human labelers
1:18:07
that would be in um involved in creation
1:18:09
of these conversation data sets they
1:18:11
will usually be usually be educated
1:18:13
expert people and you're kind of like
1:18:15
asking a question of like a simulation
1:18:17
of those people if that makes sense so
1:18:20
you're not talking to a magical AI
1:18:21
you're talking to an average labeler
1:18:23
this average labeler is probably fairly
1:18:24
highly skilled
1:18:25
but you're talking to kind of like an
1:18:27
instantaneous simulation of that kind of
1:18:29
a person that would be hired uh in the
1:18:32
construction of these data sets so let
1:18:35
me give you one more specific example
1:18:36
before we move on for example when I go
1:18:39
to chpt and I say recommend the top five
1:18:41
landmarks who see in Paris and then I
1:18:43
hit
1:18:45
enter
1:18:49
uh okay here we go okay when I hit enter
1:18:52
what's coming out here how do I think
1:18:55
about it well it's not some kind of a
1:18:57
magical AI that has gone out and
1:18:59
researched all the landmarks and then
1:19:00
ranked them using its infinite
1:19:02
intelligence Etc what I'm getting is a
1:19:04
statistical simulation of a labeler that
1:19:07
was hired by open AI you can think about
1:19:09
it roughly in that way and so if this
1:19:13
specific um question is in the
1:19:16
posttraining data set somewhere at open
1:19:18
aai then I'm very likely to see an
1:19:21
answer that is probably very very
1:19:22
similar to what that human labeler would
1:19:24
have put down
1:19:25
for those five landmarks how does the
1:19:27
human labeler come up with this well
1:19:29
they go off and they go on the internet
1:19:30
and they kind of do their own little
1:19:31
research for 20 minutes and they just
1:19:33
come up with a list right now so if they
1:19:35
come up with this list and this is in
1:19:37
the data set I'm probably very likely to
1:19:39
see what they submitted as the correct
1:19:42
answer from the assistant now if this
1:19:45
specific query is not part of the post
1:19:46
training data set then what I'm getting
1:19:48
here is a little bit more emergent uh
1:19:51
because uh the model kind of understands
1:19:54
the statistically
1:19:55
um the kinds of landmarks that are in
1:19:58
this training set are usually the
1:19:59
prominent landmarks the landmarks that
1:20:01
people usually want to see the kinds of
1:20:03
landmarks that are usually uh very often
1:20:05
talked about on the internet and
1:20:07
remember that the model already has a
1:20:09
ton of Knowledge from its pre-training
1:20:10
on the internet so it's probably seen a
1:20:12
ton of conversations about Paris about
1:20:14
landmarks about the kinds of things that
1:20:15
people like to see and so it's the
1:20:17
pre-training knowledge that has then
1:20:18
combined with the postering data set
1:20:21
that results in this kind of an
1:20:24
imitation um
1:20:25
so that's uh that's roughly how you can
1:20:28
kind of think about what's happening
1:20:30
behind the scenes here in in this
1:20:32
statistical sense okay now I want to
1:20:34
turn to the topic of llm psychology as I
1:20:36
like to call it which is what are sort
1:20:38
of the emergent cognitive effects of the
1:20:41
training pipeline that we have for these
1:20:42
models so in particular the first one I
1:20:44
want to talk to is of course
1:20:47
hallucinations so you might be familiar
1:20:50
with model hallucinations it's when llms
1:20:52
make stuff up they just totally
1:20:54
fabricate information Etc and it's a big
1:20:56
problem with llm assistants it is a
1:20:58
problem that existed to a large extent
1:21:00
with early models uh from many years ago
1:21:03
and I think the problem has gotten a bit
1:21:04
better uh because there are some
1:21:06
medications that I'm going to go into in
1:21:07
a second for now let's just try to
1:21:09
understand where these hallucinations
1:21:10
come from so here's a specific example
1:21:13
of a few uh of three conversations that
1:21:16
you might think you have in your
1:21:17
training set and um these are pretty
1:21:20
reasonable conversations that you could
1:21:22
imagine being in the training set so
1:21:24
like for example who is Cruz well Tom
1:21:26
Cruz is an famous actor American actor
1:21:28
and producer Etc who is John baraso this
1:21:31
turns out to be a us senetor for example
1:21:34
who is genis Khan well genis Khan was
1:21:37
blah blah blah and so this is what your
1:21:40
conversations could look like at
1:21:41
training time now the problem with this
1:21:43
is that when the human is writing the
1:21:46
correct answer for the assistant in each
1:21:49
one of these cases uh the human either
1:21:51
like knows who this person is or they
1:21:52
research them on the Internet and they
1:21:54
come in and they write this response
1:21:56
that kind of has this like confident
1:21:57
tone of an answer and what happens
1:22:00
basically is that at test time when you
1:22:01
ask for someone who is this is a totally
1:22:04
random name that I totally came up with
1:22:05
and I don't think this person exists um
1:22:08
as far as I know I just Tred to generate
1:22:10
it randomly the problem is when we ask
1:22:12
who is Orson kovats the problem is that
1:22:15
the assistant will not just tell you oh
1:22:18
I don't know even if the assistant and
1:22:20
the language model itself might know
1:22:23
inside its features inside its
1:22:25
activations inside of its brain sort of
1:22:27
it might know that this person is like
1:22:28
not someone that um that is that it's
1:22:31
familiar with even if some part of the
1:22:33
network kind of knows that in some sense
1:22:35
the uh saying that oh I don't know who
1:22:37
this is is is not going to happen
1:22:40
because the model statistically imitates
1:22:43
is training set in the training set the
1:22:46
questions of the form who is blah are
1:22:48
confidently answered with the correct
1:22:50
answer and so it's going to take on the
1:22:52
style of the answer and it's going to do
1:22:54
its best it's going to give you
1:22:55
statistically the most likely guess and
1:22:58
it's just going to basically make stuff
1:22:59
up because these models again we just
1:23:01
talked about it is they don't have
1:23:03
access to the internet they're not doing
1:23:04
research these are statistical token
1:23:06
tumblers as I call them uh is just
1:23:09
trying to sample the next token in the
1:23:10
sequence and it's going to basically
1:23:12
make stuff up so let's take a look at
1:23:14
what this looks
1:23:15
like I have here what's called the
1:23:18
inference playground from hugging face
1:23:21
and I am on purpose picking on a model
1:23:23
called Falcon 7B which is an old model
1:23:25
this is a few years ago now so it's an
1:23:28
older model So It suffers from
1:23:29
hallucinations and as I mentioned this
1:23:31
has improved over time recently but
1:23:33
let's say who is Orson kovats let's ask
1:23:35
Falcon 7B instruct
1:23:38
run oh yeah Orson kovat is an American
1:23:40
author and science uh fiction writer
1:23:43
okay this is totally false it's
1:23:45
hallucination let's try again these are
1:23:47
statistical systems right so we can
1:23:49
resample this time Orson kovat is a
1:23:51
fictional character from this 1950s TV
1:23:54
show it's total BS right let's try again
1:23:58
he's a former minor league baseball
1:24:00
player okay so basically the model
1:24:03
doesn't know and it's given us lots of
1:24:04
different answers because it doesn't
1:24:07
know it's just kind of like sampling
1:24:08
from these probabilities the model
1:24:10
starts with the tokens who is oron
1:24:12
kovats assistant and then it comes in
1:24:15
here and it's get it's getting these
1:24:18
probabilities and it's just sampling
1:24:19
from the probabilities and it just like
1:24:21
comes up with stuff and the stuff is
1:24:24
actually
1:24:25
statistically consistent with the style
1:24:28
of the answer in its training set and
1:24:30
it's just doing that but you and I
1:24:32
experiened it as a madeup factual
1:24:34
knowledge but keep in mind that uh the
1:24:36
model basically doesn't know and it's
1:24:38
just imitating the format of the answer
1:24:40
and it's not going to go off and look it
1:24:41
up uh because it's just imitating again
1:24:44
the answer so how can we uh mitigate
1:24:47
this because for example when we go to
1:24:48
chat apt and I say who is oron kovats
1:24:51
and I'm now asking the stateoftheart
1:24:53
state-of-the-art model from open AI
1:24:55
this model will tell
1:24:57
you oh so this model is actually is even
1:25:01
smarter because you saw very briefly it
1:25:03
said searching the web uh we're going to
1:25:05
cover this later um it's actually trying
1:25:07
to do tool use and
1:25:11
uh kind of just like came up with some
1:25:13
kind of a story but I want to just who
1:25:16
or Kovach did not use any tools I don't
1:25:19
want it to do web
1:25:22
search there's a wellknown historical or
1:25:25
public figure named or oron kovats so
1:25:27
this model is not going to make up stuff
1:25:29
this model knows that it doesn't know
1:25:31
and it tells you that it doesn't appear
1:25:33
to be a person that this model knows so
1:25:36
somehow we sort of improved
1:25:37
hallucinations even though they clearly
1:25:39
are an issue in older models and it
1:25:42
makes totally uh sense why you would be
1:25:44
getting these kinds of answers if this
1:25:46
is what your training set looks like so
1:25:48
how do we fix this okay well clearly we
1:25:51
need some examples in our data set that
1:25:53
where the correct answer for the
1:25:55
assistant is that the model doesn't know
1:25:57
about some particular fact but we only
1:26:00
need to have those answers be produced
1:26:02
in the cases where the model actually
1:26:04
doesn't know and so the question is how
1:26:05
do we know what the model knows or
1:26:07
doesn't know well we can empirically
1:26:09
probe the model to figure that out so
1:26:11
let's take a look at for example how
1:26:13
meta uh dealt with hallucinations for
1:26:16
the Llama 3 series of models as an
1:26:18
example so in this paper that they
1:26:20
published from meta we can go into
1:26:22
hallucinations
1:26:26
which they call here factuality and they
1:26:28
describe the procedure by which they
1:26:30
basically interrogate the model to
1:26:32
figure out what it knows and doesn't
1:26:34
know to figure out sort of like the
1:26:35
boundary of its knowledge and then they
1:26:38
add examples to the training set where
1:26:42
for the things where the model doesn't
1:26:44
know them the correct answer is that the
1:26:47
model doesn't know them which sounds
1:26:49
like a very easy thing to do in
1:26:51
principle but this roughly fixes the
1:26:53
issue and the the reason it fixes the
1:26:55
issue is
1:26:56
because remember like the model might
1:26:59
actually have a pretty good model of its
1:27:02
self knowledge inside the network so
1:27:04
remember we looked at the network and
1:27:06
all these neurons inside the network you
1:27:09
might imagine that there's a neuron
1:27:10
somewhere in the network that sort of
1:27:12
like lights up for when the model is
1:27:14
uncertain but the problem is that the
1:27:17
activation of that neuron is not
1:27:19
currently wired up to the model actually
1:27:21
saying in words that it doesn't know so
1:27:23
even though the internal of the neural
1:27:25
network no because there's some neurons
1:27:27
that represent that the model uh will
1:27:30
not surface that it will instead take
1:27:32
its best guess so that it sounds
1:27:33
confident um just like it sees in a
1:27:36
training set so we need to basically
1:27:37
interrogate the model and allow it to
1:27:40
say I don't know in the cases that it
1:27:41
doesn't know so let me take you through
1:27:43
what meta roughly does so basically what
1:27:45
they do is here I have an example uh
1:27:48
Dominic kek is uh the featured article
1:27:52
today so I just went there randomly and
1:27:54
what they do is basically they take a
1:27:56
random document in a training set and
1:27:58
they take a paragraph and then they use
1:28:01
an llm to construct questions about that
1:28:04
paragraph so for example I did that with
1:28:07
chat GPT
1:28:09
here so I said here's a paragraph from
1:28:12
this document generate three specific
1:28:15
factual questions based on this
1:28:16
paragraph and give me the questions and
1:28:18
the answers and so the llms are already
1:28:20
good enough to create and reframe this
1:28:23
information so if the information is in
1:28:26
the context window um of this llm this
1:28:30
actually works pretty well it doesn't
1:28:31
have to rely on its memory it's right
1:28:33
there in the context window and so it
1:28:36
can basically reframe that information
1:28:38
with fairly high accuracy so for example
1:28:40
can generate questions for us like for
1:28:42
which team did he play here's the answer
1:28:45
how many cups did he win Etc and now
1:28:47
what we have to do is we have some
1:28:48
question and answers and now we want to
1:28:50
interrogate the model so roughly
1:28:52
speaking what we'll do is we'll take our
1:28:53
questions and we'll go to our model
1:28:56
which would be uh say llama uh in meta
1:28:59
but let's just interrogate mol 7B here
1:29:01
as an example that's another model so
1:29:04
does this model know about this answer
1:29:07
let's take a
1:29:09
look uh so he played for Buffalo Sabers
1:29:12
right so the model knows and the the way
1:29:15
that you can programmatically decide is
1:29:17
basically we're going to take this
1:29:19
answer from the model and we're going to
1:29:21
compare it to the correct answer and
1:29:24
again the model model are good enough to
1:29:25
do this automatically so there's no
1:29:26
humans involved here we can take uh
1:29:29
basically the answer from the model and
1:29:30
we can use another llm judge to check if
1:29:34
that is correct according to this answer
1:29:36
and if it is correct that means that the
1:29:37
model probably knows so what we're going
1:29:39
to do is we're going to do this maybe a
1:29:41
few times so okay it knows it's Buffalo
1:29:43
Savers let's drag
1:29:45
in um Buffalo Sabers let's try one more
1:29:51
time Buffalo Sabers so we asked three
1:29:54
times about this factual question and
1:29:56
the model seems to know so everything is
1:29:58
great now let's try the second question
1:30:01
how many Stanley Cups did he
1:30:03
win and again let's interrogate the
1:30:05
model about that and the correct answer
1:30:06
is
1:30:08
two so um here the model claims that he
1:30:14
won um four times which is not correct
1:30:18
right it doesn't match two so the model
1:30:20
doesn't know it's making stuff up let's
1:30:22
try again
1:30:28
um so here the model again it's kind of
1:30:30
like making stuff up right let's
1:30:34
Dragon here it says did he did not even
1:30:38
did not win during his career so
1:30:40
obviously the model doesn't know and the
1:30:41
way we can programmatically tell again
1:30:43
is we interrogate the model three times
1:30:45
and we compare its answers maybe three
1:30:47
times five times whatever it is to the
1:30:49
correct answer and if the model doesn't
1:30:52
know then we know that the model doesn't
1:30:53
know this question
1:30:54
and then what we do is we take this
1:30:57
question we create a new conversation in
1:31:00
the training set so we're going to add a
1:31:02
new conversation training set and when
1:31:04
the question is how many Stanley Cups
1:31:05
did he win the answer is I'm sorry I
1:31:08
don't know or I don't remember and
1:31:11
that's the correct answer for this
1:31:12
question because we interrogated the
1:31:14
model and we saw that that's the case if
1:31:16
you do this for many different types of
1:31:18
uh questions for many different types of
1:31:21
documents you are giving the model an
1:31:23
opportunity to in its training set
1:31:26
refuse to say based on its knowledge and
1:31:28
if you just have a few examples of that
1:31:30
in your training set the model will know
1:31:33
um and and has the opportunity to learn
1:31:35
the association of this knowledge-based
1:31:38
refusal to this internal neuron
1:31:41
somewhere in its Network that we presume
1:31:43
exists and empirically this turns out to
1:31:45
be probably the case and it can learn
1:31:47
that Association that hey when this
1:31:49
neuron of uncertainty is high then I
1:31:52
actually don't know and I'm allowed to
1:31:54
say that I'm sorry but I don't think I
1:31:56
remember this Etc and if you have these
1:32:00
uh examples in your training set then
1:32:02
this is a large mitigation for
1:32:04
hallucination and that's roughly
1:32:06
speaking why chpt is able to do stuff
1:32:08
like this as well so these are kinds of
1:32:11
uh mitigations that people have
1:32:12
implemented and that have improved the
1:32:14
factuality issue over time okay so I've
1:32:17
described mitigation number one for
1:32:20
basically mitigating the hallucinations
1:32:22
issue now we can actually do much better
1:32:24
than that uh it's instead of just saying
1:32:27
that we don't know uh we can introduce
1:32:30
an additional mitigation number two to
1:32:32
give the llm an opportunity to be
1:32:34
factual and actually answer the question
1:32:36
now what do you and I do if I was to ask
1:32:39
you a factual question and you don't
1:32:41
know uh what would you do um in order to
1:32:44
answer the question well you could uh go
1:32:46
off and do some search and uh use the
1:32:48
internet and you could figure out the
1:32:50
answer and then tell me what that answer
1:32:52
is and we can do the exact exact same
1:32:54
thing with these models so think of the
1:32:57
knowledge inside the neural network
1:32:58
inside its billions of parameters think
1:33:01
of that as kind of a vague recollection
1:33:03
of the things that the model has seen
1:33:06
during its training during the
1:33:07
pre-training stage a long time ago so
1:33:10
think of that knowledge in the
1:33:11
parameters as something you read a month
1:33:13
ago and if you keep reading something
1:33:16
then you will remember it and the model
1:33:17
remembers that but if it's something
1:33:19
rare then you probably don't have a
1:33:20
really good recollection of that
1:33:21
information but what you and I do is we
1:33:23
just go and look it up now when you go
1:33:26
and look it up what you're doing
1:33:27
basically is like you're refreshing your
1:33:28
working memory with information and then
1:33:31
you're able to sort of like retrieve it
1:33:32
talk about it or Etc so we need some
1:33:34
equivalent of allowing the model to
1:33:36
refresh its memory or its recollection
1:33:39
and we can do that by introducing tools
1:33:41
uh for the
1:33:43
models so the way we are going to
1:33:45
approach this is that instead of just
1:33:46
saying hey I'm sorry I don't know we can
1:33:48
attempt to use tools so we can create uh
1:33:53
a mechanism
1:33:54
by which the language model can emit
1:33:56
special tokens and these are tokens that
1:33:58
we're going to introduce new tokens so
1:34:01
for example here I've introduced two
1:34:02
tokens and I've introduced a format or a
1:34:05
protocol for how the model is allowed to
1:34:07
use these tokens so for example instead
1:34:10
of answering the question when the model
1:34:12
does not instead of just saying I don't
1:34:14
know sorry the model has the option now
1:34:16
to emitting the special token search
1:34:18
start and this is the query that will go
1:34:21
to like bing.com in the case of openai
1:34:23
or say Google search or something like
1:34:24
that so it will emit the query and then
1:34:27
it will emit search end and then here
1:34:31
what will happen is that the program
1:34:33
that is sampling from the model that is
1:34:35
running the inference when it sees the
1:34:37
special token search end instead of
1:34:40
sampling the next token uh in the
1:34:42
sequence it will actually pause
1:34:44
generating from the model it will go off
1:34:46
it will open a session with bing.com and
1:34:49
it will paste the search query into Bing
1:34:52
and it will then um get all the text
1:34:54
that is retrieved and it will basically
1:34:57
take that text it will maybe represent
1:34:59
it again with some other special tokens
1:35:00
or something like that and it will take
1:35:02
that text and it will copy paste it here
1:35:05
into what I Tred to like show with the
1:35:07
brackets so all that text kind of comes
1:35:09
here and when the text comes here it
1:35:13
enters the context window so the model
1:35:15
so that text from the web search is now
1:35:17
inside the context window that will feed
1:35:20
into the neural network and you should
1:35:22
think of the context window as kind of
1:35:23
like the working memory of the model
1:35:25
that data that is in the context window
1:35:27
is directly accessible by the model it
1:35:30
directly feeds into the neural network
1:35:32
so it's not anymore a vague recollection
1:35:34
it's data that it it has in the context
1:35:37
window and is directly available to that
1:35:38
model so now when it's sampling the new
1:35:41
uh tokens here afterwards it can
1:35:44
reference very easily the data that has
1:35:46
been copy pasted in there so that's
1:35:49
roughly how these um how these tools use
1:35:52
uh tools uh function
1:35:54
and so web search is just one of the
1:35:56
tools we're going to look at some of the
1:35:57
other tools in a bit uh but basically
1:35:59
you introduce new tokens you introduce
1:36:01
some schema by which the model can
1:36:02
utilize these tokens and can call these
1:36:05
special functions like web search
1:36:06
functions and how do you teach the model
1:36:09
how to correctly use these tools like
1:36:11
say web search search start search end
1:36:13
Etc well again you do that through
1:36:14
training sets so we need now to have a
1:36:17
bunch of data and a bunch of
1:36:19
conversations that show the model by
1:36:21
example how to use web search so what
1:36:24
are the what are the settings where you
1:36:26
are using the search um and what does
1:36:28
that look like and here's by example how
1:36:30
you start a search and the search Etc
1:36:33
and uh if you have a few thousand maybe
1:36:35
examples of that in your training set
1:36:37
the model will actually do a pretty good
1:36:39
job of understanding uh how this tool
1:36:41
works and it will know how to sort of
1:36:43
structure its queries and of course
1:36:45
because of the pre-training data set and
1:36:47
its understanding of the world it
1:36:49
actually kind of understands what a web
1:36:50
search is and so it actually kind of has
1:36:52
a pretty good native understanding
1:36:54
um of what kind of stuff is a good
1:36:56
search query um and so it all kind of
1:36:58
just like works you just need a little
1:37:00
bit of a few examples to show it how to
1:37:03
use this new tool and then it can lean
1:37:05
on it to retrieve information and uh put
1:37:07
it in the context window and that's
1:37:09
equivalent to you and I looking
1:37:10
something up because once it's in the
1:37:12
context it's in the working memory and
1:37:14
it's very easy to manipulate and access
1:37:16
so that's what we saw a few minutes ago
1:37:18
when I was searching on chat GPT for who
1:37:21
is Orson kovats the chat GPT language
1:37:23
model decided Ed that this is some kind
1:37:25
of a rare um individual or something
1:37:28
like that and instead of giving me an
1:37:30
answer from its memory it decided that
1:37:32
it will sample a special token that is
1:37:33
going to do web search and we saw
1:37:35
briefly something flash it was like
1:37:37
using the web tool or something like
1:37:38
that so it briefly said that and then we
1:37:40
waited for like two seconds and then it
1:37:42
generated this and you see how it's
1:37:44
creating references here and so it's
1:37:46
citing sources so what happened here is
1:37:50
it went off it did a web web search it
1:37:52
found these sources and these URLs and
1:37:55
the text of these web pages was all
1:37:59
stuffed in between here and it's not
1:38:01
showing here but it's it's basically
1:38:03
stuffed as text in between here and now
1:38:07
it sees that text and now it kind of
1:38:09
references it and says that okay it
1:38:11
could be these people citation could be
1:38:13
those people citation Etc so that's what
1:38:16
happened here and that's what and that's
1:38:17
why when I said who is Orson kovats I
1:38:19
could also say don't use any tools and
1:38:22
then that's enough to um
1:38:24
basically convince chat PT to not use
1:38:26
tools and just use its memory and its
1:38:28
recollection I also went off and I um
1:38:33
tried to ask this question of Chachi PT
1:38:35
so how many standing cups did uh Dominic
1:38:37
Hasek win and Chachi P actually decided
1:38:40
that it knows the answer and it has the
1:38:41
confidence to say that uh he want twice
1:38:44
and so it kind of just relied on its
1:38:45
memory because presumably it has um it
1:38:49
has enough of
1:38:50
a kind of confidence in its weights in
1:38:53
it parameters and activations that this
1:38:55
is uh retrievable just for memory um but
1:38:59
you can also
1:39:01
conversely use web search to make sure
1:39:04
and then for the same query it actually
1:39:06
goes off and it searches and then it
1:39:08
finds a bunch of sources it finds all
1:39:10
this all of this stuff gets copy pasted
1:39:12
in there and then it tells us uh to
1:39:15
again and sites and it actually says the
1:39:18
Wikipedia article which is the source of
1:39:20
this information for us as well so
1:39:23
that's tools web search the model
1:39:25
determines when to search and then uh
1:39:28
that's kind of like how these tools uh
1:39:30
work and this is an additional kind of
1:39:32
mitigation for uh hallucinations and
1:39:35
factuality so I want to stress one more
1:39:37
time this very important sort of
1:39:39
psychology
1:39:40
Point knowledge in the parameters of the
1:39:43
neural network is a vague recollection
1:39:46
the knowledge in the tokens that make up
1:39:47
the context
1:39:48
window is the working memory and it
1:39:51
roughly speaking Works kind of like um
1:39:54
it works for us in our brain the stuff
1:39:56
we remember is our parameters uh and the
1:39:59
stuff that we just experienced like a
1:40:01
few seconds or minutes ago and so on you
1:40:03
can imagine that being in our context
1:40:04
window and this context window is being
1:40:06
built up as you have a conscious
1:40:07
experience around you so this has a
1:40:10
bunch of um implications also for your
1:40:12
use of LOLs in practice so for example I
1:40:16
can go to chat GPT and I can do
1:40:17
something like this I can say can you
1:40:19
Summarize chapter one of Jane Austin's
1:40:20
Pride and Prejudice right and this is a
1:40:23
perfectly fine prompt and Chach actually
1:40:25
does something relatively reasonable
1:40:27
here and but the reason it does that is
1:40:29
because Chach has a pretty good
1:40:30
recollection of a famous work like Pride
1:40:33
and Prejudice it's probably seen a ton
1:40:34
of stuff about it there's probably
1:40:36
forums about this book it's probably
1:40:37
read versions of this book um and it's
1:40:40
kind of like remembers because even if
1:40:43
you've read this or articles about it
1:40:47
you'd kind of have a recollection enough
1:40:48
to actually say all this but usually
1:40:50
when I actually interact with LMS and I
1:40:51
want them to recall specific things it
1:40:54
always works better if you just give it
1:40:55
to them so I think a much better prompt
1:40:57
would be something like this can you
1:40:59
summarize for me chapter one of genos's
1:41:01
spr and Prejudice and then I am
1:41:03
attaching it below for your reference
1:41:05
and then I do something like a delimeter
1:41:06
here and I paste it in and I I found
1:41:09
that just copy pasting it from some
1:41:11
website that I found here um so copy
1:41:14
pasting the chapter one here and I do
1:41:16
that because when it's in the context
1:41:18
window the model has direct access to it
1:41:20
and can exactly it doesn't have to
1:41:22
recall it it just has access to it and
1:41:25
so this summary is can be expected to be
1:41:27
a significantly high quality or higher
1:41:29
quality than this summary uh just
1:41:31
because it's directly available to the
1:41:33
model and I think you and I would work
1:41:34
in the same way if you want to it would
1:41:37
be you would produce a much better
1:41:38
summary if you had reread this chapter
1:41:41
before you had to summarize it and
1:41:43
that's basically what's happening here
1:41:45
or the equivalent of it the next sort of
1:41:47
psychological Quirk I'd like to talk
1:41:49
about briefly is that of the knowledge
1:41:50
of self so what I see very often on the
1:41:53
internet is that people do something
1:41:54
like this they ask llms something like
1:41:57
what model are you and who built you and
1:42:00
um basically this uh question is a
1:42:01
little bit nonsensical and the reason I
1:42:04
say that is that as I try to kind of
1:42:06
explain with some of the underhood
1:42:07
fundamentals this thing is not a person
1:42:09
right it doesn't have a persistent
1:42:11
existence in any way it sort of boots up
1:42:14
processes tokens and shuts off and it
1:42:17
does that for every single person it
1:42:18
just kind of builds up a context window
1:42:20
of conversation and then everything gets
1:42:21
deleted and so this this entity is kind
1:42:24
of like restarted from scratch every
1:42:25
single conversation if that makes sense
1:42:27
it has no persistent self it has no
1:42:29
sense of self it's a token tumbler and
1:42:32
uh it follows the statistical
1:42:33
regularities of its training set so it
1:42:36
doesn't really make sense to ask it who
1:42:38
are you what build you Etc and by
1:42:40
default if you do what I described and
1:42:43
just by default and from nowhere you're
1:42:45
going to get some pretty random answers
1:42:46
so for example let's uh pick on Falcon
1:42:48
which is a fairly old model and let's
1:42:51
see what it tells
1:42:52
us uh so it's evading the question uh
1:42:56
talented engineers and developers here
1:42:58
it says I was built by open AI based on
1:43:00
the gpt3 model it's totally making stuff
1:43:02
up now the fact that it's built by open
1:43:04
AI here I think a lot of people would
1:43:06
take this as evidence that this model
1:43:08
was somehow trained on open AI data or
1:43:10
something like that I don't actually
1:43:11
think that that's necessarily true the
1:43:13
reason for that is
1:43:14
that if you don't explicitly program the
1:43:18
model to answer these kinds of questions
1:43:20
then what you're going to get is its
1:43:22
statistical best guess at the answer and
1:43:26
this model had a um sft data mixture of
1:43:29
conversations and during the
1:43:32
fine-tuning um the model sort of
1:43:36
understands as it's training on this
1:43:37
data that it's taking on this
1:43:38
personality of this like helpful
1:43:40
assistant and it doesn't know how to it
1:43:42
doesn't actually it wasn't told exactly
1:43:44
what label to apply to self it just kind
1:43:47
of is taking on this uh this uh Persona
1:43:50
of a helpful assistant and remember that
1:43:53
the pre-training stage took the
1:43:55
documents from the entire internet and
1:43:57
Chach and open AI are very prominent in
1:44:00
these documents and so I think what's
1:44:01
actually likely to be happening here is
1:44:04
that this is just its hallucinated label
1:44:06
for what it is this is its self-identity
1:44:08
is that it's chat GPT by open Ai and
1:44:11
it's only saying that because there's a
1:44:13
ton of data on the internet of um
1:44:15
answers like this that are actually
1:44:18
coming from open from chasht and So
1:44:20
that's its label for what it is now you
1:44:24
can override this as a developer if you
1:44:26
have a llm model you can actually
1:44:28
override it and there are a few ways to
1:44:29
do that so for example let me show you
1:44:32
there's this MMO model from Allen Ai and
1:44:35
um this is one llm it's not a top tier
1:44:38
LM or anything like that but I like it
1:44:39
because it is fully open source so the
1:44:41
paper for Almo and everything else is
1:44:43
completely fully open source which is
1:44:44
nice um so here we are looking at its
1:44:47
sft mixture so this is the data mixture
1:44:50
of um the fine tuning so this is the
1:44:52
conversations data it right and so the
1:44:55
way that they are solving it for Theo
1:44:56
model is we see that there's a bunch of
1:44:58
stuff in the mixture and there's a total
1:45:00
of 1 million conversations here but here
1:45:02
we have alot to hardcoded if we go there
1:45:05
we see that this is 240
1:45:08
conversations and look at these 240
1:45:11
conversations they're hardcoded tell me
1:45:13
about yourself says user and then the
1:45:16
assistant says I'm and open language
1:45:18
model developed by AI to Allen Institute
1:45:20
of artificial intelligence Etc I'm here
1:45:22
to help blah blah blah what is your name
1:45:24
uh Theo project so these are all kinds
1:45:26
of like cooked up hardcoded questions
1:45:28
abouto 2 and the correct answers to give
1:45:31
in these cases if you take 240 questions
1:45:34
like this or conversations put them into
1:45:36
your training set and fine tune with it
1:45:38
then the model will actually be expected
1:45:39
to parot this stuff later if you don't
1:45:43
give it this then it's probably a Chach
1:45:46
by open
1:45:47
Ai and um there's one more way to
1:45:50
sometimes do this is
1:45:51
that basically um in these conversations
1:45:55
and you have terms between human and
1:45:56
assistant sometimes there's a special
1:45:58
message called system message at the
1:46:00
very beginning of the conversation so
1:46:03
it's not just between human and
1:46:04
assistant there's a system and in the
1:46:06
system message you can actually hardcode
1:46:08
and remind the model that hey you are a
1:46:11
model developed by open Ai and your name
1:46:14
is chashi pt40 and you were trained on
1:46:17
this date and your knowledge cut off is
1:46:18
this and basically it kind of like
1:46:20
documents the model a little bit and
1:46:22
then this is inserted into to your
1:46:23
conversations so when you go on chpt you
1:46:25
see a blank page but actually the system
1:46:27
message is kind of like hidden in there
1:46:29
and those tokens are in the context
1:46:30
window and so those are the two ways to
1:46:33
kind of um program the models to talk
1:46:36
about themselves either it's done
1:46:38
through uh data like this or it's done
1:46:40
through system message and things like
1:46:42
that basically invisible tokens that are
1:46:44
in the context window and remind the
1:46:46
model of its identity but it's all just
1:46:48
kind of like cooked up and bolted on in
1:46:50
some in some way it's not actually like
1:46:52
really deeply there in any real sense as
1:46:55
it would before a human I want to now
1:46:57
continue to the next section which deals
1:46:59
with the computational capabilities or
1:47:01
like I should say the native
1:47:02
computational capabilities of these
1:47:04
models in problem solving scenarios and
1:47:06
so in particular we have to be very
1:47:08
careful with these models when we
1:47:09
construct our examples of conversations
1:47:11
and there's a lot of sharp edges here
1:47:13
that are kind of like elucidative is
1:47:15
that a word uh they're kind of like
1:47:17
interesting to look at when we consider
1:47:18
how these models think so um consider
1:47:22
the following prompt from a human and
1:47:25
supposed that basically that we are
1:47:26
building out a conversation to enter
1:47:28
into our training set of conversations
1:47:29
so we're going to train the model on
1:47:30
this we're teaching you how to basically
1:47:32
solve simple math problems so the prompt
1:47:35
is Emily buys three apples and two
1:47:37
oranges each orange cost $2 the total
1:47:39
cost is 13 what is the cost of apples
1:47:41
very simple math question now there are
1:47:44
two answers here on the left and on the
1:47:46
right they are both correct answers they
1:47:48
both say that the answer is three which
1:47:50
is correct but one of these two is a
1:47:52
significant ific anly better answer for
1:47:55
the assistant than the other like if I
1:47:56
was Data labeler and I was creating one
1:47:58
of these one of these would be uh a
1:48:01
really terrible answer for the assistant
1:48:04
and the other would be okay and so I'd
1:48:06
like you to potentially pause the video
1:48:07
Even and think through why one of these
1:48:10
two is significantly better answer uh
1:48:12
than the other and um if you use the
1:48:15
wrong one your model will actually be uh
1:48:18
really bad at math potentially and it
1:48:19
would have uh bad outcomes and this is
1:48:21
something that you would be careful with
1:48:22
in your life labeling documentations
1:48:24
when you are training people uh to
1:48:25
create the ideal responses for the
1:48:27
assistant okay so the key to this
1:48:29
question is to realize and remember that
1:48:32
when the models are training and also
1:48:34
inferencing they are working in
1:48:36
onedimensional sequence of tokens from
1:48:38
left to right and this is the picture
1:48:40
that I often have in my mind I imagine
1:48:42
basically the token sequence evolving
1:48:44
from left to right and to always produce
1:48:46
the next token in a sequence we are
1:48:49
feeding all these tokens into the neural
1:48:51
network and this neural network then is
1:48:53
the probabilities for the next token and
1:48:54
sequence right so this picture here is
1:48:56
the exact same picture we saw uh before
1:48:59
up here and this comes from the web demo
1:49:02
that I showed you before right so this
1:49:04
is the calculation that basically takes
1:49:06
the input tokens here on the top and uh
1:49:09
performs these operations of all these
1:49:11
neurons and uh gives you the answer for
1:49:14
the probabilities of what comes next now
1:49:16
the important thing to realize is that
1:49:18
roughly
1:49:19
speaking uh there's basically a finite
1:49:21
number of layers of computation that
1:49:23
happened here so for example this model
1:49:25
here has only one two three layers of
1:49:29
what's called detention and uh MLP here
1:49:32
um maybe um typical modern
1:49:34
state-of-the-art Network would have more
1:49:36
like say 100 layers or something like
1:49:38
that but there's only 100 layers of
1:49:39
computation or something like that to go
1:49:41
from the previous token sequence to the
1:49:43
probabilities for the next token and so
1:49:45
there's a finite amount of computation
1:49:47
that happens here for every single token
1:49:49
and you should think of this as a very
1:49:51
small amount of computation and this
1:49:53
amount of computation is almost roughly
1:49:55
fixed uh for every single token in this
1:49:57
sequence um the that's not actually
1:50:00
fully true because the more tokens you
1:50:02
feed in uh the the more expensive uh
1:50:04
this forward pass will be of this neural
1:50:07
network but not by much so you should
1:50:09
think of this uh and I think as a good
1:50:11
model to have in mind this is a fixed
1:50:13
amount of compute that's going to happen
1:50:14
in this box for every single one of
1:50:16
these tokens and this amount of compute
1:50:18
Cann possibly be too big because there's
1:50:19
not that many layers that are sort of
1:50:21
going from the top to bottom here
1:50:23
there's not that that much
1:50:24
computationally that will happen here
1:50:26
and so you can't imagine the model to to
1:50:28
basically do arbitrary computation in a
1:50:30
single forward pass to get a single
1:50:32
token and so what that means is that we
1:50:34
actually have to distribute our
1:50:36
reasoning and our computation across
1:50:38
many tokens because every single token
1:50:40
is only spending a finite amount of
1:50:42
computation on it and so we kind of want
1:50:45
to distribute the computation across
1:50:48
many tokens and we can't have too much
1:50:51
computation or expect too much
1:50:52
computation out of of the model in any
1:50:54
single individual token because there's
1:50:56
only so much computation that happens
1:50:58
per token okay roughly fixed amount of
1:51:01
computation here
1:51:03
so that's why this answer here is
1:51:06
significantly worse and the reason for
1:51:08
that is Imagine going from left to right
1:51:10
here um and I copy pasted it right here
1:51:14
the answer is three Etc imagine the
1:51:16
model having to go from left to right
1:51:18
emitting these tokens one at a time it
1:51:20
has to say or we're expecting to say the
1:51:23
answer is space dollar sign and then
1:51:28
right here we're expecting it to
1:51:29
basically cram all of the computation of
1:51:31
this problem into this single token it
1:51:33
has to emit the correct answer three and
1:51:36
then once we've emitted the answer three
1:51:38
we're expecting it to say all these
1:51:40
tokens but at this point we've already
1:51:41
prod produced the answer and it's
1:51:43
already in the context window for all
1:51:45
these tokens that follow so anything
1:51:47
here is just um kind of post Hawk
1:51:49
justification of why this is the answer
1:51:52
um because the answer is already created
1:51:54
it's already in the token window so it's
1:51:56
it's not actually being calculated here
1:51:59
um and so if you are answering the
1:52:01
question directly and immediately you
1:52:03
are training the model to to try to
1:52:06
basically guess the answer in a single
1:52:08
token and that is just not going to work
1:52:10
because of the finite amount of
1:52:11
computation that happens per token
1:52:14
that's why this answer on the right is
1:52:16
significantly better because we are
1:52:17
Distributing this computation across the
1:52:19
answer we're actually getting the model
1:52:21
to sort of slowly come to the answer
1:52:23
from the left to right we're getting
1:52:25
intermediate results we're saying okay
1:52:27
the total cost of oranges is four so 30
1:52:29
- 4 is 9 and so we're creating
1:52:32
intermediate calculations and each one
1:52:34
of these calculations is by itself not
1:52:36
that expensive and so we're actually
1:52:38
basically kind of guessing a little bit
1:52:40
the difficulty that the model is capable
1:52:42
of in any single one of these individual
1:52:45
tokens and there can never be too much
1:52:47
work in any one of these tokens
1:52:49
computationally because then the model
1:52:51
won't be able to do that later at test
1:52:53
time and so we're teaching the model
1:52:55
here to spread out its reasoning and to
1:52:58
spread out its computation over the
1:53:00
tokens and in this way it only has very
1:53:03
simple problems in each token and they
1:53:05
can add up and then by the time it's
1:53:08
near the end it has all the previous
1:53:10
results in its working memory and it's
1:53:12
much easier for it to determine that the
1:53:13
answer is and here it is three so this
1:53:16
is a significantly better label for our
1:53:18
computation this would be really bad and
1:53:21
is teaching the model to try to do all
1:53:23
the computation in a single token and
1:53:25
it's really
1:53:26
bad so uh that's kind of like an
1:53:29
interesting thing to keep in mind is in
1:53:30
your
1:53:32
prompts uh usually don't have to think
1:53:34
about it explicitly because uh the
1:53:36
people at open AI have labelers and so
1:53:39
on that actually worry about this and
1:53:40
they make sure that the answers are
1:53:42
spread out and so actually open AI will
1:53:44
kind of like do the right thing so when
1:53:46
I ask this question for chat GPT it's
1:53:48
actually going to go very slowly it's
1:53:50
going to be like okay let's define our
1:53:51
variables set up the equation
1:53:53
and it's kind of creating all these
1:53:54
intermediate results these are not for
1:53:56
you these are for the model if the model
1:53:59
is not creating these intermediate
1:54:00
results for itself it's not going to be
1:54:02
able to reach three I also wanted to
1:54:04
show you that it's possible to be a bit
1:54:06
mean to the model uh we can just ask for
1:54:08
things so as an example I said I gave it
1:54:11
the exact same uh prompt and I said
1:54:13
answer the question in a single token
1:54:15
just immediately give me the answer
1:54:16
nothing else and it turns out that for
1:54:19
this simple um prompt here it actually
1:54:22
was able to do it in single go so it
1:54:24
just created a single I think this is
1:54:25
two tokens right uh because the dollar
1:54:28
sign is its own token so basically this
1:54:30
model didn't give me a single token it
1:54:32
gave me two tokens but it still produced
1:54:34
the correct answer and it did that in a
1:54:36
single forward pass of the
1:54:38
network now that's because the numbers
1:54:40
here I think are very simple and so I
1:54:42
made it a bit more difficult to be a bit
1:54:43
mean to the model so I said Emily buys
1:54:45
23 apples and 177 oranges and then I
1:54:48
just made the numbers a bit bigger and
1:54:50
I'm just making it harder for the model
1:54:52
I'm asking it to more computation in a
1:54:53
single token and so I said the same
1:54:56
thing and here it gave me five and five
1:54:58
is actually not correct so the model
1:55:00
failed to do all of this calculation in
1:55:02
a single forward pass of the network it
1:55:04
failed to go from the input tokens and
1:55:08
then in a single forward pass of the
1:55:09
network single go through the network it
1:55:11
couldn't produce the result and then I
1:55:14
said okay now don't worry about the the
1:55:16
token limit and just solve the problem
1:55:18
as usual and then it goes all the
1:55:20
intermediate results it simplifies and
1:55:23
every one of these intermediate results
1:55:24
here and intermediate calculations is
1:55:26
much easier for the model and um it sort
1:55:30
of it's not too much work per token all
1:55:32
of the tokens here are correct and it
1:55:34
arises the solution which is seven and I
1:55:36
just couldn't squeeze all of this work
1:55:38
it couldn't squeeze that into a single
1:55:40
forward passive Network so I think
1:55:42
that's kind of just a cute example and
1:55:43
something to kind of like think about
1:55:45
and I think it's kind of again just
1:55:47
elucidative in terms of how these uh
1:55:49
models work the last thing that I would
1:55:50
say on this topic is that if I was in
1:55:52
practi is trying to actually solve this
1:55:53
in my day-to-day life I might actually
1:55:55
not uh trust that the model that all the
1:55:58
intermediate calculations correctly here
1:56:00
so actually probably what I do is
1:56:01
something like this I would come here
1:56:02
and I would say use code and uh that's
1:56:06
because code is one of the possible
1:56:09
tools that chachy PD can use and instead
1:56:12
of it having to do mental arithmetic
1:56:14
like this mental arithmetic here I don't
1:56:16
fully trust it and especially if the
1:56:17
numbers get really big there's no
1:56:19
guarantee that the model will do this
1:56:20
correctly any one of these intermediates
1:56:22
steps might in principle fail we're
1:56:25
using neural networks to do mental
1:56:26
arithmetic uh kind of like you doing
1:56:28
mental arithmetic in your brain it might
1:56:30
just like uh screw up some of the
1:56:31
intermediate results it's actually kind
1:56:33
of amazing that it can even do this kind
1:56:34
of mental arithmetic I don't think I
1:56:35
could do this in my head but basically
1:56:37
the model is kind of like doing it in
1:56:38
its head and I don't trust that so I
1:56:40
wanted to use tools so you can say stuff
1:56:42
like use
1:56:43
code and uh I'm not sure what happened
1:56:47
there use
1:56:50
code and so um like I mentioned there's
1:56:53
a special tool and the uh the model can
1:56:56
write code and I can inspect that this
1:56:59
code is correct and then uh it's not
1:57:02
relying on its mental arithmetic it is
1:57:04
using the python interpreter which is a
1:57:05
very simple programming language to
1:57:07
basically uh write out the code that
1:57:09
calculates the result and I would
1:57:11
personally trust this a lot more because
1:57:12
this came out of a Python program which
1:57:14
I think has a lot more correctness
1:57:15
guarantees than the mental arithmetic of
1:57:18
a language model uh so just um another
1:57:21
kind of uh potential hint that if you
1:57:23
have these kinds of problems uh you may
1:57:25
want to basically just uh ask the model
1:57:27
to use the code interpreter and just
1:57:29
like we saw with the web search the
1:57:31
model has special uh kind of tokens for
1:57:34
calling uh like it will not actually
1:57:37
generate these tokens from the language
1:57:38
model it will write the program and then
1:57:41
it actually sends that program to a
1:57:43
different sort of part of the computer
1:57:44
that actually just runs that program and
1:57:46
brings back the result and then the
1:57:48
model gets access to that result and can
1:57:50
tell you that okay the cost of each
1:57:51
apple is seven
1:57:53
um so that's another kind of tool and I
1:57:55
would use this in practice for yourself
1:57:58
and it's um yeah it's just uh less error
1:58:02
prone I would say so that's why I called
1:58:04
this section models need tokens to think
1:58:07
distribute your competition across many
1:58:09
tokens ask models to create intermediate
1:58:11
results or whenever you can lean on
1:58:14
tools and Tool use instead of allowing
1:58:16
the models to do all of the stuff in
1:58:17
their memory so if they try to do it all
1:58:19
in their memory I don't fully trust it
1:58:21
and prefer to use tools whenever
1:58:23
possible I want to show you one more
1:58:25
example of where this actually comes up
1:58:27
and that's in counting so models
1:58:29
actually are not very good at counting
1:58:30
for the exact same reason you're asking
1:58:32
for way too much in a single individual
1:58:34
token so let me show you a simple
1:58:37
example of that um how many dots are
1:58:39
below and then I just put in a bunch of
1:58:41
dots and Chach says there are and then
1:58:45
it just tries to solve the problem in a
1:58:46
single token so in a single token it has
1:58:49
to count the number of dots in its
1:58:51
context window
1:58:53
um and it has to do that in the single
1:58:55
forward pass of a network and a single
1:58:57
forward pass of a network as we talked
1:58:59
about there's not that much computation
1:59:00
that can happen there just think of that
1:59:02
as being like very little competation
1:59:04
that happens there so if I just look at
1:59:06
what the model sees let's go to the LM
1:59:09
go to tokenizer it sees uh
1:59:14
this how many dots are below and then it
1:59:16
turns out that these dots here this
1:59:18
group of I think 20 dots is a single
1:59:20
token and then this group of whatever it
1:59:23
is is another token and then for some
1:59:25
reason they break up as this so I don't
1:59:28
actually this has to do with the details
1:59:29
of the tokenizer but it turns out that
1:59:31
these um the model basically sees the
1:59:34
token ID this this this and so on and
1:59:38
then from these token IDs it's expected
1:59:41
to count the number and spoiler alert is
1:59:44
not 161 it's actually I believe
1:59:46
177 so here's what we can do instead uh
1:59:48
we can say use code and you might expect
1:59:52
that like why should this work and it's
1:59:54
actually kind of subtle and kind of
1:59:55
interesting so when I say use code I
1:59:57
actually expect this to work let's see
1:59:59
okay 177 is correct so what happens here
2:00:03
is I've actually it doesn't look like it
2:00:05
but I've broken down the problem into a
2:00:08
problems that are easier for the model I
2:00:10
know that the model can't count it can't
2:00:12
do mental counting but I know that the
2:00:14
model is actually pretty good at doing
2:00:16
copy pasting so what I'm doing here is
2:00:18
when I say use code it creates a string
2:00:20
in Python for this and the task of
2:00:24
basically copy pasting my input here to
2:00:27
here is very simple because for the
2:00:30
model um it sees this string of uh it
2:00:34
sees it as just these four tokens or
2:00:36
whatever it is so it's very simple for
2:00:37
the model to copy paste those token IDs
2:00:41
and um kind of unpack them into Dots
2:00:45
here and so it creates this string and
2:00:48
then it calls python routine. count and
2:00:50
then it comes up with the correct answer
2:00:52
so the python interpreter is doing the
2:00:54
counting it's not the models mental
2:00:55
arithmetic doing the counting so it's
2:00:57
again a simple example of um models need
2:01:01
tokens to think don't rely on their
2:01:03
mental arithmetic and um that's why also
2:01:06
the models are not very good at counting
2:01:07
if you need them to do counting tasks
2:01:09
always ask them to lean on the tool now
2:01:12
the models also have many other little
2:01:14
cognitive deficits here and there and
2:01:15
these are kind of like sharp edges of
2:01:17
the technology to be kind of aware of
2:01:18
over time so as an example the models
2:01:21
are not very good with all kinds of
2:01:22
spelling related tasks they're not very
2:01:24
good at it and I told you that we would
2:01:27
loop back around to tokenization and the
2:01:29
reason to do for this is that the models
2:01:31
they don't see the characters they see
2:01:33
tokens and they their entire world is
2:01:36
about tokens which are these little text
2:01:37
chunks and so they don't see characters
2:01:39
like our eyes do and so very simple
2:01:42
character level tasks often fail so for
2:01:45
example uh I'm giving it a string
2:01:48
ubiquitous and I'm asking it to print
2:01:50
only every third character starting with
2:01:52
the first one so we start with U and
2:01:54
then we should go every third so every
2:01:57
so 1 2 3 Q should be next and then Etc
2:02:01
so this I see is not correct and again
2:02:04
my hypothesis is that this is again
2:02:06
Dental arithmetic here is failing number
2:02:08
one a little bit but number two I think
2:02:10
the the more important issue here is
2:02:12
that if you go to Tik
2:02:14
tokenizer and you look at ubiquitous we
2:02:16
see that it is three tokens right so you
2:02:19
and I see ubiquitous and we can easily
2:02:21
access the individual letters because we
2:02:24
kind of see them and when we have it in
2:02:26
the working memory of our visual sort of
2:02:28
field we can really easily index into
2:02:30
every third letter and I can do that
2:02:31
task but the models don't have access to
2:02:33
the individual letters they see this as
2:02:35
these three tokens and uh remember these
2:02:38
models are trained from scratch on the
2:02:40
internet and all these token uh
2:02:43
basically the model has to discover how
2:02:44
many of all these different letters are
2:02:46
packed into all these different tokens
2:02:48
and the reason we even use tokens is
2:02:49
mostly for efficiency uh but I think a
2:02:51
lot of people areed interested to delete
2:02:53
tokens entirely like we should really
2:02:55
have character level or bite level
2:02:56
models it's just that that would create
2:02:58
very long sequences and people don't
2:03:00
know how to deal with that right now so
2:03:02
while we have the token World any kind
2:03:04
of spelling tasks are not actually
2:03:05
expected to work super well so because I
2:03:08
know that spelling is not a strong suit
2:03:09
because of tokenization I can again Ask
2:03:12
it to lean On Tools so I can just say
2:03:14
use code and I would again expect this
2:03:16
to work because the task of copy pasting
2:03:18
ubiquitous into the python interpreter
2:03:20
is much easier and then we're leaning on
2:03:22
python interpreter to manipulate the
2:03:25
characters of this string so when I say
2:03:28
use
2:03:29
code
2:03:31
ubiquitous yes it indexes into every
2:03:33
third character and the actual truth is
2:03:35
u2s
2:03:37
uqs uh which looks correct to me so um
2:03:41
again an example of spelling related
2:03:43
tasks not working very well a very
2:03:45
famous example of that recently is how
2:03:47
many R are there in strawberry and this
2:03:49
went viral many times and basically the
2:03:52
models now get it correct they say there
2:03:53
are three Rs in Strawberry but for a
2:03:55
very long time all the state-of-the-art
2:03:57
models would insist that there are only
2:03:58
two RS in strawberry and this caused a
2:04:01
lot of you know Ruckus because is that a
2:04:04
word I think so because um it just kind
2:04:07
of like why are the models so brilliant
2:04:09
and they can solve math Olympiad
2:04:10
questions but they can't like count RS
2:04:13
in strawberry and the answer for that
2:04:15
again is I've got built up to it kind of
2:04:17
slowly but number one the models don't
2:04:19
see characters they see tokens and
2:04:21
number two they are not very good at
2:04:23
counting and so here we are combining
2:04:25
the difficulty of seeing the characters
2:04:27
with the difficulty of counting and
2:04:29
that's why the models struggled with
2:04:31
this even though I think by now honestly
2:04:33
I think open I may have hardcoded the
2:04:35
answer here or I'm not sure what they
2:04:36
did but um uh but this specific query
2:04:40
now works
2:04:42
so models are not very good at spelling
2:04:44
and there there's a bunch of other
2:04:46
little sharp edges and I don't want to
2:04:47
go into all of them I just want to show
2:04:48
you a few examples of things to be aware
2:04:50
of and uh when you're using these models
2:04:52
in practice I don't actually want to
2:04:54
have a comprehensive analysis here of
2:04:56
all the ways that the models are kind of
2:04:58
like falling short I just want to make
2:05:00
the point that there are some Jagged
2:05:01
edges here and there and we've discussed
2:05:04
a few of them and a few of them make
2:05:05
sense but some of them also will just
2:05:06
not make as much sense and they're kind
2:05:08
of like you're left scratching your head
2:05:10
even if you understand in- depth how
2:05:12
these models work and and good example
2:05:14
of that recently is the following uh the
2:05:16
models are not very good at very simple
2:05:18
questions like this and uh this is
2:05:20
shocking to a lot of people because
2:05:22
these math uh these problems can solve
2:05:24
complex math problems they can answer
2:05:26
PhD grade physics chemistry biology
2:05:29
questions much better than I can but
2:05:31
sometimes they fall short in like super
2:05:32
simple problems like this so here we go
2:05:35
9.11 is bigger than 9.9 and it justifies
2:05:38
it in some way but obviously and then at
2:05:41
the end okay it actually it flips its
2:05:44
decision later so um I don't believe
2:05:47
that this is very reproducible sometimes
2:05:49
it flips around its answer sometimes
2:05:51
gets it right sometimes get it get it
2:05:52
wrong uh let's try
2:05:57
again okay even though it might look
2:06:00
larger okay so here it doesn't even
2:06:02
correct itself in the end if you ask
2:06:03
many times sometimes it gets it right
2:06:05
too but how is it that the model can do
2:06:07
so great at Olympiad grade problems but
2:06:10
then fail on very simple problems like
2:06:12
this and uh I think this one is as I
2:06:16
mentioned a little bit of a head
2:06:17
scratcher it turns out that a bunch of
2:06:18
people studied this in depth and I
2:06:20
haven't actually read the paper uh but
2:06:22
what I was told by this team was that
2:06:25
when you scrutinize the activations
2:06:28
inside the neural network when you look
2:06:29
at some of the features and what what
2:06:31
features turn on or off and what neurons
2:06:33
turn on or off uh a bunch of neurons
2:06:36
inside the neural network light up that
2:06:38
are usually associated with Bible verses
2:06:41
U and so I think the model is kind of
2:06:43
like reminded that these almost look
2:06:45
like Bible verse markers and in a bip
2:06:48
verse setting 9.11 would come after 99.9
2:06:52
and so basically the model somehow finds
2:06:54
it like cognitively very distracting
2:06:56
that in Bible verses 9.11 would be
2:06:58
greater um even though here it's
2:07:01
actually trying to justify it and come
2:07:02
up to the answer with a math it still
2:07:05
ends up with the wrong answer here so it
2:07:07
basically just doesn't fully make sense
2:07:09
and it's not fully understood and um
2:07:13
there's a few Jagged issues like that so
2:07:15
that's why treat this as a as what it is
2:07:17
which is a St stochastic system that is
2:07:19
really magical but that you can't also
2:07:21
fully trust and you want to use it as a
2:07:23
tool not as something that you kind of
2:07:25
like letter rip on a problem and
2:07:27
copypaste the results okay so we have
2:07:29
now covered two major stages of training
2:07:32
of large language models we saw that in
2:07:35
the first stage this is called the
2:07:36
pre-training stage we are basically
2:07:38
training on internet documents and when
2:07:41
you train a language model on internet
2:07:42
documents you get what's called a base
2:07:44
model and it's basically an internet
2:07:46
document simulator right now we saw that
2:07:49
this is an interesting artifact and uh
2:07:51
this takes many months to train on
2:07:53
thousands of computers and it's kind of
2:07:55
a lossy compression of the internet and
2:07:57
it's extremely interesting but it's not
2:07:59
directly useful because we don't want to
2:08:01
sample internet documents we want to ask
2:08:03
questions of an AI and have it respond
2:08:05
to our questions so for that we need an
2:08:07
assistant and we saw that we can
2:08:09
actually construct an assistant in the
2:08:11
process of a post
2:08:14
training and specifically in the process
2:08:17
of supervised fine-tuning as we call
2:08:19
it so in this stage we saw that it's
2:08:23
algorithmically identical to
2:08:24
pre-training nothing is going to change
2:08:26
the only thing that changes is the data
2:08:28
set so instead of Internet documents we
2:08:30
now want to create and curate a very
2:08:33
nice data set of conversations so we
2:08:35
want Millions conversations on all kinds
2:08:38
of diverse topics between a human and an
2:08:42
assistant and fundamentally these
2:08:44
conversations are created by humans so
2:08:47
humans write the prompts and humans
2:08:50
write the ideal response responses and
2:08:52
they do that based on labeling
2:08:55
documentations now in the modern stack
2:08:57
it's not actually done fully and
2:08:59
manually by humans right they actually
2:09:01
now have a lot of help from these tools
2:09:03
so we can use language models um to help
2:09:06
us create these data sets and that's
2:09:07
done extensively but fundamentally it's
2:09:09
all still coming from Human curation at
2:09:11
the end so we create these conversations
2:09:14
that now becomes our data set we fine
2:09:16
tune on it or continue training on it
2:09:18
and we get an assistant and then we kind
2:09:20
of shifted gears and started talking
2:09:22
about some of the kind of cognitive
2:09:23
implications of what this assistant is
2:09:25
like and we saw that for example the
2:09:27
assistant will hallucinate if you don't
2:09:30
take some sort of mitigations towards it
2:09:32
so we saw that hallucinations would be
2:09:34
common and then we looked at some of the
2:09:36
mitigations of those hallucinations and
2:09:38
then we saw that the models are quite
2:09:39
impressive and can do a lot of stuff in
2:09:41
their head but we saw that they can also
2:09:43
Lean On Tools to become better so for
2:09:46
example we can lo lean on a web search
2:09:48
in order to hallucinate less and to
2:09:51
maybe bring up some more um recent
2:09:53
information or something like that or we
2:09:55
can lean on tools like code interpreter
2:09:57
so the code can so the llm can write
2:09:59
some code and actually run it and see
2:10:01
the
2:10:02
results so these are some of the topics
2:10:04
we looked at so far um now what I'd like
2:10:06
to do is I'd like to cover the last and
2:10:09
major stage of this Pipeline and that is
2:10:13
reinforcement learning so reinforcement
2:10:16
learning is still kind of thought to be
2:10:17
under the umbrella of posttraining uh
2:10:20
but it is the last third major stage and
2:10:22
it's a different way of training
2:10:24
language models and usually follows as
2:10:27
this third step so inside companies like
2:10:29
open AI you will start here and these
2:10:31
are all separate teams so there's a team
2:10:33
doing data for pre-training and a team
2:10:36
doing training for pre-training and then
2:10:38
there's a team doing all the
2:10:40
conversation generation in a in a
2:10:42
different team that is kind of doing the
2:10:44
supervis fine tuning and there will be a
2:10:46
team for the reinforcement learning as
2:10:47
well so it's kind of like a handoff of
2:10:49
these models you get your base model the
2:10:51
then you find you need to be an
2:10:52
assistant and then you go into
2:10:54
reinforcement learning which we'll talk
2:10:55
about uh
2:10:57
now so that's kind of like the major
2:10:59
flow and so let's now focus on
2:11:01
reinforcement learning the last major
2:11:03
stage of training and let me first
2:11:06
actually motivate it and why we would
2:11:07
want to do reinforcement learning and
2:11:09
what it looks like on a high level so I
2:11:11
would now like to try to motivate the
2:11:13
reinforcement learning stage and what it
2:11:14
corresponds to with something that
2:11:15
you're probably familiar with and that
2:11:17
is basically going to school so just
2:11:19
like you went to school to become um
2:11:21
really good at something we want to take
2:11:23
large language models through school and
2:11:26
really what we're doing is um we're um
2:11:30
we have a few paradigms of ways of uh
2:11:32
giving them knowledge or transferring
2:11:34
skills so in particular when we're
2:11:36
working with textbooks in school you'll
2:11:38
see that there are three major kind of
2:11:41
uh pieces of information in these
2:11:43
textbooks three classes of information
2:11:46
the first thing you'll see is you'll see
2:11:47
a lot of exposition um and by the way
2:11:49
this is a totally random book I pulled
2:11:51
from the internet I I think it's some
2:11:52
kind of an organic chemistry or
2:11:53
something I'm not sure uh but the
2:11:55
important thing is that you'll see that
2:11:57
most of the text most of it is kind of
2:11:59
just like the meat of it is exposition
2:12:01
it's kind of like background knowledge
2:12:03
Etc as you are reading through the words
2:12:06
of this Exposition you can think of that
2:12:08
roughly as training on that data so um
2:12:12
and that's why when you're reading
2:12:14
through this stuff this background
2:12:15
knowledge and this all this context
2:12:16
information it's kind of equivalent to
2:12:19
pre-training so it's it's where we build
2:12:21
sort of like a knowledge base of this
2:12:24
data and get a sense of the topic the
2:12:27
next major kind of information that you
2:12:29
will see is these uh problems and with
2:12:33
their worked Solutions so basically a
2:12:36
human expert in this case uh the author
2:12:37
of this book has given us not just a
2:12:39
problem but has also worked through the
2:12:41
solution and the solution is basically
2:12:44
like equivalent to having like this
2:12:46
ideal response for an assistant so it's
2:12:48
basically the expert is showing us how
2:12:50
to solve the problem in it's uh kind of
2:12:52
like um in its full form so as we are
2:12:55
reading the solution we are basically
2:12:58
training on the expert data and then
2:13:01
later we can try to imitate the expert
2:13:04
um and basically um that's that roughly
2:13:07
correspond to having the sft model
2:13:09
that's what it would be doing so
2:13:11
basically we've already done
2:13:12
pre-training and we've already covered
2:13:15
this um imitation of experts and how
2:13:17
they solve these problems and the third
2:13:20
stage of reinforcement learning is
2:13:22
basically the practice problems so
2:13:24
sometimes you'll see this is just a
2:13:26
single practice problem here but of
2:13:27
course there will be usually many
2:13:29
practice problems at the end of each
2:13:30
chapter in any textbook and practice
2:13:33
problems of course we know are critical
2:13:34
for learning because what are they
2:13:36
getting you to do they're getting you to
2:13:38
practice uh to practice yourself and
2:13:40
discover ways of solving these problems
2:13:42
yourself and so what you get in a
2:13:44
practice problem is you get a problem
2:13:46
description but you're not given the
2:13:49
solution but you are given the final
2:13:51
answer answer usually in the answer key
2:13:53
of the textbook and so you know the
2:13:55
final answer that you're trying to get
2:13:57
to and you have the problem statement
2:13:59
but you don't have the solution you are
2:14:00
trying to practice the solution you're
2:14:03
trying out many different things and
2:14:04
you're seeing what gets you to the final
2:14:07
solution the best and so you're
2:14:10
discovering how to solve these problems
2:14:12
so and in the process of that you're
2:14:13
relying on number one the background
2:14:15
information which comes from
2:14:16
pre-training and number two maybe a
2:14:18
little bit of imitation of human experts
2:14:20
and you can probably try similar kinds
2:14:22
of solutions and so on so we've done
2:14:25
this and this and now in this section
2:14:27
we're going to try to practice and so
2:14:30
we're going to be given prompts we're
2:14:32
going to be given Solutions U sorry the
2:14:35
final answers but we're not going to be
2:14:36
given expert Solutions we have to
2:14:39
practice and try stuff out and that's
2:14:41
what reinforcement learning is about
2:14:43
okay so let's go back to the problem
2:14:44
that we worked with previously just so
2:14:46
we have a concrete example to talk
2:14:48
through as we explore sort of the topic
2:14:50
here so um I'm here in the Teck
2:14:53
tokenizer because I'd also like to well
2:14:55
I get a text box which is useful but
2:14:57
number two I want to remind you again
2:14:59
that we're always working with
2:15:00
onedimensional token sequences and so um
2:15:03
I actually like prefer this view because
2:15:04
this is like the native view of the llm
2:15:06
if that makes sense like this is what it
2:15:08
actually sees it sees token IDs right
2:15:11
okay so Emily buys three apples and two
2:15:14
oranges each orange is $2 the total cost
2:15:17
of all the fruit is $13 what is the cost
2:15:20
of each apple
2:15:22
and what I'd like to what I like you to
2:15:23
appreciate here is these are like four
2:15:26
possible candidate Solutions as an
2:15:29
example and they all reach the answer
2:15:32
three now what I'd like you to
2:15:33
appreciate at this point is that if I am
2:15:35
the human data labeler that is creating
2:15:38
a conversation to be entered into the
2:15:39
training set I don't actually really
2:15:42
know which of these
2:15:44
conversations to um to add to the data
2:15:48
set some of these conversations kind of
2:15:50
set up a system equations some of them
2:15:53
sort of like just talk through it in
2:15:54
English and some of them just kind of
2:15:56
like skip right through to the
2:15:58
solution um if you look at chbt for
2:16:01
example and you give it this question it
2:16:04
defines a system of variables and it
2:16:05
kind of like does this little thing what
2:16:07
we have to appreciate and uh
2:16:09
differentiate between though is um the
2:16:12
first purpose of a solution is to reach
2:16:14
the right answer of course we want to
2:16:16
get the final answer three that is the
2:16:18
that is the important purpose here but
2:16:20
there's kind of like a secondary purpose
2:16:21
as well where here we are also just kind
2:16:24
of trying to make it like nice uh for
2:16:26
the human because we're kind of assuming
2:16:28
that the person wants to see the
2:16:29
solution they want to see the
2:16:30
intermediate steps we want to present it
2:16:32
nicely Etc so there are two separate
2:16:34
things going on here number one is the
2:16:36
presentation for the human but number
2:16:38
two we're trying to actually get the
2:16:39
right answer um so let's for the moment
2:16:42
focus on just reaching the final answer
2:16:45
if we're only care if we only care about
2:16:47
the final answer then which of these is
2:16:50
the optimal or the best prompt um sorry
2:16:54
the best solution for the llm to reach
2:16:56
the right
2:16:58
answer um and what I'm trying to get at
2:17:00
is we don't know me as a human labeler I
2:17:03
would not know which one of these is
2:17:04
best so as an example we saw earlier on
2:17:07
when we looked at
2:17:09
um the token sequences here and the
2:17:12
mental arithmetic and reasoning we saw
2:17:14
that for each token we can only spend
2:17:16
basically a finite number of finite
2:17:18
amount of compute here that is not very
2:17:20
large or you should think about it that
2:17:21
way way and so we can't actually make
2:17:24
too big of a leap in any one token is is
2:17:27
maybe the way to think about it so as an
2:17:29
example in this one what's really nice
2:17:31
about it is that it's very few tokens so
2:17:33
it's going to take us very short amount
2:17:34
of time to get to the answer but right
2:17:37
here when we're doing 30 - 4 IDE 3
2:17:40
equals right in this token here we're
2:17:43
actually asking for a lot of computation
2:17:44
to happen on that single individual
2:17:46
token and so maybe this is a bad example
2:17:48
to give to the llm because it's kind of
2:17:49
incentivizing it to skip through the
2:17:51
calculations very quickly and it's going
2:17:52
to actually make up mistakes make
2:17:54
mistakes in this mental arithmetic uh so
2:17:57
maybe it would work better to like
2:17:58
spread out the spread it out more maybe
2:18:01
it would be better to set it up as an
2:18:02
equation maybe it would be better to
2:18:04
talk through it we fundamentally don't
2:18:06
know and we don't know because what is
2:18:10
easy for you or I as or as human
2:18:12
labelers what's easy for us or hard for
2:18:14
us is different than what's easy or hard
2:18:17
for the llm it cognition is different um
2:18:20
and the token sequences are kind of like
2:18:23
different hard for it and so some of the
2:18:28
token sequences here that are trivial
2:18:30
for me might be um very too much of a
2:18:34
leap for the llm so right here this
2:18:37
token would be way too hard but
2:18:38
conversely many of the tokens that I'm
2:18:41
creating here might be just trivial to
2:18:43
the llm and we're just wasting tokens
2:18:45
like why waste all these tokens when
2:18:47
this is all trivial so if the only thing
2:18:49
we care care about is the final answer
2:18:52
and we're separating out the issue of
2:18:53
the presentation to the human um then we
2:18:56
don't actually really know how to
2:18:57
annotate this example we don't know what
2:18:59
solution to get to the llm because we
2:19:01
are not the
2:19:02
llm and it's clear here in the case of
2:19:05
like the math example but this is
2:19:07
actually like a very pervasive issue
2:19:09
like for our knowledge is not lm's
2:19:12
knowledge like the llm actually has a
2:19:14
ton of knowledge of PhD in math and
2:19:15
physics chemistry and whatnot so in many
2:19:17
ways it actually knows more than I do
2:19:19
and I'm I'm potentially not utilizing
2:19:22
that knowledge in its problem solving
2:19:24
but conversely I might be injecting a
2:19:26
bunch of knowledge in my solutions that
2:19:28
the LM doesn't know in its parameters
2:19:31
and then those are like sudden leaps
2:19:33
that are very confusing to the model and
2:19:36
so our cognitions are different and I
2:19:39
don't really know what to put here if
2:19:41
all we care about is the reaching the
2:19:43
final solution and doing it economically
2:19:46
ideally and so long story short we are
2:19:50
not in a good position to create these
2:19:53
uh token sequences for the LM and
2:19:55
they're useful by imitation to
2:19:57
initialize the system but we really want
2:20:00
the llm to discover the token sequences
2:20:02
that work for it we need to find it
2:20:05
needs to find for itself what token
2:20:07
sequence reliably gets to the answer
2:20:10
given the prompt and it needs to
2:20:12
discover that in the process of
2:20:13
reinforcement learning and of trial and
2:20:14
error so let's see how this example
2:20:18
would work like in reinforcement
2:20:20
learning
2:20:21
okay so we're now back in the huging
2:20:23
face inference playground and uh that
2:20:26
just allows me to very easily call uh
2:20:28
different kinds of models so as an
2:20:30
example here on the top right I chose
2:20:31
the Gemma 2 2 billion parameter model so
2:20:34
two billion is very very small so this
2:20:36
is a tiny model but it's okay so we're
2:20:39
going to give it um the way that
2:20:41
reinforcement learning will basically
2:20:42
work is actually quite quite simple um
2:20:45
we need to try many different kinds of
2:20:47
solutions and we want to see which
2:20:49
Solutions work well or not
2:20:51
so we're basically going to take the
2:20:53
prompt we're going to run the
2:20:55
model and the model generates a solution
2:20:59
and then we're going to inspect the
2:21:00
solution and we know that the correct
2:21:02
answer for this one is $3 and so indeed
2:21:05
the model gets it correct it says it's
2:21:07
$3 so this is correct so that's just one
2:21:10
attempt at DIS solution so now we're
2:21:12
going to delete this and we're going to
2:21:13
rerun it again let's try a second
2:21:15
attempt so the model solves it in a bit
2:21:18
slightly different way right every
2:21:20
single attempt will be a different
2:21:22
generation because these models are
2:21:23
stochastic systems remember that at
2:21:25
every single token here we have a
2:21:26
probability distribution and we're
2:21:28
sampling from that distribution so we
2:21:30
end up kind kind of going down slightly
2:21:32
different paths and so this is a second
2:21:34
solution that also ends in the correct
2:21:36
answer now we're going to delete that
2:21:39
let's go a third
2:21:40
time okay so again slightly different
2:21:42
solution but also gets it
2:21:44
correct now we can actually repeat this
2:21:47
uh many times and so in practice you
2:21:49
might actually sample thousand of
2:21:51
independent Solutions or even like
2:21:53
million solutions for just a single
2:21:55
prompt um and some of them will be
2:21:58
correct and some of them will not be
2:21:59
very correct and basically what we want
2:22:01
to do is we want to encourage the
2:22:02
solutions that lead to correct answers
2:22:05
so let's take a look at what that looks
2:22:07
like so if we come back over here here's
2:22:09
kind of like a cartoon diagram of what
2:22:11
this is looking like we have a prompt
2:22:14
and then we tried many different
2:22:15
solutions in
2:22:17
parallel and some of the solutions um
2:22:20
might go well so they get the right
2:22:22
answer which is in green and some of the
2:22:24
solutions might go poorly and may not
2:22:26
reach the right answer which is red now
2:22:29
this problem here unfortunately is not
2:22:30
the best example because it's a trivial
2:22:32
prompt and as we saw uh even like a two
2:22:35
billion parameter model always gets it
2:22:36
right so it's not the best example in
2:22:38
that sense but let's just exercise some
2:22:40
imagination here and let's just suppose
2:22:44
that the um green ones are good and the
2:22:47
red ones are
2:22:49
bad okay so we generated 15 Solutions
2:22:52
only four of them got the right answer
2:22:55
and so now what we want to do is
2:22:57
basically we want to encourage the kinds
2:22:58
of solutions that lead to right answers
2:23:01
so whatever token sequences happened in
2:23:03
these red Solutions obviously something
2:23:05
went wrong along the way somewhere and
2:23:08
uh this was not a good path to take
2:23:10
through the solution and whatever token
2:23:12
sequences there were in these Green
2:23:13
Solutions well things went uh pretty
2:23:15
well in this situation and so we want to
2:23:18
do more things like it in prompts like
2:23:21
this and the way we encourage this kind
2:23:24
of a behavior in the future is we
2:23:25
basically train on these sequences um
2:23:28
but these training sequencies now are
2:23:30
not coming from expert human annotators
2:23:32
there's no human who decided that this
2:23:34
is the correct solution this solution
2:23:36
came from the model itself so the model
2:23:38
is practicing here it's tried out a few
2:23:40
Solutions four of them seem to have
2:23:42
worked and now the model will kind of
2:23:44
like train on them and this corresponds
2:23:46
to a student basically looking at their
2:23:47
Solutions and being like okay well this
2:23:49
one worked really well so this is this
2:23:50
is how I should be solving these kinds
2:23:52
of problems and uh here in this example
2:23:56
there are many different ways to
2:23:57
actually like really tweak the
2:23:59
methodology a little bit here but just
2:24:01
to give the core idea across maybe it's
2:24:02
simplest to just think about take the
2:24:05
taking the single best solution out of
2:24:06
these four uh like say this one that's
2:24:09
why it was yellow uh so this is the the
2:24:12
solution that not only led to the right
2:24:14
answer but may maybe had some other nice
2:24:16
properties maybe it was the shortest one
2:24:18
or it looked nicest in some ways or uh
2:24:21
there's other criteria you could think
2:24:22
of as an example but we're going to
2:24:24
decide that this the top solution we're
2:24:25
going to train on it and then uh the
2:24:28
model will be slightly more likely once
2:24:31
you do the parameter update to take this
2:24:34
path in this kind of a setting in the
2:24:36
future but you have to remember that
2:24:38
we're going to run many different
2:24:40
diverse prompts across lots of math
2:24:42
problems and physics problems and
2:24:43
whatever wherever there might be so tens
2:24:46
of thousands of prompts maybe have in
2:24:48
mind there's thousands of solutions
2:24:51
prompt and so this is all happening kind
2:24:53
of like at the same time and as we're
2:24:55
iterating this process the model is
2:24:58
discovering for itself what kinds of
2:25:00
token sequences lead it to correct
2:25:03
answers it's not coming from a human
2:25:05
annotator the the model is kind of like
2:25:08
playing in this playground and it knows
2:25:10
what it's trying to get to and it's
2:25:13
discovering sequences that work for it
2:25:15
uh these are sequences that don't make
2:25:17
any mental leaps uh they they seem to
2:25:20
work reliably and statistically and uh
2:25:23
fully utilize the knowledge of the model
2:25:25
as it has it and so uh this is the
2:25:28
process of reinforcement
2:25:30
learning it's basically a guess and
2:25:32
check we're going to guess many
2:25:33
different types of solutions we're going
2:25:34
to check them and we're going to do more
2:25:36
of what worked in the future and that is
2:25:39
uh reinforcement learning so in the
2:25:41
context of what came before we see now
2:25:43
that the sft model the supervised fine
2:25:45
tuning model it's still helpful because
2:25:47
it still kind of like initializes the
2:25:49
model a little bit into to the vicinity
2:25:51
of the correct Solutions so it's kind of
2:25:53
like a initialization of um of the model
2:25:57
in the sense that it kind of gets the
2:25:58
model to you know take Solutions like
2:26:01
write out Solutions and maybe it has an
2:26:03
understanding of setting up a system of
2:26:04
equations or maybe it kind of like talks
2:26:06
through a solution so it gets you into
2:26:08
the vicinity of correct Solutions but
2:26:10
reinforcement learning is where
2:26:12
everything gets dialed in we really
2:26:14
discover the solutions that work for the
2:26:15
model get the right answers we encourage
2:26:18
them and then the model just kind of
2:26:19
like gets better over time time okay so
2:26:22
that is the high Lev process for how we
2:26:23
train large language models in short we
2:26:26
train them kind of very similar to how
2:26:28
we train children and basically the only
2:26:31
difference is that children go through
2:26:32
chapters of books and they do all these
2:26:35
different types of training exercises um
2:26:38
kind of within the chapter of each book
2:26:40
but instead when we train AIS it's
2:26:41
almost like we kind of do it stage by
2:26:43
stage depending on the type of that
2:26:45
stage so first what we do is we do
2:26:48
pre-training which as we saw is
2:26:49
equivalent to uh basically reading all
2:26:51
the expository material so we look at
2:26:54
all the textbooks at the same time and
2:26:55
we read all the exposition and we try to
2:26:58
build a knowledge base the second thing
2:27:00
then is we go into the sft stage which
2:27:03
is really looking at all the fixed uh
2:27:05
sort of like solutions from Human
2:27:07
Experts of all the different kinds of
2:27:10
worked Solutions across all the
2:27:12
textbooks and we just kind of get an sft
2:27:15
model which is able to imitate the
2:27:16
experts but does so kind of blindly it
2:27:18
just kind of like does its best guess
2:27:21
uh kind of just like trying to mimic
2:27:23
statistically the expert behavior and so
2:27:25
that's what you get when you look at all
2:27:26
the work Solutions and then finally in
2:27:29
the last stage we do all the practice
2:27:31
problems in the RL stage across all the
2:27:33
textbooks we only do the practice
2:27:35
problems and that's how we get the RL
2:27:38
model so on a high level the way we
2:27:40
train llms is very much equivalent uh to
2:27:43
the process that we train uh that we use
2:27:45
for training of children the next point
2:27:48
I would like to make is that actually
2:27:49
these first two stat ages pre-training
2:27:51
and surprise fine-tuning they've been
2:27:53
around for years and they are very
2:27:54
standard and everyone does them all the
2:27:55
different llm providers it is this last
2:27:58
stage the RL training that is a lot more
2:28:01
early in its process of development and
2:28:03
is not standard yet in the field and so
2:28:07
um this stage is a lot more kind of
2:28:09
early and nent and the reason for that
2:28:12
is because I actually skipped over a ton
2:28:13
of little details here in this process
2:28:15
the high level idea is very simple it's
2:28:17
trial and there learning but there's a
2:28:19
ton of details and little math
2:28:20
mathematical kind of like nuances to
2:28:22
exactly how you pick the solutions that
2:28:23
are the best and how much you train on
2:28:25
them and what is the prompt distribution
2:28:27
and how to set up the training run such
2:28:29
that this actually works so there's a
2:28:31
lot of little details and knobs to the
2:28:33
core idea that is very very simple and
2:28:35
so getting the details right here uh is
2:28:38
not trivial and so a lot of companies
2:28:40
like for example open and other LM
2:28:42
providers have experimented internally
2:28:44
with reinforcement learning fine tuning
2:28:46
for llms for a while but they've not
2:28:49
talked about it publicly
2:28:51
um it's all kind of done inside the
2:28:52
company and so that's why the paper from
2:28:55
Deep seek that came out very very
2:28:57
recently was such a big deal because
2:28:59
this is a paper from this company called
2:29:01
DC Kai in China and this paper really
2:29:05
talked very publicly about reinforcement
2:29:07
learning fine training for large
2:29:08
language models and how incredibly
2:29:10
important it is for large language
2:29:12
models and how it brings out a lot of
2:29:15
reasoning capabilities in the models
2:29:16
we'll go into this in a second so this
2:29:18
paper reinvigorated the public interest
2:29:21
of using RL for llms and gave a lot of
2:29:25
the um sort of n-r details that are
2:29:28
needed to reproduce their results and
2:29:30
actually get the stage to work for large
2:29:31
langage models so let me take you
2:29:33
briefly through this uh deep seek R1
2:29:35
paper and what happens when you actually
2:29:37
correctly apply RL to language models
2:29:39
and what that looks like and what that
2:29:40
gives you so the first thing I'll scroll
2:29:41
to is this uh kind of figure two here
2:29:44
where we are looking at the Improvement
2:29:46
in how the models are solving
2:29:47
mathematical problems so this is the
2:29:49
accuracy of solving mathematical
2:29:51
problems on the a accuracy and then we
2:29:54
can go to the web page and we can see
2:29:55
the kinds of problems that are actually
2:29:57
in these um these the kinds of math
2:29:59
problems that are being measured here so
2:30:01
these are simple math problems you can
2:30:02
um pause the video if you like but these
2:30:05
are the kinds of problems that basically
2:30:06
the models are being asked to solve and
2:30:08
you can see that in the beginning
2:30:09
they're not doing very well but then as
2:30:11
you update the model with this many
2:30:13
thousands of steps their accuracy kind
2:30:15
of continues to climb so the models are
2:30:17
improving and they're solving these
2:30:19
problems with a higher accuracy
2:30:21
as you do this trial and error on a
2:30:23
large data set of these kinds of
2:30:24
problems and the models are discovering
2:30:26
how to solve math problems but even more
2:30:29
incredible than the quantitative kind of
2:30:32
results of solving these problems with a
2:30:34
higher accuracy is the qualitative means
2:30:36
by which the model achieves these
2:30:37
results so when we scroll down uh one of
2:30:40
the figures here that is kind of
2:30:41
interesting is that later on in the
2:30:44
optimization the model seems to be uh
2:30:47
using average length per response uh
2:30:50
goes up up so the model seems to be
2:30:51
using more tokens to get its higher
2:30:55
accuracy results so it's learning to
2:30:56
create very very long Solutions why are
2:30:59
these Solutions very long we can look at
2:31:01
them qualitatively here so basically
2:31:03
what they discover is that the model
2:31:05
solution get very very long partially
2:31:07
because so here's a question and here's
2:31:09
kind of the answer from the model what
2:31:11
the model learns to do um and this is an
2:31:14
immerging property of new optimization
2:31:16
it just discovers that this is good for
2:31:18
problem solving is it starts to do stuff
2:31:20
like this wait wait wait that's Nota
2:31:22
moment I can flag here let's reevaluate
2:31:24
this step by step to identify the
2:31:25
correct sum can be so what is the model
2:31:27
doing here right the model is basically
2:31:30
re-evaluating steps it has learned that
2:31:32
it works better for accuracy to try out
2:31:35
lots of ideas try something from
2:31:37
different perspectives retrace reframe
2:31:40
backtrack is doing a lot of the things
2:31:42
that you and I are doing in the process
2:31:43
of problem solving for mathematical
2:31:45
questions but it's rediscovering what
2:31:47
happens in your head not what you put
2:31:49
down on the solution and there is no
2:31:51
human who can hardcode this stuff in the
2:31:53
ideal assistant response this is only
2:31:55
something that can be discovered in the
2:31:56
process of reinforcement learning
2:31:58
because you wouldn't know what to put
2:32:00
here this just turns out to work for the
2:32:02
model and it improves its accuracy in
2:32:04
problem solving so the model learns what
2:32:07
we call these chains of thought in your
2:32:09
head and it's an emergent property of
2:32:11
the optim of the optimization and that's
2:32:14
what's bloating up the response length
2:32:16
but that's also what's increasing the
2:32:18
accuracy of the problem problem solving
2:32:21
so what's incredible here is basically
2:32:22
the model is discovering ways to think
2:32:25
it's learning what I like to call
2:32:26
cognitive strategies of how you
2:32:28
manipulate a problem and how you
2:32:30
approach it from different perspectives
2:32:32
how you pull in some analogies or do
2:32:34
different kinds of things like that and
2:32:36
how you kind of uh try out many
2:32:37
different things over time uh check a
2:32:39
result from different perspectives and
2:32:41
how you kind of uh solve problems but
2:32:43
here it's kind of discovered by the RL
2:32:45
so extremely incredible to see this
2:32:47
emerge in the optimization without
2:32:49
having to hardcode it anywhere the only
2:32:51
thing we've given it are the correct
2:32:52
answers and this comes out from trying
2:32:55
to just solve them correctly which is
2:32:57
incredible
2:32:58
um now let's go back to actually the
2:33:01
problem that we've been working with and
2:33:02
let's take a look at what it would look
2:33:04
like uh for uh for this kind of a model
2:33:08
what we call reasoning or thinking model
2:33:10
to solve that problem okay so recall
2:33:12
that this is the problem we've been
2:33:13
working with and when I pasted it into
2:33:15
chat GPT 40 I'm getting this kind of a
2:33:18
response let's take a look at what
2:33:19
happens when you give this same query to
2:33:22
what's called a reasoning or a thinking
2:33:24
model this is a model that was trained
2:33:25
with reinforcement learning so this
2:33:28
model described in this paper DC car1 is
2:33:31
available on chat. dec.com uh so this is
2:33:34
kind of like the company uh that
2:33:36
developed is hosting it you have to make
2:33:38
sure that the Deep think button is
2:33:39
turned on to get the R1 model as it's
2:33:42
called we can paste it here and run
2:33:44
it and so let's take a look at what
2:33:47
happens now and what is the output of
2:33:48
the model okay so here's it says so this
2:33:51
is previously what we get using
2:33:53
basically what's an sft approach a
2:33:55
supervised funing approach this is like
2:33:57
mimicking an expert solution this is
2:33:59
what we get from the RL model okay let
2:34:02
me try to figure this out so Emily buys
2:34:03
three apples and two oranges each orange
2:34:05
cost $2 total is 13 I need to find out
2:34:08
blah blah blah so here you you um as
2:34:11
you're reading this you can't escape
2:34:14
thinking that this model is
2:34:16
thinking um is definitely pursuing the
2:34:19
solution solution it deres that it must
2:34:22
cost $3 and then it says wait a second
2:34:24
let me check my math again to be sure
2:34:25
and then it tries it from a slightly
2:34:26
different perspective and then it says
2:34:29
yep all that checks out I think that's
2:34:31
the answer I don't see any mistakes let
2:34:33
me see if there's another way to
2:34:34
approach the problem maybe setting up an
2:34:36
equation let's let the cost of one apple
2:34:39
be $8 then blah blah blah yep same
2:34:42
answer so definitely each apple is $3
2:34:45
all right confident that that's correct
2:34:47
and then what it does once it sort of um
2:34:50
did the thinking process is it writes up
2:34:52
the nice solution for the human and so
2:34:54
this is now considering so this is more
2:34:56
about the correctness aspect and this is
2:34:59
more about the presentation aspect where
2:35:01
it kind of like writes it out nicely and
2:35:03
uh boxes in the correct answer at the
2:35:05
bottom and so what's incredible about
2:35:07
this is we get this like thinking
2:35:08
process of the model and this is what's
2:35:11
coming from the reinforcement learning
2:35:12
process this is what's bloating up the
2:35:15
length of the token sequences they're
2:35:17
doing thinking and they're trying
2:35:18
different ways this is what's giving you
2:35:21
higher accuracy in problem
2:35:22
solving and this is where we are seeing
2:35:25
these aha moments and these different
2:35:27
strategies and these um ideas for how
2:35:30
you can make sure that you're getting
2:35:31
the correct
2:35:32
answer the last point I wanted to make
2:35:34
is some people are a little bit nervous
2:35:36
about putting you know very sensitive
2:35:39
data into chat.com because this is a
2:35:41
Chinese company so people don't um
2:35:43
people are a little bit careful and Cy
2:35:45
with that a little bit um deep seek R1
2:35:48
is a model that was released by this
2:35:50
company so this is an open source model
2:35:52
or open weights model it is available
2:35:55
for anyone to download and use you will
2:35:57
not be able to like run it in its full
2:36:00
um sort of the full model in full
2:36:03
Precision you won't run that on a
2:36:04
MacBook but uh or like a local device
2:36:07
because this is a fairly large model but
2:36:09
many companies are hosting the full
2:36:11
largest model one of those companies
2:36:13
that I like to use is called
2:36:15
together. so when you go to together.
2:36:17
you sign up and you go to playgrounds
2:36:19
you can can select here in the chat deep
2:36:22
seek R1 and there's many different kinds
2:36:24
of other models that you can select here
2:36:25
these are all state-of-the-art models so
2:36:27
this is kind of similar to the hugging
2:36:28
face inference playground that we've
2:36:30
been playing with so far but together. a
2:36:32
will usually host all the
2:36:33
state-of-the-art models so select DT
2:36:36
car1 um you can try to ignore a lot of
2:36:38
these I think the default settings will
2:36:40
often be okay and we can put in this and
2:36:44
because the model was released by Deep
2:36:45
seek what you're getting here should be
2:36:48
basically equivalent to what you're
2:36:49
getting here now because of the
2:36:51
randomness in the sampling we're going
2:36:52
to get something slightly different uh
2:36:54
but in principle this should be uh
2:36:56
identical in terms of the power of the
2:36:57
model and you should be able to see the
2:36:59
same things quantitatively and
2:37:00
qualitatively uh but uh this model is
2:37:02
coming from kind of a an American
2:37:05
company so that's deep seek and that's
2:37:07
the what's called a reasoning
2:37:09
model now when I go back to chat uh let
2:37:12
me go to chat here okay so the models
2:37:15
that you're going to see in the drop
2:37:16
down here some of them like 01 03 mini
2:37:19
O3 mini High Etc they are talking about
2:37:21
uses Advanced reasoning now what this is
2:37:24
referring to uses Advanced reasoning is
2:37:26
it's referring to the fact that it was
2:37:28
trained by reinforcement learning with
2:37:30
techniques very similar to those of deep
2:37:32
C car1 per public statements of opening
2:37:34
ey employees uh so these are thinking
2:37:38
models trained with RL and these models
2:37:40
like GPT 4 or GPT 4 40 mini that you're
2:37:43
getting in the free tier you should
2:37:44
think of them as mostly sft models
2:37:46
supervised fine tuning models they don't
2:37:48
actually do this like thinking as as you
2:37:50
see in the RL models and even though
2:37:52
there's a little bit of reinforcement
2:37:53
learning involved with these models and
2:37:55
I'll go that into that in a second these
2:37:57
are mostly sft models I think you should
2:37:59
think about it that way so in the same
2:38:01
way as what we saw here we can pick one
2:38:03
of the thinking models like say 03 mini
2:38:05
high and these models by the way might
2:38:07
not be available to you unless you pay a
2:38:10
Chachi PT subscription of either $20 per
2:38:12
month or $200 per month for some of the
2:38:14
top models so we can pick a thinking
2:38:17
model and run now what's going to happen
2:38:20
here is it's going to say reasoning and
2:38:22
it's going to start to do stuff like
2:38:23
this and um what we're seeing here is
2:38:27
not exactly the stuff we're seeing here
2:38:29
so even though under the hood the model
2:38:32
produces these kinds of uh kind of
2:38:34
chains of thought opening ey chooses to
2:38:36
not show the exact chains of thought in
2:38:39
the web interface it shows little
2:38:41
summaries of that of those chains of
2:38:43
thought and open kind of does this I
2:38:45
think partly because uh they are worried
2:38:47
about what's called the distillation
2:38:48
risk that is that someone could come in
2:38:50
and actually try to imitate those
2:38:52
reasoning traces and recover a lot of
2:38:54
the reasoning performance by just
2:38:55
imitating the reasoning uh chains of
2:38:58
thought and so they kind of hide them
2:38:59
and they only show little summaries of
2:39:01
them so you're not getting exactly what
2:39:02
you would get in deep seek as with
2:39:04
respect to the reasoning itself and then
2:39:07
they write up the
2:39:09
solution so these are kind of like
2:39:11
equivalent even though we're not seeing
2:39:12
the full under the hood details now in
2:39:14
terms of the performance uh these models
2:39:17
and deep seek models are currently rly
2:39:20
on par I would say it's kind of hard to
2:39:21
tell because of the evaluations but if
2:39:23
you're paying $200 per month to open AI
2:39:25
some of these models I believe are
2:39:26
currently they basically still look
2:39:28
better uh but deep seek R1 for now is
2:39:31
still a very solid choice for a thinking
2:39:33
model that would be available to you um
2:39:36
sort of um either on this website or any
2:39:39
other website because the model is open
2:39:41
weights you can just download it so
2:39:44
that's thinking models so what is the
2:39:46
summary so far well we've talked about
2:39:48
reinforcement learning and the fact that
2:39:51
thinking emerges in the process of the
2:39:52
optimization on when we basically run RL
2:39:55
on many math uh and kind of code
2:39:57
problems that have verifiable Solutions
2:40:00
so there's like an answer three
2:40:02
Etc now these thinking models you can
2:40:05
access in for example deep seek or any
2:40:07
inference provider like together. a and
2:40:10
choosing deep seek over there these
2:40:13
thinking models are also available uh in
2:40:15
chpt under any of the 01 or O3
2:40:18
models but these GPT 4 R models Etc
2:40:21
they're not thinking models you should
2:40:22
think of them as mostly sft models now
2:40:25
if you are um if you have a prompt that
2:40:28
requires Advanced reasoning and so on
2:40:30
you should probably use some of the
2:40:31
thinking models or at least try them out
2:40:33
but empirically for a lot of my use when
2:40:35
you're asking a simpler question there's
2:40:37
like a knowledge based question or
2:40:38
something like that this might be
2:40:39
Overkill like there's no need to think
2:40:40
30 seconds about some factual question
2:40:43
so for that I will uh sometimes default
2:40:45
to just GPT 40 so empirically about 80
2:40:47
90% of my use is just gp4
2:40:50
and when I come across a very difficult
2:40:51
problem like in math and code Etc I will
2:40:53
reach for the thinking models but then I
2:40:56
have to wait a bit longer because
2:40:57
they're thinking um so you can access
2:41:00
these on chat on deep seek also I wanted
2:41:03
to point out that um AI studio.
2:41:06
go.com even though it looks really busy
2:41:08
really ugly because Google's just unable
2:41:11
to do this kind of stuff well it's like
2:41:13
what is happening but if you choose
2:41:15
model and you choose here Gemini 2.0
2:41:18
flash thinking experimental 01 21 if you
2:41:20
choose that one that's also a a kind of
2:41:22
early experiment experimental of a
2:41:25
thinking model by Google so we can go
2:41:27
here and we can give it the same problem
2:41:29
and click run and this is also a
2:41:31
thinking problem a thinking model that
2:41:34
will also do something
2:41:35
similar and comes out with the right
2:41:37
answer here so basically Gemini also
2:41:40
offers a thinking model anthropic
2:41:42
currently does not offer a thinking
2:41:44
model but basically this is kind of like
2:41:45
the frontier development of these llms I
2:41:48
think RL is kind of like this new
2:41:49
exciting stage but getting the details
2:41:52
right is difficult and that's why all
2:41:54
these models and thinking models are
2:41:55
currently experimental as of 2025 very
2:41:58
early 2025 um but this is kind of like
2:42:01
the frontier development of pushing the
2:42:03
performance on these very difficult
2:42:04
problems using reasoning that is
2:42:06
emerging in these optimizations one more
2:42:08
connection that I wanted to bring up is
2:42:10
that the discovery that reinforcement
2:42:12
learning is extremely powerful way of
2:42:14
learning is not new to the field of AI
2:42:18
and one place what we've already seen
2:42:19
this demonstrated is in the game of Go
2:42:22
and famously Deep Mind developed the
2:42:24
system alphago and you can watch a movie
2:42:26
about it um where the system is learning
2:42:30
to play the game of go against top human
2:42:32
players and um when we go to the paper
2:42:36
underlying alphago so in this paper when
2:42:40
we scroll
2:42:41
down we actually find a really
2:42:43
interesting
2:42:44
plot um that I think uh is kind of
2:42:47
familiar uh to us and we're kind of like
2:42:49
we discovering in the more open domain
2:42:52
of arbitrary problem solving instead of
2:42:54
on the closed specific domain of the
2:42:55
game of Go but basically what they saw
2:42:58
and we're going to see this in llms as
2:42:59
well as this becomes more mature is this
2:43:03
is the ELO rating of playing game of Go
2:43:05
and this is leas dull an extremely
2:43:07
strong human player and here what they
2:43:10
are comparing is the strength of a model
2:43:12
learned trained by supervised learning
2:43:14
and a model trained by reinforcement
2:43:16
learning so the supervised learning
2:43:18
model is imitating human expert players
2:43:21
so if you just get a huge amount of
2:43:22
games played by expert players in the
2:43:24
game of Go and you try to imitate them
2:43:26
you are going to get better but then you
2:43:29
top out and you never quite get better
2:43:32
than some of the top top top players of
2:43:34
in the game of Go like LEL so you're
2:43:36
never going to reach there because
2:43:38
you're just imitating human players you
2:43:39
can't fundamentally go beyond a human
2:43:41
player if you're just imitating human
2:43:43
players but in a process of
2:43:44
reinforcement learning is significantly
2:43:46
more powerful in reinforcement learning
2:43:48
for a game of Go it means that the
2:43:50
system is playing moves that empirically
2:43:53
and statistically lead to win to winning
2:43:56
the game and so alphago is a system
2:44:00
where it kind of plays against it itself
2:44:02
and it's using reinforcement learning to
2:44:04
create
2:44:05
rollouts so it's the exact same diagram
2:44:07
here but there's no prompt it's just uh
2:44:10
because there's no prompt it's just a
2:44:11
fixed game of Go but it's trying out
2:44:14
lots of solutions it's trying out lots
2:44:15
of plays and then the games that lead to
2:44:18
a win instead of a specific answer are
2:44:21
reinforced they're they're made stronger
2:44:24
and so um the system is learning
2:44:27
basically the sequences of actions that
2:44:28
empirically and statistically lead to
2:44:30
winning the game and reinforcement
2:44:33
learning is not going to be constrained
2:44:34
by human performance and reinforcement
2:44:36
learning can do significantly better and
2:44:38
overcome even the top players like Lisa
2:44:41
Dole and so uh probably they could have
2:44:45
run this longer and they just chose to
2:44:46
crop it at some point because this costs
2:44:48
money but this is very powerful
2:44:49
demonstration of reinforcement learning
2:44:51
and we're only starting to kind of see
2:44:53
hints of this diagram in larger language
2:44:56
models for reasoning problems so we're
2:44:59
not going to get too far by just
2:45:00
imitating experts we need to go beyond
2:45:02
that set up these like little game
2:45:04
environments and get let let the system
2:45:07
discover reasoning traces or like ways
2:45:10
of solving problems uh that are unique
2:45:14
and that uh just basically work
2:45:16
well now on this aspect of uniqueness
2:45:19
notice that when you're doing
2:45:20
reinforcement learning nothing prevents
2:45:22
you from veering off the distribution of
2:45:25
how humans are playing the game and so
2:45:27
when we go back to uh this alphao search
2:45:29
here one of the suggested modifications
2:45:32
is called move 37 and move 37 in alphao
2:45:35
is referring to a specific point in time
2:45:37
where alphago basically played a move
2:45:41
that uh no human expert would play uh so
2:45:44
the probability of this move uh to be
2:45:46
played by a human player was evaluated
2:45:48
to be about 1 in 10th ,000 so it's a
2:45:50
very rare move but in retrospect it was
2:45:52
a brilliant move so alphago in the
2:45:54
process of reinforcement learning
2:45:56
discovered kind of like a strategy of
2:45:58
playing that was unknown to humans and
2:46:00
but is in retrospect uh brilliant I
2:46:02
recommend this YouTube video um leis do
2:46:05
versus alphao move 37 reactions and
2:46:07
Analysis and this is kind of what it
2:46:09
looked like when alphao played this
2:46:11
move
2:46:14
value that's a very that's a very
2:46:17
surprising move I thought I thought it
2:46:20
was I thought it was a
2:46:22
mistake when I see this move anyway so
2:46:25
basically people are kind of freaking
2:46:26
out because it's a it's a move that a
2:46:29
human would not play that alphago played
2:46:31
because in its training uh this move
2:46:34
seemed to be a good idea it just happens
2:46:36
not to be a kind of thing that a humans
2:46:37
would would do and so that is again the
2:46:39
power of reinforcement learning and in
2:46:41
principle we can actually see the
2:46:43
equivalence of that if we continue
2:46:44
scaling this Paradigm in language models
2:46:47
and what that looks like is kind of
2:46:48
unknown so so um what does it mean to
2:46:51
solve problems in such a way that uh
2:46:55
even humans would not be able to get how
2:46:57
can you be better at reasoning or
2:46:58
thinking than humans how can you go
2:47:00
beyond just uh a thinking human like
2:47:04
maybe it means discovering analogies
2:47:06
that humans would not be able to uh
2:47:08
create or maybe it's like a new thinking
2:47:10
strategy it's kind of hard to think
2:47:11
through uh maybe it's a holy new
2:47:14
language that actually is not even
2:47:16
English maybe it discovers its own
2:47:18
language that is a lot better at
2:47:20
thinking um because the model is
2:47:23
unconstrained to even like stick with
2:47:24
English uh so maybe it takes a different
2:47:27
language to think in or it discovers its
2:47:29
own language so in principle the
2:47:31
behavior of the system is a lot less
2:47:33
defined it is open to do whatever works
2:47:37
and it is open to also slowly Drift from
2:47:40
the distribution of its training data
2:47:41
which is English but all of that can
2:47:44
only be done if we have a very large
2:47:46
diverse set of problems in which the
2:47:48
these strategy can be refined and
2:47:50
perfected and so that is a lot of the
2:47:52
frontier LM research that's going on
2:47:54
right now is trying to kind of create
2:47:56
those kinds of prompt distributions that
2:47:57
are large and diverse these are all kind
2:47:59
of like game environments in which the
2:48:01
llms can practice their thinking and uh
2:48:04
it's kind of like writing you know these
2:48:06
practice problems we have to create
2:48:08
practice problems for all of domains of
2:48:10
knowledge and if we have practice
2:48:12
problems and tons of them the models
2:48:14
will be able to reinforcement learning
2:48:16
reinforcement learn on them and kind of
2:48:18
uh create these kinds of uh diagrams but
2:48:22
in the domain of open thinking instead
2:48:24
of a closed domain like game of Go
2:48:27
there's one more section within
2:48:28
reinforcement learning that I wanted to
2:48:29
cover and that is that of learning in
2:48:32
unverifiable domains so so far all of
2:48:35
the problems that we've looked at are in
2:48:37
what's called verifiable domains that is
2:48:39
any candidate solution we can score very
2:48:41
easily against a concrete answer so for
2:48:44
example answer is three and we can very
2:48:46
easily score these Solutions against the
2:48:48
answer of three
2:48:50
either we require the models to like box
2:48:52
in their answers and then we just check
2:48:54
for equality of whatever is in the box
2:48:56
with the answer or you can also use uh
2:48:58
kind of what's called an llm judge so
2:49:01
the llm judge looks at a solution and it
2:49:03
gets the answer and just basically
2:49:05
scores the solution for whether it's
2:49:07
consistent with the answer or not and
2:49:08
llms uh empirically are good enough at
2:49:11
the current capability that they can do
2:49:12
this fairly reliably so we can apply
2:49:14
those kinds of techniques as well in any
2:49:16
case we have a concrete answer and we're
2:49:18
just checking Solutions again against it
2:49:19
and we can do this automatically with no
2:49:21
kind of humans in the loop the problem
2:49:24
is that we can't apply the strategy in
2:49:25
what's called unverifiable domains so
2:49:28
usually these are for example creative
2:49:29
writing tasks like write a joke about
2:49:31
Pelicans or write a poem or summarize a
2:49:33
paragraph or something like that in
2:49:35
these kinds of domains it becomes harder
2:49:37
to score our different solutions to this
2:49:40
problem so for example writing a joke
2:49:42
about Pelicans we can generate lots of
2:49:43
different uh jokes of course that's fine
2:49:46
for example we can go to chbt and we can
2:49:48
get it to uh generate a joke about
2:49:51
Pelicans uh so much stuff in their beaks
2:49:54
because they don't bellan in
2:49:57
backpacks what
2:49:59
okay we can uh we can try something else
2:50:03
why don't Pelicans ever pay for their
2:50:04
drinks because they always B it to
2:50:06
someone else haha okay so these models
2:50:10
are not obviously not very good at humor
2:50:12
actually I think it's pretty fascinating
2:50:13
because I think humor is secretly very
2:50:15
difficult and the model have the
2:50:17
capability I think anyway in any case
2:50:20
you could imagine creating lots of jokes
2:50:23
the problem that we are facing is how do
2:50:25
we score them now in principle we could
2:50:28
of course get a human to look at all
2:50:29
these jokes just like I did right now
2:50:32
the problem with that is if you are
2:50:33
doing reinforcement learning you're
2:50:35
going to be doing many thousands of
2:50:36
updates and for each update you want to
2:50:38
be looking at say thousands of prompts
2:50:41
and for each prompt you want to be
2:50:42
potentially looking at looking at
2:50:43
hundred or thousands of different kinds
2:50:45
of generations and so there's just like
2:50:48
way too many of these to look at and so
2:50:51
um in principle you could have a human
2:50:52
inspect all of them and score them and
2:50:54
decide that okay maybe this one is funny
2:50:56
and uh maybe this one is funny and this
2:50:58
one is funny and we could train on them
2:51:01
to get the model to become slightly
2:51:03
better at jokes um in the context of
2:51:05
pelicans at least um the problem is that
2:51:09
it's just like way too much human time
2:51:11
this is an unscalable strategy we need
2:51:12
some kind of an automatic strategy for
2:51:14
doing this and one sort of solution to
2:51:17
this was proposed in this paper
2:51:19
uh that introduced what's called
2:51:20
reinforcement learning from Human
2:51:22
feedback and so this was a paper from
2:51:24
open at the time and many of these
2:51:25
people are now um co-founders in
2:51:28
anthropic um and this kind of proposed a
2:51:31
approach for uh basically doing
2:51:33
reinforcement learning in unverifiable
2:51:35
domains so let's take a look at how that
2:51:37
works so this is the cartoon diagram of
2:51:40
the core ideas involved so as I
2:51:42
mentioned the native approach is if we
2:51:44
just set Infinity human time we could
2:51:46
just run RL in these domains just fine
2:51:49
so for example we can run RL as usual if
2:51:52
I have Infinity humans I would I just
2:51:54
want to do and these are just cartoon
2:51:55
numbers I want to do 1,000 updates where
2:51:58
each update will be on 1,000 prompts and
2:52:01
in for each prompt we're going to have
2:52:02
1,000 roll outs that we're scoring so we
2:52:06
can run RL with this kind of a setup the
2:52:09
problem is in the process of doing this
2:52:11
I will need to run one I will need to
2:52:13
ask a human to evaluate a joke a total
2:52:15
of 1 billion times and so that's a lot
2:52:18
of people looking at really terrible
2:52:20
jokes so we don't want to do that so
2:52:22
instead we want to take the arlef
2:52:24
approach so um in our Rel of approach we
2:52:28
are kind of like the the core trick is
2:52:30
that of indirection so we're going to
2:52:33
involve humans just a little bit and the
2:52:35
way we cheat is that we basically train
2:52:37
a whole separate neural network that we
2:52:39
call a reward model and this neural
2:52:42
network will kind of like imitate human
2:52:44
scores so we're going to ask humans to
2:52:47
score um roll
2:52:49
we're going to then imitate human scores
2:52:52
using a neural network and this neural
2:52:54
network will become a kind of simulator
2:52:56
of human
2:52:57
preferences and now that we have a
2:52:58
neural network simulator we can do RL
2:53:01
against it so instead of asking a real
2:53:03
human we're asking a simulated human for
2:53:06
their score of a joke as an example and
2:53:10
so once we have a simulator we're often
2:53:12
racist because we can query it as many
2:53:14
times as we want to and it's all whole
2:53:16
automatic process and we can now do
2:53:18
reinforcement learning with respect to
2:53:19
the simulator and the simulator as you
2:53:21
might expect is not going to be a
2:53:22
perfect human but if it's at least
2:53:24
statistically similar to human judgment
2:53:27
then you might expect that this will do
2:53:28
something and in practice indeed uh it
2:53:30
does so once we have a simulator we can
2:53:33
do RL and everything works great so let
2:53:35
me show you a cartoon diagram a little
2:53:37
bit of what this process looks like
2:53:39
although the details are not 100 like
2:53:41
super important it's just a core idea of
2:53:43
how this works so here I have a cartoon
2:53:44
diagram of a hypothetical example of
2:53:46
what training the reward model would
2:53:48
look like so we have a prompt like write
2:53:50
a joke about picans and then here we
2:53:52
have five separate roll outs so these
2:53:54
are all five different jokes just like
2:53:57
this one now the first thing we're going
2:53:59
to do is we are going to ask a human to
2:54:03
uh order these jokes from the best to
2:54:05
worst so this is uh so here this human
2:54:08
thought that this joke is the best the
2:54:11
funniest so number one joke this is
2:54:14
number two joke number three joke four
2:54:17
and five so this is the worst joke
2:54:19
we're asking humans to order instead of
2:54:21
give scores directly because it's a bit
2:54:23
of an easier task it's easier for a
2:54:24
human to give an ordering than to give
2:54:26
precise scores now that is now the
2:54:29
supervision for the model so the human
2:54:31
has ordered them and that is kind of
2:54:33
like their contribution to the training
2:54:34
process but now separately what we're
2:54:36
going to do is we're going to ask a
2:54:38
reward model uh about its scoring of
2:54:41
these jokes now the reward model is a
2:54:43
whole separate neural network completely
2:54:45
separate neural net um and it's also
2:54:48
probably a transform
2:54:49
uh but it's not a language model in the
2:54:51
sense that it generates diverse language
2:54:53
Etc it's just a scoring model so the
2:54:57
reward model will take as an input The
2:54:59
Prompt number one and number two a
2:55:02
candidate joke so um those are the two
2:55:05
inputs that go into the reward model so
2:55:07
here for example the reward model would
2:55:09
be taken this prompt and this joke now
2:55:12
the output of a reward model is a single
2:55:14
number and this number is thought of as
2:55:16
a score and it can range for example
2:55:18
from Z to one so zero would be the worst
2:55:21
score and one would be the best score so
2:55:24
here are some examples of what a
2:55:25
hypothetical reward model at some stage
2:55:27
in the training process would give uh s
2:55:29
scoring to these jokes so 0.1 is a very
2:55:33
low score 08 is a really high score and
2:55:36
so on and so now um we compare the
2:55:41
scores given by the reward model with uh
2:55:43
the ordering given by the human and
2:55:45
there's a precise mathematical way to
2:55:47
actually calculate this uh basically set
2:55:49
up a loss function and calculate a kind
2:55:52
of like a correspondence here and uh
2:55:54
update a model based on it but I just
2:55:56
want to give you the intuition which is
2:55:58
that as an example here for this second
2:56:01
joke the the human thought that it was
2:56:02
the funniest and the model kind of
2:56:04
agreed right 08 is a relatively high
2:56:06
score but this score should have been
2:56:07
even higher right so after an update we
2:56:11
would expect that maybe this score
2:56:12
should have been will actually grow
2:56:14
after an update of the network to be
2:56:15
like say 081 or
2:56:17
something um for this one here they
2:56:19
actually are in a massive disagreement
2:56:21
because the human thought that this was
2:56:22
number two but here the the score is
2:56:25
only 0.1 and so this score needs to be
2:56:28
much higher so after an update on top of
2:56:31
this um kind of a supervision this might
2:56:34
grow a lot more like maybe it's 0.15 or
2:56:35
something like
2:56:36
that um and then here the human thought
2:56:40
that this one was the worst joke but
2:56:42
here the model actually gave it a fairly
2:56:44
High number so you might expect that
2:56:45
after the update uh this would come down
2:56:48
to maybe 3 3.5 or something like that so
2:56:50
basically we're doing what we did before
2:56:52
we're slightly nudging the predictions
2:56:55
from the models using a neural network
2:56:57
training
2:56:58
process and we're trying to make the
2:57:01
reward model scores be consistent with
2:57:03
human
2:57:04
ordering and so um as we update the
2:57:07
reward model on human data it becomes
2:57:10
better and better simulator of the
2:57:12
scores and orders uh that humans provide
2:57:15
and then becomes kind of like the the
2:57:17
neural the simulator of human
2:57:19
preferences which we can then do RL
2:57:21
against but critically we're not asking
2:57:23
humans one billion times to look at a
2:57:25
joke we're maybe looking at th000
2:57:27
prompts and five roll outs each so maybe
2:57:29
5,000 jokes that humans have to look at
2:57:31
in total and they just give the ordering
2:57:33
and then we're training the model to be
2:57:34
consistent with that ordering and I'm
2:57:36
skipping over the mathematical details
2:57:39
but I just want you to understand a high
2:57:40
level idea that uh this reward model is
2:57:43
do is basically giving us this scour and
2:57:45
we have a way of training it to be
2:57:47
consistent with human orderings
2:57:49
and that's how rhf works okay so that is
2:57:51
the rough idea we basically train
2:57:53
simulators of humans and RL with respect
2:57:56
to those
2:57:57
simulators now I want to talk about
2:57:59
first the upside of reinforcement
2:58:01
learning from Human
2:58:04
feedback the first thing is that this
2:58:06
allows us to run reinforcement learning
2:58:08
which we know is incredibly powerful
2:58:09
kind of set of techniques and it allows
2:58:11
us to do it in arbitrary domains and
2:58:13
including the ones that are unverifiable
2:58:16
so things like summarization and poem
2:58:18
writing joke writing or any other
2:58:19
creative writing really uh in domains
2:58:21
outside of math and code
2:58:23
Etc now empirically what we see when we
2:58:26
actually apply rhf is that this is a way
2:58:28
to improve the performance of the model
2:58:31
and uh I have a top answer for why that
2:58:34
might be but I don't actually know that
2:58:36
it is like super well established on
2:58:38
like why this is you can empirically
2:58:40
observe that when you do rhf correctly
2:58:42
the models you get are just like a
2:58:43
little bit better um but as to why is I
2:58:46
think like not as clear so here's my
2:58:47
best guess my best guess is that this is
2:58:50
possibly mostly due to the discriminator
2:58:52
generator
2:58:53
Gap what that means is that in many
2:58:56
cases it is significantly easier to
2:58:58
discriminate than to generate for humans
2:59:01
so in particular an example of this is
2:59:05
um in when we do supervised fine-tuning
2:59:08
right
2:59:09
sft we're asking humans to generate the
2:59:12
ideal assistant response and in many
2:59:15
cases here um as I've shown it uh the
2:59:19
ideal response is very simple to write
2:59:20
but in many cases might not be so for
2:59:22
example in summarization or poem writing
2:59:24
or joke writing like how are you as a
2:59:26
human assist as a human labeler um
2:59:29
supposed to give the ideal response in
2:59:31
these cases it requires creative human
2:59:33
writing to do that and so rhf kind of
2:59:36
sidesteps this because we get um we get
2:59:39
to ask people a significantly easier
2:59:41
question as a data labelers they're not
2:59:43
asked to write poems directly they're
2:59:45
just given five poems from the model and
2:59:47
they're just asked to order them and so
2:59:49
that's just a much easier task for a
2:59:51
human labeler to do and so what I think
2:59:54
this allows you to do basically is it um
2:59:57
it kind of like allows a lot more higher
3:00:00
accuracy data because we're not asking
3:00:02
people to do the generation task which
3:00:04
can be extremely difficult like we're
3:00:06
not asking them to do creative writing
3:00:08
we're just trying to get them to
3:00:09
distinguish between creative writings
3:00:11
and uh find the ones that are best and
3:00:14
that is the signal that humans are
3:00:16
providing just the ordering and that is
3:00:18
their input into the system and then the
3:00:20
system in rhf just discovers the kinds
3:00:23
of responses that would be graded well
3:00:26
by humans and so that step of
3:00:29
indirection allows the models to become
3:00:31
a bit better so that is the upside of
3:00:34
our LF it allows us to run RL it
3:00:36
empirically results in better models and
3:00:38
it allows uh people to contribute their
3:00:40
supervision uh even without having to do
3:00:42
extremely difficult tasks um in the case
3:00:45
of writing ideal responses unfortunately
3:00:48
our HF also comes with significant
3:00:50
downsides and so um the main one is that
3:00:54
basically we are doing reinforcement
3:00:56
learning not with respect to humans and
3:00:58
actual human judgment but with respect
3:00:59
to a lossy simulation of humans right
3:01:02
and this lossy simulation could be
3:01:03
misleading because it's just a it's just
3:01:05
a simulation right it's just a language
3:01:07
model that's kind of outputting scores
3:01:09
and it might not perfectly reflect the
3:01:11
opinion of an actual human with an
3:01:13
actual brain in all the possible
3:01:15
different cases so that's number one
3:01:17
which is actually something even more
3:01:19
subtle and devious going on that uh
3:01:21
really
3:01:22
dramatically holds back our LF as a
3:01:25
technique that we can really scale to
3:01:28
significantly um kind of Smart Systems
3:01:31
and that is that reinforcement learning
3:01:33
is extremely good at discovering a way
3:01:35
to game the model to game the simulation
3:01:38
so this reward model that we're
3:01:40
constructing here that gives the course
3:01:43
these models are Transformers these
3:01:46
Transformers are massive neurals they
3:01:48
have billions of parameters and they
3:01:50
imitate humans but they do so in a kind
3:01:52
of like a simulation way now the problem
3:01:54
is that these are massive complicated
3:01:56
systems right there's a billion
3:01:57
parameters here that are outputting a
3:01:59
single
3:02:00
score it turns out that there are ways
3:02:03
to gain these models you can find kinds
3:02:06
of inputs that were not part of their
3:02:08
training set and these inputs
3:02:11
inexplicably get very high scores but in
3:02:14
a fake way so very often what you find
3:02:17
if you run our lch for very long so for
3:02:19
example if we do 1,000 updates which is
3:02:21
like say a lot of updates you might
3:02:24
expect that your jokes are getting
3:02:25
better and that you're getting like real
3:02:27
bangers about Pelicans but that's not
3:02:29
EXA exactly what happens what happens is
3:02:32
that uh in the first few hundred steps
3:02:34
the jokes about Pelicans are probably
3:02:35
improving a little bit and then they
3:02:37
actually dramatically fall off the cliff
3:02:39
and you start to get extremely
3:02:40
nonsensical results like for example you
3:02:43
start to get um the top joke about
3:02:45
Pelicans starts to be the
3:02:48
and this makes no sense right like when
3:02:49
you look at it why should this be a top
3:02:51
joke but when you take the the and you
3:02:54
plug it into your reward model you'd
3:02:56
expect score of zero but actually the
3:02:57
reward model loves this as a joke it
3:03:00
will tell you that the the the theth is
3:03:03
a score of 1. Z this is a top joke and
3:03:06
this makes no sense right but it's
3:03:08
because these models are just
3:03:09
simulations of humans and they're
3:03:11
massive neural lots and you can find
3:03:12
inputs at the bottom that kind of like
3:03:15
get into the part of the input space
3:03:17
that kind of gives you nonsensical
3:03:18
results these examples are what's called
3:03:20
adversarial examples and I'm not going
3:03:22
to go into the topic too much but these
3:03:24
are adversarial inputs to the model they
3:03:26
are specific little inputs that kind of
3:03:29
go between the nooks and crannies of the
3:03:31
model and give nonsensical results at
3:03:33
the top now here's what you might
3:03:35
imagine doing you say okay the the the
3:03:37
is obviously not score of one um it's
3:03:39
obviously a low score so let's take the
3:03:41
the the the the let's add it to the data
3:03:43
set and give it an ordering that is
3:03:46
extremely bad like a score of five and
3:03:48
indeed your model will learn that the D
3:03:50
should have a very low score and it will
3:03:52
give it score of zero the problem is
3:03:54
that there will always be basically
3:03:55
infinite number of nonsensical
3:03:58
adversarial examples hiding in the model
3:04:01
if you iterate this process many times
3:04:02
and you keep adding nonsensical stuff to
3:04:04
your reward model and giving it very low
3:04:06
scores you can you'll never win the game
3:04:09
uh you can do this many many rounds and
3:04:11
reinforcement learning if you run it
3:04:13
long enough will always find a way to
3:04:14
gain the model it will discover
3:04:16
adversarial examples it will get get
3:04:18
really high scores uh with nonsensical
3:04:21
results and fundamentally this is
3:04:23
because our scoring function is a giant
3:04:26
neural nut and RL is extremely good at
3:04:29
finding just the ways to trick it uh so
3:04:34
long story short you always run rhf put
3:04:37
for maybe a few hundred updates the
3:04:39
model is getting better and then you
3:04:40
have to crop it and you are done you
3:04:42
can't run too much against this reward
3:04:46
model because the optimization will
3:04:48
start to game it and you basically crop
3:04:50
it and you call it and you ship it um
3:04:54
and uh you can improve the reward model
3:04:56
but you kind of like come across these
3:04:58
situations eventually at some point so
3:05:01
rhf basically what I usually say is that
3:05:04
RF is not RL and what I mean by that is
3:05:07
I mean RF is RL obviously but it's not
3:05:09
RL in the magical sense this is not RL
3:05:12
that you can run
3:05:14
indefinitely these kinds of problems
3:05:16
like where you are getting con correct
3:05:18
answer you cannot gain this as easily
3:05:21
you either got the correct answer or you
3:05:22
didn't and the scoring function is much
3:05:24
much simpler you're just looking at the
3:05:25
boxed area and seeing if the result is
3:05:27
correct so it's very difficult to gain
3:05:30
these functions but uh gaming a reward
3:05:32
model is possible now in these
3:05:34
verifiable domains you can run RL
3:05:36
indefinitely you could run for tens of
3:05:39
thousands hundreds of thousands of steps
3:05:40
and discover all kinds of really crazy
3:05:42
strategies that we might not even ever
3:05:44
think about of Performing really well
3:05:46
for all these problems in the game of Go
3:05:49
there's no way to to beat to basically
3:05:51
game uh the winning of a game or the
3:05:53
losing of a game we have a perfect
3:05:55
simulator we know all the different uh
3:05:58
where all the stones are placed and we
3:05:59
can calculate uh whether someone has won
3:06:01
or not there's no way to gain that and
3:06:04
so you can do RL indefinitely and you
3:06:06
can eventually be beat even leol but
3:06:09
with models like this which are gameable
3:06:12
you cannot repeat this process
3:06:14
indefinitely so I kind of see rhf as not
3:06:17
real RL because the reward function is
3:06:19
gameable so it's kind of more like in
3:06:21
the realm of like little fine-tuning
3:06:23
it's a little it's a little Improvement
3:06:26
but it's not something that is
3:06:27
fundamentally set up correctly where you
3:06:30
can insert more compute run for longer
3:06:32
and get much better and magical results
3:06:35
so it's it's uh it's not RL in that
3:06:37
sense it's not RL in the sense that it
3:06:39
lacks magic um it can find you in your
3:06:41
model and get a better performance and
3:06:44
indeed if we go back to chat GPT the GPT
3:06:47
40 model has gone through rhf because it
3:06:50
works well but it's just not RL in the
3:06:53
same sense rlf is like a little fine
3:06:55
tune that slightly improves your model
3:06:56
is maybe like the way I would think
3:06:58
about it okay so that's most of the
3:07:00
technical content that I wanted to cover
3:07:02
I took you through the three major
3:07:03
stages and paradigms of training these
3:07:05
models pre-training supervised fine
3:07:07
tuning and reinforcement learning and I
3:07:09
showed you that they Loosely correspond
3:07:11
to the process we already use for
3:07:13
teaching children and so in particular
3:07:15
we talked about pre-training being sort
3:07:17
of like the basic knowledge acquisition
3:07:19
of reading Exposition supervised fine
3:07:21
tuning being the process of looking at
3:07:23
lots and lots of worked examples and
3:07:25
imitating experts and practice problems
3:07:29
the only difference is that we now have
3:07:30
to effectively write textbooks for llms
3:07:33
and AIS across all the disciplines of
3:07:35
human knowledge and also in all the
3:07:37
cases where we actually would like them
3:07:39
to work like code and math and you know
3:07:43
basically all the other disciplines so
3:07:44
we're in the process of writing
3:07:45
textbooks for them refining all the
3:07:48
algorithms that I've presented on the
3:07:49
high level and then of course doing a
3:07:51
really really good job at the execution
3:07:53
of training these models at scale and
3:07:55
efficiently so in particular I didn't go
3:07:57
into too many details but these are
3:07:59
extremely large and complicated
3:08:01
distributed uh sort of
3:08:04
um jobs that have to run over tens of
3:08:07
thousands or even hundreds of thousands
3:08:08
of gpus and the engineering that goes
3:08:11
into this is really at the stateof the
3:08:13
art of what's possible with computers at
3:08:14
that scale so I didn't cover that aspect
3:08:18
too much
3:08:19
but um this is very kind of serious and
3:08:23
they were underlying all these very
3:08:24
simple algorithms
3:08:26
ultimately now I also talked about sort
3:08:29
of like the theory of mind a little bit
3:08:30
of these models and the thing I want you
3:08:32
to take away is that these models are
3:08:34
really good but they're extremely useful
3:08:35
as tools for your work you shouldn't uh
3:08:38
sort of trust them fully and I showed
3:08:40
you some examples of that even though we
3:08:41
have mitigations for hallucinations the
3:08:43
models are not perfect and they will
3:08:45
hallucinate still it's gotten better
3:08:47
over time and it will continue to get
3:08:48
better but they can
3:08:50
hallucinate in other words in in
3:08:52
addition to that I covered kind of like
3:08:54
what I call the Swiss cheese uh sort of
3:08:56
model of llm capabilities that you
3:08:58
should have in your mind the models are
3:08:59
incredibly good across so many different
3:09:01
disciplines but then fail randomly
3:09:03
almost in some unique cases so for
3:09:05
example what is bigger 9.11 or 9.9 like
3:09:08
the model doesn't know but
3:09:09
simultaneously it can turn around and
3:09:11
solve Olympiad questions and so this is
3:09:14
a hole in the Swiss cheese and there are
3:09:16
many of them and you don't want to trip
3:09:18
over them so don't um treat these models
3:09:22
as infallible models check their work
3:09:24
use them as tools use them for
3:09:26
inspiration use them for the first draft
3:09:28
but uh work with them as tools and be
3:09:30
ultimately respons responsible for the
3:09:33
you know product of your
3:09:35
work and that's roughly what I wanted to
3:09:39
talk about this is how they're trained
3:09:41
and this is what they are let's now turn
3:09:43
to what are some of the future
3:09:44
capabilities of these models uh probably
3:09:47
what's coming down the pipe and also
3:09:48
where can you find these models I have a
3:09:50
few blow points on some of the things
3:09:51
that you can expect coming down the pipe
3:09:53
the first thing you'll notice is that
3:09:55
the models will very rapidly become
3:09:57
multimodal everything I talked about
3:09:59
above concerned text but very soon we'll
3:10:01
have llms that can not just handle text
3:10:04
but they can also operate natively and
3:10:06
very easily over audio so they can hear
3:10:08
and speak and also images so they can
3:10:10
see and paint and we're already seeing
3:10:13
the beginnings of all of this uh but
3:10:15
this will be all done natively inside
3:10:17
inside the language model and this will
3:10:19
enable kind of like natural
3:10:21
conversations and roughly speaking the
3:10:22
reason that this is actually no
3:10:24
different from everything we've covered
3:10:25
above is that as a baseline you can
3:10:28
tokenize audio and images and apply the
3:10:31
exact same approaches of everything that
3:10:33
we've talked about above so it's not a
3:10:35
fundamental change it's just uh it's
3:10:36
just a to we have to add some tokens so
3:10:39
as an example for tokenizing audio we
3:10:41
can look at slices of the spectrogram of
3:10:43
the audio signal and we can tokenize
3:10:46
that and just add more tokens that
3:10:48
suddenly represent audio and just add
3:10:50
them into the context windows and train
3:10:51
on them just like above the same for
3:10:53
images we can use patches and we can
3:10:56
separately tokenize patches and then
3:10:59
what is an image an image is just a
3:11:01
sequence of tokens and this actually
3:11:03
kind of works and there's a lot of early
3:11:05
work in this direction and so we can
3:11:07
just create streams of tokens that are
3:11:09
representing audio images as well as
3:11:10
text and interpers them and handle them
3:11:12
all simultaneously in a single model so
3:11:15
that's one example of multimodality
3:11:17
uh second something that people are very
3:11:19
interested in
3:11:20
is currently most of the work is that
3:11:22
we're handing individual tasks to the
3:11:24
models on kind of like a silver platter
3:11:26
like please solve this task for me and
3:11:28
the model sort of like does this little
3:11:29
task but it's up to us to still sort of
3:11:32
like organize a coherent execution of
3:11:35
tasks to perform jobs and the models are
3:11:38
not yet at the capability required to do
3:11:41
this in a coherent error correcting way
3:11:44
over long periods of time so they're not
3:11:47
able to fully string together tasks to
3:11:49
perform these longer running jobs but
3:11:51
they're getting there and this is
3:11:52
improving uh over time but uh probably
3:11:55
what's going to happen here is we're
3:11:56
going to start to see what's called
3:11:57
agents which perform tasks over time and
3:12:00
you you supervise them and you watch
3:12:03
their work and they come up to once in a
3:12:05
while report progress and so on so we're
3:12:07
going to see more long running agents uh
3:12:10
tasks that don't just take you know a
3:12:11
few seconds of response but many tens of
3:12:13
seconds or even minutes or hours over
3:12:15
time uh but these uh models are not
3:12:18
infallible as we talked about above so
3:12:20
all of this will require supervision so
3:12:22
for example in factories people talk
3:12:23
about the human to robot ratio uh for
3:12:26
automation I think we're going to see
3:12:28
something similar in the digital space
3:12:30
where we are going to be talking about
3:12:31
human to agent ratios where humans
3:12:33
becomes a lot more supervisors of agent
3:12:36
tasks um in the digital
3:12:38
domain uh next um I think everything is
3:12:41
going to become a lot more pervasive and
3:12:43
invisible so it's kind of like
3:12:45
integrated into the tools and everywhere
3:12:49
um and in addition kind of like computer
3:12:51
using so right now these models aren't
3:12:54
able to take actions on your behalf but
3:12:56
I think this is a separate bullet point
3:12:59
um if you saw chpt launch the operator
3:13:02
then uh that's one early example of that
3:13:04
where you can actually hand off control
3:13:06
to the model to perform you know
3:13:08
keyboard and mouse actions on your
3:13:10
behalf so that's also something that
3:13:11
that I think is very interesting the
3:13:13
last point I have here is just a general
3:13:15
comment that there's still a lot of
3:13:16
research to potentially do in this
3:13:17
domain main one example of that uh is
3:13:20
something along the lines of test time
3:13:21
training so remember that everything
3:13:23
we've done above and that we talked
3:13:24
about has two major stages there's first
3:13:27
the training stage where we tune the
3:13:29
parameters of the model to perform the
3:13:30
tasks well once we get the parameters we
3:13:33
fix them and then we deploy the model
3:13:35
for inference from there the model is
3:13:38
fixed it doesn't change anymore it
3:13:40
doesn't learn from all the stuff that
3:13:41
it's doing a test time it's a fixed um
3:13:43
number of parameters and the only thing
3:13:45
that is changing is now the token inside
3:13:48
the context windows and so the only type
3:13:50
of learning or test time learning that
3:13:52
the model has access to is the in
3:13:54
context learning of its uh kind of like
3:13:57
uh dynamically adjustable context window
3:13:59
depending on like what it's doing at
3:14:00
test time so but I think this is still
3:14:03
different from humans who actually are
3:14:05
able to like actually learn uh depending
3:14:07
on what they're doing especially when
3:14:08
you sleep for example like your brain is
3:14:09
updating your parameters or something
3:14:11
like that right so there's no kind of
3:14:13
equivalent of that currently in these
3:14:14
models and tools so there's a lot of
3:14:17
like um more wonky ideas I think that
3:14:18
are to be explored still and uh in
3:14:21
particular I think this will be
3:14:22
necessary because the context window is
3:14:24
a finite and precious resource and
3:14:26
especially once we start to tackle very
3:14:28
long running multimodal tasks and we're
3:14:30
putting in videos and these token
3:14:32
windows will basically start to grow
3:14:34
extremely large like not thousands or
3:14:36
even hundreds of thousands but
3:14:38
significantly beyond that and the only
3:14:40
trick uh the only kind of trick we have
3:14:42
Avail to us right now is to make the
3:14:43
context Windows longer but I think that
3:14:46
that approach by itself will will not
3:14:47
will not scale to actual long running
3:14:50
tasks that are multimodal over time and
3:14:52
so I think new ideas are needed in some
3:14:54
of those disciplines um in some of those
3:14:57
kind of cases in the main where these
3:14:58
tasks are going to require very long
3:15:01
contexts so those are some examples of
3:15:03
some of the things you can um expect
3:15:05
coming down the pipe let's now turn to
3:15:07
where you can actually uh kind of keep
3:15:09
track of this progress and um you know
3:15:12
be up to date with the latest and grest
3:15:14
of what's happening in the field so I
3:15:15
would say the three resources that I
3:15:17
have consistently used to stay up to
3:15:19
date are number one El Marina uh so let
3:15:22
me show you El
3:15:24
Marina this is basically an llm leader
3:15:26
board and it ranks all the top models
3:15:30
and the ranking is based on human
3:15:32
comparisons so humans prompt these
3:15:34
models and they get to judge which one
3:15:36
gives a better answer they don't know
3:15:38
which model is which they're just
3:15:39
looking at which model is the better
3:15:40
answer and you can calculate a ranking
3:15:43
and then you get some results and so
3:15:45
what you can hear is what you can see
3:15:46
here is the different organizations like
3:15:48
Google Gemini for example that produce
3:15:50
these models when you click on any one
3:15:51
of these it takes you to the place where
3:15:54
that model is
3:15:55
hosted and then here we see Google is
3:15:57
currently on top with open AI right
3:15:59
behind here we see deep seek in position
3:16:02
number three now the reason this is a
3:16:04
big deal is the last column here you see
3:16:06
license deep seek is an MIT license
3:16:08
model it's open weights anyone can use
3:16:11
these weights uh anyone can download
3:16:13
them anyone can host their own version
3:16:15
of Deep seek and they can use it in what
3:16:17
whatever way they like and so it's not a
3:16:19
proprietary model that you don't have
3:16:20
access to it's it's basically an open
3:16:22
weight release and so this is kind of
3:16:25
unprecedented that a model this strong
3:16:27
was released with open weights so pretty
3:16:30
cool from the team next up we have a few
3:16:32
more models from Google and open Ai and
3:16:34
then when you continue to scroll down
3:16:35
you start to see some other Usual
3:16:37
Suspects so xai here anthropic with son
3:16:41
it uh here at number
3:16:43
14 and
3:16:46
um then
3:16:48
meta with llama over here so llama
3:16:51
similar to deep seek is an open weights
3:16:53
model and so uh but it's down here as
3:16:56
opposed to up here now I will say that
3:16:58
this leaderboard was really good for a
3:17:01
long time I do think that in the last
3:17:04
few months it's become a little bit
3:17:05
gamed um and I don't trust it as much as
3:17:08
I used to I think um just empirically I
3:17:12
feel like a lot of people for example
3:17:13
are using a Sonet from anthropic and
3:17:15
that it's a really good model so but
3:17:17
that's all the way down here um in
3:17:20
number 14 and conversely I think not as
3:17:22
many people are using Gemini but it's
3:17:24
racking really really high uh so I think
3:17:27
use this as a first pass uh but uh sort
3:17:30
of try out a few of the models for your
3:17:33
tasks and see which one performs better
3:17:36
the second thing that I would point to
3:17:37
is the uh AI news uh newsletter so AI
3:17:41
news is not very creatively named but it
3:17:43
is a very good newsletter produced by
3:17:45
swix and friends so thank you for
3:17:46
maintaining it
3:17:47
and it's been very helpful to me because
3:17:49
it is extremely comprehensive so if you
3:17:51
go to archives uh you see that it's
3:17:53
produced almost every other day and um
3:17:56
it is very comprehensive and some of it
3:17:58
is written by humans and curated by
3:18:00
humans but a lot of it is constructed
3:18:01
automatically with llms so you'll see
3:18:03
that these are very comprehensive and
3:18:05
you're probably not missing anything
3:18:06
major if you go through it of course
3:18:09
you're probably not going to go through
3:18:10
it because it's so long but I do think
3:18:12
that these summaries all the way up top
3:18:14
are quite good and I think have some
3:18:16
human oversight uh so this has been very
3:18:19
helpful to me and the last thing I would
3:18:20
point to is just X and Twitter uh a lot
3:18:23
of um AI happens on X and so I would
3:18:26
just follow people who you like and
3:18:27
trust and get all your latest and
3:18:30
greatest uh on X as well so those are
3:18:32
the major places that have worked for me
3:18:34
over time and finally a few words on
3:18:36
where you can find the models and where
3:18:38
can you use them so the first one I
3:18:39
would say is for any of the biggest
3:18:41
proprietary models you just have to go
3:18:43
to the website of that LM provider so
3:18:45
for example for open a that's uh chat
3:18:47
I believe actually works now uh so
3:18:49
that's for open
3:18:51
AI now for or you know for um for Gemini
3:18:55
I think it's gem. google.com or AI
3:18:58
Studio I think they have two for some
3:19:00
reason that I don't fly understand no
3:19:01
one does um for the open weights models
3:19:05
like deep SE CL Etc you have to go to
3:19:07
some kind of an inference provider of
3:19:08
LMS so my favorite one is together
3:19:10
together. a and I showed you that when
3:19:12
you go to the playground of together. a
3:19:14
then you can sort of pick lots of
3:19:16
different models and all of these are
3:19:17
open models of different types and you
3:19:19
can talk to them here as an
3:19:22
example um now if you'd like to use a
3:19:25
base model like um you know a base model
3:19:28
then this is where I think it's not as
3:19:30
common to find base models even on these
3:19:31
inference providers they are all
3:19:33
targeting assistants and chat and so I
3:19:36
think even here I can't I couldn't see
3:19:38
base models here so for base models I
3:19:40
usually go to hyperbolic because they
3:19:42
serve my llama 3.1 base and I love that
3:19:45
model and you can just talk to it here
3:19:47
so as far as I know this is this is a
3:19:49
good place for a base model and I wish
3:19:51
more people hosted base models because
3:19:53
they are useful and interesting to work
3:19:55
with in some cases finally you can also
3:19:58
take some of the models that are smaller
3:20:00
and you can run them locally and so for
3:20:02
example deep seek the biggest model
3:20:04
you're not going to be able to run
3:20:05
locally on your MacBook but there are
3:20:08
smaller versions of the deep seek model
3:20:09
that are what's called distilled and
3:20:11
then also you can run these models at
3:20:12
smaller Precision so not at the native
3:20:14
Precision of for example fp8 on deep
3:20:17
seek or you know bf16 llama but much
3:20:20
much lower than that um and don't worry
3:20:24
if you don't fully understand those
3:20:25
details but you can run smaller versions
3:20:27
that have been distilled and then at
3:20:28
even lower precision and then you can
3:20:30
fit them on your uh computer and so you
3:20:33
can actually run pretty okay models on
3:20:35
your laptop and my favorite I think
3:20:37
place I go to usually is LM studio uh
3:20:40
which is basically an app you can get
3:20:42
and I think it kind of actually looks
3:20:44
really ugly and it's I don't like that
3:20:45
it shows you all these models that are
3:20:47
basically not that useful like everyone
3:20:48
just wants to run deep seek so I don't
3:20:50
know why they give you these 500
3:20:51
different types of models they're really
3:20:53
complicated to search for and you have
3:20:54
to choose different distillations and
3:20:56
different uh precisions and it's all
3:20:58
really confusing but once you actually
3:21:00
understand how it works and that's a
3:21:01
whole separate video then you can
3:21:03
actually load up a model like here I
3:21:04
loaded up a llama 3 uh2 instruct 1
3:21:08
billion and um you can just talk to it
3:21:12
so I ask for Pelican jokes and I can ask
3:21:14
for another one and it gives me another
3:21:16
one Etc all of this that happens here is
3:21:19
locally on your computer so we're not
3:21:21
actually going to anywhere anyone else
3:21:22
this is running on the GPU on the
3:21:24
MacBook Pro so that's very nice and you
3:21:27
can then eject the model when you're
3:21:28
done and that frees up the ram so LM
3:21:31
studio is probably like my favorite one
3:21:33
even though I don't I think it's got a
3:21:34
lot of uiux issues and it's really
3:21:36
geared towards uh professionals almost
3:21:39
uh but if you watch some videos on
3:21:41
YouTube I think you can figure out how
3:21:42
to how to use this
3:21:43
interface uh so those are a few words on
3:21:46
where to find them so let me now loop
3:21:48
back around to where we started the
3:21:50
question was when we go to chashi
3:21:51
pta.com and we enter some kind of a
3:21:54
query and we hit go what exactly is
3:21:58
happening here what are we seeing what
3:22:00
are we talking to how does this work and
3:22:03
I hope that this video gave you some
3:22:05
appreciation for some of the under the
3:22:06
hood details of how these models are
3:22:08
trained and what this is that is coming
3:22:10
back so in particular we now know that
3:22:12
your query is taken and is first chopped
3:22:15
up into tokens so we go to to tick
3:22:18
tokenizer and here where is the place in
3:22:21
the in the um sort of format that is for
3:22:24
the user query we basically put in our
3:22:28
query right there so our query goes into
3:22:31
what we discussed here is the
3:22:33
conversation protocol format which is
3:22:35
this way that we maintain conversation
3:22:37
objects so this gets inserted there and
3:22:40
then this whole thing ends up being just
3:22:41
a token sequence a onedimensional token
3:22:43
sequence under the hood so Chachi PT saw
3:22:46
this token sequence and then when we hit
3:22:48
go it basically continues appending
3:22:51
tokens into this list it continues the
3:22:53
sequence it acts like a token
3:22:55
autocomplete so in particular it gave us
3:22:58
this response so we can basically just
3:23:00
put it here and we see the tokens that
3:23:02
it continued uh these are the tokens
3:23:04
that it continued with
3:23:06
roughly now the question
3:23:08
becomes okay why are these the tokens
3:23:11
that the model responded with what are
3:23:13
these tokens where are they coming from
3:23:15
uh what are we talking to and how do we
3:23:17
program this system and so that's where
3:23:19
we shifted gears and we talked about the
3:23:21
under thehood pieces of it so the first
3:23:24
stage of this process and there are
3:23:26
three stages is the pre-training stage
3:23:27
which fundamentally has to do with just
3:23:29
knowledge acquisition from the internet
3:23:31
into the parameters of this neural
3:23:33
network and so the neural net
3:23:35
internalizes a lot of Knowledge from the
3:23:37
internet but where the personality
3:23:39
really comes in is in the process of
3:23:41
supervised fine-tuning here and so what
3:23:45
what happens here is that basically the
3:23:47
a company like openai will curate a
3:23:49
large data set of conversations like say
3:23:51
1 million conversation across very
3:23:53
diverse topics and there will be
3:23:56
conversations between a human and an
3:23:58
assistant and even though there's a lot
3:23:59
of synthetic data generation used
3:24:01
throughout this entire process and a lot
3:24:03
of llm help and so on fundamentally this
3:24:05
is a human data curation task with lots
3:24:08
of humans involved and in particular
3:24:10
these humans are data labelers hired by
3:24:12
open AI who are given labeling
3:24:14
instructions that they learn and they
3:24:17
task is to create ideal assistant
3:24:18
responses for any arbitrary prompts so
3:24:21
they are teaching the neural network by
3:24:24
example how to respond to
3:24:27
prompts so what is the way to think
3:24:30
about what came back here like what is
3:24:33
this well I think the right way to think
3:24:35
about it is that this is the neural
3:24:37
network simulation of a data labeler at
3:24:41
openai so it's as if I gave this query
3:24:44
to a data Li open and this data labeler
3:24:47
first reads all of the labeling
3:24:49
instructions from open Ai and then
3:24:51
spends 2 hours writing up the ideal
3:24:53
assistant response to this query and uh
3:24:57
giving it to me now we're not actually
3:25:00
doing that right because we didn't wait
3:25:01
two hours so what we're getting here is
3:25:03
a neural network simulation of that
3:25:05
process and we have to keep in mind that
3:25:08
these neural networks don't function
3:25:10
like human brains do they are different
3:25:12
what's easy or hard for them is
3:25:14
different from what's easy or hard for
3:25:16
humans and so we really are just getting
3:25:18
a simulation so here I shown you this is
3:25:21
a token stream and this is fundamentally
3:25:23
the neural network with a bunch of
3:25:25
activations and neurons in between this
3:25:27
is a fixed mathematical expression that
3:25:29
mixes inputs from tokens with parameters
3:25:33
of the model and they get mixed up and
3:25:36
get you the next token in a sequence but
3:25:38
this is a finite amount of compute that
3:25:39
happens for every single token and so
3:25:42
this is some kind of a lossy simulation
3:25:44
of a human that is kind of like
3:25:46
restricted in this way and so whatever
3:25:49
the humans
3:25:50
write the language model is kind of
3:25:52
imitating on this token level with only
3:25:55
this this specific computation for every
3:25:58
single token and
3:26:01
sequence we also saw that as a result of
3:26:04
this and the cognitive differences the
3:26:06
models will suffer in a variety of ways
3:26:08
and uh you have to be very careful with
3:26:10
their use so for example we saw that
3:26:12
they will suffer from hallucinations and
3:26:14
they also we have the sense of a Swiss
3:26:17
model of the LM capabilities where
3:26:19
basically there's like holes in the
3:26:21
cheese sometimes the models will just
3:26:23
arbitrarily like do something dumb uh so
3:26:26
even though they're doing lots of
3:26:27
magical stuff sometimes they just can't
3:26:29
so maybe you're not giving them enough
3:26:31
tokens to think and maybe they're going
3:26:32
to just make stuff up because they're
3:26:34
mental arithmetic breaks uh maybe they
3:26:36
are suddenly unable to count number of
3:26:38
letters um or maybe they're unable to
3:26:41
tell you that 911 9.11 is smaller than
3:26:44
9.9 and it looks kind of dumb and so so
3:26:46
it's a Swiss cheese capability and we
3:26:48
have to be careful with that and we saw
3:26:50
the reasons for
3:26:51
that but fundamentally this is how we
3:26:54
think of what came back it's again a
3:26:56
simulation of this neural network of a
3:27:01
human data labeler following the
3:27:03
labeling instructions at open a so
3:27:06
that's what we're getting back now I do
3:27:09
think that the uh things change a little
3:27:11
bit when you actually go and reach for
3:27:14
one of the thinking models like o03 mini
3:27:18
and the reason for that is that GPT
3:27:20
40 basically doesn't do reinforcement
3:27:23
learning it does do rhf but I've told
3:27:26
you that rhf is not RL there's no
3:27:29
there's no uh time for magic in there
3:27:31
it's just a little bit of a fine-tuning
3:27:33
is the way to look at it but these
3:27:35
thinking models they do use RL so they
3:27:38
go through this third state stage of
3:27:41
perfecting their thinking process and
3:27:44
discovering new thinking strategies and
3:27:46
uh
3:27:47
solutions to problem solving that look a
3:27:50
little bit like your internal monologue
3:27:51
in your head and they practice that on a
3:27:53
large collection of practice problems
3:27:55
that companies like openi create and
3:27:57
curate and um then make available to the
3:28:01
LMS so when I come here and I talked to
3:28:03
a thinking model and I put in this
3:28:06
question what we're seeing here is not
3:28:08
anymore just the straightforward
3:28:09
simulation of a human data labeler like
3:28:11
this is actually kind of new unique and
3:28:14
interesting um and of course open is not
3:28:16
showing us the under thehood thinking
3:28:19
and the chains of thought that are
3:28:20
underlying the reasoning here but we
3:28:23
know that such a thing exists and this
3:28:25
is a summary of it and what we're
3:28:26
getting here is actually not just an
3:28:28
imitation of a human data labeler it's
3:28:30
actually something that is kind of new
3:28:31
and interesting and exciting in the
3:28:32
sense that it is a function of thinking
3:28:35
that was emergent in a simulation it's
3:28:37
not just imitating human data labeler it
3:28:39
comes from this reinforcement learning
3:28:41
process and so here we're of course not
3:28:43
giving it a chance to shine because this
3:28:45
is not a mathematical or a reasoning
3:28:47
problem this is just some kind of a sort
3:28:49
of creative writing problem roughly
3:28:51
speaking and I think it's um it's a a
3:28:55
question an open question as to whether
3:28:58
the thinking strategies that are
3:28:59
developed inside verifiable domains
3:29:02
transfer and are generalizable to other
3:29:06
domains that are unverifiable such as
3:29:08
create writing the extent to which that
3:29:10
transfer happens is unknown in the field
3:29:12
I would say so we're not sure if we are
3:29:14
able to do RL on everything that is very
3:29:16
verifiable and see the benefits of that
3:29:18
on things that are unverifiable like
3:29:20
this prompt so that's an open question
3:29:22
the other thing that's interesting is
3:29:24
that this reinforcement learning here is
3:29:26
still like way too new primordial and
3:29:29
nent so we're just seeing like the
3:29:32
beginnings of the hints of greatness uh
3:29:34
in the reasoning problems we're seeing
3:29:36
something that is in principle capable
3:29:38
of something like the equivalent of move
3:29:40
37 but not in the game of Go but in open
3:29:44
domain thinking and problem solving in
3:29:46
principle this Paradigm is capable of
3:29:49
doing something really cool new and
3:29:50
exciting something even that no human
3:29:53
has thought of before in principle these
3:29:55
models are capable of analogies no human
3:29:57
has had so I think it's incredibly
3:29:59
exciting that these models exist but
3:30:01
again it's very early and these are
3:30:02
primordial models for now um and they
3:30:05
will mostly shine in domains that are
3:30:07
verifiable like math en code Etc so very
3:30:10
interesting to play with and think about
3:30:12
and
3:30:13
use and then that's roughly it um um I
3:30:17
would say those are the broad Strokes of
3:30:18
what's available right now I will say
3:30:20
that overall it is an extremely exciting
3:30:23
time to be in the
3:30:24
field personally I use these models all
3:30:26
the time daily uh tens or hundreds of
3:30:29
times because they dramatically
3:30:30
accelerate my work I think a lot of
3:30:32
people see the same thing I think we're
3:30:33
going to see a huge amount of wealth
3:30:35
creation as a result of these models be
3:30:37
aware of some of their shortcomings even
3:30:41
with RL models they're going to suffer
3:30:42
from some of these use it as a tool in a
3:30:45
toolbox don't trust it fully because
3:30:48
they will randomly do dumb things they
3:30:50
will randomly hallucinate they will
3:30:51
randomly skip over some mental
3:30:53
arithmetic and not get it right um they
3:30:55
randomly can't count or something like
3:30:57
that so use them as tools in the toolbox
3:30:59
check their work and own the product of
3:31:01
your work but use them for inspiration
3:31:03
for first draft uh ask them questions
3:31:06
but always check and verify and you will
3:31:09
be very successful in your work if you
3:31:11
do so uh so I hope this video was useful
3:31:14
and interesting to you I hope you had it
3:31:15
fun and uh it's already like very long
3:31:18
so I apologize for that but I hope it
3:31:19
was useful and yeah I will see you later
Start a conversation
Ask anything about Deep Dive into LLMs like ChatGPT
100/100 free messages remaining
AI responses may be inaccurate. Verify important information.
Deep Dive into LLMs like ChatGPT
Andrej Karpathy