Maximilian: The thing that should be loaded into memory
when performing inference are, in the end, all the weights,
the so-called parameters
that are associated to these connections
between these different neurons, these different nodes,
that make up the neural network
that was used for training this model.
So a large language model, in the end,
is such a neural network.
You could say it was trained as a neural network
with a certain training algorithm.
And as a result of this training process,
we got billions of parameters
because we got billions of connections.
And every connection has a parameter,
a weight associated to it,
that simply describes how a value is transformed
when traveling through these different nodes.
Now, what is the value I'm talking about?
It's, in the end, your prompt, which, to be precise,
is actually deconstructed into so-called tokens,
where every token has a token ID.
And there are tools out there,
you'll find a link to one attached to this lecture,
that show you how a given prompt for a given AI model
is translated to tokens and then to token IDs.
And it's these IDs that are actually used by the model,
you could say.
Now, technically, it's, of course,
a bit more complex than that,
but you can think of these token IDs
being fed into this neural network.
There, every connection has a weight.
And as these IDs are fed through that trained network
where the weights, of course, have been derived
based on the training data,
in the end as output, you get new token IDs,
which are then translated back to human-readable text.
And which is then, in the end,
the response that was generated by the model.
So these parameters, these weights are important.
And you might have heard the term parameter
in conjunction with large language models before.
You also might have heard
that some of the most capable large language models,
like DeepSeek R1, have hundreds of billions of parameters.
Now, the open models I'll cover in this course
are typically a bit smaller.
For example, the Gemma 3 model,
which is the one I will use primarily in this course,
"only," in quotes, has around 27 billion parameters.
But that still is quite a lot,
even if it is far less than the 600,
700 billion parameters DeepSeek R1 might have,
which, by the way, also is an open model.
You could run that locally as well too
if you have the right hardware for that.
Because, of course, this information about the parameters
brings it back to these hardware requirements.
Because I mentioned that the model
should be loaded entirely into VRAM,
or system memory if you don't have any VRAM.
And what I mean with loading the model
is loading all the model parameters
and then all the context, as I explained before.
So all these billions of parameters
must be loaded into your computer memory,
preferably into the VRAM, the memory of your GPU.
So how much memory is that then?
How much memory is required?
Well, every parameter, typically,
is a float32 or float16 value
after that initial training process.
Now, what does this mean?
These are simply data types used in computer science,
used in programming,
and specifically, they're data types that describe fractions
or fractional numbers,
numbers that have digits after the decimal place in the end.
Now, the difference between float32 and float16
is the precision of that number.
And you don't need to understand
the internals for this course, or in general,
for interacting with large language models,
but you can simply think of a float32 number
being capable of storing more information
about such a fractional number.
It's simply able to store longer fractions,
longer decimal numbers.
Float16 is less precise.
Now, internally, on your computer,
every such number must be stored in some space and memory
because the entire model should be loaded.
And here, a float32 value takes up four bytes,
float16 value takes up two bytes in memory.
So that's four or two bytes for every single parameter,
every single weight.
And therefore,
even if you had just a 2 billion parameter model,
which is really small.
Keep in mind, we are talking about 600
or 700 billion parameter models.
And I mentioned that for this course,
I would dive into using 27 billion parameter model.
So a 2 billion parameter model is really small,
but even that would require
four to eight gigabytes of memory
if you wanted to load it as is into memory,
or into VRAM preferably.
And that's, of course, absolutely doable for many computers.
But if you, again,
think of that 27 billion parameters model,
well, that would require 54,
or maybe even around 100 gigabytes of video RAM.
And most computers and most GPUs don't have that.
So that's a problem we need to solve.