Maximilian: So here for this course,
just as in the previous section for LM Studio,
I will use the Gemma 3 model,
but you can of course play around with different models
and use all kinds of models.
Now, once you go to the model detail page of a given model,
you have a dropdown here which allows you
to choose from different flavors
and quantizations of that model.
All these identifiers here are called tags,
so a given model, Gemma 3, has different tags
that then point at different versions of that model.
Now, the obvious tags are, of course, the tags
that simply describe the different parameter sizes,
so the different amounts of parameters,
because the Gemma 3 model was published by Google
with different amounts of parameters,
but then we also got all these tags that refer
to different quantized versions of these models.
The confusing part,
however, is that also these versions here are quantized,
it's just not obvious from the tag names,
but by default, Ollama only has quantized versions
in their model catalog because you typically want
to run quantized versions on your system
for all the reasons you learned about
earlier in this course.
You essentially have pretty much no disadvantages
from using quantized versions,
but you get better performance
and lower hardware requirements.
So therefore, these versions of the model
that you find at the top here,
these tags also point at quantized versions.
You can just think of them
as the officially recommended versions,
because if you select one of these models,
let's say the 12 billion parameter model,
you find more information about it here, and guess what?
You also find quantization information here.
So that proves that this version of the model
with this tag also is quantized,
which is a good thing, of course.
You also find the size of the model here,
which is the amount of disk space this model will occupy
after being downloaded.
And you find some other pieces of information here,
which you can explore in greater detail,
like for example, the default configuration that's applied
to this model, for example here,
a quite high-temperature value for high creativity,
but we'll get back to configuring these models
and creating your own customized versions
of these models later in this section.
You also, most importantly,
find a command for running this version
of the model locally on your system.
So here, we find ollama run,
we already talked about this,
but then this here is the model identifier,
and the structure of this identifier is always the same.
It's essentially the model name, gemma3 in this case,
then a colon, and then the tag
of the specific model version you want to run,
so 12 billion in this case.
If you were to choose a different tag
because you want to force some other quantization technique,
for example, you could select that in the dropdown
and then simply use this tag here.
And here, indeed, I will use this version,
because for the Gemma 3 model specifically,
Google released some quantized versions
where they trained that version with quantization in mind.
So in theory, this version should perform a little better
than the quantized version
that was derived from the unquantized raw model,
though in reality,
it might not really make a big difference.
But here, I'll go for this version,
and I'll therefore copy this command
and go back to my command line to run it there.
Now, before I do that, though, I also wanna point
at that Readme, which you'll find on that model detail page
where you can in general learn more
about the model you're about to use,
for example, about the context window it has
and many other things, where in case of the Gemma models,
you learn about these quantization aware trained models
I just explained.
So that is indeed the model I am running here,
and where you can also learn more about the performance
of this model, how it did in various benchmarks, and so on.
This information is very similar
to what you would find on the detail page
of that model on Hugging Face.
So therefore, I will now run this command,
and what happens now is that as a first step,
the model is downloaded.
Of course, this doesn't happen
every time you run this command,
it just happens the first time
if you didn't download the model before.
By the way, just to also show you this command,
and for that, I stopped this command by pressing Ctrl+C,
besides downloading the model on demand,
when you run it the first time,
you can also use ollama pull
and then that model identifier
to just download it without running it.
Now, in many situations, you of course want to download
and then immediately run it,
but if you know that you will need that model
later that day, but not immediately,
or that you just wanna run it behind the scenes as a server,
something we'll explore later,
then you could also just download it
without running it by using all ollama pull.
Either way, once that command is done,
it will have been downloaded,
and then as a next step, you can run it with ollama run,
or if you used ollama run, it will start automatically.
And I'll be back once the download finished for me,
and now for me, that download process finished.
If you did use ollama run,
it will now already have started.
I, however, used ollama pull, so what I'll do here
to run it is I'll simply run ollama run again,
and it's that same command as before,
but now of course since it was downloaded,
it will not download it, but instead it will just run it.
And once it runs, you'll see something like this,
basically an arrow,
and then it's waiting for your input here,
because again, it doesn't come
with a graphical user interface.
Now, as you see, you can enter /? to get some help,
and that's not a bad idea
because that will tell you that there are a couple
of slash commands available
that you can run here in this mode.
So when it's waiting for your input,
you can set various parameters,
you can view information about the model that's running.
You could also save or load a session, a chat session,
essentially if you wanna save your work, quit the process
and come back to it later, for example.
You can clear your session context,
so clear the chat history if you wanna restart
without closing and restarting the program.
You can close the program by typing /bye,
and you can type this command
to view some helpful keyboard shortcuts
you might wanna use.
So what I'll do here is I'll just ask, hi, how are you?
Which is always kind of a weird question.
I mean, it's just a token generator, but still,
this allows me to show you that this is working
and up and running.
And that is indeed how we can use
an open Large Language Model with help of Ollama,
and if you wanna chat with it,
that is a great way of doing it.
You might not need a graphical user interface
like the one you get from LM Studio.
For that, you might indeed prefer
the more lightweight approach Ollama has.