-: With tools like LM Studio, but also Ollama,
you can control how the loaded model generates tokens
and data for its output.
And specifically,
you can typically tweak the temperature,
and values called top_k, top_p, and sometimes also min_p,
but what are these terms about now?
The temperature setting can be used
to modify the probabilities that are assigned
to the different tokens or token candidates.
And to understand that,
it's important to understand
that these large language models generate tokens
or token candidates.
So if we, for example, have a sequence like the sky is,
then as a next token or word,
the model may generate blue, or clear,
or visible, or some other word.
And each token it considers
has a probability assigned to it.
Now,
it's this probability
and this token that is derived by all these weights,
all these parameters that make up the model.
So based on your input, the sky, is based on these tokens,
these token candidates,
and their probabilities are generated by the model.
And with temperature, top_k, top_p, and min_p,
you can configure which of these possible tokens
are actually considered for sampling.
So which tokens are actually considered to be chosen
for the actual output, so to say.
And with temperature, as mentioned,
you can modify these probabilities, for example.
To be precise, if you set a low temperature value,
zero, or 0.1, or something like this,
then differences in probabilities are exaggerated,
which means, for example, that blue,
which in my example here has a made-up probability of 45%
would become even more likely.
The probability might get boosted to 90
or 95% behind the scenes after the tokens were generated,
so these tweaks are made after the token candidates
have been generated by the model.
So a low temperature value will boost
the likely probabilities to be even more likely
and make the unlikely probabilities even less likely.
On the other hand,
a high temperature flattens those differences
and leads to more equal probabilities.
So again, in this example with a high temperature setting,
blue might go down from 45 to 20%,
but clear might go up to 20%, for example,
that's what the temperature setting does,
And we'll see it in action in the next lecture, of course,
but we don't just have the temperature.
We also, for example, have the top_k value,
which can be set.
And with top_k,
we can limit the number of candidates
that are considered at all.
So here, it's not about boosting probabilities,
it's simply about getting rid of some candidates.
For example, if you have a K value of five,
only the five most likely values are considered.
If it's one,
it's definitely the most likely value that will be used.
So if the top K parameter were set to one here,
we would definitely pick blue
because it's the most likely token
in this made up example here.
If I set top_k to two instead, for example,
blue and visible would be considered
because these are the two highest percentages.
The dot dot dot with 22%
is meant to represent multiple alternatives
where each alternative has less than 22%,
just to be clear, so that's top_k.
Top_p is about limiting the candidates based
on their combined probabilities.
If P is set to 0.5, for example,
all candidates would be considered
that combined have more than 50%.
With 0.9,
all candidates that have more than 90% would be considered.
So again, here, if top_p were set to 0.5,
blue and visible would be considered
because combined they have more than 50%,
and we don't need any other candidates to go above 50%.
If top_p were set to 90%,
we would need blue, visible, and clear,
but even then we would still be below 90%.
So we would also include some other candidates
from those remaining tokens.
And then from all these tokens
that are included in the actual candidates,
one candidate will be chosen,
and that will then be the actual token
that makes it into the actual output.
And by the way,
it's not like the highest probability then always wins.
Instead you can think of it as a weighted dice roll,
so the higher probability candidate
is more likely to be chosen,
but the less likely one also have a chance of being chosen.
That's how temperature, top_k, and top_p work,
and they can be combined.
You can exaggerate and flatten differences with temperature,
then restrict the number of candidates with top_k,
and then in addition also use top_p, for example.
There also is min_p.
And min_p can be used to discard certain candidates
that don't meet a minimum probability threshold.
So for example, if you set top_k to five,
and you therefore have five candidates,
you could in addition set min_p to 0.5
to remove all candidates
that have an individual probability of less than 5%.
So if one of your five candidates
would only have a probability of 4%,
it would be excluded with min_p set to 0.5,
even if it would be one of the five most likely ones,
that's the idea here.
Now, I will say that in reality,
you likely will not tweak these settings all the time.
The defaults are typically pretty good,
giving you a nice mixture of creativity, randomness,
and expected behavior you could say.
But for specific use cases,
you can definitely tweak these settings.
And we will play around with them to get a better feeling
for them in the next lecture.