Hello everyone and welcome.
In today's session we will learn about inference parameters.
We will be using inference parameters throughout the course.
However, it's also important that you learn about inference parameters, since these parameters would
help you regulate and manage the responses from the large language models.
So the very first and obvious question is what are inference parameters?
When running model inference, you can adjust the inference parameters to influence the model response.
Inference parameters can change the pool of possible outputs that the model considers during generation,
or they can also limit the final response.
The following categories of parameters are commonly found across different models randomness and diversity
and length.
Let's take a deep dive in each one of them and understand what they mean.
Randomness and diversity.
Randomness and diversity refer to the amount of variation in models.
Response for any given sequence model determines a probability distribution options for the next token
in the sequence.
You can control these factors by limiting or adjusting the distribution Foundation.
Models typically support the following parameters to control the randomness and diversity in the response
temperature.
Top k.
Top p.
There are other parameters as well.
However, these three parameters are more often used to control the variation in the model's response.
Before we take a deep dive into these parameters, let's first understand certain concepts.
Sampling.
In sampling, we have greedy sampling.
So let's say if you're ordering something from restaurant, greedy sampling would be equivalent to always
ordering the single most common or most popular dish on the menu.
If the most frequently ordered dish is the Caesar salad, greedy sampling would result in ordering.
I will have the Caesar salad all the time.
This relates to the language model using top k equals one and temperature equals one, where the model
always chooses the single most likely next word according to its probability distribution.
We'll come back to top k and temperature values later in this video.
The second sampling that we have to understand is random sampling.
If you're ordering in a restaurant, random sampling would be the equivalent of choosing your order
by literally pulling a menu item at random, with no regards to what type of dish it is, or whether
it even makes sense.
So you might have ordering something like I'll have a chicken fried steak soup, or I'll have a cheeseburger.
Completely random combinations that don't form a coherent dish or a meal.
This relates to language models using moderate values of top k, such as top k equals to 50, top k
equals to 100, along with a high temperature of 1.5 to 2.0.
With such settings, the model can generate creative and surprising outputs by sampling from the broad
set of potential networks.
So let's do a quick summary.
Greedy sampling leads to coherent but uncreative outputs like always ordering the single most popular
item.
Random sampling, on the other hand, enables maximum creativity, but outputs are incoherent, like
ordering.
By pulling any menu item out of the hat randomly.
The goal is doing these parameters to achieve a desired balance between coherence that is, sticking
to a conventional menu item, and creativity that is, ordering something new or unexpected, sometimes
for the specific use case.
So let's first understand the top k parameter.
Top k limits the model's output to the top k most probable tokens at each step.
This will help reduce incoherent or non cycle output by restricting the model's vocabulary.
Let's say for the prompt, I have the words in the vocabulary and the probabilities are matched with
probability of 0.6.
Couch.
The probability of 0.2 bed.
0.1 chair.
0.05 car 0.01 bikes 0.01.
Bucket 0.3.
With top k sampling, let's say k equals three.
It does the following.
It considers only top three highest probability words in the distribution after sorting them.
So if k equals three it only considers the words mat, bucket and bed since they have the highest probability.
Top p parameter.
Top p filters out tokens whose cumulative probability is less than a specified threshold, that is,
p.
It allows for more diversity in the output, but still avoiding low probability tokens.
Let's say for the prompt, I will have the words in the vocabulary and the probabilities are solid.
0.4 Berger 0.3 pasta 0.1.
Steak as 0.08.
When you specify top p parameter value as 0.8, it will include salad with 0.4.
Burger with 0.3.
Pasta with 0.1.
Since the cumulative probability of all three of them is 0.8, this covers 80% of the probability mass
in just top three words.
And it would drop steak, which is 0.08.
Let's recap top k and top p parameters.
Top p sampling with point p equals 0.8 will consider broader, more inclusive set of words compared
to using top k equals three with top k equals three.
The model only considers top three highest probability words after the context.
No matter how low the probabilities are with top p at 0.8, it will include as many words as needed
until the cumulative probability reaches 0.8.
Now let's go and understand the third inference parameter.
Temperature.
Temperature adjusts the randomness or confidence level of the model's prediction by scaling the log
probabilities.
Higher temperature leads to more diverse but potential nonsensical outputs when lower temperature leads
to more focused and predictable responses.
Let's say we specify a low temperature value of 0.2 or 0.5, makes the model more confident and peaks
the probability distribution, whereas a high temperature value of greater than one makes the model
Predictions more spreadable and uncertain.
Now that we learned about these parameters, let's apply the inference parameters.
Consider the example prompt.
I hear the hoof beats off.
The model determines the following three words to be the candidates for the next token, and the model
also assigns probability for each word.
Horse has 0.7 probability.
Zebras has 0.2, and unicorn has 0.1 probability.
So how does this all come together?
Let's say we set the temperature at 0.8.
Top K as two and top P as 0.7.
The very first step is if you set high temperature the probability distribution is flattened and the
probabilities become less different which would increase the The probability of choosing unicorn and
decrease the probability of choosing horse.
So it depends on what you want to select.
If you select 0.80.5, it would choose horse.
But if you choose a 1.5 or 2, the model will choose unicorn as the next possible token.
The next step it would consider is the top key equals two.
Filtering.
It selects the two candidates horses and zebras.
Top P equals 0.7.
Filtering is the third step.
It applies from this the top key equal to two.
It then applies top PSP0 point seven, filtering from the highest to lowest scale probability by keeping
the order until the cumulative probability mass, which is 0.7.
So in this case, the model only considers horse because it's the only candidate which has 0.7 probability.
However, if top p value was 0.9, then the model considers horses and zebras as the next probable tokens.
These were all the inference parameters that controls the model behavior.
The other category we have is length.
Foundation model typically supports parameters that limit the length of the responses.
An example of these parameters are provided below.
Response length and exact value.
To specify the minimum or maximum number of tokens to return in the generated response.
It really helps to keep check on the model responses from having excessive long responses, which reduces
or increases actually the latency of the generated responses.
The other parameter we have is top sequences.
Specify sequence of characters that stop the model from generating further tokens.
If the model generates a stop sequence that you specify, it will stop generating after that sequence,
a third length token is the penalties.
You can specify the degree to which you can penalize the outputs in a response.
Examples include length of the response, repeated tokens in the response, frequency of tokens in a
response, and the type of tokens in a response.
I hope you understand the inference parameters very very deep.
Take a deep dive here.
It's very important to go through this.
It will really help you understand how the model response generation can be governed with these parameters.