So in this lecture, we will continue our discussion on class imbalance.
Now there's one final metric I want to discuss, which comes from the era of radar in the field of estimation
and detection.
This is where many of the statistical methods we use today were invented.
So one helpful plot that was used was the receiver operating characteristic or the rosy curve, this
curve is a plot of the true positive rate on the y axis in the false positive rate on the x axis.
Note that the false positive rate is just one minus the true negative rate.
So a typical RC curve will go up into the right and it will be steep on the left side while being more
flat on the right side.
So let's think about how this type of curve arises.
The main idea behind the RC is that your decision rule can be based on a threshold and where you define
that threshold to be will affect these true positive and false positive rates.
Now we haven't really discussed this yet, but there is typically a tradeoff between a specificity and
sensitivity in particular.
Suppose that our sensitivity is low.
That is, the number of true positives is low because we're not catching enough of the positives.
So perhaps your job is to detect enemy ships on your radar, then you're letting too many of them slip
by.
In other words, our detector is not sensitive enough.
In that case, we can simply make it more sensitive by lowering the threshold of detection.
Normally, we would choose the positive class if the probability of that class is bigger than 50 percent.
However, we don't have to use this as a threshold.
Instead, we could pick 40 percent or 30 percent.
By doing this, our classifier will make more positive detections, hopefully increasing the number
of true positives.
The tradeoff is that this won't be perfect.
We could end up having more false positives as well.
This will increase the false positive rate or, in other words, decrease the specificity.
So that brings us back to the Rosie.
Basically, the Rosie is formed by letting this threshold of detection go all the way from zero percent
up to 100 percent at zero percent.
We will predict that everything is positive, so our true positive rate will be one and our false positive
rate will also be one.
This is because we will always predict one, since our threshold is zero percent and any number output
by our model will be greater than zero percent, then our model will always predict one.
So we catch all the true positives or that enemy ships, but all the negatives will be predicted as
positive as well.
At 100 percent, a true positive rate will be zero in our false positive rate will also be zero.
In this case, we never predict positive, so we never detect any enemy ships.
But on the other hand, we never falsely identify something which is not an enemy ship as an enemy ship.
The interesting part of the Rowsey is its shape.
Typically, we draw a line from the bottom left corner up to the top right corner.
This line is the benchmark for random guessing.
That is to say this model would be very dumb and just output a random probability for every input sample.
As an exercise, I recommend thinking about why that would yield a straight line in the RC.
Now, with this line as our benchmark, we can now start to think about what kind of ROIC we actually
want to see.
Well, what we want to see, as you can probably tell, is a curve that goes to the top left of our
benchmark.
The more top left two can go, the better it is.
Let's think about why that is the case.
To understand this, let's think about the extreme case where the rosy shoots up to one immediately
and stays there.
What does this mean?
This means that no matter what my threshold is, my true positive rate is perfect.
Therefore, I can drive my false positive rate down to zero while still keeping my true positive rate
at one.
Furthermore, as you recall, a false positive rate of zero corresponds to a specificity of one.
Therefore, in this case, our classifier is perfect for both classes and does not make any mistakes.
Both the sensitivity and specificity are one, of course, in reality for most practical data sets.
The curve will be below this perfect case.
Now, here's something to think about.
We've been talking about metrics.
Our goal was to quantify the performance of a model with a single number.
But we seem to have made things worse by introducing a plot.
You have to look at.
We seem to have gone in the opposite direction as what we intended.
Of course, we are still not done.
In fact, we can get a single number from this plot.
The single number is called the AUC or the area under the curve.
As you might expect, it is simply the area underneath the rosy curves.
Note that both the height and the width of this box is one.
Therefore, the area of the box is also one.
Now, because the total area of the box is one, that means a perfect AUC is one.
It covers the whole area of the box.
Note that the benchmark splits the box in two and thus the benchmark for random guessing is zero point
five.
Thus, your models AUC is hopefully larger than zero point five, and ideally you want it to be close
to one.
So this is another alternative to the F1 score.
So here's one complication of the AUC method that makes it a bit more complex to use compared to the
F1 score in particular, this is that it requires your model to output probabilities.
Specifically, the model needs to output P of Y equals one given X.
This is assuming that your classes are assigned the values to zero and one, and that one is the positive
class, as you recall.
This is needed since the RC is formed by applying different thresholds from zero up to one.
So, for example, if the threshold is zero point three and the model outputs zero point four, then
we would assign that sample to the positive class.
But if the threshold were zero point five, then we would assign that same sample to the negative class.
So the model must be able to output this posterior probability.
Now it's worth discussing how one usually does this in the most popular Python library for machine learning,
which is I learn.
Note that if you're implementing models yourself, then this would be obvious.
But if you're using you learn it may not be obvious if you're not familiar with the API.
So it's worth reviewing how this works.
So normally when you call Model Dog, predict inside, you learn you get back in and length one dimensional
array.
Assuming that you have any input samples if you have classes in this array will contain integers from
zero up to K minus one.
So this is the typical way to get a prediction inside.
Get Learn.
The question is how do we get probabilities?
The answer is to call a different function, which is usually called predict problem.
Now keep in mind, it's like you learn as another library, which is part of the Nampai stack, which
means it is constantly changing from month to month and year to year.
So while this is the case today, it may not be the case forever.
In any case, what can we expect from this function?
Well, supposing again, that we have no samples in key classes, we will get back a two dimensional
array of size n by K. This time, the array values are not integers, but floating point numbers.
And furthermore, these floating point numbers are probabilities.
In particular, if we call the output p, then pink will be the posterior probability that the sample
belongs to Class K.
In other words, mathematically it is p of why seven equals K given exhibit.
Note that because of this convention, each row of this array will sum to one, since each row of the
matrix is a separate probability distribution.
Now, back to the binary case specifically, as you recall, we want to know the probability that the
class is equal to one in, since the output is binary.
There are only two columns in the Matrix.
In this case, we want the second column or the column at Index one, since the first column corresponds
to the probability for class zero.
So putting this all together here is how we would compute the AUC insight, could learn from a trained
binary model.
First, suppose we have our target array, which is called why this array contains only the integers
zero and one.
Suppose we have a corresponding input matrix called X.
What we need to do is get the predictions for X by calling model that predicts Prabha passing in X.
This will give us the posterior probabilities which we will call P.
As you recall, P is an end by two matrix.
Since we only want the probabilities for class one, we will grab the column at Index one.
Finally, we will pass Y in the column int index one for P into the function called RC AUC score.
This will then return a single number, which is the AUC.
Now, as a final side note, it's worth mentioning that not all classifiers inside you learn have a
function called predicts Prabha.
This is simply due to how certain classifiers work.
Some algorithms simply do not output probabilities.
One example of this is the perceptron.
Another example is the support vector machine, although there is an ad hoc method to obtain probabilities,
which has some issues.
In particular, the probabilities may not be consistent with the predictions.
And this method does not scale well for large datasets.
On the other hand, since neural networks naturally output probabilities, using the AUC for imbalanced
classes in deep learning is typically a fine choice.