So in this lecture, we will be discussing the concept of class imbalance in some typical methods for
dealing with that in machine learning.
So let's begin by discussing why class imbalance is a problem.
Suppose that you were developing a blood test for some disease.
Thus you were essentially building a binary classifier where the output is either a disease or no disease.
It should be noted that diseases by nature are generally rare in the population.
Otherwise, a large portion of people would be walking around in a disease state.
Suppose that for the disease in question, it is only present in 0.1 percent of the population and thus
ninety nine point nine percent do not have the disease.
Now, suppose that the classifier we build is very bad, such that it doesn't really do any computation
except for returning no disease for every input.
This illustrates the problem with using accuracy as a metric.
You see, even with this bad classifier, you would still end up getting ninety nine point nine percent
accuracy simply because that is the percentage of people who do not have the disease.
And this is assuming that the testing for this disease is indiscriminate.
That is, everyone in the population has an equal chance of being tested regardless of their symptoms.
This is probably not realistic, but let's just assume that is true.
So what can be done about this issue?
Well, we can start by looking at the results in a more detailed manner in order to understand where
we are going wrong.
In particular, there are four quantities we're interested in looking at instead of just one.
These are the true positives, the true negatives, the false positives and the false negatives.
So these should be pretty intuitive.
We first have to decide which class is positive and which is negative.
So for this example, let's assume that testing positive for the disease is equivalent to the positive
class.
In this case, the number of true positives is the number of people who have the disease that we correctly
predicted have the disease.
On the other hand, the number of false positives is the number of people we predicted to have the disease
but do not really have the disease.
Similarly, the number of true negatives is the number of people who do not have the disease, and we
correctly predicted that they do not have the disease.
And the number of false negatives is the number of people who we predicted to not have the disease but
really do have the disease.
Oh, and if you noticed we typically organized these numbers in a table, in particular, we call this
a confusion matrix.
Note that a confusion matrix is more general and can be extended to any number of classes.
However, the following techniques in this lecture will be focused on the binary case, although they
can be extended to the multiclass case.
They don't naturally fit into that picture.
And thus, for the rest of this lecture, let's assume that we're talking about binary classification.
So how can we make use of the numbers we've collected so far?
Well, historically there have been a few ways to do this, and whatever you use tends to depend on
what field you come from.
So if you're in the medical field or life sciences, you tend to report the sensitivity and specificity.
The sensitivity is the true positive rate.
It is equal to the number of true positives divided by the total number of positives.
Hence, true positive rate.
It is the rate at which we can detect those who are positive for the disease.
Another way to think of this is out of, however, many actual positives.
This is the percentage of those positives you'll be able to detect.
Note that in terms of the four counts we collected, it is equal to the number of true positives divided
by the number of true positives, plus the number of false negatives.
As you recall, the false negatives also have the disease.
The second item is the specificity.
This is the true negative rate, as you might be able to guess, it is equal to the number of true negatives
divided by the total number of negatives.
Hence, true negative rate if it's the rate at which we can detect those who are negative for the disease.
And again, you can derive this to be the number of true negatives divided by the number of true negatives,
plus the number of false positives.
Again, the false positives are among those who do not have the disease.
So let's think about what would happen to the sensitivity and specificity if we use our very bad classifier,
which always predicts no disease.
Suppose there are people in our study.
As mentioned, zero point zero zero one end of these people have the disease, while zero point nine
nine nine n do not.
Now, since we always predict no disease.
This means that we have zero point nine nine nine n a true negatives.
We also have zero point zero zero one n a false negatives.
We have no true positives because we never predict positive.
And we also have no false positives since we never predict positives.
Because of this, the sensitivity is equal to zero divided by zero plus zero point zero zero one N,
which is just zero.
This is because we have no true positives.
On the other hand, the specificity is equal to zero point ninety nine n, divided by zero point nine
nine and a plus zero, which is one.
This makes sense because we correctly predicted everyone who does not have the disease by always predicting
and no disease.
So how does this help?
Well, note that we now have two numbers to measure the performance of our predictor.
Previously, we only had accuracy, which was at ninety nine point nine percent, which seems good when
it is really not.
But by using two numbers, they both give us an indication of how the model is performing by looking
at the two classes separately.
In this case, the specificity is one which is a good sign, but the sensitivity is zero, which is
a bad sign if you measured the performance of your predictor and it had poor sensitivity.
You would immediately know that it is failing to detect the disease and thus, unlike accuracy, this
could be used as a measure of how useful your model would be in the real world to understand this.
Let's continue with our example of trying to predict whether or not a patient has some disease.
The sensitivity tells us how good we are at detecting the disease in people who actually have the disease.
Of course, we want this to be high so that we can provide treatment to these patients.
We would not want to leave them untreated, which is what a bad predictor would do by predicting no
disease for every patient.
On the other hand, we want our specificity to also be high.
As an extreme example, we don't want to tell everyone that they have the disease because they don't
actually need treatment.
Now, in practice, typically what happens is not that you treat the patient upon seeing a positive
test, but rather you do further tests to confirm the diagnosis.
At the same time, you don't want to scare patients by telling them that they have the disease when
they really do not.
Now it's worth thinking about what the sensitivity and specificity would be if our model made perfect
predictions.
In this case, the number of false positives is zero and the number of false negatives is also zero.
So the sensitivity is tip over TP plus F.N, which is equal to TP over TP, which is one.
Likewise, the specificity is 10 over 10 plus FP, which is equal to 10 over 10, which is also one.
Therefore, the true positive rate is one in the true negative rate would also be one.
Thus, what we are aiming for when we build a binary classifier is for both the sensitivity and specificity
to be one.
In the case of our very bad classifier, we got zero for the sensitivity, which is very bad.
Now, as mentioned, the metrics you use tend to be influenced by your field of study.
I personally find the sensitivity and specificity to make a lot of sense.
But in NLP, we tend to use precision and recall intuitively.
You can always remember what these mean because they are named for their use in document retrieval.
So let's start with recall.
The recall happens to be the same thing as the sensitivity that is the true positive rate in terms of
document retrieval or search engines.
It would be the number of relevant documents you found out of the total number of documents you should
have found.
In other words, it's the percentage of documents you were able to recall.
Now the precision is a little bit different.
This is also known as the positive predictive value.
It's equal to the ratio of true positives to true positives, plus false positives.
In other words, it's equal to the number of documents you correctly retrieved divided by the total
number of documents you retrieved.
So why does this make sense?
Well, suppose you built a very bad document retrieval system that just returned all the documents every
time.
Sure.
This would include the documents you wanted, but also tons of documents which you did not want.
Hence, it would be imprecise.
The number of false positives would be very high.
On the other hand, if you can reduce the number of false positives to zero and stop returning irrelevant
documents, then you would be very precise.
If the number of false positives is zero, then the precision would be equal to true positives over
true positives, which would be one.
Hence, this ratio is known as the precision.
How precise were you in returning relevant documents?
So we just discussed two methods of measuring the performance of a binary classifier when we have imbalanced
classes.
One option is to use sensitivity and specificity, while the other is to use precision and recall.
Note that there is one crucial way that these differ from accuracy, which is the typical performance
metric, and that is these give us back to numbers instead of just one.
The problem is this might make it too complicated when it comes to making decisions about which model
is best.
For example, suppose you're comparing two models, but one does better with precision, while the other
does better with recall.
How can you choose which model is best?
Well, one way to answer this question is to distill these metrics back into a single number.
So one method of getting a single number out of the precision and recall is to use the F1 score.
Essentially, the F1 score is the harmonica meaning of the precision and recall.
Now you don't have to worry about what that is, but I'll show you the formula in case you ever need
to implement it yourself.
Essentially, it's just two times the product of precision and recall, divided by the sum of precision
and recall.
So just think of the F1 score as a way to take the average of these two numbers.
Now, if you're curious, recall that the regular meeting is where you add up all the numbers and divide
by end, the harmonic mean is where you first invert all the numbers, then add together, then divide
by end and then invert them back.
So a simple way to think about it is that it's just another way of computing the mean.
So the F1 score is some kind of mean of the precision and recall.