In this lecture, we are going to look at a CoLab notebook that does text classification using Arnon's,
the specific text classification we're going to work on is spam detection, which, as you can probably
surmise on your own, is a very practical application.
Note that there are some other examples of a text classification, such as sentiment analysis, where
you can try to guess whether a passage is positive or negative and also just document classification.
For example, if you're a new site and you have thousands of articles, you may want to automatically
classify them into different categories on your site, such as technology, local news, lifestyle,
health, weather and so forth.
So there are lots of basic applications of text classification.
This lecture is going to walk you through a prepared CoLab notebook, although a very good exercise,
which I always recommend, is once you know how this is done, to try and recreate it yourself with
as few references as possible.
As usual, you can look at the title of the notebook to determine what notebook we are currently looking
at.
To start, we're going to load in our data set, which is a spam detection data set stored as a CSV.
We use the get method to download the data first.
You'll notice that this is a very strangely formatted file.
If we use the head commands, you'll see that there appear to be some invalid characters.
In addition, each line ends with three commas, but none of those columns contain any data.
The next step is to load in our CCV using pedigreed CSFI notice that I've used in encoding called ISO
eight eight five nine, dash one.
The default is UTSA, which does not work on this file due to invalid characters.
The next step is to do a DFT head to see what our data looks like.
It seems that the first two columns are simply called V1 and V2, and there are three completely empty
columns after that.
This makes sense since each line of our file ended with three commas.
So in the next line, I'm going to drop these three unnecessary columns.
The next step is to do a deal, how to get.
And we see that the three columns are gone as expected.
Next, because I don't like having my columns named, if you wanted me to, I'm going to name them something
a little better labels and data.
The next step is to do it the head again to check that the columns have been successfully renamed.
So currently, the labels are called ham and spam, as you know, in machine learning, we like our
columns to be integers from zero up to K minus one.
In our case, since K is two, we want our labels to be zero one one.
We'll map ham to zero and spam to one will assign this new column to B B labels.
The next step is to split our data into train and test.
The next step is to check the shape of our data sets just as a sanity check.
So in the next few blocks of code, we're going to begin writing the code for text preprocessing.
We'll start by initializing our word to index dictionary.
Currently, this will only contain the padding token, which is assigned to have the index zero.
So that's the format of this dictionary.
The key is the word and the value is its corresponding index.
We also have a variable called X, which I've initialized to one.
Basically, this variable will store our current index as we loop through the train set.
So notice it's initialized to one because zero is already taken by the padding token.
But you'll see precisely how this word to index dictionary is used very shortly.
The next step is to live through each document in the train set, our goal right now is to populate
the word to index dictionary, that is, we want to store all the words in the train set and assign
them to all unique integer indices.
So inside the loop.
We begin by grabbing the data column, lower casing the string and then calling the split method.
This is the simplest way to do tokenization.
Lots of people think tokenization is some magical operation, but in reality it's just a fancy string
split.
There are other things you can do, like slip punctuation or completely remove punctuation, but we
won't consider those scenarios.
Of course, it's easy to replace this line with anything you like.
For example, the token icer from Analytica Spacy or Gentium, it's essentially no effort to do that.
So if you want to try that on your own, please do.
The next step is to loop through each of our tokens inside this loop.
We're going to check whether or not the token is currently in our word to index dictionary.
If this is not the case, then we need to create a new entry, otherwise we don't need to do anything.
So inside the statement, we simply add this token as a key and set IDEX to be the value.
We then increment IDEX to prepare it for the next token we find.
OK, so hopefully you can see how this loop will populate the word to IDEX dictionary with unique indices
assigned to each word in the training corpus.
The next step is to simply print out the word to index dictionary to confirm that it contains what we
expect.
OK, so hopefully this is what you expected to see.
The next step is to check the size of the word to a next dictionary to see how many unique tokens we
have.
So as you can see, we have about ten thousand unique tokens.
The next step, as you recall, is that we need to convert our text data into data that can be accepted
by a neural network.
This means we have to convert each document into a list of word indices in order to do this.
We'll start by creating an empty list called train sentences as int's, hopefully for obvious reasons.
The next step is to loop through the train said.
Once again, note that I've made a little comment here, which reminds you that we could have done this
step and the previous step all at once.
That is, we could have populated word to IDEX and converted the data into lists of integers at the
same time.
So you may want to try that as an exercise.
I think doing it separately makes things a little more clear.
OK, so inside the loop, we're going to read tokenized the current document so you can see how this
is a bit inefficient since we've tokenized every document twice.
Once we have the tokens, we then map them to integers using the word to IDEX dictionary after we get
the result, which I've called Centonze as int, we then append that to our list of train inputs.
The next step is to do the same kind of loop for the test set, but this time it's going to be a bit
different.
The main reason is because we can't be sure that every word that appears in the test set also appeared
in the train said one common way of dealing with this is to simply not include those words.
Another common method is to have a special token just for unknown words.
But then you would have to train.
Your neural network waits for unknown words as well.
In this case, we're just going to ignore any words that do not appear, in our words, to IDEX dictionary.
Other than that, this loop is the same as above.
OK, so the next step is to check the size of our train and test input's.
So as you can see, they are the same as before.
The next step is to create a data generator.
This is optional since theoretically you could just loop through the data structures we already have.
However, this makes the subsequent pocketwatch code more similar to what you've seen before.
So this function is going to take in three arguments, the inputs X, the targets Y and the batch size
inside the function.
We're going to start by shuffling the inputs and targets, as you recall, when we do batch gradient
descent.
We like to shuffle the data on kapok.
We then compute the number of batches, which is the ceiling of the size of Y, divided by the batch
size.
The reason we want to use the ceiling is because if they don't divide evenly, we still want to include
the leftover points.
The next step is to live through each of our batches.
Inside the loop, we're going to start by computing the end index of the batch, so normally this would
just be a plus one times batch size.
But remember, the final batch may not be a full batch.
So we need to account for that by taking the men of the usual value and the length of y.
The next step is to Index X and Y to obtain expatriate Y batch using the indices we just found.
OK, so please double check these indices and make sure they make sense to you.
If they do not make sense to you, then apply my rule when in doubt, print it out.
OK, so currently X matches a list of documents, each represented as a list of integers.
There's one problem with this, which is that each document in our dataset could have a different length.
In fact, they most likely do by default.
Arnon's don't work with sequences of variable length.
So unless you want to write some additional code to handle that, then the typical way to deal with
this is to make each sample in the batch have the same length.
In order to do this, we first have to find the longest sequence in the batch.
Of course, this is so that we can include all the data that appears in the batch, so all the shorter
sequences will have padding and the longest sequence will not have any padding.
OK, so Maslon is just the maximum length of the documents in the batch.
The next step is to live through each sample in the batch.
Inside this loop, we first grab the sample, which is called Little X.
So little X at this point is a list of integers representing the document.
As always, when in doubt, print it out.
OK, so the next step is to determine the amount of padding.
Of course, this is just Max Len minus the length of X. This is because after padding, we want all
the documents to have the length mcauslan.
So we multiply maxilla minus the length of X by a list containing zero, which, as you recall, doesn't
actually do multiplication.
When you use the asterisk operator with a list, it simply repeats the list that many times.
So what we get back is a list of zeros.
The final step is to join the padding and X together and store this in X batch.
So this will overwrite what was in X backstage at the start of this loop.
Now, once this inner loop is complete, X batch will be a non JAGAT array.
So at this point we can convert both X and Y back into towards Tensas.
The final step in this function is to yield the expansion Y batch.
As you recall, this is a generator.
So we want to yield and not return.
The next step is to test their new function.
This will also show you how our generator is to be used.
As expected, we use it in a loop and it takes in our inputs and labels as arguments inside the loop.
We're going to print out one input target pair along with their shapes, will then break out of the
loop since we only need to see one.
OK, so here you can see the input tensor, which is a 2D array of integers as expected, where most
of the rows just start with a bunch of zeros.
This makes sense since that's the padding.
We can see that the target sensor is just an array of zeros and ones with a size of thirty two as expected.
The next step is to do the same thing, but with our test data set, just so you can see how we can
loop through that data as well.
As you can see, it's pretty much the same thing.
The next step is to set the device to be the CPU.
Next, we define the model.
This is the same as the Arnon's we looked at earlier, but within embedding layer up front, so as input
we take in the vocab size, the embedding dimension, the number of hidden units, the number of hidden
layers and the number of outputs, the two relevant things for us in this section are, of course,
the vocab size V and the embedding dimensioned since that specifies the size of our embedding matrix.
Next, we create our three layers, the embedding, the l'estang and the final dense layer.
In the forward function, we pass our data through each of the layers, first we create our initial
ASTM states, H0 and C0.
Next, we pass our data through the embedding layer.
This gives us an NBA team I.D..
Next, we pass our data through the LSM layer.
This gives us an NBA team by M next we do a global max pool.
This collapses the T dimension and gives us an N by N.
Finally, we pass our data through the last dense layer and this gives us an end by K.
Next, we instantiate the answer and move the parameters to the GPU as before, I'm going to speed through
the usual boilerplate code which we've seen.
The next step is to create the laws and optimizer.
The next step is to create simple lambda functions that can be used to call our data generators for
training tests, you'll see that these shorter names are much easier to use throughout the notebook.
The next step is to define the training function.
The thing to pay attention to here is how we use the train Jan and testing lambda functions also note
that we explicitly move the inputs and targets to the GPU.
Other than that, you've essentially seen all this before.
The next step is to call the training function.
The next step is to plot the train and test Los per iteration.
OK, so the last iteration looks good.
The next step is to calculate the accuracy again, notice that we simply use our transgene and Tashjian
Lambda functions.
All right, so we end up doing pretty well in the high 90s, so conceivably, if you wanted to say create
your own Web server that processes some type of email or digital message, you can now filter spam very
accurately.
As usual, please take this opportunity to try different type of parameters yourself and see if you
can improve the results on your own.
So please try that as an exercise and I'll see you in the next lecture.