So in this lecture, we'll be looking at the notebook to implement latent semantic analysis.
We'll begin this notebook by downloading our data set, which is a list of book titles.
The next step is to import everything we need for this script.
The only thing new here is truncated served notice how this comes from the same decomposition module
as an EMF and LDA, which should give you some idea of where this fits in the universe of machine learning
methods.
The next step is to download the data we need for NCTC.
The next step is to create a word net limits or object.
Note that in this notebook, we'll be writing our own custom tokenize here, since the data set contains
specific terms will want to remove.
The next step is to read in our book titles, note that our file is structured so that each line consists
of one title.
The next step is to get an initial list of stop words.
Note that of course, this to a set since will want to add more stop words in the next step.
The next step is to add more software to our set of stop words.
As noted in the comment, this is an excellent example of where we need to do something.
Domain specific in this instance, because we're working with book titles, we don't care about words
like Introduction Edition, series, application approach and so forth.
Normally, these wouldn't be considered stop words, but because we're working with book titles, they
would be.
These words are too generic to be meaningful.
But they appear in many book titles.
The next step is to create our tokenize, their function as input we take in a string s, which represents
a book title.
The first step is to down case the text.
The next step is to tokenize the title into tokens.
The next step is to remove any tokens that are too short for examples of tokens that are too short.
I recommend turning this off and seeing what happens.
The next step is to clematis, the tokens.
The next step is to remove stalwarts.
The final step is to remove any tokens that contain digits.
These are similar to Stoppard's, but unlike, say, upwards, any number can appear anywhere.
So it's easier to simply remove any token that contains digits.
At this point, we can return the remaining list of tokens.
The next step is to create a count victories or object note that have said binary to true, which means
that we won't count the tokens, but instead just to know whether or not they appear.
So the output will only contain values of zero and one for the tokenize area will pass in the function
we just created.
The next step is to convert our list of titles into account Matrix.
The next step is to create an index toward mapping, which we'll need later in the script.
I've left a comment here showing you conceptually what we want to do.
Essentially, we want to create a list since a list has integers as the index ends any object as the
value.
We then loop through this vocabulary, which is a word to index mapping and then simply reverse this
mapping.
So the vocabulary has the word as the key and the index as the value.
But our index, the word mapping has the index as the key and the word is the value.
Luckily, we don't actually have to do all this work since there's a function called Get Feature Names
Out, which does what we want.
The next step is to transpose our account matrix, as you recall, by default, they count victimizer,
gives us documents by terms.
However, we want to find a word vectors, which means we want to reduce the dimensionality of the term
document matrix.
The next step is to perform as be on our term document matrix, which gives us back Z.
Now, the next step is to essentially create a scatterplot for Z.
Unfortunately, CoLab doesn't have interactive plots by default.
If you ran this on your local computer, you would be able to zoom in, move around the plot and so
forth.
So if you want to try that, please feel free to do so for us.
We're going to do everything in this notebook, so we're going to install a package called plot.
The next step is to import a plot leaked express, which has an API similar to a map plot lid.
The next step is to create our scatterplot.
Note that this is where we make use of our index toward mapping, as you recall, is just a list of
data points.
But in order to know what these data points actually mean, we have to actually draw the words on the
plot.
OK, so we see right away that our plot is very interesting.
There appears to be two directions where the data spreads.
On one side, which almost goes horizontal, we can see the words computer and science on the other
side, which almost goes vertical.
We can see the words history and art.
So this is very interesting because it reflects essentially the two sides of most college offerings.
Typically, the biggest differentiator of college programs is whether they are based on science and
technology or liberal arts.
And these two axes reflect this observation.
Now, the center is where most of the words reside.
Note that you can zoom into this plot by clicking and holding while dragging your mouse to highlight
the portion you want to see.
So please try this yourself and try to find different areas of interest because there are many possible
areas to look at.
I've chosen a select few to discuss for this lecture.
So if we zoom in just a bit, we can see more of the same pattern to the right, we have words like
business, statistic, biology, engineering, probability, Earth, political and so forth.
Again, reflecting science and tech at the top, we have words like American world, global modern,
again reflecting liberal arts.
So if we zoom in again, we see more of the same.
So the rate we see words like psychology, algorithm, mathematics, machine physiology, actuarial
java, environmental and more words related to science and tech.
At the top, we see words like culture, Buddhism, narrative, Judaism, Greek literature, religion,
feminism, diaspora, in other words, related to liberal arts.
If we zoom in even more, we see the same pattern yet again to the right, we see words like thermodynamics,
geology, finance, computational quantum security, communication and physics.
Again, all words related to science and tech.
Vertically, we see words like state, Islam, sex, sexuality, Christianity, Asian East Anthology
philosophy, in other words, related to liberal arts.
Now, I don't want to go through the whole list of words, but please look through this on your own
to see whether this pattern continues as you zoom in and more and more check whether or not the results
make sense.
And think about why this happened and how it's related to speed.