So in this lecture, we'll be looking at the notebook for how to do topic modeling with LDA.
We'll begin this notebook by downloading our data set, which is the BBC news data once again.
Note that although this data set is reclassification and comes with labels, we won't be using those
labels since LDA is unsupervised.
In addition, a nice and simple exercise you can always try is to plug in a new data set into this notebook
and repeat the analysis.
The next step is to do our imports.
Note that Llda appears in the decomposition module, which is also the home of other methods like Piqué,
SVT and AMF and factor analysis.
So that should give you some idea of what a family of algorithms LDA belongs to.
Also, take notice that we are using the count of exerciser and not TFR yet.
This is because the model itself is based on word counts and not distributions based on Faria.
The next step is to download Stoppard's from Analytic.
The next step is to convert or stoppers list into a set since will be adding more storage in the next
step.
The next step is to add more words to our set of stop words.
Basically, I ran the script originally without these top words, but added them later when I saw them
appear in the topics.
So this is an example of how one would refine their code based on the results they see if you see generic
words in your output.
It's likely that they are not useful and so you can go back and simply remove them.
The next step is to load in our data using PD that reads ISV.
The next step is to call the after head to remind ourselves what this data looks like.
As you can see, the data set is a two column data frame of text and labels.
The next step is to create a town of riser passing in the stop words we previously defined.
The next step is to transform our text into account vector format.
The next block of code is a simple comment stating that you could potentially split the data into train
and test at this point and use the test set to evaluate the model.
Note that for LDA, common metrics are the log likelihood of the data or the perplexity, which is just
exponential of the negative log likelihood.
The next step is to create our Llda instance.
Note that I've chosen 10 components arbitrarily.
As usual, this can be optimized if you have some metric you care about, which is generally chosen
based on out of sample data.
For example, if you're doing classification, then you might want to use test accuracy to choose the
best end components.
Also note that I've set the random state since there's some randomness in the LDA algorithms.
The results will be different each time you run this setting, this random state will ensure that we
get the same results.
The next step is to fit the older model passing in our count matrix.
Note that the fitting process takes quite some time compared to other socket learned models.
Generally speaking, Bayesian learning and variational inference have the potential to be slower than
similar non Bayesian methods.
The next step is to define a function called plot.
Top words now the details of this function aren't that important.
So we won't be discussing what's in it.
This code is simply copied and pasted from the site.
You'll learn documentation.
And basically, it generates the plots I showed you in the previous lecture.
What's important to know is what it does, not how it works.
So basically, the job of this function is to make a bar plot for each topic.
As you recall, each topic is expressed in terms of words for each topic.
What we want is a bar plot showing the sorted word count for the top words pertaining to that topic.
In any case, this is much easier to see in a picture, so it will all make sense shortly.
Note that for simplicity, we'll plot just the top 10 words per topic, which, as you can see, we
have set as the default argument for any top words.
The next step is to call the plot top where it's function, which we just defined above.
Recall that the second argument corresponds to the feature names which are just column indices to the
LDA model.
What we really need is an index toward word mapping, which is just the reverse of the word to index
mapping, which you saw earlier in this course.
Now, since we didn't create the count matrix ourselves, we don't actually have this mapping.
However, it is stored in the count vector riser object.
Specifically, we can obtain the word to index next mapping by calling this weirdly named function and
get feature names out.
This will give us a list of feature names which we can then pass in to plot top words.
OK, so as you can see, we get a plot just like the one I showed you in the lecture.
What is nice about all this is that the results make a lot of sense.
For instance, the first topic consists of the words people, mobile technology, music and so forth.
This is clearly related to consumer technologies such as phones and digital music, which of course
are commonly played from phones.
The second topic has the words US big company, market firm, sales, growth, oil and so forth.
This is probably related to the economy, finance and the stock market.
The third topic has the words government, U.S. economic, economy, world and so forth.
Notice how the word at U.S. shows up again.
So for topics, this is acceptable since a single word can be associated with multiple topics.
This topic also seems to be related to the economy, but on a more global scale.
The fourth topic is the words TV people, broadband, search, video, web and so forth.
So this seems to be related to TV and internet services.
The fifth topic is the words Law Lord, Government Bill and so on.
So basically words related to the law in legal matters.
As an exercise, please go through topics six up to 10 now yourself.
You should find that these cover even more topics like film, music, software, politics and sports.
Also, if you recall the labels in this data set, you can relate these topics to the ground truth labels
as well.
In fact, topics are more fine grained the labels.
For example, both music and film would belong to entertainment, despite the fact that they should
likely be separate topics.
Now, we just got a chance to look at the topics, which, as you recall, is a property of the model.
The next step is to look at our documents to see how they relate to our topics.
As you recall, Llda essentially gives us two matrices.
One with documents by topics and another with topics by words.
We just saw topics by words.
So now we want to see documents by topics.
To get this matrix, we call transform onex.
And this gives us back Z.
As you recall, Z is the letter we commonly use in machine learning for hidden variables in clustering.
This would represent cluster IDs for topic modeling.
This represents a distribution over topics.
So the next step is to pick a random document and plot its distribution will begin by setting numbers
random see so that we get a consistent result.
The next step is to select a random row from our data frame, which we'll call I.
The next step is to grab the ith row of Z, which will give us a 1D array, which represents a distribution
over topics.
The next step is to define a list of topics which are just integers from one up to 10.
As you recall, topics are latent variables, which means they don't have any inherent meaning other
than what we assign to them.
The next step is to draw a bar chart plotting the topic distribution along with the topics themselves,
as well as the true label.
OK, so as you can see, essentially all of the probability goes to topic 10, the true label of sport.
So let's go back up to topic 10 and make sure that this makes sense.
OK, so Topic 10 has the top words, game England games, players time a play Wales and so forth.
So clearly this is related to some sport they play in England.
The next step is to print the article to further check that this topic assignment makes sense.
OK, so the article title is Chavez set to lose fitness bit?
So this makes sense.
Clearly, this article is about some famous athlete.
The next step is to repeat the same code for a new article.
OK, so as you can see, this article has a majority of the weight on topic seven in a small amount
of weight on topic nine and three.
Note that the label for this article is entertainment.
So let's see what these topics mean.
OK, so Topic seven is related to the words film, best world awards and so forth, so that makes sense.
Clearly, this has to do with entertainment.
So again, let's print the articles, you make sure that this really is related to entertainment and
specifically film.
OK, so the title is Oscar steer clear of controversy, which makes complete sense.
The Oscars are awards for those in the film industry.