In the last lesson, we got started using Beautiful Soup and we saw how we could
use it to parse through the HTML of a website and pull out the pieces that we're
interested in.
Now it's no fun scraping a website that you've already got access to locally.
It's much better if we can get hold of something that's currently live on the
internet. So I'm going to go ahead and comment out all of this code,
and I'm going to be using Beautiful Soup to get hold of a live website and grab
data from it.
And the website that we're going to be using is the YCombinator's Hacker
News website.
This is where anybody can post a link to a news piece that they've discovered
that's tech related, or you could show off things that you've built.
For example,
I've just been looking at this guy's website that he built called My Desk Tour
where you can post a picture of your desk setup,
and you can see all of the tools and gear that they've got.
So we're going to be scraping the main Hacker News website
which is under this particular URL.
And this is usually where I go to find the latest tech news.
We're going to copy this URL, and we're going to go back to our main.py.
And in order to download the data from that website,
we're going to be using our handy friend, which is requests.
Now requests allows us to get hold off of the data from a particular URL,
which in this case is news.ycombinator.com.
And once we've made that request,
then we can save the data that we get back in a response variable.
And once we've got the response,
we can actually print out the text of the response
and this is basically equivalent to what we did when we opened up our HTML file
and we read the file contents, the text of the file.
So now if I go ahead and run this,
then you can see that it's going to print out loads of stuff,
but this is basically the code that represents this particular page.
So in fact, if you right click and click view page source,
you'll see that this is exactly the same HTML code that we're getting back over
here. So we don't actually want all of this jumbled mess.
What we're more interested in is the specific titles and the links
for each of these pieces.
It shows by default 30 of the top articles,
and this is ranked by an algorithm.
So it's most recent and also getting a lot of traction,
so a lot of upvotes,
but it doesn't represent the most upvoted items. So you can see here, in fact,
the most upvoted at least today anyways,
is Mozilla laying off of 250 employees.
So what if I wanted to get hold of the article title and the link of the post
from this page that has the highest point.
I don't want to have to manually check all of this. I want to do it with code.
So let's go ahead and scrape it. Now,
if I right click on each of these titles and I click on inspect,
then it takes me to the precise line of code in the HTML
that's responsible for rendering this component.
This is actually a anchor tag, so that's the
a tag. And the text in the anchor tag is the title of the article,
and then the href is the link that will take me to the actual story. So
if I click on this, you can see
it takes me to the actual news piece about Joan Feynman.
So what about this point? Well,
let's go ahead and right click on it and click inspect.
You can see this is in a span and it has a class of score while this title is in
a a ref and it has a class of story link.
So with those two pieces of information,
we can use Beautiful Soup to scrape all of the titles,
all of the links and all of their points.
And we can compare all those points and figure out which one has the highest
point on this page. So let's go ahead and do that.
So I'm gonna save the yc_webpage as the response.text,
and then I'm going to use Beautiful Soup to parse that webpage.
So BeautifulSoup
and then I'm going to pass in the actual HTML document that we want to parse.
So this is the YC webpage,
and then we provide the method to which we're going to parse it.
So html.parser, with an ER at the end.
And this is our soup. Once we've created our soup,
the next step is actually to dig in the soup and find the parts that we want.
So if, for example, if I want to get hold of the title of all of that,
then I can just say print soup.title. And now you'll see
it gives me the title which is Hacker News.
And that's the same as what you see here in the tab bar.
Now, what if I didn't want the title?
What if I actually wanted to get hold of this text here,
the title of each of these articles?
See if you can figure out how to get hold of this text and print it out in your
code. Remember it has the class that's a story link,
and it's an anchor tag.
Pause the video and see if you can get this title,
so yours might be different from what I've got on screen of course.
It depends on what's showing up on Hacker News on the day you are doing this.
But get the title of the first article printed out using BeautifulSoup.
1: Okay.
0: All right. What we want to do is we want to use find.
So we're going to find the first instance from this webpage where the actual
name of the tag is equal to a, so that's an anchor tag,
and then the class is equal to the story link.
So I'm just gonna copy that and paste it in.
Remember that in order to not clash with the reserved class keyword,
we have to add a underscore afterwards.
Now this should be our article tag.
And if we go ahead and print it,
then you can see that we get this exact anchor tag.
But if we want to get hold of the text that's actually in the anchor tag,
then we have to go one step further and call the getText method
that's also from Beautiful Soup. So now when I run that, you can see
I only get the actual text of the article.
Let's work on some of the other pieces. So this is the article text.
And then if we want to get hold of the article_link and the article_upvotes.
See if you can figure out how to complete these two parts as well.
So we want the HTML link that is, of course, all of this HTTP,
et cetera. And then we also want to get hold of the upvote
which is this little number right here.
It's inside a span with a class of score.
1: Right?
0: All right. So let's do the first thing, which is article link. Well,
we can actually already tap into the same article tag we already got up here.
And instead of saying getText,
we can use the get method to get the specific value of a attribute.
So what we want is of course, the href.
And then the article_upvote,
we'll have to tap into our soup and find the tag with a name that is span
because this is what we're looking for, and has a class of score.
1: Right?
0: Like this. Finding this particular tag is not enough.
This actually just gets us the tag. If we want to go further
and we actually want to get the text that's inside that span
which is of course the 19 points,
then we have to dig one step deeper and call the
getText method like this. Now,
if I go ahead and print out the article_text and the article_link,
and also finally the article_upvote,
then you can see
I get all three pieces of data that I'm interested in. Now,
instead of getting the first occurrence, I want to get all of the ones that are
on this page, so all 30 results. Now, in order to do that,
I have to change the find to find_all both here and here.
This way
we get a list of all of the articles and I'll get a list of all of the article
_upvotes.
So now it's going to be a little bit different. In order to get all of the text
and all of the link, then I have to use a for loop.
So I'll say for article tag in articles,
so articles is of course,
this list where we find all of the anchor tags with a class of storylink,
and then I'm going to loop through each one of those and for each of the tags,
I'm going to get the text and also get the Href.
I'm going to create two new lists, articles_text, and article_links.
And then I'm going to save each of the new articles into those lists.
2: ...
0: Like this. And in fact,
we could probably simplify this a little bit by refactoring and renaming the
article text,
so the singular version into just text and the article_link
to just the link.
So now let's print out the lists. So the article_texts,
the article_links and the article_upvotes.
And this find_all
gives me a list and I can't call getText on the list.
So I'll also need to create a new list.
So I'm going to choose to use list comprehension here.
So I'm going to say for score in all of the scores,
we're going to create a list using each of those scores
and we're going to call getText in order to get each of them.
This is the same as writing out a for loop like this
but it's obviously much shorter. Now, when I hit run,
you can see that each of my lists are ordered.
So this is the first article's text, this is the first article's link,
and this is the first article's points.
What we want to do is we want to get the article_upvotes into a number format,
so an integer. And to do that,
we of course have to get rid of the points that comes afterwards.
But notice how each of these items are strings. So that means we can split the string
by the space and only get hold of the first item in that space.
Let me show you what I mean. So we've got all of the article upvotes,
let's go ahead and just print out the first item.
2: Right.
0: So now we just get the first item, which is 40 points. Now,
if I take that item and I call the split method,
then it's going to split every word in the sentence. By default,
it splits by the space. Now, if I run this code,
you can see
I get a list where I've got the first item being 40 and the second being points.
So it's basically split that string by the space.
Now the next stage is I could get hold of just the first item that comes from
that list, which is now 40.
If I now finally wrap it around an int,
then I can turn that into an actual number. Don't worry if your number changes
because you're pulling data live from a website. That upvote number can change in
any second. So this is the method
how we can get hold of the actual number from the upvotes.
Now we're going to apply all of this .split
and also getting hold of the first item into our list comprehension.
So for each of the scores that soup finds,
we're going to get hold of the text and then split the text and then get the first
item from the text. And then finally,
we wrap all of this around an int and turn it into an integer. Then if I go ahead
and uncomment all these lines of code,
then you can see I've got all of these numbers being printed out,
which means I can now sort them.
I want to get the index of the list item that has the highest value.
And then I want to use that index to pick out the title, text,
and also the link from these two lists,
because they're all ordered in exactly the same way.
So this first item corresponds to this first link corresponds to this first
upvote. And I want to pose this to you as a challenge.
Can you print out the title and link for the Hacker
News story with the highest number of upvotes?
Since we're working with three different lists at this point,
you'll have to find the index of the largest number inside the article_upvotes
list to accomplish this.
I'll give you a few seconds to pause the video before I show you the solution.
All right,
here's the solution. We can use the max function that Python comes with to get
the largest number from our article_upvotes.
And then we can print this largest number and see if it works.
So we've got 1,312. Now,
once we've gotten hold of the largest number,
then we can find its index from this list.
So we can say article_upvotes.index
and then we find the index of this largest number.
2: Right.
0: Now, if we hit run, you can see that we're getting index number 27.
So instead of just printing out that index,
we can print, instead, the article_texts with that index, so passing in the largest
index, and also the article_links and passing in the same index.
So now if I hit run,
you can see that the most popular article at the moment on this page has this
title text and this particular link. Of course for you
it will be different because it depends on what's currently showing up on Hacker
News. But if I refresh this page,
you can see that this article with 1,313 points is of course the most popular
article and it is the one that's about Mozilla.
You can imagine a use case for this where every day we scrape all the data on Y
Combinator and then we send ourselves through a text message through an email,
the most upvoted title and article
so that we can just look at that one thing.
And you've seen now how we can use the requests module to get hold of the text
the HTML code from a particular website,
and then use Beautiful Soup to parse through that website
and then to get hold of these specific parts that we want by using find_all or
find,
and then getting hold of the text or getting hold of the link or getting hold of
any other thing that we want.
So now that we've seen how we can do web scraping using Beautiful Soup, in the
next lesson
I want to talk a little bit about the ethics of scraping websites and when to do
it and what you can use the data you get from this
for. So for all of that and more, I'll see you on the next lesson.