Now there's two things you'll notice here.
One is we're only getting hold of the first, for example,
p tag or a tag, but we're not getting hold of any of the other ones.
So what if we wanted to get all of the anchor tags,
all of the paragraphs in our website? Well,
then we can use a function that comes with Beautiful Soup called find_all.
This is probably one of the most commonly used methods when it comes to
Beautiful Soup
And here we can search by a bunch of things. We could search by name,
so we can say
find all of the tags where the tag name is equal to a.
So this is going to give us all of the anchor tags,
1: right?
0: And if I print that,
you can see that it gives us a list and it gives us all three of the links that
exists in our website.
And if I change this to p for example,
then it'll find all of the paragraphs and we can change this to basically any of
the tag names in our website. Now,
what if we wanted to drill a little bit deeper?
We've got a list of all the anchor tags,
but what if I only wanted the text in that anchor tag?
So I just wanted this part. Well, how would I get hold of all of them? Well,
firstly, we would probably need a for loop.
So we could say for tag in all anchor tags and we can loop through all of those
anchor tags and use a method called tag.getText.
And now if I go ahead and print this,
you can see that it's basically going to print out all three of the text that is
in all three of the anchor tags that it found. Now,
what if I didn't want to get the text,
but instead I wanted to get hold of the actual href,
so the link, right? So let's print our all_anchor_tags again.
1: All right.
0: And you can see that there is a attribute called href
which stores the actual link that the tag goes to.
So very often, you'll want to isolate that link. So how would you do that?
Well,
you can tap into each of the tags and you can use another method called get.
And here you can get the value of any of the attributes.
So if I pass in href here and I print this,
then it's going to give me all of the links and it's not going to give me
anything else. It's basically just stripped out the link
which is what I'm interested in. Similarly, when we use find_all
we can also find things by their attribute, so the moment we're searching by
the tag name, but we can also get hold of things
by the attribute name. For example,
if I wanted to get hold of this item, I can of course search for an h1.
But what if I had lots of h1s? Well,
then I could isolate it by this ID. So I could say,
1: soup
0: .find_all which will give me a list of all of the items that match the search
query,
or I can use the find method to only find the first item that matches the query.
In my case, there's only one thing I'm looking for.
And this particular tag has a name of h1
but it's also got a ID of name.
As you can see,
this ID is equal to name and it's also an h1 tag.
So this will give us that particular element.
So if I print out this heading and let's comment out everything else,
then now you can see I've just isolated that one h1.
And this also means if I just add another h1 here,
1: ...
0: and I run this code again,
that is not going to show up because I've said it has to have a name of h1
and an ID that matches this particular value.
Now, as you can imagine,
you can also do the same thing with the class attribute.
So we can say,
1: ...
0: soup.find because again, I'm only looking for one.
And the thing that I'm looking for has a name
which is a h3
1: ...
0: but it's also got a class that's equal to heading.
So I'm just going to copy that and paste that in here. Now,
one of the things you'll get here is an error because this class keyword is a
reserved keyword in Python.
And what that means is that it's a special word
which can only be used for creating classes. Now, in this case,
we're definitely not creating a class or an object here. Instead,
we're trying to tap into an attribute.
So in order to not clash with the class keyword,
this attribute is actually called class_re.
Now it's going to look for all of the h3s where the class attribute is
equal to heading.
Let's go ahead and print this section_heading and you should see now we'll get that
h3 with the class of heading show up. And again,
if we wanted to get hold of the text
that's contained in that h3, then we simply use the getText method,
or if we want to know the name of that particular tag,
then we can say .name.
1: Okay.
0: And if we want to get hold of the value of an attribute, for example,
get the class value,
then we can do something like this. Now,
while that's a pretty good way of selecting elements from the entire website,
there's certain cases where it might not work. For example,
at the moment here, we've got our three anchor tags.
If we wanted to get hold of a specific anchor tag,
let's say we wanted this anchor tag, then what do we do?
Well,
then we could just simply find all of the anchor tags and then find the first
one. But as you can imagine,
this is a incredibly simple website.
Most websites will have thousands
if not tens of thousands of links. In that situation,
it's really hard to know which particular link you want from the list of all of
the anchor tags.
So we want to have a way where we can drill down into a particular element.
What's unique about this particular anchor tag? Well,
it sits inside a strong tag and it sits inside an emphasis tag and it sits
inside a paragraph tag, which itself is in the body.
We can narrow it down using these steps. In our current website,
nowhere else is there an anchor tag that sits inside a paragraph tag.
And you'll remember from our previous lessons on CSS that you can use CSS
selectors in order to narrow down on a particular element in order to specify
its style. And if we were to write CSS code,
then it would look something like this.
So we would select first the paragraph and then we would select the anchor tag
and then we can specify what the style should be
1: .
0: Now. When, we're using Beautiful Soup,
we can also use the CSS selectors.
I can get hold of that company URL by simply saying soup,
and instead of using find or find_all, I'm going to use select_one.
There's select and select_one. Select_one will give us the first matching
item and select will give us all of the matching items in a list.
Now we get to specify the selector as a string. And again,
I'm going to use the same selector that I showed you before.
So we're looking for a a tag which sits inside p tag.
And this string is the CSS selector. So you can write anything in here
really. This means that we'll be able to get that anchor tag.
And then once I've gotten hold of the company URL and print it out,
you can see its that exact anchor tag that we wanted.
We don't have to just stick to the HTML selectors.
You can also use the class or the ID in your CSS selector.
So remember, to select on an ID,
we use the pound sign. So let's say
we want to get hold of this h1, which has an ID of name,
we can say #name,
and now this is going to be equal to my name.
And if I now run it, you can see that last one,
the element that was picked out is the h1 with the ID of name.
And finally you can use a CSS selector to select an element by class.
So for example, here, we've got heading and here we've got heading as well.
So if we want to select all of the elements that have a class of heading,
then we could say soup.select so this will give us a list
and then the selector is the first item that goes into the method.
So similar to this, we can have this keyword argument there or we can delete it.
And this selector will be using the .heading in order to select the element
that has a class of heading.
And this is now going to be a list if we print it out.
Right here.
2: So,
0: you can use everything that you've learned about CSS selectors to select a
particular item out of an HTML file.
And this is usually really useful because a lot of these elements will be nested
inside divs and the div will have an ID
and then all you have to do is to narrow down on the div and then narrow down on the
element you want.
So you can basically drill through using CSS selectors to get to any item you
want on the page.
Now that we've looked at how to find various items from HTML using Beautiful
Soup, in the next lesson, I've got a quiz for you
for you to have a go and have some practice at selecting and finding elements
from an HTML file using Beautiful Soup.
So for all of that and more, head over to the next lesson.