In this lesson,
we're going to get started using a library called Beautiful Soup to pass an HTML
file. Parsing
the HTML file is the first step to extracting the data contained in a website. To
get started, the first thing I want you to do is to head over to
the course resources and download the starting project for today.
It's called bs4-start and once you've extracted it and opened it in
PyCharm, then I want you to take a look inside. First, we've got an empty
main.py file
which we're going to write in order to use Beautiful Soup and extract the data
that we want. And the data is going to come from this website.html.
Now, a really nice thing about PyCharm is when you have an HTML file,
you can always just click on your favorite browser icon here
and it will open up the website as it is.
And you can see that this is a simplified version of our HTML CV page that we
built on day 41. Now,
if you've skipped day 41 to 44
because you already know HTML and CSS,
then have a quick read through the HTML.
It's a very simple document with a bunch of different HTML tags,
attributes like class and ID,
and also a list. I've kept this as simple as possible
just so that it's easy for us to work through it
when we're trying to get hold of things
using Beautiful Soup. In the course resources,
I've also got a link to the Beautiful Soup documentation.
So this is where you can find out everything that you can do with Beautiful
Soup. But I wanna walk you through some of the most commonly used components.
Beautiful Soup,
as they say, is a Python library for pulling data out of HTML and XML files.
So HTML and XML are both structural languages and they're responsible for
structuring data like the data in a website using these tags.
And the great thing about Beautiful Soup
1: is it's super easy to use, and it can save you hours or,
0: days of work to get hold of the data that you want from a particular website.
And you can see the documentation's also been translated by kind users into some
other languages as well. So if you find it easier,
then these languages might help you as well. Now,
the first thing I have to do here in my main.py is to actually get hold of
this particular file.
Now you might remember from previous lessons,
how we open a file in Python.
So have a quick think and see if you can figure out how to get a hold of the
content in this HTML file as a string or as a piece of text.
Pause the video and see if you can complete this challenge. Just as a hint,
you might need the keyword with and the
1: keyword open.
0: All right. So the way we do this, as we say with open,
and then we provide the name of the file, which is website.html.
And we can open this as a alias name which we'll call file,
and now we have access to this file and we can say file.read.
Now, once we've read this file,
we can save this to a variable
which we'll call contents. And once we've gotten hold of these contents,
then we can start using Beautiful Soup.
So as always, we always start with the import,
so the name of the library that we're going to install is called bs4
and this is Beautiful Soup version four
which is currently the latest version of Beautiful Soup.
And from this particular package, we want to import the class
which is called Beautiful Soup. Now,
if you downloaded this project from the course resources,
you should see that there's no errors underlining this line because bs4 has
already been installed. If you see some squiggly underlines,
just click on the red light bulb and install the required module
which is called bs4. Now,
once we've got hold of our BeautifulSoup class
then we're ready to make soup. In order to make soup,
we use our BeautifulSoup class and we create a new object from that class
and we pass in a string. So this is the markup,
and that's the same M that you find in HTML and XML.
It's the hypertext markup language and the extensible markup language.
So the markup refers to basically all of this.
We've already gotten hold of it through this contents
so let's go ahead and pass that in.
So now that we've specified what it is we want to use to create our soup,
the next thing we have to provide is the parser.
This is going to help the BeautifulSoup module
understand what language this particular content is structured in.
As they mentioned, it can parse HTML and XML
so we have to tell it what particular type of document we've got.
And the easiest way is just to use the Python
html.parser.
So after we've parsed in the text that we want to turn into soup,
we're going to add the parser as html.parser.
And this is going to help Beautiful Soup understand these contents.
Now, depending on the website that you're working with, occasionally,
you might need to use the lxml's parser. And do that,
all you have to do is say import lxml
and then install this particular package, and here in a string
instead of using html.parser, you can use lxml.
And this is basically just a different way of parsing or understanding the
content that you're passing to Beautiful Soup.
And I found that with certain websites the html.parser might not work
and you might get an error that tells you something about your parser not
working. So then you might consider using lxml instead.
So this one line of code basically completes our parsing.
And this soup is now an object that allows us to tap in to various parts of the
website, but using Python code. For example,
if I wanted this title tag out of this whole website,
all I have to do is say soup.title.
And now if I print this soup.title,
you can see that we'll get the title tag being printed out in its entirety.
Once Beautiful Soup has made sense of this website by parsing the HTML,
we can now tap into that object
which is the HTML code as if it were a Python object.
So we can tap into the title,
but we can dig even deeper. Instead of just tapping into the title,
we can also get hold of other things like the title.name,
and this is going to give us the name of that particular title tag.
So remember this gave us the title tag,
so all of this, and this next stage drilling even deeper into the name of this
tag, you'll see that it gives us title.
So the name of this title tag is called title,
and we can also get hold of the string
which is contained in the title tag by simply using .title.string.
And you can see this is the actual string that's inside that title tag.
If we think about it, this entire soup
object now represents our HTML code.
So I can also actually just print out the entire soup object.
And you can see that this is basically just all HTML.
And if you want to, there's even a method called prettify
which will indent your soup HTML code.
So now, compared this where everything's all on one line, with this prettified
version where everything's all indented properly and easier to read.
In addition to getting the title tag, we can also get hold of,
for example, the a tag.
So this is going to give us the first anchor tag that it finds in our website,
which happens to be this one right here.
And we can swap that with maybe the first li or the first paragraph.
Essentially what we're doing with beautiful soup is we're just drilling down
into this HTML file, finding the HTML tags that we're interested in,
and then getting hold of the
either name of the tag or the actual text of the tag.
But what if we wanted all of the paragraphs or all of the anchor tags in our
website, how would we do that? In the next lesson,
we're going to dive deeper into searching through websites for all of the
components that we're looking for. For example,
all of the P tags or all of the anchor tags.
And we're going to see how we can refine our search and specify exactly what it
is that we want. So for all of that and more, I'll see you on the next lesson.