Hey guys,
welcome to Day 45 of 100 Days of Code. Now,
today, we're going to be getting back to coding with Python,
and we're going to be learning how to scrape the web for data
using a module called BeautifulSoup.
Now we've been working with APIs for quite a while now,
and we know that
we can use a website's API to access their data or to interact with the
website using code.
But some websites don't have an API or their API
doesn't let us do all the things that we want to do.
So this is where we start thinking about using web scraping
where we look through the underlying HTML code
of a website to get hold of the information that we want.
So the aim of today is to learn how to make soup,
but not this kind of soup. We're going to be making BeautifulSoup.
What exactly is BeautifulSoup? Well,
it's a module that helps developers like us make sense of websites.
We could think of a lot of websites as a bit of a spaghetti soup,
even something seemingly as simple as the Google front page,
when you right click on it and view page source,
you can see that it's horrendously complicated.
And if you wanted to make sense of this webpage and pull out the relevant parts
of the data,
then you'll need an HTML parser like BeautifulSoup so that you can
find and pull out the HTML elements that you're interested in from this
soup of jumbled HTML code.
And once we've mastered this skill,
then we'll be able to take any website, for example,
Empire's 100 Greatest Movies Of All Time,
this is a huge list of a hundred movies that apparently everyone should have
watched at some point in their life,
and we can pull out the relevant parts to us
namely the title and the ranking of each movie
and we're going to use it to compile a list of movies that we have to watch so
that we can look at the list, cross out the ones that we've already seen,
and then pick at random one from the list so that we can watch all of the
hundred movies of all time. That's the goal.
And once you're ready head over to the next lesson and we're going to get started
using BeautifulSoup.