Hello, everyone.
In this section, I want to talk about what you should always check before you start to even begin making
a web scraper.
Now, I actually want to avoid riding a web scraper, and that's kind of odd, since now I have a course
about web scrapers and they're fun enough to write.
But inherently a web scraper is sort of a hack, Which means what I mean by that is that a web scraper
is taking a website and sort of parsing it into data that you can use, like machine readable data,
like titles, images, not something for humans to consume.
Right?
So inherently we are taking things from a format that's not meant for computers and trying to pass it
into a computer.
What I'm getting to is that before you even start to make a web scraper for a site, you should always
check if the site has an API that you can use instead.
Always, always check if the site has an API that you can use instead.
That goes either by a totally public API that the site is advertising that you can use, or if you do
a little like reverse engineering on the site's source, then you get to a API that you can use.
We do an example of this using the Nordstrom website where we sort of reverse engineer how to use the
API to get results of their items instead of scraping the site instead.
And I can tell you it is you're going to save so much time and trouble if you can use an API instead
of having to scrape lots of HTML or even launch a automated web browser such as puppeteer.
The order of things that you should try to do.
It is always to try get an API first.
If you can't get an API from the site, if it doesn't have an API, not even if you look inside of the
source of the website, then try and go for using request only.
And the reason why you want to go with request request next after an API.
If you can't get an API, go for request because request is using a lot less resources than puppeteer
or something like nightmare chairs, selenium and so on and solutions like automated browsers, like
Puppeteer, selenium, Nightmare, JS and so on.
Use a lot of CPU, a lot of memory.
They're prone to crashing.
It's a lot more complex than using request.
Request is just.
Taking a URL and getting the HTML back without parsing any JavaScript and stuff like that.
However.
If you have a site.
That doesn't have a public API.
You can access right away or it's not easy for you to reverse engineer how to use the API.
And the site requires JavaScript to render, which means you can't use request because request needs.
A request doesn't have a JavaScript rendering enabled, it's just fetching the data on the URL.
Then the last option is to use an automated browser such as puppeteer or selenium nightmare JS and so
on.
That's the very last option.
And unfortunately I see a lot of my students go right to the very last that I would go to, which is
something like Puppeteer.
And they ask me, Hey Stefan, how do I make Puppeteer go to a form and type in something and search
something for me?
And I'm like.
Um.
What do you want to scrape, first of all?
Well, this side here and I go and look at the side and I see.
Well, that doesn't require JavaScript to be rendered.
You could use request with that.
And request takes a lot less memory CPU than perpetrators and it's a lot less prone to crashing.
And it's so much simpler to use than having to use something like puppeteer.
That being said, there's always a place in time for it, but it's the very last option that I would
use.
And so that's it.
So if you want to scrape something like Yelp, then don't go and write a puppeteer scraper for it,
because Yelp, for example, has a public API that you can use.
Same thing.
We have a case study on Nordstrom, for example, where we reverse engineer, find out they have an
API we can use instead of building a scraper.
So, yeah, even though I have a web scraping course here, you're watching a web scraping course.
I also teach you how to avoid actually writing a web scraper, which is perhaps the best solution.
Because once you have an API, you get nice and clean data that you can use right away in a nice Json
format, and that's just going to make your life so much easier.
That being said, there's always going to be a time and place to use something like request parse HTML
or something like puppeteer and parse the HTML.
And of course I'll also show you how to do that.
But keep in mind API first, then request.
Then puppeteer or another automated puzzle solution.
So that's it for this lesson.
I hope you got something out of it.
Remember the order of things to do it.
And you'll be.
You'll.
You'll be good.
You'll be on your way to becoming a web scraping master.
So see you in the next section.