WEBVTT

00:00.230 --> 00:01.010
Hello, everyone.

00:01.010 --> 00:05.630
In this section we're going to look at how to deploy a web scraper onto Heroku.

00:05.660 --> 00:12.470
Now, how to make it run automatically every ten minutes or any other given interval you may want.

00:12.770 --> 00:14.570
So we have a site here.

00:14.570 --> 00:17.560
We have Craigslist with the cars.

00:17.570 --> 00:24.020
So I went just to look in the cars and trucks section and we're going to scrape this side or this page,

00:24.020 --> 00:26.930
the first page every ten minutes on Heroku.

00:27.320 --> 00:31.430
The the the site is being updated pretty frequently.

00:31.430 --> 00:32.840
Pretty frequently.

00:33.620 --> 00:35.150
Pretty frequently.

00:35.150 --> 00:40.220
So that's why it's pretty interesting to do this kind of periodic scraping on it.

00:41.290 --> 00:49.880
Um, so Heroku is a place for you to deploy your NodeJS apps, so you basically deploy it to the cloud

00:49.880 --> 00:58.100
and you can either have a API up and running on Heroku or you can also just have a simple scraping script

00:58.100 --> 01:04.650
like we will have that you can run in every ten minutes or every day or every hour and so on.

01:05.860 --> 01:12.310
And then it's going to save all of our results onto a MongoDB database we have running on Mlab.

01:12.400 --> 01:19.630
And of course, you can choose any other provider you want to, but it's I'm using mlab and it's going

01:19.630 --> 01:22.090
to save all of the results here.

01:22.090 --> 01:30.730
And if there's any new results every time it runs, it's going to save a new a new object onto the database.

01:31.060 --> 01:38.660
So, so far since I've been running it for a couple of hours, it's been scraping 310 cars.

01:38.680 --> 01:43.540
And remind you, there's 120 cars on this first page.

01:43.540 --> 01:47.290
So it's just getting new cars every ten minutes or so.

01:50.240 --> 01:57.380
So now let's take a look into the code itself and we'll talk a little more about the code, how it works,

01:57.380 --> 02:00.170
its principles we had before in the course.

02:00.170 --> 02:03.620
But I'm just going to show it now so you understand what it's doing.