WEBVTT

00:00.460 --> 00:07.090
So right now we are able to scrape the job titles and the job URLs from the first page.

00:07.090 --> 00:11.140
But how do we go about scraping all the other pages as well?

00:11.920 --> 00:13.420
It's quite simple actually.

00:13.420 --> 00:20.380
You just have to create a loop that changes the page number and the URL up here for the URL.

00:21.100 --> 00:25.630
So let's try and see if we can make a simple for loop here.

00:30.750 --> 00:38.850
So we say four and we say, let I be equal to one because it starts from page one, not from page zero.

00:40.250 --> 00:43.400
And then we go up until page 14.

00:45.440 --> 00:50.300
And then let's put all of our code inside of the loop here.

00:54.120 --> 01:00.150
And then let's make sure that we use this I variable inside of the URL.

01:01.110 --> 01:05.850
So I'm going to replace these, uh, quotes with a backtick.

01:05.850 --> 01:13.260
So we can just put the variable inside of the string like this by having a dollar sign and curly brace.

01:13.410 --> 01:15.630
Then let's put the I inside here.

01:16.860 --> 01:18.120
So that's it.

01:18.120 --> 01:22.770
Basically, now we are scraping all of the different pages on the site.

01:23.510 --> 01:29.090
Now, since we want to make a big array for all of the charts and not just a single page, let's create

01:29.090 --> 01:32.810
an empty array outside of the scope of the for loop.

01:32.810 --> 01:39.140
So let's call it all chops, and it's just initialize an empty array here.

01:43.910 --> 01:50.420
And then at the end of the loop, let's push the page data onto this array.

01:52.550 --> 01:54.980
So we can say all shops dot push.

01:54.980 --> 02:02.690
And then I'm going to use the spread operator to just copy the array into this existing array, or push

02:02.690 --> 02:04.370
it at the end of the array.

02:05.610 --> 02:06.780
At the end of the loop.

02:06.780 --> 02:11.790
Let's try and do a console log with all chops dot length.

02:11.820 --> 02:16.770
It should be around 325 items we get in this.

02:17.710 --> 02:21.130
Now let's try and run the code and see what we get.

02:26.840 --> 02:29.990
And we can see we get 325.

02:29.990 --> 02:35.600
So that means we do get all of the jobs from every of the 14 pages.

02:38.550 --> 02:43.650
Okay, so now we got all of the pages with all of the job titles and the URLs.

02:43.650 --> 02:49.320
But what about the job descriptions that we have when we click on the job?

02:49.350 --> 02:51.990
We also want to get all of this text here.

02:52.110 --> 02:56.130
So how can we get those and traverse even further into the site.

02:56.160 --> 02:59.070
Let's dig into that in the next section.
