Okay, one, we got a little loop built out so that we go through each of the job URLs inside of this
list of jobs.
But we also need to have a way to limit the request rate we make on the Craigslist site so that they
don't block us from accessing the site because they think we might be making too many requests, which
is not like a normal user behavior.
So we need to limit how many requests we make per second and how do we do this?
Well, it is quite simple.
We just need to write out a generic sleep function in the JavaScript so we can write out something like
async function sleep.
And this sleep function then takes in the number of milliseconds we want it to wait.
So.
Milliseconds and then we can say something like, well, we can say return new promise.
Then we have the result callback.
Then we say set, timeout.
And we pass in the result.
Callback here and the milliseconds.
Now what this is going to do is it's going to return a new promise, which is only going to be resolved
once set timeout has run through the milliseconds we pass to it.
So set timeout is just a native JavaScript function.
It takes in the number of milliseconds and it has a callback function that's going to call once the
milliseconds has passed.
And once that callback has been called, then we say the promise has been resolved and we can move on
to other things.
So the way to use this inside of our function or inside of our loop would be something like await sleep
and then we just pass in the number of milliseconds.
So 1000 milliseconds means one second wait.
That is it.
That's how we can make a generic sleep function.
You can also use this inside of NodeJS requests.
So if you're using NodeJS request or Nightmare JS, whatever you might be using, you could use this
function to make the threat sleep.
Now keep in mind that it only works this way.
If you're using async await, if you're using something like dense classes, then you would have to
do something like this.
You would write out sleep 1000 and then have a then.
And then you would have some code here.
But I would highly recommend that you use the async await.
Instead, it makes the code so much easier to read and maintain, I think.
Okay.
So.
That's how we make the scraper sleep.
And now on to the next section.
We are going to be scraping the content text we have for each of these job descriptions.
Now, I recommend that as an exercise that you try to scrape this content of the job listing yourself.
Try and do this.
It's a similar process to what we've been doing before when we get the text content of an element.
So go ahead and try to write out something where you can scrape out the text content of a job description.
And in the next section I'm going to show you how I will do it.