Okay.
So the first thing we are going to do is to make a new directory for our robots parser, but we're just
going to call it web crawler for now because we're going to have various types of crawlers in here.
So we call it web crawlers and let's initialize NPM inside the web crawlers directory.
So I'm going to go into Web crawlers.
And say npm init.
Dash dash.
And that's it.
And now we're going to open the directory inside of Visual Studio code.
So let me go and go into red colors.
And now we are going to import some packages and write out some code.
So we're not going to build out the robots parser from scratch because that's a lot of work and thankfully
someone else has built a robots.txt parser for us.
We're going to use the robots parser from Sam Clark and it's got all the things we need and we're just
going to add that using either NPM or yarn add.
So now let's go into Visual Studio code and make sure that you open a new terminal via terminal new
Terminal and add the robots parser package.
So we say yarn add robots parser.
You can also use NPM install if you prefer to use NPM.
And then let's go and make a new file as well.
Let's call it robot parser dot js and in here we're going to have some robot robots text parsing code
that we can use globally in all of our projects for these coders we're going to build.