Hey guys, welcome to day 53
of a hundred days of Code.
Today it's time for your Capstone project,
and it's time when you review everything that we've learned
over the last 10 days or so.
Everything to do with web scraping.
The project that we're going to be tackling
is a data entry job.
Now, there's a lot of data entry jobs out there
where you're kind of just meant to transfer data
from one format to another.
So maybe you have it in a physical print copy,
or maybe it's on a website, maybe it's in a PDF,
and you just have to transfer it somewhere else,
usually typing it into a spreadsheet.
The inspiration for this project came from
when I was browsing Reddit actually
on the r/Python subreddit,
which is a really good community for you to actually look at
and see what other people are doing with Python
and seeing the latest and greatest things built
or news about Python.
Now, one of the posts I saw was asking
whether if anyone has automated their job completely,
basically using Python.
Now we've seen how powerful Python can be,
especially when we apply it to web scraping
using Beautiful Soup and Selenium.
And looking through all the comments,
there's actually a lot of people who have done this,
including this one guy
who basically pretty much automated his entire job.
And the jobs that tend to be easily automated
using Python are data entry jobs,
moving data from one format to another.
And if you think about it, if that job is in fact remote,
so if you search on Indeed.com for a remote data entry job
and you get up and running with the company
and you start doing it manually,
and then once you've understood what it is you have to do,
for example, gathering statistical data, preparing reports,
and maintaining spreadsheets,
if you realize that this is a large part of your job
and you can automate it pretty much with Python,
then you could probably get Python to do 70% of your job
and you spend the rest 30% of the day
doing the rest of the job, but still being paid
as a full on worker with full benefits.
So this is something that a lot of people
in the Python community has talked about and explored.
And this is something that we are going to be trying out
using both Beautiful Soup and Selenium in this project.
In our case, we're going to be tackling
a research data entry job
where we're researching house prices
that fit a particular criteria for a client
on the Zillow website
and then we're going to be transferring that data into a form,
which will create a spreadsheet in Google Sheets.
And that is usually how as a data entry person,
this is how we would make our money.
Now, because this is a Capstone project,
we're going to be using everything that we've learned
in this section.
That means Beautiful soup as well as Selenium.
You might have to revise up
on some of the things you learned,
especially the stuff on Beautiful Soup,
which we covered a few days ago
and we're going to combine all the skills
that you've done so far.
And this project really is going to test
all of your web scraping skills that you've acquired so far
and see how far you can run with it.
Because it's Capstone project,
there's not going to be a lot of guidance,
so you're going to have to persevere
and try to see if you can solve your own problems
and see if you can get to the end outcome.
Let's say that you have a client who wants you
to compile a list of all the places that they can rent
in San Francisco up to $3,000 per month,
and it has to have at least one bedroom.
Now, San Francisco is notorious
for really expensive housing,
and it's also really difficult
often to actually find somewhere that you want to live.
On Zillow, you can already filter on these things.
So for example, you could say, this is the area,
San Francisco, California that I want to rent
and then of course, changing it to for rent
rather than for sale,
switching the maximum price up to $3,000
and then adding in the extra requirement
that it must have at least one bedroom.
Now, we could use the live version of Zillow
to do this project,
but the problem is that websites frequently get updated.
Companies like Zillow continuously improve their site,
so they might change the structure of their HTML,
the names of their CSS classes
and have popups to the website or introduce captures
which causes issues for Selenium.
For this course, I want to make sure
that we can all practice writing our code
in a stable environment that doesn't change
and that my code solution continues to work.
That's why I've created a clone of Zillow's website
so that you can practice and test your knowledge
of Beautiful Soup and Selenium.
Open up the Zillow clone website inside your Chrome browser.
And as you can see, I've created a snapshot
of the Zillow site
where I've already narrowed down the search criteria.
I've picked San Francisco as the location,
and I've picked for rent rather than for sale,
and I specify the price as up to $3,000
for a one bedroom apartment.
So the URL you should use with Selenium for this project
should read, https://appbrewery.github.io/Zillow-Clone.
Now, in addition to using that URL,
you're also going to be using Beautiful Soup
to scrape through all of this data.
And what we want is the price, the address,
and also the URL that this will link to.
So for example, when I click on this,
it will link to the actual listing of the place.
And then once you've scraped all of that data
using Beautiful Soup, then you're going to be using Selenium
to auto fill in a Google form.
So we're going to be adding in the address of the property,
the price per month and the linked property
and of course, we're going to fill out one of these forms
per listing that we have on Zillow.
And once all of that form's been compiled,
then you'll have the option to turn it into a spreadsheet.
Whenever you create a form in Google Forms,
you can see that when you go to the responses tab,
you can click on this button
in order to create a Google sheet from the responses
that have been submitted
and this is what you end up with,
a spreadsheet with the address of the property,
the price per month, and a link to the property.
So this way, once you've done this research,
then you can send it to your client
so that they can filter down on each of the listings
that match their criteria and decide which one they want to go
and make a viewing.
So this of course makes their job a little bit easier,
and this is our research task
that we're going to complete today.
So the first part of scraping the data
for the relevant listings is going to be done
using Beautiful Soup and then the second part
where we're going to be filling in this form is going to be done
using Selenium.
So that is the project.
And once you're ready, head over to the next lesson
and take a look at the requirements of the project
and we can get started with the Capstone project.