OK, so in this lecture, we are going to start looking at some code now, although the code I provided
to you is in a CoLab notebook, you can see that I'm running this locally in a Jupiter notebook.
This is one of those rare situations where using Jupiter notebook actually makes sense.
So why do I say this?
Firstly, recall that I've stored my Adewusi credentials in a local file, which I do not have on Google
CoLab.
So if you'd like to use Google CoLab, then make sure that you store your credentials in the correct
location.
For me, it makes sense to run the code locally because that's where my attaboys credentials are stored.
The second major reason a notebook is good for this exercise is because, as you recall, some of these
steps are going to take a very long time, sometimes hours, as you'll see very shortly.
Everything you do is a job that you launch and then you check the status of your job to see whether
or not it's done.
Only when it's done can you move on to the next step.
So basically, you're going to run some code to launch various jobs and then you're going to run more
code to check the status of those jobs.
But you're going to be checking the status repeatedly until those jobs are finished.
This basically means you're hitting the run button over and over again until the job is done, which
is nice because you can just click a button.
Another reason is that when it comes to deleting a job, you'll have to go backwards in the notebook.
You'll see what I mean if you ever try to delete one of the things we're about to create.
Basically, it's a long chain of dependencies, so you can't delete your data set without first deleting
your predictor, but you can't delete your predictor without first deleting your forecast and so on
and so forth.
In any case, you'll see what each of these items is later in this section.
So because of this, it's very helpful to be able to scroll through the previous code.
You've run and then run the relevant blocks when you need.
Now, of course, you can do all that and CoLab too.
So this is more like why you might not want to do this in a regular Python script or in the Python console,
which I frequently use.
One reason you might want to avoid CoLab notebook is that, remember, it's online and you're using
shared resources.
If you stay idle for too long or your Internet disconnects, CoLab simply destroys your session and
you have to start again.
So if your model takes five hours to train, it's likely your notebook won't be running anymore by the
time it's done.
So my personal preference, if I were you, would be to just download the CoLab notebook locally as
a Jupiter notebook and run it on your own computer.
CoLab is just a convenient way for me to share the notebook with you.
So the basic idea is we're going to start by creating some caves that will be read by Adewusi forecast,
the next step is to upload these caves to S3, which is Amazon's simple storage service.
Adewusi Forecast will then read the files from S3 and train a model from them.
Once we've trained, our model will generate a forecast and then compare that to the true Time series.
So let's start by importing pandas, why finance Boto three in daytime in the first block of code will
download and transform the data will be using for the forecast.
So here we call why after download and we'll be using stock prices from the S&P 500.
We use data from January one twenty eighteen up to April 17 at twenty twenty one.
Next, we call DFG head just to remind ourselves what the data looks like.
As you recall, the date column is used as an index and then we have open high, low, close adjusted
close in volume.
So we'll be using the close column instead of the adjusted close column.
The reason for this is if you look at the open high and low prices, notice that these are not adjusted.
So in the first row, the adjusted closes is about two five three.
But the open, high, low closed prices are all around to sixty seven to sixty nine.
In the next block, we're going to start working on filling in the missing data.
The first thing we need to do in this process is generate a data frame with all the necessary dates.
To do this, we use the date range function for the start date will pass in the first date from our
data at index zero for the end date will pass in the last date from our data at index minus one.
When we print out the result, you can see that all the dates are now contiguous, by the way, note
that for this lecture, it's essentially necessary to have run this script before the actual lecture
recording, since some steps can take several hours to run.
So that's why I'm not clicking around the notebook alive, since it wouldn't make a lot of sense.
The next step is to create a new data frame with our new date range as an index, if we do a dot head,
we can see that it's an empty data frame with no columns except for the index.
Next, we call the joint function, joining our new data frame with the existing data, we use an outer
join so that all the rows are kept.
You can see from the output that our data frame now has nans for the days where there was no data before.
The next step is to fill in the missing data, as you recall, for stock prices we use forward filling.
Remember that we don't want to do something like interpellation since that requires you to look into
the future.
The only exception to this is the volume column.
We know that for days where there is missing data, those are non-trading days and the trading volume
for those days is obviously zero.
Once we've done this, we do another dot head to check the output.
So we can see here that for the days where we previously had missing data, the prices are now carried
forward, whereas the volume is zero.
Next, we're going to save our data frame to a CSV just in case we need a later, since this script
is meant to be run from start to end in one go.
This probably won't be necessary.
However, saving it lets you return to the script at another time or continue the work in a different
script.
So I always think it's helpful to actually look at the output of what we've done, so here I'm using
the head come in to check what our CSFI looks like.
As you can see, it is indeed a CSFI of our data frame, including the date column.
Note that the date column doesn't have a name, which is fine.
Next, as you recall, Adewusi forecast requires that we have a column called Item ID, therefore,
I've created a column called Item ID.
It's going to take on the value S.P.I since that's the ticker for our data set.
In practice, what you might want to do in your own experiments is a vector auto regression.
So, for example, you can include all the stocks in the S&P 500 and each of them would have their own
item ID.
As a side note, these can also be stored as separate caves since Adewusi forecast has the ability to
look at more than one caveat once.
The next step is to define the forecast length, which I've said of 30, so our forecasts will be approximately
one month into the future.
Using this, we can now define our train set, which is the data frame, D.F. up to the last 30 data
points.
The next step is to separate out the Target type series and the related Time series, so we discuss
the differences between these data sets earlier in this section.
The Target series is the series whose value we want to forecast, since that's the close price, we've
chosen the close call.
We also need the item ID column since that's required by a US forecast.
The related series is a series of features that can be used to predict the Target series.
Technically, we don't know that these would be useful, but we are making the assumption that they
are.
So for the related series, we choose open high, low volume as before.
We also need the item ID.
Next, we're going to write both of these new data frames to CSV files using the Toksvig function we
set header equals nine, a sense Adewusi forecast does not require headers in your CSS vs.
And again, it's always nice to check the output of your code to make sure it's what we expect, so
we call the head commands and look at the first five lines of our Target series, CSFI.
So each row here is the date, followed by the price, followed by the ticker, which is the item ID.
So this looks correct.
Next, we do the same thing for the related Time series here we see the date followed by a few prices,
followed by a really big number, which represents the volume followed by the ticker.
So that looks correct.
OK, so that's everything for this lecture.
Now that our data is stored in the format we want, the next step will be to upload this data to S3
to be ingested by Adewusi forecast.