In this lecture we are going to begin discussing the code for our reinforcement learning.
Stock trader you will notice that we are not using Google COLA for this which I found performs very
slowly compared to running it locally.
For this reason I'm running the scripts locally.
But as usual if you prefer to run this code in Google Ecolab all you would need to do is paste the code
and modify the data source the relevant file for this lecture is RL trader dot Pi which can be found
in the course repository
in this lecture.
We're going to describe just a few initial bits and pieces of our program.
The first function we have is get data.
Luckily I've already included the data in the course repository.
So all we have to do is important.
The data is stored in a CSP which we can load in a pen as data frame.
But of course what we are interested in are the values so we can call dot values to get a number higher.
Note that in order to simplify this problem we're going to use the closed price only.
You're welcome to look at the CSO for yourself if you want to know what's in it.
As usual this returns that he by D a sequence where t is the number of times steps and d the feature
dimensionality and in this case that's the number of stocks three here are the three stocks we'll be
looking at in case you're interested.
First we have Apple that's AAPL next we have Motorola that's MSCI and finally we have Starbucks that's
SB ex note that each row corresponds to the same date.
So for each stock we have data from about February 2013 to February 2018.
Next we have the replay buffer.
This has three functions.
The constructor store and sample batch in the constructor.
We initialize our array buffers and pointers.
First we have obs one buff and OBS to above which store the states and next states respectively.
Next we have ax buff which stores our actions.
These are represented by integers from zero up to 26 inclusive so we can use you in eight to represent
them.
Next we have the rewards buffer which stores our rewards and we have the done buffer which stores the
Dunn flag.
This can only be 0 or 1 So again it can be a U.N. 8.
Lastly we have a pointer which starts at zero the current size of the buffer is zero and the max size
of the buffer is specified as the size argument.
Next we have the store function.
This store is the state action reward.
Next state and dunk flag in their respective buffers at the index self-taught pointer.
After this we increment the pointer so that the next time we call the store function all the values
will be stored in the next position.
And remember this is circular.
So we make use of the modulo operation when incrementing the pointer would go up to max size it's set
back down to zero.
Lastly we set the current size of our memory which is equal to the minimum of the previous size plus
one.
Since we just added one more value or max size.
In the case the buffer is for
finally we have the sample batch function.
This chooses random indices from 0 up to the size of the buffer.
Then we return a dictionary containing the state's actions rewards and so forth indexed by those indices
the next thing we're going to look at is the get scalar function this takes in an environment object
since that will be used in fitting our scalar.
The idea is in order to get the right parameters for a scalar.
We must have some data in order to get this data.
It's a face is to just play an episode of randomly and store each of the states we encounter.
There is no need to have an AGM because such an agent wouldn't be trained anyway.
So you can see that when we choose an action to perform the only thing we need to do is sample a value
from the action space.
When we're done we create a standard scalar object and fit it to the states we encountered.
Now one thing you could do to make this more accurate is run it for multiple episodes.
Next we have a function called maybe make Dir.
This is more just a utility function.
It checks if a particular directory exists and if it doesn't then it creates the directory.
The reason we need this is because we're going to store our trained model and the rewards we encounter
to separate files so that we can plot them afterwards.
Next we have the MLP class which is going to represent our Q function approximated as you can see.
This is just the standard day and then with an arbitrary number of hidden layers.
So as input to the constructor be taken the number of inputs the number of actions the number of hidden
layers and the number of hidden units.
Next we create a variable M initialized to be the number of inputs and a list to store our layers.
Next we enter a loop so we can add each of our layers.
So in hidden layers times I'm going to create a linear layer with size M by hidden dim I'm going to
update m to be the size of the next layer as output.
I'm going to append to the layer to my list of layers and then I'm going to add a real you non linearity
outside the loop.
I'm going to add one final layer with output size equal to N action and then I'm going to pass all these
layers into a sequential object and then store that in the layers attribute.
Next we have the forward function which simply passes the data through our previously created sequential
object.
Next we have some special functions that are not normally part of a PI George model for saving and loading
for saving.
I just called towards that save pass in the state dict belonging to the current object and then the
path to save the file to for loading.
I call the load state dict function.
Belonging to this class and then passing the output from torch dot load path
next.
We have a few convenience functions to interact with the rest of the script.
I like the rest of the script to be library agnostic.
What I mean by that is I don't want the rest of the scripts to have to know anything about pi talk at
all.
It should work no matter how I implemented my neural network whether that's with PI torch tensor flow
jacks or any other library.
So to that end we basically need two kinds of functions that psyche learn would normally provide a training
function and a prediction function in the only interface.
These should be no higher raise or other basic Python types.
There should be no pi towards specific inputs or outputs.
So for the predict function we take in a model and a set of states.
We can assume that the set of states has the typical and by D data shape inside the function we use
torture that no grad since this is just a prediction.
First we convert the states to float 32 and then convert it into a torch tensor.
Then we pass that into the model and retrieve the output once we have the output we call dot num pi
to return the prediction as an umpire a
next we have the train ones that function.
I call it this because we're going to do one step of our optimizer for every step we take in the environment.
As input we take in the model the criterion the optimizer and a set of inputs and targets.
Now you might be like Wait a minute I thought you said you don't want any PI towards specific inputs.
So I lied a little bit.
Well you'll see later that the Asian object will have no pi talk specific arguments.
So that's kind of the true interface.
Alternatively you could create the criterion and the optimizer as global variables otherwise we'd have
to make this function part of the model including the criterion and the optimizer and so forth which
would be very unconventional pi tau.
So we don't want to do that.
So anyway the first thing we do is convert the inputs and targets into flow 32 and turn them into torch
sensors.
Next we zero the gradients.
Next we pass the inputs through the model to get the outputs.
And next we calculate the loss.
And as usual we call lost that backward and optimize it our step.
And this does one step of gradient descent.