Okay. We've mentioned
the shape of the data and the batches that
the data is split up into. It's important to
take a look at that, and let's dig into that next. The inputs are three dimensional. So for example, if we
have a window size of 30 timestamps and we're
batching them in sizes of four, the shape will be 4
times 30 times 1, and each timestamp,
the memory cell input will be a four by
one matrix, like this. The cell will also
take the input of the state matrix from
the previous step. But of course in this case, in the first step,
this will be zero. For subsequent ones, it'll be the output from
the memory cell. But other than the state vector, the cell of course
will output a Y value, which we can see here. If the memory cell is
comprised of three neurons, then the output matrix will
be four by three because the batch size coming in was four and the number
of neurons is three. So the full output of the layer
is three dimensional, in this case, 4 by 30 by 3. With four being the batch size, three being the number of units, and 30 being the number
of overall steps. In a simple RNN, the state output H is just
a copy of the output matrix Y. So for example, H_0
is a copy of Y_0, H_1 is a copy of Y_1, and so on. So at each timestamp, the memory cell gets both the current input and
also the previous output. Now, in some cases, you might want to
input a sequence, but you don't want to
output on and you just want to get a single vector for each instance in the batch. This is typically called
a sequence to vector RNN. But in reality, all you do is ignore all of the outputs,
except the last one. When using Keras in TensorFlow, this is the default behavior. So if you want the recurrent
layer to output a sequence, you have to specify
returns sequences equals true when
creating the layer. You'll need to do
this when you stack one RNN layer on top of another.