In the previous video we looked at the data, a string containing a single song,
and saw how to prepare that for generating new text.
We saw how to tokenize the data and then create sub-sentence
engrams that were labelled with the next word in the sentence.
We then one-hot encoded the labels to get us into a position where we can
build a neural network that can, given a sentence, predict the next word.
Now that we have our data as xs and ys, it's relatively simple for us to create
a neural network to classify what the next word should be, given a set of words.
Here's the code.
We'll start with an embedding layer.
We'll want it to handle all of our words, so we set that in the first parameter.
The second parameter is the number of dimensions to use to plot the vector
for a word.
Feel free to tweak this to see what its impact would be on results, but
I'm going to keep it at 64 for now.
Finally, the size of the input dimensions will be fed in, and
this is the length of the longest sequence minus 1.
We subtract one because we cropped off the last word of each sequence to get
the label, so our sequences will be one less than the maximum sequence length.
Next we'll add an LSTM.
As we saw with LSTMs earlier in the course,
their cell state means that they carry context along with them, so
it's not just next door neighbor words that have an impact.
I'll specify 20 units here, but again, you should feel free to experiment.
Finally there's a dense layer sized as the total words,
which is the same size that we used for the one-hot encoding.
Thus this layer will have one neuron, per word and
that neuron should light up when we predict a given word.
We're doing a categorical classification, so
we'll set the laws to be categorical cross entropy.
And we'll use the atom optimizer,
which seems to work particularly well for tasks like this one.
Finally, we'll train for a lot of epoch, say about 500, as it takes a while for
a model like this to converge, particularly as it has very little data.
So if we train the model for 500 epochs, it will look like this.