In the previous video we looked at the
data, a string containing a single song, and saw how to prepare that for
generating new text. We saw how to tokenize the data and
then create sub-sentence engrams that were labelled with
the next word in the sentence. We then one-hot encoded the labels to
get us into a position where we can build a neural network that can,
given a sentence, predict the next word. Now that we have our data as xs and ys,
it's relatively simple for us to create a neural network to classify what the next
word should be, given a set of words. Here's the code. We'll start with an embedding layer. We'll want it to handle all of our words,
so we set that in the first parameter. The second parameter is the number of
dimensions to use to plot the vector for a word. Feel free to tweak this to see what
its impact would be on results, but I'm going to keep it at 64 for now. Finally, the size of the input
dimensions will be fed in, and this is the length of
the longest sequence minus 1. We subtract one because we cropped off
the last word of each sequence to get the label, so our sequences will be one
less than the maximum sequence length. Next we'll add an LSTM. As we saw with LSTMs
earlier in the course, their cell state means that they
carry context along with them, so it's not just next door neighbor
words that have an impact. I'll specify 20 units here, but again,
you should feel free to experiment. Finally there's a dense layer
sized as the total words, which is the same size that we used for
the one-hot encoding. Thus this layer will have one neuron,
per word and that neuron should light up
when we predict a given word. We're doing a categorical classification,
so we'll set the laws to be
categorical cross entropy. And we'll use the atom optimizer, which seems to work particularly well for
tasks like this one. Finally, we'll train for a lot of epoch,
say about 500, as it takes a while for a model like this to converge,
particularly as it has very little data. So if we train the model for
500 epochs, it will look like this.