Here are a few phrases that were generated when I gave the neural network the sentence
Lawrence went to Dublin, and I asked it to predict
the next 10 words. We'll see the code
for that in a moment, but notice that there's a lot
of repetition of words. In this sentence, three of
the last five words are wall, and here three of the
last four are ball, and even in this one the word
relations gets repeated. This is because our LSTM was only carrying context forward. Let's take a look at
what happens if we change the code to
be bidirectional. By adding this line simply defining the LSTM
is bidirectional, and then retraining, I can see that I do converge a bit quicker as you'll
see in this chart. After training and testing, I now get these sentences. They make a little
bit more sense, but there's still
some repetition. That being said, remember this is a song where words
rhyme such as ball, all and wall, et cetera, and as such many of them
are going to show up.