OK, so the next step is to set a random seed so that we get consistent results, feel free to comment
this out in order to see different documents.
The next step is to randomly choose a document and call the spin document function on that document.
The next step is to print out our document using the text we're at module so that the text does not
go off the screen.
OK, so let's have a look at the results.
Firstly, you'll notice that sometimes the original word is replaced with the same word.
This is possible since the middle words are randomly selected.
And of course, the original word is one of the possibilities.
Note that if you wanted to force the word to change, you could temporarily set its probability to zero.
OK, so notice that the first part of the first paragraph appears, OK.
It says shares in train and plane making giant Bombardier have fallen to a 10 year low against the hands
of its chief executives.
It obviously doesn't make sense if you're a serious reader of this type of news, but it is grammatically
correct.
In fact, it's not difficult to find articles online that make even less sense.
Note that the second part of the sentence does not make sense.
It ends with in two members of the keyboard.
So let's think about why this might happen.
As you recall, a model looks at tri grams only.
It does not consider the rest of the context in the sentence.
So if you only look at this program, the keyboard, it makes complete sense.
This is because you might have a sentence like the keyboard members were present at the meeting.
But the reason this does not make sense is because the word of members appears earlier in the sentence.
Thus, this is a hint that it might be useful to consider context beyond just the previous the next
word.
Let's now consider the next sentence, which says Paul Tellier, which was also Bombards epicenter,
left the company amid an £80million restructuring.
Again, it's kind of a weird sentence, but it's not completely wrong.
It makes grammatical sense, but it does not make logical sense.
So we normally do not use the word which to refer to a person, but rather who, which was the original
word.
Of course, our model doesn't know this because the previous token in the context was just a comma.
Another funny replacement is Epicenter, which replaced the word president again, grammatically, it's
not bad, but if you wanted to improve this, you would have to take into account very long range dependencies.
Specifically, this word still refers to the first two tokens in the sentence, which are now quite
a few tokens behind.
So this suggests the need for a model that can perhaps look at a variable number of previous tokens
and also one that can learn long range dependencies.
Keep these ideas in mind when you study deep learning and neural networks.
OK, so let's check the next sentence.
Laurent Botwin, part of the family that controls the Montreal based firm, will take over the role
of CEO under a newly created management structure.
So this one makes perfect sense.
By the way, notice how, for some reason, the de tokenize is failing at these points.
So let's check out the next sentence.
Analysts believe the resignations seem to have stemmed from a boardroom dispute.
This also makes sense, but note that it changes the meaning of the sentence.
OK, so the next sentence definitely does not make sense, although it is close under Mr Teles tenure
at the subsidy, which began in July 2003, according to Cut, the world wide workforce of 75000 by
signing a movement by two thousand six were announced.
And of course, this does not make sense because the model does not account for long range dependencies.
OK, so let's move on to the next sentence.
The firm's auto division and Defense Services Unit were also sold in Bombardier started the future of
a new aircraft seating 110 to 135 passengers.
In this case, I would say that the result makes sense.
So the next two sentences mostly make sense.
Mr. Tellier had indicated he wanted to expand at the industry's top train maker and third largest manufacturer
of civil aircraft until the restructuring was complete.
But Bombardier has been charged with a declining share price and profits.
In my opinion, these replacements are not that.
So notice that the next sentence has quite a few replacements.
Earlier this month, the government said it earned $100 million.
Nineteen point two million pounds for the third quarter, down from a bid of $133 million a year ago.
Again, this sentence doesn't make logical sense, but it does make grammatical sense.
OK, so the next sentence is a bit of a mess.
Notice that the first part looks OK.
I understand the UK's concern that I would not be there for the long term.
However, in the middle, we can see that the word end is replaced by a double quote.
In the context of programs.
This makes sense, but relative to the overall sentence, it does not make sense, since the quotes
now don't balance out.
Note that the ending of the sentence also looks OK.
Mr Tellier said in a meeting on agriculture Obviously this doesn't make any sense or relative to the
semantics of the article, but grammatically it works.
OK, so since these articles are quite long, I think that's enough for this lecture.
However, I do encourage you to look at more on your own and furthermore, think about why this article
spinner behaves like it does in particular, think about things like the size of the context window
and long term dependencies.
It's also worth thinking about what level of understanding is really required to spin an article and
make sure it makes sense semantically.
That is, without changing the meaning of what it says.
Consider whether or not anything like that currently exists at this time.
The state of the art is GPT three, and the consensus is that we are close but still not there.
Also consider whether or not anything like that is possible at all what level of intelligence would
be required?
From what I've seen, this is a much bigger leap than a lot of people think, especially for those who
are new to machine learning.
Inversion, one of this course, I was surprised at how many beginners thought something like human
level understanding could just exist.
And this was five years ago before Transformers were even invented.
From my perspective, this just goes to show that there is a pretty wide gap between what I can do in
reality versus what the public thinks I can do.
So if you're new to this course or you're new to machine learning, it's a good idea to really think
about whether your expectations are based on reality or if they are based on maybe misinformation that
you got on YouTube or elsewhere.
Now that being said, there are some ways you could extend this article spinner to make it better,
even without considering deep learning and neural networks.
Consider these to be extension exercises for this section.
So as you recall, sometimes words that are replaced in a program makes sense in the context of the
program, but not in the context of the full sentence.
One way you could potentially address this issue is to take parts of speech into account.
In this way, you'll at least have a greater chance of the result making sense.
Here's another idea.
Often we neglect to consider simple yet effective tools.
This idea is to simply use a dictionary which contains synonyms.
As you recall, a synonym means words that have the same meaning.
For example, fast and quick are small and tiny.
Note that this does not require any probabilistic model at all.
Furthermore, you could also combine this approach with the parts of speech extension.
Yet another option which has its pros and cons is to simply make the context window larger.
For example, account for a more previous words and account for more future words.
However, note that this introduces a data problem.
Since you're increasing the dimensionality of the distribution, so this is another instance of the
curse of dimensionality.
Consider how unlikely you would be to encounter the same five gram more than once.
Note also that the context window does not have to be symmetric, so you could depend on it.
Two previous words, but only one next word or one previous words and two next words.