1
00:00:00,000 --> 00:00:02,340
There can be a limitation

2
00:00:02,340 --> 00:00:05,235
when approaching
text classification in this way.

3
00:00:05,235 --> 00:00:07,605
Consider the following.
Here's a sentence.

4
00:00:07,605 --> 00:00:10,050
Today has a beautiful blue.

5
00:00:10,050 --> 00:00:13,230
What do you think would
come next? Probably sky.

6
00:00:13,230 --> 00:00:16,125
Right? Today has
a beautiful blue sky.

7
00:00:16,125 --> 00:00:17,955
Why would you say that?

8
00:00:17,955 --> 00:00:20,580
Well, there's a big clue
in the word blue.

9
00:00:20,580 --> 00:00:21,980
In a context like this,

10
00:00:21,980 --> 00:00:23,790
it's quite likely that when we're

11
00:00:23,790 --> 00:00:26,145
talking about a beautiful
blue something,

12
00:00:26,145 --> 00:00:28,140
we mean a beautiful blue sky.

13
00:00:28,140 --> 00:00:30,810
So, the context word
that helps us understand

14
00:00:30,810 --> 00:00:32,580
the next word is very

15
00:00:32,580 --> 00:00:34,950
close to the word that
we're interested in.

16
00:00:34,950 --> 00:00:37,440
But, what about
a sentence like this,

17
00:00:37,440 --> 00:00:39,750
I lived in Ireland
so at school they

18
00:00:39,750 --> 00:00:42,480
made me learn how
to speak something.

19
00:00:42,480 --> 00:00:44,340
How would you finish
that sentence?

20
00:00:44,340 --> 00:00:46,175
Well, you might say Irish

21
00:00:46,175 --> 00:00:48,605
but you'd be much more
accurate if you said,

22
00:00:48,605 --> 00:00:51,020
I lived in Ireland so at school

23
00:00:51,020 --> 00:00:53,645
they made me learn
how to speak Gaelic.

24
00:00:53,645 --> 00:00:56,465
First of course, is
the syntactic issue.

25
00:00:56,465 --> 00:00:58,095
Irish describes the people,

26
00:00:58,095 --> 00:01:00,000
Gaelic describes the language.

27
00:01:00,000 --> 00:01:02,720
But more importantly
in the ML context is

28
00:01:02,720 --> 00:01:04,310
the key word that gives us

29
00:01:04,310 --> 00:01:06,440
the details about the language.

30
00:01:06,440 --> 00:01:08,180
That's the word Ireland,

31
00:01:08,180 --> 00:01:10,780
which appears much
earlier in the sentence.

32
00:01:10,780 --> 00:01:12,650
So, if we're looking
at a sequence of

33
00:01:12,650 --> 00:01:15,665
words we might lose that context.

34
00:01:15,665 --> 00:01:19,760
With that in mind an update
to RNNs is called LSTM,

35
00:01:19,760 --> 00:01:22,525
long short - term memory
has been created.

36
00:01:22,525 --> 00:01:26,600
In addition to the context
being PaaSed as it is in RNNs,

37
00:01:26,600 --> 00:01:28,880
LSTMs have an additional pipeline

38
00:01:28,880 --> 00:01:31,070
of contexts called cell state.

39
00:01:31,070 --> 00:01:34,325
This can pass through
the network to impact it.

40
00:01:34,325 --> 00:01:38,495
This helps keep context from
earlier tokens relevance

41
00:01:38,495 --> 00:01:40,580
in later ones so issues like

42
00:01:40,580 --> 00:01:43,465
the one that we just
discussed can be avoided.

43
00:01:43,465 --> 00:01:47,010
Cell states can also
be bidirectional.

44
00:01:47,010 --> 00:01:49,250
So later contexts can impact

45
00:01:49,250 --> 00:01:52,480
earlier ones as we'll see
when we look at the code.

46
00:01:52,480 --> 00:01:55,910
The detail about LSTMs
is beyond the scope of

47
00:01:55,910 --> 00:01:57,590
this course but
you can learn more

48
00:01:57,590 --> 00:02:00,330
about them in
this video from Andrew.