1
00:00:00,000 --> 00:00:02,460
Welcome back. In this video,

2
00:00:02,460 --> 00:00:03,810
you'll learn how to implement

3
00:00:03,810 --> 00:00:06,360
sequence models, in TensorFlow,

4
00:00:06,360 --> 00:00:09,090
and if you've heard of
models like an RNN,

5
00:00:09,090 --> 00:00:10,665
or Recurrent Neural Network,

6
00:00:10,665 --> 00:00:13,350
this week we'll teach you
how to implement that.

7
00:00:13,350 --> 00:00:15,420
It's important to
really take words,

8
00:00:15,420 --> 00:00:17,235
and compute embeddings for them,

9
00:00:17,235 --> 00:00:18,855
but the relative ordering,

10
00:00:18,855 --> 00:00:20,160
the sequence of words

11
00:00:20,160 --> 00:00:22,260
matters too for the
meaning of a sentence,

12
00:00:22,260 --> 00:00:23,670
and if you jumble
the words around,

13
00:00:23,670 --> 00:00:26,625
that changes, or destroys
the meaning of a sentence.

14
00:00:26,625 --> 00:00:28,290
Definitely, and so if you

15
00:00:28,290 --> 00:00:30,060
take just a very simple sentence,

16
00:00:30,060 --> 00:00:32,130
like my dog sat on my hat,

17
00:00:32,130 --> 00:00:34,215
and you swap
the words dog, and hat,

18
00:00:34,215 --> 00:00:35,340
you're now changing from

19
00:00:35,340 --> 00:00:36,810
a sentence that has
some kind of meaning,

20
00:00:36,810 --> 00:00:38,310
and some semantics, to

21
00:00:38,310 --> 00:00:40,020
a ridiculous sentence that
has no meaning, right?

22
00:00:40,020 --> 00:00:41,100
My hat sat on my dog,

23
00:00:41,100 --> 00:00:42,600
unless you have
a really smart hat,

24
00:00:42,600 --> 00:00:44,280
it doesn't really mean anything.

25
00:00:44,280 --> 00:00:47,860
So when we were doing
classification based on embeddings,

26
00:00:47,860 --> 00:00:50,890
it was very nice that we had
words with similar meanings,

27
00:00:50,890 --> 00:00:52,380
labels in a particular way,

28
00:00:52,380 --> 00:00:54,410
so that we could say okay,
this is a positive review,

29
00:00:54,410 --> 00:00:56,525
and I got a bunch of
vectors that are similar,

30
00:00:56,525 --> 00:00:57,830
this is a negative review,

31
00:00:57,830 --> 00:00:59,705
and I got these vectors
in a similar way,

32
00:00:59,705 --> 00:01:01,495
but ordering those then

33
00:01:01,495 --> 00:01:03,110
gives us that whole
extra layer of

34
00:01:03,110 --> 00:01:04,550
meaning that'll help
us to understand

35
00:01:04,550 --> 00:01:06,880
the sentence rather than
just a simple classification.

36
00:01:06,880 --> 00:01:08,090
So for a neural network,

37
00:01:08,090 --> 00:01:11,210
to take into account
the ordering of the words,

38
00:01:11,210 --> 00:01:14,300
people now use specialized
Neural Network Architectures,

39
00:01:14,300 --> 00:01:16,130
things like an RNN,

40
00:01:16,130 --> 00:01:17,660
or GRU, or LSTM,

41
00:01:17,660 --> 00:01:20,375
and you see what all these terms
mean, and a little bit,

42
00:01:20,375 --> 00:01:22,700
in order for these
Specialized Neural Networks

43
00:01:22,700 --> 00:01:24,420
to process natural language.

44
00:01:24,420 --> 00:01:26,295
Yeah. So like
something like an RNN,

45
00:01:26,295 --> 00:01:27,330
it's really interesting that,

46
00:01:27,330 --> 00:01:30,120
the context is preserved
from timestamp to timestamp,

47
00:01:30,120 --> 00:01:31,740
which can be really useful,

48
00:01:31,740 --> 00:01:35,340
but that might get lost
in longer sentences,

49
00:01:35,340 --> 00:01:37,320
and that's why I really
love LSTMs because

50
00:01:37,320 --> 00:01:39,420
LSTMs have that cell state,

51
00:01:39,420 --> 00:01:41,250
and the cell state
are almost like

52
00:01:41,250 --> 00:01:44,340
a conveyor belts carrying
contexts all the way down,

53
00:01:44,340 --> 00:01:46,485
for a long way down timestamp.

54
00:01:46,485 --> 00:01:48,110
I love that you have a favorite

55
00:01:48,110 --> 00:01:50,315
flavor on how sequence follow.

56
00:01:50,315 --> 00:01:53,885
I hold no shame, and
keeping favorites.

57
00:01:53,885 --> 00:01:57,725
So then, in a really long
sentence, like for example,

58
00:01:57,725 --> 00:01:59,360
I grew up in Ireland,

59
00:01:59,360 --> 00:02:00,680
so I went to school,

60
00:02:00,680 --> 00:02:04,325
and at school, they made me
learn how to speak something.

61
00:02:04,325 --> 00:02:06,720
It's like you could say,
okay the context from speak,

62
00:02:06,720 --> 00:02:08,605
means that it's a language,

63
00:02:08,605 --> 00:02:10,310
but you have to go
all the way back to

64
00:02:10,310 --> 00:02:12,425
the beginning of the sentence
to see that it's Ireland,

65
00:02:12,425 --> 00:02:14,690
and then in Ireland, I
would learn how to speak.

66
00:02:14,690 --> 00:02:16,920
You speak Gaelic.

67
00:02:16,920 --> 00:02:19,030
Really badly.

68
00:02:19,360 --> 00:02:22,895
Since we learn to infant
all of these things.

69
00:02:22,895 --> 00:02:25,500
Let's go on to the next video.