1
00:00:00,000 --> 00:00:02,505
Welcome back. Transfer learning

2
00:00:02,505 --> 00:00:04,530
is one of the most
important techniques of

3
00:00:04,530 --> 00:00:06,690
deep learning and
TensorFlow lets you do

4
00:00:06,690 --> 00:00:09,450
that with just a handful
of lines of codes.

5
00:00:09,450 --> 00:00:10,740
Rather than needing to train

6
00:00:10,740 --> 00:00:12,180
a neural network from scratch we

7
00:00:12,180 --> 00:00:14,820
can need a lot of data and
take a long time to train,

8
00:00:14,820 --> 00:00:16,454
you can instead download

9
00:00:16,454 --> 00:00:18,300
maybe an open-source model

10
00:00:18,300 --> 00:00:19,800
that someone else has
already trained on

11
00:00:19,800 --> 00:00:22,080
a huge dataset
maybe for weeks and

12
00:00:22,080 --> 00:00:24,660
use those parameters
as a starting point to

13
00:00:24,660 --> 00:00:27,240
then train your model
just a little bit more on

14
00:00:27,240 --> 00:00:30,180
perhaps a smaller dataset that
you have for a given task,

15
00:00:30,180 --> 00:00:31,530
so it is called
transfer learning.

16
00:00:31,530 --> 00:00:34,275
Transfer learning, and I find
it's really cool because

17
00:00:34,275 --> 00:00:35,820
we've been looking at
convolutional neural

18
00:00:35,820 --> 00:00:37,735
networks up to this point,

19
00:00:37,735 --> 00:00:38,960
and when we're training with

20
00:00:38,960 --> 00:00:40,400
a convolutional
neural network we're

21
00:00:40,400 --> 00:00:41,900
extracting features from an image

22
00:00:41,900 --> 00:00:43,070
and learning those features,

23
00:00:43,070 --> 00:00:44,750
learning the ones
that match images.

24
00:00:44,750 --> 00:00:47,210
But when we download
a large models

25
00:00:47,210 --> 00:00:49,040
something like
an inception that has

26
00:00:49,040 --> 00:00:51,140
all of these features that
have already been extracted

27
00:00:51,140 --> 00:00:53,615
off lots more images
than we can use,

28
00:00:53,615 --> 00:00:55,640
I find that really
exciting because as you

29
00:00:55,640 --> 00:00:57,440
mentioned researchers who built

30
00:00:57,440 --> 00:00:59,960
these wonderful models
of lots of images

31
00:00:59,960 --> 00:01:02,860
have been able to
share them and today

32
00:01:02,860 --> 00:01:04,640
we've been looking
at convolutions and

33
00:01:04,640 --> 00:01:06,590
convolutional
neural networks where

34
00:01:06,590 --> 00:01:07,760
from scratch we've been

35
00:01:07,760 --> 00:01:10,040
training something off
of our data and off of

36
00:01:10,040 --> 00:01:11,480
our images to be able to

37
00:01:11,480 --> 00:01:13,520
extract features from
these and learn them.

38
00:01:13,520 --> 00:01:15,410
But now these models that
have been trained off with

39
00:01:15,410 --> 00:01:17,150
lots and lots of images over

40
00:01:17,150 --> 00:01:18,560
a lot of time might have

41
00:01:18,560 --> 00:01:20,480
spotted other features
that weren't in

42
00:01:20,480 --> 00:01:22,070
our dataset and will help

43
00:01:22,070 --> 00:01:24,005
us to build better
convolutional networks.

44
00:01:24,005 --> 00:01:25,910
Yes. There's so much stuff

45
00:01:25,910 --> 00:01:27,830
for neural networks to
learn about the world.

46
00:01:27,830 --> 00:01:29,525
There are edges, corners,

47
00:01:29,525 --> 00:01:31,310
round shapes, curvy shapes,

48
00:01:31,310 --> 00:01:33,975
blobs and then there
are things like eyes,

49
00:01:33,975 --> 00:01:36,830
circles, squares, wheels.

50
00:01:36,830 --> 00:01:38,330
There's so many things
in the world that

51
00:01:38,330 --> 00:01:39,740
convolutional neural networks can

52
00:01:39,740 --> 00:01:40,980
pick up on but if you have only

53
00:01:40,980 --> 00:01:42,980
a 1,000 images or even 25,000

54
00:01:42,980 --> 00:01:44,210
images may not be

55
00:01:44,210 --> 00:01:46,280
enough data for content to
learn all those things.

56
00:01:46,280 --> 00:01:49,070
So by taking an inception network

57
00:01:49,070 --> 00:01:51,200
or some other network that
someone else's has trained you

58
00:01:51,200 --> 00:01:53,720
can basically download
all this knowledge into

59
00:01:53,720 --> 00:01:55,205
your neural network to give it a

60
00:01:55,205 --> 00:01:57,470
huge and much faster start.

61
00:01:57,470 --> 00:01:59,060
It's really exciting
to be a part of

62
00:01:59,060 --> 00:02:01,610
that community where
these things are being shared so

63
00:02:01,610 --> 00:02:04,085
that people who
don't have access to

64
00:02:04,085 --> 00:02:05,270
machinery to be able to build

65
00:02:05,270 --> 00:02:07,160
massive models like this one,

66
00:02:07,160 --> 00:02:08,600
are able to derive

67
00:02:08,600 --> 00:02:11,015
the features from the people
who have done so.

68
00:02:11,015 --> 00:02:13,985
I personally find that
really inspirational.

69
00:02:13,985 --> 00:02:16,070
Yes. Many days I wake up really

70
00:02:16,070 --> 00:02:18,649
grateful to the whole AI
community for the openness,

71
00:02:18,649 --> 00:02:20,270
so many breakthrough AI ideas are

72
00:02:20,270 --> 00:02:22,250
shared in papers on the Internet,

73
00:02:22,250 --> 00:02:24,185
on the open source code
shared on the Internet,

74
00:02:24,185 --> 00:02:25,820
and I think this has been

75
00:02:25,820 --> 00:02:29,220
a large reason for
the rapid advance of AI.

76
00:02:29,220 --> 00:02:31,400
I think to all the
learners if you end

77
00:02:31,400 --> 00:02:33,620
up doing something
consider sharing

78
00:02:33,620 --> 00:02:36,410
it on the Internet as well
freely to contribute back to

79
00:02:36,410 --> 00:02:38,245
this open community that's making

80
00:02:38,245 --> 00:02:40,315
all of us go much faster.

81
00:02:40,315 --> 00:02:41,195
Definitely.

82
00:02:41,195 --> 00:02:44,630
Transfer learning is
a technical embodiment that

83
00:02:44,630 --> 00:02:48,515
lets us use these ideas to
accelerate the whole field.

84
00:02:48,515 --> 00:02:50,870
One of the cool things
about transfer learning is

85
00:02:50,870 --> 00:02:53,910
that it's so simple to implement,

86
00:02:53,910 --> 00:02:55,970
in TensorFlow you download
a model and then you

87
00:02:55,970 --> 00:02:58,130
say set these models as

88
00:02:58,130 --> 00:02:59,960
trainable and freeze or lock

89
00:02:59,960 --> 00:03:02,455
those other layers and
then you just run.

90
00:03:02,455 --> 00:03:04,520
Yes. In the course what

91
00:03:04,520 --> 00:03:05,870
the students are going
to be looking at

92
00:03:05,870 --> 00:03:07,400
is the inception model

93
00:03:07,400 --> 00:03:09,649
which is a beautiful
image classification

94
00:03:09,649 --> 00:03:12,485
and classifies a 1,000 images
it has been trained on.

95
00:03:12,485 --> 00:03:13,970
I think it's like over a million

96
00:03:13,970 --> 00:03:15,500
images and what you're going

97
00:03:15,500 --> 00:03:16,880
to do is you're
going to take a look

98
00:03:16,880 --> 00:03:18,350
at one of the lower
levels in that.

99
00:03:18,350 --> 00:03:19,995
You're going to lock all
the way down to that.

100
00:03:19,995 --> 00:03:21,650
You're going to add a DNN

101
00:03:21,650 --> 00:03:23,400
a deep neural network
underneath that,

102
00:03:23,400 --> 00:03:24,770
and then you're just
going to retrain

103
00:03:24,770 --> 00:03:26,250
for those lower levels,

104
00:03:26,250 --> 00:03:28,030
and as a result
using all of these,

105
00:03:28,030 --> 00:03:30,365
I like to call it standing
on the shoulders of giants,

106
00:03:30,365 --> 00:03:32,360
you're going to be doing
all that to be able

107
00:03:32,360 --> 00:03:34,400
to make the classifiers
that you've been

108
00:03:34,400 --> 00:03:36,215
building in the course to date

109
00:03:36,215 --> 00:03:38,150
much more efficient
and maybe even

110
00:03:38,150 --> 00:03:40,130
quicker to be able to
reach higher levels of

111
00:03:40,130 --> 00:03:42,730
accuracy than if you're
training it from scratch.

112
00:03:42,730 --> 00:03:44,915
So this week, you'll learn how to

113
00:03:44,915 --> 00:03:47,750
implement transfer learning
and use it to get

114
00:03:47,750 --> 00:03:50,510
your models to not
only train faster but

115
00:03:50,510 --> 00:03:54,330
also get higher accuracy.
Please dive in.