Oh, yeah.
So I hope you enjoyed those lessons on convolutional neural networks.
However, we're taking a step, a bit of a step back now.
We're going to take a look at the history of deep learning, and it's a very important reason why I'm
doing this for because right now we are basically on the treadmill on the cusp of a revolution.
That's that's because of deep learning.
And I'll tell you why.
Firstly, yes, now is definitely the best time to become a deep learning project practitioner.
We're in the golden age of AI.
There's never been a time right now where things are moving so fast and everything is coming together.
I've actually had this discussion in a few meetups and with lots of other people and in Big Tech companies
like Google and Facebook and in video, and most of them share this view.
It basically is.
A graph will show you the next two slides, so I'll talk a bit about more about it.
Do.
But a lot of people underestimate the potential of a it kind of goes through a cycle where people overestimate
and people underestimate.
And is a reason for that is because continuously disappointed in the past, but something is changing
now and that something is related to both deep learning and the accessibility of faster GPUs.
GPU hardware, which is required to train these deep learning models, which is otherwise going on the
CPU, is just is not feasible.
It's GPUs have become so much faster thanks to in video meanly not not to bash in AMD, but AMD doesn't
have CUDA support.
And that's basically the low level library that all these deep learning libraries are built on.
Now AMD is developing something.
I think it's Vulkan.
I'm not entirely sure much about it, but it's coming.
So AMD is not going to be left behind.
However, in video basically has a monopoly on this market right now, and GPUs are even free right
now because as you're using Google collab, which we're using in open CME section, we're going to go
back and do inter-club very soon, actually.
And maybe the next three or four after the next three or four lectures, we'll start using CoLab to
train with GPUs over convolutional neural networks, using both PyTorch Tensor for Keros.
So you'll see how much faster these network streamer GPUs and these are you use to free up using CoLab.
Honestly, they're kind of ancient, and they were quite slow.
They are faster debuts be colab that you can pay for.
I do pay for it because it's quite cheap.
It's $10 a month and you get access to much faster P 100s or T100 GPUs if they're available.
So it's actually a pretty decent substitute when you do have to do some simple models.
So it's another thing as well.
Deep learning and deep learning libraries have become much more mature, and they're quite easy to use,
to be honest.
It's very easy to train at CNN using cameras or by touch.
It doesn't take that much knowledge to get up and running.
However, you do need to understand if you need to make improvements or tweaks.
You do need to understand the inner workings of deep learning, which you would have already achieve
if you watched the previous hour of lectures and things weren't always this way.
And let's talk a bit about that.
Let's talk about how we got here.
So let's talk about the history of in deep learning.
So it has been around for a while, actually.
I mean, a lot of science fiction writers obviously talked about machines and robots.
I could think maybe like a hundred years ago or so.
However, really, I research started like maybe after World War Two, even probably during World War
Two, where a lot of these groundbreaking computer scientists and mathematicians were working on simple
A.I. concepts like the first neuron for artificial neuron algorithm was in 1943 by these guys.
So and then they continued a lot of work.
They actually did some work where they actually use continuous, a continuous back propagation model
based on the basics of it to train some of these these networks.
So things were coming along, coming along nicely.
However, back then, and even maybe till like five, 10, 10 years ago, we just simply didn't have
the computing power and it was easily accessible to train these new neural networks.
So a lot of alternative methods and machine learning were developed, just cycle modeling and machine
learning.
These were these were quite good, actually.
However, they reached a point where they had diminishing returns, where we just couldn't get much
better performance out of them.
And oftentimes these algorithms were too slow and inefficient and also error prone.
So they also suffered from the curse of dimensionality, which means that when your dataset grows.
Just becomes exponentially harder to train that type of model using these methods.
So the deep learning did change a lot of this.
So let's talk about it a winter.
Lots of people in the world don't even know what this is, but in the 1970s and in the late 60s, lots
of people were very optimistic about it.
They thought like computers that could understand video text, even talk back to us, even like some
sort of semi consciousness was achievable within was achievable in their lifetime.
However, as the as more research went along, it was it was.
It was realized that this wasn't going to be possible.
So a lot of funding was slashed and many cleaned air, which is hype, that it wasn't possible.
So it kind of went into the depression, and this graph gives us a good example of it.
So you can see this is the debate of genesis, of a lot of hype, a lot of speculation than a lot of
failures and a lot of wasted money, which is important because a lot of this air work takes a lot of
funding to achieve.
So this was the winter.
Then maybe in the 80s, when some machine learning algorithms started to make more progress and computers
became more accessible.
There was the early Commodore 64 and your old Apple Macintoshes, so a lot of people did think maybe
we can probably start getting some simple UI stuff done.
And then those hopes vanished again.
And that was in the 90s.
And maybe it's basically since then we've seen growth.
This graph isn't really sure, surely explosive growth that we have seen in recent time, but we've
seen growth in AI.
It was incremental.
It was it was lots of good algorithms, screams, machine learning algorithms came along.
Then we started to do a simple convolutional neural networks and neural networks and elastic arms.
And you don't know what all of these terms yet, but they're just types of neural networks.
And it was it was the potential was seen and it grew and it grew and grew.
And right now, honestly, right now, I think in the last year or two, I'm seeing so much improvement
in computer vision world with all of this new models and methods we're using.
It's been it's been an explosion, and I don't see it slowing down anytime soon because there's still
so much work to be done in computer vision and in a simple, a simple thing like a robot.
Navigating inside of a home requires so much computer vision, knowledge and processing, not just because
he wants to navigate and avoid obstacles, but he has to understand his environment.
Yes, understand that this is a door.
This is a doorknob.
This is a television.
This is a sink or bathroom and you need needs.
He needs to see and visually a process and understand the scene that he's in.
So it's not that hard yet.
But what's also hard is actually making this work in real time, processes information accurately and
then make decisions out of it.
So it's also tying into tying in together all of the the building blocks of an air system.
So new cognition, cognition, sorry, was one of the first neural networks, and this was released
in 1980.
Actually, it was solid work in 1979 by these guys.
And basically, this was like the grandfather of neural networks.
And people actually did that propagation back in FORTRAN in the 70s, because back propagation is a
relatively simple algorithm.
Once you can do the numerical calculations to get the derivatives, it's relatively.
I mean, I wouldn't say it's easy to implement.
This is actually good looks, but it's probably a lot more code in this.
It's probably just a small, tiny block of it.
It's probably pretty complicated because I can imagine in Python using a lot of libraries as well.
It's still relatively hard.
I wouldn't say that difficult, but relatively hard to code a neural network from scratch and do the
training process.
However, it's a very, very good exercise if you want to understand deep learning and in reality.
So if you do, you can search Google building a neural net from scratch in Python.
There's a lot of blogs that talk about this.
It's a very good exercise to do so.
Things did pick up, though.
As you can see, I talked about this here, the basically this point, but that was actually end of
the year I went to when I started to create new hopes.
So that just mentioning that there.
So here's the thing.
GPAs and deep learning are a match made in heaven.
And yes, I know there are specialized deep learning hardware out there.
However, no one is going to beat the ubiquity and accessibility of chips.
Even though I know right now in 2021 to late 2021, I'm recording.
This is still a GPU shortage.
Like trying to get a 290 end video card is next to impossible sometimes.
There was a company I used to do some work with that was trying to get this GPU since since April,
that was about six months ago and still haven't gotten it yet.
So think of a GP use as like a four door sedan, so sorry, CPU's as a four door sedan.
They're very good all vehicles, but they're not good at specializing in one general thing.
So that's why that's why we need GP use.
But think about what GP use are and what gives them the capacity to be good at deep learning.
Well, let's loosen that analogy.
GP use are like a huge truck.
They're good for wanting, but not good for other things.
Handling is bad in trucks.
It's not going to be a very comfortable ride.
But at the same time, it can pull a lot of weight.
So in essence, this is that's what the GP who is a GP you is does some calculations, some types of
calculations, very, very efficiently, and those types of calculations can actually be skilled quite
well.
So he can use multiple cores.
That's why in Nvidia's GPUs, they have like two thousand up to 4000 CUDA cores.
And all of these things allow it to achieve parallel processing, which allows us to do so many calculations
at once, and we can see a comparison afterwards.
But for now, let's talk about that.
While GPUs are mainly were originally used for treaty processing and I'm basically rendering scenes
as well.
They were actually not really designed for deep learning at that point.
However, the reason they were used for deep learning processes is because they performed floating point
operations very, very quickly and they have dedicated memory.
So this allows high bandwidth movement of data within the GPU, so it becomes very, very good for this
sort of task specific task.
So you have come a long way because look at this here.
In 2003, the best CPU just barely had but had more theoretical computation performance than a CPU.
And that's understandable because GPUs do dedicated types of calculations and they're able to do it
very well.
However, look at the difference in computational performance GPUs made compared to CPUs.
CPUs started around here.
This is like maybe a two or three and right note.
Even this is probably a lot more now.
But even in the last five years, Djibouti CPUs have not improved that much.
The last big jump in and CPU's with some of AMD's Threadripper CPUs.
Also Apple's M1, which is a risk CPU for Abrams review, was outstanding compared to when you compared
it to performance of its CPUs.
But nevertheless is still pale in comparison to the actual theoretical.
Computational performance of CPUs and GPUs have actually made so much strides in the last few years,
like Nvidia's 3000 series is phenomenally fast.
So I hope this lesson this lecture I know it wasn't a more smooth kind of well-put to get together lecture.
However, I just wanted you to understand what's making deep learning so accessible right now.
It's a combination of GPU use becoming cheaper and faster, deep learning libraries becoming much,
much more mature and easy to use.
And then all this research, all these research that's just bombarding us from all around because tech
big big tech companies like Facebook, Google, Amazon, Netflix, Apple, Microsoft, they're they're
pouring so much money and resources into deep learning, and they're open sourcing almost all of their
software, at least all of the good software, in my opinion.
So it's becoming very, very it's becoming a very good time to get into this field.
So we'll stop there and we'll take a look now at some of the most common and popular deep learning libraries,
which you will be using in discourse.
So I'll see you in the next section.
Thank you.