1
00:00:00,750 --> 00:00:05,100
Once we have a model and a period,
then we can evaluate the model on it, and

2
00:00:05,100 --> 00:00:07,410
we'll need a metric to
calculate their performance.

3
00:00:08,970 --> 00:00:11,750
So let's start simply by
calculating the errors,

4
00:00:11,750 --> 00:00:15,280
which is the difference between
the forecasted values from our model and

5
00:00:15,280 --> 00:00:17,829
the actual values over
the evaluation period.

6
00:00:19,100 --> 00:00:22,910
The most common metric to evaluate
the forecasting performance of a model

7
00:00:22,910 --> 00:00:27,440
is the mean squared error or
mse where we square the errors and

8
00:00:27,440 --> 00:00:29,070
then calculate their mean.

9
00:00:29,070 --> 00:00:30,790
Well, why would we square it?

10
00:00:30,790 --> 00:00:33,510
Well, the reason for
this is to get rid of negative values.

11
00:00:33,510 --> 00:00:37,750
So, for example, if our error was two
above the value, then it will be two, but

12
00:00:37,750 --> 00:00:41,040
if it were two below the value,
then it will be minus two.

13
00:00:41,040 --> 00:00:43,400
These errors could then
effectively cancel each other out,

14
00:00:43,400 --> 00:00:47,550
which will be wrong because we
have two errors and not none.

15
00:00:47,550 --> 00:00:51,380
But if we square the error of value before
analyzing, then both of these errors would

16
00:00:51,380 --> 00:00:55,540
square to four, not canceling each
other out and effectively being equal.

17
00:00:57,460 --> 00:01:01,810
And if we want the mean of our errors'
calculation to be of the same scale

18
00:01:01,810 --> 00:01:05,520
as the original errors,
then we just get its square root,

19
00:01:05,520 --> 00:01:08,908
giving us a root means squared error or
rmse.

20
00:01:08,908 --> 00:01:14,866
Another common metric and one of my
favorites is the mean absolute error or

21
00:01:14,866 --> 00:01:20,068
mae, and it's also called the main
absolute deviation or mad.

22
00:01:20,068 --> 00:01:22,725
And in this case, instead of
squaring to get rid of negatives,

23
00:01:22,725 --> 00:01:25,080
it just uses their absolute value.

24
00:01:25,080 --> 00:01:29,110
This does not penalize large
errors as much as the mse does.

25
00:01:29,110 --> 00:01:33,271
Depending on your task,
you may prefer the mae or the mse.

26
00:01:33,271 --> 00:01:36,448
For example, if large errors
are potentially dangerous and

27
00:01:36,448 --> 00:01:41,120
they cost you much more than smaller
errors, then you may prefer the mse.

28
00:01:41,120 --> 00:01:42,030
But if your gain or

29
00:01:42,030 --> 00:01:46,585
your loss is just proportional to the size
of the error, then the mae may be better.

30
00:01:48,405 --> 00:01:52,355
Also, you can measure the mean
absolute percentage error or mape,

31
00:01:52,355 --> 00:01:57,965
this is the mean ratio between
the absolute error and the absolute value,

32
00:01:57,965 --> 00:02:01,525
this gives an idea of the size of
the errors compared to the values.

33
00:02:02,625 --> 00:02:06,675
If we look at our data, we can
measure the MAE using code like this.

34
00:02:07,700 --> 00:02:13,180
The keras metrics libraries include
an MAE that can be called like this.

35
00:02:13,180 --> 00:02:16,932
With the synthetic data we showed earlier,
we're getting about 5.93,

36
00:02:16,932 --> 00:02:19,190
let's consider that our baseline.