Welcome back to practical
time series analysis. In this lecture, we look at
the Akaike Information Criterion. As a way of figuring out the quality of
a model, assessing the quality of a model, there's an interesting issue
that comes and supply for us. At this point, you know that if you
have an autoregressive model or moving average model, we have techniques available to us to
estimate the coefficients of those models. We'll, of course, usually do that with
software rather than write the code by ourselves,
rather than code it by hand, but if I told you that you had a third
order autoregressive model, pretty trivially you could tell me what
good estimates are for those coefficients. A more fundamental question, perhaps,
is what is the order of the model? In general, you won't know. You don't usually have the kind of
intimate process knowledge that will tell you the order of the model. And you're going to try to infer
the order from the data set that you have in front of you. So as we look here in this lecture,
we'll try to measure the quality of the model with
the Akaike Information Criterion. I suppose I could keep apologizing for my pronunciation,
I'm just doing the best I can. We'll try to measure the quality
of model with the AIC and the SSE, the sum of the squares of the Earth. And you should be able to do that for a time series that you find
interesting after this lecture. You should also be able
to explain to a friend or a colleague in fairly simple terms
what it is that you're doing. So here's our first example we'll look at. The Loblolly data set is one
that we enjoy working with. And if you look at these tree
heights as a function of age here, a couple things I think
are immediately apparent. One is, there is certainly
a trend in this data set. As time goes on,
the trees get taller and taller. That's probably not very surprising. As I look through here, too, I do see that
there is heteroscedasticity in the sense that the variability is increasing as
I move from the left to the right. We won't worry about that
in this particular example. Right now, we're just going
to do a very simple thing. Given these data,
we'll fit a straight line to the data. Now, the green line here, you can see,
is doing an okay job capturing the trend. But I think we could do better. If you fit, instead of a first
order polynomial, a straight line. If you fit a second order polynomial,
here we've got a concave down parabola, then we're doing a much better
job of capturing the centers of each of these little parts of the data
set as we move from left to right. It's a little surprising though if you
look at our numerical measures of quality. Multiple R-squared coefficient
determination, you'll remember, is described by many people
is the amount of variability that we can explain through our model. It's up on the straight
line at around 98%. It's true. So we were from a straight
line to a parabola. Multiplying our square
does increase up to 99.3%. But still, that different is probably
not as profound as you might think, given the visual evidence that
we have over here on the left. We talked previously about
Adjusted R-squared values, as well. Those curves are analogous to the AIC,
in the sense that the Adjusted R-squared makes you pay a penalty as you bring
higher order terms into your model. As I move from a first to second order
model, I can definitely justify that. As I move from a second
to third order model, you should take the data set Loblolly and
see if you believe that the enhancement in variability explained
is worth the third order term. Now, the data sets that we find in nature
tend to be rather messy and complicated. What we're going to do in this
lecture is a simulation, and so we will simulate a second
order aggressive process. We'll have 2,000 data points so
we have a fairly large number of data points we should be
able to do our estimation pretty well. We'll do the usual things,
we'll take a look at our data set, we'll also look at the ACF and
the partial ACF. So in our simulated data,
the ACF seems to trail off, the PACF seems to exhibit
two significant terms there. This seems rather consistent
with the second order, order regressive process that we have. But the ACF and the PACF are probably
really not enough by themselves to really establish what sort of model you have. We're going to look for less objective
numerical measures of quality. So just to review, if you want to
estimate the coefficients on a model and you know the order, then you can
make a call to the arima command. It's going to give us an ar1 and
ar2 coefficient set as 0.7 and a little bit and negative 0.2 over here. We're doing a pretty good
job at our estimation. But again, that's just because we knew,
because we generated the data, that it was a second order data set. In general, you won't know what
the order of your data set is, and so you'll have to make a guess over here,
as to your P value. What we'll try to do with the AAC
is make that an educated guess. Down here, you can see that arima,
which is a pretty generic call is going to give you somethings that
it thinks are just mission critical, things that you should know. One of those, right there is the aic. So this is a fairly standard measure the people use when they're
comparing model quality. What I've done in this table
is nothing complicated, you could reproduce this yourself. I've gone through and let p equal 1,
first order regressive model. Let p equal 2, ,4 and 5,
all the way up through fifth order. And with each of these
increasingly complex models, we have estimated our coefficients. No huge surprises as we look at those and
we've also given in the table the AIC and the more prosaic SSC
some squares of the areas. It looks like as I move from the first
order to the second order model, I get a pretty good drop in these terms. As I move from a second to a third, third to a fourth,
I'm not seeing as much difference. What I'll do on the next
slide is give you a picture. So as I look here, the SSE which is pretty
obvious and well understood by us at this point, takes a good drop as they go from
a first order to a second order model. And then, it pretty much stays the same. There's a little bit of
variability there but the big payoff is as you move from
a first to a second order model. The AIC picture is telling us
pretty much the same story. As we move from a first to a second order
model we get a pretty good drop off. As we move out to a fifth order model, it's true that the AIC here is
somewhat less than on these others. But you have to ask yourself whether you
think that that very small diminution in AIC is really worth the added
complexity of the higher order model. Based upon these scree plots here, I would
definitely go with a second order model. So what does the AIC try to do? It's rather common as you're getting
the feeling at this point for sure, to give credit for models that are going to
reduce some sort of aggregate error. And we'll use the error sums of squared
as the handiest, most common example. But we'd also like to build in
some kind of penalty when out models start bringing in more and
more terms. Formal definition of the AIC is
going to have two terms in it. In a course in probability, you may have talked about
maximum likelihood estimation. In this lecture, we're not going to
dive to deep into this first term. But just keep, as people say, as you're
flying at 30,000 feet, keep the basic idea in your head that you want something
which is going to give you credit for reducing some sort of aggregate error, but make you pay some kind of penalty for
the number of parameters in the model. So you’ll see different versions
in different textbooks, software has different implementations,
the AIC, a very simple version is, let's go take the log of
the estimated variability. That'll be a term that for
a good model with low SSE will be low and then we'll delve in another term
here where n is your sample size, that's unchanging but
as your number parameters increases, you pay penalty through this term. The SSE, just to review, is very similar,
but it doesn't make you pay a penalty. Pick a potential value of your order,
figure model, and look at the aggregated error. So we've done that here. For our arima model,
we've explored with various values of p. We will produce the output Here in m. And then, we'll interrogate our model
here through the command resid. So we'll pull off the errors here
with resid(m), we'll square them and then aggregate them by adding them. At this point, you should be able to use
the AIC to measure the quality of a model. Especially when you have
several competing models, and you're trying to establish
the order of your process. And you should be able to describe in
fairly causal terms to a friend or colleague what it is that you're doing.