The RATS Software Forum

Thank you Tom and Happy Easter.

Apologies for annoying you again but I wanted to run a modified DM test using your code with LWINDOW=NEWEY and LAGS=15 as we discussed previously.

Does the MDM test have to be conducted with a truncated kernel for calculating the HAC standard errors (this is mentioned in a previous post and reflected in your code)? If so, why?

Assuming it has to be truncated, would it still be appropriate for me to run my MDM test (with the overlapping issue in my data caused by using annual forecasts updated monthly) with LWINDOW=TRUNCATED and LAGS=15?

Thank you again

The modification in the MDM test is strictly for the truncated kernel.

Hello everyone:

I did some OOS forecasting using Rats 10.0, and then utilized both @dmariano & @ClarkForeTest to test predictive accuracy. @dmariano worked very well, but failed to run @ClarkForeTest. I have downloaded the source file (clarkforetest.src) from Estima official website for sure. I wiil greatly appreciate it if anyone knows & answers me what's the problem.
.
.
@dmariano ds_t naive fores(1)
@ClarkForeTest ds_t naive fores(1)
********************************************
Diebold-Mariano Forecast Comparison Test
Forecasts of DS_T over 1996:04 to 2013:02

Forecast MSE Test Stat P(DM>x)
NAIVE 1.90517027 2.3954 0.00830
FORES(1) 1.15432041 -2.3954 0.99170

## CP18. CLARKFORETEST is not the Name of a PROCEDURE. (Did you forget to SOURCE?)
>>>>end proc<<<<
********************************************

You may have downloaded the procedure, but you didn't but it in the same directory as the other procedures. Typically, if you download a file from the internet it just goes into your standard Windows downloads directory, from which you would have to move it. If you know where you want to put a file, right click on the link and do Save Link As... which will let you browse to the directory you want.

Dear Tom:

Thank you for your reply. I dowloaded 'clarkforetest.src' and placed it in the same file with my all other Rats codes. However, the same problems still happen. I would like to make sure if the command, @ClarkForeTest, really works in your computer? I'm wondering if any other similar command(s) available in Rats version 10 (such as, Clark & West, 2006;2007)? I now can play both @GNewbold & @dmariano. Thanks for your time.

Hi Tom,

I'm unclear regarding the definition of "nested" models and The Diebold-Mariano test.

In the RATS 10.0 Users Guide UG-171 there's an example
I think there's a typo as in the paragraph above:
"For instance, consider the rolling regressions example on page UG–159" --- there's no rolling regression on UG-159, that's the beginning of the chapter, so I'm guessing it's the model on UG-168.

Are these the models?
The model generating forecasts naive: y(t) = 1.0*y(t-1)
The model generating forecasts rhat: y(t) = c + t + alpha*y(t-1)
To my understanding they are NOT nested as "one model is not a special case of the other", hence the DM test can be calculated. Is that correct?

Can the DM test be applied to?
The model generating forecasts naive: y(t) = 1.0*y(t-1)
The model generating forecasts full AR(3): y(t) = c + alpha1*y(t-1) + alpha2*y(t-2) + alpha3*y(t-3)

What if I change the naive to
mean/constant model: y(t) = c
OR
random walk with drift: y(t) = 1.0*y(t-1) + c
Can I still apply the DM test to compare
mean/constant model & full AR(3)?
random walk with drift & full AR(3)?

And if I change the full AR(3) to AR({1,3})?

many thanks,
Amarjit

Thank you for pointing out the broken page reference.

ac_1 wrote: Are these the models?
The model generating forecasts naive: y(t) = 1.0*y(t-1)
The model generating forecasts rhat: y(t) = c + t + alpha*y(t-1)
To my understanding they are NOT nested as "one model is not a special case of the other", hence the DM test can be calculated. Is that correct?

Do you really mean that without any coefficient on the "t" term? If there's a coefficient on that, then no; they nest: c=0, trend coefficient = 0, alpha=1.

ac_1 wrote: Can the DM test be applied to?
The model generating forecasts naive: y(t) = 1.0*y(t-1)
The model generating forecasts full AR(3): y(t) = c + alpha1*y(t-1) + alpha2*y(t-2) + alpha3*y(t-3)

No. c=0, alpha1=1, alpha2=0, alpha3=0.

ac_1 wrote: What if I change the naive to
mean/constant model: y(t) = c
OR
random walk with drift: y(t) = 1.0*y(t-1) + c
Can I still apply the DM test to compare
mean/constant model & full AR(3)?
random walk with drift & full AR(3)?

And if I change the full AR(3) to AR({1,3})?

AR(3) vs AR{1,3} nest. I'm not sure what the other question is.

The naive model is clear, then which model generates rhat in UG-171? If rhat is generated by y(t) = c + beta*t + alpha*y(t-1), then they nest.
Here's my question: let's say I am generating forecasts from the following "benchmark" models
(a) mean/constant model: y(t) = c
(b) naive model: y(t) = 1.0*y(t-1)
(c) random walk with drift model: y(t) = 1.0*y(t-1) + c
and I want to determine whether forecasts from each of the "benchmark" models are significantly different with forecasts from various AR(p) models, e.g. (a) vs AR(p), (b) vs AR(p), (c) vs AR(p). I cannot use the DM test as the pairs of models are nested. I have Enders (2010) AETS 3rdEdn, and there's a good section in p.84-p.97 on Forecast Evaluation with a standard recommendation to use the F-statistic, then GN & DM tests, and a mention of the Clark and McCracken (2001) test - which IMHO is complicated - designed for nested models. Please can a simple explanation with worked example and interpretation of results be provided in RATS using clarkforetest.src , or otherwise. Or is there a simpler alternative test for nested models? Many Thanks.

(c) nests (b). There is no nesting relationship among the others.

ac_1 wrote: I have Enders (2010) AETS 3rdEdn, and there's a good section in p.84-p.97 on Forecast Evaluation with a standard recommendation to use the F-statistic, ...

I should have remarked from Enders (2010) AETS 3rdEdn:
The ratio of MSPE's from two different models, the larger of the two MSPEs in the numerator, has a standard F-distribution if the following three assumptions hold:
1. The forecast errors have zero mean and are normally distributed.
2. The forecast errors are serially uncorrelated.
3. The forecast errors are contemporaneously uncorrelated with each other.
Although it is not necessarily true these three assumptions hold & the violation of any one of these assumptions means that the ratio of the MSPE's does not have an F-distribution. Thus, additional tests: Granger and Newbold (1976), Diebold and Mariano (1995), Clark and McCracken (2001) and Clark and West (2007), are proposed.