DMARIANO—Diebold-Mariano test

[u4179507]

Thank you Tom. I obtained the following output (the data file is attached just in case)

Forecast MSE Test Stat P(DM>x)
F1 0.04037088 5.1037 0.00000
F2 0.03033151 -5.1037 1.00000

Hence, the difference is 0.01004. Is there a way to extract the test statistics relating to H0: Forecast accuracy is equal.

It should be approximately 1.69 (p-value = 0.0910).

This hypothesis test basically tests whether the difference between the two forecasts is statistically significant from 0. In this specific case, it is so only at the 10% level.

Thank you again

[u4179507]

That is...after applying a loss criterion like MSE or MAE, the procedure estimates the timeseries predictive accuracy (refered to as D in the series window) that allows a test of the null hypothesis of equal accuracy. The test statistic I described in my previous post tests whether the mean difference between the loss criteria for the two competing predictions is zero, using a long-run estimate of the variance of the difference series. How I get the dmariano code to report this test statistic and its p-value?

Thank you again

[TomDoan]

Diebold-Mariano isn't testing whether the forecasts are "different" but whether one is better at matching the third series (the actual) than the other. I'm not sure where you are getting the 1.69---those two expected_xxx series are quite a bit more different than that.

[u4179507]

Thank you Tom

I see you adjusted the code recently to include the modified Diebold Mariano test. What do I need to change in order to conduct this test? Do I need to go into the @dmariano code and change something specific or do I just change the code provided below by adding the option to conduct the modified Diebold Mariano test?

open data "C:\rats\test1.xls"
calendar(m) 1990:1
data(format=xls,nolabels,org=columns,top=2) 1990:01 2011:06 actual f1 f2
@dmariano actual f1 f2

Thank you again

[TomDoan]

You would do

@dmariano(modified) actual f1 f2

[hardmann]

Dear Tom:

If I have three or four forecasting model. How can I use instructions @DMARIANO or Gnewbold to evaluate these.?

Best regard
Hardmann

[TomDoan]

They aren't really designed to pick the "best" model from a set of four. They are for pairwise comparisons to determine if the difference between two sets of forecasts is statistically significant. You would never reject the model with the best MSE, but you could also fail to reject any of the others in favor of the one with the lowest criterion.

[u4179507]

Hi Tom

If I am testing the forecast accuracy of one-step ahead estimates from two methods (f1 and f2) using the @dmariano code, should my lags be equal to zero? In a two-step ahead scenario, should lags=1...etc?

Also, I am using annual forecasts that update each month - i.e. there is a 11 month overlap between each of my annual forecast observations. Is this an issue that needs to be corrected? To my knowledge, the Diebold and Mariano (1995) asymptotic test of forecast accuracy is a model-free procedure that benefits from applicability to forecast errors that are non-Gaussian, contemporaneously and serially correlated.

Thank you

[TomDoan]

It sounds like you understand how to come up with "LAGS". The biggest problem that can come up is that the Diebold-Mariano test (as described by the authors) uses a "flat" window (not a Newey-West or some other window which is forcibly positive definite) and so could have numerical problems when the serial correlation is substantial.

[u4179507]

Thank you Tom

So in that case (high serial correlation due to my overlapping forecast windows), do you recommend using a newey LWINDOW? Your program allows for selecting between newey, bartlett, truncated, parzen and quadratic LWINDOWs - is newey the best choice for me?

Thank you again

[TomDoan]

Probably LWINDOW=NEWEY, but with a somewhat higher value of lags (to counteract the way that the Bartlett window heavily cuts the last few lags). None of the "damped" windows are really designed for use when the number of non-zero correlations is known exactly (at least theoretically), which I assume is why DM originally used a flat window. The flat window is theoretically correct, but (potentially) has numerical problems.

[u4179507]

Thank you Tom

I will use the newey-west window (LWINDOW=NEWEY) in combination with a higher value of lags. I am thinking of using 11 lags (LAGS=10) given that I am using annual forecasts that are updated monthly (hence, there is an 11 month overlap between each observation). Do you think this is appropriate?

Thank you

[TomDoan]

No. I mean LWINDOW=NEWEY with a higher value than the theoretical 10 (say 15) to allow for the fact that a Newey-West with width 10 only gives a 10% weight to lag 10.

[u4179507]

Thank you Tom

LWINDOW=NEWEY with LAGS=15 it is. Do you have any other advice?

I've attached some sample data for reference if you have a few minutes to take a look.

Thank you

[TomDoan]

No. That seems reasonable.