directional accuracy multistep ahead forecasts

[ac_1]

Hi Tom,

One-step ahead forecasts, h=1:
- I can simply calculate a correct directional percentage from the number of times the forecast and actual prices move in the same direction.
- Also, from bivariate regressions the estimated t-statistic for the beta coefficient can be considered to be a measure of the statistical significance; the RHS being a binary (or with 3 levels: +1, 0, -1) for the forecast direction and the LHS either being a binary (or with 3 levels: +1, 0, -1) for the actual difference or the log-difference/percentage change for the actual.
- There are various statistical tests e.g. Kuipers score, Henriksson and Merton (1981), Pesaran and Timmermann (1992, 2009), etc.

Multistep ahead forecasts, h>1:
The above is not so straightforward. I can calculate a correct directional percentage as above, but how do I assess the statistical significance of the forecasts and actual being "in-synch"/market-timing for h>1, the hth-step forecast; specifically in AR models where the estimated betas are small?

thanks,
Amarjit

[TomDoan]

I'm not sure why the same idea wouldn't work. Yes, multistep "sign" forecasts are fairly uninteresting: if the last observed value is above the mean, the sign of the forecast change is negative, and if the last observed value is below, the forecast change is positive. That's reality.

[ac_1]

Thanks - yes, understood.

Using either of the above regressions:

For multistep ahead forecasts:
(a) If either the sign of all the observations in the actuals or the sign of all the forecasts values are the same I get t-ratio=0, p-value=0.5.
(b) If just one point is different in sign for the forecasts (especially at the longer horizon), with varying signs in the actuals I get a large negative t-ratio, p-value=1.
(c) Interestingly, for the one-step ahead where the correct-directional accuracy can be around 50% either below/above, and the multistep ahead being much higher at (say) 65%, I can get insignificant (& sometimes negative) t-ratios for the latter, and significant (e.g. 10%) for the former.


For the LHS variable being: +1,0,-1, I have used LINREG i.e. ols, not DDV, to estimate the t-ratios which are inclusive of an intercept, any preference with the choice for LWINDOW in serial correlation correction, and similarly with the LHS being a continuous variable?

Also, if I use DDV, TYPE=MULTINOMIAL with 3 levels as the DV is there an interpretation for being "in-synch"/market-timing?

[TomDoan]

How many data points are you using for this? This is a technique which is designed to apply to 100's if not 1000's of data points (so there are ample examples of all combinations).

[ac_1]

Daily data, analysing individual years, approximately 252 data points per year.

[TomDoan]

You have an entire year of daily data with all the forecasts, or all the actuals in the same direction? That sounds like you're doing something wrong.

[ac_1]

Yes, with the forecasts in same direction (not the actuals), generated recursively, saving the hth=21 multistep ahead forecast OOS on each day.

The code is correct with the forecasts generated as per https://estima.com/docs/RATS%2010%20Use ... f#page=186

[TomDoan]

Time series forecasts aren't usually very interesting. Long-term time series forecasts are usually very, very uninteresting. I'm not sure what you're expecting, but if your series has even a very modest trend, the optimal 21 step forecast will be for a slight uptick. The forecast will effectively never be down.

Just to look at a simple example, suppose X(t)={+1 with p=.5,-1 with p=.5} i.i.d. The optimal (mean square error) "forecast" of X(t) is 0. Of course, it is a forecast which can never actually be a value of the process. The point value of a forecast really shouldn't be forced into a categorical measure which is what it sounds like you're trying to do.