UFORERRORS—Analysis of (univariate) forecast errors

[TomDoan]

@UForeErrors computes error statistics on a series of in-sample one-step forecasts. You supply the series of actual data and forecasts and it computes statistics such as the mean and root mean square errors.

Detailed description

[fan]

Dear Tom and all,

I am trying to do forecast error analysis for my study. I got an error message says "## M4. A memory request for an additional 4293258904 bytes cannot be satisfied The Error Occurred At Location 235, Line 30 of UFOREERRORS C:\Users\Public\Documents\Estima\WinRATS Std 9.2\uforeerrors.src Line 67" I do not understand what caused the error. I am seeking for your help to solve the problem

I am doing 1-step ahead forecast comparison between FF 3 factor model and a simple benchmark rate =0.78. The forecasting period is 1980:1 to 2004:12, with a moving 60-period window to estimate the model parameters

There are my codes:

Code: Select all


CALENDAR(M) 1965:7
DATA(FORMAT=XLSX,ORG=COLUMNS) 1965:07 2010:12 month MktRF RF Divyield500 Divyield Term Default SL SM SH $
 BL BM BH

clear rhat
do fstart=1980:1,2004:12
   linreg(noprint) sl fstart-59 fstart-1
   # constant mktrf smb hml
   uforecast rhat fstart fstart
end do
@uforeerrors sl rhat 19801:1 2004:12
*
* Diebold-Mariano test
*
set naive 1980:1 2004:12 = 0.78
@dmariano sl naive rhat




[TomDoan]

@uforeerrors sl rhat 19801:1 2004:12

You have a typo in your date.

[fan]

You have a typo in your date.[/quote]

@uforeerrors sl rhat 19801:1 2004:12

You have a typo in your date.

Thank you, Tom, for the quick reply. The output I have, however, is some different from the example's output. It seems that the Mean Pct Error, Mean Abs Pct Error, Root Mean Square Pct Error, Theil's Relative U are missing from my output.

Code: Select all

clear rhat1
do fstart=1988:4,2010:12
   linreg(noprint) r11 fstart-273 fstart-1
   # constant mktrf smb hml
   uforecast rhat1 fstart fstart
end do
@uforeerrors r11 rhat1 1988:4 2010:12

clear rhat2
do fstart=1988:4,2010:12
   linreg(noprint) r11 fstart-273 fstart-1
   # constant snew(1) snew(2) snew(3)
   uforecast rhat2 fstart fstart
end do
@uforeerrors r11 rhat2 1988:4 2010:12

@dmariano r11 rhat1 rhat2

set foreerror1sq = (r11-rhat1)**2
set foreerror2sq = (r11-rhat2)**2
set diff = foreerror1sq-foreerror2sq
stat diff



My output

Code: Select all

Forecast Analysis for R11
From 1988:04 to 2010:12
Mean Error             0.09607337
Mean Absolute Error    3.06732641
Root Mean Square Error 4.87751621
Mean Square Error       23.790164
Theil's U                0.434008


Forecast Analysis for R11
From 1988:04 to 2010:12
Mean Error             0.01344721
Mean Absolute Error    1.76551429
Root Mean Square Error 2.40882310
Mean Square Error        5.802429
Theil's U                0.214340


Diebold-Mariano Forecast Comparison Test
Forecasts of R11 over 1988:04 to 2010:12

Forecast    MSE     Test Stat P(DM>x)
RHAT1    23.7901644    2.7072 0.00339
RHAT2     5.8024287   -2.7072 0.99661



in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks

[TomDoan]

Percentage errors are only defined when the "actual" series is positive throughout. The Theil statistics are computed and displayed only if you do the THEIL option.

[TomDoan]

in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks

You've already done that. Twice. Just with different alternatives. If you don't understand that, you need to read up more about what Diebold-Mariano does.

[ac_1]

Hi Tom,

uforecasterrors.src computes error statistics on a series of in-sample one-step forecasts, is there a procedure to measure: correct directional accuracy or a simple sign-test?
If not, please can you include in uforecasterrors.src?

Thanks,
Amarjit

[TomDoan]

I'm not sure what you mean by "directional accuracy" but the %SIGN function converts a number to +1 or -1 depending upon the sign. So

SSTATS(mean) fstart fend (%sign(forecast)==%sign(actual))>>percentmatch

will compute the percentage of forecast and actual which have the same sign over the period from fstart to fend. @UFOREERRORS is basically a wrapper around an SSTATS instruction which computes the raw forecast error statistics---everything else is just organizing the output.

[ac_1]

Hi Tom,

I want to add MASE to uforerrors.src:

After,

if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)

Code: Select all

* MASE (Mean Absolute Scaled Error)
if startl == 1 {

   comp %%MASE = %na; * undefined

}
else {

   set num startl endl = actual-forecast
   set den / = abs(actual-actual{1})

   SSTATS(mean) startl endl den>>mean_den;
   *disp mean_den

   * or alternatively by-hand
   *SSTATS startl endl den>>sum_den; * SSTATS computes the sum, without any option
   *comp mean_den = sum_den/%NOBS
   *disp mean_den

   set ratio = abs(num/mean_den)

   SSTATS(mean) startl endl ratio>>%%MASE

}


and as MASE is defined as a ratio, output in the Percentage Errors section, i.e. after,

if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)

Code: Select all

* MASE reported in nonpos==0 section as MASE is defined as a ratio. MASE is always calculated.
   if (h==1)
      report(use=ureport,row=new,atcol=1) "MASE" %%MASE
   else
      report(use=ureport,row=new,atcol=1)


Is this correct/reasonable?

thanks,
Amarjit

[TomDoan]

1. What is H?
2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.

[ac_1]

1. What is H?

h=1 is an option for the horizon of forecasts being h-step(s) ahead in the version of UForeErrors where I have included MASE.

2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.

Here's an attempt:

The mean absolute no change error through the training sample is defined as MANCE.

Code: Select all

procedure acUForeErrors actual forecast MANCE start end
TYPE REAL MANCE

option integer  MASE 0

local series    RATIO


Just after
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)

Code: Select all

if (MASE==1)
   set RATIO = abs(ferrors/MANCE)
   SSTATS(mean) startl endl RATIO>>%%FERRMASE


And just after
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)

Code: Select all

* MASE reported in nonpos==0 section as MASE is defined as a ratio. %%FERRMASE is always calculated.
   if (MASE==1)
      report(use=ureport,row=new,atcol=1) "MASE" %%FERRMASE
   else
      report(use=ureport,row=new,atcol=1)


end acUForeErrors


To run:

Code: Select all

* MANCE is calculated from the training period:

set den / = abs(actual-actual{1})
sstats(mean) regstart regend den>>MANCE

* and then

@acUForeErrors(MASE=1) actual forecast MANCE start end


Correct?


But, am unclear as to exactly how to run as I still need to have a value for MANCE in the Parameters
@acUForeErrors(MASE=0) actual forecast MANCE start end
if option MASE=0 ???

Questions:
Does startl need to be > 1?

To clarify, are all the loss measures, including Theil's, in UForeErrors applicable for:
(a) multi-step ahead forecasts?
(b) and also, as in https://estima.com/docs/RATS%2010%20Use ... f#page=186 i.e. computing hth-step-ahead forecasts, and save only the hth forecast step from each rolling regression?


To include MASE in UForeErrors is not ideal, as MANCE i.e. mean(den) (or the scale) is for the training sample. It's probably better to use Theil's, as Theil is benchamrked against nochange within UForeErrors.src.

[TomDoan]

First of all, you want to make MANSE an option, not a parameter. (It is, pretty much by definition an option.)

OPTION REAL MANSE

Use a %DEFINED(MANSE) check to see if it's been given a value. (As written, it won't unless it is actually used in the procedure call). Calculate your statistic if and only if it's defined.

"ACTUAL" is the name of the formal parameter in the procedure; if you do the calculation outside the procedure, you need to use the series name itself (I use Y below, but it's whatever it is). Also note that the actual series isn't (necessarily) defined for regstart-1 so the usual way to calculation this would be

sstats(mean) regstart+1 regend abs(y-y{1})>>MANCE

(You can combine the SET and SSTATS into a single instruction).

The points of the Theil U and the MASE are the same: replace a scale-dependent measure such as the MAE with a scale-free measure that also provides at least a (very) rough comparison with a naive forecasting system. Neither one of them is particularly useful in looking at a single forecasting method (other than values > 1 being objectively less than promising). Instead, they provide a scale-free method to compare different forecast methods for the same series. It's just easier to look at .42 vs .45 rather than .000213 vs .000228. In comparing two different forecast methodologies for the same series, it doesn't really matter which number you use for the scaling, since it's the same for each statistic. From a practical standpoint, the MASE may be somewhat better since the scale is based upon a larger sample size; thus a >1.0 value is less likely to be a small-sample fluke. But that's the only advantage.