UFORERRORS—Analysis of (univariate) forecast errors
UFORERRORS—Analysis of (univariate) forecast errors
@UForeErrors computes error statistics on a series of in-sample one-step forecasts. You supply the series of actual data and forecasts and it computes statistics such as the mean and root mean square errors.
Detailed description
Detailed description
Last edited by TomDoan on Mon Aug 13, 2018 12:37 pm, edited 3 times in total.
Reason: Switch to description from HTML help
Reason: Switch to description from HTML help
Errors of Forecast error analysis
Dear Tom and all,
I am trying to do forecast error analysis for my study. I got an error message says "## M4. A memory request for an additional 4293258904 bytes cannot be satisfied The Error Occurred At Location 235, Line 30 of UFOREERRORS C:\Users\Public\Documents\Estima\WinRATS Std 9.2\uforeerrors.src Line 67" I do not understand what caused the error. I am seeking for your help to solve the problem
I am doing 1-step ahead forecast comparison between FF 3 factor model and a simple benchmark rate =0.78. The forecasting period is 1980:1 to 2004:12, with a moving 60-period window to estimate the model parameters
There are my codes:
I am trying to do forecast error analysis for my study. I got an error message says "## M4. A memory request for an additional 4293258904 bytes cannot be satisfied The Error Occurred At Location 235, Line 30 of UFOREERRORS C:\Users\Public\Documents\Estima\WinRATS Std 9.2\uforeerrors.src Line 67" I do not understand what caused the error. I am seeking for your help to solve the problem
I am doing 1-step ahead forecast comparison between FF 3 factor model and a simple benchmark rate =0.78. The forecasting period is 1980:1 to 2004:12, with a moving 60-period window to estimate the model parameters
There are my codes:
Code: Select all
CALENDAR(M) 1965:7
DATA(FORMAT=XLSX,ORG=COLUMNS) 1965:07 2010:12 month MktRF RF Divyield500 Divyield Term Default SL SM SH $
BL BM BH
clear rhat
do fstart=1980:1,2004:12
linreg(noprint) sl fstart-59 fstart-1
# constant mktrf smb hml
uforecast rhat fstart fstart
end do
@uforeerrors sl rhat 19801:1 2004:12
*
* Diebold-Mariano test
*
set naive 1980:1 2004:12 = 0.78
@dmariano sl naive rhat
Re: Errors of Forecast error analysis
@uforeerrors sl rhat 19801:1 2004:12
You have a typo in your date.
You have a typo in your date.
Re: Errors of Forecast error analysis
You have a typo in your date.[/quote]
My output
in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks
Thank you, Tom, for the quick reply. The output I have, however, is some different from the example's output. It seems that the Mean Pct Error, Mean Abs Pct Error, Root Mean Square Pct Error, Theil's Relative U are missing from my output.TomDoan wrote:@uforeerrors sl rhat 19801:1 2004:12
You have a typo in your date.
Code: Select all
clear rhat1
do fstart=1988:4,2010:12
linreg(noprint) r11 fstart-273 fstart-1
# constant mktrf smb hml
uforecast rhat1 fstart fstart
end do
@uforeerrors r11 rhat1 1988:4 2010:12
clear rhat2
do fstart=1988:4,2010:12
linreg(noprint) r11 fstart-273 fstart-1
# constant snew(1) snew(2) snew(3)
uforecast rhat2 fstart fstart
end do
@uforeerrors r11 rhat2 1988:4 2010:12
@dmariano r11 rhat1 rhat2
set foreerror1sq = (r11-rhat1)**2
set foreerror2sq = (r11-rhat2)**2
set diff = foreerror1sq-foreerror2sq
stat diff
My output
Code: Select all
Forecast Analysis for R11
From 1988:04 to 2010:12
Mean Error 0.09607337
Mean Absolute Error 3.06732641
Root Mean Square Error 4.87751621
Mean Square Error 23.790164
Theil's U 0.434008
Forecast Analysis for R11
From 1988:04 to 2010:12
Mean Error 0.01344721
Mean Absolute Error 1.76551429
Root Mean Square Error 2.40882310
Mean Square Error 5.802429
Theil's U 0.214340
Diebold-Mariano Forecast Comparison Test
Forecasts of R11 over 1988:04 to 2010:12
Forecast MSE Test Stat P(DM>x)
RHAT1 23.7901644 2.7072 0.00339
RHAT2 5.8024287 -2.7072 0.99661
Re: UFORERRORS—Analysis of (univariate) forecast errors
Percentage errors are only defined when the "actual" series is positive throughout. The Theil statistics are computed and displayed only if you do the THEIL option.
Re: Errors of Forecast error analysis
You've already done that. Twice. Just with different alternatives. If you don't understand that, you need to read up more about what Diebold-Mariano does.fan wrote:in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks
uforecasterrors.src
Hi Tom,
uforecasterrors.src computes error statistics on a series of in-sample one-step forecasts, is there a procedure to measure: correct directional accuracy or a simple sign-test?
If not, please can you include in uforecasterrors.src?
Thanks,
Amarjit
uforecasterrors.src computes error statistics on a series of in-sample one-step forecasts, is there a procedure to measure: correct directional accuracy or a simple sign-test?
If not, please can you include in uforecasterrors.src?
Thanks,
Amarjit
Re: UFORERRORS—Analysis of (univariate) forecast errors
I'm not sure what you mean by "directional accuracy" but the %SIGN function converts a number to +1 or -1 depending upon the sign. So
SSTATS(mean) fstart fend (%sign(forecast)==%sign(actual))>>percentmatch
will compute the percentage of forecast and actual which have the same sign over the period from fstart to fend. @UFOREERRORS is basically a wrapper around an SSTATS instruction which computes the raw forecast error statistics---everything else is just organizing the output.
SSTATS(mean) fstart fend (%sign(forecast)==%sign(actual))>>percentmatch
will compute the percentage of forecast and actual which have the same sign over the period from fstart to fend. @UFOREERRORS is basically a wrapper around an SSTATS instruction which computes the raw forecast error statistics---everything else is just organizing the output.
Re: UFORERRORS—Analysis of (univariate) forecast errors
Hi Tom,
I want to add MASE to uforerrors.src:
After,
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)
and as MASE is defined as a ratio, output in the Percentage Errors section, i.e. after,
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)
Is this correct/reasonable?
thanks,
Amarjit
I want to add MASE to uforerrors.src:
After,
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)
Code: Select all
* MASE (Mean Absolute Scaled Error)
if startl == 1 {
comp %%MASE = %na; * undefined
}
else {
set num startl endl = actual-forecast
set den / = abs(actual-actual{1})
SSTATS(mean) startl endl den>>mean_den;
*disp mean_den
* or alternatively by-hand
*SSTATS startl endl den>>sum_den; * SSTATS computes the sum, without any option
*comp mean_den = sum_den/%NOBS
*disp mean_den
set ratio = abs(num/mean_den)
SSTATS(mean) startl endl ratio>>%%MASE
}
and as MASE is defined as a ratio, output in the Percentage Errors section, i.e. after,
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)
Code: Select all
* MASE reported in nonpos==0 section as MASE is defined as a ratio. MASE is always calculated.
if (h==1)
report(use=ureport,row=new,atcol=1) "MASE" %%MASE
else
report(use=ureport,row=new,atcol=1)
thanks,
Amarjit
Re: UFORERRORS—Analysis of (univariate) forecast errors
1. What is H?
2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.
2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.
Re: UFORERRORS—Analysis of (univariate) forecast errors
h=1 is an option for the horizon of forecasts being h-step(s) ahead in the version of UForeErrors where I have included MASE.TomDoan wrote:1. What is H?
Here's an attempt:TomDoan wrote: 2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.
The mean absolute no change error through the training sample is defined as MANCE.
Code: Select all
procedure acUForeErrors actual forecast MANCE start end
TYPE REAL MANCE
option integer MASE 0
local series RATIO
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)
Code: Select all
if (MASE==1)
set RATIO = abs(ferrors/MANCE)
SSTATS(mean) startl endl RATIO>>%%FERRMASE
if %valid(nochange)
report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)
Code: Select all
* MASE reported in nonpos==0 section as MASE is defined as a ratio. %%FERRMASE is always calculated.
if (MASE==1)
report(use=ureport,row=new,atcol=1) "MASE" %%FERRMASE
else
report(use=ureport,row=new,atcol=1)
end acUForeErrors
Code: Select all
* MANCE is calculated from the training period:
set den / = abs(actual-actual{1})
sstats(mean) regstart regend den>>MANCE
* and then
@acUForeErrors(MASE=1) actual forecast MANCE start end
But, am unclear as to exactly how to run as I still need to have a value for MANCE in the Parameters
@acUForeErrors(MASE=0) actual forecast MANCE start end
if option MASE=0 ???
Questions:
Does startl need to be > 1?
To clarify, are all the loss measures, including Theil's, in UForeErrors applicable for:
(a) multi-step ahead forecasts?
(b) and also, as in https://estima.com/docs/RATS%2010%20Use ... f#page=186 i.e. computing hth-step-ahead forecasts, and save only the hth forecast step from each rolling regression?
To include MASE in UForeErrors is not ideal, as MANCE i.e. mean(den) (or the scale) is for the training sample. It's probably better to use Theil's, as Theil is benchamrked against nochange within UForeErrors.src.
Re: UFORERRORS—Analysis of (univariate) forecast errors
First of all, you want to make MANSE an option, not a parameter. (It is, pretty much by definition an option.)
OPTION REAL MANSE
Use a %DEFINED(MANSE) check to see if it's been given a value. (As written, it won't unless it is actually used in the procedure call). Calculate your statistic if and only if it's defined.
"ACTUAL" is the name of the formal parameter in the procedure; if you do the calculation outside the procedure, you need to use the series name itself (I use Y below, but it's whatever it is). Also note that the actual series isn't (necessarily) defined for regstart-1 so the usual way to calculation this would be
sstats(mean) regstart+1 regend abs(y-y{1})>>MANCE
(You can combine the SET and SSTATS into a single instruction).
The points of the Theil U and the MASE are the same: replace a scale-dependent measure such as the MAE with a scale-free measure that also provides at least a (very) rough comparison with a naive forecasting system. Neither one of them is particularly useful in looking at a single forecasting method (other than values > 1 being objectively less than promising). Instead, they provide a scale-free method to compare different forecast methods for the same series. It's just easier to look at .42 vs .45 rather than .000213 vs .000228. In comparing two different forecast methodologies for the same series, it doesn't really matter which number you use for the scaling, since it's the same for each statistic. From a practical standpoint, the MASE may be somewhat better since the scale is based upon a larger sample size; thus a >1.0 value is less likely to be a small-sample fluke. But that's the only advantage.
OPTION REAL MANSE
Use a %DEFINED(MANSE) check to see if it's been given a value. (As written, it won't unless it is actually used in the procedure call). Calculate your statistic if and only if it's defined.
"ACTUAL" is the name of the formal parameter in the procedure; if you do the calculation outside the procedure, you need to use the series name itself (I use Y below, but it's whatever it is). Also note that the actual series isn't (necessarily) defined for regstart-1 so the usual way to calculation this would be
sstats(mean) regstart+1 regend abs(y-y{1})>>MANCE
(You can combine the SET and SSTATS into a single instruction).
The points of the Theil U and the MASE are the same: replace a scale-dependent measure such as the MAE with a scale-free measure that also provides at least a (very) rough comparison with a naive forecasting system. Neither one of them is particularly useful in looking at a single forecasting method (other than values > 1 being objectively less than promising). Instead, they provide a scale-free method to compare different forecast methods for the same series. It's just easier to look at .42 vs .45 rather than .000213 vs .000228. In comparing two different forecast methodologies for the same series, it doesn't really matter which number you use for the scaling, since it's the same for each statistic. From a practical standpoint, the MASE may be somewhat better since the scale is based upon a larger sample size; thus a >1.0 value is less likely to be a small-sample fluke. But that's the only advantage.