@UForeErrors computes error statistics on a series of in-sample one-step forecasts. You supply the series of actual data and forecasts and it computes statistics such as the mean and root mean square errors.

Detailed description

Detailed description

Last edited by TomDoan on Mon Aug 13, 2018 12:37 pm, edited 3 times in total.

**Reason:** *Switch to description from HTML help*

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

Dear Tom and all,

I am trying to do forecast error analysis for my study. I got an error message says "## M4. A memory request for an additional 4293258904 bytes cannot be satisfied The Error Occurred At Location 235, Line 30 of UFOREERRORS C:\Users\Public\Documents\Estima\WinRATS Std 9.2\uforeerrors.src Line 67" I do not understand what caused the error. I am seeking for your help to solve the problem

I am doing 1-step ahead forecast comparison between FF 3 factor model and a simple benchmark rate =0.78. The forecasting period is 1980:1 to 2004:12, with a moving 60-period window to estimate the model parameters

There are my codes:

I am trying to do forecast error analysis for my study. I got an error message says "## M4. A memory request for an additional 4293258904 bytes cannot be satisfied The Error Occurred At Location 235, Line 30 of UFOREERRORS C:\Users\Public\Documents\Estima\WinRATS Std 9.2\uforeerrors.src Line 67" I do not understand what caused the error. I am seeking for your help to solve the problem

I am doing 1-step ahead forecast comparison between FF 3 factor model and a simple benchmark rate =0.78. The forecasting period is 1980:1 to 2004:12, with a moving 60-period window to estimate the model parameters

There are my codes:

- Code: Select all

CALENDAR(M) 1965:7

DATA(FORMAT=XLSX,ORG=COLUMNS) 1965:07 2010:12 month MktRF RF Divyield500 Divyield Term Default SL SM SH $

BL BM BH

clear rhat

do fstart=1980:1,2004:12

linreg(noprint) sl fstart-59 fstart-1

# constant mktrf smb hml

uforecast rhat fstart fstart

end do

@uforeerrors sl rhat 19801:1 2004:12

*

* Diebold-Mariano test

*

set naive 1980:1 2004:12 = 0.78

@dmariano sl naive rhat

- fan
**Posts:**215**Joined:**Wed Jun 19, 2013 5:14 pm

@uforeerrors sl rhat 19801:1 2004:12

You have a typo in your date.

You have a typo in your date.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

You have a typo in your date.[/quote]

Thank you, Tom, for the quick reply. The output I have, however, is some different from the example's output. It seems that the Mean Pct Error, Mean Abs Pct Error, Root Mean Square Pct Error, Theil's Relative U are missing from my output.

My output

in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks

TomDoan wrote:@uforeerrors sl rhat 19801:1 2004:12

You have a typo in your date.

Thank you, Tom, for the quick reply. The output I have, however, is some different from the example's output. It seems that the Mean Pct Error, Mean Abs Pct Error, Root Mean Square Pct Error, Theil's Relative U are missing from my output.

- Code: Select all
`clear rhat1`

do fstart=1988:4,2010:12

linreg(noprint) r11 fstart-273 fstart-1

# constant mktrf smb hml

uforecast rhat1 fstart fstart

end do

@uforeerrors r11 rhat1 1988:4 2010:12

clear rhat2

do fstart=1988:4,2010:12

linreg(noprint) r11 fstart-273 fstart-1

# constant snew(1) snew(2) snew(3)

uforecast rhat2 fstart fstart

end do

@uforeerrors r11 rhat2 1988:4 2010:12

@dmariano r11 rhat1 rhat2

set foreerror1sq = (r11-rhat1)**2

set foreerror2sq = (r11-rhat2)**2

set diff = foreerror1sq-foreerror2sq

stat diff

My output

- Code: Select all
`Forecast Analysis for R11`

From 1988:04 to 2010:12

Mean Error 0.09607337

Mean Absolute Error 3.06732641

Root Mean Square Error 4.87751621

Mean Square Error 23.790164

Theil's U 0.434008

Forecast Analysis for R11

From 1988:04 to 2010:12

Mean Error 0.01344721

Mean Absolute Error 1.76551429

Root Mean Square Error 2.40882310

Mean Square Error 5.802429

Theil's U 0.214340

Diebold-Mariano Forecast Comparison Test

Forecasts of R11 over 1988:04 to 2010:12

Forecast MSE Test Stat P(DM>x)

RHAT1 23.7901644 2.7072 0.00339

RHAT2 5.8024287 -2.7072 0.99661

in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks

- fan
**Posts:**215**Joined:**Wed Jun 19, 2013 5:14 pm

Percentage errors are only defined when the "actual" series is positive throughout. The Theil statistics are computed and displayed only if you do the THEIL option.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

fan wrote:in addition, could you please kindly help me compute DM-STAT to test null hypothesis H0: diff=0? Many Thanks

You've already done that. Twice. Just with different alternatives. If you don't understand that, you need to read up more about what Diebold-Mariano does.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

Hi Tom,

uforecasterrors.src computes error statistics on a series of in-sample one-step forecasts, is there a procedure to measure: correct directional accuracy or a simple sign-test?

If not, please can you include in uforecasterrors.src?

Thanks,

Amarjit

uforecasterrors.src computes error statistics on a series of in-sample one-step forecasts, is there a procedure to measure: correct directional accuracy or a simple sign-test?

If not, please can you include in uforecasterrors.src?

Thanks,

Amarjit

- ac_1
**Posts:**201**Joined:**Thu Apr 15, 2010 6:30 am**Location:**London, UK

I'm not sure what you mean by "directional accuracy" but the %SIGN function converts a number to +1 or -1 depending upon the sign. So

SSTATS(mean) fstart fend (%sign(forecast)==%sign(actual))>>percentmatch

will compute the percentage of forecast and actual which have the same sign over the period from fstart to fend. @UFOREERRORS is basically a wrapper around an SSTATS instruction which computes the raw forecast error statistics---everything else is just organizing the output.

SSTATS(mean) fstart fend (%sign(forecast)==%sign(actual))>>percentmatch

will compute the percentage of forecast and actual which have the same sign over the period from fstart to fend. @UFOREERRORS is basically a wrapper around an SSTATS instruction which computes the raw forecast error statistics---everything else is just organizing the output.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

Hi Tom,

Many thanks! 'SignTest' is probably a better name.

Amarjit

Many thanks! 'SignTest' is probably a better name.

Amarjit

- ac_1
**Posts:**201**Joined:**Thu Apr 15, 2010 6:30 am**Location:**London, UK

Hi Tom,

I want to add MASE to uforerrors.src:

After,

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)

and as MASE is defined as a ratio, output in the Percentage Errors section, i.e. after,

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)

Is this correct/reasonable?

thanks,

Amarjit

I want to add MASE to uforerrors.src:

After,

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)

- Code: Select all
`* MASE (Mean Absolute Scaled Error)`

if startl == 1 {

comp %%MASE = %na; * undefined

}

else {

set num startl endl = actual-forecast

set den / = abs(actual-actual{1})

SSTATS(mean) startl endl den>>mean_den;

*disp mean_den

* or alternatively by-hand

*SSTATS startl endl den>>sum_den; * SSTATS computes the sum, without any option

*comp mean_den = sum_den/%NOBS

*disp mean_den

set ratio = abs(num/mean_den)

SSTATS(mean) startl endl ratio>>%%MASE

}

and as MASE is defined as a ratio, output in the Percentage Errors section, i.e. after,

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)

- Code: Select all
`* MASE reported in nonpos==0 section as MASE is defined as a ratio. MASE is always calculated.`

if (h==1)

report(use=ureport,row=new,atcol=1) "MASE" %%MASE

else

report(use=ureport,row=new,atcol=1)

Is this correct/reasonable?

thanks,

Amarjit

- ac_1
**Posts:**201**Joined:**Thu Apr 15, 2010 6:30 am**Location:**London, UK

1. What is H?

2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.

2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

TomDoan wrote:1. What is H?

h=1 is an option for the horizon of forecasts being h-step(s) ahead in the version of UForeErrors where I have included MASE.

TomDoan wrote:2. The denominator is the mean absolute no change error through the training sample. Nothing else in @UFOREERRORS uses training (i.e. regression) sample data so there is nothing in the design to let you compute it within the procedure. It's probably simpler to compute it once and pass it through as an option.

Here's an attempt:

The mean absolute no change error through the training sample is defined as MANCE.

- Code: Select all
`procedure acUForeErrors actual forecast MANCE start end`

TYPE REAL MANCE

option integer MASE 0

local series RATIO

Just after

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's U" sqrt(meansqr/nochange)

- Code: Select all
`if (MASE==1)`

set RATIO = abs(ferrors/MANCE)

SSTATS(mean) startl endl RATIO>>%%FERRMASE

And just after

if %valid(nochange)

report(use=ureport,row=new,atcol=1) "Theil's Relative U" sqrt(relchange/nochange)

- Code: Select all
`* MASE reported in nonpos==0 section as MASE is defined as a ratio. %%FERRMASE is always calculated.`

if (MASE==1)

report(use=ureport,row=new,atcol=1) "MASE" %%FERRMASE

else

report(use=ureport,row=new,atcol=1)

end acUForeErrors

To run:

- Code: Select all
`* MANCE is calculated from the training period:`

set den / = abs(actual-actual{1})

sstats(mean) regstart regend den>>MANCE

* and then

@acUForeErrors(MASE=1) actual forecast MANCE start end

Correct?

But, am unclear as to exactly how to run as I still need to have a value for MANCE in the Parameters

@acUForeErrors(MASE=0) actual forecast MANCE start end

if option MASE=0 ???

Questions:

Does startl need to be > 1?

To clarify, are all the loss measures, including Theil's, in UForeErrors applicable for:

(a) multi-step ahead forecasts?

(b) and also, as in https://estima.com/docs/RATS%2010%20Use ... f#page=186 i.e. computing hth-step-ahead forecasts, and save only the hth forecast step from each rolling regression?

To include MASE in UForeErrors is not ideal, as MANCE i.e. mean(den) (or the scale) is for the training sample. It's probably better to use Theil's, as Theil is benchamrked against nochange within UForeErrors.src.

- ac_1
**Posts:**201**Joined:**Thu Apr 15, 2010 6:30 am**Location:**London, UK

First of all, you want to make MANSE an option, not a parameter. (It is, pretty much by definition an option.)

OPTION REAL MANSE

Use a %DEFINED(MANSE) check to see if it's been given a value. (As written, it won't unless it is actually used in the procedure call). Calculate your statistic if and only if it's defined.

"ACTUAL" is the name of the formal parameter in the procedure; if you do the calculation outside the procedure, you need to use the series name itself (I use Y below, but it's whatever it is). Also note that the actual series isn't (necessarily) defined for regstart-1 so the usual way to calculation this would be

sstats(mean) regstart+1 regend abs(y-y{1})>>MANCE

(You can combine the SET and SSTATS into a single instruction).

The points of the Theil U and the MASE are the same: replace a scale-dependent measure such as the MAE with a scale-free measure that also provides at least a (very) rough comparison with a naive forecasting system. Neither one of them is particularly useful in looking at a single forecasting method (other than values > 1 being objectively less than promising). Instead, they provide a scale-free method to compare different forecast methods for the same series. It's just easier to look at .42 vs .45 rather than .000213 vs .000228. In comparing two different forecast methodologies for the same series, it doesn't really matter which number you use for the scaling, since it's the same for each statistic. From a practical standpoint, the MASE may be somewhat better since the scale is based upon a larger sample size; thus a >1.0 value is less likely to be a small-sample fluke. But that's the only advantage.

OPTION REAL MANSE

Use a %DEFINED(MANSE) check to see if it's been given a value. (As written, it won't unless it is actually used in the procedure call). Calculate your statistic if and only if it's defined.

"ACTUAL" is the name of the formal parameter in the procedure; if you do the calculation outside the procedure, you need to use the series name itself (I use Y below, but it's whatever it is). Also note that the actual series isn't (necessarily) defined for regstart-1 so the usual way to calculation this would be

sstats(mean) regstart+1 regend abs(y-y{1})>>MANCE

(You can combine the SET and SSTATS into a single instruction).

The points of the Theil U and the MASE are the same: replace a scale-dependent measure such as the MAE with a scale-free measure that also provides at least a (very) rough comparison with a naive forecasting system. Neither one of them is particularly useful in looking at a single forecasting method (other than values > 1 being objectively less than promising). Instead, they provide a scale-free method to compare different forecast methods for the same series. It's just easier to look at .42 vs .45 rather than .000213 vs .000228. In comparing two different forecast methodologies for the same series, it doesn't really matter which number you use for the scaling, since it's the same for each statistic. From a practical standpoint, the MASE may be somewhat better since the scale is based upon a larger sample size; thus a >1.0 value is less likely to be a small-sample fluke. But that's the only advantage.

- TomDoan
**Posts:**7236**Joined:**Wed Nov 01, 2006 5:36 pm

13 posts
• Page **1** of **1**

Users browsing this forum: No registered users and 3 guests