THEIL(SETUP, other options) equations

# list of equations (if MODEL option isn’t used)

THEIL(options) start

THEIL(DUMP, other options)

 

THEIL computes statistics on forecast performance for the variables of a model. You can also use the @UForeErrors procedure to compute forecast performance statistics for univariate forecasting models.

Description

For each variable and each forecast step, THEIL computes:

 

•the mean error

•the mean absolute error

•the root mean square (RMS) error

•Theil’s U statistic: a ratio of the RMS error to the RMS error of the “naive” forecast of no change in the dependent variable.

 

Using THEIL is a three-step process:

1.Initialization, using THEIL with the SETUP option. This describes the model and is done once for each analysis.

2.Forecasting, using THEIL without the SETUP or DUMP options. This usually goes inside a loop, and does the forecasting and accumulation of forecast performance statistics. If you want to save the forecasts, use a FORECAST instruction in addition to THEIL.

3.Final Statistics, using THEIL with the option DUMP. This takes the forecast statistics and prints a table for each variable in the model.

Parameters—with the SETUP option

 

The steps and dataend parameters used in previous versions have been replaced by options of the same names.

Options—with the SETUP option

STEPS=number of forecast steps

TO=last period for sample data[standard workspace length]

Use STEPS to indicate the number of forecast steps (periods) to compute.

Use TO to indicate the last period in the sample. Since THEIL needs to compare the forecasts with actual data, it will not compute any forecasts for periods beyond that period. TO defaults to the standard workspace length.
 

MODEL=general model to be evaluated [unused]

Of the two ways to input the form of the model to be evaluated (the other is using the list on the supplementary card), this is the more convenient and is the only way to forecast with a set of FRMLs. MODELs are usually created by GROUP or SYSTEM. If you use this, omit the “list of equation” supplementary card.

 

ITERS=iteration limit for solution algorithm [50]

CVCRIT=convergence criterion [.00001]

DAMP=damping factor(1 = no damping) [1]

When evaluating a nonlinear model (created by the GROUP instruction and specified by the MODEL option), you can use these three options to control various aspects of the model-solving procedure.
 

ESTIMATE=re-estimation interval [1]

Use ESTIMATE if you re-estimate the model at some interval other than every period (or never). When you apply the forecasting form of THEIL, RATS uses the current estimates to generate sets of forecasts beginning in the re-estimation interval periods after start. You have to code the re-estimation yourself.

Parameters—Forecasting Form

Options—Forecasting Form

FROM=starting period of the forecast interval [last estimation end+1]

Starting period of forecast. This provides the same information as the start parameter.
 

FORECASTS=(input) VECT[SERIES] of forecasts

This allows you to input a series of forecasts which can't be generated by EQUATION or MODEL-based forecasting calculations. Note that you still have to provide a MODEL or set of equations in the SETUP form so THEIL knows what the dependent variable(s) are.

 

PRINT/[NOPRINT]

Use PRINT to get THEIL to print the forecasts together with the actual values. This can generate a lot of output.

Options—With the DUMP Option

REPLACE/[NOREPLACE]

With REPLACE, THEIL replaces the raw forecast statistics accumulated in the array %THEIL (defined by this instruction) with the finished statistics.
 

You can only use REPLACE when you have reached the end of the period that you are evaluating, since it destroys the raw statistics.
 

TITLE="Title of report"

WINDOW="Title of window"

You can use the TITLE option to label the output with the string you supply. This will also be used as the title for the copy of the report stored in memory. If you use the WINDOW option, the output will be displayed in a (read-only) spreadsheet window (rather than in the output window) with the window title you supply. You can export the contents of this window to a file in any of several formats using File–Export....

Description—With the DUMP Option

THEIL with the DUMP option processes and prints the statistics that the previous commands have compiled. You can include several THEIL(DUMP,...) instructions in a sequence. For instance, you can check the performance through the first five years as well as through the full period.

Using the THEIL instruction

The recommended way to use THEIL is to estimate the model through some time period before the end of the data and evaluate the performance of the model over that last stretch of data, with periodic re-estimation of the coefficients. This is as close as you can come to putting the model to an “honest” test, since you are always making the forecasts using coefficients estimated without use of the data from the forecast period itself.

 

You can easily do periodic re-estimation for vector autoregressions by using KALMAN. For ARIMA models, you need to do a new BOXJENK instruction, and to save computation time, you may decide not to do this every period.

Interpreting the Output

This is an example of the summary for one variable.

 

Forecast Statistics for Series USARGNP

Step  Mean Error Mean Abs Error  RMS Error      Theil U  N.Obs

   1 -0.002185156  0.004646036  0.006342988    0.6099094   24

   2 -0.005635293  0.008570236  0.011294940    0.6021547   23

   3 -0.010004031  0.012008270  0.016554306    0.6168643   22

   4 -0.014327303  0.016584875  0.021560530    0.6157698   21

   5 -0.018220694  0.020893524  0.026150736    0.5982887   20

   6 -0.021564104  0.024578867  0.029866540    0.5736811   19

   etc.

 

Mean Error vs. Mean Absolute Error

If the Mean Error is approximately the same magnitude as the Mean Absolute Error, the model consistently forecasts either low (if the mean error is positive) or high (if negative). For instance, the model above apparently overestimates USARGNP, especially at longer time horizons.

 

Mean Absolute Error vs. RMS Error

The RMS error will always be at least as large as the mean absolute error. They will be equal only if all errors were exactly the same size. If the RMSE is several times as large as the MAE, there may very well be an error in your program (such as a mistake on the data end parameter on THEIL(SETUP)), as this only happens if there are a few very large errors.

 

Theil’s U

Theil’s U statistic has several advantages over the RMS error when you are comparing models.

 

First, as a unit-free measurement, it is often easier to work with than the unit-bound RMSE. For instance, Theil U’s for interest rate series usually are between .8 and 1.0, while RMSE’s will vary depending upon term.

 

Second, it provides an immediate comparison of the forecasts with those of the naive scheme of forecasting no change over time.

 

A value in excess of one is not promising, since it means the model did worse than the naive method. However, a value substantially less than one should not necessarily be construed as a major success—almost any reasonable procedure will produce such a value for a series with a strong trend.

 

N. Obs column

The N. Obs column lists the number of different forecasts upon which the statistics at that horizon are based. Here, we’ve used THEIL in a loop over 24 different starting periods. As the starting period approaches the end of the sample, there is no data against which to compare the forecasts of the later steps. Thus the total number of available data points is less for the longer horizons. For example, the table shows that the one-step horizon results are based on 24 different one-step forecasts (one for each starting period used), while the two-step horizon results are based on 23 forecasts, and so on.

Examples

boxjenk(define=bjeq,sdiffs=1,ar=2,sma=1) ldeuip * 2003:12

theil(setup,estimate=6,steps=24,to=2009:12) 1

# bjeq

do time=2004:1,2009:12,6

   theil time

   boxjenk(define=bjeq,sdiffs=1,ar=2,sma=1) ldeuip * time+5

end do time

theil(dump)

 

This evaluates a Box-Jenkins model over the period 2004:1 to 2009:12. To reduce computation time, the model is re-estimated only every 6th period. The THEIL instruction inside the loop will do one sets of forecasts beginning with period TIME, another beginning with TIME+1, etc. through TIME+5.

 

system(model=canmodel)

variables canrgnp canm1s cantbill cancpinf canusxsr usargnp

lags 1 to 4

det constant

specify(tightness=.15,type=symmetric) 0.5

end(system)

 

theil(setup,model=canmodel,steps=8,to=2009:4)

estimate(noprint,noftests) * 1997:4

do time=1998:1,2009:4

   if time==2003:1 ; theil(dump)

   theil time

   kalman

end

theil(dump)

 

This evaluates a six-variable VAR. The evaluation period is 1998:1 to 2009:4. The initial ESTIMATE goes through 1997:4, then the KALMAN instructions update the coefficients. The THEIL(DUMP) after the IF statement prints the intermediate results as of 2003:1.

Variables Defined

 

The five rows hold the following raw/finished statistics:

 

1.sum of forecast errors / mean error

2.sum of absolute errors / mean absolute error

3.sum of squared errors / root mean square error

4.sum of squared errors of no-change forecasts / Theil U

5.number of forecast observations for this step

 

The REPLACE option of THEIL(DUMP) replaces the raw statistics with the finished statistics. For instance, if you replace the final THEIL(DUMP) in the second example with THEIL(DUMP,REPLACE), then %THEIL(3)(4,8) will be the Theil U (row 4) for step 8 for CANTBILL (3rd equation) and %THEIL(6)(3,5) will be the RMSE (row 3) for step 5 for USARGNP.

Technical Information

Each time you execute a THEIL in forecasting form, RATS computes forecasts for steps horizons (or fewer, if the data end period is hit first). From these and the actual data, it computes forecast errors, absolute errors, squared errors, and squared errors of the naive (flat) forecasts. The program stores running sums of these statistics, and keeps track of the number of times that statistics have been computed for each horizon. We’ll call this latter value \(N_t\)—the number of times that a forecast has been computed for horizon t (t=1 for one-step-ahead forecasts, t=2 for two-step-ahead forecasts, etc.). In the sample shown earlier in “Interpreting the Output”, \(N_1\) was 24, \(N_2\) was 23, and so on.

 

When you use THEIL(DUMP), RATS divides the sums by \(N_t\) to convert them into means, and from these it computes the RMS error and Theil U statistics. The formulas for these computations (at forecast horizon t) are given below:

 

 

Copyright © 2025 Thomas A. Doan