I am interested in evaluating density forecast accuracy.

I can evaluate distributional forecasts as per Continuous Ranked Probability Score (CRPS): https://otexts.com/fpp3/distaccuracy.html

An interesting paper:

Wagner Piazza Gaglianone & Gabriel Jaqueline Terra Moura Marins, 2017. "Evaluation of Exchange Rate Point and Density Forecasts: an application to Brazil," International Journal of Forecasting Volume 33, Issue 3, July–September 2017, pp. 707-728

Working paper: https://www.bcb.gov.br/pec/wps/ingl/wps446.pdf

I would like to compute the following accuracy measures as listed in B2 - Full-density forecast, in RATS, especially

(a) Probability Integral Transform (PIT)

(b) Log Predictive Density Score (LPDS)

and any others of interest, similar to CRPS, for the whole forecast distribution.

If I generate fan charts for 3 simple models, assuming the distribution of forecasts is normal, as below, how do I calculate the PIT distribution's (which are usually compared & plotted to U(0,1) histograms, the more similar, the better the density forecast), and LPDS, for each of the 3 models?

thanks,

Amarjit

- Code: Select all
`*===============================`

* read in data

OPEN DATA "/Users/Shared/RATS/Examples/g10xrate.xls"

DATA(FORMAT=XLS,NOLABELS,ORG=COLUMNS,TOP=2) 1 6237 USXJPN USXFRA USXSUI USXNLD USXUK $

USXBEL USXGER USXSWE USXCAN USXITA

*

*

*===============================

* choose series

set y = usxjpn

compute steps=25

compute istart=6000

compute iend=6237-steps

* (1) rwwd (rw without drift) or naive model

boxjenk(diffs=1,define=foreeq_rwwd) y istart iend

uforecast(equation=foreeq_rwwd,from=(iend+1),steps=steps+1,stderrs=stderrs_rwwd) yhat_rwwd

set l95_rwwd iend+1 iend+steps+1 = yhat_rwwd+%invnormal(.025)*stderrs_rwwd

set u95_rwwd iend+1 iend+steps+1 = yhat_rwwd+%invnormal(.975)*stderrs_rwwd

* (2) rwd (rw with drift) model

boxjenk(diffs=1,constant,define=foreeq_rwd) y istart iend

uforecast(equation=foreeq_rwd,from=(iend+1),steps=steps+1,stderrs=stderrs_rwd) yhat_rwd

set l95_rwd iend+1 iend+steps+1 = yhat_rwd+%invnormal(.025)*stderrs_rwd

set u95_rwd iend+1 iend+steps+1 = yhat_rwd+%invnormal(.975)*stderrs_rwd

* (3) mean/constant model

boxjenk(diffs=0,constant,define=foreeq_mean) y istart iend

uforecast(equation=foreeq_mean,from=(iend+1),steps=steps+1,stderrs=stderrs_mean) yhat_mean

set l95_mean iend+1 iend+steps+1 = yhat_mean+%invnormal(.025)*stderrs_mean

set u95_mean iend+1 iend+steps+1 = yhat_mean+%invnormal(.975)*stderrs_mean