I would like to compare the forecasting performance of a BVAR model with a classic VAR one, as in the example (7. Using RUNTHEIL, in the user guide.
I wonder if I can compare the "new" model that results in the Gibbs sampler "loop" after the command "compute %modelsetcoeffs(SNVAR_EG1,bdraw)" to a classic VAR model estimated using SUR?
It would be much more complicated than what @RunTHEIL does. That compares forecasts produced by point estimates of different VAR's. The Gibbs sampler instead produces a "cloud" of forecasts instead of a single set of numbers. So the performance statistics would have to somehow be averaged across simulations.
My idea is: the BVAR in the @RunTHEIL is based on the basic Minnesota prior, with the possibility of changing its hyperparameters, but the prior that I implement is the independent normal wishart one with informative Minnesota on the coefficients, and the forecasting performance resulting from the model with the latter prior is the one that I want to compare to a classic one estimated with sur to tackle the change in the errors variance.
So If there isn't a ready code for this, would you kindly give me ideas so I can implement it?
@RunTheil does a sequence of estimates using Kalman filter. You really can't do that with the Gibbs sampler without a staggering number of calculations. In order to get the "point" forecasts for the Gibbs sampler for a fixed estimation interval, you would have to generate VAR coefficients using the Gibbs sampler, generate a set of forecasts from that, and average across forecasts across sampled models. That would correspond to the point forecasts out of the simpler form of BVAR. You would need quite a few draws in order to make that sufficiently accurate. Rolling that forward would require redoing that whole sequence of calculations.
Would you kindly give me more help with some routines guiding me to the application of this. In the following paper, footnote (7) page 13, I think the same idea is applied.
On the other hand, in the same paper, Historical Decomposition, conditional and unconditional forecasting are conducted with BVAR model, would you kindly tell me how this is applied with Rats.
"In order to get the "point" forecasts for the Gibbs sampler for a fixed estimation interval, you would have to generate VAR coefficients using the Gibbs sampler, generate a set of forecasts from that, and average across forecasts across sampled models. That would correspond to the point forecasts out of the simpler form of BVAR. You would need quite a few draws in order to make that sufficiently accurate. Rolling that forward would require redoing that whole sequence of calculations."
What do you mean here by fixed estimation interval?
What do you mean across sampled models?
How many are "Quite few draws" for you?
"Rolling that forward" could be done by estimating the VAR model, before the loop, with an additional point and redo the whole sequence of calculations? This could be done by a loop for the whole code?
For what it is worth, I think you should be able to find answers to many of your questions about forecasting with Bayesian VARs in the 1997 paper in the Journal of Applied Econometrics by Kadiyala and Karlsson and in a chapter on forecasting with Bayesian VARs that Karlsson wrote for the recently published vol. 2 of the Handbook of Economic Forecasting (Elsevier).
Todd Clark
Economic Research Dept.
Federal Reserve Bank of Cleveland
I'm sorry to bother you with so many questions, I tried to implement what you recommended me to do to compare the forecasting performance of a BAVR estimated with Gibbs sampling and classical VAR one.
I need only a confirmation that what I did is correct.
I have a data sample 1974-2010,
I estimate the classical VAR for 1974-2004, then i use the command simulate to forecast with this model, is it ok to do this with a classical VAR?
Then I compute the RMSE to measure the forecating performance of the model using:
Do i=1,nvar
Set forecastEorrder0(i) fstart fend = (%modeldepvars(SBVAR_EG)(i)- Clas_simresults(i))**2
Sstats(mean) fstart fend forecastEorrder0(i)>>SSE0(i)
Compute RMS0(i) = sqrt(SSE0(i))
Display RMS0(i)
End do i
Then for the BVAR model, within the gibbs sampling loop, I do the same with each new coefficients:
I'm sorry to bother you with so many questions, I tried to implement what you recommended me to do to compare the forecasting performance of a BAVR estimated with Gibbs sampling and classical VAR one.
I need only a confirmation that what I did is correct.
I have a data sample 1974-2010,
I estimate the classical VAR for 1974-2004, then i use the command simulate to forecast with this model, is it ok to do this with a classical VAR?
Simulate(model=SBVAR_EG,results=Clas_simresults,from=fstart,to=fend) This goes forward for two years.
Then I compute the RMSE to measure the forecating performance of the model using:
Do i=1,nvar
Set forecastEorrder0(i) fstart fend = (%modeldepvars(SBVAR_EG)(i)- Clas_simresults(i))**2
Sstats(mean) fstart fend forecastEorrder0(i)>>SSE0(i)
Compute RMS0(i) = sqrt(SSE0(i))
Display RMS0(i)
End do i
Then for the BVAR model, within the gibbs sampling loop, I do the same with each new coefficients:
set forecast(i) fstart fend = forecast(i)+simresults(i)
set forestderr(i) fstart fend = forestderr(i)+simresults(i)**2
End do i
outside the loop, I compute the performance statistics as above:
Do i=1,nvar
Set forecast(i) fstart fend = forecast(i)/ndraws
Set forecastEorrder(i) fstart fend = (%modeldepvars(SBVAR_EG)(i)- forecast(i))**2
Sstats(mean) fstart fend forecastEorrder(i)>>SSE(i)
Compute RMS(i) = sqrt(SSE(i))
Display RMS(i)
End do i
Then i divide both results to get relative performance statistics of both model. I repeat the whole calculation again for 1974-2005, etc.
What do you think?
Do you recommend the command forecast instead of simulate?
No. Neither of those will be correct. SIMULATE just gives you one random simulation of the process---it's not a forecast. If there's no closed form for computing forecasts, you need to compute a large number of simulations, and take the mean or median of those as the forecast. That's what the authors of the paper say they did (in their case taking the median).
Do i=1,nvar
Set forecastEorrder0(i) fstart fend = (%modeldepvars(SBVAR_EG)(i)- Clas_simresults(i))**2
Sstats(mean) fstart fend forecastEorrder0(i)>>SSE0(i)
Compute RMS0(i) = sqrt(SSE0(i))
Display RMS0(i)
End do i
Is this correct, or I should use the command forecast instead, and replace Simulate for the BVAR also with forecast.
Using RUNTHEIL, in the user guide.