Last updated: April 2002 Volume 49, Number 1

The three papers that follow are part of a symposium on forecasting performance I organized under the auspices of IMF Staff Papers. These are interesting papers that deal with forecast evaluation in real-world situations, ones that can often be far removed from those considered in forecasting textbooks (e.g., Diebold (2001), to pick one at random) in which the evaluator has access to a long history of forecasts made by an individual or agency, knows the model used to generate the forecasts, and can reasonably assume that the forecaster's goal is to do as well as possible in predicting the actual outcome. In contrast to the textbook view, in the papers presented here:

• The history of forecasts is often very short, in many cases because the forecasts are for countries that have only existed in their present form for a few years (e.g., Poland and other transition economies);

• The model used to generate the forecasts is not known, in many cases because the forecasts are judgmental rather than model based;

• The assumption that the forecaster's goal is to do as well as possible in predicting the actual outcome is sometimes questionable. In the context of private sector forecasts, this is because the incentives for forecasters may induce them to herd rather than to reveal their true forecasts. Public sector forecasts may also be distorted, although for different reasons. Forecasts associated with IMF programs, for example, are often the result of negotiations between Fund staff and the country authorities and are perhaps more accurately viewed as goals, or targets, rather than pure forecasts.

Gallo, Granger, and Jeon on Copycat Behavior

The paper by Gallo, Granger, and Jeon presents evidence--using forecasts of growth from Consensus Forecasts, a monthly survey of mostly private sector forecasters--that there is a tendency among forecasters to conform to the mean ("consensus") forecasts. In particular, an individual's growth forecasts are strongly influenced by the consensus forecast of the previous month. This copycat behavior can sometimes lead the consensus toward convergence to a forecast value that is far from the actual value. With copycat behavior, moreover, one cannot use the distribution of individual forecasts as if it were made of independent draws, which casts doubt on the practice of using the dispersion of individual forecasts as an informal forecast confidence interval.

Musso and Phillips on the Accuracy of IMF Projections

Associated with an IMF-supported program is a set of forecasts (or projections, as Musso and Phillips prefer to call them). How do those forecasts compare with actual outcomes? This question became more prominent following 1997, when the initial IMF-supported programs with Indonesia, Korea, and Thailand called for continued growth in the near term, but real GDP contracted sharply in all three countries. Moreover, in Korea and Thailand, projections for continued current account deficits were instead followed by large surpluses and capital flight.

Was the recent experience in East Asia atypical, or in line with a persistent pattern? The paper by Musso and Phillips provides answers by studying the accuracy of projections in 69 IMF-supported programs in the mid-1990s. They study projections in three key areas--output growth, inflation, and the balance of payments--and provide evidence on the bias, efficiency, and accuracy of these projections. Readers hoping for a strong bottom-line result ("IMF projections are wonderful"; "IMF projections are terrible") will be disappointed. Like any sufficiently large set of forecasts, the IMF's projections do well on some dimensions and not so well on others.

Juhn and Loungani on Comparisons between Private Sector and IMF Forecasts

Juhn and Loungani find for a large sample of countries that there is a high degree of collinearity between output growth forecasts of the private sector taken from Consensus Forecasts and IMF forecasts reported in the World Economic Outlook (WEO). The results for directional accuracy are a statistical "dead heat" between the two sources of forecasts, whereas the evidence tends to favor the private sector forecasts as being a little more accurate than, and encompassing, the WEO forecasts. As Juhn and Loungani note, the dominance of private sector forecasts should not come as a surprise. The Consensus Forecasts are updated in a far more timely manner and with a shorter "production lag" than the WEO forecasts. The consensus is also an average of several individual forecasts, which--unless the copycat behavior highlighted in the paper by Gallo, Granger, and Jeon is highly prevalent--should tend to produce a more accurate forecast.

All told, the papers collected here provide a fascinating glimpse into the challenges of practical forecast accuracy evaluation. I hope you enjoy them as much as I did.

REFERENCES

Diebold, F.X., 2001, Elements of Forecasting (Cincinnati: South-Western College Publishing).