Have you ever thought economists were far more confident in their statements about the world than they had any right to be? Well, now there’s proof.
It comes from Emre Soyer, who just got his Ph.D in economics and decision making at the Universitat Pompeu Fabra in Barcelona, and his professor Robin Hogarth, a psychologist who has long been one of the most prominent students of how people make judgments and decisions. Their paper, “The illusion of predictability: How regression statistics mislead experts,” is the centerpiece of a special section in the July-September 2012 issue of the International Journal of Forecasting(which will cost you a fortune to read if you don’t happen to belong to an institution with a subscription). An earlier version is here.
What Soyer and Hogarth did was get 257 economists to read about a regression analysis that related independent variable X to dependent variable Y, then answer questions about the probabilities of various outcomes (example: if X is 1, what’s the probability of Y being greater than 0.936?).
When the results were presented in the way empirical results usually are presented in economics journals — as the average outcomes of the regression followed by a few error terms — the economists did a really bad job of answering the questions. They paid too much attention to the averages, and too little to the uncertainties inherent in them, thereby displaying too much confidence.
When the economists were shown the numerical results plus scatter graphs of the same data, they did slightly better. The economists who were shown only the graphs and none of the numerical results, meanwhile, actually got most of the answers right, or close to right.
The bigger point here, which Soyer and Hogarth have elaborated in other research, is that we tend to understand probabilistic information much better when it’s presented in visual form than if we’re just shown the numbers. (This was also a key argument of Sam Savage’s edifying and entertaining 2009 book The Flaw of Averages.) What’s so interesting is to learn that statistically literate experts are just as likely to glom onto the point estimate and discount the uncertainty as, say, innumerate journalists reporting the results of political polls.
When Hogarth presented the paper at the International Symposium on Forecasting in Boston Tuesday, there was some debate over whether it was really fair to pick on economists as he and Soyer had. The economist next to me thought scholars in other fields would make the same mistakes. The physicist behind me said that since physicists’ natural inclination was to believe that economists are usually wrong, they wouldn’t pay much heed to the result.
Hogarth agreed that psychologists, sociologists, physicists, and others would probably make similar errors. But he said it was important to focus on economists because they’re “very arrogant people” (he taught for 22 years at the University of Chicago’s business school, so he should know) and tend to rely heavily on regression analyses without really thinking through the implications of those analyses. The whole point of doing a regression is to make a prediction that a relationship between variables discovered in past data will hold up in the future. But economists — despite Milton Friedman’s famous claim that prediction is what the discipline is all about — seem unwilling to make those predictions explicit and express their level of confidence in them, thereby giving short shrift to the uncertainty and error inherent in their work.
“My concern,” Hogarth said, “is that when reading economics journal articles you get the impression that the world is much more predictable than it is.” That’s true enough. It’s also true of the way forecasts are usually presented outside of academic economics — in business and in government, for example. We focus on the estimate, and put the uncertainty in a footnote, if anywhere. And when somebody tries to communicate their forecasts with the uncertainty front and center, like the Bank of England with its awesome fan charts, they often catch flak for it. Wanting to be more certain about the future than we have any right to be may well be an ineradicable human trait. But hey, at least somebody has identified a treatment: More scatter graphs!
