Friday, June 12, 2009

Roberts on the economics-free application of sophisticated statistical techniques (amv)

Russ Roberts posted a very nice little question and therewith comes close to the way I feel about this topic. His post is an answer to Bryan Caplan's and to understand it you have to know both of them. So first, here is Caplan's from Econlog:

In a Perfect World, I'd Call myself a Sociologist

I've studied economics for over twenty years. The more I think about it, though, the more I realize that I don't know what "economics" means anymore.

Textbooks may say that economics is about "incentives" or "trade-offs." But you can publish papers in econ journals about the effect of birth weight on educational attainment. I don't see any incentives or trade-offs there. Or take Emily Oster's early research arguing that hepatitis, not infanticide or selective abortion, explained a lot of Asia's gender imbalance. Some economists asked, "How is this economics?" But if some economists argue that the gender imbalance is driven by incentives, how can you object if other economists say that the real explanation is medical? Or consider happiness research. Economists like Justin Wolfers are in the vanguard; but the connection to incentives or trade-offs is unclear.

You could deplore all this as a loss of focus. But I see massive progress. Economics has grown hard to define because we now focus primarily on real-world problems, not "literatures." If we want to understand income determination, we don't waste time with topological proofs. We still think about supply and demand, but we also think about policy, psychology, behavioral genetics, and much more. As a result, we come to understand the world, instead of solving unusually difficult homework problems.

What, though, is the common essence behind everything that economists now do? The only answer that works, I think, is that economics is the all-encompassing study of the social world. In the words of the Roman poet Terence, "Homo sum, humani a me nihil alienum puto" - "I am a man, I consider nothing that is human alien to me."

Unfortunately, this puts me in an awkward position. There's another field that already sounds like "the all-encompassing study of the social world": sociology. Not only does sociology have lower status than economics; with honorable exceptions, it's also well-stocked with academics who aren't fond of economics. Tactically, then, it would be foolish to start calling ourselves "sociologists." If we were picking names from scratch, though, "sociologists" is exactly what modern economists ought to proudly call ourselves.


And here is Roberts's reply at Cafe Hayek:

Sociology

[Caplan] goes on to say that what economists do, might best be called "sociology" the word we already have to describe the study of the social world, which is what Bryan is arguing economists increasingly study. I am reminded of what George Stigler said: "There is only one social science and we are its practitioners."

Having noted that, I have to disagree with Bryan on a few things. I think too much of modern empirical economics is the economics-free application of sophisticated statistical techniques that does little to actually advance our understanding of the social world. It's not just that it isn't about trade-offs or incentives, the role of trade-offs and incentives are ignored. I also don't think we've made "massive progress" in understanding the social world. We've made massive progress in publishing papers on the social world. But understanding? Not so much. We treat the natural world as if the sophisticated tools of statistics can turn reality into a natural experiment. But the world is usually (always?) too complex for the results to be reliable.

I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus—a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work.

When I asked Ian Ayres that question, someone who advocates increased use of statistical techniques in various aspects of life, his answer was the Levitt and Donahue study of abortion and crime. A strange answer as it is highly controversial and widely dismissed by skeptics.

My example used to be the Monetary History of the United States by Friedman and Schwartz that created a consensus that the Fed and the money supply play a crucial role in inflation and business cycles. But The Monetary History is a collection of facts rather than the use of the fancy techniques so in vogue today. I'm not sure there's anything sophisticated in the empirical work. Is there even a single regression in it? I don't know. But the point is that I'm not saying that facts are irrelevant to our understanding of the world. I'm saying that attempts to use statistical technique to tease out causation in a complex world is incredibly un-credible.

I am open to other suggestions. I'd like one example, please. One example, from either micro or macro where people had to give up their prior beliefs about how the world works because of some regression analysis, ideally usually instrumental variables as that is the technique most used to clarify causation.

One example. There should be dozens. Or hundreds. But I'll take one
.

If you know an example send it to me