Ayres on Super Crunchers and the Power of Data (the Library of Economics and Liberty)
October 22, 2007
Ian Ayres of Yale University Law School talks about the ideas in his new book, Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart. Ayres argues for the power of data and analysis over more traditional decision-making methods using judgment and intuition. He talks with EconTalk host Russ Roberts about predicting the quality of wine based on climate and rainfall, the increasing use of randomized data in the world of business, the use of evidence and information in medicine rather than the judgment of your doctor, and whether concealed handguns or car protection devices such as LoJack reduce the crime rate. The podcast closes with a postscript by Roberts challenging the use of sophisticated statistical techniques to analyze complex systems.
Listen to this podcast: Play time: 1:02:44 min.
|
- About this week's guest:
- Ian Ayres's Home page
- Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart, by Ian Ayres. Bantom, 2007. At amazon.
- Ian Ayres's Prediction Tools page
- "Bordeaux Wine Vintage Quality and the Weather,", by Orley Ashenfelter, David Ashmore, and Robert LaLonde. Chance, Vol. 8 No. 4, 1995. Harris School Working paper.
- "How computers routed the experts,", a commentary by Ian Ayres. Financial Times, August 31, 2007.
- "Let's Take the Con Out of Econometrics,", by Ed Leamer. The American Economic Review, Vol. 73, No. 1 (Mar., 1983), pp. 31-43. At JStor.
- Childbed Fever: A Nineteenth-Century Mystery, by Christa Colyer. Ignaz Semmelweis and hand-washing.
- The Law of Large Numbers, by Doctor Jordi. Ask Doctor Math, 02/08/2001.
- Introduction to Evidence-Based Medicine. Duke University Medical Center Library and Health Sciences Library, UNC-Chapel Hill, 2004.
- Multiple Regression. StatSoft, 2003.
- "The Economics of Crime and the Criminal Justice System,", by Steven D. Levitt. NBER. On LoJack and Ayres.
- Weinberger on Everything is Miscellaneous and the Wonderful World of Digital Information. EconTalk podcast, June 18, 2007.
- Sunstein on Infotopia, Information and Decision-Making. EconTalk podcast, May 14, 2007. Predicting the Supreme Court's decisions.
- Fooling Ourselves, by Russ Roberts. Cafe Hayek, October 22, 2007.
- Econometric Confirmation Bias, by Arnold Kling. EconLog, October 22, 2007.
About ideas and people mentioned in this podcast:
- Articles:
Web Pages:
Podcasts and Blogs:
Podcast Highlights
| Time Mark | Highlights |
|---|---|
| 0:36 | Intro. Data and statistics. What is the case for super crunching? Orley Ashenfelter wine example. Multivariate regression to try to find underlying relations between rainfall and weather and quality of the wine. Surprisingly accurate. Can't drink Bordeaux for several months. Robert Parker, traditional wine taster, originally dismissive of Ashenfelter, but has started incorporating weather into his predictions. Statistics did better than experts, but traditional experts resist the new breed of number-crunchers. Ignaz Semmelweis, doctor who discovered doctors' hand-washing helped save mothers in childbirth. Resisted by medical profession. Even today doctors tend to go from one patient to the next without washing their hands. Business leaders, Wal-mart and hurricanes example. Statistical analysis of consumer purchases after hurricanes. Story goes that statistically included pop-tarts, so they stock up on pop-tarts before hurricanes--urban myth? Does putting beer and diapers together in quick-purchase stores increase sales or is it an urban myth? Grocery store layouts, toothpaste vs. toothbrush placements. Weinberger podcast. |
| 10:54 | Randomized experiments. Why when you call a credit card company do they ask in a recording for some info and then ask you again? The recording is not telling the agent the info. Why not? Capital One example. What kind of products and services do you want? What you type in during the initial phone contact affects the routing of your call. Prediction, up-selling. Computer calculates even the interest rates you are offered by the representative. Randomized experiments based on mass mailings test what kinds of promotions work best: kitten picture vs. puppy picture? 2% teaser rate for 2 months or 1% for 4 months? Law of large numbers kicks in if you sample enough people. Distributions of the two groups will be the same if you have enough people. Randomized studies give very accurate answers if you have a large enough sample. |
| 16:54 | EBM (evidence-based medicine). Randomized studies have been done on whether taping your knee actually gives relief to knee pain, oral vs. vitamin shots, types of acupuncture. Grading of evidence creates a kind of competition between evidence-gathering styles. Physicians are in recent years for the first time starting to do patient-specific research. Previously they might read generally in the field but they didn't have a medical library and they didn't check your specific case. Now through the Internet the physician can look up various treatments and see how they are graded, based on quality of evidence. Flip side: When a patient comes into a hospital with pneumonia, he has to have antibiotics within 4 hours. However, because of that mandate, in some hospitals everyone gets chest x-rays just in case he has pneumonia. Pitfall of rules versus gut instinct. Does the data give rise to the right rules or not? Physicians have ceded their control over treatment, and focus on diagnosis instead. Next EBM revolution may result in their ceding diagnosis, digital medical records. Doctors' decisions to order a test are not always the same as your desires. Insurance companies' incentives versus lawyers' incentives. In any business with 1000 employees, it doesn't make sense to build institution around the top 10%. Taking away some of their discretion and basing it on statistical information will do better. |
| 25:26 | Prediction page has about 40 prediction tools that will help you predict things like how long you will live, your due-date if pregnant, sporting events, likelihood that a book title will become a best seller. Will give you the average length of marriage for people will your characteristics. Regression output also tells you the precision of the prediction, 95% confidence range. |
| 27:57 | Regression analysis in economics: can you successfully hold variables constant? Hard to isolate the effect of one variable on another. What's happened to the standard of living in the U.S. since the late 1970s? Stagnant, if you look at average hourly earnings corrected for inflation. But that number doesn't include fringe benefits, demographics, etc.--highly controversial number. Have to trust the outcome variable. Statistical analysis cannot make accurate predictions about all things, have to be able to measure the things you care about, have to be able to run a randomized experiment, have to have a large enough population. Some claim that half of all statistical results are wrong. Need other statistics to know if that claim is true. Claim is a relative one: That in case after case, statistical prediction does better than human prediction. Common idea is that the more subtle the event, the more humans should be relied on, but in fact it's the opposite. With even ten causal factors, statistical prediction does better than humans. Humans can't bring themselves to put the right weight on the right factor when there are many factors. 83 legal experts vs. crude statistical algorithm tried to predict Supreme Court, yet the statistical algorithm did better than the legal experts at predicting the Supreme Court's decisions. Supreme Court hates the 9th Circuit, California, but legal experts can't bring themselves to take that history into account. The algorithm wasn't precise but it still did better than humans. |
| 35:33 | Social issues: immigration, wages, Wal-mart. For LoJack, automobiles with radio chips have been found to actually reduce crime. Concealed handgun laws, John Lott, do they deter crime in the analogous way? Lott and Ayres don't agree about handgun laws. If one person has LoJack, no thief will worry; but if half the city has it would deter car theft. LoJack is never used as an offensive weapon, but concealed weapons could potentially be used to commit crime, so which effect dominates with concealed weapons? Lott has played an important role in changing the norms of data sharing. Journals are starting to require the posting of data publicly. Leamer, "Let's Take the Con Out of Econometrics." Cost of computation has become cheap. Is there a chance we will get closer to the truth with careful studies? Russ: Monetary History of the United States by Friedman and Schwartz, gold standard of simple statistical analysis. Changed opinion about how to measure the money supply. Are there more complicated examples? Ironic that example is in macro. Ayres: gold standard is Heckman on civil rights issues, micro side, 1964 Civil Rights Act affected hiring in Southern textile industries. Donahue and Levitt abortion article. Not the final word. "Clash of competing studies helps us make progress." New categories of inquiry. Does immigration increase wages of native-born workers? No consensus yet, but we won't get it non-statistically. It may not even be a big enough effect for us to come up with a credible belief in its response. Shouldn't just trust any super cruncher. Businesses may not be following the academic clash approach, may need to hire statistical auditors. |
| 46:46 | Worries: First, most findings confirm the bias of the researcher because of the range of regressions and techniques you can run, so researchers keep crunching till the results show their biases. It's a concern. Second, ordinary folks don't have the statistical sophistication to be skeptical of sloppily done findings. What is the statistical study that you believe but don't like? If you only believe the ones you like you must be a biased consumer. Easier to cook the books on regressions than on randomized trials. First results out of Move to Opportunity, study where they gave housing vouchers to poor families so they could move to middle class neighborhoods. Randomized experiment. Does it impact life-chance results? Hasn't actually improved the life very much of those who have moved. That was certainly not anticipated by those who put their money into the program. |
| 50:16 | Summary of issues. Social science research. Hand guns example. Spurious correlations in economics. Simultaneity problem. LoJack. Ayres very confident about LoJack but thinks Lott is wrong about guns. Vice versa for Lott. They go back and forth on details, but what if source is different. Neither may measure with any precision. Data may simply not be good enough to allow us to measure even the direction of the results. Results can be paraded around as scientific when they are not. Can either side in a policy debate concede that some statistical results are more convincing than others? Two-stage least squares--techniques are glamorous and elegant but may be used with data that are not up to the task. Ed Leamer: wrong conclusions happen often because researchers try so many specifications that fail and throw those out, trying again till they find something that works. Bloodletting in medieval times. Leamer quote. Faith-based empirical work. Empirical tournaments, the Iron Economist. |
Featured Speakers and Categories: Ian Ayres on Books Information
Posted by Russ Roberts
