Justin Wolfers, an economist, via Marginal Revolution:
When I watch and speak with my friends in psychology, very little of their work is about analyzing observational data. It’s about experiments, real experiments, with very interesting interventions. So they have a different method of trying to isolate causation. I am certain that we have an enormous amount to learn from them. But I am curious why we have not been able to convince them of the importance of careful analysis of observational data.
By “careful analysis of observational data” I think Wolfers means the way economists search within observational data for comparisons in which the factor of interest is the only thing that changes (which is why he says “isolate” rather than “infer”). He’s right — it really is a methodological innovation that psychologists are unfamiliar with. It lies between ordinary survey data and experiments.
Here’s why I think this innovation has had (and will have) little effect on psychology:
1. Most psychology professors are bad at math. They still use SPSS! Which is terrible but they think R is too difficult. Economics papers are full of math. That is part of the problem. Math difficulty also means they have trouble with basic statistical ideas. When analyzing data, they’re afraid they’ll do the wrong thing. For example, most psychology professors don’t transform their data. It wasn’t in some crummy textbook so they are afraid of it. Lack of confidence about math makes them resistant to new methods of analysis. Experimental data is much easier to analyze than observational data. You don’t need to be good at math to do a good job. So they not only cling to SPSS, they cling to experimental data.
2. Psychology studies smaller entities than economics. Study of the parts often influences study of the whole; the influence rarely goes the other way. This is why, when it comes to theory, physics will always have a much bigger effect on chemistry than vice-versa, chemistry a much bigger effect on biology than vice-versa. Method is different than theory but if you aren’t reading the papers — and physicists don’t read a lot of chemistry — you won’t pick up the methods.
3. There is a long history of longitudinal research in psychology. Studying one or more groups of children year after year into adulthood. The Terman Genius project is the most famous example. I find these studies unimpressive. They haven’t found anything I would teach in an introductory psychology class. I think most psychologists would agree. This makes observational data less attractive by association.
4. Like everyone else, psychologists have been brainwashed with “correlation does not equal causation”. I have heard many psychology professors repeat this; I have never heard one say how misleading it is. To the extent they believe it, it pushes them away from observational data.
5. Psychologists rarely use observational data at all. To get them to appreciate sophisticated analysis of observational data is like getting someone who has never drunk any wine to appreciate the difference between a $20 wine and a $40 wine.