A new article in The New Yorker by Jonah Lehrer is about declines in the size of experimental or quasi-experimental effects over time. For example, Jonathan Schooler, an experimental psychologist, found that if subjects are asked to describe a face soon after seeing it their later memory for the face is worse. As Schooler continued to study the effect, it appeared to get weaker. The article also describes examples from drug trials (a anti-psychotic drug appeared to become weaker over 15 years) and ecology (the effect of male symmetry on mating success got weaker over years).
It’s nice to see an ambitious unconventional article. I blogged a few weeks ago about difficulties replicating the too-many-choices effect. Difficulty of replication and the decline effect are the same thing. I could do what Jared Diamond does in Collapse: give a list of five or six reasons why this happens. (Judging by this paper, the effect, although real, is much weaker than you’d guess from Lehrer’s article.) For example, the initial report has much more flexibility of data analysis than later reports. Flexibility of analysis allows researchers to increase the size of effects.
A long list of reasons would miss a larger point (as Diamond does). A larger point is this: Science (search for truth) and profession (making a living) are not a good fit. In a dozen ways, the demands of a scientist’s job get in the way of finding and reporting truth. You need to publish, get a grant, please your colleagues, and so on. Nobody pays you for finding the truth. If that is a goal, it is several goals from the top of the list. Most jobs have customers. If a wheelwright made a bad wheel, it broke. Perhaps he had to replace it or got a bad reputation. There was fast powerful feedback. In science, feedback is long-delayed or absent. Only long after you have been promoted may it become clear anything was wrong with the papers behind your promotion. The main customers for science are other scientists. The pressure to have low standards — and thus appear better to promotion committees and non-scientists — is irresistible. Whereas if Wheelwright Y makes better wheels than Wheelwright X, customers may notice and Wheelwright Y may benefit.
There are things about making science a job that push scientists toward the truth as well, such as more money and time. When science is a job, a lot more research gets done. Fine. But how strong are the forces against finding truth? I was never surprised by the replication difficulties Lehrer writes about. I had heard plenty of examples, knew there were many reasons it happened. But I was stunned by the results of my self-experimentation. I kept finding stuff (e.g., breakfast disturbs sleep, butter improves brain function) that contradicted the official line (breakfast is the most important meal of the day, butter is dangerous). Obviously I had a better tool (self-experimentation) for finding things out. The shock was how many things that had supposedly been found out were wrong. Slowly I realized how much pressure career demands place on scientists. It is no coincidence that the person most responsible for debunking man-made global warming, Stephen McIntyre, is not a professional climatologist (or a professional scientist in any other area). Unlike them, he can say whatever he wants.
Thanks to Peter Couvares.
More In his blog, failing to see the forest for the trees, Lehrer says we must still believe in climate change (presumably man-made): climate change and evolution by natural selection “have been verified in thousands of different ways by thousands of different scientists working in many different fields.” Charles Darwin, like McIntyre, was an amateur, and therefore could say whatever he wanted.