A new article in The New Yorker by Jonah Lehrer is about declines in the size of experimental or quasi-experimental effects over time. For example, Jonathan Schooler, an experimental psychologist, found that if subjects are asked to describe a face soon after seeing it their later memory for the face is worse. As Schooler continued to study the effect, it appeared to get weaker. The article also describes examples from drug trials (a anti-psychotic drug appeared to become weaker over 15 years) and ecology (the effect of male symmetry on mating success got weaker over years).
It’s nice to see an ambitious unconventional article. I blogged a few weeks ago about difficulties replicating the too-many-choices effect. Difficulty of replication and the decline effect are the same thing. I could do what Jared Diamond does in Collapse: give a list of five or six reasons why this happens. (Judging by this paper, the effect, although real, is much weaker than you’d guess from Lehrer’s article.) For example, the initial report has much more flexibility of data analysis than later reports. Flexibility of analysis allows researchers to increase the size of effects.
A long list of reasons would miss a larger point (as Diamond does). A larger point is this: Science (search for truth) and profession (making a living) are not a good fit. In a dozen ways, the demands of a scientist’s job get in the way of finding and reporting truth. You need to publish, get a grant, please your colleagues, and so on. Nobody pays you for finding the truth. If that is a goal, it is several goals from the top of the list. Most jobs have customers. If a wheelwright made a bad wheel, it broke. Perhaps he had to replace it or got a bad reputation. There was fast powerful feedback. In science, feedback is long-delayed or absent. Only long after you have been promoted may it become clear anything was wrong with the papers behind your promotion. The main customers for science are other scientists. The pressure to have low standards — and thus appear better to promotion committees and non-scientists — is irresistible. Whereas if Wheelwright Y makes better wheels than Wheelwright X, customers may notice and Wheelwright Y may benefit.
There are things about making science a job that push scientists toward the truth as well, such as more money and time. When science is a job, a lot more research gets done. Fine. But how strong are the forces against finding truth? I was never surprised by the replication difficulties Lehrer writes about. I had heard plenty of examples, knew there were many reasons it happened. But I was stunned by the results of my self-experimentation. I kept finding stuff (e.g., breakfast disturbs sleep, butter improves brain function) that contradicted the official line (breakfast is the most important meal of the day, butter is dangerous). Obviously I had a better tool (self-experimentation) for finding things out. The shock was how many things that had supposedly been found out were wrong. Slowly I realized how much pressure career demands place on scientists. It is no coincidence that the person most responsible for debunking man-made global warming, Stephen McIntyre, is not a professional climatologist (or a professional scientist in any other area). Unlike them, he can say whatever he wants.
Thanks to Peter Couvares.
More In his blog, failing to see the forest for the trees, Lehrer says we must still believe in climate change (presumably man-made): climate change and evolution by natural selection “have been verified in thousands of different ways by thousands of different scientists working in many different fields.” Charles Darwin, like McIntyre, was an amateur, and therefore could say whatever he wanted.
Would there be a way to structure a research organization to produce extremely transparent and truth-oriented research? If so, would the quality of their aggregate conclusions be novel and high-quality enough to demonstrate the superiority of this way of doing science?
There’s also this article on a somewhat related topic:
“Placebos Are Getting More Effective. Drugmakers Are Desperate to Know Why.“
You write about the incentives faced by scientists and the ways in which you believe that distorts research on various subjects, such as climate change. You make good points. However, when you talk about climate change I’m a bit surprised that you neglect to talk about the incentives faced by the climate change “skeptics” — namely, the large amounts of money that special interests (energy companies, etc) have pumped into the debate in the interest of calling into question anthropogenic climate change and thus heading off policy changes. This is a distortion that I would guess to be at least as large as the ones that you identify.
A colleague once explained a second colleague’s professional success by saying “He’s careful not to be familiar with the literature, and never to scrutinise his own data critically”.
M, I don’t believe that Stephen McIntyre is a tool of the energy companies. His skepticism about the hockey stick graph, it has turned out, was fully justified. It is unsurprising that it took an amateur to do what for a professional would have been career suicide — make numerous Freedom of Information Act requests from someone who might review your next grant.
More specifically, in many cases there is what I call “the car mechanic problem”. The people most qualified to judge how big a problem X is are also the people who benefit most from describing X as a huge problem, whether X is your motor, climate change, youth crime or passive smoking.
LemmusLemmus, “the car mechanic effect” is a nice name for it. In practice it describes only half of the problem — assessments of how bad something is. The other half of the problem is assessments of how good something is — professors of engineering judging the work of professors of engineering, professors of English judging the work of professors of English, and so on. It’s curious there is no name for this.
Seth, as much as I enjoy reading about your self-experimentation research and think it has real value for generating hypotheses, I doubt much of it is replicable in well-controlled studies (I’ve tried a few of the interventions and they haven’t done anything for me as far as I can tell). Unless your work is replicated more robustly than the research you disparage, it seems premature to conclude that it is in any way better.
Misunderstanding of placebo continues.
“The main objections to more widespread placebo use in clinical practice are ethical”
The placebo works by the belief that it works. Therefore, just teach the patient that they can get much better due to a chemically inert placebo. Then, give them the placebo. The barrier to this is that you can’t patent the sugar pill – and placebos don’t have to be pills, but basically any ritual.
Fully generalized, if you believe that you can activate the placebo effect at will, you can, because of that very belief. (Though it’s nice to have things to reinforce the belief.)
Seth, I don’t know much about McIntyre so I can’t comment on him specifically.
I like your point about how amateurs (or insider/outsiders, as I think you sometimes say) can question in areas where “professionals” blind themselves to the truth.
When it comes to climate change, yes, professionals would have to be brave to question their peers. Call this the first set of incentives. But there is another set of incentives, operating in the other direction: a climate skeptic wouldn’t have to be as fearful as skeptics in other more obscure scientific areas. This is because special interests (energy companies, etc) provide plenty of financing to media (that heap praise on skeptics), jobs at “think tanks”, contracts through publishing houses, etc. In other words, to be a climate skeptic can have substantial financial, social, and political rewards.
I’m surprised that, when it comes to climate change, you focus on the first set of incentives, but not the second.
M, you write “to be a climate skeptic can have substantial financial, social, and political rewards.” I am talking about academia. Professors of climatology, professional scientists. I cannot think of a single professor who has reaped those rewards. Whereas the grant money given to Michael Mann, for example, is in the millions.
vic, what “interventions” do you doubt are replicable?
Seth, I don’t know if they are replicable or not, the point is I’d remain skeptical till I saw the effects replicated in some sort of systematic study in more than one person — or at least until they worked for me. A few of the ones I tried: standing on one leg and eating more animal fat didn’t seem to improve my sleep; faces in the morning didn’t seem to improve my mood; on the other hand, morning light did seem to improve my sleep.
Vic, the morning faces effect depends on several things you may not have realized. For example, fluorescent light exposure at night may cause the effect to go away. With animal fat, you may have not used a large enough dose. Other people have observed both effects. I don’t know anyone who has repeated the standing on one leg effect but I have observed and tested it in several ways. Perhaps it didn’t work for you because something else is causing you sleep trouble. For example you have too much X or too little Y — and standing on one leg affects neither.
I’ve found one-legged standing to work for me, surprisingly well and reliably, though I haven’t tried to measure it.