In this post Andrew Gelman comments on my recent post about acne self-experimentation. He makes an excellent point about drug-company studies:
How would you want to evaluate the risks and effectiveness of a new drug that was developed by a pharmaceutical company at the cost of millions of dollars? I’d be suspicious of an observational study: even if conducted by professionals, there just seem to be too many ways for things to be biased.
Right. And it’s not just observational studies. The data from any big study can be analyzed many ways. The more at stake, the greater the chance of what Andrew calls bias and I call making choices that favor the result you prefer. Independently of Andrew, Hal Pashler and I came up with what I call the Pashler-Roberts Law: The more expensive the research, the less likely the researchers will be honest about it.
You may remember that Robert Gallo, the AIDS researcher, did very expensive research. The deception (possibly self-deception) that accompanied very expensive fusion research is described in Charles Seife’s Sun in a Bottle: The Strange History of Fusion and the Science of Wishful Thinking (2008).
As Andrew says, this is a big virtue of self-experimentation. Because it’s free, it’s easy to be honest, especially about failure. The cheaper the better is a broad truth about science that’s hard to learn from books or classes or even talking to scientists.
It seems that the expense of the study is an indication of how much the person funding the study stands to make due to the desired outcome of the study. With self-experimentation, you are “funding” the study with your time (which you could use for other activities) and typically don’t stand to make any money or realize any benefit other than making your own life a little better.
David, that’s an interesting point. Whereas drug companies lose money if the experiment has a certain outcome (the drug doesn’t work), someone who self-experiments benefits with either outcome (the treatment works or it doesn’t work). Because either way the self-experimenter is closer to improving his or her life. You could say the payoff matrix is more balanced.
The CEOs of drug companies should self-experiment with their own drugs and post the results on blogs. No randomized trials, no double-blind, nothing.