After referring to Jane Jacobs (“successful city neighborhoods need a mixture of old and new buildings”), which I liked, Tim Harford wrote this, which I didn’t like:
Many medical treatments (and a few social policies) have been tested by randomized trials. It is hard to imagine a more clear-cut practice of denying treatment to some and giving it to others. Yet such lotteries — proper lotteries, too — are the foundation of much medical progress.
The notion of evidence-based medicine was a step forward in that it recognized that evidence mattered. It was only a small step forward, however, because its valuation of evidence — on a single dimension, with double-blind randomized trials at the top — was naive. Different sorts of decisions need different sorts of evidence, just as Jacobs said different sorts of businesses need different sorts of buildings. In particular, new ideas need cheap tests, just as new businesses need cheap rent. As an idea becomes more plausible, it makes sense to test it in more expensive ways. That is one reason a monoculture of evidence is a poor idea.
Another is that you should learn from the past. Sometimes a placebo effect is plausible; sometimes it isn’t. To ignore this and insist everything should be placebo-controlled is to fail to learn a lot you could have learned.
A third reason a monoculture of evidence is a poor idea is that it ignores mechanistic understandings — understanding of what causes this or that problem. In some cases, you may think that the disorder you are studying has a single cause (e.g., scurvy). In other cases, you may think the problem probably has several causes (e.g., depression, often divided into endogenous and exogenous). In the latter case, it is plausible that a treatment will help only some of those with the problem. So you should design your study and analyze your data taking into account that possibility. You may want to decide for each subject whether or not the treatment helped rather than lump all subjects together. And the “best” designs will be those that best allow you to do this.