I recently asked Robin Hanson, a professor at George Mason University, what he thought of web trials. Web trials are a way to learn how to solve difficult health problems (e.g., acne, obesity). By web trial I mean a web-based collection of data that compares different ways of solving a problem. People with the problem would go to a website, sign up for one of the treatments, follow the directions, and report the results in a standardized format. For example, a site might compare three acne treatments (treatments that anyone can try, such as dietary changes or over-the-counter medicines). The cumulated results would gradually show which treatment works best — a thousand times more efficiently (sooner, cheaper, more easily) than a clinical trial (which no one would finance because there is no profit to be made). Web trials are halfway between clinical trials and the data collection now going on at the Shangri-La Diet (SLD) forums, where people post their progress on SLD.
I asked Robin because he has pioneered a similar improvement: Prediction markets are often far better than what they replace. And his core political affiliation is “I don’t know.”
Here is a summary of what he said.
1. A selection effect is a big concern. Do people wait to report back until after it works? There is always going to be the issue of sampling, selection bias for people who stay with it.
2. How could you get people to allow you (the website) to choose for them which treatment to do? That would be the hard thing. Perhaps the website could say: “would you like to see what our advice for you is?” At most you could get randomization for your advice.
3. It doesn’t have to be restricted to health problems. It could be used to test all sorts of advice. You could just get data about what happens when people do or don’t follow some advice — romantic advice, for example. Very rarely do we have randomization in choices. When we do, we call them natural experiments. In medicine, researchers have used practice variation (variation from one doctor to the next) to look at effectiveness.
4. Perhaps you could get people to commit to this the way they do to Wikipedia. The goal would be: Let’s understand humanity — a noble cause. Let’s be part of a grand project to do this.
Robin blogs at Overcoming Bias. Tomorrow I will comment on Robin’s comments.