Yesterday I posted Robin Hanson’s comments on web trials. My comments on his comments:
1. I think Robin is right that it would be hard to get most people to allow themselves to be randomized. But I also think it doesn’t matter much. The important thing is to improve on existing methods of evaluation. Randomization of subjects to treatments isn’t an end in itself, of course. The goal is to reach the right answer: Learn which treatment works best. I think if you have what might be called a “level playing field” or a “fair comparison” (the various treatment alternatives are presented “equally” — e.g., as equally likely to work, equally attractive, equally high on a list) it will be hard to imagine how the results will be on average worse than nothing. The site can record data about each subject (age, sex, etc.) and the results can be analyzed using those factors — another way to equate subjects across treatments and to help each person decide what would be best for him or her.
2. Excellent point that web trials could be used for evaluation of any advice. Maybe it would be better to start with a non-health problem. Something where the effects are quick and easy to measure.
3. I like the Wikipedia comparison. All-to-all institutions — institutions that help connect everyone to everyone — are ancient and have been very important. Markets and money may have been the first. If I pay Sam $5 for X, and then Sam pays Peter $5 for Y, Peter and Sam have traded X and Y. Money has made this much easier. Democratic institutions allow everyone to govern everyone. Banks allow everyone to loan money to everyone. Books allow everyone to teach everyone. Wikipedia makes all-to-all teaching much easier. Web trials allow everyone to help everyone solve any problem where data would help. As Robin says, Wikipedia suggests that people will participate in all-to-all institutions when there is no obvious reward for doing so.
I would specifically worry about people losing interesting when the thing they try doesn’t work, but reporting back when it does. Perhaps based on the proportion of people who sign up for a particular treatment but don’t report back, you could estimate that for a greater number the treatment wasn’t effective.
And you’d have to deal with confounding factors. What about people who try multiple possible solutions at once? Who confuse the causality? (Maybe that would be noise? But you’d want to watch out for where there could be a bias there.)
You might ask people to list other things about their life, and see if you can find correlations. Maybe remedy X doesn’t work for people with more stress in their lives, or who are taking medication Y, but remedy X works for most other people. I’m not sure what a good way to do this would be.
“I’d worry about people losing interest and not reporting back.” That’s what’s nice about comparing two treatments at the same time — if the reporting rates differed for the two treatments, that would tell you something.
I think you’d want to restrict the data to people who try one treatment at a time.