At the Quantified Self blog, in response to a video of me talking about QS and the Ancestral Health Symposium (paleo), someone named Colin made the following comment:
Very interesting talk. I am just curious how someone can claim a study conducted with a sample size of one is “100 times better” than someone else’s study. I do not know anything about the other study mentioned, but I do know that a study based on n=1 cannot be considered scientific proof. And sure, he hears from people who have lost weight drinking the sugar water he prescribed, but it is quite possible there are 100 times as many people who didn’t email him because they didn’t see any positive results and decided to try something else. I think the QS stuff is very interesting and helpful on a personal level, but it seems like a stretch to generalize your results to others.
I responded:
I have two responses.
1. Sample size isn’t everything. Sure, a study with n=1 isn’t “scientific proof”. Nor is any other study, in my experience. “Scientific proof” has always required many studies. New scientific ideas have very often started with n = 1 experiments or observations. Later, larger experiments or observations were done. Both — the initial n=1 observation and the later n = many observations — were necessary for the new idea to be discovered and confirmed.
2. The history of biology teaches there are few exceptions to general rules. See any biology textbook. For example, a textbook might say “lymphocytes fight germs”. This means no serious exceptions have ever been found to that rule. So, as matter of biological history, the person who managed to figure out what one particular lymphocyte does turned out to have figured out what they all do. Biology textbooks have thousands of statements like “lymphocytes fight infection” meaning that this sequence of events (you can generalize from one to all, or nearly all) has happened thousands of times. There is no shadow hidden history of biology that teaches otherwise.
I think the first response is much more convincing than the second, and is really all that needs to be said on the topic. You might also explain that n=1 is much better than n=0, and that many of the things QS folks are studying simply wouldn’t be studied at all if they weren’t doing it on their own!
The second response seems shakier to me. The person who figured out what lymphocytes do must have done so with a series of observations that weren’t so conclusive, and any one of them might have been NON-generalizable. only once they were shown to be generalizable after all could they be used to support the conclusion that lymphocytes fight germs, for ex.
I think “n=1″ self-experimentation is entirely consistent with a Bayesian framework, which is all that is needed to claim the approach is scientific.
A pet peeve along those lines: too many people get away with the phrase “scientific proof.” Science is inductive inference etc etc
Also, I thought you would respond by mentioning that 1 subject is different from 1 observation. Hundreds of observations can be observed from one subject and still be “scientific”. No neuroscientist complains that H.M. was just “one” subject.
Marshall got a Nobel for N=1…
limits to n=1
1) it is expected that very many changes will work specifically to specific persons.
Hence, much Of the promise of self experimentation is exactly is that.
For some people eating carbs for example makes them healthier (with limitations), for reasons related to complexity and individuality.
There is a lot of situational
I agree that many n=1 things will replicate. But I think that many are inherently individualistic.
2) there is nothing to be ashamed about individualisitc effects. The generalization is that for anyone there are some things that can be found. Ad that lack of evidence in others doe snot imply it will not work for this person…..
What is scientific (or not) is the method. Proofs can be obtained by means of the scientific method on a n=1 or n=100 basis. It makes no difference.
3) you can get statistically significant results with n=1. Ayne who does not get this is an imbecile.
There are limits. But the limits are limited.
3.1) generality. Indeed, the formal claim of an n= 1 result is local. Tis is what is verified. Yet, like any scientific result it may be true for other similar cases…..
Other limit not now on my mind.
Note the different between my tone and the “non scientific” nonsense.
There is a confusion between “instruction for making the erfect experiment” and “scientific”.
There are endless instructions students get, most of them irelevant to study validity. Maybe 20% of the instructions are relevant to validity (i.e. blinidng, deciding the measures in advance, etc.) while most (i.e. ethics commitee, balancing for sex and hair color etc.) are nicieties.
What happens is that students get entrapped in the whole instruciton set.
1) they lose the sense of what and why is scientifically important.
2) experiments are perceived and taken as much harder than they should scientifically be
Interesting, of course the hard part about experiments, research, studies and trials owhere N=1 is doing it all with the blindfold on.