To my pleasant surprise, Mark Frauenfelder posted this call for volunteers. Will eating half a stick of butter per day or a similar amount of coconut fat improve your performance on arithmetic problems? Eri Gentry is organizing a simple trial to find out. The trial is inspired by my recent Quantified Self talk. Study details.
During the question period of my talk, I responded to a question about a trial with 100 volunteers by saying I would suggest starting with 2 volunteers. A reader has written to ask why.
What’s your reasoning behind suggesting only 2 volunteers to test the eating more butter results? You seem highly convinced earlier in the video, but if you were so convinced why not have a larger trial?
Because the trial will be harder than the people running it expect. If you’re going to make mistakes, make small ones.
This is my first rule of science: Do less. A grad student in English once told me that a little Derrida goes a long way and a lot of Derrida goes a little way. Same with data collection. A little goes a long way and a lot goes a little way. A tiny amount of data collection will teach you more than you expect. A large amount will teach you less.
My entire history of self-experimentation started with a small amount of data collection: An experiment about the effectiveness of an acne medicine. It was far more informative than I expected. My doctor was wrong, I was wrong — and it had been so easy to find out.
This may sound like I am criticizing Eri’s study. I’m not. What’s important is to do something, however flawed, that can tell you something you didn’t know. Maybe that should be the first rule, or the zeroth rule. It has the pleasant and unusual property of being easier than you might think.
Thanks to Carl Willat.
I like rule zero
Thanks for replying to my email. Your comment makes great sense in context. Making the mistakes up front with less effort gets results more quickly and may illuminate ways of improving future studies. There will always be the possibility of doing more rigorous studies later.
Seth, how do you rule out the so called placebo effect?
Human attitude/intention influences the outcome.
IMHO such trials tell the one human that observes himself if his approach worked or if he should try different. And different can start with different attitude.
Btw antibiotics reduced acne for me; at that time I believed doctors and in medicine
I will probably never use anything like that again, though, now that I am less clueless.
I haven’t thought this through, it just came into my mind while reading your blog post.
Be well.
Venkat from rinnonfarm.com had an interesting post apropos a little data versus a lot of data:
https://www.ribbonfarm.com/2010/09/28/learning-from-one-data-point/
The gist IIRC is that when you collect a lot of data, you may stop thinking deeply about a problem, whereas one piece of data tends to force you to think deeply, I think mainly about how cause and effect might act in this particular case.
Do less. I love it. Really, I do.
But, at least this once, I want to have the experience of running a study with as many participants as are willing.
It’s going to be a data mess at first. But I hope that, soon enough, online platforms will be capable of automating this type of study. However, I’ll probalby still spend the majority of my test time on myself.
Thanks so much for your curiosity and testing, Seth. It is quite the inspiration.
Eri
Eri, thanks!
Yes, better software could certainly help. R is not designed for data collection but I use it anyway.
Does anyone have any data, arguments, or guesses about what it is in the butter that has this effect, and what the mechanism might be? (Really enjoyed the talk btw.)
Forrester, here is general information:
https://www.westonaprice.org/
And here is one great blog out of many about how the conventional lipid theory is total bullshit:
https://high-fat-nutrition.blogspot.com/
It’s proven!
“This is my first rule of science: Do less.”
“Same with data collection. A little goes a long way and a lot goes a little way.”
“A tiny amount of data collection will teach you more than you expect.”
“A large amount will teach you less.”
No. Absolutely not. Science hinges on good sampling methods, and the more data, the better. You will eventually exhaust resources in order to obtain this data, this data may be difficult to collect, but a good statistician / scientist would never say “no” to more data. Especially if it is collected in a manner that sheds light on interesting relationships and phenomena, and especially if we want meaningful outcomes. A small amount of data can suggest further study, but these can never be relied upon to make confident and accurate decisions.
majamin, where do these beliefs of yours come from? Science requires a lot more than “good sampling methods”. Every experiment requires assumptions and the bigger the experiment the more assumptions — which have a higher chance of being wrong than you appreciate. A single wrong assumption may cause the whole project to fail. If you’re a scientist and haven’t managed to learn this, I’m surprised.
Hi Seth,
Of course assumptions are counter-productive, and can render any data collection to be fruitless. There are many other things that need to be considered (bias, blinding, etc.). With these in place, you have the beginning of a robust study.
“the bigger the experiment the more assumptions”
No. Every data point that is collected does not mean one more assumption. Bias in sampling can creep in, of course, but that does not mean that less data should be used. It just means that the data if you are collecting data, you need to ensure that it is collected with a good dose of randomness.
The claim made above was “less data is better”. Let’s focus on that. That’s like saying that, in sampling a product line for an estimated number of defected products, the less amount of data you collect, the better. Like I said in my first post, at one point the collection of data would become difficult (i.e. the resources required to test for defects), but how could less data teach you more? Why would you chose to sample less number of products? Can you give a good reason why doing this would result in a better understanding of the situation?
The same with any other observation of our world … how would less observation give us the ability to understand it better? Please point me to a (mathematical, if possible,) analysis that would show that less data you have, the better.
Sincerely,
majamin
majamin, I’m still curious: where do your beliefs come from? They strike me as highly incomplete.
Your puzzlement (“how would less observation give us the ability to understand it better?”) is why I wrote what I did: because it wasn’t obvious. You’re just describing the conventional point of view. I’m well aware of that view. But why do you believe it? To be even a little bit convincing you have to explain that.
Jay Cohen of UCSD Medical School has written a whole book about the idiosyncratic ways in which different patients can react to the same medication. (This is a problem that drug companies would rather sweep under the rug.)
With regard to the butter/reaction-time experiments, for example — is it not possible that your (Seth’s) results are an outlier? In order to generalize the results to the population-at-large, would we not need to replicate the experiment with a larger group of people? Not sure if that’s what Majamin is getting at here, but it’s something that occurred to me.
Seth:
I can see how my response may seem rhetorical, since I am actually echoing the established viewpoint. The reason why I adopt it, is given by the above example of a product line, where one is checking for defects. Logically, I cannot see how one can learn more from a smaller sample. This example illustrates why we should not opt out of collecting more data points. In fact, it can be shown that no useful information can be gained from doing so. We will not be able to tell whether the items in the sample are a fluke, or not, etc. So, I’ve explained my position. It’s time to back up your claim by applying it to the situation that I posed.
Your argument to “less is more”, is that we are susceptible to making more mistakes. What mistakes? Bias? We have techniques to lessen the amount of bias attributed to every sampling method. Do you disagree with these methods? If so, why?
“What mistakes?” Here’s an example. A professor at a famous university got a large grant to do a big study of how cleaning your apartment affects your chances of asthma. Most of the grant money was wasted because his research group couldn’t get enough people to volunteer for the experiment.
Alex, you ask if my results “could be an outlier”. The history of nutrition is a good guide. There are no examples in the history of nutrition where one person, apparently normal, needed a nutrient that no one else needed. There are countless examples where the results from one person turned out to generalize to everyone else.
Seth:
You still haven’t spoken to the example I gave, and how a smaller sample (“less data”) would be more beneficial. Again, product line defects: how would checking a smaller amount lead to less mistakes?
Seth:
In the large grant case, all this shows is that the study’s funds were used ineffectively, is says nothing about the efficacy of having smaller sample sizes. All we can say is that (1) a bunch of money was wasted, and (2) there wasn’t enough data to perform a meaningful analysis. If you are arguing this particular university did not issue funding in an efficient manner, I wholeheartedly agree with you. But, what does this have to do with the efficacy of small sample sizes?
If they had tried to do less — smaller sample size — they would have wasted far less money learning that one of their assumptions was wrong.
Seth:
I’m not sure if you keep missing the question I posed above, or are ignoring it. There’s no point in this conversation if you won’t even bother to apply your approach to that scenario.
I am ignoring it. I’m not interested in hypothetical examples. I was trying to find out if you had real-world experience that contradicted what I wrote. The answer seems to be no.
Seth:
That’s fine. I’m not interested in hypothetical theories.