A few weeks ago I blogged about a leukemia doctor’s disapproval of self-experimentation (“you won’t learn anything and others won’t learn from it, either”). What I wrote was reposted at The Health Care Blog, where it elicited this comment (by “rbar”):
Sigh. Mr Roberts did it again, he simply does not (want to) understand that anecdotal evidence is of little value (let me give you an example: I self experiment with traffic signals; I noted that I can considerable cut down on travel times when ignoring red lights and stop signs; there are no drawbacks whatsoever, no one get hurts, and even my gas mileage/carbon footprint got better) .
Individuals who have similar questions as Mr. Roberts should look up the following key words, because they may understand why controlled studies are far superior to anecdotal evidence:
-placebo effect
-regression to the mean
-misattribution error [apparently rbar means error in determining the cause of a change]-self limited conditions/natural fluctuation of chronic conditions
-and in terms of drawbacks of experimentation: primum non nocere, and also the fact that anecdotal evidence adds relatively little to humanity’s knowledge baseDoes all that mean that patients should not be well informed, active and making suggestions to their treating physicians? Of course absolutely not. Being knowledgeable about one’s condition is different from self experimentation. Is that intellectually challenging?
One reply to this comment said we should be aggregating data across patients. “I believe Mr. Roberts is alluding to the power of aggregating real-world data across patients to generate insights into what may and may not work, not to giving undue weight to any single anecdotal case.” No, I was looking at it from the point of view of the self-experimenting patient. If you have a health problem, and you can measure it often (daily, weekly) you can find out what works faster than your doctor — often much faster. You can test many more possible solutions. This is what Richard Bernstein taught the whole world of diabetes, starting in the 1960s, when he pioneered home blood glucose testing. Apparently rbar also objects to that.
Rbar’s comment is dismissive (“Sigh”, “Is that intellectually challenging?”) and partly obscure (“ignoring stop signs and stoplights” — huh?). Because patients who self-experiment may make “misattribution errors” they shouldn’t self-experiment? That’s like saying because people may make reasoning errors they shouldn’t reason.
The true meaning of rbar’s comment may be hidden in his statement that it’s okay for patients to “make suggestions to their treating physician.” Which shows who he thinks should be boss in the doctor-patient relationship. When a patient self-experiments, the doctor is no longer boss. Maybe rbar is a doctor. Maybe he feels threatened by self-experimentation. If so, I hope he’s right.
More A later reply to rbar put it well: ” Your list of possible pitfalls . . . is similar to lists I remember seeing back in graduate school in various research handbooks. I do not see how you go from the fact that these effects and errors are possible to the conclusion that the whole endeavor isn’t worthwhile.”
Its very problematic.
Everyone self-experiments in different areas of their life. In my time, I have seen too many people make false attribution errors through wishful thinking and other biases to not realize that while it can be useful, self-experimentation has to be done with extreme caution and its conclusions treated with extreme care. I myself have gone through long periods of my life completely convinced that certain factors were the culprits in certain situations, only to later discover to my astonishment how wrong I was, and how my biases have led me to overlook important conflicting data.
The problem is, your biases and assumptions can often literally not let you SEE relevant data. Its why *peer review* is so important.
This has made me extremely careful these days. I now believe that self-experimentation has to be done with the utmost of reservations and care – no conclusions can be treated as anything more than extremely provisional – and that it really is of limited use to people at large.
I think, really, self-experimentation has more value in areas where we can be reasonably sure that there a few factors, and that we can isolate those factors. Many situations we must admit are just too complex for that and are just useless for self-experimentation, even if what we do seemingly *works* (there are too many factors for us to know WHAT exactly worked. This does not mean we should stop DOING what works until we understand WHAT is working, only that we must be aware that our explanations for WHAT is working and WHY might be completely off base)
Also, we cannot expect others to treat our results as anything other than invitations to thought.
“stoplights and stopsigns” struck me the most relevant point rbar made. If the condition you’re trying to treat is “my commute takes too long” and you self-experiment with ignoring stoplights and record the outcomes, the data will tell you that is perfectly safe and gets you where you’re going faster…right up until the day you die in a horrible crash. If we apply that metaphor to self-experimentation, his fear is that that somebody just trying random treatments will find a false local maximum – a treatment that *seems* to help in the short run but has a worse long-term prognosis than doing nothing, due to a serious secondary effect that takes longer to develop or has some semi-random component.
Applied to conventional drug trials (eg: thalidomide), it’s a valid concern. But applied to the sort of treatments you’re usually exploring/recommending, not so much. He’s basically applying a precautionary principle (if it could cause harm, don’t do it), whereas you’re applying a proactionary principle (if it’s broken, try to fix it). Your rule makes more sense for patients trying to get better; his rule makes more sense for doctors trying not to get sued.
Let me respond to this guy
1) there are negatives to self experimentation. Likewise, there are negatives to talk with a doctor. Everything has negatives. Of only relevance is the sum of the effects.
Indeed, one got to be smart. Always. Albit self experimentation requires a little more than aferage wisdom.
2) placebo effects are over rated. It is a cultural prejudice that placebo effects are “nothing” they are very real. And anything that bring about placebo improvements is highly welcome. If self experimentation is an effective method to induce placebo healing, it is a great reason to embrace it.
Besides, when someone tries many things and only some work, the odds for placebo effects are quite lower.
3) one MUST look for the statistical significance of self experimentation results.
4) one can only look for the statistical expected value for contribution to the world knowledge. It is childish to ask whether a specific finding will contribute. The question is whether it has some chance to contribute. And it has.
Every self experimenter has a tiny chance to add for new knowledge. But it adds up.
Besides, the average doctor adds zero to overal knowledge.
And since when do patients live for adding for the world’s knowledge? How ludicrous it is to even mention this to someone with a condition! People try things to get healthy.
Two thoughts: Barry Marshall got a Nobel Prize in Medicine for self-experimentation. Any individidual who says “anecdote is not data” doesn’t understand how science is done.
Second thought, if you do the experiment that Glen Raphael described, you might also (tickets aside) come to the conclusion that stop lights are a waste of your time, and actually increase the accident rate. The town of Makkinga in the Netherlands ditched all their street signs and stop lights, and the accident rate went to zero. Other towns are now following their example…
So even the type of self experimentation that Rbar would have us not do yields useful results.
What does not yield useful results in science is a slavish adherence to precedent. Although that is the way to get through medical school…
“European Cities Do Away with Traffic Signs”
https://www.spiegel.de/international/spiegel/0,1518,448747,00.html
+1 to Glen Raphael’s comment (I especially like the last sentence).
Two more of my off-the-wall opinions, about placebos and what I will call “The Hundredth Patient Effect:”
1/ PLACEBOS: In my opinion, the reason that placebos sometimes look good relative to BigMedicine is not that placebos work. Rather, BigMedicine often doesn’t work, so there’s not much difference. But sometimes, BigMedicine actually kills the patient.
2. MEDICAL STATISTICS: I think medical statistics conceal a dirty secret of modern medicine, which is that the statistics are applied for the benefit of the doctors, and very much to the detriment of the patient’s health. The doctors are gambling with the patients, and the patients are losing.
After my bad medical experiences in 2010, when I refused a leg amputation and quadruple coronary artery bypasses (and both conditions got better with simple cheap lifestyle changes), I have often asked myself why doctors could have made such bad recommendations. Let me try to explain my reasoning:
Suppose that, over a period of time, a doctor considers whether to perform a major and expensive (profitable) procedure on 100 patients. In the past, I would have thought that the profit motive would have been an incentive for the doctor to operate. I still think that is a factor, but I now think there is another, more important factor: statistics.
If the doctor amputates 100 ailing legs, and all the patients live, the doctor make a lot of money, and he is praised for saving 100 lives.
BUT WAIT A MINUTE THERE!
Based on my experience, the body is capable of healing very sick legs and other body parts. Maybe, if no amputations had been done, one, or 10, or 50, or even 99 of those 100 patients would have recovered fully and kept their legs. So why amputate 100 legs?
Because of that hundredth patient, the one that dies. If 100 patients are not operated on, but one dies, that one death will almost certainly result in a major malpractice lawsuit and perhaps even the end of the doctor’s career. So, the doctor’s options are:
(a) amputate and make a lot of money, and be credited with saving 100 lives, or
(n) don’t amputate, and save 99 legs (and 99 lives), but get one major lawsuit and very bad publicity.
Unfortunately, I think there may be strong pressure for the doctor to take the safe (for him) choice and amputate 100 legs just to avoid one messy lawsuit.
I call my theory “The Hundredth Patient Effect.”
Glen, thanks for explaining that. This happens with modern clinical trials, too — FDA-approved drugs are found to have bad effects later and are taken off the market. So it is unclear that self-experimentation is more dangerous. As you say, I test stuff that has been safely engaged in (such as looking at faces in the morning) for very long times.
I keep coming back to the concept of risk management. I think these doctors have heard too many stories of patients who died from a burst appendix because the patient had received herbal treatment from some quack who thought he knew Chinese medicine. For high-risk situations, these physicians are right. (The TV show ‘Doc Martin’ illustrates several such situations in an entertaining fashion.)
And yet if a patient goes to a physician who assesses the condition and says, sorry, it’s a low-risk issue, nothing I can do, live with it, these things sometimes clear up over time . . .
In those situations, where the physician has already assessed it as low-risk, then why shouldn’t the patient experiment? Suppose it’s a sleep issue. What’s the risk with experimenting with light levels at night, timing of Vitamin D, drinking peppermint tea after supper, or standing on one leg?
I also argue that these ‘obey the physician’ advocates do no such thing in the case of their own diet. No matter which diet they follow, the one designed by a physician who clinically treats for obesity, I bet not one of these advocates strictly follows that specific diet. Which is fine. The risk is low.
Jordan, care to give examples to support these conclusions? My experience has led me to opposite conclusions. For example, I can point to many people who have used self-experimentation successfully. For me, the interesting comparison is not self-experimentation versus nothing — I think it’s obvious that some data is better than none. (And, of course, sometimes data is misleading.) The interesting comparison is self-experimentation versus expert advice (e.g., doctor’s advice). What has been your experience comparing the two?
“Self experimentation versus expert advice.” To repeat something posted on this site elsewhere:
I suffered a great deal of clinical and sub-clinical depression over the years. Between 1975 and 1996, I went through 4 antidepressants, was told that I needed to return to the considerable trauma I felt as a child and relive the experiences and release a lot of repressed emotion. I saw 3 clinical psychologists and 3 psychiatrists. I was a night person during this time staying up to anywhere from 2-4 a.m. on average.
No mental health professional told me to start going to bed and getting up earlier as a way to treat my depression, even though I made it very clear that I was a night person. Seth Roberts strongly impressed upon me the notion that going to bed at 3:30 a.m. was my biggest problem.
When I finally changed my circadian rhythms in December 2003, I felt a quick mood uplift, about 20 point rise on a scale of 100 in terms of baseline mood, I estimated. With baseline mood improved, I started enjoying music enormously and life became more fun.
I agree with earlier posts that self experimentation requires a certain capacity for objectivity, disciplined observation, and engaged reflection and some people probably are not able to do it effectively. But lots of people are, if they are willing to do some work, and it has helped me in many ways.
Experts have an incentive to overstate their knowledge.
All the RCTs in the world won’t tell me whether I can tolerate beta-blockers: only self-experimentation can do that.
Jimpidry,
Doctors have good reason not to plan on lifestyle changes.
Most patients do not adhere to large lifestyle changes.
Even if a doctor is sure that a list of exercises,food etc. will heal the foot, he may judiciously recommend to amputate it, if the chances a patient will do the complex changes is neAr zero. (assuming the risk from doing nothing is high)
It may sound undemocratic etc. but every suggestion you do is related to what are the chances of it being used.
I believe that there is still much research in medicine. Should not be making diagnoses closed. You can always have hope.
Hi Seth,
I’ve been reading for a while. I’ve had delayed sleep onset my whole life, and seem to have pretty much fixed it reading this blog. I am still varying the factors but what seems most important is:
1) 5000 IU liquid D3 first thing on waking
2) Blue Blocker Glasses
3) Using the xiser (xiser.com) for what I am seeing as a combination of one legged standing to exhaustion and anaerobic exercise
Don’t seem to have much effect:
1) 10,000 Lux lamp
2) Niacin to flush
3) Magnesium
4) Cold shower before bed
My sleep onset insomnia was terrible, with me often unable to sleep before 3-5am, and now I’ve been waking between 8am and 9am feeling refreshed. My wakeup time seems to be inching back slowly. Thanks for the help your blog gave me!
Best,
Brian
What does this doctor do every single day if not experiments of n=1 on his patients? Doesn’t this doctor try and assess the efficacy of the treatments that they dole out? Doesn’t this doctor change the drug that he prescribes to his patients if they don’t seem to work, even if the clinical data says that the drug is 99% effective?
And that’s why Dr Richard Bernstein – former engineer- went on to train as a physician. He continues to amass data from masses of grateful patients. Does that squash the sceptics? Nope. ‘No double blind trials… blah de blah de blah’