A New York Times article about error in a risk calculator paints an unflattering picture of doctors:
1. The risk calculator supposedly tells you your risk of a heart attack, to help you decide if you should take statins. It overestimates risk by about 100%. The doctors in charge of it were told about the error a year ago. They failed to fix the problem.
2. The doctors in charge of the risk calculator are having trouble figuring out how to respond. The possibility of a simple retraction seems to not have occurred to them. As one commenter said, “That the researchers, once confronted with the evidence it was faulty, struggled with how to handle the issue is quite telling.”
3. In the comments, a retired doctor thinks the problem of causation of heart disease is very simple:
Statins . . . are only one component in the prevention and treatment of coronary artery disease. Item number one is to have a normal weight. Item two is never smoke. Item three is exercise. Four is to eat an intelligent diet. Five is to remove stress from your life as much as practical. If everybody did these five things (all of which are free), the incidence of coronary artery disease would plummet and many fewer would need statins.
This reminds me of a doctor who told me she knew why people are fat: They eat too much and exercise too little. She was sure.
4. In the comments, a former medical writer writes:
Several years ago, I wrote up, as internal reports, about two dozen transcripts recorded at meetings with local doctors that a major drug company held all around the country. The meetings concerned its statin. Two ideas presented at these meetings by the marketing team, and agreed with by the physician attendees, were: 1) the muscle pain reported by patients was almost never caused by the statin but was the result of excessive gardening, golfing, etc; 2) many children should be prescribed a statin and told that they would have to take the drug for life.
5. Another doctor, in the comments, says something perfectly reasonable, but even her comment makes doctors look bad:
I am a physician and I took statins for 2 years. Within the first 6 months, I developed five new serious medical problems, resulting in thousands of dollars spent on treatments, diagnostic tests, more prescription medications, and lost work. Neither I nor any of my 6 or 7 different specialists thought to suspect the statin as the source of my problems. I finally figured it out on my own. It took 3 more years for me to get back to my baseline state of health. I had been poisoned. I see this all the time now in my practice of dermatology. Elderly patients are on statins and feel lousy, some of whom are also on Alzheimer’s drugs, antidepressants, Neurontin for chronic pain, steroids for fibromyalgia. These poor people have their symptoms written off as “getting older” by their primary physicians, most of whom I imagine are harried but well intentioned, trying to follow guidelines such as these, and so focused on treating the numbers that they fail to see the person sitting in front of them. The new guidelines, with their de-emphasis on cholesterol targets, seem to tacitly acknowledge that cholesterol lowering has little to do with the beneficial actions of statins. The cholesterol hypothesis is dying. If statins “work” by exerting anti-inflammatory benefits, then perhaps we should seek safer alternative ways to accomplish this, without subjecting patients to metabolic derangement.
6. A patient:
My previous doctor saw an ultrasound of my Carotid Artery with a very small buildup and told me I needed to take crestor to make it go away. That was four years ago and I still suffer from some memory loss episodes as a result. The experience was terrible and he’s toast because he denied it could happen.
7. A bystander:
For the past couple of years my job has involved working with academic physicians at a major medical school. After watching them in action — more concerned with personal reputation, funding and internecine politics than with patients — it’s a wonder any of us are limping along. And their Mickey Mouse labs and admin organizations can barely organize the annual staff holiday potluck without confusion and strife. So these botched-up results don’t surprise me at all.
I am not leaving out stuff that makes doctors look good. Maybe this is a biassed picture, maybe not. What I find curious is the wide range of bad behavior. I cannot explain it. Marty Makary argued that doctors behave badly due to lack of accountability but that doesn’t easily explain ignoring a big error when pointed out (#1), an immature response (#2), a simplistic view of heart disease (#3), extraordinary callousness (#4) and so on. In her last book (Dark Age Ahead), Jane Jacobs wrote about failure of learned professions (such as doctors) to police themselves. Again, however, I don’t see why better policing would improve the situation.
Thanks to Alex Chernavsky.
This post reminds me of an outstanding book I read a couple of years ago. It’s called, Mistakes Were Made (But Not by Me): Why We Justify Foolish Beliefs, Bad Decisions, and Hurtful Act, by Caroll Tavris and Elliot Aronson. The book documents the extreme lengths to which people will go in order to avoid admitting that they were wrong.
Seth: I agree, several of the examples can be seen as people not wanting to admit they were wrong. For example, a doctor who prescribed statins doesn’t want to admit that statins are dangerous. Maybe doctors look especially bad because they make a lot more big mistakes than people in other jobs.
It would be nice if we had a free market in the medicine market. These results are to be expected. In a free market a possible cure to these mistakes would be:
If you go in for surgery you get insurance and get paid an agreed upon amount of money if something bad happens. If a doctor has really poor results the insurance company won’t supply or will make the cost of insurance quite high do to the poor results. We will then have price signals that will help us determine which doctors are best and what practices are best.
Right now, under government health care (yes, in the USA we have had government health care for a very long time – when the industry is regulated to the point that people cannot choose anymore then it is government healthcare) there are no true or very little price signals. Which is unfortunate. Many people are suffering needlessly and even dying because of it.
I have long suspected that doctors have a strong tendency to have to be right for a combination of two reasons:
1) history – doctors used to be quite dogmatic and arrogant about their practice of medicine, so those attitudes became baked into the system; and
2) High stakes – since peoples’ health and lives are at stake, it’s hard to be wrong.
Very human and understandable, but not adequate as a way to function these days.
John Banja, assistant director of health science ethics at the Center for Ethics and associate professor of rehabilitation medicine:
Do perfectionist doctors have trouble managing medical errors?
That’s part of the idea. A bigger part, however, is that most health professionals (in fact, most professionals of any ilk) work on cultivating a self that exudes authority, control, knowledge, competence and respectability. It’s the narcissist in us all—we dread appearing stupid or incompetent. The problem, I think, is that health care is so unpredictable and stressful and contains such high stakes, that many practitioners exaggerate their competence. They come to believe that one ought never appear ignorant, uncertain, hesitant or wrong. So when this professional self-image is challenged, these persons are tempted to withdraw, or become hyper-defensive or just plain arrogant.
https://www.emory.edu/EMORY_REPORT/erarchive/2005/February/February7/sandr1.htm
One thing that’s not made clear on any article I’ve read on this subject is that the whole idea of a “risk calculator” may be bogus anyway.
What you’d really like is a list of causes, or of contributors to causes, of heart attacks. What you actually have is a list of positive correlates of heart attacks. That’s a plain different animal.
For example: is their risk calculator consistent with the unexplained rise in heart attack rates during the five decades before 1960 or so, and the unexplained decline thereafter? Not a chance, I’ll bet! In which case, it is not based on causes.
Dearieme,
I agree completely with the notion that a “risk calculator” (based on a probability model at the population level) applied to individuals is inherently bogus. Always. Any “risk” calculation (as much as that means anything) must be conditioned on the individual.
Hey Seth, could you post the code you used to make the programs you use to test your brain function? I’m starting to take flax seeds and want to test my brain to figure out the optimal dose but have no knowledge of writing code. Another thing I would appreciate would be if you could give me a non-computer idea for testing my brain. Thanks so much!
Seth: I once posted the code, it was a complete waste of time. I might share the software with you via Dropbox, as I have done for several people, if you could give me a good reason why.
My mother recently had a physical where the doctor scolded her for not taking the Crestor has been prescribing for her every year. She told him that they gave her migraines and that when she researched its efficacy that it didn’t seem to provide any benefit whatsoever for women. His response? “Your cholesterol is too high.” I asked her what his response was to her complaint of migraines. Nothing. He prescribed it again, and again she is not filling the prescription.
MD here.
I agree with you, roughly. We are, as doctors, neither trained in statistical reasoning nor any kind of critical thinking outside of the diagnose-then-medicate paradigm. With the advent of “guidelines”, things got worse instead of better, because the pharmaceutical industry now has only one attack point instead of convincing every single doctor individually, and physicians feel bound by the guidelines because deviating could be interpreted as malpractice. Every morning I go to the hospital, I know that the medical decision making part of my job could be done much “better” (more adherent to guidelines) by a piece of software. This nonsense system is ultimately frustrating. It also defies everyday experience. Whenever someone is about to die, we stop any medication except for fluids and oxygens, and suddenly: blood pressure becomes normal, pulse normal, blood sugar rock stable. What did we do before that, exactly? I wonder.
I remember seeing a study where some case was presented and the guidelines said “prescribe this”. Something like 95% said they would prescribe it, less than 70% were convinced that it is useful and less than 40% would take it themselves if they had the condition described. That’s what I tell my patients: Always ask what your doctor would do if he was in your place, never “what shall I do”. You’re more likely to get a useful answer. (This idea is a heuristic from Taleb’s Anti-fragile).
And yes, except for some rather rare cases, to me, statins are the ultimate modern-day quackery. What quackery is is pitily decided by majority vote, not by independent science, not by statistics, not by experience, not by observation or even inference. People are so shit-scared by “high cholesterol” that not taking a statin, even only for a couple of weeks, is the same as not taking an antibiotic in a severe infection. “You’re gonna die!!!!!111!!!11!eleven!11″ Erm, what?
I furthermore roughly agree with the other commentators above my comment. They all raise good points.
Seth: Thanks for your comment. It has seemed to me that doctors are trained in “critical thinking” just enough to criticize alternative medicine but not enough to criticize mainstream medicine.
How can you have a “risk calculator” when the cause of the disease is officially unknown?
It’d be interesting to see an attempt to walk back this “risk calculator”‘s basis back through all the dubious correlations to the supposed “data”.
Seth: You can calculate your risk of dying in a plane crash even though the cause(s) of plane crashes are unknown.
“The doctors in charge of the risk calculator are having trouble figuring out how to respond.”
That’s because the purpose of the “risk calculator” is to sell more drugs, and to make it more reality-based would sell less. Of course that results in a puzzle. What to do??
I’ll bet the causes of plane crashes are known rather better than the causes of heart attacks. But suppose that there’s a positive correlation between plane crashes and the proportion of passengers who are men. In the medical world that would be added to the list of risk factors because it’s a positive correlate.
One cannot reasonably calculate their true probability of dying in a plane crash because it depends on nonrandom errors. Again, it’s conditional on the precise plane trip, and it’s most likely either essentially zero or essentially one.
Elduderino, thank you for an enlightening perspective. I have wondered whether most doctors have ever read anything about cholesterol and stains that wasn’t put in their hands by drug companies, through a drug rep or by funding of favorable research.
Mark said:
One cannot reasonably calculate their true probability of dying in a plane crash because it depends on nonrandom errors. Again, it’s conditional on the precise plane trip, and it’s most likely either essentially zero or essentially one.
Fly Air India sometime & see if you still think this is true.
Always remember…
(Most) MD’s aren’t scientists. They aren’t even engineers. They are mechanics, and not very good ones.
Michael Eades has written many times about how most physicians can’t comprehend the typical journal article. Case reports yes, journal articles no.
@Audrey: John Ioannidis has written many times about how most journal articles aren’t worth comprehending.
Seth what did you mean by a good reason? I do not think there is any way it could profit you if that is what you meant. I have been reading through your archives after finding your website a week ago (I have read about half of your posts in that time), and want to find the optimal daily dose of flax seeds for me, and since I do not know how to write code I am having trouble doing it. Any help would be greatly appreciated.
Seth: Yes, that’s what I mean. I have learned that simply giving strangers code is a waste of time.
Hi Seth,
Dr. Michael Eades (Protein Power) blogged yesterday a moving piece about the frustration he dealt with from the medical system concerning his father:
https://www.proteinpower.com/drmike/statins/statin-madness/#more-5428
scroll down to the comments and look at the one from Gregory Porkrywyka
“Statins work and are safe. Period. This has been shown again and again, in men and women”