Category: evidence-based medicine

Assorted Links

Seth Roberts5 Comments
  • Top ten excuses for climate scientists behaving badly. For example, “the emails are old” and “the timing is suspicious”.
  • Scientific retractions are increasing. My guess is that retractions are increasing because scientific work has become easier to check. Tools are cheaper, for example.
  • More Dutch scientific misconduct. “Professor Poldermans published more than 600 scientific papers in a wide range of journals, including JAMA and the New England Journal of Medicine.”
  • The next time someone praises “evidence-based medicine”, ask them: What about Accutane? It illustrates how evidence-based medicine encourages dangerous drugs. You can’t make lots of money from cheap, time-tested things that we know to be safe (such as dietary changes) so the drug industry revolves around things that are not time-tested and therefore dangerous — far more dangerous than dietary changes. Evidence-based medicine, which says that certain tests (expensive) are much better than other tests (cheap), provides cover for this. Because the required tests are so expensive, they are allowed to be short.

Thanks to Allan Jackson.

Assorted Links

Seth Roberts4 Comments
  • Salem Comes to the National Institutes of Health. Dr. Herbert Needleman is harassed by the lead industry, with the help of two psychology professors.
  • Climate scientists “perpetuating rubbish”.
  • A humorous article in the BMJ that describes evidence-based medicine (EBM) as a religion. “Despite repeated denials by the high priests of EBM that they have founded a new religion, our report provides irrefutable proof that EBM is, indeed, a full-blown religious movement.” The article points out one unquestionable benefit of EBM — that some believers “demand that [the drug] industry divulge all of its secret evidence, instead of publishing only the evidence that favours its products.” Of course, you need not believe in EBM to want that. One of the responses to the article makes two of the criticisms of EBM I make: 1. Where is the evidence that EBM helps? 2. EBM stifles innovation.
  • What really happened to Dominique Strauss-Kahn? Great journalism by Edward Jay Epstein. This piece, like much of Epstein’s work, sheds a very harsh light on American mainstream media. They were made fools of by enemies of Strauss-Kahn. Epstein is a freelance journalist. He uncovered something enormously important that all major media outlets — NY Times, Washington Post, The New Yorker, ABC, NBC, CBS (which includes 60 Minutes), the AP, not to mention French news organizations, all with great resources — missed.

Evidence-Based Medicine Versus Innovation

Seth Roberts6 Comments

In this interview, a doctor who does research on biofilms named Randall Wolcott makes the same point I made about Testing Treatments — that evidence-based medicine, as now practiced, suppresses innovation:

I take it you [meaning the interviewer] are familiar with evidence-based medicine? It’s the increasingly accepted approach for making clinical decisions about how to treat a patient. Basically, doctors are trained to make a decision based on the most current evidence derived from research. But what such thinking boils down to [in practice — theory is different] is that I am supposed to do the same thing that has always been done – to treat my patient in the conventional manner – just because it’s become the most popular approach. However, when it comes to chronic wound biofilms, we are in the midst of a crisis – what has been done and is accepted as the standard treatment doesn’t work and doesn’t meet the needs of the patient.

Thus, evidence-based medicine totally regulates against innovation. Essentially doctors suffer if they step away from mainstream thinking. Sure, there are charlatans out there who are trying to sell us treatments that don’t work, but there are many good therapies that are not used because they are unconventional. It is only by considering new treatment options that we can progress.

Right on. He goes on to say that he is unwilling to do a double-blind clinical trial in which some patients do not receive his new therapy because “we know we’ve got the methods to save most of their limbs” from amputation.

Almost all scientific and intellectual history (and much serious journalism) is about how things begin. How ideas began and spread, how inventions are invented. If you write about Steve Jobs, for example, that’s your real subject. How things fail to begin — how good ideas are killed off — is at least as important, but much harder to write about. This is why Tyler Cowen’s The Great Stagnation is such an important book. It says nothing about the killing-off processes, but at least it describes the stagnation they have caused. Stagnation should scare us. As Jane Jacobs often said, if it lasts long enough, it causes collapse.

Thanks to Heidi.

Testing Treatments: The Authors Respond

Seth Roberts2 Comments

In a previous post I criticized the book Testing Treatments. Two of the authors, Paul Glasziou and Iain Chalmers, have responded. I have replied to their response. They did not respond to the main point of my post, which is that the preferences and values of their book — called evidence-based medicine — hinder innovation.

Sure, care about evidence. Of course. But don’t be an evidence snob.

Testing Treatments: Nine Questions For the Authors

Seth Roberts14 Comments

From this comment (thanks, Elizabeth Molin) I learned of a British book called Testing Treatments (pdf), whose second edition has just come out. Its goal is to make readers more sophisticated consumers of medical research. To help them distinguish “good” science from “bad” science. Ben Goldacre, the Bad Science columnist, fulsomely praises it (“I genuinely, truly, cannot recommend this awesome book highly enough for its clarity, depth, and humanity”). He wrote a foreword. The main text is by Imogen Evans (medical journalist), Hazel Thornton (writer), Iain Chalmers (medical researcher), and Paul Glaziou (medical researcher, editor of Journal of Evidence-Based Medicine).

To me, as I’ve said, medical research is almost entirely bad. Almost all medical researchers accept two remarkable rules: (a) first, let them get sick and (b) no cheap remedies. These rules severely limit what is studied. In terms of useful progress, the price of these limits has been enormous: near total enfeeblement. For many years the Nobel Prize in Medicine has documented the continuing failure of medical researchers all over the world to make significant progress on all major health problems, including depression, heart disease, obesity, cancer, diabetes, stroke, and so on. It is consistent with their level of understanding that some people associated with medicine would write a book about how to do something (good science) the whole field manifestly can’t do. Testing Treatments isn’t just a fat person writing a book about how to lose weight, it’s the author failing to notice he’s fat.

In case the lesson of the Nobel Prizes isn’t clear, here are some questions for the authors:

1. Why no chapter on prevention research? To fail to discuss prevention, which should be at least half of health care, at length is like writing a book using only half the letters of the alphabet. The authors appear unaware they have done so.

2. Why are practically all common medical treatments expensive?

3. Why should some data be ignored (“clear rules are followed, describing where to look for evidence, what evidence can be included”)? The “systematic reviews” that Goldacre praises here (p. 12) may ignore 95% of available data.

4. The book says: “Patients with life-threatening conditions can be desperate to try anything, including untested ‘treatments’. But it is far better for them to consider enrolling in a suitable clinical trial in which a new treatment is being compared with the current best treatment.” Really? Perhaps an ancient treatment (to authors, untested) would be better. Why are there never clinical trials that compare current treatments (e.g., drugs) to ancient treatments? The ancient treatments, unlike the current ones, have passed the test of time. (The authors appear unaware of this test.) Why is the comparison always one relatively new treatment versus another even newer treatment?

5. Why does all the research you discuss center on reducing symptoms rather than discovering underlying causes? Isn’t the latter vastly more helpful than the former?

6. In a discussion of how to treat arthritis (pp. 170-172), why no mention of omega-3? Many people (with good reason, including this) consider omega-3 anti-inflammatory. Isn’t inflammation a major source of disease?

7. Why is there nothing about how to make your immune system work better? Why is this topic absent from the examples? The immune system is mentioned only once (“Bacterial infections, such as pneumonia, which are associated with the children’s weakened immune system, are a common cause of death [in children with AIDS]“).

8. Care to defend what you say about “ghostwriting” (where med school professors are the stated authors of papers they didn’t write)? You say ghostwriting is when “a professional writer writes text that is officially credited to someone else” (p. 124). Officially credited? Please explain. You also say “ghostwritten material appears in academic publications too – and with potentially worrying consequences” (p. 124). Potentially worrying consequences? You’re not sure?

9. Have you ever discovered a useful treatment? No such discoveries are described in “About the Authors” nor does the main text contain examples. If not, why do you think you know how? If you’re just repeating what others have said, why do you think your teachers are capable of useful discovery? The authors dedicate the book to someone “who encouraged us repeatedly to challenge authority.” Did you ever ask your teachers for evidence that evidence-based medicine is an improvement?

The sad irony of Testing Treatments is that it glorifies evidence-based medicine. According to that line of thinking, doctors should ask for evidence of effectiveness. They should not simply prescribe the conventional treatment. In a meta sense, the authors of Testing Treatments have made exactly the mistake that evidence-based medicine was supposed to fix: Failure to look at evidence. They have failed to see abundant evidence (e.g., the Nobel Prizes) that, better or not, evidence-based medicine is little use.

Above all, the authors of Testing Treatments and the architects of evidence-based medicine have failed to ask: How do new ideas begin? How can we encourage them? Healthy science is more than hypothesis testing; it includes hypothesis generation — and therefore includes methods for doing so. What are those methods? By denigrating and ignoring and telling others to ignore what they call “low-quality evidence” (e.g., case studies), the architects of evidence-based medicine have stifled the growth of new ideas. Ordinary doctors cannot do double-blind clinical trials. Yet they can gather data. They can write case reports. They can do n=1 experiments. They can do n=8 experiments (“case series”). There are millions of ordinary doctors, some very smart and creative (e.g., Jack Kruse). They are potentially a great source of new ideas about how to improve health. By denigrating what ordinary doctors can do (the evidence they can collect) — not to mention what the rest of us can do — and by failing to understand innovation, the architects of evidence-based medicine have made a bad situation (the two rules I mentioned earlier) even worse. They have further reduced the ability of the whole field to innovate, to find practical solutions to common problems.

Evidence-based medicine is religion-like in its emphasis on hierarchy (grades of evidence) and rule-following. In the design of religions, these features made sense (to the designers). You want unquestioning obedience (followers must not question leaders) and you want the focus to be on procedure (rules and rituals) rather than concrete results. Like many religions, evidence-based medicine draws lines (on this side “good”, on that side “bad”) where no lines actually exist. Such line-drawing helps religious leaders because it allows their followers to feel superior to someone (to people outside their religion). When it comes to science, however, these features make things worse. Good ideas can come from anybody, high or low in the hierarchy, on either side of any line. And every scientist comes to realize, if they didn’t already know, that you can’t do good science simply by following rules. It is harder than that. You have to pay close attention to what happens and be flexible. Evidence-based medicine is the opposite of flexible. “ There is considerable intellectual tyranny in the name of science,” said Richard Feynman.

Testing Treatments has plenty of stories. Here I agree with the authors — good stories. It’s the rest of the book that shows their misunderstanding. I would replace the book’s many pages of advice and sermonizing with a few simple words: Ask your doctor for the evidence behind their treatment recommendation. He or she may not want to tell you. Insist. Don’t settle for vague banalities (“It’s good to catch these things early”). Don’t worry about being “difficult”. You won’t find this advice anywhere in Testing Treatments. If I wanted to help patients, I would find out what happens when it is followed.

More Two of the authors respond in the comments. And I comment on their response.

Monocultures of Evidence

Seth RobertsLeave a comment

After referring to Jane Jacobs (“successful city neighborhoods need a mixture of old and new buildings”), which I liked, Tim Harford wrote this, which I didn’t like:

Many medical treatments (and a few social policies) have been tested by randomized trials. It is hard to imagine a more clear-cut practice of denying treatment to some and giving it to others. Yet such lotteries — proper lotteries, too — are the foundation of much medical progress.

The notion of evidence-based medicine was a step forward in that it recognized that evidence mattered. It was only a small step forward, however, because its valuation of evidence — on a single dimension, with double-blind randomized trials at the top — was naive. Different sorts of decisions need different sorts of evidence, just as Jacobs said different sorts of businesses need different sorts of buildings. In particular, new ideas need cheap tests, just as new businesses need cheap rent. As an idea becomes more plausible, it makes sense to test it in more expensive ways. That is one reason a monoculture of evidence is a poor idea.

Another is that you should learn from the past. Sometimes a placebo effect is plausible; sometimes it isn’t. To ignore this and insist everything should be placebo-controlled is to fail to learn a lot you could have learned.

A third reason a monoculture of evidence is a poor idea is that it ignores mechanistic understandings — understanding of what causes this or that problem. In some cases, you may think that the disorder you are studying has a single cause (e.g., scurvy). In other cases, you may think the problem probably has several causes (e.g., depression, often divided into endogenous and exogenous). In the latter case, it is plausible that a treatment will help only some of those with the problem. So you should design your study and analyze your data taking into account that possibility. You may want to decide for each subject whether or not the treatment helped rather than lump all subjects together. And the “best” designs will be those that best allow you to do this.

The Problem with Evidence-Based Medicine

Seth Roberts11 Comments

In a recent post I said that med school professors cared about process (doing things a “correct” way) rather than result (doing things in a way that produces the best possible outcomes). Feynman called this sort of thing “ cargo-cult science“. The problem is that there is little reason to think the med-school profs’ “correct” way (evidence-based medicine) works better than the “wrong” way it replaced (reliance on clinical experience) and considerable reason to think it isn’t obvious which way is better.

After I wrote the previous post, I came across an example of the thinking I criticized. On bloggingheads.tv, during a conversation between Peter Lipson (a practicing doctor) and Isis The Scientist (a “physiologist at a major research university” who blogs at ScienceBlogs), Isis said this:

I had an experience a couple days ago with a clinician that was very valuable. He said to me, “In my experience this is the phenomenon that we see after this happens.” And I said, “Really? I never thought of that as a possibility but that totally fits in the scheme of my model.” On the one hand I’ve accepted his experience as evidence. On the other hand I’ve totally written it off as bullshit because there isn’t a p value attached to it.

Isis doesn’t understand that this “ p value” she wants so much comes with a sensitivity filter attached. It is not neutral. To get it you do extensive calculations. The end result (the p value) is more sensitive to some treatment effects than others in the sense that some treatment effects will generate smaller (better) p values than other treatment effects of the same strength, just as our ears are more sensitive to some frequencies than others.

Our ears are most sensitive around the frequency of voices. They do a good job of detecting what we want to detect. What neither Isis nor any other evidence-based-medicine proponent knows is whether the particular filter they endorse is sensitive to the treatment effects that actually exist. It’s entirely possible and even plausible that the filter that they believe in is insensitive to actual treatment effects. They may be listening at the wrong frequency, in other words. The useful information may be at a different frequency.

The usual statistics (mean, etc.) are most sensitive to treatment effects that change each person in the population by the same amount. They are much less sensitive to treatment effects that change only a small fraction of the population. In contrast, the “clinical judgment” that Isis and other evidence-based-medicine advocates deride is highly sensitive to treatments that change only a small fraction of the population — what some call anecdotal evidence. Evidence-based medicine is presented as science replacing nonsense but in fact it is one filter replacing another.

I suspect that actual treatment effects have a power-law distribution (a few helped a lot, a large fraction helped little or not at all) and that a filter resembling “clinical judgment” does a better job with such distributions. But that remains to be seen. My point here is just that it is an empirical question which filter works best. An empirical question that hasn’t been answered.

How to Base Medicine on Evidence

Seth RobertsLeave a comment

The thing to notice about what the New York Times calls “ the evidence-based medicine practiced at Intermountain hospital” is how different it is than the movement called evidence-based medicine. The Intermountain stuff, above all, is not black-and-white thinking. It is a good example of what the opposite looks like. The rules aren’t simple, they are complex, and not fixed. It is what engineers in other areas have been doing since Deming.

So many scientists — not to mention everyone else — are completely paralyzed, rendered completely useless, by their black-and-white thinking. It feels good to them — they love the certainty of it, and the power it gives them to look down on others — and they never quite realize what it has done to them. The notion of using evidence to improve health care made perfect sense — until black-and-white thinkers got a hold of it.

Any class in scientific method should be at least half about avoiding black-and-white thinking. They never are.

Evidence-Based Medicine

Seth Roberts5 Comments

In the comments, Bruce Charlton writes:

The failure to fund trials is combined with a suffocating dominance of the perspective of self-styled ’evidence-based medicine’ (EBM) – including the groundless notion that only mega-trails should be taken seriously. . . Since the vast majority of randomized trials are industry funded, EBM has meant that industry has a de facto monopoly on ’reputable’ therapeutic knowledge.

Delivering us into the hands of Big Pharma was not – of course – intended by the socialistic founders of EBM, but it has happened nonetheless.

This reminds me of something one of my students said. We were discussing male/female differences — in particular, the observation that women are more religious than men. One student said that in her experience, guys were either not religious at all or very religious.

I agree with her. I think this is why EBM has the form it does. Its male founders — not understanding the tendency that my student pointed out — went from one extreme (medical orthodoxy, unrelated to evidence) to another (evidence-based medicine). Reliance on evidence is a good idea, yes, but the founders of EBM couldn’t help making it resemble a religion. You might think that relying on evidence is the opposite of religion but they made the whole thing as religious as possible. EBM became just another way — just another excuse, really — to sneer at people.

Yes, Canker Sores Prevented (and Cured) by Omega-3

Seth Roberts21 Comments

Here is a comment left on my earlier canker-sore post by a reader named Ted:

I found out quite by accident WALNUTS get rid of [canker sores] quite quickly. The first sign of an ulcer I chew walnuts and leave the paste in my mouth for a little while (30 seconds or so).

The first time was by accident, my ulcers disappeared so quickly I knew it had to be something I ate. And the only thing I had eaten differently the past day was walnuts.

Flaxseed oil and walnuts differ in lots of ways but both are high in omega-3. My gums got much better around the time I started taking flaxseed oil. I neither noticed nor expected this; my dentist pointed it out. Several others have told me the same thing. Tyler Cowen’s gums got dramatically better. One reader started and stopped and restarted flaxseed oil, making it blindingly clear that the gum improvement is caused by flaxseed oil. There is plenty of reason to think the human diet was once much higher in omega-3. All this together convinces me that omega-3 can both prevent and cure canker sores. Not only that, I’m also convinced that canker sores are a sign of omega-3 deficiency. You shouldn’t just get rid of them with walnuts; you should change your diet. Omega-3 has other benefits (better brain function, less inflammation, probably others).

Let’s say I’m right about this — canker sores really are prevented and cured by omega-3. Then there are several things to notice.

1. Web facilitation. It was made possible by the internet. My initial interest in flaxseed oil came from reading the Shangri-La Diet forums. I didn’t have to read a single book about the Aquatic Ape theory; I could learn enough online. Tyler Cowen’s experience was in his blog. Eric Vlemmix contacted me by email. No special website was involved.

2. Value of self-experimentation. My flaxseed oil self-experimentation played a big part, although it had nothing to do with mouth health. These experiments showed dramatic benefits — so large and fast that something in flaxseed oil, presumably omega-3, had to be a necessary nutrient. Because of these results, I blogged about omega-3 a lot, which is why Eric emailed me about his experience.

3. Unconventional evidence. All the evidence here, not just the self-experimentation, is what advocates of evidence-based medicine and other evidence snobs criticize. Much of it is anecdotal. Yet the evidence snobs have, in this case, nothing to show for their snobbery. They missed this conclusion completely. Nor do you need a double-blind study to verify/test this conclusion. If you have canker sores, you simply drink flaxseed oil or eat walnuts and see if they go away. Maybe this omnipresent evidence snobbery is . . . completely wrong? Maybe this has something to do with the stagnation in health research?

4. Lack of credentials. No one involved with this conclusion is a nutrition professor or dentist or medical doctor, as far as I know. Apparently you don’t need proper credentials to figure out important things about health. Of course, we’ve been here before: Jane Jacobs, Elaine Morgan.

5. Failure of “trusted” health websites. Health websites you might think you could trust missed this completely. The Mayo Clinic website lists 15 possible causes — none of them involving omega-3. (Some of them, we can now see, are correlates of canker sores, also caused by lack of omega-3.) If canker sores can be cured with walnuts, the Mayo list of treatments reads like a list of scurvy cures from the Middle Ages. The Harvard Medical School health website is even worse. “Keep in mind that up to half of all adults have experienced canker sores at least once,” it says. This is supposed to reassure you. Surely something this common couldn’t be a serious problem.

6. Failure of the healthcare establishment. Even worse, the entire healthcare establishment, with its vast resources, hasn’t managed to figure this out. Canker sores are not considered a major health problem, no, but, if I’m right, that too is a mistake. They are certainly common. If they indicate an important nutritional deficiency (too little omega-3), they become very important and their high prevalence is a major health problem.