How to Lie with Meta-Analysis

Michael Constans drew my attention to a Consumer Reports article about spinal surgery. According to the article, a popular type of spinal surgery called vertroplasty (involving cementing vertebrae together) doesn’t work and should be stopped:

Despite the popularity of the procedure, the American Academy of Orthopedic Surgeons has just released a guideline [actually a meta-analysis of available data] saying it doesn’t work, and shouldn’t be used.

They reviewed all the literature about the procedure, and found two good-quality studies (randomized controlled trials) that show vertebroplasty works no better than a fake (placebo) procedure. There were no clinically significant differences in pain or disability, they say.

No significant difference means it “doesn’t work”? At best, it’s absence of evidence. And it’s not even that because we don’t know what the rest of the research suggests. The meta-analysis might have ignored thirty studies; it doesn’t say how many studies were ignored. Nor does it say what “placebo” means. Patients are interested in pain relief. Whether the pain relief is “all in their head” or whatever hardly matters. From a patient’s view, and a clinician’s view, a better comparison is a group that gets another plausible treatment or no treatment. The meta-analysis reports one study that compared vertrebroplasty to “conservative” treatment, which makes more sense.

So how come vertebroplasty has been used so often? Other experts have recommended the treatment in the past, including the United Kingdom’s National Institute for Health and Clinical Excellence, which said most people got some pain relief from the procedure.

This suggests that ignoring of evidence and poor choice of comparison group made a difference.

In this case, it probably comes down to what you mean by significant pain relief. The US surgeons set a strict definition—they said a difference in pain relief of less than 2 points on a 10-point scale was not meaningful for patients. Smaller differences in pain relief were recorded in the studies, but, say the surgeons, they were not big enough to make a real difference.

I think patients would prefer to decide for themselves what degree of pain relief is meaningful.

The surgeons say the evidence against vertebroplasty is strong, and they don’t expect future studies to overturn their recommendations.

Haha! Two studies with inferior comparison groups that failed to find a difference — and ignoring an unknown number of studies and arbitrarily raising the bar — adds up “strong evidence against”!

Still the Guideline and Evidence Report is useful. It reviews a range of treatments, provides citations (so you can look much further), and isn’t wedded to the placebo-comparison group. It includes studies with other comparison groups; the article just doesn’t mention them.

And I completely agree with the article’s conclusion:

If you’re considering any type of surgery, ask your surgeon to show you data about how likely it is to solve your problem.

And don’t take your surgeon’s word for it that such data exists.

This Year’s Nobel Prize in Medicine

I applaud it. The winner developed in vitro fertilization, which has helped millions of parents. In contrast to last year’s prize for telomere research, which has helped no one. Notice what in vitro fertilization is not: It is not taking a powerful poorly-tested drug for the rest of your life — the drug industry’s preferred answer to all problems. It is not expensive (given the benefits). Unlike health care in general. It is not dangerous, unlike many drugs and surgeries. It is not molecular biology. It is barely science (uncovering cause and effect). If the prize were given for research like this year after year, many biologists who now dream of winning a Nobel Prize would stop dreaming. It is not a typical Nobel Prize. They waited so long to award it that the winner became demented. Above all, the prize-winning work was not mainstream medical research. The winner and his collaborator endured “an unremitting barrage of criticism”, unlike almost any other medical researcher.

The award is unflattering to medical ethicists, who did a lot to try to prevent the prize-winning work.

Out-of-Control Drug Companies and Their Consorts


The increase in pediatric bipolar diagnosis, [Dr. Joseph Biederman’s] lawyer said, “cannot be attributed solely to Dr. Biederman’s work.”

This article is a long list of drug-company abuses, such as

Lilly produced a video called “The Myth of Diabetes” to sell Zyprexa, which became its all-time best-selling drug, even though evidence showed that Zyprexa could cause diabetes.

And

Pfizer paid more than 250 child psychiatrists to promote its antipsychotic, Geodon, at a time when it was approved only for adults, according to a government filing with the Pfizer settlement last year. High-prescribing doctors pocketed extra money in the form of research payments, speaking fees, gifts, meals and junkets.

Pharma Gossip. Thanks to Ken Feinstein.

The Corruption of Drug Trials

In a clinical trial of a new antipsychotic drug done at the University of Minnesota, a man named Dan Weiss was given a choice: be hospitalized in a psych ward or, shockingly, “take part in an industry-funded study of antipsychotic drugs”. The usual choice is between hospitalization or conventional treatment. Weiss chose to be in the clinical trial. During the trial he killed himself.

An FDA investigator named Sharon Matson decided that Weiss had not been coerced into participating! During a trial, Moira Keane, the head of the University of Minnesota Institutional Review Board, which of course is meant to protect human subjects, claimed the purpose of the board was not to protect human subjects. The purpose of the board, Keane said, was “to make sure that Olson and the trial sponsor had a plan to protect subjects.” This is false. IRBs sometimes measure compliance, not just plans.

After Weiss’s mom sued the University of Minnesota and lost,

The university filed a legal action against Mary, demanding that she pay the university $57,000 to cover its legal expenses. Gale Pearson, one of Mary’s attorneys, says that while such suits are technically permissible, she had never seen one filed in her previous 14 years of legal practice. The university agreed to drop the lawsuit against Mary only when she agreed not to appeal the judge’s decision.

The article by Carl Elliott about this case also contains excellent discussion of how drug companies shape clinical trials to get the results they want — and when that fails, hide the results. The effect is that new drugs are approved that are worse than the drugs they replace.

Thanks to James Andrewartha.

The Thick-Fingered Surgeon

Kim Øyhus, who has a proof that correlation is evidence of causation, told me this story of medical overtreatment:

At 16 I got glass splinters and sand inside my hand when a test tube broke because it was handled too hard. The small local clinic sewed the wound shut without close examination, so a glass splinter and sand remained deep inside. About 5 years later the glass splinter cut itself loose because of bowling, and for about 10 years made the hand problematic to use. It became swollen and partly numb each time I used it with force.

So, 5 years into this I decided to do something about it. My mother contacted the local clinic again. “Come back tomorrow, and we will look at it.” the doctor said to me at the first examination.
The next day I arrived to a ready operation table of the simplest kind, and just the doctor.

“I thought we should look at it today, not operate it,” I said.

“You know perfectly well that that means operating,” he said.

I took that answer as a hint that he might not be an honest person.

In addition he had nervous tics in his shoulders and arms, as well as big thick sausage fingers, as if he plowed hard soil every day.

So, there I lay on the hard mattress, arm outstretched while he plunged the local anesthetic needle hither and dither inside my hand, while my unease continued to grow. So when he took the scalpel and pointed it at my hand, I said “No. There is not going to be an operation today.” and rose from the guerney.

“You can’t just leave like that!” he said.

“It is my hand, so I decide what is to be done with it,” I answered, and left the room with my mother.

Having seen the entire ordeal silently, which is very atypical of her, she was visibly relived, and agreed entirely with my decision. She thought he was extremely nervous.

As we drove away, I saw the doctor sitting smoking on some wooden stacks outside, looking somewhat forlorn. I waved, and he waved back.

Fortunately, the needle had moved the glass splinter to a better place, so the hand was useful again for a few years after that.

When it started getting really bad again, I asked other doctors how stuff like that could be fixed, and they told me that hand operations are exceedingly difficult due to the delicate nature and lots of nerves, tendons, muscles, and so on everywhere tight together, so it requires a surgical team with a very good and experienced surgeon, long operating time, and often unconscious anesthesia. And so it was. They found sand inside nerves. I can tell you that is uncomfortable to have. The glass splinter I knew was there because I could feel it by poking hard with my fingers before the hand got swollen, was nowhere to be found.

The recovery took many years. It’s OK now. And eating omega-3s and dropping carbohydrates this last year significantly improved it. It became softer, more bendable, and more sensitive, less numb, even though it is decades old now.

The Treatment Trap by Rosemary Gibson

The Treatment Trap, a new book by Rosemary Gibson, is about the overuse of medical care — too much medicine. In this talk, Gibson tells how a woman getting a heart check-up overheard a conversation: “We’re only doing 9 bypasses a day, we need 14 a day to keep this place running.” The result of her check-up: She needed a bypass!

My encounter with too much surgery (and here). The Safe Patient Project is gathering stories of overtreatment, although it is unclear what they will do with them.

New Heart Scan Results: Good News (lipid scores)

My recent heart scan results were 50% lower (= better) than predicted. Apparently I am doing something right.
You might think that my lipid values would reflect that. Not quite. They were measured twice in the last two weeks, first with a Cholestech LDX machine (instant results); second, ordinary lab tests.
Here are the scores (first test, second test). Total Cholesterol: 210, 214, which is “borderline high” (borderline bad) according to the Cholestech LDX quick reference sheet. HDL = 17, 36, which is “low” (bad). TRG = 62, 75, which is “normal”. LDL = 180, 163, which is “high” (bad).

There is no hint in these numbers that I am doing the right thing! If anything, they imply the opposite, that I’m doing the wrong thing. This supports all those people, such as Uffe Ravnskov, who say the connection between cholesterol and heart disease is badly overstated.

New Heart Scan Results: Good News

One and a half years ago, in February 2009, I got a heart scan. It’s an X-ray measurement of how calcified your arteries are. Persons with high scores are much more likely to have a heart attack than persons with low scores. Scores in the hundreds are dangerous. Tim Russert, who died at age 58 of a heart attack, had a score of about 200 ten years before his death. Above age 40, the scores typically increase about 25% per year. That puts Russert’s score when he died at around 2000.

A few weeks ago I got another scan, at the same place with the same machine. Here are my scores. February 2009: 38 (about 50th percentile for my age). August 2010: 29 (between 25th & 50th percentile). In other words: 47% lower than expected. The earlier scan detected 3 “lesions”; the recent scan detected 2. The woman who runs the scanning center — HeartScan, in Walnut Creek, California — told me that decreases in this score are very rare. About 1 in 100, she said.

The only big lifestyle change I made between the two scans is to eat much more animal fat. After I found that pork fat improved my sleep, I started to eat a large serving of pork belly (with 80-100 g of fat) almost every day. Later I switched to 60 g of butter every day. The usual view, of course, is that to eat so much animal fat is v v bad and will “clog” my arteries. In fact, the reverse happened. Judging from this, the change was v v good.

Drug Company Corruption

This Al Jazeera documentary, called “Drug Money”, emphasizes three things.

1. Doctors get vast amounts of money from drug companies, which influences which drugs they prescribe. One influential doctor, Tom Stossel of Harvard, who has received “millions” from drug companies, sees no problem with that!

2. Drug companies encourage the prescription of drugs for unapproved uses. For this and other crimes, more than half of the major drug companies have been found guilty and fined billions of dollars. Several of the not-yet-guilty ones are under investigation. The problem is industry-wide, not due to a “bad apple”.

3. The harm done by deceptive practices isn’t trivial. One example is Risperdal. It isn’t approved to treat ADHD in children, but it is prescribed for that. Given to boys, it can cause them to grow breasts, which is extremely embarrassing. When the boys were given the drugs, their parents were unaware of this possibility. Joseph Biederman, another Harvard professor who has received millions from drug companies and an advocate of giving Risperdal to children, told a Congressional committee he had no idea that a large fraction of all Risperdal is given to children (“I have no idea how much Risperdal is used in children”).

Thanks to Anne Weiss.