The Contribution of John Ioannidis

Seth Roberts6 Comments

From an excellent Atlantic article about John Ioannidis, who has published several papers saying that medical research is far less reliable than you might think:

A different oak tree at the site provides visitors with a chance to try their own hands at extracting a prophecy. “I [bring] all the researchers who visit me here, and almost every single one of them asks the tree the same question,” Ioannidis tells me . . . “’Will my research grant be approved?'”

A good point. I’d say his main contribution, based on this article, is pointing out the low rate of repeatability of major medical findings. Until someone actually calculated that rate, it was hard to know what it was, unless you had inside experience. The rate turned out to be lower than a naive person might think. It was not lower than an insider might think, which explains lack of disagreement:

David Gorski . . . noted in his prominent medical blog that when he presented Ioannidis’s paper on [lack of repeatability of] highly cited research at a professional meeting, “not a single one of my surgical colleagues was the least bit surprised or disturbed by its findings.”

I also like the way Ioannidis has emphasized the funding pressure that researchers face, as in that story about the oak tree. Obviously it translates into pressure to get positive results, which translates into overstatement.

I also think his critique of medical research has room for improvement:

1. Black/white thinking. He talks in terms of right and wrong. (“We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine.”) This is misleading. There is signal in all that medical research he criticizes; it’s just not as strong a signal as the researchers claimed. In other words the research he says is “wrong” has value. He’s doing the same thing as all those meta-analyses that ignore all research that isn’t of “high quality”.

2. Nihilism (which is a type of black/white thinking). For example,

How should we choose among these dueling, high-profile nutritional findings? Ioannidis suggests a simple approach: ignore them all.

I’ve paid a lot of attention to health-related research and benefited greatly. Many of the treatments I’ve studied through self-experimentation were based on health-related research. An example is omega-3. There is plenty of research suggesting its value and this encouraged me to try it. Likewise, there is plenty of evidence supporting the value of fermented foods. That evidence and many other studies (e.g., of hormesis) paint a large consistent picture.

3. Bias isn’t the only problem, but, in this article, he talks as if it is. Bias is a relatively minor problem: you can allow for it. Other problems you can’t allow for. One is the Veblenian tendency to show off. Thus big labs are better than small ones, regardless of which would make more progress. Big studies better than small, expensive equipment better than cheap, etc. And, above all, useless is better than useful. The other is a fundamental misunderstanding about what causes disease and how to fix it. A large fraction of health research money goes to researchers who think that studying this or that biochemical pathway or genetic mechanism will make a difference — for a disease that has an environmental cause. They are simply looking in the wrong place. I think the reason is at least partly Veblenian: To study genes is more “scientific” (= high-tech = expensive) than studying environments.

Thanks to Gary Wolf.

How She Adjusted to Living in China

Seth Roberts7 Comments

I asked an American friend who’s been in China for a year how the year had changed her. She told a story:

I was in a restaurant in Inner Mongolia. This guy was going around smashing things, throwing glasses. He was drunk. I was shocked. I expected a strong reaction: Get out of my restaurant! That’s not what happened. There was no strong reaction. The guy finally left and the staff cleaned up the mess he made. I’ve learned not to react strongly to unusual behavior.

I love this story. That travel changes your assumptions is hardly a new idea but this says it vividly and briefly.

Chinese Reaction to Liu Xiaobo’s Nobel Prize

Seth Roberts7 Comments

I asked several Tsinghua students what they thought about Liu Xiaobo, the imprisoned Chinese dissident, winning the Nobel Peace Prize. There was a wide range of answers:

1. “It’s a sensitive subject,” said one student. And said no more.

2. “The Nobel Prize always seems to involve China,” said another student. Maybe she meant the Peace Prize in 1989 to the Dalai Lama and the more recent Literature prize to Gao Xingjian, but I’m not sure. Politely changing the subject.

3. “I don’t know much about what he stands for,” said another student (a freshman).

4. “Now is not the right time for his ideas. They would interfere with economic progress,” said a student who is a member of the Communist Party.

5. “Many people say because the European economy is bad, they gave the prize to someone who will never collect the money [because he’s imprisoned],” said another student. She added that receiving the prize will be bad for Liu. Because it was “a great shame for China” (meaning the government), they will increase his prison sentence.

How to Lie with Meta-Analysis

Seth Roberts6 Comments

Michael Constans drew my attention to a Consumer Reports article about spinal surgery. According to the article, a popular type of spinal surgery called vertroplasty (involving cementing vertebrae together) doesn’t work and should be stopped:

Despite the popularity of the procedure, the American Academy of Orthopedic Surgeons has just released a guideline [actually a meta-analysis of available data] saying it doesn’t work, and shouldn’t be used.

They reviewed all the literature about the procedure, and found two good-quality studies (randomized controlled trials) that show vertebroplasty works no better than a fake (placebo) procedure. There were no clinically significant differences in pain or disability, they say.

No significant difference means it “doesn’t work”? At best, it’s absence of evidence. And it’s not even that because we don’t know what the rest of the research suggests. The meta-analysis might have ignored thirty studies; it doesn’t say how many studies were ignored. Nor does it say what “placebo” means. Patients are interested in pain relief. Whether the pain relief is “all in their head” or whatever hardly matters. From a patient’s view, and a clinician’s view, a better comparison is a group that gets another plausible treatment or no treatment. The meta-analysis reports one study that compared vertrebroplasty to “conservative” treatment, which makes more sense.

So how come vertebroplasty has been used so often? Other experts have recommended the treatment in the past, including the United Kingdom’s National Institute for Health and Clinical Excellence, which said most people got some pain relief from the procedure.

This suggests that ignoring of evidence and poor choice of comparison group made a difference.

In this case, it probably comes down to what you mean by significant pain relief. The US surgeons set a strict definition—they said a difference in pain relief of less than 2 points on a 10-point scale was not meaningful for patients. Smaller differences in pain relief were recorded in the studies, but, say the surgeons, they were not big enough to make a real difference.

I think patients would prefer to decide for themselves what degree of pain relief is meaningful.

The surgeons say the evidence against vertebroplasty is strong, and they don’t expect future studies to overturn their recommendations.

Haha! Two studies with inferior comparison groups that failed to find a difference — and ignoring an unknown number of studies and arbitrarily raising the bar — adds up “strong evidence against”!

Still the Guideline and Evidence Report is useful. It reviews a range of treatments, provides citations (so you can look much further), and isn’t wedded to the placebo-comparison group. It includes studies with other comparison groups; the article just doesn’t mention them.

And I completely agree with the article’s conclusion:

If you’re considering any type of surgery, ask your surgeon to show you data about how likely it is to solve your problem.

And don’t take your surgeon’s word for it that such data exists.

Strong Light and Cancer

Seth Roberts5 Comments

From an excellent article about light pollution (not online) by David Owen in the 20 Aug 2007 issue of The New Yorker:

Richard Stevens, a cancer epidemiologists at the University of Connecticut Health Center, in Farmington, has suggested a link between cancer and the “circadian disruption” of hormones caused by artificial lighting. Early in his career, Stevens was one of many researchers struck by the markedly high incidence of breast cancer among women in the industrial world, in comparison with those in developing countries, and he at first supported the most common early hypothesis, which was that the cause must be dietary. Yet repeated studies found no clear link to food. In the early eighties, Stevens told me recently, “I literally woke up in the middle of the night — there was a street lamp outside the window, and it was so bright that I could almost read in my bedroom — and I thought, Could it be that? A few years later, he persuaded the [directors] of the Nurses’ Health Study . . . to add questions about nighttime employment, and the study subsequently revealed a strong association between working the night shift and an increased risk of breast cancer. [The researchers] wrote, “We hypothesize that the potential primary culprit for this observed association is the lack of melatonin, a cancer-protective agent whose production is severely diminished in people exposed to light at night.”

Exposure to strong light at night reduces the amplitude of your circadian rhythms. That causes a thousand changes. To decide that one of them (“lack of melatonin”) is the one that matters is highly premature. If reducing circadian amplitude increases cancer, it follows that getting more light during the day — which surely increases circadian amplitude — will reduce cancer.

The article also says:

Growing numbers of us pass most of our waking hours “in a box, looking at a box,” as Dave Crawford put it . . . Fewer and fewer of us spend much time outside at all, except in automobiles.

I have measured the light inside cars (front seats) several times and found it is quite strong (you are close to a big window). If the article is arguing that night light is bad and causes cancer, I am unconvinced. Night light exposure and daylight exposure are confounded — people who work night shifts get more night light and less daylight.
The Nurses Study paper: E Schernhammer, K Schulmeister. Light at night and cancer risk. Photochemistry and Photobiology, 2004, Vol 79, Iss 4, pp 316-318.

Where Does Oil Come From?

Seth Roberts6 Comments

This fascinating article describes two ideas about oil production that were new to me: 1. It is made by microbes a long way down inside the Earth. 2. It is made by nuclear reactions going on in the middle of the earth.

The first idea came from a Cornell geologist named Thomas Gold. According to the article, “some geologists were so incensed by Gold’s ideas they petitioned to have the government remove all mention of it from the nation’s libraries.” That is so strange (and no source cited) you might think the whole article is made-up but Gold explained his ideas here (short) and here (long).

Thanks to Carl Willat.

Chinese Text-Message Censorship

Seth Roberts4 Comments

An American friend wondered about Chinese reaction to Liu Xiaobo winning the Nobel Peace Prize. So I sent a text message to four Chinese friends: “Did u know chinese dissident won nobel peace prize?”

Six hours later I hadn’t received any replies. I phoned one of them. She hadn’t received my message. I saw it hadn’t gone through: “Unable to send message”. I tried again. Failed again. Then I tried to send “did u know chinese dissident won prize?” Success. I tried the earlier message, with “nobel peace prize”. Fail. Tried the shorter message. Success.

Messages almost never fail, so it was clearly censorship. One of my friends said the same about messages in Chinese: “We can’t send email or text messages” about it. She had heard about the prize from her classmates. She didn’t know how they had learned about it. Another Chinese friend read it on a website. Later it was gone.

I thought of the boy who cried wolf. One day there was a wolf. But no one believed him.

Dairy Consumption and Health

Seth Roberts6 Comments

Two studies of the effect of dairy consumption on health have recently appeared. Both suggest it is healthy. One of them– a prospective study where about 1500 people were followed for 16 years — found no association of dairy intake with overall mortality but did find a protective effect of full-fat dairy against heart disease. The study considered lots of possibilities and the authors write ” it is important to take into account the large number of comparisons considered in this study and thus we cannot rule out the possibility that the protective association between full-fat dairy intake and cardiovascular mortality was due to chance.”

I mentioned this study earlier. It gains more credence because of the other study, which is a meta-analysis. The second study found protective effects of dairy products on several outcomes, including overall mortality:

Meta-analyses suggest a reduction in risk in the subjects with the highest dairy consumption relative to those with the lowest intake: 0.87 (0.77, 0.98) for all-cause deaths, 0.92 (0.80, 0.99) for ischaemic heart disease, 0.79 (0.68, 0.91) for stroke and 0.85 (0.75, 0.96) for incident diabetes.

This is good news for me since I eat yogurt and butter every day.

Thanks to Peter Spero.

Research Fraud in China

Seth Roberts9 Comments

From the New York Times:

Last December, a British journal that specializes in crystal formations announced that it was withdrawing more than 70 papers by Chinese authors whose research was of questionable originality or rigor. . . . “Even fake papers count because nobody actually reads them,” said Mr. Fang, who is more widely known by his pen name, Fang Zhouzi, and whose Web site, New Threads, has exposed more than 900 instances of fakery, some involving university presidents and nationally lionized researchers.

Recently a Tsinghua colleague asked me to fix the English in his paper. Most paragraphs required a few changes every sentence but here and there were whole paragraphs with no mistakes. Presumably he copied them from somewhere else. The material in them was boring — it was like copying from the phone book — so it was hard to care (he wasn’t taking credit for anyone else’s ideas) but I wonder if he realized how obvious it was. I don’t mean this is typical. I have looked at several other papers by Chinese authors and found no patches of perfect English.

The article begins with a false claim by a Chinese doctor — and of course these are truly damaging. In my experience, false claims by American doctors are common. An example is my surgeon recommending an operation that, she said, evidence showed would benefit me. There was no such evidence. One value of self-experimentation is that you can find out if a medicine works, rather than take your doctor’s word for it. I became impressed with self-experimentation when it showed me that an acne medicine (tetracycline, an antibiotic) my dermatologist had prescribed didn’t work. Not at all. He didn’t express any doubts when he prescribed it. Call it forensic DNA testing (e.g., The Innocence Project) for the rest of us.

Perhaps the Chinese people, faced with even more false claims than Americans, can benefit even more from self-experimentation.

Thanks to Tim Beneke.

How to Eat a Lot of Butter

Seth Roberts25 Comments

Since I discovered that butter makes my brain work better, I have been eating half a stick (60 g) per day. Usually half in the morning and half in the evening. It is hard to eat by itself but easy to eat with other foods. I’ve tried a dozen ways of doing this. My top three additions:

1. Pu’er tea. The most convenient. As convenient as drinking tea. Put the butter in hot tea, wait till it melts. I can eat at least 20 g of butter in one cup of tea. Butter tea is common in Tibet. Thanks again to Robin Barooah.

2. Cherry tomatoes. The healthiest and fastest. Slice the tomatoes in half lengthwise, eat each half with a similar-sized piece of butter. It is like that classic Italian combination, mozzarella and tomatoes.

3. Thin-sliced roast beef. The most delicious. Wrap a piece of butter with the roast beef. However, I already eat plenty of meat, it is hard to get thin-sliced roast beef in Beijing, and it is so delicious I end up buying a lot of thin-sliced roast beef.

None of these additions affects brain function (measured by arithmetic score), as far as I can tell, although I suppose the tea wakes me up.