Month: July 2010

Learning From Mulan

Seth Roberts7 Comments

You may have seen the lovely Disney movie based on the story of Mulan, the girl who dresses as a boy to take her father’s place in the army. Even better is the original story, which is only 300-odd ancient Chinese characters. It begins like this:

Mulan was weaving. She was having trouble concentrating on her work. The previous night she had learned that her elderly father had been called to military service.

What a great beginning! Instantly you care. You could read every short story The New Yorker has published and not find a beginning as great as that. The essence of how a story should begin is so strong it reminds me of something that happened when I was a grad student. My roommates had cooked something with a lot of ginger. So that’s what ginger tastes like, I thought. I understood for the first time why ginger ale was called ginger ale.

Do Fermented Foods Improve Brain Function?

Seth RobertsLeave a comment

I’m sure we need to ingest plenty of bacteria for our digestion and immune systems to work properly. What about the brain? When I started eating lots of fermented foods, I didn’t notice any brain-related changes, such as changes in mood or sleep. Suggesting that fermented foods have little effect on the brain. But a new study in the American Journal of Psychiatry suggests I reexamine the question. The researchers followed 160,000 high-school students in Taiwan for eleven years.

The incidence rate of suicide mortality in participants with current asthma at [the start of the study] was more than twice that of those without asthma (11.0 compared with 4.3 per 100,000 person-years), but there was no significant difference in the incidence of natural deaths.

Linking immune-system dysfunction (asthma) with brain dysfunction (suicide). I believe fermented foods will substantially reduce asthma. This finding makes it more plausible they’d also improve brain function.

The Journalistic Response to Climategate

Seth Roberts14 Comments

When the Climategate emails came out, people like Bill McKibben and Elizabeth Kolbert were in enormously difficult positions. McKibben, an extremely talented writer, had centered his entire professional life around stopping climate change. Kolbert, also a very talented writer, hadn’t become an activist, like McKibben, but she had made the dangers of climate change her journalistic specialty. She wrote a book about it, for example. For them to say that the Climategate emails revealed something important — namely, that the case for man-made climate change is much weaker than the public realizes — would have been like the Pope saying God might not exist. It wasn’t going to happen. And it hasn’t happened.

But other journalists are not so committed to one side. They are free to react honestly and intelligently. One sign of what an honest and intelligent reaction would be came during a New Yorker podcast about Climategate. On one side was Kolbert, on the other — saying that Climategate mattered — was Peter Boyer. Kolbert came off as nervous and defensive; Boyer came off as reasonable.

Another sign of what an honest and intelligent reaction would be is this column by Clive Crook, an Atlantic editor. Crook ridicules the inquiries that followed for reasoning such as this:

Had Dr. [Michael “Hockey Stick”] Mann’s conduct of his research been outside the range of accepted practices, it would have been impossible for him to receive so many awards and recognitions . . .

Crook is right to ridicule this. Ranjit Chandra, a nutrition professor, received the Order of Canada, an extremely prestigious award, yet some of his research appears fabricated.

Kombucha Eliminated Heartburn

Seth Roberts10 Comments

In a comment on an old post — in which I described how a friend’s acid reflux was greatly alleviated by kombucha — Dave Schulz says he had a similar experience:

My heartburn occurs daily unless a) I stick to a strict diet with no carbs, dairy, or greasy/fatty food, like the Paleo Diet or b) I drink kombucha daily. It’s not always possible to do a), so kombucha has literally been a life saver for me.

Daily kombucha eliminates his heartburn for long periods of time, not just for a few hours after drinking it. Due to the current ban he can no longer get it and his heartburn came back. He got the idea from a friend. Before kombucha, he’d tried many remedies that didn’t work. The three doctors he saw were no help.

On the Mayo Clinic website a doctor says that “until definitive studies quantify the risks and benefits of Kombucha tea, it’s prudent to avoid it.” This is what the Protestant Reformation was about: Speaking directly to God rather than waiting for “definitive studies” by experts that “quantify the risks and benefits”.

The Zamboni MS Procedure in Canada

Seth Roberts11 Comments

Because his wife had multiple sclerosis (MS), the Italian surgeon Paolo Zamboni discovered that a simple surgical procedure helped a large fraction of patients with MS. The Canadian MS society and some Canadian neurologists have not reacted well to this discovery:

In November 2009, an elated Jamie Chalmers went to his neurologist and handed him a stack of print-outs on the new findings. Without so much as a glance, the neurologist tossed the papers in the garbage. He told Chalmers it was nothing but junk science.

In fact, cause and effect are utterly clear:

The vein-opening procedure involves snaking a balloon through the groin up to the neck and then inflating it where the veins are believed to be narrow. It didn’t hurt, says Stock. “I could feel it . . . it was like plugging your nose and blowing.”

Almost immediately afterward, says Stock, he felt a change: his compromised sense of balance had improved. By the time he touched down in Canada [the operation was in India], he was convinced he had done the right thing. Before the procedure, he couldn’t read a full paragraph. Now, he is reading whole chapters again. Before, he couldn’t stand without support for long and was always hunched over his cane. Now, he can stand and walk for as long as an hour.

Doctors have believed that MS is an autoimmune disease. For example, the Mayo Clinic’s website says:

Multiple sclerosis (MS) is a potentially debilitating disease in which your body’s immune system eats away at the protective sheath that covers your nerves.

Thanks to Anne Weiss.

The Problem with Evidence-Based Medicine

Seth Roberts11 Comments

In a recent post I said that med school professors cared about process (doing things a “correct” way) rather than result (doing things in a way that produces the best possible outcomes). Feynman called this sort of thing “ cargo-cult science“. The problem is that there is little reason to think the med-school profs’ “correct” way (evidence-based medicine) works better than the “wrong” way it replaced (reliance on clinical experience) and considerable reason to think it isn’t obvious which way is better.

After I wrote the previous post, I came across an example of the thinking I criticized. On bloggingheads.tv, during a conversation between Peter Lipson (a practicing doctor) and Isis The Scientist (a “physiologist at a major research university” who blogs at ScienceBlogs), Isis said this:

I had an experience a couple days ago with a clinician that was very valuable. He said to me, “In my experience this is the phenomenon that we see after this happens.” And I said, “Really? I never thought of that as a possibility but that totally fits in the scheme of my model.” On the one hand I’ve accepted his experience as evidence. On the other hand I’ve totally written it off as bullshit because there isn’t a p value attached to it.

Isis doesn’t understand that this “ p value” she wants so much comes with a sensitivity filter attached. It is not neutral. To get it you do extensive calculations. The end result (the p value) is more sensitive to some treatment effects than others in the sense that some treatment effects will generate smaller (better) p values than other treatment effects of the same strength, just as our ears are more sensitive to some frequencies than others.

Our ears are most sensitive around the frequency of voices. They do a good job of detecting what we want to detect. What neither Isis nor any other evidence-based-medicine proponent knows is whether the particular filter they endorse is sensitive to the treatment effects that actually exist. It’s entirely possible and even plausible that the filter that they believe in is insensitive to actual treatment effects. They may be listening at the wrong frequency, in other words. The useful information may be at a different frequency.

The usual statistics (mean, etc.) are most sensitive to treatment effects that change each person in the population by the same amount. They are much less sensitive to treatment effects that change only a small fraction of the population. In contrast, the “clinical judgment” that Isis and other evidence-based-medicine advocates deride is highly sensitive to treatments that change only a small fraction of the population — what some call anecdotal evidence. Evidence-based medicine is presented as science replacing nonsense but in fact it is one filter replacing another.

I suspect that actual treatment effects have a power-law distribution (a few helped a lot, a large fraction helped little or not at all) and that a filter resembling “clinical judgment” does a better job with such distributions. But that remains to be seen. My point here is just that it is an empirical question which filter works best. An empirical question that hasn’t been answered.

Does Lithium Slow ALS?

Seth Roberts10 Comments

In 2008, an article in Proceedings of the National Academy of Sciences (PNAS) reported that lithium had slowed the progression of amyotrophic lateral sclerosis (ALS), which is always fatal. This article describes several attempts to confirm that effect of lithium. Three studies were launched by med school professors. In addition, patients at PatientsLikeMe also organized a test.

One of Nassim Taleb’s complaints about finance professors is their use of VAR (value at risk)Â to measure the riskiness of investments. It’s still being taught at business schools, he says. VAR assumes that fluctuations have a certain distribution. The distributions actually assumed turned out to grossly underestimate risk. VAR has helped many finance professionals take risks they shouldn’t have taken. It would have been wise for finance professors to wonder how well VAR does in practice, thereby to judge the plausibility of the assumed distribution. This might seem obvious. Likewise, the response to the PNAS paper revealed two problems that might seem obvious:

1. Unthinking focus on placebo controls. It would have been progress to find anything that slows ALS. Anything includes placebos. Placebos vary. From the standpoint of those with ALS, it would have been better to compare lithium to nothing than to some sort of placebo. As far as I can tell from the article, no med school professor realized this. No doubt someone has said that the world can be divided into people focused on process (on doing things a certain “right” way) and those focused on results (on outcomes). It should horrify all of us that med school professors appear focused on process.

2. Use of standard statistics (e.g., mean) to measure drug effects. I have not seen the ALS studies, but if they are like all other clinical trials I’ve seen, they tested for an effect by comparing means using a parametric test (e.g., a t test). However, effects of treatment are unlikely to have normal distributions nor are likely to be the same for each person. The usual tests are most sensitive when each member of the treatment group improves the same amount and the underlying variation is normally distributed. If 95% of the treatment group is unaffected and 5% show improvement, for example, the usual tests wouldn’t do the best job of noticing this. If medicine A helps 5% of patients, that’s an important improvement over 0%, especially with a fatal disease. And if you take it and it doesn’t help, you stop taking it and look elsewhere. So it would be a good idea to find drugs that only help a fraction of patients, perhaps a small fraction. The usual analyses may have caused drugs that help a small fraction of patients to be considered worthless when they could have been detected.

All the tests of lithium, including the PatientsLikeMe test, turned out negative. The PatientsLikeMe trial didn’t worry about placebo effects, so my point #1 isn’t a problem. However, my point #2 probably applies to all four trials.

Thanks to JR Minkel and Melissa Francis.

Unlikely Data

Seth Roberts2 Comments

Connoisseurs of scientific fraud may enjoy David Grann’s terrific article about an art authenticator in the current New Yorker and this post about polling irregularities. What are the odds that two such articles would appear at almost the same time?

I suppose I’m an expert, having published several papers about data that was too unlikely. With Saul Sternberg and Kenneth Carpenter, I’ve written about problems with Ranjit Chandra’s work. I also wrote about problems with some learning experiments.

Assorted Links

Seth Roberts2 Comments

Thanks to Alex Chernavsky.