The Journalistic Response to Climategate

Seth Roberts14 Comments

When the Climategate emails came out, people like Bill McKibben and Elizabeth Kolbert were in enormously difficult positions. McKibben, an extremely talented writer, had centered his entire professional life around stopping climate change. Kolbert, also a very talented writer, hadn’t become an activist, like McKibben, but she had made the dangers of climate change her journalistic specialty. She wrote a book about it, for example. For them to say that the Climategate emails revealed something important — namely, that the case for man-made climate change is much weaker than the public realizes — would have been like the Pope saying God might not exist. It wasn’t going to happen. And it hasn’t happened.

But other journalists are not so committed to one side. They are free to react honestly and intelligently. One sign of what an honest and intelligent reaction would be came during a New Yorker podcast about Climategate. On one side was Kolbert, on the other — saying that Climategate mattered — was Peter Boyer. Kolbert came off as nervous and defensive; Boyer came off as reasonable.

Another sign of what an honest and intelligent reaction would be is this column by Clive Crook, an Atlantic editor. Crook ridicules the inquiries that followed for reasoning such as this:

Had Dr. [Michael “Hockey Stick”] Mann’s conduct of his research been outside the range of accepted practices, it would have been impossible for him to receive so many awards and recognitions . . .

Crook is right to ridicule this. Ranjit Chandra, a nutrition professor, received the Order of Canada, an extremely prestigious award, yet some of his research appears fabricated.

Kombucha Eliminated Heartburn

Seth Roberts10 Comments

In a comment on an old post — in which I described how a friend’s acid reflux was greatly alleviated by kombucha — Dave Schulz says he had a similar experience:

My heartburn occurs daily unless a) I stick to a strict diet with no carbs, dairy, or greasy/fatty food, like the Paleo Diet or b) I drink kombucha daily. It’s not always possible to do a), so kombucha has literally been a life saver for me.

Daily kombucha eliminates his heartburn for long periods of time, not just for a few hours after drinking it. Due to the current ban he can no longer get it and his heartburn came back. He got the idea from a friend. Before kombucha, he’d tried many remedies that didn’t work. The three doctors he saw were no help.

On the Mayo Clinic website a doctor says that “until definitive studies quantify the risks and benefits of Kombucha tea, it’s prudent to avoid it.” This is what the Protestant Reformation was about: Speaking directly to God rather than waiting for “definitive studies” by experts that “quantify the risks and benefits”.

The Zamboni MS Procedure in Canada

Seth Roberts11 Comments

Because his wife had multiple sclerosis (MS), the Italian surgeon Paolo Zamboni discovered that a simple surgical procedure helped a large fraction of patients with MS. The Canadian MS society and some Canadian neurologists have not reacted well to this discovery:

In November 2009, an elated Jamie Chalmers went to his neurologist and handed him a stack of print-outs on the new findings. Without so much as a glance, the neurologist tossed the papers in the garbage. He told Chalmers it was nothing but junk science.

In fact, cause and effect are utterly clear:

The vein-opening procedure involves snaking a balloon through the groin up to the neck and then inflating it where the veins are believed to be narrow. It didn’t hurt, says Stock. “I could feel it . . . it was like plugging your nose and blowing.”

Almost immediately afterward, says Stock, he felt a change: his compromised sense of balance had improved. By the time he touched down in Canada [the operation was in India], he was convinced he had done the right thing. Before the procedure, he couldn’t read a full paragraph. Now, he is reading whole chapters again. Before, he couldn’t stand without support for long and was always hunched over his cane. Now, he can stand and walk for as long as an hour.

Doctors have believed that MS is an autoimmune disease. For example, the Mayo Clinic’s website says:

Multiple sclerosis (MS) is a potentially debilitating disease in which your body’s immune system eats away at the protective sheath that covers your nerves.

Thanks to Anne Weiss.

The Problem with Evidence-Based Medicine

Seth Roberts11 Comments

In a recent post I said that med school professors cared about process (doing things a “correct” way) rather than result (doing things in a way that produces the best possible outcomes). Feynman called this sort of thing “ cargo-cult science“. The problem is that there is little reason to think the med-school profs’ “correct” way (evidence-based medicine) works better than the “wrong” way it replaced (reliance on clinical experience) and considerable reason to think it isn’t obvious which way is better.

After I wrote the previous post, I came across an example of the thinking I criticized. On bloggingheads.tv, during a conversation between Peter Lipson (a practicing doctor) and Isis The Scientist (a “physiologist at a major research university” who blogs at ScienceBlogs), Isis said this:

I had an experience a couple days ago with a clinician that was very valuable. He said to me, “In my experience this is the phenomenon that we see after this happens.” And I said, “Really? I never thought of that as a possibility but that totally fits in the scheme of my model.” On the one hand I’ve accepted his experience as evidence. On the other hand I’ve totally written it off as bullshit because there isn’t a p value attached to it.

Isis doesn’t understand that this “ p value” she wants so much comes with a sensitivity filter attached. It is not neutral. To get it you do extensive calculations. The end result (the p value) is more sensitive to some treatment effects than others in the sense that some treatment effects will generate smaller (better) p values than other treatment effects of the same strength, just as our ears are more sensitive to some frequencies than others.

Our ears are most sensitive around the frequency of voices. They do a good job of detecting what we want to detect. What neither Isis nor any other evidence-based-medicine proponent knows is whether the particular filter they endorse is sensitive to the treatment effects that actually exist. It’s entirely possible and even plausible that the filter that they believe in is insensitive to actual treatment effects. They may be listening at the wrong frequency, in other words. The useful information may be at a different frequency.

The usual statistics (mean, etc.) are most sensitive to treatment effects that change each person in the population by the same amount. They are much less sensitive to treatment effects that change only a small fraction of the population. In contrast, the “clinical judgment” that Isis and other evidence-based-medicine advocates deride is highly sensitive to treatments that change only a small fraction of the population — what some call anecdotal evidence. Evidence-based medicine is presented as science replacing nonsense but in fact it is one filter replacing another.

I suspect that actual treatment effects have a power-law distribution (a few helped a lot, a large fraction helped little or not at all) and that a filter resembling “clinical judgment” does a better job with such distributions. But that remains to be seen. My point here is just that it is an empirical question which filter works best. An empirical question that hasn’t been answered.

Does Lithium Slow ALS?

Seth Roberts10 Comments

In 2008, an article in Proceedings of the National Academy of Sciences (PNAS) reported that lithium had slowed the progression of amyotrophic lateral sclerosis (ALS), which is always fatal. This article describes several attempts to confirm that effect of lithium. Three studies were launched by med school professors. In addition, patients at PatientsLikeMe also organized a test.

One of Nassim Taleb’s complaints about finance professors is their use of VAR (value at risk)Â to measure the riskiness of investments. It’s still being taught at business schools, he says. VAR assumes that fluctuations have a certain distribution. The distributions actually assumed turned out to grossly underestimate risk. VAR has helped many finance professionals take risks they shouldn’t have taken. It would have been wise for finance professors to wonder how well VAR does in practice, thereby to judge the plausibility of the assumed distribution. This might seem obvious. Likewise, the response to the PNAS paper revealed two problems that might seem obvious:

1. Unthinking focus on placebo controls. It would have been progress to find anything that slows ALS. Anything includes placebos. Placebos vary. From the standpoint of those with ALS, it would have been better to compare lithium to nothing than to some sort of placebo. As far as I can tell from the article, no med school professor realized this. No doubt someone has said that the world can be divided into people focused on process (on doing things a certain “right” way) and those focused on results (on outcomes). It should horrify all of us that med school professors appear focused on process.

2. Use of standard statistics (e.g., mean) to measure drug effects. I have not seen the ALS studies, but if they are like all other clinical trials I’ve seen, they tested for an effect by comparing means using a parametric test (e.g., a t test). However, effects of treatment are unlikely to have normal distributions nor are likely to be the same for each person. The usual tests are most sensitive when each member of the treatment group improves the same amount and the underlying variation is normally distributed. If 95% of the treatment group is unaffected and 5% show improvement, for example, the usual tests wouldn’t do the best job of noticing this. If medicine A helps 5% of patients, that’s an important improvement over 0%, especially with a fatal disease. And if you take it and it doesn’t help, you stop taking it and look elsewhere. So it would be a good idea to find drugs that only help a fraction of patients, perhaps a small fraction. The usual analyses may have caused drugs that help a small fraction of patients to be considered worthless when they could have been detected.

All the tests of lithium, including the PatientsLikeMe test, turned out negative. The PatientsLikeMe trial didn’t worry about placebo effects, so my point #1 isn’t a problem. However, my point #2 probably applies to all four trials.

Thanks to JR Minkel and Melissa Francis.

Unlikely Data

Seth Roberts2 Comments

Connoisseurs of scientific fraud may enjoy David Grann’s terrific article about an art authenticator in the current New Yorker and this post about polling irregularities. What are the odds that two such articles would appear at almost the same time?

I suppose I’m an expert, having published several papers about data that was too unlikely. With Saul Sternberg and Kenneth Carpenter, I’ve written about problems with Ranjit Chandra’s work. I also wrote about problems with some learning experiments.

Assorted Links

Seth Roberts2 Comments

Thanks to Alex Chernavsky.

Two Faces Better Than One?

Seth Roberts2 Comments

Here I describe my discovery that seeing faces on TV in the morning improved my mood the next day. The details of the effect suggested that the ideal stimulus is what you’d see during a conversation. For a long time I used the C-Span show Booknotes as the main source of the faces. I watched it on a 25-inch TV. More recently I used my own face in a mirror. It was readily available and perfectly life-size. I listened to a podcast or book at the same time.

A few months ago, Caleb Cooper commented saying that he’d found that looking at two faces every morning seemed to work better than looking at one face. He found that Bloggingheads.tv expanded to full screen on a 24-inch monitor (measured diagonally) produced close-to-life-size faces, which is what he wanted.

This interested me for several reasons: 1. It might make the effect stronger. 2. Bloggingheads.tv has a big selection, offering control over size. 3. I disliked looking at my face for long times. 4. It seems more naturalistic than looking at my own face.

I’ve been trying this with a 22-inch monitor (which I already had). Perhaps 24-inch would be better. The effect does seem stronger, as Caleb said.

I asked Caleb several questions about his experience.

How did you get started doing this?

I think it started when I read your posts about standing and sleeping. This led me to read your paper on self experimentation and sleep. Like you, I often suffered from early awakenings where I would wake up around 2-3 hours early, still feeling tired but having a hard time going back to sleep.

Based on what I learned from you and other sources, I tried out the following; got a pair of blue blocker clip-ons for my glasses which I put on about two hours before bed; ordered an Apollo goLite blue light emitter that I use for about an hour in the morning, I would sometimes take 1/3 mg of melatonin nine hours after waking up, and 3mg half hour before bed, and I started standing on a high difficulty Thera-Band balance pad on one leg while looking into a mirror for 30 minutes in the morning.

What made you think it was worth a try?

Well, why not:) Most self experimentation can be easily done for practically no cost, while the potential upside is significant. There’s also satisfying curiosity, expanding self knowledge, gaining mastery over your mind and body… You had a plausible theory, had collected suggestive data, and I’d already found the appetite suppression effect of the Shagnri-La was very real, so you had a track record of introducing ideas worth paying attention to.

What happened at first?

It felt to me like my sleep modestly improved, sleeping through the night longer and having the energy to get up and go much sooner after waking. This was awhile ago though, I didn’t keep any data, and I was adding and dropping different things, so my experience doesn’t have a high enough confidence interval for drawing any general inferences.

When did you make those changes?

I’d guess around sixteen months ago.

After you made those changes (“got a pair of BlueBlocker glasses…”) did your mood change?

It improved in as much as waking up feeling rested makes you feel a lot better than trying to get up while still tired.

Tell me something about yourself (job, age, etc.).

I got into medicine through Clinical Massage Therapy. Being a high school dropout I wanted something I could get into quickly, then sink or swim on my own. Massage is one of the few fields the university-accreditation complex hasn’t sunk its tentacles deeply into (a mixed blessing; for an autodidact it lets you quickly start a great career, but the field really needs a bifurcated certification track to separate medical massage from relaxational spa massage). I live in the Pacific Northwest, near the site where they developed the atomic bomb dropped on Nagasaki. Despite all the lingering nuclear waste, it’s a nice, mid sized metro area. I’m in my mid twenties.

More Flight From Data

Seth Roberts8 Comments

I’ve blogged many times about the desire of professors to show off and how it interferes with being useful. It doesn’t just make them bad teachers, it makes them bad scientists. Here’s an example from economics (via Marginal Revolution):

“The mainstream of academic research in macroeconomics puts theoretical coherence and elegance first, and investigating the data second,” says Mr. Rogoff. For that reason, he says, much of the profession’s celebrated work “was not terribly useful in either predicting the financial crisis, or in assessing how it would it play out once it happened.”

“[Academic economists] almost pride themselves on not paying attention to current events,” he says.

Pure Veblen, who in Theory of the Leisure Class provided many examples of people, including professors, priding themselves on being useless. Men wear ties, he said, to show they don’t do manual labor (which is clearly useful).

My research is closer to biology, where you can say the same thing: much of the profession’s celebrated work has not been terribly useful. Yesterday I gave an example (the oncogene theory of cancer).

Modern Veblen: Flight From Data.