Category: scientific method

Gary Taubes on the Religious Nature of Obesity Research

Seth Roberts5 Comments

From an excellent interview with Gary Taubes:

Martin: You write that the “enterprise” of diet, obesity and disease research “purports to be a science and yet functions like a religion.” In what ways?

Taubes: Simple. The researchers and authority figures in this business seem utterly uninterested in finding out whether what they believe is true or not. It’s as though their God, whichever one that might be, told them that obesity is caused by eating too much — by gluttony and/or sloth — and so they believe that unconditionally, and no amount of contradictory evidence, no failure to explain the actual observations can convince them to question it. They have unconditional faith that they know what the truth is, and there’s no place for this kind of faith in the pursuit of science. Science requires skepticism to function. Religion requires faith.

I agree with Taubes about the facts: Obesity “authority figures” do “seem uninterested in finding out” etc. Yes, it resembles religion, not science. Taubes’s summing-up, however, is one-sided. To say “science requires skepticism” is to miss the point that science also requires paying attention — finding, noticing, thinking about facts you can’t explain. Religion doesn’t. The Atkins Diet caused millions of people to lose plenty of weight in a way that mainstream weight-control theories could not explain. No one powerful in obesity research managed to notice this was a puzzle worth trying to explain.

Science isn’t just about testing ideas (Taubes’s “skepticism”); it also requires generating them. I’m hoping if I blog about this often enough I will find a humorous way to say it.

Thanks to Dave Lull.

Experimental Mathematics

Seth RobertsLeave a comment

The journal Experimental Mathematics, started in 1992, publishes “formal results inspired by experimentation, conjectures suggested by experiments, descriptions of algorithms and software for mathematical exploration, [and] surveys of areas of mathematics from the experimental point of view.” The founder wanted to make clearer and give more credit to an important way that mathematicians come up with new ideas. As the journal’s statement of philosophy puts it, “Experiment has always been, and increasingly is, an important method of mathematical discovery. (Gauss declared that his way of arriving at mathematical truths was “through systematic experimentation.”) Yet this tends to be concealed by the tradition of presenting only elegant, well-rounded, and rigorous results.”

When John Tukey wrote Exploratory Data Analysis (1977), he was doing something similar: shedding light on how to come up with new scientific ideas plausible enough to be worth testing. Tukey obviously believed this was a neglected area of statistics research. I was told that the publisher of EDA was uninterested in it; they only published it because it was part of a two-book deal. The other book, with Frederick Mosteller, was more conventional.

My paper titled “ Self-experimentation as a source of new ideas” made the same point as Tukey about an earlier step in the scientific process: data collection. How to collect data to generate new ideas worth testing was a neglected area of scientific method. Self-experimentation, derided as a way of testing ideas, might be an excellent way of generating ideas worth testing.

I think of it as crawling back into the water. In the beginning, all math was conjecture and experimentation. In the beginning, all data analysis was exploratory. In the beginning, all science was tiny and devoted to coming up with new ideas. From these came methods of proof, confirmatory data analysis, and methods of carefully testing ideas. Human nature being what it is, users and teachers of the new methods came to greatly disparage the earlier methods. Gary Taubes told me that he spoke to several obesity researchers who thought that the field essentially began with the discovery of leptin. Nothing before that mattered, they believed.

Thanks to Dev Rana.

Google vs Yahoo: Scientific Implications

Seth Roberts3 Comments

Google vs Yahoo over several years. A fable for scientists. Yahoo is worth countless billions of dollars less than Google, in spite of a big head start. The moral: methodological complications, always seen as “improvements”, have a price. The benefit of a more complex experiment is easy to see, while the increase in cost (difficulty) usually goes unremarked.

My usual comment on proposed research is that an easier experiment — often smaller, often less “well-controlled” — would be better. I seem to be the only person who says this, yet I say it all the time.

A Modern Microscope

Seth RobertsLeave a comment

Reading this legal complaint — suing a florist who gave the plaintiffs far less than agreed-upon at a very expensive wedding — I feel I am peering into a kind of microscope. Something far away and very small in the big scheme of things — the plaintiffs’ frustration — is made very clear. It reminds me of The Devil Wears Prada (no art but lots of emotion) but the legal complaint is even more evocative.

What Causes Heart Attacks? (continued)

Seth Roberts16 Comments

Uffe Ravnskov, a Swedish doctor, wrote a paper titled “Is atherosclerosis caused by high cholesterol?” (An admirably clear title.) His answer was no. He submitted it to a medical journal. One of his empirical points was that there was no relationship between cholesterol level and atherosclerosis growth. One reviewer commented:

Lack of relationship can be explained by more factors that only absence of it: small numbers, incorrect or indirect measurements of variables of interest, imprecision in measurement, confounding factors, etc.

To which Ravnskov replied:

If it is impossible to find exposure-response between changes of blood cholesterol and atherosclerosis growth in 22 studies including almost 2500 individuals a relationship between the two, if any, must be trivial.

Which sounds reasonable. But an even larger number of clinical trials failed to find clear evidence that omega-3 supplementation reduces heart disease. Yet I am sure that, with a large enough dose, it does.

Most people believe clinical trials, which are usually double-blind when possible and placebo-controlled. “The gold standard,” they are called. Science writer Gary Taubes, for example, believes them: When the results of a clinical trial contradicted a survey result, he believed the clinical trial. His recent NY Times magazine article was based on the assumption that clinical trials are trustworthy. This is such an article of faith that he gave no evidence for it.

That the heart disease clinical trials failed to clearly show benefits of omega-3 supplementation had large and unfortunate consequences. Not only because heart disease is the leading cause of death in many places, including America, but also because I am sure proper omega-3 supplementation would reduce many other problems, including falls, memory loss, gum disease, and other diseases of too much inflammation.

I don’t know why the big clinical trials failed to point clearly in the right direction. I can think of several possibilities:

1. Too large. Hard to control quality — verify data, for example. People near the bottom doing the work have little stake in accuracy of the outcome.

2. Poor compliance. If you are taking the placebo, why bother? And the odds are fifty-fifty you are. Lots of people have trouble following SLD, which obviously works.

3. Degradation. My belief that omega-3 is powerful comes from experiments (mine) and examples involving flaxseed oil. Flax grows at room temperature. The heart disease studies used fish oil; fish live in cold water. The omega-3 fats in fish oil may degrade at room temperature. The omega-3 fat in flaxseed oil may be far more stable at room temperature.

4. Wrong dose. Self-experimentation made it easy for me to figure out the correct dosage. People studying heart disease had no similar data to guide them. They could not realistically expect people to consume as much fish oil as the Eskimos whose rate of heart disease was so low.

5. Too sure. Self-experimentation encourages skepticism about one’s results because new experiments are easy to do. If I can think of reasons to doubt my results so far, that’s a good excuse for a new experiment. The more experiments the better. Each one is easy; I just need a good story line, a good reason for each one. Whereas if you are doing an experiment that cannot be repeated, any skepticism about it — e.g., about accuracy of measurements — is discouraged: It would cast doubt on the whole enterprise.

How Much Water Should You Drink?

Seth Roberts4 Comments

According to this persuasive non-embeddable video — from a BBC series called The Truth About Food — the answer is don’t worry about it.

They compare two twins. One drinks 2 liters water/day, the other doesn’t drink any water. Not self-experimentation, but close.

I did an experiment in which I drank 5 liters of water/day. I lost a few pounds, not nearly worth the trouble. There was one surprise: Flavors intensified. Every strawberry was the best-tasting strawberry I’d ever had.

What Do Meatloaf, Acupuncture, Psychotherapy, and Clinical Trials Have in Common?

Seth RobertsLeave a comment

Jane Jacobs tells a story about a handed-down meatloaf recipe: After the loaf is made, the end is cut off. “Why?” she asked. “We’ve always done it that way,” she was told. The original recipe was for a smaller oven, it turned out; the end was cut off to make the loaf fit in the oven.

I thought of this story when I read a recent study in the Annals of Internal Medicine that compared three treatments for back pain: acupuncture, “sham acupuncture,” and “conventional therapy.” Sham acupuncture was like acupuncture except that the needles were put in “wrong” places, inserted less deeply, and not rotated after insertion. Conventional therapy was drugs, physical therapy, and exercise. The study found that acupuncture and sham acupuncture were equally effective. Both were much better than conventional therapy. The results imply that acupuncture works, but the surrounding theory (meridians, etc.) is wrong. Which I find reassuring.

Psychotherapy is essentially the same. Lots of studies show that psychotherapy helps — but many studies also imply that the surrounding theory is wrong. Untrained therapists are as effective as trained therapists. Keeping a journal has similar effects. The active ingredient may be telling your problems, just as the active ingredient of acupuncture is apparently needle insertion.

Ritual — doing something just because — can be found in meatloaf recipes, acupuncture, psychotherapy, and clinical trials. In the discussion section of the acupuncture paper, the authors wrote:

Potential limitations of this study [include] inability to blind acupuncturists to the form of acupuncture.

Just as the meatloaf cooks did not understand their recipe, the acupuncture researchers did not understand their research design. The original reason for blinding was to equate expectations. That the two forms of acupuncture came out equal in spite of unequal expectations among the therapists is better evidence that expectations were not important. The authors failed to grasp that lack of blinding worked in their favor.

Thanks to Hal Pashler.

How Accurate is Epidemiology? (part 4)

Seth RobertsLeave a comment

In Sunday’s NY Times Magazine, Gary Taubes argued that epidemiology does not provide a good basis for health decisions — it is often wrong, he claimed. By “wrong” he meant experiments were more pessimistic. Things that seemed to help based on surveys turned out not to help, or help much less, when experiments were done. A 2001 BMJ editorial disagrees:

Randomized controlled trials and observational studies are often seen as mutually exclusive, if not opposing, methods of clinical research. Two recent reports, however, identified clinical questions (19 in one report, five in the other) where both randomized trials and observational methods had been used to evaluate the same question, and performed a head to head comparison of them. In contrast to the belief that randomized controlled trials are more reliable estimators of how much a treatment works, both reports found that observational studies did not overestimate the size of the treatment effect compared with their randomized counterparts. . . . The combined results from the two reports indeed show a striking concordance between the estimates obtained with the two research designs. . . . The correlation coefficient between the odds ratio of randomized trials and the odds ratio of observational designs is 0.84 (P<0.001). This represents excellent concordance.

Here is the data:

experiment vs observation

The correlation coefficient is the wrong statistic. They should have reported the slope of a line through the points constrained to have intercept = 0. The graph above shows that the slope of such a line would be close to 1. Unlike the correlation, that is relevant to their main question — whether surveys tend to find larger risk ratios than experiments.

Part 1. Part 2. Part 3.

Addendum: A later (2005) paper by John Ioannidis, one of the authors of the 2001 paper, claims to explain, in the words of its title, “why most published research findings are false.” The above data suggest that most published research findings in Ioannidis’s area are accurate. Alex Tabarrok on the 2005 paper.

Spider Science

Seth Roberts3 Comments

The success of my self-experimentation has puzzled me. The individual discoveries (a new way to lose weight, a new way to improve mood, sleep-related stuff, the fast effects of omega-3) seem normal; someone would have found them. It’s their combination that’s strange. Scientists who study weight control do not discover anything about mood, for example. But I did.

An ancient (2001) essay by Paul Graham is about how the future lies with web-based applications. No more Microsoft Word. One of Graham’s stories sheds light on my puzzle:

I studied click trails of people taking the test drive [of Graham’s web-based application] and found that at a certain step they would get confused and click on the browser’s Back button. . . . So I added a message at that point, telling users that they were nearly finished, and reminding them not to click on the Back button. , . . The number of people completing the test drive rose immediately from 60% to 90%. . . . Our revenue growth increased by 50%, just from that change.

I studied click trails. He examined a rich data set, looking for hypotheses to test. I practiced what I’ll call spider science: I waited for something to happen. When it did, I started to study it, just as a spider moves to the part of the web with the fly. Here are examples:

1. A change in what I ate for breakfast caused me to wake up early much more often. I did many little experiments to find out why.

2. Watching TV early one morning seemed to have improved my mood the next day. This led to a lot of research to figure out why and how to control the effect.

3. After I started to stand more, my sleep improved. I made many measurements to see if this was cause and effect and if so what the function looked like (the function relating hours of standing to sleep improvement).

4. In Paris I lost my appetite. This started the research that led to the Shangri-La Diet.

5. The morning after I took some omega-3 capsules, my balance improved. This led to experiments to see if it was cause and effect and if so what the function (balance vs. amount of omega-3) looked like.

6. One day I took flaxseed oil at an unusual time. My mental scores suddenly improved. I started to study these short-term effects.

7. While studying these short-term effects, I noticed improvements shortly after exercise. I started to study the effect of exercise.

Graham studied click trails partly because he could so easily act on anything he learned, partly because it was his company and he was so committed to its success. The seven examples I have given all came about partly because I could easily act on what I noticed and partly because I would directly benefit from learning more.

Conventional scientists do not practice spider science. They do not continuously monitor or search out large rich data sets hoping to find something they can act on. They can’t afford to, it’s unconventional, it’s too risky, it won’t pay off soon enough, they probably couldn’t act on what they found, etc. Later in Graham’s essay he marvels that big companies develop any software at all. Microsoft is like “a mountain that can walk.” Likewise, I’m impressed that scientists operating under the usual constraints manage to discover anything. You might think tenure allows them to relax, wait, take chances, and do things they weren’t trained to do, but it doesn’t work out that way.

How Accurate is Epidemiology? (part 2)

Seth Roberts6 Comments

Because Gary Taubes is probably the country’s best health journalist, his article in today’s NY Times Magazine (”Do We Really Know What Makes Us Healthy?”) about the perils of epidemiology especially interested me. It’s the best article on the subject I’ve read. He does a good job explaining what’s called the healthy-user bias — people who take Medicine X tend to make other healthy choices as well. Does wine reduce heart attacks? Well, probably — but people who drink more wine also eat more fruits and vegetables.

The article falls short in two big ways. Taubes does a terrible job presenting the case for epidemiology. He mentions the discovery that smoking causes lung cancer but then disparages it by quoting someone calling it “turkey shoot” epidemiology. Actually, that discovery did more for public health than any clinical trial or laboratory experiment I can think of. Taubes fails to mention the discovery that too-little folate in a pregnant woman’s diet causes neural-tube and other birth defects. As the dean of a school of public health put it in a talk, that one discovery justified all the money ever spent on schools of public health (where epidemiology is taught). Taubes also fails to mention that some sorts of epidemiology are much less error-prone than the studies he talks about. For example, a county-by-county study of cancer rates in the United States showed a big change across a geological fault line. People on one side of the line were eating more selenium than people on the other side. Experiments have left no doubt that too-little selenium in your diet causes cancer.

Even worse, Taubes shows no understanding of the big picture. Above all, epidemiology is a way to generate new ideas. Clinical trials are a way to test new ideas. To complain that epidemiology has led to many ideas that turned out to be wrong — or to write a long article about it — is like complaining that you can’t take a bike on the highway. That’s not what bikes are for. If only 10% of the ideas generated by epidemiology turn out to be correct, well, 10% is more than zero. Taubes should have asked everyone he interviewed “Is there a better way to generate new ideas?” Judging from his article, he asked no one.

Now excuse me to take a selenium pill . . .