Category: scientific method

How Effective are Flu Vaccines?

Seth Roberts4 Comments

An article in The Atlantic, based on research by Lisa Jackson, questions the conclusion that flu vaccines work. Here is the essence of her argument from a letter to the editor by Jackson and others in The New England Journal of Medicine:

In an 8-year study of a similar population of members of a health maintenance organization, we found risk reductions among vaccinated elderly persons during the influenza season to be essentially identical to those reported by Nichol et al. (Table 1).1 However, we also found even greater reductions before the influenza season.

Emphasis added. The lack of specificity suggests that those who get vaccinated are in better health to begin with than those that don’t. Other comparisons supported this conclusion.
Thanks to JR Minkel.

Interview with Professor David Jentsch about Not Taking Drug Company Money

Seth Roberts3 Comments

Dr. J. David Jentsch is a professor of psychology at UCLA; his research area is psychopharmacology. I contacted him because Aaron Blaisdell told me that he had decided to stop accepting research money from drug companies. This is unusual; I wondered why.

1. What is your research about? What portions of it have been funded by drug-company money?

My own research over the past 12 years has focused on the etiology of mental disorders (how genetic factors influence brain chemistry and behavioral functions) and how psychoactive substances work to normalize behavior through working on those very pathophysiological mechanisms. In particular, I study the brain systems and molecular pathways in control of cognitive functions, with a very specific focus on using that knowledge to generate insights about cognitive enhancement for schizophrenia, addictions and AD/HD. I study rodents and primates.

I have received funds from drug companies for two reasons. 1) The companies appreciated my work and funded efforts to discover new mechanisms that might inform what they ultimately did. 2) The companies provided funds to my laboratory so that I could investigate how novel potential candidate mechanisms that they developed influence cognition in laboratory models.

When one does work like I do, one wants to know that information learned is moving from the bench to the real world. That always requires a connection to a drug company — they make drugs/universities do not. That being said, I’ve always been of the opinion that having the best and most rigorous academic labs undertake these collaborations is in everyone’s best interest (the quality of the work is ensured). In my case, this was always a tiny part of what I did; therefore, the quality was good, my objectivity was unquestionable and the answers were certain.

Because top scientists are increasingly withdrawing from collaborative partnerships (in part because of the negative attitudes about them), this work gets left to less competitive scientists whose objectivity may be less clear because they rely upon this type of support more heavily. I think that is quite unfortunate.

2. How does one get drug-company money for research?

Generally speaking, a company representative approaches you because of your reputation and invites you to propose a study to accomplish a mutual goal (see my answer to #1 above). A study design is drawn up, circulated and discussed and finally approved.

3. How much easier is it to get drug-company money than to money from other sources (for the same research)?

It’s hard to say. Fewer people receive drug company funds. If a company is interested in your work and approaches you, it’s not that difficult to obtain the funds. But it is difficult to be recognized to do this kind of work.

4. When did you start getting drug-company money for your research? If you’re comfortable saying how much it has been over the years (per year), that would help clarify the implications of your decision.

As a graduate student, the laboratory in which I trained participated in some studies. As a faculty member myself, I have participated in two such efforts. The total amount of funding I have received from pharmaceutical companies in all my years at UCLA (a total of 8 years) is less than the budget I obtain in a single year on my RO1 grant. It is not an immense amount, and it certainly is not the kind of funding that I would need to sustain my research program.

Because of the negative perception of these sorts of activities, it is not worth continuing to engage in them. I don’t require those funding sources. That being said, I find it a bit unfortunate. Again, it’s in everyone’s best interest if the TOP scientists did those collaborations to ensure their quality and rigor. When I don’t do them, it is possible that a less objective party does. Second, every concept I have about novel treatments that isn’t pursued because of lack of such a relationship is a potential delay in moving basic science to real use.

5. What are some examples of how the animal-rights activists publicized and complained about your use of drug-company money?

After the bombing [his car was bombed in March 2009], statements were made on the web and in the press by animal rights groups saying that people such as me used animals needlessly in a drug-company-fueled manic process of animal killing in order to get rich. As I already mentioned, this is not the case, if only because people like me often have relatively few such grants, and their size is not large (again, usually not larger than a single year of funding on an RO1 grant). Because of this, I simply decided not to take any such grants in the future.

6. The car bombing (on top of other attacks) led to the decision to stop taking drug company money?

As you can discern from the fact that I only have accepted two such awards in 8 years, I already placed a good number of criteria on accepting them. I wanted them to be only projects that I considered to be of very high scientific merit, and I wanted them to be logically and obviously related to our broader research projects.

Additionally, there is already a good deal of “negative perception” of research funded by drug companies within academic circles, and so I had already batted around the question in my mind about whether I should accept further awards. When the extremist attack on me happened in March of this year (2009), I had not had such an award in some time. That was not because I had taken a decision about the matter – simply that I hadn’t found a situation I wanted to pursue. At that point, the decision solidified.

7. Your decision to not take drug company money — what effect do you think it will have or hope it will have?

I am certain a situation will arise where I will have an idea about a novel therapeutic based upon my research that I will be unable to pursue without such a relationship to a company. What is more, the compounds in development by companies are not being evaluated by me, so they may well be evaluated by someone with a little bit less rigor and objectivity.

I believe strongly that the academic enterprise gives a crucial “objective” check on novel therapeutics when leading scientists who are not “dependent” on drug company money examine them. The alternative is that others who are more dependent, and therefore less objective, will do it.

Secrets of Infomercials

Seth Roberts11 Comments

Here is a long list of reasons, by Steve Dworman, who makes them for a living, why infomercials are the way they are. One big reason is data: you can easily do an experiment that compares two different versions of the same commercial. It is much harder to measure the effectiveness of other forms of advertising. (The lack of data involved in most advertising choices is easy to see on Mad Men.) Self-experimentation has the same advantage: It’s so much easier to test an idea.

One of his points is about the use of celebrities: It must work, or else they wouldn’t do it. (Because there is data behind how things are done.) I think this points to something hard-wired: We want to learn from other people. That’s the default. If we have a question, we search for someone who will answer it. Learning from our own experience — such as self-experimentation — is a last resort. It feels wrong, we don’t like it. I remember feeling this way when I bought a camera. Sure, I could do extensive research about which camera is best. But that would be hard. Better to ask a friend. And then the purchase would be a link between us.

Modern Biology = Cargo-Cult Science?

Seth Roberts15 Comments

At first I thought the title of this article was “Taking Back The Nobel Prizes”. My eyes widened. Someone at the New York Times has a radical thought, it appeared. I was wrong. The title is “Taking Back Nobel Prizes”; the article is about the less-than-radical idea that Henry Kissinger did not deserve a Peace Prize. Then I thought it was too bad that Richard Feynman isn’t alive. If he were, I would ask him if modern biology — the sort that wins Nobel Prizes — is an example of what he called cargo-cult science in a famous graduation speech. I would be a good person to ask that question, I thought, because he considered rat psychology cargo-cult science. Yet I used rat psychology to come up with the Shangri-La Diet, which has helped many people lose weight in counter-intuitive ways.

Cargo-cult science, according to Feynman, was activities that have the superficial trappings of science but don’t actually accomplish anything. You do all the right things, or so you think, but the planes don’t land. The sort of biology that wins Nobel Prizes has a long history of this. This year’s prize went to research that found that telomeres shorten with age. The press release, forced to say how this is useful (the Nobel Prize is supposed to be for research that benefits mankind), says

These discoveries had a major impact within the scientific community. Many scientists speculated that telomere shortening could be the reason for ageing, not only in the individual cells but also in the organism as a whole. But the ageing process has turned out to be complex [shocking!] . . . Research in this area remains intense.

. . . It was therefore proposed that cancer might be treated by eradicating telomerase. Several studies are underway in this area, including clinical trials evaluating vaccines directed against cells with elevated telomerase activity.

Some inherited diseases are now known to be caused by telomerase defects, including certain forms of congenital aplastic anemia, in which insufficient cell divisions in the stem cells of the bone marrow lead to severe anemia. Certain inherited diseases of the skin and the lungs are also caused by telomerase defects.
In conclusion, the discoveries by Blackburn, Greider and Szostak have added a new dimension to our understanding of the cell, shed light on disease mechanisms, and stimulated the development of potential new therapies.

Shameless. Note the utter absence of even one disease in one person cured or prevented. Not one. And this is supposed to be the most beneficial discovery in medicine. It’s the top prize in medicine and biology! Last year the prize was given for HIV. Do we have an HIV vaccine? No. The year before that, HPV. Do we have an HPV vaccine? No. A few years before that, the discovery that a certain bug “causes” stomach ulcers — the award that showed that the medical community and the Nobel Prize committee have a weak grasp of the concept of causality. The biologists think they do everything right — but the planes don’t land. The biologists who do this research aren’t able to solve actual problems. (Some people do — those who discovered that smoking causes cancer, for example — but they don’t get Nobel Prizes.) Could something important be missing from their view of the world? I think so.

Cargo-cult activities aren’t worthless, so long as you learn from your mistakes. The cargo cultists could see that the planes didn’t land and eventually figure out that something was missing. That’s actual knowledge, humble but useful. Feynman’s criticisms of rat psychology were reasonable. Those doing rat psychology learned from their mistakes, I think, and eventually the field improved and produced the research behind the Shangri-La Diet. Modern biology isn’t worthless, just as cargo cults aren’t worthless. Obviously “useless” knowledge can eventually become useful, as has happened many times. But these overblown claims for the value of modern biology truly cost the rest of us — a great deal, I believe. Because the first step in getting somewhere, as Feynman liked to say, is to confront reality. At least in their public statements about the value of their research, modern biologists are living in a dream world. It’s always “potential” this and “future” that and “insight into disease mechanisms” — without ever curing or preventing a disease.

Thanks to Eric Meltzer.

Exercise and Its Confounds: The London Bus Study

Seth Roberts4 Comments

The Financial Times recently ran an article about Jerry Morris, a London epidemiologist who did the most famous study of the effect of exercise. He compared London bus drivers with the ticket takers on the same buses. The ticket takers got a lot more exercise than the drivers. The health differences between them were attributed to exercise:

“There was a striking difference in the heart-attack rate. The drivers of these double-decker buses had substantially more, age for age, than the conductors [= ticket takers].” [said Morris]

The data were so telling because drivers and conductors were men of much the same social class. There was only one obvious difference between them. “The drivers were prototypically sedentary,” explains Morris, “and the conductors were unavoidably active. We spent many hours sitting on the buses watching the number of stairs they climbed.” The conductors ascended and descended 500 to 750 steps per working day. And they were half as likely as the drivers to drop dead of a sudden heart attack.

Passengers entering a London bus in the 1940s
Morris found that bus conductors had fewer heart attacks than sedentary drivers

Today, almost everyone understands that physical exercise can help prevent heart disease, as well as cancer, diabetes, depression and much else besides. But on that day in 1949 when Morris looked at the bus data, he was the first person to see the link. He had inadvertently — “mainly luck!” — “ stumbled on a great truth about health: exercise helps you live longer.

It’s not that simple. There are two big confounds in the study (two other differences between drivers and ticket takers) that surely caused Morris to overestimate the benefits of exercise. One is well-known to epidemiologists: Bus driving is very stressful. Much more stressful than a dozen other equally sedentary jobs. Stress certainly causes heart disease. The other is based on my discovery that standing a lot improves sleep. (The standing needn’t involve movement.) I don’t know if better sleep specifically reduces heart disease but it certainly increases resistance to infection and heart disease seems to have an infectious component. The ticket takers were on their feet all day, the drivers were not.

You may remember that James Fixx, a famous advocate of jogging, died of a heart attack.

Thanks to Dave Lull.

Easy versus Hard: Hunting, Agriculture, Etc.

Seth Roberts10 Comments

Coming across this sentence

The more intensive the agricultural system, the more work required for a unit of food.

in Charles Maisel’s The Emergence of Civilization (1990, p. 35) made me think for a while and make a list:

  1. Hunting: Easy.
  2. Agriculture: Hard. In agriculture you have to start from scratch in a way you don’t when hunting.
  3. Self-Experimentation: Easy.
  4. Ordinary Science: Hard. It is much harder to discover something useful via ordinary science than via self-experimentation.
  5. Fermentation: Easy. It is easy to make yogurt or kombucha, for example.
  6. Medical Drugs: Hard. Hard to invent, hard to make, hard to sell, hard to get, hard to afford, not to mention dangerous. It is much easier to cure/prevent problems by eating fermented foods, such as yogurt.

What’s interesting is the starkness of the differences. Hunting and agriculture are two answers to the same question. I suppose we backed into agriculture because we over-hunted. In the other two pairs, I think the basic Veblenian dynamic was/is at work: The more useless, the more high status. Scientists must be elaborately theoretical and high-techy and wasteful to be high-status. Likewise with home remedies (such as fermented food) versus medical drugs: To be high-status, doctors had to promote elaborate, obscure, hard-to-get remedies.

John Tukey and GPS

Seth Roberts2 Comments

In this amusing article Emily Yoffe tells about her troubles with GPS. She fails, unfortunately, to look on the bright side — to say how flawed GPS is better than no GPS. After a talk by John Tukey, the statistician, at Berkeley, I told him that I had found the tools he wrote about in Exploratory Data Analysis to be really helpful. (For example, smoothing my data led me to discover that eating breakfast made me wake up too early.) Tukey replied that if the tools are helpful half the time, that’s good. It isn’t easy to make an interesting response to a compliment!

Something is better than nothing.

How Much Should We Trust Clinical Trials?

Seth Roberts7 Comments

Suppose you ask several experts how to choose a good car. Their answers reveal they don’t know how to drive. What should you conclude? Suppose these experts build cars. Should we trust the cars they’ve built?

Gina Kolata writes that “experts agree that there are three basic principles that underlie the search for medical truth and the use of clinical trials to obtain it.” Kolata’s “three basic principles” reveal that her experts don’t understand experimentation.

Principle 1. “It is important to compare like with like. The groups you are comparing must be the same except for one factor — the one you are studying. For example, you should compare beta carotene users with people who are exactly like the beta carotene users except that they don’t take the supplement.” An expert told her this. This — careful equation of two groups — is not how experiments are done. What is done is random assignment, which roughly (but not perfectly) equates the groups on pre-experimental characteristics. A more subtle point is that the X versus No X design is worse than a design that compares different dosages of X. The latter design makes it less likely that control subjects will get upset because they didn’t get X and makes the two groups more equal.

Principle 2. “The bigger the group studied, the more reliable the conclusions.” Again, this is not what happens. No one with statistical understanding judges the reliability of an effect by the size of the experiment; they judge it by the p value (which takes account of sample size). The more subtle point is that the smaller the sample size, the stronger the effect must be to get reliable results. Researchers try to conserve resources so they try to keep experiments as small as possible. Small experiments with reliable results are more impressive than large experiments with equally reliable results — because the effect must be stronger. This is basically the opposite of what Kolata says.

Principle 3. In the words of Kolata’s expert, it’s “Bayes theorem”. He means consider other evidence — evidence from other studies. This is not only banal, it is meaningless. It is unclear — at least from what Kolata writes — how to weigh the various sources of evidences (what if the other evidence and the clinical trials disagree?).

Kolata also quotes David Freedman, a Berkeley professor of statistics who knew the cost of everything and the value of nothing. Perhaps it starts in medical school. As I blogged, working scientists, who have a clue, don’t want to teach medical students how to do research.

If this is the level of understanding of the people who do clinical trials, how much should we trust them? Presumably Kolata’s experts were better than average — a scary thought.

The Pashler-Roberts Law: Expense versus Honesty

Seth Roberts3 Comments

In this post Andrew Gelman comments on my recent post about acne self-experimentation. He makes an excellent point about drug-company studies:

How would you want to evaluate the risks and effectiveness of a new drug that was developed by a pharmaceutical company at the cost of millions of dollars? I’d be suspicious of an observational study: even if conducted by professionals, there just seem to be too many ways for things to be biased.

Right. And it’s not just observational studies. The data from any big study can be analyzed many ways. The more at stake, the greater the chance of what Andrew calls bias and I call making choices that favor the result you prefer. Independently of Andrew, Hal Pashler and I came up with what I call the Pashler-Roberts Law: The more expensive the research, the less likely the researchers will be honest about it.

You may remember that Robert Gallo, the AIDS researcher, did very expensive research. The deception (possibly self-deception) that accompanied very expensive fusion research is described in Charles Seife’s Sun in a Bottle: The Strange History of Fusion and the Science of Wishful Thinking (2008).

As Andrew says, this is a big virtue of self-experimentation. Because it’s free, it’s easy to be honest, especially about failure. The cheaper the better is a broad truth about science that’s hard to learn from books or classes or even talking to scientists.

Self-Tracking: What I’ve Learned

Seth Roberts10 Comments

I want to measure, day by day, how well my brain is working. After I saw big fast effects of flaxseed oil, I realized how well my brain works (a) depends on what I eat and (b) can change quickly. Maybe other things besides dietary omega-3 matter. Maybe large amounts of omega-6 make my brain work worse, for example. Another reason for this project is that I’m interested in how to generate ideas, a neglected part of scientific methodology. Maybe this sort of long-term monitoring can generate new ideas about what affects our brains.

So I needed a brain task that I’ll do daily. When I set out to devise a good task, here’s what I already knew:

1. Many numbers, not one. A task that provides many numbers per test (e.g., many latencies) is better than a task that provides only one number (e.g., percent correct). Gathering many numbers per test allows me to look at their distribution and choose an efficient method of combining (i.e., averaging) them into one number. (E.g., harmonic mean, geometric mean, trimmed mean.) Gathering many numbers also allows me to calculate a standard error, which helps identify unusual scores.

2. Graded, not binary. Graded measures (e.g., latencies) are better than binary ones (e.g., right/wrong).

Every experimental psychologist knows this. What none of them know is how to make the task fun. If I’m going to do something every day, it matters a great deal whether I enjoy it or not. It might be the difference between possible and impossible. People enjoy video games, which is a kind of existence proof. Video games have dozens of elements; which matter? Here’s what I figured out by trial and error:

3. Hand-eye coordination. Making difficult movements that involve hand-eye coordination is fun. My bilboquet taught me this. Presumably this tendency originated during the tool-making hobbyist stage of human evolution; it caused people to become better and better at making tools. Ordinary typing involves skilled movement but not hand-eye coordination. This idea has worked. I led me to try one-finger typing (where I look at the keyboard while I type) instead of regular typing. And, indeed, I enjoy the one-finger typing task, whereas I didn’t enjoy the ordinary typing tasks I’ve tried.

4. Detailed problem-by-problem feedback. Right/wrong is the crudest form of feedback; it doesn’t do much. What I find is much more motivating is more graded feedback based on performance on the same problem.

5. Less than 5 minutes. The longer the task the more data, sure, but also the more reluctant I am to do it. Three minutes seems close to ideal: long enough for the task to be a pleasant break but not so long that it seems like a burden.

Experimental psychology is a hundred years old. Small daily tests is an unexplored ecology that might have practical benefits.