In The Death and Life of Great American Cities, Jane Jacobs wrote, “Old ideas can use new buildings, but new ideas need old buildings.” New ideas need old buildings because old buildings are cheap. New ideas cannot be expected to be especially profitable, or profitable at all, at first. This is why self-experimentation should have a permanent place in the ecology of science: It provides a cheap way to develop new ideas.
Category: self-experimentation
Science in Action: Omega-3 (materials)
Self-experimentation is for everyone just as blogging is for everyone. You don’t need fancy equipment to be an interesting blogger and you don’t need fancy equipment to do interesting self-experimentation — at least I think my recent omega-3 results are interesting. Here are the easy-to-find materials I used:
1. To measure my balance, I used a bamboo cutting board ($15), a 0.5-inch pipe plug ($1), a booklet with about 20 pages (free), and a stopwatch ($10). Here they are:
This shows the size of the pipe plug:
I put the booklet on the floor and put the pipe plug in a cut-out hole in its center. The hole goes about halfway into the booklet (e.g., 10 pages out of 20).
Then I balance the cutting board on top of the pipe plug:
I stand on my right foot on the cutting board; the measure is how long I can balance on it before my left foot touches the floor.
2. To vary the amount of omega-3 in my brain, I used these oils:
Walnut oil and flaxseed oil are high in omega-3 fats; sesame oil is low.
If you are interested in doing similar experiments, feel free to contact me.
Science in Action: Omega-3 (background)
The omega-3 story began with the circulatory system. In the 1960s, two Danish scientists wondered why Eskimos rarely die of heart disease. Could the answer explain the sharp decrease in heart disease mortality in Norway during World War II? In spite of this promising beginning, the heart and mortality benefits are still not clear. A 2006 meta-analysis of heart disease studies concluded that “omega 3 fats do not have a clear effect on total mortality, combined cardiovascular events, or cancer.”
You can find lots of recommendations to consume omega-3 fats in various forms — fish, supplement, and so on. On the other side, Marion Nestle, the author of What To Eat, seems to believe the advantages claimed for omega-3 are “ hype.” Most researchers are less certain. From a recent New York Times article about Martek, a company that makes an omega-3 food supplement:
“A lot of the claims made for DHA [a form of omega-3] are in the realm of hypotheses,” said David Schardt, senior nutritionist at the Center for Science in the Public Interest, an advocacy organization based in Washington. “They are certainly worth pursuing, but there’s not yet enough proof to warrant telling people to go out of their way to take DHA.”
The exceptions, Mr. Schardt said, are people with a history of heart disease and premature infants, who need an extra boost of DHA for proper brain and eye development to compensate for their early exit from the womb.
Martek’s scientists, when pressed, generally agreed with Mr. Schardt. The data showing any health benefits of DHA beyond those related to the heart or premature infants, while encouraging, is not quite conclusive, they say.
The typical experimental study of omega-3 takes two groups of people with a pre-existing problem, gives one group omega-3 and the other group a placebo, and measures outcomes several months later. A 2005 study in Pediatrics, for example, compared two groups of children (n = about 60/group) with Developmental Coordination Disorder. Most of them had ADHD. One group was given an omega-3 supplement; the other group was given a placebo. The children were tested before treatment and after three months of treatment. (The reading, spelling, and behavior scores of children in the supplement group improved more than the scores of children in the placebo group.) Studies like this are hard.
In summary, there is considerable uncertainty about the effects of omega-3; and the methods used to reduce that uncertainty are slow and difficult. This is why self-experimentation might help.
My recent data. The Queen of Fats (2006) by Susan Allport, a science writer, is an excellent introduction to the subject.
Science in Action: Omega-3 (balance results)
Because many SLD dieters reported better sleep, I wondered if omega-3 improved sleep. I increased my omega-3 intake by switching from olive oil, which has little omega-3, to walnut oil and flaxseed oil, which have much more — especially flaxseed oil. The amount of oil stayed roughly the same. The night after the change, my sleep got better. To my surprise, so did my balance. The next morning, I found I could more easily put on my shoes while standing up. I had been putting on my shoes standing up for 2-3 years and it had never been this easy. (I put on my shoes standing up because I thought it might improve my balance.)
I devised a simple measure of balancing ability. I stood on one foot on a platform balanced on a small metal cylinder (a pipe plug). (I will post pictures.) The parts were easy to find. I tried cylinders of different sizes until the balancing was neither too easy nor too hard. The measure was how long I could stand on one foot on the platform, which measured with a stopwatch. I made these measurements in blocks of 20 (the first 5 were warmup, leaving 15).
My early attempts had two problems: (1) The dose was too low. I had been taking the flaxseed oil as capsules (10 1000-mg capsules/day). I started taking 1 T/day in liquid form (much faster). Then I increased the amount of flaxseed oil/day from 1 T to 2 T. My sleep improved: I woke up more rested. Because the sleep effect was now perfectly clear, I thought measuring the effect on my balance would be a good idea. (2) Practice effects were too large. How well I could balance depended on how often I measured my balance. To avoid practice effects, I measured my balance no more than once/day.
I did a baseline period of several days; then I replaced the walnut oil and flaxseed oil with the same volume of sesame oil, which is low in omega-3. I continued this period until the effects seemed beyond doubt. Then I did another baseline period with the original amounts of walnut and flaxseed oil.
Here are the balance results. Each point is a geometric mean over 15 trials. The bars are standard errors. After one day, my balance got worse with sesame oil. When I returned to the high-omega-3 oils, my balance returned to its baseline level. To measure the clarity of the effect, I compared the 17 baseline days with the last 4 sesame-oil days. This gave t (19) = 4.1. A very clear effect.
I made this graph in a cafe. The person sitting next to me asked what I was working on. I showed her the graph. I explained that I measured my balance as a way of measuring how well my brain was working. The results suggested that the type of fat in my diet affected how well my brain worked. She said the results were very interesting because most people will have diets closer to sesame oil than walnut oil and flaxseed oil. Many people will be interested in these results, she said. I hope so, I said.
I will post later on the background of these results, the questions they raise, and procedural details. If you can’t wait, read the posts in the omega-3 category. If you are interested in doing a similar experiment, please let me know.
Yet More about Omega-3
Perhaps inspired by USA Today, the New York Times discusses DHA, an omega-3 fat sold as a food additive. “Magical or overrated?” is the question posed by the headline. According to Marion Nestle, overrated:
“My experience in nutrition is that single nutrients rarely produce miracles,” said Marion Nestle, a professor in the department of nutrition, food studies and public health at New York University and the author of “What to Eat,” published last year. “But it’s also been my experience that companies will put anything in their food if they think the extra marketing hype will help them sell more of it.”
Single nutrients rarely produce miracles? There is a long history in nutrition of just that: The story of the discovery of vitamins. One single-nutrient miracle after another. Given that history, the claims for omega-3 are plausible. If Nestle has an alternative explanation for the many results that point to the benefits of omega-3, that would be interesting to hear. It wasn’t provided in the article. “Companies will put anything in their food if they think the extra marketing hype will help them sell more of it”? Well, B vitamin supplementation of flour has cut the rate of neural tube birth defects roughly in half, a huge benefit, a huge amount of averted misery. Given that success, it is reasonable to think that other supplementation might also be helpful — to everyone. I discuss derogatory treatment of food companies (”will put anything in their food if . . . hype will help them sell more of it”) in the last chapter of The Shangri-La Diet. Curiously enough, Jane Jacobs once said, you can only change something if you love it.
I have done more self-experimentation about omega-3s and will describe the results in a week or two. Previous posts about omega-3 here, here, here, here, here, here, here, here, here, and here
Robin Hanson on Web Trials
I recently asked Robin Hanson, a professor at George Mason University, what he thought of web trials. Web trials are a way to learn how to solve difficult health problems (e.g., acne, obesity). By web trial I mean a web-based collection of data that compares different ways of solving a problem. People with the problem would go to a website, sign up for one of the treatments, follow the directions, and report the results in a standardized format. For example, a site might compare three acne treatments (treatments that anyone can try, such as dietary changes or over-the-counter medicines). The cumulated results would gradually show which treatment works best — a thousand times more efficiently (sooner, cheaper, more easily) than a clinical trial (which no one would finance because there is no profit to be made). Web trials are halfway between clinical trials and the data collection now going on at the Shangri-La Diet (SLD) forums, where people post their progress on SLD.
I asked Robin because he has pioneered a similar improvement: Prediction markets are often far better than what they replace. And his core political affiliation is “I don’t know.”
Here is a summary of what he said.
1. A selection effect is a big concern. Do people wait to report back until after it works? There is always going to be the issue of sampling, selection bias for people who stay with it.
2. How could you get people to allow you (the website) to choose for them which treatment to do? That would be the hard thing. Perhaps the website could say: “would you like to see what our advice for you is?” At most you could get randomization for your advice.
3. It doesn’t have to be restricted to health problems. It could be used to test all sorts of advice. You could just get data about what happens when people do or don’t follow some advice — romantic advice, for example. Very rarely do we have randomization in choices. When we do, we call them natural experiments. In medicine, researchers have used practice variation (variation from one doctor to the next) to look at effectiveness.
4. Perhaps you could get people to commit to this the way they do to Wikipedia. The goal would be: Let’s understand humanity — a noble cause. Let’s be part of a grand project to do this.
Robin blogs at Overcoming Bias. Tomorrow I will comment on Robin’s comments.
Near-Celebrity Near-Endorsement
People have often told me that what the Shangri-La Diet needs is endorsement from a celebrity. I agree. From the current (2007-01-15) issue of Star magazine, here is a step in that direction. Not quite an endorsement but close; and on the same page as two celebrities.
I am giving a talk tonight (Tuesday, January 9, 7:30 pm) at PARC (Palo Alto) about self-experimentation that is open to the public. Audio and PowerPoint will be posted.
Methodological Lessons from Self-Experimentation (part 4 of 4)
6. Curiosity helps — because it provides a wide range of knowledge. Pasteur made a similar point when he said luck favors “the prepared mind” by which he meant the well-stocked mind. To come up with my theory of weight control you needed to know both obesity research and animal learning because the theory is based on basic facts about weight control and basic facts about Pavlovian conditioning. I knew the weight control facts because I had taught introductory psychology and lectured on weight control. I knew the basic facts about Pavlovian conditioning because my graduate training was in animal learning. It was unusual to know both sets of facts. Few obesity researchers knew much about animal learning; few animal-learning researchers knew much about weight control. The same thing happened with my mood research: Facts that I had learned from teaching introductory psychology showed me that my findings made sense and were important. I had taught introductory psychology because I was curious about psychology.
These two examples (weight control, mood) surprised me. I may have heard this point made a few times but I didn’t know any examples. Since then, however, I have come across examples not involving me that make the same point. Luca Turin is a biophysicist who has come up with a far better explanation of how the nose works than any previous theory. His recent book The Secret of Scent tells the story. “In order to solve the structure/odor problem,” he wrote, “you need to know at least three things: (a) biology, (b) structure and (c) odor. Each of these three things taken individually is not difficult” (p. 166). The problem had gone unsolved because no one before Turin knew all three.
7. Publish in open-access journals. Because my long self-experimentation paper was published in an open-access journal, anybody could read it within minutes. My friend Andrew Gelman blogged about it, which caused Alex Tabarrok at Marginal Revolution to mention it. This brought it to the attention of Stephen Dubner, who with Steven Levitt wrote about it in their Freakonomics column in the New York Times. That led to a contract to write two books — one about weight loss, the other about self-experimentation in general. That anyone could download my paper made it spread much faster. In the old days, with photocopies and libraries and mailed reprints . . . no talk tonight.
A summing-up, if you want to figure something out via data collection: 1. Do something. Don’t give up before starting. 2. Keep doing something. Science is more drudgery than scientists usually say. 3. Be minimal. 4. Use scientific tools (e.g., graphs), but don’t listen to scientists who say don’t do X or Y. 5. Post your results.
Read Part 1, Part 2, and Part 3. You no longer need to register to comment. My talk Tuesday night (tomorrow Jan 9) 7:30 pm at PARC (Palo Alto) is open to the public.
Methodological Lessons from Self-Experimentation (part 3 of 4)
4. There are serious defects in the way science is usually done. I found a new and powerful way of losing weight — yet I’m an outsider to that area. Although obesity is a huge problem, and hundreds of millions of dollars go into obesity research every year, I was completely outside that group of people and resources. If science is being done properly, there should be a relation between input and output — the more input, the more output. That failed here. Professional obesity researchers, given vast input, failed to discover this; whereas I, given zero input, managed to do so. You might say this was a weird fluke except the same thing happened again with mood: I discovered a powerful way of changing mood, even though I was an outsider to the study of mood. Depression is a huge problem, vast resources go into trying to do something about it.
What the serious defects are has no simple answer. After the next lesson learned I’ll try to explain what I think is wrong.
5. There are serious strengths in the way science is usually done. I relied heavily on conventional science and could never have gotten where I did without it. Ramirez and Cabanac did brilliant research. I say there are serious strengths in conventional science because I used conventional scientific methods and conclusions to find a new solution to a serious problem — obesity is a serious problem. I didn’t just use conventional scientific tools; I also used self-experimentation, which is unconventional. But self-experimentation alone wouldn’t have gotten very far, I’m sure. The turning point in my weight control research was reading a paper by Ramirez about rat experiments. Not only did I use a vast number of conclusions from conventional science, I also used conventional experimental designs and standard, common tools for data analysis, such as programs for plotting data.
To say that science is glorified common sense has a lot of truth to it. To say that science is a collection of methods to help us understand and control the world also has a lot of truth to it. But science is far more than a collection of tools; it is a whole community and culture, with beliefs as well as tools. Like any culture, many of its beliefs are based on faith.
Here is a story to illustrate what happens. It’s pure human nature. Suppose someone gave you a power saw. Your first thought is: Wow, I have a power saw. There are many things I can now do that I couldn’t do before. It seems like a pure benefit. No negatives. You learn how to use the power saw and you become better and better at using it. Eventually you start to make a living using the power saw — other people, who don’t have a power saw, pay you to saw stuff for them. You become a power-saw professional and, along with other professionals, you establish rules about how to use power saws. To save the public from bad power saw usage, you establish a licensing test to become a power-saw professional. Your view of yourself is: I know how to use a power saw. And if there is a problem to be solved, you try to solve it with your power saw — that’s what you know how to do best. All this makes perfect sense to you. Hundreds of professions have followed this path. What is hard for you to notice is that in certain ways you have become weaker — if a problem doesn’t call for power-saw usage, you are less likely to find the solution. Because you are too busy making a living using your power saw.
I hope my point is obvious. Budding scientists go to graduate school where they learn a bundle of specialized research methods that varies from one research area to the next. That is their power saw. After graduate school, they make a living using the techniques that they have learned. After graduate school, they are in better shape to make a living; but they are in worse shape to solve problems for which the techniques that they have learned are not appropriate. Conventional scientific methods could go part of the way toward finding the Shangri-La Diet; but they could not go all the way. Other techniques were needed — very simple ones, pre-power-saw. So conventional science never found it.
In Dark Age Ahead, Jane Jacobs gives another example of this. During a recent heat wave in Chicago, two nearby neighborhoods, similar in many ways, had very different death rates. A good explanation of the difference was provided by a graduate student in sociology, who used very simple very low-cost methods. In contrast, a task force of scientists from the Centers For Disease Control, with vast resources and great methodological sophistication, failed to explain the difference. They were blinded by their expertise. They failed to see that their methods weren’t working.
Methodological Lessons from Self-Experimentation (part 2 of 4)
3. Be minimal. In other words, do the easiest, simplest thing that will that will tell you something important and new, that will provide significant progress. This I learned by failure. In the early days of my self-experimentation — and of my rat research, too — I constantly tried to do experiments that broke this rule, and again and again they failed to work. (Slow learner.) The more complex the design, the more untested assumptions it makes. And untested assumptions, at least mine, are often wrong. I’ve been good about following this rule in my own research in recent years so to give an example of how it is broken I will describe someone else’s research. I sat in on a planning session for an experiment about asthma at a highly-ranked school of public health. The experiment was expensive — the grant to pay for it was many hundreds of thousands of dollars. There were to be about 50 families in the treatment group and 50 families in the control group. They had done some pilot work involving three families. They proposed to begin the full experiment. I suggested that they do a larger pilot experiment — maybe four families in each group. There were several professors and several more people with Ph.D.’s at the meeting. No one agreed with me. Several people explicitly disagreed: “I don’t think we need to do any more pilot work.” As it turned out, I was right. They began the full experiment and it failed miserably because recruitment turned out to be far more difficult than expected.
Almost all proposed research I hear about breaks this rule, which is fascinating in a train-wreck kind of way. I have never seen a book about research design that makes this point. As a result, I suspect that books about research design are often counter-productive: The student would have been better off if he or she hadn’t read them. The textbook teaches this or that complication to people who can barely do basic stuff. The poor student wastes time using complex designs that fail in cases where a simpler design would have succeeded.
Part 1 is here. Part 3 is here. You no longer need to register to comment.