The Trouble With Rigor

This is an easy question: When writing down numbers, when is it bad to be precise? Answer: When you exceed the precision to which the numbers were measured. If a number was measured with a standard error of 5 (say), don’t record it as 150.323.

But this, apparently, is a hard question: When planning an experiment, when it is bad to be rigorous? Answer: When the effort involved is better used elsewhere. I recently came across the following description of a weekend conference for obesity researchers (December 2006, funded by National Institute of Diabetes & Digestive & Kidney Diseases):

Obesity is a serious condition that is associated with and believed to cause much morbidity, reduced quality of life, and decreased longevity. . . . Currently available treatments are only modestly efficacious and rigorously evaluating new (and in some cases existing) treatments for obesity are clearly in order. Conducting such evaluations to the highest standards and so that they are maximally informative requires an understanding of best methods for the conduct of randomized clinical trials in general and how they can be tailored to the specific needs of obesity research in particular. . . . We will offer a two-day meeting in which leading obesity researchers and methodologists convene to discuss best practices for randomized clinical trials in obesity.

Rigorously evaluating new treatments”? How about evaluating them at all? Evaluation of new treatments (such as new diets) is already so difficult that it almost never occurs; here is a conference about how to make such evaluations more difficult.

This mistake happens in other areas, too, of course. Two research psychiatrists have complained that misguided requirements for rigor have had a very bad effect on finding new treatments for bipolar disorder.

More Reason to Crazy-Spice

Spices are good for you, I blogged, because they are high in antioxidants. A new study, done in Singapore with elderly subjects, supports this conclusion. It found that curry-eaters do better than others on a mental test. The abstract:

Curcumin, from the curry spice turmeric, has been shown to possess potent antioxidant and antiinflammatory properties and to reduce ß-amyloid and plaque burden in experimental studies, but epidemiologic evidence is lacking. The authors investigated the association between usual curry consumption level and cognitive function in elderly Asians. In a population-based cohort (n = 1,010) of nondemented elderly Asian subjects aged 60-93 years in 2003, the authors compared Mini-Mental State Examination (MMSE) scores for three categories of regular curry consumption, taking into account known sociodemographic, health, and behavioral correlates of MMSE performance. Those who consumed curry “occasionally” and “often or very often” had significantly better MMSE scores than did subjects who “never or rarely” consumed curry. The authors reported tentative evidence of better cognitive performance from curry consumption in nondemented elderly Asians, which should be confirmed in future studies.

Tze-Pin Ng, Peak-Chiang Chiam, Theresa Lee, Hong-Choon Chua, Leslie Lim and Ee-Heok Kua. Curry Consumption and Cognitive Function in the Elderly. American Journal of Epidemiology 2006 164(9):898-906

Brain Food (part 9: supporting data, and a problem)

I reduced the amount of omega-3 in my diet. I stopped taking flax-seed oil capsules (I had been taking 10 1000-mg capsules/day) and started drinking extra light olive oil (2 tablespoons/day) instead of walnut oil. I made the change at midnight: Tuesday high, Wednesday low. The graph below shows measurements of my balance.

From Saturday through Tuesday, and preceding days, my intake of omega-3 was high; on Wednesday and Thursday it was low.

My balance was worse Wednesday morning than expected by extrapolation, which supports the idea I started with: omega-3 affects my balance. The time course of the change (the impairment was clear in hours) resembles the original observation: I could put on my shoes while standing much more easily the morning after the day I increased my omega-3 consumption.

But, as you can see, there was a problem: My balance rapidly improved during the low omega-3 condition. Although the results support my original idea, they don’t support it as strongly as they might. A comment on a previous post was “Aren’t you worried that your expectation of worse balance will skew the results?” No, I’m not I thought when I read it. I had several reasons for not worrying about the effect of expectations, and now another has come along: Surprising results, which imply that expectations have little effect. I did not expect significant improvement from practice. I had believed that because I balance everyday for hours while standing and walking, there would not be a large practice effect. I was wrong.

Psychologists don’t know much about motor learning. There are few well-established empirical generalizations about what makes motor learning faster or slower. Another gap in our knowledge is about the nature of the underlying change. When you get better with practice, how does your brain change?

After I shifted to low omega-3, I was surprised not only by how much I improved but also by how quickly. Was my improvement due simply to more tests? I plotted my scores versus test number:

This graph suggests that I improved more per test (greater slope) during the low-omega-3 condition than during the high-omega-3 condition. I think it is a spacing effect: During the low-omega-3 condition, I tested more often. During the high-omega-3 condition, I did 14 tests in 3.5 days — 4.0/day. During the low-omega-3 condition, I did 11 tests in 1.2 days — 9.1/day. I tested more often because I wanted to track the decrease. I think this difference in test rate is the reason for the slope difference. This effect is the opposite of the usual spacing effect in learning experiments, in which close-together (”massed”) practice is less effective than widely-spaced practice.

Relevant to the theme of inspiration via self-experiment, these results and my experience gave me several new (at least to me) ideas about motor learning. One was the existence of this spacing effect. Another was that practice changes the brain by increasing how much of the brain is devoted to the task. (The areas used for other tasks shrink.) Practice increases accuracy because more neurons become involved. The output, the action, is an average from a larger sample. One reason I thought of this is that after lots of practice, and I became quite accurate, the circular area on which I was balancing seemed larger. The notion that the brain area used by the task gets larger helps explain the spacing effect. Spacing is important because the brain doesn’t care how often you have done something in the distant past; what matters is how often you are doing it now. Thus the spacing effect helps make efficient use of scarce resources (neurons). The spacing effect occurs because neural activity causes an increase in something (call it X) that slowly fades away. If later activity happens while X is above a threshold, neural rewiring occurs.

The big practice effects and the idea that practice is more powerful when more frequent should interest anyone who wants to improve their balance (and probably other motor skills), from athletes to the elderly. In a simple cheap easy safe way I got better quickly–too quickly, actually. What happened reminds me of Little League: My batting got much better when I started swinging a bat in my backyard.

The lesson for my experimental design is that I should reduce and keep more constant how often I test.

Brain Food (part 8: a little more baseline)

As I mentioned earlier, while measuring my balance I’ve been listening to a book called Cod: The Fish that Changed the World. Around Hour 4 of the book I realized it was related to what I was doing: fish, brain food. Duh!

Each test of balance consists of 5 warmup trials followed by 15 regular trials. Each trial generates one number, a duration: how long I stand on one foot before the other foot touches the floor. It’s is a bit like surfing–balance, balance, balance, balance, balance, wipe out. (Surfers, skateboarders, skiers, snowboarders, gymnasts . . . this may interest you.) I enter the stopwatch times directly into my laptop. Each test lasts about 12 minutes. Because of the book, they’re pleasant.

I made several more baseline measurements of my balance with two changes:

1. To reduce fluctuations in the concentration of omega-3 in my brain, I did my best to take the flaxseed oil capsules as evenly spaced as possible. The general rule was to take 1 every 2.4 hours (= 10 per day). I didn’t take the capsules with me when I left home but I did follow that rule when I was home (not waking up to take them, however).

2. To make the distribution of (log) balance times more Gaussian (normal), I raised the maximum possible time from 30 seconds to 60 seconds. Previously I had stopped the test at 30 seconds; now the cutoff was 60 seconds. The problem was 30 seconds was too common — my balance was too good. Before the change, 3% of baseline measurements (6 out of 210) were 30 seconds. After the change, 12% of measurements (25 out of 210) were between 30 and 60 seconds and <1% (1 out of 210) were 60 seconds.

The graph below shows results (mean & standard error) for 28 sessions.

The early problem (first 10 tests), discussed in my previous post, was that the means were fluctuating too much. A one-way ANOVA, with each test a different level, gave F (9, 140) = 2.6, p = 0.008. This is why I started trying to evenly distribute the flax capsules over the day. This seemed to work. For the last 18 tests, F (17, 252) = 1.0, p = 0.4. Unfortunately there is obviously an upward trend but that is okay because the change I am going to make — much less omega-3 — should if anything impair balance.

Brain Food (part 7: looking for a steady baseline)

Thanks a lot to those who commented on my previous post, very helpful comments. Barleyblair said that after greatly increasing her omega-3 intake she too found her balance greatly improved — not only could she put her socks on while standing she could put her shoes on while standing, which she hadn’t even dreamed of being able to do. Bekel said her sleep is deep and restful because of flaxseed oil. Pauls referred me to a Real-Age test of balance where you stand on one foot with your eyes closed. I tried it. It was way too easy: After two minute I opened my eyes and stopped the test. The table that tells you what the results means only goes up to 28 seconds. If the table is not completely bogus, then my balance is much better than average. Which is consistent with my working hypotheses that (a) the average American gets far too little omega-3 and (b) my brain function — indexed by my ability to balance — greatly improved when I increased my omega-3 intake. Keep in mind that according to conventional recommendations I ate plenty of fish (several servings per week) before increasing my omega-3 intake.

Before doing a simple test of the effects of omega-3 on my balance, I would like to establish a steady baseline and get an idea of what normal variation is. If possible, I would like to reduce normal variation — reduce background noise, in other words. With this goal I have measured my balance 13 times under roughly the same conditions: barefoot, listening to a book (a fascinating book, by the way: Cod: A Biography of the Fish that Changed the World by Mark Kurlansky) while doing the test, 20 trials per test. Each trial consists of standing on one foot on the cutting board on the 0.5-inch platform (see equipment here) and measuring how long until my other foot touches the ground. Each test takes about 10 minutes. I like the book so I enjoy the tests.
Below are the results from all 13 tests as a function of trial number. They show that there is a warmup period lasting 5 trials.

That just refines warmup measurements I posted previously. This is completely new:

The x axis shows when the test was done; points that are close together on the x axis were done close together in time — e.g., an hour apart. I hoped for a steady baseline so that I could go on to more interesting stuff. That is not what I found. I did a one-factor F test to see if there was significant heterogeniety. I used only the last 15 trials of each test, dropping the first 5 “warmup” trials. There were 13 levels (the 13 tests) of one factor. There was a highly reliable (p = .003) effect of test, meaning the variation from one test to the next was too large to be sampling error. And this test did not take into account the obvious clustering — tests close in time had similar results. The clustering makes it even more likely there were real differences in balancing ability from one test (or rather cluster of tests) to the next.

Apparently my balancing ability can change substantially in several hours! (For example, the time between the last test and the next-to-last test, the last in a cluster of three, was 7 hours.) And my test is sensitive enough to detect this! Forgive the exclamation marks. Nothing in my knowledge of psychology makes it obvious or even likely that this would be true — that a measure of quality of brain function would vary so much in hours that it could be detected by single measurements. Or that single measurements would be precise enough to detect such a change. Of course brain function (e.g., alertness) may get worse as you get sleepy but in this case my balance was much better in the evening than in the morning. The differences in my scores had no correlate that I could notice — I didn’t feel noticeably different when I did worse than when I did better.

What might be causing the differences? Body temperature or other circadian rhythm: Unlikely, because one would expect best performance when body temperature is highest, around 4 pm, which does not fit the results very well. More plausible: blood concentration of omega-3. It will be relatively low in the morning because while I was asleep I took no flaxseed oil or walnut oil. It will rise during the day as I consume these. This is consistent with the high measurements in the early evening.

Whatever the cause, these data suggest that something in ordinary life (which includes omega-3 consumption) can improve brain function within hours. If you, dear reader, know of other data that suggests this conclusion please let me know. Drugs and alcohol can quickly change brain function but they are not involved here. Nor am I listening to music, also believed to improve brain function (slightly). I am going to try to reduce fluctuations in omega-3 blood levels and see if I get more uniform measurements. I had a cup of tea with caffeine this afternoon; caffeine consumption is something else I will better control (by eliminating it).

Brain Food (part 6: a little more progress)

I did two balance experiments with a warmup of 8 trials. In one, the order of feet (which foot I stood on) was left, then right; in the other, right, then left. In both experiments I did much better (i.e., balanced longer) on my right foot than my left foot, p s < 0.001. This surprised me; I had never heard of such an asymmetry. The difference was so large that the platform size (0.75 inch) good for the left foot was too easy for the right foot.

To make things as simple and easy as possible I decided to stop testing both feet and to only measure balancing on my right foot (and to use a 0.5-inch platform to make it more difficult and avoid a ceiling effect). I tested my balance (a) in silence and (b) listening to a book. The results were similar so I decided the standard condition will be listening to something. I want to make my balance test fast and pleasant.
I came across several promising related facts:

1. On the Shangri-La Diet (SLD) forums, spacehoppa said she felt “solid on [her] feet” — which may mean her balance has improved. If so, the improved balance that I noticed may be widely true. She also said “my mind feels clearer,” another effect I noticed from omega-3’s, and more reason to think omega-3 improve brain function.

2. On the SLD forums, porkypine wrote, “I have a very strong reaction to the 1500 mg of OmegaBrite that I have begun taking in the morning. . . . During the day, I am not just happier, but actually chipper, which is not a normal state for me. I have wondered if I am getting too much Omega-3.” This supports one of the assumptions behind my upcoming tests of the effects of omega-3 on balance: the effects of omega-3 on the brain happen quickly. It also highlights an advantage of measuring balance rather than something else, such as mood — namely, it is reasonable to assume that the better your balance, the better your brain is working. As this quote shows, the mapping between mood and goodness of functioning is not so clear.

3. In a book about neurology (Defending the Cavewoman by Harold Klawans), including Creutzfeldt-Jacob disease, I read: “A [Fore] woman in late pregnancy who was unable to walk easily across a narrow tree trunk bridging a gorge knew from that change in her balance that she had kuru and that she would die of it. The physicians examined her and thought she was normal, but in less than one year, she was dead.” This shows that balance is an especially sensitive measure of brain function, at least under demanding conditions. It’s relatively easy to notice worse balance.

Balance is also much easier to quantify than many other measures of brain function, such as mental clarity.

Brain Food (part 5: a little progress)

I’ve been doing small experiments on my balance to learn what affects it. Most research using new tools follows a progression. Step 1: you learn what people already knew. Step 2: you find new information that isn’t very interesting. Step 3: you find interesting new information. Earlier I found that I could balance on one foot longer on a wider platform — Step 1.

Now Step 2. I’ve done a few experiments comparing different footwear (sandals, shoes, barefoot). In each experiment I ran several conditions, each consisting of 12 trials standing on my left foot followed by 12 trials standing on my right foot. These trials had gaps of seconds between them. Different conditions (different footwear) were separated by at least 10 minutes and usually more.

The right-foot average was always more than the left-foot average. You can see examples of this in my earlier results. I doubt that the right foot/leg is actually better than the left so this suggests there is a substantial warmup effect, as there is in most tasks.

To make measurements more precise, it would help to have a warmup period before collecting the main, more stable data. How long should it be? The graph below shows data from many of the conditions I have run arranged by trial number, with a lowess summary line.

The y axis is in log seconds, not seconds; I used a log transform to make the distribution of the data more symmetrical. The maximum time was 30 seconds. (Log(30) = 3.4.) If I kept my balance for 30 seconds, I stopped, and recorded the result as 30 seconds.

The graph shows an early warmup period that lasts 6-8 trials long, followed by a slow improvement that lasts at least 24 trials. Here is something new and not very interesting: details about the warmup effect.

Brain Food (part 4: measuring balance)

Why is now a great time to be alive? Because Philip Weiss, one of my favorite writers, has a blog. Today’s entry mentioned a story about teaching the Torah while standing on one foot.

Speaking of standing on one foot . . . I devised a way to measure my balance. (To recap: I want to measure my balance to see if omega-3 improves it. When I increased my omega-3 consumption via walnut oil and flaxseed oil, it suddenly became much easier to put on my shoes while standing, which I’d been doing for years. The omega-3 also improved my sleep. Maybe omega-3 makes much of the human brain work better, especially the most-recently-evolved portions. Maybe this effect happens within hours.)

Here is the method. Equipment. At a hardware store I bought a series of 6 pipe caps, caps for 0.5 inch pipe, 0.75 inch pipe, 1.0 inch pipe, 1.25 inch pipe, 1.5 inch pipe, and 2.0 inch pipe (total $24). At a new-age pharmacy I bought a thick foot-sized cutting board (made of bamboo, $15). Below is a picture of these items and my stopwatch, which measures times to 0.01 second. Procedure. I put the board on one of the caps and balance on the board on one foot. I measure with a stopwatch how long I can balance on it before putting the other foot down. After 30 seconds, the trial stops — 30 seconds is the maximum possible score. I stand on my left foot for several trials (e.g., 12), then switch to my right foot for several trials.

The reason for six different caps — six different platforms — is to be able to adjust the difficulty so that it is neither too easy nor too hard — if either were the case the measurements wouldn’t be telling me much. With a little trial and error, the 0.75-inch cap seemed to be best. Below is data from that cap and the smaller and larger caps. With each foot I balanced 12 times; the graph shows the means and standard errors on a log scale. The sequence of conditions was: (1) 0.75-inch cap, (2) 0.5-inch cap, (3) 1.0-inch cap, (4) 0.75-inch cap. I balanced on each foot 12 times in each of the 4 conditions.

The results make sense: the smaller the platform, the less time I could balance on it. There appears to be a practice effect — better scores with more practice. I hope with more experience this effect will go away. The next step is to do these measurements several times per day for several days so that I can get some idea of how much they vary “naturally” — what the background variation is.

Brain Food (part 3)

In Brain Food (part 2) I found that when I reduced my flaxseed-oil intake my sleep got worse that very night. (Presumably because I reduced the amount of omega-3 fatty acids in my diet and therefore my blood.) Several people, including me, have found that their sleep improved the night after taking more omega-3 — that is, hours later. For example, if the omega-3 was begun Monday afternoon, they slept better Monday night.

How could the effect turn on and off so quickly? An article in the current issue of Journal of Nutrition supplies an answer:

Of the lipids found in the brain, polyunsaturated fatty acids (PUFAs) play an important role, serving as a major component of the phospholipids that form cell membranes, being precursors of signaling molecules such as prostaglandins and leukotrienes, modulating gene expression through the activation of transcription factors, and forming the microenvironment around membrane-bound proteins.

In other words, omega-3 has several effects in the brain, with vastly different sensitivities to changes in omega-3 blood levels. Changes in cell membranes probably happen very slowly; changes in “the microenvironment around membrane-bound proteins” could happen very quickly.

That a necessary nutrient (omega-3) could have fast-acting changes (within hours) is counter-intuitive. It doesn’t agree with previous experience. Other necessary nutrients take much longer for a deficiency to become apparent. And it doesn’t agree with common-sense design notions. Evolution builds our bodies out of what is in our blood. Human design is quite different — cars are not built out of what flows around them (oil and gas). Nor is anything else. There isn’t a everyday analogy that shows that a deficiency of a construction element can have fast-acting effects. If all the raw metal in the world disappeared, your car would run fine for a long time. Nor does it fit with general trends in nutrition research. Nutrition researchers study the whole body, most of which changes slowly by comparison to the brain. Sure, food can change the brain — make you sleepy, make you alert (caffeine), but these are not changes that interest most nutrition researchers, who usually emphasize optimal functioning. During the Stone Age, our diets did not contain much caffeine so it is obviously not a necessary nutrient, even if it can improve memory. No dietician tells clients to consume more caffeine. There are hundreds of substances like caffeine that change mental functioning, of course, and with the right definition of improvement all of them can be considered to improve mental functioning. But none of them interest nutrition researchers, with the exception of Adelle Davis (fascinated by LSD). An experimental psychologist, on the other hand, . . .

Brain Food (part 2)

Do omega-3 fatty acids improve brain function? I blogged earlier that switching from olive oil (low omega-3) to walnut oil (high omega-3) and flaxseed-oil capsules (very high omega-3) caused my sleep, my balance, and maybe my mood to improve. If you are interested in duplicating what I did, here are details:

Supplies. I take (a) 2 tablespoons/day walnut oil (Spectrum Organic refined). Store locator at www.spectrumorganics.com will help you locate this. Total 240 calories. (b) 10 1000-mg capsules/day of flaxseed oil (Longs cold-pressed softgels). Longs drugstore house brand, which is only available at Longs drugstores. Total 100 calories. I store both in the refrigerator but they are in stores at room temperature (reasonably enough, since walnut trees and flax plants live at room temperature). Procedure. I take both between meals. I divide the walnut oil into 2 doses of 1 T each that I take at least several hours apart. I take 80% of this stuff after noon. I spread the flaxseed capsules out throughout the day, take about 3 at a time. But read on for more helpful info — you may not want to start with exact duplication.

With the new oils, my sleep was consistently and unusually good for about two weeks, making it was clear that the improvement was caused by the new oils (or more precisely, the difference between the new oils and the oil they replaced). Less clear was what aspect of the dietary change made the difference. I switched to walnut oil and flaxseed oil because they were high in omega-3; but they differ from olive oil in other ways as well.

It would be great to know more — both to maximize the effect on myself and to help others get the effect. The wonderful thing about finding a food component that improves sleep — if that isn’t wonderful enough — is that it is likely to improve the brain in all sorts of other ways, too. (In contrast to my previous sleep research on the effects of non-food-components, such as standing and breakfast, where the improvements were probably specific to sleep.) The data about omega-3 support this view: A wide range of improvements in mental function have been observed. Assuming omega-3 causes a single change in the brain, that change causes (a) a reduced rate of Alzheimer’s, (b) less depression, and (c) better sleep — so it is likely to be widespread in the brain.

Since my earlier post, I’ve gathered some new and helpful data.

First, a Shangri-La-Diet forum poll found that most people who used olive oil for the diet had better sleep (10 out of 12), even though olive oil is relatively low in omega-3. Can even a small amount of omega-3 improve sleep? (Small compared to my current dose. SLD dieters consume large amounts of olive oil compared to everyone else.) Or is some other component of olive oil causing the change?

Second, after reading my earlier post, Catherine Johnson remembered that “I realize that I started sleeping miserably when I stopped taking Omega 3s.”
I trust that sort of thing. I had had a similar now-I-understand experience. After figuring out that lots of standing improves sleep, I remembered that several years earlier I had sleep very well the night after visiting lots of art studios during an Open Studios day. At the time I had guessed that it was all the art-inspired thinking that had caused my much-better-than-average sleep. But it was also a day with much more standing than usual.

Third, I reduced my flaxseed-oil intake by half: I took 5 capsules instead of 10. To my great surprise, I woke feeling as I felt with the olive oil. I hadn’t felt that way in weeks. The next day I went back to 10 capsules and again woke up feeling great. Obviously this strengthens the plausibility of omega-3 –> better sleep because the crucial ingredient is apparently in high quantities in the flaxseed oil capsules.

The stunning thing, the reason I was so surprised, is this: I didn’t expect the flaxseed change to make a difference so quickly. When someone ate a zero-folate diet to learn about the effects of folate, it took months for the effects to become clear. Although I had noticed the sleep improvement caused by the new oils the very next morning, I had assumed that was because I was quite deficient — like someone with scurvy noticing fast improvement with Vitamin C. Someone who is not Vitamin-C-deficient will have to go without Vitamin C for months before scurvy occurs. I had expected to wait weeks before seeing sleep degradation.

If you read about why omega-3 is important, you will read endlessly that our brains are made of it — the fraction of our brain that is omega-3 fatty acids is 10% (Wikipedia?), 60% (a Whole Foods employee), whatever. That is what I had assumed: that omega-3 is a structural element of our brains. Which is no doubt true. I have never heard that it is a metabolic element of our brains. Cars are “made of” carburetors, fan belts, computers, tires, and the like (structual elements); they “run on” gasoline and electricity (metabolic elements). Structural elements are parts. Metabolic elements are fuel. Failure to replace a perfectly good carburetor or other structual element will eventually cause trouble, but it may be several years. Failure to replace gasoline or electricity will cause trouble much sooner. Thus my little experiment suggested that omega-3 was a metabolic element.

If an effect can be turned on and off quickly it is much easier to study than if it takes weeks or months to turn on or off. Upcoming attractions: How I am studying it.