Elements of Personal Science

To do personal science well, what should you learn?

Professional scientists learn how to do science mostly in graduate school, mostly by imitation, although they might take a statistics class. Personal scientists rarely have anyone to imitate, so have more need to understand basic principles. There are five skills/dimensions that matter. Here are a few comments about each one:

1. Motivation. In conventional science, the scientist does it as part of a job and subjects are paid. Neither works here: It isn’t a job and you can’t pay yourself. My original motivation was wanting to learn how to do experiments (for my job — experimental psychologist). After I discovered how useful it could be, I started doing personal science to solve actual problems, including early awakening and overweight. On these two subjects (sleep and weight control) conventional scientists seemed to have made and be making little progress, with a few exceptions (such as Sclafani, Cabanac, and Ramirez) in the area of weight control. Here my motivation was lack of plausible alternatives. Now I now see personal science like playing the lottery, except it costs almost nothing. Most of the time nothing happens, once in a long while there is a big payoff. An example of the lottery-like payoff is that for ten years I measured my sleep, trying to figure out what was causing my early awakening. One day it suddenly got worse (when I changed my breakfast). That led me to realize many things. Another example is I measured my brain function with an arithmetic test for several years. One day it suddenly improved (due to butter).

2. Measurement. Conventional scientists almost always use already-established measures because they improve communication. In contrast, a personal scientist wants a measure that is especially sensitive to the problem (e.g., insomnia) to be solved or the question to be answered (e.g., did flaxseed oil improve my balance?). Communication is much less important. Psychologists use Likert scales (rating scales with 5 or 7 possible answers) to measure internal states but they almost always use inexperienced and unmotivated subjects. When I’ve measured internal states (e.g., mood), I have a lot of motivation and eventually have a lot of experience and find I can make much finer distinctions. Unlike conventional research, I care enormously about the convenience of the measurement. For example, it should be brief.

3. Treatment choice. You don’t want to do a lot of experiments that don’t find any effect, so you need to choose wisely the treatments you test. Scanning the internet (what has cured insomnia?) and reading scientific papers (what are standard treatments for insomnia?) hasn’t worked for me, although it’s better to try anything than to try nothing. One thing that’s worked is to test large surprising effects I hear about. An example is Tara Grant’s discovery that restricting her Vitamin D to the morning improved her sleep. Also successful is measuring the problem for a long time, in search of outliers. When the problem suddenly gets better or worse, I test whatever unusual happened just before that. For example, when I switched from oatmeal breakfast to fruit breakfast, my early awakening suddenly got worse. I started testing various breakfasts. A third successful strategy is to combine the first two strategies with evolutionary thinking, giving bonus points if the treatment I’m thinking of testing provides something present in Stone Age life but absent now. For example, this is one reason I decided to test the effect of standing a lot. Stone Age people must have been on their feet more than most of us.

4. Experimental design. The hard part is knowing how fast the treatment effect rises and falls. If it rises and falls quickly, your experiment should be very different than if it rises and falls slowly. In most cases, what I study rises and falls slowly and the best design is some variation of ABA. Do A for several days, do B for several days, do A for several days. It is much easier to do a condition for too few days than too many so I try to err on the side of too many days. The hardest lesson to learn was to realize how little I know and avoid complex designs with untested assumptions.

5. Data analysis. Statistics books and classes emphasize statistical tests, whereas in practice what matters are simple graphs (e.g., what you measure versus time). I make one or more new graphs every time I collect new data (e.g., I make a plot of my weight versus time every time I weigh myself) but rarely do t tests and the like. I’ve learned to make several graphs at different time scales (e.g., last week, last month, etc.), not just one graph.

I believe these factors combine in a multiplicative way to determine how much you learn. If any is poor, you will learn little. They provide a way of asking yourself what you’ve learned after you’ve done some personal science. For example, where did I get the idea for the treatment? Presumably, with experience, you slowly get better at each of them.

Thanks to Brian Toomey for encouraging me to write this.