The Difficulty of Finding a Good Experiment to Do

My self-experimentation began because of one tiny thing — an article I noticed in the Brown University Science Library about teaching mathematics to college students. “The best way to learn is to do,” it started. Which made sense. To learn how to do experiments — one of my goals as a graduate student — I started doing self-experiments. Let’s imagine I had never seen that article, which is entirely possible. In this parallel universe, I become a psychology professor and then one day notice that someone else has done the personal science I have actually done. Has written “ Self-experimentation as a source of new ideas,” The Shangri-La Diet, and so on, including the experiments I’ve described on this blog. How would I react?

Many things about it would not impress me. You devised an arithmetic test to measure your brain function — so what? You measured yourself for a long time — big deal. You did an experiment — yawn. I might be slightly impressed by the experimental designs, which are simple and effective. Most experimental psychology uses more complex designs. What would baffle me would be the discovery of safe powerful beneficial treatments. How did this guy find these treatments? For example, the first experiment in “Self-experimentation as a source of new ideas” is about the effect of breakfast on early awakening. Eliminating breakfast reduced the fraction of days with early awakening from about 50% to 10%. Not eating breakfast is easy and perfectly safe. I don’t know of anything like this in all sleep research.

To a psychology professor, doing an experiment on one’s sleep is nothing. Finding something naturalistic (= not a drug) and sustainable that caused a big improvement, however, would be . . . unprecedented? Seemingly impossible? Psychology professors study everyday topics of great interest, such as memory and problem-solving and happiness, quite often. They would love to find easy safe sustainable non-drug ways of improving these things by large amounts. But I can’t think of a single example.

I thought about how hard it is to find big beneficial experimental effects (it’s easy to make things worse) when I read this post by the economist Yanis Varoufakis. He is excited about working for an online game company (Valve) because the nature of their game will allow experimental study of economics.

Econometrics is a travesty! . . . Econometrics purports to test economic theories by statistical means. And yet what it ends up testing is whether some ‘reduced form’, an equation (or system of equations), that is consistent with one’s theory, is also consistent with the data. The problem of course is that the ‘reduced form’ under test can be shown to be consistent with an infinity of competing theories. Thus, econometrics can only pretend to discriminate between mutually contradictory theories. All it does is to discover empirical regularities lacking any causal meaning. [Why is he sure they lack any causal meaning? — Seth]. . . The reason for this unavoidable failure? None other than our inability to run experiments on a macroeconomy such as rewinding time to, say, 1932, in order to see whether the US would have rebounded without the New Deal (or to 2009 to see what would have happened to the US economy without Ben Bernanke’s Quantitative Easing). Even at the level of the microeconomy, keeping faith with the ceteris paribus assumption (i.e. keeping all other things equal in order to measure, e.g., the relationship between the price of and the demand for milk) is impossible (as opposed to just hard).

In sharp contrast to our incapacity to perform truly scientific tests in ‘normal’ economic settings, Valve’s digital economies are a marvelous test-bed for meaningful experimentation. . . . We can change the economy’s underlying values, rules and settings, and then sit back to observe how the community responds, how relative prices change, the new behavioural patterns that evolve. An economist’s paradise indeed…

I find this baffling. It’s like thinking: Now I can write. Soon I will be writing stuff that the world wants to read! Okay, now he can do experiments. Good. After a few of them, I suppose, he will learn what every experimental scientist knows and confronts every working day: it is incredibly hard to do interesting experiments. The “sharp contrast” between the new setting and the old one has yet to be demonstrated.

Okay, how did I find a bunch of big beneficial safe sustainable effects? I am now finishing a paper in which I try to answer this question. To be brief: 1. As I’ve said, I believe that the distribution of surprise/observation follows a power-law-like distribution. Almost all observations, very little surprise, a tiny fraction of observations, great surprise. Which is pretty obvious. 2. The “slope” (parameter) of the distribution depends on subject-matter knowledge (more knowledge = more favorable slope, i.e., “chance favors the prepared mind”), scientific skill (more skill = more favorable slope), and novelty (more novelty = more favorable slope). I was in good shape on all three. For example, when I studied sleep, I knew a lot about sleep. Novelty is enormously important. In my personal science I could easily study treatments (e.g., not eating breakfast) and dimensions (e.g., how rested I felt when I awoke) that had rarely if ever been studied before. I could do this again and again, keeping novelty high and thus keeping the slope very favorable. (Varoufakis will get a burst of novelty when he begins experimentation (the situation is new) but forced to use that situation for all his experiments the novelty will run down, making the slope of the distribution less favorable.) 3. My cost/observation was very low and the benefit/observation remarkably high (I was improving my own health). So I was very motivated to make observations. My answer in the paper is a little more complicated but that’s most of it.