For exercise (Dance Dance Revolution) and self-tracking, I decided to buy a Wii. My first attempt, I was scammed. It arrived in August. With difficulty, I took it and accessories unopened to China. That was hard. It was even harder — for no obvious reason — to install it in China. The box sat unopened next to my TV, easily visible, for two months.
Finally I opened the box, took out the parts, put them together, added batteries, plugged it into the TV in my apartment. And nothing happened! Was my TV at fault? Or the Wii? Wii’s aren’t sold in China. I imagined bringing it back to America to get the problem fixed. After a few days, I tested my TV using video output from a neighbor’s Apple computer. My TV worked. After the test, my Wii also worked. When I replaced the Apple input with the Wii input I saw the Wii input for the first time. I don’t understand it, but that’s what happened.
In my experience, this is how science works. It is much harder than expected, then it pays off in ways that defy understanding. The concept of self-experimentation is simple: I will measure X (sleep, productivity) about myself. I will test different ways to improve X, learn what works, and thereby improve X. The reality is different. For years I measured my sleep and tried to improve it. It was hard to deal with the data. Even worse, every idea I had was wrong. That seemed like a huge obstacle — like my Wii needing repair. But I kept plugging away, because it was better than doing nothing, and . . . got somewhere. Out of nowhere and nothing. Not only did I improve my sleep, I arrived at a broader idea about health that turned out to be very helpful (that our bodies are designed for Stone-Age conditions and self-experimentation can help determine those conditions, which aren’t obvious). Just as we overvalue big steps (e.g., well-funded prestigious research), we undervalue small ones (e.g., cheap research with no prestige).
Science is basically a bunch of little steps. Many little experiments that explore cause-effect space. If you find a new example of cause and effect, the payoff is unpredictably large. Scientists don’t like thinking of themselves as wandering ants. But that’s how they are most effective. This goes against human psychology because wandering (Nassim Taleb calls it “tinkering”) is low status and lonely. The payoff is too rare and too unclear. It isn’t supported by powerful institutions, such as research universities and medical schools. Imagine an ant who says “I know where food is!” This is a way to get many ants to follow him, to feel important, to have high status, to get support from his employer. That’s why he does it. But he doesn’t know. The effect on the rest of us, the potential beneficiaries of progress, is that instead of having a thousand ants wandering everywhere, we have a thousand ants following one ant who doesn’t know what he’s doing.
I was and still am unprepared for how unsuccessful my experiments are. Although I don’t have a true tally, the number is certainly less than 20%, and I think probably a fair and accurate count would probably put that number at around 5%. It seems like that number is similar to your success rate with self-experiments. While it’s personally hard to deal with that level of failure, I think it’s even harder at the institutional level. Most institutions couldn’t tolerate employees who get such a low success rate for their experiments even if the resulting success brings much more valuable information. So on a personal level thats why I try to spend time on things that are much more likely to work because it’s easier to defend your work that way. That’s why I like your self-experiments because they allow for more risk that would probably kill the career of a normal experimenter.
If you enjoy Dance Dance Revolution, definitely check out Dance Dance Immolation at Burning Man.
When I worked in R/D the upper management wanted predictability in “findings” so we went our way in testing ideas so we ended up with a pipeline of hidden successes and then we metered it out in a “predictable” manner. We ended up predictable but slow in comparison to the rest of the competitor. Upper management patted themselves on the back for being able to manage and to be predictable.
There is a lot of randomness in R/D and the quicker one built up a pile of “failures” the quicker one found the nugget(s).
The philosophy I found to work the best is TREE-test randomly, evaluate and elect-and the team should consist of a bunch of people with cognitive diversity and experience diversity.
Andrew, by science I meant empirical science, where the main goal is gathering data from which you learn how the world works. This isn’t the main goal of statistics professors. They don’t gather data nor focus on substantive issues. But I agree, you make a good point. Even within this subset of science I agree that one can pursue a path where progress is more predictable. In terms of the scientist = ant analogy, an ant has a choice of joining a trail of many ants to a food source or wandering around by itself to find a new food source. Likewise a scientist who gathers data has the choice of exploiting a known cause-effect relationship (e.g. doing variations on it, trying to explain it) or trying to find new cause-effect relationships. I don’t know what a similar choice would be for statistics profs.
Andrew, yes, there are exceptions to the broad statements I made. But most scientists (95%?) gather data in one way or another. I think physics and astronomy are the only sciences where there are a lot of pure theorists (such as Feynman), and even in those fields I think the data collectors outnumber them. I think you’re an exception, too — as far as I can tell, most statistics profs spend little of their time trying to answer substantive questions. Most statistics profs are most interested in developing new methods. Only a few statistics profs are also profs in a substantive area, such as political science. I say all this partly because I wonder what would be a “high-risk” line of research for a statistics professor. Maybe developing methods to do something unconventional, such as generate ideas?