The data I posted that showed a sudden improvement in my arithmetic ability is among the most interesting data I’ve ever collected. Not because it revealed something wildly new — I was already sure flaxseed oil helped — but because it revealed something intriguing and new (the time course of the improvement is puzzling).
I collected the data in an unusual way — watchful waiting. I didn’t do an experiment, the way experimental psychology data is usually collected. I didn’t do a survey, the way epidemiological data is collected. In the emphasis on one person it resembles a case report in medical journals — but I didn’t have a problem to be solved and the data is far more numerical and systematic than the data in a case report.
And this rarely-used scientific method paid off. Hmm. I think the scientific methods currently taught have a big weakness: They focus almost entirely on idea testing, whereas idea generation is just as important. Tools that work well for idea testing work poorly for idea generation. The effect of this imbalance — a kind of nutritional deficiency in intellectual diet — is that scientists don’t do a good job of coming up with new ideas.
What should scientists be doing? I would like to find out. My watchful-waiting data collection is/was part of trying to find out. That it paid off pretty quickly is a good sign. It’s the third step in a long process. Step 1. When I was a grad student, my acne self-experimentat led me to realize that one of my prescribed medicines didn’t work — a surprising and useful new idea. Step 2: Later self-experiments had the same effect: Generated surprising and useful ideas. At a much higher rate than my conventional experiments. Why? Perhaps because it involves cheap frequent tests of something important. Step 3: Arrange such a situation — cheap frequent tests of something important — and see what happens.
Some of my research involves understanding causal cognition in rats. The very nature of this research precludes self experimentation (unless I were a rat!). But, I find talking to people from very diverse disciplines to be very helpful in the idea generation stage of my research. I was just at the Eastern Psychological Association meeting in Pittsburgh, and visiting a group of philosophers at Carnegie Mellon University to talk about my rat work. These philosophers are causal interventionists (e.g., Peter Spirtes, David Danks, Clark Glymour), but there was also a philosopher who does rat experiments to understand exploratory behaviors. I met with these folk and we had an extremely enriching and invigorating discussion. I came back to UCLA with a bag full of new ideas with very interesting and profound (in my mind) implications.
I’ve had the same reaction discussing my work with evolutionary anthropologists and ethologists (e.g., Rob Boyd, Clark Barret, Dan Fessler, and Joan Silk) at UCLA’s anthropology department. And in talking with ethologists and psychologists at UC Davis.
These sources of ideas are a bit more conventional than the use of self experimentation, but still widely unpracticed by many practitioners of science who stick close to their circle of like-minded colleagues.
Competent idea testing takes so much time and trouble that most of us are bubbling over with way more ideas than can ever be tested. The problem is to narrow the field to the few that are actually in our own field, and that we can afford to test, and to figure out ways to (afford to) test them.
Finding ways to make such testing cheaper and quicker goes a long way to helping use up ideas. A clearinghouse for ideas each of us can’t afford to test right just yet would eliminate any such shortage easily.
Reading Gary Taubes, Good Calories, Bad Calories, has caused me to think about scientific research and the scientific method in new ways. Taubes spends a lot of time discussing the flaws of scientific study and how testing ideas, if the researcher is convinced he is correct, leads to misinterpreting the research. I think your point about the scientific method being terrible for idea generation is excellent. Certainly research and scientific method is important, but a way to stimulate creativity would be incredibly valuable too.