A few days ago a graduate student in economics asked me what I thought of behavioral economics. On the positive side, I said, some of the phenomena are impressive. For example, the endowment effect, which is so strong I would demonstrate it in class. On the negative side, none of the researchers use experiments to generate ideas. They don’t merely not do it; they seem unaware of the possibility of doing it. The graduate student wondered how it can be done. I said there were three main ways:
1. Do something extra. Do a little more than necessary so that your experiment tells you about something that isn’t the focus of interest. For example, vary a factor that you think is not important. This is Saul Sternberg’s idea. I did this in my peak-procedure experiments: measured how long rats held down the bar. This was irrelevant to the purpose of the experiments, which was to understand how rats measured time. These measurements greatly surprised me. For years, I misunderstood them. Eventually they led to a new line of research about the control of variability.
2. Measure a function, not a point. Ask how your treatment changes a whole function, not just this or that numerical measure. This is what I did in my peak procedure experiments: The experiments generated for every condition an entire function showing response rate as a function of time. I saw how treatments changed the entire function. This talk describes some of the new ideas this led to.
3. Make your experiment easy and fast. The easier and faster it is, the more you can do it in lots of variations. Our ignorance of behavior being great, some fraction of these are likely to generate unexpected – and therefore inspiring — results. This is one reason self-experimentation is good for generating ideas: It is easy and fast.
I am not aware of any other written answers to this question, strangely enough.
I suggest spending a lot of time looking at patterns in the raw data using graphs and spreadsheet analysis. Do subjects react in the same way? in a non-linear way? Regressions are so brute force that you lose a lot of subtle details at the individual level. Also compare your results to those of others and try to understand the variation. I am working on a paper because of this: my subjects saw more data in a cooperation (PG) game — they were less “cooperative” and more “reciprocal” — relative to the other experiment. My result fits the literature better and so I’m excited to push it ahead.
Yeah, I agree. That’s advice about what to do after the experiment is over, however. My comments are about what to do before that — while planning your experiment, for example.