In this 10-minute talk I discuss what I think are the two main principles of experimental design:
- Something is better than nothing. You learn more from doing something than from thinking about what to do.
- When you do something, do the smallest easiest thing that will help, that will tell you something you don’t know.
Grad students often fail to understand Principle 1: They worry too much about what to do. Early in grad school, that was my big mistake. Professors often fail to understand Principle 2: They do something more complex than necessary. Failure is much more likely than they realize. My previous post was about such a failure (using genes to predict disease). When I was an assistant professor, I often made this mistake.
Makes sense. In grad school I had what some would call an overbearing mentor. I called him an enthusiastic and impatient workahaulic that expected the same of his students. I thrived in that environment and after four years in grad school earned my Ph.D. and accumulated 12 publications.
Regarding the second principle, I learned that in grad school, too. In fact, one of our favorite phrases to kick around lab meetings when we were discussing experimental designs was KISS (Keep It Simple, Stupid). Most of my most successful and profound experiments used very simple designs. Some of my most spectacular failures used quite complex designs. The simple designs generated the biggest surprises, as well.
To this day I adhere to these principles and try to entrain them in my own students.
The one drawback with Principle 1 is that I sometimes have jumped too soon-so to speak. That is, I wanted to do something but it turned out I should have given a little more thought to the problem and solution space before doing something that was not optimal or had some methodological flaws. Nevertheless, I have benefited more than been harmed by the use of the first principle. The second one has almost always been helpful.
what about trying out many new things at once, like when you went to paris or beijing. those seemed like fruitful enterprise, and didn’t seem like ‘the smallest, easiest thing.’ going to those places seemed to expose you to lots of strange stimuli, and new things happened to you, which you then sought explanations for. this to my inexpert mind seems like a very valuable approach, as long as you don’t do anything that could lead to non-trivial risks of serious harm. one could try to eliminate stimuli to see if effects remain, to isolate the actual causes. but perhaps there is a serious flaw in this thinking?
Mike, going to Paris is a good way to get new ideas but it isn’t doing an experiment. Experiments are usually done to test ideas, not generate them.
what if i were to take a bunch of different supplements at once to see if i notice anything new happening to me. i’m impatient, so i don’t do one supplement at a time, but a whole host of them, while being cautious not to poison myself in any known way.
i then observe what happens to me–say i get a dry mouth and i sleep an extra hour a night. i then remove half of the supplements to see if the results remain, and they do. i then divide the supplements i am taking into half again and see if the results remain, and they don’t, so i take the half i had just stopped taking, et c. would this constitute a reasonable experiment? i’ve heard warnings about testing multiple things at once, and have wondered if the above example is included or is different.
Mike, the sort of thing you propose works only in people’s dreams, as far as I know. Things just aren’t that simple.
thanks–i defer to you on this given your experience–is there a particular difficulty that emerges from such an approach? just thinking about it, and not having much experience with experimentation in general, i have trouble coming up with a serious problem with it.
here are two possible difficulties: 1. it will be hard to take so many supplements. 2. the effects won’t be clear.