Before the invention of statistical tests, such as the t test, science moved forward. People gathered data, computed averages, drew reasonable conclusions. As far as I can tell, modern ways of analyzing data improved the linkage between data and conclusion because they reduced a big source of noise: How the data were analyzed. Procedures became standardized. Hypothesis testing improved. Hypothesis formation, however, did not improve. Knowing how to do a t test and the philosophy behind it will not help you come up with new ideas. Yet data can be used to generate new ideas, not just test the ones you already have.
Our understanding of outliers is in a kind of pre-t-test era. People use them in an unstructured way. As Howard Wainer’s analysis of his blood sugar data indicates, better use of them will improve hypothesis formation. A kind of standardized treatment should help generate ideas, just as the t test and related ideas helped test ideas. Here are some questions I think can be answered:
1. Cause. What causes outliers? It’s a step forward to realize that outliers are often caused by other outliers. Howard has found that unusually high blood sugar readings are caused by eating unusual (for him) foods.
2. Inference. I’m fond of saying lightning doesn’t strike twice in one place for different reasons. The longer version is if two outliers could have the same explanation, they probably do. I think this principle can be improved.
3. Methodology. To test ideas, you want variation to be low. To generate ideas, you want outlier rate to be high. Howard could make progress in understanding what controls his blood sugar by deliberately testing foods that might produce outliers. In genetics, x-rays and chemical mutagens have been used to increase mutation rates; mutations are outliers. (Discovery of a white-eyed mutant fruit fly led to a wealth of new genetic ideas.) In physics, particle accelerators increase the outlier rate in order to discover new subatomic particles. There are no comparable procedures for psychology. Self-experimentation increased my rate of new ideas because it increased my outlier detection rate. It increased that rate for three reasons: 1. I kept numerical records. 2. I analyzed my data using the same methods as Howard. 3. I did experiments. Travel is like experimentation; there too it helps to keep numerical records and analyze them. The question: What are the basic principles for increasing outlier rate?
Part 1.
“There are no comparable procedures for psychology.”
What about the rorschach test? Granted, I had to stretch all the way to psychoanalytic theory to find an example.
Well, Rorschach tests haven’t generated any new psychological ideas, as far as I know. But they may be the best possible answer for psychology, as you say.
Hypothesis testing improved. Hypothesis formation, however, did not improve. Knowing how to do a t test and the philosophy behind it will not help you come up with new ideas.
Exactly! I have had the misfortune of taking a graduate degree and one of the most irritating aspects of it was the fetishism on hypothesis testing and rather intentional devaluation of hypothesis formation. This resulted in quite a few, extremely intelligent and creative (and subsequently frustrated) people being sidelined and passed over for their (relative) weaknesses in statistical methodology, while their unfairly devalued talents and insights clearly lay in what you call hypothesis formation. I always thought this was a tremendous waste of human capital.
Economists found botulism spores in honey by studying SIDS and diet and data mining. They find a number of things.