Self-experimentation is an example of the more general idea that non-experts can do valuable research. Another example is that two New York teenagers have shown that fish sold in New York City is often mislabeled. They gathered samples from 4 sushi restaurants and 10 grocery stores and sent them to a lab to be identified using a methodology and database called Barcode of Life. They found that “one-fourth of the fish samples with identifiable DNA were mislabeled . . . [and concluded] that 2 of the 4 restaurants and 6 of the 10 grocery stores had sold mislabeled fish.”
The article, by John Schwartz, appeared in the Science section, which makes the following sentence highly unfortunate:
The sample size is too small to serve as an indictment of all New York fishmongers and restaurateurs, but the results are unlikely to be a mere statistical fluke.
This is a Samantha-Powers-sized blunder. It could hardly be more wrong. How much you can generalize from a sample to a population depends on how the samples were chosen. Sample size has very little to do with it. (John Tukey had the same complaint about the Kinsey Report: Stop boasting about your sample size, he said to Kinsey. Your sampling methods were terrible.) To know to what population we can reasonably generalize these results we’d need to know how the two teenagers decided what grocery stores and restaurants to sample from. (Which the article does not say.) If the 14 fish sellers were randomly sampled from the entire New York City population of grocery stores and restaurants, it would be perfectly reasonable to draw broad conclusions.
I have no idea what it could mean that the results are “a mere statistical fluke”.
The effect of these errors is that Mr. Schwartz places too low a value on this research. It’s impressive not only for its basic conclusion that there’s lots of mislabeling but also for showing what non-experts can do.
The end of the article did see the big picture:
In a way, Dr. Ausubel said, their experiment is a return to an earlier era of scientific inquiry. “Three hundred years ago, science was less professionalized,” he said, and contributions were made by interested amateurs. “Perhaps the wheel is turning again where more people can participate.”