Scientists are fond of placing great value on what they call skepticism: Not taking things on faith. Science versus religion, is the point. In practice this means wondering about the evidence behind this or that statement, rather than believing it because an authority figure said it. A better term for this attitude would be: Value data.
A vast number of scientists have managed to convince themselves that skepticism means, or at least includes, the opposite of value data. They tell themselves that they are being “skeptical” — properly, of course — when they ignore data. They ignore it in all sorts of familiar ways. They claim “correlation does not equal causation” — and act as if the correlation is meaningless. They claim that “the plural of anecdote is not data” — apparently believing that observations not collected as part of a study are worthless. Those are the low-rent expressions of this attitude. The high-rent version is when a high-level commission delegated to decide some question ignores data that does not come from a placebo-controlled double-blind study, or something similar.
These methodological beliefs — that data above a certain threshold of rigor are valuable but data below that threshold are worthless — are based on no evidence; and the complexities and diversity of research imply it is highly unlikely that such a binary weighting is optimal. Human nature is hard to avoid, huh? Organized religions exist because they express certain aspects of human nature, including certain things we want (such as certainty); and scientists, being human, have a hard time not expressing the same desires in other ways. The scientists who condemn and ignore this or that bit of data desire a methodological certainty, a black-and-whiteness, a right-and-wrongness, that doesn’t exist.
“The scientists (…) desire (…) a black-and-whiteness, a right-and-wrongness, that doesn’t exist.”
Relatedly, it is remarkable that statistical associations are either “significant at conventional levels” or not, when the actual statistic is continuous.
Saying there is no evidence for the beliefs that some studies provide more trustworthy directional implications than others, and that anecdotes can be misleading, is just silly. Scientists seek out the most perfect studies so that they can update their Bayesian predictions most efficiently. All of these “methodological beliefs” come from a simple analysis of random sampling and the logic of science, of data.
If you want to get down to brass tacks, start wagering. The people whose income depends on predicting the future know exactly how much to weight evidence towards a correlation or cause. And it’s not going to be as much as well-structured studies.
We may be quick to seize upon a study’s failings because it’s often so easy to do it right. A simple calculation. A sentence of instruction to the subject. Once you know how much better it is to randomize faces in a lineup, would you ever congratulate a police department for neglecting this?
Lemmus: the p = .05 is there as a courtesy to the reader, no? Not because there is a step-function of a conclusion’s relevancy.
“Some studies just don’t have enough data to be relevant.” What’s an example? I find it hard to believe there is a threshold for relevance: above the threshold, relevant, below the threshold, not relevant.
NE1, I’m not saying there is no point being discriminating. Of course some studies are more meaningful than others. It is the black-and-whiteness of the judgments that is the problem. The way studies that don’t meet some arbitrary criterion are ignored. They really are ignored, completely discounted; I’m not making this up.
Whether the p-must-be-.05 -or-less convention is helpful or harmful is a hard question, at least for me. It gives scientists who know very little something to shoot for; but it ignores the reality that there is no important difference between p = .04 and p = .06. It distinguishes things that are the same, in other words. Should bikes be built with permanent training wheels?
Seth, the problem is that “the scientific method” is not an epistemology but rather an evolved set of hacks optimized for, among other things, improving the quality of naive human epistemology. All of the forms of pseudo-skepticism you complain about involve rules that deviate from optimal epistemology, but by default humans also deviate from optimal epistemology. These are the best known rules that we have, not for reaching truth, but for indoctrinating into a crowd of people so that those people can do a better job of reaching truth than they would by default.
“Lemmus: the p = .05 is there as a courtesy to the reader, no? Not because there is a step-function of a conclusion’s relevancy.”
Oh, in the stuff that I read (I’m a sociologist), it’s pretty much treated like a “step-function”. p = .06 equals “no association”.
Michael, yeah, I agree, a binary function is better than a flat function. That’s what I meant by “gives scientists who know very little something to shoot for.” A little scientific education is a dangerous thing.
It’s definitely worse to build bikes with permanent training wheels than to teach people to ride bikes without training wheels, but better than to have a population of bikers who are always falling down. The question is, do we (collectively) know any fairly reliable way to teach people to use th “no training wheels” version of rationality instead of just “faking it” with the “scientific method”. If so, do we (personally) know how to get there from here, e.g. how to influence the culture in the relevant manner. Honestly, I think ‘no’ and ‘no’, but I think I’m closer to the second bit than to the first.
Were I to teach the basics of science, I would tell a bunch of stories chosen to show the value of a wide range of evidence, experimental, non-experimental, case reports, etc. I would say: we need all types, and explain why. A story-driven approach is a lot different than the current approach (e.g., evidence-based medicine) with its emphasis on rules (do this, do that) and value judgments (this is good, that is bad).
Andrew, thanks for clarifying that. Your paper argues that the Kanazawa study was too small to have a good chance of finding results with p < .05 given the likely size of the effect. Sure. I don’t agree that this means its info is useless. You write: “A study of this size is not fruitful for estimating variation on this scale.” That is too strong, I believe. I don’t think it’s true that nothing can be learned from the Kanazawa data. The Kanazawa data make some ideas more plausible, other ideas less plausible.
Susan Haack was once taking questions after a talk, and was asked “What, essentially, do you think the scientific method is?” Her (prescriptive) response was: “Trying really hard to figure out the truth.”
The problem I have with this is that many scientific advances are made when scientists almost dogmatically hold onto beliefs in the teeth of contrary evidence. Their perseverance leads to the uncovering of evidence that does support their position …
Darius,
This link seems to have the full article: “Creativity versus skepticism within science: more harm has been done in science by those who make a fetish out of skepticism, aborting ideas before they are born, than by those who gullibly accept untested theories.”
https://www.encyclopedia.com/doc/1G1-154238670.html
This may be tangential to your discussion, but have you run across https://www.institutionalreviewblog.com/ The blogger there is a history teacher at Geo. Mason who has been thwarted by IRB rules intended, of course, to protect human subjects, but misapplied (he feels) to the social sciences. I’m currently going through the online IRB training through CITI and, while I totally agree that psychological and medical experiments need oversight, is it really necessary if I’m testing a user interface for data entry?
For whatever reason, I’m seeing parallels between this series of posts and my recent reading of horror stories related to getting IRB approvals. Another article about the blog is at https://insidehighered.com/news/2007/01/19/irb