My Frontiers of Psychology class read Daryl Bem’s new paper Feeling the Future that reports nine experiments that show an effect of the future on the present. I have a different take than anything I’ve read: I think there are several good reasons to take it seriously. But in this post let’s start with how it could have been better:
1. Lack of background. There have been lots of experiments along these lines. What did they show? This question is not clearly answered. The prior probability of these claims is enormously important. As I told my students, if seeing the future was common and easy for even a small fraction of people, we wouldn’t have businesses, such as casinos, making money on gambling. But the existence of such businesses doesn’t rule out weak effects.
2. Lack of exact repetition. An obvious criticism is that Bem slanted the data analysis to favor the results he wanted. In any data analysis of unfamiliar data, you must choose — how to transform the data, what test to use, and so on. You must also choose how many subjects to run and how many trials to give them. There are rules for these choices (Bem doesn’t seem to know how to choose a transformation) but nevertheless they allow favoritism to creep in. Drug trials have big problems along these lines — severe slanting of the analysis to make the results more favorable — which is why when you register a clinical trial you must specify the endpoints. The answer to the criticism that your data-analysis choices made your favored result more likely is to do a data analysis with no choices at all. This cannot be done from scratch. You need to do the experiment once, make all the necessary choices, and then do the same experiment again (same everything as much as possible) and analyze the data exactly the way you analyzed the data from the first experiment. Bem never does this. Instead each experiment is different from all the rest. This is what experimental psychologists traditionally do but here it is a bad idea. Better to have taken the two simplest and clearest effects (priming and word learning) and repeated them several times exactly.
3. Were experiments left out? Let’s say you observe a weakly-significant result, p = 0.03. Now you do the same experiment eight more times. How likely is it that each of the eight replications will also find a significant difference? Quite low. Yet Bem finds a weakly significant difference in each of his nine experiments. This is highly unlikely. Bem appears unaware of the problem. Mendel had the same problem (data too good to be true). Ultimately Mendel was proved right. But again it stresses that Bem should do exact repetitions and report the results no matter what if he wants to be more persuasive.
As I tried to argue in the related case of Alternative/ Complementary healing – https://qjmed.oxfordjournals.org/content/95/10/643.full – I believe that the problem for this kind of study, of a phenomenon whose nature is unknown, is that ‘controlled trials’ are strictly impossible/ nonsensical.
In this instance, since the nature of hypothetical psi is unknown, and the features that influence it, it is not possible to ‘control’ the trial.
When a causal phenomenon is understood, then a trial can be controlled – and the better controlled the experiment, the clearer will be the result.
The level of significance in frequentist statistical testing is merely a distraction – a mere arbitrary convention applied to a no-causal hypotheses (the ‘null hypothesis’ has nothing to do with causation).
And significance testing has nothing to do with (is orthogonal to) causality: https://charltonteaching.blogspot.com/2010/10/scope-and-nature-of-epidemiology.html.
I used to subscribe to the Skeptical Inquirer, and I read a lot of stuff by James Randi and Martin Gardner. There’s usually some plausible, mundane explanation for these types of PSI effects. Extraordinary claims require extraordinary evidence.
@Alex,
Is an ‘extraordinary claim’ relative to a given agent’s belief system? (for example, telepathy might be extraordinary for person a, but not for b)