Poor Replication Rate in Psychiatric Genetics Research

With the ability to measure individual genes has come interest in learning what they do. Perhaps Person X is depressed and Person Y is not depressed because Person X’s genes differ from Person Y’s. A whole generation of psychiatry researchers now believes this is plausible. There are “general reasons to expect that GxEs [gene by environment interactions] are common,” says a new review paper in the American Journal of Psychiatry. By “common” they mean large enough and common enough to do research about.

I don’t agree with this conclusion. Sure, twin studies show that genes matter for psychiatric diagnoses. Identical twins are more likely to be concordant (= have the same diagnosis) than fraternal twins, for example. But this is a very long way from indicating that single genes matter. Twins results are entirely consistent with the possibility that a large number of genes each matter a little. If this is true — and I find it far more plausible, when it comes to psychiatry, than the single-gene idea — then searching for one gene that does this or that is a waste of time. Individual genes are too weak. To do psychiatric gene research you have to dismiss or ignore the many-tiny-effects possibility, because if true it would mean what you are doing is bound to fail. The new review paper I mentioned ignores it.

The new review paper surveys all of the research papers about GxEs during the first decade of research (2000-2009) in this area — about 100 papers. It asks (a) if initial findings have been repeatable and (b) how much we should trust the repetition attempts. To answer the first question, they found that only a third (10 of 37) of initial findings were repeated when tested a second time. If things were working well, all of the initial findings would have been repeatable. The low replication rate doesn’t mean that two-thirds of the initial findings were false. Perhaps the replication attempts were poorly done and all of the initial findings would have held up if they were better done (e.g., larger samples). Or perhaps the replication attempts were biased toward positive results and none of the initial findings would have held up if they were better done.

The review paper also found that positive replication attempts had much smaller samples (median sample size about 150) than negative replication attempts (median sample size about 380). This suggests that the negative replication attempts are more trustworthy than the positive ones. The true replication rate is probably lower than one-third.

The findings, in other words, support my initial belief that the whole field is a waste of time. Amusingly, the authors of the review (one at Harvard, the other at the University of Colorado) conclude the opposite. Here’s what they say:

This review should not be taken as a call for skepticism about the G×E field in psychiatry. . . . True progress in understanding G×Es in psychiatry requires investigators, reviewers, and editors to agree on standards that will increase certainty in reported results. By doing so, the second decade of G×E research in psychiatry can live up to the promises made by the first.

Of course their findings support skepticism about GxE research. This isn’t slanting your conclusions to be more convenient, this is bending them backwards. And failure to mention the many-tiny-effects possibility, a plausible explanation for all the results they describe, is another sign that this area of research is not to be trusted.