This semester at Tsinghua University — the most selective college in China — I taught a freshman seminar about recent psychology research. Three weeks ago I gave my students a choice of five articles from Psychological Science, all published in 2008. They were to read one of them and comment.
Mostly I try to teach appreciation but three weeks ago we focused on how articles could be improved. I have never tried to teach this, yet the students made some very good points. Here are some of their comments:
1. This article said that we believe women make better leaders when there is within-group conflict and that men make better leaders when there is between-group conflict. One student pointed out that Rwanda was a good example. After the genocide (within-group conflict), far more women were elected to office.
2. This article studied the effect of cleanliness on moral judgments. One experiment compared two groups: subjects in one group had recently washed their hands, subjects in the other group had not. Before the time when the handwashing happened, both groups saw unpleasant scenes from a movie. Students pointed out an important confounding not mentioned in the article: The two groups differed not only in handwashing but also in the time from movie to test (because handwashing took time). Perhaps subjects who washed their hands remembered the movie less well.
3. The name-letter effect is a tendency to favor outcomes (broadly defined) that involve the first letter of your name. This study involved Belgium workers. The authors found that workers were more likely to be employed by a company whose first letter matched the first letter of their name. The correlation was small but reliable. Two students pointed out that this might reflect the company’s choice of whom to hire rather than the employee’s choice of where to work. One student pointed out that the correlation might be due to name-place correlations across Belgium. Perhaps certain regions favor certain names for both people and companies. As you move closer to the French border, perhaps French names become more common among both people and companies.
In all of these cases, had I been the editor, I would have required the authors to change their article appropriately.
James Surowiecki’s The Wisdom of Crowds described cases where averages of estimates made by non-experts did very well, sometimes out-performing experts. These three examples don’t involve numerical judgments nor averaging, but they do show non-experts (freshmen) doing better than experts (journal editors and reviewers) in certain ways. Each paper was read by about eight students.
More It isn’t easy to convey how impressed I was. The comments about Rwanda and about name localization certainly deserve a letter to the editor (if Psychological Science published them). Both of them are sophisticated methodological comments. The Rwandan one says that after you write an experimental article, try to find out if real-world events support your findings. That may be a helpful lesson in many cases. The name localization one suggests that psychologists who use survey data should be learning more about how to analyze survey data. Several other times my students surprised me with how good their comments were. One was during a discussion of possible reasons for the Holocaust, another was about why women in ancient China bound their feet, a third involved proposals for Mindless-Eating-type experiments.
Seth, while your post is interesting, it seems to be based on the assumption that the “experts”, that is journal editor and reviewers, would catch all of the major criticisms or flaws of the manuscript during the review phase. This is not a safe assumption. I’ve recognized from early in my career that if you give a draft of a manuscript to 12 people, you will get back 12 sets of partially overlapping sets of comments and criticisms. Sure there are a lot of common things picked up by the majority of reviewers, but each reviewer will invariably catch one or two flaws or see a potential confounding factor or account that all the others hadn’t. It’s like writing computer code. No matter how many programmers work on a piece of software, it will never be completely bug-free.
Have you considered the possibility that the students were much more motivated to pay close attention than journal reviewers generally are?
The students were doing this as a one time exercise, with the very immediate rewards of engagement with their peers, and with you, an encouraging professor whom they presumably respect. Furthermore, you were listening to and respecting them for their criticism of published papers, something I suspect is not a consistent experience.
For journal reviewers, on the other hand, the task has presumably become ‘work’, and it seems likely that many individual papers simply won’t be of engaging interest to them. There is little immediate social reward for playing ‘the game’ of catching faults, and indeed doing so may carry some costs in terms of increased administration.
I don’t mean to suggest that journal reviewers are not taking their role seriously or professionally. I am just pointing out that it may sometimes be the scientific equivalent of ‘guard duty’, and that to maintain a consistent level of attention to each paper could be very challenging compared to someone who is doing it for the first time.
Aaron, I wouldn’t say that i expect reviewers and editors to notice all ways an article could be improved — just that they didn’t notice these particular ways. But you have a point: Perhaps the editors and reviewers picked up various problems that the students would have missed. Still, I was surprised how good their critiques were. That’s what I was trying to say here.
Andrew, thanks. You might be right about the subject-matter match (psychology students reading psychology articles). I never gave this assignment in Berkeley so I can’t directly compare — but I would have been surprised if more than maybe 10% of my Berkeley students made comments like this.
Robin, you’re probably right. Although the articles were not in their native language, the students were quite motivated to understand them and make comments that would interest me. I am a tough grader. They couldn’t get the highest possible grade unless they told me something I didn’t know.