Month: February 2010

Dealing With Referee Reports: What I’ve Learned

Seth Roberts12 Comments

Alex Tabarrok discusses a proposal to make referee reports and associated material publicly available. I think it would be a good thing because it would make writing a self-serving review (e.g., a retaliatory review) more dangerous. If Reviewer X writes an unreasonable review, the author is likely to complain to the editor. If the paper gets published, the unreasonableness will be highlighted — and nominal anonymity may not be enough to hide who wrote it. On the other side, as a reader, it would be extremely educational. You could learn a lot from studying these reports and the replies they generated, especially if you’re a grad student. I would like to know why some papers got accepted. For example, my Tsinghua students pointed out serious flaws in published papers. Were the problems noted by reviewers and ignored, or what?

My experience is that about 80% of reviews are reasonable. Many of those are ignorant, but that’s no crime. (A lot of reviewers know more than me.) The remaining 20% seem to go off the rails somehow. For example, Hal Pashler and I wrote a paper criticizing the use of good fits to support quantitative models. The first two reviewers seemed to have been people who did just that. Their reviews were ridiculous. Apparently they thought the paper shouldn’t be published because it might call their work into question. A few reviews have appeared to be retaliation. In the 1990s, I complained to the Office of Research Integrity that a certain set of papers appeared to contain made-up data. (ORI sent the case to the institution where the research was done. A committee to investigate did the shallowest possible review and decided I was wrong. I learned my lesson — don’t trust ORI — which I applied to the Chandra case.) After that allegation, I got stunningly unfair reviews from time to time, presumably from the people I accused. A small fraction of reviews (5%?) are so lazy they’re worthless. One reviewer of my long self-experimentation paper said it shouldn’t be published because it wasn’t science. The author (me) should go do some real science.

The main things I’ve learned about how to respond are: 1. When resubmitting the paper (revised in light of the reviews), go over every objection and how it was dealt with or why it was ignored. Making such a list isn’t very hard, it makes ignoring a criticism much easier (because you are explicit about it), and editors like it. This has become common. 2. When a review is unreasonable, complain. The theory-testing paper I wrote with Hal is one of my favorite papers and it wouldn’t have been published where it was if we hadn’t complained. Another paper of mine said that some data failed a chi-square test many times — suggesting that something was wrong. One of the reviewers seemed to not understand what a chi-square test was. I complained and got a new reviewer.

I’m curious: What have you learned about responding to reviewers?

North Korea and Penn State

Seth Roberts27 Comments

In an excellent talk last week about North Korea — linked to his book The Cleanest Race — Brian Myers, a professor in South Korea, said that people don’t fear dying, they fear dying without significance. Without their life having meant something. Life in North Korea is far more attractive than Americans realize, he said. The border between North Korea and China is easy to cross, and about half of the North Koreans who go to China later return, in spite of North Korea’s poverty. How does the North Korean government do such a good job under such difficult circumstances? Partly by playing up external threats (U.S. troops in South Korea), the obvious way politicians win support, but also by telling the North Korean people they are special. Maybe it plays this card because it has to — they can’t afford a police state — but there is no denying how well it works. In contrast, Myers said, the South Korean government offers its citizens no more than consumerism. That doesn’t work well, and South Korea, in spite of high per capita income, has high rates of depression and suicide.

I think the attractiveness of North Korean life has a lot to do with why Penn State students like Penn State so much. This American Life did a show about Penn State a few months ago. Life at the nation’s top party school said the description. Sounds boring, I thought, so I waited to listen to it until I’d run out of stuff to listen to. It turned out to be one of their best shows ever. Mostly it’s about the large amount of drinking — this is why they did the show — but at the very end is a short segment about how much Penn State students love their school. Not much detail but I was convinced. The attractive school cheer (“We Are Penn State”) comes up in conversation! A few people reading this won’t know that Penn State has an extremely successful football team. A large fraction of the students attend its games. After graduation, a lot of them continue to attend the games.

Here is a powerful and neglected force in human life. The bland technical term is group identity. As the South Korea comparison indicates, governments don’t routinely use it to govern. As Penn State exceptionalism indicates, colleges don’t routinely use it either. Faculty routinely disparage football. Beer and Circus: How Big-Time College Sports Has Crippled Undergraduate Education was written by a professor — of course. The Penn State chancellor seemed mystified that his students were so proud and supportive of their school. (They’re just that way, he seemed to say.) A lot of my self-experimentation has been about discovering what we need to be healthy, such as morning faces. I can’t self-experiment about this but I would if I could. It’s yet another thing that people must have routinely gotten in Stone-Age life but don’t get any more — unless you happen to be a rabid sports fan or an alumnus of a college with a sufficiently successful football team. Or live in North Korea.

Exploratory Versus Confirmatory Data Analysis?

Seth Roberts33 Comments

In 1977, John Tukey published a book called Exploratory Data Analysis. It introduced many new ways of analyzing data, all relatively simple. Most of the new ways involved plotting your data. A few involved transforming your data. Tukey’s broad point was that statisticians (taught by statistics professors) were missing a lot: Conventional statistics focussed too much on confirmatory data analysis (testing hypotheses) to the omission of exploratory data analysis — data analysis that might show you something new. Here are some tools to help you explore your data, Tukey was saying.

No question the new tools are useful. I have found great benefits from plotting and transforming my data. No question that conventional statistics textbooks place far too little emphasis on graphs and transformations. But I no longer agree with Tukey’s exploratory versus confirmatory distinction. The distinction that matters — at least to historians, if not to data analysts — is between low-status and high-status. A more accurate title of Tukey’s book would have been Low-Status Data Analysis. Exploratory data analysis already had a derogatory name: Descriptive data analysis. As in mere description. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff — like complicated regression. That the new stuff wasn’t terribly useful (compared to graphs and transformations) mattered little. Like all academics — like everyone — they cared enormously about showing high status. It was far more important to be impressive than to be useful. As Veblen showed, it might have helped that the new stuff wasn’t very useful. “Applied” science is lower status than “pure” science.

That most of what statistics professors have developed (and taught) is less useful than graphs and transformations strikes me as utterly clear. My explanation is that in statistics, just as in every other academic area I know about, desire to display status led to a lot of useless highly-visible work. (What Veblen called conspicuous waste.) Less visibly, it led to the best tools being neglected. Tukey saw the neglect –Â underdevelopment and underteaching of graphs, for example — but perhaps misdiagnosed the cause. Here’s why Tukey’s exploratory versus confirmatory distinction was misleading: Because the tools that Tukey promoted for exploration also improve confirmation. They are neglected everywhere. For example:

1. Graphs improve confirmatory data analysis. If you do a t test (or compute a p value in any way) but don’t make an associated graph, there is room for improvement. A graph will show whether the assumptions of the computation are reasonable. Often they aren’t.

2. Transformations improve confirmatory data analysis. That a good transformation will make the assumptions of the test more reasonable many people know. What few people seem to know is that a good transformation will make the statistical test more sensitive. If a difference exists, the test will be more likely to detect it. This is like increasing your sample size at no extra cost.

3. Exploratory data analysis is sometimes thought of as going beyond the question you started with to find other structure in the data — to explore your data. (Tukey saw it this way.) But to answer the question you started with as well as possible you should find all the structure in the data. Suppose my question is whether X has an effect. I should care whether Y and Z have an effect in order to (a) make my test of X more sensitive (by removing the effects of Y and Z) and (b) assess the generality of the effect of X (does it interact with Y or Z?).

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don’t. A big simplification.

Exploration (generating new ideas) and confirmation (testing old ideas) are outputs of data analysis, not inputs. To explore your data and to test ideas you already have you should do exactly the same analysis. What’s good for one is good for the other.

Likewise, Freakonomics could have been titled Low-status Economics. That’s essentially what it was, the common theme. Levitt studied all sorts of things other economists thought were beneath them to study. That was Levitt’s real innovation — showing that these questions were neglected. Unsurprisingly, the general public, uninterested in the status of economists, found the work more interesting than high-status economics. I’m sensitive to this because my self-experimentation was extremely low-status. It was useful (low-status), cheap (low-status), small (low-status), and anyone could do it (extremely low status).

More Andrew Gelman comments. Robin Hanson comments.

Climate Science Slowly Becomes Less Settled

Seth Roberts12 Comments

Andrew Gelman, in a comment on the previous post, said that he believes the science of climate change is “much more settled” than I do. He’s right — in the sense that I believe the state of the world is different (less certain) than claimed. Andrew sees correct certainty; I see false certainty. Because science slowly becomes more accurate, I think the science will slowly shift toward “less settled” — a prediction I don’t think Andrew would make. Here’s an example of such a shift. According to the Mail on Sunday, Phil Jones

admit[s] that there is little difference between global warming rates in the Nineties and in two previous periods since 1860 and accept[s] that from 1995 to now there has been no statistically significant warming.He also leaves open the possibility, long resisted by climate change activists, that the ’Medieval Warm Period’ from 800 to 1300 AD, and thought by many experts to be warmer than the present period, could have encompassed the entire globe.

Phil Jones slowly shifts.

Confirmation of Stunning MS Claim

Seth Roberts5 Comments

I blogged earlier about an Italian med school professor named Paulo Zamboni who, studying his wife, came up with an entirely new theory about multiple sclerosis (MS): It’s caused by restricted blood outflow from the brain. Almost all MS patients had this condition, Zamboni found. The great value of this theory is that blood outflow can often be improved with surgery. In at least some cases, this surgery has reduced MS symptoms.

Now, a study done in Buffalo has found results that support Zamboni’s idea. MS patients were twice as likely as healthy people to have restricted blood flow. This is a weaker correlation than Zamboni found but I make nothing of it — there are lots of ways to mess things up, so that you get noisier results. (And there are lots of ways to push results in a preferred direction.)

Zamboni wasn’t an MS expert. He made this breakthrough because his wife had MS and he had technical skills (including surgical skills — his specialty is surgery).

Thanks to Anne Weiss.

More A more detailed description.

Elizabeth Kolbert Confronts Climategate

Seth Roberts5 Comments

The New Yorker website has a weekly podcast called The Political Scene. I’ve listened to almost all of them. This week’s was unlike any other.

The brief description is “Elizabeth Kolbert and Peter J. Boyer discuss recent attacks on climate science.” Never before have the discussants been so far apart. They should have replaced discuss with debate. Boyer hasn’t written a word about climate science — or even science. He moved from the New York Times to The New Yorker after he wrote an (excellent) book about television. Recently he’s covered politics. Kolbert has written dozens of articles and a book about climate science. In spite of this, the moderator (Dorothy Wickenden, executive editor of the magazine), asked Boyer to describe the Climategate emails and their significance. They showed, he said, “an intolerance [by the scientists] of skepticism of their narrative . . this was a real shock to the system and a real shock to the global warming consensus.” I think any unbiased observer would agree.

Then Wickenden asked Kolbert what she thought:

KOLBERTÂ I don’t agree with him [Boyer] . . . One of the things that comes out in these emails is the climate scientists’ frustration with having to deal with people who use the data in all sorts of irresponsible ways. . . I’m not aware of any instances where people have had to go back and had to say “you’re right and the conclusion we drew was wrong.”

BOYER Perhaps we could say that language was used in these communications that would allow for an interpretation that perhaps there was fudging or something going on that needed to be obscured. There was a whole tone of intolerance of questioning of their data or — and this was what was so disturbing to hear from scientists — any questioning of what sounded an awful lot like their mission.

Boyer went on and on — as if he were the expert. (And he clearly knew what he was talking about.) Then Wickenden turned to the United Nations IPCC report and asked Kolbert what she thought of recent criticism (which Wickenden learned about from the NY Times).

KOLBERT . . . [The error was in Part 2.] In [Part 2 of] this report, which was literally 986 pages long, there were a couple of things inserted that weren’t from the peer-reviewed literature. . . .

BOYER Well, Betsy, I’m sorry, these aren’t just 986 pages of Scripture, and then a couple of little awkward errant notations on the side. The IPCC isn’t an inconsequential body. Al Gore and Mr. Pachauri shared the Nobel Prize. They are granted a level of authority when they speak. These reports were certainly granted authority. . .

KOLBERT [interrupting] I guess I should ask you: What is your point? . . .

BOYER . . . The consensus about the consensus has begun to crack. That’s just the political reality . . . There is a crack in the consensus.

Kolbert has published hundreds of thousands of words about global warming in the most prestigious magazine in the world. That she is unable to see or at least say this basic truth but must have someone else say it is another sign of problems with her reporting.

Until now, all speakers on The Political Scene have sounded calm and confident. On rare occasions they disagree, but never like this. And the conversation always has a relaxed tone. Not this time. Boyer sounded calm and confident but I thought Kolbert sounded nervous and upset. With good reason: It struck me as a huge and public rebuke from her employer. She’s been the expert. Now someone with no credentials has been allowed to say she’s wrong — has been brought on the program, apparently, in order to disagree with her. As if it’s no longer clear she’s right. And her dismissal of the Climategate emails, as if they taught her nothing, didn’t help her. The debate with Boyer was preceded and followed by softball questions by Wickenden to Kolbert. They struck me as attempts to soften the blow, as did a comment at the end by Boyer about a Super Bowl commercial.

Written With A Straight Face? Dept.

Seth Roberts18 Comments

Jonathan Cole used to be provost of Colombia University. He has written a book called T he Great American University, in which, according to this review,

He lists their dazzling achievements, which in biology and medicine include findings on gene-splicing, recombinant DNA, retroviruses, cancer therapies, coch­lear implants, the fetal ultrasound scanner, the hepatitis B vaccine, prions, stem cells, organ transplantation and even a treatment for head lice. . . . In a chapter on the social sciences, he cites, among many others, such useful innovations as theories of human capital and social mobility, research in linguistics and even the use of prices to reduce traffic jams.

“Research in linguistics”? Yes, that sounds dazzling. I’m sure those “theories of human capital” have been v v “useful”. And who would have thought that if you raise the price of something (“use of prices to reduce traffic jams”) . . . people use less of it? Which was traffic engineering, not social science. Did the reviewer, an economics professor at Harvard named Claudia Goldin, write this with a straight face?

The “dazzling achievements” in biology and medicine are only slightly less unconvincing.”Gene splicing” and “recombinant DNA” research are different names for the same thing. Fetal ultrasound scanners may cause autism. Vaccines were not invented by an American university professor. The discovery of prions has had no obvious non-laboratory use — besides being questionable. Stem-cell research has yet to produce anything of use outside of labs. To be fair, gene splicing has been used to produce human insulin, which is better than the insulin previously available, but conspicuously absent from the list of accomplishments is prevention of diabetes — not to mention allergies, obesity, depression, arthritis, stroke, or any of the other lifestyle problems that a large fraction of Americans suffer from. Such achievements would be truly useful. Great American universities haven’t given us any of those . . but they have given us a treatment for head lice.

There’s a reason for the term ivory tower. Apparently Cole, conscious of the term, is trying to argue against it — but merely shows why it exists. (I’m assuming the review is accurate.) It reminds me of the time that top Chinese students, visiting top American colleges such as Harvard and Yale, found the American students ignorant and arrogant. The theme of Cole’s book is that American universities are in trouble and need more support. What useful stuff they’ve accomplished is central to his argument. When I was an undergrad, I read Thorstein Veblen’s bitter The Higher Learning in America, which said American universities were dysfunctional. He mentioned “committees for the sifting of sawdust.”

More “Graduate school in the humanities is a trap” (via Marginal Revolution).

A GAPS Testimonial

Seth Roberts4 Comments

Gut And Psychology Syndrome (GAPS) is a book by Natasha Campbell-McBride about how to treat allergies, autism, and similar conditions. In this entry, Cheeseslave talks about her own son and then quotes another mother about the effects of the GAPS diet (plus other changes) on her autistic son.

Kevin lacked oxygen at birth, so in the first year of life, I already saw that he was not developing like my other kids (he is our 5th). His motor skills lagged and he cried a lot, didn’t sleep so well, etc.At two, his behavior was just not right. He never responded right to correction, would throw things in anger or frustration, cried all the time, especially when waking up, basically never happy. He didn’t walk until two and then he would fall down constantly.

He also began to always be starving. When he was really hungry, his face would get distorted and frozen in a strange way. I now think he was having seizures of sorts.

We did not vaccinate at all and we figured out that if we fed him lots of protein type foods like meats, he would relax his body and face and be able to go play for a bit until it happened all over again in a short time.

I do think that because we didn’t vaccinate and figured out to keep feeding him this way, we were able to “coast along” like this for years. He had learning disabilities, lacked social skills and continued to have autistic traits like sensory issues, hiding under blankets, reacting to sounds, not liking people around, rigid in routines, and spinning and going on his head along with head banging.

Long story shorter, we did get a diagnosis of Aspergers at one point. We took him to doctor after doctor, specialist after specialist to no avail. He also strangely was NEVER once sick (we later learned that his immune system was not working a bit).

At 9 years old, he got pneumonia, followed by asthma and allergies. His eating [problems] had escalated to the point of feeding him every 20-30 minutes or he would have gigantic meltdowns. We eventually could not even have people over.

He was given an inhaler for the asthma and suddenly, without us making the connection, he began to not respond when called, became extremely hyperactive and began to run away at all hours of the day and night requiring police to find him and being very dangerous. (we once lost him in the middle of downtown Chicago). He would also try to jump out of moving vehicles, out of windows and required constant restraining.

The seizures got bad, he would fall down the stairs and lose consciousness several times per day. They tried psych drugs and he almost died twice from his reaction to them (I am now grateful that we couldn’t go that route).

We became so desperate that we brought him home from hospital and got deadbolts to keep him from running, did all our own restraining and called alternative docs to help us.

We began kefir and diet from nutritionist (basically a BED [Body Ecology Diet]/GAPS version), took him off inhaler. His allergies were totally out of control, he could barely open his eyes from swelling, and his chin was deformed and swollen, his belly too, his whole body. He would only eat junk food and fast foods and it was incredibly difficult to transition him to the diet.

The DAN (Defeat Autism Now) protocols we followed, made him worse in lots of ways b/c the chelation made him extremely violent, the B12 shots kept him awake for nights on end without any sleep, the antifungals and all those other interventions were nightmarish for him.

Eventually, I resolved to use only foods and do this without any kind of doctors. So for this past year, I researched and researched and was determined to bring him back from this state. We have done a combo of GAPS (and BED) very successfully along with lots of fermented foods and drinks.

The allergies and asthma are 100% gone, the seizures we have had only one in 65 days and very mild (compared to 5-10 per day). He sings every morning and has cried once in the last 2.5 months (he used to cry for 1-3 hours at a time each day) and he can go outside again without running away. He is in martial arts, acting appropriately at church, having eye contact, no autistic traits of late and learning academics after two years of not being able to open a book. He reads before bed at an 8th grade level.

This story has many interesting elements. 1. Huge improvement. Very plausible that it’s due to the dietary change. 2. Autism and allergies go away at the same time, suggesting same cause. 3. Treatment with fermented foods. 4. A different “radical” solution failed, meaning there is no reason to think this is a placebo effect. 5. The mystery of why an inhaler made things worse. 6. Autism not due to vaccination.