One of my colleagues at Berkeley didn’t return library books. He kept them in his office, as if he owned them. He didn’t pay bills, either: He stuck them in his desk drawer. He was smart and interesting but after he failed to show up at a lunch date — no explanation, no apology — I stopped having lunch with him. He died several years ago. At his memorial service, at the Berkeley Faculty Club, one of the speakers mentioned his non-return of library books and non-payment of bills as if they were amusing eccentricities! I’m sure they were signs of a bigger problem. He did no research, no scholarly work of any sort. When talking about science with him — a Berkeley professor in a science department — it was like talking to a non-scientist.
David Freedman, a Berkeley statistics professor who died recently, was more influential. He is best known for a popular introductory textbook. The work of his I found most interesting was his comments on census adjustment: He was against adjusting the census to remove bias caused by undercount. This was only slightly less ridiculous than not returning library books — and far more harmful, because his arguments were used by Republicans to block census adjustment. The undercounted tended to vote Democrat. The similarity with my delinquent colleague is the very first line in Freedman’s obituary: He “fought for three decades to keep the United States census on a firm statistical foundation.” Please. A Berkeley statistics professor, I have no idea who, must have written or approved that statement!
The obituary elaborates on this supposed contribution:
“The census turns out to be remarkably good, despite the generally bad press reviews,” Freedman and Wachter wrote in a 2001 paper published in the journal Society. “Statistical adjustment is unlikely to improve the accuracy, because adjustment can easily put in more error than it takes out.”
There are two kinds of error: variance and bias. The adjustment would surely increase variance and almost surely decrease bias. The quoted comments ignore this. They are a modern Let Them Eat Cake.
Few people hoard library books, but Freedman’s misbehavior is common. I blogged earlier about a blue-ribbon nutrition committee that ignored evidence that didn’t come from a double-blind trial. Late in his career, Freedman spent a great deal of time criticizing other people’s work. Maybe his critiques did some good but I thought they were obvious (the assumptions of the statistical method weren’t clearly satisfied — who knew?) and that it was lazy the way he would merely show that the criticized work (e.g., earthquake prediction) fell short of perfection and fail to show how it related to other work in its field — whether it was an improvement or not. As they say, he could see the cost of everything and the value of nothing. That he felt comfortable spending most of his time doing this, and his obituary would praise it (“the skeptical conscience of statistics”), says something highly unflattering about modern scientific culture.
For reasonable comments about census adjustment, see Eriksen, Eugene P., Kadane, Joseph B., and Tukey, John W. (1989). Adjusting the 1980 census of population and housing. JASA, 84, 927-943.
You seem to be saying that Freedman was a bad person because you disagree with his view on censuses and because republicans agree with him. You think it obvious that he was wrong, but I don’t see anything obviously wrong with it.
“Bad person”? No, bad behavior. You’re right, Dennis, that he was wrong might not be obvious to a someone not professionally involved with estimation — and I suppose at least a few statistics professors even agreed with him. (But when I brought up Freedman’s position with my friends who were statistics professors, they agreed with me.) That’s why I gave a citation with a detailed explanation of why his position was wrong. No one doubted that the census undercounted some people more than others. Freedman was against fixing this so that you could no longer predict who was undercounted. Suppose a store systematically overcharged you. What would you think of someone who opposed solving the problem using the best methods available so that, on average, you were charged the correct price? That Republican politicians agreed with him is not part of why I dislike his behavior. It was why his behavior was influential.
Seth,
He may have been a bastard, he may have been ideologically blinded. But the time to address those issues was when he was alive.
Obituaries are romanticized for the benefit of the survivors.
–Ed
Ed, I made these points when Freedman was alive, too. Many times. I hope you’re not saying no one should criticize a dead person or the culture that produced him.
I’m not saying he was a bastard or ideologically blinded, certainly not. For all I know he was a Democrat.
The problem with the obituary is not “romanticization”. My point is not that Freedman did this stuff but that other professors saw this as okay, even good. My point is about the culture, in other words. It is one reason academia is called the ivory tower. Romanticization in obituaries is fine. Romanticization would have been to leave the census stuff out of the obituary. Freedman did a lot of perfectly good other stuff.
Seth,
I didn’t mean to say that one shouldn’t criticize the dead.
And you didn’t call Freedman a bastard. Anyone’s obituary, bastard or ideologue is romanticized.
I think I understand now… The issue is the “Census Stuff” that should have been frowned upon and challenged, wasn’t, even worse it was celebrated in his obituary. Didn’t get that on first or second reading of the post, maybe I need more flaxseed oil.
Anyway, thank you for your posts on academia.
–Ed
It is not clear if census adjustment would have reduced bias (at the cost of variance) as you state. Indeed my bet is that David Freedman is correct: model adjustment would have made the bias worse. The issue is not one of just random sampling. Much model based extrapolation was being proposed. The devil is in the details, and vague gestures do not an argument make. A hint of the complexity is that by the 2000 census the partisan impact of the undercount was probably the reverse of what you conjecture. For the purposes of apportionment, the south west (e.g., states like Texas) would have gained Congressional seats and hence electoral college votes at the expense of the north east. So, a correction would have been a win for the Republicans in national elections. Note that apportionment counts illegal immigrants, although they cannot vote. Much like many academics, the political parties took their positions based on weak evidence and held their ground even when the facts shifted. Of course the scientific issue is that the undercount is difficult to fix and who is being undercounted is constantly changing. The problem is more difficult than calculating the partisan impact, and most people can’t even get that part right.
Jas, where is it argued that census adjustment was more likely to increase bias than reduce it? I have never heard that argument nor do I understand it.
Freedman argued that census adjustment would introduce bias of unknown magnitude. He proved this by following essentially the same methodology but making minor changes, and showing that substantial differences would arise in the adjustment. As Jas says, the devil is in the details, which must be understood here to follow what is going on.
Following the census, there is a survey which attempts to measure the undercount which occurred in the census. This survey itself is subject to the same undercount problem as the census. A fundamental assumption which makes adjustment possible is that these two undercounts are INDEPENDENT. Freedman argues that this is not true — a person who is undercounted in the census is also likely to be undercounted in the subsequent survey. This makes eminent sense. If there is correlation between the undercounts, this will lead to bias in the adjustment. How large this bias can be is assessed in the paper cited below, which shows that it is likely to be so large as to make the adjustment worse than useless.
K.W. Wachter and D.A. Freedman. “The fifth cell: Correlation bias in U.S. census adjustment.” Evaluation Review, vol. 24 (2000) pp. 191—211
I don’t think anyone is disputing whther biases exist in the census adjustments. On any given census you can be sure that it will be biased one way or another depending on the methodologies used. Given the likelihood that undercounts are correlated, I agree with Mr. Zaman, Wachter and Freedman, in that the adjustments are likely “less than useless”.
taw
Terry, you wrote “given the likelihood that undercounts are correlated” — I don’t think anyone disputes this either. This is the puzzle behind Freedman’s criticisms: he seemed to equate lack of perfection with uselessness. I haven’t read the Wachter and Freedman yet, but I will, and then after that maybe I will understand Freedman’s criticisms better.
Once the decision is made to adjust the census results, the adjustment will be just one more political issue. There is no guarantee that unbiased statistical methods will be used. Do you think the average (or above-average) Congressman understands the difference between variance and bias? Would he even want to? All he has to know is how to get his party’s statisticians on the committee that does the adjustments.
On a related question, how would everyone feel about using statistical methods to adjust the vote totals in elections? The arguments would be similar. A lot of people don’t vote, we could get a more accurate outcome with adjustment. Right?
To summarize: you read Freedman’s conclusions about the census in the obituary, decided they were self-evidently false without reading the arguments supporting them, and concluded that he was a bad and/or dishonest statistician. Now there’s good science!
Mike, I knew about Freedman’s census ideas long long before his obituary.
Asad Zeman, I’ve now read the citation you gave. You are wrong when you write “a fundamental assumption that makes adjustment possible is that these two undercounts are INDEPENDENT.” The assumption that makes adjustment possible is that the two undercounts are not PERFECTLY CORRELATED. Which is far more plausible. Wachter and Freedman, as far as I can tell, ignore this point. No sane person ever thought census adjustment would eliminate bias; they merely thought it was more likely to reduce it than increase it.
https://www.stat.berkeley.edu/users/census/612.pdf
provides detailed arguments explaining how census adjustments can be harmful. Your rhetoric is not just to disagree with Freedman, but to claim the conclusions are “ridiculous.” If you want to take the scientific high ground, say something about the actual arguments in the paper.
Mike, thanks for the reference. I’ve read that paper. Before I read it I understood that adjustments are inevitably inaccurate and thus increase the variation of the final estimate. Their value is that they are more likely to decrease bias than to increase it — at least, this is the heart of the argument for adjustment. Going on and on about whether the adjustments increase or decrease “accuracy” as that paper does is just a distraction.
Technically you are right that there is a possibility of adjustment when the sample/resample undercounts are not perfectly correlated.
Practically speaking, I do not know of any census adjustment schemes which allow for and adjust for such dependence. Erikson-Kadane et. al (EK). whom Freedman was arguing against, did assume independence in producing their adjustments.
It is strange that the central issue: whether or not an adjustment will produce a more accurate count, is viewed as a distraction by you.
I propose to subtract random numbers to adjust the census — this is likely to improve the census, since it is known that there is an undercount.
EK proposed a scheme which, as Freedman demonstrated, was essentially equivalent. He showed this in three different ways.
A: Following exactly the same methodology, but using an alternative series to the one picked by EK (out of 18 equivalent series) produces seriously different adjustments.
B: Changing some of the theoretical assumptions (in particular independence) required for the adjustments to more plausible ones, leads to seriously different numbers for adjustment.
C: “Shrinkage,” the Bayesian methodology for producing adjustments used by EK requires specification of a prior distribution. This is elegantly finessed via the empirical Bayes method actually used by EK, where prior assumptions are hidden at a deeper level. By changing these, one can arrive at substantially different numbers. In particular, someone who knows the game well can produce numbers which favor one state over another or to suit his own political preferences.
“Whether adjustment will produce a more accurate count.” As I said, to talk about “accuracy” is confusing. Accuracy has two components: variance and bias. Adjustment will very likely increase variance and reduce bias.
The EK scheme was “equivalent” to a scheme where you “subtract random numbers”? Another confusing statement. Random numbers can be anything.
Seth,
As you say, adjustments will likely increase variance and reduce bias. A large increase in variance may not be worth a small reduction in bias, so it is not clear that adjustments are a good idea. To talk about “accuracy” is simply to talk about the net effect, which is what we care about. It seems much more misleading to focus on only one component at a time.
Mike, the average person doesn’t know there are two components. Lumping them together under the name of “accuracy” glosses over the question of how to weight them. I don’t see how it is misleading to say that adjustments will likely increase variance and reduce bias. Complicated, yes, misleading, no.
Seth,
You initially claimed that it was “ridiculous” to recommend against trying to remove bias. It is not ridiculous if the attempt to reduce the bias increases the variance so much that the mean squared error increases overall, which is exactly what the quote (and the paper) from Freedman says. I bring up mean squared error since that is the thing of which variance and (squared) bias are components. Freedman didn’t say “mean squared error” since the average person would not have understood that. Instead he said “accuracy.” The misleading quote from you is “He was against adjusting the census to remove bias caused by undercount. This was only slightly less ridiculous than not returning library books.” You have offered no argument for this statement; Freedman on the other hand, offers a detailed argument for his position in the paper cited.
Mike, your “mean squared error” is only one of an infinity of ways of combining bias and variance to get an overall measure of goodness/badness. Given that we value fairness (lack of bias) more than uniformity (lack of variance), it is hardly an obvious choice. Freedman’s idea that his particular relative weighting of bias and variance was so sure — on such a “firm statistical foundation” — as to not need mentioning or discussion, that no reasonable person could disagree, really does resemble not returning library books in its self-centeredness. For more about why I thought Freedman’s position resembled not returning library books, read the rest of the post.
Seth,
Mean squared error is indeed arbitrary, as is the notion of variance itself. That is not the main point. The main point is that pretty much however you would like to define “total error,” it is possible to increase the total error while reducing bias (unless the only thing you care about is bias, in which case what you want to do is pick a really random sample of one person).
Freedman argues that this actually happens, for a reasonable (and conventional, though arbitrarily so) measure of total error, when the usual techniques are applied to adjust the census. You are claiming, I think, that the bias-reduction benefits of census adjustment outweigh the costs of increasing other kinds of error. This isn’t self evidently true or false. Freedman has given some reasons for believing it is false. What are your reasons for believing it is true?
Mike, no I’m not claiming that. I’m claiming that it was a bad idea to omit the bias/variance distinction when discussing the “accuracy” of the census and the effects of adjustment.
In a certain way I agree with you. It doesn’t necessarily resemble not returning library books to claim the census shouldn’t have been adjusted. If someone made clear the bias/variance difference and explained their relative weighting scheme — why they chose to weight bias like this and variance like that — that would be reasonable. I saw nothing like that in Freedman’s writings.
Section 14 of the paper discusses estimation of bias and variance from adjustment methods, and makes explicit that they are being combined in the conventional way: by adding the squared bias to the variance to get mean squared error.
Maybe mean squared error is not the right way to measure accuracy, but it’s not like Freedman invented it just to make his argument! In fact, he’s responding to arguments in favor of adjustment which also cast the discussion in this framework. For better or for worse, most of the common statistical tools (mean, variance, regression) are closely tied to MSE.
At least, MSE has the virtue that the random and systematic errors can be separated additively, which makes it a lot easier to discuss their relative contributions. If, for example, we were primarily interested in mean absolute error, another reasonable candidate, we would have no nice way to decompose it into systematic and random parts, and “variance” (MSE of an unbiased estimate) would have no particular importance.
So, if your criticism of Freedman boils down to “he used MSE without justifying it” – I agree he is guilty of that, along with most of the rest of the statistical profession. However, it doesn’t seem to justify the harsh words in your original post.