The Roberts-Schwartz correspondence continued. I replied to Schwartz:
“Dining establishments”? [His previous email stated: “Four restaurants simply cannot represent the variety of dining establishments in New York City”] I thought the survey was about sushi restaurants. Places where raw fish is available.
Quite apart from that, I am sorry to see such a fundamental error perpetuated in a science section. If you don’t believe me that the teenagers’ survey was far better than you said, you might consult a friend of mine, Andrew Gelman, a professor of statistics at Columbia.
John Tukey — the most influential statistician of the last half of the 20th century — really did say that a well-chosen of sample of 3 was worthwhile when it came to learning about sexual behavior. Which varies even more widely than sushi restaurants. A sample of 4 is better than a sample of 3.
Schwartz replied:
The survey included 4 restaurants and 10 stores.
The girls would not disclose the names of any of the restaurants, and only gave me the name of one store whose samples were not mislabeled. Their restaurants and stores might have been chosen with exquisite care and scientific validity, but without proof of that I could not say it in the article.
I wrote:
I realize the NY Times has an “answer every letter” policy and I am a little sorry to subject you to it. Except that this was a huge goof and you caused your subjects damage by vastly undervaluing their work. Yes, I knew the survey included 4 restaurants and 10 stores. That was clear.
As a reader I had no need to know the names of the places; I realized the girls were trying to reach broad conclusions. They were right not to give you the names because to do so might have obscured the larger point. It was on your side that the big failing occurred, as far as I can tell. Did you ask the girls about their sampling method? That was crucial info. Apparently The Times doesn’t correct errors of omission but that was a major error in your article: That info (how they sampled) wasn’t included.
He replied:
I could have been more clear on the subject of sample size, but I did not commit an error. Neither do my editors. That is why they asked me to write a letter to you instead of writing up a correction.
I don’t feel I have been “subjected to” anything, or that this is some kind of punishment. This is an interesting collision between the precise standards of someone with deep grounding in social science and statistical proof and someone who tries to write intelligible stories about science for a daily newspaper and a general interest audience. But I am not sorry that you wrote to me, even a little sorry.
i wrote:
“I did not commit an error.” Huh? What am I missing? Your article had two big errors:
1. An error of commission. You stated the study should be not taken seriously because the sample size was too small. For most purposes, especially those of NY Times readers, the sample size was large enough.
2. An error of omission. You failed to describe the sampling protocol — how those 10 stores and 4 restaurants were chosen. This was crucial info for knowing to what population the results should be generalized.
If you could explain why these aren’t errors, that would be a learning experience.
Did you ask the girls how they sampled?
His full reply:
We’re not getting anywhere here.
Not so. After complaining he didn’t have “proof” that the teenagers used a good sampling method, he won’t say if he asked them about their sampling method. That’s revealing.
Something similar happened with a surgeon I was referred to, Dr. Eileen Consorti, in Berkeley. I have a tiny hernia that I cannot detect but one day my primary-care doctor did. He referred me to Dr. Consorti, a general surgeon. She said I should have surgery for it. Why? I asked. Because it could get worse, she said. Eventually I asked: Why do you think it’s better to have surgery than not? Surgery is dangerous. (Not to mention expensive and time-consuming.) She said there were clinical trials that showed this. Just use google, you’ll find them, she said. I tried to find them. I looked and looked but failed to find any relevant evidence. My mom, who does medical searching for a living, was unable to find any completed clinical trials. One was in progress (which implied the answer to my question wasn’t known). I spoke to Dr. Consorti again. I can’t find any studies, I said, nor can my mom. Okay, we’ll find some and copy them for you, she said, you can come by the office and pick them up. She sounded completely sure the studies existed. I waited. Nothing from Dr. Consorti’s office. After a few weeks, I phoned her office and left a message. No reply. I waited a month, phoned again, and left another message. No reply.
More. In spite of Dr. Consorti’s statement in the comments (see below) that “I will call you once I clear my desk and do my own literature search,” one year later (August 2009) I haven’t heard from her.
I also recently consulted a Berkeley surgeon about an inguinal hernia. He confessed to me that he has TWO of them, both untreated for the last two years! I said, “Wow, with what you know I’d think you’d leap on the table.” He replied, “Well, I’m not sure I trust anyone else with my body.” Interesting.
Back East for a physical with a friend who is a Western MD running an integrative (wholistic) medicine practice, I learned that he has an untreated hernia, as well. He told me there’s no need to treat it.
Both doctors agreed that if it gets in the way of things you want to do, or especially if it becomes painful (a very dangerous situation), then surgery would be an option. (Even then, it can be done endoscopically with a small incision and the insertion of a piece of mesh over the weak area of the abdomnial wall.) As it is, I’m sticking with Chinese medicine which does not immediately force one to pick up a knife.
Thanks for this telling story. I really think that a doctor should be obliged to provide the patient with the actual literature reporting relevant research if asked by the patient. Of course, the medical establishment would be against this because it removes their power as the priests of the information and, perhaps more importantly from their point of view, it removes their protection from being vetted by patients who question their authority and expertise.
Aaron, yes, I completely agree. I think that’s the moral of this story. Of course a surgeon will recommend surgery — I don’t think that’s a big surprise. But claiming there is nonexistent evidence for that recommendation — that’s going too far. I could check her claim very thoroughly but most people to whom she makes such claims would have to take it on faith.
Seth, you are completely right. But…
But I’m surprised he took the time to answer – your communications seemed confrontational to me. We should embrace guys like Roberts-Schwartz – yeah, he ain’t got it all right, and yeah, it would be cool if he did, since its a science section. But he seems to value research and have a genuine scientific bent.
Thanks for publishing the correspondence.
As for the hernia issue, when I asked my surgeon about how to avoid hernias, he said there is no way to avoid them. ‘They show up, we repair them.’ Why should they be repaired? Because they may become strangulated. What is the probability of your hernia becoming strangulated? That’s the tough question. It’s the risk management decision that those of us with hernias are being forced to make, with no information.
As for the columnist, he may be a bit out of his depth having to do with appropriate sampling sizes. (Sort of like I am in this particular area.) You might want to rephrase the issue as a set of Action Items for him. Give him specific tasks. For example, if it’s important to know the geographical spread of the sushi restaurants, then write the exact questions that should be posed to the original authors. If it’s important to know the number of sushi restaurants in a particular area, ask if a Gayot search for ‘sushi’ (which shows 62 listings) is sufficient. If it’s important to know that the restaurants have different owners and not all part of the same chain, state that as a question. What would be best is if he can use this follow-up investigation to construct another article. Then he has motivation.
Seth – great to see people with responsibility for public health or the perception of it being held to account via the public medium of a blog. I am trying the same approach on my blog with a campaign to bring health food stores to account for the volumes of sugar in some of their products:
The Worst Sugar Pushers of all – Health Food Stores
So far one of them has responded and made an equally poor job as Schwartz of defending their position (also published.) Suddenly the power shifts to the letter writer and it is no longer good enough for the recipients to fob them off with inadequate and ill-judged responses….
I looked around and found what looks like a relevant hernia study: https://www.ncbi.nlm.nih.gov/pubmed/16418463?dopt=Abstract
The dirty little secret is that some doctors are not as driven by evidence as we might hope. That is why the evidence-driven medicine movement is gaining momentum. Remember how medical journals initially refused to published evidence that helicobacter pylori was a cause of ulcers and not stress?
A number of doctors pretend to be driven by evidence, but they are much more driven by a god complex that says, I’m the expert listen to me.
The sampling question is another example of enumerative versus analytic thinking in the world of statistics. Most people think that all statistics is enumerative. Refer the journalist to any of the key articles by Deming.
Sean, I’d like a reference to the key articles by Deming if you have them.
Regarding the question of whether or not one can trust a doctor’s recommendation, what about dentists? In the early 20th there were some in the profession who made claims that dentistry had reached the point in history where it was about to start doing more good than harm. Were they right? Have we ever reached that point? How can we know?
Seth, While I am in the process of finding papers in the literature to satisfy your scientific curiosity on why this hernia should or should not be fixed I am additionally trying to care for around 30 new patients referred to me for their new cancer diagnosis in the last 3 months. This may or may not explain why I have not been motivated to answer your call regarding your ambivalence about fixing your hernia. Yes, it is small and runs the risk of incarceration at some time. I will call you once I clear my desk and do my own literature search. Thanks for the update. Eileen Consorti
Dr. Consorti, to call my question “scientific” curiosity is highly unfair. I’m sure everyone, not just scientists, cares whether a proposed surgery will do more good than harm. My question is not “why” my hernia should be fixed — it’s whether it should be fixed. My calls that have gone unanswered were not about my “ambivalence” — they were trying to determine what happened to the studies you said, several months ago, you would find and copy for me. But thank you for finally responding.
While I did not have a hernia, I just wanted to say I had a horrible experience w/ Dr. Eileen Consorti. I had extra breast tissue removed from my armpit, and she did a horrible job. First off, I would like to say how insulted I was that she did not have a paper gown in her office to adequately fit my breasts. She showed up late for my surgery, did it in half the time she said it would take, and stitched me up like a one-armed blind person who’d never been to med school. I used to be self-conscious about my underarm breast, but it’s nothing compared to the hideous clown smile under my arm now. Oh yeah, and the puckers at both ends of the incision look like the nipples of a girl just entering puberty. DR. CONSORTI SUCKS!
Seth:
Here is a good PhD thesis topic:
“I. What proportion of medical treatment or procedures have a basis in evidence? II. Looking at the universe of ailments, what proportion of treatments in [1950] are recognized to be (a) harmful; or (b) useless today? III. Given II., what should we infer about accepted treatments today that are not evidence-based?”
IV. Is the current emphasis on “evidence-based medicine” helpful or harmful? I don’t mind that a doctor recommends a treatment for which there is no study showing it works. That is okay. It was that my doctor claimed the existence of non-existent studies that bothered me.
About the hernia problem, you are in good company. Clinical Evidence, the database of evidence-based medicine run by (I think) the British Medical Journal, has a monograph on “expectant management” (doing nothing) compared to repairing an inguinal hernia: “We found no systematic review, RCTs, or cohort studies of sufficient quality” to say which strategy is better for improving symptoms. Also from Clinical Evidence: “Compared with expectant management (in people with minimally symptomatic hernia) Open mesh repair may be no more effective at reducing mortality (very low-quality evidence).”
But the bigger problem is that most people have no idea how much of our clinical practice is not guided by solid evidence, and never will be because there are just too many questions and good studies are so hard to do. As a physician I often tell people “nobody really knows what the right thing to do is,” but I think it tends to make many patients uncomfortable so I find myself trying to gauge whether someone just wants my advice or is really interested in hearing my evaluation of the evidence as it may apply to them. I certainly try to be as complete as possible if people ask, and I have often been prompted by patients’ questions to do literature searches, but I don’t print out studies for people even if they ask because I don’t have the time. If they are that motivated they can do it themselves.
I do not disagree with this writing…
Johann, you write, “I don’t print out studies for people even if they ask because I don’t have the time. If they are that motivated they can do it themselves.” That makes sense. But consider this: Dr. Consorti was going to make a lot of money if I followed her recommendation. That may be why she didn’t tell me she was busy.