Category: General

The Parable of the SAMe

Seth Roberts4 Comments

SAMe is a drug well known to help depression. For example, “a popular dietary supplement called SAMe may help depressed patients who don’t respond to prescription antidepressant treatment, a new study shows.” But there’s something important few people know about SAMe.

While talking to a Seattle woman about how Vitamin D3 first thing in the morning helped her with depression, she told me the following story:

When I was 47, I just wanted to be healthier. I kept gaining weight. I knew what foods are healthy. I just didn’t seem to eat them. A naturopath suggested SAMe. I tried it — Twin Labs SAMe. That was really fabulous for me. For the first time I got a glimpse of what being not depressed was like. Cravings weren’t there any more. Went from a size 24 to a size 14. Lost 70 pounds. I’m 5′ 8″. I didn’t feel deprived. I was eating plenty of food. going to yoga. Feeling really great.

Then Twin Labs discontinued it. It was made in Japan. I tried every other SAMe out there, eight different brands. None of them worked. I gave each of them a month. I tried different dosages.

I started slipping back into depression. Not being able to cope. I was sleeping more. Sugar cravings returned.

[why did Twin Labs stop making it?]

It wasn’t a good seller for them. So fucking wrong. I wrote letters to try to get them to start making it again. I did a campaign. People found pockets of what was left in the country and sent it to me. But it finally ran out.

The moral(s) of the story? 1. So much for word of mouth. You might have thought it would make the good SAMe sell well, better than the bad SAMe. Apparently not. 2. So much for the placebo effect. 3. Clinical studies (e.g., of SAMe) may higher-quality versions of what they are testing than the versions available to the rest of us. 4. So much for quality control in the supplement industry — except maybe in Japan. There can be substantial quality variation among supplements, undetected by the industry. I have to believe the companies selling the useless SAMe didn’t realize it. Surely they thought that good SAMe would be a better product for them than bad SAMe.

This resembles the Vitamin D3 story I have been telling. Tara Grant said she’d heard countless times that Vitamin D is good. She hadn’t heard once that it must be taken in the morning. I’ve heard countless times that SAMe is good. This was the first time I heard about huge quality control issues. In both cases individual self-observation uncovered a crucial truth that an industry had overlooked. They didn’t want to miss it. The Vitamin D Council didn’t want to miss the time-of-day effect. They just did.

This also resembles what I said about ultrasound machines: A lot of them are broken, unbeknownst to their operators and the people (often pregnant women) being scanned. The countless “experts” (doctors) who recommend ultrasound don’t seem to know this.

Which is why personal science (trusting data, not experts) is more valuable than experts want you to think.

What Is a Good Word For This?

Seth Roberts45 Comments

Can you help me? I am looking for a word — maybe a new word — to describe the transformation of an activity from (a) something done only by trained specialists, as part/all of their job to (b) something done by the general public, not as a job. For example:

  • word processing software has made producing an attractive manuscript something that you no longer need hire a secretary to do — you can do it yourself.
  • digital cameras and software have made producing high-end photographs something you no longer need a professional photographer to make.
  • When I was a graduate student I hired a professional to make publication-quality figures for my scientific papers. Now I make them myself.

The transition I am talking about is part of a longer historical sequence that goes like this:

  1. Hobby
  2. Part-time job
  3. Full-time job
  4. Specialization (= division of labor)
  5. [new word goes here]

The best word I can think of is deprofessionalization. Unfortunately that has been used with a different meaning. Amateurization doesn’t work because amateur often means hobbyist. Popularization doesn’t work because the status of the activity has changed — from something done as part of a job to something done not as a job. It is one of several ways a job can change:

  • More efficient. New tools, materials, etc., make it possible to do the same job in a shorter period of time or at lower cost.
  • Higher quality. New tools, etc., make it possible to do a better job.
  • More exclusive (= higher barriers to entry). Something (e.g., licensing requirements) makes it harder for others to compete with you.
  • Less exclusive. Something (e.g., the Internet) makes it easier for others to compete with you.
  • ???. People no longer need to hire you or someone like you to do what you do. They do it themselves.

I care because personal science (science done to help oneself) is an example. For a long time, non-trivial science was done only by professional scientists. Now it is being done by non-professionals.

More What about publicization? Or is it too ugly? I looked up democratization as a possibility but found this under “democratization of photography”:”Serious photography has gone from being the preserve of the reasonably well off to something that just about anyone can take up with minimal expense”. That isn’t what I mean here — that the price of something comes down. Hoipolloization is too long. What about massification?

Still More It really is DIY, I hadn’t thought of that. That exactly conveys the transition from job to non-job. DIYing (or should it be DIYization?) has a nice ring to it, is very short, is not pompous, and would not need to be defined. I also like promethization, deguilding, democratization, and deprofessionalization.

Vitamin D3 in Morning Has No Clear Effect on Sleep (Story 12)

Seth Roberts12 Comments

Alex Chernavsky, who has used the Shangri-La Diet successfully for two years, recently commented as follows (emphasis added):

For what it’s worth, I’ve taken Vitamin D at different times of the day, and I’ve never noticed any effect on my sleep. Of course, my sleep is already pretty good, in the sense that I fall asleep quickly and don’t usually wake up during the night. (My sleep is not good in the sense that I don’t get enough of it.)

By email, I learned that Alex is now taking Vitamin D3 — this particular product, which is vegan (“plant-source”) — at 5000 IU every other day. On weekdays, he takes it at about 8:00 am, on weekends, 9:30-10:00 am.

What might explain Alex’s failure to notice better sleep?

1. Not enough D3. I found that 2000 IU/day had no noticeable effect, whereas 4000 IU/day did produce noticeable benefit. Alex is getting 2500 IU/day — or less, if he takes it too late on the weekends.

2. His source of D3.

3. Individual differences large enough to matter. If you do sensitive psychology experiments, you will learn there are individual differences in everything.

4. Ceiling effect. Alex’s sleep is too good to notice improvement.

Those are the just the obvious possibilities.

Google Analytics: “Make This Version Default”

Seth Roberts4 Comments

I use Google Analytics to measure web traffic to my blog, forums, and website. A few months ago a new version was introduced and, as far as I can tell, made the default. The new version is worse than the old version. Every time I use GA, I click on “old version”.

In an excess of confidence, the new version isn’t merely the default, it also has a link called “make this version default”. The old version has no such link. There seems to be no way to make the old version the default. It is a Google version of “this sentence is false.”

More Apparently someone from Google read this or was told about it. The problem has been fixed.

Even More And then someone changed it back — it remains unclear how to make the old version the default.

SOPA Strike

Seth Roberts6 Comments

SOPA is an example of what Thorstein Veblen called “the vested interests” trying to prevent change. In an essay called “ The Vested Interests and the Common Man” he pointed out “the existence of powerful vested interests which stand to gain from the persistence of the existing, but outdated system of law and custom.” Jane Jacobs said much the same thing. The most important conflict in any society, she wrote at the end of The Economy of Cities, isn’t between the rich and poor or management and labor; it is between those who benefit from the status quo and those who benefit from change. If those who benefit from the status quo usually win, problems stack up unsolved.

iTunes For Windows is Horrible

Seth Roberts13 Comments

May I interrupt my usual posts to complain about something? Something minor?

It is that iTunes for Windows — from Apple, the maker of what are said to be brilliantly-designed products — is horribly designed. I have two examples.

1. Suppose I want to see what’s in the iTunes Store. I open a new window. I can’t close that window without closing iTunes! And if, after closing the whole program, I open it again, it still gives me the Stores window! Maybe the Stores window went away after a few weeks…I don’t want to even think about it.

2. I pressed the wrong button and started 181 downloads. There is no way to cancel them! If I stop the whole program, they will resume the next time I start it. This is software design from the 1960s.

And this is iTunes version 10.something, not version 0.3.

 

Butter and Arithmetic: How Much Butter?

Seth Roberts25 Comments

I measure my arithmetic speed (how fast I do simple arithmetic problems, such as 3+ 4) daily. I assume it reflects overall brain function. I assume something that improves brain function will make me faster at arithmetic.

Two years ago I discovered that butter — more precisely, substitution of butter for pork fat — made me faster. This raised the question: how much is best? For a long time I ate 60 g of butter (= 4 tablespoons = half a stick) per day. Was that optimal? I couldn’t easily eat more but I could easily eat less.

To find out, I did an experiment. At first I continued my usual intake (60 g /day). Then I ate 30 g/day for several days. Finally I returned to 60 g/day. Here are the main results:

The graph shows that when I switched to 30 g/day, I became slower. When I resumed 60 g/day, I became faster. Comparing the 30 g/day results with the combination of earlier and later 60 g/day results, t = 6, p = 0.000001.

The amount of butter also affected my error rate. Less butter, less errors:

Comparing the 30 g/day results with the combination of earlier and later 60 g/day results, t = 3, p = 0.006.

The change in error rates raised the possibility that the speed changes were due to movement along a speed-accuracy tradeoff function (rather than to genuine improvement, which would correspond to a shift in the function). To assess this idea, I plotted speed versus accuracy (each point a different day).

If differences between conditions were due to differences in speed-accuracy tradeoff, then the points for different days should lie along a single downward-sloping line. They don’t. They don’t lie along a single line. Within conditions, there was no sign of a speed-accuracy tradeoff (the fitted lines do not slope downward). If this is confusing, look at the points with accuracy values in the middle. Even when equated for accuracy, there are differences between the 30 g/day phase and the 60 g/day phases.

What did I learn?

1. How much butter is best. Before these results, I had no reason to think 60 g/day was better than 30 g/day. Now I do.

2. Speed of change. Environmental changes may take months or years to have their full effect. Something that makes your bones stronger may take months or years to be fully effective. Here, however, changes in butter intake seemed to have their full effect within a day. I noticed the same speed of change with pork fat and sleep: How much pork fat I ate during a single day affected my sleep that night (and only that night). With omega-3, the changes were somewhat slower. A day without it made little difference. You can go weeks without Vitamin C before you get scurvy. Because of the speed of the butter change, in the future I can do better balanced experiments that change conditions more often.

3. Better experimental design. An experiment that compares 60 g/day and 0 g/day probably varies many things besides butter consumption (e.g., preparing the butter to eat it). An experiment that compares 60 g/day and 30 g/day is less confounded. When I ate less butter, I ate more of other food. Compared to a 60 g/0 g experiment, this experiment (60 g/30 g) has less variation in other food. Another sort of experiment, neither better nor worse, would vary type of fat rather than amount. For example, replace 30 g of butter with 30 g of olive oil. Because the effect of eliminating 30 g/day of butter was clear, replacement experiments become more interesting — 30 g/day olive oil is more plausible as a sustainable and healthy amount than 60 g/day.

4. Generality. This experiment used cheaper butter and took place in a different context than the original discovery. I discovered the effect of butter using Straus Family Creamery butter. “One of the top premium butters in America, ” says its website, quoting Food & Wine magazine This experiment used a cheaper less-lauded butter (Land O’Lakes). Likewise, I discovered the effect in Berkeley. I did this experiment in Beijing. My Beijing life differs in a thousand ways from my Berkeley life.

The results suggest the value of self-experimentation, of course. Self-experimentation made this study much easier. But other things also mattered.

First, reaction-time methodology. In the 1960s my friend and co-author Saul Sternberg, a professor of psychology at the University of Pennsylvania, introduced better-designed reaction-time experiments to study cognition. They turned out to be far more sensitive than the usual methods, which involved measuring percent correct. (Saul’s methodological advice about these experiments.)

Second, personal science (science done to help yourself). I benefited from the results. Normal science is part of a job. The self-experimentation described in books was mostly (or entirely) done as part of a job. Before I collected this data, I put considerable work into these measurements. I discovered the effect of butter in an unusual way (measuring myself day after day), I tried a variety of tasks (I started by measuring balance), I refined the data analysis, and so on. Because I benefited personally, this was easy.

Third, technological advances. Twenty years ago this experiment would have been more difficult. I collected this data outside of a lab using cheap equipment (a Thinkpad laptop running Windows XP). I collected and analyzed the data with R (free). A smart high school student could do what I did.

There is more to learn. The outlier in the speed data (one day was unusually fast) means there can be considerable improvement for a reason I don’t understand.

The Genomera Buttermind Experiment.

Dr. Eileen Consorti and Patient Power

Seth Roberts30 Comments

My alternative to Testing Treatments (199 pages), I said recently, is three words: Ask for evidence. Ask your doctor for evidence that their recommendation (drugs, surgery, etc.) is better than other possibilities. A few years ago, I asked Dr. Eileen Consorti, a Berkeley surgeon, for evidence that the surgery she recommended (for a hernia I couldn’t detect) was a good idea. Surgery is dangerous, I said. What about doing nothing?

To reread what I’d written about this (here and here), I googled her. I learned she has a blog. It contains only one post (June 21, 2011). That post is only seven words long. I also learned she has two very similar websites (here and here). Both use her full name and title where most people would use she. Perhaps I caused the blog and websites.

Here’s what happened:

1. In 2008, during a routine physical, my primary-care doctor finds that I have a hernia, so small I hadn’t noticed it. He says I should see Dr. Consorti. Do I need surgery for something so small? I ask. Ask her, he says.

2. Dr. Consorti examines my hernia. She recommends surgery (that she would perform). Why? I ask. It could get worse, she says.

3. Eventually I realize that’s a poor reason. Anything can get worse. Influenced by Robin Hanson, I speak to Dr. Consorti: Surgery is dangerous. What about doing nothing? Is there evidence that the surgery you recommend is beneficial? Dr. Consorti says, yes, there is evidence supporting her recommendation. She says I can find it (studies that compared surgery and no surgery) via Google.

4. I try to find the evidence. I use Google and PubMed. I can’t find it. My mom, who used to be a medical librarian at UC San Francisco, is an expert at this. She has done thousands of medical searches. She too cannot find any studies supporting Dr. Consorti’s recommendation. Moreover, she finds an in-progress study that compares surgery for my problem with doing nothing. Apparently some researchers think doing nothing may be better than surgery.

5. I tell Dr. Consorti that my mom and I couldn’t find the studies she said exist. Dr. Consorti says she will find them. She will let me know when she’s found them and make copies. I can pick them up at her office.

6. Months pass. I call her office twice. No response.

7. In August 2008, I blog about Dr. Consorti’s continuing failure to produce the studies she seemed sure existed.

8. A reader named kirk points out “ what looks like a relevant hernia study“. It concludes: “Watchful waiting is an acceptable option for men with minimally symptomatic inguinal hernias. Delaying surgical repair until symptoms increase is safe.” This argues against Dr. Consorti’s recommendation. No one points out studies supporting her recommendation.

9. Two weeks after my post, someone who appears to be Dr. Consorti replies. She’s busy. She has 30 new patients with cancer. She terms my question “scientific curiosity”. She says “I will call you once I clear my desk and do my own literature search.”

10. More than a year passes. In 2010, I receive a call from Dr. Consorti’s office. An assistant asks me to remove my blog post about her failure to provide the studies. Why? I ask. It makes her look bad, he says. He says nothing about inaccuracy. I say I would be happy to amend what I wrote to include whatever Dr. Consorti wants to say about it. The assistant asks if I have any “further questions” for her. No, I say. The conversation ends.

11. A little later, I realize I do have a question. In 2008, during the conversation when I asked Dr. Consorti for evidence, I had said surgery is dangerous. In response, she had said no one had died during any of her surgeries. By 2010, I realized that such an answer was seriously incomplete. Many bad things can happen during surgery. Death is only one bad outcome. How likely were other bad outcomes? Dr. Consorti hadn’t said. She knew about these other bad effects much better than I did, yet, in a discussion of the safety of surgery, she hadn’t mentioned them. By not mentioning them, she made surgery sound safer than it actually is. Why had she not mentioned them? That’s my question. I call Dr. Consorti’s office and reach the person who had called me. I ask my question. As I wrote ,

He tried to answer it. I said I wanted to know Dr. Consorti’s answer. Wait a moment, he said. He came back to the phone. He had spoken to “the doctor”, he said. She wasn’t interested in “further dialogue”. She would contact a lawyer, he told me.

I haven’t heard from her since then.

This story illustrates a big change. As recently as twenty years ago, the doctor-patient balance of power was heavily weighted toward the doctor, in the sense that the doctor exerted considerable influence on the patient (e.g., to have surgery). One reason, Robin Hanson has emphasized, is human nature: The more fearful we are, the more we trust. Patients are often fearful. Another reason for the power imbalance was information imbalance. The doctor knew a lot about the problem (had encountered many examples, had read a lot about it). The patient, on the other hand, knew almost nothing and could not easily learn more.

During the last twenty years, of course, this has changed dramatically. Patients can easily learn a great deal about any health problem. Google, PubMed, on-line forums, MedHelp, CureTogether, and so on. The story of Dr. Consorti and me illustrates what a difference the new access to information can make.

Personal science (science done to help yourself) has two sides. One is: collect data. My self-experimentation is an example. To improve my health, I gathered data about myself. It worked. My skin improved, I lost weight, slept better, improved my mood, and so on. The other side is: use data already collected. That’s what I did here. My search for data (including my mom’s search) showed that data already in existence (including the absence of evidence supporting surgery) contradicted Dr. Consorti’s recommendation. My search was not biassed against her recommendation. I didn’t care whether she was right or wrong. I just wanted what was best for me. As Feynman said, science is the opposite of trusting experts — including doctors. My first glimpse of the power of self-experimentation was when it showed me that one of the two medicines my dermatologist had prescribed didn’t work.

Overtreatment is an enormous problem in America. Overtreated by Shannon Brownlee and Overdiagnosed by H. Gilbert Welch, Lisa Schwartzl and Steve Woloshin are recent books about it. Overtreatment could easily be why Americans pay far more for health care than people in any other country yet die earlier than people in many countries. A large fraction of our health care may do more harm than good. A common view is that the incentives are wrong. As one commenter put it, pay for treatment, you get treatment. The solution, according to this view, is to change the incentives. That’s a good idea but will not happen soon. I believe overtreatment can be reduced now. You can (a) ask for evidence (as I did) and (b) search for evidence (as I did). The difference in lifespan between America and other countries suggests this might add years to your life.

I would like to find out what happens when people ask for evidence and/or search for evidence. Please send me your stories or post them in the comments.

More Two days after I posted this, Dr. Consorti replied to this post and the earlier one with essentially the same comment, which is here.

Testing Treatments: Nine Questions For the Authors

Seth Roberts14 Comments

From this comment (thanks, Elizabeth Molin) I learned of a British book called Testing Treatments (pdf), whose second edition has just come out. Its goal is to make readers more sophisticated consumers of medical research. To help them distinguish “good” science from “bad” science. Ben Goldacre, the Bad Science columnist, fulsomely praises it (“I genuinely, truly, cannot recommend this awesome book highly enough for its clarity, depth, and humanity”). He wrote a foreword. The main text is by Imogen Evans (medical journalist), Hazel Thornton (writer), Iain Chalmers (medical researcher), and Paul Glaziou (medical researcher, editor of Journal of Evidence-Based Medicine).

To me, as I’ve said, medical research is almost entirely bad. Almost all medical researchers accept two remarkable rules: (a) first, let them get sick and (b) no cheap remedies. These rules severely limit what is studied. In terms of useful progress, the price of these limits has been enormous: near total enfeeblement. For many years the Nobel Prize in Medicine has documented the continuing failure of medical researchers all over the world to make significant progress on all major health problems, including depression, heart disease, obesity, cancer, diabetes, stroke, and so on. It is consistent with their level of understanding that some people associated with medicine would write a book about how to do something (good science) the whole field manifestly can’t do. Testing Treatments isn’t just a fat person writing a book about how to lose weight, it’s the author failing to notice he’s fat.

In case the lesson of the Nobel Prizes isn’t clear, here are some questions for the authors:

1. Why no chapter on prevention research? To fail to discuss prevention, which should be at least half of health care, at length is like writing a book using only half the letters of the alphabet. The authors appear unaware they have done so.

2. Why are practically all common medical treatments expensive?

3. Why should some data be ignored (“clear rules are followed, describing where to look for evidence, what evidence can be included”)? The “systematic reviews” that Goldacre praises here (p. 12) may ignore 95% of available data.

4. The book says: “Patients with life-threatening conditions can be desperate to try anything, including untested ‘treatments’. But it is far better for them to consider enrolling in a suitable clinical trial in which a new treatment is being compared with the current best treatment.” Really? Perhaps an ancient treatment (to authors, untested) would be better. Why are there never clinical trials that compare current treatments (e.g., drugs) to ancient treatments? The ancient treatments, unlike the current ones, have passed the test of time. (The authors appear unaware of this test.) Why is the comparison always one relatively new treatment versus another even newer treatment?

5. Why does all the research you discuss center on reducing symptoms rather than discovering underlying causes? Isn’t the latter vastly more helpful than the former?

6. In a discussion of how to treat arthritis (pp. 170-172), why no mention of omega-3? Many people (with good reason, including this) consider omega-3 anti-inflammatory. Isn’t inflammation a major source of disease?

7. Why is there nothing about how to make your immune system work better? Why is this topic absent from the examples? The immune system is mentioned only once (“Bacterial infections, such as pneumonia, which are associated with the children’s weakened immune system, are a common cause of death [in children with AIDS]“).

8. Care to defend what you say about “ghostwriting” (where med school professors are the stated authors of papers they didn’t write)? You say ghostwriting is when “a professional writer writes text that is officially credited to someone else” (p. 124). Officially credited? Please explain. You also say “ghostwritten material appears in academic publications too – and with potentially worrying consequences” (p. 124). Potentially worrying consequences? You’re not sure?

9. Have you ever discovered a useful treatment? No such discoveries are described in “About the Authors” nor does the main text contain examples. If not, why do you think you know how? If you’re just repeating what others have said, why do you think your teachers are capable of useful discovery? The authors dedicate the book to someone “who encouraged us repeatedly to challenge authority.” Did you ever ask your teachers for evidence that evidence-based medicine is an improvement?

The sad irony of Testing Treatments is that it glorifies evidence-based medicine. According to that line of thinking, doctors should ask for evidence of effectiveness. They should not simply prescribe the conventional treatment. In a meta sense, the authors of Testing Treatments have made exactly the mistake that evidence-based medicine was supposed to fix: Failure to look at evidence. They have failed to see abundant evidence (e.g., the Nobel Prizes) that, better or not, evidence-based medicine is little use.

Above all, the authors of Testing Treatments and the architects of evidence-based medicine have failed to ask: How do new ideas begin? How can we encourage them? Healthy science is more than hypothesis testing; it includes hypothesis generation — and therefore includes methods for doing so. What are those methods? By denigrating and ignoring and telling others to ignore what they call “low-quality evidence” (e.g., case studies), the architects of evidence-based medicine have stifled the growth of new ideas. Ordinary doctors cannot do double-blind clinical trials. Yet they can gather data. They can write case reports. They can do n=1 experiments. They can do n=8 experiments (“case series”). There are millions of ordinary doctors, some very smart and creative (e.g., Jack Kruse). They are potentially a great source of new ideas about how to improve health. By denigrating what ordinary doctors can do (the evidence they can collect) — not to mention what the rest of us can do — and by failing to understand innovation, the architects of evidence-based medicine have made a bad situation (the two rules I mentioned earlier) even worse. They have further reduced the ability of the whole field to innovate, to find practical solutions to common problems.

Evidence-based medicine is religion-like in its emphasis on hierarchy (grades of evidence) and rule-following. In the design of religions, these features made sense (to the designers). You want unquestioning obedience (followers must not question leaders) and you want the focus to be on procedure (rules and rituals) rather than concrete results. Like many religions, evidence-based medicine draws lines (on this side “good”, on that side “bad”) where no lines actually exist. Such line-drawing helps religious leaders because it allows their followers to feel superior to someone (to people outside their religion). When it comes to science, however, these features make things worse. Good ideas can come from anybody, high or low in the hierarchy, on either side of any line. And every scientist comes to realize, if they didn’t already know, that you can’t do good science simply by following rules. It is harder than that. You have to pay close attention to what happens and be flexible. Evidence-based medicine is the opposite of flexible. “ There is considerable intellectual tyranny in the name of science,” said Richard Feynman.

Testing Treatments has plenty of stories. Here I agree with the authors — good stories. It’s the rest of the book that shows their misunderstanding. I would replace the book’s many pages of advice and sermonizing with a few simple words: Ask your doctor for the evidence behind their treatment recommendation. He or she may not want to tell you. Insist. Don’t settle for vague banalities (“It’s good to catch these things early”). Don’t worry about being “difficult”. You won’t find this advice anywhere in Testing Treatments. If I wanted to help patients, I would find out what happens when it is followed.

More Two of the authors respond in the comments. And I comment on their response.