Category: scientific method

Which Should You Trust: Scientific Literature or Anecdote?

Seth Roberts16 Comments

In a comment on a BMJ paper critical of alternative medicine (the author submitted a fictional abstract to a conference then criticized the program committee for not rejecting it), a retired chemist named Joe Magrath said:

The scientific literature tells us that acupuncture, cupping and reflexology are all nonsense.

I haven’t looked into it but I’ll take his word for it.

Around the time Magrath said that, James Fallows said this:

During our years in Malaysia in the 1980s, and more recently in China, my wife and I became unlikely converts to a lot of Asian medical practices. I had serious back pain cured by an acupuncturist (who used needles the size of aluminum baseball bats) in Kuala Lumpur. In her book, my wife describes how the gruesome-seeming therapy of fire-cupping, applied in an all-night massage parlor in the city of Yueyang, snapped her out of a serious bout of the flu. Sure, she had big red welts on her back for the next ten days, but her fever was gone!

Which do you believe?

The Decline Effect and Kitty Kelley

Seth Roberts5 Comments

A few posts ago I commented on Jonah Lehrer’s article about replication difficulties, which Lehrer called the decline effect. I concluded it was an indication how poorly science (truth-seeking) and profession (making a living) fit together. Scientists are always under pressure to do what’s good for their career rather than find and report the truth.

Journalism is another kind of truth-seeking. It has the same problem. Journalists are always under pressure to do what’s good for their career rather than find and report the truth. In an essay about unauthorized biographies, Kitty Kelley makes this point:

[Michael] Hastings said that reporters like [Lara] Logan do not report negative stories about their subjects in order to assure continued access. No reporter would admit to tilting a story toward favorable coverage to keep entrée, but they do, and that is one of the dirty little secrets of journalism today.

Just as no reporter admits this, I have never heard a scientist admit it, with two exceptions: 1. The inventor of the aquatic ape theory of human evolution (Alister Hardy) said he stopped talking about it to avoid hurting his career. It fell to a non-scientist (Elaine Morgan) to develop it. 2. In that famous graduation speech, Richard Feynman pointed out how the first determination of the charge on an electron used the wrong value for the viscosity of air and later determinations, which did not involve that viscosity, tended to confirm the mistaken value. Unfortunately, Feynman went on to say: “We’ve learned those tricks nowadays, and now we don’t have that kind of a disease.” As if human nature had changed.

I conclude that both science and journalism will work best with systems where amateurs and professionals both have substantial power. Kelley doesn’t mention that authorized biographers have important truth-seeking advantages over non-authorized ones (e.g., access to old letters).

The Decline Effect

Seth Roberts14 Comments

A new article in The New Yorker by Jonah Lehrer is about declines in the size of experimental or quasi-experimental effects over time. For example, Jonathan Schooler, an experimental psychologist, found that if subjects are asked to describe a face soon after seeing it their later memory for the face is worse. As Schooler continued to study the effect, it appeared to get weaker. The article also describes examples from drug trials (a anti-psychotic drug appeared to become weaker over 15 years) and ecology (the effect of male symmetry on mating success got weaker over years).

It’s nice to see an ambitious unconventional article. I blogged a few weeks ago about difficulties replicating the too-many-choices effect. Difficulty of replication and the decline effect are the same thing. I could do what Jared Diamond does in Collapse: give a list of five or six reasons why this happens. (Judging by this paper, the effect, although real, is much weaker than you’d guess from Lehrer’s article.) For example, the initial report has much more flexibility of data analysis than later reports. Flexibility of analysis allows researchers to increase the size of effects.

A long list of reasons would miss a larger point (as Diamond does). A larger point is this: Science (search for truth) and profession (making a living) are not a good fit. In a dozen ways, the demands of a scientist’s job get in the way of finding and reporting truth. You need to publish, get a grant, please your colleagues, and so on. Nobody pays you for finding the truth. If that is a goal, it is several goals from the top of the list. Most jobs have customers. If a wheelwright made a bad wheel, it broke. Perhaps he had to replace it or got a bad reputation. There was fast powerful feedback. In science, feedback is long-delayed or absent. Only long after you have been promoted may it become clear anything was wrong with the papers behind your promotion. The main customers for science are other scientists. The pressure to have low standards — and thus appear better to promotion committees and non-scientists — is irresistible. Whereas if Wheelwright Y makes better wheels than Wheelwright X, customers may notice and Wheelwright Y may benefit.

There are things about making science a job that push scientists toward the truth as well, such as more money and time. When science is a job, a lot more research gets done. Fine. But how strong are the forces against finding truth? I was never surprised by the replication difficulties Lehrer writes about. I had heard plenty of examples, knew there were many reasons it happened. But I was stunned by the results of my self-experimentation. I kept finding stuff (e.g., breakfast disturbs sleep, butter improves brain function) that contradicted the official line (breakfast is the most important meal of the day, butter is dangerous). Obviously I had a better tool (self-experimentation) for finding things out. The shock was how many things that had supposedly been found out were wrong. Slowly I realized how much pressure career demands place on scientists. It is no coincidence that the person most responsible for debunking man-made global warming, Stephen McIntyre, is not a professional climatologist (or a professional scientist in any other area). Unlike them, he can say whatever he wants.

Thanks to Peter Couvares.

More In his blog, failing to see the forest for the trees, Lehrer says we must still believe in climate change (presumably man-made): climate change and evolution by natural selection “have been verified in thousands of different ways by thousands of different scientists working in many different fields.” Charles Darwin, like McIntyre, was an amateur, and therefore could say whatever he wanted.

Feeling the Future: Room For Improvement

Seth Roberts3 Comments

My Frontiers of Psychology class read Daryl Bem’s new paper Feeling the Future that reports nine experiments that show an effect of the future on the present. I have a different take than anything I’ve read: I think there are several good reasons to take it seriously. But in this post let’s start with how it could have been better:

1. Lack of background. There have been lots of experiments along these lines. What did they show? This question is not clearly answered. The prior probability of these claims is enormously important. As I told my students, if seeing the future was common and easy for even a small fraction of people, we wouldn’t have businesses, such as casinos, making money on gambling. But the existence of such businesses doesn’t rule out weak effects.

2. Lack of exact repetition. An obvious criticism is that Bem slanted the data analysis to favor the results he wanted. In any data analysis of unfamiliar data, you must choose — how to transform the data, what test to use, and so on. You must also choose how many subjects to run and how many trials to give them. There are rules for these choices (Bem doesn’t seem to know how to choose a transformation) but nevertheless they allow favoritism to creep in. Drug trials have big problems along these lines — severe slanting of the analysis to make the results more favorable — which is why when you register a clinical trial you must specify the endpoints. The answer to the criticism that your data-analysis choices made your favored result more likely is to do a data analysis with no choices at all. This cannot be done from scratch. You need to do the experiment once, make all the necessary choices, and then do the same experiment again (same everything as much as possible) and analyze the data exactly the way you analyzed the data from the first experiment. Bem never does this. Instead each experiment is different from all the rest. This is what experimental psychologists traditionally do but here it is a bad idea. Better to have taken the two simplest and clearest effects (priming and word learning) and repeated them several times exactly.

3. Were experiments left out? Let’s say you observe a weakly-significant result, p = 0.03. Now you do the same experiment eight more times. How likely is it that each of the eight replications will also find a significant difference? Quite low. Yet Bem finds a weakly significant difference in each of his nine experiments. This is highly unlikely. Bem appears unaware of the problem. Mendel had the same problem (data too good to be true). Ultimately Mendel was proved right. But again it stresses that Bem should do exact repetitions and report the results no matter what if he wants to be more persuasive.

Epilepsy’s Big, Fat Miracle …

Seth Roberts3 Comments

… is the title of a New York Times Magazine article about the ketogenic diet, a treatment for childhood epilepsy, which I’ve blogged about several times (here, here, here, here, here). It’s a very-high-fat diet. It interests me for two reasons: (a) It connects a high-fat diet with proper brain function, as my self-experiments have done. A curious feature of the ketogenic diet is that it isn’t permanent. After several years the child can go off it. My self-experimentation suggests that Americans eat far too little of certain fats. Perhaps eating enough of these fats would prevent childhood epilepsy. (b) It shows how someone who cares enough — in this case, Jim Abrahams, whose son had epilepsy — can be more effective than professional researchers and doctors. Abrahams rediscovered the diet. He saw its value, the professionals didn’t. I’ve argued that this is part of why my self-experimentation found new solutions to common problems: because I had those problems. I cared more about finding a workable solution than researchers in those areas, who had several other concerns (publication, funding, acceptance, etc.).

The details of the article reminded me of something I learned in the BBC series The Story of Science. For hundreds of years, medical students were told, following Aristotle, that the liver has three lobes. It doesn’t. You might think that examination of thousands of actual livers would have dispelled the wrong idea, but it didn’t. The article contains many examples of doctors ignoring perfectly good evidence in favor of nonsense they read in a book or heard in a lecture. Epilepsy is easy to measure. If a child has 100 seizures per day, and has been having them at this rate for years, and this goes down to 5 shortly after he starts the ketogenic diet, and goes up again when the child goes off the diet, there is no doubt the diet works. As early as the 1930s, this had been observed hundreds of times. This was overwhelming evidence of effectiveness. Doctors ignored it, probably based on the modern equivalent of the three-lobed liver. They complained, according to the article, that there was “no evidence it worked” or that the evidence wasn’t “controlled” or “scientific” (whatever that means). A study published in 2008 “answered doubts about keto’s clinical effectiveness” — as if doctors needed the equivalent of a very-large-type book to be able to read what most of us can read with normal-sized type.

According to the article, “by 2000, more people were asking about keto, but most pediatric neurologists still would not prescribe it” — as if the parents needed the approval of their doctor to try it. You don’t need a prescription to buy food.

Thanks to Tim Beneke, Michael Bowerman, Alex Chernavsky, David Cramer, and Peter Couvares.

How to Choose A Research Topic

Seth Roberts3 Comments

A few weeks ago, a female biology professor from Berkeley gave a talk at Tsinghua as part of a women-in-science series. During the question period, a student asked how to choose a research topic. You have a choice of labs; which should you choose? You have a choice of research questions; which should you choose? An excellent question: Every young scientist wonders about this.

The speaker’s answer: Believe in yourself. Huh? This came from her personal history. When she was a grad student (at Berkeley) she proposed a certain line of research to her advisor. Her advisor said it was a bad idea. She switched to Harvard and pursued her idea there. It paid off. A sign of her success is that her lab gets $1 million/year in grants.

I wasn’t there. The friend who told me the professor’s unhelpful answer asked how I would answer the same question. During graduate school, I thought a lot about it — about how to do research that anyone will care about in fifty years. I can answer it only for experimental psychology.

First，invent a new method or study a large puzzling experimental effect. With either one you can generate a steady steam of publications. Inventing a new method mean inventing a better way — usually, a faster way — of measuring something important. You can then apply your new method all over the place. With a large experimental effect you can vary all sorts of things and narrow in on an explanation. As a grad student, I took the first route: I used a new way of studying animal time discrimination. I didn’t invent it but its inventor hadn’t seen its value. An example of the second route is the career of John Garcia. In graduate school, he discovered that making rats sick after eating a new flavor caused them to dislike the flavor. The sickness could come hours after the flavor. Garcia made a whole career out of doing variations on this.

Second, take advantage of whatever is unusual about you. If you are unusually interested in X, study X. I differed in two ways from most experimental psychologists: I was better at math, and I cared more about writing. Taking advantage of this, I spent a lot of time on data analysis and writing. Both paid off. I suppose my paper were better written than necessary but the time spent on writing paid off because I got good ideas while writing.

Third, collect a rich data set. New experimental effects are enormously important — if you manage to find one you can spend the rest of your career studying it — but are also very difficult to find. You can’t do experiments whose main purpose is to look for them. The chances of success are too low. To find them, you set up your research so that a conventional experiment has the possibility of finding them. For that you need a rich data set — a data set with many factors and many levels of each factor, ideally. The new way of studying timing that I used provided a rich data set. Quite soon this led to discovering a new effect when some of the data changed in a surprising way.

Dangers of Supplements

Seth Roberts2 Comments

Via Robin Hanson I found this study of the effects of antioxidant supplements. It studied five (e.g., Vitamins A and C). Overall they were slightly harmful, except selenium.

This isn’t intuitive — why should they differ? — but fits well with previous work:

1. Evidence for benefits of selenium is overwhelming. You can look at a county-by-county map of US cancer rates and see a sharp drop along a certain line in the northeast. The line separates different geology. There is much more selenium in the soil on the low-cancer side of the line. Yet another case where correlation is powerful evidence for causation. An experiment with selenium supplementation found a reduction in cancer.

2. Several years ago, two experiments found Vitamin A supplements increased lung cancer. (Another study.) Later experiments cast doubt on Vitamins C and E. As one of Robin’s readers put it: “two of which were previously well known to be bad for you.”

Given this previous research, which is far more persuasive than the current study, the interesting contribution of the new study is methodological: will a meta-analysis of epidemiological studies reach the right conclusions? Will the signal outweigh the many sources of bias and error? In fact, it did. Again suggesting that severe critics of epidemiology, such as John Ioannidis, go too far.

Placebo Non-Disclosure

Seth RobertsLeave a comment

A new study in the Annals of Internal Medicine asked how often placebo-controlled medical studies made clear what the placebo was. The abstract says:

Most studies did not disclose the composition of the study placebo.

Which may give the wrong impression. About 90% of the studies they looked at did not say what the placebo was.

The paper illustrates the problem with an example:

In one of these studies [where the placebo was not described], the authors commented that “The lack of any overall effect in patients with myocardial infarction might be related to the unexpectedly low mortality rate in the placebo group.” The possibility that the placebo composition may have influenced this “unexpectedly low mortality” was apparently not considered.

Thanks to Gunnar Schröder.

The C.I.A. and Self-Experimentation

Seth Roberts1 Comment

I learned of The Human Factor: Inside the CIA’s Dysfunctional Intelligence Culture by Ishmael Jones (a pseudonym) from an interview on the New Yorker website. This comment by the author interested me:

Once the C.I.A. became a place to get rich, effective operations ended. Today, more than ninety per cent of C.I.A. employees live and work entirely within the United States, in violation of the C.I.A.’s founding charter [to supply only foreign intelligence].

I could say much the same about science: Once it became a place to get rich (or at least get large grants), effective science became a lot less common. A great deal of science is done by drug companies. They pay a lot. Some of their scientists are surely brilliant but their talents are wasted by the need to find solutions that will be highly profitable. My self-experimentation found solutions that cost nothing and make far more intellectual sense. I was able to do something that didn’t produce a lot of publications because it wasn’t my job.

Many skills make good full-time jobs. Science doesn’t. There is too much pressure for short-term results. Without short-term results, you may lose your job or your grant. (Or, in China, most of your income.) Nor is science a good source of status. If you want your science to provide your status, you will be under great pressure to conform. Yet for practically all scientists, it’s their full-time job and their main source of status. This may not make it impossible for them to do good work but I suspect it comes close to doing so. My self-experimentation was effective not only because it was fast and cheap (per experiment) but also because I could be slow (per publication) and do something low-status.

A Chinese Physicist Resigns

Seth Roberts2 Comments

A Chinese physicist recently resigned from his job (pure research) at a Beijing research institute. His salary was too low. The base salary is something like $200/month, with something like $1200 for each paper you publish. He explained his decision in a letter to his bosses, which he posted on the Internet. From Google Translate:

Dear leaders:

Hello!

August 2006, I single-handedly carried the mat, one hand holding the quilt to the school to report to work. Slept on the floor in the office 3 nights later, Frank and others XX XXX Street, shares a house, 800 yuan per month. Themselves feel better. However, when my wife came to see me when to Shanghai, but a cry. She did not expect this to write beautiful prose, in English is superb, the monthly salary of ten years ago, men who have three thousand dollars so come down: the room to work without a decent table, there is no place to sit, could only sit bed; office also can take place without her. Yes, until now, my office is a chair, a common HP laser printer or the wife gave me a birthday present. His wife’s insistence, in March 2007, after six months sharing with others, I moved to Village X XX X, X round room (Reference: College on XXXX XXX), monthly rent of 1,600 yuan.

I come to school only task is to do research. Not in class, not with the students. I do specialty is theoretical physics. In my opinion, the current basic problems of theoretical physics can be divided into three areas, but also three levels: the top and most shallow, which is the quality of particle physics, neutrino problem, particle state mixing problem ; Secondly, the harder is self-consistent description of the gravitational field and unify quantum field; final, most difficult, is the true cosmology, the concept of the universe all things are included, the true cosmological should a TOE (Theory of Everything). Cosmology is now called, should be called observational cosmology, from a theoretical point of view, can only be called up to fit the observational cosmology. Three aspects of the course, or interrelated, each is at stake.

Second half of 2006, I understand the unified description of the main energy to the gravitational field and quantum field theory and mathematical tools needed which, in an arbitrary manifold reflects the quantum characteristics of the mathematical tools may be harmonic analysis. But I do not master any of harmonic analysis on manifolds.

Time to flash the first half of 2007, the annual meeting of gravity, I consider the possible use of Finsler geometry to try to do quantum gravity, began to self-Finsler geometry, the same year by the end of September, I made use of Finsler geometry re-expressed in general relativity. However, re-expressed with the Finsler geometry of general relativity does not make me to the direction of quantum gravity there is any progress, Finsler geometry is very strict rigid mathematical structure, the introduction of any features of the quantum structure itself will be incompatible with the geometry. Of course, you can put Finsler space-time structure of dark energy as an inevitable result, however, computability theory, the ability to compare theory and observations are not good.

July 2007, I bought a house in Shanghai, the total price of 96 million, 30-year loans to 66 million. After that, our economic pressure to the limit: my wife rented a house in Beijing, XXX XX Garden Park Unit X X X Building, Room (Reference: Beijing XXX Company XX), monthly rent of 2,500 yuan; my house in Shanghai, monthly rent of 1,600 yuan; also close to 6,000 yuan per month mortgage. All of my salary: two accounts, one CCB, more than 2,000 points (up once already, I am now in January 2669.27 yuan (March 2010)); another investment line, 1,800 yuan per month. Do not say that I run in Beijing and Shanghai both transport costs, my salary to feed my already running out. So we got married in 2005 had never to children.

Under the pressure of the huge costs climb in 2007, first half of 2008, I intend to do some simple problems, the observation of neutrino cosmology application, or vice versa, according to some astronomical observations to discuss the micro- Mass of sub-limits. However, read the literature, in-depth period of time to do that: this work is not a person with a PC machine can be done, at least a group of some value to make it possible to do the work. So, do I still have to go back I am a person with a PC, will be able to do mathematical physics. At this time, there have been two things: First, my wife is pregnant, should be joyous, married 4 years, she has XX years of age. Second, in June 2008, my body feels discomfort, severe cough began after the end of August, the end of September to do CT, in the chest, heart on top, between the lungs lymphatic tumor, a copy of the specific report, see CT.

08 expiration of the contract work my wife returned to Shanghai from Beijing. August 30, we arrived in Shanghai, after living for a month in a hotel (of course, is my wife’s company out of accommodation), I X, X, Room XXX Village renters (references, XXX), 3,300 yuan monthly rent. Our own new home in the June 30 submitted to, because my physical problems, and his wife was pregnant, there is no way to decoration. Only in November 2008 leased to our new home workers rough (references XXXX), monthly rent of 1,800 yuan.

My CT report came out, there are three doctors that the disease: sarcoidosis best results, followed by lymphoma or lung cancer. Six City Hospital, Zhongshan Hospital and the City Chest Hospital’s doctors agree that: To do mediastinal thoracic endoscopy to confirm the diagnosis. However, my child is born in November 2, and before that in October, I can not do chest examination. October 21, 2008, in my guarantee, Chest Hospital, XXXX decided in accordance with sarcoidosis, I began to hormone (prednisone) therapy. In fact, I know, if it is worse than the result of sarcoidosis, I have only the choice of suicide: I can not let my white mother in the cold begging to others: “Save my son… “; or my wife holding an infant child:” save my husband… ”

Thank bliss, I am still alive and well. But, my friends were not so lucky: Mao XXXX, 2005 in High Energy suicide; von XXX, one of my best friends, in January 2007 committed suicide at Tokyo University. Their death, and they chose Theoretical Physics, chose to do research are closely related. Von XXX, 28 years old when he died less, 8 years old his father died, take pains to support his mother and younger siblings. In the Babaoshan, watching Von XXX’s mother, that Ganchangcunduan, piercing woman, I was deeply sad but misery. To von XXX, I think I should continue to do theoretical physics.

2009, I went back to quality issues, problems and mixed particle state mixing matrix. An elegant theory that all the observed phenomena should be explained with the natural, but the existing weak unified theory, have failed. After several months of effort, I think, rest mass operator should be introduced to the rest mass of the quantum field operator as is the rest mass of the intrinsic value, so that uniform interpretation of quantum states can be mixed and mixed matrix. Done in July, found the problem difficult to overcome, and because of physical reasons, was closed. After the October, and XXX, XXX discussions and continue to do for some time. I found a little forward, but there are fundamental problems from a distance, need to be more profound thoughts.

In 2009, my body slowly getting better. Done 4 CT, 4 after March, the condition improved slowly. And then to Shanghai Pulmonary Hospital treatment is the same method. September 2, to Nantong XXXX hospital medicine treatment, medication in January, costs over 3,500 yuan (can not be reimbursed, easy to find the invoice.) To October 2009, my medical insurance card, no money already. At this time, CT showed that half of the tumor has disappeared, the results see the copy. Since then I have to give up treatment, on the one hand do not want to spend more money on the one hand I believe I can beat the rest of the half.

2008 year-end assessment, I did not write. Do write to me to do research, but failed; or I was sick; or I just add a small baby at home? Write my difficulties, I do not want to let others sympathetic to buckle my money? For a limb still, physical and mental health of man, which is a shame.

Thus, starting from March 2009, my investment line of Cary, 900 yuan less per month, only 900 yuan.

Finally, I deeply know that I am the circle around me, most people are not doing scientific research. Doing scientific research is false, is an issue, published an article to change some money, to live. (Above paragraph, is not directed against any person or thing, I seriously do research and those who pay tribute to the depths of his heart. Even if someone is writing papers for money, as long as there is no plagiarism, is beyond reproach, and now the rule is like that. I can not do, but my poor ability to adapt.) von XXX, XXXX Mao may also, like me, are the idealists, does not fit in this circle there to do, so they died.

Yes, I suddenly realized: I do not fit inside the circle to do the research now. Otherwise, I will and Feng XX, XXX Mao same fate. In the November 25, 2009 made a report terms of rest mass operator, I decided to leave. November 30, in the center of the meeting, I proposed to my research was no longer suitable.

All see, I know, the year-end 2009, has not filled out the necessary assessment: whether what I was made or not made anything, I knew.

So, I Merchants Bank of Cary, from the beginning in January 2010, a month is 0. I can start another kind of living law.

Determined from the high school to study physics to me, from the self to the Thermodynamics of Newtonian mechanics, to quantum mechanics; from the self-study calculus to linear algebra; learn from differential geometry, group theory, and then learn topology. Into his own youth, twenty years into his effort, and now had to give up, is a frustration.

Dreamed last night: spring, towards the sea, I lingered under the weeping willow, think about the physics, think of the beauty of the universe!

I know, only to leave, I can live. When people have forgotten that the earth is the soul of the physical sacrifice, I still worried about soul.

Must live with dignity, I can only leave. In the vast universe, the short life of individuals, such as fireworks, energy and all the leaders met, life is fate. Besides, I also get more help and the central leadership to take care, keep in mind when I hope to have the opportunity to return.