Andrew Gelman astutely noted that the three researchers (Michel Cabanac, Anthony Sclafani, and Israel Ramirez) whose work I used the most to come up with the Shangri-La Diet were not at Harvard or Yale or Rockefeller University. Isn’t that where breakthrough research is supposed to come from? This wasn’t the only way that development of the Shangri-La Diet was not quite “right”:
1. The research of Cabanac et al. got little recognition. The set point idea arose in the 1950s, or even earlier. In the 1970s, Cabanac saw very clearly that your set point depends on what you eat. With Rabe, he did an excellent experiment supporting this view. Not one weight-control researcher took note. No other lab built on this work.
2. I was not a weight-control researcher. In graduate school, I studied animal learning. Weight control is not just a different field of psychology; it is usually studied in a different department (nutrition or physiology).
3. The research I did was not funded. Given my lack of credentials and previous experience, it is not obvious it could ever be funded.
The hard-core defender of the system would say: SLD is rubbish. Just another fad diet. The open-minded defender would say: What do you want? You were a tenured professor at Berkeley. You published your work in Behavioral and Brain Sciences. It is well-recognized that really new ideas often take a while to be appreciated. The open-minded critic would say: All three points are correct. In addition, why isn’t Israel Ramirez still a scientist? After all that brilliant research. What a loss. The hard-core critic would say: SLD reached the public as a self-help book. Why give the system any credit? Lots of non-scientists have published influential self-help books.
The hard-core views ignore reality. The SLD forums make it clear SLD is not rubbish. The notion that the system should get no credit at all ignores the fact that I made a living as a scientist at a well-respected place and published my work in a well-respected journal. The open-minded views, however, are both reasonable.
Yes, I agree. That’s part of what I meant when I said the open-minded defender of the system made reasonable points.
As a student it was interesting that most of the professors at Stanford didn’t come from top research institutions. (Math and physics seemed to be exceptions.) At first I thought maybe being at the top 10 made you soft, but eventually I decided that education quality at the top 400 schools is very similar. Research universities are more like awards or exploiting a promising line of research than where ideas originate. The guy who figured out ulcers are from bacteria could get a good position with a Nobel in his pocket, but since it isn’t a whole new area of research that people could start working on he probably wasn’t immediately that interesting.
One part of your problem is follow up. How do others build on your work or reference it in their own papers? In a summer research job (pretend grad school) the professor I worked with said something like “these folks have some interesting results, try it out and see what happens when we do it this way.” And a more independent grad student needs to think this is an interesting approach, I can add my approach to SLD and the grants will flow freely.
Self experimenting might be too easy. Grad students have these tools they want to use and it seems like anyone could start record keeping and drawing inferences. Instinctively they are (I was) drawn to work that can only be done after mastering the complex information from all the classes and books and studying.
It is also too hard. Key points in the process involved insight and some serendipity. “Come up with an insight” is a very hard mission for people and they would prefer less risky work.
For the acceptance problem it’s hard to distinguish exactly what it is that separates your findings from the enormous number of cures in the health food store (which I’ve been encountering as I search for flax oil). Every one of the 2000 ingredients seems to cure many things for everyone. I credit what you say because of the old Victorian science idea of a “reliable observer.” You are careful, aware of placebo and confirmation bias effects and tend to find things which are “surprising” to you and discard much else. But that’s a little elitist of me. (Health store guy: Oh for sleeping and mental function, try this saw palmetto too, it totally opened my mind. Me: I’ll just do this for now.) These seems to be where larger studies should come in, but as you point out we’ve way over engineered double blind studies.
Back to the system: it is great at funding big promising projects like human genome, a fab for new kinds of chips, or the effects of global warming. And for having lots of play in the system for personally interesting work. Not very good at recognizing true ideas quickly or nimbly trying out the avenues with large possible impact but small staffs.
Silicon valley is very good with the last two. At various startups we’ve just “tried out stuff on people” all the time, I wonder if a human research committee would have approved facebook or youtube. “Hmm intense, sometimes hurtful social interaction. Privacy invasions. And what is the control?” Ebay and paypal both depended on existing where the law was just ambiguous enough to not apply to them. It would be nice if scientific standing or true finding points could be converted to money to pay off VC’s and angel investors. They might then invest in the young Israel Ramirez in a way that the current system apparently didn’t. Even so, modern professional research universities (the system since 1950) have produced remarkable work and such different work than what the rest of society produces that it’s frightening to think about messing it up.
That giant step in difficulty when you want to validate an idea. Ideally it would look as close as possible to a smooth slop. Right now it’s get idea, test idea, ask friend colleagues, ask colleagues, [small leap] some publication,[big leap] get $2 million for giant double blind study for a binary result which might be wrong, major publication, and finally acceptance.
How about a giant ongoing longitudinal study where researchers could ask the participants to try things out? So instead of needing to start from scratch with each study you do the equivalent of posting to the SLD forums. Some giant study is intensively monitoring everything they can about 30,000 people. Yearly physicals, monthly phone interviews, diaries, automated sensor monitoring. Someone says they are trying SLD or the Atkins diet and that’s noted and the data can be tracked. There are currently longitudinal studies but they don’t try interventions and they don’t investigate lives very deeply – they have interviews every 5 years or so. You don’t worry about representative, you pick people that live close to the research institution and who are convenient, because you want to see what happens when they do something different. People will try the very most mainstream things and the craziest on the edge ear candle things imaginable and you track it all. Anyone can post ideas to the forum, but researchers can support each other by commenting “I think this might work.”
Then ideas like Atkins, SLD, or weight watchers can be evaluated as they come up. You can say our participants who started Atkins, lost weight and raised cholesterol and 5 years later they were right back where they started. There is an opportunity for surprise and multivariate causality, maybe spinach + peanuts is a miracle food. Some people can get as intense as you Seth about recording everything, some people can get intense with automated monitoring (sensors on the cellphone, or gps, or logging heart rate monitors), and some people can just wait for the monthly interview. Participants could share the data with whoever they liked. So I could show Seth my whole record or just the part he’s interested in.
This is a massive very expensive step, but it might enable lots of gentler slops for lots of other research. And a place for grad students to frolic merrily in meadows of data.
If I were reading me, I would believe my conclusions about flaxseed oil because I had experimental data that anyone could see. The effects were big and repeated, the designs were good. And there was plenty of supporting evidence.
I heard a talk in which the speaker said engineers were good at going from research to device usable by tech person and from device usable by tech person to device usable by expert but poor at the final stage: going from device usable by expert to device usable by ordinary person. Science funding has the opposite problem: it is poor at the first stage, funding research that will generate ideas.