And how do you learn what matters?

When I was a grad student, I read Stanislav Ulam’s memoir Adventures of a Mathematician. I was impressed by something Ulam said about John von Neumann: that he grasped the difference between the trunk of the tree of mathematics and the branches. Between core issues and lesser ones. Between what matters more and what matters less. I wanted to make similar distinctions within psychology. Nobody talked about this, however. Not even other books.

Some research will be influential, will be built upon. Some won’t. To put it bluntly, some research will matter, some won’t. I once thought of teaching a graduate course where students learn to predict how many citations an article will receive. You take a 10-year-old journal issue, for example, and try to predict how many citations each article will receive. I like to think it would have been a helpful class: The key to a successful scientific career is writing articles that are often cited. I even had a title: “What Will You Do After You Stop Imitating Your Advisor?”

When I was a grad student the short answer to “what matters?” in experimental psychology was clear enough:

1. New methods. The Skinner box, for example, was a new way to study instrumental learning. Skinner didn’t discover or create the first laboratory demonstration of instrumental learning; he simply made it easier to study.

2. New effects. New cause-and-effect linkages. For example, John Garcia discovered that if you make a rat sick after it experiences a new flavor it will avoid foods with that flavor.

My doctoral dissertation was about a new way to study animal timing.

A few months ago I had coffee with Glen Weyl, a graduate student in economics at Princeton. We discussed his doctoral research, which is about how to test theories. One of Glen’s advisors had told him about a paper by Hal Pashler and me on the subject. Hal and I argued that fitting a model to data is a poor way to test the model because there is no allowance for the model’s flexibility. The first reviewers of our paper didn’t like it. “You don’t realize how hard it is to find a model that fits,” one of them wrote.

Glen’s interest in this question began during a seminar in Italy, when he realized the speaker was more or less ignoring the problem. The speaker was comparing how well two different theories could explain the same data without taking into account their different amounts of flexibility. Glen’s thesis proposes a Bayesian framework that allows you to do this. His main example uses data of Charness and Rabin from choice experiments. (Matt Rabin is a MacArthur Fellow.) Taking flexibility into account, he reaches a different conclusion than they did.

I wondered how Glen decided this was important. (It’s a method, yes, but a highly abstract one.) I asked him. He replied:

Sadly, despite my interest in the history of economic thought, I don’t have a lot of insight about why I came upon these thoughts. But one thing: my interests are very interdisciplinary . . . My work is based on drawing connections between economics, philosophy of science, and computer science (and meta-analysis from psychology and bio-statistics). Most of my work takes this form: as you’ll see on my website, I’ve used theoretical insights from economics and computer science as well as evidence from neuroscience, psychology and biology to critique the individualist foundations of liberal rights theory; I’ve used ideas from decision theory to lay firmer foundations for goals set out by computer scientists designing algorithms; I’ve used tools from information theory to instantiate insights from psychology to help understand the design of auctions; and I’ve used computational neuroscience to model biases in economic information processing. Broad interests are hard to have, because they limit the time for learning a particular area in depth, but I prefer to read broadly and draw connections rather than to read deeply and chip away at open questions.

That was interesting. I read broadly, and so does Hal, who knows more about the philosophy of science than I do. I wrote to Glen:

The usual comment about interdisciplinary knowledge is that it’s good because you can bring ideas from one area, including solutions and methods, to solve problems in another area. . . . But maybe it’s also good because by learning about different areas you absorb a range of different value systems and this makes you less sensitive to fads (which vary from field to field), more sensitive to longer-lasting and more broadly-held values.

The more trees you know, the easier it is to see the forest.

Evaluating new product ideas.

I asked a friend of mine why she was a good boss. “I was nurturing,” she said. A big study of managers reached essentially the same conclusion: Good managers don’t try to make employees fit a pre-established box, the manager’s preconception about how to do the job. A good manager tries to encourage, to bring out, whatever strengths the employee already has. This wasn’t a philosophy or value judgment, it was what the data showed. The “good” managers were defined as the more productive ones — something like that. (My post about this.)

The reason for the study, as Veblen might say, was the need for it. Most managers failed to act this way. I posted a few days ago about a similar tendency among scientists: When faced with new data, a tendency to focus on what’s wrong with it and ignore what’s right about it. To pay far more attention to limitations than strengths. Here are two examples:

1. Everyone’s heard “correlation does not imply causation”. I’ve never heard a parallel saying about what correlation does imply. It would be along the lines of “something is better than nothing.”

2. Recently I attended a research group meeting in which a postdoc talked about new data she had gathered. The entire discussion was about the problems with it — what she couldn’t infer from it. There could have been a long discussion about how it added to what we already know, but there wasn’t a word about this.

Some of the comments considered this behavior a kind of Bayesian resistance to change in beliefs. But it occurs regardless of whether the new data support or contradict prior beliefs. There’s nothing about prior beliefs in “correlation does not imply causation.” The post-doc wasn’t presenting data that contradicted what anyone knew. Also, similar behavior occurs in other areas besides science (e.g., how managers manage) in which the Bayesian explanation doesn’t fit so well.

I think it’s really strong. I was guilty of it myself when discussing it! I made very clear how this tendency is a problem, giving the analogy of a car that could turn left but not right. Obviously bad. I said nothing about the opportunities this tendency gives everyone. My self-experimentation is an example. The more that others reject useful data, the more likely it is that useful data is lying around and doesn’t require much effort to find. I have called this behavior dismissive; I could have called it generous. It’s like leaving money lying on the ground.

A related discussion at Overcoming Bias. What should “correlation does not imply causation” be replaced with?

Addendum. Barry Goldwater weighs in: “I’m frankly sick and tired of the political preachers across this country telling me as a citizen that if I want to be a moral person, I must believe in ‘A,’ ‘B,’ ‘C,’ and ‘D.’” Indeed, preachers spend far more time on what we are doing wrong (and should do less of) than on what we are doing right (and should do more of). The preacher Joel Osteen has taken great advantage of this tendency. “I think most people already know what they are doing wrong,” he told 60 Minutes.