Conway’s Law and Science

Conway’s Law is the observation that the structure of a product will reflect the structure of the organization that designed it. If the organization has three parts, so will the product. In the original paper (1968), Conway put it like this:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Here is an example:

A contract research organization had eight people who were to produce a COBOL and an ALGOL compiler. After some initial estimates of difficulty and time, five people were assigned to the COBOL job and three to the ALGOL job. The resulting COBOL compiler ran in five phases, the ALG0L compiler ran in three.

A consumer — someone outside the organization who uses the product — wants the best design. Conway’s Law implies they are unlikely to get it.

I generalize Conway’s Law like this: It is hard for people with jobs to innovate — for reasons that outsiders know nothing about. Whereas persons without jobs have total freedom. An example is a politician who promises change but fails to deliver. The promises of change are plausible to outsiders (voters) so they elect the politician. However, being outsiders, they barely understand how government works. When the promised changes don’t happen, the voters are “disillusioned”.

To me, the most interesting application of the generalized law is to science. In my experience, people who complain about “bad science”, such as John Ioannides and Ben Goldacre, have the same incomplete view of the world as the “disillusioned” voters. They fail to grasp the constraints involved. They fail to consider that the science they are criticizing may be the best those professional scientists can produce, given the system within which they work. Better critiques would look at the constraints the professional scientists are under, the reasons for those constraints, and how those constraints might be overcome.

“Much research is conducted for reasons other than the pursuit of truth,” writes Ioannidis. Well, yes — people with jobs want to keep them and get promoted. They want to appear high status. That’s not going to change. It’s absolutely true that drug company scientists slant the evidence to favor their company’s drug, as Irving Kirsch explains in The Emperor’s New Drugs. But if you don’t understand what causes depression and you’re trying to produce a new anti-depressant and you want to keep your job . . . things get difficult. The core problem is lack of understanding. Lack of understanding makes innovation difficult. Completely failing to understand this, Ioannidis recommends something that would discourage new ideas: “We must routinely demand robust and extensive external validation—in the form of additional studies—for any report that claims to have found something new.”

Truly “bad science” has little to do with what Ioannides or Goldacre or any quackbuster talks about. Truly bad science is derivative science, science that fails to find new answers to major questions, such as the cause of obesity. Failure of innovation isn’t shown by any one study. Given the rarity of innovation, it is unwise to expect much of any one study. To see lack of innovation clearly you need to look at the whole distribution of innovation. Whether the system is working well or poorly, I think the distribution of innovation resembles a power law: most studies produce little progress, a tiny number produce large progress. The slope of the distribution is what matters. Bad science = steep downward slope. With bad science, even the most fruitful studies produce only small amounts of innovation.

Just as outsiders expect too much from professionals, they fail to grasp the innovative power of non-professionals. Mendel was not a professional scientist. Darwin was not a professional scientist. Einstein did his best work while a patent clerk. John Snow, the first person to use data (a graph) to learn the cause of an infection, was a doctor. His job had nothing to do with preventing infection. To improve innovation about health (or anything else), we should give more power to non-professionals, as I argued in my talk at the First Quantified Self Conference.

Thanks to Robin Barooah.