Science is the engine of society, and the decline of truly disruptive research is a warning sign for all of us.
If you measure by the sheer quantity of papers published, we’re in a golden age of science. There are more scientists than ever; there are more publications than ever; and while a lot of great work remains underfunded, there’s far more funding than ever before. Federal funding for research and development has grown from $3.5 billion in 1955 to $137.8 billion in 2020, a more than tenfold increase even once you adjust for inflation.
Fields like AI and biotechnology seem to be booming, but outside of a few specific areas, like AI and biotechnology, it doesn’t really feel like we’re in a golden age of science. The early 20th century saw discovery after discovery that radically changed our comprehension of the world we lived in, and upended industry: nitrogen fixation, which made it possible to feed billions; the structure of the atom and of DNA; rocketry, plate tectonics, radio, computing, antibiotics, general relativity, nuclear chain reactions, quantum mechanics … the list goes on and on.
There might be more science now, but it feels like it adds up to little that compares to the 20th century in terms of discoveries that actually change the world. It feels like we’re doing more research and getting less out of it.
That’s the thesis explored in a new Nature paper, “Papers and patents are becoming less disruptive over time,” which attempts to systematically explore what I said above: more science, but less world-changing science.
The Nature paper looks at patents and papers, and attempts to measure how much future research was built on a given publication, or how much a given piece of work served to “push science and technology in new directions.”
The finding: Yeah, it does seem that there are fewer radical innovations than there used to be.
Are we getting worse at transformative science?
This isn’t a new question. As the Nature paper notes, previous studies “document declining research productivity in semiconductors, pharmaceuticals, and other fields. Papers, patents, and even grant applications have become less novel relative to prior work and less likely to connect disparate areas of knowledge, both of which are precursors of innovation. The gap between the year of discovery and the awarding of a Nobel Prize has also increased, suggesting that today’s contributions do not measure up to the past.”
But those are fairly narrow measures of progress, many of them limited to a single field or highly subjective (like the judgments of the Nobel Prize committee). The Nature researchers aimed to look at a more comprehensive measure. So they evaluated 25 million papers (1945–2010) and 3.9 million patents (1976–2010) according to a new metric, the so-called “CD index,” which judges whether papers are mostly “consolidating” (or building on) knowledge in the field, or whether they’re “disrupting” the field and pointing toward new, fresh avenues of research.
The idea is that if a paper builds on previous work, citations of that paper will generally also cite previous work. If a paper blazes a new research direction, then citations of that paper are less likely to cite previous work. The lower the CD score, the less disruptive the research.
For example, the 1953 paper on the structure of DNA by James Watson and Francis Crick scores very high as “disrupting” on the CD index — it proposed a new view of DNA, and papers citing it didn’t bother citing the old, wrong models of DNA that it corrected.
The Nature authors suspected that “disrupting” papers, ones that change the field and point in new research directions, are on the decline. And indeed, that’s what they found – and the decline is incredibly dramatic.
In the “social sciences,” “the average CD5 dropped from 0.52 in 1945 to 0.04 in 2010.” In “physical sciences,” “the average CD5 decreased from 0.36 in 1945 to 0 in 2010.” For “drugs and medical” patents, “the average CD5 decreased from 0.38 in 1980 to 0.03 in 2010.” For “computer and communications” patents — one area where we might expect meaningful progress — “the average CD5 decreased from 0.30 in 1980 to 0.06 in 2010.”
Why is science getting harder?
One possibility is that we just found all the most disruptive ideas already. At the beginning of the 20th century, there was a lot of very basic work that hadn’t yet been done. Of course, the first person ever to study antibiotics would make far more progress than one of 1,000 researchers at a pharmaceutical company 100 years later. Think of it as the “low-hanging fruit” theory.
Relatedly, scientists now tend to make their important discoveries at an older age and as part of a larger team, perhaps because it requires more time and more effort to learn everything you need to know before you can even make it to the forefront of a field.
But this feels a little circular as an answer. Why aren’t scientists discovering new things? Maybe because we already discovered all the transformative and crucial things. Why do we think we may have discovered all the transformative and crucial things? Well, because scientists aren’t finding any new ones!
It seems entirely possible that the slowdown in science is not an inevitable natural law, but a result of policy choices. The way we hand out scientific grants is flawed, for instance. Despite the record level of funding, we know that visionaries with transformative ideas — like Katalin Karikó, who did crucial early work to invent mRNA vaccines — struggled for years to get grant money. And getting money requires jumping through a growing number of hoops — many leading scientists now spend 50 percent of their time writing grant proposals so they can spend the other 50 percent actually doing science.
“I think because you have to publish to keep your job and keep funding agencies happy, there are a lot of (mediocre) scientific papers out there … with not much new science presented,” wrote Kaitlyn Suski, a chemistry and atmospheric science postdoc at Colorado State University, for a 2016 Vox survey of scientists on what’s wrong with their field.
Saying that the science slowdown is inevitable because our predecessors already grabbed all the good ideas might blind us to the possibility that science is slowing down because we’re actively mismanaging it, directing researchers away from the best uses of their time and the most crucial research and toward small incremental papers that keep funders — and tenure review committees — happy.
The decline of science has huge and wide-reaching societal implications. Disruptive papers often mean new innovations that increase productivity, improve quality of life, raise wages, and save lives. Some have speculated that much of the flattening of productivity and wages in the US is driven by the slowing of scientific innovation.
In reality, of course, the decrease in innovative papers is probably the product of many factors, some we can control and some we can’t. But the new Nature paper makes it clear the effects are huge. And since science is the engine of productivity and prosperity, figuring out why it’s not running as well as it once did couldn’t be more important.
A version of this story was initially published in the Future Perfect newsletter. Sign up here to subscribe!