A measured response
to reproducibility problems

The research enterprise has been enduring a crisis of confidence. Many reports have been published over the past several years suggesting the biomedical research enterprise is generating irreproducible research and wasting billions of taxpayer dollars. In June, the National Institutes of Health released two notices (1, 2) describing new policies and procedures the agency would implement in 2016 to improve reproducibility in research . While improving reproducibility in research is a laudable goal, papers that make sweeping generalizations or conduct incomplete analyses about science have the potential to slow the pace of research.

One paper published in June in the journal PLoS Biology suggested the United States spends roughly $28 billion each year on irreproducible research. This paper used fairly broad criteria to determine whether research was irreproducible. Using these criteria, the authors suggested that roughly half of all research is irreproducible and then extrapolated to say that roughly half of all money invested in biomedical research is spent on irreproducible findings. The 1:1 correlation between irreproducibility and investment is simplistic and ignores the possibility that many of the findings in a paper are reproducible and useful. By using such a broad definition for irreproducible research, the authors surely mislabel useful findings and inflate their estimate of how much research is irreproducible.

Another paper published in August, in the Proceedings of the National Academy of Sciences, suggests research efficiency, determined by dividing the number of new drugs approved in a year by the NIH dollars spent in that year, has fallen in recent decades. This report makes its case on new drug approvals being the primary output of NIH-funded research. However, NIH-funded research can improve human health by more than just the discovery of new drugs. Discoveries that result in new diagnostics that promote early disease detection or technologies that improve drug delivery methods can have profound effects on human health. Additionally, drugs approved for use in humans decades ago can be repurposed to treat different diseases. These benefits of NIH-funded research were not accounted for in this analysis. Thus, this paper undersells the effectiveness of NIH-funded research and the contributions made by the scientific community.

Reports such as the two described above do little to improve research reproducibility and efficiency. The American Society for Biochemistry and Molecular Biology is concerned that the hype surrounding these papers and other reports and opinion articles will provoke Congress or research-funding agencies to implement policies that are overly prescriptive. Another concern is the potential misperception by the general public that generating new drugs is the sole purpose of NIH-funded research. This could put pressure on the agency to make a more deliberate transition from funding basic to translational science.

It is important to ensure the research enterprise is producing the best possible research, and the ASBMB long has supported efforts to improve reproducibility and efficiency. Scientists long have been excellent stewards of the taxpayer investment in biomedical research, and the scientific community should constantly be working to ensure it generates useful, well-founded knowledge. But only with measured analyses of the functioning of the research enterprise can policies be made that improve research reproducibility and efficiency.

Photo of Chris Pickett Chris Pickett is a policy analyst at the ASBMB.