Let’s talk about reproducibility

Published March 01 2017

In 2015, Leonard Freedman, the founder and president of the Global Biological Standards Initiative, and colleagues published a report that claimed $28 billion worth of biomedical research is irreproducible. The amount accounts for more than 50 percent of all preclinical research grants.

This analysis, published in PLOS Biology, shook scientists and science advocates alike. Articles and blog posts critical of the study have been published in the 18 months since Freedman’s analysis was published, including by us and others, that were critical of his study, citing concerns over Freedman’s potential conflicts of interest.

The inability of scientists to reproduce other scientists’ findings is an issue that the biomedical research enterprise should be concerned about, especially given that so much of the research is funded by the federal government, which is to say taxpayers. Being good stewards of taxpayer money must mean conducting sound science. Sound science requires reproducibility. But is the amount of research irreproducibility in the life sciences really so clear-cut that we can chastise an entire community? The fact is that we need more data before we can draw firm conclusions.

Researchers at the Center for Open Science recently began their own analysis on reproducibility, specifically in cancer research projects. Their preliminary results are complicated. Project researchers are examining 29 papers published in Science, Nature and Cell since 2012, repeating the same experiments and asking whether or not they can reproduce the findings. One unique aspect of this reproducibility effort is its transparency. Unlike previous efforts that have not shared which papers they attempted to reproduce, this group is publishing their findings from beginning to end — the papers they reviewed, the procedures and methods they followed, and their entire results are all available for review and scrutiny. Sean Morrison, a senior editor of eLife, told Nature, “For people keeping score at home, right now it’s kind of two out of three that appear to have been reproduced”.

Conversations about reproducibility can be different among scientists because the importance of the issue may vary significantly across different disciplines. Some efforts, like one led by the Federation of American Societies for Experimental Biology, set the issue of reproducibility as the need for scientists to be transparent, careful, diligent about recordkeeping, and mindful of reducing potential pitfalls, such as confirmation bias and poor study design, while conducting research. Scientists rightfully feel proud of the enormous contributions made by biomedical research, but they can get defensive about their work, which can complicate the issue. Moreover, even the definition of what constitutes reproducibility is murky, because science, particularly biological science, is complex. Small environmental differences from one lab to another or unclear methodology explanations can affect the ability of one researcher to reproduce the results of another.

Nevertheless, rather than overreacting to criticisms regarding reproducibility, let’s commit to the goal of making our scientific work careful, scholarly and impactful. If we articulate this goal clearly, then the enterprise will be better protected against any future criticisms, and perhaps then we’ll show the criticisms to be irreproducible.

Benjamin Corb Benjamin Corb is director of public affairs at ASBMB.