A new reproducibility initiative at the NIH

Recent articles in The Economist have highlighted a growing problem within biomedical research — high-profile findings relevant to human health are often irreproducible outside of the lab making the original report (see here and here). This problem may be new to the public, but it is not new to scientists. Several articles have been published in scientific journals recently discussing issues of reproducibility in research (see here, here and here). Many of these discussions place the blame for the lack of reproducibility on the community’s bias against reporting negative results and insufficient reporting of methodological approaches or erroneous approaches altogether — few papers report blinded or randomized trials in animal models or sample-size power calculations. To be clear, this is not an issue of intentional fraud but honest omission, lack of an attention to detail and systemic poor training in experimental design in preclinical research of all types.
Most, if not all, of the research called into question was funded in some part by the National Institutes of Health, and reports of irreproducible research have spurred the agency to action. Last year, an ad hoc committee appointed by NIH Director Francis S. Collins identified three underlying problems:

  • • poor training in experimental design;
  • • poor evaluation of submitted manuscripts and grant applications by journals and grant reviewers, respectively; and
  • • perverse reward incentives, or as NIH Deputy Director Lawrence Tabak put it, “It’s publish or perish on steroids out there.”

Over the past several months, Tabak and other NIH institute and center directors have been discussing the findings of the ad hoc committee with each institute’s and center’s governing council as well as making presentations to individual scientific societies. The ad hoc committee established five principles and two suggestions for rectifying the reproducibility issue:

  1. 1. Raise community awareness: Institute and center directors are disseminating information about this initiative to all of their advisory councils, and council members are encouraged to further distribute the information to their colleagues.
  2. 2. Enhance formal training: The Office of Intramural Research is going to create and pilot a teaching module for students on experimental study design. When complete, the Office of Extramural Research will post the module online as a model for the extramural community.
  3. 3. Improve the evaluation of grant applications: Select institutes and centers will examine methods to improve the grant-review process to value study design, including asking reviewers to review not only the grant application but also the main papers on which the application is based.
  4. 4. Protect the integrity of science by adopting more systematic review processes: The NIH will partner with journals to determine the value of recently adopted reporting guidelines. (For example, see here and here.)
  5. 5. Increase stability for investigators: Improve the biosketch so that a scientist can indicate his or her contribution to the work listed.

    Suggestion A:
    Consider the use of guidelines or checklists to evaluate grants: This would encourage reviewers to evaluate grants based on a checklist of topics the grant must communicate. This will be piloted by select institutes and centers.

    Suggestion B:
    Fund reproducibility studies and centers. The National Institute of Neurological Disorders and Stroke and the National Institute on Aging have piloted funding replication studies prior to launching clinical trials.

Tabak, in his presentations on the issue, has stressed that the NIH leadership understands there are many problems with implementing any or all of the principles to improve the reproducibility of research.
First, asking overburdened grant reviewers to put more work into the grant applications they review could doom the entire initiative. Second, policies that improve reproducibility in one field may be overly burdensome in another, and a one-size-fits-all policy may not be appropriate. Third, the NIH wants to minimize any differential effects of these policies on early vs. late-stage investigators. Finally, publishing or otherwise disseminating negative results is essential for improving reproducibility, but the best mechanism to do this is not clear.
The American Society for Biochemistry and Molecular Biology and its members have a significant role to play in this process. The NIH Reproducibility Initiative is still a new program with many kinks to work out. While guidelines to improve reproducibility have been established, the agency still is soliciting comments from the extramural community about how best to implement these guidelines, while various institutes and centers are piloting several possible solutions.
However, the success of this program is important not just for the progress of science and improving human health but also for public perception. The NIH and scientists across the nation would suffer a loss of credibility should their work be viewed as largely irreproducible. This might even invite congressional action. To prevent this loss of credibility while maintaining the rigor of basic biomedical research, the ASBMB will continue to advise the NIH on how best to implement reproducibility policies without negatively affecting basic researchers. In addition, the ASBMB is engaging with a number of congressional offices to ensure that lawmakers understand the complexities of scientific research.

Photo of Chris PickettChris Pickett (cpickett@asbmb.org) is the senior science policy fellow at the ASBMB.