Review the reviewers

A large fraction of the National Institutes of Health’s budget is spent in support of extramural research. These funds are the lifeblood of biomedical research in the United States. NIH grant applications are reviewed, in most instances, by members of 176 study sections organized by the Center for Scientific Review. How can we know whether these reviewers are doing a good job? Are the reviewers reviewed?

The review of reviewers happens in spades at the Howard Hughes Medical Institute. The senior leadership of the HHMI, including its president, Robert Tjian, sits in on each and every investigator review. They also listen to the talks of candidates for new appointments to the HHMI. The HHMI leaders continually judge the capabilities of their contracted reviewers too. “We monitor reviewer performance on an ongoing basis. If a reviewer exhibits anything other than substantial competence, that person is relieved of his or her responsibilities,” Tjian told me.

The NIH is composed of 27 institutes, each headed by a director. On average, these 27 institutes disburse around $500 million each in external funding per year. This is about half the amount of funding disbursed annually by the HHMI. If the HHMI can review its reviewers effectively in the disbursement of $1 billion annually, it should be possible for individual institutes of the NIH to do the same.

Each institute director should care as passionately about the disbursement of his or her institute’s funds as the president of the HHMI cares about the distribution of his institute’s own funds. Like the president of the HHMI, the NIH institute directors should keep a finger on the pulse of the review process controlling disbursement of their precious funds. As such, I offer that it is only reasonable to ask that NIH institute directors pay as much attention to the research review process as does the president of the HHMI. NIH institute directors and their senior staff should be at study section meetings just as the president of the HHMI and its senior leadership are at each investigator review.

How might it be possible for the NIH to review its reviewers? Let’s consider what would be expected of NIH institute directors were they to pay keen attention to how their funds are distributed to extramural researchers. Knowing that there are 176 study sections and 27 institutes, each institute director would — on average — care about the operation of roughly six or seven study sections. The director of the National Cancer Institute should care about the five to 10 study sections that review research proposals in the field of oncology; the director of the National Institute of Mental Health should care about the handful of study sections that review proposals relating to neuroscience; and on and on.

If each study section meets three times a year, and if each institute were paired with the handful of study sections covering the research most relevant to its mission, each institute director and his or her leadership team would need to sit in on roughly 20 study section meetings per year. A bit less than every other week, institute directors would spend two days ensuring the quality of the study section and its decisions about funding. This would consume about a quarter of the time of each institute director.

If institute directors and their leadership staffers were embedded deeply in the review process, they could monitor the performance of reviewers directly. If institute directors and their leadership staff noted anything less than outstanding performance by a study section reviewer, much less the person chairing a study section, they would be expected to relieve that person and replace him or her with a competent one.

For all I know, there might be federal rules in place that directly prevent this. If so, how incredibly foolish it would be to prevent NIH institute directors from directly monitoring and guiding the spending of their substantive budgets.

Recognizing that these ideas likely will be deemed ridiculous and impossible to implement, I close by asking a simple question: Were we just starting to devise a system to disburse federal grant dollars in support of biomedical research, would we choose the hands-on methods of the HHMI or the hands-off methods of the NIH?

Steven McKnight Steven McKnight is president of the American Society for Biochemistry and Molecular Biology and chairman of the biochemistry department at the University of Texas-Southwestern Medical Center at Dallas.