Funding decisions: the NIH method

Number of HHMI investigators and NAS members who were on certain study sections in 2014

  • Cell Signaling and Regulatory Systems – 3
  • Development 1 – 9
  • Genetic Variation and Evolution – 3
  • Immunity and Host Defense – 3
  • Macromolecular Structure and Function C – 4
  • Molecular Genetics A – 3
  • Prokaryotic Cell and Molecular Biology – 3
  • Vector Biology – 3

My column last month briefly described how the Howard Hughes Medical Institute appoints new investigators and reviews existing investigators. Now let’s switch to the grant review system at the National Institutes of Health, the largest funding source of biomedical research worldwide.

For the purpose of simplicity, I have in two ways restricted my analysis of the NIH grant review system. First, I’ve limited it to R01 grants and R37 grants (also known as MERIT awards). Second, I’ve limited my analysis to the 177 study sections chartered by the Center for Scientific Review. These 177 study sections assist in the disbursement of roughly $10 billion annually in the form of R01 and R37 awards.

Each NIH grant application is evaluated first by three or four members of a study section and then scored by the study section as a whole, which is composed of 20 to 30 scientists. Reviewers judge the application using five criteria: significance, investigator, innovation, approach and environment. Although study sections may weigh the importance of these criteria to differing degrees, it seems to me that feasibility of success of the proposed research plan (approach) tends to dominate. I will endeavor to provide a quantitative assessment of this in next month’s essay. Unlike at the HHMI, NIH reviewers do not interview applicants in person, and past performance is not of enhanced significance.

As reported last month, 72 percent of reviewers serving the HHMI are members of the National Academy of Sciences. How do things compare at the NIH? Data kindly provided by the CSR indicate that there were 7,886 reviewers on its standing study sections in 2014. Evaluation of these data reveals the following:

  1. 48 out of 324 HHMI investigators (15 percent) participated in at least one study section meeting.
  2. 47 out of 488 NIH-funded NAS members (10 percent) participated in at least one study section meeting.
  3. 11 of these reviewers are both funded by HHMI and NAS members.

These 84 scientists constituted roughly 1.1 percent of the reviewer cadre utilized by the CSR.

Where do these HHMI investigators and NIH-funded NAS members serve? As shown in Table 1, eight study sections enjoyed the participation of three or more HHMI and/or NAS members during 2014. All eight of these study sections can readily be categorized as being focused on basic biomedical research. As such, it is clear that when they participate in the CSR review process, HHMI investigators and NAS members gravitate toward study sections charged with the review of basic research to a far greater extent than clinical research.

Herein I compare the means by which two organizations operate in the important task of scientific review. I summarize several differences in the manner of review employed by the HHMI compared with the NIH.

  1. The HHMI reviews applicants in person; the NIH does not.
  2. The HHMI employs a smaller review team to disburse its funds ($20 million disbursed per reviewer); the NIH employs a much larger team ($1.3 million disbursed per reviewer).
  3. The HHMI pays attention to the track record of a scientist; the NIH seems to pay more attention to the details of the proposed research plan.
  4. The HHMI relies heavily on highly accomplished scientists to conduct its reviews (72 percent of HHMI reviewers are NAS members); the NIH does not (less than 2 percent of NIH study section reviewers are HHMI investigators, NAS members or both).

I close with several comments based on these differences.

First, I emphasize that there are many, many highly accomplished scientists who are not HHMI investigators or NAS members, and it is imperative that these scientists participate in the review of NIH grant applications. I have focused on the HHMI investigators and NAS members because it is straightforward to identify them and quantify their participation in the review process. It is my belief that HHMI investigators and NIH-funded members of the NAS are substantively accomplished. I readily admit that scientific accomplishment does not necessarily equate to effective capacity to review. I do, however, believe that a reasonable correlation exists between past scientific accomplishment and capacity to choose effectively between good and poor bets. This contention is open for debate and is — to me — of significant importance.

Second, even if all HHMI investigators and NIH-funded NAS members were to participate in NIH study sections, they would constitute only 9 percent of the full roster (707 out of 7,886). The only way to change this percentage would be to reduce substantively the total number of reviewers. I will address this concept in next month’s essay.

Third, it is clear that HHMI investigators and NIH-funded members of the NAS participate in study sections charged with the review of basic research to a far greater extent than clinical research. It is my belief that study sections involving HHMI investigators and NAS members benefit from the involvement of highly accomplished scientists. If that is correct, the quality of certain basic science study sections may be high.

I assume that it is a common desire of our biomedical community that all sources of funding, be they private or public, find their way to the support of our most qualified scientists — irrespective of age, gender, ethnicity, geographical location or any other variable. In subsequent essays, I will offer ideas as to how the NIH system of grant award distribution might be altered to meet this goal.

Steven McKnight Steven McKnight is president of the American Society for Biochemistry and Molecular Biology and chairman of the biochemistry department at the University of Texas-Southwestern Medical Center at Dallas.