Two kinds of grants?
Last month I speculated that in its review of grant applications, the National Institutes of Health may apply heavy focus to the feasibility of the science. In an attempt to assess this speculation more carefully, I asked if the Center for Scientific Review might share with me data collected over the past year on R01 grant applications. Regrettably, the CSR was unwilling to share these data.
Short of access to the requested CSR data, I solicited the input of biomedical researchers working at my home institution, the University of Texas Southwestern Medical Center at Dallas. My request was in the form of a simple e-mail seeking scoring information on grant applications reviewed in 2014 (irrespective of whether funded or not). I received 86 responses, allowing deduction of the correlative relationships shown in the figure. View larger image
Recall that the NIH uses five criteria in the review of most grant applications: significance, investigator, innovation, approach and environment. The most substantive correlative relationship with an application’s “overall impact score” was the approach criterion (r = .86). Less striking but important correlations were seen for the significance (r = .65) and innovation (r = .54) criteria. Little or no correlation with overall impact score was seen for the environment (r = .37) or investigator (r = 0.36) criteria.
Admittedly, the 86 UTSWMC investigators included in these analyses represent but a small sampling of the thousands of researchers whose grant applications were reviewed by the CSR in 2014. Still, I believe certain conclusions can be drawn from these data – even if those conclusions are relevant for only my colleagues here in Texas.
First, approach is the 800-pound gorilla dictating whether a grant application is to achieve a fundable score. Second, significance and innovation contribute to overall impact score with a positive correlative influence. Third, neither the environment score nor the investigator score appears to contribute significantly to the overall impact score.
I was not surprised in seeing the strongest correlation between approach and overall impact score. Indeed, in 2009 Jeremy Berg, then director of the National Institute of General Medical Sciences, came to the same conclusion with data he presented to the NIGMS council. Berg’s analysis derived from the evaluation of 360 NIGMS grant applications, in which he found a correlation coefficient of r = .74 between approach and overall impact score.
What did surprise me from my more limited and recent data was the near absence of correlation between the environment and investigator scores and overall impact score.
These data stand as evidence that the NIH is fundamentally different from the Howard Hughes Medical Institute in weighing criteria that dictate funding decisions. The NIH wants the research plan to be sound but is largely unconcerned by the qualifications of the scientist. The HHMI bets its money on the scientist instead of the details of his or her research plan.
In thinking about the data in the figure. I offer that it might be best to split NIH grant applications into two categories. One category would be for research proposals wherein the exact nuts and bolts of the research plan are of paramount importance. These applications would be contract-like in helping fit the desires and objectives of the different research missions of the NIH’s institutes and centers. If they are sound and highly likely to work, they should be funded. I will call these ASI grants (for approach, significance and innovation) and emphasize two things. First, ASI grants would be similar to most grants currently funded by the NIH (approach-dominated). Second, the product of this research should be of critical importance to the various missions of the NIH divisions.
The track record of the scientist and the quality of his or her environment should mean little to the reviewers of these sorts of applications, offering an unappreciated opportunity. I suggest that these applications be reviewed with the applicants’ identities and institutions redacted. This proposed anonymity offers a means of combating the clublike behavior that I consider a detriment to our federal granting system in the biomedical sciences. If properly organized, this class of grant application/evaluation would place applicants on a level playing field. Simply put, the value of “club membership” would be mitigated.
A fundamentally different class of grants would be of the IE category (short for investigator and environment). These grant applications would be judged on the track record of the applying scientists rather than on the details of his or her proposal. Encouragingly, the NIGMS is now piloting its Maximizing Investigators’ Research Award program – a program designed along the lines of the IE category of applications recommended herein.
Personally, I would recommend no more than minor attention to the environment criterion for IE grants. I say this for the simple reason that environment is difficult to measure. The most dominant influence of environment is the small microcosm of colleagues available for daily intellectual interaction. Dominant biomedical research centers may have hundreds of these micro-environments; smaller institutions might have no more than a handful. The worst microcosm of a superlative biomedical research center might well be inferior to micro-environments found in less-famed institutions.
Finally, I close with the question of how it might work to score the promise of scientists, young and old, in grant programs that would be, essentially, betting on the jockey. First and foremost, analysis of past performance should be strictly limited to the five-year period preceding submission of the grant application. The long-past accomplishments of an established scientist should bear no weight on this analysis.
With respect to young scientists, it is universally the case that grant applicants have had ample opportunities to demonstrate their capabilities and to amass notable accomplishments. For decades, my colleagues and I in the Department of Biochemistry have conducted job searches for the appointment of new, freshly independent members of our faculty. Whereas our track record in choosing winners has not been perfect, it is darned good.
How do we make our choices? We look at the track records of applicants just as reviewers should for the proposed IE category of grant applications. If a young scientist made a significant discovery as either a graduate student or a postdoctoral fellow, that accomplishment is precisely what makes us think that he or she will continue on a successful trajectory. It should be no harder for review bodies to grade the promise of young scientists than it is for the evaluation of established scientists: What has the applicant discovered over the past five years?
The key question to be asked of reviewers of IE grant applications would be this: “Has the applicant made a discovery of significance over the past five years?” I emphasize this in contrast to any general measure of so-called productivity. The quantity of papers published by a scientist may or may not be indicative of that scientist having contributed a discovery of significance. Thinking in the most optimistic of terms, if this proposed metric of review were adhered to in a strident manner, applicants for IE grant awards would evolve toward the generation of a discovery corps of American scientists.
Author’s note: At press time, I was made aware of a new article in Science. Economists Danielle Li and Leila Agha conclude that the peer-review system is effective at selecting and funding high-impact science using approach-dominated methods. The authors contend that it is healthy for grant funding to be largely divorced from the past accomplishments or institutional affiliations of applicants. I see this as a strong affirmation of the idea of ASI grants being reviewed without identifying information on the applicants and their home institutions.
is president of the American Society for Biochemistry and Molecular Biology and chairman of the biochemistry department at the University of Texas-Southwestern Medical Center at Dallas.