July 2011

The challenge of reviewing grant applications


Given the status of the U.S. budget deficit, it is unlikely that the National Institutes of Health and the National Science Foundation can expect to see significant increases in funding any time soon. Indeed, the NIH has just enacted across-the-board budget cuts, and the percentage of grants that will be able to be funded is approaching a dangerously low level. This makes the process of application evaluation incredibly important – and given the fact that the NIH alone received 77,000 applications last year, the task of evaluation may never have been more challenging.

For students and postdoctoral fellows not familiar with the workings of the NIH, grant applications are evaluated by review panels (or study sections) composed of scientists from across the country. A scientific review officer oversees the panels and ensures that meetings follow specific guidelines. The SRO assembles the panel of expert scientists, seeking to ensure representation of men, women, under-represented minorities and institutions from all around the U.S. A member of the panel is appointed to chair the proceedings.

Although research is funded by one of the many institutes or centers at the NIH (e.g., the National Institute of General Medical Sciences or the National Cancer Institute), review oversight is provided by the Center for Scientific Review, whose sole task is to oversee application review. Thus, a panel may review applications that have the potential to be funded by different institutes. In rare cases, an institute may assemble its own review panels. The job of the panel is to rank applications in relation to the other applications evaluated in that general area of research. That information is then provided to the relevant institute, and funding decisions are made by the institute, not by the CSR, depending on the institute’s budget and priorities. Individual review panels may be approach-based; for example, they may evaluate only applications in structural biology. Alternatively, they may be much broader in scope, with a single panel evaluating topics as diverse as chaperones, protein folding, membrane trafficking and mitochondrial function.

Review panels score an application after three reviewers read it (before the meeting), present their critiques to the group (during the meeting), and discuss its strengths and weaknesses. The reviewers suggest scores, and the entire panel votes based on their understanding of reviewer presentations and their assessment of how that science ranks in significance relative to other applications under discussion. The reviewers don’t always agree, and it is the responsibility of the panel’s chairman either to help achieve consensus or to identify the points of contention for the group. Panels usually meet three times a year and consider about 85 applications per meeting.

During the past several years, the NIH has sought to identify the highest impact applications, focusing all discussion on the significance of the science proposed. This is important: There is more science that can be done than dollars available, and it is essential that the NIH spend funds on the most important and impactful science. What many new applicants have trouble understanding is the fact that review panels simply rank the applications that are received. Thus, if a famous scientist in a given area submits an application at the same time as a new faculty member, they will be compared and ranked accordingly. You can submit an excellent application and still rank lower based simply on who else submitted an application at the same time. Lucky for new investigators, their applications have a special demarcation, and institute staff members reach farther down the ranked list to ensure increased success for new applicants.

One of the most valuable aspects of a review panel discussion relates to relative significance. The chairman plays an incredibly important role: Because the entire panel votes, it is the chairman’s responsibility to ensure that everyone understands the application and contributes to the discussion of its overall significance for the field. Members of the panel need to be encouraged to ask questions relating to significance, and the discussion can involve difficult questions, such as: Why is this chaperone study of greater impact to the field than another proposal to study mitochondrial fusion? How many other labs are also asking the same question using the same approaches?

The CSR faces many challenges, including recruiting the very best reviewers available. Some scientists claim to be too busy to serve; some prefer not to lose time to travel. Yet every panel must include experts with the requisite knowledge to evaluate the science under discussion. To manage this challenge, the CSR sometimes turns to telephone or internet-based reviews. There is no question that if a proposal utilizes an unusual technique it is important for an expert to be able to provide expertise regarding one or two applications. However, when reviewers phone in their comments, they don’t usually listen in to the entire meeting’s proceedings and are thus less able to contribute to the broader discussions that are so important when diverse science has to be ranked successfully. Similarly, internet-based review can be valuable when a small set of applications is under consideration. Three referees usually can reach a consensus or even carry out a heated debate in a bloglike forum. However, it is much more difficult for a larger group to discuss the relative significance of diverse areas of science using this format. The CSR is to be applauded for trying new technologies to save all of us travel time and money. Nevertheless, this applicant hopes that in-person reviews will continue to be supported, as the pursuit of high significance requires a level of discussion that is hard to achieve on a blog. In the past few years, the CSR has begun alternating meetings between the east and west coasts, a positive development that can save time and money for panel members and for the NIH.

Some American Society for Biochemistry and Molecular Biology members have written to me to share their frustration with the shorter format of critiques now provided by review panels. In previous years, when a higher percentage of submitted applications could be funded, these critiques provided important clues to aid applicants in crafting revised applications. Now, the NIH permits submission of only one revised application. This new rule was instituted so that there would not be backlogs of revised applications receiving priority over new and exciting submissions. While well intended, the new rule has frustrated many scientists, because at the moment, even outstanding proposals are not receiving a score that the institutes can fund. Perhaps the rule can be modified so that applicants who obtain a priority score in the 20th percentile or better would be able to submit one additional revised version (“A2”). An additional frustration stems from the fact that review panel rosters can change from one meeting to the next. Thus, a proposal revised in response to one set of comments may fail on resubmission due to a completely new set contributed by a different group of reviewers. Tight budgets also drive reviewers to find reasons not to fund something rather than to try to find reasons in favor of funding. This can lead to very good proposals being nitpicked to death over trivial issues of experimental detail. It is the responsibility of the panel chairman to stop this trend, but once a discussion has any negative tone, it is very difficult to turn it around.

While one can learn a great deal about the grant process by serving on a panel, I usually discourage junior faculty from serving until after they have obtained tenure. More senior scientists can provide a broad perspective in terms of what constitutes the most innovative science and what will offer the most significant advances, hopefully without nitpicking the details provided by a researcher with a strong record of previous accomplishment. These individuals also may have more time to commit to grant reading and critique writing, which is significant. How can we encourage more top scientists to serve on panels? ASBMB has provided the CSR with a long list of members who are willing to serve. ASBMB Past-president Gregory Petsko has called for a jury pool system where all grant recipients must be willing to serve if called upon; I support this approach wholeheartedly. I also encourage you to contact SROs in your research area and suggest names of senior experts who would add depth and knowledge to current panels. Encourage your colleagues to serve, because there has never been a more important time for us to help out. By working together with the SROs at the CSR, we can enhance the review process. Thanks also to our members Bruce Alberts, Etty Benveniste, Heidi Hamm, David Korn and Keith Yamamoto, who advise the CSR, and to all ASBMB members who volunteer to review applications for the NIH and the NSF at this critical time in research funding.

Suzanne PfefferASBMB President Suzanne Pfeffer (pfeffer@stanford.edu) is a professor of biochemistry at the Stanford University School of Medicine.

found= true1425