Given the status of the U.S. budget deficit, it is unlikely that the National Institutes of Health and the National Science Foundation can expect to see significant increases in funding any time soon. Indeed, the NIH has just enacted across-the-board budget cuts, and the percentage of grants that will be able to be funded is approaching a dangerously low level. This makes the process of application evaluation incredibly important – and given the fact that the NIH alone received 77,000 applications last year, the task of evaluation may never have been more challenging.
For students and postdoctoral fellows not familiar with the workings of the NIH, grant applications are evaluated by review panels (or study sections) composed of scientists from across the country. A scientific review officer oversees the panels and ensures that meetings follow specific guidelines. The SRO assembles the panel of expert scientists, seeking to ensure representation of men, women, under-represented minorities and institutions from all around the U.S. A member of the panel is appointed to chair the proceedings.
Although research is funded by one of the many institutes or centers at the NIH (e.g., the National Institute of General Medical Sciences or the National Cancer Institute), review oversight is provided by the Center for Scientific Review, whose sole task is to oversee application review. Thus, a panel may review applications that have the potential to be funded by different institutes. In rare cases, an institute may assemble its own review panels. The job of the panel is to rank applications in relation to the other applications evaluated in that general area of research. That information is then provided to the relevant institute, and funding decisions are made by the institute, not by the CSR, depending on the institute’s budget and priorities. Individual review panels may be approach-based; for example, they may evaluate only applications in structural biology. Alternatively, they may be much broader in scope, with a single panel evaluating topics as diverse as chaperones, protein folding, membrane trafficking and mitochondrial function.
Review panels score an application after three reviewers read it (before the meeting), present their critiques to the group (during the meeting), and discuss its strengths and weaknesses. The reviewers suggest scores, and the entire panel votes based on their understanding of reviewer presentations and their assessment of how that science ranks in significance relative to other applications under discussion. The reviewers don’t always agree, and it is the responsibility of the panel’s chairman either to help achieve consensus or to identify the points of contention for the group. Panels usually meet three times a year and consider about 85 applications per meeting.
During the past several years, the NIH has sought to identify the highest impact applications, focusing all discussion on the significance of the science proposed. This is important: There is more science that can be done than dollars available, and it is essential that the NIH spend funds on the most important and impactful science. What many new applicants have trouble understanding is the fact that review panels simply rank the applications that are received. Thus, if a famous scientist in a given area submits an application at the same time as a new faculty member, they will be compared and ranked accordingly. You can submit an excellent application and still rank lower based simply on who else submitted an application at the same time. Lucky for new investigators, their applications have a special demarcation, and institute staff members reach farther down the ranked list to ensure increased success for new applicants.
One of the most valuable aspects of a review panel discussion relates to relative significance. The chairman plays an incredibly important role: Because the entire panel votes, it is the chairman’s responsibility to ensure that everyone understands the application and contributes to the discussion of its overall significance for the field. Members of the panel need to be encouraged to ask questions relating to significance, and the discussion can involve difficult questions, such as: Why is this chaperone study of greater impact to the field than another proposal to study mitochondrial fusion? How many other labs are also asking the same question using the same approaches?
The CSR faces many challenges, including recruiting the very best reviewers available. Some scientists claim to be too busy to serve; some prefer not to lose time to travel. Yet every panel must include experts with the requisite knowledge to evaluate the science under discussion. To manage this challenge, the CSR sometimes turns to telephone or internet-based reviews. There is no question that if a proposal utilizes an unusual technique it is important for an expert to be able to provide expertise regarding one or two applications. However, when reviewers phone in their comments, they don’t usually listen in to the entire meeting’s proceedings and are thus less able to contribute to the broader discussions that are so important when diverse science has to be ranked successfully. Similarly, internet-based review can be valuable when a small set of applications is under consideration. Three referees usually can reach a consensus or even carry out a heated debate in a bloglike forum. However, it is much more difficult for a larger group to discuss the relative significance of diverse areas of science using this format. The CSR is to be applauded for trying new technologies to save all of us travel time and money. Nevertheless, this applicant hopes that in-person reviews will continue to be supported, as the pursuit of high significance requires a level of discussion that is hard to achieve on a blog. In the past few years, the CSR has begun alternating meetings between the east and west coasts, a positive development that can save time and money for panel members and for the NIH.