January 2012

The NSF’s two-criteria review

Broader impacts are here to stay


Joseph Provost mentors undergraduate Lisa Magstadt while conducting research funded by the NSF.

Preliminary data? Check. Budget worked out and passed through institutional review? Check. Fantastic (and transformative) hypothesis with strong experimental design? Seems like it. Broader impacts? Huh? What’s that? Never mind; I’ll write up some outreach bit and worry about it later.


Although the National Science Foundation’s two merit-review criteria have been around for nearly 15 years, this scene continues to be played in the office of many a well-meaning proposal writer. It doesn’t take long to find articles or faculty opposing this system; however, the criteria are here to stay, and, in fact, criterion two, “broader impacts,” is changing.

At a recent national meeting, I met with a number of scientists and asked what they knew about broader impacts and how they incorporated them into their proposals. Surprisingly, only a few seemed to understand what the two-merit-criteria system was all about. Some got upset about having to consider diluting their time with another activity. In 1997, the NSF began requiring two components in all proposals. Since 2002, both criteria must be addressed in a proposal or the work will not be reviewed. The first criterion, “intellectual merit,” is the discipline-specific review of the proposed work. The second, “broader impacts,” has increased in importance and is better understood by the general community. This second criterion was developed in response to the Clinton administration’s requirement of accountability and fits the NSF’s goal to integrate research and education throughout its programs. The current definition of broader impacts requires the proposal writer to work outside of a strictly scientific aim and to align some part of his or her project with a greater societal need. What is hard for some to come to terms with is that this criterion typically is best met neither by the impact of the research results nor by the training of graduate students and postdocs. Both criteria are part of the review process and incorporated in funding decisions, and, as a result, broader impacts must be taken quite seriously.

In 2010, the National Science Board and the NSF initiated a review of both criteria and in the summer of 2011 requested feedback from the stakeholders. The resulting assessment was that “the two review criteria of intellectual merit and broader impacts are in fact the right criteria for evaluating NSF proposals, but that revisions are needed to clarify the intent of the criteria, and to highlight the connections to NSF’s core principles” (1). Part of the upcoming changes to the review criteria will include gauging how a proposal reflects one or more of the national goals as defined by the America COMPETES Reauthorization Act of 2010 (2). This U.S. House resolution states that the NSF must include broader impacts as a review criterion to achieve a range of national goals (see box).

EPD_boxHow might we best meet these needs? One way not to do it is to tack on some activity or presentation to a K – 12 class not related in any way to your proposed work. Another way to diminish a proposal is to focus on only the research results. If the results of your research meet one of the criterion, then relate how this happens. To create a stronger, broader impact takes time, just like creating a well-supported hypothesis and scientific approach. Make a coherent plan that incorporates some facet of your science into the activity. Find support for your activity and, just as you would when finding a research collaborator, include letters of support from your institution to help with your activity. If you are planning to work with a regional school, get a supporting letter from a teacher, principal or counselor showing real commitment. If your broader impact activity is to create a new course to expand scientific understanding, then include how it will be incorporated into your institution’s curriculum. Provide evidence of support from your departmental chair or dean for the class and, if needed, resources to sustain the activity beyond funding the research. Many universities have existing outreach programs you can build upon.

Next, assess how you will evaluate these goals. If your activities include students or other participants, does your budget reflect your commitment? Statements about including teachers and undergraduates or increasing access for students without a budget to support their work over a summer are a red flag for some reviewers. Most importantly, take time to think about what you want to do, find support and don’t be afraid to ask a program director for guidance and insight. Ask early and ask often. He or she will be very interested in helping you work through the current and pending changes to the broader-impacts criteria.

It is critical for us to become accountable to our funders by meeting the public need. The private and public sectors are simply running short on STEM-trained workers. What is worse is that this challenge to America’s competitiveness has grown over the past five years. Who better to build this workforce and change the nature of how science is perceived by our society than our own community?

  1. 1. NSB/NSF seeks input on proposed merit review criteria revision and principles. www.nsf.gov/nsb/publications/2011/06_mrtf.jsp  
  2. 2. America COMPETES Reauthorization Act of 2010. www.nsf.gov/statistics/about/BILLS-111hr5116enr.pdf 

EPD_provostJoseph Provost (joseph.provost@ndsu.edu) is at the Center for Biopharmaceutical Research and Production at North Dakota State University and serves on the ASBMB Education and Professional Development Committee.

found= true1622