This is part 3 of a discussion of reviewer service in DEB. In parts 1 and 2 we talked about the roles of panelists and ad hoc reviewers and how contacting us to volunteer as a reviewer doesn’t often work. With that as background we can provide some insight into actual reviewer selection and assignment in DEB.

Let’s mention a few caveats. First, all reviewers are chosen based on three key principles: expertise, interest, and lack of conflicts. In choosing a reviewer we strive for individuals who are not only highly qualified to evaluate the proposal, but are also very interested in the topic of the research (and therefore likely to do a thorough review) and who have no apparent conflicts of interests with the individuals and institutions involved in the proposal.

Second, every time a proposal is submitted, the PI has the opportunity to submit a list of potential reviewers (a “single copy document” which does not get seen by any reviewers). Program Officers try to use at least a few of those potential reviewers after first checking that they are appropriate experts and are not in conflict with anyone (or institution) involved in the proposal. BTW, if you want to make your Program Officer happy, when submitting a proposal include ~8 such expert reviewers, free of conflicts, and provide full names, institutions, and email addresses (only about 1 in 5 PIs submits this information).

Third, Program Officers often can think of a few highly appropriate reviewers for any given proposal they are managing simply because they know the literature and lots of scientists in their field. Still, this is rarely enough for what we need. Ideally, we want 3 ad hoc reviews for each full proposal, to inform the three panelist reviewers. Given a response rate of ~50%, that means requesting at least 6 ad hoc reviews

So how do we find and solicit reviewers’ participation? It’s a little different depending on the reviewer role: ad hoc reviewer or panelist.

Ad hoc reviewers

First, think of 5 or 6 keywords which in combination are specific to your research expertise.

We can wait….

Got ‘em?

Good. Now open a new browser tab or window and navigate to your favorite academic search (e.g., Google Scholar, Scholar Universe, Web of Science, etc.) or even just a plain search engine.

Search on your combination of keywords.

Do you come up? If not, who does? Where are you? Are you even in the results? If you’re somewhere near the top, kudos! Otherwise….

This is how we find you in a nutshell. Except of course we don’t have your custom list of keywords; we use descriptions from proposals submitted by others. When we are identifying potential reviewers and panelists we’re looking for good matches. If you don’t even match your own description it’ll be hard to find you when we’re searching based on someone else’s project. Seriously, the most important thing you can do to make yourself available for review is to maintain a searchable presence on the web[i]. If you’re a post-doc or grad student and don’t have a lab but want to get started, use professional and academic social media to create a presence.

But that’s only step 1. Now we need to determine if you’re both an appropriate match and have a reasonable likelihood of actually responding to our request and completing a review.

What is behind the hyperlinks our search brought up? Boilerplate departmental faculty descriptions, publication lists that stop in 2009, old slide decks you never knew were out there, or an active lab website with descriptions of projects and recent publications? There’s a huge trust and confidentiality issue with proposal review — we’re not just going to send researchers’ best ideas, pre-publication data, etc. out to an unknown entity[ii] and hope it arrives in caring hands. You may come up at the top of the results but if those results look like an online ghost town, we’ll be skipping down to the next name on the list.

Once we’ve found an active online presence, we need to actually compare the content of the proposal and the potential reviewer’s expertise, check for conflicts, and then putting our judgment to the test ask ourselves “Will this person want to see this proposal?” If we think so, then there’s a good chance they’ll provide a thorough review or at least decline politely.

Panelists

As we mentioned above, finding panelists differs from matching individuals for ad hoc reviews. For one, the stakes are higher with a panelist. For a given set of proposals, an individual panelist plays a larger role in providing reviews than an individual ad hoc reviewer. An ad hoc review that is superficial, cursory, or goes totally off the rails can be ignored and those that never arrive have no weight at all. The same issues arising with a panelist, however, can undermine the discussion and evaluation of multiple proposals. While getting new faces onto panels is valuable to us, we also need to manage the risks of bringing in inexperienced participants.

How do we manage these risks?

Often, we look to people who have or previously had an award from us to become panelists. We’ve seen your proposal and report writing. Plus, while not required, we hope your sense of community includes giving back through review service.

We identify potential panelists through a professional network. Professional relationships actually play an important role when it comes to populating a panel. We often rely on suggestions from current reviewers and rotating Program Officers for new panelists since they both know the requirements of review service and have knowledge of different professional circles.

We first offer opportunities to gain experience through ad hoc review and service on smaller panels. Especially for early career faculty, panels outside of the core programs, like DDIGs, are often used to provide an introduction to reviewership.

Since we don’t have a database of reviewer/panelist expertise, each cluster in DEB has to rely on keeping and sharing notes to maintain institutional knowledge of the panelist pool: who has served, wants to serve, and might be able to provide needed expertise at specific future dates? Most members of a panel come out of this pool in response to a broad invitation noting the dates of upcoming panels. Of those available to serve, volunteers are selected based on the expected needs of the panel. This is often done well in advance of the proposal submission deadline and includes a few “swing” spots to round out the assembled expertise with later invitations. Those swing spots are where new people often enter the process, but we’re filling specific needs and so finding these folks more-or-less follows the same path as finding ad hoc reviewers.

So what can you do to become involved in review?

Drawing from the across this and the previous posts, we offer the following suggestions, in order of importance:

1) Make it easy for us to find you online in connection with your scientific work.

2) If you have no NSF review experience, talk to colleagues who are already involved in review and ask them to suggest you as a reviewer.

3) Recognize that, if you’re going to reach out to us, there’s not much we can do with your information if it’s not an immediate match to a specific need.

4) Say yes when you can, since more opportunities often follow. It’s ok if you can’t take on a review when we ask; tell us — we understand. But, please don’t ignore the request or we will get the idea that you have no interest in review.

[i] While we don’t ask you or your close collaborators to review your proposal, seeing those names (yours and theirs) in search results is a good sign.

[ii] With respect to having served within the NSF review system.

This is part 3 of a discussion of reviewer service in DEB. In parts 1 and 2 we talked about the roles of panelists and ad hoc reviewers and how contacting us to volunteer as a reviewer doesn’t often work. With that as background we can provide some insight into actual reviewer selection and assignment in DEB.

Let’s mention a few caveats. First, all reviewers are chosen based on three key principles: expertise, interest, and lack of conflicts. In choosing a reviewer we strive for individuals who are not only highly qualified to evaluate the proposal, but are also very interested in the topic of the research (and therefore likely to do a thorough review) and who have no apparent conflicts of interests with the individuals and institutions involved in the proposal.

Second, every time a proposal is submitted, the PI has the opportunity to submit a list of potential reviewers (a “single copy document” which does not get seen by any reviewers). Program Officers try to use at least a few of those potential reviewers after first checking that they are appropriate experts and are not in conflict with anyone (or institution) involved in the proposal. BTW, if you want to make your Program Officer happy, when submitting a proposal include ~8 such expert reviewers, free of conflicts, and provide full names, institutions, and email addresses (only about 1 in 5 PIs submits this information).

Third, Program Officers often can think of a few highly appropriate reviewers for any given proposal they are managing simply because they know the literature and lots of scientists in their field. Still, this is rarely enough for what we need. Ideally, we want 3 ad hoc reviews for each full proposal, to inform the three panelist reviewers. Given a response rate of ~50%, that means requesting at least 6 ad hoc reviews

So how do we find and solicit reviewers’ participation? It’s a little different depending on the reviewer role: ad hoc reviewer or panelist.

Ad hoc reviewers

First, think of 5 or 6 keywords which in combination are specific to your research expertise.

We can wait….

Got ‘em?

Good. Now open a new browser tab or window and navigate to your favorite academic search (e.g., Google Scholar, Scholar Universe, Web of Science, etc.) or even just a plain search engine.

Search on your combination of keywords.

Do you come up? If not, who does? Where are you? Are you even in the results? If you’re somewhere near the top, kudos! Otherwise….

This is how we find you in a nutshell. Except of course we don’t have your custom list of keywords; we use descriptions from proposals submitted by others. When we are identifying potential reviewers and panelists we’re looking for good matches. If you don’t even match your own description it’ll be hard to find you when we’re searching based on someone else’s project. Seriously, the most important thing you can do to make yourself available for review is to maintain a searchable presence on the web[i]. If you’re a post-doc or grad student and don’t have a lab but want to get started, use professional and academic social media to create a presence.

But that’s only step 1. Now we need to determine if you’re both an appropriate match and have a reasonable likelihood of actually responding to our request and completing a review.

What is behind the hyperlinks our search brought up? Boilerplate departmental faculty descriptions, publication lists that stop in 2009, old slide decks you never knew were out there, or an active lab website with descriptions of projects and recent publications? There’s a huge trust and confidentiality issue with proposal review — we’re not just going to send researchers’ best ideas, pre-publication data, etc. out to an unknown entity[ii] and hope it arrives in caring hands. You may come up at the top of the results but if those results look like an online ghost town, we’ll be skipping down to the next name on the list.

Once we’ve found an active online presence, we need to actually compare the content of the proposal and the potential reviewer’s expertise, check for conflicts, and then putting our judgment to the test ask ourselves “Will this person want to see this proposal?” If we think so, then there’s a good chance they’ll provide a thorough review or at least decline politely.

Panelists

As we mentioned above, finding panelists differs from matching individuals for ad hoc reviews. For one, the stakes are higher with a panelist. For a given set of proposals, an individual panelist plays a larger role in providing reviews than an individual ad hoc reviewer. An ad hoc review that is superficial, cursory, or goes totally off the rails can be ignored and those that never arrive have no weight at all. The same issues arising with a panelist, however, can undermine the discussion and evaluation of multiple proposals. While getting new faces onto panels is valuable to us, we also need to manage the risks of bringing in inexperienced participants.

How do we manage these risks?

Often, we look to people who have or previously had an award from us to become panelists. We’ve seen your proposal and report writing. Plus, while not required, we hope your sense of community includes giving back through review service.

We identify potential panelists through a professional network. Professional relationships actually play an important role when it comes to populating a panel. We often rely on suggestions from current reviewers and rotating Program Officers for new panelists since they both know the requirements of review service and have knowledge of different professional circles.

We first offer opportunities to gain experience through ad hoc review and service on smaller panels. Especially for early career faculty, panels outside of the core programs, like DDIGs, are often used to provide an introduction to reviewership.

Since we don’t have a database of reviewer/panelist expertise, each cluster in DEB has to rely on keeping and sharing notes to maintain institutional knowledge of the panelist pool: who has served, wants to serve, and might be able to provide needed expertise at specific future dates? Most members of a panel come out of this pool in response to a broad invitation noting the dates of upcoming panels. Of those available to serve, volunteers are selected based on the expected needs of the panel. This is often done well in advance of the proposal submission deadline and includes a few “swing” spots to round out the assembled expertise with later invitations. Those swing spots are where new people often enter the process, but we’re filling specific needs and so finding these folks more-or-less follows the same path as finding ad hoc reviewers.

So what can you do to become involved in review?

Drawing from the across this and the previous posts, we offer the following suggestions, in order of importance:

1) Make it easy for us to find you online in connection with your scientific work.

2) If you have no NSF review experience, talk to colleagues who are already involved in review and ask them to suggest you as a reviewer.

3) Recognize that, if you’re going to reach out to us, there’s not much we can do with your information if it’s not an immediate match to a specific need.

4) Say yes when you can, since more opportunities often follow. It’s ok if you can’t take on a review when we ask; tell us — we understand. But, please don’t ignore the request or we will get the idea that you have no interest in review.

[i] While we don’t ask you or your close collaborators to review your proposal, seeing those names (yours and theirs) in search results is a good sign.

[ii] With respect to having served within the NSF review system.

In the prior post we described the roles of panelists and ad hoc reviewers in the DEB merit review process and how others have highlighted the value of taking part in this process. We left off with an observation that has popped up in several comment threads on those other discussions: “I volunteered but no one ever called.” This post addresses why that happens.

Foremost among the reasons we haven’t called you back is a possibly startling admission: we don’t have an actual “reviewer database”. We have a reviewer records system but it’s an old architecture, built for a smaller scientific enterprise[i], and lacks the sort of substantial or topical content (not even research keywords) that would be needed for it to function as a database for identifying appropriate reviewers. Even with such content, because it is primarily for record-keeping and not discovery, it would simply search reviewers we’ve used in the past and wouldn’t help us to identify “new blood.” Yes, this is incredibly, woefully behind-the-times compared to what many journals use[ii]. This is an issue for which NSF is trying to find a solution.

The next major roadblock for self-identifying reviewers is timing. DEB programs need to find large numbers of reviewers in narrow windows of time. If you’re just sending us emails whenever the thought strikes you, they are likely landing on our desks in-between when we’re looking for reviewers. You can check out the review process calendar we posted previously to see when this happens for various programs. The take-home point is that we’re often looking for panelists before proposals have even been submitted (and months before a panel meets) and we are looking for specialist reviewers during just the few weeks when we are sending out proposals for “ad hoc” reviews. Hitting those periods can help put you in the right place at the right time, but is no guarantee, simply because…

Any email is one among numerous offers being passed around daily and is likely to have passed out of memory and been supplanted by other more recent offers by the time we have a need. Suffice it to say, there’s a lot of low-information-value noise in our in-boxes: CVs without context, boilerplate introductory letters where “review” appears as an afterthought, etc.

Even with a strategically-timed, well-written introduction, however, a potential reviewer may not best match our needs for a current proposal or panel and, ultimately, we don’t have anywhere to put the information and retrieve it efficiently. Any system we put in place for keeping tabs on volunteer offers is going to be 1) competing with the whole of the internet to quickly identify sufficient numbers of relevant experts and 2) filled with dead ends unless it is regularly updated. Your CV might get dropped in a shared folder and might turn up in a document search while you’re still at the same job, but it just as easily can get buried amongst the daily deluge of emails.

While advice around the academic web to “just send in your information” reflects individual experience, it is based on a perception of cause and effect when the reality is often just coincidence. You may get a call to review from someone who never saw your email if you come up in a search result. Around the time most people are first entering the potential PI/reviewer pool, they are also developing a professional web presence. And, as we said above, without a dedicated reviewer database, we put the whole internet to work to find reviewers. So, being searchable and showing up at the right moment can make sending us any sort of introduction moot. We’ll address this further in part 3.

[i] Even just 10 years ago we were dealing with only ~half the proposals we see today.

[ii] We would like to think we are pretty adept at working around this information system deficiency. Between panelists and individual (ad hoc) reviewers, DEB manages to obtain some ~10,000 separate evaluations of proposals each year. And, taking some pride in facing this adversity, we note that lots of folks never notice our lack of a proposal & reviewer matching database until they come here and we tell them.