This DEB Numbers post is a continuation of our previous post, here, where we laid out some of the measures of collaboration that are available to us in DEB. If you are new to DEB Numbers posts, we suggest you read our introductory message to familiarize yourself with some of the conventions we use here to describe what can be confusing information.

How many collaborators?

Beyond the presence/absence of GPG-defined collaborative proposals or presence/absence of Co-PIs on a project, we have some additional information that may shine a light on other facets of the collaboration question.

Clearly, an immediate follow-up question to presence/absence of collaboration is “how many are involved?”  In answer to this we can take a look at the average number of names on a project’s cover sheet(s) or the average number of institutions on a project’s budget(s).  The following tables exclude preliminary proposals because those projects are submitted by single institutions without budget information and so do not reliably provide complete information for persons or institutions. (The Excel template required for January, 2013 preliminary proposals was an attempt to improve capture of such info.)

Mean PI/Co-PI Names on Full Proposal Project Cover Sheet(s) for DEB Core Program Panel Submissions and Awards FY2007-FY2013

Mean Institutions in Full Proposal Project Budget(s) for DEB Core Program Panel Submissions and Awards FY2007-FY2013

*Tentative numbers for FY2013 under the Continuing Resolution budget scenario.

For both mean persons and mean institutions per project submission, we do not see any large change in FY2013 which would signify some sort of catastrophic impact on collaboration habits in project submissions.

The numbers for both mean persons and mean institutions on the awards, however, may give one pause.  Especially if plotted, the visualization can seem frightening:

For all four measures of the awards, FY2013 is down from FY2012. In three of the four cases the FY2013 drop is the largest single year change in the set.  Since we saw in the previous post on collaborations that the proportions of single investigator versus collaborative project awards had not changed, this observed change requires that the size of groups involved in collaborative projects expected to be awarded for FY2013 are somewhat smaller than in previous years.  How much smaller? Well, the mean changes by ~.25 in the largest drop so we are looking at one fewer participant for every four multi-investigator project awards.  This is definitely something we are keeping an eye on but, given the unique circumstances in FY2013, our interpretation is it does not yet signal a problem with the review system but rather a trade-off with the budget.

Such a decrease actually makes sense in the context of a limited and uncertain budget.  Especially given the focus on funding rates and maximizing the number of projects awarded, programs have an incentive to spread the available funding over as many projects as possible.  Programs also seek balance between multi-investigator and single investigator project awards. If multi-investigator projects with smaller collaborative groups cost less than similar projects supporting larger groups of PIs and Co-PIs, the funds saved on the less costly projects could enable more awards to be made in total.

Inter-generational Collaboration

The last measure we will take a look at in this discussion highlights a specific aspect of the collaboration concern that was raised in some quarters, namely that the submission limit would stifle collaboration between younger and older researchers.  The best measure we have to look at such inter-generational collaboration is the terminal degree year of the researchers. This number is part of each individual’s FastLane profile.  We can capture that data for the participants in each multi-investigator project and calculate the number of years between the most recent and most remote terminal degree years on each project.  The distribution of these degree year ranges was compiled for DEB Core Program Panel Multi-Investigator Projects awarded from FY2007 through FY2013.  These distributions are presented below, without further commentary, as a series of box-and-whisker plots.

We hope these two posts have started you thinking about collaboration and look forward to continuing the discussion in the comments and future posts.

A topic we have been interested in since long before the launch of the two-stage review process is how collaboration plays into the review process in DEB.  In this post, we explore the various definitions of collaborative proposals and look at measures of the extent of collaboration in DEB project submissions and awards.  If this is your first DEB Numbers post, I suggest you read our introductory message to familiarize yourself with some of the conventions we use here to describe what can be confusing information.

What does collaboration mean in DEB?

We mentioned in a previous post that the NSF GPG recognizes two different arrangements as “collaborative proposals“: 1) single proposal jackets with additional institutions as subawards, and 2) linked proposal jackets submitted by multiple institutions. Only the second arrangement is explicitly labeled as “Collaborative Research:…” in the project title.  The common feature of these two arrangements is that the full proposal projects contain budgetary information submitted by two or more organizations.

In conversation, DEBers often use the shorthand term “collabs.”  This use usually refers to only those projects consisting of multiple linked proposal jackets with “Collaborative Research:…” titles.  We are particularly interested in this subset of collaborative proposals internally because they pass through review as a single unit but become individual grants if awarded and that has effects on the processing workflows.

DEB Numbers posts report collaborative proposal information incorporating both arrangements described in the GPG.

We also recognize that there is more to collaboration than organizations coordinating project budgets. Counts of GPG defined “collaborative proposals” do not account for the vast majority of cooperative arrangements that fall under a reasonable general-public understanding of collaboration.  For instance, Co-PIs from the same university in the same or complementary fields are clearly taking part in collaboration but only a single institution receives an award.  A foreign counterpart providing access to a collection or dataset may be thought of as a collaborator (and may even have provided what solicitations call a “letter of collaboration” confirming their willingness to provide support to a project) but is not a PI or Co-PI and their organization is not an awardee. Neither of those constitutes a “collaborative proposal” but they are aspects of collaboration that interest us. However, data on such collaborations are not always systematically captured during the submission process or award life-cycle.

Our abilities to scrape meaningful data on these deeper facets of collaboration from NSF records vary, often from difficult to currently unavailable.  But, there is a lot of promise in the future.  The development of altmetrics in the wider community, StarMetrics within federal agencies, and the continued upgrading and replacement of old systems with new ones designed with data needs in mind suggest that in a couple of years we will have operational tools to better explore project collaboration.

One immediate development in this direction is the migration of project reporting from FastLane to Research.gov. Current grantees should have heard about this by now.  The switch will make the information in awardee reports easier to draw out and analyze. However, it will also be a fresh start.  We do not expect to have backward conversion of old records.  At the program level, we do not know what all the outputs and data products from the new form will look like (either internal or external to NSF). It will definitely require time and exploration to get enough data into the reporting system to figure out how to recombine it with existing data sources and produce new insights.

Those limitations in mind, there are several pieces we can look at today to give a picture of the landscape of collaboration over the last several years.

For starters we can look at the numbers of individuals appearing on project cover pages.  We can also look at the numbers of institutions represented in project budgets.  With these numbers we can look for trends in the representation of various collaborative arrangements in the submission and award portfolios.

Proportions of Three Collaboration Strategies in the Portfolio of Project Submissions to DEB Core Program Panels FY2007-FY2013

^Institutional involvement beyond the lead organization on a proposal jacket is captured in budgetary information.  This data is not generated in the submission of preliminary proposals.

*Tentative numbers for FY2013 under the Continuing Resolution budget scenario.

Here we have the data for three types of collaboration strategies:  Collaborative Proposals (GPG-definition requiring 2 or more institutions involved in the project budget), Multi-investigator projects (a broader concept of collaboration including all projects with Co-PIs even if from a single institution), and Single Investigator Projects (no named collaborations with other institutions or Co-PIs).  There is not much to interpret, the relative contributions of each collaboration strategy to the submitted project portfolio has been amazingly constant even through the FY2010 post-stimulus submission spike and preliminary proposal debut.  The only notable feature is the apparent relative increase in formal collaborative proposals at the FY2013 full proposal stage.  That change would appear to run counter to some of the concerns voiced at the launch of the two-stage review process.  However, that number is also a decent, if imperfect, proxy for the proportion of collaborative proposals in invited preliminary proposals.  When viewed in that context, it is less exciting.

Proportions of Three Collaboration Strategies in the Portfolio of Awards or Invitations from DEB Core Program Panels FY2007-FY2013

*Tentative numbers for FY2013 under the Continuing Resolution budget scenario.

Here we see a bit more variation over the reviewed period, but we are also considering a much smaller population.  Again, however, the numbers coming out of FY2013 are not out-of-place when compared against the range of values encountered since FY2007.  If the X in this table were replaced with the 43% collaborative proposals from the FY2013 full proposal submission portfolio, it too would fit right in. Regarding the preliminary proposals in FY2012, Multi-Investigator Projects appear to have fared a little better than normal at the point of invitation but we have only a single data point and that difference must be interpreted in light of several factors: single investigator CAREER projects were not part of that review stage, and both panelists and Program Officers were aware of publiczied concerns about negative impacts on collaboration and could have responded with behavioral changes.

These data initially suggest that the two-stage review process is not having an effect on collaborative proposal submissions or outcomes. “But,” you may say, “this data only reflects presence/absence of a technical marker of collaboration, what about measures of ‘how much’ collaboration is happening?”  Currently such information pushes the limits of our ability to glean data from the records, but we will take a look at what we can in part two.

A topic we have been interested in since long before the launch of the two-stage review process is how collaboration plays into the review process in DEB.  In this post, we explore the various definitions of collaborative proposals and look at measures of the extent of collaboration in DEB project submissions and awards.  If this is your first DEB Numbers post, we suggest you read our introductory message to familiarize yourself with some of the conventions we use here to describe what can be confusing information.

What does collaboration mean in DEB?

We mentioned in a previous post that the NSF GPG recognizes two different arrangements as “collaborative proposals“: 1) single proposal jackets with additional institutions as subawards, and 2) linked proposal jackets submitted by multiple institutions. Only the second arrangement is explicitly labeled as “Collaborative Research:…” in the project title.  The common feature of these two arrangements is that the full proposal projects contain budgetary information submitted by two or more organizations.

In conversation, DEBers often use the shorthand term “collabs.”  This use usually refers to only those projects consisting of multiple linked proposal jackets with “Collaborative Research:…” titles.  We are particularly interested in this subset of collaborative proposals internally because they pass through review as a single unit but become individual grants if awarded and that has effects on the processing workflows.

DEB Numbers posts report collaborative proposal information incorporating both arrangements described in the GPG.

We also recognize that there is more to collaboration than organizations coordinating project budgets. Counts of GPG defined “collaborative proposals” do not account for the vast majority of cooperative arrangements that fall under a reasonable general-public understanding of collaboration.  For instance, Co-PIs from the same university in the same or complementary fields are clearly taking part in collaboration but only a single institution receives an award.  A foreign counterpart providing access to a collection or dataset may be thought of as a collaborator (and may even have provided what solicitations call a “letter of collaboration” confirming their willingness to provide support to a project) but is not a PI or Co-PI and their organization is not an awardee. Neither of those constitutes a “collaborative proposal” but they are aspects of collaboration that interest us. However, data on such collaborations are not always systematically captured during the submission process or award life-cycle.

Our abilities to scrape meaningful data on these deeper facets of collaboration from NSF records vary, often from difficult to currently unavailable.  But, there is a lot of promise in the future.  The development of altmetrics in the wider community, StarMetrics within federal agencies, and the continued upgrading and replacement of old systems with new ones designed with data needs in mind suggest that in a couple of years we will have operational tools to better explore project collaboration.

One immediate development in this direction is the migration of project reporting from FastLane to Research.gov. Current grantees should have heard about this by now.  The switch will make the information in awardee reports easier to draw out and analyze. However, it will also be a fresh start.  We do not expect to have backward conversion of old records.  At the program level, we do not know what all the outputs and data products from the new form will look like (either internal or external to NSF). It will definitely require time and exploration to get enough data into the reporting system to figure out how to recombine it with existing data sources and produce new insights.

Those limitations in mind, there are several pieces we can look at today to give a picture of the landscape of collaboration over the last several years.

For starters we can look at the numbers of individuals appearing on project cover pages.  We can also look at the numbers of institutions represented in project budgets.  With these numbers we can look for trends in the representation of various collaborative arrangements in the submission and award portfolios.

Proportions of Three Collaboration Strategies in the Portfolio of Project Submissions to DEB Core Program Panels FY2007-FY2013

^Institutional involvement beyond the lead organization on a proposal jacket is captured in budgetary information.  This data is not generated in the submission of preliminary proposals.

*Tentative numbers for FY2013 under the Continuing Resolution budget scenario.

Here we have the data for three types of collaboration strategies:  Collaborative Proposals (GPG-definition requiring 2 or more institutions involved in the project budget), Multi-investigator projects (a broader concept of collaboration including all projects with Co-PIs even if from a single institution), and Single Investigator Projects (no named collaborations with other institutions or Co-PIs).  There is not much to interpret, the relative contributions of each collaboration strategy to the submitted project portfolio has been amazingly constant even through the FY2010 post-stimulus submission spike and preliminary proposal debut.  The only notable feature is the apparent relative increase in formal collaborative proposals at the FY2013 full proposal stage.  That change would appear to run counter to some of the concerns voiced at the launch of the two-stage review process.  However, that number is also a decent, if imperfect, proxy for the proportion of collaborative proposals in invited preliminary proposals.  When viewed in that context, it is less exciting.

Proportions of Three Collaboration Strategies in the Portfolio of Awards or Invitations from DEB Core Program Panels FY2007-FY2013

*Tentative numbers for FY2013 under the Continuing Resolution budget scenario.

Here we see a bit more variation over the reviewed period, but we are also considering a much smaller population.  Again, however, the numbers coming out of FY2013 are not out-of-place when compared against the range of values encountered since FY2007.  If the X in this table were replaced with the 43% collaborative proposals from the FY2013 full proposal submission portfolio, it too would fit right in. Regarding the preliminary proposals in FY2012, Multi-Investigator Projects appear to have fared a little better than normal at the point of invitation but we have only a single data point and that difference must be interpreted in light of several factors: single investigator CAREER projects were not part of that review stage, and both panelists and Program Officers were aware of publicized concerns about negative impacts on collaboration and could have responded with behavioral changes.

These data initially suggest that the two-stage review process is not having an effect on collaborative proposal submissions or outcomes. “But,” you may say, “this data only reflects presence/absence of a technical marker of collaboration, what about measures of ‘how much’ collaboration is happening?”  Currently such information pushes the limits of our ability to glean data from the records, but we will take a look at what we can in part two.