• Video with audio description (mp4)
• Transcript (PDF)

Introduction

NASA’s Space Technology Mission Directorate (STMD) is strongly committed to promoting a culture that actively encourages diversity and inclusion and removes barriers to participation. One important way of achieving this objective is to ensure that the review of proposals is performed in an equitable and fair manner.

The Flight Opportunities and Small Spacecraft Technology programs use dual-anonymous peer review (DAPR) to evaluate proposals for the TechFlights solicitation, an Appendix of the SpaceTech-REDDI umbrella. Under this system, the identity of reviewers will not be shared with proposers, and the identity of proposers will not be shared with reviewers until after the technical merit has been evaluated for all of the anonymized proposals.

Motivated by a successful study conducted for the Hubble Space Telescope, our partners in NASA’s Science Mission Directorate (SMD) piloted DAPR to evaluate proposals for several ROSES-2020 program elements. These pilot implementations showed improvements, both in terms of the overall quality of the review process, as well as in the demographics of awardees.

Guidance for Proposers

Proposers should ensure that they adhere to the specific requirements for the particular Appendix to which they are proposing. Useful tips for preparing an anonymized proposal for a solicitation that uses DAPR are highlighted below. Additional information can also be found in the “Guidelines for Proposers to SpaceTech-REDDI Dual-Anonymous Peer Review (DAPR) Programs” document available under “Other Documents” on the NSPIRES page for TechFlights 2022.

Essential Points

Do not include the names of the personnel associated with the proposal or their organizational affiliations. This includes but is not limited to, page headers, footers, diagrams, figures, or watermarks. This does not include references to past work, which should be included whenever relevant (see item 4 below). 

References must be written in the form of a number in a square bracket (e.g., [1]), which will then correspond to the full citation in the reference list. The purpose of using the square-bracket approach is to remove attribution to a particular investigator or group in the main body of the proposal. This will help in masking for the reviewers who the proposers may be during the reading of the proposal. 

When citing references or previous experience, use third person neutral wording and do not claim ownership of past work. This especially applies to self-referencing. For example, replace phrases like “as we have demonstrated in our previous work [17], …” with “as previously demonstrated [17], …”

Depending on the Appendix, it may be occasionally important to cite exclusive access to resources (e.g., hardware, testing facilities, non-public software, unpublished data, findings that are not citable). Each of these may reveal (or strongly imply) the identity of the proposers. In these instances, proposers must use language such as “obtained in private communication” or “from private consultation” when referring to such potentially identifying work. If proposers include this type of citation, do not include with whom the personal communication took place (i.e., do not refer to the names or roles of individuals or provide a description of a team or group).

Depending on the Appendix, it may be occasionally important to cite specific proprietary hardware being used or improved in the project.  Each of these may reveal (or strongly imply) the organization performing on the proposal. In these instances, proposers must use language such as “provided by the manufacturer/developer” or “with permission from the manufacturer/developer” when referring to potentially identifying hardware use. If proposers include this type of citation, do not include who provided the permission or hardware (i.e., do not refer to the names or roles of individuals or provide a description of the organization.)

Example Anonymized Proposals 

Dual-anonymous peer review has been used by several programs in the Science Mission Directorate (SMD) for the past few years. SMD has a website for proposers to become more familiar with DAPR and its incorporation into the NASA proposal preparation and review process at: https://science.nasa.gov/researchers/dual-anonymous-peer-review. 

While the focus of the website is on the preparation of scientific proposals rather than proposals for technology development, testing, and demonstration, it contains several examples of how proposal text can be anonymized. These examples may be useful for proposers to the SpaceTech-REDDI appendices that are using DAPR as well. 

Guidance for Reviewers

The overarching objective of dual-anonymous peer review is to ensure that the review of proposals is performed in an equitable and fair manner. To ensure this goal, the review panels will be instructed to evaluate proposals on their technical merit without taking into account the proposing team’s qualifications. Sample points of the guidance provided to reviewers include:

1. Consider proposals solely on the technical merit of what is proposed.
2. Do not spend any time attempting to identify the PI or the team. This applies even if you think you know the identities of the team members. Remember to discuss the technology and proposed plans, not the people.
3. In the panel discussions, do not make guesses on identities, insinuate the likely identities, or instigate discussion on a possible team’s past work.
4. When writing evaluations, use neutral pronouns (e.g., “what they propose”, or “the team has previously evaluated similar data”).