FAQs: 12 frequently asked questions (and answers) about the anonymous review study
1. What is the ‘anonymous review’ study?
The Office of the Women in STEM Ambassador is leading a national trial to study the effects of anonymising grant applications for in-demand scientific equipment, like telescopes, synchrotrons, and supercomputers.
In this trial, applicants are asked to remove all investigator names and gender pronouns from their research applications. They also need to remove reference to where they work and any language that identifies previously published research as their own. By removing this information, reviewers don’t know who the applicants are. This process is known as anonymous peer-review.
The trial results will be compared to data from previous (non-anonymous) years to measure any differences. The study will assess whether the anonymised review of applications reduces unconscious bias in allocating time on scientific equipment.
2. Who is involved?
Four large Australian research organisations are involved in the trial and study:
- ANSTO (Australian Nuclear Science and Technology Organisation)
- NCI (National Computational Infrastructure)
- AAL (Astronomy Australia Limited)
3. Why move to anonymous reviews of applications?
Several studies have shown that a reviewer’s attitude toward a submission may be affected, either consciously or unconsciously, by the lead author’s identity (referred to as the ‘Principal Investigator’). This is known as unconscious bias (see FAQ 4 below).
Grant and resource allocation applications led by women have systematically lower success rates than those led by men. A meta-analysis of many studies from around the world found that men had 7% higher odds of fellowship and grant funding than women applicants. This trend, in part, reflects the disproportion in the number of women in different researcher fields and at senior levels. But a large body of research* suggests that the trend is also partially due to unconscious bias. For example, a Canadian study found that women grant applicants are equally successful when peer reviewers assess the science but not when they assess the scientist.
Research* suggests that removing identifying information, such as names and gender pronouns, can reduce bias and level the playing field — not just for women but also other marginalised groups and early career researchers.
*For relevant studies, take a look at the annotated bibliography compiled for a recent study on NASA’s dual-anonymous review process.
4. What is unconscious bias?
We all have unconscious biases. Unconscious biases are a result of our brains taking mental shortcuts. Our brains take in massive amounts of information all the time — more than they can process. So, we rely on mental shortcuts to help us sort through that information quickly. Although these shortcuts can be helpful, they can also negatively influence our thoughts and decisions.
Many studies* show that we are drawn to those who think, look and act like us. There are evolutionary, physiological, cultural and many other reasons for this behaviour.
There are many different types of bias. Examples are:
- Performance bias: we tend to underestimate women’s performance and overestimate mens’.
- Confirmation bias: the tendency to search for, interpret, focus on and remember information in a way that confirms one’s preconceptions.
- Affinity bias: we gravitate toward people like ourselves in appearance, beliefs, and background.
*For relevant studies, look at the literature review on unconscious bias compiled by Advance HE (Higher Education).
5. Are truly anonymous applications even possible? Can reviewers guess?
Even in relatively small and niche research communities, it is not as likely as one might believe to guess the author of an application correctly. Studies from other similarly small fields suggest that most reviewers of anonymised grant applications and conference papers could not guess the Principal Investigators’ identity accurately.
So, while a system that provides perfectly unbreakable anonymity would be ideal, our goal is to obscure identity, discourage guessing, reduce unconscious bias, and not make the Principal Investigator (and their team) the focus of the evaluation of an application. Simply put, our goal is to focus on assessing the science rather than the science teams.
6. Is there scientific evidence that anonymising applications reduces bias?
Yes. Literature is abundant on this topic from many different fields.
In a recent example, NASA adopted anonymous evaluation of applications for observing time on the Hubble Space Telescope. For the first time in the 17 years of appropriate record-keeping, applications with female Principal Investigators had a higher success rate than those led by men. Three years after the move to anonymous assessment, the gap between women and men’s success rates has shrunk from 6% to less than 1%. The results have prompted other organisations, such as the European Southern Observatory (ESO) and European Space Agency, to follow suit. A similar trial by the Irish Research Council recently led to an immediate 10% increase in women recipients of STEM postdoctoral awards. It reversed the male-skewed success rates that had persisted for many years.
Some relevant journal articles on this are available in the annotated bibliography compiled for a recent study of NASA’s dual-anonymous review process.
7. What does this mean for the people applying for instrument time (applicants)?
The anonymous review process requires some changes in the way applicants write their applications.
Researchers applying to use certain telescopes, synchrotrons, and supercomputers are asked to write their application in Anonymous Third Person (A3P). They must remove all investigator names and gender pronouns from their research applications. They are also required to remove any language that identifies previously published research as their own. We have guidelines to help them conceal their identifies.
The A3P changes for applicants are mainly in the style, structure, and grammar used to describe their work and the research they propose to do with the requested equipment. The change means that applicants cannot re-submit the same application they wrote in past years, as their previous applications would likely not be in A3P format.
This study primarily uses NASA’s same applicant guidelines: STScI Hubble Space Telescope (HST).
8. What does this mean for the reviewers of the applications?
The anonymous review process requires some changes in the way reviewers evaluate the applications. We have guidelines to help reviewers assess anonymised applications.
The reviewers do not know the applicants’ identity and should resist the urge to guess who they are. They are encouraged to focus on the scientific merit of the application. The primary objective is to select the best science, not the best science teams.
First, the reviewers review and score the A3P applications that are assigned to them. Then, the review committee (made up of all reviewers for the grant round) collectively rank, make decisions and recommendations about the A3P applications during a committee meeting.
A ‘leveller’ is used during the review committee meeting. The leveller is an external and impartial individual who monitors the committee review meeting to ensure it stays focused on the science and not the science teams. If necessary, the leveller may interject and refocus the discussion.
This study primarily uses the same reviewer guidelines used by NASA: STScI Hubble Space Telescope (HST).
9. How does the process ensure that awards do not go to science teams without the right expertise?
The review committees may include an unblinded post-review round (as a fail-safe). This step involves revealing the applicants’ identities (only after the review) to cross-check the team’s veracity. The committee may disqualify an application based on a significant lack of expertise to carry out the proposed research project.
10. What happens to applications that are not sufficiently anonymised?
Anonymisation of the applications is mandatory for compliance, and applicants must follow the applicant guidelines.
To support applicants in making this change, some organisations have opted to offer an optional pre-submission compliance check to provide feedback and guidance to applicants on any issues or questions they have in preparing their anonymised application.
Once final applications are submitted, if an application is found to quite obviously and blatantly disregard the anonymising guidelines, the application will be deemed non-compliant and may be withdrawn from further consideration. Some organisations have opted to screen applications and contact applicants of non-compliant applications to make any changes to their application and re-submit before the reviewer’s assessment begins.
11. Does anonymisation prevent applicants from leveraging their expertise and reputations?
No. If applicants follow the applicant guidelines, they can still present their past work and expertise in the field and write an A3P-compliant application.
Applicants do not need to ‘water down’ or obscure their science, methods, or tools. It is simply their responsibility to write about them in the third-person, in a way that does not intentionally identify them.
12. How does the process deal with conflicts of interest?
In some respects, reviewer conflicts of interest with a given application are a bit simpler. If the reviewers do not know who the applicants are, how can they be conflicted?
As always, reviewers can (and should) identify any applications that they are listed on as team members or any other personal conflicts of interests that they need to identify. The organisations (and systems) ensure that reviewers are not assigned their own application for review.