What’s the verdict?

Feedback on the National Evaluation Guide for STEM gender equity programs

Many people who run STEM gender equity programs (or any programs) do not feel confident or equipped to evaluate them [1]. Earlier this year, we asked program leaders to tell us how they feel about evaluation. They used words such as: Trepidation. Daunted. Slightly panicked. Underqualified. Time-consuming. 

With such negative associations, it’s no wonder that program evaluation often falls by the wayside. For example, of 331 STEM gender equity programs in Australia in 2018, only three had publicly available evaluation findings [2]. Yet, the lack of evaluation means that we don’t know what is working, or how to improve what isn’t.  

Why we need to evaluate STEM gender equity programs 

STEM gender equity programs seek to dismantle barriers to attract and keep girls and women in STEM. The end goal: a diverse and gender-balanced STEM workforce to confront important world issues. With such significant outcomes at stake, we need to know if what we are doing is working.   

Evaluation of STEM gender equity programs is a subject of national interest. This is why evaluation is a key priority of the Australian Government’s Advancing Women in STEM strategy and the Women in STEM Decadal Plan. 

The National Evaluation Guide: A pilot 

In May this year, the Office of the Women in STEM Ambassador delivered on this key national priority. We produced an evaluation guide to make evaluation simple and easy for program leaders. The ‘Guide’ is a user-friendly, how-to resource that breaks evaluation down into a simple four-step process. 

The Office piloted the Guide to find out if it offered what program leaders need. We invited a group of recipients of Women in STEM and Entrepreneurship (WISE) grants to use the Guide and give us feedback. We extended the same invitation more broadly to anyone who accessed the Guide. A total of 41 people gave feedback through an online survey between May and July 2020*. 

Feedback on the Guide

The feedback on the Guide is mostly positive. Respondents said that the Guide is useful, practical, comprehensive, well-organised and easy to understand. The most valuable aspects are the templates, practical worksheets and real-life examples.  

A general concern is that the Guide is too long. Respondents suggested creating a shorter ‘quick-reference’ version to complement the longer version. They also suggested including a bank of recommended questions to choose from for their evaluations. 

One respondent suggested that the Guide should have a focus on systemic change:  

 “If the examples could be changed to be focused on systemic change projects this would be extremely valuable and would help push the STEM sector in the right direction […] The issue at the moment in the STEM sector (and more widely) is not only that initiatives aren’t effectively evaluated, but that the ‘projects’ themselves are unlikely to lead to long term change as they aren’t addressing the underlying causes of inequality – we need to encourage people to start in the right place AND evaluate their work.” 

A short summary report provides an overview of the feedback. 

Next steps 

We are taking this valuable feedback on board as we develop the second version of the Guide. The second version will be coupled with a shorter ‘quick-reference’ document. Both will be available at the end of 2020.  

From 2021, the Department of Industry, Science, Energy and Resources will recommend the Guide to WISE-funded projects as the preferred evaluation resource. We will also work with funders of STEM gender equity programs to promote the use of the Guide to evaluate funded programs. The long-term goal is to build a bank of consistent and comparable program evaluations to understand if what we are doing is working.