Example Program Evaluation: Science in Australia Gender Equity (SAGE) Pilot

Problem.

Women are poorly represented in senior roles in HER institutions. Studies show that women academics and researchers in STEMM are squeezed out of science careers by structural barriers and organisational culture. These barriers include: limited access to career development opportunities, lack of flexible work arrangements, sexual harassment, stereotypes and flawed recruitment, progression and recognition practices, among others.

Audiences.

The program audiences are STEMM HER institutions who apply to gain SAGE accreditation for Athena Swan Institutional Bronze Award by undertaking a structured self-assessment and reflective examination of institutional policies, practices and data guided by the 10 key principles of the Charter.

The evaluation audiences—those interested in knowing or applying the evaluation findings—include many stakeholders: SAGE governing bodies and its management, government, Australian STEMM sectors as well as academics, researchers and students.

Goals.

In the first instance (in 2015-2016), SAGE will pilot the adapted UK Athena SWAN Charter in Australia with participating institutions and raise awareness of gender inequity in the STEMM HER sector (short-term outcome). From 2017, the learning and recommendations from the pilot will be implemented and the pilot will be expanded to all publicly-funded STEMM HER institutions in Australia (medium-term outcomes). From 2020, the SAGE’s goal is to reach the entire STEMM HER sector to drive systemic, structural and cultural organisational changes. The aim is to improve the participation, retention and success of women and other gender minorities and support gender diversity in STEMM HER in Australia (long-term impact).

Activities.

STEMM HER organisations who participate in SAGE must adopt the 10 key principles of the Athena SWAN Charter, undertake a structured evidence-based approach and devise action plans that embed these principles within their policies, practices, and culture to achieve bronze accreditation. In doing so, they must establish an internal cross-disciplinary self-assessment team (SAT) to perform an analysis of the gender equity and diversity to understand the root causes of inequities within their institute and devise an actionable four-year plan. SATs then submit this information to be assessed by a broad range of expert peer reviewers, in a process managed by SAGE and overseen by independent moderators. SAGE then accredits institutions via the Athena SWAN Award to successful institutions.

The inputs for these activities include: guidance materials and resources, specialist workshops and training program delivered by SAGE and regional network meetings enabled by funding from the Australian Government and subscription fees paid by participating institutions, and an annual symposium, SAGE team and management board, an expert advisory group and the SAGE peer review panels who assess applications.

The outputs from the activities include: SAGE subscribers (HER institutions who are applying for accreditation) applications for accreditation and Athena SWAN Bronze awards to institutions who successfully complete the accreditation process.

Evaluation.

The Australian Council for Educational Research (ACER) was engaged to undertake the evaluation.

The three evaluation priorities include:

1. Evaluate the appropriateness and effectiveness of the pilot implementation undertaken to from 2015 to 2017 (Design; Efficacy)
2. Evaluate the Athena SWAN framework and accreditation processes to the Australian context for use in Australian tertiary education and research institutional environments (Design).
3. Develop and establish a data framework to inform future assessment and reporting on impact and benefits realisation from the implementation of the Athena SWAN framework in the Australian context (Outcomes and Impacts; Lessons Learnt).

The first priority sought to answer the following key evaluation questions: What went well? What didn’t go well? What can to be improved and how? Key questions and indicators were identified for each of the evaluation priorities (see Evaluation Planning Tool on pg. _ above).

Between April and October 2017, ACER conducted widespread consultations involving more than 140 people, including senior leaders from SAGE member institutions, peak bodies, government agencies and a range of external stakeholders. This involved individual interviews, regional workshops and on-site focus groups (posttest, qualitative).

A precis of ACER’s report Evaluating the introduction of Athena SWAN into Australia: putting gender on your agenda, communicates the key findings of ACER’s evaluation of the SAGE Pilot. It was publicly published on the SAGE website and shared with stakeholders, funders and government.