Example Program Evaluation: STEM Women

Problem.

Women are underrepresented across all STEM sectors in Australia. For example, in Australian science-related stories, 33% of direct sources are women. At conferences, male speakers regularly dominate unless targets are explicitly set and adhered to. The underrepresentation of women in STEM spans media, conferences and events, boards and committees, awards nominations and workplaces.

Audiences.

STEM Women targets two key program audiences. The first is women working or studying in STEM or applying STEM skills. The second is organisations and individuals who are looking for a STEM expert for an upcoming opportunity either within their organisation (e.g. board position, mentoring) or a public-facing opportunity (e.g. media story, or speaking opportunities)

The evaluation audiences—those interested in knowing or applying the evaluation findings—include many stakeholders. For example, STEM Women was funded by the Australian Government through the Women in STEM and Entrepreneurship Program and overseen by an organising committee of key STEM organisations, all of which would be interested in understanding the impact of STEM Women. Further resources are required to continue to grow STEM Women, therefore some findings will be relevant to potential partners. The users of the site and the general STEM community will also be interested in the findings and proposed next steps.

Goals.

In the long-term (impact), STEM Women would like to see greater representation of women in STEM in the media, at conferences, at events and on boards. To reach that goal in the short-term, STEM Women would like to see 3000 profiles that represent women from across Australia and a broad range of STEM expertise. Over time, STEM Women would like to be a staple, well known and regularly used tool for the STEM community (medium-term). 

Activities.

To achieve these goals, the Australian Academy of Science worked with the STEM community (input) to build STEM Women, an online database of women in STEM. Women in STEM provide their time (input) to create and update a profile (output). Seekers then search the profiles based on their requirement (input) and shortlist and message relevant profiles with opportunities (output).

STEM Women project was developed within the Academy overseen by a dedicated part-time Project Officer, developed by the ICT team, and promoted by the communication team. Overall, development, launch and initial promotion of the site cost $200,000 and took eleven months to build a functional product (inputs).

Evaluation.

Evaluation of STEM Women was undertaken from August 2019 to August 2020. The purpose of the evaluation was to determine the impact of the platform on increasing women’s participation in STEM and how the site is being used and by whom. As STEM Women transitioned from its initial development and launch phases (2019-2020), the aim was also to incorporate the evaluation findings to further shape the development of the platform beyond 2020. The STEM Women staff identified three evaluation priority areas—discovery, connection, and awareness. They then developed key questions to explore under these areas:

• Discovery: Does STEM Women showcase a diverse range of women? Is it an accessible and usable platform?
• Connection: Does STEM Women help women in STEM to share their expertise, knowledge and experiences with the community?
• Awareness: Is the STEM community aware of STEM Women and its functions?

These high-level key questions shaped the design of the data collection tools and analysis, which included qualitative and quantitative surveys, case studies and data analysis. The evaluation involved the two program audiences: women in STEM who created profiles (called ‘members’) and users of the site searching for women in STEM (called ‘seekers’).

• Surveys.
  • Members (download the survey): A 20-question survey with Likert scales and open-ended questions (posttest, mixed methods), aimed to understand usability, access to opportunities, and impacts and perceptions of STEM Women.
  • Seekers (download the survey): An 18-question survey with Likert scales and open-ended questions (posttest, mixed methods), aimed to understand usability, ease of offering opportunities, outcomes of connections and perceptions of STEM Women.
• Case Studies.
  • In January and August 2020, a selection of seekers and members were contacted via email to provide a description and comment capturing their experiences with the STEM Women site (time series, qualitative).
• Analytics.
  • De-identified user data was exported from the STEM Women servers, along with Google Analytics and Campaign Monitor (email distribution platform) data for the period of 1 August 2019 – 1 August 2020 (time series, quantitative).

The report STEM Women: One Year of Impacts and Learnings, communicates the key findings of the STEM Women program in its initial development and launch phases (2019-2020). It was publicly published on the Australian Academy of Science website and shared with stakeholders, funders and government.