Example Program Evaluation: Techgirls Competition
If you are evaluating your STEM gender equity program—or any program, for that matter—take a look at how the Tech Girls Movement Foundation evaluated the Techgirls Competition program.
The example applies the 5 steps of the National Evaluation Guide for STEM gender equity programs: Define, Plan, Design, Execute and Share.
Download the National Evaluation Guide
It contains practical advice, worksheets and templates to evaluate your program.
Techgirls Competition: A 12-week program changing the way girls perceive and engage in STEM
The Tech Girls Movement Foundation has a vision of a society in which girls confidently lead in STEM entrepreneurship and contribute to their community and the economy. It does this by championing Australian schoolgirls using hands-on learning to transform their future and encourage equity in the technology industry. The Tech Girls Movement Foundation aims to help girls get excited and connected with technology in a way that is meaningful and life-changing. One of their programs—the annual Search for the Next Tech Girl Superhero competition (now called Techgirls)—engages student teams and their mentors/coaches in designing and building an app and business plan to solve a local community problem.
The image below is a completed Evaluation Planning Tool worksheet for the STEM Women program evaluation (see Appendix A of National Evaluation Guide for a blank version of the worksheet). Use this example as a template for your program evaluation. Each section of the worksheet is explained in greater detail further down. You can also access the suite of evaluation reports from 2016.
Girls and women are underrepresented in STEM and entrepreneurship, in-part due to factors that influence girls’ participation in STEM education and careers. These include: lack of role models and mentors, lack of awareness of STEM career options, interest and attitudes towards STEM, confidence in their abilities and lack of opportunities to engage with STEM, among many others.
The program audiences are girls (students) between the ages of 7 to 17 years old. Other audiences include the coaches (teachers and parents/carers) and women industry mentors who provide mentorship and support to the student teams.
The Tech Girls Movement Foundation’s business model relies primarily on funding from corporate sponsors, government grants, and partnerships. Therefore, the evaluation audiences—those interested in knowing or applying the evaluation findings—include the above stakeholders. Parents/carers, the wider school community, the corporate community of mentors involved are also important evaluation audiences.
In the long-term (impact), the Tech Girls Movement Foundation has a vision for a society in which girls confidently lead in STEM entrepreneurship and contribute to their community and the economy. To achieve this, the Techgirls Competition aims to provide hands-on experiences intended to: improve girls’ coding knowledge, confidence in using technology and increase their interest in STEM as a future career choice (short-term outcomes). The aim is that this will influence and encourage girls to pursue STEM education, and ultimately STEM careers (medium-term outcome). In addition, the program also aims to provide an enriching experience for the mentors and coaches (short-term outcome).
The 12-week program/competition engages student teams in design thinking to build an app and business plan to solve a local community problem that they care about. The competition requires girls to form a team of 2 to 5 students in school or outside of school and enlist a coach to be their contact point for the program. The coach registers their team online, and Tech Girls matches them with a female industry mentor and encourages them to meet before the program starts.
First, teams brainstorm problems in their local community and decide on one to solve by doing a feasibility study. Second, the teams research how others have previously tried to solve the problem, and then they design a solution to the problem in the form of an app. Third, they document their work in a business plan and build a working prototype of the app. Finally, the teams pitch and demonstrate their idea, plans and prototype and submit everything for judging.
Some of the inputs for these activities include: Tech Girls staff, industry mentors and judges (human resources), registration platform and venues for showcase events (material resources), funding (financial resources), etc.
Some of the outputs from these activities include: showcase events, prototype apps built by students (tangible solutions to local problems), etc.
Since 2015, The Tech Girls Movement Foundation has evaluated the Techgirls Competition. The evaluation targets the three program audiences involved in the 12-week program:
- Schoolgirls aged 7-17 years old
- Coaches (usually teachers or parents/carers)
- Industry mentors
The evaluation has three main evaluation priorities:
- Student outcomes and impacts (examines four topic areas):
- Current STEM activities involved in at school or otherwise
- Current technical knowledge in relation to coding
- Confidence and interest in entrepreneurial learning
- Awareness of science and technology careers as an option
- Mentor/coaches outcomes and impacts (examines three topic areas):
- What they hoped to gain from participating in the competition
- What motivated them to join
- Would you participate again
- Lessons learned from mentors/coaches (examines three topic areas):
- What worked well and not so well
- Advice for mentors/coaches
- Advice for expanding the program
These topics helped shape the structure for the pre- and post-surveys (pretest-posttest). The surveys consisted of open (qualitative) and closed (quantitative) questions (∴ mixed methods) and were delivered via the Techgirls Competition Portal. The surveys use questions from existing questionnaires, such as the ‘STEM Career Interest Survey (STEM-CIS)’ developed by researchers in the field of STEM education. The pre- and post-surveys for the three cohorts are publicly available on the Tech Girls website (scroll down to the bottom of their page).
The three cohorts of participants were surveyed before they started the 12-week program, and again at the completion of the competition during submission of their entries, or soon after. The overall response rate was 40-60%. The analysis of pre versus post data is especially important given the time frame of the program.
The outcomes of each evaluation over the years feed into continual improvement of the program. The outcomes are also of value to current and potential future sponsors and partners. The reports and summary infographics present the changes in each of the cohorts and communicate the key findings of the program evaluations. They demonstrate not only what the program does well, but what can be improved, why and how. The reports are publicly published on the Tech Girls website and shared with stakeholders, funders and government.