Example Program Evaluation: Techgirls Competition

Problem.

Girls and women are underrepresented in STEM and entrepreneurship, in-part due to factors that influence girls’ participation in STEM education and careers. These include: lack of role models and mentors, lack of awareness of STEM career options, interest and attitudes towards STEM, confidence in their abilities and lack of opportunities to engage with STEM, among many others.

Audiences.

The program audiences are girls (students) between the ages of 7 to 17 years old. Other audiences include the coaches (teachers and parents/carers) and women industry mentors who provide mentorship and support to the student teams.

The Tech Girls Movement Foundation’s business model relies primarily on funding from corporate sponsors, government grants, and partnerships. Therefore, the evaluation audiences—those interested in knowing or applying the evaluation findings—include the above stakeholders. Parents/carers, the wider school community, the corporate community of mentors involved are also important evaluation audiences.

Goals.

In the long-term (impact), the Tech Girls Movement Foundation has a vision for a society in which girls confidently lead in STEM entrepreneurship and contribute to their community and the economy. To achieve this, the Techgirls Competition aims to provide hands-on experiences intended to: improve girls’ coding knowledge, confidence in using technology and increase their interest in STEM as a future career choice (short-term outcomes). The aim is that this will influence and encourage girls to pursue STEM education, and ultimately STEM careers (medium-term outcome). In addition, the program also aims to provide an enriching experience for the mentors and coaches (short-term outcome).

Activities.

The 12-week program/competition engages student teams in design thinking to build an app and business plan to solve a local community problem that they care about. The competition requires girls to form a team of 2 to 5 students in school or outside of school and enlist a coach to be their contact point for the program. The coach registers their team online, and Tech Girls matches them with a female industry mentor and encourages them to meet before the program starts.

First, teams brainstorm problems in their local community and decide on one to solve by doing a feasibility study. Second, the teams research how others have previously tried to solve the problem, and then they design a solution to the problem in the form of an app. Third, they document their work in a business plan and build a working prototype of the app. Finally, the teams pitch and demonstrate their idea, plans and prototype and submit everything for judging.

Some of the inputs for these activities include: Tech Girls staff, industry mentors and judges (human resources), registration platform and venues for showcase events (material resources), funding (financial resources), etc.

Some of the outputs from these activities include: showcase events, prototype apps built by students (tangible solutions to local problems), etc.

Evaluation.

Since 2015, The Tech Girls Movement Foundation has evaluated the Techgirls Competition. The evaluation targets the three program audiences involved in the 12-week program:

• Schoolgirls aged 7-17 years old
• Coaches (usually teachers or parents/carers)
• Industry mentors

The evaluation has three main evaluation priorities:

1. Student outcomes and impacts (examines four topic areas):
   
   • Current STEM activities involved in at school or otherwise
   • Current technical knowledge in relation to coding
   • Confidence and interest in entrepreneurial learning
   • Awareness of science and technology careers as an option

2. Mentor/coaches outcomes and impacts (examines three topic areas):
   
   • What they hoped to gain from participating in the competition
   • What motivated them to join
   • Would you participate again

3. Lessons learned from mentors/coaches (examines three topic areas):
   • What worked well and not so well
   • Advice for mentors/coaches
   • Advice for expanding the program

These topics helped shape the structure for the pre- and post-surveys (pretest-posttest). The surveys consisted of open (qualitative) and closed (quantitative) questions (∴ mixed methods) and were delivered via the Techgirls Competition Portal. The surveys use questions from existing questionnaires, such as the ‘STEM Career Interest Survey (STEM-CIS)’ developed by researchers in the field of STEM education. The pre- and post-surveys for the three cohorts are publicly available on the Tech Girls website (scroll down to the bottom of their page).

The three cohorts of participants were surveyed before they started the 12-week program, and again at the completion of the competition during submission of their entries, or soon after.  The overall response rate was 40-60%. The analysis of pre versus post data is especially important given the time frame of the program.

The outcomes of each evaluation over the years feed into continual improvement of the program. The outcomes are also of value to current and potential future sponsors and partners. The reports and summary infographics present the changes in each of the cohorts and communicate the key findings of the program evaluations. They demonstrate not only what the program does well, but what can be improved, why and how. The reports are publicly published on the Tech Girls website and shared with stakeholders, funders and government.