Evaluation of Trio Sci Cymru: executive summary

This report presents the key findings from the final evaluation of the Trio Sci Cymru (TSC) Operation, undertaken by Miller Research in 2024. 

TSC is a circa £6.2m Operation supported by the European Social Fund (ESF) 2014-2020 Programme for West Wales and the Valleys. The Operation is funded specifically under Priority Axis 3: Youth Employment and Attainment, Specific Objective 3, which was designed: “to increase the take up and attainment in Science, Technology, Engineering and Maths (STEM) subjects^{[footnote 1]} amongst 11–19-year-olds.” 

The overarching aim of TSC was to increase take up of and attainment in STEM subjects amongst 11-to-19-year-olds in Wales, through a range of interventions that enrich but do not duplicate mainstream education and other extra-curricular activities. 

The operation was managed by the Welsh Government (the Lead Beneficiary), who were responsible for coordinating activity with Joint Beneficiaries and reporting progress to the Welsh European Funding Office (WEFO). TSC was delivered through a partnership between four Joint Beneficiaries which included Aberystwyth University, Bangor University, Cardiff University and Swansea University. The Institute of Physics (IOP) were also named as a Joint Beneficiary on the initial bid, however the Institute of Physics withdrew from the operation prior to delivery due to difficulties working within the funding constraints set.

The operation delivery was initially due to commence in October 2018, however operation delivery was delayed until early 2019, due to delays in receiving the initial funding confirmation from WEFO. The operation was initially due to come to a close in September 2021 but was extended until December 2022.  This adjustment accounted for the disruption posed by the COVID-19 pandemic, which prevented in-person engagement in schools.

Delivery

Activities were delivered through a mix of in-person and virtual workshops. 

TSC engaged a total of 6,095 learners, despite challenges posed by COVID-19 related disruption and the withdrawal of the Institute of Physics. However, due to challenges with data capture and enrolment forms, only 3,808 participants engaged were able to be verified as eligible under WEFO compliance requirements which included the provision of the participant’s name, date of birth and full address. Bangor University’s ethics board disagreed with the sharing of full address data with WEFO, and as such only 63 participants (5.4%) of Bangor University’s 1,173 participants could be claimed. This contributed to the overall reduced number of participants who could be claimed at the operation level, which fell below TSC’s revised target of 4,500 participants engaged. One Welsh Government stakeholder noted that the collection of post code data should not have been an unrealistic requirement and felt that this looks like an administration problem on behalf of Bangor University.

One of the key aims of TSC was to inspire more girls to pursue STEM education and careers. While Joint Beneficiaries took a general approach to learner engagement, girls constituted 53% of the total participants engaged by TSC. In addition, some Joint Beneficiaries made concerted efforts to include female staff in their STEM enrichment delivery to highlight STEM career pathways and provide learners with role models. Joint Beneficiaries and stakeholders welcomed this inclusion and hoped it would have an effect on the patterns of engagement of girls in STEM. 

Operational challenges

The operation faced significant delays in receiving funding approval from WEFO early in the process leading to delays in the commencement of delivery from 2018 to early 2019. Delivery was then further hindered by a Welsh Government moratorium on recruitment during the COVID-19 pandemic, leading to under resourcing, which substantially limited the Welsh Government's capacity as Lead Beneficiary to effectively coordinate the operation. This limited opportunities to facilitate more effective collaboration and sharing of best practise among Joint Beneficiaries. Whilst Joint Beneficiaries did express that they had taken on opportunities to meet and collaborate as a consortium that this was felt to be hindered by the limited ability of Welsh Government to engage and provide timely responses to their queries or concerns.

COVID-19 posed substantial challenges to the delivery of TSC and more broadly learner engagement with schools and enrichment activities. One key disrupted element of delivery was the ability to deliver enrichment activities in-person due to school closures and restrictions. 

Joint Beneficiaries highlighted a need to adapt their delivery models ‘overnight’ to the COVID-19 pandemic which significantly disrupted the delivery of the operation. Delivery had to be pivoted towards virtual resources and digital modes of delivery at a time when learners and teachers were adapting to new ways of teaching and learning. Prolonged school closures and restrictions hindered in-person relationship-building and the ability to embed STEM enrichment through TSC within school timetables, as priorities shifted to delivering the core curriculum. Despite these challenges, Joint Beneficiaries adapted by pivoting to online delivery models, offering virtual workshops and accessible resources. This shift enabled a broader reach beyond the original target group but introduced complexities in evidencing impact. 

Adaptation varied among Joint Beneficiaries, with some leveraging pre-existing digital capabilities, while others, such as Swansea University, trialled innovative solutions like home experiment kits to maintain engagement. However, the fully online approach struggled to replicate the hands-on nature of STEM learning and learners expressed a strong preference for in-person delivery. A positive legacy of this transition is the digital STEM resources created through the project, which one Joint Beneficiary is seeking to update align with the new Curriculum for Wales and continue to support STEM enrichment delivery. 

Despite the move to online delivery, the lack of in-person engagement negatively affected the planned delivery of TSC as a key element of the value proposition that TSC offered was the ability to deliver more hands-on, practical opportunities to engage with STEM subjects. Additionally, in some cases online resources were also open to non-TSC students so the any potential observed difference between the outcomes of TSC and non-TSC learners would have been reduced.

Joint Beneficiaries and stakeholders noted that practical and hands-on learning experiences typically reduce following the transition to secondary school, which is associated with a reduction in interest and engagement. While the shift to virtual modes of engagement enabled greater access to resources shared initially through open access platforms, including for non-TSC schools, it posed challenges for the evaluation of engagement and impact for the Joint Beneficiaries in a way which was WEFO compliant. However, Joint Beneficiaries did welcome the flexibility shown by WEFO in its approach to the how delivery of the operation could be modified in light of the COVID-19 pandemic.

The ability to effectively evidence output progress against targets  was negatively affected by data collection and retention challenges, including the destruction of key forms during the flooding of Welsh Government Offices, as well as some confusion from Bangor University around the level of post code data needed to evidence outputs. This led to a number of outputs which could not be effectively evidenced due to non-compliant data collection. One teacher also noted some cases in which parents of learners did not want their data shared or disagreed with some of the data collection categories which meant that their participation could not be evidenced toward the formal operation outputs.

Overall, in spite of the operational challenges faced, the TSC operation delivered substantially above initial participant targets, however not all of these participants could be claimed due to the challenge outlined above. TSC successfully engaged a substantial number of learners across Wales, contributing to improved STEM awareness and engagement. While the operation faced notable challenges, particularly during the pandemic, Joint Beneficiaries were able to adapt their provision to deliver virtual learning opportunities, although it was noted by both Teachers and Learners that there was a strong preference for in-person delivery for future STEM enrichment programmes.

Feedback from Joint Beneficiaries and wider stakeholders indicated that it was felt that the role of the Lead Beneficiary was substantially limited by the insufficient resources allocated internally to the operation due to a Welsh Government moratorium on recruitment during the COVID-19 pandemic and staff turnover. This meant that the Lead Beneficiary role was not fully resourced as set out in the initial WEFO application, which set out an expected six FTE staff to be employed to lead on the operation and manage the relationships with Joint Beneficiaries. Whilst Joint Beneficiaries did hold some joint meetings as a consortium to check in with each other and share best practise, Joint Beneficiaries felt that the lack of lead beneficiary capacity affected communication with the Welsh Government. This included the ability to receive a timely response to queries raised, and led to Joint Beneficiaries making decisions regarding their delivery largely independently based on their experience of what had worked previously for similar operations. While one Joint Beneficiary highlighted attempts to work collaboratively as part of a wider consortium of Joint Beneficiaries, another Joint Beneficiary did not feel that the collaboration was sufficient and described often more ‘siloed’ ways of working.

Differences in models of delivery

Joint Beneficiaries welcomed the ability to develop their own delivery work streams based on their institution’s research expertise, strengths and prior experience. Most Joint Beneficiaries had experience delivering STEM enrichment previously and were able to build on these established approaches and best ways of working in order to deliver an approach that would work with their schools in the individual contexts in which they operate. This led to considerable variation among Joint Beneficiaries in terms of the mode of delivery, whether learners were engaged at their schools or brought onto university campus, as well as whether delivery would be delivered through general classroom approaches or smaller group interventions. However, one Joint Beneficiary expressed concern over the ability to evaluate the relative merits of the different approaches take and expressed a desire to have seen more steer from the Welsh Government on approaches taken. 

Additionally, there was variation in the topics covered, with some Joint Beneficiaries only being able to offer activities on select science subjects and having to pair up with another Joint Beneficiary to offer options across the sciences. Where it was provided, teachers welcomed the ability to select from a range of topics, however, would have liked to have been able to select activities from all three science subjects. Although one stakeholder noted some challenges with Joint Beneficiaries being able to provide enrichment across Biology, Chemistry and Physics given the limitations on their institutional expertise and capacity.

There was some disagreement between Joint Beneficiaries as whether TSC should have been delivered STEM enrichment opportunities in addition to the curriculum, or whether the topics covered should aim to align more with core curriculum content. It was noted by Joint Beneficiaries and teachers that while teachers valued the ability of TSC to deliver activities that were more ‘novel’ and ‘hands-on’, they also appreciated and were more likely to take up delivery which aligned with the curriculum.

COVID-19 pandemic adaptations

Joint Beneficiaries highlighted a need to adapt their delivery models ‘overnight’ to the COVID-19 pandemic which significantly disrupted the delivery of the operation, particularly in engaging schools and promoting STEM uptake among learners. Prolonged school closures and restrictions hindered in-person relationship-building and the ability to embed STEM enrichment through TSC within school timetables, as priorities shifted to delivering the core curriculum. Despite these challenges, Joint Beneficiaries adapted by pivoting to online delivery models, offering virtual workshops and accessible resources. This shift enabled a broader reach beyond the original target group but introduced complexities in evidencing impact. 

Adaptation varied among Joint Beneficiaries, with some leveraging pre-existing digital capabilities, while others, such as Swansea University, trialled innovative solutions like home experiment kits to maintain engagement. However, the fully online approach struggled to replicate the hands-on nature of STEM learning and learners expressed a strong preference for in-person delivery. A positive legacy of this transition is the digital STEM resources created through the project, which one Joint Beneficiary is seeking to update align with the new Curriculum for Wales and continue to support STEM enrichment delivery. 

Impact of the operation

TSC has demonstrated some successes in some of its intended outcomes, particularly in increasing short-term engagement with STEM subjects and fostering interest among previously disengaged learners. Teachers interviewed highlighted an increased interest in and engagement with STEM among learners in the short term through delivery of ‘novel’ practical, hands-on activities, which offered engagement opportunities outside of core curriculum delivery. However, teachers were uncertain about the sustainability of such impacts in the long term and their translation to increased take up of STEM qualifications and attainment in STEM qualifications. 

Evidence from GCSE attainment data further indicates that it may have had an impact, with TSC schools showing a statistically significant increase in A* grades compared to non-TSC schools between 2019 and 2023. However, this improvement was not consistently reflected across other higher grade categories (e.g., A and B grades), where TSC schools saw less favourable outcomes than their non-TSC counterparts. 

Teachers noted however the challenges in observing changes in triple science take up and achievement as a result of TSC, owing to the lack of baseline data collected and disruption caused by the COVID-19 pandemic which was viewed as having a substantially negative impact on learners and their overall engagement with school and attainment. 

There is some evidence to indicate that TSC’s efforts to incorporate female role models and challenge gender stereotypes in STEM were appreciated, with data demonstrating some improvement in the attainment and uptake of STEM A-Levels by females in TSC schools from 2022 onwards. 

Despite its achievements, TSC faced significant challenges in achieving its broader objectives. The operation struggled to demonstrate a measurable impact on sustained participation in STEM subjects or increased uptake of triple science GCSEs, partly due to the disruption caused by the COVID-19 pandemic, which limited both learner engagement and partly due to the lack of baseline monitoring data being captured^{[footnote 2]} consistently across interventions monitoring data not supporting assessment of TSC’s impact.

Teachers expressed uncertainty regarding the operation's long-term influence on outcomes like STEM achievement and progression, and qualitative feedback suggested mixed results in building Science Capital among learners. Furthermore, while some schools benefited from strengthened relationships with TSC Joint Beneficiaries and access to resources, the operation’s capacity to deliver widespread STEM enrichment across Wales was hindered by logistical challenges, such as the early loss of a key Joint Beneficiary and the impact of pandemic-related restrictions. 

Due to limitations in capturing systematic data on the impact that TSC interventions had on learners, it was determined that there is insufficient data to undertake the initially planned comparison of cost effectiveness between different approaches adopted by Joint Beneficiaries. The overall value for money assessment is also limited beyond a calculation of the cost per pupil for TSC overall. This means that no firm conclusions can be reached regarding the  value for money of the TSC operation compared to other STEM enrichment programmes or the relative cost effectiveness of the different models Joint Beneficiaries undertook as part of the operation.

Feedback from Joint Beneficiaries and stakeholders highlighted that the Lead Beneficiary’s role was constrained by insufficient internal resources. The Welsh Government’s moratorium on recruitment during the COVID-19 pandemic, coupled with staff turnover, meant the team was smaller than originally planned. This limited capacity affected communication with Joint Beneficiaries, leading them to make independent delivery decisions based on prior experience. While some Joint Beneficiaries collaborated within a consortium, others felt that engagement remained fragmented, impacting the overall cohesion of the operation.

Joint Beneficiaries valued the flexibility to tailor their delivery to their institutions’ expertise and past experience. This resulted in diverse approaches, varying in terms of on-campus versus in-school engagement, group sizes, and teaching methods. While this adaptability was welcomed, one Joint Beneficiary raised concerns about the challenge of evaluating the relative effectiveness of different approaches and suggested that clearer guidance from the Welsh Government would have been beneficial.

The scope of topics also varied, with some Joint Beneficiaries limited in their subject coverage. Teachers appreciated the ability to select from a range of science topics but expressed a preference for a more comprehensive offering across Biology, Chemistry, and Physics. Some Joint Beneficiaries faced challenges in covering all three subjects due to institutional capacity constraints.

There was also debate over whether the programme should align more closely with the curriculum or focus solely on enrichment. While teachers valued the novel, hands-on approach of the programme, they were more likely to engage with content that complemented core curriculum topics.

The pandemic significantly disrupted the operation, forcing Joint Beneficiaries to rapidly shift to online delivery models. Schools’ priorities shifted towards core curriculum delivery, making it difficult to embed STEM enrichment. Virtual workshops and online resources expanded the programme’s reach but introduced challenges in measuring engagement and impact. Adaptation varied across institutions, with some, like Swansea University, trialling home experiment kits to maintain engagement. However, learners expressed a strong preference for in-person activities, highlighting the limitations of digital STEM learning. A positive legacy of this shift is the development of digital STEM resources, which one Joint Beneficiary aims to update for the new Curriculum for Wales.

The programme succeeded in boosting short-term STEM engagement, particularly among previously disengaged learners. Teachers noted that hands-on activities helped make STEM subjects more engaging, fostering discussions and strengthening home-school links. GCSE attainment data suggests that TSC schools saw a statistically significant increase in A* grades compared to non-TSC schools between 2019 and 2023. However, this improvement was not mirrored across other high-grade categories, where non-TSC schools often performed better.

Efforts to promote gender diversity in STEM were well received, with some evidence indicating an increase in STEM A-Level uptake by female learners in TSC schools from 2022 onwards. Despite these achievements, the programme struggled to demonstrate a long-term impact on sustained STEM participation or the uptake of triple science GCSEs. The pandemic limited learner engagement, and inconsistent baseline data collection hindered a comprehensive assessment of impact.

Teachers expressed uncertainty about the programme’s lasting influence on STEM achievement and career pathways. While some schools benefited from stronger ties with Joint Beneficiaries and access to additional resources, the programme faced logistical challenges, including the early withdrawal of a key Joint Beneficiary and pandemic-related restrictions.

Due to data limitations, the planned comparison of cost-effectiveness between different delivery models was not possible. As a result, no definitive conclusions could be drawn regarding the overall value for money of the operation or the relative effectiveness of the different approaches adopted.

Overall, while TSC has achieved some important short-term successes, its ability to deliver sustained and systemic changes in STEM participation, attainment, and gender balance remains uncertain.

Recommendations for similar operations in Wales

Recommendation 1

Based on feedback from Joint Beneficiaries, it was felt that the lack of collection of baseline data prior to interventions with learners as well as longitudinal data collection of learner outcomes made it more challenging to evaluate the impact that the TSC operation had on learners as a whole, as well as the ability to assess the relative merits of the different approaches employed by Joint Beneficiaries to engage learners with STEM. It is recommended that future initiatives build in a standardised plan for data collection on learner progress prior to the start of the operation to enable a more robust assessment of the impact of TSC on the learner’s baseline.

For example, this could include tracking a sample of learners throughout their learning journey over a number of years, with surveys being conducted periodically to assess factors relating to STEM Capital. Additionally, a baseline survey could have been used with all or a sample of participants at the start of their engagement with TSC, again collecting data relating to STEM Capital. Such data can provide baseline information on learners’ interest in STEM subjects, their understanding of subjects’ value to or role in their own lives, career aspirations, and general awareness of the links between STEM subjects and career opportunities. The use of anonymised data can also capture the relative impact of external factors, such as the quality of STEM teaching in schools, the availability of STEM options, and the impact of wider social and economic trends on the appreciation of STEM.

Recommendation 2

Feedback from the Lead Beneficiary, Joint Beneficiaries, and WEFO indicated a shared understanding that the Lead Beneficiary role was consistently under resourced throughout the operation due to a Welsh Government moratorium on recruitment and staff turnover. It is recommended going forward that similar initiatives ensure that there is a Lead Beneficiary which is sufficiently resourced to enable a more rapid mobilisation of Joint Beneficiaries, providing sufficient steer for delivery and facilitation of opportunities for Joint Beneficiaries to collaborate and share best practise.

Recommendation 3

Based on feedback from learners and teachers it was felt that years 7-9 was generally felt to be the right age group to target to avoid the drop off in learners engagement with STEM between primary and secondary school. It was felt to be important to engage learners prior to years 10-11 where learners would be making key decisions about their subject choices going forward, and teachers would struggle to timetable wider STEM enrichment activities alongside core curriculum delivery in years 10-11 where learners would be working towards their GCSEs. Wider stakeholders, however, did note the importance of complementing STEM enrichment at years 7-9 with wider engagement throughout the learner pathway, noting how early young people begin to form perceptions around whether certain subjects and jobs are for them, as well as noting the challenges of keeping learners engaged with STEM through the year 6-7 transition. It is therefore recommended that future initiatives continue to target this age group but also seek to tie in with wider STEM enrichment initiatives which target STEM engagement earlier in the learner pathways to ensure continuity of engagement.

Recommendations for Joint Beneficiaries

Recommendation 4

Teachers and learners expressed an appreciation of more practical ‘hands-on’ activities through TSC. Teachers noted that they were often unable to deliver similar activities due predominantly to lack of resources and specialised equipment but also due to curriculum limitations and having insufficient staffing to manage more involved activities. It was felt that a more hands-on approach to delivery was welcomed by all learners, but particularly learners who had previously been disengaged by more traditional STEM lessons by enabling access to STEM though more ‘novel’ experiences and using accessible entry points to STEM topics which learners would be familiar with. Therefore, it is recommended that future STEM enrichment initiatives continue to engage learners in STEM through more hands-on activities, as well as facilitating access to STEM engagement through novel experiences which schools often struggle to facilitate due to a lack of resource and specialised equipment.

Recommendation 5

Teachers expressed positive views on their relationships with Joint Beneficiaries. They welcomed the ability of Joint Beneficiaries to build positive working relationships with their schools as well as having a firm understanding of the context in which schools are operating and their individual needs. During the COVID-19 pandemic in particular teachers felt that the Joint Beneficiaries were responsive to the needs of teachers and were able to adapt their provision to address the particular barriers schools were facing to engage with the operation.  Rural schools also welcomed the extra effort taken by Joint beneficiaries to enable school visits. Therefore, this evaluation recommends that STEM enrichment initiatives build on this success and be developed in a way which is responsive to the needs of schools and teachers, by for example offering a greater menu of topic options or offering more tailored delivery options to schools depending on their needs.

Recommendation 6

While TSC was predominantly pitched as an enrichment operation to offer novel hands-on opportunity above and beyond core curriculum delivery, teachers expressed an interest in future STEM enrichment operations having a greater alignment with the Curriculum for Wales. They recommended that this be done through developing a clearer careers message through delivery, and enabling linkages between STEM subjects and wider subjects on the curriculum, such as Physical Education, the Arts, as well as the Digital Competency framework through greater embedding of tech into delivery. There were some positive examples of a more cross disciplinary approach enabled through Joint Beneficiary delivery, however it was felt that these could be expanded further. 

[1] For the purpose of our analysis STEM subjects included: Additional Science, Biological Sciences, Chemistry, Design and Technology, Information and Communication Technology, Mathematics, Other Science, Physics and Single Science.

[2] Due to delays in appointing an evaluator until after TSC was delivered.

Report author: Research and Evaluation Unit

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:

Research and Evaluation Unit
Welsh Government
Cathays Park
Cardiff
CF10 3NQ

Email: Research.Evaluation@gov.wales 

Social research number: 44/2026
Digital ISBN: 978-1-83745-095-4