Final evaluation of Sêr Cymru II: summary

This report provides the final evaluation of the European Regional Development Fund (ERDF) funded components of the Sêr Cymru II programme, which ran from December 2015 to June 2023 ^{[footnote 1]}. The programme aimed to increase research capacity linked to Science, Technology, Engineering, Mathematics and Medicine (STEMM) in Wales’ Higher Education Institutions (HEIs) and build a stronger science base in Wales.  It did this by attracting world class talent and providing targeted support to help HEIs build research capacity.   The programme funded a number of different awards targeted at researchers at different stages of their career, including:

• 30 Research Fellows: aimed at candidates 3-5 years post PhD
• 4 Precision Medicine Fellows (PMFs): similar to Research Fellows but aimed specifically at research related to precision medicine. 
• 10 Rising Stars: prestigious and competitive positions, designed to attract the very best ‘rising stars’ of academic research.
• 3 Recapturing Talent fellowships: these provided support for stellar researchers returning to work following a career break
• 5 Research Chair packages: these aimed to attract the very best senior academic researchers to Wales and establish new research groups

In September 2019, Welsh Government was successful in securing additional funding to expand Sêr Cymru II. These new funding opportunities included an additional Research Chair at Aberystwyth University, five Industrial Fellowships which aimed to strengthen relationships between Welsh academic researchers and industrial partners ^{[footnote 2]},19 Accelerator Awards focused on building research capacity in strategic areas (e.g. research staff or capital equipment) or building partnerships with other universities, and additional funding to support Wales’ three National Research Networks (NRNs), originally funded through Sêr Cymru (Life Sciences Network, Low Carbon Energy and Environment Network, and the Advanced Engineering and Materials Network).

The aims of this evaluation were to:

• review the strategic fit of the programme and the need for future programmes that build on and enhance the outcomes of Sêr Cymru II
• assess whether Sêr Cymru II achieved the ERDF indicators, aims and objectives set out in the business plan
• assess the effectiveness of management, delivery and monitoring processes
• assess the success of Sêr Cymru II in meeting the objectives set for WEFO’s Cross Cutting Themes (CCTs)
• assess the impacts of the programme on research quality and capacity in Wales, including counterfactual analysis
• provide recommendations for future programmes

A range of research methods were used including document reviews, analysis of monitoring data and progress reports, two surveys of grant recipients (a destination survey undertaken by Welsh Government and an online survey undertaken by Hatch), semi-structured consultations with 30 stakeholders and 32 grant recipients, and a counterfactual analysis of the impact of Sêr Cymru II on research income, research publications and citations.

Need for Sêr Cymru programme

The evaluation finds that the challenges that the Sêr Cymru II was designed to address (i.e. the proportionally low share of UK research income and research staff in Wales HEIs) still persists. The latest data shows Wales still only accounts for 3.6% of the UK’s STEMM research income and 3.9% of STEMM research staff; both below the target set by Welsh Government (5%).

Although Sêr Cymru II has not closed the overall gap in terms of research funding, this was never a realistic (or expected) outcome of the programme given the scale of investment.  When the analysis is more focused on the subject areas that have received support through the programme, there is evidence that this has helped them to grow their research income at a faster rate than comparator institutions and increase their share of UK research income ^{[footnote 3]}.

However, there is significant variation in the amount of research income secured across the programme, with the strong overall performance driven by a small number of awards. Given that less funding is likely to be available, it suggests that future rounds of the programme will need to be more selective than was the case for Sêr Cymru II and focus on a smaller number of investments which offer greatest potential to generate social and economic impact in Wales.

Strategic fit of Sêr Cymru II

The policy context in Wales and the UK continues to be highly supportive of the aims of Sêr Cymru II, including specific references to the programme in Welsh Government’s most recent Innovation Strategy. It is also consistent with the UK Government’s policy to increase investment in Research Development and Innovation (RD&I) outside London and the South East. 

The recent policies stress the importance of public investment in RD&I being more strategic, commercially focused and mission-orientated. If Wales is to remain competitive for UK funding opportunities, investment will therefore need to be focused on areas where Wales is internationally competitive, where it can contribute to national priorities, and where there is clear potential for economic and social benefits. Although there was some consideration of these factors for Sêr Cymru II, the programme funded a wide range of awards, many of which were not clearly aligned with existing research groups or building on recognised strengths. This also suggests future rounds of Sêr Cymru will need to be more selective and strategic than was the case for Sêr Cymru II.

Performance against targets in the business plan

This evaluation shows that Sêr Cymru II has successfully met almost all of the targets in the business plan, and in many cases has significantly overachieved. This includes the target for research income (£78.8m^{[footnote 4]} compared to a target of £45m) and the number of new researchers in supported institutions (179 compared to a target of 102), which relate to the main objectives of the operation. 

The business plan did not set specific targets for all of the different types of awards, but the monitoring data shows Research Chairs have performed particularly well and have accounted for the majority of the outputs achieved for a number of indicators (driven by the strong performance of three Chairs), including research income and partners cooperating in a research project. Research Fellows also delivered a very large number of research publications which exceeded the expectations in the business plan ^{[footnote 5]} (although it could be argued these expectations lacked ambition). 

The monitoring data suggests the SCII+ awards have also performed well despite the challenges imposed by the Covid pandemic. There is some uncertainty about the amount of research income and co-operations with partners secured by these awards, as a large number were found to be ineligible or lacking the evidence required by WEFO and so were not claimed. However, in many cases this appears to be for administrative reasons and there is good evidence to suggest that the outputs claimed with WEFO understate the achievements of these awards.

Impacts of Sêr Cymru II

The evaluation has assessed the impacts of Sêr Cymru II using the methodology recommended in the inception evaluation.  Different methods were used to assess different outcomes. This included the following.

• A benchmarking exercise was used to assess impacts of the original Sêr Cymru II awards on research income and research staff. This compared the change in the number of research staff/research income in those subject areas that have received support through Sêr Cymru II with the change in the same subject area at similar institutions ^{[footnote 6]}. Similar institutions were identified based on their ranking in the Research Excellence Framework (REF) and baseline research income for the relevant subject area. This is not a robust counterfactual approach but was considered to be the most feasible with the data available.  The findings were then triangulated with the findings from other research strands including analysis of monitoring data and interviews with grant recipients and stakeholders. 
• Counterfactual impact analysis was used to assess the impacts of Sêr Cymru II on the quantity, breadth and quality of research undertaken by fellows ^{[footnote 7]}. A control group of similar researchers was identified based on their characteristics (including their gender, experience, research interests and volume of research publications prior to the start of Sêr Cymru II). The performance of Sêr Cymru II fellows against a number of outcome variables was then compared with the control group over time. This measured the difference between the averages for two time periods; the pre-treatment average for the period 2012-2015 and the post-treatment average for 2016-2022.  If the difference between the pre and post-treatment average is higher for the treatment group than the control group, and the difference is statistically significant, it can be assumed that the difference can be attributed to Sêr Cymru II.
• Theory-based approaches were used to assess impacts on the personal and career development of researchers and all of the impacts of SCII+ awards. This drew upon a range of information sources including monitoring data, surveys and interviews with grant recipients.  These were used to assess whether the outcomes expected in the theory of change have come about, and the extent to which these can be attributed to Sêr Cymru II.

The evaluation finds that Sêr Cymru II has had a number of positive impacts.

• Research income in subject areas that received support through Sêr Cymru II (e.g.  biosciences at Cardiff University, physics at Swansea University and engineering at Bangor University) has grown at a faster rate than the same subject areas in similar institutions and the UK average.  Although there are some limitations to this analysis, it suggests Sêr Cymru II has had an additional effect on research income which would not have occurred in the absence of the programme. This is reinforced by consultation evidence, particularly Research Chairs and stakeholders from the host institutions, all of whom said the increase in research income would not have been possible without Sêr Cymru II funding. 
• Sêr Cymru II had a significant, positive additional impact on the number of research publications by supported fellows ^{[footnote 8]}. The average number of publications per annum by Sêr Cymru II fellows increased from 1.5 before the programme to 2.8 after the programme, representing an increase of 1.3. This was significantly higher than the increase for the control group (+0.1).  This suggests that the additional impact of Sêr Cymru II was 1.2 publications per researcher per annum (1.3-0.1).
• There is some evidence the programme increased the breadth of subject areas covered by fellows' research (indicating increased interdisciplinary working). This was assessed by measuring the average number of subject areas in which fellows’ publications have appeared for the pre-treatment and post treatment periods and calculating the difference.  For Ser Cymru II fellows, the difference between the two periods was +2.6, meaning their publications cover a broader range of subjects after Ser Cymru II than before it.  However, for the control group, the difference was -0.2, meaning the breadth of their research has remained roughly the same. The additional impact of the programme was estimated to be +2.8 subject areas per researcher (2.6 - (-0.2)). However this finding requires some caution as the number of subject areas covered by Sêr Cymru II fellows’ research publications was already increasing at a faster rate than the control group prior to the start of the programme, and therefore could be explained by other factors. 
• There is mixed evidence about the impact of the programme on the quality of research by fellows.  The programme had a significant, positive impact on the number of publications in journals ranked in the top 10% by CiteScore. While the number of articles in the top quartile for citations also increased by more than the control group, the difference was not statistically significant. Similarly, there was no significant difference in the change in the proportion of outputs in top journals or in the top quartile for citations. 

The evaluation has found that Ser Cymru II has had only a modest impact on research capacity.  Although 180 new researchers were recruited as part of the programme, only 74 are still employed by their host institution as of December 2023 (a retention rate of 41%). The retention rate was lowest at Swansea University (22%) and highest at Aberystwyth University (78%).  The reasons for this variation are uncertain, but it is likely to be explained by the extent to which researchers were aligned with, and integrated into, research groups with sustainable income streams.  Consultees from Swansea University noted that too many of the fellows were not well integrated and, as such, lacked supervision and a supportive network. Many of these left once the grant period had ended. 

Several consultees described the difficulty of retaining high quality researchers after the funding period ended, citing both the continued strain on university finances after the pandemic and uncertainty about the future of Sêr Cymru. This resulted in a number of researchers leaving and restrictions on any future recruitment. It was also noted that a number of those who have stayed at their host institutions have done so in either lecturing or in combined lecturing and research roles.

Effectiveness of management, delivery and monitoring processes

Although the hard work, dedication and flexibility of the delivery team in Welsh Government was recognised, it is clear there was a significant gap between the administrative requirements of the programme and the resources available within the delivery team. This meant that resources had to be dedicated to programme administration, which came at the cost of maximizing the strategic impact of Sêr Cymru II and engaging with the scientific research that had been funded. This was exacerbated by the departure of the incumbent Chief Scientific Adviser (CSA) and the original project manager, both of whom had played key roles in the design and early momentum of the operation. 

The focus on administration meant there was limited follow-up to progress reports and fewer meetings with Rising Stars and Research Chairs after the early phases of the programme. This meant Welsh Government lacked a strategic overview of which awards were performing strongly and which were not, what further support was needed and how benefits could be maximised. This also means opportunities were likely missed to celebrate the successes and achievements of Sêr Cymru II and raise the profile of Welsh science. 

Finally, it is clear that a number of early career researchers were not adequately supported by their host universities during their Sêr Cymru fellowship and this left many feeling isolated. In general, it appears those fellows who were integrated into existing research groups were better supported than those who were not. Future rounds of Sêr Cymru will need to give greater consideration to how early career researchers will be integrated and supported.

Contributions to Cross Cutting Themes

Sêr Cymru II's main contribution to CCTs were as follows.

• Equal opportunities and gender mainstreaming: The Recapturing Talent fellowships were designed to help stellar researchers restart their careers in academia following a career break. They were therefore suitable for women who wished to re-enter academia after a period outside the workforce (e.g. to start/raise a family). The operation also took a number of steps to ensure applicants were not disadvantaged by their characteristics including sex, ethnic background, sexual orientation or socio-economic background. Although the operation did not meet its target for Recapturing Talent fellowships, it did meet its target for at least 40% of researchers to be female. 
• Sustainability: The main contributions to sustainability have been through the research funded by the programme Around half of the awards have been related to the grand challenge area of low carbon, energy and the environment and are therefore contributing to this theme. 

The operation made only a limited contribution to tackling poverty and social exclusion. However this could be a longer term outcome if some of the research funded by Sêr Cymru II is commercialised and results in job creation in Wales. There are also a number of examples of where Sêr Cymru II funded researchers have undertaken volunteering and outreach activities which is relevant to this theme. 

There are a small number of examples of where projects have contributed to the promotion of the Welsh language. This includes school outreach activities, speeches at events and Welsh language awareness sessions. However this has generally only occurred where the fellow or a member of the research team was able to speak Welsh and was a strong advocate for promoting the language.

The potential to maximise and monitor contributions to CCTs was constrained by two factors.

1. Limited capacity and high staff turnover in the Welsh Government delivery team, which meant the role of CCTs Champion changed hands multiple times and was vacant for a long period. 
2. The expectations around CCTs not being made clear to grant recipients from the outset and providing no feedback to the information in the quarterly progress reports. 

The evaluation finds that Sêr Cymru II is contributing to most of the seven goals of the Wellbeing of Future Generations Act, with the only exception being a 'Wales of Cohesive Communities' as this is less relevant to this operation as there are no clear links between research activities and the goal of creating “attractive, viable, safe and well-connected communities”.

Rationale and design of future rounds of Sêr Cymru

Welsh Government should continue to support and deliver the Sêr Cymru programme. There is still clear evidence of need for Sêr Cymru in order to build research capacity and  world leading research excellence at Wales' HEIs. However, given that future rounds are likely to have substantially less funding available, this will need to be more strategic and selective than Sêr Cymru II. This should also set specific targets which more closely reflect the outcomes and impacts which are achievable. 

The future of Sêr Cymru should be guided by a clear, long-term vision and strategy. The stop-start nature of Sêr Cymru has created significant uncertainty for Wales's HEIs. This has made it difficult for the Research Chairs and their groups to plan for the future and build on their successes. An urgent priority is therefore for Welsh Government to work with the universities to establish an agreed vision for Sêr Cymru and its long-term priorities for the next ten years. 

Focus investments on internationally excellent research which builds on Wales' strengths and addresses national priorities. Future rounds of Sêr Cymru will need to avoid spreading resources too thinly. To maximise impact this should be focused on a small number of investments where there is a compelling strategic and economic opportunity, linked to Wales' national priorities, the existing strengths of universities and the distinctive assets, strengths and opportunities for Wales as a country. 

Ensure that industry is engaged in determining priorities. Linked to the above, it is essential that Welsh Government engages with industry and offers industry stakeholders a chance to influence priorities for investment. This had been missing from Sêr Cymru II which was focused mainly on scientific excellence and funded some research which was less relevant to the needs of industry.

Provide more flexibility and structure funding in a way that allows universities to leverage other funding. Future rounds of the programme should aim to offer more flexibility, including funding for PhD studentships which was not permitted for ERDF funded investments. It is also important that future funding is structured in such a way that it allows universities to leverage other funding opportunities. For example, Sêr Cymru funding could be used as leverage to secure new programmes such as Centres for Doctoral Training (CDTs) which offer another route to recruiting more PhD students to Wales. Structuring funding in such a way would allow Wales' HEIs to gain a competitive advantage for securing these programmes. 

Outcomes and progress should be rigorously scrutinised. Given the funding constraints and the need to focus on a small number of high impact investments it is essential that the progress and outcomes are closely monitored. Welsh Government and universities could agree a number of milestones and outcome indicators for each investment and monitor progress against these. If it is found that projects are not delivering the Welsh Government should not be afraid to terminate projects early. 

Welsh Government will need to work with universities and heads of department to ensure research groups are protected. There are concerns that some of the gains made by the research groups supported through Sêr Cymru II will be lost. It is clear that funding constraints have already led to the loss of a number of high quality researchers recruited through the operation, even in those groups which have secured a large amount of new research income, due to wider financial constraints in these organisations and pressure to focus on teaching. It is therefore important that the polices and priorities of Welsh Government and universities are harmonised as far as possible (recognising the financial pressures facing universities and the effect this has on research), and to work with Pro Vice Chancellors and heads of department to ensure they provide an environment for these research groups to flourish.

Leadership, management and delivery

Welsh Government needs to forge close relationships with universities and be a powerful voice for Welsh science. The long-term vision and strategy referred to above needs to secure the buy-in of Wales' HEIs to gain as much leverage as possible and to identify areas where universities can collaborate around shared national priorities. Given the commitment of the UK government to increase RD&I investment outside the South East, it is essential that Welsh Government is prepared to advocate forcefully within UK government circles on behalf of Welsh science and to set out a compelling narrative about its achievements and what its universities can offer. A key part of this should be celebrating the achievements of Sêr Cymru II and raising the profile of its 'stars'. In return, Wales’ HEIs need to provide Welsh Government with compelling evidence for world leading research. 

Ensure there is sufficient capacity and scientific expertise overseeing delivery. It is clear that the effective management and delivery processes in place at the start of the operation was not sustained after the pre-award phase, and that there was a lack of scientific expertise and capacity to manage and monitor the operation effectively. It is essential that this is addressed on future rounds of Sêr Cymru and that scientific rigour is applied in the post-award phase as well as the pre-award phase. Welsh Government should consider using an international panel of scientific experts to monitor the progress of projects once in delivery as well as in the pre-award phase. Alternatively, both the award and monitoring process could be subcontracted to a trusted external organisation with established, robust processes for awarding research funding and monitoring progress. 

Ensure early career researchers are supported and integrated into existing research teams. If future investments involve the recruitment of new researchers, the selection process should give far greater consideration to how they will be supported and integrated in to existing or new research teams, and ensure they are being supported by their host institutions on an ongoing basis. 

Streamline reporting and claims processes. Quarterly monitoring reports should be streamlined in line with research council norms and the claims process should be simplified to reduce the administrative burden and resource requirements for universities, grant recipients and Welsh Government.

[1] The Sêr Cymru II programme also contained a Marie Skłodowska-Curie COFUND funded operation, which is not included in this evaluation.

[2] These provided funding for academics to spend time working in a commercial organisation or for someone from a commercial organisation to spend time in an academic environment.

[3] Increasing the success of Wales’s HEIs in attracting research funding is a longstanding priority for Welsh Government and one of the specific objectives in the ERDF Operational Programmes for West Wales and the Valleys and East Wales, under which Ser Cymru II was funded. The rationale underpinning this specific objective was that growing the research base “will help encourage inward investment, business expenditure on research and the development of research-intensive innovative clusters”.

[4] Excluding the research income which was claimed by the HEIs but not included in claims to WEFO as it was considered to be ineligible or not supported by evidence.

[5] The business plan did not set a target for publications by research fellows but noted it was “possible that the fellows will be writing and submitting articles to journals at some point during their fellowship”

[6] Using data from the Higher Education Statistics Agency (HESA)

[7] Using data from Scopus, Elsevier’s database containing detailed bibliographic information on researchers, publications and citations.

[8] Excluding Research Chairs

Report author: Oliver Chapman

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:

Research and Evaluation
Social Research and Information Division
Welsh Government
Cathays Park
Cardiff
CF10 3NQ

Email: research.evaluation@gov.wales

Social research number: 45/2026
Digital ISBN: 978-1-83745-079-4