Evaluation of the Out of Work Service: final report (summary)

The OoWS is a Welsh Government, European Social Fund (ESF), and Substance Misuse Area Planning Boards funded programme, which provides holistic employment support via peer mentors who use their own lived experience to provide empathetic and non-judgemental support and guidance to participants.

ICF was commissioned by the Welsh Government in March 2022 to undertake a second evaluation of OoWS, following an earlier evaluation commissioned in 2018 and completed in 2020. The second evaluation had four key objectives.

1. To update the findings of the first evaluation, providing an assessment which explains current performance and how COVID-19 has impacted on the achievement of targets and Cross Cutting Themes indicators.
2. To explore the impact of the COVID-19 pandemic on the OoWS, including experiences for participants and those delivering the service.
3. To examine the extent to which the recommendations from the first evaluation were met.
4. To identify learning and best practice for the Welsh Government and partners.

The primary aims of the OoWS were to reduce the number of 16 to 24 year olds who are not in education, employment or training, and to reduce the number of people aged over 25 years and above (with substance misuse and/or, mental health issues) who are long-term unemployed or economically inactive. This was done by offering one-to-one support from peer mentors and by supporting participants to gain qualifications, carry out job searching, engage in work experience or volunteering opportunities, and enter employment. Since the beginning of the COVID-19 pandemic in March 2020, the OoWS has largely been delivered remotely/online, meaning in large part that contact between peer mentors and participants took place by phone, text message or through online platforms.

The evaluation was mainly based on evidence collected through case study interviews in each of the seven area ‘Lots’ across Wales: North Wales, Gwent, Dyfed, Western Bay, Cwm Taf, Powys, and Cardiff and Vale. Across the seven Lots, interviews were carried out between May and August 2022 with nine Lot leads, 22 peer mentors, five employment specialists, 10 delivery partners, 4 employers, and 35 participants (service users). The evaluation was also informed by analysis of numerical monitoring information collected by service providers.

Participant recruitment

Findings on participant recruitment were largely informed by an analysis of programme monitoring information collected by OoWS providers between August 2016 and May 2022. According to the data, 18,110 participants were recruited to the OoWS up to May 2022, and at this time all of the area Lots had achieved at least 89% of their recruitment target.

The most common referral route on to the service was through external agencies, usually the Jobcentre Plus (JCP). Other routes included a medical professional, Citizen’s Advice Bureau, or mental health charities. Participants were attracted to the OoWS initially to get support with health and wellbeing challenges. Gaining employment and work-related training was a less common reason for joining the programme.

Peer mentors and Lot leads said that COVID-19 had a significant impact on their ability to recruit new participants to the programme. Unsurprisingly, the period during which the first national lockdowns were introduced (year four, quarter four) saw a significant drop in the number of participants recruited onto the programme.

Provider leads and peer mentors considered that a good proportion of referrals were somewhat inappropriate because the participants had high-level needs and were experiencing complex mental health problems and were, therefore, far-off from entering employment. This was likely related to the conditions of the pandemic whereby people were facing significant challenges and services were less accessible than usual.

Overall, 8,366 (46%) of the participants left the programme earlier than planned. There was significant variation in the proportion of early leavers between the operational areas. West Wales and the Valleys saw a higher proportion of leavers across all categories (58%) than East Wales (44%). The main reasons for early leaving over the pandemic period are likely to be problems engaging with online platforms, diminished motivation to find work, and unmet expectations regarding specific needs, such as provision of goods and services and access to training courses.

Regarding the challenges faced by participants, mental ill-health  was the predominant issue for participants: over half were recorded in this category. A much lower proportion (10%) were categorised as recovering from only substance misuse. The remainder experienced both mental ill-health  and substance misuse.

Implementation of the Out of Work Service

The use of peer mentors was highly valued by both participants and delivery partners. However, high turnover of peer mentors was a key challenge, the reasons for which include stress (sometimes occasioned by the role and in some cases due to pre-existing mental health issues), not having the right skill set and/or experience to undertake the role, challenges in dealing with the procedural and data driven aspects of the role (such as evidencing the achievement of targets) and dissatisfaction with levels of pay.

The initial mandatory training for peer mentors was generally well regarded. The length of subsequent on-the-job training and shadowing varied by provider, with some offering just a few days and some offering up to six-weeks. Some peer mentors felt that shorter timeframes were insufficient. Key areas of training that would be beneficial in the future include supporting more severe mental health problems and crisis situations such as risk of suicide, and maintaining boundaries (i.e., the line between counselling and mentoring).

In the context of the pandemic, participants often presented with relatively high-level and complex needs related to mental health, including social anxiety, social isolation, fear of leaving the house and substance use, such that a large proportion of participants were a long way off from employment and may need longer-term support to address complex barriers. Consequently, the emphasis of support was ‘well-being’ related rather than directly employment related.

While the shift to online/remote delivery models produced challenges, a key advantage of this was that participants with social anxiety issues were able to better manage social interactions and they did not need to leave the house if this was too challenging for them. Remote contact also afforded a greater degree of accessibility for some participants, especially where there are transport barriers. Peer mentors had more flexibility in their working day because they did not need to work to fixed face-to-face appointments and could fit calls in when appropriate.

Peer mentors and participants considered social contact, ‘checking-in’, and meaningful wellbeing related conversations to be crucial elements in maintaining wellbeing and progressing towards employment and education. Alongside this, peer mentors provided much needed help with practical life tasks and were able to refer participants to internal wellbeing activities and courses, covering confidence, understanding anxiety, stress awareness, managing mental health. Other activities were socially orientated, the purpose of which was to get people out of the house and to interact with other people.

Peer mentors having lived experience of mental health issues and/or substance misuse was highly valued by participants. The support given is thought more likely to be non-judgemental and empathetic. However, while peer mentors have a strong skill set emanating from their lived experience, a potential challenge is they may lack higher-level technical and professional skills and competences. Another challenge was that in the context of participants showing high-level, complex mental health needs over the pandemic, peer mentors were at risk of being drawn into offering psychological-therapeutic interventions, something they were not equipped for.

The timing and balance of wellbeing related support with employability related support is crucial to maintaining engagement and achieving longer-term job and education outcomes. Some peer mentors did not feel equipped or confident to offer employability related support, and therefore an employment specialist role can add distinctive value.

Volunteering opportunities are a valuable option for many participants because they are often linked to their interests and goals, are community focused, and flexible depending on the needs of participants. Offering short courses, especially with a vocational orientation, also worked well, often representing a step closer to work or having the potential to directly lead to work.

Achievement of project outcomes  

The OoWS was especially successful in supporting participants to gain qualifications. The West Wales and the Valleys 16 to 24 programme exceeded the target by 33% while the two East Wales programmes (age groups of 16 to 24 and 25 and above) exceeded the target by 15%. The West Wales and the Valleys 25 and above programme exceeded the target for gaining qualifications by 14%.

Regarding the outcome target for participants to enter employment on leaving the service, as of May 2022, for the 16 to 24 age group, in West Wales and the Valleys, 68% of the target was achieved, and in East Wales 77% of the target was achieved. For the 25 and over group, the achievement was 55% of the target in West Wales and the Valleys and 71% in East Wales. Therefore, across the overall programme, 10% of participants had entered employment by May 2022 against an overall target of 15%. Notably, the success rate was maintained over the period of COVID-19 protective measures. The figure drops to 2% of participants in maintained employment six months after leaving the service. However, there were challenges in maintaining contact with participants once they had left the service, so this may not reflect what has been achieved in practice.

It was relatively more challenging to support participants to enter education or training on leaving the service. As of May 2022, for the 16 to 24 age group, in West Wales and the Valleys, 41% of the target was achieved. In East Wales, 21% of the target was achieved (this target was not relevant for those 25 years and over).

By May 2022, 10% of participants had completed a work experience placement or volunteering opportunity. In relation to targets, for the 16 to 24 age group, in West Wales and the Valleys, 62% of the target was achieved, and in East Wales, 47% of the target was achieved. For the 25 years and over group, the achievement was similar: 67% in West Wales and the Valleys and 49% in East Wales. The success rate was not significantly lower during the period of COVID-19 protective measures despite volunteering events and placement opportunities being significantly curtailed at this time.

Given that levels of need and complexity had risen since the onset of COVID-19, the achievement of soft outcomes is even more important than the pre-pandemic situation. Many service users were reported to have a long way to go before they were job-ready and small steps, such as going outside for the first time in months, were described as very important. This was echoed by participant interviewees who reported significant improvements in mental health, general well-being, and confidence. Overcoming social isolation frequently went hand-in-hand with a restored sense of purpose and confidence derived from spending more time with other people within the community, partaking in social activities, support groups or volunteering placements.

Recommendation 1

To retaining peer mentors, funders and providers should ensure salaries are locally competitive, that employment contracts are of a sufficient duration, and there is a clear progression and training plan.

Recommendation 2

The training and development of peer mentors should be formalised and systematic, including continuous professional development options such as accredited courses at Level 3 or Level 4.

Recommendation 3

While many peer mentors have a unique skill set based on their lived experience and are highly motivated to help people, providers should give fuller consideration to defining and limiting their role to avoid peer mentors being drawn into providing psychological therapeutic interventions for which they may not be trained.

Recommendation 4

Given the high-level needs of many participants, providers and peer mentors should have procedures and pathways in place to identify when a participant should move from wellbeing related support to receiving employability support/beginning job search and entering employment. For example, this could include the use of toolkits to ascertain how ready someone is to begin work.  

Recommendation 5

Employment specialists can play a distinctive and valuable role and can also help to source volunteering placements. Providers should install employment specialists and ensure their role with peer mentors is defined and that there are clear pathways between the two roles.

Recommendation 6

Providers should develop hybrid models of support (online and face-to-face) which can ensure that accessibility and flexibility of support is maximised. This can be done by developing or procuring short courses or vocational courses that are delivered online.

Recommendation 7

Providers should prioritise short courses as an engagement mechanism and steppingstone to work, while funders should recognise this outcome in contractual/payment structures.

Recommendation 8

Providers should more fully develop roles and workstreams to source volunteering opportunities. This could mean employing a dedicated volunteer coordinator to broker volunteering placements and support participants whilst placed. Also, hosts should be supported to develop a structured placement whereby learning or soft skills objectives are identified for that participant. 

Recommendation 9

Funders and providers should ensure that peer mentors have full awareness and understanding of local services and the procedures for signposting and/or referral. This may also require systematic coordination of local services to link into a peer mentoring service. 

Report authors: Lucy Loveless, David Scott, Oli Taylor, Izabela Jamrozik, and Emma Lovatt (ICF Consulting Services Ltd.)

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:
Laura Entwistle
Email: research.healthandsocialservices@gov.wales

Social research number: 11/2023
Digital ISBN 978-1-80535-272-3

Image

Image