National Survey for Wales: equality data review (summary)

Welsh Government needs to be able to measure whether it is succeeding in its national well-being goal of “A More Equal Wales” as set out in the Well-being of Future Generations (Wales) Act 2015. To achieve this, evidence is needed on a range of indicators that can then be broken down by protected and associated characteristics.

The National Survey for Wales (National Survey) is one of the main social surveys commissioned by the Welsh Government of people aged 16+. It began running in 2012 and covers a wide range of topics including education, exercise, health, community cohesion, well-being and finances. The National Survey is a random sample survey with topics varying each year. Some questions are repeated each year, while others vary depending on policy needs.

The survey also captures data for 16 of the National Well-being Indicators to measure progress towards the 7 goals in the Well-being of Future Generations (Wales) Act 2015. The National Survey is therefore a key data source on National Indicators.

Research aims

This project aimed to assess to what extent the National Survey is representative of Welsh equality groups and make recommendations on how to improve the quality and level of detail of equality data collected in future iterations of the National Survey.

The main objectives of the research were to: consider the usefulness of the National Survey for reporting on equality groups in Wales; and to conduct analysis to see if pooling National Survey data over several years improves usability.

The methodology had two parts, a desk-based assessment of the National Survey and pooled analysis on equality characteristics. The desk-based assessment compared the latest National Survey equalities questions to current harmonised standards and Census for Wales 2021 data.

The pooled analysis combined four years of National Survey data (2016 to 17 to 2019 to 20) on four National Indicators. These National Indicators were used as they were collected in all survey years, giving the largest pooled sample. This helped assess whether pooling enables better-quality equalities data analysis.

The desk-based assessment was limited as harmonised standards were not assessed against academic literature or government use beyond Great Britain. The Census for Wales 2021 and National Survey periods are different so not directly comparable, and the pooled data analysis does not explore a range of variables, so the conclusions are limited to the National Indicators. Even so, it is likely the conclusions would still be accurate, and challenges are likely greater for topics included less frequently and asked to only a sub-set of people.

Survey review: how protected characteristic information is collected and how it compares to the Census for Wales 2021

Because the National Survey is based on a random sample, the survey results may differ from the ‘true’ value for the population is likely to fall (that is, the figure we would get if the survey covered the entire population). We can use statistical methods to calculate confidence intervals, sometimes known as the ‘margin of error’, for each survey estimate. The confidence intervals around an estimate give a range within which the true value for the population is likely to fall.  So, for example we found that when investigating material deprivation by seven age categories for single year data, confidence intervals were less than +/- 4%. This means that we think the true population figure lies within 4 percentage points of the sample estimate. Confidence intervals are generally wider for smaller groups of respondents. For shorthand, in this report we refer to this as “less precise”.  Similarly, confidence intervals for larger groups of respondents are generally narrower, and we refer to this as “more precise”.

For the purposes of this project, we defined sufficient precision as confidence intervals around an estimate being narrower than plus or minus 2% for a particular category (e.g. 16 to 24’s or females).

Main findings

Variables where we have sufficiently precise data

Age

The survey captures age through date of birth or age at last birthday, with data then grouped into age categories for reporting. The results are broadly consistent with Census 2021 breakdowns.

Disability

The survey asks two questions, which when asked together are aligned with the Equality Act 2010 definition, capturing conditions that limit daily activities, rather than the Social Model of Disability, which the Welsh Government commits to. The National Survey found a higher proportion of disabled people compared with the 2021 Census.

Marriage and civil partnership

The National Survey captures marital status with categories mostly corresponding to the harmonised standard. The results are broadly comparable to Census 2021.

Sex

Guidance relating to sex and gender survey questions has been developed and refined further in recent years, including updates to the harmonised standard. These developments are reflected in changes made in the National Survey over recent years. Sex breakdowns are broadly comparable between the National Survey and 2021 Census.

Variables where do not have sufficiently precise data

Pregnancy and maternity

Data on pregnancy is collected, but maternity is not captured. There is no harmonised standard for pregnancy or maternity. Census 2021 captures maternity leave but not pregnancy, so comparisons were not possible.

Marriage and civil partnership

While marital status can be broken down relatively precisely, Civil Partnership and separated categories lack precision.

Household type

The National Survey captures household type and on the whole this is fairly precise. However, some categories had wide confidence intervals and the ‘married couple pensioner’ category had very small sample sizes so was suppressed.

Race and ethnicity

The survey captures ethnicity and national identity, rather than race, following both harmonised standards. Ethnicity data is published and is similar to Census 2021 results. National identity data from the National Survey is not routinely published but is available on request.

Religion and belief

The survey captures religion but not belief. The way religion is captured follows the harmonised standard. The National Survey shows the largest group are people with ‘no religion’, whereas Census 2021 shows it to be ‘Christians’.

Gender reassignment

The National Survey does not collect gender reassignment data; therefore, no Census Comparisons were made.

Sexual orientation

The National Survey follows the harmonised standards; however, this differs from the question used in Census 2021. Sexual orientation data in the National Survey is broadly comparable to the Census 2021.

Pooled analysis

The pooling of four years of data allows for larger samples and greater precision. This enables more granular analysis of some protected characteristics and associated characteristics by National Indicators. Age, sex and disability are not discussed below as precision was suitable in single year data, however pooling increased this further, and this is discussed within the main report. Small sample sizes continue to be an issue for some characteristics meaning robust granular analysis, particularly for ethnicity, religion and sexual orientation, is not possible.

Marriage and civil partnership

Single year breakdowns of marital and civil partnership status by National Indicator lacked precision. Pooling data improved this, but sample sizes were still too small to provide precise data on to civil partnerships.

Household type

Generally, household types can be broken down by National Indicators in single year data in a way that reflects the population, but some sub-categories had low precision and small sample sizes which pooling data greatly improved. 

Pregnancy and maternity

A question on pregnancy is asked of women of childbearing age who are in the half of respondents asked about BMI. Given the low proportion of the population who are pregnant at any one time this means that even when pooling 4 years of data, samples were too small. Due to these data limitations, analysis was not undertaken.

Race and ethnicity

Single-year sample sizes were not large enough for robust analysis beyond three high-level groups with wide confidence intervals. Pooling data allowed for five categories to be reported. However, data was still fairly imprecise meaning National Indicators cannot be precisely broken down by detailed ethnicity groups.

Religion and belief

Single-year data on religion can only be grouped into three categories, because of low sample sizes. Pooling data allowed for four of the desired eight religion categories to be produced. Pooling data improved precision however certain categories still had wider confidence intervals.

Sexual orientation

Single year data provided imprecise breakdowns into two categories. Pooling four years allowed for data to be broken down into four categories, improving granularity, however conclusions were still limited in terms of precision.

Pooling four years of data increases sample sizes of certain protected characteristics, in general, was shown to improve precision of results. To maximise the potential data for analysis, especially when pooling, it is key to consider asking questions that underpin National Indicators to more National Survey respondents, particularly those from under-represented equality groups. This should be balanced with considerations around limiting overburdening these groups. However, there are categories where pooling even four years of data does not generate sufficient sample for robust data so further steps are required.

To improve precision of results for the smaller subgroups, Welsh Government may need to explore options for targeted sample boosts for specific equality groups and possibilities of further data linkage that capture additional equality characteristics so analysis can cover a wider range of characteristics. A full list of recommendations can be found in the main report.

Report authors: Owens, A., Gibson, J; Poole, J

Views expressed in this report are those of the researchers and not necessarily those of the Welsh Government.

For further information please contact:

Victoria Saynor
Equality, Race and Disability Evidence Units
Equality, Poverty and Children’s Evidence and Support Division
Welsh Government
Cathays Park
Cardiff
CF10 3NQ

Email: EqualityEvidenceUnit@gov.wales

Social research number: 60/2025
Digital ISBN: 978-1-80633-011-9