Trade Survey for Wales: quality and methodology information

This Trade Survey for Wales (TSW) Quality and Methodology Information (QMI) report discusses the technicalities (e.g., sample & response rate distribution, compliance cost, etc.) for the 2021 data collection. It also encompasses the overall methodology employed over multiple years and outlines the quality aspects of the survey methodology and statistics produced. The separate main statistical release details the results of the survey. Accompanying data tables are published alongside the report, and provide the latest revised estimates, with sales data backdated to 2018. Detailed analysis from previous surveys can also be found online.

Unless otherwise stated the year of TSW referred to throughout this report is TSW year four (TSW 2021) which collected 2021 data.

The Trade Survey for Wales (TSW) generates ‘official statistics in development’ ^{[footnote 1]} estimating the value (in current prices) ^{[footnote 2]} of trade conducted by businesses located in Wales. It collects data relating to the values of annual trade (sales and purchases) that business units located in Wales conduct with other parts of the UK, the EU and the rest of the world (Non-EU countries).

The sales and purchases are broken down by type (goods or services) and the broad geographical location of the trade partner. The questions cover both inter UK and international trade. Survey responses allow statistical estimates to be produced covering the value of trade for businesses in Wales and the number of businesses conducting this trade.

To date the survey has been run annually since 2019 when the Welsh Government (WG) commissioned IFF Research (IFF) to pilot the TSW on their behalf. Fieldwork for TSW (2021) was carried out between the 22 September 2022 and 4 January 2023. The survey produced statistics from information captured directly from businesses on trade flows to and from Wales in 2021.

The rationale for the TSW stemmed from the need to have a more detailed evidence base to underpin WG’s policymaking. Increased evidence would enable WG to:

• have a better understanding of Wales’ economy, including interlinkages between Wales-based businesses and those in other parts of the UK and abroad
• more accurately assess potential impacts of the UK’s proposed future trading relationships, on businesses in Wales and the wider economy

Welsh Government Analysts were responsible for:

• scoping the TSW content
• drawing the sample
• maintaining a governance framework for TSW oversight with key WG stakeholders
• commissioning the main-stage fieldwork and the analysis of survey results
• quality assuring the analysis and compiling the publications

IFF were commissioned to:

• collect data from respondent businesses
• chase businesses that had not completed the survey by the deadline date
• validate respondent business data through desk research and clarification phone calls where businesses gave their consent to be recontacted
• undertake imputation to model missing responses to the survey (within year imputation has been carried out every year, but this is the first year that across years imputation for sales has been achieved. See ‘Imputation’ in the Analysis section below for more information)
• produce estimates of Wales’ trade from weighted survey responses
• prepare and deliver data updates, and the final data file, to WG

The UK Government commissioned reviews of official economic statistics  have long recommended improvements to regional statistics. The 2004 Allsopp Review looked at statistics for economic policymaking, which made recommendations to improve regional trade statistics, and for the design of new and existing surveys to take account of the need for regional statistics. The later 2016 Bean Review of UK Economic Statistics (HM Treasury, Cabinet Office, The Rt Hon Matt Hancock MP, and The Rt Hon George Osborne) made further recommendations to improve ‘inadequate’ regional statistics, but highlighted the potential for administrative data to fill some of the gaps alongside improved coverage of business surveys.

Following the EU Exit referendum in 2016, the Welsh Government established its own Trade Policy team. The requirement for trade evidence grew, with the Welsh Government’s Trade Policy: The Issues for Wales (2018) calling for improved evidence to underpin trade policy decisions, and highlighted gaps in the trade data for Wales as well as potential methodological issues identified with existing data sources. There was no existing data on inter-UK trade from Wales, and limited data available according to business characteristics (such as size and sector).

HM Revenue and Customs (HMRC) publish quarterly trade statistics for Wales on the movement of goods, with value (in current prices) and volume data sourced from customs declarations. Much of the HMRC data analysis apportions trade for UK-wide businesses to Wales based on their employment split. This means that that the HMRC data for Wales may not accurately reflect the specific export values and markets of Welsh business units. The HMRC Regional Trade Statistics is published at the Standard International Trade Classification (SITC) 2 digit level ^{[footnote 3]}, and does not include any detail about business demographics (e.g. business size). This is not sufficiently disaggregated for Welsh Government evidence needs. Further, due to EU Exit, there have been changes to how HMRC collects GB-EU trade data, moving from the EU Intrastat system to customs declarations for exports in January 2021 and imports in January 2022. This resulted in a break to the Regional Trade Statistics time-series, making comparisons over time more difficult.

The HMRC data does not include trade in services. The Office for National Statistics (ONS) administer an Annual Survey of International Trade in Services (ITIS) that produces experimental statistics at the Wales level, with ONS publishing below-Wales level estimates for both trade in goods and services (also published in current prices) since 2021.

Given the limitations with data at the time, TSW was piloted in 2019 with the aim of capturing robust information directly from businesses on trade flows to and from Wales.

The statistics produced from the TSW have provided Welsh Government policy makers with improved evidence with which to inform Ministers. The inter-UK trade figures have been particularly useful in gaining a more detailed understanding of the UK internal market than was previously possible. This data is used by the Office of the Internal Market (OIM) in their annual reports on the operation of the UK internal market (OIM). International and domestic TSW data is also being used by the ONS to produce experimental and provisional UK interregional trade in goods and services estimates. These are due for publication in 2024, at the International Territorial Level 1 (ITL 1) which will cover Northern Ireland, Scotland and Wales, and the nine English regions.

The Welsh Government’s Export Action Plan, a Ministerial commitment in the 2021 to 2026 Programme for Government, also draws on the TSW as part of its evidence base. The survey additionally gives Welsh Government the ability to undertake follow up research with respondents agreeing to re-contact, allowing further insight into key policy questions highlighted in the results. Two small-scale follow up research reports (recontacting 2020 respondents) have been published to date on Rules of Origin and Mode 5 Services.

In 2023, Welsh Government launched a three-year project examining the feasibility of constructing a full regional economic account for Wales (digital and data blog), known as an Input Output (IO) table. The purpose of this is to enable an analysis, not only of how the economy ‘works’, but also how it fits within other systems – for example, UK and global economies, Wales’ natural environment, or the climate. It is anticipated that TSW data will contribute to this project.

The methods employed for the survey are still in development and there are some known limitations in terms of data quality. The volatility of estimates are assessed, with ways to further improve their quality explored. Official statistics in development are published with a view to improving their quality over time through feedback from users and stakeholders and developing the methodology.

The TSW estimates will remain official statistics in development while development work continues to improve their coverage and quality. Limitations which have led to these statistics being badged as ‘in development’ are listed within this section. However, the data and analysis are still of value provided that users view them in the context of the data quality information provided.

Strengths

• The TSW is the only direct collection of trade data specific to business operations located in Wales. Businesses directly report the activity of their Welsh locations. This contrasts with existing regional statistics which collect UK level responses from which Wales level results are modelled.
• TSW is the only trade survey collecting inter-UK trade data, broken down by all UK nations, by businesses in Wales.
• TSW results are the only source of inter-UK trade conducted by businesses in Wales.
• Results are broken down by business size.
• Results are broken down by grouped SIC classifications.
• Goods and services trade are collected on a consistent basis.
• The methodology allows estimates of the total trade (goods and services) businesses in Wales make with all parts of the UK, the European Union (EU) and the rest of the world (Non-EU).
• The survey allows for follow up research with consenting businesses on policy relevant topics and issues e.g. Mode 5 Services and Rules of Origin.

Limitations

• As a voluntary and online survey there are low response rates compared with other, more costly (or mandatory), methods of data collection. This increases the likelihood of non-response errors which affects the ability to produce some lower breakdowns.
• Variability in the achieved sample across years affects the ability to make robust comparisons across years, although with the added across-years imputation methodology it makes high-level comparisons across the years more achievable, which the results report presents. Breakdowns that are more granular have shown greater variability, specific comparisons are discussed in the comparability and coherence sections of the results reports.
• Incomplete coverage of businesses, sectors and total Welsh trade in our sampling approach means the estimates do not cover all trade activity in Wales and should not be used to work out a trade balance, nor compared to total GVA/GDP. For example, only business purchases are covered, direct to consumer purchases are missing. This means Welsh consumer spending abroad (e.g. tourism imports) and consumer spending in other parts of the UK has not been captured, whereas the expenditure of tourists buying from businesses in Wales is captured.
• Some respondents provided estimated values in their responses so the values provided will have varying degrees of reliability.
• Feedback suggests that some respondents found it difficult to allocate sales to customers based in other parts of the UK.
• Respondents were asked to allocate trade based on their immediate partner in the supply chain, not the original source or final destination of a purchase or sale. The movement of goods through supply chains within the UK may therefore mask an international origin or final destination in the TSW results, which may be of particular significance for the retail sector using distribution hubs within the UK.
• Within the analysis there are a number of instances where ‘unallocated’ sales or purchases data is highlighted, where the respondent was unable to allocate to a specific destination. The effect of unallocated can at times be inflated due to the weighting. This may have led to underestimates of the overall figures.
• The six sector groupings used to categorise businesses by SIC code within the survey are broad to ensure base sizes are large enough for analysis. This means it is not possible to filter results by some specific sectors of interest which fall within wider categories e.g. food and drink.
• TSW relies on ONS Inter-Departmental Business Register (IDBR) (ONS) sample frame which has a time lag. This means each year of IDBR might not be reflective of ‘real time’ data for the same year. Business liquidation/insolvency in any one year might mean that the weighting and grossing could be incorrect.
• The survey asks questions on trade value but not trade volume. This makes it challenging to understand changes over time in light of changes in inflation.

Mode

A push-to-web approach was selected as the most appropriate method to meet the aims and objectives of the TSW given the survey’s scale. Respondents are invited to take part through an initial postal mail out with further reminder letters and telephone chasing survey, but the questionnaire is completed online. The sample frame is obtained from the ONS Inter-Departmental Business Register (IDBR) (ONS), a comprehensive record of business addresses, which ensures the TSW sample coverage is consistent with other businesses surveys. However, the IDBR contains limited email addresses, so an initial postal mail-out letter contacting businesses and inviting them to respond to the online questionnaire is necessary.

Sample size and frame

8000 businesses are selected for the sample.

The sample frame includes businesses in the UK with local units in Wales. The sample is selected from the ONS IDBR that lists businesses from either VAT or PAYE records. The sample frame is limited to reporting units that have local units in Wales and have existed long enough to be able to give the full year of data being asked for.

Businesses with employment of fewer than 3 are excluded, along with the list of sectors below. The sector exclusions were decided in-line with the measurement of trade for national accounts as well as seeking to harmonise exclusions made by other, similar, surveys. Businesses of certain legal statuses were also excluded from the sample. These were local authority organisations, central government and most non-profit bodies (though universities were included).

Excluded Standard Industrial Classification (SIC) ^{[footnote 4]} Sections:

• O - Public administration and defence; compulsory social security
• Q - Human health and social work activities
• T - Activities of households as employers; undifferentiated goods-and services-producing activities of households for own use
• U - Activities of extraterritorial organisations and bodies

Further to the SIC sections list above, a number of lower SIC classifications are also excluded:

• 06 - Extraction of crude petroleum and natural gas
• 84 - Public administration and defence; compulsory social security
• 86 - Human health activities
• 87 - Residential care activities
• 88 - Social work activities without accommodation
• 93 - Sports activities and amusement and recreation activities
• 94 - Activities of membership organisations
• 96 - Other personal service activities
• 97 - Activities of households as employers of domestic personnel
• 98 - Undifferentiated goods and services producing activities of private households for own use
• 99 - Activities of extraterritorial organizations and bodies
• 642 - Activities of holding companies
• 791 -Travel agency and tour operator activities
• 851 - Pre-primary education
• 852 - Primary education
• 8299 - Other business support service activities not elsewhere classified (n.e.c.)
• 9101 - Library and archive activities
• 03110 - Marine fishing
• 47799 - Retail sale of other second-hand goods in stores (not incl. antiques)
• 49320 - Taxi firms
• 56290 - Other food services
• 56301 - Licenced clubs
• 64110 - Activities of central bank
• 79901 - Activities of tourist guides
• 79909 - Other reservation service activities n.e.c.

Sample design

Stratified sampling takes place at the reporting unit level. The sample was stratified by employment size band, and industrial sector SIC at the 2-digit level).

An overarching aim of the sample is to capture as large a proportion of the total Welsh economy (business turnover) as possible. To date the sample has selected 8000 reporting units, with a focus on larger businesses.

The structure and sample design are reviewed each year, as these statistics continue to develop. The strata and number of records selected each year are provided below.

After considering all exclusions, the survey participants were selected from a population of approximately 33,000 reporting units with activity in Wales and employment of three or more people.

Welsh Government Statisticians selected companies using a stratified approach based on employment. Some strata (20+ employment) took a census approach, making up the majority of the sample.

For TSW3, an element of convenience sampling was introduced to the small and micro-business (3-19 employment) strata to include businesses that previously responded in an aim to reduce respondent variability.

A stratified random sample was taken of the remaining businesses in the 3-19 employment size band (those with at least one local unit in Wales). These businesses were further stratified by employment size (‘3 to 9’ and ‘10 to 19’) and industrial sector (SIC section), see Table 1.

The sample was allocated based on the SIC section turnover; sectors with higher total turnover received a greater proportion of the sample. Further adjustments were made to the sample allocation to avoid under or over sampling within sub-bands.

Table 1 shows the sample breakdown for TSW (2021).

Source: IDBR

Sampling a small proportion of businesses with 3 to 19 employees ensured coverage of all size bands while reducing the overall burden placed on businesses and maintaining a practical and cost-effective sample size. The rationale for over-representing larger businesses within the sample was to ensure a large proportion of the Welsh economy (total Welsh IDBR turnover) was captured. The 8,000 sampled businesses were estimated to account for 98.7% of turnover in Wales (from businesses with employment of 3 or more).

The survey was conducted using a push-to-web approach. Sampled businesses were initially sent a bilingual (Welsh and English) postal invitation letter (copies included in Communications Annex) encouraging them to go online to complete the TSW.

Figure 1: Overview of the data journey

Image

Description of Figure 1: A flow chart showing the fieldwork process from sample drawn and prepared through to distribution of the survey link, data cleaning, processing and finally data and reporting outputs.

Source: Trade Survey for Wales (2021),  Welsh Government

As with previous iterations of the survey, TSW was voluntary; businesses were invited to take part via an initial invitation letter, followed by a phone call to confirm receipt of the initial invitation letter, one reminder letter, and two subsequent reminder emails (where an email address for the business was held) (see Table 2). In addition, IFF undertook call chasing throughout the fieldwork period in order to answer any questions from businesses and encourage completion of the survey.

If respondents needed to complete the survey in more than one sitting (or more than one person needed to fill in the survey), they could do so by re-entering their access code.  When they re-entered the survey, respondents were immediately taken to the last point in the survey they had reached. It was only possible to use each access code for a single survey response, so each business could only submit one return.

Throughout the data collection process, all data was stored securely in IFF’s Unicom Intelligence online storage system.

As part of regular progress updates between IFF and WG during the fieldwork phase, weekly response rate reports were compiled using survey data (which showed, for example, information on which survey sections respondents were pausing at, and respondents’ business size and sector).

Survey design, testing and piloting

Cognitive testing was conducted with 32 businesses ahead of the first TSW in 2019 and a further pilot undertaken ahead of the third in 2021. Feedback from businesses informed question development and the collection mode used. Further details of the cognitive testing and development of TSW year one can be found in TSW 2018: Technical Report.^{[footnote 5]}.

The majority of the year four TSW survey (collecting 2021 data) remained the same as the previous year (collecting 2020 data). There were some minor changes introduced to the year four survey, with some light-touch question testing undertaken with a small number of businesses:

• One new closed response question was added because of policy interest in the topic area of environmental goods and services. This question was asked only to those who made sales of goods and/or services: 

In 2021, did the Wales-based operation of the business make any SALES of environmental GOODS/SERVICES? Environmental goods/services are products manufactured primarily for environmental protection purposes, and/or for natural resource management.

• An amendment was made to a question on Mode 5 services following feedback from users that it was not well understood in the previous survey. The amended question included in the Year 4 survey asked respondents:

In the process of selling those goods, are there any services that are provided alongside the product as part of the sale of the good? For example, an ongoing subscription for maintenance and repair, or training, or a related software subscription such as cloud storage with a phone.

• A question on public procurement, introduced in TSW (2019),  was removed since this was no longer considered a priority question for policy colleagues.

Have you ever bid for, or won, public sector and/or government funded contracts to deliver goods or services either in the European Union or in overseas markets outside of Europe (i.e. Rest of World)?”)

To make the survey as short as possible and reduce respondent burden, questions were routed where possible to allow respondents to automatically miss pages that were not relevant to them.

Although respondents could not skip forward through the questions (due to routing constraints), they could move backwards in the survey to review and edit their previous answers before submitting the survey.

Timeline

The mainstage fieldwork commenced September 20 2022 and closed on 4 January 2023, running for approximately 15 weeks.

Source: Trade Survey for Wales (2021)

Postal and email contact with businesses

Sampled businesses were initially sent a bilingual invitation letter informing them of the survey and how they could complete it [Table 2 details the key dates for the postal and email contact]. Two versions of the letter were created, one for businesses which had not completed the previous survey and another thanking businesses who had completed the previous survey. Each letter was accompanied by an overview of the survey content and process. Copies of all communication materials (letters and emails) can be found in the Communications Annex.

A bilingual postal reminder letter was subsequently sent [see Table 2 for dates] to those who had not yet completed the survey, explaining that the deadline had been extended. A reminder email was also sent to those businesses who had asked to have the survey emailed to them following contact with the IFF call handlers. Although the responses which immediately followed each step cannot be directly attributed, an indication of corresponding response rates around these dates is provided in the further information column in Table 2. 

Response metrics

The survey was accessible 24 hours a day during the fieldwork period, although the peak periods of activity were usually during office hours. Over each survey year there had been a clear trend of survey completion primarily on weekdays during typical business hours, most often between 11am and 12pm.

There are two sources of data which may be considered when measuring time taken to complete the survey: data collected by the system when surveys were completed online, and self-reported figures provided by respondents within the survey. These are shown in Table 3.

Source: Trade Survey for Wales, 2021

The Code of Practice for Statistics advises public bodies collecting statistical data that the burden of the survey for respondents should be proportionate to the benefits arising from use of the survey data collected. A question therefore was included within the survey asking businesses for an estimate on how much time they felt it had taken them to complete the survey. The question asked respondents to include:

• time taken to become familiar with the questionnaire
• time of everyone who helped the business complete the questionnaire
• time spent extracting and preparing information from their systems
• any other time spent in relation to the questionnaire

Table 4 shows that the vast majority of businesses reported completion of the survey in under two hours ^{[footnote 6]}.

Source: Trade Survey for Wales (2021),  Welsh Government

[Note 1] The number of micro businesses (123)

[Note 2] The number of small businesses (584)

[Note 3] The number of medium businesses (333)

[Note 4] The number of large businesses (165)

Reported cost of completing the survey

A question in the TSW asked respondents to estimate how much completing the survey cost their business, including the time of anyone involved in filling out the survey. 

Of those who reported estimated costs incurred in completing the survey, more than half estimated the survey cost their business less than £50 to complete. This compares similarly to previous rounds of the TSW. Table 5 outlines the self-reported estimated costs broken down by company size.

Source: Trade Survey for Wales (2021),  Welsh Government

Base: 1214 businesses

Note 1:Micro businesses (123)

Note 2:Small businesses (587)

Note 3:Medium businesses (337)

Note 4:Large businesses (167)

Compliance costs

Compliance costs for businesses completing the TSW were estimated as per the GSS guidance. The total estimated cost for all businesses to complete the survey was calculated ^{[footnote 7]} to be £27,993. This financial cost is calculated based on the time taken to complete the questionnaire, an appropriate hourly rate and external costs incurred (for example, costs to the business of a bookkeeper or accountant to aid in completion of the survey).

Based on the self-reported question in the survey the median completion time for all responses was approximately 60 minutes. Subsequent contact between IFF and respondents to validate responses is also included in calculations since it contributes to respondent burden. 47 businesses were re-contacted over the phone for validation. The average duration of these calls was approximately 19 minutes (calls under 2 minutes were discounted).

Completes in Welsh and English

The TSW complies with the Welsh Language Standards, meeting the WG’s commitment to treating the Welsh and English languages equally. Respondents were given the opportunity to complete the survey in either Welsh or English and could switch easily between the two languages throughout when completing the online survey.

The system only captured language information for those who clicked ‘submit’ on the final page of the survey; this captured the language they were viewing on the final page. Of the 1214 respondents (who submitted on the final page), 1212 submitted the survey in English and two did so in Welsh, although some may have completed parts of the survey in Welsh.

Telephone data collection

In instances where businesses were unable to complete online, IFF call handlers were able to collect data over the phone. Due to the complexity of the information being collected, this was only provided as a last resort, where technical issues or infrastructure within the business prevented them from completing online, or where the business’s data was relatively simple. One respondent completed TSW (2021) over the phone, this compares to five which completed over the phone for TSW (2019).

Incoming contact with businesses

Businesses were able to contact IFF via phone and email with any queries they had relating to the survey. Table 6 shows that, in total, 1,042 queries were received either by email (265) or by telephone (672) ^{[footnote 8]}; a much higher volume than the 389 queries received in year two.

Source: Trade Survey for Wales (2021),  Welsh Government

Call handlers logged ‘dead letters’, a letter that was undeliverable and typically with an incorrect address, and then conducted desk research to find alternative details for these businesses. In total, 182 letters were returned to IFF. The low number of dead letters did not affect the overall 16% response rate when deducted from the 8000 sample. 

Privacy Notice

A requirement under UK GDPR when collecting any personal data, a privacy notice was published online detailing the exact lawful basis for processing their personal data.

Frequently asked questions (FAQ) documents

Alongside the initial invitation letter, businesses were sent a FAQs sheet which contained:

• background detail on the TSW and its importance to the WG
• information about IFF
• how to verify the survey was genuine
• how businesses were selected
• reassurance about confidentiality
• the deadline for completion
• how to complete the survey if the business was not primarily based in Wales, had no data to report or had entered into administration/receivership.

The online survey site contained a longer version of the FAQs, with more detail than could be included with the invitation letter. Additional information included in the online FAQs related primarily to the survey itself, including:

• more detail on how data provided would be used
• the process for reporting any technical issues experienced with the survey
• how to provide feedback on the survey to the WG
• how IFF securely received respondents’ personal details.

Response chasing

To encourage businesses to respond, IFF undertook telephone chasing. The try-count records the number of contact attempts made to each business during the chasing exercise. Most businesses were reached within one or two calls, although there were some that required further chasing [Table 7]. The average number of contact attempts made was 2.24. All businesses with a phone number were called at least once. Those recorded as ‘0’ include a combination of businesses who had already completed the survey before they were called as well as those for whom a telephone number could not be found.

Source: Trade Survey for Wales (2021),  Welsh Government

On average, a respondent took 8 days to submit their data from the point at which they ended their conversation with the call handler. Of the 326 respondents who received a chasing call and who then went on to successfully complete the survey (including those who did not explicitly agree to complete, such as those who were left a voicemail message), an average of 20 days elapsed between receiving first contact and submitting the survey.

Response rates

A response rate of 16% was achieved, with 1,264 respondents taking part in the survey out of a possible 8,000 businesses. This equated to a total just short of TSW (2019) (1,272). Only responses where both the sales and/or purchases sections of the survey had been completed, and where no discrepancies within the data were flagged, were included within the analysis. Progress had lagged behind that of the previous surveys throughout the fieldwork period.

Historically, the survey has received response rates of between 14% and 17% (Y1: 14%, Y2: 16%, Y3: 17%, Y4: 16%).

In terms of response rates for businesses according to business size (based on UK employment values within the IDBR), Medium-sized businesses had the highest response rate (19%), whilst Micro businesses had the lowest (10%) [Table 8].

Source: Trade Survey for Wales (2021),  Welsh Government

The response rate was fairly consistent across different business sectors, but those in Construction and Manufacturing had the highest response rates (17%) [Table 9].

Source: Trade Survey for Wales (2021),  Welsh Government

Business size bands

Throughout the data processing and analysis of TSW, businesses are grouped into categories based on UK employee values (business size). Four size bands are used in the analysis (see Table 8). However, only three size bands (small – which encompasses micro businesses, medium and large) have been used for the purposes of reporting.

UK employee values are used in order to ensure base sizes are large enough for analysis, improve comparability to HMRC & ONS data, and due to anecdotal evidence from cognitive testing that most businesses make trade decisions based on their overall UK operations.

Sector groupings

The IDBR provides a SIC code for each business in TSW, these are grouped to ensure base sizes are large enough to analyse.

Sector Groupings

• Business and Other Services
  • Information and communication
  • Financial and insurance activities
  • Real estate activities
  • Professional, scientific and technical activities
  • Administrative and support service activities
  • Arts, entertainment and recreation
  • Other service activities
• Construction
• Manufacturing
• Primary Sector and Utilities
  • Agriculture, forestry and fishing
  • Mining and quarrying
  • Electricity, gas, steam and air conditioning supply
  • Water supply; sewerage, waste management and remediation activities
• Trade, Accommodation and Transport
  • Wholesale and retail trade; repair of motor vehicles and motorcycles
  • Transportation and storage
  • Accommodation and food service activities
• Non-market Services
  • Education
  • Human health and social work activities (excluded)
  • Public administration and defence; compulsory social security (excluded)

Non-market services are combined with business and other services due to low base sizes, resulting in the following five categories being used throughout the data processing and analysis of TSW:

• Business and Other Services
• Construction
• Manufacturing
• Primary Sector and Utilities
• Trade, Accommodation and Transport

Imputation

To maximise the data that could be analysed from the survey, imputation was carried out to model missing responses to the survey within years (for Sales and Purchases data) and across years (for sales data only):

• Within year imputation (sales and purchases): Applied to the data provided by businesses who took part in the survey year but were not able to provide certain values e.g. they could provide their total sales but could not apportion this to goods and services. Missing values were based on other given values within the survey (and average proportions) provided by other similar businesses.
• Across year imputation (sales): Applied to sampled businesses who did not take part in the survey year. Missing responses were based on their given responses to other survey years and responses from other similar businesses in the missing survey year. Development is underway to achieve across-years imputation for purchases in future Trade surveys.

Within year imputation

Throughout the survey, respondents had the option of indicating to which destinations (for example, Wales, Rest of UK, EU, non-EU) they sold to, and then a subsequent question asking for the values of sales and purchases to these selected destinations. Within year imputation was carried out when respondents indicated the destination but could not provide the subsequent actual values. Therefore, values were imputed where a business selected that they had goods sales in Wales and Rest of UK (and not EU or non-EU), but then had not apportioned out their total goods sales value between these two destinations.

Missing values were imputed based on apportioning the relevant total figure provided by the respondent in line with average proportions of sales/purchases made to each destination among businesses of a) a similar type (explained in following section) and b) selecting the same combination of destinations.

Imputation was also carried out for businesses who selected that they had sales/purchases of goods and services but were unable to apportion these total values between goods and services. Missing values were imputed based on apportioning the total sales/purchases figure provided by the respondent in line with average proportions of sales/purchases made to goods and services among businesses of a similar type.

Once a business has had sales or purchases values imputed within the survey, no further imputations are calculated i.e. if their UK sales value is imputed, their disaggregated values for England, Scotland and NI are not eligible for imputation.

Businesses of similar type

Average proportions were calculated within imputation groups according to business size and sector, meaning imputed values were only calculated using values given by similar businesses. Values were only calculated within each group if there were at least 10 other businesses who had provided exact values in the same group as the record requiring imputation. Consequently, three imputation groups were created (detailed below):

• Imputation Group 1 was based on the 21 sector codes and 4 size bands
• Imputation Group 2 merged the sector codes into 5 broader groups but retained the 4 size bands
• Imputation Group 3 retained the 5 broader sector groups as in Group 1, but also merged the size bands into 2 broader bands

If a business requiring imputation fell into an imputation group containing fewer than 10 businesses giving exact values (e.g. imputation group 1), then they were allocated to the next level imputation group (imputation group 2).

Imputation Group Categorisation

Imputation Group 1

Industry sector included

1. Accommodation and food service activities
2. Activities of extraterritorial organisations and bodies
3. Activities of households as employers; undifferentiated goods-and services-producing activities of households for own use
4. Administrative and support service activities
5. Agriculture, forestry and fishing
6. Arts, entertainment and recreation
7. Construction
8. Education
9. Electricity, gas, steam and air conditioning supply
10. Financial and insurance activities
11. Human health and social work activities
12. Information and communication
13. Manufacturing
14. Mining and quarrying
15. Other service activities
16. Professional, scientific and technical activities
17. Public administration and defence; compulsory social security
18. Real estate activities
19. Transportation and storage
20. Water supply; sewerage, waste management and remediation activities
21. Wholesale and retail trade; repair of motor vehicles and motorcycles

Size band included

1. Micro
2. Small
3. Medium
4. Large

Imputation Group 2

Industry sector included

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size band included

1. Micro
2. Small
3. Medium
4. Large

Imputation Group 3

Industry sector included

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size band included

1. Micro/Small
2. Medium/Large

Same combination of destinations

Apart from the questions which asked for a total value to be split by goods and services, for each type of question, there were a number of different combinations of destinations that businesses could have selected [Table 10]. Average proportions were calculated only among businesses who had also selected the same combination of answers within the imputation group cell.

Source: Trade Survey for Wales (2021),  Welsh Government

Within the imputation groups containing similar business types and the same combination of destinations, the average proportions of sales or purchases allocated across these destinations were calculated, as opposed to average monetary values. Missing values were then calculated for each record by applying these average proportions to the actual total value provided by that business [Table 11]. For example, if a business said it had goods sales of £1,000 in a given year but had not said how this was split between Wales, the rest of the UK, EU and non-EU the average proportions would be applied as below.

Source: Trade Survey for Wales (2021),  Welsh Government

In total, within year imputation was calculated across 32 variables, allowing a total of 652 values, across 112 business, to be imputed. Across these 32 variables, the highest number of values per variable that were able to be imputed was 85 and the lowest was 0. The highest percentage of total values imputed ranged from 0% to 14.5%.

Across year imputation

Businesses who had not completed the survey were eligible to have their responses imputed if they met all of the following criteria:

• They hadn’t taken part in the current survey year (“target” year) but had been sampled i.e. invited to take part. It would not have been suitable to impute responses for business who were never intended to be included that year.
• They had completed or had their response imputed into the “achieved” year, thus enabling a response to be imputed for the “target” year. The achieved year includes both reported and imputed values.
• They had an IDBR UK turnover of £100 million or more, or had a Welsh turnover of £10 million or more, in the “achieved” year. This was to focus imputation analysis on businesses who would have a larger impact on the total turnover being reported for the “target” year.
• They have more than 20 employees in Wales in the “target” year.  All reporting units (a census) with employment of 20 or more are included each year of TSW, and made up the majority of the sample.

There were 249 businesses which met these criteria in Year 4 and were eligible to have their survey response imputed. Businesses had their survey response imputed for the “target” year based on “achieved” responses from either:

• Responses they had given in the previous survey year i.e. how much turnover they had reported in the Year 3.
• Responses they had had imputed into the previous survey year (Year 3) based on turnover they had reported in the years prior to that (e.g. Year 2) ^{[footnote 10]}.

A percentage change was applied to these “achieved” responses to reflect that turnover is assumed to not be the same year on year. The percentage change applied was calculated by:

• Identifying businesses who had responded to both the “achieved” and “target” years ^{[footnote 11]}. This was 572 businesses in Years 3 and 4 (47% of all Year 4 businesses).
• Summing the total turnover reported for these two years within similar businesses (four size bands by five broad sector groups).
• Calculating the percentage change between these two years of turnover for each of these size by sector groups e.g. Large Manufacturing businesses saw an 8% increase in reported turnover between the two years.

Percentages were only applied if at least 10 other businesses had taken part in both the “achieved” and “target” years (for Year 4, 12 of the 20 size by sector groups had 10 or more businesses within them). If fewer than 10 businesses had done so, the percentage change applied to an individual business was instead based on the change in UK turnover (as provided in the IDBR) for that business between the “achieved” and “target” years instead.

Once the percentage changes were calculated and assigned to individual businesses, broader quality checks with other data sources were conducted in collaboration with Welsh Government to ensure the model and values were robust. These checks resulted in some businesses having a percentage change of 0% applied or not being included within the imputation process. 

In total 249 businesses had their responses imputed into Year 4.

The same percentage change was applied to sales responses (not purchases responses) and ‘coded’ sales responses were copied into the data file.

Weighting

Response data from the survey is weighted and grossed up to the total population of businesses (derived from the sample frame), to produce Wales level estimates. Value and unit weights are applied to produce estimates of the value of trade and the number of businesses engaging in trade.

The weighting of results takes account of the unequal selection probabilities of businesses and the effect of non-response on the achieved sample.

As development work continues for these statistics, the sampling and weighting approach will be reviewed each year. The weights for sales and purchases were different in the current analysis because of the introduction of across-years imputation applied to the sales data.

In total, four sets of weights were created: two for sales and two for purchases. The two weights within sales and purchases accounted for the fact that roughly half of the survey collected value-based answers and the other half collected responses for tick-box questions which suited being weighted to represent the business population of Wales in terms of reporting units. The ‘unit-based’ weights were used for analysis of ‘tick-box/coded’ based questions and ‘value-based’ weights were used for analysis of turnover and cost based questions.

The weighting/grossing process involved the following steps:

• Step 1: Calculating two non-response weights (for unit-based and value-based questions) comparing to the starting sample of the survey.
• Step 2: Calculating ‘unit-based’ and ‘value-based’ grossing weights for both sales and purchases.
• Step 3: Multiplying the non-response weights with the sales and purchases weights to create the final four weights needed.

Step 1: Non-response weights

As the survey was not mandatory, there was some natural variation in response rates between different types of businesses which resulted in the survey profile differing slightly from that of the starting sample of the survey. To adjust for these differences, non-response weights were calculated, using a random iterative method (RIM), and applied to bring the survey profile in line with the sample. Non-response weights were calculated by comparing the profile of achieved interviews with the 8,000 businesses in the starting sample. The variables were compared, and where there was a difference of 2 or more percentage points between the achieved interview profile and that of the sample profile, a non-response weight was applied.

Sector Employment (5 bands)

1. Primary Sector and Utilities
2. Manufacturing
3. Construction
4. Trade, Accommodation and Transport
5. Business and Other Services

Size (IDBR UK employees, 4 bands)

1. Micro (<10 employees)
2. Small (10 to 49 employees)
3. Medium (50 to 249 employees)
4. Large (250+ employees)

IDBR Turnover (banded)

1. <£1 million
2. £1 million - <£5 million
3. £5 million - <£10 million
4. £10 million - <£20 million
5. £20 million - <£50 million
6. £50 million - <£100 million
7. £100 million+

Region (IDBR)

1. AA. North East
2. BA BB. North West
3. DC. Yorkshire and the Humber
4. ED. East Midlands
5. FE. West Midlands
6. GF. Eastern
7. GG. Eastern
8. HH. London
9. JG. South East
10. KJ. South West
11. WW. Wales

No of branches in Wales (banded)

1. 0
2. 1
3. 2
4. 3+

Whether business is entirely Welsh

1. Yes
2. No

Step 2: Grossing ‘unit-based’ and ‘value-based’ weights for sales and purchases

The non-response weights were applied to the data, and then two different weighting grids were calculated, grouping cells by four size bands and five sector bands as outlined above (the smallest number of businesses in any one cell was six and so no cell merging took place).

The two different weighting grids compared:

• number of units achieved in the survey compared to the population]
• total turnover (sales & purchases) in the survey compared to the population

For the unit grid, a weight was created for each cell of the grid to gross the number of records in each cell to the number of businesses in that cell in the population (i.e. if there were 20 records in the cell and there were 100 in the population then each record received a weight of 5). The UK employment values reported within the survey were not used because data cleaning revealed these were not as reliable compared to the IDBR employment data. This was likely as the survey would in some cases have been completed by staff with knowledge of the Wales based business but not necessarily with the UK-wide overview. The unit grid was therefore calculated using the employment values provided within the IDBR.

For the turnover grid, a weight was created for each cell of the grids to gross the sum of turnover for each cell (from the population and from our achieved sample) and applied to each £ of turnover so that each respondent in that cell would get a weight relative to their exact amount of turnover. The IDBR turnover figure was used because, although the survey asked about Welsh turnover specifically, the IDBR turnover field relates to UK turnover and a reliable turnover figure to gross to was required.

Step 3: Creating the final four weights

These ‘unit-based’ weights and ‘value-based’ weights were then multiplied by the non-response weights to create the final weights for both sales and purchases [Table 12].

Source: Trade Survey for Wales (2021),  Welsh Government 

Non-response error

The largest limitation on the accuracy of results is the non-response error. Response rates for online surveys are known to be typically low compared with other, more costly, methods of data collection such as telephone or face-to-face. Further, the TSW is a voluntary survey, while similar surveys conducted by the ONS are mandatory leading to response rates of some 60% or more.

Non-response leads to reduced base sizes, therefore breakdowns of the Wales level estimates are only provided where base sizes allow.

The across-years imputation approach is one way of modelling non-responses to improve the accuracy of the estimates.

Sampling error

Sampling error arises because the estimates are based on a random sample of the population rather than the whole population. The results obtained for any single random sample are likely to vary by chance from the results that would be obtained if the whole population was surveyed (i.e. a census), and this variation is known as the sampling error. In general, the smaller the sample size the larger the potential sampling error.

The sample design for TSW aims to minimise this by taking a census approach to the largest businesses (by employment) and focusing the remaining sample allocation to the industrial sectors with the largest turnover.

The sampling error will be explored further as part of the as part of the work programme associated with these official statistics in development.

Missing answers

Missing answers occur for several reasons, including refusal or inability to answer a particular question. Routing within the survey prevents respondents seeing questions that are not applicable. Within-years imputation is used to model some missing responses.

The results include an ‘unallocated’ category which is a result of respondents partially completing the questionnaire, for example, a business providing the value of total goods sales to the UK, but unable to provide a breakdown of sales to the separate UK nations. Although this limits the usefulness of some results, understanding the difficulty respondents find in answering particular questions is important while development work continues.

Incorrect answers

A potential cause of bias is respondent error when providing answers. The TSW online survey has multiple validation checks in place to prevent this. The online survey allows automatic checks to be conducted as the respondent is completing the questionnaire and flag questionable. Validation checks include;

• Logic checks that confirm breakdowns sum to the totals provided.
• Providing respondents a summary of answers given, in numeric and written form, overcoming potential confusion when inputting large values.

Where responses fail automatic checks the respondent is asked to review and correct the response or confirm it is correct (with the option of providing a reason).

Incorrect answers are also minimised by routing respondents to applicable questions only, based on responses to filter questions.

Final validation

Automatic validation checks are run on each submitted survey response to flag any remaining discrepancies which would invalidate the response. Flagged responses are reviewed by the contractor, data is amended if necessary; through a combination of desk-based research and respondent call-backs where consented to.

The contractor is also required to carry out other quality assurance checks on the data once fieldwork had finished. This included looking at the comments provided within the survey to assess whether any respondents had flagged particular issues completing the survey which might impact the reliability of their responses.

Quality assurance

Welsh Government statisticians perform quality assurance checks and analysis on the response data and results files provided by the contractor. All published statistics undergo quality assurance in line with GSS Quality statistics in government guidance.

Each wave of the TSW sample is drawn from an annual extract of the IDBR. The IDBR is updated using administrative and survey-based data sources with a variety of reference dates which means that business data could have been updated at any point up to the date of extract. This means that the business characteristic data used in sampling and weighting for the TSW is only as up to date as the latest version of the IDBR.

TSW fieldwork commences in Autumn, collecting data for the previous calendar year. The results are then analysed and published the following year, for example; 2018 data was collected in 2019 and results were published in 2020. However, due to issues encountered with the results from the TSW (2020), the findings for that year were not published immediately. The inconsistent results for 2020 were no doubt impacted heavily by the impacts of Covid-19 on trade and data collection in general. This led to a review of the methodology and the introduction of across-years imputation for sales. Results for 2020 have been published in 2023, along with a revised back series of sales figures to 2018.

Data

All TSW data is provided in open format accessible spreadsheets.

Disclosure control

We take care to ensure that individual businesses are not identifiable from the published results. We follow the requirements for confidentiality and data access set out in the Code of Practice for Statistics. Respondents are directed towards the TSW Privacy Notice for further information about the security and confidentiality of the data they provide.

Language requirements

Welsh language standards are adhered to for TSW data collections and outputs. Our invitation letters, questionnaire, telephone helpline, website, results publications, technical reports, datasets are produced in both Welsh and English. We aim to write clearly (using plain English/‘Cymraeg Clir’).

Comparing the results produced by TSW to existing data sources is a key component of our development work.

Annual results reports contain a detailed comparison of the latest TSW estimates against previous years and alternative statistics published by HMRC and ONS. Discrepancies are discussed and limitations in our estimates highlighted.

Development work on TSW continues and plans are in place to improve estimates by bringing in data from other surveys. We continue to seek feedback on our plans and on the outputs produced to date. Internal stakeholders are regularly engaged through the TSW Project Board which meets quarterly to discuss the ongoing development of the survey. Wider feedback and communication is always welcomed by the Trade Team and any user is welcome to get in contact at any time. Feedback can be sent to stats.trade@gov.wales.

[1] Official statistics in development (Office for Statistics Regulation) were previously referred to as 'experimental statistics'

[2] Current prices data are not adjusted to remove the effects of inflation.

[3] For a comprehensive list of SITC categories (un.org)

[4] UK Standard Industrial Classification of Economic Activities (SIC) (ONS)

[5] Trade Survey for Wales, 2018: Technical Report.

[6] These figures exclude seven businesses who said it took them zero minutes to complete. It also excludes six businesses who said the survey took them 24 hours or longer to complete.

[7] Where: Burden = [{Number of respondents x median completion time (hours)} + {Number of businesses recontacted for validation x Median time (hours) taken for re-contact}] x hourly_rate (based on the Annual Survey of Hours and Earnings, table 14.6A 2022 using managers, directors and senior officials median hourly pay).  Final calculation: Burden = [(1,264 x 1) + (51 x 0.12)] x 22.04 = £27,993.44.

[8] This does not sum the total number of queries as multiple queries could be included within one email or phone call.

[9] Sectors were grouped into five categories throughout data processing and analysis. After consultation within WG, it was agreed to group sectors using a standard sector grouping used in the Employer Skills Survey, but “Non-market services” were combined with “Business and other services” due to low base sizes.

[10] Businesses can only have their responses imputed from a previous sequential “achieved year”. Responses cannot ‘jump’ forward a year i.e. year 4 responses cannot be imputed based on year 1 responses. A business could be imputed across multiple past years, but this would always be done by imputing values forward one year at a time. 

[11] Imputed data was not used to calculate percentage changes between years within size and sector groups.

Statistician: Jonathan Burton
Researcher: Mair Smith
Email: stats.trade@gov.wales

Media: 0300 025 8099