scotland’s large scale surveys and survey harmonisation

SCOTLAND’S LARGE SCALE SURVEYS AND

SURVEY HARMONISATION

1 JULY 2008

Scottish Household Survey

Introduction

• Started in February 1999, as a result of devolution

• c.15,000 households per year• Two part interview

– Highest Income Householder or spouse/partner• Information about the households in Scotland• Characteristics of HIH help characterise the household as a

whole

– Randomly Selected Adult• Information about adults in Scotland

• 45 minutes long

Topics Covered

• 3 Priority Topic Areas– Transport – Local Government – Communities

• Many other areas– Health & Care– Anti-Social Behaviour– Culture and Sport– Recycling

Reporting

• Can produce – Results for Scotland every quarter– Results for larger LAs every year– Results for the urban/rural classification every

year– Results for all LAs every two years

Survey Data

• The SHS is an important source of data to aid policy development and evaluation

• The survey provides the data for 5 of the SG’s Spending Review 07 indicators:– Neighbourhood Rating– Satisfaction with local services– Smoking– Journeys affected by congestion– Travel to work

Smokers in Scotland

Adults (16+): Percentage who smoke

0

5

10

15

20

25

30

35

1999 2000 2001 2002 2003 2004 2005 2006

Year

% S

mo

kers

Target – 22% by year 2010

Neighbourhood RatingAdults (16+): Pecentage who rate their neighbourhood as a Very Good or Fairly Good place to live

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

SIMD deciles

%

Very good Fairly good

Recent Changes

• Significant de-clustering of the sample

• Introduction of modularisation

• Consortium of three companies for 2007-2010– Ipsos Mori– TNS– Scottish Centre

Looking forward

• Questionnaire Review for 2009– Harmonisation– Policy Requirements under the new

administration– Improving interview “flow”

Future Plans

• Methodology Developments– Weighting– Income Imputation

• Dissemination & Promotion– Training– New Publications– Web-based analysis

The Scottish House Condition Survey

The SHCS – basics

• 3,000 occupied households every year;

• Simple Random sample of Local Authorities - unclustered

• Minimum of 80 surveys in each Local Authority

• 2-stage Survey• Social Survey; followed by • Physical Survey

• Purpose: Measures the quality of people’s experience of housing against the quality of housing in Scotland;

The SHCS – Basic outputs

Surveyor expertise

-

50

100

150

200

250

pre-1919 1919-1944 1945-1964 1965-1982 post-1982

Age of dwelling

Num

ber

of

dw

elli

ngs

(thousa

nds)

DetachedSemi-detachedTerracedTenementOther flats

The SHCS – Basic outputsSocial Survey expertise

Tenure by age group of highest income householder (%)

0

10

20

30

40

50

60

70

80

16-30 31-44 45-54 55-64 65-74 75+

Grouped age of HIH

Perc

enta

ge

Owner-occupierLA/other publicHA/co-opPrivate renter

The SHCS – Basic outputsSocial and Physical Survey – number of bedrooms and

household relationships, genders and age of residents

3 or more bedrooms above

standard7%

2 bedrooms above standard

19%

I bedroom above standard

39%

Compliance with bedroom standard

32%

Below Standard3%

The SHCS – Key Outputs Domestic Dwellings Energy The SHCS – Key Outputs Domestic Dwellings Energy EfficiencyEfficiencyDwellings becoming more energy efficient – but still a long way to Dwellings becoming more energy efficient – but still a long way to gogo

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1991 1996 2002 2003/2004

2004/2005

2005/2006 Survey period

Good 7-10

Moderate 3-6

Poor 0-2

% of dwellings

National Home Energy Rating

The SHCS – Key outputs:

Fuel Poverty (Income, dwelling heating characteristics, size and fuel costs)

Hhlds in Fuel Poverty 1996-2005/6 (% and CIs)

1996 2002 2003/42004/5 2005/60

5

10

15

20

25

30

35

40

Survey year

Perc

enta

ge

The SHCS – Key outputs:

Scottish Housing Quality Standard

•be above the statutory Tolerable Standard

•be free from serious disrepair

•be energy efficient

•have modern facilities and services

•be healthy, safe and secure.

The SHCS – Key outputs: Scottish Housing Quality Standard

0

500

1000

1500

2000

2500

No of dw

ellings 000s

2002 2003/4 2004/5 2005/6

>4 Failures

4 Failures

3 Failures

2 failures

1 Failure

0 Failures

The SHCS medical variables:

• 2002 Health Report

http://www.scotland.gov.uk/Resource/Doc/933/0039909.pdf

• Nothing since – handout with current health questions.

• Abandoned Mental health module but in survey in 1991, 1996, 2002, 2003/04, 2004/05 2005/06 and 2007.

• Abandoned Coughing and wheezing but in survey in 1991, 1996, 2002, 2003/04, 2004/05, 2005/06 2007.

• Now have simplified asthma type questions (2008 onwards)

Mental Health Module GH1 Have you recently been able to concentrate on whatever you're doing?

GH2 Have you recently lost much sleep over worry?GH3 Have you recently felt that you are playing a useful part in things?

GH4 Have you recently felt capable of making decisions about things?

GH5 Have you recently felt constantly under strain?

GH6 Have you recently felt you couldn't overcome your difficulties?

GH7 Have you recently been able to enjoy your normal day-to-day activities?

GH8 Have you recently been able to face up to your problems?

GH9 Have you recently been feeling unhappy and depressed?

GH10 Have you recently been losing confidence in yourself?

GH11 Have you recently been thinking of yourself as a worthless person?

GH12 Have you recently been feeling reasonably happy, all things considered?

Coughing and Wheezing Module

HE2 Have you had any wheezing or whistling in your chest at any time in the last 12 months?

HE3 Have you been at all breathless when wheezing noise was present?

HE4 Have you had this wheezing or whistling when you did not have a cold?

HE5 Have you been woken up with a feeling of tightness in your chest at any time in the last 12 months?

HE6 Have you been woken by an attack of shortness of breath at any time in the last 12 months?

HE7 Are you currently taking any medicine (including inhalers, aerosols, or tablets) for asthma?

Coughing and Wheezing Module

HE8 Have you ever had asthma?HE9 Was this confirmed by a doctor?HE10 Have you had an attack of asthma in the last 12 months?HE11 How old were you when you had your first attack of asthma?HE12 How old were you when you had your most recent attack of

asthma?HE13 How many attacks of asthma have you had in the last 12 months?HE14 Do you have any nasal allergies including hay feverHE15 Have you ever suffered a two year period or more during which

you experienced coughing and phlegm on most days?

HE16 And would you say that you suffered that problem for periods of 3 months or more each of those years?

Coughing and Wheezing Module (HE17-26 asked about children in HH)

HE17 Has {sub} had any wheezing or whistling in the chest in the last 12 months?

HE18 How many attacks of wheezing has {sub} had in the last 12 months?

HE19 In the last 12 months how often, on average, has {sub2} sleep been disturbed due to wheezing?

HE20 In the last 12 months has wheezing ever been severe enough to limit {sub}'s speech to only one or two words at a time between breaths?

HE21 Has {sub} been woken by an attack of wheezing in the last 12 months?

HE22 In the last 12 months has {sub} chest sounded wheezy during or after exercise?

Coughing and Wheezing Module (HE17-26 asked about children in HH)

HE23 In the last 12 months has {sub} had a dry cough at night, apart from a cough associated with a cold or chest infection?

HE24 Has {sub} had more than 3 courses of antibiotics for respiratory infections (chest, ears or throat) in the last 12 months?

HE25 Is {sub} currently taking any medicine (including inhalers, aerosols or tablets) for asthma?

HE26 Has {sub} had an attack of asthma in the last 12 months?

Scottish Health Survey

History

• Designed to provide information on population health, health-related behaviour and use of health services in Scotland.

• Unique aspect of survey is nurse visit where measurements such as BP, lung function and blood samples are taken

• Previously run in 1995, 1998 and 2003

• Reviewed in 2005/2006

Review Recommendations

• Continuous

• Core and modular structure

• Scotland results annually, HB results at least every 4 years

• Unclustered sample

• NHS boards able to purchase boosts to their samples

Survey Design

• Annual sample of around 6,400 adults and 2,000 children

• HB minimum sample of 784 after 4 years

• Sample stratified by HB and SIMD

• Unclustered design (after 4 years)

• HBs able to purchase boosts (3 have for 2008)

Survey Design - Structure

• Core and modular structure• Core questions (around 35 mins) to full sample• Two modules of questions (around 10 mins) to

partial sample– Module A – range of topics – some questions being

asked annually, some biennially Given to one third of the full sample

– Module B - Knowledge, Attitudes and Motivations module (previously HEPS) – asked annuallyGiven to two thirds of the full sample but only asked of one adult per household

What Happens Now?

• Contractor appointed July 2007

• 2008 questionnaire finalised end December 2007

• Fieldwork began end January 2008

• First publication planned for summer 2009

Labour Force Survey &

Annual Population Survey

LFS & APS

• Survey of households• Covers:

– Employment hours worked, industry, occupation, earnings

– Unemployment and economic activity – Education and training:

qualifications held and current education/training

- Health problems limiting ability to work

UK-wide LFS

• Quarterly survey (since 1992) • Sample size – 52,000 households in UK

– Approx 8,000 in Scotland (in five ‘waves’)

• Panel design – Individuals stay in sample for five quarters– Each quarter a fifth of sample replaced– Sample contains five ‘waves’ of c.11,000 households

• Unclustered sample – small bias in Scotland: partial coverage north of

Caledonian Canal

UK-wide LFS

• Changed in 2006 from seasonal quarters (Mar-May, Jun-Aug etc) to calendar quarters (Jan-Mar, etc) in line with EU

• Publications – monthly Labour Market First Release (with

regional data)Headline figures employment, unemployment, economic activity

– quarterly supplement to Labour Market Trends

Employment, unemployment and economic inactivity rates by gender, Scotland, 1992-2006

0

10

20

30

40

50

60

70

80

90

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

Rat

e

Employment Male Employment Female Employment Male

Unemployment Female Inactive Male Inactive Female

Annual Population Survey

• SG funds an annual boost to LFS in Scotland – From 8,000 (unique households) to 23,000

• Calendar years 2004 to 2006 – plus quarterly rolling annual (Apr-Mar, Jul-Jun etc.)

• More reliable estimates for local authorities• Most headline estimates available each year for

LAs• Annual publication by SG

Future Developments

• Looking at including communal establishments

• Harmonisation of questions – UK Integrated Household Survey

• Pilot of ethnicity boost (this Summer, funded by SG)

Further Information

Data available from Nomis & data archive

Websites

SG:http://www.scotland.gov.uk/Topics/Statistics/Browse/Labour-Market

ONS:http://www.statistics.gov.uk/

Nomis:https://www.nomisweb.co.uk/Default.asp

E-mail: labour-market.statistics@scotland.gsi.gov.uk

The Scottish Crime The Scottish Crime and Justice Surveyand Justice Survey

About the Scottish Crime and About the Scottish Crime and Justice SurveyJustice Survey

Face to face household surveyFace to face household survey Random sample (some stratification, Random sample (some stratification,

clustering)clustering) Up to 2006:Up to 2006:

– Every 2 to 3 yearsEvery 2 to 3 years– Sample of 5,000Sample of 5,000

2008 onwards2008 onwards– ContinuousContinuous– Sample of 16,000Sample of 16,000

Rationale and limitationsRationale and limitations

Helping fill the gap in official statistics Helping fill the gap in official statistics on crimeon crime– Victim focusVictim focus– Unreported crimeUnreported crime

BUTBUT Private households onlyPrivate households only 16+ only16+ only Language and disability may be Language and disability may be

barriersbarriers

Counting crimeCounting crime

How much crime?How much crime? What type of What type of

crime?crime? How many victims?How many victims?

Prevalence of victimisation 1992 to 2005/06

0

5

10

15

20

25

1995 1999 2002 2003/4 2005/6

Year

Per

cen

t of p

opul

atio

n

Household crimes Personal crimes All SCVS crimes

Measuring attitudesMeasuring attitudes

Once secondary; Once secondary; becoming just as becoming just as important?important?

Keeping policy Keeping policy makers in tune makers in tune with people’s with people’s perceptionsperceptions

Measuring Measuring performanceperformance

Perceptions of change to the local crime rate

Less crime than 2

years ago, 8% More crime

than 2 years ago,

33%

About the same, 59%

SCVS 2006

Counting crime since 1970sCounting crime since 1970s Scottish Crime Survey (SCS), Scottish Crime and Scottish Crime Survey (SCS), Scottish Crime and

Victimisation Survey (SCVS) and Scottish Crime and Justice Victimisation Survey (SCVS) and Scottish Crime and Justice Survey (SCJS) Survey (SCJS)

Face to faceFace to face ‘‘British’ Crime Survey (BCS) since 1982 (now covering British’ Crime Survey (BCS) since 1982 (now covering

England and Wales only)England and Wales only)Face to faceFace to face

Northern Ireland Crime Survey (NICS)Northern Ireland Crime Survey (NICS)Face to faceFace to face

International Crime and Victim Survey/ European Crime and International Crime and Victim Survey/ European Crime and Victim Survey (ICVS)Victim Survey (ICVS)

Mixed methods (phone)Mixed methods (phone) United States National Crime Survey (NCS)United States National Crime Survey (NCS)

PhonePhone

Crime surveys in ScotlandCrime surveys in Scotland

1982 1988 1982 1988 Lowland Scotland covered in the ‘British’ Crime SurveyLowland Scotland covered in the ‘British’ Crime Survey Face to face, pen and paperFace to face, pen and paper19931993 1996 2000 2003 1996 2000 2003 Scottish Crime SurveyScottish Crime Survey Face to face, pen and paperFace to face, pen and paper 5,0005,000

REVIEW in 2003REVIEW in 200320042004 20062006 Scottish Crime and Victimisation SurveyScottish Crime and Victimisation Survey Telephone interviewing (CATI) in 2004 discontinuedTelephone interviewing (CATI) in 2004 discontinued Back to face to face, CAPI/CASI in 2006Back to face to face, CAPI/CASI in 2006 3,000 (2004) and 5,000 (2006)3,000 (2004) and 5,000 (2006)20082008 Scottish Crime and Justice SurveyScottish Crime and Justice Survey

The ‘new’ Scottish Crime and The ‘new’ Scottish Crime and Justice SurveyJustice Survey

A balance – meeting stakeholder needs, but A balance – meeting stakeholder needs, but affordable and using gold standard survey affordable and using gold standard survey methodmethod

Continuous – data every yearContinuous – data every year 16,000 per year – data at police force area 16,000 per year – data at police force area

levellevel Bigger sample – more modules means more Bigger sample – more modules means more

space for questionsspace for questions Face to face household interviews using Face to face household interviews using

CAPI/CASICAPI/CASI In the field in April 2008In the field in April 2008

SCJS todaySCJS today

Cognitive testing of new sections Cognitive testing of new sections Small scale pilot to test script and Small scale pilot to test script and

systemssystemso Main pilot completed February 2008:Main pilot completed February 2008:

o Stage 1 (800 quota)Stage 1 (800 quota)o Stage 2 (200 RPS)Stage 2 (200 RPS)

Next steps….into the field!Next steps….into the field! Data? Data?

Survey Harmonisation in Survey Harmonisation in Scotland Scotland

an overview of the theoretical and the an overview of the theoretical and the practicalpractical

The Background…….The Background…….

• Many Government surveysMany Government surveys• Independent/isolated developmentIndependent/isolated development• One purposeOne purpose

Which meant……Which meant……

• Poor comparability of survey outputsPoor comparability of survey outputs• Confusion for usersConfusion for users• Potential embarrassment for Scottish Potential embarrassment for Scottish

GovernmentGovernment• Timings offTimings off• Duplicated costs Duplicated costs

The Theory………The Theory………

Survey IntegrationSurvey Integration

OROR

Survey HarmonisationSurvey Harmonisation

Survey Integration……Survey Integration……

• Merge 4-5 key surveys into 1 integrated surveyMerge 4-5 key surveys into 1 integrated survey• Ask core set of economic and social questions of whole Ask core set of economic and social questions of whole

samplesample• Ask non-core questions across subsets of the sampleAsk non-core questions across subsets of the sample• Ask specialist topic questions in modules across subsets Ask specialist topic questions in modules across subsets

of the populationof the population• Re-brand component surveysRe-brand component surveys• e.g. ONS Integrated Household Surveye.g. ONS Integrated Household Survey

Drawbacks………

•Costly•Complicated•Case management system•Component Surveys may lose their identity•Slow to disseminate (like the Census)•No way back?

Survey harmonisation………

•Identify key surveys with harmonisation potential

•Embed standardised questions and outputs into each survey

•Each survey should adopt a core and modular structure

•Each survey should demonstrate unclustered sampling

•Policy driven dissemination from larger (pooled) samples

Benefits………

•More flexible than Integrated Survey model

•Surveys retain their unique identities and purpose

•Avoid duplication of effort and costs

•Topic experts e.g. Health Survey Statistician definitive authority on health related questions.

•Singing from same song sheet

•Communication within Scottish Government improved

•Harmonisation potential continues to be identified

Survey harmonisation appears to be the more effective model for Scotland’s large scale surveys although survey integration has not been ruled out.

A decision has yet to be made but in the meantime survey harmonisation continues.

The story so far…….

Main focus has been on Scotland’s 5 large scale Surveys:

•Scottish Household Survey•Scottish House Condition Survey•Scottish Crime and Justice Survey•Scottish Health Survey•Labour Force Survey

Other surveys involved GUS, FRS, SSAS and SEABs.

The Census 2011 is important in the harmonisation process.

What is practical?What is practical?

• Socio-economic coreSocio-economic core• Survey designSurvey design

Harmonisation potential:Harmonisation potential:

• Respondent permission/consent incl. follow-upsRespondent permission/consent incl. follow-ups• Survey letters and leafletsSurvey letters and leaflets• ProcurementProcurement• Complaint handlingComplaint handling• Data Access PanelData Access Panel

Socio-economic coreSocio-economic core

• Established a 20 question core by examining Established a 20 question core by examining common survey questions and important common survey questions and important questionsquestions

• Set up Scottish Harmonisation Working Group Set up Scottish Harmonisation Working Group to review each question and its outputs to to review each question and its outputs to build a harmonised question bank build a harmonised question bank

• Increasing focus on harmonised outputs Increasing focus on harmonised outputs – survey to survey across Scotlandsurvey to survey across Scotland– UK wide UK wide – European European – Comparability with economically similar countries Comparability with economically similar countries

P e o p l e a n d P l a c e s

Gender Age/DOB Marital Status Household Relationships Residence one year ago

I d e n t I t y

Cultural/Ethnic Group Religion

H e a l t h

Limiting long term illness/disability Self perception of general health

H o u s I n g

Tenure Accommodation Type Accommodation Size (no. bedrooms, overcrowding)

E m p l o y m e n t

Economic status (ILO) Full/part time Banded Household Income Attendance on government training scheme

E d u c a t I o n

Education status Highest qualification held

T r a n s p o r t

Access to car Mode of transport to work/education/school

Survey designSurvey design

• Adoption of a core and modular structureAdoption of a core and modular structure

• Unclustered samplingUnclustered sampling

• Methodology Forum Sub-group (MFSG)Methodology Forum Sub-group (MFSG) •samplingsampling

•weightingweighting

• time seriestime series

• sample additivity sample additivity

• omnibus etc.omnibus etc.

  Harmonisation Programme Work Flow  

SURVEY MANAGERS' NETWORK

----------SCOTTISH POPULATION SURVEYS CO-ORDINATING COMMITTEE  

METHODOLOGY FORUM GROUP  

SCOTTISH HARMONISATION WORKING GROUP

METHODOLOGY FORUM SUB-GROUP (SURVEY HARMONISATION)    

What next?What next?

• Website : Question bank Website : Question bank • Website : Surveys’ Customer Service sectionWebsite : Surveys’ Customer Service section• Promote core questions and survey design to smaller Promote core questions and survey design to smaller

Government surveys, local authorities, health boards etcGovernment surveys, local authorities, health boards etc• Adopt a naming convention for variablesAdopt a naming convention for variables• Administrative Sources …… data linking?Administrative Sources …… data linking?• Work on non-core questionsWork on non-core questions• Continue to add specialist topic modules onto existing Continue to add specialist topic modules onto existing

surveys rather than developing new surveys surveys rather than developing new surveys (e.g. culture and (e.g. culture and sport module)sport module)

• Decommission standalone surveys if they will sit within one Decommission standalone surveys if they will sit within one of the five large scale surveys as a module (e.g. HEPS)of the five large scale surveys as a module (e.g. HEPS)

• Survey Integration or Harmonisation?Survey Integration or Harmonisation?

Issues to considerIssues to consider

• Pooled survey data? Pooled survey data? – What will/can we do with it?What will/can we do with it?– Who will disseminate it?Who will disseminate it?– Will pooled sample be stored in data archive?Will pooled sample be stored in data archive?– Will results from it be classed a National Statistic?Will results from it be classed a National Statistic?– Do we need respondent permission to pool?Do we need respondent permission to pool?– Who/how do we check for double counting?Who/how do we check for double counting?

• Respondent Burden Respondent Burden – Case management system?Case management system?– Fieldwork capacity issuesFieldwork capacity issues