NDC Household Survey 2006

An overview

Rachel Williams, Jessica Vince and Leon Page

2

Session objectives

Background information about this year’s household survey

Review data outputs available and give practical tips on reading and interpreting data

The Extranet (www.ipsos-mori.com/ndc)

Sources of help at Ipsos MORI

3

Background to the household survey

4

Overview

Survey aims to provide information on the key outcomes from the NDC programme and to show change over time

Third survey in the series

initial survey in 2002

2004 and 2006 surveys are longitudinal – going back to as many of the original 2002/4 respondents as possible plus a cross-sectional top-up sample

Ipsos MORI/GfK NOP conducted 15,792 interviews between May and October 2006 – c.400 in each NDC area

National benchmarks - from existing surveys and new Omnibus survey – are NDC areas ‘catching up’?

5

Changes for 2006

Sample size reduced to 400 interviews

Longitudinal and top-up addresses issued in the same proportion

as 2004

Questionnaire length reduced:

25 minute core questionnaire

Partnership specific questions optional

No research among movers, beneficiaries or business

6

Questionnaire

7

Core questionnaire – asked in all NDC areas

Core questionnaire covers each “theme” in national evaluation

Housing and the local environment

Crime

Worklessness

Education

Health

Plus profile information, basic income questions and general

perceptions of area/community

8

Core questionnaire – 2

Wherever possible questionnaire kept identical to 2002/04 to allow

changes in views to be compared

Designed with NET – following outcome review and review following each

wave

Full pilot in similar regeneration areas each wave

Some questions ‘follow-up’ answers given by respondents in 2004 e.g. Last

time you were very worried about being burgled now you are not very

worried – why have your views changed?

Includes national benchmarks to compare against the country as a whole

This year each partnership could add 5 minutes worth of their own

questions

9

Sampling

10

Why do we sample?

Not feasible to question all residents of a particular area – e.g. 17,000 residents in the East Brighton NDC area alone

Interview a smaller number on the basis that they share the same characteristics as the population generally

Population…

Sampling is making an inference about a…

…from a… …Sample

Works but relies on strict procedures

11

Bigger is better…but size isn’t everything

S o u rc e : M O R I

Bigger samples are better - but not proportionately

0

5

10

15

20

0 100 200 300 400 500 600 700 800 900 1000 1100Sample size

12

Household survey – two samples

Return to respondents interviewed in 2004

Longitudinal sample

New addresses selected at random to top up to 400 per area

Top-up sample

13

Longitudinal sample

Longitudinal

Return to individuals interviewed in 2004 and attempt to interview them

If named respondent has moved or died then randomly select a substitute

10,770 successful interviews at original addresses

14

Top–up sample

Random selection of addresses from within each of the NDC partnerships

Random selection of one property/dwelling or household unit at each sampled address

Random selection of one adult within each selected household (5,022 successful interviews)

Top-up sampling involves a number of steps…

Means every person has an equal chance of being interviewed

15

Sample structure - example

EXAMPLE – Salford NDC

400 original 2004

addresses sent out

Interviewed 283 people at these

addresses

238 original respondents

45 new respondents

Select 237 new addresses

- 124 new respondents

407 interviews in total – 238 longitudinal and 169 top-up

16

Fieldwork

17

Fieldwork

Fieldwork May - October 2006

Full briefings for all interviewers (28 sessions) dummy interviews before starting work where possible using the same interviewers as 2002/4

Advance letter and FAQ sheet sent to respondents

Interviews conducted using Computer Assisted Personal Interviewing (CAPI)

Minimum of 6 calls, including two at weekend or evenings, plus further re-issues.

Quality control - back-checking and accompaniment

Over 370 interviews conducted in other languages

Response rates improved on 2002 and 2004

18

Household survey outputs

19

Household survey outputs

Range of outputs provided by Ipsos MORI team Key indicators (most accessible) Topline questionnaires Computer tabulations (simple sub-group analysis) SPSS dataset (most ‘technical’)

Will suggest how and when you might want to use these

Extranet

20

What are key indicators?

Show summaries of some key questions from the survey

Easy to read if showing to non-researchers in your NDC

Useful if you want a quick overview of the results and change over time

Can be used to create a summary of findings for internal use e.g. Board meetings etc.

Cover all the theme areas (Crime, Housing etc)

21

What are key indicators?

Columns giving partnership and aggregate data for all three waves, and national benchmark

Summary of all those very/fairly satisfied with accommodation. Does not show % dissatisfied

Theme heading

22

What are toplines?

Show all of the questions from the survey in the order these were asked on the questionnaire

All responses to each question are shown (e.g. very and fairly satisfied/dissatisfied) not just summaries

Questions are grouped by theme to make them easier to find – e.g. CR questions (CR1, CR2 etc) are from the Crime theme

Use the marked-up questionnaire if you want overall results but do not want to analyse by sub-groups e.g. men vs. women

Fairly easy to read for those not experienced in research

23

What are toplines?

Theme heading and question number Full question text

List of all responses to the question

24

What are toplines?

Filter – an indication of who is asked the question

Base – a definition of who was asked the question and how many people this includes

National source – most recent benchmarking figures available (mostly for England)

25

Toplines - Interpretation

Number of pitfalls to be aware of when reading and interpreting toplines:

Check the bases underneath the question – is everyone being asked or not?

Some demographic questions are based on all household members rather than all respondents. Gives more accurate information on age, gender and work status

Read the introductory page!

26

Topline document is now in Excel rather than Word

Toplines - functionality

Advantages Display only question text Display only benchmarked questions Quickly calculate change between years using formulas

=IF(H17="*",0,H17) etc Run charts from the data

Will be available in PDF version

Working on formulas to produce automated significance testing (TBC)

27

Statistical reliability - 1

Not all differences between 2002, 2004 and 2006 data will be significant

E.g. if satisfaction with area has increased by 2 percentage points between 2004 and 2006 this will not mean that there has been a significant increase in satisfaction

Whether changes are significant also depends on how many people were asked a question

The fewer people asked a question, the greater the difference will have to be for this to be significant

E.g. more difficult to find significant difference where 50 respondents have been asked compared with where 400 have been asked

28

Statistical reliability - 2

Where all respondents have answered a question need differences of:

8 percentage points or more between 2002, 2004 and 2006 figures to be sure they represent actual change and are NOT due to chance

6-7 percentage points between individual NDCs and the aggregate or national benchmark figures

Can use our “ready reckoner” – available on the Extranet

An example…

29

Statistical reliability calculator

Testing change in proportion feeling unsafe between 2004 and 2006 for Partnership X

30

What are computer tabulations?

All responses to the survey are broken down by around 40 different sub-groups

Sub-groups are respondents who are grouped together in the data because they have something in common. This may be demographic (e.g. their age) or attitudinal (e.g. they want to move out of the area)

Examples of sub-groups are: Men (demographic) Those who earn less than £100 per month (demographic) Those who have heard of NDC (attitudinal)

31

What are computer tabulations?

Why would you want to use these?

You may want to see if all sections of the population are benefiting equally from NDC

E.g. fear of mugging may be going down overall, but are any groups being left behind?

Can compare 2006 results with 2004 results – young people may be 2% less likely to be mugged than in 2004 compared with a drop of 15% overall

Can also use for baseline/monitoring data – are Black residents more likely than average to be earning £100 a month or less?

32

What are computer tabulations?

Question on tenure – who are most likely to be private renters?

16-34 year olds significantly more likely to rent privately (text shown in bold)

33

What are computer tabulations?

BUT: Crucial not to compare sub-groups with small bases as these are unreliable

Only 13 people in overall base

Can’t compare refugee status of men vs. women

34

Computer tables - significance testing

Letters used to identify which results are significantly different from each other

“x” = total column

Results for male respondents are in column “a”, females in “b”

35

Computer tables - significance testing

Each column is tested against the total to see whether the differences in results are significantly different

a letter under a result indicates it is

Fewer in column “a” (men), column “e” (45-64 year olds) and column “f” (aged 65+) are lone parent families than average

More in column “b” (females) and column “d” (35-44 year olds) are lone parent families than average (i.e. column “x”)

36

A few things to watch…

Some tables run over two pages because the list of possible answers is so long – i.e. 4 pages per question (e.g. WO1)

As well as the actual answers that can be given, also include derived summaries at the end of the list (eg WO1 ILO Unemployed)

There may be tables for the same question but with different bases. E.g. at WO1 there are tables based on all respondents and also on all working age respondents

Sometimes have ‘summary tables’ where a battery of questions are asked

There are a few derived tables e.g. HO3 cross-tabulated by HO4

37

SPSS data

A software program that allows you to conduct your own analyses (Statistical Package for the Social Sciences)Allows you to do a wide range of analyses that are not included on the Computer TablesE.g. You could find out the characteristics of ‘Trapped residents’ – those who want to move but do not think they will, by cross-tabulating two survey questions against each otherEvery NDC has their own datasetFile contains some additional derived variables eg workless householdsBUT – your NDC may not have bought SPSS software so you may not be able to use this data (LA may have it though …)Ask the Ipsos MORI team if you need any help with using SPSS

38

A note on weighting

Two sets of weights applied to the survey data

Selection weighting to correct for unequal selection probabilities (every person should have equal chance of selection)

Profile weighting to ensure that survey is representative of the population as a whole (based on Census)

All data presented in the survey outputs is WEIGHTED

EXCEPT, the final column on the computer tables - this presents the raw data

Correct weights must also be applied to the SPSS data

NB weighted samples are less accurate – need to bear in mind when checking for significant differences

39

Extranet

40

Extranet - 1

You may have used this with the 2002 and 2004 data

Analysis tool taken off-line recently to add the new 2006 data

Can still access survey documentation

2006 data will be available in February next year

41

Extranet - 2

Extranet allows you some of the flexibility of an analysis programme like SPSS but easier to use

Can analyse your partnerships’ results by demographic factors such as age or gender

Can compare your results against those of other relevant partnerships and against the aggregate data set

Can filter results e.g. the findings just for women aged 16-24

Can compare change in data over time

42

Extranet - 3

An on-line resource containing all NDC data

Run your own tables on issues of interest

Download toplines, key findings and computer tables

Data available for all partnerships

Possible to compare yourself with other “local” NDCs or those with similar characteristics

Each partnership has their own log in

Benchmarking data and links to useful sites

Information about the household survey Background to each wave

Glossary of research terms and training slides

43

Some on-screen examples

44

Using the data

45

Using the data

Background information/reality check along with other sources of data

eg if feelings of safety have increased how does this relate to crime statistics/what the police are saying locally. 

Partnership specific questions used to evaluate specific projects

Post coding data to provide neighbourhood level information

Wider area benchmark surveys

46

Thank youFor further information or help please contact:

jessica.vince@ipsos-mori.com (020 7347 3148)

leon.page@ipsos-mori.com (020 7347 3152)

hayley.muller@ipsos-mori.com (020 7347 3252)