Client logo

Telephone Polls: Opportunities and Pitfalls

Martin Boon, ICM Research

Telephone polls: Opportunities and pitfalls

The catalyst: 1992

• The 1992 election was a defining point in political opinion polling in the UK.

• 1992: five main pollsters, one data collection methodology, one wrong outcome.

• The scene is set for the pollsters’ methodological debate.

Telephone polls: Opportunities and pitfalls

• The most visible aspect of methodological change: the switch to telephone.

• The basic method: RDD sampling using approximately 12,000 telephone numbers.

Telephone polling: a short history & method

* 2005 Evening Standard prediction poll by telephone

1995

1997 2003

1999

NOP

2006

2008

Telephone polls: Opportunities and pitfalls

Telephone polling: the outcome

• ICM final prediction telephone poll average error in 1997: 1.2% average error. Peter Kellner, 2nd May 1997: “Congratulations. The Evening Standard is on the streets….”

• It the only prediction poll to understate the Labour lead in any General Election since 1983.

• In 2001, the best telephone poll had an average error of 1.6%.

• In 2005, we saw the great success of NOP in producing a statistical bulls-eye (average error of 0.25%).

• Other types of telephone polling have also produced accurate predictions. Polls across marginal seats, constituency polling and at least one poll in Scotland did well in estimating the actual result.

Telephone polls: Opportunities and pitfalls

But is it about the data collection method?

• The introduction of a telephone methodology is but one aspect of the evolution of political opinion polling.

• Over time, data collection change was reinforced and refined with wording change, turnout filters, past vote weighting and adjustment of partial refusers.

• Most of these changes had a greater impact on vote intentions than data collection switchover. The switch to telephone should not be seen as the single most important contribution to the development of polling methods.

• By the 1997 General Election, Gallup had also switched to random telephone methods but had not innovated beyond that. Their campaign polls looked more like face-to-face polls rather than ICM’s telephone equivalent:

Telephone polls: Opportunities and pitfalls

But is it about the data collection method?

Con%

Lab%

LD%

Other%

Lead%

Harris (6) 30 51 13 6 21

MORI (8) 30 51 13 6 21

NOP (6) 29 49 15 7 20

Gallup (15) 31 50 13 6 19

ICM (6) 33 45 16 6 12

Result 31 44 17 7 13

Average 1997 campaign poll ratings, by company

Telephone polls: Opportunities and pitfalls

The BIG issues that confront us today

• Non-Internet opinion polls in Britain suffer from a persistent, long standing Labour bias. The great challenge for pollsters is to remove or at least account for pro-Labour bias.

• On a standardised measure of deviation across these polls we find that Labour recall is the least volatile (1.13), followed by the Liberal Democrats (1.2) and the Conservatives most accurate but most volatile (1.27).

• i.e recall for Labour may be hopelessly wrong, but at least it’s wrong in the most consistent way.

8.7% 1.8% 6.6%

Average faulty recall by party, ICM polls Nov 08 – Dec 09

Telephone polls: Opportunities and pitfalls

Why do telephone polls overstate Labour?

• Suggestions include:

• Inaccurate demographic profiles - but demographics have a negligible impact on voting; demographically representative surveys do not mean they are politically representative.

• We get through to too many Labour voters for systematic reasons:

• low response rates

• differential refusal

• public sector workers

• spiral of silence

Telephone polls: Opportunities and pitfalls

The practical difficulties

• There can be little doubt that telephone polling is becoming harder, and will get harder still.

• An increasingly unresponsive public. N = 1,000 depends on c.12,000 RDD records. ICM telephone omnibus alone conducts 100,000 interviews per year, implying that on a 12:1 ratio our pool of the general public is being depleted by over a million households per year on a worst case scenario.

• Sales, marketing and research drive more and more to TPS and tele-screening gadgets.

• 21 million UK households, 12% (2.5m) are mobile only. Cardiff (29%) is the mobile capital of the UK (August 2009). How do we contend with mobile only households and future possible non-geographic landline numbers?

• Non-response among specific groups. We typically achieve 64% of the required interviews with 18-24 year olds.

Telephone polls: Opportunities and pitfalls

Is it a case of papering over the cracks?

Telephone polls: Opportunities and pitfalls

Telephone polls are still up to the job!

• When all is said and done we have to be rational and make decisions on what we think is the best methodology over the medium to long term. We still believe that telephone trumps online.

• The record of telephone polls does stand up by and large when tested in electoral conditions.

• RDD sampling combined with quota methods is still able to draw a representative sample and demographic weighting is no big deal.

• Past vote weighting, or other strategies can and do contend with the political balance issue.

• On the practical level our telephone opinion research continues to be successful and doesn’t currently suffer from unsustainable refusal rates.

• Telephone polls can be undertaken quickly and cost effectively.

Telephone polls: Opportunities and pitfalls

Innovation

Telephone polls: Opportunities and pitfalls

Crystal ball gazing….

• Innovation was the key when it came to dealing with 1992 issues. ICM and others innovated within the telephone interviewing framework; YouGov and others innovated with the online solution.

• Telephone polling will have to change in order to meet the conditions the great British public impose on us. In 20 years time what might a telephone poll look like?

• We can speculate that any of the following might contribute to orthodox opinion research:

• SMS polling (it’s here now)• Mixed method polling• Mobile-only random samples • Telephone panels

• We reserve the right to make any changes that we think will help to improve accuracy – so long as we are transparent about what we do, then others can judge both in terms of suitability and effectiveness.