the 41st election
TRANSCRIPT
1
The 41st Election:
A Polling Retrospective
By Frank Graves
Ottawa, Ontario
September 22, 2011
Implications for Methodology and Democracy
2
Outline
1.0 Evaluating Our Polls and Industry Overall
2.0 Understanding the Gap between Final Polls and Re sults
2.1 First Hypothesis: Flawed Poll 2.2 Second Hypothesis: Late Shift2.3 Third Hypothesis: Turnout
3.0 Implications for Improving Polling Methodology a nd Reporting
4.0 Implications for Democracy, the Country, and Soc ietal Engagement
3
1.0 Evaluating Our Polls andOverall Industry
“…an unprecedented amount of polling (failed) to forecast accurately the key regional results that determined the shape ofthe House of Commons. For example, no company released results that were close to correct about Conservative support inOntario, which provides a third of the seats in the House.”
-Jennifer Espey
4
• Successes:
– Overall accurately / swiftly understands key themes of the
electorate
– First detected NDP surge / Bloc collapse
– Accurately predicted Liberal Party decline / Green Party collapse
– Correctly noted the new fault lines (generational, knowledge class)
• Failures:
– Final Conservative majority outcome was unexpected (but this
turns out to be a failure to forecast the voter turnout accurately)
Successes and Failures of EKOS Polling
5
39
35
36.4
34.5
0 10 20 30 40 50
Nanos (April 29th)
EKOS (April 29th)
Angus Reid (April 27th)
Environics (April 25th)
Summary of Pre-Election Polls – Friday, April 29 th
% Conservative
• Remarkable Consensus
– Late shift
– All biased?
– Turnout?
6
• EKOS’ federal vote intention polls are conducted using Interactive Voice Response (IVR) technology,
which allows respondents to enter their preferences by punching the keypad on their phone, rather than
telling them to an operator
– In an effort to reduce the coverage bias of landline only random digit dialing (RDD), we created a
dual landline/cell phone RDD sampling frame
• Advantages
– Perhaps closest to national population
– Minimises social desirability
– Cost-effective
– Higher reliability due to large sample sizes
• Disadvantages
– Higher non-response
– Survey must be shorter
– Some design limits
– Intrusiveness
– Reputation
IVR versus Other Polling Methodologies
7
4333
25 20
80
32 34 34
13
87
32 37
14
85
31 33 36
15
85
31
0
25
50
75
100
Actual Population IVR Landline CATI Land-line IVR Dual Frame Land-Mobile
12
36 35
186
2943
22
3
27
46
248
3140
21
0
25
50
75
100
Age
<25 25 to 44 65+45 to 64
High school College ImmigrantUniversity Born in Canada
Education Immigrant Status(%)
IVR versus CATI
BASE: IVR: Canadians; December 2010 (n=1,976) CATI: Canadians; January 2011 (n=3,009)
8
2.0 Understanding the Gap between Final Poll and Results
• Hypothesis & Test Results:
1) Flawed poll (bias) – measurement / sample
2) Late shift (“blue” Liberals in Ontario?)
3) “Get-out-the-vote” / Turnout
9
• Evidence shows polls were accurate in a number of key areas:
1. Polls correctly captured 2008 vote throughout campaign
2. Polls correctly captured 2011 vote post hoc (reproduced
election result)
3. Samples were close to census on all key parameters
• Question: Why would samples accurately give their 2008 and 2011
Conservative vote choices (behaviour) but erroneously give vote
intentions? (makes no sense)
– Conservative vote intention is lower in non-voters
• Prompting Green → mixed social desirability / turnout effect?
– Evidence shows lingering Green Party effect in both intentions
and behaviour
2.1 - First Hypothesis: Flawed Poll
10
18.2
26.3
6.8
10.0
1.2
15.1
27.1
7.49.7
37.7
2.8
37.9
0
10
20
30
40
CPC NDP LPC GP BQ Other
2008 Election Results EKOS Poll (April 29-May 1)
Other
BASE: Those who voted in 2008; April 29-May 1, 2011 (n=2,789)
Bias Test 1 – 2008 Vote Behaviour in Final Poll
Q. How did you vote in the federal election held in 2008?
Note: The data is based on those who say they voted in 2008. Our survey also finds that 11.7% of respondents did not
vote.
11
39.6
30.6
18.9
3.96.0
0.9
37.9
29.3
20.3
6.0 5.3
1.2
0
10
20
30
40
CPC NDP LPC GP BQ Other
May 2nd Election Results EKOS Poll (May 20-28)
Other
BASE: Those who voted on May 2nd; May 20-28, 2011 (n=2,209)
Note: The data is based on those who say they voted on May 2nd. Our survey also finds that 20.9% of respondents either
did not respond or did not vote.
Q. How did you vote in the most recent federal election, held on May 2nd?
Bias Test 2 – 2011 Vote Using Same Method 3 Weeks La ter
12
• Did Liberal Party supporters (particularly in Ontario) move to the
Conservative Party in fear of an NDP-led coalition?
– For this to be the case, there had to be a feedback loop to the
polls
– For this to be the case, late shifters had to move overwhelmingly
to the Conservative Party (particularly in Ontario)
• Evidence / Questions:
1) Was your final choice your original choice?
2) When did you make your final decision?
3) What was the reason for this shift?
4) Did you follow the polls?
5) Did the polls influence your choice?
2.2 - Second Hypothesis: Late Shift
13
73
21
6
Yes No DK/NR
Q. Were you originally leaning towards voting for a different party at the beginning of the election campaign?
BASE: Voters; May 20-28, 2011 (n=2,203) / Those who changed their mind; May 20-28, 2011 (n=414)
3
21
37
22
20
0 10 20 30 40
Q. [IF YES] When did you make your final decision regarding how you were going to vote?
Before the election campaign
After the leader’s debate
In the last week of the election campaign
On the day of the election
DK/NR
Timing of Final Decision
Of the 21% who changed their minds, 21% did so on the day of the election. Therefore, just 4.2% (21% of 21%) made their decision on election day.
14
3
25
29
43
0 10 20 30 40 50
3
8
13
76
0 20 40 60 80
Not at all (1-2)
Somewhat closely (3)
Very closely (4-5)
DK/NR
BASE: Those who voted on May 2nd; May 20-28, 2011 (n=2,203)
Not at all (1-2)
To a moderateextent (3)
To a great extent (4-5)
DK/NR
Q. Throughout the election campaign, how closely did you follow public opinion polls?
Impact of Public Opinion Polls
Q. To what extent did public opinion polls affect your final decision on Election Day?
15
43
32
6
19
0
10
20
30
40
50
60
It was time for a
majority government
We need to stay on a
sound economic
trajectory
Concerned about an
NDP-led coalition
DK/NR
Biggest Factor in Switching (Conservative Shifters Only)
BASE: Those who switched to the Conservative Party during the campaign; May 20-28, 2011 (n=99)
Q. What was the biggest factor in your final decision?
16
• With the endemic bias and late shift hypotheses
failing tests, we turn to turnout / get-out-the-vote
• Evidence:
– Analysis of demographic patterns of non-voters
– Regression / Log-linear analysis of voter
preferences of these groups
– Analysis of the cell-only population
→ mirrors these overall patterns
2.3 - Third Hypothesis: Get-Out-the-Vote
17
• While both sample and measurement biases inflate voting
intention and reported behaviour, the relative patterns are
revealing
• Age is crucial with non-voters (those under 25 are six times
more likely to not vote than those over 65)
• Even Gen X voters (i.e., 25-44) were twice as likely not to vote
as boomers
• Virtually all seniors claim to vote and vast majority actually do
• Age interacts with gender – young women even less likely to
vote
Results / Patterns of Non-Voting
18
1.5
6.4
6.0
21.0
31.2
33.9
0 10 20 30 40
Other
BQ
GP
LPC
NDP
CPC
0.9
6.0
3.9
18.9
30.6
39.6
0 10 20 30 40
Other
BQ
GP
LPC
NDP
CPC
2.5
7.0
9.3
24.3
32.1
24.8
0 10 20 30 40
Other
BQ
GP
LPC
NDP
CPC
EKOS final pre-election poll (April 29-May 1)
Actual results Estimated non-voter support
Voters (61.4%)
Non-Voters (38.6%)
Estimated Non-Voter Support
19
4
7
15
24
11
0 5 10 15 20 25
10
11
16
11
0 5 10 15 20
Non-Voters by Key Demographics
Q. How did you vote in the most recent federal election, held on May 2nd?
Age Phone type
% who did not vote % who did not vote
Overall
<25
25-44
45-64
65+
Overall
Cell only
Both cell & landline
Landline only
BASE: Canadians; May 20-28, 2011 (n=2,724)
20
• Non-voting is strongly linked to age
• Support for Conservative Party is strongly linked to age
• Cell-only use is strongly linked to age and non-support for
Conservative Party
• Cell-only use mirrors broader pattern of linkage of non-voting
and non-Conservative support
• Therefore, including cell phone-only in the sample produced
the larger gap
• Indeed, with the cell-only users out, our results are within the
margin of error of the election results
Impact of Cell Phone-Only Respondents
21
37.7
29.2
22.2
5.4 4.6
0.9
24.1
33.1
21.1
7.6
11.2
3.0
0
10
20
30
40
CPC LPC NDP GP BQ Other
Both landline and cellphone Cell phone only
Other
Cell Phone Only versus Cell/Landline Households
BASE: Decided voters; April 29-May 1, 2011 (n=1,836)
Note: The data on federal vote intention are based on decided and leaning voters only.
Q. If a federal election were held tomorrow, which party would you vote for?
22
3.0 Implications for Improving
Polling Methodology and Reporting
23
• Is the link between election outcome and final polls a good yardstick of
polling quality?
– Increasingly, no. Why not?
→ growth of non-voters (now over 50% of those under 45)
→ strong correlation between non-voting and non-support of
incumbent
• How to “fix” the forecast:
– Commitment index
– Remove <45
– Remove new voters / Eliminate non-voters
What Does This Mean?
24
• Our commitment index is an aggregation of five factors:
– the ease with which a respondent revealed/declared their voting preference
– the self-rated level of enthusiasm with current voting choice
– the expressed intensity of attachment to current voting choice
– the self-expressed likelihood of actually voting on May 2nd
– the respondent’s willingness to consider other parties
Commitment Index
25
39.6
30.6
18.9
3.96.0
0.9
38.7
31.9
18.1
5.0 5.5
0.8
0
10
20
30
40
CPC NDP LPC GP BQ Other
May 2nd Election Results EKOS Voter Commitment Index (April 29-May 1)
Other
BASE: Decided voters; April 29-May 1, 2011 (n=2,876)
Q. If a federal election were held tomorrow, which party would you vote for?
EKOS Commitment Index versus Actual Results
Note: The data on federal vote intention are based on decided and leaning voters only.
26
• We have no assurances these adjustments would work again
(e.g., they would have blown up in US 2008 election)
• Commitment / Enthusiasm may be crucial (taps the emotional
domain)
– In particular, older, conservative voters are more
enthusiastic / committed than older, non-conservative
voters
• Self-rated likelihood to vote means nothing
• No one would seriously argue that ignoring the younger half of
the population or new market entrants is a sound research
method for any other area of the private or public centre
What Does This Mean?
27
44
15
33
4 31
48
13
32
3 4
38
18
34
52 21
0
10
20
30
40
50
CPC NDP LPC GP BQ Other
Overall Men Women
Other
BASE: Decided voters; April 29-May 1, 2011 (n=2,876)
Q. If a federal election were held tomorrow, which party would you vote for? (highly enthusiastic voters only)
Interaction of Enthusiasm, Gender, and Party Suppor t
Note: Results are delimited to those who say they are “highly” enthusiastic regarding their current choice (i.e., 6 or 7 on a 7-point scale)
28
• More circumspect final forecast
• Greater awareness that nearness of final poll to
electoral outcome is an increasingly flawed indicator
of polling quality (it was a good test in the past)
• Shift media focus away from vanity sweepstakes of
final prediction to deeper understanding of campaign
dynamics and implications for future
• Better understanding of turnout
– Can’t assume future will resemble the past
Recommendations
29
4.0 Implications for Democracy,
the Country, and SocietalEngagement
30
• The new generational chasm
• Most of those under 45 aren’t voting (more so than in the past)
• This may be in part a product of conscious political strategy (suppression)
• Profound questions about implications for a societal “succession” strategy
• Profound differences between older and younger Canada
– Composition• Diversity
• Education
• Digital world view/ horizontality
– Values and Interests (younger Canada tilt)• Knowledge and skills
• Climate and post Carbon economy
• Cosmopolitan
• Less security-focussed
Implications for Democracy and Societal Engagement (1/2)
31
• Unusual demographic skew today. Median age is 42, up from 26 at
our centennial.
– Gerontocracy? Political agenda reflecting imagined / exaggerated
fears of older Canada? Good politics but good policy?
– Healthy economies balance grace and enthusiasm; prudence and
exuberance
– Brewing generational storm?
– Lesson of election polling should not be to ignore those voices
which weren’t registered – if anything, this heightens need to
include “unheard” voices
– Shift from obsessive focus on non-existent horse race to informed,
reflective, and representative national conversations on key issues
Implications for Democracy and Societal Engagement (2/2)
32
43
17
39
1
58
15
25
2
0
10
20
30
40
50
60
Dissatisfied (1-2) Neither (3) Satisfied (4-5) DK/NR
Overall Youth (<25)
Satisfaction with Election Results
BASE: Those who voted on May 2nd; May 20-28, 2011 (n=2,203)
Q. Overall, how satisfied were you with the results to the last election?
33
• Frank Graves. Accurate Polling, Flawed Forecast: An Empirical Retrospective on Election
41 (June 2011)
http://www.ekospolitics.com/wp-content/uploads/accurate_polling_flawed_forecast.pdf
• Frank Graves. The Great Canadian Poll-Off (March 2011)
http://www.ekospolitics.com/wp-content/uploads/the_great_canadian_poll_off.pdf
• EKOS Research Associates. Interactive Voice Response: The Past, Present, and Into the
Future (January 2011)
http://www.ekos.com/admin/articles/2011-01-21-MRIA.pdf
• EKOS Research Associates. EKOS’ Observations on MRIA Study – Canadian Online
Panels: Similar or Different? (January 2010)
http://ekos.com/admin/articles/MRIA-Comparison-Panel-Study-2010-01-27.pdf
For Further Reading
34
FOR MORE INFORMATION:
Frank Graves, President
EKOS Research Associates
p. 613.235.7215