Social Desirability Social Desirability EffectsEffects

in Survey Researchin Survey Research

Gov 1011Gov 1011

Shauna L. ShamesShauna L. Shames

shames@fas.harvard.edushames@fas.harvard.edu

Social Desirability: Def. & ExamplesSocial Desirability: Def. & Examples What Weisberg would classify as "Measurement Error Due What Weisberg would classify as "Measurement Error Due

to Respondents," a sub-category of "Response Accuracy" to Respondents," a sub-category of "Response Accuracy" (see Figure 2.1, from Weisberg Chapter 2, p. 19(see Figure 2.1, from Weisberg Chapter 2, p. 19

Phillips and Clancy (1972) definition:Phillips and Clancy (1972) definition:• "Broadly conceived, 'social desirability' as a response "Broadly conceived, 'social desirability' as a response

determinant refers to the tendency of people to deny socially determinant refers to the tendency of people to deny socially undesirable traits or qualities and to admit to socially undesirable traits or qualities and to admit to socially desirable ones" (923).desirable ones" (923).

Def. & Examples, Cont’dDef. & Examples, Cont’d

Sociological literature divides into two Sociological literature divides into two components (see Phillips and Clancy (1972): components (see Phillips and Clancy (1972): 'Some Effects of 'Social Desirability' in Survey 'Some Effects of 'Social Desirability' in Survey Studies," in Studies," in AJS, AJS, and also earlier work by the and also earlier work by the same authors):same authors):• - trait desirability (personality construct, has to do with - trait desirability (personality construct, has to do with

how the respondent describes self to the interviewer or how the respondent describes self to the interviewer or within the survey)within the survey)

• - need for social approval (quality of the measurement - need for social approval (quality of the measurement items – has to do with the relationship between the items – has to do with the relationship between the interviewer/survey giver and the respondent, and the interviewer/survey giver and the respondent, and the respondent's quest for social approval from interviewer respondent's quest for social approval from interviewer or survey giver)or survey giver)

Roots of ConceptRoots of Concept The idea of social desirability traces its roots back to the The idea of social desirability traces its roots back to the

sociology and psychology concept of response sets, "where the sociology and psychology concept of response sets, "where the individual responds to item content in such a way as to portray individual responds to item content in such a way as to portray himself in other than a true light (e.g. responding to items in himself in other than a true light (e.g. responding to items in terms of the social desirability of the answers)" (O'Neill 1967, terms of the social desirability of the answers)" (O'Neill 1967, 95).95).

Response set: "a personal tendency to respond in a specified Response set: "a personal tendency to respond in a specified way within a testing or interview situation" (Smith 1967, 87).way within a testing or interview situation" (Smith 1967, 87).

Roots, Cont’dRoots, Cont’d Converse in the early 1960s shocked political science (and Converse in the early 1960s shocked political science (and

himself) by finding that most people's opinions are ill-himself) by finding that most people's opinions are ill-considered, inconsistent, and pretty ignorant – in his words, considered, inconsistent, and pretty ignorant – in his words, lack "constraint" (Converse 1964).lack "constraint" (Converse 1964).

Recently, John Zaller has thrown a similar monkey wrench Recently, John Zaller has thrown a similar monkey wrench into the gears of public opinion research, with the major into the gears of public opinion research, with the major contention of his 1992 book (contention of his 1992 book (The Nature and Origins of The Nature and Origins of Mass OpinionMass Opinion) being that public opinion surveys do not ) being that public opinion surveys do not reveal deep underlying preferences of the public, but rather reveal deep underlying preferences of the public, but rather are "top of the head" replies by people trying to come up are "top of the head" replies by people trying to come up with answers to survey questions!with answers to survey questions!

Some Effects of Soc. DesirabilitySome Effects of Soc. Desirability

Some of the sociological and political science social desirability studies Some of the sociological and political science social desirability studies reveal, for instance, that people will say they voted when they haven't, will reveal, for instance, that people will say they voted when they haven't, will inflate their reported incomes, will describe themselves in socially-inflate their reported incomes, will describe themselves in socially-sanctioned ways (such as saying they are happier than they are in their sanctioned ways (such as saying they are happier than they are in their marriages), and will edit politically "incorrect" responses (such as marriages), and will edit politically "incorrect" responses (such as prejudice)prejudice)

Effects, Cont’dEffects, Cont’d Several studies by Bishop and co-authors found that respondents are Several studies by Bishop and co-authors found that respondents are

differentially likely to offer opinions on fictitious issues, when asked to do differentially likely to offer opinions on fictitious issues, when asked to do so in a survey. Not exactly that all respondents will offer such opinions, so in a survey. Not exactly that all respondents will offer such opinions, but that certain people are more likely to do so than others (included lower but that certain people are more likely to do so than others (included lower status or less educated respondents). So people who are sensitive to being status or less educated respondents). So people who are sensitive to being thought dumb or uneducated are more likely to overcompensate, by thought dumb or uneducated are more likely to overcompensate, by offering opinions on nonexistent political issues (see, for example, Bishop offering opinions on nonexistent political issues (see, for example, Bishop 1986).1986).

Cross-comparative survey of women in US and Costa Rica – has to do with Cross-comparative survey of women in US and Costa Rica – has to do with cultural connotation of topic, so saying "Have you had as many children as cultural connotation of topic, so saying "Have you had as many children as you want" in Costa Rica implies that if a woman says yes, she's going to you want" in Costa Rica implies that if a woman says yes, she's going to have an abortion (even if she really does have enough for her) (Class have an abortion (even if she really does have enough for her) (Class Notes, 3/1/07)Notes, 3/1/07)

Acquiescence: the tendency to say yes repeatedly (generally believed Acquiescence: the tendency to say yes repeatedly (generally believed that Latinos have more of a tendency to acquiesce to survey questions, that Latinos have more of a tendency to acquiesce to survey questions, because of cultural socialization – Class Notes, 3/1/07)because of cultural socialization – Class Notes, 3/1/07)

1980s study on whether racism has declined in America: puzzle that white 1980s study on whether racism has declined in America: puzzle that white Americans say they are less racist on surveys, but strongly resist programs Americans say they are less racist on surveys, but strongly resist programs designed to achieve racial equality, such as school desegregation and designed to achieve racial equality, such as school desegregation and affirmative action (McConahay, Hardee, and Batts 1981)affirmative action (McConahay, Hardee, and Batts 1981)

Effects of Study ModeEffects of Study Mode In a recent study, Presser and Stinson (1998) found that people were about In a recent study, Presser and Stinson (1998) found that people were about

a third less likely to report weekly attendance of religious services when a third less likely to report weekly attendance of religious services when the survey was self-administered rather than conducted by an interviewer.the survey was self-administered rather than conducted by an interviewer.

Mode Effects, Cont’dMode Effects, Cont’d This turns out to have major impacts on our understanding This turns out to have major impacts on our understanding

of trends in religious service attendance:of trends in religious service attendance:

"This difference in measurement approach… alter[s] the "This difference in measurement approach… alter[s] the observed trend in religious attendance over time: In contrast to observed trend in religious attendance over time: In contrast to the almost constant attendance rate recorded by conventional the almost constant attendance rate recorded by conventional interviewer-administered items, approaches minimizing social interviewer-administered items, approaches minimizing social desirability bias reveal that weekly attendance has declined desirability bias reveal that weekly attendance has declined continuously over the past three decades" (137).continuously over the past three decades" (137).

Weisberg suggests (see Chap. 12) that different modes of Weisberg suggests (see Chap. 12) that different modes of survey call up different “scripts” and therefore invite survey call up different “scripts” and therefore invite different social behaviors different social behaviors

• Face-to-face interview in a person’s home calls up an Face-to-face interview in a person’s home calls up an “interviewer as guest” script, and respondents act more polite. “interviewer as guest” script, and respondents act more polite. Telephone interviews call up a “solicitor” script, while self- Telephone interviews call up a “solicitor” script, while self-administered paper surveys summon a “multiple-choice exam” administered paper surveys summon a “multiple-choice exam” script, both of which invite possible negativity.script, both of which invite possible negativity.

““Fixes” for Soc. Desirability Effects Fixes” for Soc. Desirability Effects in Surveysin Surveys

1.) Change mode of survey (self-administered surveys have 1.) Change mode of survey (self-administered surveys have greater chance of what Steven Colbert calls "truthiness")greater chance of what Steven Colbert calls "truthiness")• For a sensitive survey, want self-administered (could be web or For a sensitive survey, want self-administered (could be web or

paper)paper)

2.) Carefully reword questions to (as far as possible) remove any 2.) Carefully reword questions to (as far as possible) remove any implications or cuesimplications or cues

Fixes, Cont’dFixes, Cont’d 3.) Deflect attention away from respondent (not 3.) Deflect attention away from respondent (not

"what do YOU think about racial conflict," but "what do YOU think about racial conflict," but "what does your NEIGHBOR think")"what does your NEIGHBOR think")

• example: recent WHP studies (see most recent example: recent WHP studies (see most recent study at: study at: http://thewhitehouseproject.org/v2/press/2007/http://thewhitehouseproject.org/v2/press/2007/February/20070216-PressRelease.htmlFebruary/20070216-PressRelease.html ) )

ReferencesReferences Bishop, George F.; Tuchfarber, Alfred J.; and Olendick, Robert W.: 1986. Bishop, George F.; Tuchfarber, Alfred J.; and Olendick, Robert W.: 1986.

"Opinions on Fictitious Issues: The Pressure to Answer Survey Questions," "Opinions on Fictitious Issues: The Pressure to Answer Survey Questions," Public Opinion QuarterlyPublic Opinion Quarterly Vol. 50, No. 2 (Summer): 240-50. Vol. 50, No. 2 (Summer): 240-50.

Converse, Philip: 1964. "The Nature of Belief Systems in Mass Publics,: in Converse, Philip: 1964. "The Nature of Belief Systems in Mass Publics,: in David Apter, David Apter,

ed., ed., Ideology and Discontent. Ideology and Discontent. New York: The Free Press, pp. 206-261.New York: The Free Press, pp. 206-261. McConahay, John B.; Hardee, Betty B.; and Batts, Valerie: 1981. "Has McConahay, John B.; Hardee, Betty B.; and Batts, Valerie: 1981. "Has

Racism Declined in America? It Depends on Who is Asking and What is Racism Declined in America? It Depends on Who is Asking and What is Asked," Asked," The Journal of Conflict Resolution, The Journal of Conflict Resolution, Vol. 25, No. 4 (December): pp. Vol. 25, No. 4 (December): pp. 563-579.563-579.

O'Neill, Harry W.: 1967. "Response Style Influence in Public Opinion O'Neill, Harry W.: 1967. "Response Style Influence in Public Opinion Surveys," Surveys," Public Opinion Quarerly Public Opinion Quarerly Vol. 31, No. 1 (Spring): 95-102.Vol. 31, No. 1 (Spring): 95-102.

References, Cont’dReferences, Cont’d Phillips, Derek L. and Clancy, Kevin J.: 1972. "Some Effects of 'Social Phillips, Derek L. and Clancy, Kevin J.: 1972. "Some Effects of 'Social

Desirability' in Survey Studies," Desirability' in Survey Studies," American Journal of Sociology American Journal of Sociology Vol. 77, No. Vol. 77, No. 5 (March): 921-40.5 (March): 921-40.

Presser, Stanley and Stinson, Linda: 1998. "Data Collection Mode and Presser, Stanley and Stinson, Linda: 1998. "Data Collection Mode and Social Desirability Bias in Self-Reported Religious Attendance," Social Desirability Bias in Self-Reported Religious Attendance," American American Sociological ReviewSociological Review Vol. 63, No. 1 (February), pp. 137-145. Vol. 63, No. 1 (February), pp. 137-145.

Smith, David Horton: 1967. "Correcting for Social Desirability Response Smith, David Horton: 1967. "Correcting for Social Desirability Response Sets in Opinion-Attitude Survey Research," Sets in Opinion-Attitude Survey Research," Public Opinion Quarterly Public Opinion Quarterly Vol. Vol. 31, No. 1 (Spring): pp. 87-94.31, No. 1 (Spring): pp. 87-94.

Watson, P.J.; Morris, Ronald J; Foster, James E.; and Hood, Ralph W., Jr.: Watson, P.J.; Morris, Ronald J; Foster, James E.; and Hood, Ralph W., Jr.: 1986. "Religiosity and Social Desirability," 1986. "Religiosity and Social Desirability," Journal for the Scientific Study Journal for the Scientific Study of Religion of Religion Vol. 25, No. 2 (June), pp. 215-232.Vol. 25, No. 2 (June), pp. 215-232.

Weisberg, Herbert F.: 2005. Weisberg, Herbert F.: 2005. The Total Survey Error Approach. The Total Survey Error Approach. Chicago: Chicago: University of Chicago Press.University of Chicago Press.

Zaller, John: 1992. Zaller, John: 1992. The Nature and Origins of Mass Opinion. The Nature and Origins of Mass Opinion. New York: New York: Cambridge University Press.Cambridge University Press.