RC-33 Eighth International Conference on Social Science Methodology

Timothy P. Johnson and Paul J. Lavrakas

AAPOR’s Transparency AAPOR’s Transparency InitiativeInitiative

1

Problem:Problem: There is not There is not enough transparency in the enough transparency in the Social and Behavioral Social and Behavioral Sciences.Sciences.Premise 1: Too often, there is

inadequate transparency of research methods and statistical methods when the results of social and behavioral science research are disseminated, especially by non-academic sectors.

Premise 2: This is a serious detriment to progress in these scientific domains.

2

Openness is fundamental to Openness is fundamental to scientific inquiry.scientific inquiry.

AAPOR’s Transparency Initiative (TI) is envisioned to encourage openness in survey research so the field can advance.

Openness is fundamental to credibilityTI also is envisioned to enhance the

credibility of our profession by encouraging disclosure of methodological information and educating the public on how to use it.

3

Transparency Initiative GoalsTransparency Initiative Goals

•Main: • Advance the science and reputation of survey

research.

•Secondary:• Provide professional education on survey

documentation and disclosure practice• Educate survey sponsors and the public on

value of transparency

4

A Brief Historical OverviewA Brief Historical Overview

Transparency Initiative proposed by then AAPOR President (and Sociologist) Peter V. Miller in May 2010

5

Membership RequirementsMembership Requirements•Membership in AAPOR’s TI is anticipated to be

open to any survey organization or consultant, regardless of individual AAPOR affiliations or organizational members

•Membership requires commitment to the principles and practices of the Transparency Initiative

•Mainly, what will be required:• A pledge to conform to the AAPOR Code of

Professional Ethics and Practice regarding Disclosure (Section III)

• Willingness to train employees regarding appropriate transparency practices• AAPOR will develop an online training tool

6

AAPOR Code, Section III AAPOR Code, Section III SummarySummaryReport Immediately (a)• Who sponsored,

conducted & funded the research

• Exact question wording• Definition of the population• Geographic location• Sample frame description• Sample design• Sample size & error• Weighting & cluster

adjustments• Results based on parts of

sample only• Method(s) and dates of

data collection

Within 30 Days (b-d)• Interviewer/respondent

instructions• Relevant stimuli (show

cards)• Sampling frame’s coverage• Methods of panel

recruitment (for pre-recruited panels)

• Sample design details (eligibility, screening, oversamples, incentives)

• Sample dispositions• Weighting details• Data verification details• Response rates• All of the above for each if

multiple samples or modes7

Anticipated Benefits of Anticipated Benefits of Membership in TIMembership in TI

TI participants will benefit in the commercial marketplace and in the marketplace of ideas

Clients will learn that publicly released reports will have enhanced credibility if they are based on transparent research methods

The news media will learn that participation in the Transparency Initiative is an important indicator of whether or not to pay attention (i.e., give credence) to a survey organization’s reports

8

Qualification ProcessQualification Process

Two models were considered:Safe Harbor ModelInternational Organization for

Standardization (aka, ISO) Model

9

Safe Harbor ModelSafe Harbor Model•Run by US Dept of Commerce to provide a

mechanism for US research companies to comply with EU data protection legislation

•Requires organizations to develop and publish a formal privacy policy and to self-certify via letter and recertify (on an annual basis) that it complies with the standards

•Pro: No significant cost burden to organizations

•Con: No formal verification process, thus may be viewed as weak

10

ISO ModelISO Model•Certification only achieved after auditor

visits organization, examines processes, and conducts staff interviews regarding procedure adherence

•Annual surveillance audits ensure continued compliance

•Pro: Credibility, is based on external certification according to independent standards

•Con: Serious costs both to the certifying and the applicant organizations

11

Current AAPOR TI Proposal Being Current AAPOR TI Proposal Being ConsideredConsidered

Self-certification via following steps (annually):1. Organization agrees in writing to abide

by disclosure requirements of AAPOR Code2. Organization agrees to meet the

disclosure requirements of the Code via whatever mechanism AAPOR ultimately develops

3. Organization agrees to train employees in disclosure policy and committing to annual refresher training

12

Proposed Monitoring ProcessProposed Monitoring ProcessLike Safe Harbor, the TI would rely on

research community and the public at large for general monitoringA “complaint driven” process

Complaints of failure to disclose would be filed much like current AAPOR Code violation complaints

Details not yet worked out, but the process would likely be a streamlined version of the current process AAPOR employs for Code violations

13

Proposal for Handling Proposal for Handling ViolationsViolations

“Three Strikes You’re Out” standardSingle violation would not cause an

organization to have its TI membership revokedAAPOR will provide constructive feedback

designed to bring violators back into conformanceThree unrelated incidents within a given time

period would result in recommendation to AAPOR Council that the organization be disqualified from TI membership for one year

This is all under discussion, no final decisions have been made

14

2011-2012 Activities 2011-2012 Activities

TI Steering CommitteeMembers

David Cantor, Leah Christian, Marjorie Connelly, Liz Hamel, Melissa Hermann, Tim Johnson, Scott Keeter, Courtney Kennedy, Paul J. Lavrakas (chair), Peter Miller, Rich Morin, Joe Murphy, Chuck Shuttles, Susan Tibbitts (ex officio)

Working SubcommitteesSubcommittee on Structure, Qualification, and Compliance

(Cantor, chair)Subcommittee on Database Process and Infrastructures

(Miller, chair)Subcommittee on Internal and External Communication and

Education (Morin, chair)Subcommittee on Costs and Staffing (Lavrakas, chair)

TI External Advisory Committee (Kennedy, chair)15

2011-2012 Activities2011-2012 Activities

Pilot Test 1Conducted February-March 2012Eight volunteer organizations participated

Based on what was learned, TISC proposed to AAPOR Council that:Transition TISC into the TI Coordinating

Committee (TICC) with a stable, multi-year membership

A second pilot test will be run in 2012 based on lessons learned

16

2011-2012 Activities2011-2012 Activities

Pilot test focus groupConducted May 1, 2012Included representatives from 6 of 8

organizations who participated in the first pilot test

Additional participants included:Representatives from other TI Advisory Committee

membersTISC members

17

Key Lessons from Focus Key Lessons from Focus GroupGroup

There is strong verbal commitment among many AAPOR members to the concept of transparency.

Uncertainty regarding optimal scope of TI.Should focus be on:

Education?Certification?Audit function?

Uncertainty as to whether TI materials should be housed centrally on AAPOR web site or de-centrally on participant web sites.

18

Key Lessons from Focus Key Lessons from Focus GroupGroupConcerns with compliance burden.

Cost of participation too much for some organizations?

Uncertainty as to what surveys should be covered by TI:All surveys conducted by participating

organizations?Only surveys publically released by

participating organizations?Public Opinion Polls only? What about reports, press releases,

presentations, other media formats?

19

Key Lessons from Focus Key Lessons from Focus GroupGroup

Uncertainty as to best strategy for implementing TI:All-at-once?In multiple phases?Should there be multiple levels of TI

membership?It is clear that each of these issues

needs to be further considered before we move towards another pilot test.

20

Current Activities and PlanCurrent Activities and Plan

Providing membership with ongoing updates of TI activities via AAPOR web page and AAPORnet

Developing TI application and monitoring protocols

Finalizing policies for certification & decertification

Submit proposed TI policies and procedures to AAPOR Council for review and approval

Second pilot test in Fall 2012Final recommendations to AAPOR Council

Winter/Spring 2013

21

Transparency Initiative Transparency Initiative Coordinating Committee (TICC) Coordinating Committee (TICC) 2012-132012-13

Leah Christian Melissa HerrmannTim Johnson (Chair) Scott KeeterCourtney Kennedy

Mary LoschRich MorinTrevor Tompson Tim Triplett

22

23

Thank YouThank You

24