rc-33 eighth international conference on social science methodology timothy p. johnson and paul j....
Post on 16-Jan-2016
217 Views
Preview:
TRANSCRIPT
RC-33 Eighth International Conference on Social Science Methodology
Timothy P. Johnson and Paul J. Lavrakas
AAPOR’s Transparency AAPOR’s Transparency InitiativeInitiative
1
Problem:Problem: There is not There is not enough transparency in the enough transparency in the Social and Behavioral Social and Behavioral Sciences.Sciences.Premise 1: Too often, there is
inadequate transparency of research methods and statistical methods when the results of social and behavioral science research are disseminated, especially by non-academic sectors.
Premise 2: This is a serious detriment to progress in these scientific domains.
2
Openness is fundamental to Openness is fundamental to scientific inquiry.scientific inquiry.
AAPOR’s Transparency Initiative (TI) is envisioned to encourage openness in survey research so the field can advance.
Openness is fundamental to credibilityTI also is envisioned to enhance the
credibility of our profession by encouraging disclosure of methodological information and educating the public on how to use it.
3
Transparency Initiative GoalsTransparency Initiative Goals
•Main: • Advance the science and reputation of survey
research.
•Secondary:• Provide professional education on survey
documentation and disclosure practice• Educate survey sponsors and the public on
value of transparency
4
A Brief Historical OverviewA Brief Historical Overview
Transparency Initiative proposed by then AAPOR President (and Sociologist) Peter V. Miller in May 2010
5
Membership RequirementsMembership Requirements•Membership in AAPOR’s TI is anticipated to be
open to any survey organization or consultant, regardless of individual AAPOR affiliations or organizational members
•Membership requires commitment to the principles and practices of the Transparency Initiative
•Mainly, what will be required:• A pledge to conform to the AAPOR Code of
Professional Ethics and Practice regarding Disclosure (Section III)
• Willingness to train employees regarding appropriate transparency practices• AAPOR will develop an online training tool
6
AAPOR Code, Section III AAPOR Code, Section III SummarySummaryReport Immediately (a)• Who sponsored,
conducted & funded the research
• Exact question wording• Definition of the population• Geographic location• Sample frame description• Sample design• Sample size & error• Weighting & cluster
adjustments• Results based on parts of
sample only• Method(s) and dates of
data collection
Within 30 Days (b-d)• Interviewer/respondent
instructions• Relevant stimuli (show
cards)• Sampling frame’s coverage• Methods of panel
recruitment (for pre-recruited panels)
• Sample design details (eligibility, screening, oversamples, incentives)
• Sample dispositions• Weighting details• Data verification details• Response rates• All of the above for each if
multiple samples or modes7
Anticipated Benefits of Anticipated Benefits of Membership in TIMembership in TI
TI participants will benefit in the commercial marketplace and in the marketplace of ideas
Clients will learn that publicly released reports will have enhanced credibility if they are based on transparent research methods
The news media will learn that participation in the Transparency Initiative is an important indicator of whether or not to pay attention (i.e., give credence) to a survey organization’s reports
8
Qualification ProcessQualification Process
Two models were considered:Safe Harbor ModelInternational Organization for
Standardization (aka, ISO) Model
9
Safe Harbor ModelSafe Harbor Model•Run by US Dept of Commerce to provide a
mechanism for US research companies to comply with EU data protection legislation
•Requires organizations to develop and publish a formal privacy policy and to self-certify via letter and recertify (on an annual basis) that it complies with the standards
•Pro: No significant cost burden to organizations
•Con: No formal verification process, thus may be viewed as weak
10
ISO ModelISO Model•Certification only achieved after auditor
visits organization, examines processes, and conducts staff interviews regarding procedure adherence
•Annual surveillance audits ensure continued compliance
•Pro: Credibility, is based on external certification according to independent standards
•Con: Serious costs both to the certifying and the applicant organizations
11
Current AAPOR TI Proposal Being Current AAPOR TI Proposal Being ConsideredConsidered
Self-certification via following steps (annually):1. Organization agrees in writing to abide
by disclosure requirements of AAPOR Code2. Organization agrees to meet the
disclosure requirements of the Code via whatever mechanism AAPOR ultimately develops
3. Organization agrees to train employees in disclosure policy and committing to annual refresher training
12
Proposed Monitoring ProcessProposed Monitoring ProcessLike Safe Harbor, the TI would rely on
research community and the public at large for general monitoringA “complaint driven” process
Complaints of failure to disclose would be filed much like current AAPOR Code violation complaints
Details not yet worked out, but the process would likely be a streamlined version of the current process AAPOR employs for Code violations
13
Proposal for Handling Proposal for Handling ViolationsViolations
“Three Strikes You’re Out” standardSingle violation would not cause an
organization to have its TI membership revokedAAPOR will provide constructive feedback
designed to bring violators back into conformanceThree unrelated incidents within a given time
period would result in recommendation to AAPOR Council that the organization be disqualified from TI membership for one year
This is all under discussion, no final decisions have been made
14
2011-2012 Activities 2011-2012 Activities
TI Steering CommitteeMembers
David Cantor, Leah Christian, Marjorie Connelly, Liz Hamel, Melissa Hermann, Tim Johnson, Scott Keeter, Courtney Kennedy, Paul J. Lavrakas (chair), Peter Miller, Rich Morin, Joe Murphy, Chuck Shuttles, Susan Tibbitts (ex officio)
Working SubcommitteesSubcommittee on Structure, Qualification, and Compliance
(Cantor, chair)Subcommittee on Database Process and Infrastructures
(Miller, chair)Subcommittee on Internal and External Communication and
Education (Morin, chair)Subcommittee on Costs and Staffing (Lavrakas, chair)
TI External Advisory Committee (Kennedy, chair)15
2011-2012 Activities2011-2012 Activities
Pilot Test 1Conducted February-March 2012Eight volunteer organizations participated
Based on what was learned, TISC proposed to AAPOR Council that:Transition TISC into the TI Coordinating
Committee (TICC) with a stable, multi-year membership
A second pilot test will be run in 2012 based on lessons learned
16
2011-2012 Activities2011-2012 Activities
Pilot test focus groupConducted May 1, 2012Included representatives from 6 of 8
organizations who participated in the first pilot test
Additional participants included:Representatives from other TI Advisory Committee
membersTISC members
17
Key Lessons from Focus Key Lessons from Focus GroupGroup
There is strong verbal commitment among many AAPOR members to the concept of transparency.
Uncertainty regarding optimal scope of TI.Should focus be on:
Education?Certification?Audit function?
Uncertainty as to whether TI materials should be housed centrally on AAPOR web site or de-centrally on participant web sites.
18
Key Lessons from Focus Key Lessons from Focus GroupGroupConcerns with compliance burden.
Cost of participation too much for some organizations?
Uncertainty as to what surveys should be covered by TI:All surveys conducted by participating
organizations?Only surveys publically released by
participating organizations?Public Opinion Polls only? What about reports, press releases,
presentations, other media formats?
19
Key Lessons from Focus Key Lessons from Focus GroupGroup
Uncertainty as to best strategy for implementing TI:All-at-once?In multiple phases?Should there be multiple levels of TI
membership?It is clear that each of these issues
needs to be further considered before we move towards another pilot test.
20
Current Activities and PlanCurrent Activities and Plan
Providing membership with ongoing updates of TI activities via AAPOR web page and AAPORnet
Developing TI application and monitoring protocols
Finalizing policies for certification & decertification
Submit proposed TI policies and procedures to AAPOR Council for review and approval
Second pilot test in Fall 2012Final recommendations to AAPOR Council
Winter/Spring 2013
21
Transparency Initiative Transparency Initiative Coordinating Committee (TICC) Coordinating Committee (TICC) 2012-132012-13
Leah Christian Melissa HerrmannTim Johnson (Chair) Scott KeeterCourtney Kennedy
Mary LoschRich MorinTrevor Tompson Tim Triplett
22
23
Thank YouThank You
24
top related