survey research operations survey research center institute for social research design issues in an...
TRANSCRIPT
![Page 1: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/1.jpg)
Survey Research OperationsSurvey Research CenterInstitute for Social Research
Design Issues in an Academic Survey Organization
Beth-Ellen Pennell
Director, Survey Research Operations
Survey Research Center
Institute for Social Research
University of Michigan
![Page 2: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/2.jpg)
SRO – SRC – ISR
Outline
• What’s different about an academic survey research center?
• Introduction to survey research operations• Project management 101• Communication• The triple constraints in project management• A note about quality• Communication• Can we make it more complex?
2
![Page 3: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/3.jpg)
SRO – SRC – ISR
What’s different about an academic survey research
center?
3
![Page 4: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/4.jpg)
SRO – SRC – ISR
Institute for Social Research
4
![Page 5: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/5.jpg)
SRO – SRC – ISR
Survey Research Center
• Established in 1946 • Productivity: In the past ten years faculty and staff have:
produced 59 books, 232 book chapters, and 1116 scientific articles
produced datasets that have been downloaded by thousands of researchers worldwide resulting in a new scientific article every three days
submitted ~3 proposals per week achieved higher than average success rate on proposal funding
from NIH and NSF
•
5
![Page 6: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/6.jpg)
SRO – SRC – ISR
Survey Research Center
• Research Volume: ISR approximately ~$100M + per year in sponsored
research; SRC about 70-80% of volume• Smallest of large-scale survey research
organizations but largest of academic-based • ~80% from US Federal government, predominately
National Institutes of Health, National Science Foundation
Investigator initiated grants (very few contracts)• Increased flexibility in design and execution
6
![Page 7: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/7.jpg)
SRO – SRC – ISR
Survey Research Center
• Faculty and Staff 70 faculty, most with joint appointments
in academic departments 275 staff ~up to 2000 interviewers ~100 students
• Teaching mission of University
7
![Page 8: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/8.jpg)
SRO – SRC – ISR
Educational Programs
• Summer Institute in Survey Research Techniques
• Michigan Program in Survey Methodology Masters PhD Certificate
• Joint Program in Survey Methods
• Joint Institute with Peking University
8
![Page 9: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/9.jpg)
SRO – SRC – ISR
Introduction to Survey Research Operations
9
![Page 10: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/10.jpg)
SRO – SRC – ISR
Survey Research Operations
• Complete about ~100 projects per year with staff of ~130
• Process ~100 proposals per year
• Historically, ~ half of SRO clients are outside of ISR
10
![Page 11: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/11.jpg)
SRO – SRC – ISR
Types of Projects
• Interviewer-administered CAPI CATI (both centralized and decentralized) CASI and A-CASI as components of CAPI
instruments
• Self-administered Web Mail
• Many projects use a combination of modes, attempting to optimize quality and cost
11
![Page 12: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/12.jpg)
SRO – SRC – ISR
Types of Projects
• Our volume is comprised of: large, national CAPI studies regional CAPI studies smaller telephone, mail or web-based studies
• Our organizational infrastructure accommodates large projects that can be modified for smaller or more specialized efforts
12
![Page 13: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/13.jpg)
SRO – SRC – ISR
Project Management 101
13
![Page 14: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/14.jpg)
SRO – SRC – ISR
Project Manager/Team Roles
• The project manager is the person responsible for accomplishing the project objectives
• The project team is comprised of the people who have assigned roles and responsibilities for completing aspects of the project
14
![Page 15: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/15.jpg)
SRO – SRC – ISR
COMMUNICATION
15
![Page 16: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/16.jpg)
SRO – SRC – ISR
Communication
“. . . 75 to 90 percent of the project manager's time on a project is spent
communicating with one or more stakeholders” (Corpedia)
16
![Page 17: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/17.jpg)
SRO – SRC – ISR
Project Management
Managing a project includes:
• Identifying requirements
• Establishing clear and achievable objectives
• Adapting/balancing approach with various stakeholders
• Managing the team
• Balancing the competing demands (scope (includes quality), time & cost)
17
![Page 18: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/18.jpg)
SRO – SRC – ISR
Project Management
• Application of knowledge, skills, tools and techniques to project activities with following phases: Initiation Planning Risk assessment/ Plan B
• Example Executing and controlling Closing
18
![Page 19: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/19.jpg)
SRO – SRC – ISR
Stakeholders
• Individuals and organizations actively involved in the project, or whose interest may be affected by the project
• Project management team’s job is to: Identify stakeholders Determine requirements and expectations Manage stakeholders’ influence
19
![Page 20: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/20.jpg)
SRO – SRC – ISR
Key Stakeholders
• Sponsor
• Client (if different from sponsor)
• Users (if different from above)
• Project management organization
• Performing organization
• Project manager and team members
• Respondents
Example
20
![Page 21: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/21.jpg)
SRO – SRC – ISR
Stakeholder Influence
21
![Page 22: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/22.jpg)
SRO – SRC – ISR
Management Plan
• Purpose: Documents actions necessary to define, prepare,
integrate, and coordinate project activities May include several subsidiary plans
• Progressively elaborated
• Multiple authors
• Version control important (hard to do after the fact)
22
![Page 23: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/23.jpg)
SRO – SRC – ISR
Management Plan Broad Outline
• Background
• Initiation activities
• Project planning/risk assessment
• Implementation and control
• Closing
23
![Page 24: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/24.jpg)
SRO – SRC – ISR
Launching a Project
• Project design also involves: Creating a team Developing a team Managing a team
24
![Page 25: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/25.jpg)
SRO – SRC – ISR
Team Management
• Complexities of managing resources:
Delineate roles & responsibilities
Acquire staff with needed skills
Develop and build the team
Manage individual performance
25
![Page 26: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/26.jpg)
Task Team Model
26
![Page 27: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/27.jpg)
SRO – SRC – ISR
The Triple Constraints
27
![Page 28: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/28.jpg)
SRO – SRC – ISR
Constraint #1: Scope (read: Design)
• Scope planning
• Scope definition
• Work breakdown structure
• Scope control
28
![Page 29: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/29.jpg)
SRO – SRC – ISR
Design Specifications (1)• Study Purpose
Objectives Funding source(s) Timing Production plan milestones/deliverables Overall design summary
• Project Coordination Plan Lead team Task teams Working groups Communication/coordination
29
![Page 30: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/30.jpg)
SRO – SRC – ISR
Design Specifications (2)
• Sample Design Type (sample frame availability and coverage
properties) Size Accessibility of population Goals (response rate, number of cases in
various domains, etc.) Replicate design Responsive design
30
![Page 31: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/31.jpg)
SRO – SRC – ISR
Design Specifications (3)
• Mode Feasibility of mode
• Infrastructure• Social desirability bias (what is sensitive)• Literacy• Interviewer safety
Mixed/multi-mode – increasing in use• Sequential – cheapest first• Driven by coverage/cost
31
![Page 32: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/32.jpg)
SRO – SRC – ISR
Design Specifications (4)
• Data Collection Ethical consideration/review
• Confidentiality/privacy• Informed consent• Sensitive topics or vulnerable populations• Age of majority• Literacy• Incentives/coercion• Local laws, regulations• Special cases: biomarkers
32
![Page 33: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/33.jpg)
SRO – SRC – ISR
Design Specifications (5)• Data Collection (continued)
Field Staffing Plan• Assess local infrastructure• Languages • Sample assignment model• Supervisory structure/reporting• Production schedule • Interviewer remuneration guidelines• Mechanisms for reporting progress• Result code definitions and response rate calculation
guidelines
33
![Page 34: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/34.jpg)
SRO – SRC – ISR
Design Specifications (6)
• Instrument /Application Development Mode Languages Special printing requirements Hardware and software requirements Local adaptation requirements
34
![Page 35: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/35.jpg)
SRO – SRC – ISR
Design Specifications (7)
Testing• Pretests• Applications/systems testing• Sample requirements
Training Plans• Supervisor/Team Leader training• Interviewer training• Special training, e.g., biomarkers
35
![Page 36: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/36.jpg)
SRO – SRC – ISR
Design Specifications (8)
• Quality Checks Application requirements Training Paradata: dashboards, control charts Verifications/validations Editing and coding Data review Site visits
36
![Page 37: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/37.jpg)
SRO – SRC – ISR
Design Specifications (9)• Data Processing Plans
Coding Data entry Staffing Production schedule Mechanism for reporting
• Sample Post-processing Weights, nonresponse or other adjustments,
imputation, variance estimation
37
![Page 38: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/38.jpg)
SRO – SRC – ISR
Design Specifications (10)
• Data Processing Quality Control Testing Training plans
• Supervisor training• Editor guide• DDE training• Coding/Special function training
38
![Page 39: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/39.jpg)
SRO – SRC – ISR
Design Specifications (11)
• Materials Development (Deliverables) Datasets
• Sampling variables (weighting, nonresponse adjustments) • Sample file (result codes to check response rate calculations)• Call records• Data/Application
Respondent materials• Coversheets• Questionnaires/ applications• Questionnaire supplements/aids (e.g., respondent booklets,
event history calendars)• Contact materials
39
![Page 40: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/40.jpg)
SRO – SRC – ISR
Design Specifications (12)
• Materials Development (Deliverables) - continued Training
• Materials (e.g., PowerPoint slides, exercises)• Manuals (supervisor/interviewer, other)• Agendas• Location/organization
Protocols• Incident/problem reporting• Technical support/back-up
40
![Page 41: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/41.jpg)
SRO – SRC – ISR
Design Specifications (13)• Materials Development (Deliverables) - continued
Codebook Reports/Quality Profile
• Sample design description• Ethics review verification• Translation reports• Pretest reports• Field progress reports• Quality control audits• Field procedures reports
41
![Page 42: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/42.jpg)
SRO – SRC – ISR
Constraint #2: Time
• Project Time Management
Activity Definition
Activity Sequencing
Activity Resource Estimation
Schedule Development
Schedule Control
42
![Page 43: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/43.jpg)
SRO – SRC – ISR
Schedule Development and Control
• Identify schedule activities
• Establish optimum project work flow
• Determine resources needed
• Determine time needed for each activity
• Lay out activities in calendar time
• Keep track of progress, take corrective action
43
![Page 44: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/44.jpg)
SRO – SRC – ISR
TASK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Kickoff Meeting *Develop Questionnaire * * * * * * * *Programming Questionnaire * * * * *Translation of Questionnaire * * *1" Round Instrument Testing & Modifications * * * * * *Add Translation to Programming * *Sample Preparation & Listing * * * * * * * * *Native Speaker Review & Testing of Programmed, Translated Modules * *Instrument Translation Modifications * * *Instrument Programming Modifications * * *Recruit/Hiring of Production Interviewers * * * *Integrate & Test Full Instrument * *Develop & Test Sample Management * *Finalize Pretest Training/Project Materials *Duplicate Training Materials/Prepare Laptops *Pretest I Interviewer Training & Practice *Pretest I *Debrief Pretest I *Instrument & Materials Modifications, Testing *Pretest II Interviewer Training & Practice *Pretest II *Debrief Pretest II *Instrument & Materials Final Modifications *Duplicate Training Materials/Prepare Laptops *Draw Sample *Sample Assignments * *Train the Trainer *Interviewer Training I *Production Interviewing * * * * * * * * * * * *Coding * * * * * * * * * * * * *Interviewer Training II *Data Preparation * * * * * * * *Data Delivery *Documentation Preparation * * * * * * * * * * * * * * * * * * * * * * *Documentation Delivery *
Month of Survey
Project Work Plan Example
44
![Page 45: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/45.jpg)
SRO – SRC – ISR
Constraint #3: Cost
• Project Cost Management Primarily concerned with the cost of
resources needed to complete schedule activities
45
![Page 46: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/46.jpg)
SRO – SRC – ISR
Cost Monitoring
• Cost management processes include Cost estimating Cost budgeting Cost control
46
![Page 47: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/47.jpg)
SRO – SRC – ISR
Design Considerations in Tenders, Bids, and Contracts
• Purpose: to thoroughly document and communicate roles, responsibilities, design details, expectations and outcomes in order to adequately budget the survey effort
• Organizational Considerations Locus of control Balancing standardization and localization Balancing funding versus contractual obligations versus expertise Methodological, substantive, local expertise Legal expertise ‘One Voice’/Point of contact
47
![Page 48: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/48.jpg)
SRO – SRC – ISR
Budgeting and Bid Assessment
• Benchmark, where possible
• Seek multiple bids, if feasible
• Ask for project experience and references
• Ask for sample materials
• Ask for staff and organizational capacity
• Ask for facilities and equipment capacity
• Allow time for clarification and several iterations of bids
48
![Page 49: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/49.jpg)
SRO – SRC – ISR
Budgeting and Bid Assessment
• Potential Problems Sample frame access Difficulty with concept of probability sampling Inadequate quality control procedures Not familiar/sensitive to/barrier to nonresponse reduction
techniques (number of attempts to contact households, refusal conversion techniques, inconsistent contact result coding schemes)
Inadequate interviewer training/interviewer incentives Inadequate facilities (hardware, software, Web access,
etc.) Inadequate budget detail Research ‘traditions’
49
![Page 50: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/50.jpg)
SRO – SRC – ISR
Budget TemplateNumber Number of Months Cost per Unit Total Cost
Base Salaries Project Manager Data Manager Fieldwork Manager Accountant Assistants Supervisors Interviewers Data entry operators Drivers Translators Computer programmers Overtime Incentive payments
Travel Allowances Project Manager Data Manager Fieldwork Manager Assistants Supervisors Interviewers Drivers
50
![Page 51: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/51.jpg)
SRO – SRC – ISR
Number Number of Months Cost per Unit Total CostMaterials Vehicle purchase Fuel and maintenance Computers Printers, etc. Publicity Computer/printer supplies Photocopier/Fax machine Office supplies Communications (phone, fax, postage, etc) Equipment maintenance Printing Costs Questionnaires Training Manuals Reports Miscellaneous (maps, listings, manuals, etc)
Consultant Costs International consultants International per diem International travel Local consultants Local per diem Local travel
Contingency (10%)
TOTAL COST
Budget Template (continued)
51
![Page 52: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/52.jpg)
SRO – SRC – ISR
Contracting
• Specific to: Funding source Contracting organizations Employee contracts and remuneration policies Countries Type of contract
• Must have local legal advice• Commercial versus academic versus government
pros / cons
52
![Page 53: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/53.jpg)
SRO – SRC – ISR
One Mechanism to Control Scope and Costs: Responsive Design
• Interviewer labor dominates costs on most surveys
• Surveys inherently uncertain (e.g. estimated costs and completed interviews)
• Maximize cost benefit by adapting design features based on incoming survey process data (make data driven decisions)
53
![Page 54: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/54.jpg)
SRO – SRC – ISR
Responsive Design Implementation Process (Groves et al., 2009)
1) Pre-identify features potentially affecting costs and errors (risks)
2) Identify indicators
3) Wherever possible, devise benchmarks for each indicator
4) Create decision rules that alter features of the design if benchmarks are not met
5) Develop effective visual displays of indicators
6) When benchmarks are not met, alter the survey design based on the decision rules
54
![Page 55: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/55.jpg)
SRO – SRC – ISR
Creating decision rules: Example
• Risk: Interviewers unable to convince eligible respondents to participate, potentially increasing non response bias
• Indicator: Cumulative average interview cooperation rate by day by interviewer by area
• Benchmark: Similar study had a 71% interview cooperation rate after 3 weeks of data collection
• Decision rule: If not at a 70% coop rate by the end of the 3rd week, identify source of problem: is it systematic across all interviewers and sample types or limited to a few interviewers or sample types, area
55
![Page 56: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/56.jpg)
SRO – SRC – ISR
A note about quality
56
![Page 57: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/57.jpg)
SRO – SRC – ISR
Quality Management
• Components Quality Planning and Assurance Quality Control Quality Profile
57
![Page 58: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/58.jpg)
SRO – SRC – ISR
![Page 59: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/59.jpg)
SRO – SRC – ISR
COMMUNICATION
59
![Page 60: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/60.jpg)
SRO – SRC – ISR
Communication
• Communications Planning Information Distribution Performance Reporting Manage Stakeholders
60
![Page 61: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/61.jpg)
SRO – SRC – ISR
Communication
• Frequent
• Detailed
• Mixed (face-to-face, e-mail, conference call, newsletters, Webcasts)
• Document Sharing Critical for timely review Version control
61
![Page 62: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/62.jpg)
SRO – SRC – ISR
Data Documentation eRoom
62
![Page 63: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/63.jpg)
SRO – SRC – ISR
Can we make this more complex?
63
![Page 64: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/64.jpg)
SRO – SRC – ISR
YES!
We could try to do this across multiple countries…
64
![Page 65: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/65.jpg)
SRO – SRC – ISR
Mode of Administration
65
![Page 66: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/66.jpg)
SRO – SRC – ISR
Cross-national Considerations (1)
• Structure dynamics Leadership Levels Number of organizations Locus of control Level of control Experience at various levels Funding
66
![Page 67: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/67.jpg)
SRO – SRC – ISR
Cross-national Considerations (2)
• Degree of standardization: Often determined by level, flow, and source of
funding Experience and research infrastructure at central
and local levels Control of data Decisions need to be made at every stage of the
survey lifecycle
67
![Page 68: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/68.jpg)
SRO – SRC – ISR
Cross-national Considerations (3)
• Laws/ethical standards• Lingua Franca• Time zones• Terminology• Research traditions and infrastructure• Technology (Web: e-mail, VOIP, web
casts, document sharing)
68
![Page 69: Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,](https://reader034.vdocuments.us/reader034/viewer/2022051515/5515f5e9550346cf6f8b5633/html5/thumbnails/69.jpg)
SRO – SRC – ISR
Danke!
69