![Page 1: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/1.jpg)
Lifecycle Processes to Insure Quality f Bl i I t i D tof Blaise Interview Data
Linda Gowen, WestatPat Clark WestatPat Clark, Westat
04/09/2008 1
![Page 2: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/2.jpg)
Lifecycle Processes to Insure Quality of Blaise i DInterview Data
• This presentation reviews some of the Westat tools d t f ilit t i t th diff t tused to facilitate processing at the different stages
throughout the lifecycle of a Blaise project, with a focus on the impact on data qualityfocus on the impact on data quality.
04/09/2008 2
![Page 3: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/3.jpg)
Survey Lifecycle OverviewSurvey Lifecycle Overview
• The major lifecycle steps are:• Design
D l t• Development • Data Collection• Data PreparationData Preparation • Data Delivery
• Westat tools are used at every stage to improve data quality.
04/09/2008 3
![Page 4: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/4.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Steps Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 4
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 5: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/5.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Steps Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 5
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 6: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/6.jpg)
Stage 1 – Requirements AnalysisStage 1 Requirements Analysis
• The goal of this stage is to define both the Blaise and non-Blaise requirements for the system.
• It is helpful to prepare a project data flow diagram showing:showing:
• All sources providing data into the Blaise Systems• All data required to feed external systems• All data transformations
04/09/2008 6
![Page 7: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/7.jpg)
Stage 2 – CAI SpecificationsStage 2 CAI Specifications
The CAI specifications document includes h l h dthe “complete” hard
copy questionnaire typically needed for regulatoryregulatory compliance, with additional detailed instructionsinstructions, necessary for programming the CAI system.
04/09/2008 7
y
![Page 8: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/8.jpg)
SpecWriter Output
Display Field Defined
Display Assignment Logic Box Display Field g
Skip Pattern from Answer Response
Numeric Range Declaration
Reference
Response
Question Number Edit
Item #
04/09/2008 8
![Page 9: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/9.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Steps Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 9
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 10: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/10.jpg)
Stage 3 - Blaise Development
• Programmers reference documents
Stage 3 Blaise Development
Programmers reference documents
• Project data flow diagram
• CAI specifications
04/09/2008 10
![Page 11: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/11.jpg)
Blaise Development Key ConsiderationsBlaise Development Key Considerations
• Proper use of data to be used as foreign keys in the creation of blocks/tables
• Review of complicated programming • Testing and QC at all stages of instrument development g Q g p
including verification of the data integrity after both forward and backward navigation
• Declaring fields to be compatible with external systemsg p y• Review relationship of question numbers to Blaise variable
names• Save needed intermediate variables• Save needed intermediate variables
04/09/2008 11
![Page 12: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/12.jpg)
Blaise Development Operational ConsiderationsBlaise Development Operational Considerations
• Identify data for external systems performing quality assurance and monitoring.
• Advance planning can assure the database structure provides for the easy extraction of dataprovides for the easy extraction of data.
04/09/2008 12
![Page 13: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/13.jpg)
Stage 4 – Internally Testing the Blaise SystemStage 4 Internally Testing the Blaise System
• A specialized testing team ensures a level of quality• CAI Specifications are tested• Issue Tracking• Feature by feature validation
i l i• Functional testing• Testing is done using all systems• All testing is based on workflow• All testing is based on workflow• Regression testing
04/09/2008 13
![Page 14: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/14.jpg)
Stage 5 – Pretest for Field ReadinessStage 5 Pretest for Field Readiness
• Testing is also performed by the project staff, data g p y p j ,management staff, and field staff.
• Role plays should simulate data collection scenarios• Role plays should simulate data collection scenarios.
• The goal is to have each real condition tested for all the responses available in the instrument.
• Files with the delivery layout are built from the mock• Files with the delivery layout are built from the mock data and frequencies run to verify that the system is capturing the data to meet the research and operational objectives
04/09/2008 14
objectives.
![Page 15: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/15.jpg)
04/09/2008 15
![Page 16: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/16.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Steps Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 16
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 17: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/17.jpg)
04/09/2008 17
![Page 18: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/18.jpg)
BFOS
BFOS provides projects with:
BFOS
• Interviewer Management System – A laptop-based application that allows interviewers to review and change assigned case status, launches the Blaise CAPI instrument, and synchronizes the management and CAPI data with the home officethe management and CAPI data with the home office.
• Supervisor Management System – This is the web-based application used by field supervisors to view and manage the work of interviewers, including case assignment and transfer , g gbetween interviewers, review and adjustment of case status, and other administrative and reporting functions.
• Home Office Management System – This provides functions d d b h ffi t ff t l d d th t dneeded by home office staff to load and manage the study
sample information and to maintain information about the field staff and their assignments.
04/09/2008 18
![Page 19: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/19.jpg)
Planning Prevents Unexpected ResultsPlanning Prevents Unexpected Results
04/09/2008 19
![Page 20: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/20.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Stages Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 20
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 21: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/21.jpg)
Stage 7 – Editing Blaise DataStage 7 Editing Blaise Data
An array of systems are available to support Blaise editing.y y pp g• Editing system • Extract system• Comment system• Data decision log
D d i i lid i• Data and interview validation• QA reconciliation systems• Study management systems• Study management systems• Metadata delivery system
04/09/2008 21
![Page 22: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/22.jpg)
Stage 7 – Editing Blaise Data
Basic steps of the editing process are:
Stage 7 Editing Blaise Data
p g p• Combine data into a master database• Maintain Blaise interview in interviewing and data editing modes
Display comments• Display comments• Editors may make changes to data.• Decisions documented.• Other, specify fields coded.• Frequencies reviewed• Reconciliation completed.p
04/09/2008 22
![Page 23: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/23.jpg)
04/09/2008 23
![Page 24: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/24.jpg)
04/09/2008 24
![Page 25: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/25.jpg)
Frequency ReviewFrequency Review
• As a final check of the data files themselves, the data ,are extracted from the final edited tables and frequency reviews are performed.
04/09/2008 25
![Page 26: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/26.jpg)
Quality AssuranceQuality Assurance
• Redundant data.
• QA system• Report discrepancies• Secure
R di i d S S d h• Reports on Editing system data, SMS data, other project source data
• Documentation of resolutions recorded in Data Decision Log.
04/09/2008 26
![Page 27: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/27.jpg)
Stage 8 – Create CodebooksStage 8 Create Codebooks
• The codebook is an important part of documentation.p p
• Our codebook generation system was designed around the metadata output from the Blaise instrument.
Data elements are versioned• Data elements are versioned.
• All derived data elements have the associated• All derived data elements have the associated derivation details present in notes.
04/09/2008 27
![Page 28: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/28.jpg)
04/09/2008 28
![Page 29: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/29.jpg)
Lifecycle ProcessesLifecycle
StWestat
S tStages
Steps Systems
Design
Design Survey Instruments
Develop Blaise System
Develop CAI Questionnaire Specifications SpecWriter
P Bl i S
Develop Blaise System
Test Blaise SystemDevelopment Blaise
Pretest Blaise System
Deploy Blaise SystemData Collection BFOS
Edit and QA Blaise Data
Create CodebooksBackend/Data
PreparationBlaise Editing System
04/09/2008 29
Deliver Data and Documentation/Prepare For Follow-upData DeliveryWestat Meta Data Delivery System
![Page 30: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/30.jpg)
Stage 9 – Data DeliveryStage 9 Data Delivery
Westat has developed a Data Delivery Metadata System p y ywhich supports delivery associated with datasets, such as:
Raw data• Raw data• Research datasets• Analytic datasets• Restricted use datasets, and• Public Use files.
04/09/2008 30
![Page 31: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/31.jpg)
04/09/2008 31
![Page 32: Lifecycle Processes to Insure Quality ofBl i I t i D tf ...blaiseusers.org/2007/slides/D4 - IBUC2007_Revised070924.pdf · Lifecycle Processes to Insure Quality ofBl i I t i D tf Blaise](https://reader031.vdocuments.us/reader031/viewer/2022020315/5ab9802a7f8b9ad3038e1fd4/html5/thumbnails/32.jpg)
SummarySummary
• Managing aspects of the processing extending across g g p p g gmultiple lifecycle stages is critical to data quality.
• Feedback from all aspects of the survey need to be considered, and iterative processing loops areconsidered, and iterative processing loops are necessary.
• Lifecycle tools can be implemented in multiple project environments and customized to the needs of individualenvironments and customized to the needs of individual surveys.
• Active participation from all team members assure that diff i ti t ddiffering perspectives are represented.
• Quality control becomes continuous from planning through delivery, ensuring accurate and efficient data
04/09/2008 32
g y gcollection and processing.