designing and using a behaviour code frame to assess multiple styles of survey items alice mcgee and...
TRANSCRIPT
![Page 1: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/1.jpg)
Designing and Using a Behaviour code frame to assess multiple styles
of survey items
Alice McGee and Michelle Gray
![Page 2: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/2.jpg)
Presentation outline
• Background to study• Aims of research• Methodology• Designing a behaviour code frame• Using the behaviour code frame• Analysing the data• Lessons learned
![Page 3: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/3.jpg)
Background to study
• English Longitudinal Study of Ageing (ELSA)• Dependent Interviewing (DI)• Two types of data item
Feed Forward (DI)Non-Feed Forward (non DI)
• Little evaluation of the impact of DI on data quality conducted to date
![Page 4: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/4.jpg)
Aims of research
• Research aims:To assess how DI affects data qualityTo explore how Rs react to feed-forward phrasesTo find whether this varies by nature and
sensitivity of topic• Methodological aim:
To explore the combination of CARI and Behaviour Coding as methodological tools
![Page 5: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/5.jpg)
Methodology
• Computer Assisted Recorded Interviewing (CARI)Computer acts as a sophisticated tape recorderUnobtrusively records interaction
• Behaviour CodingCodes systematically applied to interviewer-
respondent behavioursUncover and assess problems with questions
• Two methods combined for this study
![Page 6: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/6.jpg)
Designing the code frame
![Page 7: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/7.jpg)
Principles for good design
• Code frame adapted from Cannell et al (1989)• Short and straightforward • Few, easy to apply codes• Discrete• Broad rather than specific
![Page 8: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/8.jpg)
Behaviours coded
• Question asking behaviour for interviewers• Immediate response behaviour for respondents• Whether partner intervened (concurrent
interviews)• Final outcome of the entire exchange
![Page 9: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/9.jpg)
Two behaviour code frames
• Two code frames designed:DI (feed-forward) itemsnon DI (non feed-forward) items
• First level exchange (initial utterance)• Code what occured before other person speaks
![Page 10: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/10.jpg)
Code frame
![Page 11: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/11.jpg)
Behaviours coded
• Interviewer/Interviewer feed-forward• Respondent/Respondent feed-forward• Whether partner intervened• Final outcome
• One code per behaviour
![Page 12: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/12.jpg)
Interviewer codes
• Exact Wording/Slight Change 01• *Major change 02• *Omission 03• *Question became a statement 04• *Inaudible Interviewer/Other 05• Not applicable 99
*denotes where notes must be made
![Page 13: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/13.jpg)
Interviewer feed-forward codes
• FF item read as worded/slight change 01• *FF statement became a question 02• *FF question became a statement 03• *Other major change 04• *Omission 05• *Inaudible Interviewer/Other 06• Not applicable 99
*denotes where notes must be made
![Page 14: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/14.jpg)
Respondent codes
• Adequate Answer 01• *Inadequate Answer/Elaboration 02• *Clarification 03• Question Re-Read 04• Don't Know 05• Refusal 06• *Inaudible Respondent/Other 07• Not applicable 99
*denotes where notes must be made
![Page 15: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/15.jpg)
Respondent feed-forward codes
• *Affirmed FF item - adequate 01• *Disputed FF item - adequate 02• *Inadequate Answer/Elaboration 03• *Clarification 04• Question Re-Read 05• Don't Know 06• Refusal 07• *Inaudible Respondent/Other 08• Not applicable 99
*denotes where notes must be made
![Page 16: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/16.jpg)
Partner intervention codes
• *Yes 01 • No 02• Not applicable (no partner present) 99
*denotes where notes must be made
• Code used for where the respondents partner intervened and subsequently answered for the respondent
![Page 17: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/17.jpg)
Final outcome codes
• Adequate Answer 01 • *Inadequate Answer 02 • Don't Know 03• Refusal 04• *Inaudible/Other 99
*denotes where notes must be made
• Coding whether the final answer meet the objective of the question
![Page 18: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/18.jpg)
Technical details
![Page 19: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/19.jpg)
CARI equipment
• Equipment testingExternal microphones
• CARI built into Blaise program• Recording switched on and off at relevant items• Sound files automatically generated and saved• Sound files removed from interviewer laptops
Macro run Data sticks (USB)
![Page 20: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/20.jpg)
Behaviour coding system
• Conducted within Blaise• Coding program designed for this purpose
Weststat testnote software• Three windows displayed simultaneously
Blaise interviewing screenCoding entry screenSound file (.wav)
• Automatically routed through interview• Tags to skip to relevant data items
![Page 21: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/21.jpg)
Using the code frame
![Page 22: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/22.jpg)
Sound file
![Page 23: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/23.jpg)
Blaise interviewing screen
![Page 24: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/24.jpg)
Coding program
![Page 25: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/25.jpg)
![Page 26: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/26.jpg)
Data preparation and analysis
![Page 27: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/27.jpg)
Organising the data
• Two types of dataBehaviour codes (quantitative)Coder notes on non-standard behaviours
(qualitative)• All data automatically stored in Excel tab
delimited file• One Excel file produced for each coder• Excel files amalgamated• Exported into SPSS
![Page 28: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/28.jpg)
Data preparation
• More cleaning than expected• Two main problems:
Duplicate files (limitations of system used)Incorrect code frame used at interviewer and
respondent behaviours (DI and non DI items)
![Page 29: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/29.jpg)
Analysing the data
• SPSS• Frequencies and crosstabulations• Coder notes provided additional context• Very small base sizes at some items due to
routing
![Page 30: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/30.jpg)
Advantages and disadvantages of our approach and lessons learned
![Page 31: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/31.jpg)
What worked
• CARIUnobtrusive in natureMinimal impact on interviewers and respondents
• Behaviour codingAble to run statistical analysesAble to draw conclusionsMethod of coding easier than paper (routing)
![Page 32: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/32.jpg)
What didn’t worked
• CARIHigh number of inaudible or hard to hear cases
(1/3 of respondents)Purchased speakers to help
• Next time…Fully re-test microphonesProbe respondents reasons for not giving
consent to being recorded
![Page 33: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/33.jpg)
What didn’t worked
• Behaviour codingLengthy and costly process
Coding (approximately 45 mins per interview)
Data cleaning
Over complex code frameCoding method found cumbersome, limited and
error proneCoder judgement not measured
![Page 34: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/34.jpg)
Next time...
• One code frame only• Build in sufficient time for each stage• Clear rationale for behaviour coding• Inter-coder reliability test (Kappa score)• Adequate sample for uncommon questions• Create more sophisticated, less error prone
coding system
![Page 35: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/35.jpg)
Discussion & Questions...
![Page 36: Designing and Using a Behaviour code frame to assess multiple styles of survey items Alice McGee and Michelle Gray](https://reader035.vdocuments.us/reader035/viewer/2022062517/56649e915503460f94b96fd5/html5/thumbnails/36.jpg)