pilot surveys, cogni1ve interviews, and focus groups – oh...
TRANSCRIPT
Pilotsurveys,cogni1veinterviews,andfocusgroups–Ohmy!
Amixed-methodsapproachtowarddesigninga
longitudinalsurvey
DarnellCole,JosephKitchen,AdriannaKezar&MaFSoldner
• Introduction • Research Project Overview • Eight-Step Mixed Methods Process
1. Review of psycho-social measures/Qual. data 2. Baseline psychometric pilot 3. Baseline questionnaire administration 4. Focus group discussions 5. Qualitative case study research 6. Follow-up psychometric pilot 7. Cognitive Interviews 8. Follow-up questionnaire administration
• Conclusions and lessons learned • Q & A
SessionOverview
ResearchProjectOverview
ComprehensiveCollegeTransi1onProgram
● Overview ○ Since 1960s, foundation provided scholarships for low income
students ○ Five years of funding ○ Three campuses in university system ○ Small learning community courses, mentoring, residential
component, first year experience course, programming from staff ○ Two year program
● Purpose for project
○ How the comprehensive college transition program facilitates engagement, the development of academic self-efficacy, mattering/sense of belonging, persistence and other outcomes.
Methods• Concurrentmixed-methodsdesign
– 2Cohorts(2015,2016)– Embeddedsequen1alexploratory/explanatory– Sixyearstudy
• Twoimportanttenetsofmixed-methodsdesign:
– Methodologicaleclec1cism– Paradigmpluralism
6
Initial qualitative data
Baseline Pilot
Baseline Survey
Focus Group Interviews
Case Study Research
Follow Up Pilot
Cognitive Interviews
Follow Up Survey Longitu
dinalSurveyDe
velopm
ent
StepOne:Qualita1veData&ReviewofPsycho-SocialMeasures
Qualita1veData&ReviewofPsycho-socialMeasures
• Sitevisitswithprograms• Interviewswithprogramstaffandkeyadministra1ons• Programdocumentcollec1on• Focusgroups
• Programfaculty• 50studentsoneachcampus• 12-20campusstakeholders(e.g.,counseling,TRIO,studentaffairs)
• Literaturereviewconducteda^ersitevisits• Iden1fymeasurestotestinpsychometricpilot
Qualita1veData&
Reviewofpsychosocialmeasures:
Major
Takeaways
• Iden1fiedseveraladdi1onalconstructstomeasureinsurveyques1onnaire
• Usedliteraturereviewtoiden1fyothermeasuresthatmayberelevantgiventhecharacteris1csoftheprogramandpar1cipants
• Biannualprocessandhalf-dayretreatstocon1nueliteraturereview
• Quant.teamconsistentlyworkswithqual.teamtoinformsurveyandgetfeedback
StepTwo:BaselinePsychometricPilotSurvey
PsychometricPilotSurvey• ConductedinJuly2015• Purposes
– Testandvalidatemorethanadozenoutcomevariables,includingintermediateoutcomes– Evaluateopera1onalprotocols
• Sampleincluded972scholarsfromthe2012and2013cohort• 350respondents,about36percent.
KeyBaselineConstructs• Peer Interactions • Faculty Interactions • Time Use • Social self-efficacy • Academic self-efficacy • Career decision-making self-efficacy • Sense of belonging • Perceived academic control • Resiliency • Expectations about mattering • Financial Stress • Malleability of ability • Interpretation of difficulty
PsychometricPilot:
MajorTakeaways
• Psychometricproper1esofadaptedmeasuresareo^enintheball-parkofpublishedresults,butusingConfirmatoryFactorAnalysistoverifycanaddconfidenceinyourchoices.
• WeusedgradedresponseIRTmodelsasanaddi1onalcheckfollowingCFA,par1cularlywhenwewereconsideringscalereduc1on.
StepThree:BaselineSurveyAdministra1on
BaselineSurvey
• Primarypurpose–Toobtainanearly(expected)measureofkeyoutcomevariables(andintermediateoutcomevariables)
• Secondarypurpose–evaluateimprovedopera1onalprotocols
• BaselineSurveyadministeredinAugust2015
• Samplewas1335programpar1cipantsfromthe2015cohort(firstyearstudentsonly)
KeyBaselineConstructs• High school sense of belonging • High school interactions with peers • High school interactions with faculty • High school time use • Expectations about social self-efficacy • Expectations about academic self-efficacy • Expectations about mattering • Expectations about belonging • Expectations about likelihood of graduation
BaselineSurveyAdministra1on:
Major
Takeaways
• Ourbaselineadministra1onuncoveredatleastonephenomenonthatwasnotdetectedinthepilot,givenitspopula1onandthe1mingofadministra1on:higherthanan1cipatedbaselinescoresonself-efficacymeasures.
• Whatisthetemporalnatureofyourphenomena?Iftheyare1me-variant,howbesttomeasure?
StepFour:FocusGroupDiscussions
FocusGroupDiscussions
• Exploringasetofphenomenawiththeintentofdeterminingappropriatenessofsurveyitemdevelopment(i.e.,exploratory)
• Semi-structured,open-endedprotocolwith7guiding
ques1ons• Studentexperiences,percep1ons,andopinionsoffirstcontactwith
program,summerexperience,andtransi1onintotheprogram
FocusGroupDiscussions:
Major
Takeaways
Keyfindings
• Advising
• Schedulingconflicts
• Powerofsmallclasssize
• Highexpecta1onswithintheprogram
• Consideredaddingitemstofurtherdiscriminatekeyconstructsaroundhighexpecta1ons
• Considereddifferencesbetweenprogramandnon-programclassroomexperiences
StepFive:CaseStudyResearch
CaseStudyResearch
• BeFerunderstandingofprogramelementsoneachcampus,studentbodies,majorstakeholdersandbroadercampusenvironment
• Documentthewaysthattheprogramoperatestoshape
studentexperiences• Casestudyelementsincludedprogramobserva1ons,
interviewswithstaffandstakeholders,andsocialmedia/documentanalysis
CaseStudyResearch:
Major
Takeaways
Keyfindings• Helpediden1fytheimportantprogramelementstostudy
• Helpedwithdevelopingappropriatesurveyphrasingandwording• Shapedsurveyinstrumentdesign
• Iden1fiedimportantroleofstaffmembers;addedques1onsrelatedtothis
• Itemsaddedtoaddressstaff-studentrela1onships,engagement,andsupport
• Addedinmid-semestergradecheck
• Digitaldiaryinforma1onresultedinaddi1onof1me-useques1ons
• Observa1onsledtoaddi1onoffaculty-studentques1ons
StepSix:Follow-upPsychometricPilot
Follow-UpPilotSurvey• AdministeredinJanuary2016• Purposewastotestandvalidatemeasurementof10keyengagementconstructs
• Sampleincluded1652scholarsfromthe2013and2014cohorts
• 352respondents,abouta21%responserate
Follow-UpPilotSurvey:
Major
Takeaways
Keyfindings• Onceagain,resultsdemonstratedthe
importanceofpilottes1ngeventhoseitemsyouthinkstudentsshouldhavereasonableknowledgeof!
• Expandedresponseop1onstosevenpoints
• Timeanchorintheques1onstemmayhelptheperformanceofconstructs
• Dowecon1nuetoincludemediocreperformingconstructsforthesakeofthelongitudinalanalysis
• Minortweakstoitemswithintheconstructsares1llneededtoimproveperformance
StepSeven:Cogni1veInterviews
Cogni1veInterviews• Conductedtorevisesurveyitemsrelatedtostudentengagementwithspecificprogramelements
• EleveninterviewswereconductedinFeb.2016• Twocomponents:(a)thinkaloudmethod;and(b)ques1onprobing
• Contributedtosurveyrefinementandhelpedensurequality,validinstrument
Cogni1veInterviews:
Major
Takeaways
Keyfindings
• Manysurveyitemssufficientlystrong
• Changestosurveyresul1ngfromcogni1veinterviews:
• Usingagateques1ontodetermineeventpar1cipa1onpriortoaskingaboutengagement
• Useofgridinsteadofrepeatedsec1onsabouteacheventacrossdifferenttypesofengagement
• Includinga“notsure”op1on
StepEight:Follow-upSurveyAdministra1on
31
Follow-upSurveySample
• Samplewas1297programpar1cipants,COS,andControlfromthe2015cohort(firstyearstudentsonly)
• 951Respondents(73.3%responserate)
KeyFollow-upConstructs
• “Actual” academic self-efficacy
• “Actual” social self-efficacy
• Experience of mattering
• Experience of belonging
• Engagement with peers and faculty
• Engagement with program features
• Financial stress
• Revised graduation expectations
33
Follow-upSurvey:
Major
Takeaways
• We’re still in the preliminary phases of the substantive analysis.
• However, we did some analyses immediately that were designed to feed back into the questionnaire design phase for the next cohort.
• We also conducted analyses to evaluate the effect of methodological experiments designed to improve response, and tweak them for the Cohort 2 Baseline.
ConclusionsandLessonsLearned
35
Initial qualitative data
Baseline Pilot
Baseline Survey
Focus Group Interviews
Case Study Research
Follow Up Pilot
Cognitive Interviews
Follow Up Survey Longitu
dinalSurveyDe
velopm
ent
36
Challenges Benefits • Planning of qualitative and quantitative
sequence • Established valid, reliable measures
• Time intensive strategy • Strengthened and contextualized measures via cognitive interviews, psychometrics
• Gaining buy-in and securing collaborative relationships
• Moved beyond reliance on literature alone
• Decision-making about what to do next when established measures don’t work
• Tested items with students drawn from our population of interest
• Deciding how to edit or add new items for survey based on qual. findings
• New items added as a result of qualitative findings
• Bringing together (sometimes conflicting) qual. and quant. findings to make a cohesive argument
• Triangulation of data and clarification of findings from quant. or qual.
• Striking the right methodological balance
• Qual. work provides guidance in developing survey items and identifying relevant measures
Ques1onsandcomments
Wewelcomeyourfeedbackandideas!
PulliasCenterforHigherEduca1onRossierSchoolofEduca1on
UniversityofSouthernCalifornia
3470TrousdaleParkwayWaitePhillipsHall701LosAngeles,CA90089
Website:pullias.usc.edu