pilot surveys, cogni1ve interviews, and focus groups – oh...

Pilotsurveys,cogni1veinterviews,andfocusgroups–Ohmy!

Amixed-methodsapproachtowarddesigninga

longitudinalsurvey

DarnellCole,JosephKitchen,AdriannaKezar&MaFSoldner

•  Introduction •  Research Project Overview •  Eight-Step Mixed Methods Process

1.  Review of psycho-social measures/Qual. data 2.  Baseline psychometric pilot 3.  Baseline questionnaire administration 4.  Focus group discussions 5.  Qualitative case study research 6.  Follow-up psychometric pilot 7.  Cognitive Interviews 8.  Follow-up questionnaire administration

•  Conclusions and lessons learned •  Q & A

SessionOverview

ResearchProjectOverview

ComprehensiveCollegeTransi1onProgram

●  Overview ○  Since 1960s, foundation provided scholarships for low income

students ○  Five years of funding ○  Three campuses in university system ○  Small learning community courses, mentoring, residential

component, first year experience course, programming from staff ○  Two year program

●  Purpose for project

○  How the comprehensive college transition program facilitates engagement, the development of academic self-efficacy, mattering/sense of belonging, persistence and other outcomes.

Methods• Concurrentmixed-methodsdesign

– 2Cohorts(2015,2016)– Embeddedsequen1alexploratory/explanatory– Sixyearstudy

• Twoimportanttenetsofmixed-methodsdesign:

– Methodologicaleclec1cism– Paradigmpluralism

6

Initial qualitative data

Baseline Pilot

Baseline Survey

Focus Group Interviews

Case Study Research

Follow Up Pilot

Cognitive Interviews

Follow Up Survey Longitu

dinalSurveyDe

velopm

ent

StepOne:Qualita1veData&ReviewofPsycho-SocialMeasures

Qualita1veData&ReviewofPsycho-socialMeasures

•  Sitevisitswithprograms•  Interviewswithprogramstaffandkeyadministra1ons•  Programdocumentcollec1on•  Focusgroups

•  Programfaculty•  50studentsoneachcampus•  12-20campusstakeholders(e.g.,counseling,TRIO,studentaffairs)

•  Literaturereviewconducteda^ersitevisits•  Iden1fymeasurestotestinpsychometricpilot

Qualita1veData&

Reviewofpsychosocialmeasures:

Major

Takeaways

• Iden1fiedseveraladdi1onalconstructstomeasureinsurveyques1onnaire

• Usedliteraturereviewtoiden1fyothermeasuresthatmayberelevantgiventhecharacteris1csoftheprogramandpar1cipants

• Biannualprocessandhalf-dayretreatstocon1nueliteraturereview

• Quant.teamconsistentlyworkswithqual.teamtoinformsurveyandgetfeedback

StepTwo:BaselinePsychometricPilotSurvey

PsychometricPilotSurvey•  ConductedinJuly2015•  Purposes

–  Testandvalidatemorethanadozenoutcomevariables,includingintermediateoutcomes–  Evaluateopera1onalprotocols

•  Sampleincluded972scholarsfromthe2012and2013cohort•  350respondents,about36percent.

KeyBaselineConstructs•  Peer Interactions •  Faculty Interactions •  Time Use •  Social self-efficacy •  Academic self-efficacy •  Career decision-making self-efficacy •  Sense of belonging •  Perceived academic control •  Resiliency •  Expectations about mattering •  Financial Stress •  Malleability of ability •  Interpretation of difficulty

PsychometricPilot:

MajorTakeaways

•  Psychometricproper1esofadaptedmeasuresareo^enintheball-parkofpublishedresults,butusingConfirmatoryFactorAnalysistoverifycanaddconfidenceinyourchoices.

•  WeusedgradedresponseIRTmodelsasanaddi1onalcheckfollowingCFA,par1cularlywhenwewereconsideringscalereduc1on.

StepThree:BaselineSurveyAdministra1on

BaselineSurvey

•  Primarypurpose–Toobtainanearly(expected)measureofkeyoutcomevariables(andintermediateoutcomevariables)

•  Secondarypurpose–evaluateimprovedopera1onalprotocols

•  BaselineSurveyadministeredinAugust2015

•  Samplewas1335programpar1cipantsfromthe2015cohort(firstyearstudentsonly)

KeyBaselineConstructs•  High school sense of belonging •  High school interactions with peers •  High school interactions with faculty •  High school time use •  Expectations about social self-efficacy •  Expectations about academic self-efficacy •  Expectations about mattering •  Expectations about belonging •  Expectations about likelihood of graduation

BaselineSurveyAdministra1on:

Major

Takeaways

•  Ourbaselineadministra1onuncoveredatleastonephenomenonthatwasnotdetectedinthepilot,givenitspopula1onandthe1mingofadministra1on:higherthanan1cipatedbaselinescoresonself-efficacymeasures.

•  Whatisthetemporalnatureofyourphenomena?Iftheyare1me-variant,howbesttomeasure?

StepFour:FocusGroupDiscussions

FocusGroupDiscussions

•  Exploringasetofphenomenawiththeintentofdeterminingappropriatenessofsurveyitemdevelopment(i.e.,exploratory)

•  Semi-structured,open-endedprotocolwith7guiding

ques1ons•  Studentexperiences,percep1ons,andopinionsoffirstcontactwith

program,summerexperience,andtransi1onintotheprogram

FocusGroupDiscussions:

Major

Takeaways

Keyfindings

•  Advising

•  Schedulingconflicts

•  Powerofsmallclasssize

•  Highexpecta1onswithintheprogram

• Consideredaddingitemstofurtherdiscriminatekeyconstructsaroundhighexpecta1ons

• Considereddifferencesbetweenprogramandnon-programclassroomexperiences

StepFive:CaseStudyResearch

CaseStudyResearch

•  BeFerunderstandingofprogramelementsoneachcampus,studentbodies,majorstakeholdersandbroadercampusenvironment

•  Documentthewaysthattheprogramoperatestoshape

studentexperiences•  Casestudyelementsincludedprogramobserva1ons,

interviewswithstaffandstakeholders,andsocialmedia/documentanalysis

CaseStudyResearch:

Major

Takeaways

Keyfindings• Helpediden1fytheimportantprogramelementstostudy

• Helpedwithdevelopingappropriatesurveyphrasingandwording• Shapedsurveyinstrumentdesign

•  Iden1fiedimportantroleofstaffmembers;addedques1onsrelatedtothis

•  Itemsaddedtoaddressstaff-studentrela1onships,engagement,andsupport

•  Addedinmid-semestergradecheck

•  Digitaldiaryinforma1onresultedinaddi1onof1me-useques1ons

•  Observa1onsledtoaddi1onoffaculty-studentques1ons

StepSix:Follow-upPsychometricPilot

Follow-UpPilotSurvey•  AdministeredinJanuary2016•  Purposewastotestandvalidatemeasurementof10keyengagementconstructs

•  Sampleincluded1652scholarsfromthe2013and2014cohorts

•  352respondents,abouta21%responserate

Follow-UpPilotSurvey:

Major

Takeaways

Keyfindings•  Onceagain,resultsdemonstratedthe

importanceofpilottes1ngeventhoseitemsyouthinkstudentsshouldhavereasonableknowledgeof!

•  Expandedresponseop1onstosevenpoints

•  Timeanchorintheques1onstemmayhelptheperformanceofconstructs

•  Dowecon1nuetoincludemediocreperformingconstructsforthesakeofthelongitudinalanalysis

•  Minortweakstoitemswithintheconstructsares1llneededtoimproveperformance

StepSeven:Cogni1veInterviews

Cogni1veInterviews•  Conductedtorevisesurveyitemsrelatedtostudentengagementwithspecificprogramelements

•  EleveninterviewswereconductedinFeb.2016•  Twocomponents:(a)thinkaloudmethod;and(b)ques1onprobing

•  Contributedtosurveyrefinementandhelpedensurequality,validinstrument

Cogni1veInterviews:

Major

Takeaways

Keyfindings

•  Manysurveyitemssufficientlystrong

•  Changestosurveyresul1ngfromcogni1veinterviews:

•  Usingagateques1ontodetermineeventpar1cipa1onpriortoaskingaboutengagement

•  Useofgridinsteadofrepeatedsec1onsabouteacheventacrossdifferenttypesofengagement

•  Includinga“notsure”op1on

StepEight:Follow-upSurveyAdministra1on

31

Follow-upSurveySample

•  Samplewas1297programpar1cipants,COS,andControlfromthe2015cohort(firstyearstudentsonly)

•  951Respondents(73.3%responserate)

KeyFollow-upConstructs

•  “Actual” academic self-efficacy

•  “Actual” social self-efficacy

•  Experience of mattering

•  Experience of belonging

•  Engagement with peers and faculty

•  Engagement with program features

•  Financial stress

•  Revised graduation expectations

33

Follow-upSurvey:

Major

Takeaways

•  We’re still in the preliminary phases of the substantive analysis.

•  However, we did some analyses immediately that were designed to feed back into the questionnaire design phase for the next cohort.

•  We also conducted analyses to evaluate the effect of methodological experiments designed to improve response, and tweak them for the Cohort 2 Baseline.

ConclusionsandLessonsLearned

35

Initial qualitative data

Baseline Pilot

Baseline Survey

Focus Group Interviews

Case Study Research

Follow Up Pilot

Cognitive Interviews

Follow Up Survey Longitu

dinalSurveyDe

velopm

ent

36

Challenges Benefits •  Planning of qualitative and quantitative

sequence •  Established valid, reliable measures

•  Time intensive strategy •  Strengthened and contextualized measures via cognitive interviews, psychometrics

•  Gaining buy-in and securing collaborative relationships

•  Moved beyond reliance on literature alone

•  Decision-making about what to do next when established measures don’t work

•  Tested items with students drawn from our population of interest

•  Deciding how to edit or add new items for survey based on qual. findings

•  New items added as a result of qualitative findings

•  Bringing together (sometimes conflicting) qual. and quant. findings to make a cohesive argument

•  Triangulation of data and clarification of findings from quant. or qual.

•  Striking the right methodological balance

•  Qual. work provides guidance in developing survey items and identifying relevant measures

Ques1onsandcomments

Wewelcomeyourfeedbackandideas!

PulliasCenterforHigherEduca1onRossierSchoolofEduca1on

UniversityofSouthernCalifornia

3470TrousdaleParkwayWaitePhillipsHall701LosAngeles,CA90089

Website:pullias.usc.edu