1

SummaryofFindingsforEval^2FocusGroupwithFellowsMay11,2017CharlaHenley

SeniorFellowoftheEvaluationLabBackground:TheEval^2focusgroupwasperformedonMay4,2017with12of12studentsfromthisyear’scohortofEvaluationLabFellows.Onestudentleftearly.Thefocusgroupwasinteractiveinnature,soactivitiesinvolvedavarietyofexpressionmethodssuchasdrawing,speaking,sharingwithapartner,andindividualwriting.Questionswerecreatedtoevaluatethefollowinggoal:TheNMEvalLab’sGoal1.1:“Tosupportstudentsindevelopingtheknowledge,skills,andtoolstheyneedtodesignandimplementprogramevaluationsthatreflectorganizations’valuesandmissions.”Collaboratively,wedecidedonthefollowing:

a) Whatdidthecoursefeellikeforstudents?b) Whatdidstudentsexperienceincourse/curriculumvs.inpracticum?c) Whatdidstudentsexperienceinrelationshipswithorganizations?Whatwerethe

challengesandbenefits?d) Whatdidstudentslearnfromtheirmentors?e) Whatwastheexperienceofworkingasateammember?(Waseveryonevalued,did

everyonecontribute?)f) Whatskillscanbeattributedtothestudents’participationinthelab?g) Wouldstudentsrecommendtheprogramtootherstudents?h) Howcoulditbebetter?

Findings:

a) Whatdidthecoursefeellikeforstudents?Theanswertothisquestionwasachievedthroughadrawingactivity,wherestudentscoulddrawwhatthecoursefeltlikeonagingerbreadperson.Overall,studentseitherfeltthatthecoursewasoverwhelmingandconfusing(oftenrepresentedwithquestionmarksovertheeyesorbrain,orthebrainbe“fried”),representedbyFigure1.ortheyfeltempowered,engaged,andstimulatedbythecourse(representedbysmiles,hearts,andfeelingsofsuccess)asshowninFigure2.Insomecases,illustratedinFigure3.,studentsfeltconfusedatfirst,butempoweredinthesecondsemester.Moststudentsmentionedstressandexhaustion.

2

Figure1.Student’sexperienceincourse(confusingandoverwhelming)

Figure2.Student’sexperienceincourse(empowering&stimulating)

3

4

Figure3.Student’sexperienceofthefirstsemestervs.secondsemester

Thegingerbreadpersonactivitybroughttolighttheknowledgedifferencebetweenallparticipants,whichwasmentionedagaininthefocusgroupactivitiesthatfollowed.b) Whatdidstudentsexperienceincourse/curriculumvs.inpracticum?Studentsfrequentlyhighlightedguestpresenters,conferences,workshops,andpersonalfeedbackfromSoniaandMelissaonprojectsasbenefitstothecurriculumportionoftheEvalLab.Withinthecurriculum,studentscommentedthatthediscussionboardwasnotas

5

helpfulandthathavingadiversecohortmeantstudentswereatdifferentlevelsofunderstanding(somethinkitwasboring,othersthinkitwasoverwhelming).Duringthepracticumportionofthecourse,studentsenjoyedworkingandbuildingrelationshipswithorganizationsandhavinginformationentrustedtothem.However,studentsfeltoneofthemoredifficultpartsofthepracticumcamefromorganizationwithintheorganizations(asintheorganizationscanbeorseemdisorganized).Overall,fellowsreportedcommunicationbetweenteamsandorganizationsasthenumberoneissueinthepracticum.c) Whatdidstudentsexperienceinrelationshipswithorganizations?Whatwerethe

challengesandbenefits?Byworkingwithcommunityorganizations,studentsgainedpractical/realworlduseforacademic/evaluationresearch,wereabletobridgetheiracademictrainingintothecommunity,andfeltproudtohaveanimpactonthecommunity.Studentsfeltthebiggestchallengeintherelationshipwiththeorganizationswasthetimeschedules(thefactthatorganizationsarebusyandschedulingwashard).Somealsonotedthatitwashardtofigureoutwhattheorganizationsreallywantedevaluated.d) Whatdidstudentslearnfromtheirmentors?Overall,studentsrespondedverypositivelyabouttheirmentors,mentioningthattheirmentorshelpedthemimprovetheirwritingskills,professionalism,timemanagement,andpresentationskills.Theirbiggestconcernsaboutthementoringprocesswerethatitwassometimesunclearwhowasresponsibleforwhatandsomefeltthatallmembersoftheirteamwerenottreatedequally.Onestudentmentionedfrustrationwiththementorbysaying,“[Iwas]frustratedbythetreatmentfrommentorinteam.[I]feltsheoversteppedmeeverytimeandtreatedmeassomeonewhowasunintelligent.”Therewasnocommentonhowtoalleviatethisinthefuture.e) Whatwastheexperienceofworkingasateammember?(Waseveryonevalued,did

everyonecontribute?)Inmostcases,responsesaboutworkingasateamwerepositive.Inthepositiveresponses,studentsmentionedlearningfromtheirteam,balancingthework,andfeelingliketheywerecontributingandsupported.However,therewereafewstudentswhoreportedthattherewasfrustrationbecausetheyfelttheyweredominatedbytheotherteammembers.Studentsalsomentionedthatthechangeinclassschedulesfromonesemestertothenextwasuncomfortable/inconvenient.f) Whatskillscanbeattributedtothestudents’participationinthelab?ThenumberoneskillstudentsattributetotheirparticipationintheEvalLabiscommunication.Studentsalsohighlighteddatavisualization,presentations,publicspeakingskills,excelskills,amongafewothersthoughcommunicationwasbyfarthemostrepeated.InFigure4,theWordlebelow,communicationisevidentasthegreatestskillstudentsattributedtotheirparticipationintheEvalLab.

6

g) Wouldstudentsrecommendtheprogramtootherstudents?Everystudentwhoansweredthisquestion,answeredintheaffirmativethatyes,theywouldrecommendtheEvalLabtoafriend/fellowstudent.However,somementionedtheywouldonlyrecommendifthesuggestedchangesweremadeorwouldrecommendbutwiththedisclaimerthatthereisalotofwork,meetings,obligations,etc.Onefellowwrote,“Iwould[recommendit].However,Iwouldalsoletthemknowofallofthechallenges.Theorganizationsmaynotbeeasytoworkwith,workingwithateamcanbegreat,butalsochallenging.Allofthetimerequirements(sitevisits,posterpresentations,etc).Iwouldalsosaytonottakemorethan12credithoursandnottodoanotherinternshipwhiledoingtheEvaluationLab.Notonlyforyoursanity,butalsotoenableyoutobemoreflexible.

Meanwhile,otherswerethankfulfortheexperiencetheyhad:“Yes,IwouldrecommendtheEvalfellowshiptomostpeople.Itwasarichandhelpfulexperience.Thankyou.”“ThankyoufortheblessingandopportunitytoworkwithEvalLab.Theprogramandthefacilitator,professorsoftheEvalLabhavemadethisexperienceablessing.Pushingmehardertoworkandunderstandhowtodoresearchandorganizewithcommunityorganizations.”

7

h) Howcoulditbebetter?Hereiswhatstudentsreallyliked:

• Havingachoiceinaligningwithorganization• Notfeelingconstrainedbyevaluationexamples• Sitevisits• Mentors• MelissaandSonia• Postersession• Money/stipend(laughter)"butreallyitwasoneofthereasonsIapplied"• Workingwithorganization(willcontinuetoworkwiththemhereafter)• Diversityofcohort• SeniorFellows(helpfulbutnottotallynecessary—otherswhodidn'thavethem

didn'tfeelliketheymissedout)• Happyhourswiththeorganizations• Guestpresenters• Datavisualization

Hereiswhatstudentssuggestbedonedifferently:

• Centralizegroupfeedbackonreports,maybeinachain(onepersonatatime)• Examplesofevaluationscouldofferoutlinestowrapheadaroundwhatit

[evaluation]couldbe• Clearlystateexpectationsforsitevisitsaheadoftimeinclass• Changetimeofsitevisits(hardtofitaroundclassschedules)• Calendarshouldbepublishedfrombeginningofsemestertoendwithoutside

events(sitevisits,workshops,presentations,etc.)• Shouldofferclasstimetodiscuss/introducemarginalizationasinstitutionalproblem

(notjustmentionedassurfaceissueordemographicissue)-->"justasimportantasIRBtraining"

• Discussionboardsneedtobemorespecificifcontinuingtouse"seemedlikesomepeoplejustrespondedbecausetheyhadto,wasn'tproductive"(thisisalsorepeatedfromthespeedroundofactivities)

• Giveoptiontochoosepartner/whoyouwanttoworkwith• Workflows/materialsoutlinestosupportalllevelsoflearninginclass(camefrom

discussionofsomestudentsthinkingthecoursewastooslowwhileothersthoughtitwastoofast/overwhelming)

ConclusionsandRecommendations:Returningtothegoalofthisfocusgroup,todetermineiftheEvalLabismeetingGoal1.1“Tosupportstudentsindevelopingtheknowledge,skills,andtoolstheyneedtodesignandimplementprogramevaluationsthatreflectorganizations’valuesandmissions,”Ibelievetheanswerisyes.Thoughthestudentsdidhaveafewrecommendationsforimprovement—theyclearlyfeltthefrustrationsofworkingwithoverworkedcommunityorganizationsandin

8

teams—overall,theyfeltthattheEvalLabhelpedthemgrowincommunication,publicspeaking,workingwithExcel,andpresenting,amongothers.StudentsenjoyedtheirworkwiththeEvalLab.Inmanycases,studentsfeltstressedbutalsosupportedandbytheendtheywereveryproudoftheirproducts.ItappearsthatthestressandconfusiondidnotprohibitstudentsfromperformingwellintheEvalLaboverall,sincemanyreportedthattheirorganizationswereverysatisfiedwiththefinalproduct.Studentswerealsoverythankfulandverywillingtorecommendtheexperiencetotheirpeers.Thisisverypositivebecause,whilestudentsmayhavefoundthecoursechallenging,theyultimatelyfeltitwasapositiveexperiencethatcausedthemtogrow.Ontheotherhand,somestudentsdidn’tfeeloverwhelmed,andinsteadfeltthecurriculumofthecoursemovedtooslowlyandwasboringattimes.Studentsnoticedthattherewasalargedividebetweenthosewhoalreadyknewalotaboutevaluationandthosewhowerestartingfromscratch.Inthecasethatstudentsfeltunderwhelmed,theysuggestedthatperhapsworkflowsoradditionalmaterialcouldbesuppliedtothosethatarestrugglinginordertosupportalllevelsoflearningintheclass.Thiswasatopicthatsurfacedindrawing,writing,andspeakinginpairs.