recommendations to improve course and ......recommendations to improve course and teacher...

RECOMMENDATIONSTOIMPROVECOURSEANDTEACHEREVALUATIONS

May31,2017

Contact:SylviaAvery,ExecutiveAssistantTheMacPhersonInstituteMcMasterUniversityHamilton,ONL8S4M4(905)525-9140,ext.24540

T A B L E O F C O N T E N T S

EXECUTIVESUMMARY 1

Introduction 2

TERMSOFREFERENCE 3

COURSEANDTEACHEREVALUATIONSCOMMITTEE(CTEC) 3

TEACHINGEFFECTIVENESS 4

RECOMMENDATIONS 4

InitialRecommendations 4SubsequentRecommendations 4

POLICYONCOURSEANDTEACHEREVALUATIONQUESTIONS 6

StatementofPurpose 6Principles 6Responsibilities 7FormativeFeedback 7SummativeEvaluation 7PeerReviewofTeaching 8

CONCLUSIONSANDIMPLEMENTATION 8

APPENDIXI—COMMUNITYENGAGEMENT 9

APPENDIXII—TEACHINGEFFECTIVENESS 10

Complexity 10KeyFindings 10

APPENDIXIII—INSTITUTIONALQUESTIONS 11

Introduction 11ValidatedandReliableQuestions 11InstitutionalQuestions 12

APPENDIXIV—INCREASINGSTUDENTRESPONSERATES 13

Findings 13Recommendation1—CourseOutlines 13Recommendation2—In-ClassInformation 14Recommendation3—SuggestedRoomProtocol 14

APPENDIXV—REFERENCES 15

CourseandTeacherEvaluations Page1

E X E C U T I V E S U M M A R Y

ThemandateoftheCourseandTeacherEvaluationCommittee(CTEC)wastoexamineMcMasterUniversity’ssystemforevaluatingteachinginrelationtoinstructionalquality,instructionimprovementandcourseanddepartmental/programadministrationaswellasforthepurposesofinstructors’advancementthroughthetenure,permanenceandpromotionprocesses.Aspartofitsmandateandoverthecourseofatwo-and-a-half-yearperiod,theCommitteeundertookwide-rangingconsultationswiththemajorstakeholdersintheMcMastercommunityandexaminedtheacademicliteratureconcerningtheevaluationofteachingandlearning.StakeholdersexpressedanumberofconcernsthatledtheCommitteetoconcludethatthecurrentsystemofCourseandTeacherEvaluationsisnotservingtheMcMastercommunitynortheindividualstakeholdersaswellasitshould.Themostsignificantconcernsincludedtheperceivedlackofreliabilityinthecurrentevaluationsystem(bothsystematicbiasandlackofstatisticalreliability)aswellasthelimitedinformationprovidedbythesummativequestionforthetenure,permanence,promotionandotheradministrativeprocessesandtheoveralluse(orlackthereof)oftheevaluationdata.Basedonitsconsultationsandresearch,theCommitteerecommendstheimplementationofanewandcomprehensiveteachingandlearningevaluationprocess.ThisnewevaluationprocesswillrequirecontinuingactionfromtheUniversityinvolvingfourareas:

1. Policy,includingtheadoptionofanewcourseevaluationinstrument2. Infrastructure3. Governancestructure4. CommunicationstrategyaimedatallmembersoftheMcMasterCommunity.

Basedonthisbroadandextensiverevisiontotheteachingandlearningevaluationprocess,theCommitteerecommendsthefollowingmeasuresbeundertakenonaninitialandimmediatebasis:

1. TheadoptionofanewpolicyontheevaluationofteachingandlearningatMcMaster;2. Thecreationofanewteachingandcourseevaluationinstrumentbasedonacascadingevaluation

systemapproachthatincorporatesuniversitywideandcommonassessment(consistingofninequestions)aswellasspecificfaculty,departmental,programandindividualinstructorselectedquestions;

3. Theimplementationofarangeofspecificmeasuresdetailedinthisreporttoincreasestudentresponserates.

Theserecommendationsrepresenttheinitialstepinchangingtheteachingandevaluationsystem.Furtherworkwillbeneededtoaddresstheremainingpolicyandotherissues.


Introduction TheCourseandTeacherEvaluationCommittee(CTEC)beganitsworkin2014toexplore,frombothformativeandsummativeperspectives,theroleofstudent-completedcourseevaluationsaswellastheevaluations’roleintheprofessionaldevelopmentofinstructorsandtheiruseinCP/MandT&Pdecisions.TheCommitteealsoprovidedavehicleforinstructors,students,administratorsandstafftoprovideinputontheprocessandthestatusofcourseandteachingevaluations–aninclusiveapproachinkeepingwiththespiritofForwardWithIntegrity.TheCTECmetregularlywithandheardfromvariousmembersoftheMcMastercommunity–allofwhomexpressedatleastsomelevelofdissatisfactionwiththecurrentprocessusedtoengagestudentsintheevaluationofteaching.Studentleadersexpressedthatstudentsfeltdisengagedfromtheevaluationprocessandoftendidnotcompletecourseevaluationsbecausetheybelievedtheirfeedbackwasneitherconsideredimportant,nortakenseriously.Someprofessorsperceivedtheevaluationprocesstobebiased,unreliableandinmanycasesunfair.Personnelcommitteeswerecriticizedforoveremphasizingoneortwosummativedatapointsduringmerit,tenureandpromotiondecisions.Administratorsstruggledtobalancetheseviewsinlightofinstitutionalprioritiestomeasureteachingeffectiveness.Together,theseconditionspointedtoacourseandteachingevaluationsystemchallengedbyconcernsaboutfairness,credibilityandtrust.TheCommitteealsoheardconcernsaboutchangingthecultureofteachingandlearning.Specifically,theCommitteereceivedsuggestionstorecognizeandrewardteachingthroughmorenuancedandricherways,tofindwaysofimprovingthepeerevaluationofteaching,andtothinkofthewiderimplicationsonprogramandcurriculumreviewdrivenbymoreevidence-basedandeffectiveteachingpractices.Questionswerealsoraisedaboutclarifyingexpectationsregardingthescholarshiponteachingandlearningfromteaching-streamprofessorsaswellasexistinglimitationsontheuseofevaluationdataforpedagogicalresearchpurposes.Foracompletelistofmeetingsandpresentationsfromvariousstakeholders,pleaseseeAppendixI.OtherCanadianuniversitiessharemanyoftheseissuesandconcerns.Somehaverespondedbydrawingonbestpracticeandevidencedrawnfromtheliteraturethatguidesthedesignofaneffectivecourseandteacherevaluationsystem.Practically,effectiveevaluationsystemsinvolvefourintersectingareastocreatecontinuousimprovementinlearningandteaching.Theseareasare:(1)Policy;(2)Governance(3)Infrastructureandarobust(4)CommunicationStrategy.Thefirstdomainlaysafoundationbasedonasetofclearpoliciesandproceduresderivedfromprinciples,goalsandresponsibilitiesthatengageallstakeholders.Administrativepoliciesandproceduresdrivetheoperationalsystemsusedtoimplementandsustaingoodpracticesandprovidetheunderlyingframeworkforcontinuousimprovementinteachingandlearningacrosstheinstitution.Thesecondelementisagovernancemodelthatencompassesoperationalsystemsthatensuretheimplementationandsustainabilityofthesystem.Thethirdelementisareliabletechnologyinfrastructure.Finally,thefourthelementrequiresthatallstakeholdersbeengagedthroughacentralcommunicationstrategy.Centralizedandconsistentcommunicationisindispensibletotheprocessofestablishingculturalchangeataninstitutionallevel.Whileseveralelementsofthefourareascanbephasedin,theareasareinterconnected;weaknessesinanyoneofthesefourareaswillundermineconfidenceanddevaluetheentirecourseandteacherevaluationsystem.Thisiswhyitiscriticalforallfourareastobeadequatelysupportedandresourced.ThisreportidentifiesthreespecificrecommendationstobegintheprocessofestablishinganewsystemforcourseandteacherevaluationsatMcMasterUniversity.ThefirstrecommendationfocusesonanewUniversitypolicythatdescribestheunderlyingprinciplesandvalues.Thesecondrecommendationprovidesanevidence-basedcourseevaluationinstrumentconsistingofupto15questions.Thesequestionsincludeninecommoninstitutionalquestionsanduptosixquestionsdeterminedatthefacultyandinstructorlevels.Allquestionswillbedrawnfromavalidateditembank.Thethirdrecommendationsuggestschangestoexistingcourseevaluationpracticestoincreaseoverallstudentparticipation.


Thisreportalsoidentifiesbasicframeworksfortheremainingareas:governance,operationalresponsibilities,technologyinfrastructure,andcommunicationstrategies.Theseframeworksarenotoperationalplans,sotheCommitteerecommendsthattheProvostidentifytheappropriateindividualsand/orworkgroupsrequiredtobuildbriefreportsorguidesineachofthesefourareasinordertoprovidealevelofoperationaldetailnotprovidedinthisdocument.T E R M S O F R E F E R E N C E

TheProvostprovided theCTECwith a strongmandate tomakerecommendationstoenablethecontinuousimprovementofteachingatMcMaster,withaparticularemphasisoninnovationinteachingandlearning.Theserecommendationsweretobebasedoncombiningbest practices, evidence-based research findings and broadUniversity-wide inputtodeveloppedagogical supportmechanismsandmakethemosteffectiveuseofteachingevaluationdata.TheProvostalsoencouragedtheCTECtoconsiderandpursueboldrecommendationsinordertobetterassesseffective teachinganduseinformationfromtheevaluationprocesstoimprovethequalityofprogramsandstudentexperiencesingeneral. C O U R S E A N D T E A C H E R E V A L U A T I O N S C O M M I T T E E ( C T E C )

TheCommitteeworkedintwophases.Committeemembersarelistedalphabetically.ConsultationPhase(January2015–November2016)

• ArshadAhmad,AVPTeaching&LearningandDirector,MIIETL• NarayanaswamyBalakrishnan,Professor,MathematicsandStatistics• CarlosFilipe,Professor&Chair,ChemicalEngineering• GregFlynn,AssistantProfessor,PoliticalScience• HannaHolmes,AssistantProfessor,Economics• AnnaMoro,AssociateDean,Humanities• AlanNeville,AssociateDean,HealthProfessionalEducation• SpencerSemianiw,MSUVice-President(Education)• SusanSearlsGiroux,AVPFaculty(Chair)• SarahSymons,AssociateProfessor,SchoolofInterdisciplinaryScience

ReportWritingPhase(November2016–May2017)

• ArshadAhmad,AVPTeaching&LearningandDirector,MacPhersonInstitute(Chair)• CarlosFilipe,Professor&Chair,ChemicalEngineering• GregFlynn,AssistantProfessor,PoliticalScience• AnnaMoro,AssociateDean,Humanities• AlanNeville,AssociateDean,HealthProfessionalEducation• BlakeOliver,RyanDeshpande,MSUVice-President(Education)• SarahSymons,AssociateProfessor,SchoolofInterdisciplinaryScience

TheCommitteeisgratefultothemanycolleagueswhopresentedtheirvaluableinsights(AppendixI).ItisalsoindebtedtotheseveraldirectorsofteachingandlearningcentresacrossCanadawhoprovidedreportsandadvice,includingtheframeworkfromUniversityofToronto,whichassistedsignificantlyinthedevelopmentofkeysectionsofthisreport.Also,facilitationbyErinAspenliederandAmyGullagefromTheMacPhersonInstituteandadministrativesupportfromSylviaAveryandCaraJaneDempseywasappreciated.


T E A C H I N G E F F E C T I V E N E S S

OneoftheunderlyingtensionsindevelopingacommoncourseandteacherevaluationframeworkatMcMasteristheexistingsiloswithindepartments,programsandfacultieswithlittlesharedunderstandingorconsiderationofwhateffectiveteachingmeans.Deployingbestpracticestobuildasharedunderstandingrequiresopeningnewlinesofcommunicationformultiplestakeholderstoengageanddocumentexistingpractices.Wealsobelievethatitmeansencouragingflexibilityintheevaluationsystemsothatitisabletorecognizediverseperspectivesonapproachestoteachingwhilealsoaccommodatingdisciplinarydistinctionsandotherdifferences.Asdepartmentsandprogramstakestockofexistingevaluationstructuresandprocessesanddeveloptransparentmetricstocommunicatewhateffectiveteachingmeans,thereisasignificantbodyofresearchtoguidetheirefforts.TheresearchfindingsidentifiedinAppendixIIprovideaneffectivestartingpointforthoseconversations. R E C O M M E N D A T I O N S

Thisreportidentifiesthreerecommendationsforimmediateadoptionaswellasrecommendationsthatwillrequireadditionalthought,researchandreportstoprovidecontextandthedetailsrequiredfortheirimplementation.Initial Recommendations 1. Adoptthepolicydescribedinthisreport(page6)oncourseandteachingevaluations.2. Adoptacascadingsystemtodevelopatotalof15itemsinthecourseevaluationinstrument.Thisis

comprisedofnineUniversitywidequestions(AppendixIII)anduptosixquestionsdrawnfromavalidquestionbankbyfacultiesanddepartmentsaswellasindividualinstructors.Accesstoavaliditembankandrelatedrequestsfordata,itssecurity,storageandotherconcernsareaddressedinitem1below.

3. ImplementthesuggestionsmadetoIncreaseStudentResponseRatesbyraisingawarenessaboutcourseevaluationsandaddingrequiredinformationoncourseoutlines(AppendixIV).

Subsequent Recommendations

1. CreatetheProvost’sGuidelinesonCourseandTeacherEvaluationstoincludethefollowing:a) Governanceandresponsibilities

• Institutional(systemadministration,reportingfunctions,training,etc.)• Faculty/Departmental(guidelines,reporting,dataverification,etc.)• Shared(transition,timing,etc.)• Data(access,requests,optingout,security,storage,retention,etc.)

b) Infrastructure

• SelectingDatabase:technicalcapabilities,anonymity,credentials,permissions• Userinterface:laptop,tablet,smartphone• Responsetimes:accessibility,browsers,operatingsystems,hardware• Automationtocollate,tabulate,quantitativeandqualitativeanalysis• Externalvendor


2. CreateaculturechangebydevelopingrobustcentralcommunicationstrategiesacrosstheUniversityto:a) Faculty(allinstructionalpositions)

• Underliningtheimportanceoftheevaluationofteaching• Communicatingtransparentlywhatteachingeffectivenessmeans• Establishingguidelinesforusingcourseevaluationdataforcoursedevelopmentandpedagogical

research• Communicatingtheimportanceoffacultyengagementwiththecourseevaluationprocess,

particularlytostudentsb) Students

• Workingwithstudentsaspartnerstoprovidevaluableinput• Alertingstudentstopossibleareasofimplicitbias• Interpretingcourseandteachingevaluationdatawhenavailable

c) Administrators• Communicatingtransparentlyhowhiringandpersonnelcommitteeswillusedatatoevaluate

teachingeffectiveness• Establishingproceduresthatenablecourseevaluationdatatobeavailableforprogramreview

developmentandpedagogicalresearch• IdentifyingspecificprovisionsintheYellowDocumentandCP/Mtoaddressconcernsraisedby

faculty,studentsandstaffhighlightedinthisreportandinAppendixI• Opennewcommunicationchannelstoaddressongoingissuesrelatedtoany/allaspectsofcourse-

andteacher-evaluationprocessesAspartofthecommunicationstrategy,ashortguideforeachstakeholdergroup(faculty,students,andadministrators)shouldbecreatedtogatherrationale,bestpractice,FAQsandresources.


P O L I C Y O N C O U R S E A N D T E A C H E R E V A L U A T I O N Q U E S T I O N S

CourseevaluationsarepartofMcMasterUniversity’scomprehensiveteachingevaluationframeworkthatincludesregularpeerreview,instructorself-assessment,theuseofteachingportfolios,cyclicalprogramreviewandotherformsofassessment,asappropriate.Aspartofthisframework,courseevaluationsareaparticularlyusefultoolforprovidingfacultyandstudentswithanopportunitytoprovidefeedbackontheirownlearningexperiences.Courseevaluationsarealsoameanstopromoteaccountabilityandtransparency.ResponsibilityforthispolicywouldrestwiththeOfficeoftheProvostandVice-President,Academic.

McMasterUniversityrecognizestheneedtoconductcourseevaluationsforthefollowingreasons:1. Toprovideinstructorswithformativeinformationtocontinuouslyimprovetheirteaching.2. ToprovidestudentswithinformationaboutinstructorsandcoursesatMcMaster.3. Toprovidesummativeteachingevaluationdataaspartialinputforadministrativepurposessuchasannual

merit,tenureandpromotionreview.4. Toprovidedatausedbydepartmentsandfacultiesforcourseandprogramreview.5. Toprovideadatabaseofcourseinformationforcontinuousimprovementinteachingandlearning.

Statement of Purpose ThepurposeofthispolicydocumentistooutlinetheprinciplesandconsiderationsthatguidetheevaluationofcoursesatMcMasterUniversity.ThespecificsofhowthecourseevaluationprocesswillbeexecutedinparticularcontextswillbeoutlinedintheProvost’sGuidelinesonCourseandTeacherEvaluations.TheProvost’sGuidelineswillprovideadditionaldetailsontheadministration,useandinterpretationofcourseevaluationatMcMaster.AdditionalFacultyGuidelinesdevelopedatthefacultylevelwillindicatethespecificapproachtakentocourseevaluationwithinanindividualfaculty.Theactualcourseevaluationinstrumentwillincludeinstitutionalquestions,Faculty/Departmentquestionsandinstructor-selectedquestions.Thispolicy,andtheProvost’sGuidelinesonCourseandTeacherEvaluations,formMcMasterUniversity’sinstitutionalcourseevaluationframework.Principles McMasterUniversity’sinstitutionalcourseandteacherevaluationframeworkshould:1. Gatherinformationfromstudentsabouttheirlearningexperiences.2. Provideopportunitiesforbothformativeandsummativefeedbackonteaching.Formativereferstoboth

time–whenthecourseisongoingaswellasimprovement,whereassummativereferstotheendofthecourseandisassociatedwithevaluation.

3. Reflectinstitutionalteachingandlearningprioritiesthatcombinebestpracticesandevidence-basedresearchfindings.

4. Recognizethediversityofteachingprioritiesandstrengthsacrosstheinstitution.5. Beequitable,consistentandtransparentinthecollection,useandinterpretationofdata.6. Protecttheanonymity/confidentialityofstudentrespondents.7. Providereliableandmeaningfuldatatoinstructors,administratorsandstudents.8. Provideguidancetodepartment,programchairsandadministratorsonaspectsofinstructorfeedbackand

courseevaluationprocesses.9. Considermultiplesourcesofinformationincludingcourseevaluationdataandotherrelevantitemsina

teachingportfolioinordertoschedule,assignandallocatecourses.10. Ensurethecourseevaluationinstrumentisstatisticallyrobust.11. Encouragefacultytoinnovateandtakeriskswithoutpenalizingtheirtenureandpromotionprogress.12. Supportpedagogicalresearchtoimproveevidence-basedapproachesthatcanbeappliedatMcMaster

Universityandinbroaderacademiccommunities.


Responsibilities TheMcMasterUniversitycommunityacknowledgestheimportanceofformativefeedback,summativeevaluationsandpeerreviewofteachingascriticalprocessesinenhancingtheUniversity’scoursesandprograms.• The Institution—McMasterUniversityhasaresponsibilitytooverseetheimplementationofthispolicy

andprovideeducationaswellassupporttostudents,instructorsandadministratorsabouttheuseandimportanceofteacherandcourseevaluations.

• Our Faculties—Eachfacultyhasaresponsibilitytodevelopitsownguidelinesthatarealignedwiththeinstitutionalframeworks.

• Administrators—Thedean/chair/principal/directorisresponsibleforreviewingcourseevaluationdataincludingquantitativeandqualitativedata,asavailable,asonecomponentinassessingteachingeffectivenessandinunderstandingtheguidelinesforinterpretingcourseevaluationdata.Theyarealsoresponsibleforensuringthatarobustpeerevaluationsystemisinplacewithintheirareasofsupervisionandthatitispartoftheoverallsystemforevaluatingbothcoursesandteaching.

• Instructors—AllinstructorsareresponsibleforunderstandingtheroleofformativefeedbackandsummativeevaluationsatMcMasterUniversity;explainingtheimportanceofbothformsoffeedbackandevaluationtostudents;reviewingtheirowninstructorfeedbackandcourseevaluationsregularly;discussingtheseresultswiththeirdepartment/facultyheadandincludingtherequiredparagraphincourseoutlinesoncourseevaluationsaspertheundergraduatecoursemanagementpolicies(SeeAppendixIVforguidelines).

• Students—Havearesponsibilitytoparticipatethoughtfullyintheformativeandsummativeevaluationprocessesbyprovidingconstructivefeedbackandrecognizingtheimportanceofgivingfeedbackandcourseevaluationsasanopportunitytoinfluencethequalityoftheirexperienceinbothcurrentandfuturecourses.

Formative Feedback Everycourseinstructorisresponsibletoconductformativefeedbackeachtimeacourseisoffered.Supportforthistypeofactivityisavailablecentrally(e.g.throughtheMacPhersonInstitute)andwithinsomefaculties,(e.g.,FDAinEngineering;PFD&CASCADEinHealthSciences).Eachinstructorshalldeterminethenatureoftheformativefeedbackinquiriestobemadewithstudents,includingconsiderationofbothcoursedeliveryandcoursecontentandtheoverallstudentlearningexperience.Onceinstructorsreceivetheformativefeedback,theymayelecttoreleasethisdatatoacademicadministratorsand/ortostudents.Summative Evaluation Eachcoursewillbeassessedonasummativebasiseachtimeitisofferedbutcouldbewaivedincertainexceptionalcircumstances.Facultieswillmakeprovisionsforobtainingstudentfeedbackbyalternativemeansincoursesofanindividual/independentnature(e.g.independentstudycourses,musicstudios,practica)orcourseswithverysmallenrolmentsasdefinedbyeachfaculty.Incourseswithteachingassistantsormultipleinstructors,evaluationswillalsoassesstheindividualcontributionsofeachofthemembersoftheteachingteam.Summativecourseevaluationdatawillnotbereleaseduntiltheofficialapprovaloffinalgrades.Summativeevaluationswillbeavailableto:

§ Instructors—willhavefullaccesstoallquantitativeandqualitativedatafromcourseevaluationsconductedineachcoursetheyhavetaught.Quantitativedatawillbemadeavailablepublicallyunlesstheinstructoroptsoutforagivencourseandsemester.Exceptinextraordinarycircumstances,theinstitutional-basedquestionswillnormallybemadeavailabletotheMcMasterUniversitycommunityinkeepingwiththepurposesofthispolicy.

§ Academic Administrators—willhaveaccesstodataexceptfrominstructor-selectedquestions.§ Students—Studentswillnormallyhaveaccesstonumericaldataand/orwrittencommentsunlessthey

haveoptedoutoftheprocess.


Peer Review of Teaching Accordingtoarecentliteraturereview(Thomaset.al.,2013)thePeerReviewofTeaching(PRT)isaneffectivestrategytokick-starttransformationalreformsacrossdisciplinesandcultureswithinaninstitution,especiallyifthisisdonecollaborativelythroughtheuseofteachingportfolios.ThesameliteraturereviewidentifiedanumberofconfoundingfactorsincludingthethinlinebetweenconsensusandconformityinagreeingtothecriteriathatconstituteeffectiveteachingaswellasthetrainingrequiredforPRTskills.McMasterstakeholdersagreedthatthepolicyinthisareaasexpressedinthe“YellowDocument”isadequate,buttheimplementationandinterpretationofthepolicyisanareawheretherearestrikingdifferencesofpracticebetweendepartmentsandfromfacultytofaculty.Inconcertwithnewguidelinesfortheinterpretationofstudentcourseevaluationdata,TheMacPhersonInstituteshouldtakealeadtodevelopingsimilarappropriateguidelines.C O N C L U S I O N S A N D I M P L E M E N T A T I O N

TheCommitteeencounteredagreatdealofcommongroundintheeducationalliterature,inconsultationswithotherinstitutions,andinitsengagementwiththeMcMastercommunity.Forexample,auniversalchallengewithcourseandteacherevaluationsisthetensionbetweenglobalinstitutionalpriorities/interestsandlocalpracticeswhichvaryconsiderablyfromonedepartment/facultytoanother.Thistensioncanberesolvedbyadoptingacascadingsystemofvalidquestionsstartingwithacommoncoreandaddingmorethatareselectedbyfaculty/departmentsandinstructors.Moreimportantly,courseevaluationdataarevaluableonlyiftheyareusedwithinabroaderframeworkoftheevaluationofteaching.Cherrypickingoneortwodatapointsfromthecourseevaluationinstrumentisinsufficienttoassessteachingandoftentriggerconcernsaboutfairness,credibilityandtrust.TheCommitteehasrespondedtotheseconcernsbyencouragingmoretrainingforthoseengagedintheprocessofevaluationandtheuseofdatatoinformamoresophisticatedassessment(seenextstepsbelow).Additionalstepsthatcanbeimplementedrightawayhavealsobeensuggested.Oneofthesestepsistostronglyencourageformativepractices.Formativedataprovidesvaluablefeedbackforinstructorstotakeimmediateactionsostudentscanbenefitbeforethecourseiscompleted.Thispracticeisoneofthemostimportantdeterminantsofincreasingstudentresponserates.Studentresponseratesarealsoinfluencedpositivelybyshortquestionnaires,asmoststudentswillparticipateinseveralformativeevaluationsthroughoutthecourseandwillbecompletingfiveorsixquestionnairesinashorttimeframejustbeforepreparingforfinalassessments.Thisexplainsthelimitof15questions–ninecommonquestionsanduptosixadditionalquestionsfromfaculty/departmentsandinstructors.Finally,aneffectivecourseandteacherevaluationsystemmustsimultaneouslyconsiderfourareasthatthisreporthasidentified,includingtheadoptionofthepolicyoutlinedinthereportandthesimultaneouscreationofmechanismsforresponsiblegovernance,technologyinfrastructureandcommunication.Whileelementsofeachoftheseareascanbephasedinasoutlinedintheimplementationplanbelow,thefunctioningoftheentiresystemmustguideeachoftheelements.Weaknessesinanyoneofthesefourareascanundermineconfidenceandweakentheentirecourseandteacherevaluationsystem.ThisiswhyitiscriticalforeachdomaintosupporttheothersothatMcMasterUniversitycanrealizethebenefitsofcontinuousqualityimprovementsinteachingandlearning.TheCTECrecommendsthatataskforce/workinggroupbesetuptoimplementrecommendations.Suchataskforceshouldincludememberscriticalfortheimplementationfromtheperspectivesofinfrastructure,governanceandcommunication.


A P P E N D I X I — C O M M U N I T Y E N G A G E M E N T

MeetingsandPresentations:

• January26,2015—ConsultationCommitteemembers• February23,2015—RafaelKleiman,MUFAPresident• March11,2015—DeansFeedbackatProvost’sCouncil• March17,2015—RodrigoNarroPerez,VP,Education,MSU• March31,2015—DelHarnish,FacultyofHealthScience• April30,2015—SpencerNestico-Semianiw,VP,Education,MSU• July21,2015—JohnBell,HumanitiesMedia&Computing• September11,2015—DeniseChalmers,VisitingProfessor,UniversityofWesternAustraliaand

MickHealey,VisitingProfessor,UniversityofGloucestershire• September14,2015—ConsultationCommitteemembers• November23,2015—ConsultationCommitteemembers• December4,2015—ConsultationCommitteemembers• December15,2015—ConsultationCommitteemembers• January28,2016–BarbBloemhof,ChiefSteward,Unit2,CUPE• March22,2016—DavidWilkinson,Provost• April28,2016–PeterVilks,AssistantProfessor,DeGrooteSchoolofBusiness• June1,2016–KristaMadsen,AssistantProfessor,Kinesiology• October5,2016—TorgnyRoxa,VisitingProfessor,LundUniversitypresentedtoPC• November30,2016—ReportingCommitteemembers• February14,2017—ReportingCommitteemembers• March1,2017—ChairmetwithMUFAmembership• March15,2017—ReportingCommitteemembers• April5,2017—ReportingCommitteemembers• April12,2017—ReportingCommitteemembers• April17,2017—ReportingCommitteemembers• May4,2017—ReportingCommitteemembers• May15,2017—ReportingCommitteemembers

Ingeneral,thepresentationshighlightedcommentsinthefollowingkeyareas:

• McMasterUniversityTenureandPromotionPolicy• Peerreviewofteaching• Programandcurriculumreview• Cultureofassessment• Studentviewpoints• Teachingpractices• Scholarshiponteaching

TheMcMastercommunityhasclearconcernsthatneedtobeaddressedaspartoftheimplementation,andhasindicatedastrongorganizationalreadinessforchange.


A P P E N D I X I I — T E A C H I N G E F F E C T I V E N E S S

Complexity Entwistle(2010)providesanexcellentoverviewofinfluentialtheoriesandmodelsofteachingeffectivenessinTakingstock:Researchonteachingandlearninginhighereducation.Hesubstantiateswhatiswidelyaccepted–thateffectiveteachersareknowledgeableabouttheirsubjects,aboutteachingpracticesandabouthowstudentslearn.Hereiteratesthecurrentviewthatteachingeffectivenessiscontextualandgroundsthisinconceptionsoflearningoutcomesand“waysofthinkingandpracticinginthedisciplineorprofessionalarea”(p.4).Still,thereisadisconnectbetweenwhatispracticedandwhatisknowntoidentifyandevaluateteachingeffectiveness.Thereis,however,agreementintheliteraturethatguidesourgeneralconceptsoftheapproaches,behavioursandtraitsofeffectiveteachers(Knapper,2010,2013).Thesepointsofagreementprovideastartingpointforindividualsandgroupswithindepartmentsandprogramstoarticulate,contrastandshapetheirconceptionsofwhateffectiveteachingmeanstodayandhowthismightchangeanddevelopovertime.Key Findings Effectiveteachingisuniversallyacceptedtoembodyseveralprinciples.ThesearebestarticulatedinoneofthemostwidelyciteddocumentsbyChickeringandGamson(1987)andinclude:encouragingcontactbetweenstudentsandfaculty,developingreciprocityandcooperationamongstudents,encouragingactivelearning;givingpromptfeedback,emphasizingtimeontask,communicatinghighexpectations,andrespectingdiversetalentsandwaysoflearning.Generally,effectiveteachersmoveawayfromatraditionalfocusonthetransmissionoffacts,heavyemphasisoncontent,andassessmentsthatencouragestudentstoengageinsurfaceapproachesandrotelearningcharacterizedbysuperficialunderstanding(Biggs,1987;Biggs,Kember,Leung,2001,Marton,1976).Instead,effectiveteachersengageinlearner-centredteachingwhichencouragesstudentstoengageindeepapproachestolearningcharacterizedbymorecriticalthinking,selfdirection,integrationandintrinsicmotivation(Marton&Säöȷl̈ojö,1997).Muchhasbeenwrittenaboutlearning-centredteaching(Barr&Tagg,1995)whichshiftsthefocusfromtheteachertowhatthestudentsdo,recognizingtheimportanceofpriorlearning,intellectualdevelopment,etc.See,forexample,theinfluentialworkofDewey(1904/1965),Perry&Smart(1997)andVygotsky(1978).Thereisavastarrayofliteraturesupportingeffectiveteachingstrategiesincludinginquiry-basedapproaches(Justice,et.al,2009),problem-basedlearning(Neufeldetal.,1981;Woods,1983;Jenkins,2007;Neville&Norman,2007),service-basedlearning(Boyer,1996;Ward&Lisa,2000)andcase-basedlearning(Barnes,Christensen&Hansen,1994).Alsosee,forexample,meta-analysesthatfindapositiveeffectofteachingstrategiesonacademicachievement(Prince&Felder,2006;andSpringer,Stanne&Donovan,1999).Anotherrecenteffectiveteachingstrategyisassociatedwiththresholdconceptsinthediscipline,whichtransformthestudent’spriorunderstandingofthesubject.Byfocusingonthesethresholds,whicharealsodescribedas“portals”or“transformativewaypoints”,instructorscancreateacontextinwhichstudentlearningcanbesignificantlyaccelerated(Land,Meyer&Smith,2008;Meyer&Land,2006).


A P P E N D I X I I I — I N S T I T U T I O N A L Q U E S T I O N S

Introduction WeproposetousethemodelusedattheUniversityofToronto,amodelthatprovidesforasetofuniversalquestionsthatreflectstheoverallteachingandlearningprioritiesoftheentireinstitution.IntheMcMasterimplementation,nineinstitution-widequestionswillbeusedforeverysectionofeverycourseoffered,regardlessoffaculty,schoolordepartment.TheCourseandTeacherEvaluationCommittee(CTEC)developedthesequestions.Thesequestionsbegineverycourseevaluationform,andareseenbyeverystudentintheinstitution.Followingtheinstitution-widequestion,thesurveyformswillcascadetoasmanyasthreequestionsthatarespecifictoaparticularfaculty/divisionand/ordepartment.Thedeanorcentralcommitteeofanyfaculty,seenonlybystudentsinthatfacultyordivision,andusedtogatherfaculty-ordivision-specificdata,willdevelopthesequestions.Finally,thesurveyformswillcascadetoasmanyasthreequestionsspecifictoaparticularinstructorandselectedbyinstructorstobeansweredonlybystudentsofthatinstructor.Allquestionsaredrawnfromavalidatedquestionitembank.Thismodelcombinesthebenefitsofbothacentralizedanddecentralizedapproach,satisfyingtheneedforconsistentinstitution-widedata,whilecollectingmoregranularinformationfromeachdifferentdivision,departmentandinstructor.Itprovidesforbothglobalandlocalconcernsandstrikesafinebalancebetweeninstitutionalandlocalteachingandlearningpriorities.Validated and Reliable Questions Thecourseevaluationadministrationmustcreateandmaintainamasterlistofeveryitemallowedonanycourseevaluation,commonlycalledan“itembank”.Anyfaculty,department,orinstructorcansubmitnewitems,buteveryitemmustbethoroughlyexaminedandvalidatedbeforebeingaddedtothebank.Thisensuresthateveryitemateverylevelalignswithassessmentprinciplesforvalidresults.Forexample,anydouble-barreled,inconsistent,orunclearquestionsmustberevisedordiscarded,eitherbythecourseevaluationteamorbyoutsideexpertswhocanhelpwiththistask.Theliteraturepointstomanybestpracticesforcreatinganeffectiveitembank.Forexample,iftheassessmentpriorityisforstudentstoprovideinformationontheextenttowhichtheinstitution’steachingexpectationswaspartoftheirlearningexperiences,theitemsshouldthenbeexpressedasdeclarativestatements.Studentsshouldbeaskedtowhatextenteachitemwaspartoftheirlearningexperience(i.e.notatall,somewhat,moderately,mostly,oragreatdeal).Usingresponsescalesthatcorresponddirectlywiththeintentoftheassessmentpriorityhasbeenshowntogeneratefarbetterdatathanaskingstudentstosimplyagreeordisagreewithasetofstatements(Krosnick&Presser,2010).Ofcourse,allofthistakestimeandexpertise.Anyinstitutionlookingtosavetimeontheirimplementation,whileensuringthatallitemsarethoroughlyvalidated,canexplorelicensinganexistingitembank,ratherthanreinventingthewheelatmuchhighercost.


Institutional Questions Usingthefive-pointscalebelow,indicatetheextenttowhichyouagreewiththestatementsthatevaluateyourcourseandinstructors.

1 2 3 4 5Stronglydisagree Disagree Neutral Agree Stronglyagree

1. Theinstructormadethesubjectinteresting2. Overall,Ifoundthatthequalityofinstructor'steachinginthiscoursewasexcellent.3. Courseprojects,assignmentsandtestsand/orexamsprovidedanopportunityformetodemonstratemy

understandingofthecoursecontent.4. Comparedtoothercourses,Ifoundtheworkloadforthiscoursetobechallenging.5. Overallthequalityofmylearningexperienceinthiscoursewasexcellent.6. Iwouldrecommendthiscoursetoanotherstudent.NOTES:• Thetotalnumberofquestionsshouldnotexceed12questionsplusanopen-ended“Pleasefeelfreetomake

anyadditionalcomments”sectionattheend.

• Uptothreequestionsaretobedeterminedbythefaculty/departmentascommontoalltheircoursesanduptothreemorequestionsmaybeselectedbytheinstructoronaper-coursebasis.Thiswillallowinstructorsandprogramstoaddressquestionsofparticularinterestforeachcourse.


A P P E N D I X I V — I N C R E A S I N G S T U D E N T R E S P O N S E R A T E S

Findings Relevanteducationalliteratureidentifiesanumberofstrategiesthathaveproventobeeffectiveinincreasingstudentresponseratesincourseandteacherevaluations:1. Persuadinginstructorsandadministratorsabouttheimportanceofcourseevaluations

2. Persuadingstudentsthattheirrecommendationswillbeactedupon

3. Persuadingstudentsthattheirfeedbackwillinformcourseandinstructorchanges

4. Demonstratingimplementationoffeedbackbyusingmid-semesterevaluationsandmakingchangesbased

onfeedback

5. Makingnumericresultsofevaluationsavailabletostudents

6. Explaininghowtheinstitutionusesfeedbackfromevaluations

7. Askinginstructorsroutinelytosharehowpastfeedbackhasinformedtheirdecisionsregardingtexts,assignments,format;doingsothroughoutcourses

8. Usingincentivesforparticipationasalastresortonlyafterexhaustingotherapproachessuchasdedicatingtimeinclass,providingexamplesofpastchangesbasedonevaluations,usingformativefeedbackandannouncingresponserates

9. Explaininghowdifferentclasssizesrequiredifferentresponseratestobemeaningful–seetablefromtheUniversityofSaskatchewanreport:MarginofError ClassSize No.ofRespondents ResponseRate +/–10% 25 20 80%(20/25) +/–10% 50 33 66%(33/50)

+/–10% 100 49 49%(49/100)+/–10% 200 65 33%(65/200)

Acommonalityinthediscussionofstudentresponseratesfromotheruniversitiesandcollegesthathavechangedtheircourseevaluationprocessisthatstudentengagementwiththeevaluationprocessisheavilydependentoninstructorengagement.Evenstrongcommunicationcampaignsdirectedatstudentscannotovercomeanyperceptionofapathyamonginstructors.Recommendation 1—Course Outlines SubjecttoapprovalfromSenate,theRevisedUndergraduateCourseManagementPolicyre.courseoutlinesshouldincludethefollowingparagraph:“Towardstheendofthiscourse,youwillbeinvitedtocompleteanonlinecourseevaluationform.Theevaluationconsistsofasetofquestionsabouttheeffectivenessofthecourseanditsinstructioninthecontextofyourlearningexperiences.StudentfeedbackisavaluablepartofensuringthatthepedagogicalqualityofcoursesremainshighacrosstheUniversity.Werequestthatyoutakepartinevaluatingallcoursesthatyouaretakingandrespondthoughtfullytothequestions.Youranswerswillbeanonymousandwillbeusedforcourseimprovementandinstructorevaluationpurposes.”


Recommendation 2—In-Class Information Tosupportabetterunderstandingoftheimportanceofcourseevaluations,eachinstructorisencouragedtouseaslidedeckthatwilladdressesthefollowingquestionstobereviewedinfiveminutesand/orpostedonlineforstudents.Theslidedeckwillbepartofthecommunicationsstrategydevelopedcentrally.1. Whatcourseandteacherevaluationsinvolve?

a) Screenshotofaccessingsystem b) Screenshotoftypicalevaluationform

2. Whydoweevaluatecoursesandteachers?

a) Evidence-basedteachingb) Improvement,flexibilityandinclusivityc) Enhancedcourseandbetterteachers

3. Howiscourseandteacherevaluationdataisused?

a) Instructorfeedbackandcareerprogressionb) Courseimprovementc) Qualityassuranced) Institutionaldatae) Needasubstantialresponseratesothatresultsaresignificant

4. Howiscourseandteacherevaluationsbenefityou?

a) Opportunitytocommentb) Openresultsmeanyoucanmakebetter-informedchoicesofcoursesc) Communityparticipation

5. Whatarewereallyevaluating?

a) Notionsofqualityofteachingb) Warningsofbiasc) Professionalismd) Spaceforinstructorstogiveexamplesofhowthey'veusedevaluations,possiblediscussionof

changesmadethisyear

Recommendation 3—Suggested Room Protocol Thisprotocolhasbeenfoundtobeeffectiveinincreasingonlinestudentresponseratesatotherinstitutions:1. Announcethataparticularclasswillincludeabrieftimetoconductcourseevaluationsandaskstudentsto

bringatablet,laptop,phoneordevicethatallowsthemaccessandensuretheclassroomhasWi-Fi.

2. Setaside15to20minutesatbeginningofclass,preferableduringa‘peakattendance’day.

3. Instructor/TAsintroducescourseevaluationandleavesclass.

4. AppointstudenttotellInstructor/TAswhentoreturn.


A P P E N D I X V — R E F E R E N C E S

Barnes,B.,Christensen,R.C.,&Hansen,A.J.,(1994).TeachingandtheCaseMethod.HarvardBusinessPress.Barr,R.B.,&Tagg,J.(1995).Fromteachingtolearning-anewparadigmforundergraduateeducation.Change(Nov.Dec.),13-25.Biggs,J.B.(1987a).StudentApproachestoLearningandStudying.Hawthorn,Victoria:AustralianCouncilforEducationalResearch.Biggs,J.B.Kember,D.&Leung,C.Y.P.(2001).Therevisedtwo-factorstudyprocessquestionnaire:R-SPQ-2F.BritishJournalofEducationalPsychology,71–133-149.Boring,A.,Ottoboni,K.,&Stark,P.B.(2016).Studentevaluationofteaching(mostly)doesnotmeasureteachingeffectiveness.ScienceOpenResearch.DOI:10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1Boyer,Ernest.1996.Thescholarshipofengagement.JournalofPublicServiceandOutreach.1(1).Chickering,A.W.&Gamson,Z.F.(1987).Sevenprinciplesforgoodpracticeinundergraduateeducation.AAHEBulletin.Dewey,J.(1904/1965).Therelationoftheorytopracticeineducation.InM.L.Borrowman(Ed.),TeachereducationinAmerica:Adocumentaryhistory.NewYork:TeachersCollegePress.Entwistle,N.J.(2010).Takingstock:Anoverviewofresearchfindings.InJ.C.HughesandJ.Mighty(Eds.),Takingstock:Researchonteachingandlearninginhighereducation.MontrealandKingston:McGill-Queen'sUniversityPress.Hativa,N.,Barak,R.,&Simhi,E.(2001).ExemplaryUniversityTeachers:KnowledgeandBeliefsRegardingEffectiveTeachingDimensionsandStrategies.TheJournalofHigherEducation.Vol.72,(6),699-729Land,R.,Meyer,J.H.F.&Smith,J.(2008).Thresholdconceptswithinthedisciplines.RotterdamandTaipei:Sensepublishers.Jenkins,H.(2007).TheoriginsofinquiryinMcMaster’sArtsandScienceprogram.InC.Knapper(ed.),ProceedingsfromExperienceswithInquiryLearning,Hamilton,ON:McMasterUniversity.Justice,C.,Warry,W.,&Rice,J.(2009).Academicskilldevelopment-Inquiryseminarscanmakeadifference:Evidencefromaquasi-experimentalstudy.InternationalJournalfortheScholarshipofTeachingandLearning,3(1),9.Knapper,C.K.(2010).Changingteachingpractice:Barriersandstrategies.InJ.C.Hughes&J.Mighty(Eds.),Takingstock:Researchonteachingandlearninginhighereducation(pp.229–242.).McGill-Queen’sUniversityPress.Knapper,C.K.(2013).Theimpactoftrainingonteachereffectiveness:Canadianpracticesandpolicies.InE.Simon&G.Pleschová(Eds.),Improvingteachinginhighereducation:Thechallengesofcreatingeffectiveinstructionaldevelopmentprograms(pp.53–68).NewYork,NY:Routledge.Marton,F.&Säljö,R.(1976).Onqualitativedifferencesinlearningoutcomesandprocesses.BritishJournalofEducationalPsychology,464-11.Marton,F.&Säljö,R.(1997).Approachestolearning.InF.Marton,D.J.Hounsell&N.J.Entwistle(Eds.),Theexperienceoflearning(2nded.)(pp.39-58).Edinburgh:ScottishAcademicPress.


Meyer,J.H.F.&Land,R.(2006).(Eds.)Overcomingbarrierstostudentunderstanding:Thresholdconceptsandtroublesomeknowledge.London:Routledge.Neville,A.J.,&Norman,G.R.(2007).PBLintheundergraduateMDprogramatMcMasterUniversity:threeiterationsinthreedecades.AcademicMedicine,82(4),370–374.Neufeld,V.R.,Norman,G.R.,Feightner,J.W.,&Barrows,H.S.(1981).Clinicalproblem-solvingbymedicalstudents:across-sectionalandlongitudinalanalysis.MedicalEducation,15(5),315–322.Perry,R.P.&Smart,J.C.(Eds.)(1997).Effectiveteachinginhighereducation:Researchandpractice.NewYork:AgathonPress.Prince,M.&FelderR.(2006).Inductiveteachingandlearningmethods:Definitions,comparisonsandresearchbases.JournalofEngineeringEducation,95(2),123-138.Springer,L.,Stanner,M.E.&Donovan,S.S.(1999).Effectsofsmall-grouplearningonundergraduatesinscience,mathematics,engineering,andtechnology:Ameta-analysis.ReviewofEducationalResearch,69(1),21-51.Thomas,S.,Abraham,M.,Raj,S.J.&Beh,L-S.(2014).Aqualitativereviewofliteratureonpeerreviewofteachinginhighereducation.84(1),112-159.DOI:https://doi.org/10.3102/0034654313499617Vygotsky,L.S.(1978).Mindinsociety.Thedevelopmentofhigherpsychologicalprocesses.Cambridge,MA:HarvardUniversityPress.Ward,K.&LisaW-W.(2000).Community-CenteredServiceLearning:FromDoingFortoDoingWith.AmericanBehavioralScientist.43:767-80.Woods,D.R.(1983).IntroducingExplicitTraininginProblemSolvingintoOurCourses.HigherEducationResearch&Development2.1:79–102.

recommendations to improve course and ......recommendations to improve course and teacher...

Documents