Slide 1

Peer Reviewer Training WorkshopShea Wang & Trudi OhkiCAFSept. 22, 2014www.macewan.ca/facultycommons/ Faculty CommonsFaculty Commonswww.macewan.ca/facultycommons/ Workshop OverviewUpon completion of this workshop, you will be able to:Conduct a pre-observation meetingReview teaching materialsUse the narrative, checklist, rating, and data collection forms for classroom observationConduct a post-observation debriefWrite a final post-observation reportFaculty Commonswww.macewan.ca/facultycommons/ Activities Overview Practise using the pre-observation meeting form Practise using the review of teaching materials formPractise classroom observation skills and techniques Practise using the narrative, checklist, rating, and data collection forms for classroom observation Practise post-observation debrief Practice post-observation report writingFaculty Commonswww.macewan.ca/facultycommons/ Faculty Evaluation Policies: Instructional Faculty C5065Non-Instructional Faculty C5066www.macewan.ca/facultycommons/ Faculty CommonsFaculty Commonswww.macewan.ca/facultycommons/ Evaluation at MacEwanPolicy C5065 guided by the philosophy that we:promote a culture of teaching and learningfoster professional development and scholarly activitypromote fairness and transparencytimely and accurate feedbackcomply with collective agreementsundertake as a process involving multiple stakeholders, and a variety of assessment approachesFaculty Commonswww.macewan.ca/facultycommons/ Faculty strive for excellence in teaching, which is assessed by:teaching (including curriculum development)research, scholarly, or artistic engagement service (collegiate citizenship)collegiality (professionalism, respect for students & peers, and ethical behaviour)professional developmentEvaluation at MacEwanwww.macewan.ca/facultycommons/ Faculty CommonsFaculty Commonswww.macewan.ca/facultycommons/ C5065 Instructional FacultyRequirements based on appointment typeContinuingProbationarySessional/Sessional-extendedTeaching DossierPeer ReviewSessional-extendedat renewal (Collective Agreement 2011-14)Annual ReportStudent Online FeedbackPerformance Reviewwww.macewan.ca/facultycommons/ Faculty CommonsFaculty Commonswww.macewan.ca/facultycommons/ C5065 Probationary Requirements Teaching Dossier - AnnuallyPeer Review - Two reviews during each of the first two years (4 total)Annual Report - AnnuallyStudent Online Feedback - Every course during appointmentPerformance Review - AnnuallyFaculty Commonswww.macewan.ca/facultycommons/ IntroductionWhat is Peer Review?Scholarly publications review process to assess quality of the workEvaluation of classroom/online teaching and teaching materialsASAC position review committeesAny evaluation process where the evaluators are peersFaculty Commonswww.macewan.ca/facultycommons/ Peer Review LiteraturePeer review is appropriate forCourse content, expertise in field of studyCourse objectives and materialsAppropriate teaching methodologyCourse organizationAppropriate tests, assignments, evaluation methodsSeldin, 1999Faculty Commonswww.macewan.ca/facultycommons/ Peer Review at MacEwanTeaching effectivenessClassroom visitsTeaching materialsASAC position review committeesFaculty Commonswww.macewan.ca/facultycommons/ Qualifications of Peer ReviewersCriteria developed by FECCompletion of an appropriate peer review training course and / or demonstration of peer review experience Minimum 3 years full-time teaching experiencePedagogical expertise Possess a continuing appointmentRecommendation by Dean / Associate Dean / Director / senior administratorFaculty Commonswww.macewan.ca/facultycommons/ Attributes of Peer ReviewersUtilizes effective instructional practices (demonstrated through dossiers, annual reports, awards, etc)Demonstrates collegialityProvides constructive, objective feedback focusing on teaching behaviours and improvementModels ethical behaviourAppreciates different styles / modes of teachingFaculty Commonswww.macewan.ca/facultycommons/ ExceptionsFor all peer reviewers, should there be a deficit in meeting the recommended qualifications, the Dean, Associate Dean, Director, or senior administrator will assess and provide final approval. Faculty Commonswww.macewan.ca/facultycommons/ Choosing Peer ReviewersChairs will support and advise faculty members in the selection of two peer reviewers to conduct classroom observations and review of relevant teaching materials. It is not necessary that peer reviewers be content-area experts. Peer reviewers can expect to be asked to perform peer reviews outside of their home department(s).Faculty Commonswww.macewan.ca/facultycommons/ Defining Effective TeachingConversations about teaching (30min)Pair and Share Break into small groups and discuss the following sentences:I most enjoyed teaching when __________I knew I had a problem when __________Have a member of your group report on your discussion to the main group*Chickering and Gamson, 1987Faculty Commonswww.macewan.ca/facultycommons/ Measuring Good TeachingMythsDr. FoxExpressiveness High Ratings Reflect Lenient GradingWorkload/course difficulty Rank and experience Gender

https://facultycommons.macewan.ca/headlines/professors-from-sfu-share-mean-student-evaluations

Faculty Commonswww.macewan.ca/facultycommons/ Dr. Fox In what has come to be known as the Dr. Fox study, a trained actor, introduced as Dr. Fox, delivered a lecture on mathematical game theory to a group of medical educators (1973). Dr. Fox presented incorrect information, cited nonexistent references, and used neologisms as basic terms. Nonetheless, the great majority of Dr. Foxs audience rated his lecture favorably. The study produced a term that is still heard in discussions of ratings: the Dr. Fox effect. The term refers to the use of an entertaining style to seduce students into giving favorable evaluations to a teacher who is weak on content. Critics of ratings have seized on this study as strong evidence for the invalidity of student ratings, but rating experts are quick to point out that the study has many methodological flaws - Almost every feature of the study is problematic. Dr. Fox paradigm does not apply to student ratings of teaching.

Expressiveness The research on instructor expressiveness, like that surrounding personality andpopularity, is complicated and sometimes unsound. Some studies have established clearly thatexpressiveness tends to enhance learning and therefore cannot be considered a biasing factor(Cashin, 1995).

High Ratings Reflect Lenient Grading The correlation between effort and grading leniency is small and actually positive with some measures of student effort -- good teaching, might stimulate students to perform well in a course (and thus receive high grades) and might also lead students to give the course high ratings. The rule is that students rate most highly teachers from whom they learn most.

Workload/course difficulty Although many faculty believe that harder courses or higher workloadresults in lower evaluations, this has not been supported by the research which has producedinconsistent results (Marsh, 1987). Easy courses are not guaranteed higher evaluations.Additionally, some studies have shown that difficult courses and/or those with a higher workloadreceive more positive evaluations (Cashin, 1988).

Rank and experience dApollonia & Abrami (1997) and Arreola (2000) find that these variables donot significantly affect evaluation results. Marsh (2001) found that experience does not lead toimproved ratings and may in fact have a negative relationship with teaching effectiveness.

Gender In general, studies relating to gender have produced inconclusive results, but most haveshown that this variable has little or no impact on evaluations (Algozzine et al., 2004; Theall &Franklin, 2001; Marsh & Roche, 1997; Cashin, 1995; Arreola, 2000; Aleamoni & Hexner, 1980).

17Measuring Good Teaching

Faculty Commonswww.macewan.ca/facultycommons/ Dr. Fox In what has come to be known as the Dr. Fox study, a trained actor, introduced as Dr. Fox, delivered a lecture on mathematical game theory to a group of medical educators (1973). Dr. Fox presented incorrect information, cited nonexistent references, and used neologisms as basic terms. Nonetheless, the great majority of Dr. Foxs audience rated his lecture favorably. The study produced a term that is still heard in discussions of ratings: the Dr. Fox effect. The term refers to the use of an entertaining style to seduce students into giving favorable evaluations to a teacher who is weak on content. Critics of ratings have seized on this study as strong evidence for the invalidity of student ratings, but rating experts are quick to point out that the study has many methodological flaws - Almost every feature of the study is problematic. Dr. Fox paradigm does not apply to student ratings of teaching.

Expressiveness The research on instructor expressiveness, like that surrounding personality andpopularity, is complicated and sometimes unsound. Some studies have established clearly thatexpressiveness tends to enhance learning and therefore cannot be considered a biasing factor(Cashin, 1995).

High Ratings Reflect Lenient Grading The correlation between effort and grading leniency is small and actually positive with some measures of student effort -- good teaching, might stimulate students to perform well in a course (and thus receive high grades) and might also lead students to give the course high ratings. The rule is that students rate most highly teachers from whom they learn most.

Workload/course difficulty Although many faculty believe that harder courses or higher workloadresults in lower evaluations, this has not been supported by the research which has producedinconsistent results (Marsh, 1987). Easy courses are not guaranteed higher evaluations.Additionally, some studies have shown that difficult courses and/or those with a higher workloadreceive more positive evaluations (Cashin, 1988).

Rank and experience dApollonia & Abrami (1997) and Arreola (2000) find that these variables donot significantly affect evaluation results. Marsh (2001) found that experience does not lead toimproved ratings and may in fact have a negative relationship with teaching effectiveness.

Gender In general, studies relating to gender have produced inconclusive results, but most haveshown that this variable has little or no impact on evaluations (Algozzine et al., 2004; Theall &Franklin, 2001; Marsh & Roche, 1997; Cashin, 1995; Arreola, 2000; Aleamoni & Hexner, 1980).

18Measuring in contextTake 15 min. and complete the following sentences in your group:I knew my teacher was a good teacher when/becauseI knew my teacher was a poor teacher when/becauseFaculty Commonswww.macewan.ca/facultycommons/ 19Literature on Good TeachingEight dimensions of college instruction :Course organization & planningFaculty communicationFaculty/student interactionAssignments, exams, & gradingInstructional methods & materialsCourse outcomesStudent effort & involvementCourse difficulty, workload, & pace(Marsh, 1984 & 1987)

Faculty Commonswww.macewan.ca/facultycommons/ Literature on Good Teaching (2)Encourages student interactionEncourages student cooperationActive learningPrompt feedback to studentsEmphasizes time on taskHigh expectationsRespect(Chickering and Gamson, 1987)Faculty Commonswww.macewan.ca/facultycommons/ 21Literature on Good Teaching (3)Good teachersAre good learners;Display enthusiasm;Know how to modify their teaching strategies;Encourage learning for understanding;Demonstrate an ability to transform and extend knowledge, rather than just transmit it; Set clear goals;Provide high-quality feedbackShow respect for their students(Ramsden, 2003)Faculty Commonswww.macewan.ca/facultycommons/ 22Overall Good TeachingParker Palmer capricious chemistryGood teaching is ultimately irreducible to a simple technical formulaCharacteristics are beyond common descriptors or attributes of a checklist

Solution: multifaceted approach Student feedback, peer review, self-reflection, and administrative evaluationFaculty Commonswww.macewan.ca/facultycommons/ 23Peer Review ActivityIn your small group, reflect on your past experiences with giving and receiving peer feedback. Address the questions: What did you learn from the experience? In what ways it was a positive/negative experience? Did it influence your instruction?How did receiving and giving feedback change the way you thought about teaching and learning?

Faculty Commonswww.macewan.ca/facultycommons/ 24Peer ObservationThe two forms of observation:Summative observation - This observation is an evaluation. It can be done for administrative or supervisory reports for the purpose of job retention, promotion, salary increases, etc.Formative observation - The purpose of a formative peer observation is to help instructors become better teachers and more knowledgeable professionals.

Faculty Commonswww.macewan.ca/facultycommons/ 25Peer Review Process at MacEwanEvaluation Policy outlines two main areas of teaching assessed by peer reviewers: Teaching materials - lecture notes, course outlines, tests and exams, assignments, grading rubrics, etc.; and Instructional delivery - observation of teaching in formal lectures, as well as other delivery modes such as labs, studio, clinical, and online/distance formatsFaculty Commonswww.macewan.ca/facultycommons/ 26Peer Review Process at MacEwanFive main elements: Pre-observation meetingReview of teaching materialsClassroom observation Post-observation debriefPost-observation reportFaculty Commonswww.macewan.ca/facultycommons/ 27FormsSome of the forms availableNarrative formChecklist formRating formData Collection

(see appendices of workbook)Faculty Commonswww.macewan.ca/facultycommons/ Pre-Observation MeetingEmotionally, it is important to address distrust of evaluation, violation of personal style and space, insecurities about time, and fear of bias (Chism, 2007)Essential that trust be established - Acknowledge the emotional aspect. Clear objectives of observation should be outlined - Communicate expectationsStressed that the process is collaborative - Engage in active listening skills(Review pre-observation forms in appendices)Faculty Commonswww.macewan.ca/facultycommons/ The observer verbally puts herself in the position of learning from the person being observed by saying she had never taught that class before.The observer indirectly compliments the person being observed by saying, Sounds pretty challenging and sounds interesting.The observer pays attention to what the observed instructor is saying. She takes notes. She looks at her attentively. And she uses back-channeling, or verbal expressions to show she is paying attention. For example, OK, uh huh, sure, good, good, sounds great, and, Im looking forward to it.The instructor being observed answered every question as cooperatively and completely as she could. She tried to give the observer the information she needed in order to make an informed observation. Two important points to notice in this preparation phase are that the two instructors spoke in a friendly and respectful way to each other. They both tried to be as cooperative as possible. Also, the observer tried to get as much information as she could about the class she planned to observe.29Course MaterialsSome examples of materials include:SyllabusCourse guidesCourse packets and assigned textsPowerPoint Presentations , overheads, BlackBoard, etcHandoutsVideos and other multimedia supplementsProject assignment directions and handoutsFaculty Commonswww.macewan.ca/facultycommons/ Procedure for Evaluation of Course MaterialsImportantly, we are not focusing on student achievement measures when reviewing teaching materials. Rather, we are looking at content, design, clarity, etcMaterials enable reviewers to see an instructor's philosophy enacted, and can alert reviewers to the areas in which a given instructor excelsMaterials can promote discussion about the overall goals

(Review course materials checklist in appendices)Faculty Commonswww.macewan.ca/facultycommons/ Given the specialized nature of many course materials, peer reviewers may be unfamiliar with the content of specific courses and appropriateness of the level of communication. However, we believe that peers can review the materials for basic format, tone, and the clarity of the materials. It may be necessary to solicit the opinions of others in the field of specialization to review accuracy and currency of content.

31Classroom ObservationObservers should use the same observation form they should take separate minutes/notesAt least two observers should attend each class observation sessionAt least two visits is preferableNo unannounced visitsDiscuss with the instructor the format of the review prior to visit: Where will you sit? Will you be introduced?Faculty Commonswww.macewan.ca/facultycommons/ Classroom ObservationDuring the Classroom VisitBe unobtrusive (basically, remain quiet) Do not ask questionsDo not volunteer answersDo not correct the instructorBe attentiveNote specific examples of observed behaviours (positive & negative)

Faculty Commonswww.macewan.ca/facultycommons/ Classroom Observation ChallengesOne class may be not representativeAtypical performance (good or bad)Observer may not have a good understanding of the contextPersonal biases on what constitutes good teachingFaculty Commonswww.macewan.ca/facultycommons/ 34Remove points from here for handoutsSome SolutionsAttend 2+ classesVideo tape class?Focus-group students after class?Use the data collection form, cite examplesDiscuss visit with co-reviewer, debrief togetherContextualize the course reflect back on:Pre-observation meetingExamination of course materialsFaculty Commonswww.macewan.ca/facultycommons/ Peer Observation ExercisePre-observationIn your small group, decide on the form you will be using for review (narrative, checklist, rating)Review form and highlight areas that each reviewer will be responsible for focusing on and taking notes about

ObservationChoose one of the two scenarios to reviewReflect on the class from the following viewpoints: Viewpoint 1: students Viewpoint 2: instructor

Faculty Commonswww.macewan.ca/facultycommons/ Peer Observation ExerciseCompare your list of observer behaviors with others in your groupBe reflective and systematic through the criteria outlined in your observation formEmbed your observations & conclusions in examples of observed behavioursDid the instructors behavior appear to be what you might expect in a formal observation?

Faculty Commonswww.macewan.ca/facultycommons/ Peer Observation ExerciseRole PlayIn your small group, designate an individual to be the instructor, the reviewer, and a reporterBreak into large groups based on role as instructors & reviewers & reportersCompare the notes your groups took during the observation and discuss strategies for approaching the debrief (and how the characters might react)Break back into your small group and role play the post-observation debrief meeting

Faculty Commonswww.macewan.ca/facultycommons/ Post Observation DebriefMeet with the instructors immediately following the classroom visitAsk the instructor to describe how they felt the class went in relation to the goals, objectives and expectations outlined in the pre-observation meetingAt this time, the peer reviewers should bring up any immediate concerns that were highlighted by the observation session. **This is an important aspect of the review as it can potentially change the observers perceptions about what they saw and heard.Faculty Commonswww.macewan.ca/facultycommons/ Post Observation DebriefProcedureAsk the person who taught how s/he felt the lesson wentReconfirm the observation focus points for the final reportOffer positive comments on what you think went wellIdentify any urgent issuesAsk questions for clarification, as neededEnd on a courteous noteProvide an expected timeline for the final report and final meeting

Faculty Commonswww.macewan.ca/facultycommons/ Post-Observation ExercisePrepare yourself to care watch that you are not overly forgiving or overly critical (put yourself in their shoes)Use descriptive language rather than evaluativeBe responsive to the goals outlined in the pre-observation meetingCheck to make sure the instructor has heard your message as you intendedBe forward-looking and positive in your discussion, i.e. what can be improved over what went wrong

Faculty Commonswww.macewan.ca/facultycommons/ Peer Review Final ReportPolicy C5065 statesSection 4.4.4, Peer reviewers shall prepare brief written assessments, grounded in evidence, and submit a copy of reports to the faculty members and Chairs. Faculty Commonswww.macewan.ca/facultycommons/ Who Sees the Final Report?Faculty member used to improve teaching effectiveness, make adjustments as necessary for application for continuing status

Chair and Dean used for administrative review

ASACs used for personnel decisionsFaculty Commonswww.macewan.ca/facultycommons/ Characteristics of a Good ReportReference the goals established during the pre-observation meetingProvide a written record of what was observed and what was discussedConsider the audienceUse examples from the observationLanguage is clear, concise, avoids jargon and abbreviationsTone is supportive and phrasing is positiveAction-oriented and provides specific and practical suggestions for improvementFaculty Commonswww.macewan.ca/facultycommons/ Characteristics (contd)One, joint report per instructor - Structure of report should be agreed upon by all reviewersReturn report in a timely mannerReview with faculty member within 3 days of classroom visitConsider faculty member response before submitting for administrative reviewConfidential

(Review report template in appendices)

Faculty Commonswww.macewan.ca/facultycommons/ Peer Review for Online CoursesOnline Peer Review Form (4 categories)Student behaviourFaculty-student interaction (multiple interactive communication strategies)Technology supportLearning environment

Faculty Commonswww.macewan.ca/facultycommons/ Challenges - Peer Review for Online CoursesA connection between specific materials and instruction and student action that manifest learning objectives.

Technology and accessibility Are instructors responsible for these? Yes, no, maybe, more nowadays?

Faculty Commonswww.macewan.ca/facultycommons/ What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instruction:Clear objectives as stated in:Syllabus: must be clear. Super important.Grading Rubrics: explains how they will be graded

Contd.Faculty Commonswww.macewan.ca/facultycommons/ 48What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instruction

Clear objectives as stated in:

Modelsor Examples of quality workfrom successfully submitted and graded student work

Explanations of the relevance of course materials to learning.

WHY are we doing this? (think of students perspective).

Contd.Faculty Commonswww.macewan.ca/facultycommons/ 49What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instructionAssessment tools should be used:Throughout the course to establish:Learning styles

Skill/knowledge level

Challenge preconceptions

Clarify current learningContd.Faculty Commonswww.macewan.ca/facultycommons/ 50What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instructionOpportunities for students to construct and experience their own knowledge:

We all learn differently and its important to be clear and understanding about this.

Learning processes and outcomes are not all the same.

Contd.Faculty Commonswww.macewan.ca/facultycommons/ 51What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instructionOpportunities for students to become increasingly responsible for their own learning:

Instructor should provide consistent and timely input.Encouragement, good examples, and instructor presenceTry to develop intrinsic motivation.Contd.Faculty Commonswww.macewan.ca/facultycommons/ 52What Is A Good Online Course?Content? Easy use of technology? Interaction?Learner-centered outcomes-based instructionOptions for demonstrating learning outcomes (if appropriate)Opportunities for reflectionEffective student support servicesContd.Faculty Commonswww.macewan.ca/facultycommons/ 53What Is A Good Online Course?Clear and captivating instructional designthe writing style, the images and plug-ins selected, and the layout are appropriate to the intellectual sophistication, technical savvy and needs of the audiencea pathway to guide student learningappealing presentationContd.Faculty Commonswww.macewan.ca/facultycommons/ 54What Is A Good Online Course?Varied forms of interactivity with material, other students and the instructor

Regular Communication with peers and instructor

Varied Interaction with Content

Source: http://www.4faculty.org/includes/digdeeper/online/lc_outcomes_based_instruc.htmFaculty Commonswww.macewan.ca/facultycommons/ Online Course Material for ReviewInstructional material and resourcesAssignments and student assessment (clear and detailed instructions and guidelines, purposes, formal writing standards/academic writing requirement)Discussion forums (ability for students to give feedback and ask questions during the class is critical)E-mail messagesCourse syllabus (the syllabus is the most important document of the entire course. It is the road map or game plan for the course) Chat spaceFaculty Commonswww.macewan.ca/facultycommons/ Where to Go For ResourcesCAF website:Then click on Faculty EvaluationThen click on Peer Review

Faculty Evaluation Coordinator Faculty Commonswww.macewan.ca/facultycommons/