Slide 1

Peer Reviewer Training Workshop Shea Wang & Trudi Ohki CAF Sept. 22, 2014

Slide 2

Workshop Overview Upon completion of this workshop, you will be able to: Conduct a pre-observation meeting Review teaching materials Use the narrative, checklist, rating, and data collection forms for classroom observation Conduct a post-observation debrief Write a final post-observation report

Slide 3

Activities Overview Practise using the pre-observation meeting form Practise using the review of teaching materials form Practise classroom observation skills and techniques Practise using the narrative, checklist, rating, and data collection forms for classroom observation Practise post-observation debrief Practice post-observation report writing

Slide 4

Faculty Evaluation Policies: Instructional Faculty C5065 Non-Instructional Faculty C5066

Slide 5

Evaluation at MacEwan Policy C5065 guided by the philosophy that we: promote a culture of teaching and learning foster professional development and scholarly activity promote fairness and transparency timely and accurate feedback comply with collective agreements undertake as a process involving multiple stakeholders, and a variety of assessment approaches

Slide 6

Faculty strive for excellence in teaching, which is assessed by: teaching (including curriculum development) research, scholarly, or artistic engagement service (collegiate citizenship) collegiality (professionalism, respect for students & peers, and ethical behaviour) professional development Evaluation at MacEwan

Slide 7

C5065 Instructional Faculty Requirements based on appointment type ContinuingProbationary Sessional/Sessional- extended Teaching Dossier Peer Review Sessional-extended at renewal (Collective Agreement 2011-14) Annual Report Student Online Feedback Performance Review

Slide 8

C5065 Probationary Requirements Teaching Dossier - Annually Peer Review - Two reviews during each of the first two years (4 total) Annual Report - Annually Student Online Feedback - Every course during appointment Performance Review - Annually

Slide 9

Introduction What is Peer Review? Scholarly publications review process to assess quality of the work Evaluation of classroom/online teaching and teaching materials ASAC position review committees Any evaluation process where the evaluators are peers

Slide 10

Peer Review Literature Peer review is appropriate for Course content, expertise in field of study Course objectives and materials Appropriate teaching methodology Course organization Appropriate tests, assignments, evaluation methods Seldin, 1999

Slide 11

Peer Review at MacEwan Teaching effectiveness Classroom visits Teaching materials ASAC position review committees

Slide 12

Qualifications of Peer Reviewers Criteria developed by FEC Completion of an appropriate peer review training course and / or demonstration of peer review experience Minimum 3 years full-time teaching experience Pedagogical expertise Possess a continuing appointment Recommendation by Dean / Associate Dean / Director / senior administrator

Slide 13

Attributes of Peer Reviewers Utilizes effective instructional practices (demonstrated through dossiers, annual reports, awards, etc) Demonstrates collegiality Provides constructive, objective feedback focusing on teaching behaviours and improvement Models ethical behaviour Appreciates different styles / modes of teaching

Slide 14

Exceptions For all peer reviewers, should there be a deficit in meeting the recommended qualifications, the Dean, Associate Dean, Director, or senior administrator will assess and provide final approval.

Slide 15

Choosing Peer Reviewers Chairs will support and advise faculty members in the selection of two peer reviewers to conduct classroom observations and review of relevant teaching materials. It is not necessary that peer reviewers be content- area experts. Peer reviewers can expect to be asked to perform peer reviews outside of their home department(s).

Slide 16

Defining Effective Teaching Conversations about teaching (30min) Pair and Share Break into small groups and discuss the following sentences: I most enjoyed teaching when __________ I knew I had a problem when __________ Have a member of your group report on your discussion to the main group *Chickering and Gamson, 1987

Slide 17

Measuring Good Teaching Myths Dr. Fox Expressiveness High Ratings Reflect Lenient Grading Workload/course difficulty Rank and experience Gender https://facultycommons.macewan.ca/headlines/professors- from-sfu-share-mean-student-evaluations

Slide 18

Measuring Good Teaching

Slide 19

Measuring in context Take 15 min. and complete the following sentences in your group: I knew my teacher was a good teacher when/because I knew my teacher was a poor teacher when/because

Slide 20

Literature on Good Teaching Eight dimensions of college instruction : Course organization & planning Faculty communication Faculty/student interaction Assignments, exams, & grading Instructional methods & materials Course outcomes Student effort & involvement Course difficulty, workload, & pace (Marsh, 1984 & 1987)

Slide 21

Literature on Good Teaching (2) Encourages student interaction Encourages student cooperation Active learning Prompt feedback to students Emphasizes time on task High expectations Respect (Chickering and Gamson, 1987)

Slide 22

Literature on Good Teaching (3) Good teachers Are good learners; Display enthusiasm; Know how to modify their teaching strategies; Encourage learning for understanding; Demonstrate an ability to transform and extend knowledge, rather than just transmit it; Set clear goals; Provide high-quality feedback Show respect for their students (Ramsden, 2003)

Slide 23

Overall Good Teaching Parker Palmer capricious chemistry Good teaching is ultimately irreducible to a simple technical formula Characteristics are beyond common descriptors or attributes of a checklist Solution: multifaceted approach Student feedback, peer review, self-reflection, and administrative evaluation

Slide 24

Peer Review Activity In your small group, reflect on your past experiences with giving and receiving peer feedback. Address the questions: What did you learn from the experience? In what ways it was a positive/negative experience? Did it influence your instruction? How did receiving and giving feedback change the way you thought about teaching and learning?

Slide 25

Peer Observation The two forms of observation: Summative observation - This observation is an evaluation. It can be done for administrative or supervisory reports for the purpose of job retention, promotion, salary increases, etc. Formative observation - The purpose of a formative peer observation is to help instructors become better teachers and more knowledgeable professionals.

Slide 26

Peer Review Process at MacEwan Evaluation Policy outlines two main areas of teaching assessed by peer reviewers: 1.Teaching materials - lecture notes, course outlines, tests and exams, assignments, grading rubrics, etc.; and 2.Instructional delivery - observation of teaching in formal lectures, as well as other delivery modes such as labs, studio, clinical, and online/distance formats

Slide 27

Peer Review Process at MacEwan Five main elements: Pre-observation meeting Review of teaching materials Classroom observation Post-observation debrief Post-observation report

Slide 28

Forms Some of the forms available Narrative form Checklist form Rating form Data Collection (see appendices of workbook)

Slide 29

Pre-Observation Meeting Emotionally, it is important to address distrust of evaluation, violation of personal style and space, insecurities about time, and fear of bias (Chism, 2007) Essential that trust be established - Acknowledge the emotional aspect. Clear objectives of observation should be outlined - Communicate expectations Stressed that the process is collaborative - Engage in active listening skills (Review pre-observation forms in appendices)

Slide 30

Course Materials Some examples of materials include: Syllabus Course guides Course packets and assigned texts PowerPoint Presentations, overheads, BlackBoard, etc Handouts Videos and other multimedia supplements Project assignment directions and handouts

Slide 31

Procedure for Evaluation of Course Materials Importantly, we are not focusing on student achievement measures when reviewing teaching materials. Rather, we are looking at content, design, clarity, etc Materials enable reviewers to see an instructor's philosophy enacted, and can alert reviewers to the areas in which a given instructor excels Materials can promote discussion about the overall goals (Review course materials checklist in appendices)

Slide 32

Classroom Observation Observers should use the same observation form they should take separate minutes/notes At least two observers should attend each class observation session At least two visits is preferable No unannounced visits Discuss with the instructor the format of the review prior to visit: Where will you sit? Will you be introduced?

Slide 33

Classroom Observation During the Classroom Visit Be unobtrusive (basically, remain quiet) Do not ask questions Do not volunteer answers Do not correct the instructor Be attentive Note specific examples of observed behaviours (positive & negative)

Slide 34

Classroom Observation Challenges One class may be not representative Atypical performance (good or bad) Observer may not have a good understanding of the context Personal biases on what constitutes good teaching

Slide 35

Some Solutions Attend 2+ classes Video tape class? Focus-group students after class? Use the data collection form, cite examples Discuss visit with co-reviewer, debrief together Contextualize the course reflect back on: Pre-observation meeting Examination of course materials

Slide 36

Peer Observation Exercise Pre-observation In your small group, decide on the form you will be using for review (narrative, checklist, rating) Review form and highlight areas that each reviewer will be responsible for focusing on and taking notes about Observation Choose one of the two scenarios to review Reflect on the class from the following viewpoints: Viewpoint 1: students Viewpoint 2: instructor

Slide 37

Peer Observation Exercise Compare your list of observer behaviors with others in your group Be reflective and systematic through the criteria outlined in your observation form Embed your observations & conclusions in examples of observed behaviours Did the instructors behavior appear to be what you might expect in a formal observation?

Slide 38

Peer Observation Exercise Role Play In your small group, designate an individual to be the instructor, the reviewer, and a reporter Break into large groups based on role as instructors & reviewers & reporters Compare the notes your groups took during the observation and discuss strategies for approaching the debrief (and how the characters might react) Break back into your small group and role play the post-observation debrief meeting

Slide 39

Post Observation Debrief Meet with the instructors immediately following the classroom visit Ask the instructor to describe how they felt the class went in relation to the goals, objectives and expectations outlined in the pre-observation meeting At this time, the peer reviewers should bring up any immediate concerns that were highlighted by the observation session. **This is an important aspect of the review as it can potentially change the observers perceptions about what they saw and heard.

Slide 40

Post Observation Debrief Procedure Ask the person who taught how s/he felt the lesson went Reconfirm the observation focus points for the final report Offer positive comments on what you think went well Identify any urgent issues Ask questions for clarification, as needed End on a courteous note Provide an expected timeline for the final report and final meeting

Slide 41

Post-Observation Exercise Prepare yourself to care watch that you are not overly forgiving or overly critical (put yourself in their shoes) Use descriptive language rather than evaluative Be responsive to the goals outlined in the pre- observation meeting Check to make sure the instructor has heard your message as you intended Be forward-looking and positive in your discussion, i.e. what can be improved over what went wrong

Slide 42

Peer Review Final Report Policy C5065 states Section 4.4.4, Peer reviewers shall prepare brief written assessments, grounded in evidence, and submit a copy of reports to the faculty members and Chairs.

Slide 43

Who Sees the Final Report? Faculty member used to improve teaching effectiveness, make adjustments as necessary for application for continuing status Chair and Dean used for administrative review ASACs used for personnel decisions

Slide 44

Characteristics of a Good Report Reference the goals established during the pre- observation meeting Provide a written record of what was observed and what was discussed Consider the audience Use examples from the observation Language is clear, concise, avoids jargon and abbreviations Tone is supportive and phrasing is positive Action-oriented and provides specific and practical suggestions for improvement

Slide 45

Characteristics (contd) One, joint report per instructor - Structure of report should be agreed upon by all reviewers Return report in a timely manner Review with faculty member within 3 days of classroom visit Consider faculty member response before submitting for administrative review Confidential (Review report template in appendices)

Slide 46

Peer Review for Online Courses Online Peer Review Form (4 categories) Student behaviour Faculty-student interaction (multiple interactive communication strategies) Technology support Learning environment

Slide 47

Challenges - Peer Review for Online Courses A connection between specific materials and instruction and student action that manifest learning objectives. Technology and accessibility Are instructors responsible for these? Yes, no, maybe, more nowadays?

Slide 48

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction: Clear objectives as stated in: Syllabus: must be clear. Super important. Grading Rubrics: explains how they will be graded Contd.

Slide 49

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction Clear objectives as stated in: Models or Examples of quality work from successfully submitted and graded student work Explanations of the relevance of course materials to learning. WHY are we doing this? (think of students perspective). Contd.

Slide 50

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction Assessment tools should be used: Throughout the course to establish: Learning styles Skill/knowledge level Challenge preconceptions Clarify current learning Contd.

Slide 51

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction Opportunities for students to construct and experience their own knowledge: We all learn differently and its important to be clear and understanding about this. Learning processes and outcomes are not all the same. Contd.

Slide 52

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction Opportunities for students to become increasingly responsible for their own learning: Instructor should provide consistent and timely input. Encouragement, good examples, and instructor presence Try to develop intrinsic motivation. Contd.

Slide 53

What Is A Good Online Course? Content? Easy use of technology? Interaction? Learner-centered outcomes-based instruction Options for demonstrating learning outcomes (if appropriate) Opportunities for reflection Effective student support services Contd.

Slide 54

What Is A Good Online Course? Clear and captivating instructional design the writing style, the images and plug-ins selected, and the layout are appropriate to the intellectual sophistication, technical savvy and needs of the audience a pathway to guide student learning appealing presentation Contd.

Slide 55

What Is A Good Online Course? Varied forms of interactivity with material, other students and the instructor Regular Communication with peers and instructor Varied Interaction with Content Source: http://www.4faculty.org/includes/digdeeper/online/lc_outcome s_based_instruc.htm

Slide 56

Online Course Material for Review 1.Instructional material and resources 2.Assignments and student assessment (clear and detailed instructions and guidelines, purposes, formal writing standards/academic writing requirement) 3.Discussion forums (ability for students to give feedback and ask questions during the class is critical) 4.E-mail messages 5.Course syllabus (the syllabus is the most important document of the entire course. It is the road map or game plan for the course) 6.Chat space

Slide 57

Where to Go For Resources CAF website: Then click on Faculty Evaluation Then click on Peer Review Faculty Evaluation Coordinator