evaluation: a challenging component of teaching darshana shah, phd. pies 4-19-2011
TRANSCRIPT
Evaluation: A Challenging Component of Teaching
Darshana Shah, PhD.PIES
4-19-2011
Next 45 minutes……….
• Learning Objectives– Briefly review the characteristics of evaluation– Discuss types of evaluation instruments– Understand the framework for selecting and
developing assessment methods
What is Evaluations?
Evaluation??????Who Evaluates Whom?
Different Settings
Purpose of EvaluationWell integration of Teaching, Learning & Assessment
Evaluation/ Assessments
Assessments = Observations /measurements Individual appraisal
Evaluation = interpretations / conclusions Appraisal of all activities, curriculum, process outcomes, student
learning and teacher effectiveness
Types of Evaluations
Formative evaluation- provide feedback so the learner can modify their learning approach
Summative evaluation- done to decide if a student has met the minimum course requirements (pass or fail)-
Purpose of Evaluation
To certify individual competenceTo assure successful completion of
goals/objectivesTo provide feedback
To students To faculty, course and clerkship directors
As a statement of values (what is most critical to learn)
For Program Evaluation- evaluation of an aggregate, not an individual (ex. average ability of students to perform a focused history and physical)
Consequences of evaluation
Steering effect- exams “drive the learning”- students study/learn for the exam
Impetus for change (feedback from students, Executive Curriculum, LCME)
Characteristics of Evaluation
• Goal oriented• Value oriented• Comprehensive• Diagnostic valid and reliable
Blooms TaxonomyCognitive: mental skills (Knowledge)Affective: growth in feelings (Attitude)
Psychomotor: manual or physical skills (Skills)
FORMATIVE SUMMATIVE
Written Observed
MCQ
MEQ
Application
Essay
Practical
Note review
SP
OSCE
Oral
Longitudinal Clinical Observation
KNOWS
KNOWS HOW
SHOWS HOW
DOES
Erica Friedman’s presentation
Take 5 mins…….
Recall; the goals and objectives for your course or clerkship in the context of assessment
Identify the best methods of assessing your goals and objectives
Purpose of EvaluationWell integration of Teaching, Learning & Assessment
How to Match Assessment to Goals and Teaching Methods
Define the type of learning (lecture, small group, computer module/self study, etc)
Define the domain to be assessed (knowledge, skill, behavior) and the level of performance expected (knows, knows how, shows how or does)
Determine the type of feedback required
Classification Instructional Objectives
Learning Activities/ Teaching Method
Assessment Methods
knowledgeRecalling specific information
Recognize List Describe Identify Retrieve NameLocate/Find
Graphic Organizers Mnemonics Crosswords Puzzles Jeopardy or other games Note Taking Flash Cards
Simple Multiple Choice True/False Fill in the blank Label Diagram
KNOWS
Explaining ideas or concepts
Interpret Summarize Infer Paraphrase Classify Explain Attribute
Concept mapping Short Case Cooperative Task Write, Pair, Share Role Play Matching Games
Short Case Presentation Short Answer Concept Mapping Short Presentation Poster Session MCQ with vignettes
Classification Instructional Objectives
Learning Activities/ Teaching Method
Assessment Methods
KNOWs HOW
Applying Implement Carry out Use Execute Operate TakePerform
Simulation Lab
Discuss Rounds Cases Standardized Patients/ Virtual Patients
Chart Review Teaching Field Trips
Demonstration Case Study SOAP 360 Assessment Patient/family Assessment OSCE MCQ with vignettes
Analyzing Breaking information into parts to explore understandings and relationships
Analyse Compare Contrast Organize Deconstruct Outline Structure Integrate
Collaborative Tasks PBL Spreadsheets Research Project
5 Minute Preceptor Writing a Wiki Reflection Problem Solving Open Ended Questions Complex Case Study Concept mapping
Survey Report Paper Project Report Group Presentation Grand Rounds Create a Mind Map Reflective Writing Problem Identification Extended Matching MCQ
How we currently assess ourselvesStudent evaluations (quantitative and qualitative)-
most often summativePerformance of students on our exam and specific
sections of USMLE
Things to consider!!!!!
Focus and feedback groups (formative)Self evaluations- yearly grid completed by
course directors and core facultyConsider peer evaluation of teaching and
teaching materials
How to be successful
Students should be clear about the course/clerkship goals and the specifics about the types of assessments used and the criteria for passing
Make sure the choice of assessments is consistent with the values of your course and the school
Final judgments about students’ progress should be based on multiple assessments using a variety of methods over a period of time (instead of one time point)
Critical factors for choosing an evaluation tool
Type of evaluation and feedback desired: formative/summative
Focus of evaluation:Knowledge, skills, behaviors (attitudes)
Level of evaluation:Know, Knows how, Shows how, Does
Pros/Cons:Validity, Reliability, Cost (time, $ resources)
A Brief Overview of TechniquesFor Classroom Setting
• Assessing Course related knowledge and skills:– Background Knowledge Probe– Minute Paper– Empty outline– Memory Matrix– Focus listing