objective observations: creating rubrics for performance
TRANSCRIPT
© 2010 National Board of Osteopathic Medical Examiners www.nbome.org
OBJECTIVE OBSERVATIONS:CREATING RUBRICS FOR PERFORMANCE ASSESSMENTLaurie Gallagher D.O. Chief Physician Trainer
Jeanne Sandella D.O. Vice President for Clinical Skills Testing
AACOM/AODME
April 24, 2015
© 2015 NBOME
www.nbome.org
Rubric “Workshop”
• Background on Using Rubrics 10 mins
• Rubric Development 10 mins
• Apply Ratings 5 mins
• Questions 5 mins
Objectives
© 2015 NBOME
www.nbome.org
Challenge in Education
• Provide an objective measurement of learner behavior for various purposes
• Formative vs Summative
• Checklists vs Rubrics
© 2015 NBOME
www.nbome.org
What is a Rubric ?
• Scoring tool: Detailed scoring guide
• Coherent set of criteria to assess performance
• Includes descriptions of quality of performance
© 2015 NBOME
www.nbome.org
Why Use A Rubric ?
• Scoring is objective
• Match level of performance, not judge it
• Based on pre-set expectations and descriptions
• Reproducible outcomes
• Reusable
• Time efficient
© 2010 National Board of Osteopathic Medical Examiners www.nbome.org
Excellent Good SatisfactoryNeeds
Improvement
Componentsof the Report
Question/Purpose
Spelling, Punctuation,
Grammar
.
PERFORMANCE RATING
PERFORMANCE DESCRIPTIONS
CR
ITER
IA
© 2015 NBOME
www.nbome.org
Commonly used Rubrics
• Analytic: each criterion is evaluated independently
• Holistic: all criteria are evaluated simultaneously
© 2015 NBOME
www.nbome.org
Advantages
Analytic:• Good for formative
assessment
• Can link to instruction
• Gives feedback to student and instructor/rater
• Two columns of bulleted text
Holistic:• Good for summative
assessment
• Scoring is fast
• Achieve inter-rater reliability in a shorter time frame
• Two columns of bulleted text
© 2015 NBOME
www.nbome.org
Beginning1
Developing2
Accomplished3
Exemplary4
Score
Criteria1
Description reflectingBeginning level of performance
Description reflectingmovement towards masterylevel of performance
Description reflectingachievement of masterylevel of performance
Descriptionreflectinghighest level of performance
Criteria2
DescriptionreflectingBeginning level of performance
Description reflectingmovement towards masterylevel of performance
Description reflectingachievement of masterylevel of performance
Descriptionreflectinghighest level of performance
Criteria3
DescriptionreflectingBeginning level of performance
Description reflectingmovement towards masterylevel of performance
Description reflectingachievement of masterylevel of performance
Descriptionreflectinghighest level of performance
Criteria4
DescriptionreflectingBeginning level of performance
Description reflectingmovement towards masterylevel of performance
Description reflectingachievement of masterylevel of performance
Descriptionreflectinghighest level of performance
Analytic Scoring Rubric
© 2015 NBOME
www.nbome.org
Holistic Rubric
Score Description
5 Demonstrates complete understanding of the problem. Flawless grammar
4 Demonstrates considerable understanding of the problem.An occasional grammar error.
3 Demonstrates partial understanding of the problem. Few grammatical errors.
2 Demonstrates little understanding of the problem. Some grammatical errors
1 Demonstrates no understanding of the problem. Numerous grammatical errors
0 No Response/task not attempted
© 2015 NBOME
www.nbome.org
Challenges to Rubric Creation
• Concrete criterion examples
• Creating clear outcome measures
• The rubric is only as good as the preset definitions or descriptors
© 2015 NBOME
www.nbome.org
Benefits of a Well Constructed Rubric
Creates measurable outcomes for performance assessment
Creates reproducible outcomes for performance assessment
© 2015 NBOME
www.nbome.org
Rubric Websites and References
• http://Rubistar.4teachers.org
• www.teachnology.com• References:• Hall Wikfors E, Salmon, S, Chocolate chip cookies and rubrics, helping students understand
rubrics in inclusive settings. Teaching Exceptional Children. 2003; 4:8-11.
• Jamieson S. Likert Scales: How to (ab)use them. Med Educ. 2004; 38: 1217-8.
• McLeod, S.A. Likert Scale. Retrieved from http://www.simplypsychology.org/likert-
• Morton J. Performance-based assessment in undergraduate medical education. The Clinical Teacher 2007;4: 36-41.
•
• Whelan GP, Boulet JR, McKinley DW. Scoring standardized patient exams: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Med Teach. 2005;(3):200-6.
© 2015 NBOME
www.nbome.org
Crayon Rubric
Descriptors Descriptors Descriptors
Criterion 1
Criterion 2
Criterion 3
Criterion 4
© 2015 NBOME
www.nbome.org
Crayon Rubric
Unacceptable Satisfactory Superior
Color Quality
Line Sharpness
Fill Pattern
Texture
© 2015 NBOME
www.nbome.org
Crayon Rubric
Unacceptable Satisfactory Superior
Color Quality • Dull or muted• Not consistent
• Bright/intense• Consistent
Line Sharpness • Point dulls quickly• Thick• Edges blurred
• Point stays sharp• Thin• Crisp edge
Fill Pattern • Skip patterns/irregular• Some areas thicker • Difficult to apply
• No white gaps on paper• Even fill thickness• Easy to apply
Texture • Sticky/tacky• Doesn’t adhere to paper
• Silky smooth• Stays on paper
© 2015 NBOME
www.nbome.org
Pitfalls
• Complicated/ambiguous performance descriptors
• Irrelevant criteria
• Bias
• Drift
• Inadequate training
© 2015 NBOME
www.nbome.org
THANK YOU