#pearsoncite
Measuring Mastery: Best Practices for Assessment in Competency-Based Education
#pearsoncite
Barbara E. Rowan, Ph.D.
Director of Academic Research, Efficacy, & Evaluation
Course Design, Development, and Academic Research, Higher Education Services
Pearson North America
#pearsoncite
Agenda
• Brief introduction to today’s student
• Explanation of Competency-based Education
• Definition of competencies
• Measuring mastery of competencies through assessment
@BarbaraRowan
#pearsoncite
Typical College Student
• Not what we think – no longer 18-22
• 38% are over 25 years of age
• 58% work while enrolled in college
• 26% are raising children
• 47% support themselves financiallySource: Lumina Foundation
@BarbaraRowan
#pearsoncite
Competency-based Education
• An answer to:– Stagnant graduation rates
– Increasing prices
– On-demand education
– Need for asynchronous education
@BarbaraRowan
#pearsoncite
What is Competency-based Education?
A change from knowledge acquisition to knowledge application
A change from a single subjective or objective measure to multiple objective and subjective measures
A change from proxy assessment to authentic assessment
A change from norm referenced evaluation to criterion-referenced evaluation
@BarbaraRowan
#pearsoncite
Set term length
Credit Hours
Grades
Traditional
No set term
Mastery of Competencies
Real-world Application
CBE
@BarbaraRowan
#pearsoncite
Department of Education
• March 2013
• Students could receive financial aid if participating in an approved CBE program– Credit hour equivalency
– Direct assessment
@BarbaraRowan
#pearsoncite
Credit Hour Equivalency
• Competencies are course-based, and mapped back to credit hours
• Credit-hour equivalence is used to award student financial aid
@BarbaraRowan
#pearsoncite
Direct Assessment
• Uses a direct measure of student learning
• Disregards credit hours
• DoE has tentatively granted approval of these CBE models
@BarbaraRowan
#pearsoncite
Common Elements of CBE Programs
Competency Framework
Competency Assessments
Demonstration of Competency
@BarbaraRowan
#pearsoncite
Carraccio,Wolfsthal, Englander, Ferentz, & Martin, 2002, p. 362
Competency
Knowledge
Skills
Attitudes
Competence
@BarbaraRowan
#pearsoncite
Competencies
Clearly Defined Measurable
Related to the Knowledge/Skills
Needed for Future Endeavors
@BarbaraRowan
#pearsoncite
Assessment
• Driving force behind a strong CBE program
• Two-step Process– Assessment development and score validation
– Determining cut scores
#pearsoncite
Assessment Development
StandardsPsychometric
EvaluationValidity
#pearsoncite
Assessment Development
Standards
#pearsoncite
Assessment Best Practices
• Each competency must be measured more than one time
• Each competency must be measured more than one way
• Assessments should focus on clusters of competencies as they might appear in real world
#pearsoncite
Assessment Types
Performance-based Assessment (real-world
focus)
Objective Measures of
Competencies
Diagnostic Tests of Knowledge
@BarbaraRowan
#pearsoncite
Assessments
• Assessment maps to competencies
• Assessment measures a student’s knowledge and skills AND how well the student can integrate, synthesize, and use the knowledge and skills necessary to become a community of practice
#pearsoncite
Assessments
The more “real world” the assessment is, the more value the results hold for determining a student’s mastery of the competencies.
#pearsoncite
Assessment Development
@BarbaraRowan
StandardsPsychometric
Evaluation
#pearsoncite
Psychometric Evaluation
• Reliability – indicates consistency of test scores
• Item Analysis – indicates items that are performing poorly
• Rubric Evaluation – to ensure that the rubric is properly evaluating subjective measure
@BarbaraRowan
#pearsoncite
Assessment Development
@BarbaraRowan
StandardsPsychometric
EvaluationValidity
#pearsoncite
Score Validation
Test Content
Relations to Other
Variables
Response Processes
Test Consequences
Internal Test Structure
@BarbaraRowan
#pearsoncite
Framework for Determining Mastery
• Must determine what separates masters from non-masters
• Validity evidence required to support decisions
@BarbaraRowan
#pearsoncite
Cut Scores – Questions to Ask
What do Students Need to Know and Be Able to Do at Each Level?
What Performance Levels will be
Reported?
Are Cut Scores Necessary?
@BarbaraRowan
#pearsoncite
Cut Score Options
@BarbaraRowan
#pearsoncite
Five Basic Steps
Select Judges
Define borderline knowledge and skills
Train the judges
Collect judgments
Choose a cut score
by combining judgments
@BarbaraRowan
#pearsoncite
Choosing a Method
• No one best method
• Choice should depend on what kind of judgments can be gotten and believed
• Based on types of questions in the test (multiple choice; constructed-response; rubric-based
@BarbaraRowan
#pearsoncite
NAEP Suggested Levels
Basic Level
• Partial Mastery of knowledge and skills
Proficient Level
• Solid Academic Performance and Demonstrated Competence over Challenging Subject Matter
Advanced Level
• Signifies Superior Performance on Challenging Subject Matter
@BarbaraRowan
#pearsoncite
Cut Score Process
@BarbaraRowan
Set Provisional Cut Scores
Establish Operational Cut Scores
Document the ProcessEvaluate the Results of
Using the Cut Scores
#pearsoncite
Provisional Cut Scores
• Determined by people who are aware of what students should know and be able to do
• Determine how many score points students have to earn to demonstrate they have reached the level of knowledge and skill for each level
@BarbaraRowan
#pearsoncite
Provisional Cut Scores
• Choose Cut Score Method
Step 1
• Select Judges, that is, 5-15 subject matter experts
Step 2• Train Judges in
the Cut Score Method Chosen
Step 3
@BarbaraRowan
#pearsoncite
Operational Cut Scores
@BarbaraRowan
Is it better to pass a student who should fail?
Or fail a student who should pass?
#pearsoncite
Operational Cut Scores
Test Score Fluctuations
Pass
Fail
@BarbaraRowan
#pearsoncite
Document the Process
@BarbaraRowanRationale
Qualifications
Training
Steps
Data shared with Judges
Judges’ Ratings
Evaluation Forms from
Judges
Rationale for Adjustments
#pearsoncite
Evaluating the Results of Using Cut Scores
• Occurs after at least one round of testing and reporting of results has occurred
• Requires the validation of cut scores
• Ongoing process
@BarbaraRowan
#pearsoncite
Validity Study
• Are the results reasonable?
• Are failure rates far higher or lower than expected?
• Are outstanding students failing to reach the highest level?
• Are poorly prepared students reaching the highest level?
@BarbaraRowan
#pearsoncite
Questions?
@BarbaraRowan