the competency assessment toolkit for professional psychology catherine l. grus, ph.d., associate...
TRANSCRIPT
The Competency The Competency Assessment Toolkit for Assessment Toolkit for
Professional PsychologyProfessional Psychology
Catherine L. Grus, Ph.D.,Catherine L. Grus, Ph.D.,Associate Executive Director forAssociate Executive Director for
Professional Education and TrainingProfessional Education and TrainingEducation Directorate Education Directorate
American Psychological AssociationAmerican Psychological Association
THANK YOUTHANK YOU
Many thanks to Many thanks to
• Nadya Fouad, University of Wisconsin Nadya Fouad, University of Wisconsin
• Nadine Kaslow, Emory University Nadine Kaslow, Emory University
• Benchmarks WorkgroupsBenchmarks Workgroups
• Assessment “Toolkit” WorkgroupAssessment “Toolkit” Workgroup
OverviewOverview
• Considerations in the assessment of Considerations in the assessment of competencecompetence
• Development of the ToolkitDevelopment of the Toolkit
• Putting the toolkit into practicePutting the toolkit into practice
Competence …Competence …
• “ … “ … the habitual and judicious use of the habitual and judicious use of communication, knowledge, communication, knowledge, technical skills, clinical reasoning, technical skills, clinical reasoning, emotions, values, and reflection in emotions, values, and reflection in daily practice for the benefit of the daily practice for the benefit of the individual and community being individual and community being served.”served.”
From Epstein and Hundert (2002)From Epstein and Hundert (2002)
Competence …Competence …
• … … is developmental depending on is developmental depending on stage of professional functioningstage of professional functioning
• … … context dependentcontext dependent
• … … executed according to ethical executed according to ethical principles, guidelines, standards, and principles, guidelines, standards, and values of the profession andvalues of the profession and
• … … requires public verificationrequires public verification
Competencies …Competencies …
– ObservableObservable– MeasurableMeasurable– ContainableContainable– PracticalPractical– Derived by expertsDerived by experts– And flexibleAnd flexible
are elements of competence that are are elements of competence that are ….….
Assessment of Competence Assessment of Competence - Considerations - Considerations
• Use of both “formative” and Use of both “formative” and “summative” evaluation methods“summative” evaluation methods
• Authentic evaluationsAuthentic evaluations• Real world tasks, meaningful Real world tasks, meaningful
activitiesactivities
• [High] Fidelity: Degree to which[High] Fidelity: Degree to which the assessment representsthe assessment represents
actual performanceactual performance
• Employ a range of perspectives Employ a range of perspectives (multi-informant) and methods(multi-informant) and methods
• Challenge of evaluating attitudes and Challenge of evaluating attitudes and valuesvalues
• Psychometric issuesPsychometric issues
• Feasibility/costs Feasibility/costs
• Lifelong self-assessmentLifelong self-assessment
What methods are What methods are commonly used to assess commonly used to assess students/ trainees?students/ trainees?
• In class examinations and course In class examinations and course “grades”“grades”
• Faculty/supervisor completes rating Faculty/supervisor completes rating scalescale
• Letters of recommendationLetters of recommendation
• EPPP, state jurisprudence examsEPPP, state jurisprudence exams
Developing Developing Faculty/Supervisor Faculty/Supervisor CompetenciesCompetencies• Training in “difficult conversations”Training in “difficult conversations”
– How to “critique constructively”How to “critique constructively”• Develop awareness of potential dual roles Develop awareness of potential dual roles
in evaluation and how that could influence in evaluation and how that could influence resultsresults– Benefits of having other faculty evaluate Benefits of having other faculty evaluate
students’ performancestudents’ performance– Explicit obligations to make evaluations Explicit obligations to make evaluations
accurateaccurate• Grade inflationGrade inflation• Letters of recommendation to “market” vs. evaluateLetters of recommendation to “market” vs. evaluate
• Knowledge of a range of assessment tools, Knowledge of a range of assessment tools, access to resources “toolkits”access to resources “toolkits”
Competency Assessment Competency Assessment for “Toolkit” for for “Toolkit” for Professional PsychologyProfessional Psychology
– Charge from APA Board of Educational Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional Affairs: Develop a “Toolkit” for professional psychologypsychology
– Purpose: Promote broader implementation of Purpose: Promote broader implementation of competence assessment and provide competence assessment and provide information about application of assessment information about application of assessment methods to the assessment of competencemethods to the assessment of competence
– Coordinated with Benchmarks Work GroupCoordinated with Benchmarks Work Group
Benchmarks Core Benchmarks Core CompetenciesCompetenciesFoundational Foundational
Competencies:Competencies:• ProfessionalismProfessionalism• Reflective practiceReflective practice• Scientific knowledge Scientific knowledge
and methodsand methods• RelationshipsRelationships• Individual and cultural Individual and cultural
diversitydiversity• Ethical and legal Ethical and legal
standards and policystandards and policy• Interdisciplinary Interdisciplinary
systems systems
Functional Functional Competencies:Competencies:
• AssessmentAssessment• InterventionIntervention• ConsultationConsultation• Research and Research and
evaluationevaluation• SupervisionSupervision• TeachingTeaching• AdministrationAdministration• AdvocacyAdvocacy
How are the Benchmarks How are the Benchmarks Organized?Organized?
• Core Foundational and Functional Core Foundational and Functional CompetenciesCompetencies
• Essential ComponentEssential Component: what are the : what are the critical elements of/what critical elements of/what knowledge/skills/attitudes that make knowledge/skills/attitudes that make up this competency? up this competency?
• Behavioral AnchorBehavioral Anchor: what would it : what would it look like if you saw it (essential look like if you saw it (essential component)?component)?
Components of ToolkitComponents of Toolkit
• Background and IntroductionBackground and Introduction• Assessment Method Fact SheetsAssessment Method Fact Sheets• Grid of Assessment Methods and Grid of Assessment Methods and
Competencies/Essential Components Competencies/Essential Components Best used forBest used for
Fact Sheet: Portfolio ReviewDescriptionA portfolio is a collection of products, gathered by the person being assessed …Application
Competency Applies to1 Predominant Use of2
Developmental Level3
Professionalism E F,S 2,3,4
Reflective Practice E F,S 1,2,3,4Implementation Deciding on form (e.g., web-based or hard copy)Psychometric PropertiesReliability has not been well established due to the variable content included in a portfolioStrengthsProvides a low cost assessment strategyChallengesRequires intense commitment of time and labor intensive for the assessors and the person being assessedFuture directionsCreate and implement training modules for assessors and people being assessed in the portfolio assessment strategy
Annual review
Case review
s
Process/outcome data
Rating F
orms
Consum
er surveys
OS
CE
Portfolio reviews
Ratings Perform
ance
Record review
s
Self-assessment
Simulations/role plays
Stand. patient interview
Stand. oral exams
Written exam
s
360 Evaluations
Intervention Knowledge interventions
2 3 3 2 1 2 1 1
Intervention planning
2 1 2 1 2 2 3 1 2 2 1
Skills 2 2 1 3 2 1 2 1 2 1 1 2 Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1 Progress
evaluation3 1 2 2 2 3 2 2 1 2 2
Grid of Possible Assessment Grid of Possible Assessment Methods (Toolkit)Methods (Toolkit)
Assessment Methods in Assessment Methods in ToolkitToolkit• 360 evaluation360 evaluation• PortfolioPortfolio• OSCEOSCE• Structured Written & Structured Written &
Oral ExamsOral Exams• Case PresentationCase Presentation• Simulation/Role PlaySimulation/Role Play• Competence Competence
Evaluation Rating Evaluation Rating FormForm
• Self-AssessmentSelf-Assessment
• Ratings of live or Ratings of live or recorded performancerecorded performance
• Standardized Client Standardized Client InterviewInterview
• Client/Patient Client/Patient Process/Outcome Process/Outcome MeasureMeasure
• Consumer Satisfaction Consumer Satisfaction SurveySurvey
• End of Rotation End of Rotation Performance ReviewPerformance Review
Putting it all Together …Putting it all Together …
Benchmarks
Toolkit
Using the Toolkit and Using the Toolkit and Benchmarks to Guide the Benchmarks to Guide the
Assessment of Student Assessment of Student Learning OutcomesLearning Outcomes
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Select assessmen
t method(s)
Expected level of competence
attainment and training plan
developed with trainee
Formative assessment
Student demonstrates problems with competencies
acquisition
Student demonstrates emerging
competencies
Training plan continues
Remediation plan developed
Summative evaluation to assess readiness for next level of
training
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Program Program OutcomeOutcome: Students will demonstrate the : Students will demonstrate the following intervention (functional) competencies following intervention (functional) competencies prior to going on internship.prior to going on internship.
1.1. Knowledge of scientific, theoretical, empirical Knowledge of scientific, theoretical, empirical and contextual bases of intervention, and contextual bases of intervention, including theory, research, and practiceincluding theory, research, and practice
2.2. Formulates and conceptualizes cases and Formulates and conceptualizes cases and plan interventions utilizing at least one plan interventions utilizing at least one consistent theoretical orientationconsistent theoretical orientation
3.3. Clinical skills Clinical skills 4.4. Implements evidence-based interventions Implements evidence-based interventions
that take into account empirical support, that take into account empirical support, clinical judgment, and client diversity (e.g., clinical judgment, and client diversity (e.g., client characteristics, values, and context)client characteristics, values, and context)
5.5. Evaluate treatment progress and modify Evaluate treatment progress and modify treatment planning as indicated, utilizing treatment planning as indicated, utilizing established outcome measuresestablished outcome measures
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Select assessmen
t method(s)
Annual review
Case review
s
Process/outcom
e data
Rating F
orms
Consum
er surveys
OS
CE
Portfolio review
s
Ratings P
erformance
Record review
s
Self-assessm
ent
Sim
ulations/role plays
Stand. patient interview
Stand. oral exam
s
Written exam
s
360 Evaluations
Intervention Knowledge interventions
2 3 3 2 1 2 1 1
Intervention planning
2 1 2 1 2 2 3 1 2 2 1
Skills 2 2 1 3 2 1 2 1 2 1 1 2
Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1
Progress evaluation
3 1 2 2 2 3 2 2 1 2 2
Grid of Possible Assessment Grid of Possible Assessment Methods (Toolkit)Methods (Toolkit)
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Select assessmen
t method(s)
Expected level of competence
attainment and training plan
developed with trainee
Developmental LevelA. Knowledge of InterventionsReadiness for Practicum Readiness for Internship
Essential Component: Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence-based practice and its role in scientific psychology
Behavioral anchor: Articulates the relationship of EBP to the science of psychologyIdentifies basic strengths and weaknesses of intervention approaches for different problems and populations
Essential Component: Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice
Behavioral Anchor:Demonstrates knowledge of interventions and explanations for their use based on EBPDemonstrates the ability to select interventions for different problems and populations related to the practice settingInvestigates existing literature related to problems and client issuesWrites a statement of one’s own theoretical perspective regarding intervention strategies
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Select assessmen
t method(s)
Expected level of competence
attainment and training plan
developed with trainee
Formative assessment
Student demonstrates problems with competencies
acquisition
Student demonstrates emerging
competencies
Training plan continues
Remediation plan developed
Remediation Plan
Competency Domain/Essential Components
ProblemBehaviors
Expectations for Acceptable Performance
Trainee’s Responsibilities/Actions
Supervisors’/Faculty Responsibilities/Actions
Timeframe forAcceptablePerformance
AssessmentMethods
Dates ofEvaluation
Consequencesfor Unsuccessful Remediation
Identify expected student learning outcomes using
competency model (e.g., Benchmarks)
Select assessmen
t method(s)
Expected level of competence
attainment and training plan
developed with trainee
Formative assessment
Student demonstrates problems with competencies
acquisition
Student demonstrates emerging
competencies
Training plan continues
Remediation plan developed
Summative evaluation to assess readiness for next level of
training
For more information:For more information:
http://www.apa.org/ed/graduate/competency.htmlhttp://www.apa.org/ed/graduate/competency.html
Coming soon:Coming soon:
Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Competency Assessment Toolkit for Professional Psychology. Training and Education in Psychology. Training and Education in Professional Psychology.Professional Psychology.