assessment of trainee competencies: preliminary psychometric study
Post on 20-Jan-2016
26 Views
Preview:
DESCRIPTION
TRANSCRIPT
Assessment of Trainee Competencies: Preliminary
Psychometric StudyDavid J. Martin, Ph.D.
Harbor-UCLA Medical CenterGeffen School of Medicine at UCLA
Models of Competency
• Cube Model of Competency Development (Rodolfa, Bent, Eisman, Nelson, Rehm & Ritchie, 2005)– Functional Competency– Foundational Competency– Stages of Professional Development
Models of Competency
• Cube Model (Rodolfa et al., 2005):– Functional Competency
• Assessment-Diagnosis-Case Conceptualization
• Intervention• Consultation• Research Evaluation• Supervision-Teaching• Management-Administration
Models of Competency
• Cube Model (Rodolfa et al., 2005):– Foundational Competency
• Reflective Practice-Self Assessment• Scientific Knowledge-Methods• Relationships• Ethical-Legal Standards—policy• Individual-Cultural Diversity• Interdisciplinary Systems
Models of Competency
• Cube Model (Rodolfa et al):– Stages of Professional Development
• Doctoral Education• Doctoral Internship/Residency• Post-doctoral Supervision• Residency/Fellowship• Continuing Competency
Models of Competency
Models of Competency
• Practicum Competencies Outline – Association of Directors of Psychology
Training Clinics (ADPTC) Practicum Competencies Workgroup (February 27, 2004)
– Council of Chairs of Training Councils (CCTC) Practicum Competencies Workgroup (March 25, 2004)
– Final CCTC comments, November, 2004– Summarized by Hatcher & Lassiter (2006)
Models of Competency
• Practicum Competencies Outline– Levels of Competence
•Novice• Intermediate•Advanced•Proficient•Expert
Models of Competency
• Practicum Competencies Outline– Areas of Competence
• Relationship/Interpersonal Skills– With Patients/Clients/Families– With Colleagues– With Supervisors– With Support Staff– With Teams at Clinic– With Community Professionals– For the Practicum Site Itself
Models of Competency
• Practicum Competencies Outline– Areas of Competence
• Skills in Application of Research• Intervention Skills• Psychological Assessment Skills• Consultation Skills/Interdisciplinary• Collaborations• Diversity (Individual & Cultural
Differences)
Models of Competency
• Practicum Competencies Outline– Areas of Competence
• Ethics• Professional Development
– Practical Skills to Maintain Effective Clinical Practice
– Professional Development Competencies
• Metaknowledge/Metacompetencies• Development of Leadership Skills• Supervisory Skills
Harbor-UCLA Assessment
• Psychological assessment skills
Extern Fellow
Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individuals, couples, families, and groups
I P
Ability to utilize systematic approaches to gathering data to inform clinical decision making
I P
Knowledge of psychometric issues and bases of assessment methods
A P
Knowledge of issues related to integration of different data sources
A P
Harbor-UCLA Assessment
Competency Level
Unsatisfactory Needs Improvement
Satisfactory Strong Outstanding N/A
1. Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individual, couples, families and groups.
P 1 2 3 4 5 6
2. Ability to utilize systematic approaches to gathering data to inform clinical decision making.
P 1 2 3 4 5 6
3. Knowledge of psychometric issues and bases of assessment methods.
P 1 2 3 4 5 6
4. Knowledge of issues related to integration of different data sources.
P 1 2 3 4 5 6
Warning !
Attempts at complete review of the following slides may result in
serious information overload!
Internal ConsistencyFellows Externs
Items N α N α
Relationship/Interpersonal Skills
With Patients/Clients/Families 4 90 .938 203 .918
With Colleagues 3 85 .923 186 .902
With Supervisors 6 92 .958 206 .948
With Support Staff N/A N/A N/A N/A N/A
With Teams at Clinic 2 77 .907 184 .900
With Community Professionals N/A N/A N/A N/A N/A
For the Practicum Site Itself 3 74 .935 178 .928
Internal ConsistencyFellows Externs
Items N α N α
Skills in Application of Research 2 87 .910
-- --
Intervention Skills 8 70 .954
98 .959
Psychological Assessment Skills 7 55 .960
92 .950
Consultation Skills/Interdisciplinary Collaborations
7 60 .961
110 .933
Diversity (Individual and Cultural Differences)
3 91 .955
200 .939
Ethics 6 86 .972
188 .975
Professional Development
Practical Skills to Maintain Effective Clinical Practice
6 75 .938
161 .943
Professional Development Competencies 11 73 .964
136 .964
Metaknowledge/Metacompetencies 5 91 .914
190 .939
Development of Leadership Skills 8 24 .966
-- --
Supervisory Skills 5 56 .901
-- --
Descriptive Statistics
FellowsN Min Max Mean S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 90 3.00 5.00 4.55 .609
With Colleagues 85 3.00 5.00 4.55 .619
With Supervisors 92 3.00 5.00 4.55 .622
With Support Staff 86 3.00 5.00 4.67 .583
With Teams at Clinic 77 3.00 5.00 4.55 .650
With Community Professionals 76 3.00 5.00 4.47 .739
For the Practicum Site Itself 74 3.00 5.00 4.53 .682
Descriptive StatisticsFellows
N Min Max Mean S.D.
Skills in Application of Research 87 3.00
5.00 4.30 .688
Intervention Skills 70 3.00
5.00 4.47 .612
Psychological Assessment Skills 55 3.00
5.00 4.32 .669
Consultation Skills/Interdisciplinary Collaborations
60 3.00
5.00 4.42 .654
Diversity (Individual and Cultural Differences)
91 3.00
5.00 4.49 .639
Ethics 86 3.00
5.00 4.52 .640
Professional Development
Practical Skills to Maintain Effective Clinical Practice
75 2.17
5.00 4.38 .700
Professional Development Competencies
73 3.00
5.00 4.50 .590
Metaknowledge/Metacompetencies 91 3.00
5.00 4.48 .580
Development of Leadership Skills 24 3.00
5.00 3.96 .637
Supervisory Skills 56 3.00
5.00 4.16 .570
Descriptive Statistics
ExternsN Min Max Mean S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 203 2.75 5.00 4.20 .656
With Colleagues 186 3.00 5.00 4.28 .639
With Supervisors 206 1.50 5.00 4.23 .711
With Support Staff -- -- -- -- --
With Teams at Clinic 184 1.50 5.00 4.17 .761
With Community Professionals 153 2.00 5.00 4. 14 .776
For the Practicum Site Itself 178 2.67 5.00 4.14 .732
Descriptive StatisticsExterns
N Min Max Mean S.D.
Skills in Application of Research -- -- -- -- --
Intervention Skills 98 2.13
5.00 3.82 .731
Psychological Assessment Skills 92 3.00
5.00 4.21 .760
Consultation Skills/Interdisciplinary Collaborations
110 3.00
5.00 4.21 .597
Diversity (Individual and Cultural Differences)
200 2.00
5.00 4.13 .740
Ethics 188 1.50
5.00 4.25 .739
Professional Development
Practical Skills to Maintain Effective Clinical Practice
161 2.17
5.00 4.02 .748
Professional Development Competencies
136 2.18
5.00 4.24 .675
Metaknowledge/Metacompetencies 190 2.00
5.00 4.15 .682
Development of Leadership Skills -- -- -- -- --
Supervisory Skills -- -- -- -- --
Inter-Scale Correlations
• Fellows’ Inter-Scale Correlations Ranged From .39 (Relationship Skills with Support Staff – Leadership Skills) - .93 (Assessment Skills – Consultation Skills)
• Externs’ Inter-Scale Correlations Ranged From .63 (Relationship Skills with Team – Ethics) - .95 (Intervention Skills – Consultation Skills)
Pre – Post Comparisons
FellowsEvaluation Period t p
1 2
N M S.D. N M S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 42 4.37 .723 48 4.71 .438 2.77 .007
With Colleagues 40 4.33 .693 45 4.74 .471 3.26 .002
With Supervisors 44 4.39 .698 48 4.69 .511 2.34 .02
With Support Staff 42 4.52 .707 44 4.82 .390 2.41 .018
With Teams at Clinic 38 4.41 .761 39 4.68 .493 1.86 .066
With Community Professionals 35 4.14 .879 41 4.76 .435 3.94 .000
For the Practicum Site Itself 38 4.32 .803 36 4.75 .439 2.80 .006
Pre – Post ComparisonsFellows
Evaluation Period t p
1 2
N M S.D. N M S.D.
Skills in Application of Research
41 4.13 .775 46 4.46 .566 2.23 .028
Intervention Skills 32 4.20 .687 38 4.70 .431 3.72 .000
Psychological Assessment Skills
25 .410 .803 30 4.50 .474 2.31 .025
Consultation Skills/Interdisciplinary Collaborations
32 4.18 .798 34 4.64 .411 2.93 .005
Diversity (Individual and Cultural Differences)
44 4.32 .753 47 4.65 .463 2.51 .014
Ethics 40 4.33 .727 46 4.68 .479 2.68 .009
Professional Development
Practical Skills to Maintain Effective Clinical Practice
32 4.08 .754 43 4.59 .574 3.33 .001
Professional Development Competencies
30 4.23 .691 43 4.69 .421 3.55 .001
Metaknowledge/Metacompetencies
46 4.33 .656 45 4.62 .476 2.37 .020
Development of Leadership Skills
13 3.65 .545 11 4.32 .560 2.94 .008
Supervisory Skills 28 3.92 .602 28 4.41 .420 3.50 .001
Pre – Post Comparisons
ExternsEvaluation Period t p
1 2
N M S.D. N M S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 99 4.01 .674 104
4.38 .587 4.20 .000
With Colleagues 90 4.13 .591 96 4.41 .656 3.05 .003
With Supervisors 101 4.07 .710 105
4.38 .683 3.14 .002
With Support Staff 88 4.18 .736 93 4.51 .653 3.13 .002
With Teams at Clinic 89 3.99 .748 95 4.34 .738 3.17 .002
With Community Professionals 66 4.00 .744 87 4.26 .784 2.11 .036
For the Practicum Site Itself 89 3.97 .757 89 4.31 .665 3.26 .001
Pre – Post ComparisonsExterns
Evaluation Period t p
1 2
N M S.D. N M S.D.
Skills in Application of Research
-- -- -- -- -- -- -- --
Intervention Skills 49 3.57 .665 49 4.06 .715 3.56 .001
Psychological Assessment Skills
44 4.56 .811 48 4.92 .674 2.27 .026
Consultation Skills/Interdisciplinary Collaborations
51 4.08 .645 59 4.32 .532 2.15 .034
Diversity (Individual and Cultural Differences)
97 3.98 .771 103 4.28 .683 2.87 .004
Ethics 91 4.08 .763 97 4.42 .680 3.20 .002
Professional Development
Practical Skills to Maintain Effective Clinical Practice
77 3.86 .721 84 4.17 .743 2.74 .007
Professional Development Competencies
68 4.11 .676 68 4.376
.651 2.34 .021
Metaknowledge/Metacompetencies
91 4.01 .651 99 4.271
.651 2.64 .009
Development of Leadership Skills
-- -- -- -- -- -- -- --
Supervisory Skills -- -- -- -- -- -- -- --
Discussion
• Preliminary data suggest that competencies can be measured reliably
• Inter-scale correlations raise important questions concerning competency constructs
Discussion
• Future Directions– Need for additional data to inform
competency evaluation process– More controlled study of competency
scales (e.g., mid-term and final ratings)– Factor-analytic approaches may address
questions of inter-scale shared variance– Need for validation strategies
top related