SYSTEMATIC DEVELOPMENT OF A CLINICAL PERFORMANCE INSTRUMENT FOR AUD AND SLP STUDENTS/GRADUATES
Elaine Mormer, PhD, University of Pittsburgh Deborah Moncrieff, PhD University of Pittsburgh Deborah Dixon, MA, ASHA, Director, School Services Janet Deppe, MS, ASHA, Director, State Advocacy
BACKGROUND
Personnel preparation programs must develop valid and reliable student assessment instrumentation (ACAE 2005 standards; CAA standards 2013)
No standardized approach to clinical skills and knowledge across AuD and SLP programs
Several states have developed performance measures for classroom teachers, but not for Audiologists & Speech-Language Pathologists (Asha, 2012)
Development of student/clinician performance evaluation should follow a systematic approach, using appropriate resources
OBJECTIVE
We applied a systematic approach to design an instrument to assess and track AuD and SLP student knowledge and skills necessary for provision of services to high need children in underserved populations. The steps were as follows:
1. Identification of relevant skills and knowledge (i.e. items to be rated)
2. Application of a reliable and valid rating scale
3. Implementation of a data collection mechanism
METHOD
1. Identification of valid skill and knowledge list
• Identified resource materials e.g. publications, standards, etc.(see next panel)
• Solicited input from relevant constituents e.g. school personnel, consumers, etc.
• Peer review and feedback
• Pilot items with clinical instructors 2. Application of a valid and reliable rating scale
• Determine scale format e.g. # of points vs. checklist
• Create anchor values
• Create clearly defined descriptor for each scale point
• Considered priorities for psychometric characteristics
3. Implementation of a manageable data collection mechanism
• Format i.e. paper based vs. online?
• Availability of support staff for distribution and collection?
• Use of existing online mechanisms e.g. Typhon AHST, E*Value
RESOURCES, CONT…•Produced by the American Speech‐Language‐Hearing Association’s(ASHA) Value‐Added Project Team member requests, rapidly developing
•State‐level policies regarding accountability measures for school‐based speech‐languagepathologists (SLPs) •Objective was to identify a value‐added model designed for SLPs or onethat specifically accounted for the unique contributions of SLPs
• Team reviewed literature, attended seminars, and conducted a peer review to obtain input from related professional organizations, members, pertinent stakeholders, and researchers on value‐added models and assessments
SAMPLE SKILLS/KNOWLEDGE ITEMS
Addresses cultural /linguistic variations in
screening/assessment activities
Calculate classroom reverberation times
Participate effectively on multi-disciplinary team
Minimize barriers to curriculum access
Assist educational team member in making referrals
Train and supervise support personnel
Write appropriate IEP goals, considering academic,
behavioral, and developmental issues
Counsel regarding transition planning
FUTURE PLANS
1. Currently piloting with selected clinical instructors
2. Implementation of instrument in Fall term 2013
3. Evaluation of instrument pending responses on first round of implementation
4. Ongoing editing of items and scale as per constituent feedback
SELECTED REFERENCES
Kogan J, Conforti L, Benabeo E, Iobst W, Holmboe, E(2011). Opening the black box of clinical skills assessment via observation: a conceptual model. Medical Education 45 1048-1060 doi: 10.1111/j.1365-2923.2011.04025.x
American Speech-Language-Hearing Association (2005). Quality indicators for professional service programs in audiology and speech-Language pathology. Available from: www.asha.org/docs/html/ST2005-00186.html
American Speech-Language-Hearing Association (2012). Performance assessment of contributions and effectiveness of speech-language pathologists (PACE). Available from: http://www.asha.org/uploadedFiles/SLPs-Performance-Assessment-Contributions-Effectiveness.pdf
American Psychological Association (1999). Standards for Educational and Psychological Testing., Washington, DC: American Educational Research Association
Crossley J, Humphris G, Jolly B. (2002). Assessing Health Professionals. Medical Education 36 : 800-804
PERFORMANCE ASSESSMENT OF CONTRIBUTIONS AND EFFECTIVENESS OF SPEECH-LANGUAGE PATHOLOGISTS (PACE)
Value Added Assessment Research Findings Rationale for the development of the
PACE Goals of an assessment system Components of PACE Tools and resources
WHAT IS VALUE-ADDED ASSESSMENT?
Value-added assessment a process to accurately and fairly assess a professional’s impact on student performance and overall success of the school community.
A comprehensive, statistical method of analyzing test data that measures teaching and learning.
WHY IS VAA IMPORTANT?
Federal grant programs and waivers require states to include VAA/ teacher accountability measures in applications
VAA is viewed as an important accountability measure
Teacher accountability systems are being developed in many states
RESEARCH IMPLICATIONS
Research has primarily focused on implications of use of VAA with classroom teachers.
Notable concerns surfaced, such as difficulty linking student outcomes to one teacher and uncertainty about the accuracy of imputation models for missing student data.
RESEARCH IMPLICATIONS
Evaluating the value that an SLP brings to the school or connecting their value to specific student performance is a challenge when compared to a classroom teacher.
ASHA’s Value-Added Working Team was not able to identify any VAA models that specifically incorporated SLPs.
RATIONALE FOR THE DEVELOPMENT OF PACE
Since there was no system specifically developed for SLPs or other support personnel ASHA wanted to ensure that the assessment model for SLPs: accurately reflects the speech-language
pathologist’s (SLP) unique role in contributing to a child’s overall performance.
Demonstrates that the SLP is contributing to the success of the school community.
COMPONENTS OF AN EVALUATION SYSTEM
ASHA also wanted to make sure that the evaluation system for SLPs was: Comprehensive Used multiple measures Demonstrated valid and reliable findings Provided data for professional
development objectives Linked to the specific roles and
responsibilities of the specific job
COMPONENTS OF THE PACE
- PACE Matrix Portfolio Observation chart Teacher, student, and parent checklist Self reflection tool Observations by individual with knowledge of the roles and
responsibilities of SLP The matrix consists of a set of nine objectives by which
an SLP should be evaluated. These objectives are derived from typical roles and
responsibilities of a school based SLP The portfolio is developed to show evidence of mastery
of each objective
TOOLS AND RESOURCES
The PACE documents can be located at: http://www.asha.org/Advocacy/state/Performance-Assessment-of-Contributions-and-Effectiveness/
Additional tools/resources include: A guide for developing the portfolio for the
Matrix An evaluator’s guide Observation “Look Fors” and scoring system for
the Matrix A Step-by-Step guide for using the PACE system
DISCUSSION QUESTIONS: SYSTEMATIC APPROACH TO INSTRUMENT DEVELOPMENT
When considering an assessment
instrument, what are relevant quality markers necessary to include?
What are the challenges of evaluating related services scholars and graduates?
How can data collected be used to improve program quality?
Can we ensure “calibrated” responses across raters?