developing a context-sensitive approach to at outcomes measurement
Post on 17-Feb-2016
28 Views
Preview:
DESCRIPTION
TRANSCRIPT
Developing a Context-Sensitive Approach to AT Outcomes Measurement
OT 541Fall 2003
James A. Lenker, MS, OTR/L, ATP
What are Outcomes?
Outcomes are the result of an intervention (Scherer, 1998)
Outcomes answer the question, "Does it work?" (Oldridge, 1996)
Outcomes are the impact produced by AT devices in the lives of users and their environments (Fuhrer, 2001)
What is Outcomes Measurement?
"The evaluation process in the service delivery system that is designed to measure and establish a baseline of what works; how well something works; for which clients it works; and at what level of economic efficiency it works” (DeRuyter, 1997)
Purposes of Outcomes Measurement
Funding justification for equipment and services (assessment & training)
Measure individual's change over time
Program evaluation: "Who do we serve? How well do we serve them?"
Domains of Outcomes Measurement
Many domains of inquiry can be considered an "outcome" Medical status Activity performance Quality of Life (QOL) impact Community impact AT usage & usability
AT Outcomes Research – Developmentally Speaking
Blissful ignorance (Lenker, 2000)Denial ( “ )Exhortative (Fuhrer, 1999)Sober appraisal ( “ )Infrastructure building ( “ )Model demonstration ( “ )Getting on with it ( “ )
AT Outcomes Research: Challenges
Heterogeneity of…. End users: age, abilities, needs, support AT devices Contexts of use Reimbursement models Practitioner skills
Limitations of Previous Studies
Incomplete information about reasons for device abandonment
Inconsistent definitions for prevalence of use that have been difficult to compare across studies because use and abandonment are often defined differently
Changes in functional status when evaluating long-term device use and disuse, which bias the results toward high rates of ‘abandonment’.
Limitations of Previous Studies
Incompletely defined outcomes indicators (e.g. comfort, safety, and function), hindering replication by others
Incomplete information about device success
Measurement tools that were not sensitive to improvements in function
Low numbers of participants
Research Needs
Measurement of impact on social participation and on work (Fuhrer, 1999)
Consideration of ATD usage within context of use service delivery model, and user characteristics (Fuhrer, 1999)
Methods that embrace the user's perspective and assess impact on quality of life (Fuhrer, 1999; Jutai et al., 1996; Scherer, 1998)
Approaches that evaluate underlying factors contributing to overall success/failure of AT interventions (De Ruyter, 1994; Scherer, 1996)
Research Needs (con’t.)
Measurement of the hassle experienced by end users, comparing trade-offs against the functional gains (Sprigle & Abdelhamied, 1998)
Measurement for reduced dependence (Fuhrer, 1999)
Description of devices that are achieving desirable outcomes, which would provide the rationale for funding ATD devices and/or training services (Scherer, 1996).
The Chain of Outcomes Research (Keith, 1995)
Conceptual Models => relationships among domains
Domains => measurement toolsMeasurement tools => dataResearch design links treatment to
outcome, i.e. interpretation of data, i.e. “outcomes”
Outcomes influence practice and help build theory
So, Where Do I Start?
Know the context of AT useIdentify all relevant stakeholdersDevelop specific research questionsIdentify indicators that will answer your
questionsIdentify measurement tools
Useful Conceptual Perspectives
Quality of LifeProduct UsabilityFunctioning and HealthHAAT: Human, Activity, Assistive
TechnologyMPT: Matching Person & Technology
Quality of Life: The Bottom Line
Definition: “(Quality of Life is) the degree to
which the person enjoys the important possibilities of his or her life” (Renwick & Brown, 1996).
Quality of Life (Renwick & Brown, 1996)
Quality of Life
Physical• Personal safety • Health• Fitness• Mobility
Social• Relationships family, friends• Community
Material• Finance• Housing• Transport• Security & tenure
Development & Activity• Competence independence, choice & control• Productivity job, education home life, leisure
Emotional Well Being• Mental health• Self-esteem• Faith• Sexuality• Fulfillment
Product Usability Perspective
Effectiveness Does it achieve the desired goal?
Efficiency What are the costs associated with
achieving the desired goal? Ex: $$, time, speed, errors effort, comfort, pain, hassle
User satisfactionInternational Standards Organization. (1993). ISO CD 9241-11.2
Smith’s Parallel Interventions Model
Reduce the Impairment (Therapy)
Change technique / Increase Skill
Use AT Devices and Services
Redesign the Activity
Redesign the Environment
Use Personal Assistance
ICF – AT Model (Lenker & Jutai, 2002)
HAAT: Human, Activity, Assistive Technology (Cook & Hussey,
2002)
CONTEXTSocial ContextFamilyPeersStrangersSettingHomeSchoolEmploymentCommunityPhysical ContextSpaceLightSoundTemperature
MPT:Matching Person &
Technology (Scherer,
1998)
Getting Down to Business
What question(s) do you want to answer? This the most important consideration when
designing your study & picking a toolWho are the stakeholders?
Consumers, clinicians, administrators, manufacturers, policy makers
What is your conceptual orientation?What are your resources?
Develop Specific Research Question(s)
Must be a match between Stakeholder interests Reimbursement model User characteristics AT device Context of use
Identify Stakeholders
Who is your target audience? End users Practitioners / direct service providers Program administrators Manufacturers of AT devices Funding / policy agencies
State VR agenciesFederal agencies that sponsor rehabilitation
R&D programsState, local departments of Education A
Describe Context of Use
Reimbursement Model Educational, vocational, medical Practitioners involved in service delivery User involvement in service process
AT User Physical, cognitive, learning, sensory Ages and environments of use Needs / goals / roles
Context of Use (con’t.)Assistive Technology Devices
Wheelchair mobility, postural support, seating Adaptive computer software and hardware Alternative & augmentative communication aids Job accommodations, often ‘ergonomic’ Aids for daily living activities
Important!! AT devices are uniformly defined but they should not be uniformly studied
What data will answer your research question?
The domain of data may include Functional performance, QOL impact, pain,
usability, user satisfaction, etc.The perspective of data may be
objective (by observation) subjective (self-reported)
The nature of data may be quantitative (data are numeric) qualitative (data are language-based)
Outcomes research requires measurement tools!
The most important consideration when picking a tool? Knowing what question (s) you want to
answer
What other considerations matter when choosing a measurement tool?
Does it measure what you want it to measure? Does it measure what it purports to measure? Is it easy for participants to answer consistently?
Test-retest reliability Is it easy for multiple observers to rate
consistently? Inter-rater reliability
What demands does it place on respondents? What demands does it place on administrators?
Is tool-specific training required? Is it time consuming to administer? How difficult is it to score the results?
Accessibility of format for people with disability?
A Useful Model: ICF
Presents a functional continuum of human performance in life situations
Provides framework for outcomes research Interventions Reimbursement / service delivery models Assessment tools Outcomes questions
ICIDH-2 / WHO Model in the Context ofReimbursement Models and Assistive Technology Interventions
ICIDH-2Domain
Body Functions and Structures (Body level):
Activities (Individual level):
Participation (Role level):
Contextual Factors (Societal level):
Domain-relevant examples
Functional and structural integrity vs. impairment. Normal physiologic processes and organ systems function:Cognition, sensory, perception, motor, physiologic and psychological processes
Performance of action or activity (cognitive, sensory, motoric, and/or behavioral) that facilitates performance in multiple roles: - initiate, organize, sequence, judge, attend- sit, lift, stand, climb, reach, pinch, grasp, hold, release- relate, interact, cope- hear, speak, see, taste move- read, write, learn, understand
Involvement in life situations and performance in roles (family, work, and educational; self-care; leisure) that take place in specific contexts (physical, social, cognitive).
Features of the physical and social world that affect the individual but are not specific to the individual's ability or disability; e.g. physical barriers in public spaces, employer attitudes; funding for social service benefits; public policy;
Domain-relevant Assistive Technology
Heart pacemaker, hip/knee replacement, cochlear implant, baclofen pump
Mobility aids: wheelchairs, crutches, canes, walkers; UE orthotics; eyeglasses, hearing aid, AAC device
Job modifications of-task/device/ environment; home modification;vehicle modification; computer-based AT
Accessible public venues: buildings, transportation, communications, recreation; 3rd party reimbursement for AT services & equipment
Funding Model
Medical Rehabilitation Vocational RehabEducationalIndependent LivingEmployer
Independent LivingPublic Policy
Adapted from:Baum, C. M., & Law, M. (1997). Occupational therapy practice: Focusing on occupational performance. American Journal of Occupational Therapy, 51, 286.Gray, D. B., Quatrano, L. A., & Lieberman, M. L. (1998). Moving to the next stage of assistive technology development. In D. B. Gray & L. A. Quatrano & M. L.
Lieberman (Eds.), Designing and Using Assistive Technology: The Human Perspective (pp. 299-309). Baltimore, MD: Paul H. Brookes.Moyers, P. A. (1999). The guide to occupational therapy practice. American Journal of Occupational Therapy, 53, 247-322.World Health Organization. (1999). ICIDH-2: International classification of functioning and disability. (Beta-2 draft). Geneva: Author.
Example Research Questions
Do computer-based ATDs improve the psychological and emotional well-being of end-users?
Do computer-based ATDs increase end-user
participation in work, school, and/or leisure activities?
Do users with clearly defined goals become more successful user of computer-based ATDs?
Does impact of ATD use vary across disability populations?
Research Questions (con’t.)
Are there societal barriers that preclude or interfere with participation in school, work, or leisure activities?
Are there significant technical problems with computer-based ATDs that are limiting performance for users of these products?
Are there any themes or variables that are associated with successful or unsuccessful computer-based ATD use?
Indicators of AT “Outcome”
Psychosocial impact on the individual User satisfaction with device User satisfaction with services Achievement of user goals Prevalence of long-term AT use Tasks and roles in which AT is used Environmental influences that contribute to
functional AT use/abandonment Unmet technology needs Employment / educational success
AT Outcome Measurement Tools
PIADS: Psychosocial Impact of Assistive Device Scale
QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology
COPM: Canadian Occupational Performance Measure
MPT: Matching Person & Technology
PIADSAttribute: Quality of Life
3 sub-scales: Competence, adaptability, self-esteem
26 item, 7-point, self-reportStrengths
Not device-specific Identifies potentially useful points of departure
for exploration Quick, easy to administer Useful retrospectively or pre/post
PIADS
Additional considerations Items are abstract, some users will
struggle -- especially kids/adolescents Better to administer in person Open-ended probing questions can
provide insight about specific scores
COPM
Semi-structured interview format, approx. 30-40 minutes
Process Identify problem areas Prioritize 5 areas of greatest importance User rates each of the 5 areas on
Current performance levelSatisfaction with that performance level
COPM
Strengths User centered
Philosophically soundEffective as intervention approach; increased
consumer ‘stake’ in the outcome (?) Sensitive to clinically significant changes in
performance and satisfaction with performance Useful for variety of devices and contexts of use Outcome indicators are built-in to the
assessment process
COPM
Additional considerations Goals are likely to be interviewer-
dependent, which will affect inter-rater reliability
The specificity of goals to the end user's circumstances hinders generalizability of results to other users
Goals may not suit all stakeholder perspectives
QUEST
12 items, 5-point scale Device satisfaction Services satisfaction
Quick, straightforward to administer (10-15 mins.)
Suitable for phone or distance surveysSuitable for wide range of AT products and
user demographics
QUEST
Additional considerations Expectations often color the perceptions
of satisfaction, therefore expectations are a potential confounding factor
Satisfaction judgments are best supplemented with information that explains underlying predictors
User performance, or function, with use of AT device is not captured
MPT
Several instruments Survey of Technology Use ATD Predisposition Assessment (PA) Educational Technology PA Workplace Technology PA Health Care Technology PA
ATD Predisposition Assessment
Two perspectives are captured Professional assesses ...
Match between user abilities and ATD requirements
Psychosocial incentives/disincentives to AT use End user self-rates ...
Sensory & motor capabilitiesLife satisfactionPerceived value of device
MPT
Additional considerations AT ‘benefits’ to user are not explicitly captured Professional’s ratings of psychosocial incentives
and disincentives to AT use require knowing the consumer really well -- beyond the scope of a typical AT evaluation period
Professional’s ratings of psychosocial factors is subjective, subject to bias
Overall, does a fine job of identifying factors that could predispose user to frequent or infrequent use
Conclusion
Acknowledge the futility of identifying a single “best” AT outcomes tool that is appropriate for all contexts and research questions.
Accept the challenge of matching research methods and measurement tools to your research area
top related