FACULTY OF HEALTH SCIENCES

CENTRE FOR DISABILITY RESEARCH AND POLICY

INTERNATIONAL DEVELOPMENTS IN MEASURING OUTCOMES

Roger J StancliffeEric Emerson

1

Monitoring progress in achieving disability equality: Fulfilling Potential

September 2012

Discussion document

Discussions So Far

Next Stepsof

disability

Actions, outcomes

and indicators

Obtain wide range of

opinions and ideas to inform new strategy

Highlight current and

planned activity and

publish responses to discussion document

Outline strategic

priorities and explain how

specific actions will be

developed

Draw on statistics

and research to explore nature of

disability in the UK today

Outline specific

actions and timelines and

how progress will be

monitored

Buildingunderstanding

July2013

Co-production

December 2011

September2012

January 2013

FULFILLING POTENTIAL

REPORTS

http://odi.dwp.gov.uk/fulfilling-potential/index.php

DOMAINS & INDICATORS

› 11 domains containing - 1 or 2 headline

indicators - up to 11

supporting indicators

Friends & family

Housing

Information & access

Health & wellbeing

Transport

Social participation

Income

Choice & control

Employment

Education

0 2 4 6 8 10 12 14

Number of Indicators

5

KEY MESSAGE 1

› Co-production is key to balancing the interests of- Government- DPOs- (People with

disabilities)

Better housing

More friends/leisure activities

More choice/independence

Job change/Flexibility

More time with family

More money

Improved health

-20% 0% 20% 40% 60%

ONS Opinions Survey 2013What one thing would make the

biggest different to your quality of life?

KEY MESSAGE 2

› Monitor the inequality gap- over time - and for ‘at risk’ groups

8

LEFT BEHIND

› Framework- v1 (2009) UN Convention on the Rights of

Persons with Disabilities

- v2 (2011 onwards) Australia’s Social Inclusion Indicators Framework

› Data - Annual survey of Household Income & Labour

Dynamics in Australia (HILDA)

- Indicators matched to 44% of Framework indicators

Monitoring Changes in the Wellbeing of Young Disabled Australians

LEFT BEHIND 2014

› Between 2001 and 2012, the gap between young Australians with disabilities and their non-disabled peers has grown in 11 (of 22) areas including - Not being employed

- Being long-term unemployed

- Having low economic resources and financial stress

- Having low subjective well-being

- Not having someone to turn to in times of crisis

- Not having a voice in the community

- Experiencing entrenched multiple disadvantage.

› It has not narrowed in any area at all

10

LEFT BEHIND 2014

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20122

20 Risk of Expos-ure to En-

trenched Mul-tiple Disadvant-

age Among of Australian Ad-

olescents and Young Adults:

2001-2012

Capturing the views of service users: England’s Adult Social Care Survey

ADULT SOCIAL CARE SURVEY

› Annual survey (began 2010/11)

› How effectively are services helping users to live safely and independently? What is impact of services on their quality of life?

› Cross-sectional stratified random sample from every Council with Adult Social Services Responsibilities (CASSR, n=154)

› 2012/13 sample > 50,000

› Service users aged 18 and over in receipt of services funded wholly or in part by Social Services

› Questions about - Subjective wellbeing- Social relationships- Leisure activities- Safety- Food quality - And more ………………….

http://www.hscic.gov.uk/article/2215/User-Experience-Survey-Adult-Social-Care-Guidance-2012-13

SELECTED RESULTS 1

0%

10%

20%

30%Physical Disability, Frailty and Sensory Impairment Mental Health

Intellectual Disability

SELECTED RESULTS 2

0%

20%

40%

60%

80%2010/11 2011/12 2012/13

WHAT IS GOING ON?

› The data are valid, English disability services (especially for people with intellectual disability) are excellent and getting better

› People with intellectual disability have low expectations

› Supported responding introduces significant positive bias

‘Concentrating on mental characteristics (such as pleasure, happiness or desires) can be particularly restrictive when making interpersonal comparisons of well-being and deprivation. Our desires and pleasure-taking abilities adjust to circumstances …. deprived people tend to come to terms with their deprivation … [as such] ….. the deprivation of the persistently deprived may look muffled and muted’

SELECTED RESULTS

0%

4%

8%

12%Self-completion Completion supported by careworker9% 56%

RESEARCH INTO ACTION Implications

1. Invest in annual data collections that can be used to monitor all key aspects of disability equality

2. Use subjective measures of wellbeing with extreme caution (if at all)

3. Invest in developing supports for survey completion that are independent of service provider agencies

4. Develop alternative ways of capturing the voices of people who use disability services

FACULTY OF HEALTH SCIENCES

CENTRE FOR DISABILITY RESEARCH AND POLICY

International developments in measuring outcomes:US National Core Indicators

18

Outline

1. Describe National Core Indicators (NCI)

2. Examples of policy-relevant NCI analyses1. Choice of Living Arrangements

2. Wellbeing and Choice of Living Arrangements

3. ASD Eligibility Policies

3. How is NCI Data Used by US States?1. Example from Kentucky

4. Possible Applications in Australia

1. National Core Indicators (NCI): Features, development, current use and future expansion

Who participates in the NCI?

› Annual survey on a random sample of service users (400+ per state)- Longitudinal (multi-year) data on specific individuals not available by design (to

avoid survey fatigue)

› States opt in- Some states add questions to the standard NCI instrument to investigate issues

of specific local policy interest

- Some states focus on different parts of their service system in different years by oversampling different subgroups

› Used only by the intellectual disability and developmental disability (ID/DD) service system in each participating state

Who has access to NCI data?

› Benchmarking: For each indicator the NCI provides for:- state-by-state comparisons,

- comparisons with the national average,

- year-by-year comparisons within states

› Summary data publicly available via the NCI website http://www.nationalcoreindicators.org/

› Individual state annual reports available via NCI website

› Deidentified NCI data shared with (selected) university researchers for independent secondary analysis

NCI Indicator Framework

Individual Outcomes

Employment

Choice & Control

Relationships

Community Inclusion

Family Indicators

Information & Planning

Access to Supports

Community Connections

Choice & Control

Health, Welfare, &

Rights

Health & Wellness

Safety

Respect & Rights

System Performance

Service Coordination

Incidents & Mortality

Staff Turnover

= Adult Consumer Survey

NCI Participating States 2010-2013

2010-11 24 States 2011-12 29 States2012-13 35 States

HI

WA

AZ OK

KY

AL

NC

PA

ME

MA

SD

TX

AR

GANM

NJ

MO

NY

LA

OH

NH

DCCA

FL

IL

OR WI

IN

MI

MS

SC

VAMD

CTRI

UT

2. Examples of policy-relevant analyses using NCI data

Choice of Living Arrangements

Overall What percentage of adult service users living outside the family home choose where and with whom they live?

POLICY IMPLEMENTATION QUESTION

Overall NCI Choice Results 20086778 adult developmental disabilities service users living in non-family-homeservice settings in 26 US states (Stancliffe et al., 2011)

CONCLUSION

› Most people have no choice of where to live (55%) or whom to live with (59%).

› Policies endorsing choice of living arrangements are not being implemented satisfactorily.

Choice of Living Arrangements

Does choice of living arrangements vary by residence type and level of disability?

POLICY IMPLEMENTATION QUESTION

Mild Moderate Severe Profound0

20

40

60

80

InstitutionGroup homeApartmentOwn homeFoster careNursing facil-ity

Level of ID

Perc

enta

ge

Choosing Whom to Live With (person chose)by Level of Disability and Residence Type

CONCLUSION

› People with severe/profound intellectual disability had little or no choice of whom to live with, regardless of residence type.

Choosing Whom to Live With (person chose)by Level of Disability and Residence Type

Mild Moderate Severe Profound0

20

40

60

80

Group home

Own home

Level of ID

Perc

enta

ge

Own home

Group home

CONCLUSIONS

› For people with mild and moderate intellectual disability, choice of living companions varies dramatically by residence type:

- own home (73.5% and 57.3% chose)- group home (9.5% and 9.7% chose)

› These findings support policies promoting individualised settings, such as one’s own home or an agency apartment. - These settings do provide substantially more choice

about living arrangements, as intended.

Wellbeing and Choice of Living Arrangements

Choice of Living Arrangements

Does exercising choice of living arrangements lead to greater wellbeing? (Stancliffe et al., 2009)

POLICY IMPLEMENTATION QUESTION

NCI Wellbeing Outcomes

› Loneliness› Feeling happy

At Home

› Feeling afraid at home› Feeling afraid in your neighborhood› Home staff nice and polite› Liking home

Self-Report Data Only

› Well-being items come from Section I of the NCI Consumer Survey, which may only be completed by interviewing the person receiving services. Due to communication difficulties, some service users could not take part in the interview.

› Only included participants who were judged by interviewers to have given valid and consistent interview responses.

› These selection criteria yielded predominantly people with mild or moderate ID.

Loneliness the most widespread problem

Lonely Scared Home

Scared N'hood

Happy Staff Home

Like Home

0102030405060708090

100

53.9

79.7 79.2 83.290.6 88.6

Percent with positive outcome

Well-being outcome

% o

f tot

al s

ampl

e

Results Summary

Item Chose Who to Live with

Chose Where to Live

Loneliness Feeling happy Afraid at home

Afraid in neighbourhoodHome staff nice Like home

Personal characteristics controlled statistically in all comparisons.

Conclusion

› Choosing where to live and whom to live with each are associated with: - multiple wellbeing benefits and - no wellbeing detriments.

Policy Analyses and Outcomes: ASD Eligibility Policies

› Grouping states by common policies to evaluate the impact of these polices on service provision and client outcomes.

EXAMPLE› Hewitt et al. (2011) compared the proportion of state ID/DD service users

with and autism/ASD diagnosis by state autism/ASD service eligibility policies:

ASD Eligibility Policies No. of States

% service users with ASD diagnosis

None 6 6.6%Related condition (RC) 14 8.4%RC + autism –specific HCBS 5 9.3%

x2(2, 12,382)=17.39, p<.001

3. How is NCI Data Used by US States?

Overview of NCI Use at State Level

› Overall quality management- Set priorities for quality improvement- Report evidence to federal funders

› Report results to stakeholders- Internal state staff- Quality councils/review committees- State legislatures- Providers- Individuals and families receiving services

Kentucky: Example of State Application

› Quality Improvement Committee (QIC) convened in 2012

- Identified health and exercise as target area

- Initiative funded 8 pilot programs promoting inclusive physical fitness and healthy eating activities

› See Moseley, Kleinert, Sheppard-Jones & Hall (2013)

Kentucky Health and Wellness Initiative

Community-based Employment

› 7% of respondents from Kentucky and 14% of respondents across NCI States were reported to be working in community-based employment (settings where most people do not have disabilities)

Recommendation: Increasing integrated community employment

Changes included:

› Large increase in hourly funding rate to providers of integrated supported employment.

› Small decrease (11%) in hourly funding rate to providers of day activity services.

Friendship and Loneliness

› 72% of respondents from Kentucky and 40% of respondents across NCI States reported they feel lonely at least half of the time.

Recommendations: Loneliness and friendships.

Changes included:

› Increased hourly funding rate for all integrated services and decreased rates for all segregated services.

› Launched a ‘‘Community Belonging’’ initiative, starting with 10 agencies, to ensure that people are connected to their communities via unpaid relationships.

Expanding Public Use of NCI Data

Researchers using NCI Data- Research policy and process for requesting data and/or

tools- Formal process through NASDDDS Research Committee- Several university researchers and students approved and

currently working with data (autism, ageing, health – e.g., University of Minnesota)

Public use of NCI Data› New website with chart generator feature – customisable summary data publicly available

Chart Generator www.nationalcoreindicators.org

4. Possible Applications in Australia

OVERALL CONCLUSION

› Regular assessment of outcomes experienced by people with disability facilitates evaluation of:- Benefits to service users

- Benefits of different service types

- Benefits to service users with different characteristics

- Policy implementation and effects

- Relative disadvantage compared to the general community (subject to availability of comparison data).

› Countries that do not currently have a national system for assessing outcomes should examine the NCI for its local applicability.

Application of the NCI in Australia

Issue NCI in USA AustraliaTarget groups ID/DD only All disability types

Level of analysis State Needs discussion

Compliance burden Costs borne by each state.Sampling strategy means limited burden on individual service users and providers.

Needs discussion

Data availability Raw data: State officials, independent researchersSummary data: Public

Needs discussion

Process for identifying indicators

Consensus among participating states and organisations.

Needs discussion

NCI items used extensively in 1999 Australian national survey

Implications and actions for policy makers• Building on international examples, develop, field test,

implement and refine an Australian national system of outcomes monitoring for disability service users.

• Gather (some) outcomes data directly from disability service users.

• Use the outcomes data to evaluate and improve the effectiveness of disability services, funding and policy.

• Make deidentified data publicly available.• Create a body, with relevant expertise, to manage the

outcomes monitoring process.

Research into Action

Implications and actions for practice• Develop a consumer outcomes focus when

delivering and managing disability services.• Use outcomes data to evaluate and improve the

effectiveness of disability services and policy.• Educate stakeholders about outcomes and

outcomes data.

References

Bradley, V. J. & Moseley, C. (2007). Perspectives: National Core Indicators: Ten years of collaborative performance measurement. Intellectual and Developmental Disabilities, 45(5), 354-358.

Hewitt, A. S., Stancliffe, R. J., Johnson Sirek, A., Hall-Lande, J., Taub, S., Engler, J., Bershadsky, J., Fortune, J., & Moseley, C. (2011). Characteristics of adults with autism spectrum disorder who use adult developmental disability services: Results from 25 US states. Research in Autism Spectrum Disorders, 6(2), 741-751.

Moseley, C., Kleinert, H., Sheppard-Jones, K., & Hall, S. (2013). Using research evidence to inform public policy decisions. Intellectual and Developmental Disabilities, 51(5), 412-422. doi: 10.1352/1934-9556-51.5.412

Sheppard-Jones, K., Hall, S., & Kleinert, H. (2011). Using large-scale data sets to inform state DD policy. Community Services Reporter, 18(9), 4–10.

Sheppard-Jones, K., Prout, H., & Kleinert, H. (2005). Quality of life dimensions for adults with developmental disabilities: A comparative study. Mental Retardation, 43, 281-291.

Stancliffe, R. J., Lakin, K. C., Larson, S. A., Engler, J., Taub, S., & Fortune, J. (2011). Choice of living arrangements. Journal of Intellectual Disability Research, 55(8), 746-762.

Stancliffe, R. J., Lakin, K. C., Taub, S., Chiri, G., & Byun, S. (2009). Satisfaction and sense of well-being among Medicaid ICF/MR and HCBS recipients in six states. Intellectual and Developmental Disabilities, 47(2), 63-83.

Centre for Disability Research and Policywww.sydney.edu.au/health_sciences/cdrp/

Email: disabilitypolicy.centre@sydney.edu.au

eric.emerson@sydney.edu.au roger.stancliffe@sydney.edu.au