integrated care - aci.health.nsw.gov.au · hhs department of health and human services, the usa hit...

Integrated CarePatient reported outcome measures and patient reported experience measures - a rapid scoping review

Jack Chen MBBS PhD MBA (Exec)Associate ProfessorSimpson Centre for Health Services ResearchSWS Clinical School and Ingham Applied Medical Research Faculty of MedicineUniversity of New South Wales, Australia

AcknowledgementThe author would like acknowledge and thank the senior staff of the Agency of Clinical Innovation of New South Wales for their helpful comments regarding the scope, methodology and content of the review as well as their comments on the first draft of the report. The author is also grateful to Dr Lixin Ou for her help in searching, retrieving and reviewing the literature. The views expressed in the review are entirely those of the author and should not be interpreted as that of the Agency of Clinical Innovation, New South Wales and, needless to say, any errors are the author’s.

1 | Integrated Care

The Abbreviation List

ABS Australian Bureau of Statistics

ACI Agency of Clinical Innovation

ACSQH Australia Commission on Safety and Quality in Healthcare

AHRQ Agency for Healthcare Research and Quality

AQoL-8D Assessment of Quality of Life (8 Dimensions)

BCF Better Care Fund

BHI Bureau of Health Information

CER Comparative effectiveness research

CIHI Canadian Institute for Health Information

CALD Culturally and linguistically diverse

CMS Centre of Medicare and Medicaid

COSMIN COnsensus-based Standards for the selection of health Measurement Instruments

CPHCRIN Canadian Primary Health Care Research and Innovation Network

CTT Classical Test Theory

ED Emergency Department

EHR Electronic health record

EQ-5D The European Quality of Life(EuroQOL) five dimensions questionnaire

FACIT the Functional Assessment of Chronic Illness Therapy system

GEM GRID-enabled Measures

GHS Global health status

HCAHPS Hospital Consumer Assessment of Healthcare Providers and Systems

HHS Department of Health and Human Services, the USA

HIT Health Information technology

HRQoL Health-related quality of life

HR-PROs Health-Related Patient-Reported Outcomes

HUI Health Utilities Index

IC Integrated care

IOM Institute of Medicine

IRT Item Response Theory

KPIs key performance indicators

LHDs local health districts

MBS Medical Benefit Scheme

MCID Minimal clinically important difference

MID Minimally important difference

NCGC National Clinical Guideline Centre

NEHTA National Electronic Health Transition Authority

Integrated Care | 2

NHP Nottingham Health Profile

NIH National Institute of Health

NQB National Quality Board

NQF National Quality Forum

NHS National Health Services

NICE National Institute for Health and Clinical Excellence

NQMC National Quality Measures Clearinghouse

OECD Organisation for Economic Co-operation and Development

OSG Online support groups

PAIS Patient Accreditation Improvement Survey

PBS Pharmaceutical Benefit Scheme

PCC Patient-centred care

PCEHR Personally controlled electronic health record

PCORI Patient-Centered Outcomes Research Institute

PDA personal digital assistant

PEx patient experiences surveys

PHC Primary health care

PREM Patient reported experience measures

PRO Patient reported outcome

PROQOLID Patient-Reported Outcome and Quality of Life Instruments Database

PROM Patient reported outcome measures

PROMIS Patient-Reported Outcomes Measurement Information System

PRO-PM Patient-reported outcome based performance measures

PS Patient satisfaction

PSat patient satisfaction surveys

PSIv5 UltraFeedback’s Patient Satisfaction Instrument Version 5

QALY Quality adjusted life year

QPS QPS Patient Satisfaction Survey

QoL Quality of life

QWB-SA Quality of Wellbeing Scale

RBDM Registry of Births, Deaths and Marriages

SF-12 The Short Form (12) Health Survey

SF-36 The Short Form (36) Health Survey

SNS Social network sites

WaPEF Warwick Patient Experience Framework

WHO World Health Organisation

WHOQoL-BREF World Health Organisation Quality of Life Instrument

3 | Integrated Care

ContentsThe Abbreviation List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

1 | Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2 | Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.1 Patient reported outcomes (PROs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2 The narrow and broad definitions of PROs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3 Patient reported outcome measures (PROMs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Patient reported outcome measure (PROMs) vs patient reported experience measures (PREMs) . . . . . . . . . . . . . . . . . . 152.5 Why PROMs/PREMs are important to measure? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.5.1 From a theoretical and conceptual point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.5.2 From the clinical and health economic point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.5.3 From quality improvement and a societal point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.6 Framing PRO in a broader determinants of health model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.7 The pathways from PRO to PRO-PM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 | Selecting PROMs and PREMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.1 Where to find PROM/PREM? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2 How to choose a PROM/PREM – methodological issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2.2 Other confounding factors of choosing a PROM/PREM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.3 Implications of different methods and modes on response rate, reliability and validity. . . . . . . . . . . . . . . . . . . . . . . 263.2.4 Minimally important differences and changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.5 Response shift, adaptation and other challenges to detect true change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.2.6 Generic versus disease- or condition-specific PROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.2.7 Other methodological issues related to choosing PROM/PREMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3 Using PROM/PREM as quality measures: the desirable attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.4 Other principles of selecting PROMs/PREMs in IC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4 | Capturing PROMs and PREMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1 The challenges in capturing PROMs and the promises of PROMIS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.2 Generic HRQoL measures in primary care setting: results from an evidence review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.3 Measuring patient experience in a primary health care setting – an evidence review from Canada . . . . . . . . . . . . . . . . 374.4 Measuring IC based on the AHRQ ‘Care Coordination Atlas’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.5 Measuring patient experience in IC in the UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.6 Measuring ‘continuity of care’ based on patient experience – a systematic review of the instruments . . . . . . . . . . . . . . 564.7 The key points in measuring PREMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5 | Using PROMs and PREMs to improve health care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.1 A conceptual framework in understanding the effect of PROM/PREM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.2 IC and PROMs/PREMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.2.1 How is patient-centered care (PCC) defined? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Integrated Care | 4

5.2.2 Can the concept of PCC be measured? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.2.4 Patient experience vs patient satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.3 International experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.3.1 The USA experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.3.2 The UK experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.3.3 The Sweden experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.3.4 The Denmark experience: the generic Integrated PRO System (WestChronic) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.4 Measuring PREMs in non-primary care settings in Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685.3 Measuring PREMs in primary-care settings in Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.6 Social media, cost-effectiveness and IC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6 | Future plans & research priorities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

6.1 Psychosocial behaviours in EHRs as part of PROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.2 Integrating PROMs with EHRs and other data sources: the need for robust health information infrastructure . . . . . . 746.3 Developing PROM/PREM measures on important subgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756.4 Exploring the ways that the results can be better presented to different stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.5 Investment in understanding population norm, cut-off, MID, responsiveness, response-shift of the PROMs/PREMs 766.6 Investment in IRT/CAT technique and item banks for specific interested area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.7 Developing suitable case-mix adjustment methodology for different stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7 | Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

8 | Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

8.1 Appendix 1 – The list of systematic reviews on HR-PRO by COSMIN (489) (See attached document) . . . . . . . . . . . . . . . 808.2 Appendix 2 - Methodological issues related to the measurement properties of the PROMs/PREMs instruments (COSMIN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808.3 Appendix 3 – The sources of the instruments reviewed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Instrument Cost Notes: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 868.4 Appendix 4 – Measuring Patient Experiences in Primary Health Care Survey (Canadian) (see attached document) . 878.5 Appendix 5 – The June 2014 update of the AHRQ review results of the instrument in the care coordination measurement (Care Coordination Atlas) (see attached document) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.6 Appendix 6 – Social Media and IC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

8.6.1 Using social media to capture patient experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.6.3 Silver lining of the ‘cloud of patient experience’270 in a ‘perfect storm’271? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

8.7 Appendix 7 - Is IC cost-effective? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 938.8 Appendix 8 - Patient experience framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

8.8.1 The Picker principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.8.2 NHS Patient Experience Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.8.3 IOM Patient experience framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.8.4 The Warwick patient experience framework (WaPEF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958.8.5 The AHRQ ‘care coordination’ framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978.8.6 The patient experience frameworks and its implications on NSW IC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5 | Integrated Care

Tables

Table 2.5-1 The types and description of common health economic analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Table 2.7-1 Distinctions among PRO, PROM, and PRO-PM: Two Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Table 2.7-2 NQF Endorsement Criteria and their Application to PRO-PMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Table 3.2-1 Main characteristics of PRO methods issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Table 3.2-2 The strengths and weaknesses of generic and condition specific PROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Table 4.2-1 Overview of results from psychometric review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Table 4.3-1 Dimensions of patients’ experiences in primary health care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Table 4.4-1 Index of measures/instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Table 4.4-2 Care coordination master measure mapping table, patient/family perspective† . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Table 4.4-3 Care coordination master measure mapping table, healthcare professional(s) perspective† . . . . . . . . . . . . . . . . . . 44

Table 4.4-4 Care coordination master measure mapping table, system representative(s) perspective† . . . . . . . . . . . . . . . . . . . . 45

Table 4.4-5 The June 2014 update of AHRQ review results of instrument in care coordination measurement (Note: page number refers to the page number in the original document)10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Table 4.5-1 Existing user/carer experience measures in large national surveys36 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Table 4.5-2 The 18 supplementary questions developed for measuring IC experience in the UK setting . . . . . . . . . . . . . . . . . . 50

Table 4.6-1 Quality of measurement properties and the interpretability per instrument (Adopted from Uijen et al. 2012197) 57

Table 5.1-1 Possible outcome indicators for assessing the impact of the collection of PROM/PREM . . . . . . . . . . . . . . . . . . . . . . 61

Table 5.3-1 Selected outcomes indicators for Domain 2 of NHS Outcome Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Table 5.3-2 Elements of clinical application of PRO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Table 5.3-3 Characteristics of 22 projects involving implementations of a generic PRO system. Projects with patient level use (n=14) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Table 5.4-1 Frequency of domains used in the hospital patient experience and satisfaction surveys in Australia . . . . . . . . . . 68

Table 5.4-2 Survey tools used among St Vincent’s hospitals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Table 5.4-3 Patient experience and satisfaction surveys used in Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Table 8.2-1 Definitions of domains, measurement properties, and aspects of measurement properties . . . . . . . . . . . . . . . . . . . 81

Table 8.2-2 Quality criteria for measurement properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Table 8.6-1 Platforms and reported effects/outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Table 8.6-2 Potential sources of information for the ‘cloud of patient experience’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Table 8.6-3 Questions Asked for Anecdotal Comments and Ratings on NHS Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Table 8.8-1 A narrative commentary on IOM patient experience framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Table 8.8-2 The Warwick Patient Experiences Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Table 8.8-3 Mechanisms for Achieving Care Coordination (Domains) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Table 8.8-4 Relation Between the Care Coordination Measurement Framework and Other Key Sources . . . . . . . . . . . . . . . . . . . 98

Figures

Figure 2.2-1 Types of PROs currently used in medical research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 2.5-1 Patient-based evidence as part of the whole new evidence-base for high quality patient care . . . . . . . . . . . . . . . 15

Figure 2.6-1 Framing PROs within existing conceptual models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Figure 2.7-1 Pathway from PRO to NQF-endorsed PRO-PM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Integrated Care | 6

Figure 3.2-1 Types of respondent data and methods/modes of administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Figure 5.1-1 A hypothetical framework to understand the impact of routinely collected PROs on patient health outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Figure 5.2-1 The care coordination ring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Figure 5.3-1 Duty of quality and the NHS Outcomes Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Figure 5.3-2 Using PROM in the Swedish Hip Arthroplasty Register. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Figure 8.2-1 The diagram for completing the COSMIN checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Figure 8.2-2 COSMIN taxonomy of relationships of measurement properties. Abbreviations: HR-PRO, health related-patient reported outcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Fi Figure 8.6-1 Cumulative number of online ratings of hospitals in England on the NHS Choices website. . . . . . . . . . . . . . . . . 90

gure 8.8-1 Dimensions included in IC (i.e. person centered coordinated care) narratives (National Voice, 2013) . . . . . . . . . . . 94

Figure 8.8-2 Care Coordination Measurement Framework Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Executive Summary

Integrated Care | 8

The Agency for Clinical Innovation (ACI) conducted a rapid scoping review on Patient Reported Outcome Measures (PROMs) and Patient Reported Experience Measures (PREMs), with a particular focus on the NSW Health Integrated Care (IC) Strategy. This expert scoping paper has been produced to provide information, guidance and recommendations on patient reported measures question sets The overall aim of patient reported measures within the NSW Health IC strategy is to ‘enable patients to provide direct, timely feedback about their health related outcomes and experiences to drive improvement and integration of health care across NSW’.

The literature demonstrates that patients who are engaged in their health care tend to experience better outcomes, and choose less costly interventions, such as physical therapy for low back pain after they participate in a process of shared decision making. Measuring Patient Reported Outcomes (PROs) allows for better communication and shared decision making between patients and providers.

9 | Integrated Care

Using and selecting a measure When considering the selection and use of a PROM or PREM it is important to take into account the data source (i.e. self-report vs. proxy), the mode, method and setting of administration as well as generic vs. condition specific PRO and the needs of the target population (vulnerable populations, low literacy, language and culture, functional abilities). It is important to critically review the adopted PROM/PREM measures to see if they are suitable to be used across different population subgroups. The notable such subgroups in Australia include the Indigenous population, patients from culturally and linguistically diverse (CALD) backgrounds, women, children and the elderly, as well as patients with mental health problems or cognitive impairment. The burden of responding to the PROM/PREM on the patient and the burden of collecting these should not be underestimated.

When using a PROM/PREM as a quality measure in order to affect change, PROM/PREM could also be assessed according to the Agency for Healthcare Research and Quality (AHRQ) framework of desirable attributes of quality measures in three broad conceptual areas: (1) Importance of a measure (2) scientific soundness of a measure, and (3) feasibility of a measure.

One of the common challenges in applying PROMs in an IC setting is how to choose a generic PROM that measures HRQoL given the multitude of existing measurement instruments. To meet this challenge, the Canadian Institute of Health Research supported a group of researchers in 2013 to conduct a review. The researchers generated a short list of PROMs which included: PROMIS, SF-12/SF-36 and EQ-5D, after review of the evidence there was strong evidence to support PROMIS.

There is unlikely to be universal agreement on the key important dimensions of the patient experience to measure amongst different health jurisdictions. The choice of different dimensions/domains/subdomains of the patient experience very much reflects the value and preference of health systems, service providers and patients themselves.

InfrastructureCurrent opportunities exist to engage patients in building capacity and infrastructure to capture PROs routinely and then use these data to develop performance measures to allow for accurate appraisals of health care quality and efficiency.

Importantly, infrastructure needs to be in place for the collection of the collection of PRO data from patients, data needs to be synthesised in a meaningful way and presented back to both the patients and providers. This should also assist with developing strategies for acting on the information in a timely manner to improve the clinical care for patients (evaluating the amount, reasons and patterns of missing data is also important). The need for robust health information infrastructure to integrate PROMs with Electronic Health Records (EHRs) and other data sources is an important consideration particularly for the integration of any future PROM with the EHRs or personally controlled electronic health record (PCEHR).

Measurement and collectionMeasuring Patient Reported Outcomes is important as traditionally measured biomarkers often fail to correspond with how a patient is actually feeling. By measuring Health Related Quality of Life (HRQoL) in particular, directly acknowledges that patients often value different outcomes than their providers.

PROs will provide a key component to understanding the true burden of disease, especially in diseases that are marked by morbidity but not necessarily mortality. Measuring PROs allows for the assessment of a patient’s health status entering therapy and identifying treatable problems. Furthermore, it will determine the degree as well as the sources of the patients decreased ability to function and physical, emotional and social problems.

The systematic collection of PROs data has been shown to be more reflective of underlying health status than clinical reporting whilst also predicting meaningful clinical outcomes including survival. PROs data is feasible, efficient and valued by clinicians for documentation and clinical decision making whilst improving symptom management and patients’ overall health status. Measuring PROs in this context also allows for informed decisions regarding the change of treatment plans and predicting the course of diseases and outcomes.

InternationallyThe international experience, policy background and research interests are fast moving in relation to POM/PREM in IC. The UK, USA and other European countries have provided many policy initiatives in patient centred care (PCC)/IC as well as how to utilise PROM/PREM to

Integrated Care | 10

improve quality of care. However, the evidence base is still emerging. There is a generational change in developing and testing PROM/PREM, culminated by the success of the National Health Institute (NIH) funded PROMIS project which is based on Item Response theory (IRT) and Computerised Adaptive Test (CAT).

Overall, ten things that need to be considered when planning how to measure changes in patient and carer experience over time include:

1) Consider how the patient experience is being defined in order to inform precisely what needs are to be measured.

2) Think about why the patient experience is being measured and how the information will be used.

3) Assess whether it would be useful to combine approaches so that both qualitative and quantitative material is collected.

4) Consider whether to ask everyone using the services to provide feedback or only a sample.

5) Think about whether the best time to collect feedback is immediately after using the services when experiences are fresh in people’s minds.

6) Allocate enough time at the outset to plan and test measurement methods, particularly if these will be used in the future to monitor change over time.

7) Think about how the end result needs to be presented for a variety of audiences as this may shape how data are collected. Potential outputs include statistical averages, in-depth quotes or graphs.

8) Make sure that there is an appropriate infrastructure at an organisational level to analyse and use patient experience information.

9) Make sure that patients, carers, managers and health professionals are all comfortable with why the feedback is being collected and how it will be used. Staff need to be on board as well as patients.

10) Ensure that patient experience measures are seen as one component of a broader framework of measurement and that all of the approaches work well together without an excessive burden to either staff or patients.

1 | Introduction


1.1 BackgroundThe rising burden of chronic disease and the number of people with complex care needs in particular, require the development of delivery systems that bring together a range of professionals and skills from primary-, acute-, long-term and social-care sectors. Failure to better integrate or coordinate services along the continuum of care may result in suboptimal outcomes and the available evidence regarding integrated care (IC) programs points to a positive impact on the quality of patient care as well as improved health or patient satisfaction outcomes.

IC involves the provision of seamless, effective and efficient care that responds to all of a person’s health needs, across physical and mental health, in partnership with the individual, their carers and family. It means developing a system of care and support that is based around the needs of the individual (so-called patient-centeredness), provides the right care in the right place at the right time and makes sure funding is allocated to the most effective system of delivering healthcare.

The New South Wales Ministry of Health envisaged three clear directions for the future delivery of healthcare in NSW:

1. Keeping people heathy.

2. Providing world class clinical care.

3. Delivering truly IC.

As part of this vision, NSW Health developed a four-year strategic plan (2014-2017) and committed $120 million to implement new, innovative locally-led models of IC across the State which aims to achieve a health system with services connected across many different providers and will focus on individual patient needs.

Four areas have been prioritised for investment:

1) HealtheNet: This allows for the mapping of different patient identifiers to create a single picture of the patient’s information across all local health districts (LHDs), and is integrated with the national PCEHR to provide a comprehensive set of the patient’s information. The HealtheNet program allows patients and their clinicians, hospitals and other healthcare providers to view and share health information which will provide a more seamless healthcare experience.

2) Risk stratification: Developing tools for use by LHDs and healthcare providers to identify people at-risk of illness or chronic disease which can be followed-up by early and targeted intervention.

3) Patient Reported Outcomes Measures(PROMs): Investing in better measurement, tracking and feedback

of the patient outcomes across the system, with a view to achieving better patient follow–up and PCC.

4) Real time patient feedback: Investing in tools to measure the experience of the patient immediately after, or during treatment, providing a more realistic gauge of patient satisfaction, as well as prompt feedback loops to clinicians and managers to address any issues.

Given this policy background, commissioned by the Agency of Clinical Innovation (ACI), a scoping review was conducted to provide an analysis of the issues regarding applying PROMs/PREMs on a large scale, with a particular focus on PCC and IC.

1.2 MethodologyGiven the extreme wide range of topics covered within this scoping review, a few strategies were adopted to provide a quick scan on the literature. Structured searches were conducted of PubMed, PsychInfo, Web of Science and Scopus. A grey literature review was also conducted through Google and Google Scholar. Literature searches were also performed on related web sites including, but not limited to, the International Foundation of Integrated Care, Kings’ Fund, the AHRQ, New South Wales Ministry of Health, Bureau of Health Information (BHI) NSW, National Health Performance Agency, National Health Service (NHS) Choice, The Royal Australian College of General Practitioners (RACGP, Australia), Primary Health Care Research and Information Services (PHCRIS), etc. We initiatively focused on reviews in the area of IC, PROMs and PREMs. We then extended the search to original articles and editorials. Scopus’s top-down and bottom-up features were used to snowball the searching of key references. Overall, we scanned over 8,000 references and over 3,000 full-text documents were downloaded.

After a brief introduction of the policy background, we discussed the definitions and conceptual issues related to PROMs/PREMs, general review of issues related to selecting and capturing PROMs/PREMs was provided, as well as applying PROMs/PREM in different health care settings. Despite that the international and local experience on both PROM/PREM were described, we focused on reviewing the most prominent measurement instruments used in the IC setting, the methodological issues and the general considerations in applying the PROM and PREM in an Australian context. Finally, we outlined several future policy and research directions.

2 | DefinitionsPatient Reported Outcome (PROs), Patient Reported Outcome Measures

(PROMs), Patient Reported Experience Measures (PREMs) and Patient Reported Outcome Based Performance Measures (PRO-PM)


2.1 Patient reported outcomes (PROs)Patient reported outcome (PROs) measures include health status assessments, HRQoL, symptom reporting measures, satisfaction with care, treatment satisfaction measures, economic impact measures, and instruments for assessing specific dimensions of the patient experience such as depression and anxiety1. The USA Food and Drug Agency (FDA) adopts a much broader definition2: “A PRO is any report coming directly from patients about a health condition and its treatment”, meaning that PROs capture patients’ perspectives about how illness or new therapies impact on, for example, their general well-being.

The concept of PROs refers to any report on the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else. The National Quality Forum (NQF) of the USA has interpreted patient-reported outcomes (PROs) as an international term of art; the word “patient” is intended to be inclusive of all persons, including patients, families, caregivers and consumers. Also it is intended to cover all persons receiving support services, such as those with disabilities.

2.2 The narrow and broad definitions of PROs

It should be noted that most of the relatively old literature or the literature of a European origin tend to provide a narrow definition of PROs. That is, PROs was defined as only including HRQoL or quality of life (QoL) if defined differently from HRQoL, symptoms and side effects. Such a narrow definition did not include the patients’ reported experience, the satisfaction and expectation regarding their care as well as any compliments or complaints they may have. All these concepts are often included in the term “patient reported experience measures” in most of the literature. However, recently the Institute of Medicine (IOM), AHRQ and the NQF in the USA adopted a much more liberal definition of PROs with its key PRO domains including but limited to3:

z HRQoL (including functional status);

z Symptoms and symptom burden (e.g. pain, fatigue);

z Experience with care and satisfaction (Figure 2.2-1); and

z Health behaviours (e.g. smoking, diet, exercise).

Figure 2.2-1 Types of PROs currently used in medical research.

QoL

HRQL

Symptoms(Impairment)

Utility

PatientSatisfaction

Activitylimitations

(Disability)

15 | Integrated Care

The most commonly used PROMs assess symptoms and/or functional limitations. These are commonly referred to as HRQoL measures. The commonly used measures which generate utility values also ask about symptoms and/or functional limitations. Patient experience and satisfaction is generally concerned with issues such as the process of treatment and relationships with the clinical staff. QoL measures address need fulfilment rather than symptoms and/or functional limitations.4

2.3 Patient reported outcome measures (PROMs)

PROMs are the standard tools for directly eliciting the PROs and their use has become the standard both in regulated and non-regulated clinical trials and other quality improvement initiatives, particularly for assessment of symptoms and HRQoL. Systematic collection of PROs data has been shown to be feasible and efficient, to be more reflective of underlying health status than clinician reporting, to predict meaningful clinical outcomes including survival, to increase patient satisfaction with care, to be valued by clinicians for documentation and clinical decision making and to improve symptom management as well as the patient’s overall health status.

2.4 Patient reported outcome measure (PROMs) vs patient reported experience measures (PREMs)

With the very broad definition by the FDA, some will include the PREMs as part of the PROMs, such as the positions adopted by the NQF and AHRQ.3 However, given the long history of measuring patient experience/satisfaction in many countries, some would normally make a differentiation between the term PROMs (which specifically refers to HRQoL measures, patient symptoms and side effects) with the term PREMs (which refers to all patient care experiences, values, preferences, satisfaction, expectations, etc).5 6 In this report, we will describe both forms of patients’ reports; differentiation will be made in order to facilitate our discussion.

2.5 Why PROMs/PREMs are important to measure?

2.5.1 From a theoretical and conceptual point of view

Patient-based

evidence

High qualitypatient

care

Clinicalevidence

Economicevidence

Figure 2.5-1 Patient-based evidence as part of the whole new evidence-base for high quality patient care

The growing recognition of the importance of measuring PROM/PREM is, in part, inspired by the inclusion of PCC as one of the five key components of quality of care by the IOM and, more recently, by Trip Aims promoted by the Institute of Healthcare Improvement (IHI) which posits that the three goals of care are to improve health outcomes (including HRQoL), patient experience and reduce waste (or cost). In 2015, there will be, for the first time, a joint conference of international evidence-based medicine and international patient shared decision-making to be held in Sydney to emphasise the importance of the patient-centeredness, PROM/PREM, value and preference (so-called patient-based evidence) to shape the whole evidence-base of health care decision-making in combination with the clinical and economic evidence ( Figure 2.5-1 ) .


2.5.2 From the clinical and health economic point of viewThere are several reasons why PROM/PREMs are important from the clinical and health economic point of view:

1. Any experienced provider knows that traditionally measured biomarkers often fail to correspond with how a patient is actually feeling. For example, in diabetes clinicians usually measure the haemoglobin A1C level. This value is often used to make treatment decisions, like how aggressively to treat the diabetes, or whether to change to a new medication. But the problem is that some patients may have a low haemoglobin A1C but still feel listless or depressed despite their favourable laboratory values. In contrast, others with unfavourable levels may, nonetheless, feel upbeat and vigorous. Thus, the traditional outcome measured by healthcare providers (e.g. haemoglobin A1C levels,) may fail to capture other aspects of health.

2. Patients rarely value traditional biomarkers in the same manner as providers. For example, patients with hypertension often fail to share the same enthusiasm as their providers in achieving specific blood pressure goals but are quick to comply with therapy when their hypertension leads to headaches or dizziness. Measuring HRQoL, in particular, directly acknowledges that patients often value different outcomes than their providers.

3. PROMs provide a key component to understanding the true burden of any disease. Traditional measures of the disease burden include the prevalence of a disease, direct and indirect expenditures of a disease, and the worker productivity decrements related to a disease. However, in order to fully appreciate the true burden of a disease, it is also important to appreciate the HRQoL decrement engendered by the disease. The notion of “weighting” diseases not only by their cost and prevalence but also by their HRQoL decrement has an innate sense of fairness and is a fundamental principle of health economics7(Table 2.5-1 ). For this reason alone it is critical to carefully understand the HRQoL decrement of various diseases because that information may have policy implications when it comes to developing a healthcare budget.

4. PROM/PREMs are especially important in diseases that are marked by morbidity but not necessarily by mortality. PROM/PREMs have large clinical relevance in patients with disorders such as chronic migraine headache, sleep disorders or depression.

Table 2.5-1 The types and description of common health economic analyses

Type of analysis Description

Cost–offset study Cost analysis: Compares costs incurred with (other) costs saved; does not consider alternative use of resources elsewhere

Cost-minimisation analysis

As cost analysis but compares two or more interventions or programmes; assumes outcomes of different programmes to be broadly equivalent

Cost–consequence analysis (CCA)

Compares the costs and consequences of two or more alternatives, but does not aggregate or synthesise costs and consequences, and all health outcomes are left in natural units

Cost–effectiveness analysis (CEA)

Relates costs to a (typically single) common outcome between alternative interventions/programmes (which can also involve no intervention)

Cost–utility analysis (CUA)

Relates costs to utilities as a measure of programme effect;

results of CUA are typically expressed in terms of cost per

health year of cost per quality adjusted life year (QALY) gained

Cost–benefit analysis (CBA)

Economic evaluation that values all costs and benefits in the same (monetary) value; results of CBA are typically expressed as a ratio of costs to benefits or a sum representing the net benefit (or loss) of one programme over another

Some of the rationales put forward for measuring PROs in a clinical perspective include:

z better communication and shared decision making by patients and providers;

z assessing the health status of patients entering therapy and identifying treatable problems;

z determining the degree and sources of the patient’s decreased ability to function;


z distinguishing among different types of problems including physical, emotional, and social;

z detecting adverse effects of therapy;

z monitoring the effects of disease progression and response to therapy;

z informing decisions about changing treatment plans; and

z predicting the course of a disease and the outcomes of care.

2.5.3 From quality improvement and a societal point of view Patient and family engagement is increasingly acknowledged as a key component of a comprehensive strategy (along with performance improvement and accountability) to achieve a high quality and affordable health system. Emerging evidence affirms that patients who are engaged in their care tend to experience better outcomes and choose less costly but effective interventions if they have participated in a process of shared decision-making, such as physical therapy for low back pain. Promising approaches to involve patients and their families at multiple levels are being implemented across many countries. Such activities include consumers serving on governance boards at hospitals and contributing to system and practice redesign to make care safer and more patient-centric. There are growing interests from many different organisations to engage patients by building capacity and infrastructure to capture PROMs/PREMs routinely and then to use these data to develop performance measures to allow for accurate appraisals of quality and efficiency.3 8-16

There is also growing recognition and vision that PROM/PREM, in combination with other important data sources such as clinical data, genetic and biobank data, registry data, and administrative data within and beyond the health sector, can form the individual digital footprint that will provide better clinical management of patients as well as conducting improved comparative effective research.

2.6 Framing PRO in a broader determinants of health model

The broad definition of PRO, adopted by the top organisations in the USA, extends the PRO into a much more broad care model including preventive care (population at-risk), acute care (evaluation and initial management of patients) and sub-acute care (follow-up care)12. It is believed that PROs at each stage (such as health-related behaviour at the preventive care stage) are all important and so is the experience with care12(Figure 2.6-1).

Figure 2.6-1 Framing PROs within existing conceptual models

Genetics &biometrics

Clinical episode begins

Time

Determinants of health model

Physicalenvironment

Socialenvironment

Lifestyle &health behaviours

Populationat risk

Patient focused episode of care model

PHASE 1

Evaluation & initial

management

PHASE 2

PRO categories across the episode� HRQOL/functionalstatus� Health-relatedbehaviours� Sympton/symptonburden� Experience with care

Follow upcare

PHASE 3

2.7 The pathways from PRO to PRO-PM Another important trend is to use PRO to affect change and improve quality of care. In the USA, PRO has been extended beyond the purposes of its original clinical implication such as improving communication between patients and providers as well as monitoring the treatment effects of different therapies to be used actively in monitoring the provider and health system. In 2013, the NQF, the major US organisation that reviews and endorses quality metrics, assembled an expert panel to develop standards around the development of PRO-PMs. It aims to use PROs to affect changes and improve quality of care at all three levels: patients, providers and health systems. As a result, the white paper that described a pathway for developing such measures toward NQF endorsement3, was endorsed by the International Society for Quality of Life Research.


Two examples of the distinction of the purpose of the three concepts (i.e. PRO, PROM, and PRO-PM) were presented (Table 2.7-1).

Table 2.7-1 Distinctions among PRO, PROM, and PRO-PM: Two Examples

Concept Patients with clinical depression

Persons with intellectual or developmental disabilities

PRO (patient-reported outcome

Symptom: depression

Functional status-role: employment

PROM (instrument, tool, single-item measure)

PHQ-9, a standardised tool to assess depression

Single-item measure on National Core Indicators Consumer Survey: Do you have a job in the community?

PRO-PM (PRO-based performance measure)

Percentage of patients with diagnosis of major depression or dysthymia and initial PHQ-9 score >9 with a follow up score <5 at 6 months (NQF #0711)

The proportion of people with intellectual or developmental disabilities who have a job in the community

(Source: PROs in Performance Measurement (NQF, 2013)3)

In order to achieve such a conceptual change, the NQF developed specific guidelines on the pathways from PRO to PRO-PM (Figure 2.7-1) which required organisations to follow the explicit published endorsement criteria and seeks endorsement from the NQF. (Table 2.7-2).


Figure 2.7-1 Pathway from PRO to NQF-endorsed PRO-PM

1 Identify the quality performance issue or problem z Include input from all stakeholders including consumers and patients

4 Identify existing PROMs for measuring the outcome (PRO) in the target population of interest z Many PROMs (instrument/scale/single-item) were developed and tested primarily for research

2 Identify outcomes that are meaningful to the target population and are amenable to change z Ask persons who are receiving the care and services

z Identify evidence that the outcome responds to the intervention

5 Select a PROM suitable for use in performance measurement z Identify reliability, validity, responsiveness, feasibility in the target population

(see characteristics in Appendix C)

7 Specify the outcome performance measure (PRO-PM) z Aggregate PROM data such as average change; percentage improved or meeting a benchmark

9 Submit the PRO-PM to NQF for consideration of NQF endorsement z Detailed specifi cations and required information and data to demonstrate meeting NQF endorsement criteria

11 Use the endorsed PRO-PM for accountability and improvement z Refi ne measure as needed

12 Evaluate whether the PRO-PM continues to meet NQF criteria to maintain endorsement z Submit updated information to demonstrate meeting all criteria including updated evidence, performance and testing;

feedback on use, improvement and unintended adverse consequences

3 Determine whether patient-/person-reported information is the best way to assess the outcome of interest z If a PRO is appropriate, proceed to step 4

6 Use the PROM in the real world with the intwended target population and setting to: z Assess status or response to intervention, provide feedback for self-management, plan and manage care or

services, share decision-making

z Test feasibility of use and collect PROM data to develop and test an outcome performance measure

8 Test the PRO-PM for reliability, validity and threats to validity z Analysis of threats to validity, e.g., measure exclusions; missing data or poor response rate; case mix diff erences and risk

adjustment; discrimination of performance; equivalence of results if multiple PROMs specifi ed

10 Evaluate the PRO-PM against the NQF endorsement criteria z Importance to Measure and Report (including evidence of value to patient/person and amenable to change)

z Scientifi c Acceptability to Measure Properties (reliability and validity of PROM and PRO-PM; threats to validity)

z Feasibility

z Usability and Use

z Comparison to Related and Competing Measures to harmonise across existing measures or select the best measure

PRO

PRO

MPR

O-P

MN

QF

Endo

rsem

ent P

roce

ss




Table 2.7-2 NQF Endorsement Criteria and their Application to PRO-PMs

Abbreviated NQF Endorsement Criteria

Considerations for Evaluating PRO-PMs that are relevant to other performance measures

Unique Considerations for Evaluating PRO-PMs

Importance to Measure and Report

a. High impact

b. Opportunity for improvement

c. Health outcome OR evidence-based process or structure of care

• Does evidence support that the outcome is responsive to intervention?

• When should the evidence exception be allowed for performance measures focused solely on conducting an assessment (e.g., administering a PROM, lab test)?

• Patients/persons must be involved in identifying PROs for performance measurement (person-centered; meaningful).

Scientific Acceptability of Measure Properties

a. Reliability

1. precise specifications

2. reliability testing for either data elements or performance measure score

b. Validity

1. specifications consistent with evidence

2. validity testing for either data elements or performance measure score

3. exclusions

4. risk adjustment

5. identify differences in performance

6. comparability of multiple data sources

• Data collection instruments (tools) should be identified (e.g., specific PROM instrument, scale, or single item).

• If multiple data sources (i.e., PROMs, methods, modes, languages) are used, then comparability or equivalency of performance scores should be demonstrated.

• Specifications should include standard methods, modes, languages of administration; whether (and how) proxy responses are allowed; standard sampling procedures; how missing data are handled; and calculation of response rates to be reported with the performance measure results.

• Reliability and validity should be demonstrated for both the data (PROM) and the PRO-PM performance measure score.

• Response rates can affect validity and should be addressed in testing.

• Differences in individuals’ PROM values related to PROM instruments or methods, modes, and languages of administration need to be analysed and potentially included in risk adjustment.


3 | Selecting PROMs and PREMs


3.1 Where to find PROM/PREM?There are many sources that could be accessed to identify existing tools in PROM/PREM. The PROMIS website provides many useful links (http://www.nihpromis.org/resources /resourcehome) .

3.1.1 Online libraries and databases of PROs

A selection of online libraries and databases of PROs are as follows:

z PROQOLID(Patient-Reported Outcome and Quality of Life Instruments Database): MAPI Research Trust site: http://www.mapi-trust.org/. This database requires a subscription fee and currently includes over 900 different instruments of PROs.

z PROMs group, Oxford (http://phi.uhce.ox.ac.uk/about.php): There is an online bibliographic database accessible through the website. However, it was only updated to December 2005 and no further update is planned.

z BiblioPRO (http://www.bibliopro.org): the only virtual library of tools in Spanish for HRQoL and other PRO. The new website which started in 2011 currently includes more than 600 instruments in Spanish of which 25%, including SF‐12 and Family SF, may be downloaded directly.

z National Quality Measures Clearinghouse (NQMC), AHRQ (http://www.qualitymeasures.ahrq.gov/hhs/index.aspx) : Despite this online source not being specifically designed for PROM, it does have many potential useful measures, in particular measuring patient reported experiences as it is a clearly defined domain in the AHRQ quality matrix.

z GEM (GRID-enabled Measures) Databases: (http://cancercontrol.cancer.gov/brp/gem.html) :

GEM, currently hosted by the National Cancer Institute of NIH, is an interactive website containing behavioural, social science and other scientific measures organised by theoretical constructs. GEM enables researchers to collaborate with others, encourages the use of common measures and facilitates the sharing of harmonised data:

1) Users contribute to the virtual community by adding or editing meta-data about constructs and measures.

2) Users rate and comment to drive consensus on best measures.

3) Users search for constructs (e.g. anxiety, depression), see definitions, view theoretical foundations and download associated measures.

4) Users search for measures and see attributes (e.g. definition, associated construct, target population, author, reliability, validity).

5) Users download and share datasets using GEM measures and constructs.

The goals of GEM are to enable users to collaborate with their peers to build consensus on the use of common measures and to facilitate broad-scale data sharing and harmonisation. GEM allows users to interact with each other in an online environment. Based upon a wiki platform, users contribute to the website by adding and editing information about measures and associated constructs and by providing feedback and ratings on the measures and constructs in GEM.

GEM database currently includes 892 measures and 389 constructs and access to the information is free for registered users.

3.1.2 Systematic review and search of online bibliographic databases

A common approach to find the most relevant PRO tools is to conduct a systematic review from relevant bibliographic databases such as Medline, PsychInfo or Cochrane Online Library. The website of COSMIN currently maintains an updated list (up to June, 2014) of published systematic reviews on the HR-PRO and the list is included in Appendix 1.

3.2 How to choose a PROM/PREM – methodological issues

3.2.1 Data source, mode and method

For choosing any specific PROM/PREM, a few factors should be considered in combination with issues related to its measurement properties. These factors include the data source (self-report vs proxy), mode of administration (self-administration vs interview), setting of administration (clinic, home, other) and scoring (classical test theory vs modern test theory).

These three aspects can also be combined in various ways, e.g. a patient might use the telephone to self-administer a PRO instrument or an interviewer might use a computer to read questions and record answers17 ( Figure 3.2-1).

The main characteristics together with the strengths and weaknesses for each of these methods are summarised. The strengths and weaknesses should be carefully weighted before and after a particular instrument was chosen (Table 3.2-1).17


Figure 3.2-1 Types of respondent data and methods/modes of administration

DATA SOURCE

MODE

METHOD

Self-report vs proxy/observer

Self-administration - paper-and-pencil - telephone - computer

Interviewer-administration - paper-and-pencil - telephone - computer(Source: Cella et al 201217)


Table 3.2-1 Main characteristics of PRO methods issues

Methodological Issue Main Characteristics Strengths Limitations

Source of report

Self z Individual responds about him/herself

z Expert on own experience

z Not always possible to assess directly e.g. because of cognitive or communication deficits or age/developmental level

Proxy z Individual responds about someone else

z Useful when target of assessment unable to respond

z Can provide complementary information

z May not accurately represent subjective or other experiences

Mode of administration

Self z Individual self-administers PRO and records the responses

z Cost-effective

z May yield more participant disclosure

z Proceed at one’s own pace

z Potential for missing data

z Simple survey design (e.g. minimal skip patterns)

Interviewer z Interviewer reads questions out loud and records the responses

z More complex survey design (e.g. skip patterns)

z Useful for respondents with reading, writing or vision difficulties

z Interviewer costs

z Potential for bias (interviewer bias, social desirability bias, acquiescent response sets)

Method of administration

Paper-and-pencil z Patients self-administer PRO using a paper and writing utensil

z Cost-effective z Prone to data entry errors

z Data entry, scoring requires more time

z Less amenable to incorporation within EHR


Methodological Issue Main Characteristics Strengths Limitations

Electronic z Patient self-administers PRO using computer- or telephone-based platform

z Interactive

z Practical

z Increased comfort for socially undesirable behaviours

z Minimises data entry errors

z Immediate scoring, feedback

z Amenable to incorporation within EHR

z Cost

z Potential discomfort with technology

z Accessibility

z Measurement equivalence

Setting of administration

Clinic z Patients complete PROs when they arrive to clinic appointments

z Real-time assessment of outcomes

z Feasibility with use of electronic methods of administration

z Impact on clinic flow

z Interruptions resulting in missing data

z Patient anxiety

z Staff burden

Home z Patients complete PROs at home prior to, or in between clinic visits

z Minimises impact on clinic flow

z Minimises staff burden

z Accessibility

z Health information privacy

z Data security

z Patient safety

Other z Patients complete PROs at other types of settings (e.g. skilled nursing, rehabilitation)

z Feasibility with electronic methods of administration

z •Cognitive capacity and potential need for proxy

Scoring

Classical test theory z Raw scores z Easy to implement and understand

z All items must be administered

Modern test theory z Probabilistic approach z Enables CAT (tailored questions)

z Shorter questionnaires with more precision

z Difficult to implement and understand


3.2.2 Other confounding factors of choosing a PROM/PREM There are a number of factors that may impede or complicate the decision of choosing a particular PROM/PREM. These factors include:

z vulnerable populations (very young, old, fragile)

z low literacy

z language and culture (Indigenous people or people from a culturally or linguistically diverse background)

z functional abilities (both cognitive and physical)

3.2.3 Implications of different methods and modes on response rate, reliability and validityData collection methods

Different methods differ along a variety of dimensions. This includes the degree of interviewer involvement and the level of interaction with the respondent. Channels of communication (sight, sound, touch) used can be critical; various combinations may prompt different issues of comprehension, memory stimulation, social influence affecting judgment, and response hurdles. Finally, the degree of technology used is a major consideration.

Using a different method or mode than originally validated

Considering the implications of using a different method or mode than the one on which the PROM/PREM was originally validated is also important. Many existing PROMs/PREMs were initially validated in paper-and-pencil form. However, potential differences exist between paper-and-pencil and electronic-based PROM/PREM administration, ranging from differences in how items and responses are presented (e.g. items presented one at a time, size of text) to differences in participant comfort level in responding (e.g. ability to interact with electronic–based platform).

Implications of using multiple methods and modes

The implications of using multiple methods and modes also warrant consideration. One might choose to blend methods for one or more reasons: cost reduction, faster data collection, and optimisation of response rates. When combining methods or modes (or both), users must ensure that they can disentangle any effects of the method or mode from other population characteristics. This is especially true when respondents choose which method or mode they prefer or when access issues determine the choice of method or mode.

Accounting for the impact of non-responders

Difficulties with data collection and questionnaire completion are major barriers to the successful implementation of PROM/PREM. Missing data may be classified as either item non-response (one or more missing items within a questionnaire) or unit non-response (the whole questionnaire is missing for a patient). Evaluating the amount, reasons and patterns of missing data is important.

3.2.4 Minimally important differences and changesHistorically, research has relied upon tests of statistical significance to examine differences in PROM/PREM scores between patients or within patients over time. Attention has shifted to the concept of clinically significant differences in PROM/PREM scores.

z Minimally important differences (MIDs) represent a specific approach to clinical significance and are defined as “…the smallest difference in score in the outcome of interest that informed patients or informed proxies perceive as important.”18

z Minimum clinically important differences (MCIDs) comprise an even more specific category of MID and are defined as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management.”19

The examination of clinically significant differences carries a number of important implications.18

z aids in the interpretation of PROMs/PREMs

z emphasises the importance of the patient perspective

z informs the evaluation of the success of a clinical intervention

z assists with sample size estimation.

Currently, no methodological “gold standard” exists for estimating MIDs; 20 21 however, two primary methods are currently in-use:

1) the anchor-based method and

2) the distribution-based method.

The anchor-based method of establishing MIDs assesses the relationship between scores on the PROM/PREM and an independent measure which is interpretable18, including:

z clinical anchors which are correlated with the PROM/PREM at the r ≥ 0.30 level, including clinical trial experience22


z transition ratings which are within-person global ratings of change made by a patient.22 23

However, due to concerns about validity, it is recommended that researchers examine the correlation between pre-and post-test PROM/PREM scores and the transition rating.24 Between-person differences made by patients can also be used as anchors when establishing MIDs for PROM/PREM measures.

Additional sources such as HRQoL-related functional measures 22 23 and objective standards (e.g. hospital admissions, time away from work24) can also be used as anchors. Several limitations should be considered. Firstly, the transition rating approach to anchor selection is subject to recall bias on the part of the patient.20 Secondly, global ratings may only account for some variance in PROM/PREM scores.20 Thirdly, the anchor based method does not take into consideration measurement precision of the instrument.20

The distribution-based method represents the second method of establishing MIDs in PROM/PREM. The distribution-based method uses the statistical characteristics of the PROM scores when establishing MIDs.25

The distribution-based approach evaluates change in scores in relation to the probability that the change occurred at random.20 Despite several methods available when applying a distribution-based approach to MID establishment, there is little consensus on the benchmarks for establishing changes that are clinically significant.20

Important strategies in choosing MID include:

z multiple methods and triangulation should be used to determine the MID20 26 given limitations of the anchor- and distribution-based approaches

z the final selection of MID values should be based on systematic review and an evaluation process such as the Delphi method26

z MID values should also be informed by a stakeholder consensus, which includes patient engagement and input, regarding the extent of change considered to be meaningful

z when considering MIDs for PROM/PREM, a single MID should only be applied to a situation involving that particular PROM/PREM given that MID varies by population/context26

z the distribution around the MID be provided rather than just a single MID value27

z calculate the proportion of patients who experience a clinically significant change because the criteria for assessing clinically important change in individuals

does not directly translate to evaluating clinically important group differences.22

3.2.5 Response shift, adaptation and other challenges to detect true change The ability to detect true change over time in PROM/PREM poses another barrier to the integrity of PROM/PREM assessment. Often detecting true change is associated with the phenomenon of response shift which has been defined as:

“a change in the meaning of one’s self-evaluation of a target construct as a result of: (a) a change in the respondent’s internal standards of measurement (i.e. scale recalibration); (b) a change in the respondent’s values (i.e. the importance of component domains constituting the target construct) or (c) a redefinition of the target construct (i.e. reconceptualisation)” (p.1532).28

A change in perspective over time may result in patients’ attending to PROMs/PREMs in a systematically different way from one time point to another.29

Response shift serves as a barrier to PROM/PREM assessment:

z it threatens longitudinal PROM/PREM assessment validity, reliability and responsiveness29

z response shift can complicate the interpretation of PROM/PREM, since a change in PROM/PREM outcome may occur because of response shift, an effect of treatment or both.30

Monitoring for response shift can aid PROM/PREM users in interpreting longitudinal PROM/PREM data.31 Several strategies have been proposed to identify response shift, although each has limitations:

1) The “then test” compares an actual pre-test rating and a retrospective pre-test rating to assess for shift but it is less robust than other methods of detecting response shift29 and is confounded with recall bias.28

2) Structural equation modelling has also been proposed as a way to identify response shift; however, it is sensitive only if most of the sample is likely to make response shifts.32

3) Growth modelling creates a predictive growth curve model to investigate patterns in discrepancies between expected and observed scores, thus assessing response shift at the individual level.33 Although growth modelling enables users to detect both the timing and shape of the response shift31, it cannot differentiate between random error and response shift.28


3.2.6 Generic versus disease- or condition-specific PROM One primary factor to consider when selecting a patient-level PROM is whether to use a generic versus a condition-specific PROM. Several elements inform the selection of measures.34 35

1) Targeted population: The specific population of interest may indicate whether one opts to use a generic or condition-specific PRO. For example, if the target population comprises of mainly healthy individuals, a generic measure may be the preferred choice. Conversely, examining a specific subset of patients with a particular health concern, a condition-specific measure may be more appropriate.

2) Outcomes of interest: Generic measures may capture a different category of outcomes when compared to a condition-specific PRO. For example, a generic measure may assess the domains of general QoL, whereas a condition-specific PRO may measure symptoms expected to be directly addressed by a condition-specific intervention.

3) The assessment purpose: For example, for labelling purposes one should follow the FDA guidance which states that pharmaceutical company claims of improved QoL must be specific to the QoL domain that was measured; the agency recommends that assessment of specific symptoms is an appropriate starting point.

The strengths and weaknesses of both generic and condition-specific measures can be summarised 34 35 (Table 3.2-2).

Table 3.2-2 The strengths and weaknesses of generic and condition specific PROMs

PROMs Strength Weakness

Generic z allow for comparability across patients and populations (more suitable for comparison across groups than for individual use)

z allow assessments in terms of normative data which can be used to interpret scores

z enable evaluation against population norms or comparison with information about various disease conditions

z can be applied to individuals without specific health conditions

z can differentiate groups on indexes of overall health and well-being

z less sensitive to change than condition-specific measures

z may fail to capture important condition-specific constructs

Condition- or disease- specific

z greater sensitivity to change because they focus on the concerns pertinent to the given condition

z enable differentiation of groups at the level of specific symptoms or patient concerns

z introduces the notable difficulty of making comparisons across patient populations with different diseases or health conditions


In general, a combination of generic and condition-specific measures is likely to be the best choice for performance measurement purposes. Generic and condition-specific PRO measures may measure different aspects of HRQoL when administered in combination, resulting in a more comprehensive assessment. Consequently, hybrid measurement systems such as the Functional Assessment of Chronic Illness Therapy (FACIT) system (www.facit.org) and PROMIS, were developed to create item banks that are appropriate for use across common chronic disease conditions as well as for specific conditions, represents both global and targeted approaches.

3.2.7 Other methodological issues related to choosing PROM/PREMsThere are many other complex conceptual and methodological issues related to the measurement of PROMs/PREMs that are presented in Appendix 2.

3.3 Using PROM/PREM as quality measures: the desirable attributes

When using PROM/PREM as quality measures in order to affect change, PROM/PREM could also be assessed according to the AHRQ framework of desirable attributes of quality measures in the three broad conceptual areas: (1) importance of a measure, (2) scientific soundness of a measure, and (3) feasibility of a measure. AHRQ’s criteria for users to judge a desirable quality measure are paraphrased below (more details can be found in NQMC’s Template of Measures Attributes (http://www.qualitymeasures.ahrq.gov/about/template-of-attributes.aspx) .

1) Importance of the measure

z Relevance to stakeholders - the topic area of the measure is of significant interest and is financially and strategically important to stakeholders (e.g. patients, clinicians, purchasers, public health officers, policy makers).

z Health importance - the aspect of health that the measure addresses is important as defined by high prevalence or incidence and/or a significant effect on the burden of illness (i.e. effect on the mortality and morbidity of a population).

z Applicability to measuring the equitable distribution of health care (for health care delivery measures) or of health (for population health measures) - the measure can be stratified or analysed by subgroups to examine whether disparities in care or of health exist amongst a diverse population of patients.

z Potential for improvement - there is evidence indicating a need for the measure because there is overall poor quality or variations in quality among organisations (for health care delivery measures) or overall poor quality of health or variations in quality of health among populations (for population health measures).

z Susceptibility to being influenced by the health care system - for health care delivery measures, the results of the measure relate to actions or interventions that are under the control of those providers whose performance is being measured so that it is possible for them to improve that performance. For public health measures, the results should be susceptible to influence by the public health system.

2) Scientific soundness: clinical logic

z Explicitness of evidence - the evidence supporting the measure is explicitly stated.

z Strength of evidence - the topic area of the measure is strongly supported by the evidence, i.e. indicated to be of great importance for improving quality of care (for health care delivery measures) or improving health (for population health measures).

3) Scientific soundness: measure properties

z Reliability - the results of the measure are reproducible for a fixed set of conditions irrespective of who makes the measurement or when it is made; reliability testing is documented.

z Validity - the measure truly measures what it purports to measure; validity testing is documented. See tutorial on measure validity.

z Allowance for patient/consumer factors as required - the measure allows for stratification or case-mix adjustment if appropriate.

z Comprehensible - the results of the measure are understandable for the user who will be acting on the data.

4) Feasibility

z Explicit specification of numerator and denominator - a measure should usually have explicit and detailed specifications for the numerator and denominator; statements of the requirements for data collection are understandable and implementable. Some measures that do not have explicit and detailed specifications for the numerator and denominator (e.g. measures that have counts or means) can be feasible for quality improvement purposes when used with a specified baseline, benchmark and/or target.


z Data availability - the data source needed to implement the measure is available and accessible within the timeframe for measurement. The costs of abstracting and collecting data are justified by the potential for improvement in care or health.

3.4 Other principles of selecting PROMs/PREMs in ICIndicators can be more or less robust and meaningful depending on their characteristics and whether they meet certain criteria. Generic criteria that indicators should meet if they are to be useful include36:

z importance and relevance

z validity

z accuracy

z reliability

z feasibility

z meaningfulness

z implications for action

z avoidance of perverse incentives.

Wider considerations could also inform the selection of indicators36, such as:

z size of the population covered

z representation of important aspects of the care system

z (wholly or partly within the control of care services i.e. attributability

z change detectable within suitable time frames

z unambiguous interpretation

z likelihood of being meaningful to users, carers and the public

z likelihood of being meaningful to care professionals, managers and commissioners

z reflecting the user perspective and/or value for money perspective

z timeliness

z ability to assess the impact on inequalities between user groups and areas in

z terms of access and outcomes of care

z measurable from routinely collected data.

4 | Capturing PROMs and PREMs


4.1 The challenges in capturing PROMs and the promises of PROMIS

Despite a growing interest in the integration of PROMs into clinical practice, efforts have been hampered by a number of challenges.

These include:

1. floor and ceiling effects that limit sensitivity to change;

2. lengthy questionnaires that increase patient burden;

3. a proliferation of measures of the same outcome limiting the ability of decision makers to compare results across studies;

4. some promising PROMs have not been validated specifically in the clinical population under study;

5. a scarcity of evidence regarding the validity of PROMs, despite the FDA urging that special attention be paid to this in its guidelines on the use of PROMs for pharmaceutical labelling claims.2

Collectively, these challenges have limited the use of PRO as endpoints within clinical trials and clinical practice as well as inhibiting the adoption of key trial findings by practitioners. Due to lack of standardised instruments being validated in large heterogeneous populations, clinicians and policy makers believe that some instruments may not have decision making relevance (external validity) in clinical practice.

“The clinical outcomes research enterprise would be enhanced greatly by the availability of a psychometrically validated dynamic system to measure PROs efficiently in study participants with a wide range of chronic diseases and demographic characteristics.”

National Institute of Health, 2003

The PROMIS Network, a component of the NIH’s Re-engineering the Clinical Research Enterprise Program, seeks to overcome the limitations in existing PRO instruments by:

1. developing and testing large PRO item banks based on IRT covering a wide range of concepts and constructs such as pain, fatigue, physical functioning, emotional distress and social role participation that have a major impact on QoL across a variety of chronic diseases;

2. creating a CAT system for the assessment of PROs in clinical research; and

3. creating a publicly available and updatable system for accessing and using the item bank via the CAT system known as Assessment Centre (www.assessmentcenter.net).

This initiative applies to a wide range of disorders including cancer, congestive heart failure, depression, arthritis, and multiple sclerosis as well as chronic pain conditions. PROMIS is creating new paradigms for how clinical research information is collected, used and reported. The PROMIS initiative addresses a need in the clinical research community for a rigorously tested PROMs tool that utilises recent advances in information technology, psychometrics, qualitative research, cognitive research and health survey research.

PROMIS has many assessment options available to measure self-reported health for clinical research and practice. PROMIS assessment instruments are drawn primarily from calibrated item banks (sets of well-defined and validated items) measuring concepts such as pain, fatigue, physical function, depression, anxiety and social function. These calibrated item banks can be used to derive short forms (typically requiring 4-10 items per concept or computerised adaptive testing (CAT); typically requiring 4-7 items per concept for more precise measurement). Assessments are available for children and adults. Most PROMIS instruments are available through Assessment Centre. Those which are not yet available on Assessment Centre can be obtained by contacting the PROMIS statistical centre through [email protected]. The Assessment Centre can be utilised for online or offline computer-based administration or instruments can be downloaded for paper administration or entry into other data collection platforms. For registered users, all the instruments, documentation and necessary computer platforms are free at the writing of this report.

The instruments from PROMIS are available in the form of item banks, short forms and profiles. Item banks are calibrated items from which a summary score can be obtained from a subset of items (i.e. via CAT or short form) whereas scales are calibrated items from which a summary score should be obtained only from the complete set of items. Item pools are collections of related items that are not intended to produce a summary score but instead are to be used as single items. Short forms are static subsets of item banks and profiles are fixed collections of short forms measuring multiple concepts.


Data collection formats for Patient Reported Outcome

ConventionalItem response theory

Static paper formStatic paper formCustomised or pre-speci�edshort form

Static electronic formMay include simple branching, but no dynamic item selection

Static electronic formCustomised or pre-speci�edshort form

Dynamic electronic formFull computer adaptive itemselection: the next item isselected based on previousresponse

During the first phase of the initiative (2004 to 2009), PROMIS formed a network of researchers that developed questions or “items” to analyse five outcomes or “domains”. PROMIS is creating a psychometrically-robust CAT system based on IRT to administer these items. In addition, it has developed a web-based system to give clinical researchers access to the item banks and the CAT system. Whether administered through an iterative CAT system that allows research flexibility or by paper version short forms, PROMIS has already demonstrated improved efficiency and sensitivity in comparison with existing PROMs. Long-term trials are planned to address the issues of validity and sensitivity to changes in clinical populations. The efficiency, flexibility and sensitivity of PROMIS has the potential to become a widely-accepted, standardised PRO measurement tool that will allow greater comparability of studies with a reduced burden on patients.

During the second phase (2009-2013) of the PROMIS initiative, it continued to advance the field of patient self-reporting in clinical research and practice, by:

z developing new items and domains

z translating current and future items and domains into other languages such as Spanish and Chinese to facilitate international studies

z conducting validation studies in large-scale clinical trials in a variety of clinical populations

z making PROMIS tools accessible to a wider range of clinical researchers and patient-care communities, and optimising its usability for rapid adoption

z providing on-going education and outreach to familiarise users with new developments in PROMIS

z improving PROMIS tools to allow for better outcomes in clinical trials and, potentially, better individual and clinical decisions

z engaging stakeholders at all levels, by including interactions with other health-related federal agencies, forging new relationships with patients and patient organisations and establishing public-private partnerships to sustain PROMIS once Roadmap funding ends.


Domain development/Early validation

PROMIS Network Steering Committee (PNSC)

PROMIS Network Executive Committee (PNEC)

PROMIS Statistical Centre (PSC)Northwestern UniversityDr David Colla

Stephen HaleyPROs in children& young adultswith disabilities

Darren DeWaltPaediatrics:longitudinal studylinking paediatricsand adult itembanks

Christopher ForrestPaediatric PROMIS:advancing the measurement &conceptualisationof child health

Lisa ShulmanDevelopment/validation ofself e�cacy itembank

Donald PatrickPatient reported outcomes inroutine clinical care of patientsinfected with HIV

Kevin WeinfurtSexual functioning

Dinnesh KhannaDevelopment/validation ofPROMIS GI distress

Esi Morgan DeWittEnhancing PROMISin paediatric pain

Paul PilkonisDevelopment/validation ofmental health/sleep wakefunction

James FriesImprovingassessment of PF/drug safety inhealth & disease

PROMIS Network Centre (PNC)American Institutes forResearch (AIR)Dr San Keller

PROMIS Technology Centre (PTC)Northwestern UniversityDr Richard Gorshon

NIH

SMB

External collaborators

PROMIS II NETWORK STRUCTURE 2009-2013

New domainArthur Stone

Domain & earlyclinical validationSee below

Large clinical validation siteArnold Potosky


Source: NIH Clinical Outcomes Assessment : PROMIS Overview ( http://commonfund.nih.gov/ clinicalresearch / overview-dynamicoutcomes.aspx ; accessed in Dec. 2011)

In an ambitious move, the PROMIS initiative aims to achieve four values which almost all previous attempts to develop PROs have failed to address:

1. comparability: the measures can be used and compared across different diseases, conditions and different populations as well as across life course

2. reliability (precision) and validity: being extensively tested against existing and legacy instruments such as SF-36 in different formats (i.e. short-form, profile, scale) under different study populations and conditions and across the score continuum of the concept; it is extremely helpful in understanding the responsiveness, floor and ceiling effects of the instruments

3. flexibility: it can be administrated through paper-pen, touch-screen, smart phone, personal digital assistant (PDA) and the web. It can also incorporate specific instruments developed or adopted by individual research. It is also linkable to the EHR and other databases

4. inclusiveness: items were written simply at elementary-school reading level and cognitive interviews, for all items, were conducted. Every item was pre-tested and then field tested in individuals with low literacy. Items have been translated into Spanish and 33 other countries also requested the translation. The PROMIS II will focus on children’s measures.

PROMIS integrates the �elds of...

Healthinformationtechnology

Psychometrics

Surveyresearch

Qualitativeresearch

Clinicalresearch

PROMIS

Source: NIH PROMIS: Advancing PRO Science in Clinical Research and Patient Care: January 11, 2011 presentation (http://www.nihpromis.org/whatsnew/whatsnewhome)

PROMIS has been used in successful NIH grant applications, producing over 150 journal articles, including cancer-related publications. PROMIS researchers have presented on the topic worldwide on hundreds of occasions in the last four years. Selected journal publications by the PROMIS network since 200737-186 are included in the references.

4.2 Generic HRQoL measures in primary care setting: results from an evidence review

One of the common challenges in applying PROMs in a primary health care or IC setting is how to choose a generic PROM that measures HRQoL given the multitude of existing measurement instruments. To compare the methodological quality of these instruments is an arduous task. To meet this challenge, the Canadian Institute of Health Research supported a group of researchers in 2013 to conduct a review to answer the question “ What are the most effective ways to measure patient health outcomes of primary health care integration through Patient Reported Outcome Measurement Survey instruments?”187

The project took six steps to complete:

1) Long-list of generic PROMs: A comprehensive long-list of all generic PROM instruments.

2) Short-list of generic PROMs: To include truly generic, quantitative measures designed for adult populations with high recent citation counts.

3) Descriptive overview of short-listed PROMs: To include official translations, respondent burden including required literacy/reading level, cost for using the instrument and dimension coverage.

4) Review of PROM instrument ‘performance’: Psychometric (e.g. reliability, validity, responsiveness and interpretability) and decision-making (e.g. norm reference sets, utility/preference scoring algorithm and evidence of clinically relevant thresholds)

5) Additional information: Examples of use in a primary and community care context and PROM-related activity in other jurisdictions.

6) Workshop and recommendations: To review the evidence and identify the ‘preferred’ instrument (or instruments) for use in British Columbia (BC) primary and community care reform.


The researchers generated a short-list of PROMs which included:

1) Assessment of Quality of Life (AQoL-8D)

2) EuroQol EQ-5D

3) HowsYourHealth

4) Health Utilities Index (HUI)

5) Nottingham Health Profile (NHP)

6) PROMIS/Global Health Status(GHS)

7) Short-Form 36 (SF-36) and SF-12

8) Quality of Wellbeing Scale (QWB-SA)

9) World Health Organisation Quality of Life Instrument (WHOQoL-BREF).

A formal review approach was adopted which included study selection, data extraction and quality assessment. Data synthesis involved a scoring approach proposed by the COSMIN-initiative for systematic reviews of the measurement properties of instruments. The researchers considered aspects of each instrument’s reliability, validity and responsiveness following strict guidelines on scoring in each of these categories and included two additional categories: generalisability and comparison with other PROM instruments.

Twenty-one of the 22 articles provided information about the psychometric properties of the candidate PROM instruments. An overview of the results from the psychometric review is provided (see Table 4.2-1 ). A general conclusion is that the SF-36 performed particularly well across most psychometric dimensions and PROMIS was also a strong instrument although the evidence base was smaller.

Table 4.2-1 Overview of results from psychometric review

AQoL EQ-5D SF-36 HUI NHP QWB-SA WHOQoL-BREF

PROMIS

Internal consistency + n/a +++ ? +/- n/a +/- +++

Reliability ? +/- +++ +/- +/- ? ? ?

Content

validity

- - ++ +/- +/- +/- ? +/-

Construct validity ? ? +++ ? - ? +++ ++

Cross-cultural validity ? + +/- + +/- ? +++ ?

Criterion

validity

? +/- ? ? ? ? ? ?

Responsiveness + -- +++ +/- - +/- ? ++


It is worth noting that although most of the instruments can provide utility scores, AQoL was developed in Australia and by Australian researchers where the utility scores were derived from an Australian sample. Likewise, all the instruments provide population norms, but only AQoL , SF-12/SF-36 supply Australian population norms.

The project culminated with a workshop, held at the British Colombia Ministry of Health and involving a wide range of stakeholders. The primary objectives for the whole workshop were:

z to share the details of the review and evidence synthesis work undertaken by the research team

z to reflect and deliberate on the PROM instruments and their potential use

z to come to a consensus on which PROM instrument(s) should be recommended for use in BC.

There were two rounds of voting and discussions during the workshop which occurred after the information was presented on the review and participants were given an opportunity to visually review the items within each measure. The final votes indicated a strong preference for two instruments, PROMIS (41 votes) and SF (36 votes) and a desire to keep EQ-5D (18 votes) as a back-up.

More information on the cost, sources and instrument details is presented in Appendix 3.

4.3 Measuring patient experience in a primary health care setting – an evidence review from Canada

Hosted by the Canadian Primary Health Care Research and Innovation Network (CPHCRIN), three important and related scoping reports on measurement in primary health care (PHC) were commissioned. These three separate but related technical reports were companions to the Canadian Institute for Health Information (CIHI) suite of PHC organisational, provider, and patient experience surveys. The patient experience report188 updates and builds on work that examined patient experiences in PHC and suggested six dimensions and 15 sub-dimensions that were deemed important in measuring patient experience in PHC (Table 4.3-1). The report was based on an extensive literature review, qualitative research and consultation with stakeholders.

The review188 included domains/subdomains that were not included in the current version of the NHS patient experience framework. The domains such as trust and confidence in the PHC system provided different insights from the patient perspective. In order to operationalise the domains, the authors conducted an extensive review of many related measurement instruments and retained 17 instruments that were widely used in the PHC setting. Of these 17 instruments and surveys, ten were created in a country other than Canada, one was an international initiative (Commonwealth Fund International Health Policy survey) and nine were administered only in Canada. Through the mapping of different instrument items into the conceptual framework presented (Table 4.31), a final 87-item survey questionnaire was developed. Given the large number of domains/subdomains included in the framework, the number of questions included is fairly large which reflects on the trade-off between the completeness/richness of the content and the complexity/multiplicity of the questions.


Table 4.3-1 Dimensions of patients’ experiences in primary health care188

Dimension Sub-dimension Definition

Access

First contact accessibility The ability to obtain patient-or client-initiated needed care (including advice and support) from the provider of choice within a time frame appropriate to the urgency of the problem.[40]

Accommodation The relationship between how resources are organised to accept patients or clients (including appointment systems, hours of operation, walk-in facilities, telephone services) and the patients’ or clients’ ability to accommodate to these factors to realise access.[40]

Economic accessibility The extent to which direct or indirect costs related to care impeded decisions to access needed care or continue recommended care.

Interpersonal communication

General communication The ability of the provider to elicit and understand patient or client concerns, and to explain health and health care issues.[40, 47]

Respectfulness The ability of the primary care organisation and practitioners to provide care that meets the expectations of users about how people should be treated, such as regard for dignity and provision of adequate privacy.[40, 47]

Shared decision-making The extent to which patients or clients are involved in making decisions about their treatment.[ 47]

Whole-person care The extent to which providers address the physical, emotional and social aspects of a patient’s or client’s health and consider the community context in their care.[40]

Continuity and coordination

Relational continuity A therapeutic relationship between a patient or client and one or more identified providers that spans separate health care episodes and delivers care that is consistent with the patient’s or client’s biopsychosocial needs.[40]

Information continuity The extent to which information is used to make current care appropriate to the patient or client.

Coordination The provision and organisation of a combination of health services and information with which to meet a patient’s or client’s health needs, including services available from other community health service providers.[9, 10]

Team functioning The ability of primary health care providers to work effectively as a collaborative team to manage and deliver quality patient or client care.


Dimension Sub-dimension Definition

Comprehensiveness of services

Services provided The provision, either directly or indirectly, of a full range of services to meet patients’ or clients’ health care needs. This includes health promotion, prevention, diagnosis and treatment of common conditions, referral to other clinicians, management of chronic conditions, rehabilitation, palliative care and, in some models, social services.[40]

Health promotion and primary prevention

Health promotion is the process of enabling people to increase control over, and to improve, their health.[12] Primary prevention is directed towards preventing the initial occurrence of a disorder.[13]

Trust

An expectation that the other person will behave in a way that is beneficial and that allows for risks to be taken based on this expectation. For example, patient or client trust in the physician provides a basis for taking the risk of sharing personal information.[48]

Patient-reported impacts of care

Patient activation Patient’s or client’s ability or readiness to engage in health behaviours that will maintain or improve their health status.[49, 50]

Patient safety Patient’s or client’s reports of medication errors (given or taken the wrong drug or dose) or incorrect medical or laboratory reports and communication with their provider about not taking their prescribed medication or medication side effects.

Confidence in the PHC system

The perception that allows patients or clients of health care to make decisions since they assume (and expect) relative certainty about providers delivering safe and technically competent care.[51]

4.4 Measuring IC based on the AHRQ ‘Care Coordination Atlas’

The AHRQ Atlas Report189 described the domains and mechanisms that were deemed important for ‘care coordination’. The authors also conducted extensive and structured reviews and produced an ‘Atlas’ to mapping the existing instruments and survey questionnaires into the domains and activities as defined in its framework (Table 8.8-3).

The extensive review found 61 instruments, of which many have multiple versions for different subgroups of targeted populations (Table 4.41). These measures were then mapped into all activities and domains stipulated in its framework at three levels: patients, providers and health system representatives, resulting in three mapping tables

(Table 4.4-2;Table 4.4-3;Table 4.4-4) that could be used to develop specific instruments according to the individual project’s needs and aims.

Detailed guidelines on how to match these measure components to the evaluation project objectives is also provided.189


Table 4.4-1 Index of measures/instruments

No. Measure title

1 Assessment of Chronic Illness Care (ACIC)

2 ACOVE-2 Quality Indicators: Continuity and Coordination of Care Coordination

3 Coleman Measures of Care Coordination

4 Consumer Assessment of Healthcare Providers and Systems (CAHPS)

a. Adult Primary Care 1.0

b. Adult Specialty Care 1.0

c. Child Primary Care 1.0

5 Care Coordination Measurement Tool (CCMT)

6 Client Perception of Coordination Questionnaire (CPCQ)

7 Collaborative Practice Scale (CPS)

a. Nurse Scale

b. Physician Scale

8 Breast Cancer Patient and Practice Management Process Measures

9 Care Transitions Measure (CTM)

a. CTM-3

b. CTM-15

10 Patient Assessment of Care for Chronic Conditions (PACIC)

11 Family-Centered Care Self-Assessment Tool

a. Family Version

b. Provider Version

12 ICU Nurse-Physician Questionnaire

a. Long Version

b. Short Version

13 Primary Care Assessment Survey (PCAS)

14 National Survey of Children With Special Health Care Needs (CSHCN)

15 Head And Neck Cancer Integrated Care Indicators

16 Medical Home Index (MHI)

a. Long Version (MHI-LV)

b. Short Version (MHI-SV)

c. Medical Home Family Index and Survey (MHFIS)


No. Measure title

17 Primary Care Assessment Tool (PCAT)

a. Child Expanded Edition (PCAT-CE)

b. Adult Expanded Edition (PCAT-AE)

c. Facility Expanded Edition (PCAT – FE)

d. Provider Expanded Edition (PCAT – PE)

18 Physician-Pharmacist Collaboration Instrument (PPCI)

19 Readiness for the Patient-Centered Medical Home

20 Family Medicine Medication Use Processes Matrix (MUPM)

Measure Titles

21 Resources and Support for Self-Management (RSSM)

22 Continuity of Care Practices Survey

a. Program Level (CCPS-P)

b. Individual Level (CCPS-I)

23 Program of All-Inclusive Care for the Elderly (PACE)

24 Measure of Processes of Care (MPOC-28)

25 Care Evaluation Scale for End-of-Life Care (CES)

26 Oncology Patients’ Perceptions of the Quality of Nursing Care Scale (OPPQNCS)

27 Care Coordination Services In Pediatric Practices

28 Collaboration and Satisfaction About Care Decisions (CSACD)

29 Follow Up Care Delivery

30 Family Satisfaction in the Intensive Care Unit (FS-ICU 24)

31 Korean Primary Care Assessment Tool (KPCAT)

32 Primary Care Multimorbidity Hassles for Veterans With Chronic Illnesses

33 Primary Care Satisfaction Survey for Women (PCSSW)

34 Personal Health Records (PHR)

35 Picker Patient Experience (PPE-15)

36 Physician Office Quality of Care Monitor (QCM)

37 Patient Perceptions of Care (PPOC)


No. Measure title

38 PREPARED Survey

a. Patient Version

b. Carer Version

c. Residential Care Staff Version

d. Community Service Provider Version

e. Medical Practitioner Version

f. Modified Medical Practitioner Version

39 Health Tracking Household Survey

40 Adapted Picker Institute Cancer Survey

41 Ambulatory Care Experiences Survey (ACES)

42 Patient Perception of Continuity Instrument (PC)

43 Jefferson Survey of Attitudes Toward Physician-Nurse Collaboration

44 Clinical Microsystem Assessment Tool (CMAT)

45 Components of Primary Care Index (CPCI)

46 Relational Coordination Survey

47 Fragmentation of Care Index (FCI)

48 After-Death Bereaved Family Member Interview

49 Schizophrenia Quality Indicators for Integrated Care

50 Degree of Clinical Integration Measures

51 National Survey for Children’s Health (NSCH)

52 Mental Health Professional HIV/AIDS Point Prevalence and Treatment Experiences Survey Part II

53 Cardiac Rehabilitation Patient Referral from an Inpatient Setting

54 Cardiac Rehabilitation Patient Referral from an Outpatient Setting

55 Patients with a Transient Ischemic Event ER Visit That Had a Follow Up Office Visit

56 Biopsy Follow Up

57 Reconciled Medication List Received by Discharged Patients

58 Transition Record with Specified Elements Received by Discharged Patients (Inpatient Discharges)

59 Timely Transmission of Transition Record

60 Transition Record with Specified Elements Received by Discharged Patients(Emergency Department Discharges)

61 Melanoma Continuity of Care—Recall System


Table 4.4-2 Care coordination master measure mapping table, patient/family perspective†

FRAMEWORK DOMAINS KEY SOURCES

COORDINATION ACTIVITIES

Establish Accountability or Negotiate Responsibility

3, 4a, 4b, 4c, 6, 9b, 11a, 13, 14, 16c, 17a, 17b, 26, 32, 37, 40, 42, 45, 48

Communicate 3, 4a, 4b, 4c, 6, 9b, 10, 11a, 13, 14, 16c, 17a, 17b, 24, 25, 26, 29, 30, 31, 32, 33, 37, 38a, 45, 48, 51

Interpersonal Communication 3, 4a, 4b, 4c, 6, 10, 11a, 13, 14, 16c, 17a, 17b, 21, 33, 35, 36, 37, 38b, 39, 40, 41, 42, 45, 48, 51

Information Transfer 3, 4a, 4b, 4c, 6, 9b, 10, 11a, 13, 14, 16c, 17a, 17b, 21, 24, 26, 29, 30, 31, 32, 33, 35, 36, 37, 38a, 38b, 39, 40, 41, 42, 45, 48, 49, 51

FACILITATE TRANSITIONS‡

Across settings 9a, 9b, 13, 14, 16c, 17a, 17b, 21, 26, 31, 32, 37, 38a, 38b, 40, 42, 51

As coordination needs change 11a, 14, 24

Assess Needs and Goals 3, 4a, 4b, 4c, 6, 9a, 9b, 10, 11a, 13, 14, 16c, 17a, 17b, 21,24, 25, 26, 30, 31, 32, 33, 35, 37, 38a, 38b, 40, 41, 42, 45

Create a Proactive Plan of Care 6, 9b, 10, 11a, 16c, 21, 24, 37, 38a, 40

Monitor, Follow Up and Respond to Change

3, 4a, 4b, 4c, 6, 9b, 10, 11a, 13, 16c, 17a, 17b, 21, 24, 25, 26, 29, 31, 32, 33, 36, 37, 39, 40, 41, 45

Support Self-Management Goals 4a, 4b, 4c, 6, 9a, 9b, 10, 11a, 13, 16c, 17a, 17b, 21, 24, 25, 26, 29, 31, 32, 33, 35, 36, 37, 38a, 38b, 40, 41

Link to Community Resources 10, 11a, 16c, 17b, 21, 24, 31, 33, 38a, 38b

Align Resources with Patient and Population Needs

6, 11a, 14, 16c, 17a, 17b, 31, 38a, 38b, 51

BROAD APPROACHES POTENTIALLY RELATED TO CARE COORDINATION

Teamwork focused on Coordination 6, 11a, 16c, 24, 25, 29, 30, 35, 36, 39, 40

Healthcare Home 4a, 4b, 4c, 16c, 17a, 17b, 45, 51

Care Management 11a, 14, 21, 51

Medication Management 4a, 4b, 4c, 6, 9a, 9b, 10, 17a, 17b, 21, 32, 35, 36, 37, 38a, 38b, 42, 48

Health IT-enabled Coordination 4a

† A key to measure numbers can be found in Table 4.4-1.

‡ All measure items addressing transitions are mapped to one of the specific transition types (across settings or as coordination needs change).


Table 4.4-3 Care coordination master measure mapping table, healthcare professional(s) perspective†




5,7a, 7b, 11b, 18, 20, 22b, 38c, 38d, 38e, 43, 46

Communicate 5, 7a, 7b, 11b, 12a, 12b, 17d, 22b, 23, 38e, 38f, 43, 46

Interpersonal Communication 7a, 7b, 8, 11b, 12a, 12b, 17d, 18, 22b, 28, 43

Information Transfer 5, 8, 11b, 12a, 12b, 17d, 18, 20, 22b, 23, 27, 38c, 38d, 38e, 38f


Across settings 5, 17d, 22b, 27, 43, 38c, 38d, 38e, 38f

As coordination needs change 11b, 22b

Assess Needs and Goals 5, 11b, 12a, 12b, 17d, 20, 23, 27, 38d, 38e, 38f, 43, 46

Create a Proactive Plan of Care 5, 7b, 8, 11b, 12a, 22b, 23, 27, 38e, 38f

Monitor, Follow Up, and Respond to Change

5, 11b, 12a, 12b, 17d, 20, 22b, 23

Support Self-Management Goals 5, 8, 11b, 17d, 20, 22b, 38d, 38e, 38f

Link to Community Resources 5, 11b, 17d, 22b, 27, 38e


5, 8, 11b, 17d, 20, 38d, 38e


Teamwork focused on Coordination 7a, 7b, 11b, 12a, 12b, 18, 23, 27, 28, 43, 46

Healthcare Home 17d

Care Management 5, 11b, 22b, 27

Medication Management 17d, 18, 20

Health IT-enabled Coordination 12a, 17d




Table 4.4-4 Care coordination master measure mapping table, system representative(s) perspective†



Establish Accountability or Negotiate Responsibility 1, 2, 15, 16a, 16b, 57, 58, 59, 60

Communicate 1, 16a, 16b, 17c, 22a, 34

Interpersonal Communication 17c, 22a, 52

Information Transfer 1, 2, 15, 16a, 17c, 22a, 34, 44, 49, 50, 52, 53, 54, 56, 57, 58, 59, 60


Across settings 15, 16a, 17c, 22a, 49, 50, 55, 57, 58, 59, 60

As coordination needs change 16a, 16b, 22a

Assess Needs and Goals 1, 16a, 16b, 17c, 44, 49

Create a Proactive Plan of Care 1, 16a, 16b, 22a, 49, 52, 55, 58, 59, 60

Monitor, Follow Up, and Respond to Change 1, 2, 3, 17c, 19, 22a, 44, 49, 54, 58, 59, 60, 61

Support Self-Management Goals 1, 16a, 17c, 19, 22a, 34, 49

Link to Community Resources 1, 16a, 17c, 22a, 44, 52

Align Resources with Patient and Population Needs 1, 2, 16a, 16b, 17c, 19, 49, 52


Teamwork focused on Coordination 1, 44, 52

Healthcare Home 2, 3, 16a, 16b, 17c, 19, 47

Care Management 15, 16a, 16b, 22a, 49

Medication Management 2, 3, 17c, 57, 58, 60

Health IT-enabled Coordination 1, 16a, 17c, 19, 34, 44, 50



Please note that descriptions of care coordination activities and interventions in the literature were also often ambiguous with no consensus on definitions of the terms. The broad approach to care coordination frequently only described general processes or roles without specifying who performs what action under what circumstances. These approaches were also often with a wide scope and a goal of improving aspects of patient care beyond just care coordination. The AHRQ working definitions for each framework domain in care coordination were developed drawing from a variety of sources. After the publication of the Atlas, the authors further updated the instrument measures (June 2014) in another online appendix10 which included new instruments and updates on the old instruments included in Table 4.4-1. The newly included measure instruments were presented (Table 4.4-5) and the full details and the specific questions asked in each instrument can be found in the document10 (Appendix 5). This update also included the Canadian Survey of Experiences with Primary Health Care Questionnaire that is discussed in the current report.


Table 4.4-5 The June 2014 update of AHRQ review results of instrument in care coordination measurement (Note: page number refers to the page number in the original document)10

4d Consumer Assessment of Healthcare Providers and Systems (CAHPS)

Patient Centered Medical Home (PCMH) Supplementary Survey Adult Version 2.0 ....................................................................... 3

4e Consumer Assessment of Healthcare Providers and Systems (CAHPS) Patient Centered Medical Home (PCMH) Supplementary Survey Child Version 1.1 ..................................................................................................................................................................... 10

41b Primary Care Provider Ambulatory Care Experiences Survey (PCP ACES) .......................................................................................... 14

62 Team Survey for Program of All-Inclusive Care for the Elderly (PACE) ................................................................................................... 19

63 Medication Reconciliation for Ambulatory Care ...................................................................................................................................................20

64 Promoting Healthy Development Survey PLUS – (PHDS-PLUS) .................................................................................................................21

65 Canadian Survey of Experiences with Primary Health Care Questionnaire ...................................................................................... 36

66 Interpersonal Processes of Care Survey (IPC-II) ......................................................................................................................................................80

67 Brief 5 A’s Patient Survey .........................................................................................................................................................................................................84

68 Patient Perceived Continuity of Care from Multiple Providers ....................................................................................................................86

69 Relational and Management Continuity Survey in Patients with Multiple Long-Term Conditions ..................................91

70 Patient Perceptions of Integrated Care Survey (PPIC) ........................................................................................................................................92

71 Safety Net Medical Home Scale (SNMHS) ...................................................................................................................................................................99

72 Parents’ Perceptions of Primary Care (P3C) ............................................................................................................................................................ 100

73 Primary Care Questionnaire for Complex Pediatric Patients ..................................................................................................................... 104

74 Safety Net Medical Home Provider Experience Survey .................................................................................................................................108

75 Rhode Island Physician Health Information Technology Survey ........................................................................................................... 109

76 Primary Care Medical Home Option Self-Assessment Tool ........................................................................................................................ 116

77 Communication with Referring Physicians Practice Improvement Module(CRP-PIM) ...........................................................130

78 Safe Transitions Best Practice Measures for Community Physician Offices ..................................................................................... 135

79 National Survey of Physicians Organisations and the Management of Chronic Illness II (NSPO-2) ............................. 158

80 Patient-Centered Medical Home Assessment (PCMH-A) Tool .................................................................................................................. 202


4.5 Measuring patient experience in IC in the UK

Improved care coordination and integration of services within the health care sector as well as across health, social care and other public services, is a priority for the UK government. The expectation is that IC will lead to a more person-centered, coordinated care, improve outcomes for individuals, deliver more effective care and support and provide better value from public expenditure.36 Many initiatives were underway and planned to bring about better integration of health, social care and other services to meet people’s needs more comprehensively and seamlessly. The aim of two recent initiatives in England – the Integrated Care and Support ‘Pioneers’ and the Better Care Fund (BCF) – is to enable more effective partnerships working across the NHS and local government sectors, including the commissioning and provision of public health, health and social care services, together with other Local Authority responsibilities.

The UK Department of Health (DH) recently commissioned two short-term projects: the first ran from November 2013 to February 2014 and aimed to provide advice on indicators of IC for individual and collective progress monitoring using routine data; the second was an early evaluative study of the first 15 months of the pioneers in the context of the BCF, with a report due in mid-2015. The first project on the identification of indicators for measuring IC was, in

part, built on two earlier reports commissioned by DH: the Picker Institute et al (2013) report on options for measuring patient/user experience of IC190; and the Picker Institute/Oxford University report (2013)191identifying potential survey questions for measuring patient/ user experience of IC.

The authors of the first project produced a report on measurement of IC, guidelines on using the indicators and on how to use routine quantitative data to measure trends in IC. Part of the report (section F, Appendix B36 ) also provides a summary of the existing questions in current national surveys that have particular relevance for IC and are outlined in Table 4.5-1. The 18 questions developed in a previous review191 combined with the questions from the existing survey instruments provided a less satisfactory, but nevertheless, short-term solution for the pioneers in testing different IC models in the UK (Table 4.5-2).

One important feature of the UK’s effort so far was that the developed PREMs on IC were only captured through patient perspectives (as framed under the “I” statements), which is in contrast to the AHRQ framework that also includes the providers and health system representatives’ perspective. It should be noted that the three-level perspective adopted by the AHRQ is consistent with the three recent Canadian reports of measurement scales and items (on patient, providers and health system experiences) in primary care settings.


Table 4.5-1 Existing user/carer experience measures in large national surveys36

Indicator ID Indicator description Data source Other notes

(29) Proportion of people dying at home/place of their choosing

National End of Life Care Intelligence Network – End of Life Care Profiles

(30) Improving people’s experience of integrated care

NHS Outcomes Framework indicator 4.9; Adult Social Care Outcomes Framework indicator 3E

New ASCOF indicator from 2014/15

(31) Safety: the proportion of people who use services who say that those services have made them feel safe and secure

Adult Social Care Outcomes Framework indicator 4B

(32) GP Patient Survey questions

GP Patient Survey Q32. (For people with LTCs) In the last six months have you had enough support from local services or organisations to help you to manage your long-term health condition(s)? Please think about all services and organisations, not just health services.

Q33. How confident are you that you can manage your own health?

Q40. Do you know how to contact an out-of-hours GP service when the surgery is closed?

(33) Inpatient survey questions

Survey of adult inpatients

Q60. Did hospital staff take your family or home situation into account when planning your discharge?

Q63. Did hospital staff discuss with you whether you would need any additional equipment in your home, or any adaptations made to your home, after leaving hospital?

Q64. Did hospital staff discuss with you whether you may need any further health or social care services after leaving hospital (e.g. services from a GP, physiotherapist or community nurse, or assistance from social services or the voluntary sector)?

Q65. Did you receive copies of letters sent between hospital doctors and your family doctor (GP)?

(34) A&E survey questions Accident and emergency survey

Q38. Did hospital staff take your family or home situation into account when you were leaving the A&E Department?

Q41. As far as you know, was your GP given all the necessary information about the treatment or advice that you received in the A&E Department?


Indicator ID Indicator description Data source Other notes

(35) VOICES national bereavement survey questions

National bereavement survey (VOICES), Office for National Statistics

Q3. When he/she was at home in the last three months of life, did he/she get any help at home from any of the services listed?

Q4. When he/she was at home in the last three months of life, did all these services work well together?

Q5. Overall, do you feel that you and your family got as much help and support from health and social services as you needed when caring for him/her?

Q27. Did the hospital services work well together with his/her GP and other services outside of the hospital?

Q44. Do you think he/she had enough choice about where he/she died?

Q46. Were you or his/her family given enough help and support by the healthcare team at the actual time of his/her death?

Q52. Since he/she died, have you talked to anyone from health and social services, or from a bereavement service, about your feelings about his/ her illness and death?

(Adapted from Raleigh et al; 2014)36


Table 4.5-2 The 18 supplementary questions developed for measuring IC experience in the UK setting

Question Carer version ‘I’ statement Notes

Q3.1 Have all your needs been assessed?

1 All of my needs have been assessed

2 Some of my needs have been assessed

3 None of my needs have been assessed

4 Don’t know/can’t remember

Q3.1 Have all your needs been assessed?

1 All of my needs have been assessed

2 Some of my needs have been assessed

3 None of my needs have been assessed


All of my needs as a person are assessed

Q3.1 is being taken forward for further testing for potential inclusion in the Inpatient Survey and ASCS

Q3.2a Were you involved as much as you wanted to be in decisions about your care and support?

1 Yes, definitely

2 Yes, to some extent

3 No

Q3.2a Were you involved as much as you wanted to be in decisions about your care and support?

1 Yes, definitely


3 No

I am involved in discussions and decisions about my care, support and treatment as I want to be

Q3.2a is being taken forward for further testing for potential inclusion in the Inpatient Survey and ASCS, but not the Carers Survey (instead Q3.3a (carers version) will be tested)

Q3.2b Were you involved as much as you wanted to be in decisions about your treatment?

1 Yes, definitely


3 No

The researchers did not recommend using this question in a survey of carers

I am involved in discussions and decisions about my care, support and treatment as I want to be

This question relates to ‘treatment’, so could be seen as quite health-centric. ICQIDG therefore recommended that survey owners use Q3.2a.

Q3.3a Were your family or carer involved in decisions about your care and support as much as you wanted then to be?

1 Yes, definitely


3 No

4 There were no family or carers available to be involved

5 I didn’t want my family or carer to be involved in decisions about my care and support

Q3.3a Were you involved as much as you wanted to be in decisions about the care and support of the person you care for?

1 Yes, definitely


3 No

4 I didn’t want to be involved in decisions about care

My family or carer is also involved in these decisions as much as I want them to be

Q3.3a (carers version) has been put forward for further testing for potential inclusion in the Carers Survey



Q3.3b Were your family or carer involved in decisions about your treament as much as you wanted then to be?

1 Yes, definitely


3 No

4 There were no family or carers available to be involved

5 I didn’t want my family or carer to be involved in decisions about my treatment

Q3.3b Were you involved as much as you wanted to be in decisions about treatment of the person you care for?

1 Yes, definitely


3 No

4 I didn’t want to be involved in decisions about care

My family or carer is also involved in these decisions as much as I want them to be

Q3.4 Overall, do you feel that your carer/family has had as much support from health and social services as they needed?

1 Yes, they have had as much support as they needed

2 They have had some support but not as much as they needed

3 No, they did not want/need support

4 They did not want/need support

5 There are no family members or carers to support

Q3.4 Overall, do you feel that you have had as much support from health and social services as you needed?

1 Yes, I have had as much support as I needed

2 Yes, I have had some support but not as much as I needed

3 No, I have had little or no support

4 I did not want/need support

My carer/family have their needs recognised and are given support to care for me

Q3.4 is being taken forward for further testing for potential inclusion in the ASCS and the Inpatient Survey



Q3.5 To what extent do you agree or disagree with the following statement…

‘Health and social care staff always tell me what will happen next’

1 Strongly agree

2 Agree

3 Neither agree nor disagree

4 Disagree

5 Strongly disagree


‘Health and social care staff always tell me what will happen next’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

I know in advance where I am going, what I will be provided with, and who will be my main point of professional contact.

This question has been put forward for further testing for potential inclusion in the ASCS, Carers Survey and CMHS.

Responding to concerns raised over the notion of health and social care staff telling people what will happen next, ICQIDG has recommended that a variant should be tested as well or instead of Q3.5:

“Q3.5a To what extent do you agree or disagree with the following statement….

‘Health and social care staff always ensure I know what will happen next’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree’

The variant of Q3.5 is being tested for the ASCS, Carers Survey, CMHS and Inpatient Survey.

Q3.6 When health or social care staff plan care or treatment for you, does it happen?

1 Yes, it happens all of the time

2 It happens most of the time

3 It happens some of the time

4 No

Q3.6 Thinking about the person you care for, when health or social care staff plan care or treatment for them does it happen?

1 Yes, it happens all of the time

2 It happens most of the time

3 It happens some of the time

4 No

When something is planned, it happens

This question has been put forward for further testing for potential inclusion in the Carers Survey



Q3.7a To what extent do you agree or disagree with the following statement…

‘My care and support is reviewed as often as it should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

Q3.7a Thinking about the person you care for, to what extent do you agree or disagree with the following statement…

‘Their care and support is reviewed as often as it should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

I have regular reviews of my care and treatment, and of my care and support plan.

The carers version of this question has been put forward for further testing for potential inclusion in the Carers Survey

Q3.7b To what extent do you agree or disagree with the following statement…

‘My treatment is reviewed as often as it should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

Q3.7b Thinking about the person you care for, to what extent do you agree or disagree with the following statement…

‘Their treatment is reviewed as often as it should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

I have regular reviews of my care and treatment, and of my care and support plan.


‘My medicines are thoroughly reviewed as often as they should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

Q3.8 Thinking about the person you care for, to what extent do you agree or disagree with the following statement…

‘Their medicines are thoroughly reviewed as often as they should be’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

I have regular, comprehensive reviews of my medicines



Q3.9 Do you have a named health or social care professional who coordinates your care and support?

1 Yes

2 No, I coordinate my own care and support

3 Don’t know/not sure

Q3.9 Do you have a named health or social care professional who coordinates your care and support?

1 Yes

2 No, I coordinate my own care and support


I always know who is coordinating my care

Q3.9 is being taken forward for the Inpatient Survey and for further testing for potential inclusion in the ASCS, where the following additional answers will be included as Q3.9a:

No – I need and/or would like someone to coordinate my care and support

No – I don’t have multiple needs so my care and support does not need coordinating

No – For other reasons

Q3.10 Do you know who to contact if you need to ask questions about your condition or treatment?

1 Yes, definitely


3 No


Q3.10 Do you know who to contact if you need to ask questions about the condition or treatment of the person you care for?

1 Yes, definitely


3 No


I always know who is coordinating my care.

The carers version of this question has been put forward for further testing for potential inclusion in the Carers Survey

Q3.12 If you have questions, when can you contact the people treating and caring for you? Please tick ALL that apply

1 During normal working hours

2 During the evening

3 During the night

4 Weekends


Q3.12 If you have questions, when can you contact the people treating and caring for the person you care for? Please tick ALL that apply

1 During normal working hours

2 During the evening

3 During the night

4 Weekends


I have one first point of contact. They understand both me and my condition(s).

I can go to them with questions at any time.

Q3.13 Do you feel this person understands about you and your condition?

1 Yes, definitely


3 No

Q3.13 Do you feel this person understands about the person you care for and their condition?

1 Yes, definitely


3 No

I have one first point of contact. They understand both me and my condition(s).

I can go to them with questions at any time.



Q3.14 Do all the different people treating and caring for you work well together to give you the best possible care and support?

1 Yes, all of them work well together

2 Most of them work well together

3 Some of them work well together

4 No, they do not work well together


Q3.14 Thinking about the person you care for, do all the different people treating and caring for them work well together to give the best possible care and support?

1 Yes, all of them work well together

2 Most of them work well together

3 Some of them work well together

4 No, they do not work well together


The professionals involved with my care talk to each other. We all work as a team.

Q3.14 (carers version) has been put forward for further testing for potential inclusion in the CMHS, Inpatient Survey, ASCS and Carers Survey

Q3.15 Do health and social care services help you live the life you want as far as possible?

1 Yes, definitely


3 No

Q3.15 Do health and social care services help you live the life you want as far as possible?

1 Yes, definitely


3 No

Taken together, my care and support help me live the life I want to the best of my ability.

Q3.15 is being taken forward for further testing for potential inclusion in the ASCS and Inpatient Survey


‘In the last 12 months, health and social care staff have given me information about other services that are available to someone in my circumstances, including support organisations’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree


‘In the last 12 months, health and social care staff have given me information about other services that are available to someone in my circumstances, including support organisations’

1 Strongly agree

2 Agree


4 Disagree

5 Strongly disagree

I am told about the other services that are available to someone in my circumstances, including support organisations.

This question has been put forward for further testing for potential inclusion in the Carers Survey

(Adapted from King et al., 2014)191

(* Cognitive testing of Q3.5, Q3.5a and Q3.14 for use in the CMHS found that the cohort did not fully understand that they were being asked about their experience of a range of health and social care services, including services for both mental and physical health. The questions have therefore not been included in the 2013/14 survey; further work is planned for the 2014/15 survey in order to ensure that if included the questions are framed so as to collect the intended information.)


4.6 Measuring ‘continuity of care’ based on patient experience – a systematic review of the instruments

Like the term ‘care coordination’, the definitions of ‘IC’ were polyamorous and the different definitions may lead to different choices of measurement instruments. The concept of ‘continuity of care’ also seems entangled with ‘IC’ and ‘coordinated care’.192-203. Uijen and colleagues conducted a systematic review197 on the measurement properties of instruments in measuring ‘continuity of care’ based on the COSMIN checklist204-208. The authors searched from 1995 to October 2011 and included articles describing the development and/ or evaluation of the measurement properties of instruments measuring one or more dimensions of continuity of care as defined by the authors: (1) care from the same provider who knows and follows the patient (personal continuity), (2) communication and cooperation between care providers in one care setting (team continuity), and (3) communication and cooperation between care providers in different care settings (cross-boundary continuity). The authors included 24 articles describing the development and/or evaluation of 21 instruments. Ten instruments measured all three dimensions of continuity of care. Instruments were developed for different groups of patients or providers. For most instruments, three or four of the six measurement properties were assessed (mostly internal consistency, content validity, structural validity and construct validity). Six instruments scored positively on the quality of at least three of six measurement properties. The authors concluded that most instruments had suffered from poor methodological quality and did not include all three dimensions of continuity of care (Table 4.6-1). Based on the review results, the authors recommend the use of one of the four most promising instruments, depending on the target population: Diabetes Continuity of Care Questionnaire 209(Type II diabetes), Alberta Continuity of Services Scale-Mental Health210 211 (for patients with mental heath problem), Heart Continuity of Care Questionnaire212 (developed among patients with either congestive heart failure or atrial fibrillation) and Nijmegen Continuity Questionnaire (PHC setting: among patients with one or more chronic diseases197 202).


Table 4.6-1 Quality of measurement properties and the interpretability per instrument (Adopted from Uijen et al. 2012197)

Measurement properties Interpretability

Instrument Internal Consistency

Reliability Measurement Error

Content Validity

Structural Validity

Hypotheses Testing

Differences in scores between subgroups

Floor/ceiling effects of subdomain(s)

Minimal important change (MIC)

CPCI -- na na +++ ++ + Not reported Unknown Unknown 4, 3 positief

VCC -- na na +++ ++ na Not reported Unknown Unknown 3, 2 positief

CCI +++ --- na + +++ +/- Not reported Floor and ceiling effect

Unknown 5, 3 positief

CONNECT ? -- na ++ na ? Not reported Floor effect Unknown 4, 1 positief

CPCQ --- na na + + + Not reported Unknown Unknown 4, 3 positief

ACSS-MH +/- - na +++ --- + Reported Unknown Unknown 5, 2 positief

CCPS-I ? na na ? na na Not reported Unknown Unknown 2, 0 positief

CCPS-P ? na na + na ? Not reported Unknown Unknown 3, 1 positief

DCCS ? + na + ? - Reported Ceiling effect Unknown 5, 2 positief

HCCQ +++ na na + -- +++ Not reported Unknown Unknown 4, 3 positief

ECC-DM --- na ? na ++ ? Reported Unknown Unknown 4, 1 positief

King et al. (Nameless)

? + na +++ na na Not reported Unknown Unknown 3, 2 positief

CONTINU-UM na + ? ? na na Not reported Unknown Unknown 3, 1 positief

DCCQ + na na + ? - Not reported No floor/ceiling effect


PCCQ ? na na ? ? ++ Reported Unknown Unknown 4, 1 positief

Ahgren et al. (Nameless)

? na na + na na Not reported Unknown Unknown 2, 1 positief

CRP-PIM na ? na na - na Not reported Ceiling effect Unknown 2, 0 positief

CSI ? na na +++ ? na Not reported No floor/ceiling effect


Gulliford et al (nameless)

+ na na ? + na Reported Unknown Unknown 3, 2 positief

CCCQ +++ --- na +++ +++ ? Not reported Ceiling effect Unknown 5, 3 positief

NCQ +++ +++ ? + ? +++ Reported No floor/ceiling effect


+++ or - - - = strong evidence positive/negative result, ++ or - - = moderate evidence positive/negative result, + or - = limited evidence positive/negative result, +/- = conflicting evidence, ? = unknown, due to poor methodological quality. na = no information available.

Cross-cultural validity, criterion validity and responsiveness were not evaluated

It should be noted that the review only included those measurement instruments specifically designed for ‘continuity of care’ but not those instruments having multiple domains.


4.7 The key points in measuring PREMs

Given the vast range of different frameworks and methods included in measuring patient experience, it is not possible to suggest that a certain approach or a particular tool is most effective for measuring the patient experience. However, in a recent scoping review, 10 things that need to be considered when planning how to measure changes in the patient and carer experience over time were suggested and paraphrased below213:

1) Consider how patient experience is being defined to inform exactly what needs to be measured.

2) Think about why patient experience is being measured and how the information will be used.

3) Assess whether it would be useful to combine approaches so that both qualitative and more quantitative material is collected.

4) Consider whether to ask everyone using the service or only a sample to provide feedback.

5) Think about whether the best time to collect feedback is immediately after using the service when experiences are fresh in people’s minds.

6) Allocate enough time at the outset to plan and test measurement methods, particularly if these will be used in the future to monitor change over time.

7) Think about how the end-result needs to be presented to various audiences as this may shape how the data are collected. Potential outputs include statistical averages, in-depth quotes or graphs.

8) Make sure that there is appropriate infrastructure at an organisational level to analyze and use patient experience information.

9) Make sure that patients, carers, managers and health professionals are all comfortable with why feedback is being collected and how it will be used. Staff need to be on board as well as patients.

10) Ensure that patient experience measures are seen as one component of a broader framework of measurement and that all approaches work well together without an excessive burden for either staff or patients.

5 | Using PROMs and PREMs to improve health care


5.1 A conceptual framework in understanding the effect of PROM/PREMGreenhalgh et al (2005)214 proposed a framework (Figure 5.1-1) that depicts mechanisms between the routine collection of PROs and changes in patient outcomes. The authors posit that the multilayer mediators (i.e. changes to doctor-patient communication, monitoring treatment responses, detecting unrecognised problems, changes to patient health behaviour, changes to clinicians’ management plans, and improved patient satisfaction) have complex relationships among them. The studies that unveiled these complex relationships may help us understand whether and how routinely collected PROs work to improve the intended outcomes.

Figure 5.1-1 A hypothetical framework to understand the impact of routinely collected PROs on patient health outcomes.

A | Provision of information from health status measures to clinicians- Alone or supplemented with management guidelines- With or wothout training in interpretation of scores- Fed back once or several times- Graphical displays of scores or written summaries, with or without population norms, with or without previous scores

B | Evidence: +- Clinicians talk to patients about their feelings/health status- Develop a shared view of treatment goals/health status/reason for the visit

C | Evidence: +/-Monitor treatment response

D | Evidence: +/-Detect unrecognisedproblems

E | Evidence: -Changes to patienthealth behaviour- Visits to the clinician- Adherence to treatment

F | Evidence: -Changes to cliniciansmanagement of patient- Changes to/initiation of treatment- Referrals to other agencies- Tests to investigate problem further- Advice on problem management

G | Evidence: -Improved patient satisfaction

H | Evidence: -Improved healthoutcomes

Recently, Abernethy and colleagues215 have argued that the routine collection of PROMs has the capacity to impact not only at the patient-level but by addressing the logistics of data linkage and could ensure that the system will grow to accommodate other clinical- and health system-level issues (e.g. evaluating comparative effectiveness of treatments, monitoring quality of care, and translating basic science findings into clinical practice, Figure A). The integration


of data systems will fuel rapid learning cancer care at the national and societal levels (See Figures A and B), making many types of research and system learning possible across institutions and health sectors. The benefits and implications of such rapid learning health care systems may include, but is not limited to, strong and effective quality improvement (QI), increased transparency, accountability, public reporting, better health system performance (monitoring, planning, financing, evaluating, responding) and better quality of care.

The model starts at the patient level, with an ePRO dataset. New datasets can be sequentially added by warehousing orfederated models. The key element is patient-level linkage.

ePRO data

Clinical andadministrativedata

Clinical trials and researchdata

Molecular and biologicaldata

Health system level

National level

Societal level

Clinic level

Figure A: A data linkage framework

Figure B: A learning health care

Note : Figures A and B: adopted from Aberthnethy et al. (2010)

Combining both frameworks, we developed a list of outcome indicators from which the effectiveness of PROM/PREM should be assessed (Table 5.1-1).

(Note: We used the term ‘ Patient-provider communication’ instead of ‘doctor patient communication’ as proposed by Greenhalgh et al.(2005)214 in the current study).

Table 5.1-1 Possible outcome indicators for assessing the impact of the collection of PROM/PREM

No. Outcomes

1 Patient-provider communication

2 Monitor treatment response

3 Detect unrecognised problems

4 Changes to patient health behaviour

5 Changes to patient management

6 Improved patient satisfaction

7 Improved health outcomes

8 Strong and effective quality improvement

9 Increased transparency, accountability and public reporting

10 Better system performance (monitoring, planning, financing; evaluating and responding)

(Source: Chen J. 2012216)

5.2 IC and PROMs/PREMs

5.2.1 How is patient-centered care (PCC) defined?The IOM defines PCC as: “Providing care that is respectful of and responsive to individual patient preferences, needs and values, and ensuring that patient values guide all clinical decisions.”217 PCC supports active involvement of patients and their families in the design of new care models and in decision-making about individual options for treatment. In broad terms, PCC is a model in which providers partner with patients and their families to identify and satisfy the full range of patient needs and preferences, while simultaneously supporting the professional and personal aspirations of their staff. It is broadly described as an approach that puts the patient at the center of the care process and that it is sensitive, empathic and responsive to patients’ individual needs, preferences and values. PCC is expected to contribute to a beneficial outcome for patients (e.g. increased satisfaction with care, improved adherence to treatment and reduced symptom severity), for healthcare providers (e.g. increased job satisfaction, reduction of malpractice complaints) and for the healthcare system (e.g. appropriate use of healthcare resources, decreased costs). PCC is also one of the over-reaching goals of health advocacy, in addition to safer medical systems and greater patient involvement in healthcare delivery and design. Given that non-consumer stakeholders often don’t know what matters most to patients regarding their ability to get and stay well, care that is truly patient-centered cannot be achieved without active patient engagement at every level of care design and implementation. Despite its popularity, the term and content of ‘PCC’ were not universally agreed and it may mean different things to different people. For many, the fundamentals of what it means to be patient-centered remains unclear and many organisations struggle with how to actualise the concept into the day-to-day business of caring for patients and families.

5.2.2 Can the concept of PCC be measured?PCC is widely accepted as an essential component of high-quality health care. 217 218 Although highly recommended for providing health care, PCC is a vague and poorly conceptualised component of high-quality care.219 220 It could be viewed as a general approach for organising healthcare services as illustrated with the patient-centered medical home care model 221; a perspective for guiding the plan, delivery and coordination of the patients’ overall care; a strategy for designing and implementing particular care-related events such as discharge from hospital and


specific treatments such as self-management education222; and a style for communicating with patients.223 Differences in interpretations yield inconsistent operationalisation and application of PCC, which preclude its implementation with fidelity and the valid evaluation of its effectiveness in producing the desired outcomes within and across diverse healthcare contexts. Any effort in trying to measure such a concept should firstly clarify the essential elements that are responsible for inducing its expected outcomes to operationalise the PCC elements in the development of an instrument to measure PCC.224

5.2.3 What is IC?

IC is a concept that has been widely but variously used in many ways in different health systems. A key challenge remains the lack of a common definition for a plethora of terminologies such as ‘IC’, ‘coordinated care’, ‘collaborative care’, ‘managed care’, ‘disease management’, ‘case management’, ‘health/social care service user-centred care’, ‘chronic care’, ‘continuity of care’, ‘seamless care’ and others. There is a lack of clear delineation among the conceptual boundaries between these terms225 and an absence of a sound analytical framework through which to examine the processes of integration.226

In the USA, the term ‘care coordination’ is used more widely. A systematic review commissioned by the AHRQ found that there were more than 40 different definitions of ‘care coordination’227. The authors of the review combined the common elements from many definitions to develop one working definition for use in identifying reviews of interventions in the vicinity of care coordination and, as a result, developed a purposely broad definition:

“Care coordination is the deliberate organisation of patient care activities between two or more participants (including the patient) involved in a patient’s care to facilitate the appropriate delivery of health care services. Organising care involves the marshalling of personnel and other resources needed to carry out all required patient care activities and is often managed by the exchange of information among participants responsible for different aspects of care.”

For some purposes, they noted that other definitions may be more appropriate. This is not surprising given the multiple stakeholders involved in care coordination as depicted in Figure 5.2-1.

Figure 5.2-1 The care coordination ring

Primary care

System representative(s) perspective

Meet patient needs andpreferences in delivery of

high-quality, high-value care

Patie

nt/fam

ily pe

rspect

ive

Health care professional perspective

Inpatient care

Mental healthservices

Long-termcare

Medical historyTest results

Home care

Informalcaregivers

Patient/familyeducation& support

Medications/Pharmacy

Communityresources

Specialtycare (1)

Specialtycare (2)

(Source: AHRQ : Care coordination atlas report)189.

Integration in health care is not likely to follow a single path and variations will be inevitable, given the nature of the health-care ‘production’ process with its imprecise boundaries between stages, the complex ways that service users progress through the system and the often probabilistic nature of the treatment process.7 Thus, analysts have identified different dimensions of integration, most commonly differentiating the type, breadth, degree and process of integration. Four main forms of the integration were discussed228 229:

z Functional: integration of key support functions and activities such as financial management, strategic planning and human resource management.

z Organisational: for example, creation of networks, mergers and contracting.

z Professional: for example, joint working, group practices, contracting or strategic alliances of health-care professionals within and between institutions and organisations.

z Clinical: integration of the different components of clinical processes such as coordination of care services for individual health-care service users or care pathways.

These types can occur in ways that have been described as horizontal integration or vertical integration (also referred to as breadth of integration)229. Horizontal integration links services that are on the same level in the process of health care, for example general practice and community


care, that facilitates organisational collaboration and communication between providers. Vertical integration brings together organisations at different levels of a hierarchical structure under one management umbrella, for example primary and secondary or specialist care.

IC can be realised on a continuum of integration, referred to as the degree of integration.230 231 The degree can range from full integration in which the integrated organisation is responsible for the full continuum of care, including funding to collaboration which describes separate structures in which organisations retain their own service responsibility and funding criteria. The three levels were identified as:

z Linkage: operates through the separate structures of existing health and social services systems with organisations retaining their own service responsibilities, funding and eligibility criteria and operational rules.

z Coordination: as linkage but involves additional explicit structures and processes such as routinely shared information, discharge planning and case managers to coordinate care across the various sectors.

z Full integration: the integrated organisation/system assumes responsibility for all services, resources and funding.

One important message for those contemplating the implementation of an IC system is that:

“Based on the evidence presented here, there may be a need to revisit our understanding of what IC is and what it seeks to achieve and the extent to which the strategy lends itself to evaluation in a way that would allow for the generation of clear-cut evidence, given its polymorphous nature. Fundamentally, it is important to understand whether IC is to be considered an intervention that, by implication, ought to be cost-effective and support financial sustainability or whether it is to be interpreted and evaluated as a complex strategy to innovate and implement long-lasting change in the way services in the health and social-care sectors are being delivered and that involve multiple changes at multiple levels. Evidence presented here and elsewhere strongly points to the latter and initiatives and strategies underway will require continuous evaluation over extended periods of time enabling assessment of their impacts both economic and on health outcomes if we are to generate appropriate conclusions about program effectiveness and the application of findings to inform decision making.”7

From a measurement point of view, the lack of clarity and consensus among the exact content, nature, mechanisms and domains included in IC will hamper the efforts in developing necessary PROM/PREM to understand the process and improve the practice by patients, providers and health systems.

5.2.4 Patient experience vs patient satisfactionA distinction can be made between patient’s satisfaction and patient’s experience. Patient satisfaction surveys tend to ask patients subjective questions about their satisfaction with their care. In contrast, patient experience questions relate to the patient’s objective experience during the entire health center interaction. The patient experience is the sum of a patient’s interaction with a health center and is the patient’s perception of those interactions.

Patient satisfaction questions generally focus on how well a patient’s expectations are met, their preferences and the quality of their care. They tend to be subjective and non-specific. Examples of satisfaction questions include: How do you rate your doctor’s caring and concern for you? How satisfied are you with the appointment system in your health center?

Experience questions relate to the patient’s actual, more objective experiences in the health center and aim to avoid value judgments and the effects of existing expectations. Examples of experience questions include: In the last 12 months, how many days did you usually have to wait for an appointment when you needed care right away?

It could be argued that terms such as satisfaction and experience have distinct meanings. For instance, ‘expectations’ may refer to people’s perceptions before receiving care, ‘experience’ may relate to what happens during care and ‘satisfaction’ may refer to information collected afterwards. Alternatively, ‘experience’ could be taken to describe things that happened and the extent to which their needs are met whereas ‘satisfaction’ could relate more to how they feel about those things.

While there are some detailed theoretical and academic arguments about the difference between these terms, for simplicity, for this report ‘experience’ means any combination of satisfaction, expectations and experience so long as it relates to feedback provided by those using health services as well as their family or carers. For consistency, the term ‘experience’ is used throughout this report although terms such as ‘satisfaction’ and ‘expectations’ are used where needed to focus on those particular concepts.

Patient experience is an important part of the multicomponent of quality of care which is a multi-dimensional concept (safe, effective, efficient, equitable, timely and patient-centered, IOM). As such, patient experience does not (and should not) reflect quality in other domains. Patient experience itself is also a multi-dimensional concept and there is no universal agreement on the exact domains/components of patient


experience should be measured (Appendix 8). However, patient experience is important in its own right. Patient experience is consistently and positively associated with other quality outcomes including patient safety, and clinical effectiveness across a wide range of studies and healthcare facilities. Those providing high-quality clinical care tend to have better experiences reported by patients. Clinical quality and patient experience should be considered as distinct but inter-related aspects of quality.

5.3 International experience

5.3.1 The USA experienceIn the USA, PROM/PREM are being increasingly recognised as important for assessing clinical outcomes, advancing quality improvement and informing technology assessment and reimbursement decisions. Many national organisations have recognised the importance of PROMs in clinical care and the following significant activity has already occurred:

1) NIH funding for the PROMIS and Neuro-QoL network.

2) Legislation authorising the Patient-Centered Outcomes Research Institute (PCORI).

3) Grants from the AHRQ.

4) Activity at the Centers for Medicare and Medicaid Services.

5) Regulatory guidelines from the FDA and other public and private organisations. Integration of PROs into EHRs is also a key component of the Center for Medicare and Medicaid Services’ financial incentive program to demonstrate “meaningful use” of EHRs.

6) Recent recommendations from the IOM include the psychosocial behaviour PROMs in the EHRs.

7) The AHRQ commissioned report building the national digital infrastructure (including eEHRs and PROMs).

In the US, the health care reform movement is heading towards the development of accountable care organisations, integration of PROs into EHRs and is also becoming an important tool for measuring and demonstrating the success of care provided through accountable care organisations. The momentum is building towards national standardisation of PROs to provide real-time outcomes. A significant investment is being made in infrastructure to collect data from patients, synthesising the data in a meaningful way, presenting the data back to both patients and providers as well as developing strategies for acting on the information to improve the clinical care of patients.

5.3.2 The UK experienceThe NHS in the UK has led the application of the PROMs over the years, moving from research to clinical practice and quality improvement. More recently, PROM has been applied to healthcare quality assessment. The program launched by the UK Department of Health in April 2009 requires the collection of PROMs for selected surgical procedures (hip or knee replacement, inguinal hernia, varicose veins) within all the health facilities financed by the NHS. Several questionnaires were used before and after the procedure, thus allowing assessment of the health outcomes reported by the patient. From 2009 to 2013 the total number of questionnaires was high and steadily grew; in the period April 2009-March 2010 184,818 pre-procedure and 134,768 post-procedure questionnaires were completed by the patients and returned with a 17% and 11% increase respectively, compared to the previous period (April 2009-March 2010). The analysis of the UK’s experience offers many valuable lessons in feasibility and resources, the most appropriate methodology as well as in professional culture.

Moreover, two out of the five domains of the NHS Outcomes Framework put great emphasis on PROM/PREM9. Among the five domains of the outcome framework, Domain 2 (Enhancing QoL for people with long-term conditions) and Domain 4 (Ensuring that people have a positive experience of care) heavily rely on the collection of PROM/PREM to assess the outcomes.

Figure 5.3-1 Duty of quality and the NHS Outcomes Framework

Duty of quality

Duty

of qu

ality

Duty

of qu

ality

Duty of quality

Domain 1Preventing

people fromdying

prematurely

Domain2Enhancingquality of

life forpeople withlong-termconditions

Domain 3Helping peopleto recover fromepisodes of ill

health orfollowing

injury

Domain 4Ensuring thatpeople have

a positiveexperience

of care

Domain 5Treating and caring

for people in asafe environment

and protecting themfrom avoidable harm

NHS outcomes framework

NICE quality standards(Building a library of approx 150 over 5 years)

Commissioning/ContractingNHS Commissioning Board - certain specialist services and primary care

GP consortia - all other healthcare services

Commissioningoutcomes

framework

Commissioningguidance

Provider payment mechanisms

tari� CQUIN QOFstandardcontract


Table 5.3-1 Selected outcomes indicators for Domain 2 of NHS Outcome Framework

Overarching indicator

2. Health-related quality of life (HRQoL) for people with long-term conditions (EQ-5D)

Improvement areas

Ensuring people feel supported to manage their condition

2.1 Proportion of people feeling supported to manage their condition

Improving functional ability in people with long-term conditions

2.2 Employment of people with long-term conditions

Reducing time spent in hospital by people with long-term conditions

2.3. i Unplanned hospitalisation for chronic ambulatory care sensitive conditions (adults)

2.3.ii Unplanned hospitalisation for asthma, diabetes and epilepsy in under 19s

Enhancing quality of life for carers

2.4 Health-related quality of life for carers (EQ-5D)

Enhancing quality of life for people with mental illness

2.5 Employment of people with mental illness

5.3.3 The Sweden experience Sweden started a nationwide use of PROMs using the disease specific clinical databases (quality registers) established by the medical profession as early as in 1975. One prominent example is the nationwide, prospective, observational follow-up program which included PROMs for the Swedish Hip Arthroplasty Register. The program started in 2002 and was gradually expanded to include all units performing total hip replacement in Sweden. The self-administered PROMs protocol comprised the EQ-5D instrument, the Charnley class categorisation and visual analogue scales for pain and satisfaction. Study232 showed that patients’ response rates to the Registry were good. Patients eligible for total hip replacement generally reported low health-related QoL and suffered from pain. One year post-operatively the mean EQ-5D index increased to above the level of an age- and gender-matched population, with a considerable reduction of pain. Females, younger patients and those with a Charnley category C reported a lower EQ-5D index pre-operatively than males, older patients and Charnley category A or B,

respectively232. It also showed that despite the required structured organisation and effective data capture infrastructure, a nationwide implementation of a PROMs program is feasible and the continuous collection of PROMs permits local and national improvement and allows for a further health-economic evaluation as well as comparative effectiveness research (CER)233(Figure 5.3-2).

Figure 5.3-2 Using PROM in the Swedish Hip Arthroplasty Register.

PROM in Swedish hip arthroplasty has enabled quality and e�ciency improvementsSigni�cant e�ciency gains from PRO metrics in THAMain indications for total hip arthroplasty are pain and impaired HRQoL, making PRO key

Hence, a standardisedPROM protocol was introduced in 2002

� Including Charnley functional categories, VAS, EQ-5D instruments and EuroQol

� Covering all but one Swedish THA preformers

~8k USD cost per THR patient, of which 61% in productivity loss, and >16k patients annually, implies signi�cant QoL and cost gains potential

Based on PROM output, two alternative treatment procedures could be properly assessed

� One-stage bilateral THA cam out on top, with better HRQoL pro�le, at lower cost

Pre-operative anxiety/depression standsout as deciding factor in THA satisfaction

6

4

2

0

-2

Post-operation satisfaction (VAS)

Mobility

Selfcare

Usual a

ctivitie

s

Pain/disc

omfort

Anxiety/depressi

on

Charmley ca

tegory

Gender

(Source: AHRQ workshop discussion (2012))


5.3.4 The Denmark experience: the generic Integrated PRO System (WestChronic) 234

Similar to Sweden, Denmark also started to introduce PROs in some disease-specific national clinical registers in 2000.235 Despite the fact that logistic and scientific challenges are similar across diagnostic groups and applications, most PRO systems have been applied to a single-patient group. To achieve integration of different disease-specific groups, a system called WestChronic was developed in Denmark in 2004 to accommodate an arbitrary number of PRO projects with individual questionnaires, protocols, patients and users. For the patient and clinician, each implemented project appeared as a unique PRO project with its own logo, domain, website, email address, accompanying letters and contact information, etc.

The WestChronic also had one basic module (basic PRO data collection and logistics) and three optional elements: 1) PRO-based clinical decision support, 2) PRO-based automated decision algorithms, and 3) other forms of communication (detail see: Table 5.3-2). While the first element is ubiquitous, the others are optional and only applicable at a patient level. Each element has its unique methodological and organisational challenges. The WestChronic has, to date, implemented 22 PRO projects within 18 diagnostic groups, including cardiology, neurology, rheumatology, nephrology, orthopedic surgery, gynecology, oncology and psychiatry. Presently, WestChronic includes 1756 items in 92 questionnaires and 158 templates for personalised letters and emails. The system was designed for both the patient and provider groups. Despite not explicitly stated, the data collected by the system can also be harnessed from the health system perspective.

Table 5.3-2 Elements of clinical application of PRO.

Element Content

Content

Base element

PRO data collection and logistics

Questionnaire (items)

Criteria for inclusion and termination

Data collection modes: Web, paper, interview

Approach modes: letter, email, telephone, texting

Schedules of questionnaires/reminders

Optional element 1

PRO overview for clinical decision support

Categorisation of PRO for clinical decision support

Course-oriented graphic overview

Optional element 2

PRO-based automated decision algorithms

Decision tree

Action protocol

Optional element 3

Other forms of communication Two-way communication

One-to-many communication

(Source: Hjollund et al. 2014234 )

The most recent study234 showed that in a total of 30,174 patients, 59,232 PRO assessments were collected using 92 different PRO questionnaires. Response rates of up to 93% were achieved for first-round questionnaires and up to 99% during follow-up. For six diagnostic groups, PRO data was displayed graphically to the clinician to facilitate flagging of important symptoms as well as decision support and in five diagnostic groups PRO data was used for automatic algorithm-based decisions (Table 5.3-3).

The WestChronic system showed the feasibility and utility of the implementation of all proposed protocols for data collection and processing. The system has achieved high response rates and longitudinal attrition is limited. The relevance of the questions, the mixed-mode principle and automated procedures has contributed to the high response rates. Furthermore, development and implementation of a number of approaches and methods for clinical use of PRO has been possible without challenging the generic property. 234


Table 5.3-3 Characteristics of 22 projects involving implementations of a generic PRO system. Projects with patient level use (n=14)

D: PRO for clinical overview (AmbuFlex I)

E: PRO for automated cancelling of visits

F: PRO for screening

G: PRO for clinical decision support (Am- buFlex II)

H: Other forms of communication

Level of aggregation Patient Group/patient Group/patient Patient Patient

Implemented projects

5 3 2 3 1

Invoked elements(Figure 1)

Base + element 1 Base + element 2 Base + element 2 Base + element 1.2

Base + element 1,3

Patients Chronic heart failure Rheumatoid arthritis Renal failure

Lung cancer

Prostatic cancer

Hip/knee replacement

Endometriosis

Acute Coronary Syndrome

Heart transplant

Acute Coronary Syndrome

Heart transplant

ADHDa

Recruitment Preadmission assessment

Clinic referral Hospital registers / clinical referral

Clinic referral Clinic referral

Primary aim Clinical decision support

Efficient use of resource

Screening for depression

Clinical decision support

Communication (therapists and patient)

Extension Local National, selected hospitals

Local Regional Local

In operation from 2009 2011 2011 2012 2012

Patients (Jan 2014) 741 1639 1740 3120 23

Questionnaires/ patient

No limit 3 1/ no limit No limit No limit

Response rate (primary)

75% N/A 88% 93% N/A

Response rate follow-up

82% 97% N/A 99% N/A


Table 5.4-1 Frequency of domains used in the hospital patient experience and satisfaction surveys in Australia

WA OLD SA ACT NT TAS VIC NSW HCAHPS QPS %

Domain

Access/waiting time/admission process

x x x x x x x x x 90%

Information sharing/ communication

x x x x x x x x x 90%

Physical environment x x x x x x x x x 90%

Overall satisfaction x x x x x x x x x 90%

Involvement/participation x x x x x x x x 80%

Privacy/respect/dignity x x x x x x x x 80%

Consistency/coordination of care

x x x x x x x x 80%

Discharge/continuity of care

x x x x x x x 70%

Pain control x x x x x 50%

Safety/quality (i.e hand hygiene, patient identification)

x x x 20%

Note: HCAHPS – Hospital Consumer Assessment of Healthcare Providers and Systems

(Adapted; Source: ACQHS, 201215) QPS: QPS Patient Satisfaction Survey.

5.4 Measuring PREMs in non-primary care settings in AustraliaSeveral reports reviewed15 236-238 the issues of measuring patient experience in Australia. The recent report from the Australia Commission on Safety and Quality in Healthcare (ACSQH)15 provides a summation of the latest domains used in the hospital patient experience and satisfaction surveys in Australia (Table 5.4-1). Since the publication of the ACSQH’s report, the BHI also commissioned a report to review its in-hospital patient experience survey.237

The ACSQH’s report15 also found that St Vincent’s Health Australia had recently conducted a review of its patient experiences (PEx) and patient satisfaction (PSat) surveys in order to inform the development of a standard methodology. During this review, two potential key performance indicators (KPIs) that could be used to measure patient experience were identified. The two candidate KPIs were:

1) Likelihood of recommending the hospital to family and friends.

2) Overall rating of care.

The details of the different surveys used by St Vincent’s Hospitals are presented in Table 5.42.


Table 5.4-2 Survey tools used among St Vincent’s hospitals

State level Tool

used

Tool

description

Frequency of reporting

Valid

tool

No. of

questions

KPI 1 KPI 2

NSW SVH &

SHC, STJ

State wide since 2007, by IPSOS

Picker PEx Annual now monthly, Q4

Yes 70-90** yes yes

SVPH N/A Press

Ganey

PEx and PSat

Monthly, Q4 Yes 57 +

10 special

yes yes

Mater N/A Press

Ganey

PEx and PSat

Monthly, Q4 Yes 55 + some special

yes Yes

QLD HSNS

SVHB

SVHT

Nil but required as part of health fund contract

Local

Local

Local*

PSat

PSat

PSat

Annual

Annual

Continuous

No

No

No

72

36

65

yes

yes

yes

no

yes

yes

VIC

SV& MP

SVH M

Nil

Under development

Local

Local

PSat

PSat

Monthly

6 Monthly

No 21

65

yes

no

Yes

no

*Specific Day surgery and emergency tools used ** Current, proposed to reduce to 50-60

PEx – patient experience, PSat – patient satisfaction

SVH – St Vincent’s Hospital, SHC – Sacred Heart, STJ – St Joseph’s Hospital, SVOPH – St Vincent’s Private Hospital (public), HSNS – Holy Spirit Northside Hospital, SVHB – St Vincent’s Hospital Brisbane, SVHT - St Vincent’s Hospital Toowoomba, SV & MP - St Vincent and Mercy Private, SVHM - St Vincent’s Hospital Melbourne

During one of the surveys on day hospitals by the ACSQH, it was found that a range of survey tools were used by day hospitals. QPS Patient Satisfaction Survey is the survey developed by QPS Benchmarking which is an Australian and New Zealand based health care quality improvement organisation. A range of sampling methods was used by the day hospitals. Some surveyed all patients while others conducted random sampling. The QPS Benchmarking Patient Satisfaction Survey (Version 3) has 20 questions covering the following domains:

1) appointment/waiting times

2) information sharing/communication

3) respect and dignity

4) conduct of staff

5) physical environment

6) overall satisfaction

7) pain management

8) services and equipment

9) billing process

10) discharge/continuity of care.

The survey is conducted 1-2 weeks post-discharge. The QPS scorecards automatically provide a numerator and denominator as well as question by question graphical and numerical results.


Table 5.4-3 Patient experience and satisfaction surveys used in Australia

Provider Tool Administration Scope Tool description

Jurisdiction

Northern Territory Local In-house Hospital PEx

Australia Capital Territory

Health

Victoria Patient

Satisfaction Monitor

Mail State-wide PEx and PSat

Tasmania Local Mail State-wide till 2007.

Hospital thereafter

PEx and PSat

South Australia SACESS CATI State-wide PEx and PSat

Western Australia Local CATI State-wide PEx and PSat

Queensland Picker-based CATI State-wide PEx and PSat

Victoria Victoria Patient

Satisfaction Monitor

Mail and online State-wide PEx and PSat

Private Hospitals

Eye Tech Day Surgery, QLD QPS CATI Hospital PSat

Colin Street Day Surgery, WA QPS CATI Hospital PSat

Mater Hospital North Sydney Press Ganey CATI Hospital PEx and PSat

Healthscope (44 hospitals) HCAPS Mail National Pex and PSat

St Vincent’s Hospitals (2 public, 8 private)

Picker and Press Ganey

in NSW, local in QLD and VIC

Mail National PEx and PSat in select NSW PSat in QLD and VIC

Robina Procedure Centre,

QLD

Local In-house Hospital PEx

Centre for Digestive

Diseases, NSW


Buderim Gastroenterology Centre, QLD


Liverpool Day Surgery, NSW Local In-house Hospital PEx

The Women’s Clinic Day Hospital, WA


The Eye Hospital, Tasmania Local In-house Hospital PSat

CATI - computer aided telephone interviewing, PEx – Patient Experience, PSat – Patient Satisfaction

The key findings of the ACSQH’s report are paraphrased as follows:


z There are differences in the methodologies, administration, scope, rating scales, inclusion and exclusion criteria, sampling, data analysis and reporting methods used across public and private hospitals.

z Most jurisdictions and some private hospitals are using a combination of patient experience and satisfaction questions in their surveys.

z There are documented impacts from the use of the surveys showing how feedback has informed service delivery.

z There was no significant difference in the frequency or number of domains used between private and public hospitals.

z In some jurisdictions and private hospital ownership groups, surveys are conducted state-wide or across one private hospital ownership group while in certain instances individual institutions have developed and administered their own surveys.

z Most of the private hospitals included in this review use locally developed tools. These tools are often administered in-house.

z The surveys are not well suited for use with people who speak little or no English as translated versions are generally not available. The Northern Territory (NT) and Victoria has sought to address language barriers. In NT hospitals, meaningful pictures and symbols are incorporated within the surveys. In Victoria, the surveys are available in English and 16 community languages.

A point worth noting is that there has been little progress made since the review in 2009236 which reported that despite the existing patient experience data in acute hospitals in all Australian states this information was not utilised optimally to realise its potential value.

5.3 Measuring PREMs in primary-care settings in Australia

There is no national wide survey currently in Australia targeting patient experience in the primary care setting. However, a couple of related surveys are described below.

The National Health Services Patient Experience Survey: The Australian Bureau of Statistics (ABS) conducts this survey covering a range of health services and ten domains. It is administered at the population level. The hospital and emergency department (ED) modules focus on rates of admission to hospital within the last 12 months by age, remoteness and sex. These two modules also contain questions relating to reasons for visiting the ED or hospital, whether staff listened and showed respect and whether a satisfactory explanation of treatment was given. The survey does not target people who have had a recent hospital admission or ED presentation. The survey is not designed for hospital-level reporting.

Commonwealth Fund Patient Experience Survey: The Commonwealth Fund runs and reports population-based patient experience surveys in 11 Organisation for Economic Co-operation and Development (OECD) countries in three year cycles. Separately, specified healthcare providers and patient sub-populations are surveyed in the other years.

The ACSQH and the NSW BHI worked with the Commonwealth Fund and increased the sample size of the 2010 population survey and reported on access to and use of primary care services, use of specialists, out-of-pocket costs, prescriptions and hospital and ED experiences.239

The Royal Australian College of General Practice (RACGP) Patient Satisfaction Survey: There is no mandatory national patient experience survey in a primary-care setting in Australia by the RACGP. However, Criterion 2.1.2 of the RACGP Standards for general practice (4th edition) (the Standards) requires practices to:

z Collect feedback about their patients’ experiences by:

z Option 1: using an RACGP-approved validated patient experience questionnaire; or

z Option 2: developing and gaining RACGP approval of a practice-specific method for gaining patient feedback.

z Demonstrate that the information received from patients is used to help improve the practice.

There are four service providers in conducting “RACGP-approved validated patient feedback questionnaires” surveys for accreditation purposes.


The college believes that “The carefully developed RACGP-approved questionnaires are in line with best available evidence and scientific knowledge about questionnaire development and administration. The questionnaires have been to ensure they measure patient experiences in a reliable way. Providers offering these questionnaires will also collate and analyse your patients’ responses and provide your practice with a report of the results”

The four services providers are:

1. Insync Surveys incorporating UltraFeedback’s Patient Satisfaction Instrument (PSIv5): 1800 143 733

2. cfep Survey’s Patient Accreditation Improvement Survey (PAIS): (07) 3855 2093

3. Press Ganey Associates : (07) 5560 7400

4. SEHPA Patient Feedback Survey (available to practices within the Cities of Casey and Greater Dandenong and the Shire of Cardinia within Victoria): (03) 8792 1900

Further information on the survey can be accessed from the PAIS website. There is very limited information available from other service providers’ websites with respect to the content and cost of the surveys.

The RACGP also provides the Patient Feedback Guide: Learning from our patients (updated August 2014) which has been developed to:

z assist general practices in understanding what is required to fulfil Criterion 2.1.2 of the RACGP Standards for general practices (4th edition)

z explain the options available to practices for meeting the requirements of Criterion 2.1.2

z provide guidelines to practices wishing to develop their own practice-specific patient feedback tool.

The RACGP Toolkit for developing practice-specific questionnaires (the Toolkit) is a separate document to be used in conjunction with the Guide. The Toolkit provides a framework and a range of templates to assist the development of a valid and reliable practice-specific questionnaire. As such, applications that use the Toolkit will be approved more quickly.

The approval process can take up to 6-12 months from initial submission of the application and can be quite resource intensive due to the requirements of the Guide. Applications that utilise the Toolkit are anticipated to be approved within 8 weeks.

5.6 Social media, cost-effectiveness and IC

There is an increased use of social media in chronic disease management and in measuring PROMs/PREMs. A quick review of the evidence on its potential benefits and limitations are presented (Appendix 6). The tentative evidence review on the cost-effectiveness of the IC is presented in Appendix 7.

6 | Future plans and research priorities


This report identified several areas which need policy and research consideration in an Australia setting.

6.1 Psychosocial behaviours in EHRs as part of PROMs

In a recent report12 released by the IOM, the Committee on the Recommended Social and Behavioural Domains and Measures for Electronic Health Records was asked to recommend core measures of social and behavioural domains for inclusion in all EHRs. The Committee identified a parsimonious panel of measures that is comprehensive, interoperable and efficient.12 These “psychosocial vital signs” include four measures that are already widely collected (race/ethnicity, tobacco use, alcohol use and residential address) and eight additional measures (education, financial resource strain, stress, depression, physical activity, social isolation, exposure to violence and neighbourhood median household income). While recognising the additional time needed to collect such data and act upon it, the committee concluded that the health benefits of addressing these determinants outweigh the added burden to providers, patients and health care systems. Advances in research in the future will likely point to additional measures that should be included in the panel and periodic re-reviews should be undertaken to assess them.

There is need for consideration of what ‘psychosocial vital signs’ could be included as part of PROMs in the Australian setting.

6.2 Integrating PROMs with EHRs and other data sources: the need for robust health information infrastructure

The recent report commissioned by the AHRQ provided a full blueprint on how to build a robust information infrastructure and integrate PROM into the EHR in the USA.240 The report posits two major arguments:

z The current lack of interoperability among data resources for EHRs is a major impediment to the unencumbered exchange of health information and the development of a robust health data infrastructure. Interoperability issues can be resolved only by establishing a comprehensive, transparent and overarching software architecture for health information.

z The twin goals of improved health care and lowered health care costs will be realised only if health-related data can be explored and exploited in the public interest for both clinical practice and biomedical research. This will require implementing technical solutions that both protect patient privacy and enable data integration across patients.

Given the major push from the IOM to integrate psychosocial and behaviours measures (as part of PROMs) into EHRs, it is important to explore the policy, technical and infrastructure requirement for enabling integration of any future PROM with the EHR (or PCEHR) in Australia.

Other key findings from the report are (paraphrased):

1. The criteria for Stage 1 and Stage 2 Meaningful Use: while surpassing the 2013 goals set forth by Department of Health and Human Services (HHS) for EHR adoption, they fall short of achieving meaningful use in any practical sense. At present, large-scale interoperability amounts to little more than replacing fax machines with the electronic delivery of page-formatted medical records. Most patients still cannot gain electronic access to their health information. Rational access to EHRs for clinical care and biomedical research does not exist outside the boundaries of individual organisations. (Section 3.2)

2. Although current efforts to define standards for EHRs and to certify Health Information Technology (HIT) systems are useful, they lack an unifying software architecture to support broad interoperability. Interoperability is best achieved through the development of a comprehensive, open architecture. (Section 5.1)

3. Current approaches for structuring EHRs and achieving interoperability have largely failed to open up new opportunities for entrepreneurship and innovation that can lead to products and services that enhance health care provider workflow and strengthen the connection between the patient and the health care system, thus impeding progress toward improved health outcomes. (Section 5.1)

4. HHS has the opportunity to drive adoption and interoperability of the EHR by defining successive stages of Meaningful Use criteria that move progressively from the current closed box system to an open software architecture. (Section 5.2)

5. The biomedical research community will be a major consumer of data from an interoperable health data infrastructure. At present, access to health data is mostly limited to proprietary datasets of selected patients. Broad access to health data for research purposes is essential to realising the long-term benefits of a robust health data infrastructure. (Section 6.2)


6. The data contained in EHRs will increase tremendously both in volume and in the diversity of input sources. It will include genomic and other “omic” data, self-reported data from embedded and wireless sensors and data gleaned from open sources. Some types of personal health data, especially when combined, will make it possible to decipher the identity of the individual even when the data are stripped of explicit identifying information, thus raising challenges for maintaining patient privacy. (Section 6.3)

7. The US population is highly diverse, reflecting much of the diversity of the global population. Therefore, important research findings applicable to Americans are likely to come from shared access to international health data. Currently there is no coherent mechanism for accessing such data for research. (Section 6.4)

8. Electronic access to health data will make it easier to identify fraudulent activity but at present there is little effort to do so using EHRs.

Most of these issues are not only restricted to the USA and some are more pronounced in an Australian setting such as the fragmentation of data sources. The recent review commissioned by National Electronic Health Transition Authority (NEHTA) recommended 38 sweeping changes to its governance structure (including dissolving NEHTA), operation and name of the PCEHR (to MyHR).241 The report viewed the care integration as one of the most potential beneficial areas of meaningful use of PCEHR (citing both ABS Patient Experience Survey results and Kaiser Permanente case). While the effects of the proposed changes remain to be seen, it is important that the issues raised above are reviewed in the Australian setting. It should also be a priority to create a health information infrastructure to link PROM/EHR to other existing data sources (registry, administrative data, survey data, Medical Benefit Scheme(MBS), Pharmaceutical Benefit Scheme(PBS), Registry of Births, Deaths and Marriages(RBDM), bio banks, etc) and that measures are taken to ensure the promised potential of utilising PCEHR to facilitate care integration.

6.3 Developing PROM/PREM measures on important subgroups

While there is no need to reinvent the wheel by creating our own PROM/PREM if there is already a sound existing instrument, one should still critically review the adopted PROM/PREM measures to see if these PROMs/PREMs are suitable to be used across different population subgroups. The notable such subgroups in Australia include the Indigenous population, patients from culturally and linguistically diverse (CALD) backgrounds, women, children and the elderly as well as patients with mental health problems or cognitive impairment. The sound psychometric properties of an instrument developed in another setting and population may not be applicable to the subgroups in an Australian setting. Thus, it is critical that we test the suitability of these instruments in different Australian subgroups. In a psychometric term, it is the issue referred to as cross-culture validity or measurement invariance.

In the UK, the National Voices are currently developing narratives for IC specific to four particular user groups – mental health, children and young people, end of life care and frail older people – which will enable the development of user experience indicators specific to these groups.36 Another development in the UK is a project to develop a survey tool for measuring user-reported experience of IC among older people with a long-term condition. Its aim to support health and social care services in England and, more widely, in an international context to measure and improve the quality of IC. Led by the Nuffield Trust in collaboration with the Picker Institute, The King’s Fund, National Voices and the International Foundation for Integrated Care, the project is funded by the AETNA Foundation in the US and is due for completion in 2015.


6.4 Exploring the ways that the results can be better presented to different stakeholders

The purpose of collecting PROM/PREM is to improve quality care at three different levels (i.e. patient, providers and health systems). It is critical to understand and develop the best way to communicate the results derived from the PROM/PREM to different stakeholders. One of the reasons for lack of buy-in from doctors on collecting a PRO is that they did not see the relevance of PROs to their practice and failed to understand how the results could be used to improve their clinical practice. Useful strategies should be explored including:

z Testing different clinical applications of available tools.

z Making explicit the utility of collected PROM: screening, diagnosing, risk stratification and prognosis, indication for treatment, monitoring, consistent use along the care pathway.

z Identifying best methods for data collection (home vs health care settings, electronic vs paper and pen, classical test theory-based vs IRT based, timing of data collection).

z Best methods for feedback and interpretation, visual presentation;

z Most effective ways for training of professionals (web-based, virtual world, face-to-face, etc);

z Involving relevant stakeholders in the conceptualisation, developing and testing PROMs/PREMs and in interpreting and presenting the results.

6.5 Investment in understanding population norm, cut-off, MID, responsiveness, response-shift of the PROMs/PREMs

For any used PROM/PREM instrument, it is critically important that the population norm in an Australian setting, the clinically meaningful cut-off (or minimally important difference), the responsiveness of such a measure in tracking change over time and possible existence of response-shift are thoroughly investigated. Many of these issues if not addressed properly will threaten the validity and interpretation of the results and make the whole effort of collecting the data meaningless. These issues are complex and often require dedicated investigation and carefully designed studies. Few studies have addressed these issues adequately.

6.6 Investment in IRT/CAT technique and item banks for specific interested area

IRT/CAT has been successfully used in education and psychology for many decades and PROMIS has showed the feasibility of its application in the health sector at scale. IRT/CAT has many conceptual advantages such as comparability, precision (reliability), flexibility, inclusiveness and reduced burden for respondents. Given the foreseeable growth in numbers of PROMs/PREMs required for patients, the burden of responding to the PROMs/PREMs on patients and the burden of collecting these PROMs/PREMs on health professionals should not be underestimated. However, developing IRT/CAT based PROMs requires significant investment and expertise. It should be considered if strategic investment in this area is warranted.

6.7 Developing suitable case-mix adjustment methodology for different stakeholders

Adequate case-mix or risk-adjustment methods are an important part of developing meaningful quality measures. As the aims, settings, and targeted populations may vary for different PROM/PREM, the risk-adjustment methods are also likely to be purpose-specific. Given the recent studies from the USA on the potential misleading results of applying improper risk-adjustment methodology242 243, it is important to investigate these issues to evaluate its implication in an Australian setting and to develop the correct risk-adjustment methodology.

7 | Summary


We have conducted a rapid review of the issues on PROMs and PREMs with particular focus on their relevance to the IC strategy in NSW. Points from the review include:

1) The international experience, policy background and research interests are fast-moving in relation to PROM/PREM and IC.

2) The UK, the USA and other European countries have provided many policy initiatives in PCC/ICC as well as how to utilise PROM/PREM to improve quality of care. However, the evidence-base for most of the policy initiatives is still emerging.

3) The recent review by the World Health Organisation(WHO) showed that the cost-analysis effect of IC is weak and it further highlighted the importance of measuring patient experience in its own right in IC.

4) There is a generational change in developing and testing PROMs/PREMs, culminated by the success of the NIH funded PROMIS project which is based on IRT and CAT.

5) There is a growing number of different methods to capture patient experience with the growing popularity of social media. More research is needed to develop and evaluate different models/methods to capture patient experience and inform policy.

6) Despite the ‘perfect storm’ in transforming health care promised by the combination of PCC, PROM/PREM and social media, the risks and benefits of different policies should be carefully assessed and managed.

7) Despite the increasing number of PROMs/PREMs there is a need to assess the methodological properties of PROMs/PREMs in accordance with internationally recognised guidelines such as COSMIN before making a decision.

8) There is a good evidence-base for adopting one of the generic HRQoL questionnaires in primary care or the ICU setting. The final decision may be reached through a working panel and a chosen consensus-reaching process in NSW.

9) More disease- or condition- specific PROMs/PREMs could be added to the system. The criteria used, the process (how patients and stakeholders are involved) and the results (how results are interpreted and fed back to stakeholders, including patients) should be clearly specified.

10) One important factor to consider in choosing any particular PROM/PREM is its utility to patients, providers and health systems and its ability to affect change at different levels.

11) There is currently no international consensus on definitions and consisting domains of many popular terms such as “PCC” or ‘IC”. Health jurisdictions have to define their patient experience frameworks according to their value propositions on different underlying domains.

12) NSW Health could pioneer the work to operationalise the indicators of patient experience measures on IC in an Australian setting, learning from the existing experience and tools developed in the UK and by the AHRQ.

13) Critical leadership as well as strategy is needed to include psychosocial and behaviour variables in PROM/PREM, integrate PROM/PREM with EHRs, and build a sustainable big data infrastructure with linkage to other sources of data.

14) Challenges remain in generating solid evidence on IC and the effectiveness of PROMs/PREMs which necessitate a committed and robust research agenda.

8 | Appendices


8.1 Appendix 1 – The list of systematic reviews on HR-PRO by COSMIN (489) (See attached document)

8.2 Appendix 2 - Methodological issues related to the measurement properties of the PROMs/PREMs instruments (COSMIN)

COSMIN is an international initiative aimed to improve the selection of health measurement instruments.204-208 As part of this initiative, the COSMIN group developed a critical appraisal tool (a checklist) containing standards for evaluating the methodological quality of studies on the measurement properties of health measurement instruments (Figure 8.2-1). The COSMIN checklist was developed in an international Delphi study as a multidisciplinary, international collaboration with all relevant expertise involved. The focus was on HR-PROs, but the checklist is also useful for evaluating studies on other kinds of health measurement instruments such as performance-based instruments or clinical rating scales.

The COSMIN checklist can be used to evaluate the methodological quality of studies on measurement properties, for example in systematic reviews of measurement properties. In systematic reviews it is important to take the methodological quality of the selected studies into account. If the results of high quality studies differ from the results of low quality studies, this can be an indication of bias. The COSMIN checklist can also be used as a guide for designing or reporting a study on measurement properties.

COSMIN provides detailed definitions of each measurement property of an HR-PRO.

Figure 8.2-1 The diagram for completing the COSMIN checklist

Mark the properties that have been

assessed in the article.

INSTRUCTIONS FOR COMPLETING THE COSMIN CHECKLIST

Are IRT methodsused in the article?

A. Internal consistence �B. Reliability �C. Measurement error �D. Content validity (including face validity) � Construct validityE. Structural validity �F. Hypotheses- testing �G. Cross-cultural validity �H. Criterion validity �I. Responsiveness �J. Interprebility �

� Yes

Complete IRT box

Complete for each property you marked in

step 1 the corresponding box A to J

Complete for each property you marked in

step 1 theGeneralisability box

Step 1 Step 2 Step 3 Step 4

� No

Step 1 is to determine which measurement properties are evaluated in an article.

Step 2 is to determine if the statistical methods used in the article are based on Classical Test Theory (CTT) or on IRT.

Step 3 is to evaluate the methodological quality of the studies on the properties identified in step 1.

Step 4 is to assess the generalisability of the results of the studies on the properties identified in step 1.

A detailed description of how to use the checklist, a rationale for each item and suggestions for scoring the items are provided in the COSMIN manual.

Figure 8.2-2 COSMIN taxonomy of relationships of measurement properties. Abbreviations: HR-PRO, health related-patient reported outcome205

Quality of a HR-PRO

Validity

Constructvalidity

ResponsivenessCriterionvalidity(concurrent

validity, predictivevalidity)

Contentvalidity

Reliability

Interpretability

Responsiveness

Structuralvalidity

Cross-culturalvalidity

Hypotheses-testing

Facevalidity

Internalconsistency

Measurementerror

(test-retest, inter-rater, intra-rater)

Reliability(test-retest, inter-rater,

intra-rater)


Table 8.2-1 Definitions of domains, measurement properties, and aspects of measurement properties205

Term

Domain Measurement property

Aspect of a measurement property

Definition

Reliability The degree to which the measurement is free from measurement error

Reliability (extended definition) The extent to which scores for patients who have not changed are the same for repeated measurement under several conditions: for example, using different sets of items from the same HR-PROs (internal consistency), over time (testeretest) by different persons on the same occasion (interrater) or by the same persons (i.e., raters or responders) on different occasions (intrarater)

Internal consistency

The degree of the interrelatedness among the items

Reliability The proportion of the total variance in the measurements which is because of ‘‘true’’a differences among patients

Measurement error

The systematic and random error of a patient’s score that is not attributed to true changes in the construct to be measured

Validity The degree to which an HR-PRO instrument measures the construct(s) it purports to measure

Content validity The degree to which the content of an HR-PRO instrument is an adequate reflection of the construct to be measured

Face validity The degree to which (the items of ) an HR-PRO instrument indeed looks as though they are an adequate reflection of the construct to be measured

Construct validity

The degree to which the scores of an HR-PRO instrument are consistent with hypotheses (for instance with regard to internal relationships, relationships to scores of other instruments, or differences between relevant groups) based on the assumption that the HR-PRO instrument validly measures the construct to be measured

Structural validity

The degree to which the scores of an HR-PRO instrument are an adequate reflection of the dimensionality of the construct to be measured

Hypotheses testing

Idem construct validity

Cross-cultural validity

The degree to which the performance of the items on a translated or culturally adapted HR-PRO instrument are an adequate reflection of the performance of the items of the original version of the HR-PRO instrument

Criterion validity The degree to which the scores of an HR-PRO instrument are an adequate reflection of a ‘‘gold standard’’


Term

Domain Measurement property

Aspect of a measurement property

Definition

Responsiveness The ability of an HR-PRO instrument to detect change over time in the construct to be measured

Responsiveness Idem responsiveness

Interpretabilityb The degree to which one can assign qualitative meaning -that is, clinical or commonly understood connotations - to an instrument’s quantitative scores or change in scores.

Abbreviations: HR-PROs, health-related patient-reported outcomes; CTT, classical test theory.

a: The word ‘‘true’’ must be seen in the context of the CTT, which states that any observation is composed of two components a true score and error associated with the observation. ‘‘True’’ is the average score that would be obtained if the scale was given an infinite number of times. It refers only to the consistency of the score and not to its accuracy (ref Streiner & Norman244).

b: Interpretability is not considered a measurement property but an important characteristic of a measurement instrument.

The COSMIN provides also a checklist (Box A-J) and gives a guided procedure to check design issues as well as statistical methods issues (if applicable) to each measurement property within every domain. Note that the COSMIN covers not only CTT but also IRT on both the design and statistical methods.

As of November 2014, the dedicated COSMIN website includes 489 systematic reviews on the health measurement that can be downloaded. A list of these systematic reviews is included (Appendix 1).

Another important work that provided more detailed quantitative guidance on assessing the measurement properties of PROM is by Terwee and colleagues (2007)245 and its main table is presented(Table 8.2-2).


Table 8.2-2 Quality criteria for measurement properties 245

Property Definition Quality criteriaa,b

1. Content validity The extent to which the domain of interest is comprehensively sampled by the items in the questionnaire

+ A clear description is provided of the measurement aim, the target population, the concepts that are being measured, and the item selection AND target population and (investigators OR experts) were involved in item selection;

? A clear description of above-mentioned aspects is lacking OR only target population involved OR doubtful design or method;

- No target population involvement;

0 No information found on target population involvement

2. Internal consistency

The extent to which items in a (sub)scale are intercorrelated, thus measuring the same construct

+ Factor analyses performed on adequate sample size (7 * # items and ≥100) AND Cronbach’s alpha(s) calculated per dimension AND Cronbach’s alpha(s) between 0.70 and 0.95;

? No factor analysis OR doubtful design or method;

- Cronbach’s alpha(s) <0.70 or >0.95, despite adequate design and method;

0 No information found on internal consistency

3. Criterion validity The extent to which scores on a particular questionnaire relate to a gold standard

+ Convincing arguments that gold standard is ‘‘gold’’ AND correlation with gold standard ≥ 0.70;

? No convincing arguments that gold standard is ‘‘gold’’ OR doubtful design or method;

- Correlation with gold standard <0.70, despite adequate design and method;

0 No information found on criterion validity.

4. Construct validity The extent to which scores on a particular questionnaire relate to other measures in a manner that is consistent with theoretically derived hypotheses concerning the concepts that are being measured

+ Specific hypotheses were formulated AND at least 75% of the results are in accordance with these hypotheses;

? Doubtful design or method (e.g., no hypotheses);

- Less than 75% of hypotheses were confirmed, despite adequate design and methods;

0 No information found on construct validity.

5. Reproducibility

5.1. Agreement The extent to which the scores on repeated measures are close to each other (absolute measurement error

+ MIC < SDC OR MIC outside the LOA OR convincing arguments that agreement is acceptable;

? Doubtful design or method OR (MIC not defined AND no convincing arguments that agreement is acceptable);

- MIC ≥ SDC OR MIC equals or inside LOA, despite adequate design and method;

0 No information found on agreement.


Property Definition Quality criteriaa,b

5.2. Reliability + ICC or weighted Kappa ≥ 0.70;

? Doubtful design or method (e.g., time interval not mentioned);

- ICC or weighted Kappa < 0.70, despite adequate design and method;

0 No information found on reliability

6. Responsiveness The ability of a questionnaire to detect clinically important changes over time

+ SDC or SDC < MIC OR MIC outside the LOA OR RR > 1.96 OR AUC ≥ 0.70;

? Doubtful design or method;

-SDC or SDC ≥ MIC OR MIC equals or inside LOA OR

RR < 1.96 OR AUC < 0.70, despite adequate design and methods;

0 No information found on responsiveness.

7. Floor and ceiling effects

The number of respondents who

achieved the lowest or highest possible score

+ <15% of the respondents achieved the highest or lowest possible scores;

? Doubtful design or method;

- >15% of the respondents achieved the highest or lowest possible scores, despite adequate design and methods;

0 No information found on interpretation

8. Interpretatability The degree to which one can assign qualitative meaning to quantitative scores

+ Mean and SD scores presented of at least four relevant subgroups of patients and MIC defined;

? Doubtful design or method OR less than four subgroups OR no MIC defined;

0 No information found on interpretation

MIC = minimal important change; SDC = smallest detectable change; LOA = limits of agreement; ICC = Intraclass correlation; SD, standard deviation.

a + = positive rating; ? = indeterminate rating; - = negative rating; 0 = no information available.

b Doubtful design or method = lacking of a clear description of the design or methods of the study, sample size smaller than 50 subjects (should be at least 50 in every (subgroup) analysis), or any important methodological weakness in the design or execution of the study


8.3 Appendix 3 – The sources of the instruments reviewed.

Instrument Abbreviation Name Objective

AQoL Assessment of Quality of Life

The AQoL is a multi-attribute utility measure for use in economic evaluation, measuring health-related quality of life. The descriptive system can be used to provide health profiles.

NHP Nottingham Health Profile To provide a brief indication of a patient’s perceived emotional social and physical health problems

SF-12® / SF-12v2™

SF-36® / SF-36v2™

SF-12® Health Survey and SF-12v2™ Health Survey

SF-36® Health Survey and SF-36v2™ Health Survey

Developed to be a much shorter, yet valid, alternative to the SF-36 for use in large surveys of general and specific populations as well as large longitudinal studies of health outcomes

The SF-36 was developed during the Medical Outcomes Study (MOS) to measure generic health concepts relevant across age, disease, and treatment groups. The SF-12 is a subset of the Sf-36.

HUI®: HUI3 Health Utilities Index To describe health status, measure within-attribute morbidity and health-related quality of life, and produce utility scores

There are three versions of the HUI; HUI, HUI2 and HUI3.

WHOQoL-BREF World Health Organisation Quality of Life assessment instrument

To assess individuals’ perceptions on the quality of their life

QWB-SA Quality of Well-Being scale Self-Administered

To measure health-related quality of life, to monitor the health of populations over time, or to evaluate the efficacy and effectiveness of clinical therapies of practices using a preference-weighted self-administered measure

EQ-5D Euroqol EQ-5D To assess health outcome from a wide variety of interventions on a common scale, for purposes of evaluation, allocation and monitoring. Of note, the EQ-5D-3L consists of the original EQ-5D descriptive system and the EQ visual analogue scale (EQ-VAS).

HOWSYOURHEALTH? HowsYourHealth Research has shown shows that basic information tailored to the needs of the respondent and their doctor or nurses is most likely to make communication better, place everyone “on the same page”, and increase confidence with self-care.


Instrument Cost Notes: AQoL: “There are no fees associated with registration or use of any of the Centre for Health Economics AQoL instruments.” Information retrieved Jan 3, 2013 at: http://www.aqol.com.au/documents/v4_AQoL_User_Registration_Form_090710.pdf.

There are four identified versions of the AQoL available online: AQoL-8D; AQoL-7D; AQoL-6D; AQoL-4D. Information retrieved Jan 3, 2013 at: http://www.aqol.com.au/choice-of-aqol-instrument.html. Nottingham Health Profile: “No fees apply for commercial / pharmaceutical companies” Information retrieved Jan 3, 2013 at: http://www.outcomesdatabase.org/node/669

SF12 and SF36: “All use of QualityMetric health surveys, scoring algorithms, translations and benchmarking data requires a signed license agreement.” Information retrieved Jan. 6, 2103 at http://www.sf-36.org/wantsf.aspx?id=1 Licensing fee information request form available at: http://www.qualitymetric.com/RequestInformation/SurveyInformationRequest/tabid/263/Default.aspx (Retrieved Jan 6, 2013)

A sample of the SF12 can be reviewed (and completed), but not downloaded, at: http://www.qualitymetric.com/demos/TP_Launch.aspx?SID=52304 (Information retrieved Jan 6, 2013)

A sample of the SF-36 can be reviewed (and completed), but not downloaded, at: http://www.qualitymetric.com/demos/TP_Launch.aspx?SID=100 (Information retrieved Jan 6, 2013)

Health Utilities Index: Health Utilities Inc. does not grant permission for copies of its proprietary materials (e.g. questionnaires) to be distributed in grant proposals or reports. The HUI Application and Interpretation Package costs $CAN 4,000.00. This package includes initial consultation to determine the most appropriate questionnaires for use in a specific study and permission to use one version of an HUI questionnaire and the matching procedures manual set in one study. Additional questionnaires and manuals are available for use at a cost of $CAN 2,000.00 each per study. If the study requires more than one questionnaire the fee schedule becomes more complicated. For example, a study using two self-completed questionnaires (e.g. self-complete and self-assessed in both English and Spanish) will cost $CAN 6,000.00) while a study using a self-completed and an interviewer-administered questionnaire can cost $CAN 8,000.00 (fee reflects payment for one additional questionnaire and one additional manual). HUI grants permission for use of its

proprietary materials (e.g. instrumentation) one study at a time. (Additional information regarding the fee 55 schedule is on the web site: http://healthutilities.biz/fees.htm .Source: Email correspondence with Mr. Bill Furlong, General Manager Health Utilities Inc, 24/09/2012.)

WHOQoL-BREF can be downloaded freely from the World Health Organisation website. Information retrieved January 3, 2013 at: http://www.who.int/substance_abuse/research_tools/whoqolbref/en/ . The US WHOQoL Center distributes the WHOQoL-BREF US English Version (June 1997), with scoring instructions, free of charge as electronic files. Information retrieved Jan 3, 2013 at: http://depts.washington.edu/seaqol/WHOQOL-BREF

Quality of Well-Being Scale – Self-Administered (QWB-SA): Copyright Agreement For Non-Profits appears to indicate free use with restrictions. Information retrieved Jan 6, 2013 at: https://hoap.ucsd.edu/qwb-info/NotforProfit-Copyright.pdf . Terms and cost for use by non-profits indicates, “Scoring - ◦HSRC Scoring - $57/hr or, an algorithm for scoring can be purchased for $240. The scoring instructions are free of charge. Information retrieved Jan 6, 2013 at: https://hoap.ucsd.edu/qwb-info/price-nonprofit.aspx Quality of Well-Being Scale – Self Administered (QWB-SA) tool can be downloaded for review at https://hoap.ucsd.edu/qwb-info/# (Retrieved Jan 6, 2013). There are two English versions. English 2-page paper version “QWB-SA V1.04” available at: https://hoap.ucsd.edu/qwb-info/EnglishQWB-SA_2.pdf (Retrieved Jan 6, 2013) English 4-page paper version “QWB-SA V1.04” available at: https://hoap.ucsd.edu/qwb-info/EnglishQWB-SA_4.pdf (Retrieved Jan 6, 2013)

EQ-5D: “Licensing fees are determined by the EuroQol Executive Office on the basis of the user information provided on the registration form. The amount is dependent upon the type of study/trial/project, funding source, sample size and number of requested languages. You are not obligated to purchase by registering.” Information retrieved Jan 3, 2013 at: http://www.euroqol.org/eq-5d-products/how-to-obtain-eq-5d.html The User Guide can be downloaded from www.euroqol.org homepage.

PROMIS Short Form v1.0 - Global Health Scale can be obtained for free by email request. Information retrieved Jan 3, 2013 at: https://www.assessmentcenter.net/PromisForms. aspx . While it appears use of the GHS is free, users must “agree that the PROMIS Health Organisation and PROMIS Cooperative Group provide access to PROMIS instruments (e.g. item banks, short forms, profile measures) subject to the PROMIS Terms and Conditions (PTAC).” Information retrieved Jan 3, 2013 at: http://www.assessmentcenter.net/documents /PROMIS%20Terms%20and% 20Conditions%20v7.3.pdf.


8.4 Appendix 4 – Measuring Patient Experiences in Primary Health Care Survey (Canadian) (see attached document)

8.5 Appendix 5 – The June 2014 update of the AHRQ review results of the instrument in the care coordination measurement (Care Coordination Atlas) (see attached document)

8.6 Appendix 6 – Social Media and IC

8.6.1 Using social media to capture patient experience There is a growing number of studies that used social media as a means to capture the patient experience. The “traditional” methods have significant limitations which include social desirability bias, a time lag between experience and measurement and difficulty reaching large groups of people. Information on social media could be of value to overcome these limitations as this type of media are easy to use and are used by the majority of the population. Furthermore, an increasing number of people share health care experiences online or rate the quality of their health care provider on physician rating sites.

The question is whether this information is relevant to determining or predicting the quality of health care and how to maximise the value and minimise the risk of this information. Verhoef and colleagues (2014)246 recently conducted a scoping review which aimed to :

(1) map the literature on the association between social media and quality of care,

(2) identify different mechanisms of this relationship, and

(3) determine a more detailed agenda for this relatively new research area.

The authors describe and categorise according to the type of social media used on related research questions as well as consulting national and international stakeholders on the study results. The study identified 29 articles, of which 21 were concerned with health care rating sites. Several studies indicated a relationship between the information on social media and quality of health care. However, some limitations exist, especially regarding the use of rating

sites. For example, since rating is anonymous, rating values are not risk adjusted and therefore vulnerable to fraud. Also, ratings are often based on only a few reviews and are predominantly positive. Furthermore, people providing feedback on health care via social media are presumably not always representative of the patient population.

Social media and particularly rating sites are an interesting new source of information about quality of care from the patient’s perspective. However, this new source should be used to complement traditional methods, since measuring quality of care via social media has other serious limitations. Future research should explore whether social media is suitable in practice for patients, health insurers, and governments to help them judge the quality performance of professionals and organisations.

8.6.2 Social media in chronic disease managementThere is also increased interest in using social media in chronic disease management. Merolli and colleagues (2013)247 reviewed the literature on the effect of using social media in chronic disease management. Amongst the 19 studies identified, 10 studies were from USA; 6 from Europe and 3 from Asia/Pacific. Of these studies, 10 can be classified as “online support groups’ (OSG; 10) such as discussion forums, bulletin boards and chat tools; 5 as social network sites (SNS; 5), blogs (1) and virtual worlds (1).

Of the studies that focused on a single chronic condition (N = 11), five explored cancer (two specifically breast cancer)248-252, three examined chronic pain related conditions 253-255, one studied HIV/AIDS256, one was on diabetes257 and one on rheumatoid arthritis.258 The remaining studies (N = 8), either explored multiple chronic disease groups or examined chronic disease in general.259-266

Of these studies, 12 occurred on earlier social platforms249

250 252-254 256 259 261 263-266 such as OSG. Three of the five SNS studies were created specifically for the purpose of an intervention251 255 258, the others utilised the existing platforms of Facebook257 and DailyStrength260.

Various reasons for using SNS included: enhancing self-management of disease255; driving support and social interaction251 258; ‘friending’ other participants255 ; video-sharing service (as well as user profile creation, photos and narratives)251; fundraising, awareness, promotions and support267(the pre-existing SNS, Facebook). In the case of patients with type 1 diabetes, it appears that group members primarily use the platform to share information, request information from others and offer each other support.257 DailyStrength260, a health-related


SNS for patients and carers, centers around the formation of different support groups. Analysis of the conversation content from support groups for breast cancer, diabetes and fibromyalgia found the most common usage patterns centered around ‘support’. The authors suggest that in poorly understood and socially stigmatised chronic diseases, support is the main activity or conversation to occur, as well as coping and fitting within society.260

One study used blogging as a form of emotional catharsis in chronic disease management.248 It appears that breast cancer sufferers are one patient population that have the most to gain from narrating their experiences. For cancer sufferers, blogging has been used as a means to self-manage emotions, problem-solve and share information.248

One study investigated the potential of delivering psychosocial interventions via a virtual world, such as Second Life.262 It is suggested that virtual environments may provide an ideal platform for the delivery of these interventions for conditions such as depression and other chronic diseases involving significant psychosocial components.262


Table 8.6-1 Platforms and reported effects/outcomes.

OSG Discussion forum/bulletin board/chat

SNS Blogs Virtual worlds

Custom Facebook DailyStrength

Engagement/participation 264 251 268

Social interactions

– Peer support 252 256 259 266 269

253 264 251 268 257 260

– Empowerment: (Includes: better informed, social well-being, confidence in treatment, confidence with practitioner, self-esteem, network building, acceptance & belonging, understanding, validation, optimism, control)

252 256 259 266 269

253 254 251 248

– Information sharing 252 256 259 266 269

253 254 251 257 260 248

Disease-specific knowledge 44 255 258

Psychosocial impacts

– Emotional burden

– Catastrophising

– Pain-induced fear 255

– Depression 252 259 265 266

255 248 262

– Anxiety 249 250 252 261

255 262

– Stress 252 259 255 262

– Emotional expression & distress 252 263 255 262

– Coping 256 255

– Self-efficacy 256 263

– QOL (psychosocial) 256 263

Physical condition impacts 263 264 255

–Pain severity 264 255

– Pain-related interference/limitation 255

– Perceived disability 255

– Functional limitation 264

– QoL (physical) 256

(Adapted from: Merolli et al.; 2014247)


Few studies have investigated social media’s potential in chronic disease. However, the existing studies showed a positive impact on health status and patient experience with none indicating any adverse events. Benefits have been reported for psychosocial management via the ability to foster support and share information. There is less evidence of benefits for physical condition management. These studies covered a very limited range of social media platforms and there is an ongoing propensity towards reporting investigations of earlier social platforms, such as OSG, discussion forums and message boards. For social media to form a more meaningful part of effective chronic disease management, interventions need to be tailored to the individualised needs of patients, in particular with respect to identity, flexibility, structure, narration and adaptation. High methodological quality is required to investigate how social media can best serve chronic disease management.247

8.6.3 Silver lining of the ‘cloud of patient experience’270 in a ‘perfect storm’271?A growing number of patients are using the internet to describe their experiences of healthcare.272 A recent study showed a consistent increase in online voting on the NHS Choices website272 (Figure 8.6-1). Comparison between structured patient rating data online and patient surveys has already shown significant associations at the hospital level.273 274 The increasing availability of patients’ accounts of their care on blogs, social networks, Twitter and hospital review sites presents an unprecedented yet intriguing opportunity to advance PCC agenda and provide novel quality of care data. Greaves and colleagues (2013) described this concept as the ‘cloud of patient experience’.270 The authors argued that the collection and aggregation of patients’ descriptions of their experiences on the internet could be used to detect poor clinical care. The techniques of natural language processing and sentiment analysis can be used to transform unstructured descriptions of patient experience on the internet into usable measures of healthcare performance. These analyses, if automated, can provide real-time feedback into the healthcare system. In another commentary, Rozenblum and Bates (2013) 271 contends that the three domains—patient-centered healthcare, social media and the internet— are beginning to come together and have the potential to create a major shift in how patients and healthcare organisations connect, in effect, the ‘perfect storm’, a phrase that has been used to describe a situation in which a rare combination of circumstances result in an event of unusual magnitude.275

Figure 8.6-1 Cumulative number of online ratings of hospitals in England on the NHS Choices website.

Aug 0

8

25000

20000

15000

10000

5000

0

Oct 0

8

Dec 0

8

Feb 0

9

Apr 0

9

Jun 0

9

Aug 0

9

Oct 0

9

Dec 0

9

Feb 1

0

Apr 1

0

Jun 1

0

Aug 1

0

Oct 1

0

Dec 1

0

Feb 1

1

Apr 1

1

Jun 1

1

Aug 1

1

Oct 1

1

Dec 1

1

Cumulative number of hospital ratings

However, despite the great potential of harnessing the ‘cloud of patient experience’, the field is relatively new particularly in the healthcare sector. A study by Naslund and colleagues (2014)276 used qualitative inquiry informed by emerging techniques in online ethnography and analysed 3,044 comments posted in 19 videos on YouTube which were uploaded by individuals who self-identified as having schizophrenia, schizoaffective disorder, or bipolar disorder. The authors found peer support across four themes: minimising a sense of isolation and providing hope; finding support through peer exchange and reciprocity; sharing strategies for coping with day-to-day challenges of severe mental illness; and learning from shared experiences of medication use and seeking mental health care. These broad themes are consistent with accepted notions of peer support in severe mental illness as a voluntary process aimed at inclusion and mutual advancement through shared experiences and developing a sense of community. The study results suggested that the lack of anonymity and associated risks of being identified as an individual with severe mental illness on YouTube seemed to be overlooked by those who posted comments or uploaded videos. However, the authors believed that whether or not this platform can provide benefits for a wider community of individuals with severe mental illness remains uncertain.


The various sources of information that could be used in harnessing the ‘cloud of patient experience’ and the strengths and limitations of the approach are presented (Table 8.6-2).

Table 8.6-2 Potential sources of information for the ‘cloud of patient experience’

Type of source Examples Information that could be used

Advantages Disadvantages

Rating and feedback websites

RateMDs277

Patient Opinion278

Iwantgreatcare279

Ratings and free text descriptions of healthcare providers and individual clinicians

Comments usually directly relate to care experience

Comparatively low usage, possibility of deliberate gaming

Patient networks, discussion fora and blogs

Patientslikeme280 Mumsnet281

Epatients.net282

Patients’ and carers’ shared descriptions of their care and experiences

Authentic voice of the patient, often well used in specific patient communities

May be a selection bias towards particular demographics (with higher socio-economic status) or interest groups

Micro-blogs Twitter283 Tweets (short messages) directed towards hospitals or care providers

High volume of traffic, often tagged with service they relate to

Short, unstructured messages may contain minimal information about care quality

Social networks Facebook284

Google+285

Comments left on hospital or friends’ pages about care or specific signals of appreciation (eg likes, ‘+1’s)

High membership and usage by the public

Public rarely talks about healthcare on these platforms; Content may be from employees rather than patients

(Adapted from: Greaves et al. 2013 270)


The NHS’s public reporting website—known as NHS Choices—has been incorporating anecdotal comments from patients regarding primary and hospital care since 2007286. Publicly reporting patients’ narrative comments along with numerical ratings of their experience and clinical quality metrics presents opportunities as well as challenges for reporting systems (Table 8.6-3).

Table 8.6-3 Questions Asked for Anecdotal Comments and Ratings on NHS Choices

Question type Hospitals Primary care practices

Ratings on a scale (all on a 1-5 Likert-type scale)

How likely are you to recommend this hospital to friends and family if they needed similar care or treatment?

How satisfied are you with the cleanliness of the area you were treated in?

How satisfied are you that the hospital staff worked well together?

How satisfied are you that you were treated with dignity and respect by the staff?

How satisfied are you that you were involved in decisions about your care?

How satisfied are you that the hospital provides same-sex accommodation?

How likely are you to recommend this GP surgery to friends and family if they needed similar care or treatment?

Are you able to get through to the surgery by telephone? Are you able to get an appointment when you want one?

Do the staff treat you with dignity and respect?

Does the surgery involve you in decisions about your care and treatment?

This GP practice provides accurate and up to date information on services and opening hours.

Free text commentary Give your opinion in your own words. The more detail you can give, the more useful your review will be.

Give your opinion in your own words. The more detail you can give, the more useful your review will be.

(Source: Greaves et al. 2014286)

Greaves and colleagues (2014)286 provided an in-depth review of the five key design considerations for publicly reporting anecdotal comments—including how to collect, moderate and display comments and how to encourage the public and the health care providers to use them which was based on the UK experience. The authors concluded that while anecdotal comments might represent an untapped seam of valuable information about service quality and a potential hook for engaging patients to use comparative performance data, the jury is still out on where narrative comments fit in the complex landscape of quality measurement and reporting.

Another study (also by Greaves and colleagues, 2014)287 examined whether Tweets sent to NHS hospitals contained information about quality of care. The authors also compared sentiment on Twitter regarding hospitals with established survey measures of patient experience and standardised mortality rates. Using a mixed-method study including a quantitative analysis of all 198,499


tweets sent to English hospitals over a year and a qualitative directed content analysis of ,1000 random tweets, the authors found that 11% of tweets to hospitals contained information about quality of care, with the most frequent topic being patient experience (8%). Comments on effectiveness or safety of care were present, but less common (3%). Of all quality of care related tweets, 77% were positive in tone. Other topics mentioned in tweets included messages of support to patients, fundraising activities, self-promotion and dissemination of health information. No associations were observed between Twitter sentiment and conventional quality metrics.

The study showed that only a small proportion of tweets directed at hospitals discussed quality of care and there was no clear relationship between Twitter sentiment and other measures of quality, potentially limiting Twitter as a medium for quality monitoring. However, tweets did contain useful information to target quality improvement activity. Given the fact that the UK Department of Health’s Information Strategy suggests the use of social media aggregation and sentiment analysis to provide a rapid indicator of hospital performance and early warnings of poor care288, and that the NHS recently started aggregating and publishing social media sentiment about the NHS on a public website289, it should be noted that the recent enthusiasm by policy makers to use social media as a quality monitoring and improvement tool needs to be carefully considered and subjected to formal evaluation.287

The current evidence suggests that the real-time patient feedback harnessed from social media cannot be deemed as a perfect test of clinical performance for an organisation. However, the soft intelligence provided by this proposed approach— capturing and processing the cloud of patient experience— offers another way to look at health quality; and not just clinical quality but areas such as dignity and respect, cleanliness of the care environment, timeliness and efficiency of care, as well as ideas for improvement. In particular, the potential to channel the wisdom of patients in real-time to create unique insights into the quality and safety of care, without expensive new infrastructure, is appealing.

8.7 Appendix 7 - Is IC cost-effective?There are two separate important reviews by researchers from the WHO7 and Centre for Health Economics, University of York290 on the cost-effectiveness of IC . Despite the high expectation on the potential cost-savings promised by the concept of ‘IC’, both reports found that there were relatively weak or no evidence to support the notion that IC is cost-effective and many reasons have been offered to explain why. The reports also highlighted the fact that there is a paucity of high quality research on the effectiveness and cost-effectiveness of different IC models and the results from both reviews perhaps should be interpreted as a result of ‘absence of evidence’ instead of ‘evidence of absence’.

The key messages from the WHO review report on the cost-effectiveness of IC (paraphrased from Nolte & Pitchforth 20147) include:

z The rising burden of chronic disease and the number of people with complex care needs require the development of delivery systems that bring together a range of professionals and skills from both the cure (healthcare) and care (long-term and social-care) sectors.

z Evidence that is available points to a positive impact of IC program on the quality of patient care and improved health or patient satisfaction outcomes but uncertainty remains about the relative effectiveness of different approaches and their impacts on costs.

z This review of published reviews confirms earlier reports of the shortage of robust evidence on the economic impact of IC.

z The term ‘IC’ is often not specifically examined; the most common concepts or terms were case management, care coordination, collaborative care or a combination of these.

z Utilisation and cost were the most common economic outcomes assessed by reviewers but reporting of measures was inconsistent and the quality of the evidence was often low.

z There is evidence of cost–effectiveness of selected IC approaches but the evidence base remains weak.

z There may be a need to revisit our understanding of what IC is and what it seeks to achieve and the extent to which the strategy lends itself to evaluation in a way that would allow for the generation of clear cut evidence.

z It is important to come to an understanding as to whether IC is to be considered an intervention or whether it is to be interpreted and evaluated as a complex strategy to innovate and implement long-lasting change in the way services in the health and social-care sectors are being delivered and which involve multiple changes at multiple levels.


8.8 Appendix 8 - Patient experience framework

In order to operationalise the measurement of patient experience, a framework needs to be formed in order to guide such activities. The framework needs to be specific, clearly defined and operable given its underlying concepts. A number of generic frameworks have been developed, among them are the Picker principles (developed from the work of Gertis and colleagues291), the IOM framework, the National Health Council (2004) and the International Alliance Patients’ Organisations292-296. Some of these frameworks are outlined below.

8.8.1 The Picker principles Picker Europe and Picker Institute (USA) are dedicated to advancing the principles of PCC. The eight Picker Principles of PCC are:

1) access to reliable health

2) effective treatment by trusted professionals

3) participation in decisions and respect for preferences

4) clear, comprehensive information and support for self-care

5) attention to physical and environmental needs

6) emotional support, empathy and respect

7) involvement of and support for family and carers

8) continuity of care and smooth transitions.

8.8.2 NHS Patient Experience FrameworkIn October 2011, the NHS National Quality Board (NQB) agreed on a working definition of patient experience to guide the measurement of patient experience across the NHS. This framework outlines those elements which are critical to the patients’ experience in NHS Services. As noted by the Department of Health, this framework is based on a modified version of the Picker Institute Principles of PCC, an evidence-based definition of a good patient experience.297

z Respect for patient-centered values, preferences and expressed needs, including: cultural issues; the dignity, privacy and independence of patients and service users; an awareness of QoL issues; and shared decision making.

z Coordination and integration of care across the health and social care system.

z Information, communication and education on clinical status, progress, prognosis, and processes of care in order to facilitate autonomy, self-care and health promotion.

z Physical comfort including pain management, help with the activities of daily living, and clean and comfortable surroundings.

z Emotional support and alleviation of fear and anxiety about such issues as clinical status, prognosis and the impact of illness on patients, their families and their finances.

z Welcoming the involvement of family and friends, on whom patients and service users rely, in decision-making and demonstrating awareness and accommodation of their needs as care-givers.

z Transition and continuity as regards information that will help patients care for themselves away from a clinical setting and coordination, planning and support to ease transition.

z Access to care with attention, for example, to time spent waiting for admission or time between admission and placement in a room in an in-patient setting and waiting time for an appointment or visit in the out-patient, primary care or social care setting.

The UK government has defined ‘IC’ as ‘person-centred coordinated care’ which was based on the narratives developed by National Voices.298 The accompanying narrative of ‘I statements’ setting out a user-based perspective of how IC should be experienced (National Voices 2013).298 The key dimensions of ‘IC’ were also described (Figure 8.8-1 ). The government expects this definition of integration as coordinated and personalised care, to be adopted and delivered by all localities through health, social care and other services.

Figure 8.8-1 Dimensions included in IC (i.e. person centered coordinated care) narratives (National Voice, 2013)

Person centered coordinated care

I can plan my care with people who work together to understand me and my carer(s), allow me control, and bring together services to achieve the outcomes important to me.”

Information

Communication Transitions Decision making

My goals/outcomesCare planning

It should be noted such a definition is also extremely complex and imposes the perspective of users as its organising principle and stands in contrast to ‘service integration’ models which focus more on the interactions between, or across specific services, such as health and social care.191 298


8.8.3 IOM Patient experience framework The IOM report defined the PCC as one of the key components of quality of care dimension. The narrative commentary on each of the themes was provided by Staniszewska and colleagues (2014).296

Table 8.8-1 A narrative commentary on IOM patient experience framework

IOM theme Narrative commentary

Compassion, empathy and responsiveness

Compassion and empathy were both important themes but appeared in more subtle forms within a number of wider generic themes, e.g. communication. Responsiveness emerged as a generic theme but was focused on the responsiveness of the service and the need for an individualised approach.

Co-ordination and integration

These themes were important but fitted more appropriately into the wider generic themes of continuity of care and responsiveness.

Information, communication and education

Information and communication emerged as two key themes but were separated to reflect the different content of the sub-themes identified. Education appeared in a number of the generic themes in different ways, including within support and information.

Physical comfort Physical comfort was important but appeared in other more substantive generic themes, including responsiveness and lived experience.

Emotional support, relieving fear and anxiety

Emotional support was included in a much larger category of support. Elements of fear and anxiety were more subtle and appeared as part of a broader lived experience.

Involvement of family and friends

The role of family and friends was important and appeared in broader themes of lived experience and support.

(Source: Staniszewska and colleagues (2014)299)

8.8.4 The Warwick patient experience framework (WaPEF)The National Institute for Health and Clinical Excellence (NICE) produces clinical guidelines, quality standards, public health and health technology appraisal guidance for the NHS in the UK. NICE commissioned the National Clinical Guideline Centre (NCGC) to produce clinical guidance and a quality standard on patient experience in adult NHS services. There was uncertainty about the robustness of published frameworks, as part of the process in developing the NICE guideline and quality standards ‘Patient experience in adult NHS services: improving the experience of care for people using adult NHS services’.300. A scoping review was undertaken by the Royal College of Nursing Research Institute at the University of Warwick which, in collaboration with the NCGC, to identify the existence of qualitative evidence of patient experience in adult cancer, diabetes and cardiovascular disease populations. As a result, based on the IOM framework, a new patient experience framework (namely, the Warwick Patient Experience Framework (WaPEF)) was developed and described 299 (See Table 8.8-2).


Table 8.8-2 The Warwick Patient Experiences Framework

Generic theme Narrative description

Patient as active participant

Reflects the role of patients as potential active participants in their health care, co-creators and co-managers of their health and use of services; responsible for self-care, participators in health care, shared decision-makers, self-managers, risk managers and life-style managers.

Confidence in self-management is critical

Associated with issues of power and control. Responsiveness of services— an individualised approach

Needing to be seen as a person within the health care system

The responsiveness of health services in recognising the individual and tailoring services to respond to the needs, preferences and values of patients, taking into account both shared requirements and individual characteristics (such as individuals’ expectations of service cultural background, gender, subtle issues such as preferences for humour). Includes how well clinical needs are met (e.g. pain management) and evaluation of how well services perform from a patient perspective.

Lived experience The recognition that individuals are living with their condition and experiencing it in a unique way, that family and broader life need to be taken into account and that all of these aspects of lived experience can affect self-care. Taking into account individual physical needs and cognitive needs because of condition. Everyday experiences, hopes, expectations, future uncertainty, feelings of loss, feelings of being morally judged and feelings of blame. Some of these experiences originate ‘outside’ of the health care system but are brought with the patient into the health system; other experiences may be affected by attitudes and expectations of health professionals.

Continuity of care and relationships

Initiating contact with services, interpretation of symptoms, co-ordination, access (barriers to), and availability of services, responsiveness of services and feelings of abandonment (when treatment ends or support is not made available). Being known as a person rather than ‘a number’. Trust in health care professional built up over time. Recognition/questioning of expertise of health care professional. Respect, including respect for patient’s expertise. Partnership in decision-making. Issues of power and control.

Communication Needing to be seen as an individual; communication style and format (e.g. over telephone or in person); skills and characteristics of health care professional; body language (which can convey different information from that spoken); two-way communication and shared decision-making; compassion, empathy; the importance of the set-up of consultation (e.g. appropriate time for questions, appropriate physical environment and number of peoples present). Listening and paying attention to the patient. Enabling questions and providing answers.

Information Information to enable self-care and active participation in health care, importance of information in shared decision-making, tailored information to suit the individual, patient wanting/not wanting information and timely information. Sources of information, including outside the health service (e.g. peer-support, internet). Quality of information. Sources of further information and support. Developing knowledge and understanding, and making sense of one’s health.

Support Different preferences for support: Support for self-care and individual coping strategies. Education. Need for emotional support, and need for hope. Responsiveness of health care professionals to individual support needs (may vary according to gender, age and ethnicity). Importance of peer-support, groups and voluntary organisations. Practical support. Family and friends support. Role of advocacy. Feeling over-protected, not wanting to be a burden.


The WaPEF summarises a complex patient experience evidence base. The narrative description of each theme in Table 8.82 is illustrative, rather than exhaustive. Furthermore, the themes and sub-themes contained in the generic framework are complex and many connections exist between them.

Despite the recommendation by the authors on the possible international use of such a framework, it was suggested that the future international use of WaPEF should incorporate separate reviews of themes and subthemes in particular country settings or populations, as WaPEF has a particular relevance for the UK setting. In addition, future studies should include a more rigorous quality assessment of the studies included in the evidence base.

8.8.5 The AHRQ ‘care coordination’ framework All the above frameworks are generic and intended for the patient population as a whole without specifically aiming for ‘IC’ or ‘care coordination’ except for the National Voice framework (Figure 8.8-1). There is growing support of ‘care coordination’ concepts and practice in the USA by several national organisations including the AHRQ, the IOM, and the American College of Physicians, amongst others. While evidence is starting to build about the mechanisms by which care coordination contributes to patient-centered high-value, high-quality care, the health care community is currently struggling to determine how to measure the extent to which this vital activity is, or is not, occurring.

To address this challenge, the AHRQ launched a research project in 2010 aiming for developing an atlas to help evaluators identify appropriate measures for assessing care coordination interventions in research studies and demonstration projects, particularly those measures focusing on care coordination in ambulatory care. This effort involved three steps: 1) developing a ‘care coordination’ framework; 2) identifying all measures that include the domains and components of the ‘care coordination’ framework; 3) mapping different measurement instruments into the framework.

In order to develop such a framework in step 1, research was conducted to identify the mechanisms of the coordinated care which can be classified as either coordination activities or broad approaches. The detailed coordination activities (such as facilitate transitions between different care settings) and broad approaches (such as teamwork based coordination) were further conceptualised. Both coordination activities and broad approaches were then stratified by different perspectives (i.e. patients, providers and health system representatives). Such a framework made the review and mapping of different measurement instruments explicit. A diagram illustrating this process is presented in Figure 8.8-2 and the detailed domains/components in Table 8.8-3. Furthermore, the full definitions of all components were described in the original report and its conceptual source

of both ‘coordination activities’ and ‘broad approaches’ were presented (Table 8.8-4).

The original mapped measurement instruments were 62 and updated to 82 in June 2014.

Figure 8.8-2 Care Coordination Measurement Framework Diagram

Coordination measures

Goal: Coordinated care (see Chapter 2)

MechanismsMeans of achieving goal

Coordination activitiesActions hypothesised to support

coordination. Not necessarily executed in any structured way.

Coordination e�ectsExperienced in di�erent waysdepending on the perspective

Context: Settings, patient populations, timeframe, facilitators, barriers

Patient/familyperspective

Health careprofessional(s)

perspective

Systemrepresentative

perspective

Broad approachesCommonly used groups of

activities and/or toolshypothesised to support

coordination

Table 8.8-3 Mechanisms for Achieving Care Coordination (Domains)



Communicate

Facilitate Transitions

Assess Needs and Goals

Create a Proactive Plan of Care


Support Self-Management Goals

Link to Community Resources


BROAD APPROACHES

Teamwork Focused on Coordination

Health Care Home

Care Management

Medication Management

Health IT-Enabled Coordination


Table 8.8-4 Relation Between the Care Coordination Measurement Framework and Other Key Sources




NQF: Communication domain includes – all medical home team members work within the same plan of care and are measurably co-accountable for their contributions to the shared plan and achieving the patient’s goals.

Communicate Antonelli: Care coordination competency – communicates proficiently; care coordination function – manages continuous communication.

NQF: Framework domain – Communication available to all team members, including patients and family.

Interpersonal Communication Coiera: All information exchanged in health care forms a space; the communication space is the portion of all information interactions that involves direct interpersonal interactions, such as face-to-face conversations, telephone calls, letters, and email.

Information Transfer MPR: Care coordination activity – send patient information to primary care provider.

NQF: Communication domain includes – availability of patient information, such as consultation reports, progress notes, test results, and current medications to all team members caring for a patient reduces the chance of error.

Facilitate Transitions Antonelli: Care coordination function – supports/facilitates care transitions.

CMS Definition of Case Management: Case management services are defined for transitioning individuals from institutions to the community.

NQF: Framework domain – transitions between and off settings of care are a special case because currently they are fraught with numerous mishaps that can make care uncoordinated, disconnected, and unsafe. Some care processes during transition deserve particular attention, including involvement of team during hospitalisation, nursing home stay, etc.; communication between settings of care; and transfer of current and past health information from old to new home.

Assess Needs and Goals Antonelli: Care coordination function – completes/analyzes assessments.

CMS Definition of Case Management: Case management includes assessment and periodic reassessment of an eligible individual to determine service needs, including activities that focus on needs identification, to determine the need for any medical, educational, social, or other services. MPR: Care coordination activity – assess patient’s needs and health status; develop goals.

Create a Proactive Plan of Care Antonelli: Defining characteristic of care coordination – proactive, planned and comprehensive; care coordination function – develops care plans with families; facile in care planning skills.

CMS Definition of Case Management: Case management assessment includes development and periodic revision of a specific care plan based on the information collected through an assessment or reassessment that specifies the goals and actions to address the medical, social, educational, and other services needed by the eligible individual, including activities such as ensuring the active participation of the eligible individual and working with the individual (or the individual’s authorised health care decision maker) and others to develop those goals and identify a course of action to respond to the assessed needs of the eligible individual.

MPR: Care coordination activity – develop a care plan to address needs.

NQF: Framework domain – Proactive Plan of Care and Follow-up is an established and current care plan that anticipates routine needs and actively tracks up-to-date progress toward patient goals.




Antonelli: Care coordination function – manages/tracks tests, referrals, and outcomes.

CMS Definition of Case Management: Case management assessment includes periodic reassessment to determine whether an individual’s needs and/or preferences have changed. Case management includes monitoring and follow-up activities, including activities and contacts that are necessary to ensure that the care plan is effectively implemented and adequately addresses the needs of the eligible individual. If there are changes in the needs or status of the individual, monitoring and follow-up activities include making necessary adjustments in the care plan and service arrangements with providers.

MPR: Care coordination activities – monitor patient’s knowledge and services over time; intervene as needed; reassess patients and care plan periodically.

NQF: Plan of Care domain includes – follow-up of tests, referrals, treatments, or other services.

Support Self-Management Goals

Antonelli: Defining characteristic of care coordination – promotes self-care skills and independence; care coordination function – coaches patients/families.

MPR: Care coordination activity – educate patient about condition and self-care.

NQF: Plan of Care domain includes – self-management support.

Link to Community Resources Antonelli: Care coordination competency – integrates all resource knowledge.

CMS Definition of Case Management: Case management includes referral and related activities (such as scheduling appointments for the individual) to help an individual obtain needed services, including activities that help link eligible individuals with medical, social, educational providers, or other programs and services that are capable of providing needed services to address identified needs and achieve goals specified in the care plan.

MPR: Care coordination activity – arrange needed services, including those outside the health system (meals, transportation, home repair, prescription assistance, home care).

NQF: Plan of Care domain includes – community services and resources. The Plan of Care includes community and nonclinical services as well as traditional health care services that respond to a patient’s needs and preferences and contribute to achieving the patient’s goals.


MPR: Care coordination activity – arrange needed services, including those within the health system (preventive care with primary care provider; specialist visits; durable medical equipment; acute care).

NQF: A principle of care coordination is that care coordination is important to all patients, but some populations are particularly vulnerable to fragmented, uncoordinated care on a chronic basis, including (not mutually exclusive): children with special health care needs; the frail elderly; persons with cognitive impairments; persons with complex medical conditions; adults with disabilities; people at the end of life; low-income patients; patients who move frequently, including retirees and those with unstable health insurance coverage; and behavioural health care patients.

BROAD APPROACHES

Teamwork focused on Coordination

Antonelli: Care coordination competency – applies teambuilding skills; care coordination function – facilitates team meetings.



Healthcare Home NQF: Framework domain – Health Care Home is a source of usual care selected by the patient (such as a large or small medical group, a single practitioner, a community health center, or a hospital outpatient clinic).

Care Management See elements of

CMS case management definition mapped under other domains.

Medication Management MPR: Care coordination activity – review medications.

NQF: Transitions between and off settings domain includes medication reconciliation

Health IT-enabled Coordination Antonelli: Care coordination competency – adept with information technology; care coordination function – uses health information technology.

NQF: Framework domain – information systems – the use of standardised, integrated electronic information systems with functionalities essential to care coordination is available to all providers and patients.

Antonelli = Antonelli RC, McAllister JW, Popp J. Making care coordination a critical component of the pediatric health system: A multidisciplinary framework. New York, NY: The Commonwealth Fund. May 2009. Publication No. 1277. CMS Definition of Case Management = Centers for Medicare and Medicaid Services. Medicaid Program; Optional state plan case management services. 42 Code of Federal Regulations 441.18 2007 4 December;72(232):68092-3. Coiera = Coeira E. Guide to health informatics. 2nd ed. London, England: Hodder Arnold, a member of the Hodder Headline Group; 2003. MPR = Coordinating care for Medicare beneficiaries: Early experiences of 15 demonstration programs, their patients, and providers: Report to Congress. Princeton, NJ: Mathematica Policy Research, Inc.; May 2004. NQF = National Quality Forum. National Quality Forum-endorsed definition and framework for measuring care coordination. Washington, DC: National Quality Forum; 2006.

8.8.6 The patient experience frameworks and its implications on NSW ICThere is unlikely to be universal agreement on the key important dimensions of the patient experience to measure among different health jurisdictions. The choice of different dimensions/domains/subdomains of the patient experience very much reflects the value and preference of health systems, servicers providers and patients themselves.

These dimensions may have different priorities among health system representatives, service providers and patients. In general, the more objective patient experience measures are preferred as in the discussion provided above. It is important to reach some consensus on the most important aspects of the patient experience in a particular setting among different stakeholders, preferably with the participation of patients and careers.

References


1. Jones JB, Snyder CF, Wu AW. Issues in the design of Internet-based systems for collecting patient-reported outcomes. Quality of Life Research 2007;16(8):1407-17.

2. Guidance for Industry Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims．U.S. Department of Health and Human Services Food and Drug Administration December, 2009.

3. National Quality Forum (NQF). Patient Reported Outcomes (PROs) in Performance Measurement, 2013.

4. McKenna SP. Measuring patient-reported outcomes: Moving beyond misplaced common sense to hard science. BMC Medicine 2011;9.

5. Black N. Patient reported outcome measures could help transform healthcare. BMJ (Online) 2013;346(7896).

6. Black N, Varaganum M, Hutchings A. Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery. BMJ Quality and Safety 2014;23(7):534-43.

7. Nolte E, Pitchforth E. What is the evidence on the economic impacts of integrated care?: WHO Regional Office for Europe and European Observatory on Health Systems and Policies, 2014.

8. NICE. Developing NICE guidelines: the manual: National Institute for Health and Care Excellence (NICE), 2014.

9. NHS Department of Health the UK. The NHS Outcomes Framework 2013/14, 2014.

10. McDonald KM, Schultz E, Albin L, et al. Care Coordination Measures Atlas--June 2014 Update, 2014.

11. JASON The MITRE Corporation. A Robust Health Data Infrastructure: Prepared for:Agency for Healthcare Research and Quality, 2014.

12. IOM(Institute of Medicine). Capturing Social and Behavioral Domains and Measures in electronic Health Records: Phase 2. Washington, DC, 2014.

13. Department of Health the UK. Developing an indicator on user experience of integrated care for the NHS and Adult social care outcomes frameworks, 2014.

14. NHPA(National Health Performance Authority). Three Year Data Plan: 2014–15 to 2016–17, 2013.

15. Australian Commission on Safety and Quality in Health Care. REVIEW OF PATIENT EXPERIENCE AND SATISFACTION SURVEYS CONDUCTED WITHIN PUBLIC AND PRIVATE HOSPITALS IN AUSTRALIA, 2012.

16. Program EHC. Building a national PartnershiP network Advancing Patient-Centered Outcomes Research: Agency for Healthcare Research and Quality, 2011.

17. Cella D, Hahn EA, Jensen SE, et al. Methodological Issues In The Selection, Administration And Use Of Patient-Reported Outcomes In Performance Measurement In Health Care Settings, 2012.

18. Brozek JL, Guyatt GH, Schünemann HJ. How a well-grounded minimal important difference can

enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure. Health and Quality of Life Outcomes 2006;4.

19. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference (MCID). Control Clin Trials 1989;10(4):407-15.

20. Crosby RD, Kolotkin RL, Williams GR. Defining clinically meaningful change in health-related quality of life. J Clin Epidemiol 2003;56(5):395-407.

21. Lydick E, Epstein RS. Interpretation of quality of life changes. Qual Life Res 1993;2(3):221-26.

22. Dworkin RH, Turk DC, Wyrwich KW, et al. Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. The journal of pain : official journal of the American Pain Society 2008;9(2):105-21.

23. Guyatt GH. Making sense of quality-of-life data. Med Care 2000;38(9 Suppl):175-79.

24. Guyatt GH, Norman GR, Juniper EF, et al. A critical look at transition ratings. J Clin Epidemiol 2002;55(9):900-08.

25. Brozek JL, Guyatt GH, Schunemann HJ. How a well-grounded minimal important difference can enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure. Health Qual Life Outcomes 2006;4:69-69.

26. Revicki D, Hays RD, Cella D, et al. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol 2008;61(2):102-09.

27. Farivar SS, Liu H, Hays RD. Half standard deviation estimate of the minimally important difference in HRQOL scores? Expert Rev Pharmacoecon Outcomes Res 2004;4(5):515-23.

28. Schwartz CE, Sprangers MA. Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Soc Sci Med 1999;48(11):1531-48.

29. Nolte S, Elsworth GR, Sinclair AJ, et al. The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation. Qual Life Res 2012;21(3):487-94.

30. Ring L, Höfer S, Heuston F, et al. Response shift masks the treatment impact on patient reported outcomes (PROs): The example of individual quality of life in edentulous patients. Health and Quality of Life Outcomes 2005;3.

31. Brossart DF, Clay DL, Willson VL. Methodological and statistical considerations for threats to internal validity in pediatric outcome data: response shift in self-report outcomes. J Pediatr Psychol 2002;27(1):97-9107.

32. Ahmed S, Bourbeau J, Maltais F, et al. The Oort structural equation modeling approach detected a response shift after a COPD self-management program


not detected by the Schmitt technique. J Clin Epidemiol 2009;62(11):1165-72.

33. Mayo NE, Scott SC, Ahmed S. Case management poststroke did not induce response shift: the value of residuals. J Clin Epidemiol 2009;62(11):1148-56.

34. Cella D, Nowinski CJ. Measuring quality of life in chronic illness: the functional assessment of chronic illness therapy measurement system. Arch Phys Med Rehabil 2002;83(12 Suppl 2):10-17.

35. Shearer D, Morshed S. Common generic measures of health related quality of life in injured patients. Injury 2011;42(3):241-47.

36. Raleigh V, Bardsley M, Smith P, et al. Integrated care and support Pioneers: Indicators for measuring the quality of integrated care: Final report: The King’s Fund; Nuffield Trust; Personal Social Services Research Unit, London School of Economics; Health Services Research & Policy, London School of Hygiene & Tropical Medicine, 2014.

37. Varni JW, Thissen D, Stucky BD, et al. PROMIS® Parent Proxy Report Scales: an item response theory analysis of the parent proxy report item banks. Quality of Life Research 2011:1-18.

38. Thissen D, Varni JW, Stucky BD, et al. Using the PedsQL™ 3.0 asthma module to obtain scores comparable with those of the PROMIS pediatric asthma impact scale (PAIS). Quality of Life Research 2011:1-9.

39. Sung VW, Marques F, Rogers RR, et al. Content validation of the patient-reported outcomes measurement information system (PROMIS) framework in women with urinary incontinence. Neurourology and Urodynamics 2011;30(4):503-09.

40. Preston K, Reise S, Cai L, et al. Using the nominal response model to evaluate response category discrimination in the PROMIS emotional distress item pools. Educational and Psychological Measurement 2011;71(3):523-50.

41. Pilkonis PA, Choi SW, Reise SP, et al. Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS®): Depression, anxiety, and anger. Assessment 2011;18(3):263-83.

42. Magasi S, Ryan G, Revicki D, et al. Content validity of patient-reported outcome measures: perspectives from a PROMIS meeting. Quality of Life Research 2011:1-8.

43. Lai JS, Cella D, Choi S, et al. How item banks and their application can influence measurement practice in rehabilitation medicine: A PROMIS fatigue item bank example. Archives of Physical Medicine and Rehabilitation 2011;92(10 SUPPL.):S20-S27.

44. Junghaenel DU, Christodoulou C, Lai JS, et al. Demographic correlates of fatigue in the US general population: Results from the patient-reported outcomes measurement information system (PROMIS) initiative. Journal of Psychosomatic Research 2011;71(3):117-23.

45. Irwin DE, Stucky BD, Langer MM, et al. PROMIS Pediatric Anger Scale: an item response theory analysis. Quality of Life Research 2011:1-10.

46. Hung M, Clegg DO, Greene T, et al. Evaluation of the PROMIS physical function item bank in orthopaedic patients. Journal of Orthopaedic Research 2011;29(6):947-53.

47. Gibbons LE, Feldman BJ, Crane HM, et al. Migrating from a legacy fixed-format measure to CAT administration: calibrating the PHQ-9 to the PROMIS depression measures. Quality of Life Research 2011:1-9.

48. Fries JF, Krishnan E, Rose M, et al. Improved responsiveness and reduced sample size requirements of PROMIS physical function scales with item response theory. Arthritis Research & Therapy 2011.

49. Fries J, Rose M, Krishnan E. The PROMIS of better outcome assessment: Responsiveness, floor and ceiling effects, and internet administration. Journal of Rheumatology 2011;38(8):1759-64.

50. Flynn KE, Jeffery DD, Keefe FJ, et al. Sexual functioning along the cancer continuum: Focus group results from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Psycho-Oncology 2011;20(4):378-86.

51. Cook KF, Bamer AM, Roddey TS, et al. A PROMIS fatigue short form for use by individuals who have multiple sclerosis. Quality of Life Research 2011:1-10.

52. Cella D, Lai JS, Stone A. Self-reported fatigue: One dimension or more? Lessons from the functional assessment of chronic illness therapy-fatigue (FACIT-F) questionnaire. Supportive Care in Cancer 2011;19(9):1441-50.

53. Bajaj JS, Thacker LR, Wade JB, et al. PROMIS computerised adaptive tests are dynamic instruments to measure health-related quality of life in patients with cirrhosis. Alimentary Pharmacology and Therapeutics 2011;34(9):1123-32.

54. Amtmann D, Cook KF, Johnson KL, et al. The PROMIS initiative: Involvement of rehabilitation stakeholders in development and examples of applications in rehabilitation research. Archives of Physical Medicine and Rehabilitation 2011;92(10 SUPPL.):S12-S19.

55. Yeatts KB, Stucky B, Thissen D, et al. Construction of the pediatric asthma impact scale (PAIS) for the patient-reported outcomes measurement information system (PROMIS). Journal of Asthma 2010;47(3):295-302.

56. Varni JW, Stucky BD, Thissen D, et al. PROMIS pediatric pain interference scale: An item response theory analysis of the pediatric pain item bank. Journal of Pain 2010;11(11):1109-19.

57. Rothrock NE, Hays RD, Spritzer K, et al. Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology 2010;63(11):1195-204.


58. Riley WT, Rothrock N, Bruce B, et al. Patient-reported outcomes measurement information system (PROMIS) domain names and definitions revisions: Further evaluation of content validity in IRT-derived item banks. Quality of Life Research 2010;19(9):1311-21.

59. MacLaren VV, Best LA. Multiple addictive behaviors in young adults: Student norms for the Shorter PROMIS Questionnaire. Addictive Behaviors 2010;35(3):252-55.

60. Liu H, Cella D, Gershon R, et al. Representativeness of the patient-reported outcomes measurement information system internet panel. Journal of Clinical Epidemiology 2010;63(11):1169-78.

61. Irwin DE, Stucky BD, Thissen D, et al. Sampling plan and patient characteristics of the PROMIS pediatrics large-scale survey. Quality of Life Research 2010;19(4):585-94.

62. Irwin DE, Stucky B, Langer MM, et al. An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales. Quality of Life Research 2010;19(4):595-607.

63. Henly SJ. Editorial: The promise of PROMIS. Nursing Research 2010;59(2):77.

64. Henly SJ. The promise of PROMIS. Nursing research 2010;59(2):77.

65. Hays RD, Bode R, Rothrock N, et al. The impact of next and back buttons on time to complete and measurement reliability in computer-based surveys. Quality of Life Research 2010;19(8):1181-84.

66. Hahn EA, DeVellis RF, Bode RK, et al. Measuring social health in the patient-reported outcomes measurement information system (PROMIS): Item bank development and testing. Quality of Life Research 2010;19(7):1035-44.

67. Gershon RC, Rothrock N, Hanrahan R, et al. The use of PROMIS and assessment center to deliver Patient-Reported Outcome Measures in clinical research. Journal of Applied Measurement 2010;11(3):304-14.

68. Flynn KE, Shelby RA, Mitchell SA, et al. Sleep-wake functioning along the cancer continuum: Focus group results from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Psycho-Oncology 2010;19(10):1086-93.

69. Eisenstein EL, Diener LW, Nahm M, et al. Impact of the Patient-Reported Outcomes Management Information System (PROMIS) upon the Design and Operation of Multi-center Clinical Trials: a Qualitative Research Study. Journal of Medical Systems 2010:1-10.

70. Choi SW, Reise SP, Pilkonis PA, et al. Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research 2010;19(1):125-36.

71. Cella D, Riley W, Stone A, et al. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. Journal of Clinical Epidemiology 2010;63(11):1179-94.

72. Amtmann D, Cook KF, Jensen MP, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010;150(1):173-82.

73. Aletaha D. From the item to the outcome: The promising prospects of PROMIS. Arthritis Research and Therapy 2010;12(1).

74. Revicki DA, Kawata AK, Harnam N, et al. Predicting EuroQol (EQ-5D) scores from the patient-reported outcomes measurement information system (PROMIS) global items and domain item banks in a United States sample. Quality of Life Research 2009;18(6):783-91.

75. Revicki DA, Chen WH, Harnam N, et al. Development and psychometric analysis of the PROMIS pain behavior item bank. Pain 2009;146(1-2):158-69.

76. Klem M, Saghafi E, Abromitis R, et al. Building PROMIS item banks: Librarians as co-investigators. Quality of Life Research 2009;18(7):881-88.

77. Jeffery DD, Tzeng JP, Keefe FJ, et al. Initial report of the cancer patient-reported outcomes measurement information system (PROMIS) sexual function committee: Review of sexual function measures and domains used in oncology. Cancer 2009;115(6):1142-53.

78. Jansky LJ, Huang JC. A multi-method approach to assess usability and acceptability: A case study of the patient-reported outcomes measurement system (PROMIS) workshop. Social Science Computer Review 2009;27(2):262-70.

79. Irwin DE, Varni JW, Yeatts K, et al. Cognitive interviewing methodology in the development of a pediatric item bank: A patient reported outcomes measurement information system (PROMIS) study. Health and Quality of Life Outcomes 2009;7.

80. Hays RD, Bjorner JB, Revicki DA, et al. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Quality of Life Research 2009;18(7):873-80.

81. Fries JF, Krishnan E, Bruce B. Items, instruments, crosswalks, and PROMIS. Journal of Rheumatology 2009;36(6):1093-95.

82. Fries JF, Krishnan E. What constitutes progress in assessing patient outcomes? Journal of Clinical Epidemiology 2009;62(8):779-80.

83. Fries JF, Cella D, Rose M, et al. Progress in assessing physical function in arthritis: PROMIS short forms and computerized adaptive testing. Journal of Rheumatology 2009;36(9):2061-66.

84. Fortune-Greeley AK, Flynn KE, Jeffery DD, et al. Using cognitive interviews to evaluate items for measuring sexual functioning across cancer populations: Improvements and remaining challenges. Quality of Life Research 2009;18(8):1085-93.


85. Walsh TR, Irwin DE, Meier A, et al. The use of focus groups in the development of the PROMIS pediatrics item bank. Quality of Life Research 2008;17(5):725-35.

86. Rose M, Bjorner JB, Becker J, et al. Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology 2008;61(1):17-33.

87. Neary PC, Boyle E, Delaney CP, et al. Construct validation of a novel hybrid virtual-reality simulator for training and assessing laparoscopic colectomy; results from the first course for experienced senior laparoscopic surgeons. Surgical Endoscopy and Other Interventional Techniques 2008;22(10):2301-09.

88. Christodoulou C, Junghaenel DU, DeWalt DA, et al. Cognitive interviewing in the evaluation of fatigue items: Results from the patient-reported outcomes measurement information system (PROMIS). Quality of Life Research 2008;17(10):1239-46.

89. Castel LD, Williams KA, Bosworth HB, et al. Content validity in the PROMIS social-health domain: A qualitative analysis of focus-group data. Quality of Life Research 2008;17(5):737-49.

90. Reeve BB, Hays RD, Bjorner JB, et al. Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care 2007;45(5 SUPPL. 1):S22-S31.

91. Hays RD, Liu H, Spritzer K, et al. Item response theory analyses of physical functioning items in the medical outcomes study. Medical Care 2007;45(5 SUPPL. 1):S32-S38.

92. DeWalt DA, Rothrock N, Yount S, et al. Evaluation of item candidates: The PROMIS qualitative item review. Medical Care 2007;45(5 SUPPL. 1):S12-S21.

93. Cella D, Yount S, Rothrock N, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH roadmap cooperative group during its first two years. Medical Care 2007;45(5 SUPPL. 1):S3-S11.

94. Cella D, Gershon R, Lai JS, et al. The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research 2007;16(SUPPL. 1):133-41.

95. Ader DN. Developing the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care 2007;45(5 SUPPL. 1):S1-S2.

96. Pallanti S, Bernardi S, Quercioli L. The Shorter PROMIS Questionnaire and the Internet Addiction Scale in the assessment of multiple addictions in a high-school population: Prevalence and related disability. CNS Spectrums 2006;11(12):966-74.

97. Fries JF, Bruce B, Cella D. The promise of PROMIS: Using item response theory to improve assessment of patient-reported outcomes. Clinical and Experimental Rheumatology 2005;23(5 SUPPL. 39):S53-S57.

98. Cook KF, Bamer AM, Roddey TS, et al. A PROMIS fatigue short form for use by individuals who have multiple sclerosis. Quality of Life Research 2012;21(6):1021-30.

99. Edelen MO, Tucker JS, Shadel WG, et al. Toward a more systematic assessment of smoking: Development of a smoking module for PROMIS®. Addictive Behaviors 2012;37(11):1278-84.

100. Forrest CB, Bevans KB, Tucker C, et al. Commentary: The patient-reported outcome measurement information system (PROMIS®) for children and youth: Application to pediatric psychology. Journal of Pediatric Psychology 2012;37(6):614-21.

101. Irwin DE, Gross HE, Stucky BD, et al. Development of six PROMIS pediatrics proxy-report item banks. Health and Quality of Life Outcomes 2012;10.

102. Irwin DE, Stucky BD, Langer MM, et al. PROMIS pediatric Anger scale: An item response theory analysis. Quality of Life Research 2012;21(4):697-706.

103. Khanna D, Maranian P, Rothrock N, et al. Feasibility and construct validity of PROMIS and “legacy” instruments in an academic scleroderma clinic. Value in Health 2012;15(1):128-34.

104. Magasi S, Ryan G, Revicki D, et al. Content validity of patient-reported outcome measures: Perspectives from a PROMIS meeting. Quality of Life Research 2012;21(5):739-46.

105. Noonan VK, Cook KF, Bamer AM, et al. Measuring fatigue in persons with multiple sclerosis: Creating a crosswalk between the modified fatigue impact scale and the PROMIS fatigue short form. Quality of Life Research 2012;21(7):1123-33.

106. Oude Voshaar MAH, ten Klooster PM, Taal E, et al. Dutch translation and cross-cultural adaptation of the PROMIS ®physical function item bank and cognitive pre-test in Dutch arthritis patients. Arthritis Research and Therapy 2012;14(2).

107. Varni JW, Thissen D, Stucky BD, et al. PROMIS® parent proxy report scales: An item response theory analysis of the parent proxy report item banks. Quality of Life Research 2012;21(7):1223-40.

108. Alonso J, Bartlett SJ, Rose M, et al. The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative. Health and Quality of Life Outcomes 2013;11(1).

109. PROMIS: A management platform for software supply networks based on the linked data and OSLC. Proceedings - International Computer Software and Applications Conference; 2013.

110. Askew RL, Kim J, Chung H, et al. Development of a crosswalk for pain interference measured by the BPI and PROMIS pain interference short form. Quality of Life Research 2013;22(10):2769-76.

111. Auriault F, Thollon L, Peres J, et al. The PROMIS model to highlight the importance of the foetus to the validation of a pregnant woman model. Computer Methods in


Biomechanics and Biomedical Engineering 2013;16(SUPPL 1):182-83.

112. Barile JP, Reeve BB, Smith AW, et al. Monitoring population health for Healthy People 2020: Evaluation of the NIH PROMIS® Global Health, CDC Healthy Days, and satisfaction with life instruments. Quality of Life Research 2013;22(6):1201-11.

113. Bevans KB, Gardner W, Pajer K, et al. Qualitative development of the PROMIS® pediatric stress response item banks. Journal of Pediatric Psychology 2013;38(2):173-91.

114. Bruce B, Fries J, Lingala B, et al. Development and assessment of floor and ceiling items for the PROMIS physical function item bank. Arthritis Research and Therapy 2013;15(5).

115. DeWalt DA, Thissen D, Stucky BD, et al. PROMIS pediatric peer relationships scale: Development of a peer relationships item bank as part of social health measurement. Health Psychology 2013;32(10):1093-103.

116. Farin E, Nagl M, Gramm L, et al. Development and evaluation of the PI-G: a three-scale measure based on the German translation of the PROMIS® pain interference item bank. Quality of Life Research 2013:1-11.

117. Flynn KE, Lin L, Cyranowski JM, et al. Development of the NIH PROMIS® Sexual Function and Satisfaction Measures in Patients with Cancer. Journal of Sexual Medicine 2013;10(SUPPL.):43-52.

118. Flynn KE, Reeve BB, Lin L, et al. Construct validity of the PROMIS® sexual function and satisfaction measures in patients with cancer. Health and Quality of Life Outcomes 2013;11(1).

119. Forrest CB, Bevans KB, Pratiwadi R, et al. Development of the PROMIS® pediatric global health (PGH-7) measure. Quality of Life Research 2013:1-11.

120. Gibbons LE, Feldman BJ, Crane HM, et al. Erratum: Migrating from a legacy fixed-format measure to CAT administration: Calibrating the PHQ-9 to the PROMIS depression measures (Quality of Life Research (2011) 20 (1349-1357) DOI 10.1007/s11136-011-9882-y). Quality of Life Research 2013;22(2):459-60.

121. Gipson DS, Selewski DT, Massengill SF, et al. Gaining the PROMIS perspective from children with nephrotic syndrome: A Midwest pediatric nephrology consortium study. Health and Quality of Life Outcomes 2013;11(1).

122. Hays RD, Spritzer KL, Amtmann D, et al. Upper-extremity and mobility subdomains from the Patient-Reported Outcomes Measurement Information System (PROMIS) adult physical functioning item bank. Archives of Physical Medicine and Rehabilitation 2013;94(11):2291-96.

123. Hays RD, Spritzer KL, Fries JF, et al. Responsiveness and minimally important difference for the Patient-Reported Outcomes Measurement Information System (PROMIS) 20-item physical functioning short form in a

prospective observational study of rheumatoid arthritis. Annals of the Rheumatic Diseases 2013.

124. Hinds PS, Nuss SL, Ruccione KS, et al. PROMIS pediatric measures in pediatric oncology: Valid and clinically feasible indicators of patient-reported outcomes. Pediatric Blood and Cancer 2013;60(3):402-08.

125. Hung M, Baumhauer JF, Latt LD, et al. Validation of PROMIS® physical function computerized adaptive tests for orthopaedic foot and ankle outcome research. Clinical Orthopaedics and Related Research 2013;471(11):3466-74.

126. Hung M, Stuart AR, Higgins TF, et al. Computerized adaptive testing using the PROMIS physical function item bank reduces test burden with less ceiling effects compared to the short musculoskeletal function assessment in orthopaedic trauma patients. Journal of Orthopaedic Trauma 2013.

127. Jacobson CJ, Farrell JE, Kashikar-Zuck S, et al. Disclosure and self-report of emotional, social, and physical health in children and adolescents with chronic pain - A qualitative study of PROMIS pediatric measures. Journal of Pediatric Psychology 2013;38(1):82-93.

128. Kim J, Chung H, Amtmann D, et al. Measurement invariance of the PROMIS pain interference item bank across community and clinical samples. Quality of Life Research 2013;22(3):501-07.

129. Kratz AL, Slavin MD, Mulcahey MJ, et al. An examination of the PROMIS® pediatric instruments to assess mobility in children with cerebral palsy. Quality of Life Research 2013;22(10):2865-76.

130. Lai JS, Stucky BD, Thissen D, et al. Development and psychometric properties of the PROMIS® pediatric fatigue item banks. Quality of Life Research 2013;22(9):2417-27.

131. Paz SH, Spritzer KL, Morales LS, et al. Evaluation of the patient-reported outcomes information system (PROMIS ®) Spanish-language physical functioning items. Quality of Life Research 2013;22(7):1819-30.

132. Pilkonis PA, Yu L, Colditz J, et al. Item banks for alcohol use from the Patient-Reported Outcomes Measurement Information System (PROMIS®): Use, consequences, and expectancies. Drug and Alcohol Dependence 2013;130(1-3):167-77.

133. Schneider S, Choi SW, Junghaenel DU, et al. Psychometric characteristics of daily diaries for the Patient-Reported Outcomes Measurement Information System (PROMIS®): A preliminary investigation. Quality of Life Research 2013;22(7):1859-69.

134. Williams MS, Snyder DC, Sloane R, et al. A comparison of cancer survivors from the PROMIS study selecting telephone versus online questionnaires. Psycho-Oncology 2013;22(11):2632-35.

135. Alves FSM, Pinto RMC, Mendonça TMS, et al. Portuguese-language translation and cross-cultural adaptation of the Fatigue domain of Patient-Reported-


Outcomes Measurement Information System (PROMIS). Cadernos de Saude Publica 2014;30(5):1103-10.

136. Amtmann D, Kim J, Chung H, et al. Comparing CESD-10, PHQ-9, and PROMIS depression instruments in individuals with multiple sclerosis. Rehabilitation Psychology 2014;59(2):220-29.

137. Auriault F, Thollon L, Peres J, et al. Virtual traumatology of pregnant women: The PRegnant car Occupant Model for Impact Simulations (PROMIS). Journal of Biomechanics 2014;47(1):207-13.

138. Baum G, Basen-Engquist K, Swartz MC, et al. Comparing PROMIS computer-adaptive tests to the Brief Symptom Inventory in patients with prostate cancer. Quality of Life Research 2014:1-5.

139. Becker H, Stuifbergen A, Lee HY, et al. Reliability and validity of PROMIS cognitive abilities and cognitive concerns scales among people with multiple sclerosis. International Journal of MS Care 2014;16(1):1-9.

140. Bevans M, Ross A, Cella D. Patient-Reported Outcomes Measurement Information System (PROMIS): Efficient, standardized tools to measure self-reported health and quality of life. Nursing Outlook 2014.

141. Bjorner JB, Rose M, Gandek B, et al. Difference in method of administration did not significantly impact item response: An IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Quality of Life Research 2014;23(1):217-27.

142. Bjorner JB, Rose M, Gandek B, et al. Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. Journal of Clinical Epidemiology 2014;67(1):108-13.

143. Cella D, Choi S, Garcia S, et al. Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgment. Quality of Life Research 2014.

144. Chen WH, Lenderking W, Jin Y, et al. Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of Life Research 2014;23(2):485-93.

145. Choi SW, Schalet B, Cook KF, et al. Establishing a common metric for depressive symptoms: Linking the BDI-II, CES-D, and PHQ-9 to PROMIS Depression. Psychological Assessment 2014;26(2):513-27.

146. Christodoulou C, Schneider S, Junghaenel DU, et al. Measuring daily fatigue using a brief scale adapted from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Quality of Life Research 2014;23(4):1245-53.

147. de Castro NFC, de Rezende CHA, Mendonça TMS, et al. Portuguese-language cultural adaptation of the Items Banks of Anxiety and Depression of the Patient-Reported Outcomes Measurement Information System (PROMIS). Cadernos de Saude Publica 2014;30(4):879-84.

148. Edelen MO. The PROMIS® smoking assessment toolkit-background and Introduction to supplement. Nicotine and Tobacco Research 2014;16(SUPPL.3):S170-S74.

149. Edelen MO, Stucky BD, Hansen M, et al. The PROMIS® smoking initiative: Initial validity evidence: For six new smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S250-S60.

150. Edelen MO, Tucker JS, Shadel WG, et al. Development of the PROMIS® health expectancies of smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S223-S31.

151. Farin E, Nagl M, Gramm L, et al. Development and evaluation of the PI-G: a three-scale measure based on the German translation of the PROMIS ® pain interference item bank. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation 2014;23(4):1255-65.

152. Fernández-López JA. PROMIS®: A platform to evaluate health status and the results of interventions. Semergen 2014;40(7):355-56.

153. Forrest CB, Bevans KB, Pratiwadi R, et al. Development of the PROMIS® pediatric global health (PGH-7) measure. Quality of Life Research 2014;23(4):1221-31.

154. Fries JF, Witter J, Rose M, et al. Item response theory, computerized adaptive testing, and promis: Assessment of physical function. Journal of Rheumatology 2014;41(1):153-58.

155. Hansen M, Cai L, Stucky BD, et al. Methodology for developing and evaluating the PROMIS® smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S175-S89.

156. Hung M, Baumhauer JF, Brodsky JW, et al. Psychometric comparison of the PROMIS physical function CAT with the FAAM and FFI for measuring patient-reported outcomes. Foot and Ankle International 2014;35(6):592-99.

157. Hung M, Hon SD, Franklin JD, et al. Psychometric properties of the promis physical function item bank in patients with spinal disorders. Spine 2014;39(2):158-63.

158. Irwin DE, Atwood CA, Jr., Hays RD, et al. Correlation of PROMIS scales and clinical measures among chronic obstructive pulmonary disease patients with and without exacerbations. Quality of Life Research 2014.

159. Junghaenel DU, Schneider S, Stone AA, et al. Ecological validity and clinical utility of Patient-Reported Outcomes Measurement Information System (PROMIS®) instruments for detecting premenstrual symptoms of depression, anger, and fatigue. Journal of Psychosomatic Research 2014;76(4):300-06.

160. Kroenke K, Yu Z, Wu J, et al. Operating Characteristics of PROMIS Four-Item Depression and Anxiety Scales in Primary Care Patients with Chronic Pain. Pain Medicine (United States) 2014.

161. Lin FJ, Pickard AS, Krishnan JA, et al. Measuring health-related quality of life in chronic obstructive pulmonary


disease: Properties of the EQ-5D-5L and PROMIS-43 short form. BMC Medical Research Methodology 2014;14(1).

162. Mohammed SI. Validity of patient reported outcome measurement information system health assessment questionnaire (Promis HAQ) for assessing disease activity in Iraqi patients with active rheumatoid arthritis. International Journal of Pharmacy and Pharmaceutical Sciences 2014;6(8):332-34.

163. Niaura R. Delivering on its promises: The PROMIS® smoking initiative item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S261-S62.

164. Overbeek CL, Nota SPFT, Jayakumar P, et al. The PROMIS Physical Function Correlates With the QuickDASH in Patients With Upper Extremity Illness. Clinical Orthopaedics and Related Research® 2014.

165. Papuga MO, Beck CA, Kates SL, et al. Validation of GAITRite and PROMIS as high-throughput physical function outcome measures following ACL reconstruction. Journal of Orthopaedic Research 2014;32(6):793-801.

166. Pilkonis PA, Yu L, Dodds NE, et al. Validation of the depression item bank from the Patient-Reported Outcomes Measurement Information System (PROMIS®) in a three-month observational study. Journal of Psychiatric Research 2014;56(1):112-19.

167. Ravens-Sieberer U, Devine J, Bevans K, et al. Subjective well-being measures for children were developed within the PROMIS project: Presentation of first results. Journal of Clinical Epidemiology 2014;67(2):207-18.

168. Revicki DA, Cook KF, Amtmann D, et al. Exploratory and confirmatory factor analysis of the PROMIS pain quality item bank. Quality of Life Research 2014;23(1):245-55.

169. Rose M, Bjorner JB, Gandek B, et al. The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency. Journal of Clinical Epidemiology 2014;67(5):516-26.

170. Schalet BD, Cook KF, Choi SW, et al. Establishing a common metric for self-reported anxiety: Linking the MASQ, PANAS, and GAD-7 to PROMIS Anxiety. Journal of Anxiety Disorders 2014;28(1):88-96.

171. Selewski DT, Massengill SF, Troost JP, et al. Gaining the Patient Reported Outcomes Measurement Information System (PROMIS) perspective in chronic kidney disease: a Midwest Pediatric Nephrology Consortium study. Pediatric Nephrology 2014.

172. Senders A, Hanes D, Bourdette D, et al. Reducing survey burden: Feasibility and validity of PROMIS measures in multiple sclerosis. Multiple Sclerosis 2014;20(8):1102-11.

173. Shadel WG, Edelen MO, Tucker JS, et al. Development of the PROMIS® nicotine dependence item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S190-S201.

174. Shadel WG, Edelen MO, Tucker JS, et al. Development of the PROMIS® coping expectancies of smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S202-S11.

175. Silva e Costa ZMS, Pinto RMC, Mendonça TMS, et al. Brazilian-Portuguese translation and cultural adaptation of the sleep and wake disturbances domains of the Patient-Reported-Outcomes Measurement Information System (PROMIS). Cadernos de Saude Publica 2014;30(7):1391-401.

176. Spiegel BMR, Hays RD, Bolus R, et al. Development of the NIH patient-reported outcomes measurement information system (PROMIS) gastrointestinal symptom scales. American Journal of Gastroenterology 2014;109(11):1804-14.

177. Stachler RJ, Schultz LR, Nerenz D, et al. PROMIS evaluation for head and neck cancer patients: A comprehensive quality-of-life outcomes assessment tool. Laryngoscope 2014;124(6):1368-76.

178. Stucky BD, Edelen MO, Tucker JS, et al. Development of the PROMIS® negative psychosocial expectancies of smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S232-S40.

179. Terwee CB, Roorda LD, De Vet HCW, et al. Dutch-Flemish translation of 17 item banks from the Patient-Reported Outcomes Measurement Information System (PROMIS). Quality of Life Research 2014;23(6):1733-41.

180. Tucker CA, Cieza A, Riley AW, et al. Concept Analysis of the Patient Reported Outcomes Measurement Information System (PROMIS®) and the International Classification of Functioning, Disability and Health (ICF). Quality of Life Research 2014;23(6):1677-86.

181. Tucker JS, Shadel WG, Edelen MO, et al. Development of the PROMIS® social motivations for smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S241-S49.

182. Tucker JS, Shadel WG, Edelen MO, et al. Development of the PROMIS® positive emotional and sensory expectancies of smoking item banks. Nicotine and Tobacco Research 2014;16(SUPPL.3):S212-S22.

183. Tyser AR, Beckmann J, Franklin JD, et al. Evaluation of the PROMIS physical function computer adaptive test in the upper extremity. Journal of Hand Surgery 2014;39(10):2047-51.

184. Varni JW, Magnus B, Stucky BD, et al. Psychometric properties of the PROMIS® pediatric scales: Precision, stability, and comparison of different scoring and administration options. Quality of Life Research 2014;23(4):1233-43.

185. Varni JW, Thissen D, Stucky BD, et al. PROMIS® parent proxy report scales for children ages 5-7 years: An item response theory analysis of differential item functioning across age groups. Quality of Life Research 2014;23(1):349-61.

186. Voshaar MAHO, Ten Klooster PM, Glas CAW, et al. Calibration of the PROMIS physical function item bank in dutch patients with rheumatoid arthritis. PLoS ONE 2014;9(3).

187. Bryan S, Broesch J, Dalzell K, et al. What are the most effective ways to measure patient health outcomes of primary health care integration through PROM (Patient


Reported Outcome Measurement) instruments?: the Canadian Institutes of Health Research, 2013.

188. Wong ST, Haggerty J. Measuring Patient Experiences in Primary Health Care: A review and classification of items and scales used in publicly-available questionnaires: Centre for Health Services and Policy Research, University of British Columbia, 2013.

189. McDonald KM, Schultz E, Albin L, et al. Care Coordination Atlas Version 3 (Prepared by Stanford University under subcontract to Battelle on Contract No. 290-04-0020). AHRQ Publication No.11-0023-EF. Rockville, MD, 2010.

190. Graham C, Killpack C, Raleigh V, et al. Options appraisal on the measurement of people’s experiences of integrated care: The Kings Fund, 2013.

191. King J, Gibbons E, Graham C, et al. Developing measures of people’s self-reported experiences of integrated care: Picker Institute Eureope & University of Oxford, 2013.

192. Uijen AA, Schers HJ, Schellevis FG, et al. How unique is continuity of care? A review of continuity and related concepts. Family Practice 2012;29(3):264-71.

193. Husain A, Barbera L, Howell D, et al. Advanced lung cancer patients’ experience with continuity of care and supportive care needs. Supportive Care in Cancer 2013;21(5):1351-58.

194. Aller MB, Vargas I, Garcia-Subirats I, et al. A tool for assessing continuity of care across care levels: An extended psychometric validation of the CCAENA questionnaire. International Journal of Integrated Care 2013;13(OCT/DEC).

195. Continuity of care to optimise chronic disease management in the community setting: An evidence- based analysis. Ontario Health Technology Assessment Series 2013;13(6):1-41.

196. Uijen AA, Schers HJ, Schellevis FG, et al. Measuring continuity of care: Psychometric properties of the Nijmegen Continuity Questionnaire. British Journal of General Practice 2012;62(600):e494-e502.

197. Uijen AA, Heinst CW, Schellevis FG, et al. Measurement properties of questionnaires measuring continuity of care: a systematic review. PLoS One 2012;7(7):e42256.

198. Uijen AA, Bosch M, Van Den Bosch WJHM, et al. Heart failure patients’ experiences with continuity of care and its relation to medication adherence: A cross-sectional study. BMC Family Practice 2012;13.

199. Roland M. Continuity of care: Betrayed values or misplaced nostalgia. International Journal of Integrated Care 2012;12(OCTOBER - DECEMBE).

200. Haggerty JL, Roberge D, Freeman GK, et al. Validation of a generic measure of continuity of care: When patients encounter several clinicians. Annals of Family Medicine 2012;10(5):443-50.

201. Haggerty JL, Roberge D, Freeman GK, et al. Validation of a generic measure of continuity of care: when patients encounter several clinicians. Ann Fam Med 2012;10(5):443-51.

202. Uijen AA, Schellevis FG, Van Den Bosch WJHM, et al. Nijmegen Continuity Questionnaire: Development and testing of a questionnaire that measures continuity of care. Journal of Clinical Epidemiology 2011;64(12):1391-99.

203. Joyce AS, Adair CE, Wild TC, et al. Continuity of care: Validation of a self-report measure to assess client perceptions of mental health service delivery. Community Mental Health Journal 2010;46(2):192-208.

204. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: An international Delphi study. Quality of Life Research 2010;19(4):539-49.

205. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology 2010;63(7):737-45.

206. Mokkink LB, Terwee CB, Gibbons E, et al. Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Med Res Methodol 2010;10:82-82.

207. Angst F. The new COSMIN guidelines confront traditional concepts of responsiveness. BMC Med Res Methodol 2011;11:152-52.

208. Terwee CB, Mokkink LB, Knol DL, et al. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res 2012;21(4):651-57.

209. Wei X, Barnsley J, Zakus D, et al. Assessing continuity of care in a community diabetes program: initial questionnaire development and validation. J Clin Epidemiol 2008;61(9):925-31.

210. Durbin J, Goering P, Streiner DL, et al. Continuity of care: validation of a new self-report measure for individuals using mental health services. The journal of behavioral health services & research 2004;31(3):279-96.

211. Adair CE, Wild TC, Joyce A, et al. Continuity of mental health services study of Alberta: a research program on continuity of mental health care: University of Calgary, Canada, 2004.

212. Hadjistavropoulos HD, Biem HJ, Kowalyk KM. Measurement of continuity of care in cardiac patients: Reliability and validity of an in-person questionnaire. Canadian Journal of Cardiology 2004;20(9):883-91.

213. de Silva D. Measuring patient experience: The Health Fundation, 2013.

214. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice:


Lack of impact or lack of theory? Social Science and Medicine 2005;60(4):833-43.

215. Abernethy AP, Ahmad A, Zafar SY, et al. Electronic patient-reported data capture as a foundation of rapid learning cancer care. Med Care 2010;48(6 Suppl):S32-8.

216. Chen J. The impact of routine collection of Patient Reported Outcome Measures on patients, providers and health organisations in an oncologic setting: a rapid review. An Evidence Check review brokered by the Sax Institute, 2012.

217. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century, 2001.

218. Crisp N. Creating a patient-led NHS: Delivering the NHS improvement plan. Creating a Patient-Led NHS Delivering the NHS Improvement Plan 2005.

219. Digioia A, Lorenz H, Greenhouse PK, et al. A patient-centered model to improve metrics without cost increase: Viewing all care through the eyes of patients and families. Journal of Nursing Administration 2010;40(12):540-46.

220. Smith S, Dewar B, Pullin S, et al. Relationship centred outcomes focused on compassionate care for older people within in-patient care settings. International Journal of Older People Nursing 2010;5(2):128-36.

221. Stange KC, Nutting PA, Miller WL, et al. Defining and measuring the patient-centered medical home. Journal of General Internal Medicine 2010;25(6):601-12.

222. Hobbs JL. Editor’s Note. Nursing Research 2009;58(1):52-62.

223. Clayton MF, Latimer S, Dunn TW, et al. Assessing patient-centered communication in a family practice setting: How do we measure it, and whose opinion matters? Patient Education and Counseling 2011;84(3):294-302.

224. Sidani S, Collins L, Harbman P, et al. Development of a Measure to Assess Healthcare Providers’ Implementation of Patient-Centered Care. Worldviews on Evidence-Based Nursing 2014;11(4):248-57.

225. Kodner DL, Spreeuwenberg C. Integrated care: meaning, logic, applications, and implications--a discussion paper. Int J Integr Care 2002;2.

226. Goodwin N. The state of telehealth and telecare in the UK: Prospects for integrated care. Journal of Integrated Care 2010;18(6):3-10.

227. McDonald KM, Sundaram V, D.M. B, et al. Care coordination. In: Shojania KG, McDonald KM, Wachter RM, et al., eds. Closing the quality gap: A critical analysis of quality improvement strategies Technical Review 9 (Prepared by Stanford-UCSF Evidence-Based Practice Center under contract No 290-02-0017) Vol 7

Rockville, MD: Agency for Healthcare Research and Quality, 2007.

228. Delnoij D, Klazinga N, Glasgow IK. Integrated care in an international perspective. Int J Integr Care 2002;2.

229. Shortell SM, Gillies RR, Anderson DA. The new world of managed care: creating organized delivery systems. Health Aff (Millwood) 1994;13(5):46-64.

230. Ahgren B, Axelsson R. Evaluating integrated health care: a model for measurement. Int J Integr Care 2005;5.

231. Leutz WN. Five laws for integrating medical and social services: lessons from the United States and the United Kingdom. Milbank Q 1999;77(1):77-7110.

232. Rolfson O, Kärrholm J, Dahlberg LE, et al. Patient-reported outcomes in the Swedish Hip Arthroplasty Register: Results of a nationwide prospective observational study. Journal of Bone and Joint Surgery - Series B 2011;93 B(7):867-75.

233. Lindgren JV, Wretenberg P, Kärrholm J, et al. Patient-reported outcome is influenced by surgical approach in total hip replacement: A study of the Swedish Hip Arthroplasty Register including 42 233 patients. Bone and Joint Journal 2014;96 B(5):590-96.

234. Hjollund NHI, Larsen LP, Biering K, et al. Use of patient-reported outcome (PRO) measures at group and patient levels: Experiences from the generic integrated PRO system, WestChronic. Journal of Medical Internet Research 2014;16(2).

235. Green A. Danish clinical databases: an overview. Scand J Public Health 2011;39(7 Suppl):68-71.

236. Kalucy L, Katterl R, Jackson-Bowers E. Patient Experience of health care performance: Primary Health Care Research & Information Service (PHCRIS), 2009.

237. Ipsos Social Research Institute. Development Report: Adult Admitted Patient Survey: A report prepared for the Bureau of Health Information. Sydney (NSW): Ipsos Social Research Institute, 2013.

238. Pearse J. Review of patient satisfaction and experience surveys conducted for public hospitals in Australia: Health Policy Analysis Pty Ltd, 2005.

239. Secondary. http://www.commonwealthfund.org/News/News-Releases/2011/Nov/International-Health-Policy-Survey.aspx.

240. Jensen RE, Snyder CF, Abernethy AP, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. Journal of oncology practice / American Society of Clinical Oncology 2014;10(4):e215-22.

241. The National Electronic Health Transition Authority (NEHTA). Review of the Personally Controlled Electronic Health Record, 2013.

242. Wennberg JE, Staiger DO, Sharp SM, et al. Observational intensity bias associated with illness adjustment: cross sectional analysis of insurance claims. Bmj 2013;346:f549.

243. Welch HG, Sharp SM, Gottlieb DJ, et al. Geographic variation in diagnosis frequency and risk of death among Medicare beneficiaries. JAMA : the journal of the American Medical Association 2011;305(11):1113-8.


244. Streiner DL, Norman GR. Health measurement scales. A practical guide to their development and use. Oxford, UK: University Press, 2008.

245. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60(1):34-42.

246. Verhoef LM, Van De Belt TH, Engelen L, et al. Social media and rating sites as tools to understanding quality of care: A scoping review. Journal of Medical Internet Research 2014;16(2).

247. Merolli M, Gray K, Martin-Sanchez F. Health outcomes and related effects of using social media in chronic disease management: A literature review and analysis of affordances. Journal of Biomedical Informatics 2013;46(6):957-69.

248. Chung DS, Kim S. Blogging activity among cancer patients and their companions: Uses, gratifications, and predictors of outcomes. Journal of the American Society for Information Science and Technology 2008;59(2):297-306.

249. Høybye MT, Dalton SO, Deltour I, et al. Effect of Internet peer-support groups on psychosocial adjustment to cancer: A randomised study. British Journal of Cancer 2010;102(9):1348-54.

250. Klemm P. Effects of online support group format (Moderated vs Peer-Led) on depressive symptoms and extent of participation in women with breast cancer. CIN - Computers Informatics Nursing 2012;30(1):9-18.

251. McLaughlin M, Nam Y, Gould J, et al. A videosharing social networking intervention for young adult cancer survivors. Computers in Human Behavior 2012;28(2):631-41.

252. Setoyama Y, Yamazaki Y, Namayama K. Benefits of peer support in online Japanese breast cancer communities: differences between lurkers and posters. J Med Internet Res 2011:13.

253. Rodham K, McCabe C, Blake D. Seeking support: An interpretative phenomenological analysis of an Internet message board for people with Complex Regional Pain Syndrome. Psychology and Health 2009;24(6):619-34.

254. Schulz PJ, Rubinelli S, Mariotti G, et al. Meeting the ranging of informational needs of chronic low back pain sufferers: Conceptual design and rationale of the interactive website ONESELF. Disability and Rehabilitation 2009;31(25):2118-24.

255. Ruehlman LS, Karoly P, Enders C. A randomized controlled evaluation of an online chronic pain self management program. Pain 2012;153(2):319-30.

256. Mo PKH, Coulson NS. Developing a model for online support group use, empowering processes and psychosocial outcomes for individuals living with HIV/AIDS. Psychology and Health 2012;27(4):445-59.

257. Greene JA, Choudhry NK, Kilabuk E, et al. Online social networking by patients with diabetes: A qualitative evaluation of communication with Facebook. Journal of General Internal Medicine 2011;26(3):287-92.

258. Shigaki CL, Smarr KL, Yang G, et al. Social interactions in an online self-management program for rheumatoid arthritis. Chronic Illness 2008;4(4):239-46.

259. Bartlett YK, Coulson NS. An investigation into the empowerment effects of using online support groups and how this affects health professional/patient communication. Patient Education and Counseling 2011;83(1):113-19.

260. Chen AT. Exploring online support spaces: using cluster analysis to examine breast cancer diabetes and fibromyalgia support groups. Patient Educ Couns 2011.

261. Griffiths KM, Calear AL, Banfield M, et al. Systematic review on internet support groups (ISGs) and depression (1): do ISGs reduce depressive symptoms? J Med Internet Res 2009.

262. Hoch DB, Watson AJ, Linton DA, et al. The feasibility and impact of delivering a mind-body intervention in a virtual world. PLoS One 2012:7.

263. Lorig KR, Ritter PL, Laurent DD, et al. The internet-based arthritis self-management program: A one-year randomized trial for patients with arthritis or fibromyalgia. Arthritis Care and Research 2008;59(7):1009-17.

264. Schubart JR, Stuckey HL, Ganeshamoorthy A, et al. Chronic health conditions and internet behavioral interventions: A review of factors to enhance user engagement. CIN - Computers Informatics Nursing 2011;29(2):81-92.

265. van Uden-Kraan CF, Drossaert CHC, Taal E, et al. Participation in online patient support groups endorses patients’ empowerment. Patient Education and Counseling 2009;74(1):61-69.

266. Van Uden-Kraan CF, Drossaert CHC, Taal E, et al. Empowering processes and outcomes of participation in online support groups for patients with breast cancer, arthritis, or fibromyalgia. Qualitative Health Research 2008;18(3):405-17.

267. Bender JL, Jimenez-Marroquin MC, Jadad AR. Seeking support on facebook: A content analysis of breast cancer groups. Journal of Medical Internet Research 2011;13(1).

268. Mo PKH, Coulson NS. Living with HIV/AIDS and use of online support groups. Journal of Health Psychology 2010;15(3):339-50.

269. Van De Belt TH, Engelen LJLPG, Berben SAA, et al. Definition of health 2.0 and medicine 2.0: A systematic review. Journal of Medical Internet Research 2010;12(2).

270. Greaves F, Ramirez-Cano D, Millett C, et al. Harnessing the cloud of patient experience: Using social media to detect poor quality healthcare. BMJ Quality and Safety 2013;22(3):251-55.

271. Rozenblum R, Bates DW. Patient-centred healthcare, social media and the internet: The perfect storm? BMJ Quality and Safety 2013;22(3):183-86.


272. Greaves F, Millett C. Consistently increasing numbers of online ratings of healthcare in England. Journal of Medical Internet Research 2012;14(3).

273. Bardach NS, Asteria-Peñaloza R, John Boscardin W, et al. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Quality and Safety 2013;22(3):194-202.

274. Greaves F, Pape UJ, King D, et al. Associations between web-based patient ratings and objective measures of hospital quality. Archives of Internal Medicine 2012;172(5):435-36.

275. Perfect storm Wikipedia: The Free Encyclopedia. Wikimedia Foundation Inc. http://en.wikipedia.org/wiki/Perfect_storm. (accessed 18 Nov 2014).

276. Naslund JA, Grande SW, Aschbrenner KA, et al. Naturally occurring peer support through social media: The experiences of individuals with severe mental illness using you tube. PLoS ONE 2014;9(10).

277. Rate MDS. http://www.ratemds.com/. Accessed in Nov. 2014, 2014.

278. Patient Opinion https://www.patientopinion.org.uk/. (Accessd 18 Nov. 2014), 2014.

279. Iwantgreatcare https://www.iwantgreatcare.org/. (accessed 18 Nov 2014). 2014.

280. PatientsLikeMe http://www.patientslikeme.com/ (accessed 18 Nov 2014), 2014.

281. Mumsnet http://www.mumsnet.com/. (accessed 18 Nov 2014). 2014.

282. E-patients.net http://e-patients.net/ (accessed 18 Nov 2014), 2014.

283. Twitter www.twitter.com (accessed 18 Nov 2014), 2014.

284. Facebook www.facebook.com (accessed 18 Nov 2014), 2014.

285. Google + https://plus.google.com/ (accessed 18 Nov 2014), 2014.

286. Greaves F, Millett C, Nuki P. England’s experience incorporating “anecdotal” reports from consumers into their national reporting system: Lessons for the united states of what to do or not to do? Medical Care Research and Review 2014;71(4):65S-80S.

287. Greaves F, Laverty AA, Ramirez Cano D, et al. Tweets about hospital quality: A mixed methods study. BMJ Quality and Safety 2014;23(10):838-46.

288. Health Do. The power of information: Putting all of us in control of the health and care information we need: Department of Health, the UK, 2012.

289. Insights—NHS England [Internet]. http://www.insights.england.nhs.uk/ (accessed 18 Nov 2014).

290. Mason A, Goddard M, Weatherly H. Financial mechanisms for integrating funds for health and social care: an evidence review: Centre for Health Economics, Uiversity of York, UK. http://www.york.ac.uk/media/che/documents/

papers/researchpapers/CHERP97_Financial_mechanisms_integrating_funds_healtthcare_social_care_.pdf, 2014.

291. Gerteis M, Edgman-Levitan S, Daley J, et al. Through the Patient’s Eyes: Understanding and Promoting Patient-Centered Care. Fransisco: Jossey-Bass, 1993.

292. Sizmur S, Redding D. Core Domains for Measuring Inpatients’ Experience of Care. Oxford: Picker Institute, 2009.

293. Shaller D. Patient-centered Care: What Does it Take? United States: Picker Institute and the Commonwealth Fund, 2007.

294. International Alliance of Patients’ Organisations. What is Patient-centred Healthcare? A Review of Definitions and Principles. London: IAPO, 2007.

295. Cronin C. Patient-centered Care: an Overview of Definitions and Concepts. Washington, DC: National Health Council, 2004.

296. Sizmu S, Redding D. Key Domains of the Experience of Hospital Outpatients. Oxford: Picker Institute, 2010.

297. Department of Health the UK. NHS Patient Experience Framework. 2011.

298. National Voices. A Narrative for Person-Centred Coordinated Care 2012.

299. Staniszewska S, Boardman F, Gunn L, et al. The warwick patient experiences framework: Patient-based evidence in clinical guidelines. International Journal for Quality in Health Care 2014;26(2):151-57.

300. NICE Clinical Guideline (CG 138). Patient Experience in Adult NHS Services: Improving the Experience of Care for People Using Adult NHS Services. London: National Clinical Guidelines Centre, 2012.

integrated care - aci.health.nsw.gov.au · hhs department of health and human services, the usa hit...

Documents