a guide to patient reported measures - monmouth partners guide to patient reported... · monmouth...

MONMOUTHPARTNERS

INNOVATIONINSIGHTIMPACT

A Guide to Patient Reported Measures – Theory, Landscape and Uses

MONMOUTHPARTNERS



A Guide to Patient Reported Measures –Theory, Landscape and UsesPROMs and other forms of patient reported measures are now big news in the NHS. They have growing potential and application to trigger changes in care delivery and to inform clinical management of patients and conditions. This marks a step change in their impact and contrasts to their current use as survey, statistical and academic tools, predominantly used for national indicators, policy design and clinical trials.

As we move towards an NHS which places increasing emphasis on outcomes, we thought it would be useful to explore what exactly this terminology means. In this paper, we share the headline facts around the theory and landscape and then focus in on some of the practicalchallenges around the use of PROMs in particular. This briefi ng covers:

PROM, PREMs & satisfaction tools – what they are, what do they measure and how they fi t together.?

Why now? The current PROMs context in the NHS

Different forms of PROMs – a basic taxonomy and the standard terms defi ned

Choosing the right PROMs for the job – what you need to consider and look for from tools and suppliers

Implementation challenges

1. PROMs, PREMs & satisfaction tools – what they are, what do they measure and how do they fi t together?

Anyone encountering the patient reported measures landscape for the fi rst time could easily mistake it for an alphabetical soup. PROMs, PREMs, FFT, PCOMs and OEQ are just some of the terms used – before even starting to consider the acronyms for the specifi c measures available – EQ5D, AQLQ, OHS and OSS are just some of the most common in use today – out of several thousands of measures that have been developed globally.1 At their core, however, all tools and measures attempt to provide insight into the impact that health interventions have from the perspective of the service user. They evaluate either outcome or experience:

Outcomes

Patient Reported Outcome Measures (PROMs) are “standardised validated instruments (question sets) to measure patients’ perceptions of [factors such as] their health status (impairment), their functional status (disability), and their health-related quality of life (well-being)”.2

They are usually designed as questionnaires that measure the impact of clinical interventions in a strictly clinical sense. Did the intervention improve the patient’s physical or mental condition in a meaningful sense, and if so, by how much?

Experience – in turn sub-divided between:

• Satisfaction measures which allow patients (or their carers) to report their subjective view, or perception of, the treatment received (e.g. would you recommend the service); and, more recently

• Patient Reported Experience Measures (PREMs) which collate patients’ objective experience of care. By focusing on specifi c aspects of the process of care (e.g. were you seen on time) PREMs seek to remove the subjectivity associated with measures of satisfaction.

1Defi nition of terms. This briefi ng uses the terms tools, measures, and scores to mean:

PROMs Tools – A holistic term, describing an individual, or set of, measures, as operationalised by a supplierMeasures – A series of questions collectively used in a pre-defi ned way to determine a patient’s scoreScore – A single numerical output across one, or a series of, measures – used to describe the patient’s condition

2http://www.kingsfund.org.uk/sites/fi les/kf/Point-of-Care-Measures-of-patients-experience-in-hospital-Kings-Fund-July-2009_0.pdf

01

MONMOUTHPARTNERS


A Guide to Patient Reported Measures– Theory, Landscape and Uses

Figure 1: Differentiating between PROMs, PREMs and satisfaction measures

Together they provide an overall indicator of performance and quality

3http://www.hscic.gov.uk/article/3843/Background-information-about-PROMs

Measurement of health status /

function / psychosocial

wellbeing

Quality of life

Patient perceptions

Patient expectations

Overall experience

Process of care Patient views

Condition specifi c monitoring /

comparison of outcomes or progress

PROMs:

Usually designed to compare health status / outcomes pre and post intervention

Used to inform clinical management of patients.

PREMs:

Measure the patient’s views and experience while receiving care.

Focus on the process of care and how it impacted the patient experience.

Satisfaction surveys:

Measures patient experience and satisfaction of care received.

Provide insight into patient’s perception of services against their expectations

Please note that other measures exist to assess the process of care and provider’s performance against these. These, in turn, can impact on outcomes, experience and satisfaction such as waiting lists and referral to treatment time (RTT). However this paper does not tackle the use of such measures and solely concentrates on patient reported measures.

No one of these formats is necessarily any ‘better’ then any other – each measures something distinct about the care provided. Being able to collect and cross-reference evidence using all three formats together enables providers and commissioners of care to create a more holistic view of quality – which itself is generally deemed to be a blend of safety, clinical effectiveness and experience dimensions. In contrast, outcome measures are most relevant when (a) comparing pre and post scores following a clinical intervention (e.g. surgery) to monitor a patient’s progress and (b) to inform ongoing clinical management, such as treatment of long term conditions (LTCs).

2. PROMs, PREMS and satisfaction measures in the NHSIn addition to PROMs’ growing practical application for delivering positive changes to care and treatment of long term conditions, they are seen as a vital mechanism for creating visibility into service performance; driving up standards and facilitating competition on quality not just price. In future, policymakers are likely to expand their scope and look for ways in which to mainstream PROMs. In practice, this is likely to see a number of further innovations in:

• Support patient involvement in care: PROMs – particularly when administered electronically – can make big differences to the way patients understand & respond to changes in their own clinical conditions

• Force the redesign of pathways: PROMs can reduce the burden on providers by reducing inappropriate, or un-needed clinical interventions

• Intersect with payment fl ows: In principle PROMs could be used to incentivise, or penalise, providers

• Link with clinician revalidation – PROMs derived information could be used to support clinicians’ information packs in the revalidation process.

• PROMs – in 2009 the NHS introduced mandatory PROMs for four specifi c elective surgical procedures (hip and knee replacements, groin hernia and varicose veins).3 The objectives of the national scheme are to:

– Enable patients to make informed comparison between different providers performance

– Stimulate comparative performance measurement across hospital Trusts

– Over time, provide a method of linking payments to quality and outcomes

As it is nationally mandated the NHS has selected four suppliers to deliver the PROMs service (administering and collection of the surveys) from which NHS providers choose to contract with. National data has been published since 2011 and has been published quarterly since 2012. Some have argued that the NHS implementation of PROMs in their ‘pure’ form has been limited and half-hearted – resulting in little observed change in quality or service delivery. Critics point to the delay in receiving data, poor presentation of the outputs and reliance on local initiatives to use outputs in a meaningful way. Despite this criticisms, PROMs reporting is likely to continue and the NHS is considering expansion to cover long term conditions, different types of cancer survivors, dementia, cardiac and gynaecological procedures.

02

• The Friends and Family Test (FFT) introduced in April 2013 – is in effect a national standardised patient satisfaction survey is used as an indicator of quality and performance. The FFT covers about 4,500 NHS wards and 144 A&E services within NHS Trusts and independent sector providers – asking patients how likely they are to recommend the service to friends and family.

The fi rst results published in July 2013 have found a wide nationwide variation in response rates, with an average of 13.1% for the fi rst quarter.4 There was also wide variation observed in the scores, especially for some A&E services which saw a number of Trusts consistently obtaining negative scores in the fi rst quarter. Low response rates for individual providers had disproportionate impact on score and does not allow for robust data analysis, therefore the challenge still remains to improve response rates.

• Patient Experience metrics – post Francis and Berwick, the NHS has invested heavily in ‘the patient voice’ and involvement. This has fostered patient experience metrics and reporting mechanisms to assess access, availability, environment and continuity of care across the service.5 Ongoing programmes include:

– Patient experience dashboard – sponsored by NHS England

– Medicines Optimisation and adherence dashboards

– CQC user experience surveys

Going forwards, two new initiatives are also worth keeping a close eye on:

1. PCOM – Patient Centred Outcome Measures (PCOMs) are being developed to help patients assess their own health, quality of life and benefi ts of treatments/therapies.

2. OEQ – To bring outcomes and experience measures together, the Outcomes and Experience Questionnaire (OEQ) is being piloted within the NHS – it seeks to measure patients’ perceived value, relevance and benefi t from a recent inpatient episode of care. The OEQ is a short and simple tool which brings together the range of concerns to patients relating to outcomes of service and experience of service. The results of the survey are found to be easy to process and synthesize and complements well other detailed PROMs tools.

Professor Nick Black6, member of NHSE PROMs Advisory Group, has found that improving patient outcomes can increase patient experience ratings by 10%, similarly improving patient experience ratings will cause a 3% improvement on outcome scores. Therefore it is crucial to quantify and tackle both outcomes and experience through the use of PROMs and PREMs to obtain comparable providerperformance measures but also to design, contract and deliver quality care to the satisfaction of patients.

3. Different forms of PROM - a simple taxonomy & the standard terms defi nedThere are thousands of different PROMs tools in existence. The majority of these have been developed for use in clinical trials and economic evaluations and are focussed on specifi c clinical conditions and use cases. To simplify matters, the ‘market’ can be broadly classifi ed in two ways:

• Disease scope / relevance. PROMs tools can be either

– Generic – covering multiple clinical conditions; or

– Condition-specifi c – developed for and focusing on outcomes relevant to a particular condition

• Construct scope, i.e. what attributes the PROMs measures. The most typical constructs include:

– Symptoms (impairments) and other aspects of well-being

– Functioning (disability)

– Health status

– General health perceptions

– Quality of life (QoL)

– Health related quality of life (HRQoL)

– Reports and ratings of health care.

Where a PROMS tool measures only one of the above constructs it is termed uni-dimensional, measurement of more than one construct makes the PROMs multi-dimensional.

Since PROMs are often designed to assess different aspects of a patient’s clinical condition, it is common to combine tools to join a generic measure with a condition-specifi c measure in a single PROMs programme. This addresses the relative strengths and weakness of the two approaches – see Table 1 below for details.

4http://www.england.nhs.uk/2013/07/30/nhsfft/5http://www.midstaffspublicinquiry.com/report https://www.gov.uk/government/uploads/system/uploads/attachment_data/fi le/226703/Berwick_Report.pdf6www.healthcareconferencesuk.co.uk/news/newsfi les/nick-black_364.pdf

MONMOUTHPARTNERS



03

Tool Design & Format – PROMs are typically administered via paper forms distributed either during/after direct contact with the patient, e.g. in outpatient clinics/on discharge/via post. They therefore rely on patients being motivated to complete and submit the questionnaire. This makes survey design a highly specialised business – with a vibrant licensing and copyright trade accompanying it. The most successful instruments reconcile the natural tension between the imperative to keep survey instruments short and concise and the limited information that can be gathered. For example the EQ-5D-3L instrument has only 5 questions, each with three answer options, yet it can result in 243 different answer permutations.

Validation – All tools go through an extensive validation process to demonstrate that they are fi t for purpose, refl ecting their need to be at once both simple to complete but also capable of providing consistent and calibrated scores. Each PROMs questionnaire itself comprises a series of questions with pre-defi ned tick box responses. The order, layout and (if electronically administered) online design of the questionnaires are defi ned by the originator and must be consistently applied to minimise variation in users’ subjective response rates.

Scoring – Most PROMs tools are designed to determine and compare scores pre and post intervention. The pre-operative questionnaire responses create an initial index score against which the post-operative is compared. In the case of uni-dimensional PROMs, individual question responses are combined in a prescribed manner to produce a single score for the construct. Multidimensional PROMs either involve multiple scores – one for each construct evaluated – or, in some cases combine scores to create a composite score across multiple constructs.

Minimal Clinically Important Difference (MCID) – A numerical value which can be used to describe when patients have either made an appreciable improvement or reduction in their clinical condition. The MCID is defi ned as the smallest score on the PROMs scale that is linked to a clinically relevant difference or change that patients perceive as benefi cial or harmful and that would result in a change in treatment. MCID are derived by either (1) anchor-based methods, which link scores to external clinical or patient-based indicators; or (2) distribution-based methods, which estimate MCIDs using the distribution of scores in a sample.7

7http://www.hqlo.com/content/11/1/59

Table 1: Strengths and weaknesses of the two types of PROMs Tools

Table 2: PROMs tools in use in the NHS

Type

Clinical area Generic PROMs Condition Specifi c PROMs

Generic health status

Varicose vein surgery

Groin hernia repair

Hip replacement surgery

Knee replacement surgery

Aberdeen Varicose Vein Questionnaire

No condition specifi c measure used

Oxford Hip Score

Oxford Knee Score

Same measures used for all conditions:EQ-5D

Specifi c – populationdisease/intervention

• Broader range

• Provides comparative indicators across treatments or patient groups

• Can help capture the infl uence of co-morbidities and / or unexpected positive or negative effects of an intervention

• Clinically relevant

• Identify clinically important changes in health that result from specifi c interventions

• Are relevant, meaningful, tailored to the audience and have high patient acceptance

• Not in depth or have limited level of detail

• Limited applicability to specifi c patient populations

• Less responsive to clinically important changes in health

• Population must have the relevant health problem if disease-specifi c

• Restricted focus can limit insight into side effects / unforeseen effects

• Diffi cult or limited comparability across treatments or different diseases

Strengths Weaknesses

The measures in use in the NHS’s PROMs methodology follow this principle and combine both generic and condition specifi c PROMs to create a mini-suite of either two or three different measures for each clinical condition in the programme, see Table 2 below.

Note: In line with the original academic review (conducted by a team at the London School of Hygiene and Tropical Medicine), questions from the Patients’ Experiences of Surgery Questionnaire were required to cover post-operative complications.

Some of the other key issues and concepts to be aware of include:

MONMOUTHPARTNERS



04

4. What to look for in PROMs measures and suppliersThere are two distinct criteria that need to be fulfi lled when deciding to embark in the collection of patient outcomes:

1. Does the PROMs chosen have proven competence to measure the specifi c construct you are assessing?

2. How feasible and practical is the delivery of the chosen tool and what attributes does a supplier need to have to deliver the tool?

4.1 Proven reliable measure?Innumerable PROMs tools have been developed world-wide for different conditions or treatments; scientifi c reviews (also known as structured reviews) are the main method to select the most adequate measure. The London School of Hygiene and Tropical Medicine (LSHTM) was commissioned by the Department of Health to establish which of the PROMs tools available were best suited to measure outcomes for the four elective surgeries that are now mandated PROMs.8

The review identifi ed generic and condition specifi c PROMs and then assessed them for their psychometric qualities looking at

reliability, validity and responsiveness. The identifi ed PROMs were then evaluated against operational criteria of acceptability, interpretability and feasibility/burden and were reviewed by clinicians of the relevant surgical specialty to ensure the measures had a sound theoretical and clinical basis. The review recommended the PROMs that are now mandated by the NHS. The Patient Outcomes Measurement Group (PROM Group), based at Oxford University, are also a good source of reference as they have produced structured reviews for the main PROMs tools available and used in the UK for a number of elective procedures, cancers, LTCs and depression and anxiety.9

4.2 Marrying measures with modes of administration and supplier capabilitiesWhen considering which PROMs to use you will need to consider the above attributes as well as service supplier specifi c criteria as you may need to balance the desired characteristics of the measure with suppliers’ capabilities and/or the practicalities of administering the tool. It is important to remember that completion of PROMs is voluntary, patients are not obliged to complete PROMs tools/questionnaires. Therefore consider how the supplier can use new data collection technologies to encourage patients’ participation.

8LSHTM, Patient-Reported Outcome Measures (PROMs) for routine use in Treatment Centres: recommendations based on a review of the scientifi c evidence, 20059http://phi.uhce.ox.ac.uk/newpubs.php

Essential Qualities of the Measure

Reliability – How consistent are the items of the tool at measuring the construct and the stability of the scoring which should reproduce the similar results over time?

Validity – The extent to which the content of the tool is representative of the construct, the correlation between the scale used and construct being measured, or ensuring it is not correlated with measures of different constructs?

Responsiveness – Is the measure able to detect signifi cant change over time and comparability of before and after scores?

Acceptability – Does the measure produce quality data which can be assessed by the completeness of the data and score distributions?

Interpretability – Does the score provide a meaningful differentiation or point of reference?

Feasibility – Is the measure easy to administer and does not burden or limit respondent’s participation in any manner?

Clinical and theoretical basis – Is it clinically relevant and accurate? Do the items and scale have a sound theoretical basis for the measure?

Response rates and missing data – Self-administered postal questionnaires have lower response rates and higher levels of missing data compared to interviewer, telephone or supervised self-administered questionnaires

Authenticity of data – Respondents are more willing to report less favourable health outcomes or states through self-administered questionnaires

Convenience and cost – Postal or online self-administered questionnaires can be completed by respondents at their convenienceand are least costly

Data protection /confi dentiality – Online questionnaires need to ensure they protect patient confi dentiality

Market knowledge – Need to have a strong understanding of the PROMs market and the different tools available

Range of choice – Can provide and deliver different PROMs solutions tailored to your needs and multiple modes of administration available to ensure it meets respondents needs or preferences

Data reporting – Have appropriate and helpful data reporting capabilities which will help you used results in a meaningful manner

Proven track record – Financial track record and of delivering PROMs services successfully

Price – Of mode of administration options refl ect burden of administration

Optimising response rates – What strategies would the supplier use to increase or ensure high response rates?

Optimising response rates – What strategies would the supplier use to increase or ensure high response rates?

Feasibility of mode of administration of the tool Weaknesses

MONMOUTHPARTNERS



05

Tool Choice

Data collection

Comparability with NHS and across private sector

Sensitivity

Action & Intervention

Presentation

How well does it score against the following the attributes?

How will you implement patient recruitment within your services? How will you use the NHS Personal Demographics Service (PDS) to obtain patient and demographic information?

Will you use different approaches for different measures and what target response rates will you set? How will you tackle post-operative response rates which tend to be lower?

Will you integrate local data collectors’ views of data collection and make any necessary changes to your methodology? Will you seek patients’ views on how the tool is administered recruitment and follow-up?

How will you balance cost-effectiveness of data collection and response rate?

What tools can you use to lessen the burden for respondents e.g. freephone line or the web?

What is an adequate recruitment rate? What is an adequate post-op response rate? How will these compare to NHS response rates?

How will missing data impact results and comparability?

Will risk adjustment adequately identify poor performers?

Which generic measure should we use: EQ-5D or SF-36D?

How will we defi ne minimal clinically important differences? How it be applied across all providers? Will it be comparable to the NHS?

How will we assess the responsiveness of the PROMs tool and how will we establish it is adequate?

Will it be able to detect minimal clinically important differences?

How sensitive will the scoring be? How comparable is the sensitivity across all the measures to be used?

How will the data be used for decision making?

How will the industry respond to poor results or insuffi cient data from low response rates?

How will providers respond to published results? Will there be a requirement for remedy of poor performance? How will providers be held accountable?

What is the best way of presenting data? Can the reports be tailored to different provider needs?

What is the best indicator for comparing providers?

At what geographical level should providers be compared?

At what level of provision should comparisons be made?

How frequently should performance information be updated?

How will the information be presented to the public to ensure they can digest the information and make an informed choice?

Will the measure produce scores and data that you can use meaningfully?

06

5. From Tool choice to Implementation – A Checklist

This checklist is designed to provide support when considering how to choose and implement a PROMs tool, with a focus on the practicalities necessary to successful uptake and application.

Reliability

Validity

Responsiveness

Acceptability

Interpretability

Feasibility

MONMOUTHPARTNERS



For further information about issues raised in this paper, please contact Carolina Henning, [email protected] or 07769 723 444

MONMOUTHPARTNERS


a guide to patient reported measures - monmouth partners guide to patient reported... · monmouth...

Documents