Local Clinical Audit , Success and Failure

(How can measuring care lead to improvement)

Cormac Kelly, Clinical Lead for AuditClaire Bowler , Senior Quality Outcomes FacilitatorAmanda McFie, Quality Outcomes Facilitator Eric Robinson, Quality Outcomes Manager

Robert Jones and Agnes Hunt HospitalOswestry, Shropshire

RJAH Orthopaedic Hospital

• Leading orthopaedic “centre of excellence”• During 2012 a total of 11960 procedures

took place• Top of the Patient Experience review• Top for Hospital Food• Worst waiting times in 2013

The Team

Why I /you bother with CA• I /you want to do a good job• I /you want to know what is working well so I can do

more of it• I /you want to know what is working badly so I can

stop doing it• I /you want to be able to give patients honest answers

to questions such as How often does the operation work in your hands ?What complications have you seen ?• I want to be better than my colleagues at everything• I want my colleagues to think like me

Success and Failure

Views of doctors on Clinical Audit (2003)

• “When has CA ever produced anything of Quality or Value ?”

• What is in it for me or my patients ?• Love to, but no time• We are doing OK surely ?• Its all too complicated

Dealing with Difficult Professionals

• I need an audit before the end of the month because…

• Ok its not audit .. Lets call it service evaluation ?• If it is not approved I will do the study anyway…

Freedom of Information Act….• Stop hastleing me about the report.. I presented it

at the BOA last week• Here is the paper.. Do whatever you want with it

• Patient representative 2012• CCG representative 2012• Chaired by Former Medical Director• Usual enthusiastic Clinical audit Facilitators• Structured minutes• Poor attendance of Departmental Leads,

Senior management/Executive• No Non Exec representation

The Clinical Audit Committee

• Half day twice yearly, 3 hours• All elective activity “cancelled”• Invitation to present your audit to Doctors , nurses, Allied Health professional,

managers• Update on CA , Clinical audit lead• Guest speaker• Opportunity to show off good projects and offer constructive criticism of poorer

projects• Alteration of Action plan• Increasing attendance and enthusiasm (average attendance 50 staff)• Cancelled in 2009• Incorporates Mortality /morbidity meeting• We want more………“£100,000.00 of lost revenue to the Trust “

The Multidisciplinary Clinical Audit Meetings (MDCAM) 2004

• Oh, I can….• To share good experience of clinical audit • To enthuse others• To do audit across hospitals• To get clever people to educate us • Supported by Specialist Orthopaedic Alliance

and HQIP• Annual meeting

Its all in the name

• 2012. Amalgamation of Clinical audit and Outcomes Unit

• Is it time to change the emphasis from CA to QI• Quality Outcomes and Improvement . REJECTED

The problem with Junior doctors

• Target is 2 audits in every 6 month cycle• No CA training in postgraduate curriculum• Until recently no scrutiny of CA activity• Still seen as a chore rather than a quality

improvement process • Action plan and improvement rarely witnessed

HQIP knowledge among postgraduate Orthopaedic trainees

• 3 questions• Have you had any formal training in CA

methodology• What is HQIP ?• Where can you access online training ?

Answers

RESULTS

Poor knowledge and little training HQIP……….“High quality Intercourse training”

Poor knowledge among trainees

Our solution

• HQIP Intermediate Clinical audit Course online for doctors

• Clinical audit training is now part of Mandatory training for all staff, e-learning

• Linked up with CASC• 3 successful Training days for staff (at budget

cost) inside and outside Trust• Assign junior doctors to audits at the start of

their attachments

Patient InvolvementThe Out-patient Survey

• Not an audit per se• Designed and conducted with

patient involvement• Soft end point• No hard stats

Results

• Patients love us• Some areas where improvements can be made• Was waiting an hour to be seen but “it was

worth it”

• Illustrates the weakness of using patient opinion only

Fasting for Theatre Audit

• Simple• Easy to understand the benefit• Easy to measure• Not Priority One• Produced a meaningful action plan• Re audit demonstrated a big improvement• RESULT patients are more comfortable

arriving for surgery

Local anaesthetic for Hand surgery

• “A new method of local anaesthesia”• One doctor• Collected, and analysed own results• ? Data collected before audit proposal submitted• Technique Not supported by some colleagues• No patient satisfaction measures• No medium term outcome measures or

complications recorded

Consent Audit

• In association with OICAN• Across 4 sites• Agreed audit tool• Allows comparison of outcome• The medical Director likes it• Keen to try another• Re audit shows improvement in documentation• Revision to include more patient perspectives

So What should I audit ?

• Not enough top down• Not enough linkage to Risk Management• What are the real issues in the Trust• Not enough patient input into choice of topic• Who decides if an audit goes ahead ? • ? Assessing group ?• Does it really need to be audited at all ?• Clinical Audit cannot cure obvious deficiencies• LISTEN TO WHAT PATIENTS ARE TELLING YOU

“Trying to boil the ocean…… you just get wet “

The problem with Service Evaluation and other projects

• Who is responsible for quality control• Does it need proper number crunching ?• Rarely involve the Clinical Audit department• Often not registered• Often slip through the net and are not registered• Many produce conclusions that are not based on

results, are biased, are presented outside the Trust as “Research”

• BIG PROBLEM

The Action Plan• The Action plan needs to be reasonable , do-able

and supported by the whole team including budget holders

• So where does risk get identified and acted upon• Who is responsible for the action plan happening• Use an Action Planner to facilitate the chase• CA should chase action plan but when does the

chasing stop ?• ? Datix when actions don’t seem to be going

anywhere

Linkage to other committees

• How are concerns based on audit elevated to other committees

• Is there clear guidance and protocols on this ?• Is risk identified and acted upon ?• Is results disseminated to the relevant people

Problems not going away…

• Many units continue to struggle locally in many aspects of Clinical audit

• Increasing pressures on organisations to “get it right” without good knowledge of Quality improvement methodology

• Big increase in National Audits , NICe guidance , NHSLA requirements pushes local interest audits onto backburner

Disillusioned CA Staff

• Poor job structure• Poor training• Too much learning on the job• Much grief from management many of whom don’t understand

CA objectives • Lots of chasing uninterested medical staff• Too much to do eg National Audits “poorly designed” • Feel distant from the Board where the big decisions are made• When something goes wrong …”clinical audit is not working “

C L I N I C A L A U D I T SS E R V I C E

E VA L U AT I O N S

P R O M S O U T C O M E M E A S U R E S

R E VA L I D AT I O N

D ATA P R O V I S I O N

C Q U I N S

P E E R R E V I E W S M O R TA L I T Y M O N I T O R I N G

PAT I E N T E X P E R I E N C E

Q U A L I T Y M E A S U R E S

N H S L A

C O N F I D E N T I A L

E N Q U I R I E SC Q CC O M M I S S I O N E R

R E Q U I R E M E N T S

N I C E G U I D A N C E

Things that annoy Little Miss Audit Facilitator

• I have a name, please use it…… No, its not Audit Girl. • Dear Mr / Mrs highly trained medical professional completing

out a proposal form isn’t brain surgery …….fill it out• No, “ the patient will have an average pain score” is not a

standard• Even if you don’t want my help ….. Still register your audit• Asking me on a Friday afternoon (4 pm) to analyse 125 patients

worth of data by Monday is not ok ………..i do have a life outside work

X

CASC Survey 2013

• MUCH TO BE DONE• RE-INVIGORATION PROJECT 2008 PROBABLY

DID NOT WORK

Clinical audit and Quality improvement- time for a rethink ?

Still a tick box exercise for manyContrasting levels of audit knowledge, skills and experienceCA considered time consuming, bureaucratic, managerially driven,…with potential to apportion individual blame and which has no associated personal or professional reward

Bowie et al, Jnl of Eval in Clin Practice. 18(2012) 42-48

….Assurance to the Board..

(assurance to the bored)• What about assurance to the Clinical audit

Committee ?• Is there anybody listening ?• “Chief Quality Officer” Dr Nick Black “The need for the Board’s of Trusts to engage more effectively with the quality of their services, giving as much attention to quality as is customarily given to finance, was recognized by Robert Francis (February 2013) and has recently been endorsed by Jeremy Hunt (King’s Fund, May 2013). “

‘Less pain for more gain…

…..how to survive Clinical Audit’

EDUCATION, EDUCATION, EDUCATION

Educate your staff on quality audit practicePrioritise auditsFewer audits of higher quality with more names as authorsHave a clear policy on non-audit projectsNo

• Revalidation has helped focus the mind OF DOCTORS

• Certificate of attendance at MDCAM• Summary of Involvement in Clinical audit• Enclosed in Clinical Governance package

Linking to Revalidation for Doctors

• Lay Member of CA committee• Outpatient Survey designed by patients and

conducted by patients• Planned Dementia audit• Inspirational……patients do ask the right

questions…

Involving Patients in CA

Clinical Audit Plan

Quality Improvement Programme

Thank You• Questions

• Answers

• Take some notes

Local Quality Improvement – Successes and Failures

Katharine (Kat) Young MASenior Quality Improvement Lead, Royal Berkshire NHS FT

Chair, National Quality Improvement and Clinical Audit networkMember, National Advisory Group for Clinical Audit and Enquiries

Katharine.young@royalberkshire.nhs.uk @Clin_Q

www.hqip.org.uk

Royal Berkshire NHS Foundation Trust

• Large District General, Reading • Strategic objectives:

– exceed patient and customer expectations – work together to create a modern and sustainable

healthcare system – deliver the best healthcare in the best possible place for

patients – provide the best place to work, train and learn.

• Re-organisation 2011 • Move to three Care Groups (Urgent, Planned and

Networked) & Clinical Quality Improvement Unit • Need to maximise clinical and financial efficiency

Where we were at

• Quality by name but not in nature

• The candy factory – I love Lucyhttps://www.youtube.com/watch?v=8NPzLBSBzPI

• ‘Insanity: Doing the same thing over and over again and expecting different results’Albert Einstein/Benjamin Franklin/Anon

• Work harder?

Role of the Quality Improvement Team

• Champion and promote culture of continuous improvement

• Leadership, expert advice• Project support• Building Capability and Capacity (facilitated

learning)

Working together to improve quality

RBFT Improve Framework

Shared Vision

The vision should

answer the question

‘where do we want to get to?’ and should be

the inspiration

and framework

for planning

Assurance

CQIU involvement:

-Involvement in completion & challenge of Quality Impact Assessments

- Monitoring of balancing measures

- Research capability

SMART Aims

The aims set should be:

Specific

Measurable

Achievable

Realistic

Timeframe

The Quality Improvement

Approach

Improve it!

Improvement opportunities

Measure

Plan

RoI

Outcomes

Vital behaviours

Evaluate

Vital Behaviours

Desirable – what’s in it for them?

Enable – provide information and skills / deliberate practice

Stakeholders – crucial conversations – team and beyond

Influencers – senior engagement and support; opinion leaders

Rewards – what are the incentives?

Environment –providing the physical means to achieve the outcome

Governance

Project Management Office (PMO) involvement:

-Projects grouped by value (P1-P3)

-Tracking & monitoring

-Project documents

-Risk assessments

-QIPP Reports -Programme Board

Outcomes

What are the Process measures?

What are the Outcome Measures?

Sustainability

To sustain the improvement requires:

-Patient & staff engagement

-Alignment with goals & structures

-Infrastructure

-Credible evidence

-Adaptability

-Continual monitoring of progress

Shared Learning, show casing of examples such as MEMC, clinical leadership programme etcRBFT Quality Improvement Training Programme (training, master classes, visits to centres of excellence)Communication Strategy –sharing plans, outcomes & celebrating success

Quality Improvement Framework: Our journey towards excellence

Shared Vision

The vision should

answer the question

‘where do we want to get to?’ and should be

the inspiration

and framework

for planning

Assurance

CQIU involvement:

-Involvement in completion & challenge of Quality Impact Assessments

- Monitoring of balancing measures

- Research capability

SMART Aims

The aims set should be:

Specific

Measurable

Achievable

Realistic

Timeframe

The Quality Improvement

Approach

Improve it!

Improvement opportunities

Measure

Plan

RoI

Outcomes

Vital behaviours

Evaluate

Vital Behaviours

Desirable – what’s in it for them?

Enable – provide information and skills / deliberate practice

Stakeholders – crucial conversations – team and beyond

Influencers – senior engagement and support; opinion leaders

Rewards – what are the incentives?

Environment –providing the physical means to achieve the outcome

Governance

Project Management Office (PMO) involvement:

-Projects grouped by value (P1-P3)

-Tracking & monitoring

-Project documents

-Risk assessments

-QIPP Reports -Programme Board

Outcomes

What are the Process measures?

What are the Outcome Measures?

Sustainability

To sustain the improvement requires:

-Patient & staff engagement

-Alignment with goals & structures

-Infrastructure

-Credible evidence

-Adaptability

-Continual monitoring of progress

Shared Learning, show casing of examples such as MEMC, clinical leadership programme etcRBFT Quality Improvement Training Programme (training, master classes, visits to centres of excellence)Communication Strategy –sharing plans, outcomes & celebrating success

Quality Improvement Framework: Our journey towards excellence

RBFT Improve: Approach

Improvement opportunity

Measure Plan RoI Outcomes Vital Behaviours

Evaluate

Ho

w…M

eth

odo

logy

… do we want to

improve?

…. do we generate

ideas?

… good are we and how do we know?

… do we make the changes?

… do we prioritise?

…timescale?

… do we demonstrate it’s worth it

…are things different

from before?

… will our behaviours support the

change

… will we know we

have made a difference, and how do

we keep improving?

Brainstorm

P&L

Process Map

Model for Improvement

Spaghetti Diagram

RAG Study

Benchmark

Audit

Historical analysis

Baselines

Lean

Pareto

Staff +Patient

Engagement

5 D’s

PDSA

PMO Docs

Human / Financial cost

and saving

SPC

Business Cases

Project Plan

Owners

Timescales

Rapid Improvemen

t Events

Six Sigma

Open to change

See the benefit or

bigger picture

Engagement

Review performance

Communicate change

Ensure sustainability

Celebrate Success

The Quality Improvement Approach: IMPROVE

Improvement opportunity

Measure Plan RoI Outcomes Vital Behaviours

Evaluate

Ho

w…M

eth

odo

logy

… do we want to

improve?

…. do we generate

ideas?

… good are we and how do we know?

… do we make the changes?

… do we prioritise?

…timescale?

… do we demonstrate it’s worth it

…are things different

from before?

… will our behaviours support the

change

… will we know we

have made a difference, and how do

we keep improving?

Brainstorm

P&L

Process Map

Model for Improvement

Spaghetti Diagram

RAG Study

Benchmark

Audit

Historical analysis

Baselines

Lean

Pareto

Staff +Patient

Engagement

5 D’s

PDSA

PMO Docs

Human / Financial cost

and saving

SPC

Business Cases

Project Plan

Owners

Timescales

Rapid Improvemen

t Events

Six Sigma

Open to change

See the benefit or

bigger picture

Engagement

Review performance

Communicate change

Ensure sustainability

Celebrate Success

The Quality Improvement Approach: IMPROVE

Doing things differently

• Clinical Leadership• Board/Executive support• Reducing bureaucracy• Interaction / workshops• Staff and patient involvement – all levels• Facilitated learning

Behavioural change

• Vision – they why• What do you need

to make it happen?

• How?

• DESIRABLE What’s in it for them

• ENABLE Provide info / skills

• STAKEHOLDERS Team & beyond

• INFLUENCE Supervisor / Seniors

• REWARDS Incentives

• ENVIRONMENT e.g. checklist, rota

Patient Leaders

• Recruitment based on values (12 to date)

• 7 day modular Patient Leader programme includes QI

• Supporting/leading Quality Improvement projects – working in collaboration

Quality Improvement Methodology

• Root Cause Analysis• Pareto• Driver diagrams• Importance of measurement: run charts• Model for Improvement / Clinical Audit• Process Mapping• Lean / Six Sigma

A model for learning and change

When you combine the 3 questions with the…

…the Model for Improvement

PDSA cycle, you get…

49

WHY are We doing it?

HOW we will do it

Move away from traditional clinical audit

Move from traditional audit cycles to real-time, dynamic improvement change audit

Importance of Measurement: Use of run charts

Perla R. BMJ Qual Saf 2011; 20: 46-51

The change seems to be associated with an improvement

The change is not associated with an improvement; if there had been no baseline measurement before making the change, the change mioght have been mistakenly interpreted as making a difference

The change seems to be associated with an improvement initially but the effect does not appear sustained

Remember measure little and often

What are we doing differently: Structure

Quality PerformanceCommittee

Care GroupClinical Governance

SpecialtyClinical Governance

DQORG

Trust ClinicalGovernance

Clinical Outcomes & Effectiveness group

NED: Janet Rutherford

DQORG = Data Quality Outcomes Review Group

What are we doing differently?

• Training/learning - basic• Model for improvement • Rapid audit cycles/PDSA• Importance of measurement

/ run charts• Driver diagrams

• 1:1 Coaching/support• Value of personal interaction

What are we doing differently?

SMART Aims Outcome focussed

What are we doing differently?

•Junior doctor availability•Opportunities for completion•Time needed for completion

Primary Drivers

Secondary Drivers

Discharge decision

•Ward round times•Ward round structure•Consultant-led decision making•Elective admission- predictable•Emergency admission

•TTO completion•Opening times•Ward pharmacist•portering

•Information to GP•Mandatory•format•coding

EDL write up

Pharmacy

Content

AIM: Increase number of EDLs completed within 24 hours of discharge to 70% within 3 months on ward B to enable effective discharge planning

Topic selection

Successes: improvement in outcomes / sustainability

Week (starting October 2013)

IMPROVE: Hospital Acquired Pneumonia

HAP on 8 wards

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

Week

Numb

er of

patie

nts w

ith H

AP

Total on 8 wards Mean

Bed heads raised to 30 degrees at week 9 (17Dec) Oral hygiene

measures Victoria Ward week 25 (8Apr) Oral hygiene

measures across 8 wards week 29 (6May)

Trust w ide roll out w eek 43 (12 Aug)

Successes: improvement in outcomes / sustainability

0123456789

10HAP on 8 wards - progress chart

Num

ber

of p

atien

ts w

ith

HA

P

HAP prevention measures re- launched

Week (starting October 2013)

2012 Prevalence audits on 4 wards

012345

1 2 3 4

2013 HAP prevalence 8 wards

0123456789

10

1 2 3 4 5 6

Week

IMPROVE: Hospital Acquired Pneumonia

Successes: rapid improvements

Trainees experience

“For the patient a QIP means an improvement in the quality of service they receive from the simplest of things to more complex issues all with the intention of improving patient experience and quality of life.” Anna Brown, CMT1

“This has been a very valuable learning experience into clinical quality improvement as well as being brilliant for my CV. The MEMC team have been supportive and encouraging throughout and there has always been someone available to talk to if I have ever needed any help.” Anna Weil, FY2

“My supervisor was very supportive and motivating throughout…I would definitely do a QIP again.”

Ram Jeeneea CT1

“It was very useful, and its good to know that you can really make a difference - I am now more aware of what we can do, and how to take action.”

Faraz Siddiqui FY1“You hear about projects and they sound really huge, but this has opened my eyes to how you can do little things and make small changes that make a big difference.”

Anna Roche FY2

“I am definitely empowered, and now have the understanding that I can make changes.”

Kunal Kulkarni FY1

“Overall it is very satisfying to engage with trainees and to supervise a QIP as they have an enthusiasm and motivation which is a real joy to work with. I very much enjoyed supervising a QIP project this year”. Maeve McKeogh, Consultant Supervisor

“The DVD produced as part of “Making Every Moment Count” will make an enormous difference to patients on the run up to their surgery, it will ease fears before surgery and calm nerves on the day of surgery”. Olivia Johnson, a patient involved in MEMC

Successes: changing culture

Feedback 1 year on

Lessons learnt

• Need to change culture / mindsets• Make it personal • Lots of QI Methods: learning needs• Traditional clinical audit switches a lot of people off

• Use standards where they exist, you can make improvements without them

• Need greater focus on outcomes / return on investment at start of process

• Staff including Junior Doctors and Patient Leaders are real assets

Future

• Roll out Making Every Moment Count (junior doctor) programme to all Trust staff

• Continue focus on outcomes• Increase ability to assess Return on Investment• Sustainability • Increase facilitated learning • Share our learning

Scenarios

Scenarios

What do you need to do more of / less of?

CULTURE • In relation to clinical audit / quality improvement what is the culture in

my organisation?• What behaviours need to change? How to change? • Are we are learning organisation?• Do we have meaningful patient engagement/involvement?

STRUCTURE • Where is quality compromised by silo working?• Who are your clinical leaders?• How engaged are your Board/Executive? • How are we hearing the patient’s voice?

Scenarios

PROCESS • Is process geared around your customers/staff or the

clinical audit team?• Are people put off by your registration process? • How accessible is the clinical audit / QI team?OUTCOMES• Are outcomes measured / shared?• Is the impact of intervention on outcome known?• Is return on investment known/ shared?

Resources

Learning to make a difference : Making Every Moment Count PIlot

• https://www.youtube.com/watch?v=brZv_tftn_M • http://www.clinmed.rcpjournal.org/content/12/6/520.full• https://www.rcplondon.ac.uk/projects/learning-make-diff

erence-ltmd

• https://www.youtube.com/watch?v=sNKXOEPIe2Q&feature=youtu.be

Scenario 1 WHO checklist Audit

• Problem is surgeons are not completing sign out at the end of surgery

• Every theatre doing it differently• PLAN: To audit the WHO checks in order to

make it all better, please help• audit tool includes details of every step in

audit process

• Obvious problem, correct it• Create an SOP• Allow it all to bed in over a year

• Then audit against the SOP

SCENARIO 2Pain Service audit Conclusions/ action plan

• Authors Pain Nurse• 80 % of patients happy with everything• 55% of patients felt psychological needs not

met• ACTION PLAN includes• Appoint a Psychologist to the team part time

Shopping list phenomenon

• I want , I want ,,, I want…• Using the CA process to bypass the standard

process of business planning