Evidence-Based Practice

AUPHA Annual Meeting, June 20, 2013, Monterey

If doctors can do it managers can do it?

Evidence?

outcome of scientific research,

organizational facts & data,

benchmarking, best practices,

collective experience, personal

experience, intuition

All managers base their

decisions on ‘evidence’

However ...

Many managers pay little

or no attention to the

quality of the evidence they

base their decisions on

Trust me, 20 years of management experience

Teach managers how to

critically evaluate the validity,

and generalizability of the

evidence and help them find

‘the best available’ evidence

Evidence-based decision

What is the added value of evidence- based

practice for managers within the field of

hospital care?

Proof of concept

Teaching Hospital

- 6 managers

University Hospital

- 4 managers

Evidence based pilot

Phase 1: Training managers in the principles of EBP

Phase 2: Examination of the current decision making

processes that managers are using

Phase 3: Evaluation of 4 completed projects from an

EB perspective (retrospective)

Phase 4: Making EB recommendations for 4 new

projects (prospective)

Phase 5: Evaluation

Evidence based pilot

Some preliminary results

Decision making process

Decision making process

Focus on procedures instead of evidence

Internal politics and power struggles

No critical appraisal of the evidence at hand

Relying on anecdotal evidence (workshops,

best practices, popular management books,

consultants)

One option (sometimes two)

Bias: Outcome, Halo, Confirmation, etc.

Post mortem analysis

Evidence-based perspective

NOT: Did we made the right decision?

BUT: Is there evidence from scientific research to

support (or call into question) the approach

taken?

Best available evidence?

Post mortem: leadership training

leadership training: dm process

No problem definition

No organizational evidence consulted

Selection of training companies based on

experience, recommendation or reputation

No explicit selection criteria / procedure

‘Best’ presentation has won: one size fits all

leadership training: scientific evidence

15 meta analyses, 5 relevant

37 (‘systematic’) reviews, 2 relevant

Lots of relevant primary studies

leadership training: scientific evidence

Long history (30 yrs): moderate effect sizes

Senior & middle managers tend to benefit more

than managers at the supervisory level

Effect on ‘poor’ leaders is limited.

Leadership trainings that focus on

interpersonal / social skills show higher effect

sizes than those based on a specific leadership

‘model’

Reactions

Who knew?

Denial

Anger

Bargaining

Acceptance

Prospective /

EB recommendations

Questions / projects

360 degree feedback

Financial incentives

Lean Six Sigma

Hand Hygiene

Goal setting

Value Based Health Care

Downsizing

Evidence-based perspective

NOT: What works?

BUT: What are, given the target group, the problem and

the context involved, the main factors determining the

success or failure of the project that need to be taken into

account?Best available scientific

evidence?

Prospective: Multi Source Feedback

Multi Source Feedback: background

IFMS: based on multi source feedback

Regulating bodies and insurance companies

(KPI’s – prices/ revenue)

Based on CANMEDS, no standard method

New market: consulting firms

Process

Scoping session: inventory of the aspects

relevant to the question

Session with leading academic

Search in relevant databases

Critical Appraisal

Summary / research synthesis

Recommendations / guidelines

Multi Source Feedback: scientific evidence

223 primary studies on MSF, 42 relevant

6 meta analyses or systematic reviews on

MSF, 3 relevant

18 meta analyses or systematic reviews on

‘feedback’ or ‘performance appraisal’,

5 relevant

Content of the feedback (neg vs pos)

Way of the delivery of the feedback

Interpretation of the feedbackPersonality of the

rateeFeedback orientation

of the ratee

Type and number of ratersSelection of

raters

Rater reliablity (patients, nurses,

colleagues)Type of

response scale

Development vs perfomance appraisal

Organizational culture

Perceived procedural justice

Multi Source Feedback: main factors

Reactions

Who knew?

Wow!

Great!

Good stuff!

Relevant!

Lessons learned I

New approach

Recalibrates the power dynamics (accountability!)

The profit is in the process

Different (better?) decisions were made

Doctors love it!

Lessons learned II

Hard for individual managers

It starts with the senior management team

It’s all about accountability

Support system

EBP > Planning & Control

One day, maybe …

Chief Evidence Officer