1 session 3: benefits definition and measurement v4.0 may 2009 ehealth benefits management toolkit
TRANSCRIPT
1
Session 3:
Benefits Definition and Measurement
v4.0May 2009
eHealth Benefits Management Toolkit
2 Benefits Management
Session 3 Outline
Benefits measurement categorisation
Benefits profile
template
Benefits Realisation
Plan
Transition to business
as usual
3 Benefits Management
At the end of the session you will have an understanding of:
• what methods can be used to define quantifiable benefits and estimate their value;
• how to finalise detailed benefit definitions using SMARTT and ensure the benefits are measurable;
• how to pull all the outputs of the previous sessions together to create the Benefits Realisation Plan; and
• what further activities need to take place at the end of the project.
Session 3 purpose
4 Benefits Management
Benefits Measurement Categorisation
Benefits measurement categorisation
Benefits profile
template
Benefits Realisation
Plan
Transition to business
as usual
5 Benefits Management
Specific Definitions of Benefits
Benefits should be:• clearly and succinctly defined• aligned with the projects objectives and the overall NHS
strategy• focused on improving service and patient care, not focused on
IT
Effective measurement and monitoring will:• indicate the extent to which benefits are being realised;• give early warning of potential problems;• create the opportunity to adapt the benefits or changes to
enable the overall objectives; and• ensure that achieved benefits are measured, reported and
communicated.
6 Benefits Management
Measurable Outcomes
The next step is to:
• devise measures that will show when the benefits have been delivered and to what degree;
• determine which benefits can be quantified;
• determine the financial value of benefits where possible;
• investigate additional data capture activities required; and
• establish baseline, targets and benchmarking figures.
7 Benefits Management
Defining Measures
• Use existing measures where possible• Agree all new measures with those concerned• Ask yourself:
– Is it achievable? – Is it an appropriate and valid indication that the benefit has been
delivered? – Is it directly attributable to a particular change in the project?
• Consider the effort and cost of capturing and collating the data• Decide who will be responsible for ensuring that the data is available• Ensure the measures encourage the right type of behaviour• Define when and how often the measurements should be taken
8 Benefits Management
Benefits Categorisation(JW)
Degree of explicitness of measures
Financial By applying a cost, price or other valid financial formula to a quantifiable benefit, a financial value can be calculated.
Quantifiable Sufficient evidence exists to forecast how much improvement / benefit should result from the changes.
Measurable This aspect of performance is currently being measured or an appropriate measure could be implemented. But it is not possible to estimate by how much performance will improve.
Observable By use of agreed criteria, specific individuals or groups will decide to what extent the benefits has been realised, based on their experience and judgement.
9 Benefits Management
Non Quantifiable Benefits
Observable benefits • lowest category of benefits in term of robust measuring• often the softer / people benefits• can help get buy-in essential to the implementation of changes and the
realisation of their resulting benefits• agree realisation criteria and who will say whether they are met
Measurable benefits • performance is easily measured; but• potential improvement won’t be known until after the event • take baseline measurements before any changes are made for
comparison
10 Benefits Management
Quantifiable Benefits
Quantifiable benefits
• aspect of performance easily and consistently measured
• improvement can be predicted and targets defined
• usually applicable to embedded processes where performance is well documented and predictable
Financial benefits
• highest category of benefits in term of robust measuring• focus on benefits that can easily be translated in terms of financial gain
or reduced cost: if it is too complicated or not robust don’t assign a monetary value to it
• don’t confuse value with savings
11 Benefits Management
Progressing Non Quantifiable Measures to Quantifiable Measures
The following methods can be used to get from measurable to quantifiable measures:
• Modelling or simulation: extrapolate expected benefits from existing data. It may be necessary to establish trends/times series.
• Benchmarking: evaluate changes in relation to “best practices” in the comparable organisations.
• References sites: can also be used to establish a comparative base but assess relevance and feasibility in your own organisation.
• Pilots: test the new way of working and system on a small scale.
12 Benefits Management
Examples for Benefits Template
Category Benefit Descriptions and Numbers Measure Value / Improvement Level
Financial e.g. “increase utilisation of theatres resources by allowing patient-cancelled appointments to be filled at short notice from reserve list” expected increase in utilisation = 1 in 3 cancellations filled = 200 slots = £1000 per slot = £200,000
cancellations filled x resource value (£1,000)
£200,000
Quantifiable e.g. “elimination of referrals from SCI gateway being printed onto paper for transfer to secondary care system” number of referrals, and therefore those being printed, known before implementation; integration will eliminate these
% of referrals processed through new system
100%
Measurable e.g. “reduce unnecessary x-rays by avoiding repeat request” not known how many x-ray requests are repeats, but assumption is that these will be eliminated, reducing overall number of x-ray requests
number of x-ray requests
N/A
Observable e.g. “reduce staff frustration due to need for double data entry” staff satisfaction survey with regard to duplicate data entry
% of satisfied or very satisfied
N/A
13 Benefits Management
Benefits Profile Template
Benefits measurement categorisation
Benefits profile
template
Benefits Realisation
Plan
Transition to business
as usual
14 Benefits Management
SMARTT BenefitsDetailed benefit descriptions contain the following categories:
Benefit Characteristic
Definition Example
SpecificDescription of the benefits Number of re-xray forms completed.
Quality dimensions enabled Safety, efficiency, timeliness
Measureable Variable to be evaluated Number of forms filled in per month
Achieveable
How high a priority is the benefit?
Scored on a scale of 1-5 (where 5 is the highest priority)
How likely is the benefit to be realised?
Scored on a scale of 1-5 (where 5 is the highest likelihood)
RealisticWhat is the benefit dependent on?
Strategic direction, process, information management, skills/ behaviours, organisation/ roles, infrastructure/ physical environment, technology integration
Time-bound
Realisation timeframe E.G. Measure benefit from 3 months pre-go-live until 1 year post go-live
Measurement interval recommendation
E.G. Per minute, per hour, per week, etc.
Frequency of measurements
E.G. One week out of each month, one month out of each year, etc.
TargetedBenchmark E.G. 20 forms/month
Target E.G. 20% reduction
15 Benefits Management
Filling in the Benefit Profiles
ACTION:Expand on the benefits information from the previous workshops in the template for each priority benefit identified:
•Utilise measurements that are already being used by the Health Board where possible•Take into account pre-existing government targets of efficiency estimates when setting benchmarks and targets for benefits•Ensure that dependencies are taken into account when setting realisation timelines
Benefit IDEnter ID number from BDN
Benefit descriptionEnter benefit description
Benefit ownerEnter the name of the person or group who will receive the benefit
Benefit sponsorEnter name and role of the person who will ensure that the benefit is achieved and instigate remedial actions if required
SafeTick if relevant
EffectiveTick if relevant
TimelyTick if relevant
EfficientTick if relevant
Patient-centredTick if relevant
EquitableTick if relevant
Measure/Variable to be evaluatedEnter description of measure to be collected
Benefit explicitnessEnter category: financial (high), quantifiable, measurable, observable (low)
Value/Level of improvementOnly applicable for quantifiable and financial benefits
ImpactHow important is the benefit? Scale of 1 (low) to 5 (high)
LikelihoodHow likely is the benefit to be realised? 1 (low) to 5 (high)
Strategic directionEnter ID and description of objective(s) the benefit contributes to
Changes requiredEnter ID and owner of required changes (see change profile for details)
IM&T requiredEnter ID and description of required IM&T capabilities
Benefit realisation dateEnter date the benefit will be realised (or dates and degree of realisation for staged roll-out)
Measurement timeframeEnter start and end dates of monitoring and reporting
Measure intervalEnter description of the period of time the measurement covers
BaselineEnter value, source and date if applicable
Benchmark or targetEnter value, source and date if applicable
16 Benefits Management
Preparing to Measure
• Are there pre-set requirements for benefits realisation timelines?
• Who is going to be collecting the measurements?
• How will the measurement be collected? (data sources)
• Who will oversee the benefits collection?
• Where will the results be reported to?
• Who will conduct benefits reviews?
• Who will be responsible for communicating about the benefits side of the project?
17 Benefits Management
Preparing for Benefits Monitoring
In order that the benefits monitoring is as productive as possible:
• define a robust mechanism to monitor progress and ensure that reports are distributed to the appropriate people;
• ensure that the Benefits Management templates are up to date;• hold regular benefits reviews to track the progress of benefits realisation
against the project and programme objectives;• use benefits reviews as an opportunity to highlight areas where benefits
are not being achieved, and identify the barriers to achieving them;• for those benefits that are under-realised, refer to the dependencies
shown in the BDN for potential root causes;• assign responsibilities for taking and coordinating corrective actions
where benefits are not being met; and• identify and communicate successes too!
18 Benefits Management
Benefits Realisation Plan
Benefits measurement categorisation
Benefits profile
template
Benefits Realisation
Plan
Transition to business
as usual
19 Benefits Management
Formalising it
The Benefit Realisation Plan provides evidence to:
• Justify investment decisions• Prioritise competing work in context of finite resources• Establish clear ownership / governance arrangements• Realise benefits• Demonstrate value and effective use of resources
Store the Benefit Realisation Plan with the other key project management work.
20 Benefits Management
Benefits Realisation Plan (BRP)
When drafting the Benefit Realisation Plan remember to:
• Consider the dependencies identified between benefits, changes and IS&T capabilities;
• Identify timeline and milestones that take those dependencies into account;
• Work with stakeholders towards minimising the negative impact of identified disbenefits; and
• Extend the plan beyond the system implementation date (often viewed as the end of the project) to schedule regular reviews of benefits realised and consider potential further benefits.
21 Benefits Management
BRP Content
The Benefit Realisation Plan should include:
• the completed Benefits Dependency Network (BDN)• the completed benefit profiles, including dates for capture of baseline
figures at an early stage• the completed change templates• the completed stakeholders analysis
Prioritise the benefits so that the most important always has the most focus. This ensures that the project makes the greatest impact.
Identify dates for expected delivery of the benefits.
22 Benefits Management
Benefits Strategy
A benefits strategy should also be prepared for the business case, including:
• the rationale linking the drivers, objectives and benefits• defined roles and responsibilities• a mechanism for regular measurements and reporting• a mechanism for regular reviews and corrective action• arrangements for the transfer of responsibilities at close of project
23 Benefits Management
Providing all this, the business case will demonstrate that:• benefits outweigh disbenefits;• a shared vision is held which is strategically aligned;• the BDN clearly shows how the shared vision will be achieved;• the change required, when considered in relation to the overall schedule
of change planned, is not overwhelming for any stakeholders; and,• the benefits realised will be worth the effort required to achieve them.
It will also show that:• there is a clear process for identifying, monitoring and realising the
benefits; and• the baseline benefits position has been recorded to enable comparison
with projected targets for monitoring the achievement of benefits.
Linking Business Case & Benefits Management
24 Benefits Management
Transition to Business as Usual
Benefits measurement categorisation
Benefits profile
template
Benefits Realisation
Plan
Transition to business as usual
25 Benefits Management
Close of project activities
Following full implementation of information systems, the achievement of the business case should be reviewed.
The review will produce 3 reports:
• formal end-of-project review
• lessons learned
• potential further benefits
26 Benefits Management
End of Project Review
The end of project review will:
• assess to what extend the project was successful in delivering the benefits;
• decide what further actions are required, by whom and when they should be reviewed; and
• detail handover arrangements for any tasks outstanding (including benefits monitoring and reporting).
The Benefits Plan should be updated as relevant to ensure a reference document is available to handover to the “business as usual” team.
27 Benefits Management
Lessons Learned
The purpose of the “lessons learned report” is to:
• reflect on the lessons learned for future investments;• disseminate useful lessons learned during the project for the benefit of
other projects, internal and external;• provide a useful control as part of the functions of an independent quality
assurance group;• consider how much effort was required to complete the benefits
management documentation and monitor the benefits; and• make recommendations for future enhancement or modification of the
benefits management method.
28 Benefits Management
Further Benefits
The Review Team should also consider and communicate:
• what opportunities exist for the identification and realisation of additional benefits; and
• who will take the business case for them forward.
Additional benefits should be defined, planned and implemented following the same methodology as the initial project benefits.
29 Benefits Management
Transition to Business As Usual
• Leave no loose ends: each outstanding activity needs to be allocated to a named individual
• Have end dates for decommissioning of legacy systems to ensure the new IM&T are used – this will ensure new processes are used too and related benefits are delivered
• Monitor the implementation of change over a period: – people may revert to the “old ways” once the initial enthusiasm has
worn off– people may find work-arounds where the technology or new
processes are not working instead of reporting the problems– address teething problems
30 Benefits Management
Summary of Benefits Management LifecycleProject lifecycle Benefits management lifecycle Outputs
Projectinitiation
Identify and structure benefits Benefits Dependency Network (BDN)Identify change activities required to realize the
benefits
Prepare benefit plan and business case Benefit profiles Change profiles
Stakeholder analysis
Run first benefits analysis Baseline data
Delivery management
Use benefit plan to inform project management, system development and change management
activities
Regularly review project progress against benefits analysis
Updated figures
Make adjustment to system capabilities, business changes or benefit definitions
as required
Amended templates and BDN
CloseRun end of project benefits analysis (formal
review)Formal report to
Board
Business as usual
Identify need for further reviews and potential for further benefits
31 Benefits Management
References and Further Reading
• The Clinical Indicators Team has produced a presentation on statistical process control (SPC):
http://www.indicators.scot.nhs.uk/Workshops/SPC.html
• HM Treasury’s “Green Book” provides examples of methods used to estimate the financial value of activities or resources:
http://www.hm-treasury.gov.uk/d/green_book_complete.pdf
• No Delays Scotland have devised a methodology for measuring benefits:
http://www.nodelaysscotland.scot.nhs.uk/ServiceImprovement/Tools/Pages/IT211_Methodology_for_measuring_benefits.aspx
• Statistics for NHS England and Wales for benchmarking on:
http://www.ic.nhs.uk/statistics-and-data-collections (cont’d)
32 Benefits Management
References and Further Reading (cont’d)
• Benchmark statistics (Scotland) are available from ISD:
http://www.isdscotland.org/isd/782.html
• Quarterly and annual figures on patient safety are published by NPSA:
http://www.npsa.nhs.uk/nrls/patient-safety-incident-data/quarterly-data-reports/
• The Department of Health publishes annual and quarterly healthcare performance statistics:
http://www.dh.gov.uk/en/Publicationsandstatistics/Statistics/Performancedataandstatistics/index.htm
• Population numbers by NHS Boards – births, deaths, totals, by age, by gender – are available from GRO:
http://www.gro-scotland.gov.uk/statistics/publications-and-data/population-estimates/index.html
33 Benefits Management
Congratulations on completing the benefits
management programme and on all your efforts
to ensure that maximum benefit is gained from
eHealth Systems.