mie2009 good evaluation practice workshop pirkko nykanen 1 guidelines for good evaluation practices...

1MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Guidelines for Good Evaluation Practices in

Health Informatics - A shared networked initiative

EFMI and IMIA

CoordinatorsPirkko Nykänen, Tampere University

Jytte Brender, Aalborg University

2MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Motivation• Need to have good practice guidance

for evaluation • No single, global methodology exists• Good practice guidelines should

serve as:– Framework to design evaluation studies, to

select methodologies, to conduct studies– Support for health professionals and users to

understand evaluation and contribute to evaluation studies

3MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Objectives• Generic practical guidelines that

provide evaluators, users, health professionals with structured, scientifically justified, comprehensive and understandable set of rules for good practice– To design and carry out evaluation studies in

health informatics domain

4MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Objectives – practically• Guidelines:

– Criteria and aspects, both in quantitative and qualitative terms, to consider at each evaluation stage

– Carry out evaluation of the specific aspect, criteria at specific stage

– Design and manage the evaluation study

• Guidelines from all stakeholders viewpoints: third party evaluators, users, health professionals, managers, decision makers, health economists,...

5MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

How GEP-HI guidelines have been developed

• Existing guidelines development principles• Existing literature and material on evaluation

studies, methodologies, reported evaluation experiences, guidelines for good clinical practice, codes of ethics, good implementation practices critical study and assessment, review

• Spiral, consensus making approach: – Coordinators - editorial team - external experts by

HISEVAL website / mailing list – Revised versions – comments – input – revisions…– Presentations / workshops in MIE, MEDINFO-

confereneces

6MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

ExplorationReport Buyer

Team: iterative discussion

and consensus

Selection of MethodsInternalReport

First Study DesignReport Buyer

Study DesignIncluding:

Goal and Mandate Report Buyer

Participants

InternalReports

InterimReports

Buyer

(?) Participants

Final Study Report Buyer

Participants

Pub

lica

tion

1...

n

Study ImplementationThe Study

Project Controlling& Risk Management

Receiver

C Weßel 2006-12-06 GEP-HI Suggestion for a Flowchart

First start GEP-HI guidelines

7MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

First studydesign

Studyexploration

Operational.of methods

Detailed Planning

EvaluationImplement.

Funder/ client

Proj.Managem.

Proj.Managem.

Proj.Managem.

Proj.Managem.

Eval.consultant

Eval.consultant

Eval.consultant

Eval.consultant

Objectorganisation

Evaluationteam

Evaluationteam

Evaluationteam

Evaluationteam

Informationneed

Agreement Designoutline

Method Plans &procedures

Finalisation of the evaluation study.

GEP_HI guidelines flowchart

Project control and risk management

8MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

GEP-HI guidelines• Study exploration

– Starting question • First study design

– Preliminary design• Operationalisation of methods

– methodological approach, methodical aspects• Detailed study plan

– detailed planning • Evaluation study implementation

– Actual accomplishment of the planned evaluation study• Project control and risk management

– Good project management practices• Finalisation of the evaluation study

– Accounting, archiving, reporting of evaluation study results

9MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Status now and progress

• Guidelines-paper has been accessible on iig.umit.at/efmi-website since 2006

• Comments and input received• Final revision is ongoing

– Final version will be available by end 2009

• Comments and input received will be incorporated and the guidelines finalised for publication

10MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Contact

• Feedback and input Pirkko.Nykanen@uta.fi Jytte.Brender@v-chi.dk

• paper accessible: iig.umit.at/efmi

Editorial team: Elske Ammenwerth, Nicolette de Keizer, Jan Talmon, Michael Rigby, Jytte Brender, Pirkko Nykänen

11MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

A walk-through of the key guidelines issues

Pirkko Nykänen, Tampere UniversityJytte Brender, Aalborg University

12MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

First studydesign

Studyexploration

Operational.of methods

Detailed Planning

EvaluationImplement.

Funder/ client

Proj.Managem.

Proj.Managem.

Proj.Managem.

Proj.Managem.

Eval.consultant

Eval.consultant

Eval.consultant

Eval.consultant

Objectorganisation

Evaluationteam

Evaluationteam

Evaluationteam

Evaluationteam

Informationneed

Agreement Designoutline

Method Plans &procedures

Finalisation of the evaluation study.

GEP_HI guidelines flowchart

Project control and risk management

13MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Guidelines summary

1. Study exploration2. First study design3. Operationalisation of methods4. Detailed study plan and project plan5. Evaluation study implementation

5.1 Project control and risk management

6. Finalisation of the evaluation study

14MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 1: Study explorationfocuses on the starting question of the evaluation study

1.1 The information need1.2 Primary audience 1.3 Identification of the buyer / sponsor / study funding party1.4 The context of the evaluation study1.5 A first identification of stakeholders1.6 A first identification of (external) consultants1.7 A first sketch of the setting1.8 First exploration of evaluation methods to be used 1.9 Exploring the restrictions of study execution and publication 1.10 Budget1.11 Ethical, moral and legal issues

1.12 Result of Study Exploration= Outline, draft design of the study

1.13 Formal accept to proceed to the next phase

15MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 2: First study design

focuses on the preliminary design of the evaluation study

2.1 Elaboration of the rationale for the study2.2 Key evaluation issues/questions2.3 Budget2.4 Establishment of the design team2.5 Stakeholder analysis/Social Network analysis2.6 Study constraints2.7 Methods2.8 Organisational setting, the study context2.9 Technical setting, the study context2.10 Participants from the organisational setting2.11 Material and immaterial resources2.12 Time and timing2.13 Risk analysis2.14 Ethical, moral and legal issues2.15 Strategy for reporting and dissemination of results

2.16 Result of First Study Design = a more detailed evaluation study plan

16MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 3: Operationalisation of methods

focuses on making the design and evaluation methods concrete and compliant with the organizational setting and the information need, while taking into account the known pitfalls and perils

3.1 Study type 3.2 Approach3.3 Assumptions3.4 Pitfalls and perils3.5 Skills3.6 Frame of reference3.7 Timing3.8 Justification3.9 Usability studies3.10 Outcome measures3.11 Quality Control on data (measures)3.12 Participants3.13 Study flow

3.14 Result of Operationalisation of Methods = Decisions on the methodological and methodical approach

3.15 Ethical, moral and legal issues = required approvals, ethical committee permissions achievable and

prepared

17MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 4: Detailed study plan and project plan

focuses on providing plans, prescriptions and procedures detailed to the level necessary for the specific study

4.1 Project management4.2 Evaluation activity mapping4.3 Quality management4.4 Risk management4.5 Communication strategy4.6 Recruitment of necessary additional staff

4.7 Result of Detailed Study Plan and Project Plan =

Practical plans with incorporated tools. Upon approval of plans the evaluation study can now be started

18MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

focuses on activities related with the actual accomplishment of the designed evaluation study

5.1 Establishment of the frame of reference5.2 Observation of changes5.3 Quality control of findings5.4 Interpretation of observations5.5 Continuous project management, quality management and

risk management5.6 Regular reports

5.7 Final result of Evaluation Study Implementation = The final evaluation report summarising the study plan and results, and presenting conclusions

Phase 5: Evaluation study implementation

19MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 5.5 Project controlling and risk management

Focuses on the good project management practices and risk management specifically for an evaluation study

- Dedicated project management tools - Quality management in relation to the objectives

and milestones of the study > Develop a quality management plan

- Risk management - Prospectively during planning- Retrospectively with the risks realised

20MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Phase 6: Finalisation of the evaluation study

focuses on accounting, archiving of materials, and reporting of evaluation studies in terms of the STARE-HI guidelines

6.1 Accountings6.2 Reports and publications6.3 Archiving6.4 Reporting guidelines6.5 Reporting scope6.6 The reporting message6.7 Authorship6.8. Ethical and moral aspects6.9 Preparation of reports / publications

- Follow STARE-HI guidelines / Statement on reporting of evaluation studies in health informatics, IJMI 78, 2009, 1-9

6.10 Usability documentation in the context of the EU regulations and standards

21MIE2009 Good Evaluation Practice Workshop Pirkko Nykanen

Generalisability of the guidelines

• Designed for all types of evaluation study types, also applicable on usability evaluation study planning

• The guidelines are for– to plan an evaluation study– to carry out an evaluation study

• The guidelines are NOT for– Which methods to use– How to use the methods

• Handbooks available, see e.g. Brender 2006