DomainScoring Guide group Non-Scoring Guide group

ICC (95% C.I.) p-value ICC (95% C.I.) p-valueDomain 1 0.821 (-0.303~0.995) 0.043 -0.333 (-0.806~0.966) 0.512Domain 2 0.769 (-0.675~0.994) 0.068 0.769 (-0.679~0.994) 0.069Domain 3 0.796 (0.394~0.954) 0.002 0.424 (-0.710~0.871) 0.155Domain 4 -1.333 (-15.940~0.941) 0.670 0.000 (-6.260~0.975) 0.422Domain 5 0.888 (0.431~0.992) 0.005 0.272 (-2.696~0.950) 0.312Domain 6 0.667 (-4.814~1.000) 0.182 0.792 (-2634~1.000) 0.116Overall 0.869 (0.753~0.939) <0.001 0.662 (0.362~0.841) <0.001

DomainScoring Guide group Non-Scoring Guide group

ICC (95% C.I.) p-value ICC (95% C.I.) p-valueDomain 1 0.815 (-0.344~0.995) 0.046 0.682 (-1.307~0.992) 0.116Domain 2 0.430 (-3.137~0.986) 0.251 -0.762 (-11.791~0.955) 0.595Domain 3 0.722 (0.175~0.938) 0.011 0.473 (-0.565~0.882) 0.121Domain 4 0.718 (-1.048~0.993) 0.096 -0.296 (-8.411~0.967) 0.503Domain 5 0.424 (-1.925~0.960) 0.229 0.273 (-2.693~0.950) 0.312Domain 6 0.000 (-16.443~0.999) 0.391 0.000 (-16.443~0.999) 0.391Overall 0.826 (0.671~0.918) <0.001 0.680 (0.395~0.850) <0.001

Statistics • To identify the distribution Descriptive analysis (Domain specific)

• To evaluate the reliability Intra-class correlation (Domain specific)

• To evaluate the consistency Association among appraisers (Each items)

 

Study design

I. Background and Purpose

IV. Discussion

• Scoring guide reduce the inter-rater disagreemet and improve the overall

reliability of the K-AGREE II instrument. Those effects remarkable in low level CPGs development Inter-rater disagreement reflects the healthcare environment characteristics

II. Methods

Table 1. Inter-rater reliability of K-AGREE domain scores in CPG AⅡ

Effects of Korean-AGREE Scoring Guide on ⅡImproving the Reliability of the Scores

Reliability• Higher ICC in SG

Observed in almost domain and stastically significant in overall assessment

Moo-Kyung Oh 1, Heuisug Jo1,2,3, Youkyoung Lee3,4,5

1Department of Preventive Medicine, Kangwon National University Hospital, Chuncheon, Korea

2 Department of Health Management and Policy, Kangwon National University School of Medicine, Chuncheon, Korea3Executive Committee for clinical practice guideline, The Korean Academy of Medical Sciences

4Department of Laboratory Medicine and Genetics, Soonchunhyang University Bucheon Hospital, Bucheon, Korea5Department of Laboratory Medicine and Genetics, Soonchunhyang University College of Medicine, Cheonan, Korea

Table 2. Inter-rater reliability of K-AGREE domain scores in CPG BⅡ

Appraiser 1 2 3 41 1 .622 .348 .4812 .622 1 .393 .5963 .348 .393 1 -.0524 .481 .596 -.052 1

Fig. 1. Distribution of K-AGREE Ⅱdomain scores according Scoring Guide users in CPG A

Scope and

Purpose

Stake involve-

ment

Rigour de-

velop-ment

Clarity and

Presen-tation

Applica-bility

Editorial Inde-pen-

dence

Do

mai

n S

core

s

Scope and

Purpose

Stake involve-

ment

Rigour de-

velop-ment

Clarity and

Presen-tation

Applica-bility

Editorial Inde-pen-

dence

Fig. 2. Distribution of K-AGREE Ⅱdomain scores according Scoring Guide users in CPG B

III. Results

Distribution• Higher Scores and higher variability in Non-SG

Distinctive in domain 2, 3, and 5

Consistency• Higher association in SG

Association improve in SG, especially distinctive in CPG B

Table 3. Association of SG in CPG A Table 4. Association of Non-SG in CPG A

Table 5. Association of SG in CPG B Table 6. Association of Non-SG in CPG BAppraiser 1 2 3 4

1 1 .853 .453 .4912 .853 1 .651 .7493 .453 .651 1 .6414 .491 .749 .641 1

Appraiser 1 2 3 41 1 -.225 -.434 -.4592 -.225 1 .373 .5023 -.434 .373 1 .8334 -.459 .502 .833 1

Appraiser 1 2 3 41 1 .556 .441 .1272 .556 1 .479 .1283 .441 .479 1 .3724 .127 .128 .372 1

Random assignment AnalysisAppraisal

Scoring Guide Group(Korean AGREE II + Scoring

Guide)

Non-Scoring Guide Group(Korean AGREE II only)

Distribution

Reliability

Consistency

2 CPG

14Appraisers

Stakeholder involvement

Applicability

Domain Limitation Scoring Guide

Providing clear standards regarding the stakeholder

and the level of participation

Providing clear definition of implementation

Providing the methodologies and resources in detail

Lack of experience in stakeholder involvement

Different understanding of stakeholder

Low level ofCPGs implementation

Confuse implementation with dissemination

• Current status of Clinical Practice Guidelines in Korea

This study aim to examine the effects of the K-AGREE II scoring guide

to reduce inter-rater differences.

• Korean medical guideline information center (KoMGI)

• A couple of CPGs development handbook • More than 100 of guidelines of the last decade

• Korean AGREE II instrument

• Web-based evaluation systems

• KoMGI

Development

Dissemination

Implementation

ApprovalAppraisal

Inter-rater disagreement

A lack of experiences in evaluation of CPGs

Differences of healthcare environments

Scoring Guide