chain of trust, a web quality assessment tool

The Chain of Trust — Quality and Credibility in

Health Information Sources

PF Anderson, pfa@umich.eduApril 23, 2008

© 2008 Regents of the University of Michigan. All rights reserved.

BackgroundIn 2006 I was asked by two nursing

faculty to develop: a quality assessment tool for both health information and term paper

sourcesthat could be used with incoming freshmanas an assignment.

Existing tools were either too specific, too difficult to use, or not appropriate for the target audience.

My earlier tools Evaluation Tool for Clinician-Oriented Web Sites

http://www-personal.umich.edu/~pfa/pro/courses/EvalClin.pdf

Evaluation Tool for Patient-Oriented Web Sites http://www-personal.umich.edu/~pfa/pro/courses/

EvalPtEd.pdf

NOTE: In prior testing, these tools had proven effective, but difficult to use in teaching, and time consuming for scoring.

Why?

Everyone wants the good informationNobody agrees what that isLike “match the tool to the task”,

information one person recommends won’t be right for another person

Censoring what information is available or “acceptable” creates a lack of trust

Criticizing information choices impairs communication (and trust)

Why? (cont.)

What’s good depends on: The question Who’s asking Literacy Prior educational context Prior knowledge in this topic Prior experiences Attitudes & assumptions Intangible personal preferences Other context

What’s missing Professional codes define professional responsibilities with

ever-greater accuracy. Huge efforts also go into ensuring trustworthy performance. . . .The efforts to prevent abuse of trust are gigantic, relentless and expensive; and inevitably their results are always less than perfect. Have these countermeasures begun to restore trust, or just to reduce suspicion? . . . Patients, it is said, no longer trust doctors . . . and in particular no longer trust hospitals or hospital consultants. "Loss of trust" is in short a cliché of our times. O'Neill, Onora. A Question of Trust. (BBC Reith Lectures 2002: Lecture 1. "Spreading Suspicion,"

April 6, 2002, pp. 2-3.) Retrieved February 3, 2004, from http://www.bbc.co.uk/radio4/reith2002/pdf/lecture1_text.pdf; also available at http://www.bbc.co.uk/radio4/reith2002/1.shtml

Cited in: MLA Guide, Introduction: http://www-personal.umich.edu/~pfa/mlaguide/free/power.html

What’s needed

Trust Respectful dialogueTools to support dialogue

Solution?

Yes, “trust your gut”, but …Train the “gut feeling”

Discuss & define common elementsArticulate contextual elementsDescribe environmental constraintsArticulate criteria

In other wordsSpell it out

Potential audience

Students, undergradStudents, grad & professionalPatientsHealthcare professionalsFriends and familyPersonal useAnyone

Challenges Easy to use Easy to remember Broad audience / multipurpose

Incoming freshmen Training of healthcare professionals

Broad range of information available Blurred media boundaries

traditional print peer-reviewed publications popular press & news media general web

Potential range of literacy skills Potential range of awareness of health information

quality issues & importance

Process

Review as many existing similar tools as discoverable, for all audiences, from elementary school children to professionals

Collect and organize concepts represented in the various tools Note frequency of distribution for common

concepts Note unique and valuable concepts

Prioritize Seek unifying structure

Other information quality assessment

tools

ABC (The Good, The Bad, & the Ugly): Accuracy, authority, bias/objectivity, currency,

coverage http://lib.nmsu.edu/instruction/evalcrit.html

Internet Detective http://www.vts.intute.ac.uk/detective/where.html

Five criteria (Cornell) http://www.vts.intute.ac.uk/detective/where.html

MedlinePlus: Guide to Healthy Web Surfing: http://www.nlm.nih.gov/medlineplus/

healthywebsurfing.html

Aha moment - “Great home page” Ahmad

Risk Full text:

<http://www-personal.umich.edu/~pfa/pro/greatrisk.html> Candour Honesty Quality Informed consent Privacy Professionalism Responsible partnering

That the website chooses its partner with care and that those partners strengthen the chain of trust among its community.

Accountability

Strategy

Keyword basedConcept clusteringLikert scaleLevels of evidence - extended

Assumptions

All information sources have some good information

All information sources have some bad information

Peer review tends to be better information, but cannot be guaranteed

Branding or certificate programs likewise tend to be better but cannot be guaranteed

Who watches the watchers?

Chain

CandorHonestyAccountabilityInformation qualityNeighborly

Trust

TimelinessRelevantUnbiasedScopeTrustworthy

Scoring

0 = No way5 = I really really like it!Lowest possible score = 0Highest possible score = 50Perfectly alright to use the tool without

the numbers - it is the ideas that are important.

Connect your gut feeling to the

evidence

Places information in contextMakes more overt aspects such as

Type of question being askedRange of information availableWhat is the best evidence availableEnvironmental factors or constraints

Levels of evidence in EBHC

A broader view

How do they relate?

An alternate view

What is “best available evidence”?

Example 1: Common condition, common topic, large research base? Gold standard: systematic reviews, meta-analyses,

randomized controlled clinical trials (RCTs, CCTs)

Example 2: Rare condition, little to no research? Expert opinion, If you can find it Patient perspective, support groups, personal web

pages

… and all the range between the two …

Get your own copy

From this page: http://www-personal.umich.edu/~pfa/pro/courses/

Horizontal version: http://www-personal.umich.edu/~pfa/pro/courses/

ChainOfTrustLoEHoriz2.pdf

Vertical version: http://www-personal.umich.edu/~pfa/pro/courses/

ChainOfTrustLoEVert2.pdf

More information

Links:http://del.icio.us/rosefirerising/

assessment+authority.credibility