developing a decision support system for tobacco use
TRANSCRIPT
Refereed papers
Developing a decision support system fortobacco use counselling using primarycare physiciansTheodore W Marcy MD MPHOffice of Health Promotion Research, and Vermont Cancer Center, University of Vermont College ofMedicine, USA
Bonnie Kaplan PhDYale Center for Medical Informatics, Yale University School of Medicine, Department of Biomedical andHealth Information Services, University of Illinois at Chicago, and Kaplan Associates, Hamden, CT, USA
Scott W Connolly EdD MPHOffice of Health Promotion Research, University of Vermont College of Medicine, USA
George Michel MS MBA
Richard N Shiffman MD MCISYale Center for Medical Informatics, Yale University School of Medicine, USA
Brian S Flynn ScDOffice of Health Promotion Research, and Vermont Cancer Center, University of Vermont College ofMedicine, USA
ABSTRACT
Background Clinical decision support systems
(CDSS) have the potential to improve adherence
to guidelines, but only if they are designed to work
in the complex environment of ambulatory clinics
as otherwise physicians may not use them.
Objective To gain input from primary care phys-
icians in designing a CDSS for smoking cessation toensure that the design is appropriate to a clinical
environment before attempts to test this CDSS in a
clinical trial. This approach is of general interest to
those designing similar systems.
Design and approach We employed an iterative
ethnographic process that used multiple evaluation
methods to understand physician preferences and
workflow integration. Using results from our priorsurvey of physicians and clinic managers, we devel-
oped a prototype CDSS, validated content and design
with an expert panel, and then subjected it to
usability testing by physicians, followed by iterative
design changes based on their feedback. We then
performed clinical testing with individual patients,
and conducted field tests of the CDSS in two
primary care clinics during which four physicians
used it for routine patient visits.
Results The CDSS prototype was substantially
modified through these cycles of usability and clin-
ical testing, including removing a potentially fataldesign flaw. During field tests in primary care clinics,
physicians incorporated the final CDSS prototype
into their workflow, and used it to assist in smoking
cessation interventions up to eight times daily.
Conclusions A multi-method evaluation process
utilising primary care physicians proved useful for
developing a CDSS that was acceptable to phys-
icians and patients, and feasible to use in theirclinical environment.
Keywords: medical informatics, qualitative re-
search, smoking cessation
Informatics in Primary Care 2008;16:101–9 # 2008 PHCSG, British Computer Society
TW Marcy, B Kaplan, SW Connolly et al102
Introduction
Computer-mediated clinical decision support systems
(CDSSs) are software systems that match characteristics
of a patient with a knowledge base of information onrecommended care in order to provide patient-specific
recommendations as well as other information man-
agement services.1–7 A number of CDSSs intended to
support the use of clinical guidelines have improved
physician adherence to guidelines, but others have
been unsuccessful.1,2,8,9
To be successful, a CDSS must work within the
complex environment of ambulatory clinics where thistechnology interacts dynamically with clinicians, patients
and existing office systems.10,11 Workflow integration
is one of the grand challenges for health information
technology. If physicians find these tools too difficult
to incorporate into clinical workflow they will be aban-
doned.12–14 To avoid such problems, a CDSS should
first reflect the needs and preferences of the users (e.g.
physicians) and the organisational systems (e.g. am-bulatory clinics) within which it works.10,14–17 Such a
CDSS should then be introduced into clinical practice
only after a ‘rigorous schedule of iterative usability
testing and formative evaluation’10 during which the
CDSS is modified to reflect the needs of the user and
the demands of the clinical environment.
We followed these recommendations during devel-
opment of a CDSS to assist physicians in using theUnited States Public Health Service (USPHS) Guideline
on Tobacco Use and Dependence Treatment. This
guideline recommends physicians perform the five
‘As’:
1 identify patients’ smoking status (‘ask’)
2 advise those who smoke to quit
3 assess readiness to quit
4 assist in quit attempts and
5 arrange for follow up.18
A CDSS for this guideline could guide physicians in
choosing and prescribing pharmacotherapy, facilitate
referral of patients to counselling resources and pro-
vide for patients a tailored handout with this informa-
tion, potentially improving adherence and effectiveness
and saving time.We employed the three-phase development process
of definition, usability testing, and clinical testing recom-
mended by Wyatt and Spiegelhalter (Figure 1).19 This
process used a multi-method ethnographic approach20
that included surveys of key stakeholders, iterative
usability testing with primary care physicians, validity
testing and consultation with an expert panel, initial
clinical testing with patients, and then pilot testing byphysicians.10,21,22
In the definition phase we surveyed 600 Vermont
primary care and subspecialty physicians and 93 clinic
office managers to determine current practice, the
environment within these ambulatory clinics, percep-
tions of barriers to performing smoking cessation
interventions and preferences among potential infor-
mation management services.4,23 This paper describesthe second and third phases of iterative design and
testing of the CDSS, including initial clinical testing
that demonstrated it was feasible for physicians to use
this CDSS in two primary care clinics.
Methods
Iterative usability and validity testing
The initial prototype was developed by the Yale Center
for Medical Informatics based on the responses to the
surveys,23 on the smoking cessation resources in
Vermont and neighbouring states and on the content
of the USPHS Guideline.18 The purpose of the second
phase was formative evaluation of the evolving CDSS.
Figure 1 A depiction of the process used to develop the tobacco use treatment computer-mediated decision
support system (CDSS)
Developing a decision support system for tobacco use counselling using primary care physicians 103
Our evaluation was guided by two theories relevant to
developing information technology that favor adop-
tion by users: the Technology Acceptance Model 2 and
Rogers’ Diffusion of Innovations.24,25 Semi-structured
interview items were constructed to address the fol-
lowing attributes from these two theories of the CDSSas viewed by the physicians:
. perceived usefulness including job relevance and
the quality of output. relative advantage over their current smoking cess-
ation interventions. compatibility with clinic systems. perceived complexity of the CDSS compared to
other information technology and their. intention to use the CDSS if it were available.
Because the small number of participants would make
valid analyses difficult, we chose not to utilise quanti-
tative measures of these perceptions. Qualitative methods
were chosen to complement our prior surveys and to
provide more rich, in-depth and nuanced information
than a quantitative questionnaire could provide.
Physician panels for usability testing
The working prototype was first subjected to usability
testing with three physicians active in the tobacco control
community. In the second round of testing, four
physicians were randomly selected from a list of primary
care physicians in Chittenden County, Vermont. Allseven physicians invited to participate agreed to do so.
Usability testing consisted of each physician using the
CDSS during hypothetical patient encounters presented
by a test monitor (SC) while an observer (TWM)
recorded the interactions. The testing combined three
sources of data:
1 a think-aloud protocol
2 handwritten field notes during observation
3 audio taped ethnographic interviews that included
the items derived from the two theories described
above.26
These interviews and observations were analysed by
two of the investigators (TWM and SC) using codes
based on the two theories. Validity was addressed by
reviewing our conclusions with the participants,27,28
by discussing results with expert panel members and
by having one investigator (BK), who was not present
at these sessions, review data analysis and interpret-
ation. Following these analyses, we developed a list of
design changes that were then incorporated into the
revised CDSS used in the next round of usability
testing.
Expert panel
We formed an expert panel consisting of three experts
on tobacco use treatment, one each in behavioural
counselling, pharmacotherapy and patient education,
and an additional physician with expertise in ambu-
latory clinic processes. These individuals reviewed the
CDSS’s validity as an implementation of the USPHS
Guideline and provided guidance on the content and
redesign of the CDSS throughout this process. We alsoconsulted with additional individuals who had exper-
tise in clinic information systems, Vermont’s tobacco
use cessation services, readability of patient handouts
and Medicare billing documentation and compliance.
Clinical testing
Testing with patients
The initial clinical testing of the modified CDSS was
performed in the ambulatory clinic of one of the
investigators (TWM). Once technical issues of trans-
ferring administrative data to the CDSS and printing
accurate documents were resolved, the investigator usedthe CDSS with consenting patients identified through
the clinic’s standard screening process as current or
recent (within one month) smokers. Attempts were
then made to interview these patients by telephone
within two weeks of this visit to assess the patients’
perceptions of the encounter and the CDSS.
Field tests in primary care clinics
Two physicians and the staff in each of two primary
care clinics agreed to field test the CDSS. These two
clinics were selected because they were not involved in
any other current outpatient research studies and they
used a common patient administrative database pro-
gram (GE Healthcare). At the end of the testing, each
physician was interviewed separately by TWM using asemi-structured interview guide based on the same
items as those in the usability testing. The interview
was audio taped for transcription and review. The
protocols for the usability and clinical testing were
reviewed and approved by the University of Vermont’s
Institutional Review Board.
Results
Phase 2: iterative usability testing
In our 2003 surveys, we found that 94% of the clinics
had a computerised registration system, but only 20%
TW Marcy, B Kaplan, SW Connolly et al104
of Vermont ambulatory clinics had an electronic health
record (EHR) into which a CDSS could be integrated,23
similar to a contemporary national estimate.29 There-
fore, a CDSS intended for wide adoption would need
to function in offices in which physicians might not
otherwise use a computer during patient visits. How-ever, the CDSS could potentially utilise information
from a computerised administrative database. Both
physicians and clinic office managers preferred that
the CDSS be on a handheld computer (PDA) for both
space and cost considerations23 and both groups
wanted the CDSS to:
1 provide patient-specific information
2 generate tailored patient handouts
3 utilise state of residence and type of health insur-
ance in forming recommendations and
4 document the intervention for the medical record.23
The initial prototype of our smoking cessation PDA
decision support system (SC-PDA) and all subsequent
prototypes of this CDSS used a web browser on a PDA
to connect via a wireless local area network to a serverin a ‘client–server’ relationship. Each day, a software
routine on the server pulled a flat file of data on
scheduled patients from the administrative database
that included the primary physician, insurance cover-
age, medical record number, date of birth and resi-
dence. The server also contained the CDSS algorithms
based on the USPHS Guideline18 and information about
local cessation resources. Based on input by the physi-cian on the PDA screens during the patient–physician
interaction, the server compiled and printed documents
at the checkout station with information specific to
the patient.30 Figure 2 provides a schematic of the
prototype system used in the clinical testing and
Figure 3 shows examples of two of the screens.
Prior research demonstrated that an active promptto use a CDSS was more effective than a passive system
relying on a physician to remember to use the sys-
tem.1,2 We, therefore, designed the first prototype so
that the vital signs and smoking status were recorded
on a computer at intake and then communicated via
the network to the physician’s PDA as an electronic
prompt that would alert the physician to consider
using the SC-PDA.All seven physicians who used the SC-PDA in simu-
lated clinical encounters provided numerous sugges-
tions for improvements or alterations. We modified
the SC-PDA in response to this feedback and that of
the expert panel as summarised in Table 1.
For illustrative purposes, we review the process that
led to a major revision in the SC-PDA: changing the
electronic alert to a paper-based prompt to use the SC-PDA. Two of three physicians in Round 1 of usability
testing had negative opinions about the electronic
alert. Without an EHR in the clinic they saw no value
in entering vital signs electronically other than for the
sole purpose of identifying smoking status, and viewed
this process as interfering with current systems for
patient intake. To gauge preferences, in the second round
of usability testing we presented both the electronicalert and an alternative paper-based prompt for using
Figure 2 A schematic of the final prototype that was tested in the clinical testing. Stickers on the clinic vital
sign record would prompt the physician to use the SC-PDA with appropriate patients based on smoking status.
The server acted as a central repository for patient administrative data and guideline information. The
physician communicated with this server over a wireless network via a PDA from which additional information
could be entered about the patient and through which information about patient-specific guideline
recommendations and resources were displayed. The server compiled and printed patient specific documents
based on these data elements
Developing a decision support system for tobacco use counselling using primary care physicians 105
the SC-PDA. The alternative method retained the
transfer of patient administrative data to the physician’s
PDA, but not the vital signs or smoking status. Instead,
the clinic’s existing vital signs record was adapted toindicate the smoking status by a coloured sticker that
could alert the physician to open the SC-PDA pro-
gram, select the specific patient from their roster,
confirm the smoking status and then proceed with
the intervention. Three of four primary care physicians
preferred the paper-based visual prompt because it
required the least change in their clinic intake system.The fourth physician did not like either version because
it was his opinion that asking about smoking status
every visit would alienate patients. Based on the majority
Figure 3 Examples of two of the screens on the PDA with which the physician interacts on the SC-PDA. In the
screen on the left, the physician would determine and then check any of the boxes corresponding to some of
the factors that would be cautions or contraindications to the use of one or more FDA approved medications
for smoking cessation. The screen on the right would then display the recommended medications in green
(light grey in the figure) and any cautioned or contraindicated medications in yellow or red (dark grey). In this
example, Bupropion is contraindicated because of the risk of lowering seizure threshold in patients with a
prior history of seizures or head injury
Table 1 Major design changes in the SC-PDA resulting from usability testing
Problem noted Design alterations
1 Electronic alert to use SC-PDA with appropriatepatients incompatible with clinic workflow and
systems
Alternative paper-based visual prompt on vital signssheet tested with physicians and then substituted
2 Time to obtain health history to review for
medication cautions/contraindications viewed asexcessive
Nicotine nasal spray deleted from medication
choices to reduce number of screened conditionsTwo-stage screening to exclude common
contraindications to initial medication use first
(pregnant, unstable angina, serious arrhythmia)
3 Unable to cycle from ‘not motivated’ to
‘motivated’ if patient changes intention to quit
during counselling
Button on screen recycles user back to ‘Assess’
readiness to consider quit attempt
4 Insufficient counselling guidance Optional screen with prompts allows MD toprovide more counselling if desired
5 Layout of tailored handouts not inviting;
language at too high a reading level
Tailored handout scripts reviewed and edited by
education and readability consultants
6 Need health record documentation that supports
Medicare billing
The documentation note was redesigned to supplement
the standard documentation of the visit and to
comply with requirements for billing for tobacco
cessation counselling
TW Marcy, B Kaplan, SW Connolly et al106
responses, we changed to the paper-based visual prompt
in the subsequent prototype.
Phase 3: clinical testing
Testing with patients
Nine patients (three female; six male) were recruited
to have one investigator, a physician, use the CDSS
with them. Seven of the patients were current smokers
and two were recent quitters. The sequence of screens
appropriate for each patient’s smoking status and
treatment preferences was completed in an average
time of ten minutes (range 8–13 minutes). All but one
of the nine patients actively looked at the PDA screenswith the physician during the process. Five of the nine
completed a structured telephone interview with
another investigator (SC) within two weeks of the
visit; one declined the interview, one did not recall the
SC-PDA, and two were unable to be contacted. Four of
the five interviewed patients rated the SC-PDA posi-
tively on its usefulness in assisting the discussion on
smoking; the other patient was neutral and none hadany negative comments about the SC-PDA or having
the physician use it with them. Some stated that being
able to follow the PDA screens made the questions
easier to understand and facilitated their decision-
making. Several design problems were discovered during
this phase of the testing, and the subsequent design
changes are outlined in Table 2.
Field tests in primary care clinics
Neither of the two participating clinics had a system
for identifying a patient’s smoking status. Working with
the clinic staff, the process of obtaining and recording
vital signs was adapted to include determination of
smoking status by intake personnel and using stickers
to indicate this status on the vital sign sheet. This
process was implemented before any further changes
were made so that adjusting to this system would beless likely to affect SC-PDA testing. After two weeks,
physicians were each given a PDA and trained on the
SC-PDA program in one session of approximately 60
minutes. Clinic personnel were trained on the hand-
ling of the tailored patient handouts, fax referral forms
and chart documentation notes. The participating
physicians used the SC-PDA during a trial period of
three weeks. Each physician’s assigned PDA only dis-played the patients on his or her own schedule. An
additional entry labelled ‘generic’ allowed the program
to be used with unscheduled patients, though without
the benefit of the administrative information.
The field tests in the two clinics were completed as
scheduled except for a single day when the SC-PDA
was inoperative because of a server malfunction. In the
physician interviews following the trial periods, thefrequency of use of the SC-PDA estimated by each of
the four physicians ranged between only four times in
three weeks to as many as eight times in one day out of
18–20 daily patient visits, with the most cited frequencies
as one and four times a day. The physician using the
Table 2 Design alterations adopted after initial clinical testing with patients
Problem noted Design alterations
1 In recent quitters, no ability to increase dosage
to address significant withdrawal symptoms
SC-PDA changed to allow physician to choose
appropriate dose of nicotine products based on
withdrawal symptoms
2 Prescription instructions not available at the end
of the SC-PDA algorithm when most physicians
write prescriptions
Last screen of the algorithm for each patient
provides prescription instructions for
recommended medications if any selected
3 If nicotine inhaler selected, physician needed to
be prompted to write a script
Pop-up screen added to remind physician that the
inhaler is not over the counter and requires a script
4 Printing order of medication instructions did
not follow recommendations if two selected
In some patients, nicotine patch is first medication
initiated, with addition of lozenge or gum only if
continued strong urges. Order of medication
instructions changed to correspond with this usual
sequence
5 Names of several of the MAO inhibitors
unfamiliar to physicians as they checked for
this contraindication to Bupropion use
Information button allows physicians to review
generic and brand names of medications with this
property while in screen to review medical history
cautions and contraindications to specific
medication use
Developing a decision support system for tobacco use counselling using primary care physicians 107
SC-PDA the least stated that he had few opportunities
as there were few active smokers in his panel of patients.
The clinic support staff corroborated the frequency of
use of the SC-PDA based on their handling of the
patient tailored handouts distributed at checkout.
Two of the physicians reported that the ability toshow the screens to the patient and have them ‘rally
around’ the computer and follow along with the steps
of making recommendations was a relative advantage
of the SC-PDA over standard counselling.
‘... so it was sort of a different approach. Was not just a
provider saying ‘‘look, I want you to quit’’. It was a
provider saying, ‘‘I want you to quit and we have this
program that we’re using that could really maximize your
chances at success’’.’ (Physician 1)
All four physicians commented favourably on the
ability to personalise both the spoken and written
counselling, check for medication cautions and contra-
indications, and document their services to support
billing as well as insurance coverage of medications.
The two physicians in the first trial clinic found thetime required to use the SC-PDA was a relative dis-
advantage. With probing, we learned to reduce user
time by emphasising during training that the SC-PDA
could print a tailored handout even if physicians chose
not to do the intervention and that physicians could
skip the quit date screen if they preferred. Prior to the
trial in the second clinic, we adapted the training to
highlight these options and eliminated a counsellingselection screen. The two physicians in the second trial
clinic both reported that they could use the SC-PDA
efficiently. The physician in this clinic who used it the
most stated;
‘It actually is pretty quick and easy. I suspect that most of
my interventions were in the lower end of three to ten
minutes.’ (Physician 4)
This physician also commented that she would tend tobill for the service when using the SC-PDA, whereas
before she had not, partially compensating for the
additional time spent. Notably this physician had not
previously used a PDA.
Discussion
We developed a CDSS for smoking cessation inter-
ventions hand-in-hand with physicians – the intended
end users. We used an ethnographic approach in order
to design a CDSS that provided advantages over how
physicians usually advise patients about smoking, thatwas compatible with clinic workflow, and that was inte-
grated with how physicians conduct patient visits.14,21
Through usability testing, we were better able to
understand the workflow of the clinical practice, trans-
late these preferences into the design of our SC-PDA
and then implement the SC-PDA in two ambulatory
clinics. This experience illustrates the value of a staged
multi-method ethnographic process for obtaining
detailed end user feedback during the developmentof health information technology.
For example, we first incorporated an electronic
alert into the prototype SC-PDA because this charac-
teristic had been associated in the literature with im-
provements in physician adherence. Usability testing
demonstrated, however, that physicians were resistant
to a different method of identifying smoking status
because of its perceived impact on existing office sys-tems. This opinion was not apparent in our surveys of
physicians and clinic managers. Had we retained the
electronic alert, we could have experienced multiple
failed attempts during implementation. Additional
problems with the SC-PDA became apparent only
when using it with patients in the clinical setting.
Our pilot tests in two clinics demonstrated that
physicians did use the final SC-PDA prototype in actualclinical settings, and did see a relative advantage of the
SC-PDA over usual practice. The time required to use
the SC-PDA is a recognised barrier to preventive care
in general and tobacco counselling in particular.31,32
We took several measures to address this in our design.
First, additional information about medications and
counselling was provided in screens that were available if
desired, but it was not necessary to work throughthese. Second, we acted on physician recommendations
to streamline information and reduce the content and
number of screens. Third, the SC-PDA could produce a
tailored handout for a patient even if the physician
opted not to use the intervention during a patient visit.
An unanticipated observation was that both patients
and physicians saw benefits in having the patient view
the screens along with the physician. This sharedviewing of screen contents engaged the patient and
appeared to improve patient understanding. A CDSS
designed for patients to use by themselves would not
capitalise on this interaction. The future development
of decision support systems should exploit a CDSS’s
potential for interactive, collaborative decision making.
There are limitations to this approach to CDSS
development. The time-intensive nature of usabilitytesting reduces the number of end users who can
provide feedback. Those who accept an invitation to
do usability testing may not be representative of other
physicians. In addition, a CDSS developed through
this process still may not increase guideline adherence
or improve patient outcomes even if physicians use
the CDSS. Clinical trials are necessary to address these
questions. However, this type of CDSS developmentshould precede any large-scale testing to avoid ex-
pensive null results from a CDSS that physicians will
not use during these clinical trials.
TW Marcy, B Kaplan, SW Connolly et al108
A particular limitation to our methods was that one
of the developers (TWM) recruited the physicians and
clinics selected for usability testing and was present
during the usability testing. However, other evaluators
(SC and BK) were not involved in the SC-PDA devel-
opment. Wears et al recommend that the developersof a health information technology system should not
be the evaluators of the system as there is the potential
for bias.11 The developer/evaluator may selectively
record criticisms and the subjects may be less open
with the developer about problems they perceive.
Conclusions
Successful integration of health information tech-
nology into clinical practice will require collaborative
development of these systems with physicians, patients
and support staff. The combination of multiple and
iterative ethnographic methods incorporating surveys,usability testing and expert panels is feasible and
useful. Through this process we avoided costly and
potentially fatal errors in the design of our CDSS.
ACKNOWLEDGEMENTS
We gratefully acknowledge the participation of our
physician and patient subjects in this study and the
assistance from members of our expert panel. Finan-cial support of this research was provided by grants
from the National Cancer Institute: NCI R03 CA111640
and Dr Marcy’s career development award NCI K07
CA102585–01.
REFERENCES
1 Garg A, Adhikari N, McDonald H et al. Effects of
computerized clinical decision support systems on prac-
titioner performance and patient outcomes: a systematic
review. Journal of the American Medical Association 2005;
293:1223–38.
2 Kawamoto K, Houlihan C, Balas E and Lobach D.
Improving clinical practice using clinical decision sup-
port systems: a systematic review of trials to identify
features critical to success. British Medical Journal 2005;
330:765–72.
3 Blumenthal D and Glaser J. Information technology
comes to medicine. New England Journal of Medicine
2007;356:2527–34.
4 Shiffman R, Brandt C, Liaw Y and Corb G. A design
model for computer-based guideline implementation
based on information management services. Journal of
the American Medical Informatics Association 1999;6:99–
103.
5 Committee on Quality of Health Care in America of the
Institute of Medicine. Crossing the Quality Chasm: a new
health system for the 21st century. Washington, DC:
National Academy Press, 2001.
6 Frist W. Health care in the 21st century. New England
Journal of Medicine 2005;352:267–72.
7 James B. Making it easy to do it right. New England
Journal of Medicine 2001;345:991–3.
8 Hunt D, Haynes R, Hanna S and Smith K. Effects of
computer-based clinical decision support systems on
physician performance and patient outcomes. Journal of
the American Medical Association 1998;280:1339–46.
9 Shiffman R, Liaw Y, Brandt C and Corb G. Computer-
based guideline implementation systems: a systematic
review of functionality and effectiveness. Journal of
the American Medical Informatics Association 1999;6:
104–14.
10 Hesse B and Shneiderman B. eHealth research from the
user’s perspective. American Journal of Preventive Medi-
cine 2007;32:S97–S103.
11 Wears R and Berg M. Computer technology and clinical
work: still waiting for Godot. Journal of the American
Medical Association 2005;293:1261–3.
12 Lee J, Cain C, Young S, Chockley N and Burstin H. The
adoption gap: health information technology in small
physician practices. Health Affairs 2005;24:1364–6.
13 Wachter R. Expected and unanticipated consequences
of the quality and information technology revolutions.
Journal of the American Medical Association 2006;295:
2780–3.
14 Elson R. Clinical practice guidelines: the role of tech-
nology in perspective. Disease Management Health Out-
comes 1997;1:63–74.
15 Kaplan B, Brennan P, Dowling A, Friedman C and Peel
V. Towards an informatics research agenda: key people
and organizational issues. Journal of the American Medi-
cal Informatics Association 2001;8:235–41.
16 Friedman C. Information technology leadership in aca-
demic medical centers: a tale of four cultures. Academic
Medicine 1999;74:795–9.
17 Littlejohns P, Wyatt J and Garvican L. Evaluating
computerised health information systems: hard lessons
still to be learnt. British Medical Journal 2003;326:860–3.
18 Fiore M, Jaen C, Baker T et al. Treating Tobacco Use and
Dependence: 2008 update. Clinical Practice Guideline
Rockville, MD: US Department of Health and Human
Services Public Health Service, May 2008.
19 Wyatt J and Spiegelhalter D. Evaluating medical expert
systems: what to test and how? Medical Informatics 1990;
15:205–17.
20 Kaplan B. Evaluating informatics applications: clinical
decision support systems literature review. International
Journal of Medical Informatics 2001;64:15–37.
21 Kaplan B and Shaw N. Future directions in evaluation
research: people, organizational, and social issues.
Methods of Information in Medicine 2004;43:215–31.
22 Kaplan B. Addressing organizational issues into the
evaluation of medical systems. Journal of the American
Medical Informatics Association 1997;4:94–101.
23 Marcy T, Skelly J, Shiffman R and Flynn B. Attitudes and
opinions of physicians and clinic managers towards
clinical decision support systems to improve adherence
to the tobacco use treatment guidelines. Preventive
Medicine 2005;41:479–87.
Developing a decision support system for tobacco use counselling using primary care physicians 109
24 Venkatesh V and Davis F. A theoretical extension of the
technology acceptance model: four longitudinal field
studies. Management Science 2000;46:186–204.
25 Rogers E. Diffusion of Innovations (5e). New York, NY:
Free Press, 2003.
26 Kaplan B, Drickamer M and Marottoli R. Deriving
design recommendations through discount usability
engineering: ethnographic observation and thinking-
aloud protocol in usability testing for computer-based
teaching cases. In: Musen M (ed) Proceedings of the 2003
AMIA Annual Symposium. Bethesda, MD: AMIA, 2003,
pp. 346–50.
27 Kaplan B and Maxwell J. Qualitative research methods
for evaluating computer information systems. In:
Anderson J and Aydin C (eds) Evaluating the Organ-
izational Impact of Healthcare Information Systems. New
York: Springer, 2005, pp. 30–55.
28 Smith III A. Design and conduct of subjectivist studies.
In: Friedman C and Wyatt J (eds) Evaluation Methods in
Medical Informatics. New York: Springer, 1997, pp. 223–
53.
29 Hing E and Burt C. Office-based Medical Practices:
methods and estimates from the National Ambulatory
Medical Care Survey. Advance Data from Vital and
Health Statistics. Hyattsville, MD: National Center for
Health Statistics, 2007.
30 Michel G, Marcy T and Shiffman R. A wireless, handheld
decision support system to promote smoking cessation
in primary care. In: Friedman C (ed) Proceedings of the
American Medical Informatics Association Symposium,
2005. Bethesda, MD: AMIA, 2005, pp. 530–4.
31 Jaen C, Stange K, Tumiel L and Nutting P. Missed
opportunities for prevention: smoking cessation coun-
seling and the competing demands of practice. Journal of
Family Practice 1997;45:348–54.
32 Yarnall K, Pollak K, Ostbye T, Krause K and Michener J.
Primary care: is there enough time for prevention?
American Journal of Public Health 2003;93:635–41.
CONFLICTS OF INTEREST
None.
ADDRESS FOR CORRESPONDENCE
Theodore W Marcy MD MPH
Office of Health Promotion Research
University of Vermont College of Medicine
1 South Prospect Street, 4th Floor Arnold
Burlington
Vermont 05401–3444, USATel: +1 802 656 3650
Fax: +1 802 656 8826
Email: [email protected]
Accepted May 2008
Preliminary findings reported in this paper were presented in an abstract at the 2006 American Society of
Preventive Oncology and in an abstract at the 2006 Translating Research into Practice and Policy (TRIPP)
conference.