an evaluation framework for health information
TRANSCRIPT
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
journa l homepage: www. int l .e lsev ierhea l th .com/ journa ls / i jmi
An evaluation framework for Health InformationSystems: human, organizationand technology-fit factors (HOT-fit)
Maryati Mohd. Yusofa,∗, Jasna Kuljisb,Anastasia Papazafeiropouloub, Lampros K. Stergioulasb
a Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysiab School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK
a r t i c l e i n f o
Article history:
Received 15 December 2005
Received in revised form
12 August 2007
Accepted 12 August 2007
Keywords:
Information Systems
Health Information Systems
Evaluation
Framework
Human factors
Organizational factors
a b s t r a c t
Background and purpose: The realization of Health Information Systems (HIS) requires rig-
orous evaluation that addresses technology, human and organization issues. Our review
indicates that current evaluation methods evaluate different aspects of HIS and they can
be improved upon. A new evaluation framework, human, organization and technology-fit
(HOT-fit) was developed after having conducted a critical appraisal of the findings of existing
HIS evaluation studies. HOT-fit builds on previous models of IS evaluation—in particular, the
IS Success Model and the IT-Organization Fit Model. This paper introduces the new frame-
work for HIS evaluation that incorporates comprehensive dimensions and measures of HIS
and provides a technological, human and organizational fit.
Methods: Literature review on HIS and IS evaluation studies and pilot testing of developed
framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary
care organization in the UK. The case study was conducted through observation, interview
and document analysis.
Results: The main findings show that having the right user attitude and skills base together
with good leadership, IT-friendly environment and good communication can have positive
influence on the system adoption.
Conclusions: Comprehensive, specific evaluation factors, dimensions and measures in the
new framework (HOT-fit) are applicable in HIS evaluation. The use of such a framework is
argued to be useful not only for comprehensive evaluation of the particular FIS system under
investigation, but potentially also for any Health Information System in general.
with each other, the greater the potential of HIS. Most existing
1. IntroductionThe benefits derived from a Health Information Systems (HIS)require rigorous evaluation. It is claimed that organizationaland social issues are the main components of such a sys-
DOI of original article:10.1016/j.ijmedinf.2007.08.004.∗ Corresponding author. Tel.: +603 8921 6649.
E-mail addresses: [email protected], [email protected] (M.1386-5056/$ – see front matter © 2007 Elsevier Ireland Ltd. All rights resdoi:10.1016/j.ijmedinf.2007.08.011
© 2007 Elsevier Ireland Ltd. All rights reserved.
tem [1]. The more technology, human and organization fit
Mohd. Yusof).
evaluation studies of HIS focus on technical issues or clinicalprocesses, which do not explain why HIS works well or poorlywith a specific user in a specific setting [2–7].
erved.
a l i n
dpaddwfiiatw[t[
tactwettc6s
2
TaoijeAicMtHMH
piHrItfbtsbtd
i n t e r n a t i o n a l j o u r n a l o f m e d i c
The central purpose of this paper is to show how weerived a framework for HIS evaluation that incorporates com-rehensive dimensions and measures of HIS and providestechnological, human and organizational fit. A case study
emonstrates the application of the proposed framework forescribing the critical adoption factors of a particular HISith a particular user in a particular setting. This proposed
ramework, human, organization and technology-fit (HOT-fit)s constructed so that it is capable of being useful in conduct-ng a thorough evaluation study. It is also possible that it willssist researchers and practitioners to unfold and understandhe perceived complexity of HIS evaluation. The new frame-ork builds on previous work on the review of HIS evaluation
8,9]. It also makes use of two models of IS evaluation, namelyhe IS Success Model [10,11] and the IT-Organization Fit Model12].
This paper is organized as follows. Section 2 discusseshe theoretical background of the proposed framework. Theforementioned models are presented to explore their appli-ability in improving on those used in Health Informatics. Thishen forms the basis for our first proposed evaluation frame-ork for HIS presented in Section 3. Section 4 discusses an
xisting Fundus Imaging System in a primary care organiza-ion that is used as a research case study. Section 5 presentshe research methodology for applying our framework to thisase study. The case study findings are presented in Section. Finally, discussion and conclusions are given in the lastection.
. Theoretical background
he proposed human, organization and technology-fit evalu-tion framework was developed after a critical investigationf the existing findings of HIS and IS evaluation stud-
es (discussed in the preceding paper in this issue of theournal, [13] and from feedback gained from two confer-nce presentations of earlier stages in the research [8,9]).
review of success determinants of Inpatient Clinical ISndicates that the categories for success in the IS Suc-ess Model can be used to assess HIS [14]. The IS Successodel has then been identified as being complementary
o another model in fulfilling the limitations of existingIS evaluation frameworks, namely the IT-Organization Fitodel. As a result, both models are utilized in constructingOT-fit.
The IS Success Model is adopted because of its com-rehensive, specific evaluation categories, extensive val-
dation and its applicability to HIS evaluation [10,11].OT-fit makes use of the IS Success Model in catego-
izing its evaluation factors, dimensions and measures.n addition, the IT-Organization Fit Model [12] is usedo complement the IS Success Model by integrating itseatured organizational factors and the concept of fitetween the human, organizational and technological fac-ors. The two models are discussed in the last two
ub-sections of this section prior to HOT-fit’s developmenteing explained in Section 4. But first some background tohe human and organizational aspects is necessary for lateriscussions.f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 387
2.1. Human and organizational factors
The importance of human and organizational factors in thedevelopment and implementation of IS has been advocated inthe IS literature. Rigorous evaluation of HIS can be undertakenby addressing these factors as well as the fit or alignmentbetween them. According to Willcocks [15], the alignment oforganization, technology and human is an important startingpoint in IT implementation as it is one of the strategies thataffect IT investment.
Barriers to using HIS are also important to consider in HISevaluation as they explain the failure and success of thesesystems. Culture and process changes are reported to be thebarriers to the wider use of health care systems [16]. Studiescited in Anderson [17] identified a number of barriers to directphysician use of HIS including low level of expertise, lack ofacceptance, lack of medical staff sponsorship and alterationof traditional workflow patterns. Examples of organizationalchallenges include hospital culture, such as being risk adverse,reluctance to invest much in IT and resistance to change[18,19].
A study on factors influencing success and failure of HIS bya group of medical informaticians identified 110 success fac-tors and 27 failure criteria from a wide range of socio-technicalissues [20]. These factors are assessed for six types of HIS.Highest success factors for Clinical Information Systems (CIS)are: collaboration and cooperation, setting goals and courses;while for educational system is user acceptance. Highest fail-ure criteria for CIS is response rate; while for administrativesystem is not understanding the organizational context andnot foreseeing the extent to which new HIS affects the orga-nization, its structure and/or work procedures.
In short, human and organizational factors are as impor-tant as technical issues with regards to system effectiveness[4]. Human, organizational and technical elements should alsohave a mutual alignment or ‘fit’ in order to ensure successfulHIS implementation. It is crucial that HIS fit organizationalaspects as well as align with work routines, managementassumptions, patient care philosophies and users’ needs asthe introduction of a system affects different dimensionsof fit in complex ways [3]. A number of studies in HealthInformatics have included the concept of ‘fit’ in explainingthe interdependent relationship between human, organiza-tion and technology factors [21,22]. Aarts et al. [21] propose amodel which illustrates the stages of information and systemchanges and their relative specific “personal requirements”.They argued that changes in both technology and health prac-tices affect each other to a similar extent. Berg [22] makesuse of the socio-technical approach in HIS evaluation; workpractices are seen as integrated networks of various relatedelements such as people, tools, organizational processes,machines, and documents.
Southon et al. [23] found that the lack of fit among mainorganizational elements contributes to a large number ofsystem failures in public health. The fit between techni-cal, organizational, and social factors is analyzed to identify
gaps between current health care systems and new sys-tem features [24]. Kaplan [3] shows that poor fit betweensystem developers’ goals and clinicians’ cultural values con-tributes to user reluctance to use Clinical Decision Support
i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
Fig. 2 – The MIT90s (IT-Organization Fit Model) (adapted
388 i n t e r n a t i o n a l j o u r n a l o f m e d
Systems (CDSS). Executive support, understanding the busi-ness, IT-business relations, and leadership are identified asboth enablers and inhibitors of the fit of IT and business[25].
2.2. IS Success Model
In order to structure numerous results from research on ISsuccess categories; a comprehensive taxonomy is introduced[10]. A model is constructed which consists of six success cate-gories or dimensions; they are linked causally and temporallyas success is viewed as a dynamic process instead of a staticstate. The multidimensional relationships among the mea-sures of IS success have been tested extensively in a numberof IS studies [26]. Based on these studies, an updated versionof this model is presented [11] (see Fig. 1).
These measures are included in the six system dimensions:System Quality (the measures of the information processingsystem itself), Information Quality (the measures of IS output),Service Quality (the measures of technical support or service),Information Use (recipient consumption of the output of IS),User Satisfaction (recipient response to the use of the outputof IS) and Net Benefits (the overall IS impact).
Because of its circularity, the framework might lead to spi-ral behaviors in both positive and negative directions. Forexample, effective use of the system will result in higher netbenefits which lead to more intensive use of the system. Incontrast, insufficient system use will yield to lower net ben-efits; thereby acting as a disincentive to system use. In bothcases, the IT-Organization Fit Model highlights the importantrole of the organization in managing the process of changesthat take place during system introduction [12]. Effective pro-cess management can lead to increased user acceptance andparticipation in system use, which will in turn help achievehigh net benefits and subsequently increase system use.
In comparison with existing HIS frameworks, DeLone andMcLean’s IS model illustrates clear, specific dimensions of ISsuccess or effectiveness and the relationships between them.However, it does not include organizational factors that arepertinent to IS evaluation. In addition, Van der Meijden etal. [14] discovered that a number of measures such as userinvolvement during system development and organizationalculture do not match any of the dimensions of the framework.
The extension of this framework is recommended by addingthe organizational factors, their dimensions and clinical mea-sures related to the healthcare domain.Fig. 1 – Information System Success Model (Source: DeLoneand McLean [11]).
from Scott Morton, [12]).
2.3. IT-Organization Fit
Management in the 1990s (MIT90s) is a well-known IT-Organization Fit Model, which includes both internal andexternal elements of fit [12]. Fig. 2 illustrates the conceptof fit between the main organizational elements. Internal fitis accomplished by a dynamic equilibrium of organizationalcomponents including business strategy, organizational struc-ture, management processes, and roles and skills. Externalfit is achieved by formulating organizational strategy basedon environmental trends and changes such as market, indus-try and technology. Within this internal and external fit as itsenabler, IT is expected to affect the management process, thusimpacting on organizational performance and to some degree,its strategy.
In order to realize the benefits of IT, three prerequi-sites are required for successful IT transformation. First,organizational vision and the reasons behind it have to beclear to organizational members to get them prepared fororganizational changes and hence reduce the challenges inmanaging transformation. Second, organizational corporatestrategy (business and IT), information technology and organi-zational dimensions have to be aligned with each other. Third,a robust IT infrastructure such as an electronic network andunderstood standards should be equipped within the organi-zation. These three prerequisites as well as the internal andexternal fit may be used to identify the problems in IT imple-mentation. This model was relatively new and has not beenextensively utilized in healthcare [27]. The model was alsoidentified as being capable of identifying the main organiza-tional elements which can affect IS as well as emphasizing theessential alignment or fit between them. Moreover, the modelis comprehensive as it includes the following factors: technol-ogy (IT), human (roles and skills) and organization (strategy,structure and management process). However, these factorscan be categorized into more detailed dimensions to providemore specific evaluation dimensions. For instance, IT can befurther classified into system quality and information quality,as proposed by DeLone and McLean [10]. Similarly, roles andskills can be associated with use and user satisfaction.
Based on the strengths and limitations pointed out in bothmodels, IT-Organization Fit and the IS Success Model comple-ment each other in presenting a comprehensive evaluationframework. Organizational factors, which are lacking in the IS
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 389
ion-
Sllce
3
BtiseMteufMwF
•
••
•
•
cbsQ
Fig. 3 – Proposed human-organizat
uccess Model, are featured in the IT-Organization fit. Simi-arly, specific evaluation dimensions and measures which areacking in the IT-Organization fit, are featured in the IS Suc-ess Model. Based on the two models explained above, a newvaluation framework is presented in the next section.
. Proposed evaluation framework
uilding on previous studies on the evaluation approach [8,9],he proposed evaluation framework was developed after a crit-cal appraisal of the existing findings of HIS and IS evaluationtudies. It makes use of the IS Success Model in categorizing itsvaluation factors, dimensions and measures. The IS Successodel is adopted based on its comprehensive, specific evalua-
ion categories, extensive validation and its applicability to HISvaluation. In addition, the IT-Organization Fit Model is alsosed to incorporate the concept of fit between the evaluation
actors: human, organization and technology. The IS Successodel was extended by the addition of the following featureshich are explained in the following part of this section (see
ig. 3):
Organization factors, their dimensions (Structure and Envi-ronment) and evaluation measures (listed in Table 1).Fit between technology, human and organization factors.Two-way relationships between these dimensions: Informa-tion Quality and System Use, Information Quality and UserSatisfaction, Organizational Structure and Environment, Organi-zational Structure and Net Benefits, Organizational Environmentand Net Benefits.One-way relationship between these dimensions: Structureand System Use.New evaluation measures pertinent to HIS and IS in general.
Human, organization and technology are the essential
omponents of IS; the impacts of HIS are assessed in the netenefits. These three factors and the impacts of HIS corre-pond to eight interrelated dimensions of HIS success: Systemuality, Information Quality, Service Quality, System Use, Usertechnology fit (HOT-fit) framework.
Satisfaction, Organizational Structure, Organizational Environmentand Net Benefits. Each of these dimensions is associated with anumber of evaluation measures. Examples of evaluation mea-sures according to their corresponding dimension and factorare listed in Table 1.
These evaluation dimensions influenced each other in atemporal and causal way:
• System Quality, Information Quality and Service Quality singu-larly and jointly affect both System Use and User Satisfaction.
• Organizational Structure and Organizational Environmentaffects System Use.Some of these relationships are two ways:
• System Use, which relies on user knowledge and training,can influence the Information Quality, since the user’s knowl-edge in using the system can affect reports, images andprescriptions produced by the system.
• The level of System Use can affect the degree of User Satis-faction and vice versa, for both positive and negative cases.Effective System Use yield to higher User Satisfaction as useris able to explore and make full use of system featuresand functions; higher User Satisfaction subsequently moti-vate/lead user to increase System Use.
• Similarly, the Organizational Environment factors such asgovernment policy and politics can affect OrganizationalStructure while factors in Organizational Structure will affectthe population served in the Organizational Environment.
• System Use and User Satisfaction are direct antecedents ofNet Benefits. Net Benefits subsequently affect System Use andUser Satisfaction. Similarly, Organizational Structure and Envi-ronment are direct antecedents of Net Benefits. Net Benefitssubsequently have impact on organizational Structure andEnvironment.
The concept of fit is perceived as complex, abstract andsubjective. It can be viewed in terms of strategic planning
(formulating IS plan according to organizational plan) andstrategic alignment (managing IT closely with organizationalneeds) perspectives [23]. In the HOT-fit context, fit is concernedwith the ability of HIS, human (HIS stakeholders and clinical
390 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l
Tabl
e1
–Ex
amp
les
ofth
eev
alu
atio
nm
easu
res
ofth
ep
rop
osed
fram
ewor
k
Tech
nol
ogy
Hu
man
Org
aniz
atio
nN
etbe
nefi
ts
Syst
emq
ual
ity
Info
rmat
ion
qu
alit
ySe
rvic
eq
ual
ity
Syst
emu
seU
ser
sati
sfac
tion
Stru
ctu
reEn
viro
nm
ent
Dat
aac
cura
cy,
dat
acu
rren
cy,
Dat
abas
eco
nte
nts
,eas
eof
use
,eas
eof
lear
nin
g,av
aila
bili
ty,
use
fuln
ess
ofsy
stem
feat
ure
san
dfu
nct
ion
s,fl
exib
ilit
y,re
liab
ilit
y,te
chn
ical
sup
por
t,se
curi
ty,
effi
cien
cy,
reso
urc
eu
tili
zati
on,
resp
onse
tim
e,tu
rnar
oun
dti
me
Imp
orta
nce
,re
leva
nce
,u
sefu
lnes
s,le
gibi
lity
,for
mat
,ac
cura
cy,
con
cise
nes
s,co
mp
lete
nes
s,re
liab
ilit
y,ti
mel
ines
s,d
ata
entr
ym
eth
ods
Qu
ick
resp
onsi
ven
ess,
assu
ran
ce,
emp
ath
y,fo
llow
up
serv
ice,
tech
nic
alsu
pp
ort
Am
oun
t/d
ura
tion
:(nu
mbe
rof
inq
uir
ies,
amou
nt
ofco
nn
ect
tim
e,n
um
ber
offu
nct
ion
su
sed
,nu
mbe
rof
reco
rds
acce
ssed
,fre
qu
ency
ofac
cess
,fre
qu
ency
ofre
por
tre
qu
ests
,nu
mbe
rof
rep
orts
gen
erat
ed),
use
byw
hom
?(d
irec
tvs
.ch
auff
eure
du
se,)
actu
alvs
.re
por
ted
use
,nat
ure
ofu
se(u
sefo
rin
ten
ded
pu
rpos
e,ap
pro
pri
ate
use
,ty
pe
ofin
form
atio
nu
sed
,)p
urp
ose
ofu
se,l
evel
ofu
se(g
ener
alvs
.sp
ecifi
c,)r
ecu
rrin
gu
se,r
epor
tac
cep
tan
ce,p
erce
nta
geu
sed
,vo
lun
tari
esof
use
,mot
ivat
ion
tou
se,a
ttit
ud
e,ex
pec
tati
ons/
beli
ef,
know
led
ge/e
xper
tise
,acc
epta
nce
,re
sist
ance
/rel
uct
ance
,tra
inin
g
Sati
sfac
tion
wit
hsp
ecifi
cfu
nct
ion
s,ov
eral
lsa
tisf
acti
on,
per
ceiv
edu
sefu
lnes
s,en
joym
ent,
soft
war
esa
tisf
acti
on,
dec
isio
nm
akin
gsa
tisf
acti
on
Nat
ure
,(ty
pe,
size
)cu
ltu
re,p
lan
nin
g,st
rate
gy,
man
agem
ent,
clin
ical
pro
cess
,au
ton
omy,
com
mu
nic
atio
n,
lead
ersh
ip,t
opm
anag
emen
tsu
pp
ort,
med
ical
spon
sors
hip
,ch
amp
ion
,m
edia
tor,
team
wor
k
Fin
anci
ng
sou
rce,
gove
rnm
ent,
pol
itic
s,lo
cali
zati
on,
com
pet
itio
n,
inte
r-or
gan
izat
ion
alre
lati
onsh
ip,
pop
ula
tion
serv
ed,
exte
rnal
com
mu
nic
atio
n
Cli
nic
alp
ract
ice
(Job
effe
cts,
task
per
form
ance
,p
rod
uct
ivit
y,w
ork
volu
me,
mor
ale,
)ef
fici
ency
,eff
ecti
ven
ess
(goa
lach
ieve
men
t,se
rvic
e),d
ecis
ion
mak
ing
qu
alit
y(a
nal
ysis
,acc
ura
cy,
tim
e,co
nfi
den
ce,
par
tici
pat
ion
),er
ror
red
uct
ion
,co
mm
un
icat
ion
,cli
nic
alou
tcom
es(p
atie
nt
care
,m
orbi
dit
y,m
orta
lity
,)co
st
i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
practices) and setting to align with each other. Thus, fit can bemeasured and analyzed from multiple compatibility betweenhuman, organization and technology (human-organization,human-technology, organization-technology) using a numberof measures defined in the three factors including systemflexibility, system’s ease of use, system usefulness, informa-tion relevancy, user attitude, user training, user satisfaction,organizational culture, planning, strategy, management andcommunication. For example, an effective use of a HIS can beattributed to the fit between system flexibility and clinical pro-cess (system-human). The fit between human, organizationand technology is illustrated by the bold arrows in Fig. 3.
Based on its comprehensive dimensions and outcomemeasures, the framework could be used to evaluate theperformance, effectiveness and impact of HIS or IT in health-care settings. Effectiveness refers to the accomplishment ofspecific goals with accuracy and completeness, as well asthe correct utilization of appropriate resources [28]. In thisresearch, effectiveness is defined as the ability of a health-care organization to continuously accomplish goals usingoptimum resources within a specified time. The three evalua-tion factors can be evaluated through out the whole systemdevelopment life cycle namely planning, analysis, design,implementation, operation and maintenance. Meanwhile, Netbenefits can be anticipated before implementation and appar-ently evaluated after implementation. As mentioned above,each phase focuses on different issues. This framework canbe applied using qualitative, quantitative or a combination ofboth approaches. Subsequently, any data collection methodsfrom both approaches can be employed while conducting theevaluation.
The following subsections explain the evaluation factors,dimensions and measures in detail.
3.1. Technology
The studies on System Quality are often associated with systemperformance. System Quality in a healthcare setting measuresthe inherent features of HIS including system performanceand user interface. Examples of system quality measuresare ease of use, ease of learning, response time, usefulness,availability, reliability, completeness, system flexibility, andsecurity [29–33]. Ease of use assesses whether healthcare pro-fessionals regard HIS as satisfactory, convenient and pleasantto use. Availability refers to the up time of HIS while flexibilityis concerned with the ability of HIS to adapt to a healthcaresetting and integrate with other systems. Even systems thatoften work are often not used as anticipated. Thus, it is impor-tant to determine whether the system (1) meets the need ofthe projected users, (2) is convenient and easy to use, (3) fitsthe work patterns of the professionals for whom it is intendedand the overall health system [30].
Measures of Information Quality are concerned with infor-mation produced by HIS including patient records, reports,images and prescriptions. Information quality measures can
be subjective, as they are derived from the user perspec-tive. Criteria that can be used for HIS quality are informationcompleteness, accuracy, legibility, timeliness, availability, rel-evancy, consistency and reliability [23,29,30,33,34].
a l i n
ewhSqs
3
Tihtf
opmtmia
[prAtmfst(tbtritpvs
ItetrH
3
TftbcLs
i n t e r n a t i o n a l j o u r n a l o f m e d i c
Service Quality is concerned with the overall support deliv-red by the service provider of HIS or technology, regardless ofhether the service is delivered by the internal department ofealthcare organization or outsourced to external providers.ervice quality can be measured through technical support,uick responsiveness, assurance, empathy and follow upervice.
.2. Human
he IS Success Model includes the use of system and user sat-sfaction while MIT90s incorporates roles and skills as part ofuman factors elements. HOT-fit combines those human fac-ors in both frameworks in addition to other factors of humanactors as mentioned earlier.
System Use is concerned with the frequency and breadthf HIS inquiries and functions. The use of information out-ut such as reports appears to be one of the most frequenteasures to assess the success of IS. The actual use of the sys-
em as a measure of IS success refers to voluntary instead ofandatory use. System Use also relates to the person who uses
t, their levels of use, training, knowledge, belief, expectationnd acceptance or resistance [30,33].
Knowledge is concerned with computer literacy and skills7,17]. Expectation refers to the anticipation of improvedatient care delivery from the use of HIS [7]. Jiang et al. [35]egard resistance as an important factor of system success.s different types of systems are usually related with a par-
icular type of function and user, the reasons for resistanceight differ among system types. Resistance can be viewed
rom one of the following theories: (1) people-oriented, (2)ystem-oriented and (3) interaction-oriented. People-orientedheory describes resistance to system results from user’sgroups or individuals) internal factors. Personal characteris-ics such as age, gender, background, value and belief haveeen suggested as influencing individual’s attitude towardsechnology. System oriented theory suggests that resistanceesults from system design factors or relevant technologyncluding user interface and system characteristics. Interac-ion theory explains resistance from the interaction betweeneople and system factors; thus, assessment of a systemaries across settings and users. Job insecurity and fear areome examples of interaction resistance.
User Satisfaction is often used to measure system success.t is subjective in nature as it depends on whose satisfac-ion is measured. User satisfaction is defined as the overallvaluation of a user’s experience in using the system andhe potential impact of the system. User Satisfaction can beelated to user’s perceived usefulness and attitudes towardsIS which are influenced by his/her personal characteristics.
.3. Organization
he nature of a healthcare institution can be examinedrom its structure and environment [36]. Organization struc-ure consists of nature including type and size (number of
eds), culture, politic, hierarchy, autonomy, planning andontrol systems, strategy, management and communication.eadership, top management support and medical staff spon-orship can also be measured from the organization factorsf o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 391
[17,23]. The environment of a healthcare organization can beanalyzed through its financing source, government, politics,localization, the type of populations being served, competi-tion, inter-organizational relationship, population served, andcommunication.
3.4. Net benefits
A system can benefit a single user, a group of users, anorganization or an entire industry. Net Benefits capture thebalance of positive and negative impacts on user, whichincludes clinicians, managers and IT, staff, system devel-opers, hospitals or the entire healthcare sector. Individualimpact is the effect of information on the behavior ofthe recipient. It is associated with performance as well aschanges in user task (clinical practice) such as job perfor-mance, change in work activity and improved productivity[7,30]. Thus, individual Net Benefits can be assessed using jobeffects, efficiency, effectiveness, decision quality, and errorreduction.
Organizational impact is the effect of information on orga-nizational performance. In the healthcare context, clinicaloutcomes can be used as a means of measurement. Exam-ples of these measures include costs reduction, which is dueto fewer medication errors and adverse drug effect (ADE);improved efficiency in patient care delivery, specifically per-taining to tests and drug orders and increased use of genericdrug brands and number of consultations and length of wait-ing lists [7,37]. Clinical outcomes are also measured throughtwo criteria: morbidity (the rate of incidence of a disease) andmortality (death rate). Apart from these quantitative mea-sures, clinical impacts can also be assessed qualitatively usingthese measures: quality of care, impact on patient care andcommunication, such as change in communication style andfacilitation of information access [38].
4. The case of Fundus Imaging System (FIS)
Our research design consists of a case study strategy. The casestudy serves dual purposes: (1) to evaluate the adoption fac-tors of HIS in the context of the phenomena under study; (2) tovalidate the proposed HOT-fit evaluation framework. The casestudy is also undertaken to obtain a comprehensive view andunderstanding of the development process of a HIS describedin this section. The case study facilitated the conceptualisa-tion of HIS adoption tending towards success and failure, inassociation to the factors involved and their relationships. Wemake use of the case study to answer our research question,as well as to test the applicability of the proposed frameworkin being a useful evaluation tool.
A case study is conducted in the clinical settings of a pri-mary care organization (PCO) and two of its collaboratingspecialist hospitals, all members of the UK National HealthService (NHS). The GP practice was established in the 1920sand now serves around 6000 patients who are mainly the
elderly. The practice is actively engaged in medical and nurs-ing research, training and education. A number of HIS werealready in place, including telemedicine and General PracticeInformation Systems (GPIS).
i c a l
holders. During this period, in addition to the FIS, the use of aGeneral Practitioner Information Systems was also observedin order to obtain an overall view of the general applicationand the attitudes of the PCO staff towards Health Informa-
Table 2 – List of participants for the case of FIS
Participant Total (N) Initial used indata analysis
User (a GP) 1 Dr. ABCSenior partner 1 Dr. MNOGP 2 Dr. DEFNurse 5 STUPhysiotherapist 1 GHIIT staff 1 PQR
392 i n t e r n a t i o n a l j o u r n a l o f m e d
This study evaluates the development of a digital FundusImaging System (FIS) for diabetic retinopathy. While evaluat-ing FIS, a GPIS was also studied in order to investigate theoverall adoption factors of HIS of the specific users in theparticular setting. FIS consists of a Fundus camera and dig-ital imaging software that is used to capture the eyes imagesof patients with diabetes. FIS has functions for image cap-ture, image manipulation, patient data management, imagetransfer, viewing and temporary storage. For each capturedeye image, the system assembles the image, patient data,and timer data into a data file. These data files are trans-ferred to the GPIS by attaching each image to associatedpatient.
The UK government has recommended that all patientswith diabetes have their annual retinal screening using digitalretinal fundoscopy [39]; the FIS is part of this initiative. FIS isrecognized for its importance in preventing blindness amonga large population of diabetic patients and in improving thecontrol of diabetic care.
Traditionally, retinal fundoscopy for such patients has beencarried out in a hospital or by an optometrist. A GP had to refera patient to a hospital to be invited for annual screening. Therecall system in hospital is seen as inefficient in terms of trav-eling cost and accessibility, particularly to the elderly patientwho represents a significant population of the PCO. The inef-ficiency of traditional screening motivated the partners at thePCO to implement a retinal fundoscopy screening within theirpractice.
The idea to develop the system was initiated by the seniorpartner, who is a GP himself, after a Fundus camera waspurchased. The primary care screening aims to improve thepatient pathway by providing better patient education—GPscould educate patients during the screening by showingthem their eye images and immediately discuss the clin-ical implications and subsequently improve their diabeticcontrol. In addition, the patient pathway could be morecost effective through reduced time, effort and expense oftravel and double appointments, which allow patients tohave both retinal screening and diabetic health check on thesame day.
The retinal screening of the FIS is done by a GP, Dr. ABC, whois also the sole user of the system. The screening took place ina separate examination room equipped with a digital Funduscamera with a touch screen display, and PCs to transfer patienteye images into patient records which are featured in the GPIS.However, the retinal screening is temporarily discontinued, asthe existing camera does not comply with the guideline of theNational Screening Committee, which requires more advanceequipment.
The attempt to implement the FIS was continued by col-laborating with a leading specialist hospital. A group of eyespecialists who include Consultant Ophthalmologists and aReading Centre Advisor was liaised with to obtain expertadvice in terms of selecting alternative patient pathways,purchasing digital camera and relevant software, and train-ing for using the system and grading the eye images. During
this study period, major system development and improve-ment decisions with regards to the technology procurement,staffing, quality assurance and possible number of patientpathways are still under negotiation.i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
5. Methodology
The approach used in this study was that of a subjectivist, casestudy strategy employing qualitative methods. A formativeevaluation was undertaken of the adoption of FIS to identifysystem problems as they emerged and to improve the systemas it was developed [40]. A subjectivist approach was employedin order to gain an extensive understanding of the health-care context surrounding the FIS through detailed, insightfulexplanation of the study [34]. Further, qualitative methodswere employed to generate a fuller description of the health-care setting and its cultural issues and to understand why thesystem functioned well or poorly in a particular setting. Thefield study was conducted between April and September 2005by one of the authors. A number of data collection methodswere employed, including interviews, participant observationand document/artifact analysis.
A purposeful, snowball sampling method [41] was used inorder to gain in-depth information from key informants aboutthe development of the FIS. Participants were identified fromthe researcher’s initial contact with individuals known to staffmembers. After identifying an initial group of participants, anetwork was built by asking these first participants to suggestadditional participants for interview. In the end 15 participantswere found among clinicians, staff and patients who wereassociated with FIS, both from the primary care organizationand from the two specialist hospitals involved in the study. Inparticular, the list of participants is as follows (Table 2).
Our research approach to the evaluation of the FISconsisted of six iterative phases, which included problemidentification, the development of an initial evaluation frame-work, the selection of a research strategy and methods, systemevaluation, framework validation, and refinement of the eval-uation framework (Fig. 4) [34,36].
All phases were completed. Evaluation problems (issues,questions and concerns) were identified through a literaturereview as well as observations made during an immersion. Theimmersion was carried out to set the general context of theresearch, as well as to establish rapport with relevant stake-
Ophthalmologist Consultant 1 Mr. VWXReading Centre Advisor 1 Dr. JKLPatients 2
Total 15
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 393
m Fr
tibspbcaa
eoaisAeps
awpmstmaccaoot
tt
Fig. 4 – Research design (adapted fro
ion Systems. Initial data collection was gathered during themmersion. An initial evaluation framework was constructedased on the findings from this first phase. The researchtrategy and methods were selected based on the researchroblem. The original framework in Part I of this paper haseen employed initially for the FIS case study. After the con-lusion of the study, the framework was eventually validateds well as refined in terms of its evaluation measures based onn appropriate mix of current literature and case study results.
The HOT-fit framework was used as a guideline in thevaluation of the FIS. During system evaluation, participantbservations of daily clinical routines, meetings, discussionsnd social events took place in different departments, clin-cs and nursing homes of the primary care organization andpecialist department of the collaborating specialist hospitals.
good rapport between the researcher and participants wasstablished at both formal and informal occasions. Partici-ants were aware of the role of the researcher in their clinicalettings.
During observations and face-to-face interviews, individu-ls including users, clinicians and IT staff that were involvedith the system were queried about their system use andatient pathways. General background information on the pri-ary care organization and its clinical and administrative
taff were gathered to obtain a holistic and in-depth view ofhe clinical setting. This view included aspects of manage-
ent, facilities, vision, culture, politics, conflict, leadershipnd of the staff in terms of their way of working, communi-ation, relationships and rapport, as well as attitudes towardsomputer and IT literacy. Meanwhile, patients were queriedbout their perception about the system. Data were collectedn planned occasions as well as spontaneously in a numberf iterative cycles. The data were audio- and hand-recorded,
ranscribed into field notes, and analyzed.Based on the HOT-fit framework, four techniques were usedo analyze the results: coding, analytic memos (such as reflec-ion notes, displays, and concept maps), and contextual and
iedman and Wyatt [33]; Kaplan [35]).
narrative analysis [40]. The field notes were fully transcribedon margin-marked paper. The margins were used to note anyreflections, themes relevant to HOT-fit and statements whichwere unclear or needed to be confirmed with the participants.The data were coded and categorized under similar themesor concepts of the HOT-fit framework and refined throughout a series of analyses. These codes corresponded to eachfactor, dimensions and measures described in the HOT-fitframework (see Fig. 3). Further, texts under the same cate-gory were compared to identify variations and nuances inmeanings. Categories were compared to discover connectionsbetween themes. Concept maps were drawn to understandthe relationship between the number of concepts involved inthe evaluation of FIS. Contextual and narrative analyses weredone based on the themes assigned to the codes and these arefurther covered in the next section of this paper.
Two tests were used to establish the quality of this empir-ical research [42]:
1. Construct validity: establishing correct operational mea-sures for the concepts being studied. This is concerned withexposing and reducing subjectivity, by linking data collec-tion questions and measures to research questions [43]. Inthis study, the evaluation measures in the interview ques-tions were identified based on the proposed frameworkand then linked with the research objectives. The resultsshowed that the interview questions addressed each of theresearch questions to a reasonable degree.
2. Reliability: demonstrating that the operation of the study– such as the data collection produced – can be repeatedwith the same results. This was achieved through detaileddocumentation of procedures and appropriate record keep-ing [43]. Activities during immersion were recorded in a
detailed fieldwork log.Potential bias generally acknowledged in qualitativeresearch approach was overcome by conducting a reliability
i c a l
394 i n t e r n a t i o n a l j o u r n a l o f m e dtest and data triangulation. Triangulation was done throughthe use of multiple evidences from different sources to con-firm the same fact or finding [42]. For example, a certain factobtained from a GP was confirmed with a different GP as wellas the organization documentation and observation. Feedbackfrom participants regarding a particular conclusion in the set-ting studied is also gathered to reduce bias.
6. Discussion of the findings from theFundus Imaging System case
Based on observations that took place during the immer-sion and pilot study, a number of emerging themes from theframework, namely human, organizational and technologicalfactors and net benefits have been identified. These issues arepresented in the next sections, which are organized in termsof the three themes.
6.1. Technology
The general practice where the study took place is some-what advanced with its use of modern technology and is wellequipped with computing and telecommunications devices.Currently, all staff members are using a General Practice Infor-mation System which features electronic patient records aswell as service and management for GPs. A number of theassociate GPs dislike the GPIS. According to Dr. DEF, it is com-plicated and hard to learn - “I hated it!” was her literal response.In comparison to the GPIS, the old system that she used inher previous workplace “was template based and made typ-ing and data entering much easier”. In contrast to the GPIS,the ease of use of the Fundus Imaging System (FIS) can beseen immediately as a user friendly, simple to use interfaceand straightforward data entry mode. Captured images canbe manipulated, stored and attached to the patient record.
However, the inefficiency of FIS is viewed from its incon-venient storage capacity. The frequent need to delete andtransfer files from a small size memory card (attached to thecamera) to a hard drive when it gets full was seen as tediousand time consuming. The user, Dr. ABC said “it is a pain to keepon deleting the [images on the] memory card when it is full andit gets full quickly because it has small storage capacity”. Shehad to take the card out and upload the patient’s images to thePC and linked them with the patient records and saved themin a specific folder. The contents of the memory card will bedeleted for future use.
In addition to system inefficiency, slow response time is alsoseen as a disincentive for using the GPIS. For example, physio-therapist GHI has to recall a number of exercises appropriateto a patient’s condition in order to suggest which of themshould now be followed by the patient. Although she couldaccess their details on the system, she does not use the sys-tem because from her point of view, it takes too long for her toprint the list of exercises for the patient (2 min). So she requirespatients to remember their own lengthy list of exercises.
With regards to the system under investigation (FIS), thereare some problems related to the less mature technologiesinvolved in its development. For example, the existing cam-era had to be replaced, and the use of the current FIS was
i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
temporarily halted because the camera did not comply withthe National Screening Committee guideline, which was pub-lished after the camera was purchased.
In terms of Information Quality, two aspects were analysed:relevancy and completeness. The relevancy of the eye imageslies in their usefulness in educating patients to take controlof their diabetes. In addition to educating patients effectively,it is very important to produce high-quality, accurate imagesto avoid erroneous diagnosis. However, images taken with theexisting camera are not as detailed and as accurate comparedto those provided by hospitals. According to the Reading Cen-tre Advisor JKL, in terms of screening standards, the imagesare not good enough to detect eye abnormalities; however, inpractice and for the intended purpose, the images are goodenough.
Service Quality can be observed and determined from theusage of GPIS. The service providers of the primary care organi-zation comprise of external vendors and a member of internalIT staff. The internal IT specialist is trusted by the senior part-ner of the practice, Dr. MNO who argued, “He is difficult to dealwith but he is always right about something”. It seems thatthe service providers give little empathy to the clinicians thatthey are serving. According to the senior partner, the exter-nal service providers “do not care about doctors’ needs at all;they just care about their businesses”. It is also perceived thatthe user also receives too little support from the IT staff whentechnical assistance is needed. When she requested technicalsupport for a particular camera feature, a member of the ITstaff, Mr. PQR responded by saying “There’s nothing that canbe done about it”, adding “He started giving me all the jargon”.
6.2. Human
The Fundus Imaging System (for diabetic retinopathy) wasdeveloped for a very specific clinical purpose—to have a GPuser educate patients with diabetes in improving their con-trol of diabetic care. This purpose was viewed as impracticalby the collaborating specialists in terms of cost since it is muchcheaper to employ technicians to screen the patient via theFIS. This view was not taken into serious consideration untilmonths after FIS was in place, where a large amount of timeand effort were already allocated by the GP in planning, devel-oping, training and using the system.
According to nurse STU, although initially most of the usersof the primary care organization have minimal IT skills, theirwillingness to use the system was a key contributing factorthat put the system in place. As mentioned earlier, some of theGPs were not supportive of the system because, in their view,it was difficult to use. The remaining staff believe that the useof GPIS and other HIS had assisted them in performing theirjobs better; tasks were completed faster and communicationsbetween staff and doctors were improved.
The user of FIS on the other hand has acknowledged herlimited IT skills. However, the user was motivated to use thesystem with the assistance provided by a staff member at theoutset of FIS implementation. The barrier to using the sys-
tem can be seen from the lack of familiarity of the user inusing basic file organization functions, such as copying files,and in performing time consuming and tedious tasks such asdeleting and transferring files. This process resulted in user
a l i n
dipm
otei
6
TntaHwuzFdcdbnbrwppsbataot
oiaappeabatIctptlisse
i n t e r n a t i o n a l j o u r n a l o f m e d i c
issatisfaction because in her view, it is taking a lot of her lim-ted time. Another example of unfamiliarity was seen when aop up window was displayed and the user said, “I hate theseessages, I don’t understand what they are talking about”.Despite her busy schedule, the user attended a series
f training organized outside the practice. She needs to berained as an accredited screener to be able and eligible toxamine the images and inform patients of the initial screen-ng results.
.3. Organization
he senior partner played a major role in shaping the orga-izational culture and establishing the use of technology inhe organization. He was the leader with a long-term visionnd strategy to keep abreast with technology advancement.e also formed a number of collaborations and partnershipsith other primary and secondary care organizations, andniversities to exploit new technologies in medicine. Organi-ational readiness has been established in several occasions.or instance, research meeting is held on a regular basis toiscuss existing and new technology adoption and clinicaloncern pertinent to technology, including the FIS. Informaliscussions during lunchtime between various staff mem-ers, which includes the senior partner, associate GPs andurses, have build a strong rapport and good communicationetween them. Everyone is respected and treated equallyegardless of their position. Everyone is also encouraged toork as a team in performing their daily tasks. As nurse STUut it, “This place is different from other places. The seniorartner really made a difference by creating a teamworkpirit”. The teamwork spirit among the staff members cane seen in the decision making process. They tend to seekdvice from their colleagues who have expertise in an areahey themselves do not feel very confident with. For instance,
GP seek advice from the nurses regarding the critical casef a home care resident and a physiotherapist consulted a GPo check on the effect of a drug–drug interaction.
In addition to the rather advanced computing facilities, therganization has an environment that is conducive to work
n; for example, manual and electronic records are kept inn orderly manner. According to the user, motivation, person-lity, and encouragement are the catalytic factors that madeossible the adoption and use of the system. After the seniorartner assigned the role of FIS user to her, he continued toncourage the GP to change her perception about her role assole service provider and convincing her that she is capa-
le of using the system. The initiative to use the system waslso encouraged by other staff members. Without these fac-ors, Dr. ABC pointed out that “. . .. I would not take the role.t was not easy for me to accept that role initially, it was a bighange. I was very reluctant because I have a different pic-ure of a GP’s role”. In her view, a GP only comes in, sees 30atients, writes the letters, checks the results, and sends themo hospital if necessary. She believed that the senior partner’seadership has changed her perception, which made her real-
zed that she can develop professionally and change. Dr. ABCtated that change can be done in two ways: nurtures a per-on to make him feel that he can do it or it puts him down; forxample, “you start to criticise your colleague by saying you aref o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 395
not good enough, you should do more. Usually encouragementwill work. It’s like the parenting role”. She used this exampleto refer to the initiation of the FIS in this general practice.
The uptake of the FIS is also attributable to medical spon-sorship in the practice. The user recalled the time when thecamera was not being used after it was purchased. The seniorpartner kept on teasing the user, using his ‘gentle persuasive’skill, prompting her to use the system by saying somethinglike “Why don’t you have a play with it?” After a few reluctantinitial attempts, the user finally took pictures of a nurse’s eyes,which was the starting point of diabetic retinopathy screeningin this practice.
Three types of communication were observed in thisclinical setting: patient–physician, physician–physician andstaff–physician. In terms of working style, although eachGP and nurse consults their patients in different ways,a commonality in their communication is made obvious:patient–physician relationships in terms of interpersonalcommunication and eye contact are essential during consul-tations.
The communication between the user and eye specialistis limited by their busy schedule. On several occasions, a jointmeeting was postponed by a few months and this had affectedthe speed of FIS implementation. This lack of communicationhas resulted in a difference of understanding about projectimplementation. In addition, the conflict of interest betweenthe project leader, the user and the eye specialist team in termsof technology procurement, training cost and the role of userhas also delayed the uptake of FIS. After months of meetingand training, the PCO has considered assigning the role of cur-rent user, the GP, to an external screener since the time spentby the GP for training, screening and grading would be moreexpensive than hiring the screener. This situation, however,has been envisaged by one of the Consultant Ophthalmolo-gists earlier and he had informed the user but it was not takenseriously.
Meanwhile, although the communication between staff isgood, there is a common lack of communication between ITstaff and clinicians, which results in conflicts. This is an indi-cation of a typical problem of communication gap betweentechnical staff and users. The IT staff also uses jargon thatmakes the communication more difficult and confusing to thedoctors. The senior partner commented that one of the IT staff“always leaves us in a mess. He does not tell us exactly whatis going on and when the service provider meets the staff, welook like idiots!” He also said that, “All service providers arerubbish!”
Focusing on the external environment of this organization,there are some problems with the communication betweenthis primary care and secondary care organization, whichinclude miscommunication and error in reporting. For exam-ple, a patient received the wrong medical report—the pain wasreported to be in the left shoulder instead of the right one. Inanother instance, physiotherapist GHI stated, “The communi-cation between hospital and the primary care organization isdreadful in terms of patient medical history. The hospital did
not give any referral letter or information that explains theprevious medical diagnoses and it is difficult and time con-suming to get hold of these documents. I have to guess toidentify the causes of the patient’s pain”.
i c a l
396 i n t e r n a t i o n a l j o u r n a l o f m e d6.4. Net benefits
In terms of the net benefits of the new system, it seems thatmost of the practitioners have acknowledged certain effectson patient care and clinical practice. According to Dr. ABC, thebenefits of primary care screening can be assessed through: (1)patients’ control in their diabetic care, (2) equivalency or bettertreatment in primary care than the traditional screening interms of speedy treatment and (3) low error rate and patient’ssatisfaction.
It has been recognized that the FIS has the potential toimprove the delivery of patient care by providing better patienteducation. For example, Dr. ABC said to one of the patientsthat “these are your blood vessels, they are so tiny; if there’sany problem with them, it will cause major problems withyour vision. The cause of the problem is poor diabetic con-trol”. She believes that getting patients see their eye picturesmay enable them realize that their diabetic control is notgood and they are at a risk of losing their vision. Accord-ing to Dr. ABC, one way to find out whether the diabeticcontrol is improved by this sort of patient education is bya blood test in the diabetic clinic. The GP quoted an exam-ple of an obese patient who took a lot of liberty with hisdiet and had very bad diabetic control. The GP changed thepatient’s attitude by showing to the patient pictures of hiseyes and discussing possible problems that may occur in rela-tion to his diet. As a result, the patient came back, havinglost a couple of kilos and told the GP that she effectivelymade him change the whole way he thought about his dia-betes condition. In addition, the senior partner, Dr. MNOsaid, “By her (the GP’s) involvement in taking pictures ofthe patients’ eyes, she took up a marvellous opportunity tochange her patients’ attitude disorder. That is a fantastic wayof influencing patients’ behaviour about managing their ownhealth”.
The partners of this GP practice envisaged that the imple-mentation of FIS will benefit patient by saving their timeand cost of travelling, thus enable them to be screened ear-lier. This shorter patient pathway will also reduce possibleerror rates as data are stored and access directly within thepractice.
On the other hand, some negative effects have also beenrecognized. The use of FIS has increased the GP’s work vol-ume. She has to spend additional hours in the retinal cliniccapturing the eye images, as well as storing and trans-ferring them to the patient notes. In addition, she has tospend a few hours per month for regular training sessions.As a result, there have been certain occasions when, dueto accessibility problems in the FIS, she was late for hersurgeries.
6.5. Fit between human, organization and technology
Based on these three factors, their fit with each other hasbeen recognized. The uptake of FIS was contributed by theuser’s strong acceptance and personality to learn using the
system (fit between human and technology) and the technicalsupport provided by a staff member that acted as a systemchampion. However, a lack of internal fit can also be seenbetween technology and human: storage inconvenience andi n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
time and effort required from the busy GP user, as well aspoor technical support and user need for assistance. Moreover,the mismatch between clinical processes with technology wasillustrated by the GP’s busy schedule with time required fortraining. External fit of the practice is achieved by formulat-ing its strategy according to current IT trends and advances.However, the lack of external fit in FIS was shown in thenon-compliance of current FIS equipment with the Nationalguideline, which caused the system to be discontinuedtemporarily.
7. Further discussion and conclusions
The case of FIS has demonstrated the importance of havingthe right user attitude and knowledge in order to be able touse the system effectively and efficiently. Arguably, choos-ing the right people to fill the user role is more importantthan possessing the required skills, as skills can be acquiredlater. This can be seen from the use of FIS, which was pri-marily driven by the willingness of the GP to learn and useit after being persuaded by the senior partner and despiteher limited computing literacy. Knowledge can be acquiredthrough appropriate training; however, physicians have a verybusy schedule. Thus, alternatives such as having replacementphysicians can be taken to enable the user to attend thistraining.
The practice is distinguished by being a research practiceand having a good leadership. Such organizational culture andleadership has created awareness of technical advancementsamong the staff members as well as expedited the adoptionof HIS, as shown by the uptake of telemedicine, GPIS and FIS.The alignment of the organizational strategy with IT and theexistence of up-to-date computing infrastructure have alsofacilitated the implementation of HIS in the practice.
The barriers to system use are contributed by the user per-ception, ease of use, response time and clinical process. Theuser was initially reluctant to use the system due to her per-ception of her role solely as a service provider. GPs have such aheavy daily workload. This explains why response time is cru-cial in the adoption decision. The same goes with the trainingissue. It is quite impossible for a GP in the PCO to attend aseries of time consuming training sessions while still havingto attend her regular surgery sessions.
Meanwhile, the communication gap between cliniciansand the IT staff is obviously caused by the knowledge gap aswell as individual characteristics towards being more sensitiveto different stakeholders’ needs.
Communications between all staff at all levels are crucialto ensure that the purposes and benefits of an HIS are under-stood since communication problem can be costly. One way toachieve effective communication is through leadership. In thiscase study, leadership as well as top management support hasproven to be an important starting point and has major influ-ence in the realization of HIS. In addition to leadership andtop management support, medical sponsorship also played a
significant role in changing user perception and encouragingsystem use.Furthermore, external factors such as government policycan largely affect the viability of HIS. For example, although
a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 397
Fhlaw
iccps
aasmmaefictp
wwnottttcturtmoorwtatcto
trusatsftk
Summary points
What was known before the study:
• There is a large number HIS evaluation frameworkslooking at different aspects of these systems.
• The existing evaluation methods do not provideexplicit evaluation categories.
• More work on human and organizational issues iscalled for as most existing evaluation studies of HISfocus on technical issues or clinical processes which donot explain why HIS work well or poorly with a specificuser in a specific setting.
What the study has added to the knowledge:
• Previous work on the evaluation of Information Sys-tems is reviewed.
• An evaluation framework for HIS, which incorpo-rates the concept of fit between human, organizationand technology (HOT-fit), is proposed using a multi-disciplinary approach.
• The application of the proposed evaluation frameworkis demonstrated in a real-life, practical context whereformal evaluation methods have not been or could nothave been used.
• Insights shed from the findings of the case study that
r
i n t e r n a t i o n a l j o u r n a l o f m e d i c
IS had already been in place and operational, its use had to bealted due to the fact that the camera did not comply with the
atest National Guidelines. Thus, it is very important to plannd develop a system and keep it up-to-date in accordanceith the latest policy.
FIS can potentially improve healthcare delivery by provid-ng better patient education. Other potential benefits of FISan be identified in terms of patient control of their diabeticare and the provision of equivalent or better treatment inrimary care than secondary care organizations in terms ofpeedy treatment and patient satisfaction.
A number of challenges have been encountered whilepplying the HOT-fit framework in data analysis. First, therere a number of data that share the same evaluation mea-ures; a careful consideration has to be taken in selecting theost appropriate measures for each data. For example, com-unication can be classified under organizational structure
nd environment; problems in using the system can be cat-gorized under of technology factors (ease of use) or humanactors (system use). Otherwise, a repetition of the same datan different category can be used where necessary. Second, thelassification of data analysis according to the three evalua-ion factors has affected the flow of the narrative approach ofresenting the data and can result in confusion to the reader.
In order to validate its usefulness, the proposed frameworkas tested in a real clinical setting. The system put on the testas a Fundus Imaging System in a NHS primary care orga-ization. The description of the case study and its findingsffer a strong indication of the applicability of the frameworko HIS evaluation. A number of critical factors to the adop-ion of FIS have been identified; factors that had influencedhe adoption negatively include: system usefulness, responseime, technical support, empathy of service quality, user per-eption and user skills. Meanwhile, factors contributing tohe positive adoption of FIS include: information relevancy,ser attitude, leadership, medical sponsorship, organizationaleadiness, clinical process and external communication withhe inter-organizational system (the eye specialist). The align-
ent of IT and organizational strategy has led to the initiationf a number of systems, including FIS. The strong willingnessf the user to change her perception and clinical practicesesulted in the uptake of FIS. However, the adoption of FISas disrupted with the incompatibility of the system with
he National Guidelines, as well as a lack of technical supportnd limited communication between the technical staff andhe collaborating partners of the specialist’s hospital. We thusonclude that human, organizational and technological fac-ors and the fit between them are essential in the realizationf FIS.
Although our case study focused on a specific setting,he proposed evaluation framework is potentially useful toesearchers and practitioners for conducting thorough eval-ation studies of other HIS or IT applications in healthcareettings. As proposed here, the framework can and should bepplied in a flexible way, taking into account different con-exts and purposes, stakeholders’ point of views, phases in
ystem development life cycle, and evaluation methods. Theramework is not the solution to any problem; it is a struc-ured debating tool that stakeholders can access in order tonow their own system health better.can be used to inform decision making.
Acknowledgements
We gratefully acknowledge the funding received from both thePublic Service Department of Malaysia and from the UniversitiKebangsaan Malaysia (National University of Malaysia) thathelped sponsor this study.
e f e r e n c e s
[1] D.F. Sittig, B.L. Hazlehurst, T. Palen, J. Hsu, H. Jimson, M.C.Hornbrook, A clinical information system research agendafor Kaiser Permanente, Permanente J. 6 (3) (2002).
[2] B. Kaplan, Evaluating informatics applications—clinicaldecision support systems literature review, Int. J. Med. Inf.64 (1) (2001) 15–37.
[3] B. Kaplan, Evaluating informatics applications-somealternative approaches: theory, social interactionism, andcall for methodological pluralism, Int. J. Med. Inf. 64 (1)(2001) 39–56.
[4] B. Kaplan, N. Shaw, People, organizational and social issues:evaluation as an exemplar, IMIA 2002 Yearbook (2002)91–102.
[5] B. Kaplan, N.T. Shaw, Future directions in evaluationresearch: people, organizational, and social issues, Methods
Inf. Med. 43 (3) (2004) 215–231.[6] J. Brender, C. Nohr, P. McNair, Research needs and prioritiesin health informatics, Int. J. Med. Inf. 58–59 (2000) 257–289.
[7] E. Coiera, Guide to Health Informatics, Hodder Arnold, 2003.
i c a l
398 i n t e r n a t i o n a l j o u r n a l o f m e d[8] M.M. Yusof, R.J. Paul, L. Stergioulas, Health InformationSystems Evaluation: A focus on Clinical Decision SupportsSystem. Proceedings of the XIX International Congress ofthe European Federation for Medical Informatics (MIE2005),Geneva, Switzerland, 2005, pp. 855–860.
[9] M.M. Yusof, R.J. Paul, L.K. Stergioulas, Towards a Frameworkfor Health Information Systems Evaluation. Proceedings ofthe 39th Hawaii International Conferences on SystemSciences, Kauai, Hawaii, USA, 2006.
[10] W.H. DeLone, E.R. McLean, Information systems success: thequest for the dependent variable, Inf. Syst. Res. 3 (1) (1992)60–95.
[11] W.H. DeLone, E.R. McLean, Measuring e-commerce success:applying the DeLone & McLean Information SystemsSuccess Model, Int. J. Electron. Commerce 9 (1) (2004) 31–47.
[12] M.S. Scott Morton, The Corporation of the 1990s, OxfordUniversity Press, New York, 1991.
[13] M.M. Yusof, A. Papazafeiropoulou, R.J. Paul, L.K. Stergioulas,Investigating evaluation frameworks for health informationsystems, Int. J. Med. Inf. (2007).
[14] M.J. Van der Meijden, H.J. Tange, J. Troost, A. Hasman,Determinants of success of inpatient clinical informationsystems: a literature review, J. Am. Med. Inform. Assoc. 10 (3)(2003) 235–243.
[15] L. Willcocks, Managing technology evaluation—techniquesand processes, in: R.D. Galliers, B.S.H. Baker (Eds.), StrategicInformation Management: Challenges and Strategies inManaging Information Systems, Oxford, 1994, pp. 365–381.
[16] K.A. Kuhn, D.A. Giuse, J.L. Talmon, The HeidelbergConference: setting an agenda for the IMIA working Groupon Health Information Systems, Int. J. Med. Inf. 69 (2–3)(2003) 77–82.
[17] J.G. Anderson, Clearing the way for physicians’ use ofclinical information systems, Commun. ACM 40 (8) (1997)83–90.
[18] K. Bottles, Critical choices face healthcare in how to useinformation technology, Medscape Gen. Med. 1 (1) (1999).
[19] N.M. Lorenzi, R.T. Riley, Organizational ISSUES = change, Int.J. Med. Inf. 69 (2–3) (2003) 197–203.
[20] J. Brender, E. Ammenwerth, P.N. Nyakanen, J. Talmon,Factors influencing success and failure of HealthInformation System – a pilot delphi study, Methods Inf. Med.45 (1) (2006) 125–136.
[21] J. Aarts, V. Peel, G. Wright, Organizational issues in healthinformatics: a model approach, Int. J. Med. Inf. 52 (1–3) (1998)235–242.
[22] M. Berg, Patient care information systems and health carework: a sociotechnical approach, Int. J. Med. Inf. 55 (2) (1999)87–101.
[23] F.C.G. Southon, C. Sauer, C.G. Grant, Information Technologyin Complex Health Services: organizational impediments tosuccessful technology transfer and diffusion, J. Am. Med.Inform. Assoc. 4 (2) (1997) 112–124.
[24] R. Heeks, D. Mundy, A. Salazar, Why Health Care InformationSystems Succeed or Fail, Institute for Development Policy
and Management, University of Manchester, 1999.[25] J. Luftman, Assessing business-IT alignment maturity,Commun. AIS 4 (14) (2000) 1–50.
[26] W.H. DeLone, E.R. McLean, Information systems successrevisited, in: Proceedings of the 35th Hawaii International
i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398
Conference on System Sciences (HICCS), Big Island, HI, USA,2002.
[27] F.C.G. Southon, C. Sauer, K. Dampney, Lessons from a failedinformation systems initiative: issues for complexorganizations, Int. J. Med. Inf. 55 (1) (1999) 33–46.
[28] N.K. Roderer, Outcome measures in clinical informationsystems evaluation, in: Medinfo. 2004, San Francisco, CA,USA, 2004, pp. 1096–1100.
[29] A.P. Stoop, M. Berg, Integrating quantitative and qualitativemethods in patient care information systems evaluation:guidance for the organizational decision maker, MethodsInf. Med. 42 (4) (2004) 458–462.
[30] J.G. Anderson, C.E. Aydin, Overview: theoretical perspectivesand methodologies for the evaluation of health careinformation systems, in: J.G. Anderson, C.E. Aydin, S.J. Jay(Eds.), Evaluating Health Care Information Systems; Methodsand Applications, Sage, Thousand Oaks, CA, 1994, pp. 5–29.
[31] B. Doebbeling, Information Technology and Primary Care atVA: Interdisciplinary Partnership Opportunities forProviders, Managers, and Researchers, Health ServiceResearch & Development Series, 2003.
[32] J. Smith, Health Management Information Systems: AHandbook for Decision Makers, Open University Press,Buckingham, 2000.
[33] T. Lippeveld, Routine Health Information Systems: The Glueof a Unified Health System, Workshop on Issues andInnovation in Routine Health Information in DevelopingCountries, Bolger Centre for Leadership DevelopmentPotomac, Maryland, 2001.
[34] C.P. Friedman, J.C. Wyatt, Evaluation Methods in MedicalInformatics, Springer-Verlag, New York, 1997.
[35] J.J. Jiang, W.A. Muhanna, G. Klenin, User resistance andstrategies for promoting acceptance across system types,Inf. Manage. 37 (1) (2000) 25–36.
[36] B. Kaplan, Organizational evaluation of medical informationresources, in: C.P. Friedman, J.C. Wyatt (Eds.), EvaluationMethods in Medical Informatics, Springer-Verlag, New York,1997, pp. 255–280.
[37] A. van’t Riet, M. Berg, F. Hidemma, K. Sol, Meeting patients’needs with patient information systems: potential benefitsof qualitative research methods, Int. J. Med. Inf. 64 (1) (2001)1–14.
[38] N.T. Shaw, ‘CHEATS’: a generic information communicationtechnology (ICT) evaluation framework, Comput. Biol. Med.32 (3) (2002) 209–220.
[39] The National Screening Committee (NSC), Transfer andManagement of Patient Information in Diabetic RetinopathyScreening Programmes Version 2.0, July 2005. Retrieved 2August, 2005 from: http://www.nscretinopathy.org.uk (2004).
[40] B. Kaplan, J. Maxwell, Qualitative research methods forevaluating computer information systems, in: C.E. Aydin,J.G. Anderson, S.J. Jay, S.J. (Ed.), Evaluating Health CareInformation Systems; Methods and Applications, Sage,Thousand Oaks, 1994, pp. 45–67.
[41] M.Q. Patton, Qualitative Research & Evaluation Methods,
Sage Publications, Thousand Oaks, 2002.[42] R.K. Yin, Case Study Research: Design and Methods, SagePublications, Thousand Oaks, 2003.
[43] J. Rowley, Using case studies in research, Manage. Res. News25 (1) (2002) 16–27.