it sophistication in health care
DESCRIPTION
Theera-Ampornpunt N. Article review: IT sophistication in health care - an instrument validation study among Canadian hospitals. Presented at: Health Informatics Journal Club; 2008 Oct 9; Division of Health Informatics, University of Minnesota, Twin Cities, MN. Invited speaker. Based on Paré G, Sicotte C. Information technology sophistication in health care: an instrument validation study among Canadian hospitals. Int J Med Inform. 2001 Oct;63(3):205-23. Available from: http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6T7S-43P24W1-6&_user=616288&_coverDate=10%2F31%2F2001&_rdoc=6&_fmt=high&_orig=browse&_srch=doc-info(%23toc%235066%232001%23999369996%23259465%23FLA%23display%23Volume)&_cdi=5066&_sort=d&_docanchor=&_ct=9&_acct=C000032378&_version=1&_urlVersion=0&_userid=616288&md5=ee026786822e5e65c12b5fcbd430386eTRANSCRIPT
An Instrument Validation Studyamong Canadian Hospitals
Nawanan Theera-Ampornpunt, M.D.M.S. & Ph.D. Student in Health Informatics
UMN Health Informatics Journal Club (Oct. 9, 2008)
Based on Paré G, Sicotte C. Information technology sophistication in health care: an instrument validation study among Canadian hospitals. Int J Med Inform. 2001 Oct;63(3):205‐223.
3 Themes of Focus in Article
Concept: IT Sophistication/Adoption
What, Why, How
Survey Instrument Evaluation
IT Sophistication Profile of Sampled Hospitals
Background
Health care has become increasingly complex and are in various stages of reform
IT has been shown in MIS to have positive impacts on organizational performance
Managers and clinicians are realizing the value of IT in high quality care delivery while holding down costs
Background
To identify effects of IT on health care delivery, IT needs to be characterized for operationalization purposes
No existing characterization and validated instrument of IT in terms of its level of sophistication in health care organizations
Such instrument would allow profiling and enable comparison
Study Objectives
Primary:
To develop and validate a measurement instrument of IT sophistication in hospitals
Secondary
To present and compare IT sophistication profile of Canadian hospitals
Existing Models
Nolan’s Stages of EDP Growth (1973, 1979)
Evolution of systems growth in stages
Introduces the concept of “IS Maturity”
The ultimate stage of computing growth in organizations where information resources are fully developed and computer‐based systems are fully integrated
Focuses on organizational information systems
Empirical validity has been contested
EDP - Electronic Data Processing IS - Information Systems
Cited in: Paré & Sicotte (2001)
Existing Models
Cheney & Dickson (1982)
Investigated the relationship between “Technological Sophistication” (hardware, software, nature of application systems), “Organizational Sophistication”(information resources management activities), and system performance
Cited in: Paré & Sicotte (2001)
Existing Models
Saunders & Kellers (1984)
IS maturity referred to as the “sophistication of the
mix of applications provided by the IS function”
Raymond & Paré (1992)
IT sophistication defined as “a multi‐dimensional
construct, which includes aspects related to
technological support, information content,
functional support, and IT management practices”Cited in: Paré & Sicotte (2001)
Conceptual Framework
IT Sophistication in Hospitals
“The diversity of technological devices and software applications used to support patient management and patient care, clinical support, and administrative activities” as well as “the extent to which computer‐based applications are integrated (electronic and automatic transfer of information)”
Conceptual Framework
3 Dimensions of IT Sophistication
Technological Sophistication
○ Diversity of hardware devices (e.g. medical imaging, bar coding devices, data warehousing, wireless networks, PACS equipment)
Functional Sophistication
○ Proportion and diversity of processes or activities (e.g. vital sign recording, medication administration, staff scheduling)
Integration Sophistication
○ Degree to which computer‐based applications are integrated both internally and externally
Conceptual Framework
3 Health Care Activity Domains
Patient Management and Patient Care
○ Patient management, order entry/results reporting, clinical notes, care planning, vital signs recording, ER, OR, etc.
Clinical Support
○ Laboratory, pharmacy, radiology
Administrative
○ Billing & finance, HR, materials management
Conceptual Framework
Source: Paré & Sicotte (2001)
Conceptual Framework
Technological Sophistication
Office automation systems
Human‐computer interaction devices
Storage and compression devices
Data distillation systems
Connectivity devices
Each of these can be used in different activities
Conceptual Framework
Functional Sophistication
Technology alone doesn’t measure an
organization’s IT sophistication well
One technology can be used for different
activities, each of which has varying impact
on care
Can be viewed as the level of
computerization in the work processes
Conceptual Framework
Integration Sophistication
Electronic and automatic transfer of
information
Internal Integration
○With other systems in the hospital
External Integration
○With external entities’ systems
Instrument Evaluation
Validity
An instrument does what it is intended to
do (Nunnally, 1978)
The degree to which a test measures what it
was designed to measure (Wikipedia, 2008)
The degree to which the measurement
measures what the investigator wants to
measure (Friedman & Wyatt, 2005)
Instrument Validation
ReliabilityThe extent to which measurements are repeatable, by different persons, on different occasions , with alternative instruments for measuring the same thing (Nunnally, 1978)
The consistency of a set of measurements or measuring instrument (Wikipedia, 2008)
The degree to which measurement is consistent or reproducible (Friedman & Wyatt, 2005)
Validity vs. Reliability
Source: http://ibis.health.state.nm.us/resources/ReliabilityValidity.html
Types of ReliabilityTest‐retest reliability
Does the measure deliver the same answer when
applied in different time periods?
Internal consistency reliability
Multi‐item instrument
○ Does the measure yield consistent results across
different indicators (items)?
Multiple raters: Inter‐rater reliability
○ Does the measure yield consistent results when different
observers rate the same thing?Source: Neuman (2005)
Types of Validity
Face Validity
On the face of it, do people believe that the
definition and method of measurement fit?
Source: Neuman (2005)
Types of Validity
Content Validity
Is the full content of a definition
represented in a measure?
Source: Neuman (2005)
Types of ValidityConstruct Validity
Do the various indicators operate in a consistent
manner?
Convergent Validity, Discriminant Validity
Source: Neuman(2005)
Types of Validity
Criterion Validity
Does a measure correlate with some
standard or criterion?
Concurrent Validity, Predictive Validity
Source: Neuman (2005)
Methods: Content Validity
20 in‐depth interviews with health care and IT
specialists at the Montreal Jewish Hospital (MJH)
Respondents asked to identify technologies and
computer‐based applications in their organization and
elsewhere, processes supported by the systems, &
degree of integration
Inputs used to modify instrument
Pretest with 3 HIS directors with in‐depth interviews
to refine instrument (mostly cosmetic)
Methods: Reliability, Construct & Concurrent Validity
Survey of HIS directors in 2 of the largest Canadian provinces
Sampling frame: a government list of names, addresses, and phone numbers of each medical center in Quebec (n = 80) and Ontario (n = 106)
Subjects contacted by phone to request participation (10 refused)
Mail survey with 6‐week follow‐up letter
Total period: 4 months
ResultsResponse rate 62.4% (Quebec 73.8%, Ontario 53.8%)
Source: Paré & Sicotte (2001)
Results
Source: Paré & Sicotte (2001)
ResultsReliability: Cronbach’s alpha for the technological,
functional, and integration dimensions of each
activity domain and for overall dimensions
○ Low: items don’t measure the same thing or too few items
○ High (usually > 0.70): items “internally consistent”
Source: Paré & Sicotte (2001)
ResultsConstruct validity: interdimension correlations and correlations
with an adjusted overall dimension score (after removing scales
of that dimension)
○ Interpretation: High & significant correlation means consistent results
between different measures (dimensions + domains) of IT sophistication,
thus construct validity
Source:Paré & Sicotte (2001)
ResultsCriterion validity: correlation of each dimension and 6
other variables: the hospital’s present stage of IT maturity
based on Nolan’s 5‐stage model, annual budget, annual IT
budget, number of IT staff, HIS director’s educational level
and IT management experience
Source: Paré & Sicotte (2001)
ResultsOverall functional sophistication ‐ Not significant between Quebec & Ontario
Some significant variations exist e.g.
○ Higher percentage of electronic signature for medical chart documentation in Ontario
○ Higher percentage of OR computerized processes in Ontario
Medication administration, staff scheduling, order transcription, and historical record keeping are most computerized nursing activities in both provinces
Vital signs recording and other nursing activities are among the least computerized ones
Source: Paré & Sicotte (2001)
ResultsBoth provinces exhibit low level of technological
sophistication overall (not significant different)
Telemedicine, expert systems, voice recognition systems
for notes transcription are not available in most hospitals
Smaller percentages of Quebec hospitals installed
workstations in the hallways or at bedside or used
portable devices
PACS are diffused more widely in Ontario
Lower number of Quebec hospitals have a Web site
Source: Paré & Sicotte (2001)
ResultsLow level of systems integration overall, with Quebec
significantly less integrated.
ER and OR systems are somewhat less integrated than
ADT, scheduling
In many clinical support departments (labs, radiology,
pharmacy), systems are mostly stand‐alone
In general, clear sign of lack of internal and external
integration
Source: Paré & Sicotte (2001)
Conclusion
Instrument is reliable and valid
Findings support breakdown of IT sophistication into technological, functional, and integration dimensions
IT sophistication instrument is helpful for
○ Describing a hospital’s state of IT sophistication(cross‐sectional, longitudinal)
○ Diagnosis of aspects with low sophistication within a hospital for future improvements
○ Comparisons between groups of hospitals
Limitations/Critique*
IT Adoption has 2 aspects: Depth & Breadth
○ Depth: Level of functionalities computerized
○ Breadth: Extent of adoption across organization
○ IT Sophistication instrument focuses only on depth
Instrument developed and validated in 2001
Internal consistency reliability among items may not be
appropriate for heterogeneous (“formative/causal”)
indicators. Instead, should use inter‐rater reliability or test‐
retest reliability
* Presenter’s own opinion. Not discussed in the original article
References
Friedman CP, Wyatt JC. Evaluation methods in biomedical
informatics. 2nd ed. New York (NY): Springer; 2005, c2006.
386 p.
Neuman WL. Social research methods: qualitative and
quantitative approaches. 6th ed. Boston (MA): Allyn & Bacon;
2005, c2006. 592 p.
Nunnally JC. Psychometric theory. 2nd ed. New York (NY):
McGraw‐Hill; c1978. 701 p.
Paré G, Sicotte C. Information technology sophistication in health
care: an instrument validation study among Canadian
hospitals. Int J Med Inform. 2001 Oct;63(3):205‐223.
References
Rogers EM. Diffusion of innovations. 5th ed. New York (NY): Free
Press; 2003. 551 p.
Wikipedia [Internet]. San Francisco (CA): Wikimedia Foundation;
c2001‐2008. Reliability (statistics); [modified 2008 Sep 16;
cited 2008 Oct 5]; [about 4 screens]. Available from:
http://en.wikipedia.org/wiki/Reliability_(statistics)
Wikipedia [Internet]. San Francisco (CA): Wikimedia Foundation;
c2001‐2008. Validity (statistics); [modified 2008 Aug 30; cited
2008 Oct 5]; [about 7 screens]. Available from:
http://en.wikipedia.org/wiki/Validity_(statistics)
Appendix
What is Health IT?
Can be viewed as a subset of innovations
What is Adoption?
Rogers’ Diffusion of Innovations Theory (1962‐2003)
Knowledge Persuasion Decision Implement-ation Confirmation
AdoptionRejectionSource: Rogers (2003)