simon de lusignan reader in general practice & informatics [email protected] general practice...
TRANSCRIPT
Simon de Lusignan
Reader in General Practice & Informatics
[email protected] General Practice & Primary CareDivision of Population Health Sciences & EducationSt George’s – University of London
Pushpa Kumarapeli
Lecturer in Computing and Information [email protected] of Computing, Information Systems and Mathematics,Kingston University, London
Head-to-head comparison Head-to-head comparison ....using ALFA....using ALFA
ALFA (Aggregation of Log Files for Analysis)An open source toolkit for the precise observation of the clinical consultation & the impact of using technology
www.biomedicalinformatics.info/alfa/www.clininf.eu
Pushpa’s academic work
PhD research on – process modelling of computerised clinical consultations; a video study– Developing the ALFA method as a tool-kit for consultation observation, analysis and evaluation – Design and development of the software tools; to record time structured observational data (the ODC tool), recording of doctor’s computer use
(UAR), verbal interactions (VAR), and designing of a tool to aggregate observations from multiple sources (LFA)– Developing a UML (Unified Modelling Language) specification and XML schema to produce outputs useful for software engineers/system
designers
Worked with Simon on various research projects that were based on ‘routinely collected general practice data’– Data extraction using MIQUEST, query authoring, developed a system for auto generation of meta-data, experienced in cleaning, processing and
analysis of data from GP systems
Lecturer in Computing and Information Systems– Database management systems, software quality, system analysis and design
(a former dental student, qualified as an Information System Designer, worked at Sun Microsystems as a Webtechnologist)
My practice & academic work
GP in Guildford (30 miles SW of London)– 11,800 patient practice– 6.5 Whole time equivalent GPs / 8 partners– Computerised since 1988 – EMIS brand since 1994– Involved in “Practice Based Commissioning” and UK’s
First ICO (integrating Care Organisation)
Head of GP & Primary Care, St. Georges, London– Teaching network - >200 practices >1,000 placements
– Primary Care Informatics (PCI) How IT enables quality improvement?
(1)Impact of IT on the consultation - ALFA (2) Using routinely collected data for QI
- Data linkage & pseudonymisation- Chronic Kidney Disease (CKD)
(3) Scholarship: - Chair PCI Working group of European Federation for Medical Informatics (EFMI) & UK rep to EFMI- Editor of Informatics in Primary Care
Just accepted Chair at University of Surrey– http://gp2.sgul.ac.uk
www.woodbridgehillsurgery.co.uk
1. Obsession with interoperability (or linkage) over usability
The great National Pilot of IT...
2. Time to put some science into developing usable systems...
Changes driven by better patient care
3. Observing and analysing the influence of computer on the clinical consultation
to support better information system solutions
Context
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Interoperability (and linkage) ahead of usability...
England:– The National Programme for IT has achieved some successes
– Lot of emphasis on linkage / interoperability– Systematising development (Waterfall) rather than pragmatic with users (agile)
Scotland – Focus on record linkage (unique ID) – functional links between records
Loss of focus on:– System usability– Improving outcomes for patients – Usability within the consultation
1. Obsession with interoperability over usability
The great National Pilot of IT...
2. Time to put some science into developing usable systems...
Changes driven by better patient care
Context
1. Observing the consultation – the ALFA tool-kit
2. Understanding the computerised GP consultation – Overview of the consultation – content and context– time spent on performing core tasks– their associated process
3. Changing our approach to developing systems
Overview
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Outline of the presentation
Part 1: Development of multi-channel video - ALFA toolkit– Developed video methods of observing the consultation– Additional tools developed:
– Keyboard & mouse use (User Action Recording)
– Time stamp to record start & end of speech (Voice Activation Recording)
– Coding tool (Observational data capture)
– Combining & managing may files (Log File Aggregation) Part 2: The clinical consultation is a messy business
– Initial results from -167 consultations -15 GPs – 4 EPR systems Part 3: The computer as the third actor
– The computer can initiate the consultation– How the stage is set is critical... ...if the computer is to play its part
Discussion– Time to inject some science into developing more usable systems– Serious about using the computer –change your consulting room layout...
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Free on-line access to ALFAwww.biomedicalinformatics.info/alfa/
Supporting information
– Comparison with other tools– Overview of technical process– Demonstrations
– Manuals– Presentations– Publications
ALFA components free to download– ODC (Observational Data Capture)– VAR (Voice Activity Recorder)– LFA (Log File Aggregation tool)– UAR (User Activity Recorder)
Open source GPL licence– Freely available
1. Developing video methods for observing the consultation2. Considerable variation in GP clinical systems3. ALFA is needed because we lack an effective method to:
– Compare or develop EPR – Capture context of data recording
Part 1: Introduction & Rationale for ALFA
www.biomedicalinformatics.info/alfa/ www.clininf.eu
10-year development pathway of ALFA
1999 Video evaluation of new software – single video camera...
- Not enough data
2002 Multi-channel professional video
- Great but too expensive & cumbersome
Pattern recognition software
- Fun – but not yet able to make sense of results
2005 Cheap digital video
- Research resumes
2006 Qualitative study to explore number of channels & layout
Use of multi channel video to look at simulated consultations
- Lab environment (clinical skills rooms) felt wrong
- Computer use & room layout variable in own premises
- Developed utilities to allow additional data capture
2008 Created ALFA as an open source toolkit...
- Set up in 20 to 30 mins
- Data manageable aggregated onto just two files
www.biomedicalinformatics.info/alfa/ www.clininf.eu
2006 Low-cost digital video! No of Cameras & layout
Low cost video equipment– 4 channel video for 1000 Euros– Synchronisation “easy” No studio needed– High quality sound recording– Mix in free or low cost applications– Smaller
Qualitative study – How many cameras? What layout?– 3, 4 and 5 channels experimented with– Elite survey of medical educationalist used to assessing video– Need to see the patient’s body language
Leong A, Koczan P, de Lusignan S, Sheeler I. A framework for comparing video methods used to assess the clinical consultation: a qualitative study. Med Inform Internet Med. 2006 Dec;31(4):255-65.
Sheeler I, Koczan P, Wallage W, de Lusignan S. Low-cost three-channel video for assessment of the clinical consultation. Inform Prim Care. 2007;15(1):25-31.
www.biomedicalinformatics.info/alfa/ www.clininf.eu
2007/8 Simulated consultation – Lab & own consulting room
Compared three brands of EPR system, in simulated simple consultation - Scenario: A medication review and BP check
- Used a clinical skills lab- Trained students as raters - 4 to 5 raters assessed each consultation
- Computer used for >25% of the consultation, very high intra-class correlation coefficient between raters (>0.928, p<0.001)
- Limitations: “Contrived” environment; short duration items hard to code
Compared three brands of EPR system, in simulated consultation to assess CVS risk - Scenario: Two patients (actor) patients who needed their CVS assessing
- Some GPs systematic (form led) others personalised concentrating on one risk factor- Systematic: Mean 3.5 items coded taking 33.9% of consultation- Personalised: Mean 1.9 items coded taking 25.8% of consultation
- One GP used risk calculator on the Internet
Moulene MV, de Lusignan S, Freeman G, van Vlymen J, Sheeler I, Singleton A, Kumarapeli P. Assessing the impact of recording quality target data on the GP consultation using multi-channel video. Stud Health Technol Inform. 2007;129(Pt 2):1132-6.
www.biomedicalinformatics.info/alfa/ www.clininf.eu
2006-8 Three tools to capture detailed information & an output format for software engineers :
ODC – Observational Data Capture
– Allows coding of the multichannel video– Records room layout
UAR – User Action Recording– Captures the precise time each key is pressed & mouse
co-ordinates – Measures navigation/data entry ratios– Manual capture of precise times was challenging
VAR – Voice Activation Recording– Time stamps the start & end of speech. – Needed if linking to transcript text
Output a software engineer can interpret– UML sequence diagrams & XML data files
www.biomedicalinformatics.info/alfa/
Observational Data Capture (ODC)
Collection and analysis of observational data about doctor-patient and doctor-computer interactions
Watching the multi-channel video and identifying the occurrence of different interactions and their duration. e.g. Dr talking to Pt, Dr looking at the screen, Making eye contact, Prescribing
Method 1;•Watch the full video.•Set of key board keys assigned as a label for a specific interaction.•Appropriate key pressed to mark the start and end time of interactions
Method 2;•Go directly to the known part of the video.•Identify the start of the interaction, mark it,•Identify the end of interaction, mark it•Select the type of interaction from the drop down list
www.biomedicalinformatics.info/alfa/
16
User Action Recording (UAR)
Recording clinician’s use of computer keyboard and mouse during the consultation
Captures the key strokes and the mouse movements
Value of the pressed key and the coordinates of the mouse pointer are written into two separate log files with time stamps
Time stamp
Key strokes
Location of mouse pointer (X,Y)
www.biomedicalinformatics.info/alfa/
Voice Activity Recording (VAR) for linking to consultation transcripts
Time log of verbal interactions. • Log file created based on the doctor’s
and patient’s voice activity• Combining this with the transcription
gives a time stamped conversation log.
VAR interface and log file output
Consultation transcript
Time stamped consultation transcript
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Kumarapeli P, de Lusignan S, Koczan P, Jones B, Sheeler I. The feasibility of using UML to compare the impact of different brands of computer system on the clinical consultation. Inform Prim Care. 2007;15(4):245-53.
Kumarapeli P, De Lusignan S, Ellis T, Jones B. Using Unified Modelling Language (UML) as a process-modelling technique for clinical-research process improvement. Med Inform Internet Med. 2007 Mar;32(1):51-64.
Output a software engineer could interpret!
UML: Unified Modeling Language – Sequence diagram
To develop these we also needed precisely time-stamped information about keyboard and mouse use....
Aggregating and analysing the data
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Multiple files for analysis
One or more of the following files for analysis
UAR x2
– User Action Recording – Key board & Mouse
VAR– Voice Activation
Recording ODC
– Observational Data Capture
www.biomedicalinformatics.info/alfa/
LFA Tool – links all results into two files:
File 1:
Multi-channel video recording (MCV)
User action recording (UAR)Up to 13 time stamped outputs
File 2:
Observational Data Capture (ODC)Voice Action Recorder (VAR)
Outputs:
Graphs and tables linked to coded events – linked to video files
Data for statistical packages
Data for software engineers
• Set up in <20 minutes
Setting up the ALFA toolkit
Hardware
Screen recording device(VGA2USB frame grabber)
Hard drive based camcorders(JVC Everio HDD 30GB)
Laptop for screen recording
Low profile tripods
ALFA kit packs into a rucksack and tripod bag
...readily deployable
www.biomedicalinformatics.info/alfa/
Setting-up three cameras
High capacity batteries run for 3 hours
Optional additional web-cam
Use camera microphone
Adjust the three camera angels;1 - Dr’s upper body 2 - Patient’s upper body3 - Wide angle view of Dr, Patient and computer screen
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Audio-Visual recording of consultation
Three video cameras record the consultation. Screen capture software records the clinical computer system’s monitor
output
1
24
3
Multi-channel video recording (MCV)
www.biomedicalinformatics.info/alfa/
MCV Cameras Angles
Wide angle view
The camera capturing the wide angle shot should have a clear view of the Dr’s computer screenThis is important for synchronising the cameras footages with the screen recording
Patient’s camera
It is important to capture patient's hands
Doctor’s camera
Dr’s view should cover Dr’s upper body and the computer keyboard
www.biomedicalinformatics.info/alfa/
Frame grabber copies monitor activity to a laptop
Locate the VGA graphics output of the Dr’s computer and disconnect it.
Connect the VGA splitter cable into the computers VGA output.
A laptop – usually under the Drs desk captures what happens on their computer
www.biomedicalinformatics.info/alfa/
Producing the multi-channel video
• The three camera video channels are synchronised using a standard package
• Adobe Premiere Elements
Variation in brands of computer system
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Variability between brands of EPR
UK brands of EPR appear to have developed organically
Four brands account for around 75% of the market– EMIS LV (Most used – old CHUI interface)– EMIS PCS (GUI version – auto-coding – 3 windows )– INPS Vision (4 windows, tabs and icons)– iSoft Synergy (3 windows, multi-tabs & icons )
If only we could combine the best features of each?
www.biomedicalinformatics.info/alfa/ www.clininf.eu
31
Variation in GP computer systems
EMIS LV EMIS PCS
INPS Vision iSoft Synergy
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Variation in how BP is recorded in GP computer systems
EMIS LV EMIS PCS
INPS Vision iSoft Synergy
www.biomedicalinformatics.info/alfa/
Variation in how cardiovascular risk is recorded:
EMIS LV EMIS PCS
INPS Vision
iSoft Synergy
www.biomedicalinformatics.info/alfa/
Variation in coding in the consultation
Coding took place in >80%
No data were coded in:– EMIS LV (30%) – EMIS PCS (18%)– iSoft synergy (4.5%)– INPS Vision (7.4%)
In EMIS LV and PCS coding was less frequent– 1 item was coded in 69%– 2 items in 23%– Only 6 consultation 4 or more items
More items were coded in iSoft Synergy and INPS Vision– Mean 2.9 (iSoft) & 2.6 (vision) items per
consult– Mean 1.5 (LV) & 1.6 (PCS)
≤4 or ≥5 codes per consult
% consultations with no coded data
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Contrasting methods used for clinical coding
in PCS - copying and pasting from previously entered codes from sub window for record summary
in Vision – selecting from main menu add item, needs more hand-eye coordination/interactions. Despite having number of icons for common coded data entry, they were rarely used
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Contrasting time taken for clinical coding
Non-parametric test suggest that EMIS PCS is significantly faster than EMIS LV, or Vison (p<0.001); and no different than Synergy (p=0.083)
However, data linkage means more coding happens in Vision INPS and iSoft practices UML diagrams provide more insight...
www.biomedicalinformatics.info/alfa/ www.clininf.eu
1. Dr: Transition: Default Consultation ModeData entry
2. Dr: Select Heading: Problem: Double click3. EPR: Present Code list4. Dr: Navigation (Mouse): SubWin 1 SubWin2:
Problem List5. Dr: Selection: Problem List (Past History item)
Double click6. EPR: Add code { Problem List (Past History item)}7. Dr: Transition: Data Entry Coded Data Entry Free
Text
• Coded data entry embedded to free text data entry
• Less Transition time between Free text entry Coded entry Free text entry
• ‘Semantic Auto Completion’ feature: offers a picking list of coded data items as you type on
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Vision1. Dr: Transition: Default Consultation ModeData entry2. Dr: Navigation (Mouse):SubWin 3: Coded entry 3. Dr:Type Search string4. Dr:Submit Search String5. EPR: Present item:16. Dr:Expand data entry7. EPR: Present data entry form8. EPR: Present picking list9. Dr: Select item10. Dr: Submit Item selection11. EPR: Accept Selection12. Dr: Transition: Coded Data entryFree text entry
• A small sub-window for Coded data entry displayed in default screen
• May need to expand the data entry window, if the desired coded data item is not presented as the first result in the picking list
• Initial transition time less, but may result in spending more time in code selection if the full picking list is needed
• Free text entry always linked to a coded entry
SynergyVision
LV
Variation in BP recording time
PCS – BP entered using other data entry forms as well Vision – standard BP entry form used always PCS – single method of activation used for the standard BP form Vision – different activation methods used, mostly determined by the current location
within the patient record
www.biomedicalinformatics.info/alfa/
Variation in BP recording time Recording BP data in the EPR took between 5.9 and 27.6
seconds– xU 12.1 sec (x̃ 10.20) – SD 5 sec
PCS xU =13.6 (IQR=6.4), Vision xU =11.2 (IQR=1.5) PCS – GPs spend time reviewing, commented on past values Vision – default values left unchanged to minimise any delay
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Consultation mode
Txt: BP
SW, NP, Code search
GP GPS:LV Patient
Exm: BP, Pulse
Select link: Exm
39sComments: SymptomsQ: Smoking status
A: Smoking status
Comment: BP values
SW, SP, Heading: Exm.
XSelect link: Code search
Txt: BPSW, NW, BP template
Ssy, Dias txt area
txt: Sys & Dias Comment: Symptoms
Comment: BP values, normal range
GP GPS:PCS Patient
Examination Request: BPExm Response: Agrees
Enter Txt: BPDisplay template: BPExamination: BP
Save BP results
Past BP, Normal range
Select Heading: Exm
Select, txt: Dias BPSelect, txt: Sys BP
Insert Exm header to EMR: BP
GP GPS:Vision Patient
Request: Examination findingsHgt, Wgt, BPSelect link BP
Past BP readings x 6Exm Request: BP
Exm Response: AgreesSelect: Add BPBP template
Date, Time, HCP, Read code, Position, Cuff detail, location
Examination: BP
Select, txt: Dias BPSelect, txt: Sys BP
Select, Code: BP normalComment: BP range
Select link: Past BP Save Exm results
Past BP readings x 6 Explains: BP change
Comment: BP values
GP GPS:Synergy Patient
Exm: BP, PulseQOF reminders
BP, Smoking, Smear, Summary
Comment: BP review
63s
Comment: Bp medicationSelect link: Add BP
BP template Free txt area
txt: Sys & Dias BPExm: Pulsetxt: Pulse, Ftxt
12s
UML diagrams allow software engineers to understand the process…
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Acute prescribing time
55% (91/167) consultation have an acute precipitin
Little variation between brands – range 58% to 45%
The EMIS LV character user interface appears on video to offer advantages in speed of Rx
The mouse driven systems need more hand-eye coordination and were slowest where a completely new Px was started
PCS users (xU =26.3) spent slightly longer durations compared to Vision (xU =23.9)- Vision provided more default values for data entry- Vision also presented more prompts, mostly ignored
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Encounter reviewing Encounter reviewing observed in 93% of consultations PCS has a sub window in the default consultation view, Vision users need
to navigate into a separate section Similar durations spent , (PCS; x̃ =30.6s, Vision x̃ =33.9s), but more
episodes noticed in Vision (x̃ =9) than in PCS (x̃ =5)
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Free text entry & prompts Majority of free text entries in
Vision were associated with coded data entry – ‘strict problem orientation’
More patient history linked prompts in PCS.
Vision prompts & warnings mostly linked with prescriptions
Discussion
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Summary
ALFA captures the context Key differences in time and process:
– Review of notes– Problem recording– Free text entry– BP recording– Acute prescribing
If only we could have a new system with all the best features....
ALFA should enable the development of more effective EPR systems to use in the clinical consultation.....
www.biomedicalinformatics.info/alfa/ www.clininf.eu
Reminder – change your consulting room layout!
The room layout inhibit the role of the computer as an actor
– Implications for shared decision making Consider inclusive layout with two seats
– 2/3rds time patient sits in semi-inclusive Dr chair– Younger patients more often sit in the inclusive chair
www.biomedicalinformatics.info/alfa/
The evolution of the clinical consultation?
Doctor and Patient
Doctor , Patient ( and computer )Doctor, ( Patient ) and computer !
Handout:
Thanks for listening:Simon de Lusignan
Pushpa [email protected]
www.biomedicalinformatics.info/alfa/
www.clininfo.eu