Certification and Conformance Approaches for
Canada’s EHR
Elaine Sawatsky, Senior Management Consultant, Privacy
& Security, Canada
Conformance: Canada’s EHR
• Infoway Conformance Services in Canada
• Supporting the Implementation of pan-Canadian Standards
• Provincial/Territorial work
• On the ground experiences
Conformance: Canada’s EHR
• Sharing information requires a quality assessment of products to ensure interoperability, to reduce risk to patients, purchasers and the ‘system’.
• Standards support sharing data and bringing it together
• Improves comparability and harmonisation between products.
• Certification: “a process of validation of products in a given context against specific validation criteria.”
• Certification criteria are – even regarding the same functionality – up to a certain degree similar but also different between jurisdictions in:
» The way they are formulated & validated» The importance given to each of them
• Certification should be unambiguous, fair and trustworthy
Conformance Services Vision
• To support the successful implementation of pan-Canadian Standards within projects through the provision of conformance testing artefacts and tooling.– To improve the consistent implementation of
pan-Canadian Standards– To reduce the effort required to implement pan-
Canadian Standards– To reduce the cost of implementing pan-
Canadian Standards• Strategic Intent of Conformance Services:
– Ensure that there is a national service to support the vendor conformance and certification of iEHR, public health and EMR solutions.
SK EHR Blueprint and Concepts
Provincial Infostructure
Point-of-Service (POS) Infostructure
Registries Data & Services
Data WarehouseEHR Data & ServicesAncillary Data & Services
Client Registry (SCI)
Provider Registry (PRS)
Location Registry
LRS
Outbreak Management
Shared Health Record
Drug Information
PIP
Diagnostic Reports &
Imaging PACS
Lab Repository
SLRR
PHS Reporting
Health Information
Surgical Care Registry
POS Infostructure will continue to evolve as future needs are identified
BusinessRules
MessageStructures
EHRIndex
Modified: October 2, 2006
HIAL
User Registry
SecurityManagement Data
PrivacyData
ConsentIntegrationSecurity
TerminologyServices
IndexNormalizationOrchestration
CANADA
V3.1
Organizational Infrastructure Access Systems
SK Cancer Agency
Clinical Management
Service (Varian)
Prov Lab
Lab (LIMS)
Registry(Varian)
Regional Systems
Drug (WinPharm, RxTFC, PCSI)
Lab (TriWin, SCC)
Clinical Information System (SCM)
Regional ADT(WinCIS, Enovation)
Health Records (WinRecs, 3M)
Transcription (WinCIS,
HealthSuite,...)
Radiology (RIS)
RHA Integration Hub (Cloverleaf)
Clinical Viewer (SCM) Provincial
Medication Profile Provincial PACS Provincial Lab
Results
Vendor PoS
Community Pharmacies Medication Profile
Clinical EMR’s Primary Health Care
(PHC) LAB Medication Profile Shared Health Record
(SHR)
eHealth Portal
EHR Desktop Lab Results (SLRR) Medication Profile (PIP) DI Image & Reports Shared Health Record (SHR) Public Health Surveillance
(PHS) Client Id. (SCI) Health Insurance (PHRS) Provider Id. (PRS) Wait List (Surgical Care)
Individual Viewers
Client ID Viewer (SCI)
Health Insurance Viewer (PHRS)
Medication Profile (PIP)
Diagnostic Imaging Viewer (PACS)
...
Long Term Care (MDS)
Home Care(Procura)
Surgical (SIS)
Public Health Immunization Public Health
Surveillance
Primary Health CareChronic Disease Mgmt.
(CDM)
The need: Bridging The Gap
The challenge is to bridge the gap that exists between the health informatician (modeler) and the clinical integration specialist (solution developer).
EHR in
OperationEHR in
Operation
Health InformaticsModeler
Integration Specialist
Modeling Culture• Delivering
Specifications
Project and Product Culture• Delivers Production
Solutions
Standards DevelopmentStandards Implementation,
Conformance & Maintenance
Need Need Options Options Develop Develop SFU SFU
Authoring Vocabulary
Profiles Assessment Certification
Release Change Helpdesk
Tooling
6
Conformance: Canada’s EHR
Conformance: Canada’s EHR
Approach• There is recognition that there is a need for
conformance testing to ensure alignment• The approach is founded on international standards
• Prototype & Learn – conformance testing is not well understood yet
• Build Methods & Processes with stakeholder input • Build tooling & leverage : • Standards – HL7 v3, LOINC, SNOMED, XDSI
• Tooling• Component model – integration is key!• Sharing of testing artefacts – huge value to
stakeholders
Conformance: Canada’s EHR
Key Messages
• There is recognition that we all need to do end to end and integrated testing
• We are committed to leveraging work of others
• We are making good progress in developing processes and tooling to assist our jurisdictions in implementing “Conformant” solutions
Conformance Continuum
Semantic testing
Most value / Difficult
Functional testing
Easier / Little value
Messaging
Messaging, Vocabulary
Messaging, Vocabulary, Sequencing
Workflow, Conformance Statements, Business Processes
Ex: an ‘add client’ message cannot be sent until a ‘client query’ message has been sent
Conformance Continuum(Scope of ‘Conformance’)
Conformance includes:– Message content and structure– Vocabulary and terminology– Interoperability profiles and
message sequencing– Sending and receiving
applications and intermediaries
Conformance Testing includes:– Functionality of application– Basic privacy and security
services related to the functionality
– Workflow (to an optimal level determined by conformance continuum)
10
Our Approach
• First, develop Profiles for all pan-Canadian standards• Develop a process & mechanisms linking the pan-
Canadian Profiles through to conformance test • Develop, deploy, continuously improve “quick-hit”
tools• Provide business and technical support for Investment
Projects to set up and execute conformance testing• Document & refine requirements for long-term solution• Harvest artefacts for sharing with other jurisdictions• Rationalize jurisdiction artefacts to create pan-
Canadian versions• Establish a pan-Canadian Working Group of early
participant jurisdictions and their vendor partners to provide guidance and assistance in the development of Conformance Testing artefacts
Options Research
and Analysis
Standards Development
Needs Identification
& Business Definition
Maintenance
TrainingEducation
Conformance
Implementation
Test Pilot
Pan-Canadian Standards
Pan-Canadian Standards
EvaluationEvaluation
AdoptionStrategiesAdoptionStrategies
Tools andTemplatesTools andTemplates
StandardsKnowledge
Management
StandardsKnowledge
Management
LiaisonLiaison
Promotion & CommunicationPromotion &
Communication
LicensesLicenses
Change Management
Change Management
Standards Selection Strategy
Stable for Use
Final Approval
Scope of Part 4
Our Approach
Conformance: Canada’s EHR
Benefits:• Financial savings through reduced duplication of efforts in
test authoring• Provinces leverage other provinces’ Business Rules
• Financial savings though tools based test message development
• Provinces use Infoway tools and other provinces’ messages
• Breadth of testing improved through increased quantity and quality of test cases
• Provinces are committed to sharing test artefacts.
Conformance: Canada’s EHR
Benefits:• Depth of testing improved through consistent test rules
and tooling• Provinces’ use of tooling has improved UAT testing Too early to
accrue benefits
• Quality and consistency of conformance testing improved between Investment Projects
• Too early to accrue benefits
• Improved consistency in Investment Project implementations of pan-Canadian standards
• Too early to accrue benefits
Benefits
Providing Implementation support for the development of Test Authoring and Test Execution will enable jurisdictions to leverage and share Conformance Profiles and testing artefacts, leading to:
– Improved quality of testing– Financial savings through reduced duplication of efforts– Breadth of testing improved through increased quantity
and quality of test cases – Improved quality and consistency of conformance testing – Financial savings though tools based test message
development– Depth of testing improved through consistent test rules
and tooling– Improved consistency in implementations of pan-
Canadian standards
Conformance: Canada’s EHR
Conformance Services Overview• Main elements influencing content of test
criteria:Legal, national, cultural (business and social)• Sophistication of the industry area• Environment• Certification is the process• Conformance is the result• Must be good enough, not best or everyone
will fail• Area subject matter experts – what will the
medical professional need?
Stakeholder Engagement• Governance
• Collaboration on Test Authoring components• Collaboration on supporting tools development and implementation
• Registry of vendor systems & certification status • Registry of implementation projects• Ongoing liaison/communication with jurisdictions and vendors
• Promotion and Education around Conformance Services
Overview of Conformance Services
Test Authoring Services• Business Rule Management• Conformance Profile Management• Vocabulary Maintenance• Test Case Authoring• Test Data Management• Message Instance Authoring
Test Execution Services•Conformance self assessment & evaluation•Conformance independent verification & validation•Conformance certification
Technology• Test Execution Tools
Conformance ValidationTest Case ValidationSelf-test, monitoring
• Web presence for jurisdictions and vendorsRegistration, artifacts library, test results
• Test Authoring ToolsRepository, remote user accessshared authentication
17
Strategic Direction Analysis Project
• Project Objective:– Develop a preferred option for the future delivery of Conformance
Services. • Define and identify existing conformance services models;• Develop a shared vision for the delivery of Conformance Services for
the Canadian healthcare sector• Develop conformance services delivery options for the Canadian
healthcare sector; and detail the preferred option.
• Defined Strategic Intent of Conformance Services:– “Ensure that there is a national service to support the vendor
conformance and certification of iEHR, public health and EMR solutions.”
RFP Release February 15 ‘08
Project Start March 15 ‘08
Draft recommendations June 30 ‘08
Final Recommendations Available August 30 ‘08
18
Investment Project Support Overview
• Project Objective:– Provide immediate Standards Implementation Support for Test
Authoring & Test Execution to key Infoway Investment projects• Project Approach:
– Develop a process & mechanisms linking the pan-Canadian Profiles through to conformance test
– Develop, deploy, continuously improve “quick-hit” tools– Rollout to qualified projects (if proof of concept successful)– Harvest artefacts for sharing with other jurisdictions– Provide lessons learned and input into CS Strategic Direction Project– Provide transition and support to recommended Strategic Direction
Project Kick-off November 1 ‘07
Proof of concept (2-3 projects) March 31 ’08
Proposed Project Extension:
Limited rollout (4-6 projects) September ‘08
19
Investment Project Support Activities
• The specific services that Infoway is offering to assist projects
• Assistance in developing test strategies– Sort out overlap between conformance testing vs. UAT
• Education on development of business rules & test cases• Technical education on development of message
instances and test data• Sharing of testing artefacts• Quality review of artefacts developed by investment
projects• Full set of tools to assist getting to Conformance Testing • Assistance to jurisdiction in rationalizing and integrating
their testing tools with Infoway conformance tools• Harvesting of testing artefacts for sharing with
jurisdictions• Sharing of Lessons Learned• Facilitating inter-jurisdiction collaboration on conformance
testing
20
Quality Review Harvest artefacts
(2-3 days)
Quality Review Harvest artefacts
(2-3 days)
33
Introduction Workshop
Introduction Workshop
00
21
Engagement Process to Conformance
Introduction Workshop
Introduction Workshop
00
JurisdictionConformance Pilot
Testing, Quality ReviewHarvesting artefacts,
Lessons learned(3-5 days)
JurisdictionConformance Pilot
Testing, Quality ReviewHarvesting artefacts,
Lessons learned(3-5 days)
55JurisdictionEnd-to-End Test
Case Development(20-40 days)
JurisdictionEnd-to-End Test
Case Development(20-40 days)
44Quality Review Harvest artefacts
(2-3 days)
Quality Review Harvest artefacts
(2-3 days)
33
JurisdictionEnd-to-End Test
Case Development (10%) (15-20 days)
JurisdictionEnd-to-End Test
Case Development (10%) (15-20 days)
22Test case Examples
Harvestedartefacts
Test case Examples
Harvestedartefacts
End to EndTest AuthoringExamples
End to EndTest AuthoringExamples
Jurisdiction Test ArtefactsJurisdiction Test Artefacts
ToolsTools
End to EndTest ExecutionExamples
End to EndTest ExecutionExamples
Jurisdiction Test Authoring Artefacts
Jurisdiction Test Authoring Artefacts
Jurisdiction still needto complete, but ready to perform ConformanceTesting
Jurisdiction still needto complete, but ready to perform ConformanceTesting
Ongoing HarvestingOngoing Harvesting
End to EndConformance
Process Workshop(2-3 days)
End to EndConformance
Process Workshop(2-3 days)
11
22
Conformance Constructs – End-to-End Process
JurisdictionInput
BestPractice
External Application
ConformanceProfiles
StandardsArtefacts
TestCases
HL7 TestHarness
Test CaseAuthoring
TestData
InstanceEditor
(Messages)
Vocabulary Data
- “Quick-hit” tools
- “Existing” tools
- HL7 V3
Business &Technical Rules
ImplementationGuides
Conformance: Canada’s EHR
Major Tooling Components• Business Rules Manager- Maintains Business Rules• Conformance Profile Manager - browse pan-Canadian
Conformance Profiles• Implementation Conformance Profile Manager - browse profiles
and create/maintain ICPs• Static Model Constraint Tool - captures changes to models,
and traceability information• Instance Editor - creates Conformance Test Scenario Files,
Message Fragments and Parameter Files as XML output. • Test Case Author - Maintains Test Cases• Test Authoring Manager - The main user interface• Test Execution Manager - links the Test Authoring Manager to
control the test execution• Test Authoring Reports Manager - run reports
Conformance: Canada’s EHR
Conformance Test Execution ScenariosThere are different ways to perform
Conformance Test Execution. This process uses the pCCT Test Execution manager and / or the HL7 Test Harness to allow the user to test both POS and iEHR systems as “real” or “simulated” applications under test.
1. Scenario 001 - Real POS with Simulated iEHR Application 2. Scenario 002 - Simulated POS with Simulated iEHR
Application 3. Scenario 003 - Simulated POS with Real iEHR Application 4. Scenario 004 - Real POS with Real iEHR Application
Lessons Learned – Considerations for AB
• Testing artefact reuse is seen to be a big win• One province has 100 business rules & 300 test cases –
target is 500-700 test cases• Another target will be significantly higher (~2,500 test cases)• Others will leverage oand likely add more
• Toolset are not all or nothing• One province using blend of Infoway tools and home grown• Uses Infoway tools and processes first to perform testing
then conformance testing of PoS• One province using Infoway tools for go forward & PoS testing• Specific initiatives will likely use blend
• Infoway is investing in tools so jurisdictions don’t have to!
Conformance: Canada’s EHR
Tooling Vision• Create a suite of integrated Conformance Test tools based on
Open Source principles and accepted pan-Canadian standards that allows jurisdictions, and vendors to perform conformance testing
• Includes:– Well documented processes and procedures– A repository of structured conformance profiles, test cases,
test data and business rules that can be reused– A suite of re-usable conformance testing solution components
able to easily exchange structured data between each component and projects
– An Enhanced Web Based Presence to allow remote administration, self registration, shared resources management, self testing / monitoring and publication of results
Lessons Learned – Considerations for AB
• Many jurisdiction projects have left UAT & Conformance Testing until very late in their project lifecycle
• There is not a clear understanding of the continuum of conformance testing
• Profile Business rules Test cases Messaging Conformance test
• Traceability is key to conformance test• Profile Business rules Test cases Messaging
Conformance test• Using Profiles to help them define user roles
• What does a physician do vs. project focus• Tools can and are providing immediate benefit to
jurisdiction projects for publishing• Consolidated view of all business rules• Publication of specification for vendor
27
Conformance Profiles
Pan-CanadianConformance Profile (pCCP)
ImplementationConformance Profile (ICP)pCCP
Extension
Constraint
Example:BC ICP #84#84
BC ExtensionsBC
Constraint
LikelyInitial State
DesiredState
#47
#84
Library of Conformance Profiles
Sask
BC
Nfld
Sask
BC
Nfld
+
+
Conformance: Canada’s EHR
Tooling for Test Authoring Components (TAC)– Use prototyping methodology to conduct staged
development – Test with jurisdictional review and input – Document processes, requirements and changes– Garner feedback and experience to inform the refinement
of detailed specifications for long-term open source tooling development
– Address immediate jurisdictional needs & provide tangible benefits
• All test artefacts exportable• Work with jurisdictions to rationalize and integrate
tooling – Align test cases, business rules across jurisdictions– Promote early sharing and leveraging of work by
jurisdictions
Conformance: Canada’s EHR
Tooling for Test Execution– Leverage an existing open source test harness (previous Infoway
investment), adding ability for jurisdictions to easily configure to reflect local technology solutions such as transport and security
– Leverage Eclipse Instance Editor for message and vocabulary creation
– Use data exported from test authoring tools• Key is to ensure tools export into a format that can be
consumed by HL7 Test Harness• Invest in documenting processes, requirements, integration
– Rationalize and integrate tools
– Develop guidelines for local Test Execution prototyping environments including a reference implementation
– Develop processes to ensure data integrity between different groups of users during prototyping phase
Conformance: Canada’s EHR
It’s not easy:• In order to get to a conformance test, you need to
start with the Profile, know what the business rules are and create test data and message instances that will then test a scenario
• Once you have done this for every business and technical scenario, only then can you think about performing a conformance test
• By our simple calculations, this will be thousands and thousands of test cases!
0 – Step 0 Engagement1 – Step 1 Workshop2 – Step 2 Activities3 – Step 3 Workshop4 – Step 4 Activities5 – Step 5 Workshop
0 – Step 0 Engagement1 – Step 1 Workshop2 – Step 2 Activities3 – Step 3 Workshop4 – Step 4 Activities5 – Step 5 Workshop
PlannedPlannedScheduled/UnderwayScheduled/UnderwayCompletedCompleted
PlannedPlannedScheduled/UnderwayScheduled/UnderwayCompletedCompleted
0,10,10,10,1 0000
00004,54,54,54,50,1,2,30,1,2,30,1,2,30,1,2,3
PanoramaPanorama
0000
0,0,0,0, 1111 0000
0,10,10,10,1
0000
32
13-Mar-08
Conformance Services Activity in Canada
Conformance: Canada’s EHR
• Provinces have done functional Certification for EMRs specifically.
• Other certification means defining differences in function, scope, perspective (e.g. Social interaction between users, staff and patients),
• Legal requirements are different between provinces, policies are different: Infoway cannot set policy but provinces must
• Provinces must work in collaboration with Health Authorities and professional bodies.
• Conformance criteria will be different from one business environment to another.
Conformance: Canada’s EHR
• Work is based on existing provincial models– Establish business rules to support policy set by
Ministries of Health, College of Physicians and Surgeons and College of Pharmacists
– Publish compliance specifications that include all the transactions needed at the point-of-service (e.g. pharmacy claims, pharmacy prescriptions, Client Registry, Provider Registry, Common, Lab, Immunization, etc.)
– Provide support to vendors during their development
– Conduct compliance testing of vendor software once they pass a pre-test
– Support change agents (chain stores, vendors, etc.) during their training and deployment activities
Conformance: Canada’s EHR
POS Software Compliance Principles• POS systems
– Must enhance patient safety & Support clinician workflow– Must be secure and ensure privacy of patient medical records– Must support interoperability standards– Are integral in creating and maintaining quality data for use
throughout the health care system• POS vendors must be able to differentiate their products• Compliance documents
– Are referred to in some provinces legislation– Communicate what rules need to be followed, not how the rules
are implemented in vendor software– Provide one place to house Ministry policy, College rules and
application capabilities– Provide a level playing field for all vendors regardless of size
Conformance: Canada’s EHR
POS Conformance Specifications• Define Business rules
– Based on existing business rules from compliance documents and input from Clinical Working Group
– Identify items that need further clarification• Develop Compliance specifications
– Based on existing compliance documents and published message specifications from Infoway
– Builds on Infoway Conformance Profile definitions– Leverage, where possible, work from other jurisdictions
• Materials include:– POS Software Vendor Compliance specifications;– Vendor registration and development support processes;– Compliance testing process;– Process for approving vendors with compliant software
• Stakeholders included in joint review of material prior to publishing
Conformance: Canada’s EHR
Compliance – Lessons Learned• It takes time for the vendors to develop code to the
level that warrants compliance testing• Vendors may present software that was not stable
enough to warrant compliance testing• Vendors can require multiple attempts at compliance
testing• Thorough compliance testing takes time, especially on
the very first test of new software or a major release change
• Good rules are needed on what constitutes an initial test and a follow-up test for each version of the vendor software
• Good rules are needed on when penalty and charges are applied to the test
Conformance: Canada’s EHR
• Sample tests were provided for vendors• Focus needed to ensure that the vendors could prove they
ran the tests and the tests worked before scheduling a compliance testing
• Need mechanisms to ensure compliance team provided consistent responses to all vendors throughout the compliance support process
• Good rules are needed around mandatory training that a vendor must provide e.g. training plans, training outcomes, etc.
• Good test facilities are needed for vendors and for compliance testing
• Multiple training environments are needed for each vendor• Good tools are needed for the compliance testers that
assist them in developing the test cases and support them during the testing
Conformance: Canada’s EHR
• Clinical guidance was critical during vendor development and compliance testing to clarify options and workflow– Requires ongoing clinical involvement
• Specifications, clinical guidance and support needs to include details
• Compliance team requires good communication and negotiation skills and need to be able to stand their ground under pressure
• Compliance documentation needs to be targeted to specific audiences and also clearly identify elements common across audiences
• A tool that enables easy linking and/or packaging of modules is highly desirable. Documents that are difficult to maintain don’t get maintained
• Business context for the compliance material is just as important as the rules themselves
Conformance: Canada’s EHR
Conformance includes:– Message content and structure– Vocabulary and terminology– Interoperability profiles and message sequencing– Sending and receiving applications and
intermediaries
Conformance Testing includes:– Functionality of application– Basic privacy and security services related to the
functionality– Workflow (to an optimal level determined by
conformance continuum)
Contact
Standards Collaborative Information Desk Bureau d'information de l'Unité
collaborative de normalisation d’Inforoute 1-877-595-3417 or [email protected] or
[email protected] and Security, SME
Holding slide