2nd exl quality oversight conf szpindor in process vendor audit

Our purpose

We enable people with life-altering conditions to lead better lives1

2nd exl Quality Oversight ofClinical Vendors Conference

A Case Study – In Process Audit Strategy

Stan Szpindor, M.S. Director, R&D Quality Assurance

To be as brave as the people we help

Agenda

• Traditional Qualification Audit• Assumptions and Limitations of a Qualification Audit• Paradigm Shift and Related Framework• In-Process Audit Strategy• Audit Fusion – Vendor Audit to Process Audit – A Case

Study• Supporting Quality Beyond the Audit


2


Disclaimer

As a component of this presentation is a “Case Study”, the names, facts, accounts, and outcomes have been altered to protect the confidentiality of the firms and personnel involved.


3


Traditional Vendor Qualification Audit

• Audit Plan Drafted from SOW

• Two-day Onsite Audit

• Scope Includes:

• Quality Management System

• Inspection History

• Review of Select SOPs

• Interview of Personnel

• Training Program

• Personnel Qualifications

• Infrastructure (physical, IT)

Debrief with Sponsor Team

QA Qualification Audit

Note: data covers timeframe of 1/1/2003 through 12/31/2010

Transactional Vendor Selected

CAPAs

Comprehensive Audit Report

“Green Light” to Proceed (or Not)


Assumptions and Limitations of a Qualification Audit

• Transactional Vendor

• Little to No Previous Experience with Vendor• No Pre-conceived Intentions of Future Work• Cost and Timing Selection Drivers

• Narrow Audit Focus• Based on Service Area Needs of One Study or Program• Infrastructure Related Items Barely Touched Upon at Best• Training Curriculum and Requirements (not effectiveness)• Reading of SOPs and Discussion of Processes


Assumptions and Limitations of a Qualification Audit

• Inability to Gauge Compliance and Efficiency

• Review of SOPs Can’t Predict Real Life Execution• Discussion of Quality Management System with No Real Ability

to Know it is Effective • No Data or Deliverables to Sample or Assess• “Dog & Pony Show” Please!

At the End of the Day, Can I truly Know with Confidence the Vendor WILL Deliver as Promised?

To be as brave as the people we helpWe continue to move forward at a faster pace. It’s a timing thing!


Paradigm Shift and Related Framework


Sourcing Strategy Alignment• Seek an integrated view of the sourcing

strategy across the study components and R&D functions

• Ensure framework for objective vendor selection and periodic reviews

• Consolidate base of providers from many Transactional to few Strategic

• Identify economies of scale and functional service provider (FSP) opportunities

Vendor Management

• Select metrics for each vendor from comprehensive menu based on applicability of services being performed

• Monitor metrics / KPIs / KQIs for vendors (in conjunction with functions and teams)

• Report executive level dashboard • Conduct In-process audits which gauge

deliverables, outputs, and commitments.

Vendor Evaluation• Shift to Functional Assessments from Pre-

Qualification Audits• Functions and Teams are accountable to

determine “fit for use” based on evaluation of vendor SOPs, personnel, and capacity against project specifications

• Provide standard approach and adaptable outcome capture tools for consistency across R&D functions and teams

Vendor Governance• Establish Relationship Management

Committee with cross-functional representation including QA

• Utilize Quality Agreements to ensure alignment between outsourcing strategy and quality expectations

• Regularly provide / discuss audit and inspection trends from studies

• Hold After-Action-Reviews to foster continuous improvement


Anatomy of The Functional Assessment

• “Don’t Call Me Francis!”

• The Assessment Team• Consider availability and Subject Matter Expertise (SME) of your

Team/Functions• Consider availability of members of both your Team AND

members of other Teams that are using the same Vendor • Representatives should be aligned with requested outsourced

services and should be functional area expert or SME • Establish pool of “Core Away Team” personnel that are Subject

Matter Experts in Key Areas AND Trained / Experienced in conducting Functional Assessments


The Functional Assessment

• Onsite Activities

• Confirm Firm’s Capabilities (project and beyond)

• Organization – how structured, recent changes

• Competencies – partners/sub-contracting, delivery sphere

• Personnel – qualifications, turnover, transition provisions

• Procedures - Do the Vendor’s SOPs meet or exceed Sponsor standards for your project?

• Management – communication channels, senior management involvement, issue escalation, workload management

• Functional Level Service Delivery – compatibility, alignment, systems/tools, QC robustness

• Reporting

• Customizable report containing “must items” as well as those relevant to Vendor and service type

• Notes a list of follow-up items


10


Benjamin Franklin

“Diligence is themother of good luck”


In-Process Audit Strategy

Master Audit PlanMaster Audit Plan

Frequency Assignment: Annual Audit = Strategic Partners, High Risk Partners 2-Yr. Audit = Standard Level 3-Yr. Audit = Low Volume Partners, Low Risk Partners

Frequency Assignment: Annual Audit = Strategic Partners, High Risk Partners 2-Yr. Audit = Standard Level 3-Yr. Audit = Low Volume Partners, Low Risk Partners

GCP Risk Factors: Inspection History Audit History Global Exposure Key vs. Non-key Study Key Quality & Compliance Indicator Aggregates Quality Agreement Adherence

GCP Risk Factors: Inspection History Audit History Global Exposure Key vs. Non-key Study Key Quality & Compliance Indicator Aggregates Quality Agreement Adherence



Master Audit PlanMaster Audit Plan

Business Risk Factors: Relationship Type – Strategic vs. Transactional Service Type – Full Service, Niche Service, FSP Volume of Work and Overall Spend Global Exposure Merger & Acquisition Activity Key Performance Indicator Aggregates Contract Adherence

Business Risk Factors: Relationship Type – Strategic vs. Transactional Service Type – Full Service, Niche Service, FSP Volume of Work and Overall Spend Global Exposure Merger & Acquisition Activity Key Performance Indicator Aggregates Contract Adherence



Benchmarks, Benchmarks, Benchmarks!• Service Agreement / TORO / Statement of Work Compliance• Quality Agreement Compliance• SOP / Work Instruction Compliance • Functional Plan Compliance

Commitment Fulfillment• Audits

• Responses & CAPAs Process & Structural Improvements noted in Sponsor Audits and Select CRO Audits

• Vendor Scorecard• Inspections

• Responses & CAPAs to Agencies• Governance

• Outcomes of Relationship Committees, Study-level Issues Trackers

“Trust, but Verify”



Outputs and Deliverables Follow the Process• Approval of Study Team Personnel

• Evidence of Qualification Review, Selection, and Approval• Evidence of ongoing GCP, SOP, Protocol, and Therapeutic

Training

• Monitoring Reports• Logistics: timing, format, review, approval, distribution, storage• Content: visit frequency, SDV approach, PI interaction, issue

identification/escalation/resolution



Outputs and Deliverables Follow the Process• Clinical Database Lock Package

• Set up, Edit Specification Development, EDC Screen Design, Testing/Validation, Data Transfers, Data Cleaning, Safety Reconciliation, QC, Freeze/Lock

• Programming and Statistics• Blind Break, SAS Datasets, Program Development, Macro

Library and Controls, Mock Displays, Programming Outputs, Data Reviews, TLF Outputs, QC



Risk Focus• Indicator Patterns and Anomalies

• Determination of systemic items, Outliers, Root-Cause

• SOP/Work Instruction Cohesiveness• M&A Impact at either Sponsor or Vendor• Clear Definition of which Processes in Effect at Any Point in

Time• SOP / QMS Improvement Initiatives

• Infrastructure (Baseline and Incremental Views)• Network Topography, Validation Program, Back-up/Recovery• Facility Security, Business Continuity


Audit Fusion – Vendor Audit to Process Audit – Case Study

Background• Biometrics

• Vendor 1: FSP Audit Annually• Vendor 2: Biometrics Legacy Vendor Audited Biennially• Process Audit: Biometrics Function on Biennial Cycle

• A “Fused Audit” • Identification of a True End to End View Combining the Vendor

and Process Audits• Comprehensive Audit Strategy Plan with Modules for the

Process Audit and Two Vendor Audits• Focus on Expectation Setting, Oversight, Hand-offs, Process,

Actual Deliverables, Internal Review/Acceptance, Issue Resolution, and Feedback – Sponsor Oversight


Process Audit Vendor Audit 1 (FSP) Vendor Audit 2

Management: Oversight (Functional & Communication Plans), Expectation Setting (Quality Agreement), Personnel

Management: Project & Relationship Management, Expectation Setting (Quality Agreement Adherence), Personnel

Management: Project Management (Functional Plan & SOW Adherence), Expectation Setting (Quality Agreement), Personnel

SAS Application: Access Control, Deployment Types, Validation Package, Add-on Tools, SAS Macro Library

SAS Application: Access Control, Deployment Types, Validation Package, Add-on Tools, SAS Macro Library (Sponsor & Study Specific)

SAS Application: Access Control, Deployment Types, Validation Package, Add-on Tools, SAS Macro Library (Study Specific), Plans for Upgrades, Migration, and Project Impact

Programming: Procedures, Data set Specifications, Data Conversions, Data Transfers, Blinded Data Review

Programming: Procedures, Data set Specifications, Data Conversions, Derived Variable Checks, Macro Development/ Testing/ Revision, Output QC & Validation, Data Transfers, Blinded Data-set Production

Programming: Procedures, Data set Specifications, Data Conversions, Derived Variable Checks, Macro Development/ Testing/ Revision, Output QC & Validation, Data Transfers, Blinded Data-set Production

Statistical Analysis:Procedures, Random. Code Mgt., Analysis Population Define, Data Pooling, TFL Creation & QC, Interim & Ad-hoc Analysis, TFL Version Control

Statistical Analysis:Procedures, Random. Code Mgt., Data Pooling, TFL Creation & QC, Interim & Ad-hoc Analysis, TFL Version Control

Statistical Analysis:Procedures, Data Pooling, TFL Creation & QC, Interim & Ad-hoc Analysis, TFL Version Control


“Fused Audit” Approach

• Audit Planning More Strategic

• Fewer Audits Cost Saving

• Audit Intensity and Duration may increase

• Promotes Consistency Across like-audits

• Rotating Scope for Vendor.:

• Services A & B annually

• Service C every 3 years

• Infrastructure (physical, IT) biennially

• Ability to clearly see Total Picture

Debrief and Audit Report

Audit Yields/Benefits

Coordinated Multiple In-Process Vendor Auditsand Internal Process Audit

Verification and Effectiveness of Previous Commitments

CAPAs

Enhancement of Process, Expectations, and Relationship


FrameworkIntegrated view of sourcing picture

to identify economies of scale opportunities

Expectations Collaborate to

set jointExpectations

through QualityAgreement, TORO,

and Plans

MetricsStandard Menu of

Meaningful Metrics that can be Aggregated

Relationship Management

Audits

In-process,Full View,CAPA &Issue

Resolution

Regular, Collaborative,

Forward Looking, Strategic

FunctionalAssessment

Operation teams accountable

to determine“fit for use”

Sourcing Strategy Alignment

Vendor Evaluation

VendorManagement

Quality

VendorGovernance

Commitment Fulfilment

Supporting Quality Beyond the Audit

2nd exl quality oversight conf szpindor in process vendor audit

Health & Medicine

qualification audit

vendor representatives

evaluation of vendor

teams vendor governance

vendor evaluation shift

objective vendor selection

functional assessments

team personnel