1

IVT Validation Week

Automate The Automate The Validation ProcessValidation Process

Tyson M. Mew, PresidentOfni Systems Inc.

October 22, 2010 2

About the Speaker

Ty MewTy Mew is President of Ofni Systems, Inc., a is President of Ofni Systems, Inc., a software provider and Part 11 consulting firm. He software provider and Part 11 consulting firm. He has a strong background in software development has a strong background in software development and computer system validation, and has spoken at and computer system validation, and has spoken at many conferences and workshops on Part 11. Ty many conferences and workshops on Part 11. Ty regularly conducts training sessions for regularly conducts training sessions for organizations to educate employees about the organizations to educate employees about the specific requirements of the rule and how to specific requirements of the rule and how to incorporate it into daily practices. He is an active incorporate it into daily practices. He is an active member of ISPE, PDA, SQA and the Triangle member of ISPE, PDA, SQA and the Triangle PEERS group.PEERS group.

2

Getting Started

Why Validation is well-suited to automation Formalizing your strategyPreparing templates for documents and certain types of systemsDetermine and execute your testing strategy

3

Quick Poll

Are you primarily validating:• Commercial Off-The Shelf (COTS) without access

to code (black box testing)• Open systems with access to system code (white

box testing)Do you execute protocols electronically?

Automating Computer System Validation

Software is Well-Situated to Automation

Use automated software tools to • Analyze Software• Detect Objects (Screens, Reports, Code)• Identify Missing Objects• Create test scripts

4

Formalizing Validation Strategy

Create a General Strategy for All Projects:• Identify Validation Tactics/Testing Approaches for

Common Elements– Data Inputs– Formulas, Calculations– Reports, Charts

• Define Strategies for Specific Software TypesValidation Strategy Documented in Validation Master Plan

Formalizing Validation Strategy

Unifies Validation ApproachEnsures Internal Agreement with Validation StrategyImproves Training of Validation Professionals Facilitates Review/Approval ProcessProvides chance to implement best practices for lean project management

5

Validation TemplatesTemplates Improve document consistency and

efficiency of document creationTemplates Include:• SOP identified Sections (Objectives, Scope, etc.)• Instructions for completing specific sections• Template text with variable names• Accepted Formatting

Hint: Fewer templates are better than more templates. Use variables to create constant text.

Automated Document Generation

Extension of TemplatesIncludes specific verbiage with variables.

Variables are replaced with specific data.Requirement lists automatically inserted into appropriate sections of validation documentsThis approach works well with similar

programs with similar requirements (Groups of spreadsheets, etc.)

6

Example Variable NamesSystem NameProgram Version NumberFile Name, Data File Name, Security File Name, External Code Library NameFile LocationCompany Name, Company AddressOwning Department, System OwnerDocument Names, ID#s, Abbreviations, TitlesProtocol Names, Abbreviation

Determining Testing Strategy

Determine what you are going to do, then execute your planValidation Professionals test according to planQuality review/approve documents based on existing standards

7

Automate the Generation of Validation Documents

Breaking down systems into discrete validatable objects Collecting testable requirements Integrating Risk Assessments into the processGenerating Design Specifications directly from source codeAutomatic generation, tracking and updating of the Requirements Traceability Matrix

Breaking Down Systems intoDiscrete Validatable Objects

Define Distinct Parts of the Program•Forms, Reports•Worksheets, Charts•Workflows

Automatically Create List of Validatable ObjectsNote: We use software to analyze programs, record logical order for forms and reports. Software also identifies missing objects

8

Collecting RequirementsDefine requirements during project walk-through

Additional requirements can be gathered from key end users, developers, system owners.

Note: GAMP 5 specifically allows combining/separating of specification documents to facilitate validation.

Hint: Use software which allows you to record requirements as the validatable objects are recorded

Integrated Risk AssessmentValidation Resources are limited; a Risk-Based Validation approach is necessary to focus limited resourcesRisk Assessment is performed at the requirement level• Functional Specification• Design Specification

Ask Developers/Key System Users/System Owners where system risk is. They know and you usually get a better ROI than trying to analyze every requirement.

9

Define Testing for Each Object

Forms: Input, Processing, Output, SecurityReports: Data, Calculations, FormattingSpreadsheets: Data Entry, Calculations, Data Outputs, LabelsCharts: Data Ranges, Calculations, Labels/FormattingWorkflows: Process Flow

Generating Design Specifications

Once requirements are defined, use software tools to analyze objects and produce suitable outputOnly applicable to open source code.

Hint: Use software scripts to analyze common objects and produce suitable content for the Design Specification.

10

Example of Generating Design Specifications: Data Input Form

Examples of Generating Design Specifications: Spreadsheet Calculations

11

Automatic Requirement Traceability

Several Programs exist which will automatically generate the Requirement Traceability MatrixUpdates if Requirements are Moved/EditedDisplays if Functional Requirements exists without corresponding Design specificationDisplays if Requirements exist which do not have a defined Test Case

12

Test Cases and Test Plans

Testing based on Risk AssessmentsInput Testing vs. Challenge Testing, Workflow TestingAutomating with micro-test casesAuto-creation of test scripts from your requirementsCreate specific test cases for use with change control

How Much Is Enough?Automating Testing allows more time for Testing, Less Time spent on Documentation.Defining specific requirements based on Risk allows for use of different templates based on specific functional needs.Automated creation of Test Cases/Protocols allows validation resources to focus on high risk sections of program functionalityRequired business functionality should receive special attention.

13

Input vs. Challenge TestingHigh Risk Fields Should have Challenge Tests

Status Fields • Do not accept <NULL>• If Drop-Down List Used, Status not accept other values• Default Value should not be Pass, Success, etc.

Dates• Do not accept clearly incorrect dates• Do not accept unordered dates (Date of Test before Date

Received, etc.)

Workflow Testing

Extremely useful technique for systems with a defined process or workflow (Sample Management, Manufacturing Process, etc.)• Verify that entered data can move through entire

workflow• Verify that fail-points prevent data from moving

forward• Verify that data cannot be accessed from

inappropriate parts of the workflow

14

Using Micro-Test CasesMicro-Test Cases are test cases for common requirements have pre-written verbiage (with variables) which is inserted into appropriate sections of the document.Parallel sections inserted into Requirement Documents, Design Documents and Testing DocumentsSmaller test cases combined into larger test cases

Using Micro-Test CasesMicro-Test Cases most useful for common, repeated test steps.• Adding/Closing Forms or Reports• Data Entry into Fields• Screen Navigation• Printing, Generating Charts, Etc.• Confirmation of Formulas/Calculations• Any place in Test Protocol where similar operations are

repeated with similar resultsContent of Micro-Test case modified with variables

15

Enter Required Variables

Lines of Test Protocol Generated

16

Automatic Generation of Test Protocols

In certain systems, software can be used to automate the generation of entire test protocols based on defined inputsData Entry ScreensIndividual Spreadsheet WorksheetsSecurity Testing

Automatic Generation of Test Protocols

17

Test Cases for Change Control

Certain Test Cases Are Particularly Useful when verifying functionality after Change Control• Regression Testing• Functionality Identified as High RiskWherever possible, Test Cases should be written mindful of reuse during Change Control

Protocol Execution, Documentation and Managing Deviations

Paper vs. Electronic executionCapturing actual results and screen shots for maximum effectivenessGenerating, processing and closing test and script deviations

18

Paper vs. Electronic Execution

Electronic Protocol Execution• Replaces traditional pen and paper techniques• Just as word processors improve writing, electronic

protocol execution improves paper executionsDocuments can be exported and reviewed by traditional means (“Type-writer Rule”)

Note: Electronic Protocol Software must be compliant with 21 CFR 11, with audit trails, security, electronic signatures, etc.

Example of Electronic Protocol Execution

19

Documenting Protocol Execution with Screen Shots

As each step is completed, collect screen shotInsert Screen shots automatically into finished protocolConfigure Screen shots to include user name, date/time, etc. which further document testing

Hint: We use screen shot software, integrated with Electronic Protocol Execution software. This significantly reduces time organizing and numbering screen shots.

Warning!Screen shots do not prove a test step was successful,

they only provide evidence that the test was performed.

Documenting Protocol Execution with Screen Shots

20

Automating Deviation Reporting

When a test step fails, deviation is automatically createdUser identifies deviation type, based on this determination, some fields are automatically filled.Deviations can be viewed in Real-TimeDeviations can be resolved according to each companies procedures

21

October 22, 2010 41

Validation Case Study ReviewsEnsure compliance with 21 CFR 11• Technological• Procedural

Risk AssessmentGather Requirements• Screen Specific• Workflow Specific• Security

Write Requirement DocumentsTesting Documents

Case StudiesSpreadsheets for CRO• Applied tools to spreadsheets to provide

technological controls, including audit trails, user level security and electronic signatures

• Spreadsheet requirements generated from micro templates.

• Requirements verified through protocol executionResult: Entire spreadsheet library quickly and

efficiently validated

22

Case StudiesEngineering system for Pharmaceutical Manufacture• Requirements gathered from key users and developers• Software emphasized moving through exact workflows, so

testing plan modified to focus on workflows over data entry screens with associated reports

• Requirements verified through protocol execution

Result: Emphasized workflows over input/output model when appropriate, which improved testing and overall validation effort.

Case StudiesElectronic Document Capture System• Worked with developers to demonstrate compliance with 21

CFR 11• Program used Client/Server model, with digital pen inputs• Extensive use of virtual machines to model each section of

program – different OS, different sections• Tested specific form events• Requirements verified through protocol execution

Results: Validation effort allowed quick and efficient compliance with 21 CFR Part 11 regulations. After the validation process was complete, the validation results were reviewed and approved by a third party auditor

23

Case Studies

Purchasing System for Pharmaceutical Manufacturer• Requirements gathered from company SOP and

instructions• Worked with 3rd party vendor to add security• Met with both End users and developers• Requirements verified through protocol execution

Result: Third party software made compliant with 21 CFR 11, with implemented Security requirements

Case Studies

Clinical Research Database from CRO• Clear requirements, no requirement drift• Used existing templates, validation document

generation and electronic protocol execution• Requirements verified through protocol execution

Result: Validation completed in 12 business days.

24

Live Examples

Demonstration of the concepts described in this presentation

(Time Permitting, of Course)

October 22, 2010 48

Thank You!

Questions? Tyson M. Mew(919) 844-2494Ofni Systems Inc.Ty.Mew@OfniSystems.comwww.OfniSystems.com

References and additional materials will be available at www.OfniSystems.com at the conclusion of the conference.

25

Additional Reference Material

Solutions for Common Software

Web Based Applications Custom Databases (MS Access, FoxPro, SQL Server, Oracle, etc.)Spreadsheets (MS Excel, Quattro, Lotus, etc.)Connections to LIMS

In addition to system specific issues, all systems must demonstrate control of data, and meet any applicable business requirements.

26

Web Based Applications

Define System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/ComputationsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security RequirementsMake sure all functions are testedMake sure all functions are tested

October 22, 2010 52

Custom DatabasesAccess, FoxPro, SQL Server, Oracle, etc.Access, FoxPro, SQL Server, Oracle, etc.

Define System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/ComputationsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security Requirements

27

Spreadsheets

Excel, Quattro, Lotus, etc.Define Data Input Cells/RangesDefine Spreadsheet Processing• Calculation Cells• Internal FunctionsDefine System Outputs/ChartsDefine Security Requirements

Connections to LIMS

Define Data Type and RequirementsDefine Data Workflow• Define Data Input to LIMS • Define Output from LIMS• Calculations are performed inside

LIMS and exported• Calculations are performed outside

LIMS and re-imported

28

Gathering Requirements

Interviews with:• Key System End-Users• System Owner• Developer

Good communication between the validation team and all end users improves all aspects of the project and adds more value to the project.

Gathering RequirementsScreen Specific• Input tests

– Make sure that all fields accept appropriate data entry

– Use challenge tests where appropriate• Processing

– Make sure all functions (buttons, etc.) operate as expected

– Make sure all calculations produce correct results

29

Gathering RequirementsWorkflow Specific• Follow specific data entry throughout the workflow.• Follow data transitions in workflows

Security• Define specific groups • Define specific workflows available• Define specific functionality• Test what users can and cannot do

Risk Assessment

Determine what parts of the software create the most risk and focus testing there.Risk Assessment should be multi-disciplinary, and should include validation professionals, key end users, quality review, developers and system owner.Focus on known areas first