Slide 1

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com The Challenges of Validating R Managing R in a Commercial Environment

Slide 2

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What does validation mean Why validate R The challenges of validating R Validation tools ValidR Questions Agenda

Slide 3

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What does validation mean? Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product, meeting its predetermined specifications and quality attributes U.S. Food and Drug Administration (2013). 21 CFR Part 11: Electronic Records, Electronic Signatures, http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1

Slide 4

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Why validate R? No guarantee that packages meet specifications Must comply with industry regulations to be used in regulated environments

Slide 5

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Why validate R? R Foundation response to CFR Part 11: R: Regulatory Compliance and Validation Issues A Guidance Document for the Use of R in Regulated Clinical Trial Environments 1 Document relates to only base R and recommended packages R CMD CHECK provides no guarantee that an R add-on package meets its specifications Authors do not have to write tests 1. http://www.r-project.org/doc/R-FDA.pdf

Slide 6

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com ## WARNING: ## This code is a complete hack, ## may or may not work, etc.. ## Use your own risk. You have been warned. The recommended R package: codetools, version 0.2-8. Why validate R?

Slide 7

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What is validation? Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product, meeting its predetermined specifications and quality attributes U.S. Food and Drug Administration (2013). 21 CFR Part 11: Electronic Records, Electronic Signatures, http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1

Slide 8

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com The challenges of validating R Define Requirements Build Software Test Software Against Requirements Create Documentation Typical Software Development Process Investigation Remediation Certification

Slide 9

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Challenge 1 : Defining requirements Packages dont typically come with a list of requirements How do we determine the intended use? Package descriptions and help files Package vignettes Previous experience of using the packages

Slide 10

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Challenge 2 : Understanding a package Software is already built Source code is available How do we gain an understanding of the package structure? The functionMap package

Slide 11

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com The functionMap package Describes the relationship of functions within an R package Parse R code Create dynamic graphics of function relationships Create a network object to show relationships Creates an interactive graphic See which functions a given function calls Determine functions that are called within each function

Slide 12

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com

Slide 13

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com

Slide 14

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Challenge 3 : Testing the requirements Package authors dont have to write tests How do we test the requirements? Writing specific unit tests for requirements Understand the level of testing with testCoverage

Slide 15

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com The testCoverage package Determines which code in a package is covered by unit tests Reads package code and adds tracepoints Runs package unit tests and marks points as hit Creates a full report of the code hit

Slide 16

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com

Slide 17

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Challenge 4 : Validation documentation Creating consistent documentation for each R package Automatic report template generation knitr Audit trail

Slide 18

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Challenge 5 : Deployment Installation Qualification (IQ) Operation Qualification (OQ) Performance Qualification (PQ)

Slide 19

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com IQ - Is everything installed as expected? Perform an audit of the installed files OQ - Does everything operate as expected? Run the tests used in validation locally PQ - Does everything perform as expected? Measure the performance of the installed software VALIDATION Qualification process

Slide 20

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What is ValidR? R (currently 3.0.2) Core and Recommended Packages 60+ additional packages Validation Packages Validation Documentation Windows Installer Linux RPMs

Slide 21

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What are the benefits? Industry Gives analysts the tools they want use Provides assurance to the IT department Reduces risk and cost to business managers Accelerated analytical cycle R community Bug fixes patches Increased user base

Slide 22

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com The ValidR roadmap September 2014 EARL Conference Release testCoverage and functionMap on CRAN Easter 2015 Release of validated R 3.1.x Summer 2015 Release of validated additional packages

Slide 23

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com What does validation mean Assurance, evidence Why validate R Provide assurance, evidence The challenges of validating R Validation tools ValidR Methodology, software, documentation Benefits Summary

Slide 24

DRAFT Richard Chandler-Mant R Consultant rchandler-mant@mango-solutions.com Questions?