Adverse Event Reporting Adverse Event Reporting at FDA,at FDA,

Data Base Evaluation and Data Base Evaluation and Signal GenerationSignal Generation

Robert T. O’Neill, Ph.D.Robert T. O’Neill, Ph.D.

Director, Office of Biostatistics, Director, Office of Biostatistics, CDER, FDACDER, FDA

Presented at the DIMACS Working Group

Disease and Adverse Event Reporting, Surveillance, and Analysis

October 16, 17, 18, 2002; Piscataway, New Jersey

Outline of TalkOutline of Talk

The ADR reporting regulationsThe ADR reporting regulations

The information collected on a report The information collected on a report formform

The data base, its structure and sizeThe data base, its structure and size

The uses of the data base over the yearsThe uses of the data base over the years

Current signal generation approaches - Current signal generation approaches - the data mining applicationthe data mining application

Concluding remarksConcluding remarks

OverviewOverview Adverse Event Reporting System (AERS)Adverse Event Reporting System (AERS)

Report SourcesReport Sources

Data Entry ProcessData Entry Process

AERS Electronic Submissions (Esub) AERS Electronic Submissions (Esub)

Production ProgramProduction Program

E-sub Entry ProcessE-sub Entry Process

MedDRA CodingMedDRA Coding

Adverse Event Reporting Adverse Event Reporting System (AERS) Database System (AERS) Database

Database Origin 1969Database Origin 1969

SRS until 11/1/97 ; changed to AERSSRS until 11/1/97 ; changed to AERS

3.0 million reports in database3.0 million reports in database

All SRS data migrated into AERSAll SRS data migrated into AERS

Contains Drug and "Therapeutic" Biologic Contains Drug and "Therapeutic" Biologic Reports Reports

exception = vaccines exception = vaccines VAERS 1-800-822-7967 VAERS 1-800-822-7967

Adverse Event Reporting Adverse Event Reporting SystemSystem

Source of ReportsSource of Reports Health Professionals, Consumers / Health Professionals, Consumers / PatientsPatients

Voluntary : Direct to FDA and/or to Voluntary : Direct to FDA and/or to ManufacturerManufacturer

Manufacturers: Regulations for Manufacturers: Regulations for Postmarketing Reporting Postmarketing Reporting

Current Guidance on Current Guidance on Postmarketing Safety Postmarketing Safety Reporting (Summary)Reporting (Summary)

1992 Reporting Guideline1992 Reporting Guideline

1997 Reporting Guidance: Clarification of What 1997 Reporting Guidance: Clarification of What to Reportto Report

1998 ANPR for e-sub1998 ANPR for e-sub

2001 Draft Reporting Guidance (3/12/2001) 2001 Draft Reporting Guidance (3/12/2001)

2001 E-sub Reporting of Expedited and Periodic 2001 E-sub Reporting of Expedited and Periodic ICSRs (11/29/2001)ICSRs (11/29/2001)

Adverse Events Adverse Events Reports to FDA Reports to FDA

1989 to 20011989 to 2001

0

50000

100000

150000

200000

250000

300000

350000

89 90 91 92 93 94 95 96 97 98 99 00 01

Direct15-dayPeriodic

Despite limitations, it is our Despite limitations, it is our primary window on the real primary window on the real

worldworld

What happens in the “real” world very What happens in the “real” world very different from world of clinical trialsdifferent from world of clinical trials

Different populationsDifferent populations

ComorbiditiesComorbidities

CoprescribingCoprescribing

Off-label useOff-label use

Rare eventsRare events

AERS FunctionalityAERS Functionality Data EntryData Entry

MedDRA CodingMedDRA Coding

RoutingRouting

Safety EvaluationSafety Evaluation

InboxInbox

SearchesSearches

ReportsReports

Interface with Third-Party ToolsInterface with Third-Party Tools

AutoCode (MedDRA) AutoCode (MedDRA)

RetrievalWare (images)RetrievalWare (images)

AERS

AERS Esub ProgramAERS Esub ProgramHistoryHistory

Over 4 yearsOver 4 years

Pilot, then production.Pilot, then production.

PhRMA Electronic Regulatory Submission (ERS) PhRMA Electronic Regulatory Submission (ERS) Working GroupWorking Group

PhRMA eADR Task ForcePhRMA eADR Task Force

E*Prompt Initiative E*Prompt Initiative

Regular meetings between FDA and Industry Regular meetings between FDA and Industry held to review status, address issues, share held to review status, address issues, share lessons learnedlessons learned

Adverse Event Reporting Adverse Event Reporting SystemSystem

Processing MEDWATCH formsProcessing MEDWATCH forms Goal: Electronically Receive Expedited and Goal: Electronically Receive Expedited and

Periodic ISRsPeriodic ISRs

Docket 92S-0251Docket 92S-0251

As of 10/2000, able to receive electronic 15-As of 10/2000, able to receive electronic 15-day reports day reports

Paper ReportsPaper Reports

Scanned upon arrivalScanned upon arrival

Data entered Data entered

Electronic and Paper ReportsElectronic and Paper Reports

Coded in MedDRACoded in MedDRA

Electronic Submission of Electronic Submission of Postmarketing ADR Postmarketing ADR

ReportsReports MedDRA coding 3500AMedDRA coding 3500A

Narrative searched with AutocoderNarrative searched with Autocoder

MedDRA coding E-subMedDRA coding E-sub

Narrative searched with AutocoderNarrative searched with Autocoder

Enabled: companies accept their termsEnabled: companies accept their terms

AERS Esub ProgramAERS Esub ProgramAdditional InformationAdditional Information

www.fda.gov/cder (CDER)www.fda.gov/cder (CDER)

www.fda.gov/cder/aers/regs.htm (AERS)www.fda.gov/cder/aers/regs.htm (AERS)

Reporting regulations, guidances, and updatesReporting regulations, guidances, and updates

www.fda.gov/cder/aerssub (PILOT)www.fda.gov/cder/aerssub (PILOT)

aersesub@cder.fda.gov (EMAIL)aersesub@cder.fda.gov (EMAIL)

www.fda.gov/cder/present (CDER www.fda.gov/cder/present (CDER PRESENTATIONS)PRESENTATIONS)

AERS Esub ProgramAERS Esub ProgramAdditional Additional

Information(cont’d)Information(cont’d) www.fda.gov (FDA)www.fda.gov (FDA)

www.fda.gov/oc/electronicsubmissions/www.fda.gov/oc/electronicsubmissions/interfaq.htm (GATEWAY)interfaq.htm (GATEWAY)

Draft Trading Partner Agreement, Frequently Draft Trading Partner Agreement, Frequently Asked Questions (FAQs) for FDA’s ESTRI Asked Questions (FAQs) for FDA’s ESTRI gatewaygateway

edigateway@oc.fda.gov (EMAIL)edigateway@oc.fda.gov (EMAIL)

www.fda.gov/medwatch/report/mfg.htm www.fda.gov/medwatch/report/mfg.htm (MEDWATCH)(MEDWATCH)

Reporting regulations, guidances, and updatesReporting regulations, guidances, and updates

16

AERS Esub ProgramAERS Esub ProgramAdditional Information(cont’d)Additional Information(cont’d)

www.ich.org (ICH home page)www.ich.org (ICH home page)

www.fda.gov/cder/m2/default.htm(M2)www.fda.gov/cder/m2/default.htm(M2)

ICH ICSR DTD 2.0ICH ICSR DTD 2.0

www.meddramsso.com (MedDRA MSSO)www.meddramsso.com (MedDRA MSSO)

http://www.ifpma.org/pdfifpma/M2step4.PDFhttp://www.ifpma.org/pdfifpma/M2step4.PDF

ICH ICSR DTD 2.1ICH ICSR DTD 2.1

http://www.ifpma.org/pdfifpma/e2bm.pdfhttp://www.ifpma.org/pdfifpma/e2bm.pdf

New E2BM changesNew E2BM changes

http://www.ifpma.org/pdfifpma/E2BErrata.pdfhttp://www.ifpma.org/pdfifpma/E2BErrata.pdf

Feb 5, 2001 E2BM editorial changes Feb 5, 2001 E2BM editorial changes

AERS UsersAERS Users

AERS

FDA Contractor

Safety Evaluators

Compliance

FOIA

Uses of AERSUses of AERS

Safety Signal DetectionSafety Signal Detection

Creation of Case ProfilesCreation of Case Profiles

who is getting the drugwho is getting the drug

who is running into troublewho is running into trouble

Hypothesis Generation for Further Hypothesis Generation for Further StudyStudy

Signals of Name ConfusionSignals of Name Confusion

Other referencesOther references

C. Anello and R. O’Neill. 1998, C. Anello and R. O’Neill. 1998, Postmarketing Surveillance of New Postmarketing Surveillance of New Drugs and Assessment of Risk, p 3450-Drugs and Assessment of Risk, p 3450-3457; Vol 4 ,3457; Vol 4 ,Encylopedia of Encylopedia of Biostatistics,Biostatistics, Eds. Armitage and Eds. Armitage and Colton, John Wiley and SonsColton, John Wiley and Sons

Describes many of the approaches to Describes many of the approaches to spontaneous reporting over the last spontaneous reporting over the last 30 years30 years

Related work on signal Related work on signal generation and modelinggeneration and modeling

Finney , 1971, WHOFinney , 1971, WHO

O’Neill ,1988O’Neill ,1988

Anello and O’Neill, 1997 -OverviewAnello and O’Neill, 1997 -Overview

Tsong, 1995; adjustments using external drug use Tsong, 1995; adjustments using external drug use data; compared to other drugsdata; compared to other drugs

Compared to previous time periodsCompared to previous time periods

Norwood and Sampson, 1988Norwood and Sampson, 1988

Praus, Schindel, Fescharek, and Schwarz, 1993Praus, Schindel, Fescharek, and Schwarz, 1993

Bate et al. , 1998; Bayes,Bate et al. , 1998; Bayes,

ReferencesReferences

O’Neill and Szarfman, 1999; The O’Neill and Szarfman, 1999; The American Statistician , Vol 53, No American Statistician , Vol 53, No 3; 190-195 Discussion of W. 3; 190-195 Discussion of W. DuMouchel’s article on Bayesian DuMouchel’s article on Bayesian Data Mining in Large Frequency Data Mining in Large Frequency Tables, With an Application to the Tables, With an Application to the FDA Spontaneous Reporting FDA Spontaneous Reporting System (same issue)System (same issue)

Recent Post-marketing Recent Post-marketing signaling strategies :signaling strategies :

Estimating associations Estimating associations

needing follow-upneeding follow-up

Bayesian data miningBayesian data mining

Visual graphicsVisual graphics

Pattern recognitionPattern recognition

The structure and content of The structure and content of FDA’s database: some known FDA’s database: some known

features impacting model features impacting model developmentdevelopment

SRS began in late 1960’s (over 1.6 million reports)SRS began in late 1960’s (over 1.6 million reports)

Reports of suspected drug-adverse event associations Reports of suspected drug-adverse event associations submitted to FDA by health care providers (voluntary, submitted to FDA by health care providers (voluntary, regulations) regulations)

Dynamic data base; new drugs, reports being added Dynamic data base; new drugs, reports being added continuously ( 250,000 per year)continuously ( 250,000 per year)

Early warning system of potential safety problemsEarly warning system of potential safety problems

Content of each reportContent of each report

Drugs (multiple)Drugs (multiple)

Adverse events (multiple)Adverse events (multiple)

Demographics (gender,age, other covariates)Demographics (gender,age, other covariates)

The structure and content of The structure and content of FDA’s database: some known FDA’s database: some known

features impacting model features impacting model developmentdevelopment

Quality and completeness of a report is Quality and completeness of a report is variable, across reports and variable, across reports and manufacturersmanufacturers

Serious/non-serious - known/unknownSerious/non-serious - known/unknown

Time sensitive - 15 daysTime sensitive - 15 days

Coding of adverse events (COSTART) Coding of adverse events (COSTART) determines one dimension of table - determines one dimension of table - about 1300 termsabout 1300 terms

Accuracy of coding / interpretationAccuracy of coding / interpretation

The DuMouchel Model and The DuMouchel Model and its Assumptionsits Assumptions

Large two-dimensional table of size M (drugs) x N (ADR Large two-dimensional table of size M (drugs) x N (ADR events) containing cross classified frequency counts - sparseevents) containing cross classified frequency counts - sparse

Baseline model assumes independence of rows and columns - Baseline model assumes independence of rows and columns - yields expected countsyields expected counts

Ratios of observed / expected counts are modeled as mixture Ratios of observed / expected counts are modeled as mixture of two, two parameter gamma’s with a mixing proportion Pof two, two parameter gamma’s with a mixing proportion P

Bayesian estimation strategy shrinks estimates in some cellsBayesian estimation strategy shrinks estimates in some cells

Scores associated with Bayes estimates used to identify Scores associated with Bayes estimates used to identify those cells which deviate excessively from expectation under those cells which deviate excessively from expectation under null modelnull model

Confounding for gender and chronological time controlled by Confounding for gender and chronological time controlled by stratificationstratification

The Model and its The Model and its AssumptionsAssumptions

Model validation for signal generationModel validation for signal generation

Goodness of fitGoodness of fit

‘‘higher than expected’ counts informative of true higher than expected’ counts informative of true drug-event concernsdrug-event concerns

Evaluating Sensitivity and Specificity of signalsEvaluating Sensitivity and Specificity of signals

Known drug-event associations appearing in a Known drug-event associations appearing in a label or identified by previous analysis of the label or identified by previous analysis of the data base; use of negative controls where no data base; use of negative controls where no association is known to be presentassociation is known to be present

Earlier identification in time of known drug-event Earlier identification in time of known drug-event associationassociation

Finding “Interestingly Large” Cell Finding “Interestingly Large” Cell Counts in a Massive Frequency Counts in a Massive Frequency

TableTable Large Two-Way Table with Possibly Millions of CellsLarge Two-Way Table with Possibly Millions of Cells

Rows and Columns May Have Thousands of CategoriesRows and Columns May Have Thousands of Categories

Most Cells Are Empty, even though Most Cells Are Empty, even though N..N.. Is very Large Is very Large

““Bayesian Data Mining in Large Frequency Tables”Bayesian Data Mining in Large Frequency Tables”

The American StatisticianThe American Statistician (1999) (with Discussion) (1999) (with Discussion)

Analyzed SRS Database with 1398 Drugs and 952 AE CodesAnalyzed SRS Database with 1398 Drugs and 952 AE Codes

NNijij = Count of Reports Containing Drug = Count of Reports Containing Drug ii and Event and Event jj

Only 386K out of 1331K Cells Have Only 386K out of 1331K Cells Have NNijij > 0 > 0

174 Drug-Event Combinations Have 174 Drug-Event Combinations Have NNijij > 1000 > 1000

Naïve Baseline Frequencies Naïve Baseline Frequencies EEijij = = NNi. i. NN.j.j / / NN....

Extension to Stratification: Sum Independence Frequencies Defined Extension to Stratification: Sum Independence Frequencies Defined Separately over Strata Based on Age, Sex, etc.Separately over Strata Based on Age, Sex, etc.

Associations of Items in ListsAssociations of Items in Lists ““Market Basket” Data from Transaction DatabasesMarket Basket” Data from Transaction Databases

Tabulating Sets of Items from a Universe of Tabulating Sets of Items from a Universe of KK Items Items

Supermarket Scanner Data—Sets of Items BoughtSupermarket Scanner Data—Sets of Items Bought

Medical Reports—Drug Exposures and SymptomsMedical Reports—Drug Exposures and Symptoms

Sparse Representation—Record Items PresentSparse Representation—Record Items Present

PPiijkjk = Prob(= Prob(XXii = 1, = 1, XXjj = 1, = 1, XXkk = 1), ( = 1), (i < j < ki < j < k) )

Marginal Counts and Probabilities: Marginal Counts and Probabilities: NNi i ,, NNij ij , , NNijk ijk , …, …PPi i , , PPij ij , , PPijkijk

Conditional Probabilities: Prob( Conditional Probabilities: Prob( XXii| | XXj j , , XXkk) = ) = PPijkijk / /PPjk jk , etc., etc.

PPii Small, but Small, but ii PPii (= Expected # Items/Transaction) >> 1 (= Expected # Items/Transaction) >> 1

Search for “Interestingly Frequent” Item SetsSearch for “Interestingly Frequent” Item Sets

Item Sets Consisting of One Drug and One Event Reduce to the GPS Item Sets Consisting of One Drug and One Event Reduce to the GPS Modeling ProblemModeling Problem

Definitions of Interesting Definitions of Interesting Item SetsItem Sets

Data Mining Literature: Find All (Data Mining Literature: Find All () Associations) Associations

E.g., Find all Sets (E.g., Find all Sets (XXi i , , XXjj , , XXkk) Having Prob( ) Having Prob( XXi i | | XXj j , , XXkk) > ) > Prob(Prob(XXi i ,, X Xj j , , XXkk) >) >

Complete Search Based on Proportions in Dataset, with No Complete Search Based on Proportions in Dataset, with No Statistical ModelingStatistical Modeling

Note that a Triple (Note that a Triple (XXi i , , XXj j , , XXkk) Can Qualify even if ) Can Qualify even if XXii Is Is

IndependentIndependent of ( of (XXj j ,, XXkk)!)!

We Use Joint P’s, Not Conditional P’s, and Bayesian ModelWe Use Joint P’s, Not Conditional P’s, and Bayesian Model

E.g., Find all (E.g., Find all (i, j, ki, j, k): Prob(): Prob(ijkijk = = PPijkijk//ijk ijk > > 00| | DataData) > ) >

ijkijk are are BaselineBaseline Values Values

Based on Independence or some other Null HypothesisBased on Independence or some other Null Hypothesis

Empirical Bayes Shrinkage Empirical Bayes Shrinkage

EstimatesEstimates Compute Posterior Geometric Mean (Compute Posterior Geometric Mean () and 5th Percentile () and 5th Percentile (0505) of ) of

Ratios Ratios

ij ij = = PPij ij //ijij , , ijk ijk = = PPijk ijk //ijk ijk , , ijklijkl = = PPijkl ijkl //ijkl ijkl , etc. , etc.

Baseline Probs Baseline Probs Based on Within-Strata Independence Based on Within-Strata Independence

Prior Distributions of Prior Distributions of s Are Mixtures of Two Conjugate Gamma s Are Mixtures of Two Conjugate Gamma DistributionsDistributions

Prior Hyperparameters Estimated by MLE from Observed Prior Hyperparameters Estimated by MLE from Observed Negative Binomial RegressionNegative Binomial Regression

EB Calculations Are Compute-Intensive, but merely Counting EB Calculations Are Compute-Intensive, but merely Counting Itemsets Is More So Itemsets Is More So

Conditioning on Conditioning on NNijk ijk > > n*n* Eases Burden of Both Counting and EB Eases Burden of Both Counting and EB EstimationEstimation

We Choose Smaller We Choose Smaller n*n* than in Market Basket Literature than in Market Basket Literature

The rationale for stratification The rationale for stratification on gender and chronological on gender and chronological

time intervalstime intervals

New drugs added to data base over timeNew drugs added to data base over time

Temporal trends in drug usage and exposureTemporal trends in drug usage and exposure

Temporal trends in reporting independent of drug: Temporal trends in reporting independent of drug: publicity, Weber effectpublicity, Weber effect

Some drugs associated with gender-specific exposureSome drugs associated with gender-specific exposure

Some adverse events associated with gender independent Some adverse events associated with gender independent of drug usageof drug usage

Primary data-mining objective: are signals the same or Primary data-mining objective: are signals the same or different according to gender (confounding and effect different according to gender (confounding and effect modification) modification)

A concern: number of strata, sparseness, balance between A concern: number of strata, sparseness, balance between stratification and sensitivity/specificity of signalsstratification and sensitivity/specificity of signals

The control group and the issue The control group and the issue of ‘compared to what?’of ‘compared to what?’

Signal strategies compareSignal strategies compare

a drug with itself from prior time periodsa drug with itself from prior time periods

with other drugs and eventswith other drugs and events

with external data sources of relative drug usage and with external data sources of relative drug usage and exposureexposure

Total frequency count for a drug is used as a relative Total frequency count for a drug is used as a relative surrogate for external denominator of exposure; for ease of surrogate for external denominator of exposure; for ease of use, quick and efficient; use, quick and efficient;

Analogy to case-control design where cases are specific Analogy to case-control design where cases are specific ADR term, controls are other terms, and outcomes are ADR term, controls are other terms, and outcomes are presence or absence of exposure to a specific drug. presence or absence of exposure to a specific drug.

Other metrics useful in Other metrics useful in identifying unusually large identifying unusually large

cell deviationscell deviations

Relative rateRelative rate

P-value type metric- overly influenced P-value type metric- overly influenced by sample sizeby sample size

Shrinkage estimates for rare events Shrinkage estimates for rare events potentially problematicpotentially problematic

Incorporation of a prior distribution on Incorporation of a prior distribution on some drugs and/or events for which some drugs and/or events for which previous information is available - e.g. previous information is available - e.g. Liver events or pre-market signalsLiver events or pre-market signals

Interpreting the empirical Interpreting the empirical Bayes scores and their Bayes scores and their

rankings:rankings: the Role of visual the Role of visual

graphicsgraphics(Ana Szarfman)(Ana Szarfman)

Four examples of spatial maps Four examples of spatial maps that reduce the scores to patterns that reduce the scores to patterns and user friendly graphs and help and user friendly graphs and help to interpret many signals to interpret many signals collectivelycollectively

All maps are produced with All maps are produced with CrossGraphs and have drill down CrossGraphs and have drill down capability to get to the data capability to get to the data behind the plotsbehind the plots

Example 1Example 1

A spatial map showing the A spatial map showing the “signal scores” for the most “signal scores” for the most frequently reported events frequently reported events (rows) and drugs (columns) (rows) and drugs (columns) in the database by the in the database by the intensity of the empirical intensity of the empirical Bayes signal score (blue Bayes signal score (blue color is a stronger signal color is a stronger signal than purple)than purple)

Example 2Example 2

Spatial map showing Spatial map showing ‘fingerprints’ of signal ‘fingerprints’ of signal scores allowing one to scores allowing one to visually compare the visually compare the complexity of patterns for complexity of patterns for different drugs and events different drugs and events and to identify positive or and to identify positive or negative co-occurrencesnegative co-occurrences

Example 3Example 3

Cumulative scores and Cumulative scores and numbers of reports numbers of reports according to the year when according to the year when the signal was first detected the signal was first detected for selected drugsfor selected drugs

Example 4Example 4

Differences in paired male-Differences in paired male-female signal scores for a female signal scores for a specific adverse event specific adverse event across drugs with events across drugs with events reported (red means females reported (red means females greater, green means males greater, green means males greater)greater)

Why consider data Why consider data mining approachesmining approaches

Screening a lot of data, with multiple Screening a lot of data, with multiple exposures and multiple outcomesexposures and multiple outcomes

Soon becomes difficult to identify Soon becomes difficult to identify patternspatterns

The need for a systematic approachThe need for a systematic approach

There is some structure to the FDA There is some structure to the FDA data base, even though data quality data base, even though data quality may be questionablemay be questionable

Two applicationsTwo applications

Special population analysisSpecial population analysis

PediatricsPediatrics

Two or more item associations Two or more item associations

Drug interactionsDrug interactions

Syndromes (combining ADR Syndromes (combining ADR terms)terms)

Pediatric stratificationsPediatric stratifications (age 16 and younger) (age 16 and younger)

NeonatesNeonates

InfantsInfants

ChildrenChildren

AdolescentsAdolescents

GenderGender

Item AssociationItem Association OutcomesOutcomes

Drug exposures - suspect and othersDrug exposures - suspect and others

EventsEvents

CovariatesCovariates

ConfoundersConfounders

Uncertainties of information in each fieldUncertainties of information in each field

dosage, formulation, timing, acute/chronic dosage, formulation, timing, acute/chronic exposureexposure

Multiplicities of dimensionsMultiplicities of dimensions

Why apply to pediatrics Why apply to pediatrics ??

Vulnerable populations for which Vulnerable populations for which labeling is poor and directions for use is labeling is poor and directions for use is minimal - a set up for safety concernsminimal - a set up for safety concerns

Little comparative clinical trial Little comparative clinical trial experience to evaluate effects ofexperience to evaluate effects of

Metabolic differences, use of drugs is Metabolic differences, use of drugs is different, less is known about dosing, different, less is known about dosing, use with food, formalations and use with food, formalations and interactions Gender differences of interactions Gender differences of interestinterest

Challenges in the Challenges in the futurefuture

More real time data analysisMore real time data analysis

More interactivityMore interactivity

Linkage with other data basesLinkage with other data bases

Quality control strategiesQuality control strategies

Apply to active rather than passive Apply to active rather than passive systems where non-reporting is not systems where non-reporting is not an issuean issue