1

Audit of the Columbia Suicidality Classification Project

September 13, 2004Meeting of Psychopharmacological Drugs Advisory Committee and

Pediatric Advisory Committee

Solomon Iyasu, M.D., M.P.H.Medical Team LeaderDPDD/OCTAP/CDERFood and Drug Administration

2

Outline

• Background

• Objectives of the FDA audit

• FDA audit method, process

• Results of the audit

• Limitations/strengths

• Conclusion

3

Objective of the FDA Audit

• Assess the reproducibility and reliability of the Columbia suicidality classification methodology and process

4

The FDA Audit Methods and Process:Event Narratives

• Columbia reviewed and classified ALL sponsor-submitted event narratives

• FDA audit team received a computerized line listing of these event narratives from the Div. Neuro-pharmacological Drug Products

• Drew a sample of events for review

5

The FDA Audit Methods and Process

• Grouped event narratives into 4 predefined strata

• Selected event narratives for review via a stratified simple random sampling strategy

• Over-sampled difficult to classify and reclassified events

6

The FDA Audit Methods and Process:Strata and Definitions

• Stratum 1: Events reclassified by Columbia to non-suicidal or other events (n=2)

• Stratum 2: Events newly identified and classified by Columbia as possibly suicide related or other categories (n=29)

• Stratum 3: Events that were difficult to classify defined as events with discordant initial independent ratings by Columbia reviewers (n=56)

• Stratum 4: Events that are straight forward cases defined as events with concordant initial ratings by Columbia reviewers (n=336)

7

The FDA Audit Methods and Process:Sample Size and Selection

• Sample size: 64 of the 423 event narratives selected for review (15%)

– Stratum 1: All selected ( n=2)– Stratum 2: 1/3rd selected (n=10)* – Stratum 3: 1/3rd selected (n=19) – Stratum 4: 1/10th selected (n=33)

• Events from stratum 2 – 4 were selected by a simple random technique using a random number table

* Four of the sampled records were also ‘difficult to classify”

8

Composition of the FDA Audit TeamPlanning Group

• Div. of Pediatric Drug Development, OCTAP, CDER– Solomon Iyasu, MD, M.P.H.,

Audit Team Leader– Susan Cummins, M.D., M.P.H. – Rosemary Addy, Project Manager– Kristin Phucas, Project Manager

• Div. of Neuro-pharmacological Drug Products, CDER– Thomas Laughren, M.D.

• OND– Armando Oliva, M.D.

Clinical Reviewers

• Div. of Pediatric Drug Development, OCTAP, CDER– Hari Sachs, M..D., (Pediatrician)– ShaAvhree Buckman, M..D., Ph.D.

(Pediatrician)

• Div. of Neuro-pharmacological Drug Products, CDER– Cara Alfaro, PharmD. (Pharmacist)– Robert Levin, M..D. (Psychiatrist)

Consensus meeting facilitator

• Div of Scientific Investigations – Robert Stasko, M.D. (psychiatrist),

None of the clinical reviewers had previously reviewed the event narratives

9

FDA Audit Team Training

• FDA Audit Team received a 2-hour teleconference training by Kelly Posner about the suicidality classification scale, method and process including– A review of the classification scale, categories and

construct

– A review of examples of case narratives for each of the classification categories

– Independent classification of test event narratives by participants to evaluate the training

10

The FDA Audit Methods and Process:Review Assignments

• Each sampled event randomly assigned to three of four reviewers for independent and blinded review

• Each reviewed and rated 48 events (total of 192 reviews)

• Reviewers blinded to treatment assignment, sponsor, diagnosis and to the Columbia ratings

11

The FDA Audit Methods and ProcessReview Procedures/Instructions

• A memo outlining the procedures of the audit was prepared and provided to the audit team

• Reviewers were not allowed to discuss the events among themselves or with colleagues during the independent review

• Allowed reviewers to call Kelly Posner, Columbia University to obtain clarification on the classification scale but no discussion of the specifics of any case was allowed

• Required reviewers to record on rating form if they consulted with Columbia during the review process

• Modified and pre-coded rating form used

12

13

The FDA Audit Methods and Process• Reviewers returned completed rating forms in sealed envelopes

• Ratings double key entered into an Excel database • Discordant ratings identified and discussed during a consensus

meeting facilitated– by a board certified child and adolescent psychiatrist external

to DNDP– previously not involved in the review of these records or had

knowledge of the Columbia ratings score

• Final consensus ratings entered into an Excel database and compared to the final Columbia ratings.

• Discordant ratings between FDA and Columbia discussed with Columbia to better understand the reason for the differences

14

Criteria for Evaluation of Concordant Ratings

• Columbia Definition of Concordance

– Categories 1, 2, 3, 6, and 10 required exact match

– Categories 4, 5 or 11 all describe self-injurious behavior with no suicidal intent and considered equivalent rating

– Categories 7, 8, 9 or 12 all describe “other” (non-suicidal and non-self-injurious behavior) and are considered equivalent rating

15

Results:Concordance among FDA reviewers

• 47 of 64 initial ratings were concordant

• 17 of 64 initial ratings were discordant

– Event narratives discussed during the FDA reviewer’s consensus meeting

– Final consensus ratings reached for all 17 events

16

Results:Comparison of Final FDA and Columbia Ratings

• Concordance on 57 of 64 final event ratings

– Percent agreement of 89% ( k=0.84)

• Discordance analysis (n=7)

– Severity Hierarchy• (1 or 2) > 6 > 3 > (4 or 5 or 11) > 10

– Compared to Columbia, the FDA audit team classified • six events with higher severity (3 of these events were

classified as ‘not enough information’ vs. ‘other’)• one event with lower severity

17

Limitations

• Neither the quality of the narratives nor the clinical source material for the narratives were evaluated

• Validity of the Columbia classification method not assessed

– there is no gold standard

18

Strengths

• Achieved a high level of concordance between the two independent review teams despite

– Differences in expertise and experience

– Short FDA timeline for training and review

– Intentional over-sampling of difficult to classify events for review

19

Conclusion

• The Columbia Suicidality classification methodology is robust and reproducible.

20

Acknowledgments

• OCTAP/CDER: Hari Sachs, ShaAvhree Buckman, Rosemary Addy, Kristin Phucas, Susan Cummins

• DNDP/CDER: Robert Levin, Cara Alfaro, Thomas Laughren, Alice Hughes, Tarek Hammad

• DSI: Robert Stasko

• OND: Armando Oliva

• Columbia University: Kelly Posner

21

Thank you!