validation of a computerized assessment of functional capacity · international congress on...

INTERNATIONAL CONGRESS ON SCHIZOPHRENIA RESEARCH | COLORADO SPRINGS | MARCH 28 – APRIL 1, 2015

Validation of a Computerized Assessment of Functional Capacity

RICHARD KEEFE, PHD Professor of Psychiatry & Behavioral Sciences and Psychology & Neuroscience

Duke University Medical Center

Stacy A. Ruse, Nathan B. Spagnola, Vicki G. Davis, Alexandra S. Atkins, Thomas L. Patterson, Meera Narasimhan, Philip D. Harvey

2 INTERNATIONAL CONGRESS ON SCHIZOPHRENIA RESEARCH | COLORADO SPRINGS | MARCH 28 – APRIL 1, 2015

Financial DisclosuresPast Three Years

CONSULTANT/AD BOARD/SERVICE PROVIDER Abbvie, Akebia, Amgen, Astellas, Asubio, AviNeuro/ChemRar, Biogen Idec, BiolineRx, Biomarin,

Boehringer-Ingelheim, Eli Lilly, EnVivo/FORUM, GW Pharmaceuticals, Helicon, Lundbeck,

Merck, Mitsubishi, Novartis, Otsuka, Pfizer, Roche, Shire, Sunovion, Takeda, Targacept

RESEARCH FUNDINGDepartment of Veteran’s Affairs, Feinstein Institute for Medical Research, GlaxoSmithKline,

NIMH, Novartis, Psychogenics, Research Foundation for Mental Hygiene, Singapore Medical

Research Council

FOUNDER OF NEUROCOG TRIALS, INC. Providing rater training, data quality assurance and consultation to several pharmaceutical

companies and other consortia

SHAREHOLDER Sengenix

ROYALTIES Brief Assessment of Cognition in Schizophrenia (BACS), MATRICS Consensus Cognitive Battery

(MCCB), Virtual Reality Functional Capacity Assessment Tool (VRFCAT)


Co-primary Measures for Clinical Trials in Schizophrenia

• Largely ignored

• 50% of the necessary regulatory signal!

• Disability measures such as the UCSD Performance-based Skills Assessment (UPSA) may have ceiling effects for some patients and cultural adaptability is a consideration

• Interview-based measures require an objective informant • Patient report of cognitive impairment without informant

input has zero to very low correlation with actual cognitive performance

3


Introduction

We developed the Virtual Reality Functional Capacity Assessment

Tool (VRFCAT) as a reliable assessment of functional capacity for

use in such trials.

Using a virtual reality environment, the VRFCAT assesses the

ability to complete activities associated with a shopping trip.


VRFCAT Components

Searching the pantry Shopping in a store

Making a list Paying for the purchases

Taking the correct bus Getting home


VRFCAT Components


Methods

The present study assessed the validity, sensitivity and reliability

of the VRFCAT in patients with schizophrenia and healthy controls

The discriminability of patients with schizophrenia and

healthy controls was compared on VRFCAT measures

The relationship between VRFCAT outcomes and cognitive

performance on the MATRICS Consensus Cognitive Battery

(MCCB) was also assessed

VRFCAT performance was compared to the UCSD

Performance-based Skills Assessment (UPSA-2-VIM)


Methods

166 healthy controls (HCs) and 158 patients with schizophrenia

(SZ) were recruited from three sites: University of California San

Diego, University of Miami Miller School of Medicine, and

University of South Carolina. NOTE: One HC was removed due to extremely low test scores suggesting this individual was not healthy.


HC (N = 165)

SZ (N = 158)

Age, Mean (St Dev) 42.6 (13.93) 43.6 (11.84)

Male, N (%) 88 (53) 87 (55)

Non Hispanic, N (%) 136 (82) 128 (81)

English as Primary Language, N (%) 157 (95) 151 (96)

Unemployed, N(%)* 54 (33) 135 (85)

Comfortable with PC, N (%)* 160 (97) 140 (89)

Years of Education, Mean (St Dev)* 14.7 (2.41) 12.8 (1.99)

Mother’s Years of Education, Mean (St Dev) 12.9 (2.98) 12.5 (3.33)

Demographics

* Indicates significant differences between HC and SZ at the 0.05 significance level.


Procedure

MCCB administered at Visit 1

The VRFCAT and UPSA-2-VIM were completed at Visit 1 and 2.

Items on the VRFCAT were compared for the HCs and SZs.

Analyses examined test-retest reliability, performance

differences, and correlations between VRFCAT measures, the

MCCB Composite T-score and the UPSA-2-VIM Total Score.

Clinician SCoRS Total, Mean (SD) 38.2 (9.88)

PANSS Total, Mean (SD) 71.6 (21.93)

Clinician SLOF Total, Mean (SD) 120.8 (14.42)

Note: The SCoRS, PANSS, and SLOF were only administered to the SZ group.

SCoRS, PANSS, and SLOF Total Scores for SZ at Visit 1


1 Pick up the recipe on the counter.

2 Look for ingredients in your

cabinets and refrigerator.

3

Cross off the ingredients that you

already have in your apartment

and pick up the bus schedule on

the counter.

4 Pick up the bill fold on the

counter.

5 Leave the apartment and head to

the bus stop.

6 Wait for the correct bus to the

grocery store and then board it

when it arrives.

7 Add up the exact amount of bus

fare in your hand

8 Select a food aisle to begin

shopping.

9 Continue shopping for the

necessary food ingredients and

checkout.

10 Add up the exact amount for your

purchase in your hand.

11 Wait for the same bus that took

you to the grocery store and then

board it when it arrives.

12 Add up the exact amount of bus

fare in your hand.

VRFCAT Measures


VRFCAT Summary T-scores

VRFCAT summary measures were converted into age- and

gender-corrected T-scores using a regression-based approach

Appropriate regression models containing age and gender as

predictors were fit to data from the HC sample

Linear regression for log transformed total time

Poisson regression for total errors

Logistic regression for progression due to errors or time


• Each schizophrenia subject’s predicted score based on age and gender (using parameter estimates from the regression model) was subtracted from their actual score to yield a residual value.

• The sign of the residuals was reversed so that higher values reflected better performance and then transformed into T-scores using the SD of the residuals from the regression model on the HC sample.

• The resulting T-score reflects how the SZ subject performed relative to expectations for a healthy control subject of the same age and gender

VRFCAT Summary T-scores


HC (N = 165)

SZ (N = 158)

Cohen’s d

MCCB Composite Score, Mean (SD)* 44.0 (13.19) 28.1 (12.91) 1.22

VRFCAT Total Time T-score, Mean (SD)* 49.7 (11.51) 32.5 (16.60) 1.21

VRFCAT Total Errors T-score, Mean (SD)* 49.4 (11.62) 37.6 (22.37) 0.67

VRFCAT Progression T-score, Mean (SD)* 49.7 (10.16) 40.5 (13.62) 0.77

UPSA-2-VIM, Mean (SD)* 83.2 (9.03) 71.0 (11.85) 1.16

Discrimination Between Schizophrenia Patients & Healthy Controls

The three VRFCAT summary measures, the MCCB Composite

Score, and the UPSA-2-VIM all demonstrated significant

differences between HC and SZ at the first visit.

* Indicates significant differences between HC and SZ at the 0.05 significance level.


VRFCAT Total Time T-score Discrimination Between Healthy Controls & Schizophrenia

Patients

0

2

4

6

8

10

-25 -15 -5 5 15 25 35 45 55 65 75

Freq

uen

cy

VRFCAT Total Time T-score

Healthy Controls Schizophrenia Patients


Test-retest Reliability & Practice Effects

Assessment

Visit 1

Mean (SD)

Visit 2

Mean (SD)Cohen’s d

Intraclass

Correlation

Coefficient

(ICC)

HC SZ HC SZ HC SZ HC SZ

VRFCAT Total Time T-score50.1

(11.12)

32.3

(16.78)

50.9

(11.52)

31.8

(17.62)0.07 -0.03 0.65 0.81

VRFCAT Total Errors T-score49.7

(11.48)

37.1

(22.74)

49.8

(12.94)

36.7

(22.07)0.01 -0.02 0.54 0.65

VRFCAT Progression T-score49.8

(10.20)

40.4

(13.66)

50.3

(10.51)

40.8

(13.58)0.05 0.03 0.29 0.61

UPSA-2-VIM*83.4

(9.06)

70.7

(11.83)

86.7

(9.07)

74.5

(12.07)0.36 0.32 0.75 0.78

*Indicates significant differences between HC and SZ at the 0.001 significance level for HC and SZ.


Relationship To MCCB & UPSA-2-VIM

Assessment VTT T-score VTE T-score VP T-score MCCB UPSA

VRFCAT Total Time T-score --- 0.75 0.60 0.68 0.60

VRFCAT Total Errors T-score 0.69 --- 0.70 0.50 0.52

VRFCAT Progression T-score 0.70 0.64 --- 0.35 0.40

MCCB Composite T-score 0.57 0.39 0.45 --- 0.74

UPSA-2-VIM 0.59 0.41 0.43 0.70 ---

All correlations p-values were < 0.001

Pearson Correlation Coefficients Between The VRFCAT, UPSA-2-VIM & MCCB

For Healthy Controls & Schizophrenia Patients

Healthy Controls Schizophrenia Patients


VRFCAT Total Time & UPSA-2-VIM vs. MCCB Composite Score Schizophrenia Patients at Visit 1

25

35

45

55

65

75

85

95

105

-25

-5

15

35

55

75

95

-5 5 15 25 35 45 55 65 75

UP

SA-2

-VIM

VR

FCA

T To

tal T

ime

T-sc

ore

MCCB Composite Score

VRFCAT Composite T-score UPSA-2-VIM

Linear (VRFCAT Composite T-score) Linear (UPSA-2-VIM)

PEARSON CORRELATIONVRFCAT & MCCB: 0.57UPSA-2-VIM & MCCB: 0.70


Results from this study suggest the VRFCAT has

Good test-retest reliability

Strong discrimination between patients and HCs

Strong correlations with the MCCB and UPSA-2-VIM

Minimal practice effects

These data provide support for the VRFCAT as a co-primary outcome measure of functional capacity assessment for use in schizophrenia trials.

Conclusions


• Need to develop non-English and non-US English versions

• Determine the capacity of the VRFCAT to demonstrate treatment sensitivity

• Use in other disorders (healthy aging, MCI, AD) under investigation

• Use as a predictor of illness or decline in functioning

Future Studies


Acknowledgments

We are indebted to the subjects who volunteered their time and energy to participation in this study

Funding provided by the National Institute of Mental Health Grant Number 1R43MH084240-01A2 and 2R44MH084240-02

Software development provided by Virtual Heroes, a division of Applied Research Associates, Inc.

validation of a computerized assessment of functional capacity · international congress on...

Documents