This research was supported by the United States Department of Homeland Security (DHS) through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) at the University of Southern California (USC) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE.

Analyzing the Return on Investment of Homeland Security Research

Detlof von Winterfeldt Director, CREATE

University of Southern California 1150 S. Olive Street, Suite 1700

Los Angeles, CA 90015 (213) 740-3863 create@usc.edu

May 15, 2019

ADSA20 Boston, MA

2

Carl Crawford asks: So what? Who cares?

• The Problem. The Office of University Programs at DHS has been challenged to show the return on investment of the research conducted by its Centers of Excellence (COEs)

• Who Cares? Congressional appropriations committees, OMB, DHS S&T leaders, OUP leaders, COE leaders, HS researchers

• The Solution. Conduct selected benefit-cost analyses of about 20 OUP-funded research products (TTKPs=Tools, Technologies, and Knowledge Products)

• The Results. – One research product (CT Datasets and ATR) produces over $100 million in

benefits at a cost of $5 million, approximate ROI = 2,000% – 10 other research products (across multiple COEs) produce about $400

million in additional benefits, ROI = 856% – Two research products from CREATE produced enough benefits to pay for

$45 million = 12 years of OUP funding of CREATE 2

3

Steps in the Benefit-Cost Analysis

• Selection of research products

• Baseline analysis

• Cost analysis

• Benefits analysis

• Sensitivity and uncertainty analysis

• Results: Net Benefits, BCR, and ROI

4

Application to 20 Research Products (Selected by USCG, TSA, and CBP)

1. The likelihood of a successful transition (transitioned and used = 1)

2. The beneficial impact on customer, if successfully transitioned

Note: Selections are not representative of all OUP-funded projects

5

Example: CT Datasets and Algorithms

• Improve detection technology at TSA checkpoints

• CT Datasets enable better detection algorithms

• No need for classified or sensitive data

• Reduced false (nuisance) alarms

6

Cost Calculations for CT Datasets

6

Cost Category Start EndAmount in

Real Dollars

Amount in

2017 DollarsSource

Pre-project costs (COE)

Pre-project costs (other funding)

Project costs (COE) 9/1/13 6/30/18 740,000$ 748,315$ COE/OUP

Project costs (university cost share)

Oversight cost at the COE 9/21/10 6/30/18 210,000$ 216,987$ COE/OUP

Oversight cost at OUP 9/21/10 6/30/18 160,000$ 165,324$ OUP Overhead

Transition development cost 9/21/10 2/27/15 3,555,000$ 3,768,300$ S&T, TSA

Implementation start up cost

Implementation cost (User)

Implementation cost (COE)

Implementation cost (Other users)

TOTAL COST 4,665,000$ 4,898,926$

7

The Benefits of CT Datasets

• Reduces false alarms

• Saves time of passengers

• Saves time of TSA officers

7

National Academy of Science: Each 1% reduction in false alarms saved $25 million

8

Benefit Calculations for CT Datasets

8

Input Variable BaseNumber of Travelers/Year 719,000,000

Current False Positive Rate 10.0%

Decrease in False Positives 25%

Time in Secondary (Min) 7.5

Cost of One Minute of Wait Time $0.79

Cost of One Minute of TSA Time $0.27

Percent attributable to CT Vendor 25%

Percent attributable to ATR 30%

Cost of a Successful Attack 25,000,000,000

Current Cost of False Positives/Year 571,605,000$

3D Cost of False Positives/Year 428,703,750$

Attributabe Benefits/Year 10,717,594$

Discount Factor 0.030

Total Value of 3D CT Scanners for 10 Years 107,175,938$

Total Net Benefits for Ten Years 102,277,012$

9

Costs vs. Benefits of CT Datasets

Cost (in 2017 $)

$4,898,926

Benefits (in 2017 $)

$107,175,938

BCR = 22 ROI = 2,087%

10

Summary of 10 Additional BCAs

10

Tool, Technology, or Knowledge Product

(TTKP)

Cost (in Thousands, in 2017 $)

Median Net Benefit (50th Percentile)

Years of Use for Net Benefit Calculations

Research Products with Past Applications PROTECT $710 $35,505 6 Past & 4 Future Years

ARMOR $1,057 $28,969 10 Years, Past Use

CgSARVA $803

$5,247 One Time (Sandy)

Research Products with Potential Future Applications ADCIRC $36,893 $256,411 10 Years, Future Use

Engineered Swabs $1,867 $22,528 10 Years, Future Use

GeoXray $273 $18,404 10 Years, Future Use

TraffiCop $1,413 $10,444 10 Years, Future Use

HOAX Calls $183 $4,646 10 Years, Future Use

BOARD $1,018 $2,435 10 Years, Future Use

E-CAT $942 $806 10 Years, Future Use

TOTAL $45,161 $386,395 ROI=856%

11

Conclusions

• Benefit-cost analyses can be successfully conducted on OUP-funded research products

• Uncertainty about future benefits • Nevertheless: Benefits for a dozen selected research

products exceed OUP funding by a wide margin • A few research products are the main contributors to

benefits (ADCIRC, IRIS, CT Datasets, Enhanced Millimeter Wave Scanner)

• Start considering success criteria and benefits and benefit-cost analysis early – at proposal stage

• Nine benefits criteria and models developed by CREATE help (e.g., reducing false alarms, increasing detection, reducing threat, informing decision-making)

12

Thank you! For more on CREATE, go to

create.usc.edu

13

Supplementary Slides

14

Sensitivity Analysis for CT Datasets

14

5.0%

5

25%

$0.50

20%

20%

$0.18

719,000,000

0.1

0.9

10,000,000,000

20.0%

10

40%

$1.00

30%

30%

$0.35

719,000,000

0.4

0.9

50,000,000,000

$20,000,000 $40,000,000 $60,000,000 $80,000,000 $100,000,000 $120,000,000 $140,000,000 $160,000,000 $180,000,000 $200,000,000 $220,000,000 $240,000,000

Current False Positive Rate

Time in Secondary (Min)

Percent attributable to ATR

Cost of One Minute of Wait Time

Decrease in False Positives

Percent attributable to CT Vendor

Cost of One Minute of TSA Time

Number of Travelers/Year

Number of Attack Attempts Per Year

Detection probability with current system

Cost of a Successful Attack

Net Benefits for 10 Years

15

Uncertainty Analysis for CT Datasets

15

• Using wide ranges for all uncertain inputs (min, mode, max) • Using triangular distributions for each input

$- $50,000,000 $100,000,000 $150,000,000 $200,000,000 $250,000,000 $300,000,000 $350,000,000 $400,000,000

0

500

1000

1500

2000

2500

Net Benefits for Ten Years

Freq

uen

cy

16

10 Additional Research Products •ADCIRC - ADvanced CIRCulation Next generation modeling suite to predict coastal flooding

CHC - Coastal Hazards Center of Excellence

•ARMOR - Assistant for Randomized Monitoring Over Routes Software that randomizes schedules, plans, or actions for security agencies CREATE – Center for Risk and Economic Analysis of Terrorism Events

•BOARD - Bus Operator Awareness and Research Development Training program for bus operators to respond to security threats NTSCOE – National Transportation Security Center of Excellence

•CgSARVA – Coast Guard Search and Rescue Visual Analytics Interactive analysis of trends, patterns, anomalies, and distribution of

Search and Rescue cases VACCINE – Visual Analytics for Command, Control and Interoperability Environments

• E-CAT – Economic Consequence Analysis Tool Provides rapid estimates of economic impact of threats

CREATE – Center for Risk and Economic Analysis of Terrorism Events 16

17

• Engineered Detection Swab Swabs to improve detection of trace explosives on luggage and persons

ALERT - Awareness and Localization of Explosives-Related Threats

• GeoXray Tool to display geographic area and overlay relevant emergency response

CCICADA – Command, Control and Interoperability Center for Data Analysis

• HOAX Calls State-of-the-art tool for identifying hoax emergency calls

CCICADA – Command, Control and Interoperability Center for Data Analysis

• PROTECT – Port Resilience Operational/Tactical Enforcement to Combat Terrorism

Software to randomize schedules in a port environment CREATE – Center for Risk and Economic Analysis of Terrorism Events

• TraffiCop Analysis of Twitter and social media to detect human trafficking

CCICADA – Command, Control and Interoperability Center for Data Analysis

18

ADCIRC - ADvanced CIRCulation Storm surge/inundation modeling

COE: CHC - Coastal Hazards Center of Excellence

• Storm surge model that combines rain, atmospheric pressure, and wind forecasts

• Predicts when, where, and

to what extent flooding will inundate a coastal community

Median Benefit (50th Percentile)

$293,804,000

Cost (in 2017 $)

$36,893,000

18

19

E-CAT – Economic Consequence Analysis Tool Rapid estimates of the economic impact of threats

COE: CREATE- Center for Risk and Economic Analysis of Terrorism Events

Median Benefit (50th Percentile)

$1,746,000

Cost (in 2017 $)

$942,000 • Provides quick estimates of the economic impact of terrorist attacks, natural disasters, and technological accidents

20

Summary of Costs and Benefits

20

Tool, Technology, or Knowledge Product

(TTKP)

Cost (in Thousands, in 2017 $)

Median Net Benefit (50th Percentile)

Years of Use for Net Benefit Calculations

Research Products with Past Applications PROTECT $710 $35,505 6 Past & 4 Future Years

ARMOR $1,057 $28,969 10 Years, Past Use

CgSARVA $803

$5,247 One Time (Sandy)

Research Products with Potential Future Applications ADCIRC $36,893 $256,411 10 Years, Future Use

Engineered Swabs $1,867 $22,528 10 Years, Future Use

GeoXray $273 $18,404 10 Years, Future Use

TraffiCop $1,413 $10,444 10 Years, Future Use

HOAX Calls $183 $4,646 10 Years, Future Use

BOARD $1,018 $2,435 10 Years, Future Use

E-CAT $942 $806 10 Years, Future Use

TOTAL $45,161 $386,395 ROI=856%

21

Nine Benefits Criteria and Models • Cost savings with the same risk level

– Savings in operational costs (ARMOR, E-CAT)

– Savings by stretching expenses (cgSARVA)

• Increased security at same cost

– Reduction of threat (HOAX Calls)

– Reduction of vulnerability (PROTECT)

– Reduction of negative consequences (TraffiCop)

• Improved signal detection

– Reduction of false alarm rates (Engineered Swabs)

– Improvement of detection rates (3D CT Scans)

• Value of information

– Reduction of uncertainty (GeoXray)

– Improvement of decision quality (ADCIRC)

21