1

An Empirical Analysis of Software Vendors’ Patching Behavior

Rahul Telang

WithAshish Arora, Ramayya Krishnan and Yubao Yang

Carnegie Mellon University

2

“Developing and deploying patches is an increasingly important part of the software development process” –

Joseph Dadzie, MICROSOFT

3

Motivation

• Software vendors’ incentives to invest in product quality, product maintenance has been a keen area of interest (Krishnan et al 2000, Slaughter et al 2000)

• Many believe that vendors provide suboptimal levels of quality and security and applying product liability rules to software is being debated.

• One key aspect of better and more secure software is timely and reliable patching by vendors.

• The quality of a software product critically depends on the ex-post support a vendor provides (Arora, Caulkins and Telang 2005).

• In this paper, we derive empirical estimates for the drivers of vendor’s patching behavior.

4

Motivation• In particular, our focus in on the patching of security

related vulnerabilities by vendors

– The number of such vulnerabilities found and reported has exploded in last few years (CERT/CC published more than 3000 vulnerabilities in 2003 alone)

– Many such vulnerabilities are particularly costly to customers. Thus patching is of paramount interest to customers.

– Unlike many other measures of security, patching speed can be measured more reliably.

5

Motivation• Unique feature of software vulnerabilities is that anyone can find

them (randomly or by exerting effort) and can make the information regarding this flaw public. This can exert significant negative externalities on all users using that software.

• Many such discoverers do not trust the vendors to provide timely and reliable patches. In many instances, they resort to disclosing vulnerability information in public forums in the hope of pressuring vendors.

• There is no consensus on “what is the optimal time”. Vendors want more time to be given and have taken the users to court for disclosing the vulnerabilities (Cisco, 2005)

• There is little empirical work to quantify vendors’ incentives to patch.

6

Motivation

• "The truth is that unpatched Windows flaws have a value to the underground community, and it is not at all uncommon to see these things sold or traded among certain groups who use them by quietly attacking just a few key targets. So, the longer Microsoft takes to patch vulnerabilities the longer they are leaving customers exposed” –Marc Maiffret, eEye

7

Literature

• How process improvements lead to better quality and lower maintenance costs (Krishnan et al 1999; Slaughter et al 1998).

• Arora, Caulkins, Telang (2003) highlight the incentives of vendors to “Sell First and Fix Later”.

• Telang and Wattal (2004) show that disclosure is costly to vendors and hence provides incentives to vendors to improve the quality of their software

• Disclosure of vulnerabilities and how it affects patching behaviors– Arora, Telang, and Xu (2004) outline a model for how long one

should wait before disclosing. (Cavusoglu et al 2004)

• Nizovtsev and Thursby (2005) model the incentives of individuals to disclose software vulnerabilities through an open public forum.

• August and Tunca (2005) analyzes the patching behavior of users and how it affects vendors’ incentives to improve network security.

8

• Security vulnerabilities are costly to vendors and customers.

(i) Vendors incur cost of patching. One would expect that more time they have for patching less it costs. In short, they would like to delay the patch as much as possible.

(ii) Vendors’ customer incur loss if the vulnerabilities are exploited when they are breached. Vendor cares about customer loss (reputation loss, sometimes contractual obligations). How much they care depends on the market structure.

• Vendors trade-off these costs.

• Vendor characteristics, vulnerability characteristics, and disclosure affects these costs and hence patching behavior.

• The factors that increase “customer loss” should force vendor to accelerate patching. Factors that increase “patching” cost should slow patching.

Vendors’ Incentives to Patch

9

• Larger vendors patch faster.

– Larger customer base and worry of reputation loss that may spillover to other products. They may have less average patching cost.

• Severe vulnerabilities are patched faster.

– Severe vulnerabilities increase customer loss .

• Publicly traded firms patch faster.

– Public firms may experience the consequences of a fall in reputation more directly through reduction in share price (see Telang and Wattal 2004).

Predictions of Analytical Model

10

• Open Source Vendors – Large debate in academic and trade publications on comparisons between open source and closed source vendors. Patching is also part of the “quality”.

• Finally, disclosing the vulnerabilities would increases customer loss and should force vendor to patch faster.

• The policy makers have struggled with “how long should vendors be given to patch”. Our paper can provide key empirical estimate on how changes in disclosure affect patching time. This “quantification” is necessary to understand both vendors response to disclosure and hence the efficacy of disclosure as a tool.

Predictions of Analytical Model

11

Data source

• Vulnerabilities are published by many sources. We focus on CERT/CC and Securityfocus (SF). In particular, we focus on the vulnerabilities that are published by both sources to reduce any inherent bias.

• CERT/CC (Center for Emergency Response/ Coordination Center) is a federally funded organization which disseminate vulnerability information. It researches the vulnerability when a user notifies it, and then contacts the vendor and provide it with “protected period” before making it public (A typical protected period is 45 days).

• Securityfocus (an online open forum) has less quality control over the vulnerabilities published on its forum. It also does not provide any “protected period” unless individuals choose to do so. In short, individuals can post the vulnerability information even without patches.

• Fortunately, there are exception to these policies on both CERT and SF which allows us the variation in our independent variables.

12

Data• Vulnerabilities published by SF and/or CERT/CC from 9/26/2000 to

8/11/2003. – Vendor Notification date (CERT provided us this data and from SF

website)– Patching date (from vendor websites and from CERT and SF).

• Vendor information – from Hoover’s online and vendor’s website

• Vulnerability characteristics – from the national vulnerability database (NVD, previously ICAT

database)

• After cleaning up the data we have 1280 observations, related to 255 unique vendors and 303 unique vulnerabilities.

13

Example of vulnerability publication by CERT

14

Example of vulnerability publication by SF

15

Disclosure• Once the vendor is notified of vulnerability, it is possible

that information get disclosed before the patch is out.

• Many times notification and disclosure happens at same time.

• First vendor to patch may disclose the information, CERT may disclose it (sometimes before protected period), SF always discloses it, any third party may disclose it.

• In short, disclosure is “time-varying”.

• In fact, for the same vulnerability different vendors may face different disclosure time.

notification disclosure patch

16

Descriptive statistics

Patching Time (in days)

Mean 56.5 (114.8)

Median 19

% patched 90%

Disclosure Time (in days) *

Mean 15.6 ( 43.0)

% instant disclosure 67.1%

Number of observation facing disclosure 985

* Elapsed time between Vendor Notification and Disclosure (in days)

17

Descriptive statistics

Mean Std. Dev.

Vendor employee size (in 000’s) 17.60 66.12

Public firm 0.34 0.47

Open source 0.23 0.42

Vendor characteristics (N=142)*

* Include only the vendors for which the reliable information is available

Vulnerability severity metric*

Mean Std. Dev. Min Max

Vulnerability severity metric 14.82 16.48 0 108.16

Number of affected vendors/vulnerability

8.23 21.53 1 242

* Vulnerability severity measurement by CERT/CC

18

Compare between CERT and SF

Published byCERT/CC and/or SF

Published bySecurityFocus alone

For All Observations

% patched 93.21% 62.66%

Disclosure Time 15.49 (38.06) 16.15 (72.20)

Severity Metric 25.43 (22.06) 8.65 (4.40)

Obs / vuls 1311 / 349 158 /89

For Patched Observations

Patching time 55.38 (114.09) 70.88 (122.52)

Standard errors are in parenthesis

20

Kaplan-Meier survival estimates0

.00

0.2

50

.50

0.7

51

.00

0 100 200 300 400analysis time

disclosure = 0 disclosure = 1

Kaplan-Meier survival estimates, by disclosure

21

Empirical Strategy

• OLS is inappropriate.

• Proportional hazard model (PHM), assuming the baseline hazard has the form of Weibull distribution (Cox leads to similar estimates)

• In PHM, covariates shift the hazard function proportionally (needs to test for it).

• Key covariates – vendor size, severity, Open source/closed source, publicly traded firm, CERT dummy and disclosure.

• Disclosure is a time varying covariate– For a vulnerability i and vendor j, if disclosure happens at some

time t1 such that t0 < t1 < t then, disclosure = 0 before t1 and disclosure = 1 after t1

22

Clustering and Heterogeneity

• We control for the unobserved heterogeneity across vendors.

• Same vulnerabilities affect multiple vendors. We control for such clustering.

• We run what is known as “frailty” models (more or less these are like “random-effect” models in regression)

23

Estimates (Hazard Ratio) Weibull Hazard Model (N=1469)

(1) With frailty (2) Without frailty

Haz. ratio Std. Error Haz. ratio Std. Error

CERT_effect 3.04*** 0.35 3.10*** 0.33

Disclosure 2.37*** 0.20 2.19*** 0.17

Small 0.84 0.51 0.80** 0.09

Vendor size 1.01 0.03 1.00 0.01

Public firm 1.32 0.26 1.28*** 0.11

Open source software 1.71*** 0.30 1.61*** 0.14

Severity metric (log) 1.24*** 0.03 1.20*** 0.03

Post 9_11 2.08*** 0.14 2.05*** 0.13

ln_p (Weibull shape parameter) -0.58*** 0.02 -0.65*** 0.02

σ (frailty parameter) 0.32 0.10

Log Likelihood -2978 -3014

• 10% level, ** 5% level and *** 1% level. • The constant for the proportional hazard model is not identified when hazard ratio is estimated.

24

Interpretation

• Disclosure increases vendors’ patching speed by 137%.

• Open source vendors patch 70% faster than closed source vendors.

• Severe vulnerabilities are patched faster.

• Vendors respond more favorably to CERT than SF. (almost 200% faster). In short vendors do not care as much if an individual or SF informs them about a vulnerability. Credibility of “who” is informing them matters.

• Small vendors do not patch as fast.

25

Estimated hazard ratio with and without disclosure

26

The Estimated Effect of Disclosure

Disclosure time T Expected patching time (in days) Effect of disclosure(in days)disclosed at time T without disclosure

0 (Instant disclosure) 33.03 61.77 28.74

1 36.90 61.77 24.87

2 38.60 61.77 23.17

3 39.98 61.77 21.79

4 41.16 61.77 20.61

5 42.21 61.77 19.56

6 43.18 61.77 18.59

7 44.07 61.77 17.70

8 44.90 61.77 16.87

9 45.69 61.77 16.08

10 46.43 61.77 15.34These calculations are done setting the covariates at their average sample values namely that the vulnerable vendor is a public firm and has the average employee size of our sample; the vulnerability is published after 9/11 event, handled by CERT/CC and has the average severity metric of our sample.

27

Does it matter who is disclosing?

• Apparently it does. Vendors care about the “source” of the disclosure just like they care about “who” informs them of vulnerabilities.

29

External Validity

• Last month, we found another data source which is similar to ours.

• Brian Krebs of Washington Post is was also interested in finding out whether disclosure affects the patching time. He collected data from Microsoft between 2003-2005. He collected information on the time when Microsoft was informed and the time when Microsoft released patch.

• For some of the vulnerabilities, instant disclosure happened.• Almost all of these vulnerabilities were then reported by

Microsoft to CERT and SF and posted on their pages along with a patch.

• We can use this data to verify what the impact of disclosure is.

30

Summary statistics

• N = 99• No disclosure N = 83, mean patching time = 130 (80)

days, average severity score = 7.06• Disclosure N = 16, mean patching time = 62 (58)

days, average severity score = 6.4.

parameter estimate

disclosure 3.71** (1.17) severity 1.69* (.50)

Log Likelihood -352.

31

Results

• Very strong effect of disclosure. In fact, more pronounced than in our data set.

• Suggests that disclosure is a very important determinant of vendor patching behavior and confirmed by two independent data sources.

32

Robustness

• Our results are not sensitive to the assumption of the Weibull distribution since the Cox non parametric specification yields similar results

• Controlling for the source of disclosure, and for unobserved heterogeneity in vulnerabilities also does not qualitatively affect the results.

• Controlling for vulnerability specific random effect (rather than vendor specific random effects) doesn’t change the results.

• The impact of disclosure and CERT/CC remains qualitatively unchanged with including only patched observations or dropping patching time of more than 1 year.

33

Proportionality assumption

• Schoenfeld residuals Proportionality assumption cannot reject the proportionality hypothesis for the disclosure effect

• But test results show that with time the effect of CERT/CC on patching time changes

• We interact the CERT dummy with time to correct for lack of proportionality

• Proportionality test after including the interaction term between the CERT dummy and time corrects for proportionality assumption.

34

Conclusion

• One of the first papers to examine vendors patching behavior.

• Disclosure forces the vendors to patch faster – increases vendors’ patch delivery speed by almost 137%.

– instant disclosure force a vendor to release the patch 29 days earlier. This does not mean this is the right thing to do. But we have some quantifiable measure of vendor response.

• Our model is still reduced form. A structural model may identify the effects separately (i.e. customer loss and patching cost).

• Do not control for the quality of the patch. Disclosure could be endogenous? [Though we control for severity and vendor size]. Might need to find an instrument.