Lars Kurth Community Manger, Xen Project

Chairman, Xen Project Advisory Board

Lead CentOS Virtualization SIG

Director, Open Source Business Office, Citrix lars_kurth

Was a contributor to various projects

Worked in parallel computing, tools, mobile and now virtualization

Community guy at Symbian FoundationLearned how NOT to do stuff

Community guy for the Xen ProjectWorking for CitrixMember of OSS Business OfficeAccountable to Xen Project Advisory BoardChairman of Xen Project Advisory Board

Users Safe

Think of your vulnerability process as a team-effort to ensure that …

• All doors are closed

• All doors are locked

• All windows are boarded up

• Fences have no weaknesses

• …

Encourage discoverers to report security issues to security@yourproject

Discoverers are in control You can’t stop them from releasing/using informationA robust vulnerability process encourages discoverers to work with you

Ensure that your project fixes security issuesas quickly as possible

You don’t want unaddressed vulnerabilities

Exposure time to security issues is minimizedA maximum of users apply patches quickly

Minimize risk

ResponsibleF A X

FullF XA

I: Vulnerability Introduced

D: Vulnerability Discovered

R: Vulnerability Reported

A: Vulnerability Announced

F: Fix Available

X: Fix Deployed

Vulnerability exists: we do not know whether it is exploited in the wild

Vulnerability is known about by a privileged and small group of users

Vulnerability is known publicly

R

R

I D

Linux Kernel/LXC/KVM if reported via OSS Security

Linux Kernel/LXC/KVM if reported via security@kernel.org

OpenStack for low impact issues

Full

Linux Kernel/LXC/KVM if reported via OSS Security Distros

Linux Distributions (both open source and commercial)

QEMU, Libvirt, oVirt, ...

Responsible

“Distro Model”

OpenStack for intermediate to high impact issues

OPNFV, OpenDayLight : process modeled on OpenStack

Xen Project for all issues (also handles 3rd party issues, e.g. QEMU)

Responsible

“Cloud Model”

Not clearly

stated

Docker : states responsible disclosure; but policy docs / CVE pages are empty

Cloud Foundry : no clearly stated process; no published CVE’s

CoreOS: just a mail to report issues

Kubernetes: no information (or could not find it, which is an issue in itself)

Approach Used by Projects

Open-source software projects are often well intended, but security can take a back seat to making the code work. OpenDaylight, the multivendor software-defined networking (SDN) project, learned that the hard way last August after a critical vulnerability was found in its platform. It took until December for the flaw, called Netdump, to get patched …

PC World, March 2015

Source: yanilavigne.net viadomics.me

Wide range of approachesNo consistent Best Practice across projectsNewer projects are lagging behind

Using the pre-dominant model as baselineApplies to Linux Distros, OSS Sec Distros, QEMU, …

Mike Licht @ Flickr

A X

Typically fixed time during which the security issue is handled secretly

Depends on discoverer’s wishes

R: Vulnerability Reported

T: Triage

P: Vulnerability Pre-disclosed

A: Vulnerability Announced

F: Fix Available

X: Fix Deployed

Vulnerability is known by the reporter and the security team

Note: It may also be known and used by black hats

Vulnerability is known about by a privileged and small group of users

Vulnerability is known publicly

Description, CVE

allocation, …

Pre-disclosure period

What can and can’t be done

with privileged information

can differ significantly between

projects

R

Patch/fix creation

and validation

FT P

Micolo J @ Flickr

mindfulness @ Flickr

F A XR

Disclosure Time

I personally don't like embargoes. I don't think they work. That means that I want to fix things asap.

Linus Torvalds, 2008

People are less willing sometimes to brush the problem [of fixing security issues] under the mat, and leave it up to vendors that have disclosures, like infinitely long disclosure times.

Linus Torvalds, 2015

Long disclosure times discredit responsible disclosureFrom a few days to many months (recent example: Apple)

Long disclosure times create a disincentive for reporters to work with youIncreases the risk of 0 day exploits

Pre-defined disclosure times help manage vendorsExample later

Most successful projects have a 2-3 weeks disclosure period

The capability to fix issues within the recommended time

Larger and distributed projects can struggle to fix all issues in time

The capability to handle the entire process

in secret

Assigning CVE numbers is best practice in by

established projects and vendors in the

Linux/Cloud ecosystem

CVE databases (such as www.cvedetails.com) can be used to evaluate your project

This shows Xen Project CVE stats

Before 2012, we didn’t have fewer vulnerabilities than after

We just didn’t have a process requiring creation of CVEs

A fair comparison between projects/technologies using CVE data is not easily possible

Not all projects/products create CVEs for all their issues Example: Linux/QEMU only do so for severe onesPolicies are not always published

Some projects don’t assign CVEs at all

Some technologies/products cannot be easily identified in databases Example: KVM, LXC

Sometimes CVEs can affect several productsBut are counted only against oneOpen source product definitions on cvedetails are often sloppy

Mike Licht @ Flickr

Description, CVE

allocation, …

A D

Pre-disclosure period

R

Patch/fix creation

and validation

FT P

What happens here depends

on your process goals

Make sure that a fix is available before disclosure

Make sure that downstream projects and products (e.g. distros) can package and test the fix in their environment

Allow service providers that use your Software to start planning an upgrade (at scale this can take a week)

Allow service providers that use your Software to deploy an upgrade before the embargo completes

What is allowed during pre-disclosure

Who is privileged and trusted to be on the pre-disclosure mailing list

Disclosure Time

Make sure that a fix is available before disclosure

Make sure that downstream projects and products (e.g. distros) can package and test the fix in their environment

Allow service providers that use your Software to start planning an upgrade (at scale this can take a week)

Allow service providers that use your Software to deploy an upgrade before the embargo completesCloud Model

Distro Model

Emerged recently!

Recognizes the needs of service providers

Pre-Cloud Computing!

Services and their users are vulnerable

during pre-disclosure period

More Cloud/Service users than direct users of your software

Example:

AWS stated in 2014 that they have > 1M users (and a lot more instances)AliCloud claims that they have > 1M users…

Just imagine what the reputation damage would have been, if Xen had put AWS, Rackspace, SoftLayer, … users at real risk of a vulnerability.

There were 100’s of

stories at the time,

despite the fact that

users were never put

at risk, but merely

inconvenienced !

Pre-disclosure list membership: more members, more risk of leakage

In the Distro Model, the number of privileged users is typically <10In the Cloud Model, the number could be an order of magnitude higher (50-100)This increases risk of information being accidentally released

Restricting pre-disclosure list membership

Restricting membership to large service providers to minimize riskThat creates issues of “fairness” Which may be incompatible with your communities' values

How the Xen Project got to its Vulnerability Process

xenproject.org/security-policy.html

Moyan Brenn @ Flickr

2011 2012 2013 2014 2015 2016

Goals: Allow fixing, packaging and testing; Allow service providers to prepare (but not deploy) during embargo

Pre-disclosure: Membership biased towards distros & large service providersNo predefined disclosure time

1.0

2011 2012 2013 2014 2015 2016

July 2012: CVE-2012-0217, Intel SYSRETAffected FreeBSD, NetBSD, Solaris, Xen and Microsoft Windows

A large pre-disclosure list member put pressure on

key members of the Xen Project Community to get an embargo

extension

They eventually convinced the discoverer to request an extension

1.0

2011 2012 2013 2014 2015 2016

Centered on:

Predetermined disclosure schedule: 1 week to fix, 2 weeks embargo

Who should be allowed on the pre-disclosure listFairness issues between small and large service providersDirect vs. indirect Xen consumersThe risk of larger pre-disclosure list membership

1.0

2011 2012 2013 2014 2015 2016

Strongly recommended disclosure scheduleInclusive pre-disclosure list membership Changes to application procedure (based on checkable criteria)

1.0 2.0

2011 2012 2013 2014 2015 2016

Sept 2014: CVE-2014-7118

Leading to the first Cloud Reboot

AWS pre-announced cloud reboot to their customers

Other vendors didn’t.

Policy was interpreted differently by vendors.

This highlighted ambiguities in the project’s security policy

(what can/can’t be said/done during an embargo)

1.0 2.0

2011 2012 2013 2014 2015 2016

Goals: Allow fixing, packaging and testingAllow service providers to prepare (and normally to deploy) during embargo

Pre-disclosure: Clearer application criteriaPublic application process (transparency) Clear information on what is/is not allowed during an embargo (per XSA)Means for pre-disclosure list members to collaborate

1.0 2.0 3.0

2011 2012 2013 2014 2015 2016

Conducted XSA-133 Retrospective upon requestProcess change: Earlier embargoed pre-disclosure without patches

May 2015: CVE-2015-3456

First time we were affected by a branded bug

QEMU bug, which was handled by several security teams: QEMU,

OSS Distro Security, Oracle Security & Xen Project

From a process perspective: were not able to provide a

fix 2 weeks before the embargo date ended

1.0 2.0 3.0

Larger pre-disclosure list has not caused a single issues in two years of operating an inclusive approach

We have not had a single 0-day vulnerability

A well run vulnerability process builds trust

Willingness to adapt to your stake-holders needs builds more trust

It creates collaboration and understanding of stake-holders

Fairness is a difficult issue

There will always be practical issues, e.g. “interpretations of policy”, etc.

The Xen Project’s process is the only example case, where this issue has been tackled through a community consultation.

To Contrast:

OpenStack does not publish who is on their pre-disclosure list

OpenStack does not have a formal application process

Avoids dealing with the “fairness” issue head-on

Security stories are “hot”

Xen is widely used, thus security stories “sell”

It’s too easy for reporters to write a story

Reporters just have to check our page,

and know when the next story comes

Source: yanilavigne.net viadomics.me

Very wide range of approaches vs.The reality that SW stacks contain many layers Consider the weakest link in your SW stack

Best Practice appears to be emergingOlder projects seem slow to changeNew projects, don’t build security management into their culture from the beginning

New Post-Snowden era pressuresHow to effectively deal with media Hype?