scale14x: are today's foss security practices robust enough in the cloud era final
TRANSCRIPT
![Page 1: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/1.jpg)
Are Today's FOSS Security Practices Robust Enough in the Cloud Era?
Lars Kurth Community Manger, Xen ProjectChairman, Xen Project Advisory BoardDirector, Open Source/Xen Project, Citrix
lars_kurth
![Page 2: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/2.jpg)
About MeWas a contributor to various projectsWorked in parallel computing, tools, mobile and now virtualizationCommunity guy for the Xen ProjectWorking for CitrixMember of the group that develops XenServerChairman of Xen Project Advisory Board
![Page 3: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/3.jpg)
Software bugs happenSome will be security
vulnerabilities
![Page 4: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/4.jpg)
I
I: Vulnerability IntroducedD: Vulnerability Discovered
D
Discoverer exploits
issue for his own purpose
![Page 5: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/5.jpg)
I
I: Vulnerability IntroducedD: Vulnerability Discovered
Discoverer reports security issues to security@yourproject
D
![Page 6: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/6.jpg)
A team-effort to ensure that …•All (known) doors are closed•All (known) doors are locked•All (known) windows are
boarded up•Fences have no (known)
weaknesses•…
Vulnerability ManagementProcess =
![Page 7: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/7.jpg)
XF
Pattern: Full Disclosure
R: Vulnerability ReportedT: TriageA: Vulnerability AnnouncedF: Fix Available X: Fix Deployed
Vulnerability is known by the reporter and the security teamNote: It may also be known and used by black hats Vulnerability is known publicly with no fix availableVulnerability is known publicly with fix available
BasicDescription
R T A
Patch/fix creation and validation
![Page 8: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/8.jpg)
Pattern: Responsible Disclosure
X
R: Vulnerability ReportedT: TriageP: Vulnerability Pre-disclosedA: Vulnerability AnnouncedF: Fix Available X: Fix Deployed
Vulnerability is known by the reporter and the security teamNote: It may also be known and used by black hats Vulnerability is known about by a privileged and small group of users Vulnerability is known publicly
APre-disclosure period
R PPatch/fix creation and validation
FT
![Page 9: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/9.jpg)
Purpose of Vulnerability Process = Keeping Your Users Safe
![Page 10: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/10.jpg)
What does safety mean?
Encourage discoverers to report security issues to security@yourprojectDiscoverers are in control You can’t stop them from releasing/using informationA robust vulnerability process encourages discoverers to work with you
![Page 11: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/11.jpg)
What does safety mean?
Ensure that your project fixes security issues as quickly as possibleYou don’t want unaddressed vulnerabilities
![Page 12: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/12.jpg)
What does safety mean?
Exposure time to security issues is minimized A maximum of users* apply patches quickly Minimize risk
![Page 13: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/13.jpg)
Linux Kernel/LXC/KVM if reported via OSS SecurityLinux Kernel/LXC/KVM if reported via [email protected], QEMU, … for low impact issues
Full
Linux Kernel/LXC/KVM if reported via OSS Security DistrosLinux Distributions (both open source and commercial)QEMU, Libvirt, oVirt, ...OpenStack for intermediate to high impact issuesOPNFV, OpenDayLight : process modeled on OpenStackXen Project for all issues (also handles 3rd party issues, e.g. QEMU)Docker : states responsible disclosure; but policy docs empty / some CVEs
Responsible
Cloud Foundry : no clearly stated process; no published CVE’sCoreOS: just a mail to report issuesKubernetes: : just a mail to report issues (when I wrote this talk in Aug, no info)
Not clearly stated
Approaches used by Cloud ProjectsApproach Used by Projects
![Page 14: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/14.jpg)
Example: OpenDayLight & Netdump
Open-source software projects are often well intended, but security can take a back seat to making the code work. OpenDaylight, the multivendor software-defined networking (SDN) project, learned that the hard way last August after a critical vulnerability was found in its platform. It took until December for the flaw, called Netdump, to get patched …
PC World, March 2015
![Page 15: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/15.jpg)
Anatomy of ResponsibleDisclosure In DetailUsing the pre-dominant model as baselineApplies to Linux Distros, OSS Sec Distros, QEMU, …
Mike Licht @ Flickr
![Page 16: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/16.jpg)
Responsible Disclosure In Detail
A X
Typically fixed time during which the security issue is handled secretly Depends on discoverer’s wishes
R: Vulnerability ReportedT: TriageP: Vulnerability Pre-disclosedA: Vulnerability AnnouncedF: Fix Available X: Fix Deployed
Vulnerability is known by the reporter and the security teamNote: It may also be known and used by black hats Vulnerability is known about by a privileged and small group of users Vulnerability is known publicly
Description, CVEallocation, …
Pre-disclosure period
RPatch/fix creation and validation
FT P
What can and can’t be done with privileged information can differ significantly between projects
![Page 17: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/17.jpg)
Small differences, canhave large consequences
mindfulness @ Flickr
![Page 18: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/18.jpg)
Example: Disclosure Time
F A XR
Disclosure Time
![Page 19: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/19.jpg)
Long disclosure times discredit responsible disclosureFrom a few days to many months
Long disclosure times create a disincentive for reporters to work with youIncreases the risk of 0 day exploits
Pre-defined disclosure times help manage vendorsExample later
Most successful projects have a 2-3 weeks disclosure period
![Page 20: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/20.jpg)
Example:To CVE or not to CVE?Assigning CVE numbers is best practice in by established projects and vendors in the Linux/Cloud ecosystem
![Page 21: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/21.jpg)
CVE databases (such as www.cvedetails.com) can be used to evaluate your project
This shows Xen Project CVE statsBefore 2012, we didn’t have fewer vulnerabilities than afterWe just didn’t have a process requiring creation of CVEs
![Page 22: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/22.jpg)
A fair comparison between projects/technologies using CVE data is not easily possibleNot all projects/products create CVEs for all their issues Example: Linux/QEMU only do so for severe onesPolicies are not always published
Some projects don’t assign CVEs at allSome technologies/products cannot be easily identified in databases Example: KVM, LXC
Sometimes CVEs can affect several productsBut are counted only against oneOpen source product definitions on cvedetails are often sloppy
![Page 23: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/23.jpg)
How do Projects that use Responsible Disclosure differ?
Mike Licht @ Flickr
![Page 24: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/24.jpg)
Description, CVEallocation, …
Revisit: Responsible Disclosure
A DPre-disclosure period
RPatch/fix creation and validation
FT P
What happens here dependson your process goals
![Page 25: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/25.jpg)
Common Process GoalsMake sure that a fix is available before disclosureMake sure that downstream projects and products (e.g. distros) can package and test the fix in their environment
Allow service providers that use your Software to start planning an upgrade (at scale this can take a week)Allow service providers that use your Software to deploy an upgrade before the embargo completes
![Page 26: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/26.jpg)
Your Process Goals Determine …What is allowed during pre-disclosureWho is privileged and trusted to be on the pre-disclosure mailing list
Disclosure Time
![Page 27: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/27.jpg)
Make sure that a fix is available before disclosureMake sure that downstream projects and products (e.g. distros) can package and test the fix in their environment
Allow service providers that use your Software to start planning an upgrade (at scale this can take a week)Allow service providers that use your Software to deploy an upgrade before the embargo completesCloud Model
Distro Model
![Page 28: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/28.jpg)
Emerged recently! Recognizes the needs of service providers
Pre-Cloud Computing! Services and their users are vulnerableimmediately after disclosure
![Page 29: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/29.jpg)
Responsible Disclosure by ProjectsApproach Used by Projects
Linux Kernel/LXC/KVM if reported via OSS Security DistrosLinux Distributions (both open source and commercial)QEMU, Libvirt, oVirt, ...
OpenStack for intermediate to high impact issuesOPNFV, OpenDayLight : process modeled on OpenStackXen Project for all issues (also handles 3rd party issues, e.g. QEMU)Docker: depends on severity, details only available on request
![Page 30: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/30.jpg)
Trade-Offs:Distro vs. Cloud Model
![Page 31: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/31.jpg)
Risk for usersMore Cloud/Service users than direct users of your softwareExample: AWS stated in 2014 that they have > 1M users (and a lot more instances)AliCloud claims that they have > 1M users…
![Page 32: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/32.jpg)
Risk to your Project’s Reputation
Just imagine what the reputation damage would have been, if Xen had put AWS, Rackspace, SoftLayer, … users at real risk of a vulnerability.
There were 100’s ofstories at the time,despite the fact thatusers were never putat risk, but merely inconvenienced !
![Page 33: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/33.jpg)
Risk of Information LeakagePre-disclosure list membership: more members, more risk of leakageIn the Distro Model, the number of privileged users is typically <10In the Cloud Model, the number could be an order of magnitude higher (50-100)This increases risk of information being accidentally released
![Page 34: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/34.jpg)
FairnessRestricting pre-disclosure list membershipRestricting membership to large service providers to minimize riskThat creates issues of “fairness” Which may be incompatible with your communities' values
![Page 35: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/35.jpg)
War Stories:How the Xen Project got to its Vulnerability Processxenproject.org/security-policy.html
Moyan Brenn @ Flickr
![Page 36: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/36.jpg)
2011 2012 2013 2014 2015 2016
V1.0 : Modelled on DebianGoals: Allow fixing, packaging and testing; Allow service providers to prepare (but not deploy) during embargoPre-disclosure: Membership biased towards distros & large service providersNo predefined disclosure time
1.0
![Page 37: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/37.jpg)
2011 2012 2013 2014 2015 2016
July 2012: CVE-2012-0217, Intel SYSRETAffected FreeBSD, NetBSD, Solaris, Xen and Microsoft Windows
A large pre-disclosure list member put pressure on key members of the Xen Project Community to get an embargo extension They eventually convinced the discoverer to request an extension
1.0
![Page 38: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/38.jpg)
2011 2012 2013 2014 2015 2016
Community Consultation to improve our processCentered on: Predetermined disclosure schedule: 1 week to fix, 2 weeks embargoWho should be allowed on the pre-disclosure listFairness issues between small and large service providersDirect vs. indirect Xen consumersThe risk of larger pre-disclosure list membership
1.0
![Page 39: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/39.jpg)
2011 2012 2013 2014 2015 2016
V2.0 : Clarifications Strongly recommended disclosure scheduleInclusive pre-disclosure list membership Changes to application procedure (based on checkable criteria)
1.0 2.0
![Page 40: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/40.jpg)
2011 2012 2013 2014 2015 2016
Sept 2014: CVE-2014-7118 Leading to the first Cloud Reboot
AWS pre-announced cloud reboot to their customersOther vendors didn’t. Policy was interpreted differently by vendors.This highlighted ambiguities in the project’s security policy(what can/can’t be said/done during an embargo)
1.0 2.0
![Page 41: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/41.jpg)
2011 2012 2013 2014 2015 2016
V3.0 : Deploy & OptimizationsGoals: Allow fixing, packaging and testingAllow service providers to prepare (and normally to deploy) during embargo
Pre-disclosure: Clearer application criteriaPublic application process (transparency) Clear information on what is/is not allowed during an embargo (per XSA)Means for pre-disclosure list members to collaborate
1.0 2.0 3.0
![Page 42: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/42.jpg)
2011 2012 2013 2014 2015 2016
Conducted XSA-133 Retrospective upon requestProcess change: Earlier embargoed pre-disclosure without patches
May 2015: CVE-2015-3456First time we were affected by a branded bug
QEMU bug, which was handled by several security teams: QEMU,OSS Distro Security, Oracle Security & Xen ProjectFrom a process perspective: were not able to provide a fix 2 weeks before the embargo date ended
1.0 2.0 3.0
![Page 43: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/43.jpg)
Lessons LearnedLarger pre-disclosure list has not caused a single issues in two years of operating an inclusive approachWe have not had a single 0-day vulnerability
A well run vulnerability process builds trustWillingness to adapt to your stake-holders needs builds more trustIt creates collaboration and understanding of stake-holders
Fairness is a difficult issue
There will always be practical issues, e.g. “interpretations of policy”, etc.
![Page 44: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/44.jpg)
On FairnessThe Xen Project’s process is the only example case, where this issue has been tackled through a community consultation.
To Contrast:OpenStack does not publish who is on their pre-disclosure listOpenStack does not have a formal application processAvoids dealing with the “fairness” issue head-on
![Page 45: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/45.jpg)
Trade-off: On Media Hype and Security Practices
![Page 46: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/46.jpg)
![Page 47: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/47.jpg)
Why was Xen singled out?Security stories are “hot”Xen is widely used, thus security stories “sell”It’s too easy for reporters to write a story
Reporters just have to check our page,and know when the next story comes
![Page 48: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/48.jpg)
Are Security Practices Robust Enoughin the CloudEra?
Source: yanilavigne.net viadomics.me
![Page 49: Scale14x: Are today's foss security practices robust enough in the cloud era final](https://reader035.vdocuments.us/reader035/viewer/2022062901/58f0f0591a28ab461e8b45a7/html5/thumbnails/49.jpg)
Very wide range of approaches vs.The reality that SW stacks contain many layers Consider the weakest link in your SW stack
Best Practice appears to be emergingOlder projects seem slow to changeNew projects, don’t build security management into their culture from the beginning
New Post-Snowden era pressuresHow to effectively deal with media Hype?