case study: seagate’s openstack swift security rodney ... · pdf fileipmi bmc ilo drac...

Copyright © 2014 Seagate Technology

Case Study: Seagate’s OpenStack Swift Security

Presentation for Cloud Security Alliance (CSA) Congress

Rodney Beede, Seagate Technology

September 2014


Introduction● Rodney Beede

○ Cloud Security Engineer○ Seagate Technology

● M.S. in Computer Science○ University of Colorado○ “A Framework for Benevolent Computer Worms” 2012

● Doing computer security since 2001○ Primary interests are web and cloud security

● Tech blog○ https://www.rodneybeede.com/○ The views expressed in this blog are my personal view and have not been reviewed or

approved by Seagate.

https://www.rodneybeede.com/

https://www.rodneybeede.com/


What Is OpenStack Swift?● Object Storage

○ HTTP REST API web service○ Stores objects

Object Storage API example

PUT /v1/my_account/container/example_obj.txt HTTP/1.1User-Agent: curl/7.32.0Host: localhost:8080Accept: */*X-Auth-Token: authtokenhereContent-Length: 38Expect: 100-continue

------------------------------------------------------

HTTP/1.1 201 CreatedContent-Length: 118Content-Type: text/html; charset=UTF-8Etag: f7d40eceffdd9c2ecab226105737b2a6

Image from “OpenStack Installation Guide for Red Hat Enterprise Linux, CentOS, and Fedora” and licensed under the Apache License, Version 2.0


What Do We Use It For?● Testing hard drives

○ Benchmarking○ Measuring drive temperature○ Measuring drive vibration

● IT uses○ Backups

■ Petabytes of storage space

● EVault○ Long term storage (tape replacement)


Some Supporting Technologies

● Linux

● Puppet - puppetlabs.com○ Centralized automated config

● Center for Internet Security (CIS)○ Security hardening benchmarks


The Short Version - Swift Security● Lots of components to secure

○ OS○ Web server○ Authentication store○ Network traffic○ Remote console interface

● Lack of guidelines (until now)○ http://docs.openstack.org/sec/○ …”we do not have specific

guidance related to configur-ation of the storage projects…”

○ So I wrote my own

http://docs.openstack.org/sec/



Question: Network Protocol● Which

services have encrypt-ion over the wire?


Remote Consoles - Overlooked● Commonly known as

○ IPMI○ BMC○ iLO○ DRAC

● Security problem○ Default password○ IPMI protocol insecure

● A Penetration Tester's Guide to IPMI and BMCs - Rapid 7○ HD Moore, Metasploit, July 2, 2013 12:22:49 PM○ Unpatched firmware - admin access○ Admin hash vulnerability unpatchable

Image care of https://flic.kr/p/trJkJ Licensed under CC BY-SA 2.0. Modified with red highlight. Original by Cloned Milkmen and titled “T2000 USB and LED Close-up” 2006.

https://community.rapid7.com/community/metasploit/blog/2013/07/02/a-penetration-testers-guide-to-ipmi




https://flic.kr/p/trJkJ

https://creativecommons.org/licenses/by-sa/2.0/


Remote Consoles - Remediation1. Network segregation

2. Patch to latest firmware

3. Rotate your passwords often

a. Typically manual process

b. IPMI has standard protocol for user management

i. Hint: Linux ipmitool command

c. We use CyberArk with custom plugin


Securing Services - General● Standard OS configuration file permissions

# chown -R root:swift /etc/swift/

# find /etc/swift/ -type f -exec chmod 640 {} \;

# find /etc/swift/ -type d -exec chmod 750 {} \;

● Run services as “swift”○ Not root○ You won’t need TCP ports < 1024 either


Real World Mistake● Question: What are these files?

[user@host] $ ls /etc/certs/ca.crtintermediate.crtserver.crtserver.csrserver.pem


Real World Mistake - Problem● Question: What is the matter here?

[user@host] $ ls -la /etc/certs/

drwxrwxr-x 2 www www 4096 Jul 22 12:15 .-rw-r--r-- 1 www www 2110 Jan 2 2014 server.crt-rw-r--r-- 1 www www 1813 Jan 2 2014 server.csr-rw-r--r-- 1 www www 3243 Jan 2 2-14 server.pem


Real World Mistake - Correct Security● Question: Which is correct?

A. -r--r----- 1 www www server.pem

B. -rw-r----- 1 root www server.pem

C. -rw-r----- 1 root root server.pem

D. ---------- 1 root root server.pem


Real World Mistake - Answer● Answer: All of them can be

A. -r--r----- 1 www www server.pem

B. -rw-r----- 1 root www server.pem

C. -rw-r----- 1 root root server.pem

D. ---------- 1 root root server.pem


Securing Storage ServicesThe following are the default listening ports for the various storage services:

Service name Port Type

Account service 6002 TCP

Container service 6001 TCP

Object service 6000 TCP

Rsync 873 TCP


Object Storage "Account" TerminologyAn object storage "account" isn’t what you think it means

OpenStack Object Storage Account

Collection of containers; not user accounts.

Supports ACLs to associate “users” with the account.

OpenStack Object Storage Containers

Collection of objects.

Supports ACLs.

OpenStack Object Storage Objects

The actual data objects.

Supports ACLs.


Another way of thinking about it:

● A single shelf (Account) holds zero or more

○ Buckets (Containers)

● Buckets (Containers) each hold zero or more

○ Objects

● A garage (Object Storage cloud environment) may have

○ Multiple shelves (Accounts)

○ Each shelf may belong to zero or more users.

Object Storage "Account" Terminology


Securing Proxy Services● This is the consumer facing service


Securing Proxy Services1. Use SSL/TLS

a. Not the built-in web serverb. Apache, Nginx, etc. with mod-wsgi

2. Run web server as non-roota. “swift” system account is one optionb. Listen on port > 1024

i. URL is typically not typed anyway

https://swift.cloud.example.org:44443/v1/AUTH_1234

3. SSL Load Balancera. Common pitfall - early termination




Load Balancerwith SSL offload


Identity (Keystone)● Formally used SWAuth

○ Stored all info as Swift objects

● Now using Identity Service API ver 2.0○ Custom implementation versus Keystone○ Going to add LDAP for internal use

● Basic principles○ Manage credentials

■ Password length, complexity■ Account termination

○ Audit your logs for policy conformity


Security Testing● CVE list

○ http://www.cvedetails.com/vulnerability-list/vendor_id-11727/Openstack.html

● Authentication token validation○ Lots of CVEs for invalid tokens○ Distributed clustered system causes auth delays

● Check for exposed storage node ports○ TCP ports 6000-6002 (default)○ Also think about packet sniffing internally

■ Network switch != secure (think ARP flood)

http://www.cvedetails.com/vulnerability-list/vendor_id-11727/Openstack.html




Security Testing● It’s a web service

○ Similar approach as other web services○ WSDL is optional, Swift is REST

● Encoding attacks in URIs○ Ex: Third party web applications tend to “trust” the web

service data without proper output encoding

○ Create an object with a name like

“test<script>alert(‘xss’)</script>me”

■ Valid name to Swift■ Can trip up third party web application consumers


Interesting Problem & Solution


Interesting Problem & Solution

These nodes have no Internet or corporate network access


Options● NAT

○ Requires more resources○ Additional security rules to manage

● HTTP Proxy○ Must configure each node to use it

■ Puppet module can do this○ Possibility of caching is a bonus

● Local package repo○ Provides local mirror

■ Faster■ Available even if Internet is not


Trial and Error● NAT

○ Worked for small scale○ Security misconfiguration

● HTTP Proxy○ Wrote Puppet module

■ Provided auto-configuration○ Not helpful if Internet goes down


Solution● Local mirror repository

○ Controlled package versions

○ No constant Internet connection required

○ Use Puppet for client configuration


Puppet● Automation is the key

○ Account management■ Ex: “swift” OS system account

○ Cloud installation■ Public (not Seagate developed) modules

● puppetlabs/swift● puppetlabs/apache

■ Private (Seagate developed) modules● HP OpenView● Security hardening● SSH key management

● Configuration○ Hiera driven (Hierarchical Database)○ Allows separate environments

● Con: Mistake in Puppet config can take down entire environment○ Although usually can recover via Puppet too


Puppet - Exampleuser { ‘swift’:

ensure =>present,comment =>‘OpenStack’,expiry =>absent,forcelocal =>true,password => ‘*’, # no password loginshell =>‘/sbin/nologin’,system =>true,

}


Future● Detailed object reporting

○ What ACLs exist on objects■ Something open to anonymous read-only■ Document marked confidential?

● Account management○ Orphaned accounts○ Verification of terminated users

● Automated unit tests for API security○ Verification of token life cycle○ ACL’s on objects○ Malicious character names (persistent XSS)


References● OpenStack Security Guide

○ http://docs.openstack.org/sec/

● Learning Puppet

● Kinetic○ New Seagate Breakthrough Fundamentally Alters Cloud Economics

- Oct 22, 2013

○ Seagate Kinetic Open Storage Platform

○ Kinetic Open Storage Documentation Wiki



http://docs.puppetlabs.com/learning/manifests.html

http://docs.puppetlabs.com/learning/manifests.html

http://www.seagate.com/about/newsroom/press-releases/Seagate-breakthrough-fundmentally-alters-cloud-economics/




http://www.seagate.com/solutions/cloud/data-center-cloud/platforms/

http://www.seagate.com/solutions/cloud/data-center-cloud/platforms/

https://developers.seagate.com/display/KV/Kinetic+Open+Storage+Documentation+Wiki

https://developers.seagate.com/display/KV/Kinetic+Open+Storage+Documentation+Wiki


Thank You


BACKUP MATERIAL


Kinetic● Internet has published information


Kinetic - OpenStack Swift


Kinetic - SecurityFrom “Kinetic Open Storage Value Proposition”:Security

The security of storage services within the cloud datacenter is a difficult task. The Kinetic drive and interface library supports:

● Authentication - A full cryptographic authentication of servers that have access permission to the drive.

● Integrity - Full integrity check of the command and the data.● Authorization - A clear set of roles by server as to what the application is

allowed to do. Typical roles are read, read/write, management of the drive and management of the security in the drive.

● Transport Layer Security - For the security of very sensitive data and/or management commands, a full industry standard TLS suite is also provided.

https://developers.seagate.com/display/KV/Kinetic+Open+Storage+Value+Proposition



case study: seagate’s openstack swift security rodney ... · pdf fileipmi bmc ilo drac...

Documents