EMC WHITE PAPER

OPENSTACK SWIFT OBJECT STORAGE ON EMC ISILON SCALE-OUT NAS

ABSTRACT The EMC Isilon scale-out storage platform provides object storage by exposing the OpenStack Object Storage API as a set of Representational State Transfer (REST) web services over HTTP. The objects that you store through the Swift API can be accessed as directories and files through NFS, SMB, and HDFS. The result is a standard method of securely integrating data-intensive applications with the Isilon storage platform and then sharing the data with other applications, such as Hadoop and Apache Spark.

November 2014

2

To learn more about how EMC products, services, and solutions can help solve your business and IT challenges, contact your local representative or authorized reseller, visit www.emc.com, or explore and compare products in the EMC Store

Copyright © 2014 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H13637

3

TABLE OF CONTENTS

INTRODUCTION ........................................................................................ 5

USE CASES ................................................................................................ 5

SOLVING DATA MANAGEMENT PROBLEMS ................................................ 6

BENEFITS ................................................................................................. 7

THE ONEFS SWIFT IMPLEMENTATION ...................................................... 7 Data Protection Overview .................................................................................. 8

Client Libraries ................................................................................................. 8

Supported HTTP Requests .................................................................................. 8

AUTHENTICATION .................................................................................... 8 Token Generation ............................................................................................. 9

CONCLUSION ............................................................................................ 9

APPENDIX .............................................................................................. 10 Standard TempAuth Authentication .................................................................... 10

Libcloud Authentication with Python Code ........................................................... 10

Libcloud Authentication with OpenStack Identity Service ....................................... 10

Libcloud Authentication with the RackSpace Extension .......................................... 11

OBJECT STORAGE REQUESTS .................................................................. 12 Example Requests Submitted with Curl ............................................................... 12

GET Account ............................................................................................... 12

GET Container ............................................................................................. 13

GET Object ................................................................................................. 13

PUT Container ............................................................................................. 14

PUT Object .................................................................................................. 14

Copy Object ................................................................................................ 14

POST Account .............................................................................................. 14

POST Container ........................................................................................... 14

POST Object ................................................................................................ 15

HEAD Account ............................................................................................. 15

4

HEAD Container ........................................................................................... 15

HEAD Object ............................................................................................... 15

DELETE Container ........................................................................................ 16

DELETE Object ............................................................................................ 16

Paging Through Objects ................................................................................ 16

Working with Libcloud ...................................................................................... 16

Create a Swift Driver .................................................................................... 16

Configure a Swift Driver for OneFS ................................................................. 16

Submitting Example Requests with Libcloud .................................................... 17

5

INTRODUCTION The EMC® OneFS® distributed operating system integrates OpenStack Object Storage with the EMC® Isilon® scale-out storage platform. The OpenStack Object Storage project, code named Swift, stores content and metadata as objects by using an application programming interface. The API furnishes a set of Representational State Transfer (REST) web services over HTTP.

OneFS exposes the Swift API by implementing the OpenStack Object Storage proxy server on every storage node in an Isilon cluster. The proxy server handles API requests. The Swift API presents the distributed OneFS file system as a set of accounts, containers, and objects.

Although OneFS flattens the multi-tiered architecture of OpenStack Object Storage into a single scale-out storage cluster, the containers and objects that you store with the Swift API can be simultaneously accessed as directories and files by using the other protocols that OneFS supports—NFS, SMB, HTTP, FTP, and HDFS.

This seamless, shared access to Swift containers and objects lets applications interact with all the data stored on an Isilon cluster, regardless of the protocol that stored the data. You can, for example, store application data as objects through the Swift API, analyze the data with Hadoop through the OneFS HDFS interface, and then export the results of MapReduce jobs to Microsoft Windows workstations with SMB.

USE CASES By implementing OpenStack Object Storage, OneFS provides a standard, cost-effective method of integrating data-intensive applications with the EMC Isilon scale-out storage platform to securely manage application data with enterprise solutions. Object storage on an Isilon cluster addresses the following uses cases:

• Consolidate storage for applications regardless of protocol.

• Automate data-processing applications to store objects on an Isilon cluster and then analyze the data through the OneFS HDFS interface.

• Automate the sharing of information by storing data with Swift and then seamlessly access the objects as files with SMB, NFS, HTTP, FTP, and HDFS.

• Store files with SMB, NFS, and other protocols and then access the files as objects through Swift.

• Provide secure multitenancy for applications while uniformly protecting the data with enterprise security capabilities like Kerberos authentication, fine-grained access control, and identity management.

• Manage data from Swift applications with enterprise storage features like deduplication, tiering, performance monitoring, snapshots, and NDMP backups.

• Protect data reliably, efficiently, and cost-effectively with forward error correction instead of inefficient replication.

• Automate the dissemination of data to web sites and mobile devices.

• Support second- and third-platform workloads.

Storing objects instead of files on OneFS improves scalability and performance to handle the velocity and volume of large workloads. More importantly, Swift empowers you to automate the collection of petabytes of data and store them in an Isilon data lake for later analysis through other protocols, such as HDFS, SMB, and NFS. OneFS seamless interoperates between object and file, as the following diagram illustrates:

6

Figure 1. OneFS seamless interoperates between object and file.

A Swift connection processes the data as an object, while an HDFS, SMB, or NFS connection processes the same exact data as a file.

SOLVING DATA MANAGEMENT PROBLEMS Object storage's place in the Isilon architecture helps you solve an array of problems. The Isilon architecture combines a secure, scalable multiprotocol data lake with enterprise storage solutions like deduplication to produce a unique storage platform, and the result is that the platform can help solve unique problems.

You can, for example, give a set of users access to the results of MapReduce jobs by having them connect to the cluster with their SmartPhones and access the files with a REST application.

Another use case is to automate applications to retrieve objects that contain data. The applications can then process the data as, for example, business objects and then distribute the information to other applications downstream.

Another Swift use case relates to big data, including metadata. The Swift protocol gives you control over vast amounts of metadata, the data that describes your objects. You can exploit that control to tap into your metadata for analysis, potentially yielding the transformational insights that are the promise of big data analytics. For example, after you store your objects and their metadata on an Isilon cluster with Swift—metadata is stored on Isilon as an alternate data stream—you can automate an application to request the metadata of the stored objects through the REST API. You can then transform the data into a structure that suits your objectives.

Combining Swift data ingestion with Hadoop analytics are two components of a data lake—a technology strategy that lays the foundation to transform your organization into an information-driven enterprise:

7

Figure 2. Swift data ingestion with Hadoop analytics are two components of a data lake technology

strategy

The Swift protocol, coupled with a scale-out data lake powered by an Isilon cluster, streamlines the collection of vast amounts of data for analysis. Data stored with Swift can be analyzed in place with Hadoop or Apache Spark through the OneFS HDFS interface. For more information, see EMC Isilon Scale-Out NAS for In-Place Hadoop Data Analytics.

BENEFITS With an Isilon cluster, you can point at the cluster applications that use Swift to store your data, saving the time, expense, risk, and complexity of building and supporting a new storage infrastructure for the data.

Since Isilon bases its Swift protocol on an open OpenStack Object Storage standard, you can use an existing Isilon cluster to store data from Swift applications without vendor lockin.

Whether you use a new or an existing Isilon cluster, storing object data on an Isilon cluster produces efficiencies at multiple levels:

• Manage only one storage system to avoid additional operating expenses.

• Store object data more efficiently with forward error correction instead of Swift's replication.

• Tap the excess capacity of an existing Isilon cluster to keep storage costs low.

• Easily manage the object data with enterprise capabilities like security, storage pools, tiering, snapshots, replication, and NDMP.

• Increase the return on investment for your Isilon cluster by supporting object data.

• Eliminate storage silos that undermine the benefits of an enterprise data hub.

• Set up and support object storage with ease.

The OneFS implementation of OpenStack Object Storage disregards georeplication as a use case: You cannot use Swift to distribute data over geographically dispersed storage sites.

THE ONEFS SWIFT IMPLEMENTATION OneFS exposes the Swift API by implementing an instance of the OpenStack Object Storage Proxy Server and the OpenStack Storage Server on every storage node in an Isilon cluster. The distributed OneFS operating system combines both the OpenStack Proxy Server and the OpenStack Storage Server into a single server that runs on every Isilon node. As a result, client computers that connect to an Isilon cluster with Swift gain access to the distributed Isilon file system's single volume, ifs, while taking advantage of the entire cluster's performance.

To work with object storage, you connect to an Isilon cluster with HTTP and then use standard REST calls such as PUT, GET, and POST to perform API operations. The API presents the home directories as accounts, directories as containers, and files as objects. All objects have metadata. The API operations that you submit with REST can store and manage containers, objects, and metadata in the OneFS file system.

8

Each home directory in the OneFS file system maps to a Swift account. The directories and subdirectories in a home directory map to containers and subcontainers. Files appear as objects. Since each object has a URL, you can access a file by its URL. A file in a user's home directory, for example, might look something like this: /ifs/home/admin/engineering/samplefile.txt

With Swift, you can access the file as an object by using a GET, PUT, or POST operation at the following URL, which contains the SmartConnect zone of an example cluster: http://examplezonename:28080/v1/AUTH_admin/engineering/samplefile.txt

OneFS uses Port 28080 for all Swift requests. Each object can be as large as 5 GB, which is the Swift default.

By default, OneFS uses a round-robin algorithm to distribute API requests across all the nodes in the cluster. For optimal performance, you should provision the external network interfaces of the Isilon nodes with dual high-throughput (10GbE) interfaces.

DATA PROTECTION OVERVIEW The OneFS operating system efficiently and reliably protects data with forward error correction (FEC) codes, which consume less space than replication but provide better protection. Swift replicates an object three times to protect it and to make it highly available. Instead of replicating the object, OneFS stripes the object's data across the cluster over its internal InfiniBand network.

FEC is a highly efficient method of reliably protecting data. FEC encodes an object's data in a distributed set of symbols, adding space-efficient redundancy. With only a part of the symbol set, OneFS can recover the object's data. In an Isilon cluster with five or more nodes, FEC delivers as much as 80 percent efficiency. As you add nodes to a cluster, the efficiency improves.

Striping data with FEC codes consumes much less storage space than replicating every object three times. With the Isilon data protection scheme, more than 80 percent of an Isilon cluster’s capacity can be used, bringing efficiency to applications that store objects with Swift. See High Availability and Data Protection with EMC Isilon Scale-Out NAS.

CLIENT LIBRARIES The OneFS Swift implementation is compatible with the following Swift client applications and APIs:

• The Swift command-line client and the Python-Swift client library; see http://docs.openstack.org/developer/python-swiftclient.

• Apache Libcloud, which is a Python library that supports Swift along with many other different object storage provider APIs through a unified API; see http://libcloud.readthedocs.org.

SUPPORTED HTTP REQUESTS To work with these libraries, the first version of the OneFS Swift service, once activated with a license key, supports the following Swift HTTP requests:

• Authentication: Use TempAuth or Libcloud

• GET: retrieve an object or the contents of a container or account.

• PUT: upload an object or create a container.

• DELETE: delete an object or container.

• POST: store metadata for an object, container, or account.

• HEAD: retrieve metadata for an object, container, or account.

• COPY: create a copy of an object.

To obtain a license key, contact your EMC Isilon representative. For more information about supported requests and the capabilities of the OneFS Swift service, see support.emc.com. For information on the Swift RESTful API, see OpenStack Object Storage API documentation.

AUTHENTICATION Authentication for a Swift connection takes place in an access zone—a virtual security context in which OneFS connects to directory services, authenticates users, and controls access to a segment of the file system. By default, a cluster has a single access zone for the entire file system. You may create additional access zones to give users from different identity management systems, such as two untrusted Active Directory domains, access to different OneFS resources by using a destination IP address or SmartConnect zone

9

name. Access zones provide multitenancy: You can set up a cluster to work with multiple identity management systems, Swift namespaces, SMB namespaces, and HDFS namespaces.

The main purpose of an access zone is to define a list of identity management systems that apply only in the context of a zone. As such, a key use is consolidating data sets from different storage silos into a single storage system but continuing to expose each data set with a unique root directory and then limiting access to a group of users.

When a Swift user submits an authentication request to the cluster, OneFS checks the directory services to which the user’s access zone is connected for an account for the user. If OneFS finds an account that matches the user’s login name, OneFS authenticates the user.

During authentication, OneFS creates an access token for a Swift user in the same way that OneFS creates tokens for users who connect with other protocols. The token contains the user’s full identity, including group memberships, and OneFS uses the token later to check access to directories and files. The OneFS Swift implementation will not, however, create a home directory account for a user, even if a home directory has been specified in, for example, Active Directory. Before a user connects with Swift, you must explicitly create a home directory for the user in the ifs global directory with a method other than through the Swift protocol.

For more information on how OneFS authenticates connections, creates access tokens, and authorizes access to resources with access control lists, see OneFS Multiprotocol Security Untangled. For more information on multitenancy, see EMC Isilon Multitenancy for Hadoop Big Data Analytics.

With OneFS, you can submit a Swift authentication request in two ways:

• The standard Swift format, which is used by the OpenStack TempAuth and Swauth modules.

• The OpenStack Identity Service, which libcloud uses as its primary authentication method.

• The Rackspace extension to the OpenStack Identity Service.

The appendix includes examples of how to authenticate by using the standard format and libcloud.

TOKEN GENERATION When a user authenticates successfully, OneFS generates a unique authentication token for the user. A string of 32 hex characters prefixed with AUTH_tk, the token lets a user perform Swift requests without providing a username and password each time. Instead, the token appears in the X-Auth-Token or X-Storage-Token fields of the header. By default, a token lasts for 24 hours. After it expires, a new token must be obtained by submitting an authentication request. A token works only in the OneFS access zone in which the authentication request that generated it took place.

CONCLUSION The EMC Isilon OneFS operating system exposes the Swift API by implementing an instance of the OpenStack Object Storage Proxy Server and the OpenStack Storage Server on every storage node in an Isilon cluster. Client computers that connect to an Isilon cluster with Swift gain access to the distributed Isilon file system's single volume as well as data stored by other protocols, including NFS, SMB, and HDFS. The result is a standard method of securely integrating data-intensive applications with the Isilon scale-out storage platform and exchanging data with other applications.

10

APPENDIX STANDARD TEMPAUTH AUTHENTICATION You can authenticate a user with the TempAuth method by submitting a GET request in the following format: http://examplezonename:28080/auth/v1.0

The request provides the username and password for the authentication request in the HTTP header in the following format: X-Auth-User: <account>:<username> X-Auth-Key: <password>

OneFS ignores the value of the account field. It can be set to any value, but the field must include a value to maintain compatibility with the OpenStack Object Storage standard. The standard uses the account field to support multitenancy, but because OneFS provides multitenancy with access zones, the value is irrelevant.

The following example demonstrates how to authenticate with TempAuth by using curl, a command-line utility for transferring data with URL syntax (see the curl web site at http://http://curl.haxx.se/). The first instance of 'admin' in 'X-Auth-User: admin:admin' is where the client specifies the account name—the field that OneFS ignores. The username is 'admin' and the password is 'test'. curl -H "X-Auth-User: admin:admin" -H "X-Auth-Key: test" -v "http://examplezonename:28080/auth/v1.0" -X GET

A successful response looks like the following example. The response includes an access token and a storage access token and URL, which the client uses for subsequent requests. HTTP/1.1 200 OK Content-Length: 101 Content-Type: application/json; charset=utf-8 Date: Thu, 19 Jun 2014 09:21:34 PDT X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e X-Storage-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e X-Storage-Url: http://examplezonename:28080/v1/AUTH_admin X-Trans-Id: tx070e772fe79c4d948a016-0053a30e0e { "storage": { "cluster_name": "http://examplezonename:28080/v1/AUTH_admin", "default": "cluster_name" } }

LIBCLOUD AUTHENTICATION WITH PYTHON CODE Here is an example that demonstrate how to authenticate with Libcloud by using code written in Python: from libcloud.storage.types import Provider from libcloud.storage.providers import get_driver import sys ip = "192.0.2.250" cls = get_driver(Provider.OPENSTACK_SWIFT) my_username=”admin” my_password=”testpwd” driver = cls(my_username, my_password, region='ISILON', ex_force_auth_url='http://'+ip+':28080', ex_force_base_url='http://'+ip+':28080/v1/AUTH_ISILON', ex_force_auth_version='2.0', ex_force_service_type='object-store', ex_force_service_name='swift')

LIBCLOUD AUTHENTICATION WITH OPENSTACK IDENTITY SERVICE You can authenticate a user with the OpenStack Identity Service by submitting a request with Curl in the following format: curl -H "Content-Type: application/json; charset=utf-8" -v http://<node-ip>:28080/v2.0/tokens -X POST -d '{"auth":{"tenantName":"<any-value-works-here>", "passwordCredentials":{"username": "<username>", "password":"<password>"}}}'

Here is an example that replaces the variables with sample values: curl -H "Content-Type: application/json; charset=utf-8" -v http://examplezonename:28080/v2.0/tokens -X POST -d '{"auth":{"tenantName":"isilon", "passwordCredentials":{"username": "myusername", "password":"mypassword"}}}'

11

The response looks like this: HTTP/1.1 200 OK Content-Length: 467 Content-Type: text/html; charset=UTF-8 Date: Mon, 18 Aug 2014 13:13:54 PDT X-Trans-Id: txdc10f3db5b1444a6a4521-0053f25e82 * Connection #0 to host examplezonename left intact { "access": { "token": { "expires":"1600-12-31 16:02:20", "id":"AUTH_tk71e9c58f3e432da1adf3ba70dc236dd8", "tenant":{ "id":"10", "name":"isilon" } }, "serviceCatalog": [ { "endpoints": [ { "region":"ISILON", "internalURL":"http://192.0.2.250/v1/AUTH_isilon", "publicURL":"http://examplezonename:28080/v1/AUTH_isilon" } ], "type":"object-store", "name":"swift" } ], "user": { "id":"10", "roles":[ { "tenantId":"10", "id":"0", "name":"object-store:myusername" } ], "name":"mypassword" } } }

LIBCLOUD AUTHENTICATION WITH THE RACKSPACE EXTENSION You can authenticate a user with the Rackspace method by submitting a POST request in the following format: http://examplezonename:28080/v2.0/tokens

With the Rackspace method, you format the username and password in JSON and set it in the content section of the HTTP request so it looks like this: { "auth": { "RAX-KSKEY:apiKeyCredentials": { "username": "<username>", "apiKey": "<password>" } } }

Here is an example of how to submit an authentication request using the Rackspace method. The username is 'admin' and the password is 'test'. curl -H "Content-Type: application/json; charset=UTF-8" -v "http://examplezonename:28080/v2.0/tokens" -X POST -d '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "admin", "apiKey": "test"}}}'

12

A successful response looks like the following example. The response includes an access token and a storage access URL, which the client uses for subsequent requests. HTTP/1.1 200 OK Content-Length: 467 Content-Type: text/html; charset=UTF-8 Date: Thu, 19 Jun 2014 09:33:21 PDT X-Trans-Id: tx3cfed971658f46eeb33c4-0053a310d1 { "access": { "serviceCatalog": [ { "endpoints": [ { "internalURL": "http://examplezonename/v1/AUTH_isilon", "publicURL": "http://examplezonename:28080/v1/AUTH_isilon", "region": "ISILON" } ], "name": "swift", "type": "object-store" } ], "token": { "expires": "1600-12-31 16:02:20", "id": "AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e", "tenant": { "id": "10", "name": "isilon" } }, "user": { "id": "10", "name": "admin", "roles": [ { "id": "0", "name": "object-store:admin", "tenantId": "10" } ] } } }

OBJECT STORAGE REQUESTS After you acquire a token, you can submit object storage requests by using REST over HTTP. The syntax of the requests is described in the OpenStack Object Storage API Reference and other documents at http://docs.openstack.org/.

EXAMPLE REQUESTS SUBMITTED WITH CURL The following examples submit requests with curl from the command line of a client computer. The examples use an IP address of examplezonename for all requests.

GET Account The following example submits a GET request to obtain account information: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin?format=json" -X GET

A successful response looks like this: HTTP/1.1 200 OK

13

Content-Length: 84 Content-Type: application/json; charset=utf-8 Date: Thu, 19 Jun 2014 14:54:32 PDT Last-Modified: 2014-06-19 14:54:29 X-Account-Bytes-Used: 13 X-Account-Container-Count: 2 X-Account-Object-Count: 2 X-Timestamp: 130476887138427115 X-Trans-Id: tx83b138929e134d24a4cd7-0053a35d1b [ { "count": 1, "bytes": 4, "name": "container" }, { "count": 1, "bytes": 9, "name": "container2" } ]

GET Container The following example submits a GET request to obtain information about a container: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2?format=json" -X GET A successful response looks like this: HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 217 Content-Type: application/json; charset=utf-8 Date: Thu, 19 Jun 2014 14:59:44 PDT Last-Modified: 2014-06-19 14:58:48 X-Container-Bytes-Used: 9 X-Container-Object-Count: 1 X-Timestamp: 130476887284112371 X-Trans-Id: tx169d4d5da58647dea7ef0-0053a35d50 [ { "hash": "ef614d88b44d9b768ca06befb206cf3c", "last_modified": "2014-06-19 14:58:48", "bytes": "9", "name": "obj2.txt", "content_type": "application/octet-stream" } ]

GET Object The following example submits a GET request to obtain information about an object: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X GET

A successful response looks like this: HTTP/1.1 200 OK Content-Length: 9 Content-Type: application/octet-stream Date: Thu, 19 Jun 2014 15:01:13 PDT ETag: ef614d88b44d9b768ca06befb206cf3c Last-Modified: 2014-06-19 14:58:48 X-Timestamp: 130476887284112371

14

X-Trans-Id: tx2572012006224369ae08f-0053a35da9 obj2 data

PUT Container The following example submits a PUT request for a container: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container3" -X PUT

A successful response looks like this: HTTP/1.1 201 Created Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 10:54:50 PDT X-Trans-Id: tx0ceffc19132b45d1ba91d-0053a4756a

PUT Object The following example submits a PUT request for an object: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X PUT -d "obj content"

A successful response looks like this: HTTP/1.1 201 Created Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 10:47:52 PDT ETag: e3cf5b79108fc1cc822bd0f9b5b67cef X-Trans-Id: tx5e366ec5f04041f9b8500-0053a473c8

Copy Object The following example submits a COPY request to create a server-side copy of container2/obj2 and name it obj2_copy.txt in container/sub-container: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X COPY -H "Destination: container/sub-container/obj2_copy.txt"

A successful response looks like this: HTTP/1.1 201 Created Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:38:18 PDT X-Trans-Id: tx5c765fa5e1e64f5bb6864-0053a47f9a

POST Account The following example submits a POST request to set the custom metadata tags of an account to Value1 and Value2: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin" -X POST -H "X-Account-Meta-Name1: Value1" -H "X-Account-Meta-Name2: Value2"

A successful response looks like this: HTTP/1.1 204 No Content Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:07:42 PDT X-Trans-Id: txa819a723b6f449d593966-0053a4786e

POST Container The following example submits a POST request to set the custom metadata tags of a container to Value1 and Value2: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container3" -X POST -H "X-Container-Meta-Name1: Value1" -H "X-Container-Meta-Name2: Value2"

A successful response looks like this: HTTP/1.1 204 No Content

15

Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:09:56 PDT X-Trans-Id: tx1250a35c0ac14f2480612-0053a478f4

POST Object The following example submits a POST request to set the custom metadata tags of an object to Value1 and Value2: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X POST -H "X-Object-Meta-Name1: Value1" -H "X-Object-Meta-Name2: Value2"

A successful response looks like this: HTTP/1.1 204 No Content Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:12:36 PDT X-Trans-Id: tx7205bce69fe547508c176-0053a47994

HEAD Account The following example submits a HEAD request to retrieve account statistics, metadata, and the values of the custom metadata tags: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin" -X HEAD

A successful response looks like this: HTTP/1.1 204 No Content X-Account-Meta-Name1: Value1 X-Account-Meta-Name2: Value2 Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:16:23 PDT Last-Modified: 2014-06-20 11:07:42 X-Account-Bytes-Used: 15 X-Account-Container-Count: 3 X-Account-Object-Count: 2 X-Timestamp: 130477604900823291 X-Trans-Id: txe0e5f861dca641268ce4e-0053a47a76

HEAD Container The following example submits a HEAD request to retrieve a container's metadata: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2" -X HEAD

A successful response looks like this: HTTP/1.1 204 No Content X-Container-Meta-Name1: Value1 X-Container-Meta-Name2: Value2 Accept-Ranges: bytes Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:14:40 PDT Last-Modified: 2014-06-20 11:09:56 X-Container-Bytes-Used: 0 X-Container-Object-Count: 0 X-Timestamp: 130477613968019069 X-Trans-Id: tx56d23577d9e04dad95d73-0053a47a10

HEAD Object The following example submits a HEAD request to retrieve an object’s metadata, including its etag: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X HEAD

A successful response looks like this:

16

HTTP/1.1 204 No Content X-Object-Meta-Name1: Value1 X-Object-Meta-Name2: Value2 Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:13:57 PDT ETag: e3cf5b79108fc1cc822bd0f9b5b67cef Last-Modified: 2014-06-20 11:12:36 X-Timestamp: 130477615561086205 X-Trans-Id: tx4951a255ec3e43e5b6c09-0053a479e5

DELETE Container The following example submits a DELETE request to remove a container: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container3" -X DELETE

A successful response looks like this: HTTP/1.1 204 No Content Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:34:26 PDT X-Trans-Id: tx04f0072a9573448ea0e52-0053a47eb2

DELETE Object The following example submits a DELETE request to remove an object: curl -H "X-Auth-Token: AUTH_tk9b2f1d4d640b31fee0b3f6d644aab52e" -v "http://examplezonename:28080/v1/AUTH_admin/container2/obj2.txt" -X DELETE

A successful response looks like this: HTTP/1.1 204 No Content Content-Length: 0 Content-Type: text/html; charset=UTF-8 Date: Fri, 20 Jun 2014 11:33:35 PDT X-Trans-Id: txbfe6a6b594904ad1a55fd-0053a47e7f

Paging Through Objects You can page through a large list of objects in a container or a large list of containers in an account by using the limit and marker URL parameters. For example, say there are five objects in a container named small_container and the names of the objects are obj1, obj2, obj3, obj4, and obj5. To page through the items in blocks of two items at a time, you can use the following command: curl -H “X-Auth-Token: <token>” “http://<ip>:28080/v1/<account>/small_container?limit=2” -X GET

This command returns obj1 and obj2. To get obj3 and obj4, you can run this command: curl -H “X-Auth-Token: <token>” “http://<ip>:28080/v1/<account>/small_container?limit=2&marker=obj2” -X GET

And then you can get obj5 like this: curl -H “X-Auth-Token: <token>” “http://<ip>:28080/v1/<account>/small_container?limit=2&marker=obj4” -X GET

Working with Libcloud The OneFS ObjectStorage API accepts requests from libcloud. For more information on libclould, see the libcloud reference documentation.

Create a Swift Driver Here is an example of how to create a Swift driver so that you can submit requests with libcloud: from libcloud.storage.types import Provider from libcloud.storage.providers import get_driver import sys cls = get_driver(Provider.OPENSTACK_SWIFT)

Configure a Swift Driver for OneFS Here is an example of how to configure a driver. The username is 'admin' and the password is 'test': driver = cls('admin', 'test', region='ISILON',

17

ex_force_auth_url='http://examplezonename:28080', ex_force_base_url='http://examplezonename:28080/v1/AUTH_ISILON', ex_force_service_type='object-store', ex_force_service_name='swift')

Another method of configuring a driver is to use a connection endpoint URL instead of a base URL: driver = cls('admin', 'test', region='ISILON', ex_force_auth_url='http://examplezonename:28080', ex_force_service_type='object-store', ex_force_service_name='swift') driver.connection.endpoint_url = "publicURL"

Submitting Example Requests with Libcloud Get a list of containers in the account: containers = driver.list_containers()

Get a list of objects in the first container: objs = driver.list_container_objects(containers[0])

Download the first object to /tmp/test.txt: driver.download_object(objs[0], "/tmp/test.txt")

Upload /tmp/test.txt to the first container and name it libcloud.txt: driver.upload_object("/tmp/test.txt", containers[0], "libcloud.txt")

Delete the first object: driver.delete_object(objs[0])

Create a new container named libcloud_container: driver.create_container("libcloud_container")

Iterate through containers and delete the one named libcloud_container: for container in driver.iterate_containers(): if container.name == "libcloud_container": driver.delete_container(container) break