devops with suse with suse: how suse manager, ... • qa environment for testing new builds of a...

DevOps with SUSE:How SUSE Manager, SUSE Studio and SUSE Cloud APIs Facilitate Continuous Software Delivery

Joachim WernerSenior Product Manager

SUSE/joe@suse.com

Wolfgang EngelEngineer

SUSE/wengel@suse.com

2

Why?

4

5

Traditional Collaboration

Development

Operations

HW

OS

APP

CODE

6

Get Together!

7

The DevOps Way

Development

Operations

HW

OS

APP

CODE

8

Continues Delivery

What If You Want To...

10

Automate repetitive tasks?

Generate customized reports from the data in a system?

Hook systems together that weren't built to work together?

Orchestrate several separate systems or tools?

Write my own UI because I have different needs than what the authors of the tool had in mind?

11

• Development‒ Ease package creation

‒ Automate tests and code deployment

‒ Control application stack

‒ Metrics

• Operations‒ Automate system installation/creation and configuration

‒ Control systems

‒ Monitoring

12

Then You Need APIs!

Setting The Stage...

14

Objectives of This Session:

• Demonstrate the power of using APIs in SUSE's Cloud and Management products in a real-life example

• Provide starting points and hints for your own work with the APIs

Not:

• A ready to run solution (just limited-scope proof of concept)

• Comprehensive API tutorial (just the tip of the iceberg, much more to discover below sea level)

15

Scenario

• QA environment for testing new builds of a custom RPM package

• Package should be tested

‒ as part of an appliance image

‒ as an RPM maintenance update to a running instance

• Everything should be fully automated from the time a package is checked in to running the tests

16

Solution Components

Open Build Service‒ Build your RPMs

SUSE Studio™

‒ Build a new appliance image

‒ Deploy it to the Cloud

SUSE® Manager‒ Deploy RPM-based update

‒ Run test suite

SUSE Cloud‒ Run test instances

17

Open Build Service (OBS)

• OBS builds binary packages for many distributions and platforms

• OBS makes them available for download

18

SUSE Studio™

+ Simplify Maintenance

+ Simplify Deployments

+ Streamline the OS

SUSE Studio is a collection of tools designed to improve the efficiency of building, managing and maintaining software virtual and cloud applications

19

SUSE® Manager

Modular Approach

SUSE Manager delivers completelifecycle management for Linuxservers through its management,provisioning, and monitoringmodules

20

SUSE® Cloud

Private cloudinfrastructure solutionpowered by OpenStack

Integrated platform and tools for hybrid clouds

SUSE Linux Enterprisein public clouds

21

Optional Components

GIT or Subversion(everybody should be using a version control system)

Jenkins(to orchestrate)

• http://jenkins-ci.org/

• Jenkins is like “cron on steroids” :-)

Getting Prepared...

23

Prerequisites (1)For replicating the full setup you need:

• Open Build Service‒ available here: www.open-build-service.org/

‒ or on the SUSE Linux Enterprise SDK

• SUSE Studio™

‒ www.suse.com/products/susestudio/

‒ Talk to your sales contact for an evaluation version

• SUSE® Manager‒ www.suse.com/products/suse-manager/

‒ Eval available on the website

• SUSE Cloud‒ www.suse.com/products/suse-cloud/

‒ Eval available on the website

24

Prerequisites (2)For replicating the full setup you need:

• On the machine or instance the API automation is run:‒ Python (part of SUSE® Linux Enterprise)

‒ the “requests” module for Python:

‒ http://docs.python-requests.org/en/latest/index.html

‒ http://software.opensuse.org/package/python-requests

• Hardware:‒ SUSE Studio™ Onsite needs a dedicated server

‒ The rest can be virtualized

‒ You need a lot of RAM for Studio, Manager, Cloud

‒ The API automation script can be run on anyOS that has Python.

25

Prerequisites for “just playing”:Some of the components used here are available online ...

• You can use the openSUSE®

instance of theOpen Build Service:

‒ https://build.opensuse.org

• You can use the freeStudio Online:

‒ http://susestudio.com/

The Script...

27

Workflow

• Create a project in OBS

• Create a specfile for an RPM and add it to the project

• Build a new version of the RPM

• Update the RPM in SUSE Studio™

• Update the RPM in SUSE® Manager

• Trigger SUSE Studio image rebuild

• Trigger update of package with SUSE Manager

• Webhook to trigger test deployment of cloud image from SUSE Studio

• Use SUSE Manager to trigger test run of “test suite”

28

Automating an RPM Package QA Environment:

RPM“MyApp”

GIT

1 Check insources

2 Check outsources

3 Rebuild RPM

4

SUSE Studio

Sync repository/upload RPM

5 Rebuild image

6 Redeployimage SUSE Cloud

Instance 1

Instance 2

Instance 1

7 Sync repository

SUSE Manager

8 UpdateRPM

Instance 2

9 Run test

Instance 1

Instance 2

29

And One RingScript To Rule Them All:

12

3

45

67

89

30

Integration Via

• Python-based web service process‒ In the presentation we are skipping the “web service” part

‒ Demos are short executable Python scripts

• OBS, Studio, Manager, and Cloud APIs

31

We Are Going To Use Python Because ...

• xmlrpc support is built in

• http and url handling built in

• OBS client API is usable as aPython module

• optional: python-requests module

The setup can be replicated in other programming languages of your choice, like

‒ Perl

‒ Ruby

‒ Java

Open Build Service

33

Open Build Service API

• The OBS API is a RESTful API

• Authentication is done by sending aBasic HTTP Authorisation header (every time)

• Documentation:‒ https://api.opensuse.org/apidocs/

(API Reference)

‒ http://en.opensuse.org/images/d/df/Obs-cheat-sheet.pdf(Cheat Sheet for the command line client)

34

All code examples arein Python 2 syntax!

And remember:In real life you need to do exception

handling and other boring stuff!

35

A “Hello World!” Example ...Using the hard way: urllib2

#!/usr/bin/env python

import urllib

import urllib2

url = 'https://api.opensuse.org'

user = 'username'

pwd = 'password'

pwdmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

pwdmgr.add_password(None, url, user, pwd)

authhandler = urllib2.HTTPBasicAuthHandler(pwdmgr)

opener = urllib2.build_opener(authhandler)

urllib2.install_opener(opener)

response = urllib2.urlopen(url + '/about')

print response.read()

36

The Response ...

<about> <title>Open Build Service API</title> <description>API to the Open Build Service</description> <revision>2.3.0.git201207101028</revision></about>

37

Now Let's Introduce The“Requests” Module ...

• “HTTP for Humans” (quote from the website)

• ISC License (permissive BSD-style license)

• Very easy to use

• Well documented

38

Doesn't This Look Much Nicer?

#!/usr/bin/env python

import requests

url = 'https://api.opensuse.org'

user = 'username'

pwd = 'password'

r = requests.get(url + '/about', auth=(user, pwd))

print r.text

39

Or We Can Use The osc CommandLine Client As A Library

What is osc?‒ A command line client for OBS that you can use with a syntax

just like svn to check out, modify, and commit packages to OBS

‒ But it can also be used to call the api directly!

‒ You can extend osc with your own commands.

‒ And: you can load it as a library from a Python script.

40

Preparing For Launch ...

zypper in osc

osc

Your user account / password are not configured yet.

You will be asked for them below, and they will be stored in

/home/joe/.oscrc for future use.

Creating osc configuration file /home/joe/.oscrc ...

Username: username

Password: password

41

Now You Can Use osc Like This:

osc api about

<about>

<title>Open Build Service API</title>

<description> API to the Open Build Service </description>

<revision>2.3.0.git201207101028</revision>

</about>

42

This is step 3in our workflow

Or From A Python Script,As Promised:

#!/usr/bin/env python

from osc import core, conf

conf.get_config()

url = conf.config['apiurl']

x=core.rebuild(url, 'home:joachimwerner', 'treeline', 'SLE_11_SP2', 'x86_64')

print x

3

SUSE Studio™

44

SUSE Studio™ API (1)

Documentation:

• http://en.opensuse.org/openSUSE:SUSE_Studio_API(Background information, examples etc.)

• http://susestudio.com/help/api/v2(API Reference)

Client libraries:

• http://susestudio.com/help/api/libraries_and_clients.html

45

SUSE Studio™ API (2)

• The SUSE Studio API is a RESTful API

• Authentication is done by sending aBasic HTTP Authorisation header (every time)

• You need to create an API key for that in the web UI (once)

46

Creating the API Key

Sync The Updated RPM Repository To SUSE Studio™

4

48

This Is A New API Call!

POST /api/v2/user/repositories/<id>/refresh

• As of today, this is only in SUSE Studio™ Online

• Planned for SUSE Studio Onsite 1.3

• Workaround is to use the API calls for uploading single RPMs

Rebuild The Image5

50

Trigger The Rebuild:

#!/usr/bin/env python

import requests

url = 'http://susestudio.com/api/v2/'

user = 'username'

pwd = 'api_key'

parameters = {

'appliance_id': '9999999',

'force': 'true',

'image_type': 'vmx'}

r = requests.post(url + 'user/running_builds', params=parameters, auth=(user, pwd))

print r.text

You can look this up in the web UI!

51

A build id Is Returned:

<build> <id>12345</id></build>

You can use this id to query the build status later ...

Redeploy Image6

53

Webhooks

54

Redeploy Image

• We use a webhook for that

• Here is a nice explanation on how to use them:‒ http://susestudio.com/help/api/webhooks.html

• You need your script to either start a web server (easy with Python) or e.g. run in Apache as a CGI script

• When the rebuild in Studio is finished, the webhook gets called

55

Create a New Image In OpenStack's“Glance” Image Store

Example, using the current (soon to be deprecated) glance command line:

glance image-create --name=#{data[:name] + " " + data[:build][:version]} --is-public=True --disk-format=qcow2 --container-format=bare --copy-from #{data[:build][:download_url]}

SUSE® Manager

57

SUSE® Manager API

• The SUSE Manager API uses XML-RPC.

• Authentication is done by requesting a session key with the “auth.login” method call

Documentation:https://www.suse.com/documentation/suse_manager/• API Reference

• Manager Quick Start Guides and reference documentation

58

Connecting to SUSE® Manager

#!/usr/bin/env python

import xmlrpclib

url = "https://servername/rpc/api"

user = "username"

password = "password"

api = xmlrpclib.Server(url, verbose=0)

session = api.auth.login(user, password)

print api.user.listUsers(session)

59

What's So Convenient About TheXML-RPC API?

• In Python, Java and other languages there are XML-RPC modules that allow to transparently communicate over the protocol

• Methods are just called like local methods and return Python or Java data structures (e.g. a Python list) that can be used directly

• No need for parsing the response. All XML parsing is done transparently

Sync The Updated RPM RepositoryTo SUSE® Manager

7

Update RPM On Test Instance8

Run Validation Tests9

SUSE® Cloud

64

SUSE® Cloud/OpenStack APIs (1)

• The OpenStack APIs are RESTful

• They can use Jason or XML

Documentation:• http://api.openstack.org/

(API Reference)

• http://docs.openstack.org/api/quick-start/content/(API Quick Start Guide)

• http://docs.openstack.org/api/openstack-compute/programmer/content/(Programming OpenStack Compute API with Shell and Python)

• More here:http://wiki.openstack.org/Documentation#OpenStack_APIs

65

SUSE® Cloud/OpenStack APIs (2)

• Authentication works by requesting a token from the “keystone” authentication service and using that during a session.

• Similar to Kerberos ...

66

You Can Download An RC File To Configure Your Client For A SUSE® Cloud

67

Get A Token:Example using curl, for a change. ;-)

curl -X 'POST' -v https://servername:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "username", "password":"password"}, "tenantId":"453c05b"}}' -H 'Content-type: application/json'

68

Keystone Returns A Token InJason Format

Tip:

From Python you can use the“jason” module to easily parseJason responses

69

List Available Images

curl -H "X-Auth-Token:ed6b342" https://servername:9292/v1/images

Wrapping Up...

Thank you.

71

If you haven't yet, you should really start looking into using our APIs!

Thank you.

72

Send an e-mail to joe@suse.com with the subject “APIs rock!” to request more information and stay in touch on this topic!

Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany

+49 911 740 53 0 (Worldwide)www.suse.com

Join us on:www.opensuse.org

73

Unpublished Work of SUSE. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.