headquarters / operating unitvia campanini 620124 milanotel: +39 02/66.732.1 – fax: +39 02/66.732.300

operating unitp.zza san benedetto da norcia 3300040 pomezia (rm)tel: +39 06/9826.9600 – fax: +39 06/9826.9680

vat number: 12938200156c.c.i.a.a. milano n.1599095incorporation number 12938200156capital stock € 2.418.433,00

Roberto Polli, Solutions Architect18.04.2015

Test Driven Deployment with (i)python and nosetest

2

Agenda

• Who? What? Why?• Who fears Continuous Delivery?• Test Driven Deployment• A TDD Case Study: the LDAP infrastructure• Document with code• From test to setup• Contributing to opensource

3

Who? What? Why?

• Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java and Python. Red Hat Certified Engineer and Virtualization Administrator.

• Par-Tec – Proud sponsor of this talk ;) provides expertise in IT Infrastructure & Services and Business Intelligence solutions + Vertical Applications for the financial market. Contributes to various FLOSS.

• Implement a Test Driven Deployment strategy on big, legacy infrastructures: not an LDAP talk!

4

Who fears Continuous Delivery?

• Legacy infrastructures:– Compliance to laws and regulations– "Old" software / libraries– resilient to new technologies ;)

• Enterprise infrastructures– Slow release cycle– Eterogeneous environment– Different IT departments involved

5

Who fears Continuous Delivery?

• Legacy infrastructures:– Compliance to laws and regulations– "Old" software / libraries– resilient to new technologies ;)

• Enterprise infrastructures– Slow release cycle– Eterogeneous environment– Different IT departments involved

6

Who fears Continuous Delivery?

• Why not add...

• requiring...

• and:– add external repositories → Licenses, NOC– review upgrade policies SOC→– modify network policies NOC→– audit new services SOC→

2.7+

7

Who fears Continuous Delivery?

• Switch to Continuous Delivery is hard:– customers are reluctant to distruptive changes;– operation teams must adapt to new technologies.

• Use Test Driven Deployment and python:– testing is not disruptive;– ipython is easy to learn and use

– shell compatibility – self-completion

8

Test Driven Deployment

• Write down infrastructural requirements

• Create tests for each requirement– test configurations– test behavior

• Setup/Update infrastructure

• Reuse tests for monitoring– monitor configuration changes too!

9

Test Driven Deployment with python!

• Python is the way:– already installed on all Linux servers– PSL: batteries included (POP, SMTP, IMAP, HTTP, Telnet, ...)– modules already in distribution repositories– further modules packaged as a zip file– interactive

• TL;DR– no external repositories– no policy changes– no external teams involved

10

TDD Case study

• How we apply TDD to this infrastructure:– ~30 LDAP server– replication and proxy topologies– different people involved in management

11

TDD Case Study

• Achievements– eliminate maintenance issues– simplify major upgrades– identify issues and their effects on the platform– document with code

• Without reinventing the wheel!– python-ldap in all linux distros– reused bugfix script from 389 Team

• Small footprint on the platform:– python already present on all nodes– further libs packed as one .zip file

12

TDD Strategy: Collect and Create

• Collect all LDAP-related platform requirements, eg:– passwords expire periodically;– fine-grained user ACL;– backend/frontend replication;– certificates validity;– …

• Create tests for every requirement– failures explicits the symptoms on the platform

eg. user can't change password– human-readable configurations

eg. master-slave relations between hosts

13

Nose: tests are easy

• Run every file and function starting with "test_"#nosetests ­v 

• Filter tests with -m:#nosetests ­v  ­m "replication" 

USE EXPLICIT NAMES!

#

# test_ldap.py 

#

def test_proxy_access():

  ...

def test_replication_user_expiration():

  ...

def test_password_policy_tree_A():

  ...

15

Writing tests: Document with code

• Describe the infrastructure with dicts, eg:

## Specify slaves for each master#replication_t = { 'be5.foo.it': [ 'fe9.foo.it', 'fe10.foo.it'],'be6.foo.it': [ 'fe9.foo.it', 'fe10.foo.it'],'be7.foo.it': [ 'fe9.foo.it', 'fe10.foo.it'],}

17

Writing tests: for readability

• Testing functions should be simple

• Parametrize your tests with Nose yield feature

def test_replica():    for master, slaves in replication_t.items():        for slave in slaves:            # generate a single test case            # for each master­slave agreement            yield has_replica, master, slave

# nosetests ­v ­m replicatest_replica('be1.foo.it', 'fe9.foo.it') ... oktest_replica('be1.foo.it', 'fe10.foo.it') ... FAILtest_replica('be1.foo.it', 'fe11.foo.it') ... ok

def has_replica(master, slave):    """test if master replicates to slave"""

18

Writing tests: explicit is better than implicit

• Be verbose with hostnames and features

• Goals– clarity versus operation team– avoid this:

'be{hostid}.foo.it'.format(hostid=hostid)

– can use socket.gethostname directly– add hosts from other domains, eg: be1.foo.recovery

backends_config = { 'be1.foo.it': [  {'name': 'userDb', 'cacheSize' : 1*GB, ..},  {'name': 'adminDb', 'cacheSize' : 1*MB, ..}, ], ..}

backends_config = { '1':  [ ['user', (1<<30)], ['admin', (1<<20)]

 ],.. }

19

Writing tests: for usability

• Developed the first tests, iteratively improving the ldap library.

• Evolution of code in time

– before

– after

20

Writing test: for usability

• Operation team learnt python via auto-completion and test code.• Code was written to be user-friendly for interactive use:

>>> from dsadmin import DSAdmin

# verbose output via logs>>> logging.basicConfig(level=DEBUG)

# connect to server using uris>>> cli = DSAdmin("ldaps://be1.foo.it")

# get all databases on a server#  easier than bash# ldapsearch … '(&(objectclass=nsBackendInstance)(..))'>>> cli.backend.list() 

21

Test completed, and now?

...write the setup code: next deployments will go unattended!

• Use logging to produce an installation log.

• Reuse infrastructure tests as functional tests for the setup code.

• Use the library with Ansible in new projects :D

def create_proxy_user():     ... def create_backends():   my_backends = backends[getfqdn()]   for backend in my_backends:     ...create_backend with the given attributes...

22

Contribute to open source

• Reusing OpenSource gave us the opportunity to contribute, widespread python...

• And meet interesting people!

headquarters / operating unitvia campanini 620124 milanotel: +39 02/66.732.1 – fax: +39 02/66.732.300

operating unitp.zza san benedetto da norcia 3300040 pomezia (rm)tel: +39 06/9826.9600 – fax: +39 06/9826.9680

Thank you!