intermediate Python techniquesfor the automation journey

April 2015Jeremy Schulman@nwkautomaniac

Hardly Enough TheoryBarely Enough Code

Agenda

● Automation - Why Me?

● Learn key concepts

● Learn cool Pythonic techniques

Why Automate?

Don’t be the bottleneckEmpower your team

Increase agility and productivity

Keep the business running

Automate the mundaneFree up your time

Improve uptime and reliability

Your time and your expertise

Theory● Composers, constructors

and consumers

● Decomposition

● What matters most

● Zero, one, or infinity

What You will LearnCode● Iterators and generators

● Files - JSON, YAML, CSV

● Validation - JSON schema

● Templating - Jinja2○ Basics○ Programming logic○ Custom extensions

TheoryJust a few slides ...

Role RelationshipsDesign things providing instructions to othersCombination of art and scienceCreative processes and best-practices

Composers

Build finished goods based on the instructionsPracticed skills - years to master the craftSpecialized domain knowledge and tools

Constructors

Use finished goodsGenerally does not require much skillSometimes requires use of tools / products

Consumers

Decomposition

Everyone has favorites:● Modeling techniques ...● Diagramming techniques ...● Process techniques ...

Breaking a complex problem or system into partsEasier to conceive, understand, program and maintainAlso known as factoring

Entity Relationship DiagramFlowchart Diagram

Apply Decomposition

Consider a switch configuration file ...

● Create shared and unique data-sets● Define variables into explicit and

derived / calculated items● Understand cohesion of data and

code● Reuse the data for other purposes,

like validation and operations● Data can be used for multi-vendor

Step 1: separate "data" from "code"

Step 2: decompose "data"

Step 3: decompose "code"

Step 4: goto step 2 until manageable

Why do this?

What Matters Most?Does it do what it is supposed to do?Are there even any requirements?Is high availability / fault tolerant a consideration?

Functionality

How fast does it run?Is speed really even an issue?Performance

Does it need to handle the growth of your environment?Do you need to deal with concurrent processing?Scale

When it breaks, can you fix it (quickly)?Can other people work on it?What happens if you leave?

Maintainability

Zero, One, or Infinity

No need to do something so just don't do itZero

I must think about everyone else that might use this or have to deal with it one day. Empathy.

Infinity

Doing it only for myself and I don't need to worry about anyone else using it or what happens when I'm gone

One

CodeIterators and Generators

IteratorsLoop over things● list, dictionary, set, tuple● files - each line in text file● strings - each character in string● class objects supporting the iter

protocol● generators - see next slide

List iteration code example

Example output

Why use Generators?Lazy Evaluation

Evaluation strategy which delays the evaluation of an expression until its value is needed

Benefits● Reduce memory● Generate data on-demand● Generate “infinite” data● Performance increases ● Use as abstract objects

○ as function args○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation

GeneratorsProduce sequence of things● behave like iterators, but lazy● generator functions use yield● generator expressions are

comprehensions enclosed by parentheses

Generator expression example

Generator function example

CodeLoading Data: JSON, YAML and CSV

Loading Data from Files - JSON

Example file Example code

Example output

Loading Data from Files - YAML

Example file Example code

Example output

Loading Data from Files - CSV

Example file Example code

Example outputDo you see the differences?1. id is a string, not integer2. name is not the key

How to fix this?

Loading Data from Files - CSV (fixed!)

Example code

Example output

Demo: Putting it all togetherTask Load all of the files, of different types, into a single dictionary variable

Pythonic techniques● iterators and generators● lambda functions● reduce function● iterables.chain● functools.partial

Example Code

Demo: Filtering while loadingTaskWhile reading the data files, filter out records that match a given criteria

id >= 20

Pythonic techniques● generator● compression● … extract first/only dict item● … with if statement

Example Code

Example output

CodeTemplating with Jinja2

Templating with Jinja2 - IntroLoaded from text filesCreated dynamically by programCommonly used for configuration

{% if age > 12 %} ...

{% endif %}

Hello {{ name | capitalize }}

Hello {{ name }}

Template content is text

Programmatic controls

Filters and extensions

Variable string interpolation

Very Simple Example - Jinja2

Programmatic Controls - Jinja2{% if age > 20 %}{% else %}{% endif %}

if / then / else

{% for this in some_iterable %}{% endfor %}loop over iterators

{% set myvar = some_func() %}create variables

{% include "this file.j2" %}{% include file_name %}include other files

Load template searching directories

Jinja2 "Loader" determines how to find the template files

Jinja2 "Environment" allows you to control aspects template loading and rendering

Example: Program Control

Custom Filters - Jinja2Create a sorted key list by using the inner data dictionary

Bind the function to the Jinja2 environment "filters" dictionary

Example Output

Including other files - Jinja2

Example Output

The state_list variable using in the template was created in the python code using the groupby function, see next slide ...

Example Code: Report

https://docs.python.org/2/library/itertools.html#itertools.groupby

CodeData Validation

Data Validation - JSON SchemaUsed to validate dataVerify the syntactic contentsBoth inputs and outputsSchema is written in JSON

Other schema mechanism:● ASN.1 - SNMP● YANG - NETCONF● XSD - XML

Example Data

Example Schema

More interesting exampleData Schema

● One "record" per file● Record key is max 16 alpha-chars● Only the properties:

○ state○ id○ job

JSON Schema - Docs

http://json-schema.org/latest/json-schema-validation.html#toc

Automation Summary

Programming is a toolnot a career change

It's not about youit's about everyone else

Try being right nowrather than being right

More?● JSON - https://docs.python.org/2/library/json.html● JSON Schema - http://json-schema.org● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation● CSV - https://docs.python.org/2/library/csv.html● Jinja2 - http://jinja.pocoo.org/● itertools - https://docs.python.org/2/library/itertools.html● jq - https://github.com/stedolan/jq ● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac

Jeremy Schulman@nwkautomaniac

Thank you!