device apis: the force awakensd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brksdn-1119.pdf · device...

Device APIs: The Force Awakens

Shelly Cadora, Principal Engineer

BRKSDN-1119

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

The Olden Days

r2d2# conf t

Enter configuration commands, one per line. End with CNTL/Z.

r2d2(config)#

BRKSDN-1119 3


The New Bar: Lulea Datacenter

Agence France-Presse/Getty Images

25000 - 45000

1 Ops Engineer

Servers?????

BRKSDN-1119 4


Delfina Eberly, Director of Data Center Operations, Facebook

“We want to hang onto our talent. The way you do that is to give them the opportunity to work on high-value tasks. We want them to stay and improve. This matters to us.”

Networking Is High Value…Not CLI

BRKSDN-1119 6

• Why This Matters: APIs vs Protocols

• NETCONF

• RESTful APIs

• RESTCONF

• GRPC

• Telemetry

• Conclusion

Agenda

• Github:

• https://github.com/CiscoDevNet/clus-brksdn1119

• Jupyter (formerly known as iPython) Notebook

• Postman

• Device Toolkits / Sandboxes

• Pyang

• YANG Explorer

• YDK

Shadow Agenda (Toolchains)


Types of Interactions:

Bootstrap

Configure

Monitoring

Extend

What’s Evolving:

Manual -> Automated

Static -> Dynamic

Generic -> Custom

Distributed -> Centralized

Why:

Agility

Scale

Cost

Complexity

Evolving Our Interaction with Network OS

BRKSDN-1119 9


CLI vs API: To Whom You Are Talking Matters

CLI API

Easy to read

Introspectible

Self-describing

Forgiving

Easy to parse

Well-defined

Consistent

BRKSDN-1119 10


An API is a Contract

This is how you make

a request of me…

This is how I will

reply…

A

P

I

Picky!

“We certainly tend to focus more on programming language constructs like methods with parameterswhen describing APIs and on processes exchanging messages when describing protocols, but ultimately these concepts describe, from a different point of view, how programs communicate with each other.” --https://theamiableapi.com/2012/02/26/api-design-meets-protocol-design/

BRKSDN-1119 11


The Control Plane is All About APIs

OSPF, LLDP,

LACP, BGP….

Monitoring

Configuration

Routing

SwitchingLots of “APIs”

Lots of CLIs

BRKSDN-1119 12


SNMP is a Management API…Right?

RFC 3535: SNMP works “reasonably well for device monitoring”

• Typical config: SNMPv2 read-only community strings

• Typical usage: interface statistics queries and traps

• Empirical Observation: SNMP is not used for configuration

Lack of Writeable

MIBs

Security Concerns

Hard to Replay / Rollback

Requires special

applications

Google announces intent to disable SNMP for monitoring by 2017

Poor Scaling

Lack of Extensibility

Proprietary MIBs

BRKSDN-1119 13


Analytics-Ready Data

Networking

• MIBs

• NetFlow Records

• Unstructured Strings

• XML

Cloud / Web / IT

• JSON

• GPB

• Thrift

• …

Two Views of Encoding Data

Network-

specific

tooling

Generic

tooling in

many

languagesGoogle

Twitter

Mesos

Facebook

Cloudera

Evernote

BRKSDN-1119 14


A Model of a Network Device API

Transport

RPC / Ops

Encoding

Data Model

Raw Data Configuration and Operational

YANG -- The Boring Revolution!

JSON, GPB, XML

NETCONF, REST, RESTCONF, JSON-RPC, gRPC

Stuff we know – UDP, TCP, SSH, HTTP(S), HTTP/2

BRKSDN-1119 15


The Automation Journey

A

P

I

Must…

understand…

what’s…

happening…

curl, wget, ssh, pyang, …

SDKs

Libraries

Clients

BRKSDN-1119 16

NETCONF


• NMS

• Script

• Plugin

• Cut-n-Paste

NETCONF Uses a Client-Server Model

ServerClient

SSH

Connection-Oriented

Authenticated

Reliable

Trustworthy

Secure

BRKSDN-1119 18

Demo: NETCONF Transport


Operations: What Clients Do to ServersOperation Description

<get-config> Retrieve all or part of specified configuration datastore

<edit-config> Loads all or part of a configuration to the specified configuration datastore

<copy-config> Replace an entire configuration datastore with another

<delete-config> Delete a configuration datastore

<commit> Copy candidate datastore to running datastore

<get> Retrieve running configuration and device state information

<lock> / <unlock> Lock or unlock the entire configuration datastore system

<close-session> Graceful termination of NETCONF session

<kill-session> Forced termination of NETCONF session


NETCONF Data Stores: Target of Operations

• Named containers that may hold an entire copy of the configuration

• Not all data stores are supported by all devices

• Running is the only mandatory data store

• Not all data stores are writable

• Check the device’s capabilities

• To make changes to a non-writeable data store, copy from a writable one

• URL is supported by IOS (for config-copy)

<running> <startup> <candidate> <URL>

BRKSDN-1119 21


NETCONF Encodes Everything In XML

<?xml version='1.0'

encoding='UTF-8'?>

<rpc message-id="1001">

<get-config>

<source>

<running/>

</source>

</get-config>

</rpc>

eXtensible Markup Language

• XML describes data

• <> delimit markup text (tags)

• Machine and human readable

• W3C Recommendation

• Self-descriptive

• Verbose

• Highly Structured

BRKSDN-1119 22

Demo: NETCONF


NETCONF – A Programmer’s Perspective

Tool Language Getting it

ncclient Python library pip install ncclient

https://github.com/ncclient/ncclient

confd Python client https://developer.cisco.com/site/confD/

libnetconf C library https://github.com/CESNET/libnetconf

JNC Java library https://github.com/tail-f-systems/JNC

Free and/or Open Source Tools to Get You Started

SSH

Ctrl-C

Ctrl-V

BRKSDN-1119 24

https://github.com/ncclient/ncclient

https://developer.cisco.com/site/confD/

https://github.com/CESNET/libnetconf

https://github.com/tail-f-systems/JNC


NETCONF Consumers

DevOps Controllers

RESTful APIs


REST and NETCONF: Parallel Histories

2006

NETCONF 1.0

RFC 4741RESTful

Web APIs

Network Engineers Web, Cloud, Mobile

BRKSDN-1119 27


REST= REpresentational State Transfer

Proposed by Roy Fielding in 2000

Developed by W3C in parallel with HTTP 1.1

REST is An Architectural Style

Simple CRUD using HTTP

Stateless client-server model

Uses URIs to identify resources of interest

BRKSDN-1119 28


RESTful Operations: What Clients Do to Servers

Operation Description

GET Retrieves data from the specified object

POST Creates the object with the supplied information.

PUT Update the specified object (replace)

PATCH Updates the specified object (merge)

DELETE Deletes the specified object

BRKSDN-1119 29


REST Follows a Familiar Model

HTTP GET

HTML

Describes how data

should be displayed to

please human viewer

HTTP GET

JSON/XML

Describes data in a

format applications

can understand

{"ids":[303776224, 19449911, 607032789,

86544242, 2506725913, 17631389],

"next_cursor":0, "next_cursor_str":"0",

"previous_cursor":0, "previous_cursor_str":"0"}

Web Browsing REST API

BRKSDN-1119 30

Demo: REST in Space


Web APIs Are Trending Toward JSON

JavaScript Object Notation (JSON)

• Language-independent data format

• Well-aligned to RESTful goals

• Light-weight, open standard, easy

to use

• Compact alternative to XML

• RFC 4627

{

"name": "iss",

"id": 25544,

"latitude": 9.7362723127761,

"longitude": 66.973785291468,

"altitude": 403.44131856635,

"velocity": 27613.805418786,

"visibility": "eclipsed",

"footprint": 4422.2421795783,

"timestamp": 1463679110,

"daynum": 2457528.2304398,

"solar_lat": 19.966430937013,

"solar_lon": 276.17221638742,

"units": "kilometers"

}

BRKSDN-1119 32


JSON is Well-Aligned to Ruby, Javascript, Python…

{

"name": "iss",

"id": 25544,

"latitude": 9.7362723127761,

"longitude": 66.973785291468,

"altitude": 403.44131856635,

"velocity": 27613.805418786,

"visibility": "eclipsed",

"footprint": 4422.2421795783,

"timestamp": 1463679110,

"daynum": 2457528.2304398,

"solar_lat": 19.966430937013,

"solar_lon": 276.17221638742,

"units": "kilometers"

} JSON Object

Builtin Python Data TypeBRKSDN-1119 33

Demo: REST Script


There Are LOTS of RESTful APIs

BRKSDN-1119 35


RESTful APIs on IOS-XE, NX and ASA

XE 3.10 (CSR1kV), XE 3.14.0S (ASR1K)

NX 7.0(3)I2(2): 3132Q-XL, 3172PQ-XL, 3172TQ-XL, 3164Q, 31128PQ, NX 9000 Series

ASA 5500-X and ASAv: Winter-2014

BRKSDN-1119 36

Demo: RESTful Device APIs


REST Resources

• Deploy ASAv on VMware Fusion with REST-API: https://www.youtube.com/watch?v=Apw8QZMvD-E

• Postman for Chrome: http://www.getpostman.com/

• Chrome and Certs:http://blog.getpostman.com/2014/01/28/using-self-signed-certificates-with-postman/

38BRKSDN-1119

https://www.youtube.com/watch?v=Apw8QZMvD-E

http://www.getpostman.com/

http://blog.getpostman.com/2014/01/28/using-self-signed-certificates-with-postman/

RESTCONF


RESTCONF: Best of Both Worlds?

NETCONF/YANG

– SSH

– XML

+ JSON

+ HTTP(S)

RESTCONF

NETCONF RESTYANG

JSON

HTTP

“the simplicity of the HTTP protocol with the predictability

and automation potential of a schema-driven API”

BRKSDN-1119 40


RESTCONF is Not A Replacement for NETCONF

• Enables Web Applications to access configuration and operational data

• A simplified transaction model

• Data can be XML or JSON

• Operations include: DELETE, PATCH, POST, PUT, GET

NETCONF Operations Not Supported By RESTCONF

• Config locking

• Candidate config

• Startup config

• Validate

• Confirmed commit

Submitted for publication 5/2/16

https://tools.ietf.org/id/draft-ietf-netconf-restconf-13.txt

BRKSDN-1119 41


RESTCONF Uses Same Tools As Any RESTful APIscadora@C3P0:~$ curl -X GET -

H "Accept:application/yang.data+json, application/yang.errors

+json” -

H "Authorization: Basic Y2lzY286Y2lzY28=" http://172.30.8.5

1/restconf/data/bgp:bgp/neighbors/neighbor=172.30.8.52/confi

g/description?content=config

{

"bgp:neighbor": [

{

"neighbor-address": "172.30.8.52",

"config": {

"description": "This is a great neighbor"

}

}

]

}

CURL: Command-line URL (Linux Utility)

Dear 172.30.8.51,

Please give me the configured description for your BGP neighbor 172.30.6.52 in JSON format.

Thanks!

BRKSDN-1119 42

http://172.30.8.51/restconf/data/bgp:bgp/neighbors/neighbor=172.30.8.52/config/description?content=config

Demo: RESTCONF with POSTMAN


gRPC: Like REST But Different

Runs over HTTP/2• Optimize for page load time

• Server push, header compression, multiplexing

• RFC 7540 (May 2015)

• Preserves most HTTP1.1 syntax

Defines Services (“RPCs”)

Encodes Using Google Protocol Buffers (“protobufs”)

• Services and Messages

• Auto-generate code in many languages http://www.grpc.io/docs/#hello-grpc

BRKSDN-1119 45


GPB: Google-Style Serialization

Design Goals

• Simplicity

• Performance

• Forward/Backward Compatibility

Non-Goals

• Human-Readable

• Self-Describing

• Text-based

Google Protocol Buffers (GPB)

Call them

“protobufs”

for short “Protocol buffers are Google's language-neutral, platform-neutral, extensible

mechanism for serializing structured data – think XML, but smaller, faster, and

simpler.”

BRKSDN-1119 46


Protobufs are Not Self-Describing

• GPB serialized data has no descriptive tags or attributes.

• Each side needs to know in advance the structure of the data being sent.

• The “magic decoder ring” is a .proto file.

BRKSDN-1119 47


gRPC / GPB Resources

• GPB: https://developers.google.com/protocol-buffers/

• GPB on github: https://github.com/google/protobuf

• GPB overview: http://blogs.cisco.com/sp/streaming-telemetry-with-google-protocol-buffers

• gRPC specification: http://www.grpc.io

• gRPC on XR: https://github.com/CiscoDevNet/grpc-getting-started

48BRKSDN-1119

https://developers.google.com/protocol-buffers/

https://github.com/google/protobuf

http://blogs.cisco.com/sp/streaming-telemetry-with-google-protocol-buffers

http://www.grpc.io

https://github.com/CiscoDevNet/grpc-getting-started

Data Models & YANG


Same Data, Different Data Models

Transport

RPC / Ops

Encoding

Data Model

Raw Data

BRKSDN-1119 50


Why Data Models?

• Separation of models from RPC, Transports, Encoding

• Devices become self-describing:

• Including definition of constraints

• We can apply tool chains:

• Simpler to generate API language bindings

• Simpler to setup data transformation pipelines

BRKSDN-1119 51


YANG: Yet Another Next-Gen (IETF RFC 6020)

YANG Model XML/JSON/GPB

BRKSDN-1119 52


YANG Is A Modeling Language

module ietf-interfaces {import ietf-yang-types {

prefix yang; }container interfaces {

list interface { key "name"; leaf name {

type string;}leaf enabled {

type boolean; default "true";

}…

Edited

for

Brevity

Self-contained top-level hierarchy of nodes

Import or define data types

Leaf nodes for simple data

Lists for sequence of entries

Containers group related nodes

Other YANG Features

• RO or RW

• Optional nodes (?)

• Choice

• Augment

• When

• Arbitrary XML

• RPC

• etc

BRKSDN-1119 53


Useful Tree View of IETF-Interfaces

module: ietf-interfaces+--rw interfaces| +--rw interface* [name]| +--rw name string| +--rw description? string| +--rw type identityref| +--rw enabled? Boolean

+--ro interfaces-state+--ro interface* [name]

+--ro name string+--ro type identityref+--ro oper-status enumeration+--ro last-change? yang:date-and-

time...

54BRKSDN-1119https://github.com/mbj4668/pyang

module ietf-interfaces {import ietf-yang-types {

prefix yang; }container interfaces {

list interface { key "name"; leaf name {

type string;}leaf enabled {

type boolean; default "true";

}…

https://github.com/mbj4668/pyang


Where Yang Models Come From

Vendors IETF OpenConfig Other

55BRKSDN-1119

Who Writes Them

How You Get Them Real-Time (using NC)

Offline (repository)

Demo: PYANG and Github Models


Models are Abstract, Instantiation is Concrete

module: ietf-interfaces+--rw interfaces| +--rw interface* [name]| +--rw name string| +--rw description? string| +--rw type identityref| +--rw enabled? Boolean+--ro interfaces-state

...

57BRKSDN-1119

<interfaces> <interface>

<name>eth0</name><enabled>false</enabled>

</interface></interfaces>

Model Instantiation

{"ietf-interfaces:interface": {

"name": "GigabitEthernet1","enabled": true,

}}


Software: Automate Instantiation From Models

BRKSDN-1119 58

Demo: YANG Explorer


YANG Automation Workflow (Hard)

YANG • github

• Device

XML • YANG explorer

Code • app

Required Skill Set

• SSH, HTTP(/2)

• NETCONF, RESTCONF, gRPC

• JSON, XML, GPB

• Data Model / YANG

• Networking

• Programming

BRKSDN-1119 60


YANG Automation Workflow (Easier)

YANG

Code

Required Skill Set

• Data Model / YANG

• Networking

• Programming

BRKSDN-1119 61


YDK and YDK-GEN

ietf-interfaces.yang ietf-interfaces.py

YDK-GEN

https://github.com/CiscoDevNet/ydk-gen

YDK-PYhttps://github.com/CiscoDevNet/ydk-py

BRKSDN-1119 62

• Python Classes Today

• Other Languages In Future

Demo: YDK


YANG Resources

• YANG Models: https://github.com/YangModels/yang

• Pyang: https://github.com/mbj4668/pyang

• YANG Explorer: https://github.com/CiscoDevNet/yang-explorer

• YDK-GEN: https://github.com/CiscoDevNet/ydk-gen

• YDK-PY: https://github.com/CiscoDevNet/ydk-py

• YDK samples: https://github.com/CiscoDevNet/ydk-py-samples

• YDK Community: https://communities.cisco.com/community/developer/ydk

• YDK Devnet: https://developer.cisco.com/site/ydk/

64BRKSDN-1119

https://github.com/YangModels/yang

https://github.com/mbj4668/pyang

https://github.com/CiscoDevNet/yang-explorer

https://github.com/CiscoDevNet/ydk-gen

https://github.com/CiscoDevNet/ydk-py

https://github.com/CiscoDevNet/ydk-py-samples

https://communities.cisco.com/community/developer/ydk

https://developer.cisco.com/site/ydk/

Telemetry


Network Data is Bottlenecked

sensing &

measurement

Where Data Is Created Where Data Is Useful

syslog

SNMP

CLIstorage &

analysis

BRKSDN-1119 66


The New Paradigm: Free the Data

sensing &

measurement

Where Data Is Created Where Data Is Useful

storage &

analysis

As Much Data

As Fast

As Useful

As Easy

As Possible

BRKSDN-1119 67


Initial Goal: Validate the Big Data Proposition

Push Not Pull

SNMP Pull Telemetry Push

Model Based Feature Feature Feature

✔×

BRKSDN-1119 68


Instruction on:

• What data to collect

• With what cadence

• And send to where

Ultra-high level picture

RouterReceiving

unit

BRKSDN-1119 69



Router

Instruction on:




Receiving

unitTable 3 Table 2 Table 1

BRKSDN-1119 70

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSDN-1119 71

Instruction on:




RouterReceiving

unitTable 3 Table 2

Interface ifInErrors ifOutErrors ifHCOutOctets …

HundredGigabitEthernet 0/1/0/2 10 0 123456789 …

Bundle-Ether 42 3 0 234567890 …

… … … … …

Table 1


“I am the interface counters table”


Model Driven Telemetry (IOS XR 6.1.1)

RP/0/RP0/CPU0:obiwan#show run telemetryTue May 3 18:01:19.745 UTCtelemetrydestination 1encoding self-describing-gpbprotocol grpdestination-address ipv4 172.30.8.4 2122

!sensor-group 1sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-

statistics/interfaces/interface/latest/generic-counters!subscription 1sensor-group-id 1 sample-interval 30000destination-id 1

Follows the Open Config Telemetry Config YANG model

Native YANG model

for interface statistics

gRPC with GPB encoding

BRKSDN-1119 72


Data Plane: Encoder Output Remote

Management

Station

{"collection_id": "6","base_path":

"Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters",

"collection_start_time": "1462297164163","msg_timestamp": "1462297164163","fields": [

{"name": "InterfaceName","string_value": “GigabitEthernet0_0_0_0"

},{

"name": "PacketsReceived","uint64_value": "0"

},{

"name": "BytesReceived","uint64_value": "0"

},<snipped for brevity>

JSON GPB

BRKSDN-1119 73

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 74BRKSDN-1119

Different Customers, Different Models

Logstash

ElasticSearch

Kibana

ST Input Codec

Output Codec

Kafka

BYO Black

Box

S

S

T

Custom Open Source, Customizable

Proprietaryor OS-based

S

S

T

Commercial Stack

Prometheus Panda


Lesson Learned: It’s Not Hard to Beat SNMP

• 10 second poll / push

• 3 pollers / telemetry receivers

• 288 100Gig E Interfaces (Line Rate)

• SNMP: IF-MIB (query by row)

BRKSDN-1119 75


Fast Data -> High Resolution

Telemetry – 10 Second Push

SNMP – 1 Minute Polling

G0/0/0/0

G0/0/0/1

Is the bundle

polarized?

BRKSDN-1119 76


Telemetry Resources

• Ten Lessons From Telemetry (NANOG 67): https://youtu.be/tIN8BjHwpNs

• Basic Telemetry Demo: https://youtu.be/F_S9-ctNFe0

• Telemetry Demo with ELK: https://www.youtube.com/watch?v=9El0PCtNxtg

• Collector Script: https://github.com/cisco/bigmuddy-network-telemetry-collector

• Collector Stacks: https://github.com/cisco/bigmuddy-network-telemetry-stacks

• XR Docs: https://xrdocs.github.io/telemetry/

• Telemetry Demo with Signal FX: https://www.sdxcentral.com/resources/sdn-demofriday/cisco-ios-xr-signalfx-demo-monitoring-your-modern-network

77BRKSDN-1119

https://youtu.be/tIN8BjHwpNs

https://youtu.be/F_S9-ctNFe0

https://www.youtube.com/watch?v=9El0PCtNxtg

https://github.com/cisco/bigmuddy-network-telemetry-collector

https://github.com/cisco/bigmuddy-network-telemetry-stacks

https://xrdocs.github.io/telemetry/

https://www.sdxcentral.com/resources/sdn-demofriday/cisco-ios-xr-signalfx-demo-monitoring-your-modern-network

Conclusion


Model-Driven Programmability Stack

Models

Tooling

Apps

Protocol TransportEncoding

Model-Driven APIs

YANG Development Kit (YDK)

YANG Data Models

App1 App2 App3

Configuration

Streaming

Telemetry

NetworkNETCONF RESTCONF gRPC XML JSON GPB SSH HTTP/2

BRKSDN-1119 79


Yoda

“For my ally is the Force. And a powerful ally it is.”


How Will You Use the Force?


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKSDN-1119 82

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSDN-1119 83

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you


Network Programmability Cisco Education OfferingsCourse Description Cisco Certification

Integrating Business Applications with Network

Programmability (NIPBA);

Integrating Business Applications with Network

Programmability for Cisco ACI (NPIBAACI)

Learn networking concepts, and how to deploy and troubleshoot

programmable network architectures with these self-paced courses.

Cisco Business Application

Engineer Specialist Certification

Developing with Cisco Network Programmability

(NPDEV);

Developing with Cisco Network Programmability

for Cisco ACI (NPDEVACI)

Learn how to build applications for network environments and effectively

bridge the gap between IT professionals and software developers.

Cisco Network Programmability

Developer Specialist Certification

Designing with Cisco Network Programmability

(NPDES);

Designing with Cisco Network Programmability

for Cisco ACI (NPDESACI)

Learn how to expand your skill set from traditional IT infrastructure to

application integration through programmability.


Design Specialist Certification

Implementing Cisco Network Programmability

(NPENG);

Implementing Cisco Network Programmability

for Cisco ACI (NPENGACI)

Learn how to implement and troubleshoot open IT infrastructure

technologies.


Engineer Specialist Certification

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKSDN-1119 87

http://learningnetwork.cisco.com/

mailto:[email protected]

device apis: the force awakensd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brksdn-1119.pdf · device...

Documents