GRAPHCONNECT

Modeling IT Infrastuctureusing

The Assimilation Project

#AssimProj @OSSAlanR

http://assimproj.org/

http://bit.ly/AssimGC2013

Alan Robertson <alanr@unix.sh>Assimilation Systems Limited

http://assimilationsystems.com

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 2/25

GRAPHCONNECT

Upcoming Events

GraphConnect San Francisco (today!)

Open Source Monitoring Conference - Nürnberg

NSA / Homeland Security Assimilation Technical Talk

Large Installation System Administration Conference - DC

Colorado Springs Open Source User’s Group

linux.conf.au – Awesome Australian Linux Conf - Perth

Details on http://assimilationsystems.com/

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 3/25

GRAPHCONNECT

Talk Outline

● Project Overview● Infrastructure Schema● Python Object-Graph-Mapping API

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 4/25

GRAPHCONNECT

Assimilation Project History

● Inspired by 2 million core computer (cyclops64)

● Concerns for extreme scale● Topology aware monitoring● Topology discovery w/out security issues

=►Discovery of everything!

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 5/25

GRAPHCONNECT

Project Scope

Zero-network-footprint continuous Discoveryintegrated with extreme-scale Monitoring

● Continuous extensible discovery– systems, switches, services, dependencies

– zero network footprint

● Extensible exception monitoring– more than 100K systems

● All data goes into central graph database

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 6/25

GRAPHCONNECT

Discovery

Discovering● systems you've forgotten● what you're not monitoring● whatever you'd like● without setting off security alarms

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 7/25

GRAPHCONNECT

Why Discovery?

● Documentation: incomplete, incorrect● Dependencies: unknown● Planning: Needs accurate data● Best Practices: Verification needs

data● ITIL CMDB (Configuration Mgmt

DataBase)

Our Discovery: continuous, low-profile

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 8/25

GRAPHCONNECT

Why Neo4j (graph db)?

● Dependency & Discovery information: graph● Speed of graph traversals depends on size

of subgraph, not total graph size● Root cause queries graph traversals –

notoriously slow in relational databases● Visualization of relationships● Schema-less design: good for constantly

changing heterogeneous environment● Graph Model === Object Model

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 9/25

GRAPHCONNECT

Assimilation Communication Neighbor-Rings

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 10/25

GRAPHCONNECT

Ring Representation Schema

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 11/25

GRAPHCONNECT

ssh -> sshd dependency graph

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 12/25

GRAPHCONNECT

Switch Discovery Datafrom LLDP (or CDP)

CRM transforms LLDP (CDP) Data to JSON

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 13/25

GRAPHCONNECT

OGM – Object Graph Mapping

● Managing the Graph Nodes “disappears”● The Object Model is the Graph Model● Significant Improvement in Thinking

Clarity– The “mud” is gone

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 14/25

GRAPHCONNECT

OGM rules

● Don't use Constructors● Relationships replace hash tables and

object references and so on● Constructor parameter names match

attribute names

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 15/25

GRAPHCONNECT

OGM sample

@RegisterGraphClassclass Person(GraphNode): 'A Person - or at least an electronic representation of one' def __init__(self, firstname, lastname, dateofbirth=None): GraphNode.__init__(self, domain='global') self.firstname = firstname self.lastname = lastname if dateofbirth is not None: self.dateofbirth = dateofbirth else: self.dateofbirth='unknown' @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['lastname', 'firstname']

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 16/25

GRAPHCONNECT

OGM sample

@RegisterGraphClassclass TestSystem(GraphNode): 'Some kind of semi-intelligent system' def __init__(self, designation, roles=[]): GraphNode.__init__(self, domain) self.designation = designation.lower() if roles == []: roles = [''] self.roles = roles

def addroles(self, role): if self.roles[0] == '''': del self.roles[0] if isinstance(role, list): for arole in role: self.addroles(arole) elif role not in self.roles: self.roles.append(role)

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 17/25

GRAPHCONNECT

OGM sample @staticmethod def __meta_keyattrs__(): 'Return our key attributes in order of significance' return ['designation']

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 18/25

GRAPHCONNECT

OGM sample

# (seven)-[:formerly]->(Annika)

Annika = store.load_or_create(Person, firstname='Annika', lastname='Hansen')

seven = store.load_or_create(Drone, designation='SevenOfNine', roles='Borg')

store.relate(seven, 'formerly', Annika)store.commit()

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 19/25

GRAPHCONNECT

OGM sample

@RegisterGraphClassclass TestDrone(TestSystem): def __init__(self, designation, roles=[]): TestSystem.__init__(self, designation=designation) if isinstance(roles, str): roles = [roles] roles.extend(['host', 'Drone']) System.__init__(self, designation, roles=roles)

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 20/25

GRAPHCONNECT

Current State

● First release was April 2013

● Great unit test infrastructure

● Nanoprobe code – works well

● Service monitoring works● Lacks digital signatures, encryption, compression

● Reliable UDP comm code working

● Several discovery methods written

● CMA and database code restructuring near-complete

● UI development underway

● Licensed under the GPL, commercial license available

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 21/25

GRAPHCONNECT

Future Plans● Production grade by end of year

● Purchased support

● “Real digital signatures, compression, encryption

● Other security enhancements

● Much more discovery

● GUI

● Alerting

● Reporting

● Add Statistical Monitoring

● Best Practice Audits

● Dynamic (aka cloud) specialization

● Hundreds more ideas

– See: https://trello.com/b/OpaED3AT

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 22/25

GRAPHCONNECT

Get Involved!

Powerful Ideas and Infrastucture

Fun, ground-breaking project

Looking for early adopters, testers!!

Needs for every kind of skill● Awesome User Interfaces (UI/UX)● Evangelism, community building● Test Code (simulate 106 servers!)● Python, C, script coding● Documentation● Feedback: Testing, Ideas, Plans● Many others!

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 23/25

GRAPHCONNECT

Resistance Is Futile!

#AssimProj @OSSAlanR

#AssimMon

Project Web Sitehttp://assimproj.org

Blogtechthoughts.typepad.comlists.community.tummy.com/cgi-bin/mailman/admin/assimilation

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 24/25

GRAPHCONNECT

The Elder GeekGirl

GraphConnect4 October

2013

© 2013 Assimilation Systems Limited 25/25

GRAPHCONNECT

Younger GeekGirl's Computer

Running LinuxOf Course!