netflix oss season 1 episode 3

NetflixOSSSeason 1, Episode 3

Lightning talks

Pytheas

● Web based resource and rich UI framework

● Dashboards

● Web console

● Exploring / visualizing structured + unstructured data

Pytheas - based on OSS frameworks

● Guice● Jersey● FreeMarker● JQuery● DataTables● D3● JQuery-UI● Bootstrap

Pytheas - Modular Design

● Single jar per module● Bundles

○ Guice Module○ Jersey Resources○ FreeMarker Templates○ CSS + Img + JS code○ Properties configuration○ Pytheas Module class

● Minimal bootstrap code to wire modules together into a dashboard

Pytheas

What's next - Angular JS - css/js preprocessing - Scaffold generation

https://github.com/Netflix/pytheas

Conformity Monkey

Help keep your instancesand clusters followingbest practices

Conformity Monkey

● Mark and Notify approach● Customize rules● Customize rule set● Add new rules● Customize execution time and frequency● Can migrate to other cloud platforms

Sample Conformity Rules

● Security Groups● Instance Configuration In Eureka● ELB and ASG mismatch● Cluster Naming Conventions

Conformity Monkey

https://github.com/ Netflix/SimianArmy

Zuul

● Netflix's edge tier○ In place of Nginx, Apache Http or Traffic Server

● Dynamic filters that act on HTTP requests● Filters written in Groovy / Dynamically

updated○ Any JVM language Supported

Zuul within Netflix

ELB

Zuul

API Service

Netflix Services

ELB

Zuul

Streaming Service

ELB

Zuul

Website Service

Request Lifecycle

HTTP Request

"pre" filters "routing" filter(s) "post" filters

Origin Server

"custom" filters

Http Request Http Response

"error" filters

How we Use Zuul

● Authentication● Insights● Stress Testing● Canary Testing● Dynamic Routing● Load Shedding● Security● Static Response handling● Isthmus

Zuul Components

https://github.com/Netflix/zuul

http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html

ICE: AWS Usage Tool

● Background

● Design

● Usage

● Next Steps

Background

● There is increased resource usage in the cloud

● That is very decentralized

● But lacks broad and deep insights about resource usage

ICE Design Approach

● Cloud resource usage has many dimensions: account,

region, zone, product, usage type, user-defined tags

● ICE writer tags each line item in billing file with different

tags

● ICE UI provides a way for users to query cloud resource

usage freely in arbitrary dimensions

ICE Usage

● ICE provides a birds-eye view of large and complex

cloud landscape from a usage and cost perspective.

● ICE UI is easy and intended to be used by:

● Managers

● Reservation Planners

● Engineers

● ICE OSS at https://github.com/Netflix/ice

ICE Next Steps

● Accessing billing files though IAM role and allow

multiple payer accounts

● Create Wiki page and Google Group

● Configurable cost alerts

Genie - Motivation

Genie - Configuration Service

● Abstraction of physical details of back-end Hadoop clusters in the cloud○ Status: up, out of service, or

terminated○ Cluster naming/tagging for job

submissions○ Site-specific Hadoop, Hive and

Pig configurations

Genie - Job Execution Service

● Simple API to run Hadoop, Hive and Pig jobs○ Auto-magic submission of jobs to

the right Hadoop cluster○ Abstracting away cluster details

from clients

Genie - Use Cases Enabled at Netflix

● Running 100% of all SLA jobs, and a high percentage of ad-hoc jobs

● Running nightly short-lived “bonus” clusters to augment ETL processing

● Re-routing traffic between clusters● “Red/black” pushes for clusters● Attaching stand-alone gateways to clusters

http://techblog.netflix.com/2013/06/genie-is-out-of-bottle.html

https://github.com/Netflix/genie

Lipstick

Visualization of Pig workflows at a logical level.● quickly spot common errors / inefficiencies● visually monitor execution● empowers users to support themselves● facilitates communication between

infrastructure team and users

Lipstick

Lipstick in our Architecture

Lipstick UI

Lipstick UI - Pig Workflow Closeup

https://github.com/Netflix/Lipstick

http://techblog.netflix.com/2013/06/introducing-lipstick-on-apache-pig.html

Overview and Roadmap

34 Projects and counting...

Excellent mindshare and vendor adoptionToo hard to get started for individual users

State of NetflixOSS

Where to Start?

● Recipe app(s)● AMI's - Asgard available, more coming!● Hangouts● Workshop(s)● One button install.... not there yet.

Build, Deploy, Tools

Libraries Libraries, Cont-d.

Services ZK / C* Big Data Recipe Apps

Aminator Archaius Karyon Eureka Curator Genie Recipes-RSS

Asgard Blitz4J Netflix Graph

Edda Exhibitor Lipstick Flux Capacitor

GCViz Denominator Ribbon EvCache Astyanax ACME Air

Ice EvCache RxJava Turbine Priam

Simian Army

Feign Servo Zuul CassJMeter

Pytheas Governator Hystrix

Categories

3rd Party demos

Acme Air OSS Sample and Benchmark

● Fictitious airline sample application● Mobile app (iOS/Android) and desktop browser versions

● NodeJS and Java versions with multiple NoSQL data tier impls● Benchmarking framework that allows “Web Scale” runs● Previously run at 4.3 Billion customer facing REST calls per

second (50,000 req/sec) using IBM middleware and cloud

● Re-implemented using Netflix OSS● Microservices architecture● Eureka, Hystrix, Karyon, Ribbon● Asgard

● Run at (peak) ~ 13,000 req/sec – 1.1 Billion per day● Replicate this yourself in ~ 4 hours with the OSS

Learn more at http://bit.ly/acmeairblog and from @aspyker

Eucalyptus, AWS, and NetflixOSS

More details: https://github.com/eucalyptus/eucalyptus/wiki/Netflix-OSS-Tools

Asgard manages autoscaling across both clouds.Eucalyptus/AWS compatibility makes it possible!

SomeNetflixOSS

servicescan run in

either cloud

Simian Army

Edda

Aminator

Priam

Servo30+ NetflixOSS services

...and more on the way!

Confidential and Proprietary

● Multi Data Center Support● Single Dashboard to manage OpenStack & AWS ● Access multiple Tenants in Single Login● One-Stop-Shop to manage

● Compute, Storage, Network● Security● Cloud Templates

● Asgard Framework and Architecture● Why Asgard?● Janitor Monkey for OpenStack● Future Plans● Demo

HIGHLIGHTS

Riot Games rCloud●Aminator +chef-solo provisioner●Asgard●Edda +Security Audits●Simian Army●Ice●Eureka●Archaius + Configuration Service●Ribbon, Karyon, Hystrix●And more

Progress and Updates

NetflixOSS Cloud Prize

State of the Cloud Prize

Forks of github.com/netflix/cloud-prize ~700 Mailchimp signup form complete ~120Public active submissions ~20

Two more months for procrastination!Stickers in return for Mailchimp signup...

Bootcamp/WorkshopsAMIs, Charms, Cookbooks, Ansible Models

More incentives and joint prizes

Make it easier to get started

Juju Charm Championshiphttps://juju.ubuntu.com/charm-championship/Helping solve ease-of-deploy for NetflixOSS

Mark Shuttleworth joins Netflix Cloud Prize judgesAdrian Cockcroft joins Charm Championship judges

Ubuntu Juju

Citrix are offering $10K prize for best integration with NetflixOSS

Apache Cloudstack

First prototype!

Cloud Prize Trophy

Inspiration

Cloudified

Cloud Monkey

It Bleeps!

Designed by bleeplabs.com

Q+A and Demos

Food and Drink in Building B