streamline hadoop devops with apache ambari

Streamline Hadoop DevOps with Apache Ambari

Jayush LuniyaHadoop Summit, Tokyo

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Speaker

Jayush LuniyaStaff Software Engineer @ HortonworksApache Ambari PMCjluniya@apache.org

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Open-source platform to provision, manage and monitor Hadoop clusters

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Why Ambari?

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Why Ambari?

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari 2.0 Ambari 2.1 Ambari 2.2 Ambari 2.4April '15 Jul - Sept'15 Dec'15 - Feb'16 Aug'16 - Sept '16

0

500

1000

1500

2000

2500

3000

1690 1864

797

2251

277

206

34379

488#.#.2#.#.1GA

No.

of J

IRA

s

Ambari Releases

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari Architecture

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Exciting Enterprise Features in Ambari 2.4

New Services: Log Search, Zeppelin, Hive LLAP

Role Based Access Control

Management Packs

Grafana UI for Ambari Metrics System

New Views: Zeppelin, Storm

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Deploy

Secure

Config

Upgrade

Monitor

Extend

Operations - Lifecycle

Ease-of-Use

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Deploy

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Deploy On Premise

Ambari handles all of these combinations and makes recommendations based on host specs.

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Deploy In The Cloud

Certified environmentsSysprepped VMsHundreds of similar clusters

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Deploy with Blueprints

Systematic way of defining a cluster

Export existing cluster into blueprint/api/v1/clusters/:clusterName?format=blueprint

Configs Topology Hosts Cluster

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Create Cluster with Blueprints{ "configurations" : [ { "hdfs-site" : {

"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}

{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"

} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"

}, { "fqdn" : "worker002.ambari.apache.org"

}, … { "fqdn" : "worker099.ambari.apache.org"

} ] } ]}

1. POST /api/v1/blueprints/my-blueprint

2. POST /api/v1/clusters/my-cluster

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Create Cluster with Blueprints{ "configurations" : [ { "hdfs-site" : {

"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}

{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"

} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"

}, { "fqdn" : "worker002.ambari.apache.org"

}, … { "fqdn" : "worker099.ambari.apache.org"

} ] } ]}

1. POST /api/v1/blueprints/my-blueprint

2. POST /api/v1/clusters/my-cluster

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Create Cluster with Blueprints{ "configurations" : [ { "hdfs-site" : {

"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}

{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"

} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"

}, { "fqdn" : "worker002.ambari.apache.org"

}, … { "fqdn" : "worker099.ambari.apache.org"

} ] } ]}

1. POST /api/v1/blueprints/my-blueprint

2. POST /api/v1/clusters/my-cluster

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Create Cluster with Blueprints{ "configurations" : [ { "hdfs-site" : {

"dfs.datanode.data.dir" : "/hadoop/1, /hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : "master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : "worker-host", "components" : [ { "name" : "DATANODE" }, { "name" : "NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.5" }}

{ "blueprint" : "my-blueprint", "host_groups" :[ { "name" : "master-host", "hosts" : [ { "fqdn" : "master001.ambari.apache.org"

} ] }, { "name" : "worker-host", "hosts" : [ { "fqdn" : "worker001.ambari.apache.org"

}, { "fqdn" : "worker002.ambari.apache.org"

}, … { "fqdn" : "worker099.ambari.apache.org"

} ] } ]}

1. POST /api/v1/blueprints/my-blueprint

2. POST /api/v1/clusters/my-cluster

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Blueprints for Large Scale

Kerberos, secure out-of-the-box High Availability is setup initially for

NameNode, YARN, Hive, Oozie, etc Host Discovery allows Ambari to

automatically install services for a Host when it comes online

Stack Advisor recommendations

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

POST /api/v1/clusters/MyCluster/hosts

[ { "blueprint" : "single-node-hdfs-test2", "host_groups" :[ { "host_group" : "slave", "host_count" : 3, "host_predicate" : "Hosts/cpu_count>1” }, { "host_group" : "super-slave", "host_count" : 5, "host_predicate" : "Hosts/cpu_count>2& Hosts/total_mem>3000000" } ] }]

Blueprint Host Discovery

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Secure

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Comprehensive Security

LDAP/AD• User Auth.• Sync

Kerberos• MIT KDC• Keytab

Management

Atlas• Governance• Compliance• Data Classify• Lineage & History

Ranger• Security policies• Audit• Authorization

Knox• Perimeter Sec.• LDAP/AD• Sec. REST/HTTP• SSL

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Kerberos

Ambari manages Kerberos principals and keytabs

Works with existing MIT KDC or Active Directory

Once Kerberized, handles

Adding Services

Adding Hosts

Adding Host Components

Moving Host Components

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Role Based Access Control (RBAC)

As Ambari & organizations grow,so do security needs

Ambari integrates with external authentication systems & LDAP

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

RBAC Terms

Users belong to groups

A group has a role

Users can also have additional roles

Roles are applied to Resources. E.g.,Ambari, particular Cluster, particular View

Roles have permissionse.g., add services to cluster

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

New RBAC Roles

only view

↑, except change configs

↑, except alter cluster topologyor install components

Ambari Admin

Cluster Admin

Cluster Op

Service Admin

Service Op

Read-Only

↑, except add services, Kerberos,manage alerts & upgrades

↑, except manage permissions

all

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Config

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Config Management

Config Groups Different config settings for individual host components

Config Versioning Revert back to old configs

Smart Configs Highlight most important configs

Stack Advisor Recommend configurations

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Smart Configs

Widgets

- Sliders- Combos- Toggles- Spinners- Lists

30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Stack Advisor

KerberosHTTPSZookeeper ServersMemory Settings…High Availability

atlas.rest.address = http(s)://host:port

# Atlas Serversatlas.enabletTLS = true|falseatlas.server.http.port = 21000atlas.server.https.port = 21443

Example

Configurations

31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Upgrade

32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Upgrade Terminology

Manual Upgrade

The user follows instructions to upgrade the stack Incurs downtime

33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Upgrade Terminology

Manual Upgrade

The user follows instructions to upgrade the stack Incurs downtime

Rolling Upgrade

Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact

34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Background: Upgrade Terminology

ExpressUpgrade

Automated Runs in parallel across hosts Incurs downtime

Manual Upgrade

The user follows instructions to upgrade the stack Incurs downtime

Rolling Upgrade

Automated Upgrades one component per host at a time Preserves cluster operation and minimizes service impact

35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Check Prerequisites

Review the prereqs to confirm your cluster configs are ready

Prepare

Take backups of critical cluster metadata

Register + Install

Register the HDP repository and install the target HDP version on the cluster

Automated Upgrade: Rolling/Express

Perform Upgrade

Perform the HDP upgrade. The steps depend on upgrade method: Rolling or Express

Finalize

Finalize the upgrade, making the target version the current version

36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Process: Rolling Upgrade

ZooKeeper

Ranger

Hive

Oozie

Falcon

Kafka

Knox

Storm

Slider

Flume

Finalize or Downgrade

Clients HDFS, YARN, MR, Tez, HBase, Pig. Hive, etc.

Core Masters

Core Slaves

HDFS

YARN

HBase

37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Monitor

38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alerting FrameworkAlert Type Description Thresholds (units)

WEB Connects to a Web URL. Alert status is based on the HTTP response code

Response Code (n/a)Connection Timeout (seconds)

PORT Connects to a port. Alert status is based on response time

Response (seconds)

METRIC Checks the value of a service metric. Units vary, based on the metric being checked

Metric Value (units vary)Connection Timeout (seconds)

AGGREGATE Aggregates the status for another alert % Affected (percentage)

SCRIPT Executes a script to handle the alert check Varies

SERVER Executes a server-side runnable class to handle the alert check

Varies

39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Alert UI

40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Grafana for Ambari Metrics

Grafana as a “Native UI” for Ambari Metrics

Pre-built DashboardsHost-level, Service-level

Supports HTTPS

System Home, Servers

HDFS Home, NameNodes, DataNodes

YARN Home, Applications, Job History Server

HBase Home, Performance

FEATURES DASHBOARDS

41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Grafana includes pre-built dashboards for visualizing the most important cluster metrics.

42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

The HDFS NameNodedashboard highlightsfile system activity.

43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Storm Monitoring View

44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Log SearchSearch and index Hadoop logs!

Capabilities• Rapid Search of all Hadoop component logs• Search across time ranges, log levels, and for

keywords

Solr

LogsearchAmbari

45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Log Search

W O R K E RN O D E

L O G F E E D E R

Solr

L O G S E A R C H

U I

Solr

Solr

A M B A R I

Java ProcessMulti-output SupportGrok filters

Solr CloudLocal Disk Storage

46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Extend

47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Packs Improved Release Management:

Decouple Ambari core from stacks releases

Support Add-ons:–Release vehicle for 3rd party services, views–Self-contained release artifacts–Stack is an overlay of multiple management

packs

48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Overlay of Management Packs

inclu

ded

by

includ

ed by

included byincluded byinherits from 2.3

inherits from 2.4

inherits from 2.5

49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Management Pack++

Short Term Goals (Ambari 2.4) Retrofit in Stack Processing Framework Enable 3rd party to ship add-on services

Future Goals Management Pack Framework Include Views

50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Service Level Extensions

Service Role Command Order Service Advisor Service Repos Service Upgrade Packs

51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Future

52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Future of Ambari Cloud Focus Multiple Service Instance (Two ZK quorums) Multiple Service Versions (Spark 1.6 & Spark 2.0) YARN Assemblies Granular Upgrades: Patch, Component, Service Ambari High Availability

53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thank You