apache ambari bof - overview - hadoop summit 2013

17
© Hortonworks Inc. 2013 Welcome to Ambari BoF Meetup! Hadoop Summit June 2013

Upload: hortonworks

Post on 27-Jan-2015

108 views

Category:

Technology


1 download

DESCRIPTION

Apache Ambari BOF Meet Up @ Hadoop Summit 2013 Overview Slides http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/

TRANSCRIPT

Page 1: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Welcome to Ambari BoF Meetup!

Hadoop SummitJune 2013

Page 2: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Agenda: June 25, 2013

1. Ambari Overview, Current State, Recent Developments

Yusaku Sako @ Hortonworks

Siddharth Wagle @ Hortonworks

2. APIs and SPIs – How to Integrate with AmbariTom Beerbower @ Hortonworks

Nate Cole @ Hortonworks

3. Ambari BlueprintBrian Swan @ Microsoft

Sumit Mohanty @ Hortonworks

4. Ambari and OpenStackHimanshu Bari @ Hortonworks

Page 2

Page 3: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

“Make Hadoop Management Easy for Everyone”

Page 3

Provision, Manage,

Monitor

Hadoop Clusters

Hadoop Operators

Analyze Jobs

(MR, Hive, Pig, etc)

Hadoop Users

Integrate Hadoop management/monitoring capabilities to software

Software Vendors

Ambari’s Mission

Page 4: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Apache Ambari – 100% Open Source!

• Active and growing community• 50+ Contributors / 20+ Committers• 140 Ambari User Group Members

• Steady progress / short release cycle

Page 4

Release Version

Release Date

JIRAs Resolved

0.9.0 Sep 2012 402

1.2.0 Feb 2013 441

1.2.1 Mar 2013 134

1.2.2 Apr 2013 106

1.2.3 Jun 2013 515

1.2.4 Jul 2013 109+

1.2.5 Jul 2013 131+

Current Release

Today’s Demo

Page 5: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari System Architecture

5

Ambari Server

Host

Agent

gmond

Host

Agent

gmond

GangliaServer

Agent

Host

Agent

gmondgmetadgmond

Ambari Web

DB

REST/clusters

NagiosServer

Agent

Page 6: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari Today

• Live Demo of a cluster installed with Ambari 1.2.5

Page 6

Page 7: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari Architecture

Page 7

DBOrchestrator SPI

REST API

Request Dispatcher

Ambari Web

Ambari

Server

Metrics

Auth P

rovider

/clusters/services/hosts/workflows/jobs/users, …

UserStore

java

RDBMS

javascript

RDBMS

AD/ LDAP

REST API for integration

Auth Provider

Cluster Configurations

Web Client100% REST

Ambari

Agents ganglia nagios

AlertsPluggable

Service Providersfalcon

Data Mgmt

jmx

python

puppet

Page 8: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Hadoop Security - Background

Page 8

• Non secure mode = No auth• Kerberos tickets allow NN/JT to mutually

authenticate (principals/keytabs)• Guidelines for Kerberos based security

– Daemon principals and keytabs– Mapping of secure user to FS user– Re-configuring individual daemons– Re-starting services and testing for secure setup

KDC

HM

JT

NN Kerberos client Client

TT

DNKerberos client

Page 9: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Hadoop Security with Ambari

Page 9

• Ambari 1.2.5 introduces Security Wizard– Re-configuring services with security enabled– Mapping of secure users to FS user– Re-starting services in secure mode and run

smoke tests– Note: User needs to generate principals and

keytabs

• Future:– Automatically create principals and keytabs– Secure distributions of keytabs

Page 10: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Password Encryption and Web Access

Page 10

• Allow trusted signed certs to be imported into Ambari– Keystore: Open SSL with PKCS 12 keystore– User provided cert is used for server-agent

communication

• Database and LDAP passwords no longer in clear-text– JCEKS keystore to store credentials, locked with

a user-provided master key

Page 11: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

REST API – Centralized & Consistent

Page 11

Ambari REST API

Alerts

Job History

MetricsConfigurations

ConfigDB

Nagios

ServerGanglia

Server

HTTP GET, POST, PUT, DELETE

:8080

HTTP Status Code / JSON

core-site.xml

core-site.xml

Config files

Config filesConfig files

JMX

Realtime Historical*-site.xml…

Job HistoryDB

Hosts / ServicesCluster

Page 12: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari + Teradata Viewpoint Integration

Page 12

• Ambari = Key enabler for integrating Hadoop monitoring capabilities to Teradata Viewpoint

• Get Hadoop metrics from a non-Ambari deployed cluster– Ambari REST API + Custom Service

Providers

Page 13: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Stack Definitions

• Design Goals–Ambari should be able to support choice of Hadoop stacks–Ambari should enable adding new components to an existing stack

• Define which Services are available (services)• Define where to get the packages (repos)

13

S S S SStack Brepos

services

S S S SStack Arepos

services

S S S SStack CextendsStack B

repos

services

S

S+

Page 14: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari + Redhat GlusterFS Integration

• Using Ambari to deploy / manage cluster with distributed file system other than HDFS–HCFS: GlusterFS as first implementation–Pluggability with other HCFS’s–See AMBARI-1817

Page 14

MapReduce

Hive

Distributed File System

HDFS

GlusterFS

HBasePig

Other HCFS …

Page 15: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Ambari + Accumulo Integration

• Using Ambari to deploy / manage cluster with Accumulo–Google Summer of Code project–See AMBARI-1930

MapReduce

Hive

Distributed File System

HBasePig

Page 16: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Splunk + Ambari Integration

• Head over to Splunk’s booth tomorrow to see Ambari integrated into Splunk’s Management UI for Hadoop monitoring and management

Page 16

+

Page 17: Apache Ambari BOF - Overview - Hadoop Summit 2013

© Hortonworks Inc. 2013

Thanks!

• Next Up: APIs and SPIs – How to Integrate with Ambari

Page 17