apache ambari - what's new in 2.4

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ambari 2.4.0What’s NewAugust 2016


What is Apache Ambari?

A completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop.


What Ambari Does

Simplified Installation, Configuration and Management

Centralized Security Setup

Full Visibility into Cluster Health

Highly Extensible and Customizable

• Wizard-driven and automated cluster provisioning• Smart Configurations and Cluster Recommendations• Automated Rolling and Express cluster upgrades

• Reduce complexity to administer security across the platform• Automate setup Kerberos• Simplify the configuration of Apache Ranger

• Predefined alerts based on operational best practices• Advanced metrics visualization with Grafana

• Seamlessly fit into your enterprise environment• Bring custom Services under management via Ambari Stacks• Customize the UI with Ambari Views


What’s New in Ambari 2.4

Alerts: Customizable SCRIPT Parameters (AMBARI-14898)

Alerts: Retry Check Counts (AMBARI-15686)

Alerts: New HDFS Alerts (AMBARI-14800)

New Host Page Filtering (AMBARI-15210)

Remove Service (AMBARI-14759)

Support for SLES 12 Technical Preview (AMBARI-16007)

Stability: Database Consistency Checking (AMBARI-16258)

Customizable Ambari Log + PID Dirs (AMBARI-15300)

New Version Registration Experience (AMBARI-15724)

Log Search Technical Preview (AMBARI-14927)

Operational Audit Logging (AMBARI-15241)

Role-Based Access Control (AMBARI-13977)

Automated Setup of Ambari Kerberos (AMBARI-15561)

Automated Setup of Ambari Proxy User (AMBARI-15561)

Customizable Host Reg. SSH Port (AMBARI-13450)

Core Features Security Features

View URLs (AMBARI-15821), View Refresh (AMBARI-15682)

Inherit Cluster Permissions (AMBARI-16177)

Remote Cluster Registration (AMBARI-16274)

Views Framework Features


AgendaWhat’s New in Ambari 2.4.0

Feature Highlights: Alert Retry Check Counts


Alert Check Counts

Customize the number of times an alert is checked before dispatching a notification Avoid dispatching an alert notification (email, snmp) in case of transient issues


Configuring the Check Count

Set globally for all alerts, or override for a specific alert

Global Setting

Alert Override


State Change Types

SOFT state changes do not perform a dispatch HARD state changes (to non-OK) perform dispatch Regardless of change:

– The Ambari Web UI will show the current state (OK/WARN/CRIT)– The state change is written to ambari-alerts.log

2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)

Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.


Example: Check Count = 3

Check 1/3State: OK

Change: n/a

Check 1/3State: OK

Change: n/a

Check 1/3State: CRIT

Change: SOFT


Change: n/a


Change: HARD

Check 1/3State: OK

Change: HARD

DISPATCH

Check Interval Check Interval Check Interval Check Interval Check Interval

no state changestate changes to CRIT

performing multiple checks

back to OK



Feature Highlights: Alert Customizable Params


Alert Types and Thresholds

Ability to customize Thresholds for SCRIPT and SERVER alerts Ability to customize Connection Timeout for METRIC alerts

Alert Type Description Thresholds (units)

WEB Connects to a Web URL. Alert status is based on the HTTP response code. Response Code (n/a)Connection Timeout (seconds)

PORT Connects to a port. Alert status is based on response time. Response (seconds)

METRIC Checks the value of a service metric. Units vary, based on the metric being checked.

Metric Value (units vary)Connection Timeout (seconds)

AGGREGATE Aggregates the status for another alert. % Affected (percentage)

SCRIPT Executes a script to handle the alert check. Varies

SERVER Executes a server-side runnable class to handle the alert check. Varies

NEW!


Alerts: Customizable METRIC Connection Timeout

Ability to set Connection Timeout threshold via Ambari Web UI

NEW!


Alerts: Customizable SCRIPT Thresholds

Ability to set various thresholds via Ambari Web UI


Alerts: NEW!!! Ambari Server Performance Alert

Measures the Ambari Server REST API and Backend Database response



Feature Highlights: New HDFS Alerts


New HDFS Alerts Watch Trends

NameNode Client RPC Queue Latency (Hourly/Daily) NameNode Client RPC Processing Latency (Hourly/Daily) NameNode Service RPC Queue Latency (Hourly/Daily) NameNode Service RPC Processing Latency (Hourly/Daily) NameNode Heap Usage (Daily/Weekly) HDFS Storage Capacity Usage (Daily/Weekly)



Feature Highlights: New Host Filtering


New Host Filtering Control in Ambari Web

Ability to perform complex host filtering from Ambari Web Make it easier to find hosts

NEW!


Search by Host Attribute, Service or Component


Host Attribute Filtering

Host Name IP Host Status Cores RAM Stack Version + Version State Rack


Service Filtering


Component Filtering


Host Filter: Examples



Feature Highlights: Remove Service


Remove Service

Ability to perform Remove Service from Ambari Web Eliminates need to use Ambari REST API Checks for Service dependencies Service must be stopped All configuration information and history is also removed This operation is not reversible

NEW!



Feature Highlights: Other Items


Customizable Ambari Log + PID Dirs (AMBARI-15300)

Ambari Server and Agents write log activity output to log files and use a PID-file that contains the process identification number (PID) for their running process.

Log Location PID Location

Ambari Server /var/log/ambari-server/ambari-server.log /var/run/ambari-server/ambari-server.pid

Ambari Agent /var/log/ambari-agent/ambari-agent.log /var/run/ambari-agent/ambari-agent.pid


Customize Ambari Server Log + PID

vi /etc/ambari-server/conf/ambari.properties

pid.dir=/var/run/ambari-server

vi /etc/ambari-server/conf/log4j.properties

ambari.log.dir=${ambari.root.dir}/var/log/ambari-server

Ambari Server PID Ambari Server Log

1. Stop Ambari Server prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory

ownership + permissions to allow the Ambari Server process access.


Customize Ambari Agent Log + PID

vi /etc/ambari-agent/conf/ambari-agent.ini

[agent]

logdir=/var/log/ambari-agent

piddir=/var/run/ambari-agent

1. Stop Ambari Agent prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory

ownership + permissions to allow the Ambari Agent process access.


Customizable Host Registration SSH Port

Customize SSH Port when performing Host Registration automatically

NEW!


Stability: Database Consistency Checking

On Ambari Server start, Ambari runs a database consistency check looking for issues. If any issues are found, Ambari Server start will abort and a message will be printed to

console “DB configs consistency check failed.” Check Ambari Server log file for more details:/var/log/ambari-server/ambari-server-check-database.log

Ability to “skip” check and force Ambari Server startambari-server start --skip-database-check

Important: if you “skip” the check to force Ambari Server start, do not make any changes to your cluster topology or perform a cluster upgrade until you correct

the database consistency issues.



Feature Highlights: View Framework Enhancements


View URLs (AMBARI-15821)

Ability to create a “short URL” or “vanity URL” for view instances Provide users with a non-version or instance specific URL to a view

/#/main/views/{viewName}/{viewVersion}/{viewInstanceName}/#/main/view/{viewName}/{shortURL}


View Refresh (AMBARI-15682)

Automatically deploy new views into Ambari Server w/o a restart

1. Copy view archive to: /var/lib/ambari-server/resources/views/

2. Ambari Server detects the new view, automatically extracts + deploys

3. View is available for creating instances

4. Click “Refresh” in Views UI


Remote Cluster ConfigurationAMBARI-16274


Background: View <-> Cluster Communication

Deployed Views “talk” with cluster using REST APIs (as applicable)

CLUSTER

ATS

RM

Ambari Server

Tez UIView

Tez UI View talks with cluster using

REST APIs toATS and ResourceManager

Ambari DB

LDAPAuthN


Background: Operational vs. Standalone Ambari Server

Ambari Agent

Host

Ambari Agent

Host

Ambari Agent

Host

Standalone Ambari ServerOne or More Ambari Server Instances

No Agents, no requirement to operate the cluster

Operational AmbariOne Ambari Server Instance

Talking with Agents, Managing the cluster

Ambari Server

Ambari DB

LDAPAuthN

Ambari Server

Ambari DB

LDAPAuthN


Background: Local Cluster vs. Non-Local

Ambari Server

Ambari DB

LDAPAuthN

Ambari Server

Ambari DB

LDAPAuthN

Standalone Ambari ServerOne or More Ambari Server Instances

No Agents, no requirement to operate the cluster

Operational AmbariOne Ambari Server Instance

Talking with Agents, Managing the cluster

LOCALCLUSTER

NON-LOCAL

CLUSTER


Introducing Remote Cluster Configuration (AMBARI-16274)Option DescriptionLocal Cluster When you select this Local Cluster option, Ambari will automatically determine the

cluster configuration properties needed for the view instance.Criteria:• Ambari Server running the views is also managing the cluster

Remote Cluster When you select Remote Cluster option, Ambari will automatically determine the cluster configuration properties needed for the view instance.Criteria:• The cluster is not local to the Ambari Server running the views (i.e. Standalone)• Cluster is being managed by Ambari

Custom When you select Custom option, you must enter all configuration information, and are responsible for updating if the cluster configuration changes.Criteria:• The cluster running the view is not local to the Ambari Server• The cluster is not being managed by Ambari

NEW!


Local vs Remote View Configuration

Ambari Server

Views

Cluster

Ambari Server

Views

ClusterAmbari Server

LOCAL CLUSTER

REMOTE CLUSTER

Operational Ambari

Manages cluster

Standalone Ambari

Manages cluster

Talks to cluster

Obtains view config

Obtains view config

Talks to cluster

Operational Ambari


View Configuration: Minimizing Need for Custom

Cluster Config Ambari Server Cluster Mgmt Ambari 2.2 or Earlier Ambari 2.4

No HA, No Kerberos Operational Ambari Local Local

HA or Kerberos Operational Ambari Custom Local

No HA, No Kerberos Standalone Ambari Custom Remote

HA or Kerberos Standalone Ambari Custom Remote

No HA, No Kerberos Standalone Non-Ambari Custom Custom

HA or Kerberos Standalone Non-Ambari Custom Custom


Inherit Cluster PermissionsAMBARI-16177


Inherit Cluster Permissions (AMBARI-16177)

Ability to automatically grant View “Use” permission based on Cluster role Note: Option is only available when using a Local Cluster Configuration

Explicitly grant users and groups Use

permission

Automatically grant users and groups Use permission based on

Cluster roles



Feature Highlights: Log Search

TECH PREVIEW


Log Search

Solr

A M B A R ILog

Search

Search Cluster Component Logs from within Ambari

Goal: When issues arise, be able to quickly find issues across all components

⬢ Capabilities– Rapid Search of all cluster component logs– Search across time ranges, log levels, and for keywords

⬢ Core Technologies: – Apache Ambari– Apache Solr– Apache Ambari Log Search

Tech Preview


Log Search Architecture

A M B A R I

L O G F E E D E R

L O G F E E D E R

L O GF E E D E R

L O GF E E D E R

L O G F E E D E R

L O G F E E D E R

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

W O R K E RN O D E

Solr

L O G S E A R C H

U I

Tech Preview


Log Search Details

W O R K E RN O D E

L O G F E E D E R

Solr

L O G S E A R C H

U I

Solr

Solr

A M B A R I

Java ProcessMulti-output SupportGrok

Solr CloudLocal Disk StorageTTL

Tech Preview


Considerations

Log Feeders are CPU intensive, consider 1 dedicated core Solr instances should use dedicated hardware with at least 32GB of RAM dedicated to

the Solr instance By default, logs will age out after 7 days

Tech Preview



Feature Highlights: RBAC


New Role Based Access Control

Introducing new “roles” for more granular division of control for cluster operations

Old Permission New Role Notable Permissions

Operator Cluster Administrator Full operational control, including upgrades. Ambari Admins are implicitly granted this Role.

Cluster Operator Adding and removing hosts.

Service Administrator Manage configurations, move components.

Service Operator Service stop and start and service-specific operations such as HDFS Rebalance.

Read-Only Cluster User View cluster service and host information.

Note: Users flagged as “Ambari Administrators / Ambari Admins” are implicitly granted Cluster Administrator permission.


Managing Cluster RolesAssign roles to users or groups

Manage roles in Block or List View

layouts


Managing Cluster Roles View users or groups

Change current role assignment



Feature Highlights: Security Enhancements


Summary of Security Enhancements

Automatic Setup of Ambari Server as a Proxyuser Automatic Setup of Ambari Server for Kerberos


Automatic Setup of Ambari Server as a Proxyuser


Background: Proxyusers

HDFS and WebHCat (as part of Hive) support the concept of a Proxyuser Proxyuser allows UserA to access the service on behalf of UserB (i.e. the proxyuser is

allowed to impersonate other users) Proxyuser is a commonly used capability of Hadoop

HDFS

“UserA” is setup as a proxyuser

UserA can access HDFS as “UserA” on behalf of “UserB”

HDFS ops performed are as “UserB”


Background: HDFS Proxyuser Setup

A proxyuser needs to be configured in core-site.xml configuration:hadoop.proxyuser.{proxyuser-name}.hosts

hadoop.proxyuser.{proxyuser-name}.groups

If these settings are not present, impersonation will not be allowed and connection to the service via proxyuser will fail


Background: Ambari + Proxyuser

Ambari Views use proxyuser to access the cluster (such as Hive View and Pig View) Ambari Server needs to access a service on behalf of an authenticated user

Ambari Server HDFS

(running as user “ambari”)

“joe” authenticates to Ambari

(setup for proxyuser “ambari”)hadoop.proxyuser.ambari.hosts=*hadoop.proxyuser.ambari.groups=*

Ambari Server can talk to HDFS as “ambari” proxyuser on behalf of “joe”

Configuration of proxyuser is commonly “missed” when setting up Ambari Views


New: Automatic Ambari Server Proxyuser Setup

Proxyuser configurations are automatically added for HDFS and WebHCat For example: Ambari Server as running as “ambari”, the following configurations are

added during HDFS service installhadoop.proxyuser.ambari.hosts

hadoop.proxyuser.ambari.groups


Automatic Setup of Ambari Server for Kerberos


Background: Hadoop + Kerberos

Strongly authenticating and establishing a user’s identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster.

Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).

Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenticate with each other to avoid potential malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.


Background: Hadoop + Kerberos

Service Component

A

Service Component

B

Hadoop Cluster

KDC

keytabkeytab

Service Component

C

keytab

Service Component

D

keytab

Service Component

X

Service Component

X

keytabkeytab

Service Component

X

keytab

Service Component

X

keytab

Kerberos is used to secure the

Components in the cluster. Kerberos

identities are managed via

“keytabs” on the Component hosts.

Principals for the

cluster are managed in

the KDC.


Background: Automated Kerberos Setup with Ambari


Background: Principal and Keytab Generation & Distribution

1. User provides KDC Admin Account credentials to Ambari

2. Ambari connects to KDC, creates principals (Service and Ambari) needed for cluster

3. Ambari generates keytabs for the principals

4. Ambari distributes keytabs to Ambari Server and cluster hosts

5. Ambari discards the KDC Admin Account credentials (optional)

AmbariServer KDC

1 2

4

3

5

Cluster


Background: Ambari + Hadoop + Kerberos

Ambari Server communicates with the cluster to retrieve information (such as metrics) Especially important for Ambari Views (e.g. Files, Hive, Pig) Therefore: Ambari Server ALSO needs to be “setup for Kerberos”

Ambari Server Cluster

Kerberos enabled


Background: Manual Setup of Ambari Server for Kerberos

Manual setup of Ambari Server for Kerberos (outside of “Enable Kerberos” wizard):

1. Create principal for Ambari Server

2. Generate keytab for Ambari Server

3. Place keytab on Ambari Server host

4. Run “ambari-server setup-security” on Ambari Server

5. Restart Ambari Server

Configuration of Ambari Server for Kerberos is commonly “missed” when setting up Ambari Views


New: Automatic Setup of Ambari Server for Kerberos

When enabling Kerberos and choosing an automated option (MIT or AD), Ambari Server will be setup for Kerberos automatically:

1. A principal will be created for Ambari Server

2. A keytab will be generated and placed on Ambari Server

3. Ambari Server is setup for Kerberos

Note: you will still need to perform the Ambari Server restart for the Kerberos identity to get picked-up by Ambari.


What about Proxyuser + Kerberos?


New: Automatic Proxyuser Setup with Kerberos

When a cluster has Kerberos enabled, the proxyuser needs to be configured based on the primary part of the Kerberos principal name

hadoop.proxyuser.{principal-name-primary}.hosts

hadoop.proxyuser.{principal-name-primary}.groups

Ambari will adjust proxyuser configurations during Kerberos setup

Ambari Server HDFS

(running as user “ambari”)(setup with principal “[email protected]”

“joe” authenticates to Ambari

(setup for proxyuser “ambari-server”)hadoop.proxyuser.ambari-server.hosts=*hadoop.proxyuser.ambari-server.groups=*

Ambari Server can talk to HDFS as “ambari-server” proxyuser on behalf

of “joe”



Feature Highlights: Ops Audit Logging


Operational Audit Logging

Ambari will create entries in an audit log as Ambari + Cluster operations are performed Using the audit log, you can determine who performed the operation and when the

operation was performed as well as other operation-specific information The Ambari Audit log can be found at: /var/log/ambari-server/ambari-audit.log


List of Operations

Stop/Start Service Stop all Services Add Service Move Component Turn On/Off Maintenance Mode Download Client Configurations Blueprint Export Update Configuration **

Login (success/failed) / Logout Create User, Group Delete User, Group Change Group Membership Change User Status, Admin Change User Password Grant/Revoke User, Group Cluster Roles

Service Operations User Operations

** Note: When a Service Configuration change is made, an entry is also written to a specific log file ambari-config-changes.log for configuration changes that provides even more detail on the change.


List of Operations (continued)

Add/Remove Host Enable/Disable/Edit Alert Add/Update/Delete Alert Group Add/Upgrade/Delete Notification Enable/Disable Kerberos Regenerate Kerberos Keytabs Rename Cluster Add/Remove Remote Clusters

Register/Deregister Version Cluster Upgrade

Cluster Operations Upgrade Operations

Create/Delete View Instance Edit View Instance Grant/Revoke View Permissions Create/Delete View URLs

View Operations


Example: Change Group Membership

Add/Remove group members creates a “Membership change” audit entry

2016-06-02T23:12:09.930Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser)2016-06-02T23:12:34.700Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser, mike)


Example: Stop ZooKeeper

A single operation (like “Stop ZooKeeper”) might generate multiple audit entries Relate entries via RequestId()

2016-06-02T23:14:35.206Z, User(admin), RemoteIp(192.168.64.1), Operation(INSTALLED: ZOOKEEPER_SERVER/ZOOKEEPER on c6401.ambari.apache.org (MyCluster)), Host name(c6401.ambari.apache.org), RequestId(7), Status(Successfully queued)2016-06-02T00:31:56.016Z, User(admin), Operation(Stop ZooKeeper Server), Status(IN_PROGRESS), RequestId(7)2016-06-02T00:31:56.025Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(QUEUED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)2016-06-02T00:31:57.370Z, User(admin), Operation(Stop ZooKeeper Server), Status(COMPLETED), RequestId(7)2016-06-02T00:31:57.370Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(COMPLETED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)



Feature Highlights: Version Registration Experience


Introducing the Version Definition File (VDF)

This is a meta file describing which Services are included and at which version


Ambari will “discover” Available Versions

Tabs for list of available Stacks

List of discovered Versions

List of Services w/version #


“Default Version Definition” for Backwards Compat

Ambari provides a “default” Version

Definition.


Add New Version via File Upload or URL


Changes in Install / Version Registration Flow

Scenario Ambari 2.4 ChangeInternet Access / Public Repositories No change.

No Internet Access / Local repositories - Upload a VDF for the Local Repository you created- Set the Local Repository URLs

OR

- Choose the Default Version Definition- Set the Local Repository URLs


Other UX Changes: Local vs. Public Repository Radio

Explicit Choice


Other UX Changes: Local vs. Public Repository Radio

Choose Local

Must enter Base URLs


Other UX Changes: OS Add/Remove


Other UX Changes: RedHat Satellite/Spacewalk

Explicit Choice

- Ambari will not write the .repo files

- User must register the repositories channels via Satellite


Other UX Changes: Viewing, Install and Upgrade


Other UX Changes: Managing Versions


Thank You

apache ambari - what's new in 2.4

Technology