Download - Apache Ambari - What's New in 2.0.0
Page 1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Apache Ambari 2.0.0
April 2015
What’s New
Page 2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
What is Ambari?
Apache Ambari is the open source
operational platform to provision, manage
and monitor Hadoop clusters
Page 3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
What’s New in Ambari 2.0.0
Core Platform
Ambari Alerts (AMBARI-6354)
Ambari Metrics (AMBARI-5707)
Simplified Kerberos Setup (AMBARI-7204)
Automated Rolling Upgrade (AMBARI-7804)
HA Stack Support
Hive Metastore HA (AMBARI-6684)
HiveServer2 HA (AMBARI-8906)
Oozie HA (AMBARI-6683)
Ambari Platform
Handle umask 027 setting (AMBARI-7796)
Ambari Agent non-root (AMBARI-1596)
Blueprints API
Add Host (AMBARI-8458)
For a complete list of changes:
https://issues.apache.org/jira/browse/AMBARI/fixforversion/12327486
Page 4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Alerts
Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Alerts
• Ambari Web provides centralized management of Health Alerts
• Alerts are configured by default based on the Services in your cluster
Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Viewing Host Alerts
• View all Health Alerts for each host
Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Modifying Alerts
• Control thresholds, check intervals and response text
Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Alert Notifications
• What: Create and manage
multiple notification targets
• Control who gets notified when
• Why: Filter by severity
• Send only certain notifications to certain
targets based on severity
• How: Control dispatch method
• Support for EMAIL + SNMP
Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Alert Groups
• Create and manage groups of
alerts
• Group alerts further controls what alerts
are dispatched which notifications
• Assign group to notifications
• Only dispatch to interested parties
Page 10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Terminology
Term Definition
Alert Definition Defines the alert including the description, check interval, type and
thresholds.
Type The type of alert, such as PORT, METRIC, WEB, AGGREGATE, SCRIPT.
State Indicates the state of an alert definition (enabled or disabled). When
disabled, no alert instances are created.
Alert Instance Represents the specific alert instances based on an alert definition. For
example: the alert definition for “DataNode process” will have an alert
instance per DataNode component in the cluster.
Status The status (or “severity”) of the alert instance, such as: OK, CRIT.
Threshold The thresholds assigned to each alert status (dependent on type).
Alert Group Grouping of alert definitions, useful for handling notifications targets.
Notification A notification target for when an alert instance status changes. Methods of
notification include EMAIL and SNMP.
Page 11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Alert Types
Type Description StatusThresholds
Configurable?
PORTWatches a port based on a configuration property as the uri.
Example: Hive Metastore ProcessOK, WARN, CRIT Yes (seconds)
METRICWatches a metric based on a configuration property.
Example: ResourceManager RPC LatencyOK, WARN, CRIT Yes (variable)
AGGREGATEAggregate of status for another alert definition.
Example: percentage NodeManagers AvailableOK, WARN, CRIT Yes (percentage)
WEBWatches a Web UI and adjusts status based on response.
Example: App Timeline Web UIOK, WARN, CRIT No
SCRIPTUses a custom script to handle checking.
Example: NodeManager Health SummaryOK or CRIT No
Page 12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Alerting System
1. User creates or modifies cluster
2. Ambari reads alert definitions
from Stack
3. Ambari sends alert definitions to
Agents and Agent schedules
instance checks
4. Agents reports alert instance
status in the heartbeat
5. Ambari responds to alert instance
status changes and dispatches
notifications (if applicable)
Ambari
Server
1
2
4
Stack definition
alerts.json
5
Ambari
Agent(s)
3
snmp
Page 13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Notable Alert REST APIs
REST Endpoint Description
/api/v1/clusters/:clusterName/alert_definitions The list of alert definitions for the cluster.
/api/v1/clusters/:clusterName/alerts The list of alert instances for the cluster. Use partial response syntax to filter.
Example: find all alert instances that are CRITITAL/api/v1/clusters/c1/alerts?Alert/state.in(CRITICAL)
Example: find all alert instances for “ZooKeeper Process” alert def/api/v1/clusters/c1/alerts?Alert/definition_name=zookeeper_server
_process
/api/v1/clusters/:clusterName/alert_groups The list of alert groups.
/api/v1/clusters/:clusterName/alert_history The list of alert instance status changes.
/api/v1/alert_targets/ The list of configured alert notification targets for Ambari.
Page 14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Configuring Alerts During Cluster Install
• Ambari Alerts are configured by
default based on the Services
included in your cluster
• Setting-up email notification is
optional
• Customize Services > Misc
Page 15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Alerts <-> Nagios Integration Patterns
Pattern Description Ambari<->Nagios Integration
1Use Ambari Alerts for
definitions, instances and
instance status.
• Integrate with Ambari REST API.
• Generate Nagios cfgs based on alert information from Ambari.
• Nagios watches Ambari.
• Nagios dispatches notifications based on Ambari reported status.
2Use Ambari Alerts for definition
and instances.
• Integrate with Ambari REST API.
• Generate Nagios cfgs based on alert information from Ambari.
• Nagios watches the cluster.
• Nagios dispatches notifications based on status determined by Nagios.
3 Do not use Ambari Alerts.
• Use Ambari REST API to obtain cluster information.
• Define and write your own alerts for Nagios cfgs.
• Nagios watches cluster.
• Nagios dispatches notifications based on status determined by Nagios.
Page 16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
EXAMPLE: /api/v1/clusters/MyCluster/alert_definitions/15
{
"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/alert_definitions/15",
"AlertDefinition" : {
"cluster_name" : "MyCluster",
"component_name" : "ZOOKEEPER_SERVER",
"description" : "This host-level alert is triggered if the ZooKeeper server process cannot be determined",
"enabled" : true,
"id" : 15,
"ignore_host" : false,
"interval" : 1,
"label" : "ZooKeeper Server Process",
"name" : "zookeeper_server_process",
"scope" : "ANY",
"service_name" : "ZOOKEEPER",
"source" : {
"default_port" : 2181.0,
"reporting" : {
"ok" : {
"text" : "TCP OK - {0:.3f}s response on port {1}"
},
"warning" : {
"text" : "TCP OK - {0:.3f}s response on port {1}",
"value" : 1.5
},
"critical" : {
"text" : "Connection failed: {0} to {1}:{2}",
"value" : 5.0
}
},
"type" : "PORT",
"uri" : "{{zookeeper-env/clientPort}}"
}
}
}
Page 17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
EXAMPLE: /api/v1/clusters/MyCluster/alerts/1
{
"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/alerts/1",
"Alert" : {
"cluster_name" : "MyCluster",
"component_name" : "AMBARI_AGENT",
"definition_id" : 42,
"definition_name" : "ambari_agent_disk_usage",
"host_name" : "c6401.ambari.apache.org",
"id" : 1,
"instance" : null,
"label" : "Ambari Agent Disk Usage",
"latest_timestamp" : 1425530280340,
"maintenance_state" : "OFF",
"original_timestamp" : 1425484200341,
"scope" : "HOST",
"service_name" : "AMBARI",
"state" : "OK",
"text" : "Capacity Used: [1.34%, 7.0 GB], Capacity Total: [525.3 GB]"
}
}
Page 18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Pattern #1: Alert definitions, instances and status
1. Ambari reads alert definitions and
creates instances for the cluster
2. Use Ambari REST API to read
alert instance information and
generate Nagios .cfg files
3. Nagios watches Ambari REST API
for alert instance status changes
4. Ambari receives alert instance
status info from Agents
5. On status changes, Nagios
dispatches notifications
Nagios
Ambari
Server1
2
.cfg
4
3
5email
snmp
CLUSTER
(Optional: Not shown) Configure additional alerts in Nagios to watch
components of the system (Ambari Server, Agents, Hosts)
Page 19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Pattern #2: Alert definitions in Ambari
1. Ambari reads alert definitions and
creates alert instances for cluster
2. Use Ambari REST API to read
alert definition information
3. Use Ambari REST API to read
cluster configuration information
for generating Nagios .cfg files
4. Nagios watches the cluster
5. On status changes, Nagios
dispatches notificationsNagios
Ambari
Server1
3
.cfg
4
5email
snmp
CLUSTER
2
(Optional: Not shown) Configure additional alerts in Nagios to watch
components of the system (Ambari Server, Agents, Hosts)
Page 20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Pattern #3: No alerts in Ambari
1. Define your own alerts and use
Ambari REST API to read cluster
configuration information for
generating Nagios .cfg files for
those alerts
2. Nagios watches the cluster
3. On status changes, Nagios
dispatches notifications
Nagios
Ambari
Server
1
.cfg
2
3email
snmp
CLUSTER
(Optional: Not shown) Configure additional alerts in Nagios to watch
components of the system (Ambari Server, Agents, Hosts)
Page 21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Metrics
Page 22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Metrics System
Built using Hadoop technologies
Default uses local filesystem for
metrics storage (“embedded”) **
Local Filesystem **
HBaseATS
Phoenix
** Tech Preview “distributed” storage option to use existing HDFS
Page 23 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Terminology
Term Definition
Ambari Metrics (“AMS”) The built-in metrics collection system for Ambari (“AMS”).
Metrics Collector The standalone server that collects metrics, aggregates metrics, serves
metrics from the Hadoop service sinks and the Metrics Monitor. Analogous
to gmetad.
Metrics Monitor Installed on each host in the cluster to collect system-level metrics and
forward to the Collector. Analogous to gmond.
Metrics Hadoop Sinks Plugs into the Service sinks to send Hadoop metrics to the Collector.
Page 24 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Collection System
1. Metric Monitors send system-
level metrics to Collector
2. Sinks send Hadoop-level
metrics to Collector
3. Metrics Collector service
stores and aggregates metrics
4. Ambari exposes REST API for
metrics retrieval
Ambari
Server
Metrics
Monitor
Metrics
Collector
Host1
Sink(s)
3
Metrics
Monitor
Host1
Sink(s)Metrics
Monitor
Hosts
Sink(s)
1 2
4
Page 25 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Simplified Kerberos Setup
Page 26 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Simplified Kerberos Setup
• Wizard driven and automated
Kerberos support
• Works with existing Kerberos
infrastructure, including Active
Directory
• Eliminates time consuming
administration of Kerberos
Page 27 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Automated Kerberos Operations
Once enabled, Ambari works directly with the Kerberos infrastructure to
automate these common tasks:
• Add/Delete Host
• Add Service
• Add/Delete Component
• Regenerate Keytabs
• Disable Kerberos
Page 28 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Terminology
Term Definition
Service Principals Principals required for HDP Service Components.
Ambari PrincipalsHeadless principals used by Ambari to perform “smoke tests” and “health
alert checks”.
KDC Admin AccountAn administrative account that will be used by Ambari to create principals
and generate keytabs in KDC.
Page 29 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Principal and Keytab Generation and Distribution
1. User provides KDC Admin
Account credentials to Ambari
2. Ambari connects to KDC, creates
principals (Service and Ambari)
needed for cluster
3. Ambari generates keytabs for the
principals
4. Ambari distributes keytabs to
Ambari Server and cluster hosts
5. Ambari discards the KDC Admin
Account credentials
Ambari
Server KDC
1 2
4
3
5
HDP
Cluster
Page 30 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari + Service Keytab Files
Ambari
Server
HDP
Cluster
Hosts
Keytabs for
Ambari
Principals
Keytabs for
Service +
Ambari
Principals
KDC
Service Principals
Ambari Principals
Ambari and
Service
Principals
Page 31 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Customizable Principal Attributes
• Customize the directory principal attributes for your environment
• Particularly useful when using Active Directory for Kerberos
Page 32 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Manage Kerberos Clients
• Ambari installs Kerberos clients
on cluster hosts
• Optional to not have Ambari
manage krb5.conf client config
OS Client
RHEL/CentOS/OEL krb5-workstation
SLES 11 krb5-client
Ubuntu 12 krb5-user, krb5-config
Page 33 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Configure Ambari and Service Identities
Page 34 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Automated Rolling Upgrade
For HDP Stack
Page 35 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Rolling Upgrades
Minimized
Service Disruption &
Degradation
Minimizes service disruption by avoiding services
entirely not being available for a period of time, or
suffering performance degradation.
Reliable, Safe, Efficient
Process
Dependable, certified, check-pointed process.
Keep data, configurations, key operational state safe.
Page 36 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Process: Rolling Upgrade
HDP has a certified process
for Rolling Upgrades
Services are switched over to
new version in rolling fashion
ZooKeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Finalize
HDFS, YARN, MR,
Tez, HBase, Pig.
Hive
HDFS
YARN
HBase
Page 37 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Process: Rolling Downgrade
ZooKeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
DowngradeV
V
V
V
V
V
V
V
V
V
V
V
V
Finalize
Page 38 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Packaging: Side-by-Side Versions
/usr/hdp/current/usr/hdp/2.2.4.0-2633
/usr/hdp/2.2.0.0-2041
/hadoop-hdfs-namenode/hadoop-hdfs
/hadoop-yarn
/hive
/hadoop-hdfs
/hadoop-yarn
/hive
/hadoop-hdfs-datanode
Page 39 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Automated Rolling Upgrade
Register InstallPerform
UpgradeFinalize
Page 40 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Register New Version
• Register new version with
Ambari
• Provide the Repository Base
URLs for new version
Register InstallPerform
UpgradeFinalize
Page 41 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
New Version Available Register InstallPerform
UpgradeFinalize
Page 42 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Install Packages
Click “Install Packages” to start the install of the version across hosts
Register InstallPerform
UpgradeFinalize
Page 43 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Perform Upgrade Register InstallPerform
UpgradeFinalize
Page 44 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Perform Upgrade
Click “Perform Upgrade” to start the Rolling Upgrade Wizard
Register InstallPerform
UpgradeFinalize
Page 45 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Upgrade In Process
Badge indicates upgrade “In Process” or “Action Required”
Register InstallPerform
UpgradeFinalize
Page 46 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Wizard Driven Experience Register InstallPerform
UpgradeFinalize
With
verification
and validation
Page 47 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Rolling Upgrade – Success! Register InstallPerform
UpgradeFinalize
Page 48 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Ambari Agent Non-Root
For HDP Stack
Page 49 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Non-Root Ambari Agent
Agent Runs Commands From the Ambari Server
• Configuration Change
• Service Start
• Service Stop
Some Command require root level access• /bin/su hdfs -l -s /bin/bash -c /usr/hdp/current/hadoop-client/sbin/hadoop-
daemon.sh --config /etc/hadoop/conf start datanode
Sudo Leveraged
• Configuration for:
– Customizable Users (su hdfs, yarn, etc.)
– Non-Customizable Users (su mysql)
– Commands (yum, mkdir, touch, test, etc.)
Ambari
AgentAmbari
AgentAmbari
Agent
python
Page 50 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Configuring Agent for Non-Root
1. Create and configure a sudoer account
2. Manually bootstrap Ambari Agents
3. Set run_as_user in ambari-agent.ini for the sudoer account
Page 51 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Updated umask Handling
Page 52 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
What about umask?
Unix Permissions Basics: (user, group, other)
4 – read
2 – write
1 – execute
rwxr-xr-x == 755
Previous Behavior:
• If (umask > 022); Warning during agent pre-req check
• Installations would fail if ignored
New Behavior:
• If (umask > 027); Warning during agent pre-req check
• Installation will fail if ignored
Page 53 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Blueprints
Add Host
Page 54 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Blueprints Add Host
Add hosts to a cluster based on a
host group from a Blueprint
Add one or more hosts with a
single call
POST /api/v1/clusters/MyCluster/hosts
{
"blueprint" : "myblueprint",
"host_group" : "workers",
"host_name" : "c6403.ambari.apache.org"
}
POST /api/v1/clusters/MyCluster/hosts
[
{
"blueprint" : "myblueprint",
"host_group" : "workers",
"host_name" : "c6403.ambari.apache.org"
},
{
"blueprint" : "myblueprint",
"host_group" : "workers",
"host_name" : "c6403.ambari.apache.org"
}
]
https://issues.apache.org/jira/browse/AMBARI-8458
Page 55 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
High Availability
For HDP Stack
Page 56 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
High Availability Support
HDP Stack Mode Ambari 2.0.0
HDFS: NameNode HDP 2.0+ Active/Standby
YARN: ResourceManager HDP 2.1+ Active/Standby
HBase: HBaseMaster HDP 2.1+ Multi-master
Hive: HiveServer2 HDP 2.1+ Multi-instance
Hive: Hive Metastore HDP 2.1+ Multi-instance
Oozie: Oozie Server* HDP 2.1+ Multi-instance
* Oozie Server must not use Derby database and needs external load balancer to complete HA solution.
Page 57 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Hive HA
Services > Hive > Service Actions
+ Add Hive Metastore
+ Add HiveServer2
Page 58 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Learn More
Page 58
Resource Location
Apache Ambari Project Page http://ambari.apache.org
Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI
Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI