hortonworks data in motion series part 3 - hdf ambari
TRANSCRIPT
Harnessing Data-in-Motion with Hortonworks DataFlow
HDF 2.0: Apache Ambari Integration
Ali BajwaPrincipal Partner Solutions Engineer
Yolanda DavisSenior Software Engineer
Recap
HDF 2.0: Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm, Apache Ambari
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data-in-MotionCore
InfrastructureSources
Constrained High-latency Localized context
Hybrid – cloud / on-premises Low-latency Global context
RegionalInfrastructure
Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm are trademarks of the Apache Software Foundation
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Ambari?
Health Checks, Alerts Stacks, Views
Lifecycle controls, Rolling Restarts, Decommission/Re-
commission
Host Groups, Versioning, Compare, Revert,
Recommendations, Security Setup
Install Wizard (UI),Blueprints (API)
ConfigManagement
ExtensibilityMonitoring
ServiceManagement
ClusterProvisioning
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaApache Ambari Integration in HDF 2.0• Deployment• Configuration• Debugging• Monitoring
Ease of Deployment
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Deploy Apache NiFi through:– Apache Ambari install wizard or – Operationalize via blueprints automation
Choose which nodes Apache NiFi can be installed on
So either:– Choose Apache NiFi hosts at time of
cluster install
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment– Or add Apache NiFi to existing host (post-cluster install) and then start it
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment–Can configure which user/group Apache NiFi runs as (when cluster installed or when Apache NiFi is added to existing cluster)
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment Can remove Apache NiFi service from Apache Ambari
– Note: this does not remove bits Can stop/start/configure Apache NiFi across cluster via:
– Apache Ambari UI or – Apache Ambari REST API
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment Can stop/start Apache NiFi on individual hosts via Apache Ambari UI and also via
REST API
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Deployment
Quicklink for easy access to Apache NiFi UI – URL dynamically determined based on:
• whether SSL enabled and • which ports user configured
Demo #1: Deploy HDF 2.0
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo #1: Deploy HDF 2.0
•Deploy HDF cluster using Apache Ambari Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
–Install Apache Ambari–Install HDF mpack–Open Apache Ambari WebUI–Run through Install Wizard
Ease of Configuration
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
Configure once via Apache Ambari (instead of repeating config on each node) Support for most Apache NiFi config files (e.g. nifi.properties, bootstrap.conf etc) Assistance for admin during Apache NiFi configuration:
– Help text displayed on hover with property descriptions– Checkboxes instead of true/false values– User friendly labels and default values– ‘Computed’ values can be automatically handled (e.g. node address)
Get benefit of Apache Ambari config features:– Config history:
• diff between versions, revert to older version etc– Update configs via Apache Ambari REST API
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration•Manage host specific configs using Config groups feature
–e.g. to bump memory for Apache NiFi on one of the nodes, click Override button:
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration ...create new config group and add desired host to it
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
‘Common’ configs grouped together and exposed in first config section (‘Advanced nifi-ambari-config’) to allow configuration of:– Ports (nonSSL, SSL, protocol)– Initial and max memory (Xms, Xmx)– Repo default dir locations (provenance, content, db, flow file)– ‘Internal’ dir location - contains files NiFi will write to
• ‘conf’ subdir for flow/tar.gz, authorizations.xml• ‘state’ subdir for internal state• Can change subdir names by prefixing the desired subdir name with
‘{nifi_internal_dir}/’– Sensitive props key (used to encrypt sensitive property values)– Zookeeper znode for nifi
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
nifi.properties exposed under ‘Advanced nifi-properties’ as key/value pairs with helptext– values replaced by Apache Ambari shown surrounded by double braces e.g.{{ }} but
can be overridden by end user– Properties can be updated or added via ‘Custom nifi-properties’ and will get written
to all nodes– ‘Computed’ values handled:
• ‘Node address’ fields replaced with each host's own FQDN
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration
Other property based config files exposed as jinja templates– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden by
end user– Properties can be added/updated and will get written to all nodes
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Configuration Other xml based config files also exposed as jinja templates
– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden – Elements can be updated/added and will get written to all nodes
– Config files written out with either 0400 or 0600 perm
• Why? • Some property files contain plaintext passwords
Ease of Debugging
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Debugging Logsearch integration for ease of visualizing/debugging Apache NiFi logs w/o connecting via SSH to
hosts (e.g nifi_app.log, nifi_user.log, nifi_bootstrap.log) By default, monitors FATAL,ERROR,WARN messages (for all HDF services)
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Debugging Can view/drill into errors at component level or host level Can filter errors based on severity (fatal, error, warn, info, debug, trace) Can exclude ‘noisy’ messages to find the needle in the haystack
Ease of Monitoring
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Apache NiFi Service check: Used to ensure that Apache NiFi UI is up. Can also be
invoked via REST API for automation
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Apache NiFi Alert: Host level Apache Ambari alert when Apache NiFi process
goes down– Can temporarily be disabled by turning on maintenance mode
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Alerts tab allows:
– Disabling and Configuring alerts (e.g. change interval)
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Can setup email or
SNMP notifications to admins
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Apache Ambari Metrics (AMS) integration
– Service autocreates Apache Ambari reporting task, pointing to AMS collector host/port (autodetected)
36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of MonitoringThis is achieved by setting up an initial flow.xml (configurable via Amari)...
Tip: can auto-deploy any Apache NiFi flow by pasting its flow.xml here
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring...and passing arguments via bootstrap.conf Ensures Apache NiFi autodetects updated location of AMS collector (in case it’s moved)
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Results in AMS dashboard
for Apache NiFi metrics, such as:– Flowfiles sent/received– MBs read/written– JVM usage/thread counts
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring
Dashboard widgets can:– be drilled into to see
results from last 1,2,4 hours, day, week etc
– export metrics data as csv or JSON
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ease of Monitoring Same metrics can be viewed in Grafana dashboard
– Access via quicklink under ‘Ambari metrics’ service– Allows users to easily create custom dashboards for each component
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Summary - Apache NiFi in Apache Ambari
• HDF available via management pack (mpack) that defines required services, stack definitions etc
• Wizard Driven Setup• Provides access to Apache NiFi configurations, start/stop/restart• Has a Reporting Task for Apache Ambari Metrics• In Cluster Mode by Default• Leverages Zookeeper on HDF for State Management (not Embedded Zookeeper)• Some Stack Advisor configurations (Ranger integration, Apache NiFi Certificate
Authority)• Audit logging via Apache Ambari Infra (Internal Solr)
* You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)
Demo #2: Explore HDF 2.0
43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo #2: Explore HDF 2.0
Explore HDF 2.0– Apache NiFi configurations– Configuration groups– Metrics– Logsearch integration– Service checks– Alerts
Try it out!
45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Try it out!
Deploy HDF cluster using either:– a) Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
– b) Blueprints: https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html
* Reminder: You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)
46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions? https://community.hortonworks.com/
Hortonworks Community Connection:Data Ingestion and Streaming