docs.hortonworks.com

Hortonworks DataFlow Sep 14, 2016

ii

Hortonworks DataFlow: Release NotesCopyright © 2012-2016 Hortonworks, Inc. Some rights reserved.

Hortonworks DataFlow (HDF) is powered by Apache NiFi. A version of this documentation originallyappeared on the Apache NiFi website.

HDF is the first integrated platform that solves the real time challenges of collecting and transportingdata from a multitude of sources and provides interactive command and control of live flows with full andautomated data provenance. HDF is a single combined platform that provides the data acquisition, simpleevent processing, transport and delivery mechanism designed to accommodate the diverse dataflowsgenerated by a world of connected people, systems and things.

Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of ourcode back to the Apache Software Foundation. Hortonworks DataFlow is Apache-licensed and completelyopen source. We sell only expert technical support, training and partner-enablement services. All of ourtechnology is, and will remain free and open source.

Please visit the Hortonworks page for more information on Hortonworks technology. For moreinformation on Hortonworks services, please visit either the Support or Training page. Feel free to ContactUs directly to discuss your specific needs.

Except where otherwise noted, this document is licensed underCreative Commons Attribution ShareAlike 3.0 License.http://creativecommons.org/licenses/by-sa/3.0/legalcode

Hortonworks DataFlow Sep 14, 2016

iii

Table of Contents1. Hortonworks DataFlow 2.0 Release Notes ................................................................... 1

1.1. Apache Component Support ............................................................................ 11.2. New Features ................................................................................................... 11.3. Unsupported Features ...................................................................................... 3

1.3.1. Technical Preview Features .................................................................... 31.3.2. MiNiFi Native Agent download locations ................................................ 41.3.3. Community Features .............................................................................. 41.3.4. Unsupported Customizations ................................................................. 5

1.4. HDF 2.0 Repo Locations ................................................................................... 51.5. Behavioral Changes .......................................................................................... 71.6. Apache Patch Information ................................................................................ 8

1.6.1. NiFi ........................................................................................................ 81.6.2. Kafka .................................................................................................... 91.6.3. Ranger ................................................................................................. 101.6.4. Storm .................................................................................................. 11

1.7. Common Vulnerabilities and Exposures ........................................................... 131.8. Known Issues ................................................................................................. 151.9. Third-Party Licenses ........................................................................................ 171.10. Fixed Issues ................................................................................................... 17

Hortonworks DataFlow Sep 14, 2016

iv

List of Tables1.1. New Features ........................................................................................................... 11.2. Technical Previews .................................................................................................... 31.3. Community Features ................................................................................................. 41.4. HDF repo locations ................................................................................................... 51.5. Behavioral Changes .................................................................................................. 7

Hortonworks DataFlow Sep 14, 2016

1

1. Hortonworks DataFlow 2.0 ReleaseNotes

This document provides you with the latest information about the HDF 2.0 release and itsproduct documentation.

• Apache Component Support

• New Features

• Unsupported Features

• HDF 2.0 Download Locations

• Behavioral Changes

• Apache Patch Information

• Common Vulnerabilities and Exposures

• Known Issues

• Third-Party Licenses

• Fixed Issues

1.1. Apache Component SupportHDF 2.0 includes the following Apache components:

• Ambari 2.4.0.1

• Kafka 0.10.0.1

• NiFi 1.0.0

• Ranger 0.6.0

• Storm 1.0.1

• ZooKeeper 3.4.6

1.2. New FeaturesHDF 2.0 is a major release and contains the following new features, improvements, andbug fixes.

Table 1.1. New Features

Apache Component Feature

Kafka Streamlined Operations:

• Kafka Rack Awareness (KIP-36)

Enterprise Readiness:

Hortonworks DataFlow Sep 14, 2016

2

Apache Component Feature

• Improved Kafka SASL Support (KIP-43)

NiFi • Core Enhancements

• Multi-tenant authorization and internal authorization and policy management

• Zero master clustering (NIFI-1727, NIFI-1678, NIFI-1563, NIFI-483)

• Deterministic template export (NIFI-826)

• Site-to-Site proxy server support (NIFI-1857)

• Enhanced User Experience

• UI upgrade (NIFI-1323)

• Enhanced expression language support (NIFI-1974, NIFI-2208, NIFI-2407)

• Deeper Ecosystem Integration

• GenerateTableFetch processor (NIFI-2126)

• JoltTransformJSON processor (NIFI-361)

• MQTT Processor (NIFI-1808)

• Reporting Task to offload provenance data and send to NiFi through Site2Site

• Encrypted passwords in configuration files (NiFi-1831

• Support for configuring SSL as a JMS provider

• Ranger Integration (RANGER-938 NIFI-1733)

• Ambari Integration (RMP-5266, RMP-6154, RMP-6155)

• MiNiFi Java Agent (RMP-5260, RMP-6332, RMP-6333, RMP-6334)

Ranger Advanced Security:

• Ranger policy model to support data masking (RANGER-873)

• Ranger policy model to support row-filtering (RANGER-908)

• Dynamic Tag Based Access Policy (RANGER-274)

• Time and Prohibition-based Access Policy (RANGER-606)

• Location-based Access Policy (RANGER-596)

Enterprise Readiness:

• Ranger KMS – Safenet Luna HSM Integration (RANGER-868)

• Improvements on Reports page in Ranger Admin : Enhanced search by resource access,tags, etc. download report to a Excel, csv (RANGER-913)

• Remove support for DB based auditing (RANGER-900)

• Script and process to migrate existing audit from RDBMS to Solr (RANGER-271)

• Support multiple OU in LDAP search for Ranger usersync (RANGER-803)

• UserSync : limit the groups associated with users based on the group-search results(RANGER-869)

• Provide support to delete Users and Groups from Ranger Admin UI (RANGER-888)

• Add Ranger related rules to SmartSense

Storm Developer Productivity:

Hortonworks DataFlow Sep 14, 2016

3

Apache Component Feature

• Storm Windowing and State Management(STORM-1167)

• New Storm Connectors (STORM-1075, STORM-845, STORM-851)

• Kafka Spout using new Client APIs (STORM-822)

• Storm Distributed Log Search (STORM-902)

• Storm Dynamic Log Levels (STORM-412)

Performance:

• Storm Performance Improvements (STORM-1151, STORM-1526, STORM-1537,STORM-1539)

Streamlined Operations:

• Storm Management View via Ambari (AMBARI-16095)

• Storm Topology Event Inspector (STORM-954)

• Storm Dynamic Worker Profiling (STORM-1157)

• Storm Resource Aware Scheduling (STORM-894)

Enterprise Readiness:

• Automatic Back Pressure (STORM-886)

• Distributed Cache (STORM-876)

• Improved Nimbus High Availability (STORM-166)

• Pacemaker Storm Daemon (STORM-885)

1.3. Unsupported FeaturesSome features exist within HDF 2.0, but Hortonworks does not currently support thesecapabilities.

• Technical Preview Features

• MiNiFi C++ Download Locations

• Community Features

• Unsupported Customizations

1.3.1. Technical Preview FeaturesThe following features are available within HDF 2.0 but are not ready for productiondeployment. Hortonworks encourages you to explore these technical preview featuresin non-production environments and provide feedback on your experiences through theHortonworks Community Forums.

Table 1.2. Technical Previews

Component Feature

NiFi New in this release:

• Hive processors:

• PutHiveQL

Hortonworks DataFlow Sep 14, 2016

4

Component Feature

• GetHiveQL

• HiveStreaming

• OrcFormatConversion

• GetEmail

• ElasticSearch processors:

• PutElasticSearchHTTP

• FetchElasticSearchHTTP

• MiNiFi Native Agent (C++)

Storm New in this release:

• Automatic back pressure

• Pacemaker daemon

• Resource-aware scheduling

• Connectivity Enhancements: Kinesis spout, openTSDBbolt

1.3.2. MiNiFi Native Agent download locationsFor information on downloading and using the MiNiFi Native Agent, see the Apache NiFiMiNiFi website.

1.3.3. Community FeaturesThe following features are developed and tested by the Hortonworks community butare not officially supported by Hortonworks. These features are excluded for a varietyof reasons, including insufficient reliability or incomplete test case coverage, declarationof non-production readiness by the community at large, and feature deviation fromHortonworks best practices. Do not use these features in your production environments.

Table 1.3. Community Features

Component Feature

Kafka Introduced in a previous release:

• New Consumer API

NiFi • ConvertCSVToAvro

• ConvertJSONToAvro

• DebugFlow

• DeleteDynamoDB

• DeleteHDFS

• ExtractEmailAttachments

• ExtractEmailHeaders

• ExtractMediaMetadata

• GetDynamoDB

Hortonworks DataFlow Sep 14, 2016

5

Component Feature

• GetHDFSEvents

• GetSNMP

• ListenLumberjack

• ListenSMTP

• ListS3

• ModifyBytes

• PutDynamoDB

• PutIgniteCache

• PutKinesisFirehose

• PutLambda

• PutSlack

• PutTCP

• PutUDP

• QueryDNS

• SetSNMP

• StoreInKiteDataset

• AWSCredentialsProviderControllerService

• DataDogReportingTask

1.3.4. Unsupported Customizations

Hortonworks cannot guarantee that default NiFi processors are compatible withproprietary protocol implementations or proprietary interface extensions. For example,we support interfaces like JMS and JDBC that are built around standards, specifications, oropen protocols. But we do not support customizations of those interfaces, or proprietaryextensions built on top of those interfaces.

1.4. HDF 2.0 Repo LocationsUse the following table to identify the HDF 2.0 repo location for your operating system andoperational objectives.

Table 1.4. HDF repo locations

OS Format Download location

Repo http://public-repo-1.hortonworks.com/HDF/centos6/2.x/updates/2.0.0.0/hdf.repo

RMP tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos6/2.x/updates/2.0.0.0/HDF-2.0.0.0-centos6-rpm.tar.gz

Tars tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos6/2.x/updates/2.0.0.0/HDF-2.0.0.0-centos6-tars-tarball.tar.gz

Red HatEnterpriseLinux / CentOS 6(64-bit):

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/centos6/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Hortonworks DataFlow Sep 14, 2016

6

OS Format Download location

Repo http://public-repo-1.hortonworks.com/HDF/centos7/2.x/updates/2.0.0.0/hdf.repo

RMP tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos7/2.x/updates/2.0.0.0/HDF-2.0.0.0-centos7-rpm.tar.gz

Tars tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos7/2.x/updates/2.0.0.0/HDF-2.0.0.0-centos7-tars-tarball.tar.gz

RedHatEnterpriseLinux / CentOS 7(64-bit):

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/centos7/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Repo http://public-repo-1.hortonworks.com/HDF/ubuntu12/2.x/updates/2.0.0.0/hdf.list

Deb Tarball http://public-repo-1.hortonworks.com/HDF/ubuntu12/2.x/updates/2.0.0.0/HDF-2.0.0.0-ubuntu12-deb.tar.gz

Tars tarball http://public-repo-1.hortonworks.com/HDF/ubuntu12/2.x/updates/2.0.0.0/HDF-2.0.0.0-ubuntu12-tars-tarball.tar.gz

Ubuntu Precise(12.04) (64-bit):

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/ubuntu12/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Repo http://public-repo-1.hortonworks.com/HDF/ubuntu14/2.x/updates/2.0.0.0/hdf.list

Deb tarball http://public-repo-1.hortonworks.com/HDF/ubuntu14/2.x/updates/2.0.0.0/HDF-2.0.0.0-ubuntu14-deb.tar.gz

Tars tarball http://public-repo-1.hortonworks.com/HDF/ubuntu14/2.x/updates/2.0.0.0/HDF-2.0.0.0-ubuntu14-tars-tarball.tar.gz

Ubuntu Trusty(14.04) (64-bit):

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/ubuntu14/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Repo http://public-repo-1.hortonworks.com/HDF/debian7/2.x/updates/2.0.0.0/hdf.list

Deb tarball http://public-repo-1.hortonworks.com/HDF/debian7/2.x/updates/2.0.0.0/HDF-2.0.0.0-debian7-deb.tar.gz

Tars tarball http://public-repo-1.hortonworks.com/HDF/debian7/2.x/updates/2.0.0.0/HDF-2.0.0.0-debian7-tars-tarball.tar.gz

Debian 7:

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/debian7/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Repo http://public-repo-1.hortonworks.com/HDF/suse11sp3/2.x/updates/2.0.0.0/hdf.repo

RMP tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/suse11sp3/2.x/updates/2.0.0.0/HDF-2.0.0.0-suse11sp3-rpm.tar.gz

Tars tarball http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/suse11sp3/2.x/updates/2.0.0.0/HDF-2.0.0.0-suse11sp3-tars-tarball.tar.gz

SUSE EnterpriseLinux 11 SP3(64-bit):

HDFManagementPack

http://public-repo-1.hortonworks.com/HDF/suse11sp3/2.x/updates/2.0.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-2.0.0.0-579.tar.gz

Additional download options

SoftwarePackage

Format Download location

NiFi tar file http://public-repo-1.hortonworks.com/HDF/2.0.0.0/HDF-2.0.0.0-579.tar.gzNiFi onlyinstallationpackage

NiFi zip file http://public-repo-1.hortonworks.com/HDF/2.0.0.0/HDF-2.0.0.0-579.zip

Tar file Apache Download SiteMiNiFi JavaAgent – ApacheDownload

Zip file Apache Download Site

Hortonworks DataFlow Sep 14, 2016

7

1.5. Behavioral ChangesBehavioral changes denote a marked change in behavior from the previously releasedversion to this version of software. In HDF 2.0, behavioral changes affect the followingHadoop components.

Table 1.5. Behavioral Changes

HortonworksBug ID

ApacheComponent

Apache JIRA Summary Details

BUG-60199 Ranger RANGER-1025 Ranger APIchange ofbehavior forHDP 2.4.0

Component Affected: Ranger admin

Scenario: Search filter is not working as expected.

For example : If expected search result is after firstrecord and if in search request page size is 1.

http://localhost:6080/service/public/api/policy?repositoryName=Sandbox_hadoop&resourceName=test&pageSize=1

Above search policy URL not returning a policy asexpected. It seems as if the filtering is happeningafter retrieving the first n policies where n is thepageSize, while it should fetch all matching resultsfirst and apply the pageSize limit later.

Previous Behavior: Searches for a policy by namelimits the search to the default page size, searchreturns no records if the policy is farther downthe list. If pageSize is big enough to include thepolicies we are searching for then we get theresults we expect. It seems as if the filtering washappening after retrieving the first n policieswhere n is the pageSize.

New Behavior: Able to search across all policiesafter new implementation. New paginationimplementation shall send results according torequested page size after filtering the result.

RMP-5035 Ranger AMBARI-15914

AMBARI-15916

RANGER-271

RANGER-900

Ranger:Remove optionto store auditin DB

In RangerAudits, Auditto DB isno longeravailable. Usersusing Auditto DB mustmigrate to Solr.Use the HDPSecurity Guide- MigratingAudit Logsfrom DB toSolr in AmbariClusters.

Scenario: Ranger Audits users who are currentlyusing Audit to DB must migrate to Audit to Solr.

Previous Behavior: Ranger Audit can beconfigured to go with any of the followingdestinations: DB, SOLR, and HDFS.

New Behavior: Ranger Audit can no longer beconfigured to the destination DB. Ranger Auditcan only be configured to go with the followingdestinations: SOLR and HDFS.

During upgrade to HDP 2.5, If you have notenabled ranger-audit to SOLR, then you willhave to configure audit to Solr post-upgrade.Otherwise, you will not see audit activities inRanger UI. You can either use an externallymanaged Solr or Ambari managed Solr. For detailson configuring these, refer to the Solr Auditconfiguration section in installation guide.

BUG-62267 Storm STORM-1202 MigrateAPIs toorg.apache.storm,but try toprovide

Component Affected: Storm core / trident APIs

Scenario: Package name changed from:

backtype.storm

Hortonworks DataFlow Sep 14, 2016

8

HortonworksBug ID

ApacheComponent

Apache JIRA Summary Details

backwardscompatibilityas a bridge

to:

org.apache.storm

Previous Behavior: You need to create adependency on storm-core to build topologiesand also import relevant classes in their code. Forexample:

import backtype.storm.topology.BasicOutputCollector;

New Behavior: With Apache Storm 1.0, all ofthe core and trident classes are moved frombacktype.storm to org.apache.storm. You canimport the same storm-core and trident APIclasses by using org.apache instead of backtype.

import org.apache.storm.topology.BasicOutputCollector;

For existing topologies you can deploy withoutchanging the code by using the following class.

client.jartransformer.class: org.apache.storm.hack.StormShadeTransformer

Make sure you add the following configuration tostorm.yaml.

BUG-63146 Storm N/A Parametertype change inorg.apache.storm.spout.Scheme

Component Affected: Storm

Scenario: Any user who is implementing theScheme interface from Storm.

Previous Behavior: Pass the byte[] parameter tothe Scheme interface.

New Behavior: Instead of passing byte[], passByteBuffer to the Scheme interface.

See the following link for a code example: https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka/src/jvm/org/apache/storm/kafka/StringScheme.java

1.6. Apache Patch InformationThe following sections list patches in each HDF 2.0 component beyond what was fixed inthe base version of the Apache component.

1.6.1. NiFi

HDF 2.0 provides NiFi 1.0.0 and the following Apache patches:

• NIFI-1966: NiFi nodes disagree about dataflow content at startup.

• NIFI-2628: NiFi nodes do not immediate save flow returned by cluster coodinator to disk.

• NIFI-2681: Update the Persistent Provenance Repository to stop caching Index Searchersto improve reliability.

• NIFI-2678 : Clicking a processor Stop button may fail to stop the processor.

Hortonworks DataFlow Sep 14, 2016

9

• NIFI-2679 : Unable to disable invalid processors using the Operate palette.

• NIFI-2721: Cannot use the UI to access referenced Controller Services.

• NIFI-2649: Controller Services are not automaticall refreshed after creating one inlinefrom a Reporting Task.

• NIFI-2704: DataTransferResource should utilize the DataTransferAuthorizable to run thesame authorization logic.

• NIFI-2687, NIFI-2694: RPGPort REST endpoint tries to retrieve RPG with RPGPort ID.

• NIFI-2712: The DatabaseFetch processor max-value columns do not work as expected.

• NIFI-2717: Site-to-Site should ensure that the remote instance commits a transactionbefore the local instance.

• NIFI-2708: Site-to-Site Details response need to be merged.

• NIFI-2686 : Improve the tls.toolkit.sh log message about no specified host.

• NIFI-2688 : ConfigEncryptionToolTest causes build to fail when building using 4 lettertime locales (e.g. AEST).

• NIFI-2722 : Statistics on the canvas are no longer updating.

• NIFI-2732 : ConsumeKafka 0.9 and 0.10 are not handling partition reassignmentsufficiently.

• NIFI-2718: HTTP Site-to-Site does not report port authorization failure well.

• NIFI-2739 : ConsumeKafka and ConsumeKafka_0_10 may hang indefinitely if unable tocommunicate with the Kafka broker.

• NIFI-2746: Error when accessing the User page.

• NIFI-2745 : Add Flow File attribute source destination for the ConsumeJMS processor.

• NIFI-2766: Ensure that the root group is included in /resources.

• NIFI-2768 : Ensure that write permissions are required on a parent process group whenupdating connection destination.

• NIFI-2765: PutHiveStreaming does not work with Kerberos.

1.6.2. KafkaThis release provides Kafka 0.10.0.1 and the following Apache patches:

• KAFKA-3258: Delete broker topic metrics of deleted topics.

• KAFKA-3393: Updated the docs to reflect the deprecation of block.on.buffer.full andusage of max.block.ms.

• KAFKA-3660: Log exception message in ControllerBrokerRequestBatch.

• KAFKA-3683: Add file descriptor recommendation to ops guide.

• KAFKA-3704: Revert "Remove hard-coded block size in KafkaProducer."

Hortonworks DataFlow Sep 14, 2016

10

• KAFKA-3717: Support building aggregate javadoc for all project modules.

• KAFKA-3718: Propagate all KafkaConfig __consumer_offsets configs to OffsetConfiginstantiation.

• KAFKA-3721: Put UpdateMetadataRequest V2 in 0.10.0-IV1.

• KAFKA-3728: EndToEndAuthorizationTest offsets_topic misconfigured.

• KAFKA-3747: Close `RecordBatch.records` when append to batch fails.

• KAFKA-3784: TimeWindows#windowsFor calculation is incorrect.

• KAFKA-3785: Fetcher spending unnecessary time during metrics recording.

• KAFKA-3787: Preserve the message timestamp in mirror maker.

• KAFKA-3789: Upgrade Snappy to fix snappy decompression errors.

1.6.3. RangerThis release provides Ranger 0.6.0 and the following Apache patches:

• RANGER-1090: Revoke command with grant option does not disable delegated adminpermission for users/groups in the corresponding policy.

• RANGER-1094: One way SSL (when Kerberos is enabled) for Ranger and its plugins.

• RANGER-1096: Revert to jceks scheme for credential store related operations.

• RANGER-1097: Ranger KMS Plugin should not fails to download policy when UGI ticketexpires.

• RANGER-1099: Keyadmin user is not able to create service/repo using public APIs.

• RANGER-1100: Hive authorizer does not block update when row-filter/column-mask isspecified on the table for the user.

• RANGER-1101: JCEKS keystore is not created successfully after enabling SSL for AtlasRanger plugin.

• RANGER-1103: Added maven version enforcer and moved the plugin to be run as part ofmaven compile.

• RANGER-1104: Catching and Logging DB transaction exceptions during Ranger startup.

• RANGER-1105: Ranger should provide configuration to do hdfs audit file rollover atabsolute time.

• RANGER-1106: Issue after upgrade on ranger hive policy page.

• RANGER-1111: Enhancements to the db admin setup scripts.

• RANGER-1113: Ranger Hive authorizer updated to get query string from HiveConf.

• RANGER-1114: Nimbus, Storm UI server stopped after disabling ranger plugins.

• RANGER-1116: Ranger HivePluginUnitTest fails due to Hive Metastore version check.

Hortonworks DataFlow Sep 14, 2016

11

• RANGER-1119: Exclude test jars from RANGER-admin plugin folders as dependency.

• RANGER-1120: Need a java patch to handle upgrade of hive servicedef.

• RANGER-1121: Resolving circular dependency of spring beans by enabling lazyinitialization of the beans.

• RANGER-1123: Keyadmin user is not able to make getservice call using rest v2 public api.

• RANGER-1124: Good coding practices in Ranger recommended by static code analysis -UI .

• RANGER-1126: Authorization checks for non existent file/directory should not berecursive in Ranger Hive authorizer.

• RANGER-1127: Ranger HA Handle scenarios for request with X-Forwarded-Server.

• RANGER-1128: Data Masking label changes for ranger policies.

• RANGER-1129: Ability to specify 'audit all accesses' via Ranger admin configuration.

• RANGER-1132: Ranger Storm Plugin should include commons-codec jar as a dependency.

• RANGER-1134: Audit to Secure solr fails in case of Ranger Knox Plugin due to MDCcontext issue.

• RANGER-1135: Knox and Storm plugins should use secure policy download endpoint inkerberos mode.

• RANGER-1135: Modified InMemory JAAS configuration to use parent config - if exists.

• RANGER-1136: Ranger audit to HDFS fails with TGT errors in Ranger HiveServer2 pluginwhen UGI -TGT expires in audit thread.

• RANGER-1141: Null pointer exception while retrieving the key during copy file.

• RANGER-1143: Added RANGER-plugins-cred lib for tagsync deployment.

1.6.4. StormThis release provides Storm 1.0.1 and the following Apache patches:

• STORM-1136: Command line module to return Kafka spout offsets lag and display instorm UI.

• STORM-1575: Fix TwitterSampleSpout NPE on close.

• STORM-1674: Idle KafkaSpout consumes more bandwidth than needed.

• STORM-1694: Kafka Spout Trident Implementation Using New Kafka Consumer API.

• STORM-1698: Asynchronous MetricsConsumerBolt.

• STORM-1700: Introduce 'whitelist' / 'blacklist' option to MetricsConsumer.

• STORM-1705: Cap number of retries for a failed message.

• STORM-1709: Added group by support in storm SQL standalone mode.

Hortonworks DataFlow Sep 14, 2016

12

• STORM-1719: Introduce REST API: Topology metric stats for stream.

• STORM-1720: Support GEO in storm-redis.

• STORM-1723: Introduce ClusterMetricsConsumer.

• STORM-1728: TransactionalTridentKafkaSpout error.

• STORM-1730: LocalCluster#shutdown() does not terminate all storm threads/threadpools.

• STORM-1742: More accurate 'complete latency'.

• STORM-1771: HiveState should flushAndClose before closing old or idle Hive connections.

• STORM-1833: Simple equi-join in storm-sql standalone mode.

• STORM-1839: Kinesis Spout.

• STORM-1841: Address a few minor issues in windowing and doc.

• STORM-1842: Forward references in storm.thrift cause tooling issues.

• STORM-1848: Make KafkaMessageId and Partition serializable to support.

• STORM-1849: HDFSFileTopology should use the 3rd argument as topologyName.

• STORM-1850: State Checkpointing Documentation update.

• STORM-1851: Fix default nimbus impersonation authorizer config.

• STORM-1859: Late tuples in windowed mode.

• STORM-1862: Flux ShellSpout and ShellBolt can't emit to named streams.

• STORM-1864: StormSubmitter should throw respective exceptions and log respectiveerrors for registered submitter hook invocation.

• STORM-1865: Update command line client document.

• STORM-1866: Update Resource Aware Scheduler Documentation.

• STORM-1868: Modify TridentKafkaWordCount to run in distributed mode.

• STORM-1873: Implement alternative behaviour for late tuples.

• STORM-1874: Update logger private permissions.

• STORM-1878: Flux can now handle IStatefulBolts.

• STORM-1882: Expose TextFileReader public.

• STORM-1884: Prioritize pendingPrepare over pendingCommit.

• STORM-1887: Fixed help message for set_log_level command.

• STORM-1888: Add description for shell command.

• STORM-1893: Support OpenTSDB for storing timeseries data.

• STORM-1902: Add a simple & flexible FileNameFormat for storm-hdfs.

Hortonworks DataFlow Sep 14, 2016

13

• STORM-1906: Window count/length of zero should be disallowed.

• STORM-1907: PartitionedTridentSpoutExecutor has incompatible types that causeClassCastException.

• STORM-1909: Update HDFS spout documentation.

• STORM-1911: IClusterMetricsConsumer should use seconds to timestamp unit.

• STORM-1914: Storm Kafka Field Topic Selector.

• STORM-1919: Introduce FilterBolt on storm-redis.

• STORM-1924: Adding conf options for Persistent Word Count Topology.

• STORM-1925: Remove Nimbus thrift call from Nimbus itself.

• STORM-1930: Kafka New Client API - Support for Topic Wildcards.

• STORM-1934: Fix race condition between sync-supervisor and sync-processes.

• STORM-1945: Fix NPE bugs on topology spout lag for storm-kafka-monitor.

• STORM-1950: Change response json of "Topology Lag" REST API to keyed by spoutId,topic, partition.

• STORM-1956: Disabling Backpressure by default.

• STORM-1959: Add missing license header to KafkaPartitionOffsetLag.

• STORM-1960: Add CORS support to STORM UI REST API.

1.7. Common Vulnerabilities and ExposuresCVE-2016-8748: Apache NiFi XSS vulnerability in connection details dialogue

Severity: Moderate

Versions Affected:

• HDF 2.0.0

• HDF 2.0.1

• HDF 2.1.0

Description: There is a cross-site scripting vulnerability in the connection details dialogwhen accessed by an authorized user. The user supplied text is not properly handled whenadded to the DOM.

Recommended Action: To access the available fix:

• HDF 2.0.0 and 2.0.1 users should upgrade to 2.0.2 or 2.1.1.

• HDF 2.1.0 users should upgrade to 2.1.1 or later.

CVE-2107-5635: Apache NiFi Unauthorized Data Access In Cluster Environment

Severity: Important

Hortonworks DataFlow Sep 14, 2016

14

Versions Affected:

• HDF 1.2.x

• HDF 2.0.x

• HDF 2.1.x

Description: In a cluster environment, if an anonymous user request is replicated toanother node, the originating node identity is used rather than the “anonymous” user.

Mitigation: A fix is available to remove the negative check for anonymous users beforebuilding the proxy chain and throwing an exception, and evaluating each user in the proxychain iteration and comparing against a static constant anonymous user.

Recommended Action: If you are using HDF 2.x, upgrade to HDF 2.1.2. If you are using HDF1.x, upgrade to HDF 1.2.1.

See the Release Notes for the appropriate release for download information:

• HDF 1.2.1 Release Notes

• HDF 2.1.2 Release Notes

CVE-2107-5636: Apache NiFi User Impersonation In Cluster Environment

Severity: Moderate

Versions Affected:

• HDF 1.2.x

• HDF 2.0.x

• HDF 2.1.x

Description: In a cluster environment, the proxy chain serialization and deserialization isvulnerable to an injection attack where a username could impersonate another user andgain their permissions on a replicated request to another node.

Mitigation: A fix has been provided to modify the tokenization code and sanitize user-provided input. This fix was applied as part of NIFI-3487, and is available in HDF 2.1.2 andHDF 1.2.1.

Recommended Action: If you are using HDF 2.x, upgrade to HDF 2.1.2. If you are using HDF1.x, upgrade to HDF 1.2.1.

See the Release Notes for the appropriate release for download information:

• HDF 1.2.1 Release Notes

• HDF 2.1.2 Release Notes

CVE-2016-5395: Apache Ranger Stored Cross Site Scripting vulnerability

Severity: Moderate

Vendor: Hortonworks

Hortonworks DataFlow Sep 14, 2016

15

Versions Affected: All HDP 2.3/2.4 versions including Apache Ranger versions 0.5.x

Users Affected: All users of ranger policy admin tool.

Impact: Apache Ranger was found to be vulnerable to a Stored Cross-Site Scripting inthe create user functionality. Admin users can store some arbitrary javascript code to beexecuted when normal users login and access policies. See BUG-60647 .

Fix detail: Added logic to sanitize the user input.

Recommended Action: Users should upgrade to HDP 2.5+ (with Apache Ranger 0.6.1+)

1.8. Known IssuesHortonworksBug ID

Apache JIRA ApacheComponent

Summary

BUG-78260 NIFI-3520 NiFi Description of Problem: When multiple Kerberos principalsare used between multiple HDFS processors, the processorinstances will be able to login to Kerberos with theirconfigured principals initially, but will not properly relogin.

Workaround: There is currently no workaround for thisisue.

BUG-40773 N/A Kafka Description of Problem: Kafka broker fails to start afterdisabling Kerberos security.

Workaround: Before disabling Kerberos, you need to stopKafka brokers.

1. Run the following command as the Kafka user:

./bin/kafka-run-class.shkafka.admin.ZkSecurityMigrator --zookeeper.aclunsecure --zookeeper.connect 'hostname:2181'

2. Follow the instructions for disabling Kerberos throughAmbari.

3. Restart Kafka nodes.

BUG-62296   NiFi/ Ambari Problem:The Ambari UI displays Ranger TagSync as anavailable option, but this component is not supported. Youare not able to deploy Ranger TagSync.

Workaround: None.

BUG-62345   NiFi/Ambari Problem: Ambari Metrics System in distributed mode andAmbari Metrics System HA are not supported.

Workaround: None.

    NiFi/Ambari Problem: NiFi encrypted properties in configuration are notsupported in Ambari.

Workaround: None.

    NiFi Problem: PublishKafka and ConsumeKafka do not workwith Kafka 0.10 brokers.

Workaround: Use ConsumeKafka0_10 andPublishKafka0_10, if you are working with Kafka 0.10brokers.

    NiFi/Ranger Problem: You cannot use Ranger to create NiFi policiesbased on groups.

Workaround: None.

Hortonworks DataFlow Sep 14, 2016

16

HortonworksBug ID

Apache JIRA ApacheComponent

Summary

  NIFI-2785 NiFi Problem: Templates can be uploaded to any Process Group.This is driven by the URL used during the upload request.Currently, the URL in the UI for the upload request isinitialized with the root group ID and never updated. As aresult, you can only upload templates through the UI to theroot group.

Workaround: You have two options to work around thisissue:

• Option 1: You can upload templates to descendantgroups using a request directly to the REST API.

• Option 2: You can set policies for templates uploadedto the root group so that they can be shared with otherusers.

  NIFI-2797 NiFi Problem: When you are authenticating with LDAP orKerberos, some NiFi actions require one-time passwords.These passwords are only accepted at certain endpoints,and these endpoints are not correct. As a result, you cannotperfom the following activities:

• Downloading a template

• Downloading content through provenance or queuelisting

• Viewing content when you are in a clusteredenivornment

Associated error message: Unable to perform the desiredaction due to insufficient permissions. Contact the systemadministrator.

Workaround: You have two options to work around thisissue:

• Option 1: Use client certificates for authentication.

• Option 2: Download templates and content using a curlcommand explicitly adding the Authorization header.

  NIFI-2822 NiFi Problem: You cannot access the JoltTransferJson customUI when you are logged in with LDAP or Kerberosauthentication.

Associated error message: An error has occurred loadingthe editor.

Workaround: To work around this issue, use clientcertificate authentication.

  NIFI-2824 NiFi/Ambari Problem: When you are working in an Ambari-managedHDF cluster, you cannot save configuration settings fromthe UpdateAttribute or JoltTransformJson custom UI. Thisissue prevents saving configuration from a Custom UI.

Workaround: There is no workaround for this issue whenyou are working in an Ambari-managed HDF cluster.

To work around this issue, you can use a standalone NiFiinstance.

BUG-63132 N/A Storm Summary: Solr bolt does not run in a Kerberosenvironment.

Associated error message: The following is an example:[ERROR] Request to collection hadoop_logs failed due

Hortonworks DataFlow Sep 14, 2016

17

HortonworksBug ID

Apache JIRA ApacheComponent

Summary

to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http:[...] Error401 Authentication required

Workaround: None at this time.

BUG-63535 N/A Storm Description of Problem: Storm does not support rollingupgrade from a previous HDP version to HDP-2.5.

Solution: If your cluster is managed by Ambari, the rollingupgrade process will ask you to stop Storm topologies,perform the upgrade, and redeploy your topologies.

If your cluster is not managed by Ambari, perform thefollowing manual upgrade steps for Storm before startingthe rolling upgrade process:

1. Stop all topologies.

2. Stop all storm daemons.

3. Delete storm.local.dir contents on all nodes.

4. Delete storm.zookeeper.root node.

Next, upgrade the cluster to HDP-2.5.

To finish the Storm upgrade process: start the stormdaemons, and then redeploy the topologies.

BUG-64028 N/A Ranger Component Affected: Create Policy Audit

Description of Problem: When attempting to view thedetails of a Audit record associated with a deleted rangerrepository, the admin UI shows Page Not Found Error Page(401).

Workaround: Currently, there is no workaround for this.This will be addressed in a future release.

1.9. Third-Party LicensesHDF 2.0 deploys numerous third-party licenses and dependencies, all of which arecompatible with the Apache software license. For complete third-party license information,see the licenses and notice files contained within the distribution.

1.10. Fixed IssuesFixed issues represents selected issues that were previously logged via HortonworksSupport, but are now addressed in the current release. These issues may have beenreported in previous versions within the Known Issues section; meaning they were reportedby customers or identified by Hortonworks Quality Engineering team.

Potential Data Loss

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-57918 HDFS HDFS-10178 Permanent write failures canhappen if pipeline recoveriesoccur for the first packet.

BUG-58254 HBase HBASE-15811 Batch Get after batch Putdoes not fetch all Cells

Hortonworks DataFlow Sep 14, 2016

18

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-61028 NiFi NIFI-2087 The NiFi ProvenanceRepository is removingevents from Lucene Index.

Security

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-53072 Kafka KAFKA-2854 The full Kerberos principalmust be passed through forincoming requests to Kafka.

BUG-55917 Storm STORM-1711 HiveUtils.authenticatereuses current UGI, mixes upprinciples in use.

BUG-61511 NiFi NIFI-2173 If the LDAP referral strategyin the login-identity-providers.xml file is set toIGNORE, NiFi fails to startand does not recognize theIGNORE option.

Incorrect Results

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-49254 Ranger RANGER-798 Ranger "Access > Audit"not showing anything withSource "Solr" due to timefilter issue (GMT).

BUG-64652 NiFi NIFI-2620 HBase tables createdthrough API code and NiFithe standard processor arenot in sync.

Stability

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-56772 HDFS, Ranger HADOOP-12423,HADOOP-12950,HADOOP-12993,RANGER-891

ShutdownHookManagershould have a timeout foreach of the Registeredshutdown hook.

BUG-51259 NiFi NIFI-1413 If the templates are out ofsync with the cluster, a NiFinode may fail to start or jointhe cluster.

Upgrade

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-54675 Ambari, Kafka AMBARI-16027 RU does not handleKafka .8.* topics well(Changes broker IDS).

BUG-54742 Ambari, Ranger RANGER-910 Error during EU whileapplying Ranger Javapatches.

BUG-61600 Ambari, Ranger AMBARI-16756 [EU] Ranger restart is nothappening after upgrade.

Usability

Hortonworks DataFlow Sep 14, 2016

19

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-42190 Ambari, Storm   Ambari start Storm UIreports failure, although UIstarts.

BUG-62656 NiFi NIFI-1956 Add a keyboard-interactiveoption to SFTPTransfer,to support keyboard-interactive login.

Performance

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-52844 Ranger RANGER-794, RANGER-836,RANGER-843, RANGER-844

Policy downloadoptimizations.

BUG-57379 NiFi NIFI-2065 Provenance data queryperformance improvements.

Processor improvements

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-59197 NiFi NIFI-1898 Flume processors do notstart.

BUG-58695 NiFi NIFI-2020 JoltTransformJSONprocessor should supportcustom transforms.

BUG-58217 NiFi NIFI-1900 NiFi should not allowconnections to be movedbetween processors.

BUG-57475 NiFi   Backpress and max queuesize values are not set bydefault.

BUG-55110 NiFi NIFI-1052 Changing the name of acustom processor may causeit not to start.

BUG-53438 NiFi NIFI-1296 Add capabilities to KafkaNAR to use the new KafkaAPI (0.9).

Other

Hortonworks Bug ID Apache Component Apache JIRA Summary

BUG-60199 Ranger RANGER-1025 Policy search REST APIimplemented in public apiV1 is not returning results asexpected.

BUG-57802 NiFi NIFI-1877 FlowFile first in first out(FIFO) prioritization doesnot work.