improving hadoop resiliency and operational efficiency with emc isilon

1© Copyright 2015 EMC Corporation. All rights reserved.

IMPROVING HADOOP RESILIENCY & OPERATIONAL EFFICIENCY WITH EMC ISILON

1

MODERNIZE


A LITTLE BIT ABOUT ME AND WHAT I DO FOR EMC.

BONI BRUNO, CISSP, CISM, CGEITPRINCIPAL SOLUTIONS ARCHITECT, ANALYTICSEMERGING TECHNOLOGIES DIVISION | EMC

2


Agenda

Analyze Hadoop’s behavior under different failure scenarios.

Review how EMC Isilon improves Hadoop resiliency and operations.


Hadoop Deployment Considerations


DataNode Failures…

DataNode failures affect the availability of job input and output data and also delay read and write data operations which are central to Hadoop’s performance…


DataNode Shutdown

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down: DataNode failed volumes:/data2/dfs/current;2016-04-22 13:01:00,112 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:svc-platfora (auth:SIMPLE) cause:java.io.IOException: Block blk_2910942244825575033_338680521 is not valid.2016-04-22 13:01:00,112 INFO org.apache.hadoop.ipc.Server: IPC Server handler 50 on 50020, call org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol.getBlockLocalPathInfo from 172.28.10.40:55874: error: java.io.IOException: Block blk_2910942244825575033_338680521 is not valid. java.io.IOException: Block blk_2910942244825575033_338680521 is not valid.

Log message:

Note: HDFS does not support *decommission* of one single disk now. HDFS DataNode can only be decommissioned as a whole.


hdfs-site.xml

<property> <name>dfs.datanode.failed.volumes.tolerated</name> <value>0</value></property>

<property> <name>dfs.datanode.data.dir</name> <value>/data1/dfs,/data2/dfs,/data3/dfs</value></property>


Recovering Data NodesThe fix and work around for the above error log requires the replacement of any failed disks associated with /data2 volume and to recreate the data directory structure as defined by “dfs.datanode.data.dir”.

Recovery steps:

1. replace failed hardware2. restore data volume using OS utilities to recreate the file system and mount.3. mkdir /data2/dfs4. chown hdfs:hadoop /data2/dfs5. service hadoop-hdfs-datanode start


TaskTracker Failures…

TaskTracker failures are equally important because they affect running tasks as well as the availability of intermediate data, i.e. map outputs.


What’s the impact???

Surprisingly, a single failure can lead to large and unpredictable variations in job completion time.

For example, the running time of a job that takes 220swithout failures can vary from 220s to as much as 1000sunder TaskTracker failures and 700s under DataNode failures.

Ref: Florin Dinu & Eugene Ng, Rice University


Why???

• Hadoop’s speculative execution (SE) algorithm can be negatively influenced by the presence of fast advancing tasks. DataNode failures are one cause of such fast tasks.

• Hadoop tasks are not good at sharing failure information. The unfortunate effect is that multiple tasks could be left wasting time discovering a failure that has already been identified by another task.

• Temporary overload conditions such as network congestion or excessive end-host load can lead to TCP connection failures.


Isilon Scale-Out NAS Architecture

OneFS Operating Environment

Intra-cluster Communication LayerClient/Application Layer Ethernet Layer

Sing

le F

S/Vo

lum

eCIFSNFS

FTPHTTP

HDFS for HadoopREST for Object

Gig-e10 Gig-eNetwork

Protocols


HDFS: Standard Hadoop Cluster

HDFSfile

filecopy2filecopy3nodeinfo

file

nodeinfofilecopy2filecopy3

file


file


NodereplyNodereplyNodereplyNodereplynodereply

MAPReduceMAPReduceMAPReduceMAPReduceMAPReduce

nodeinfo

MAPReduceMAPReduceMAPReduceMAPReduce

DataCompute

MAPReduceMAPReduceMAPReduceMAPReduceMAPReduceMAPReduceMAPReduceMAPReduceMAPReduce

ComputeData

Name node

3X

NFSName node

Decision SupportDatabases

Web Clickdata

OLAP

EDW

HTTPCIFSFTPNFS

Landing Zone Servers

Step 1:Data is copied into the Landing Zone

Step 2:Data is copied into

the Cluster (3 times)Step 3:

Hadoop Jobs are run


HADOOP WITH ISILON SCALE-OUT NAS STORAGE

1 Multi Protocol Scale-Out Storage Platform– NFS, CIFS, FTP, HTTP, HDFS

2 Highly resilient, Predictable Scalability– Distributed NameNode & DataNode

3 Enterprise Data Protection & Governance– SnapshotIQ, SyncIQ, SmartLock, ACLs..

4 Industry-Leading Storage Efficiency– >80% Storage Utilization

5 Independent Scalability with Optimized QoS– Optimally Scale Storage & Compute

6Consolidate Data Silos

– Industry Standard Protocols– Bring Applications to Shared Data

HDFS


Better Hadoop--What If You Could…? Have implicit high availability--automatically Elastically & independently scale compute & storage Efficiently protect data with “erasure coding” Use your HDFS system for non-Hadoop processing Automatically have differentiated QoS Run multiple Hadoop distros at the same time


Isilon OneFS: Built for Big DataMassive Scalability

• Up to 20 PB in a single file systems

Management Simplicity – automates activities “unfit for humans”

Industry-leading Reliability and Self-Healing

Application and Workflow Consolidation

Unmatched Performance• Up to 118 GB/s of concurrent throughput• Up to 740 MB/s single stream throughput• Up to 25.1 TB Global Cache

17

• Symmetric scale-out architecture• Fully distributed, fine-grained services• Unified IP storage (NFS, SMB, Object, HDFS)


Ethernet

Hadoop Architecture – DAS vs Isilon

NameNode

Data Node + Compute Node






Ethernet

Compute Node Compute Node Compute Node

Compute NodeCompute Node Compute Node

name node

name node

name node

data node


SMB, NFS, HTTP, FTP,

HDFS

nodeinfo

nodeinfo

nodeinfonodeinfo

MAPReduceMAPReduceMAPReduceMAPReduce

HDFS: Integrated Isilon and Hadoop

name node data node

Isilon

name nodename nodename node

NFS

Decision SupportDatabases

Web Clickdata

OLAP

EDWStep 1:

Much or all of the Data lives on the Isilon/Hadoop Cluster

Step 2:Jobs are run

Hadoop Cluster


DAS Hadoop = at least 5 copies

Existing Virtualized Data Center DAS Hadoop Infrastructure

Unstructured Data2

Existing Primary Storage

3 4 4 4 4 4

1

5 3 4 5 3 4 5 3 4 5

3 4 5

2

Primary DataCopy of DataHDFS Rep Count = 3

1

It takes >24 hours to transfer 100TB into DAS Hadoop over 10GB Ethernet Network

http://www.google.com/url?sa=i&rct=j&q=file+folder+computer+image&source=images&cd=&cad=rja&docid=RIRIN7x8ZSTU6M&tbnid=DkZgWX30S_FZGM:&ved=0CAUQjRw&url=http://www.wpclipart.com/computer/icons/folders/yellow_file_folder.png.html&ei=W6hkUfW9L4es8QT-hYD4Aw&bvm=bv.44990110,d.eWU&psig=AFQjCNHj-4J6mOyb6D5f72yAN2wBRgk31A&ust=1365637557242641




















Data Center Network

Time-to-Results

Data Copy Analysis In-Place Analysis


Hadoop on a Stick

Have you ever copied 100TB from Primary Storage to a Hadoop system?How long does it

take to copy 100TB from one place to another over a 10Gb link?

>24 Hours

Data Center Network


Hadoop Compute Nodes

Reading relevant data to analysis



















http://www.google.com/url?sa=i&rct=j&q=document&source=images&cd=&cad=rja&docid=CvuNvLB3gqK0kM&tbnid=aubfJN_kFZJ-tM:&ved=&url=http://www-03.ibm.com/software/lotus/symphony/gallery.nsf/GalleryClipArtAll/8CE0CE18AB15F52185257596003225D4&ei=xaFoUdCKLejKigKtioDACw&bvm=bv.45175338,d.cGE&psig=AFQjCNE-MxjVtOS7CeP5fqSHsiiab-FmmQ&ust=1365898054040540


Existing Virtualized Data Center


ISILON ENTERPRISE HADOOP

1

No replication required (Use your existing data)

Store 1 copy instead of 5 Industry Leading Time to

Results – no need to wait to transfer data into HadoopNew Hadoop Compute Nodes

Unstructured Data

Use Native HDFS Protocol

Primary Data1

11

1

Start analyzing Data immediately – no need to wait >24 hours to start








Isilon HDFS Interface Isilon supports the HDFS

interfaces for the DataNode and NameNode to host data and metadata

Underlying file system is OneFS As simple as pointing the HDFS

clients to the DNS name of the Isilon cluster!

HDFS


Scale-out Isilon for Scale-out Hadoop

ComputeNodes

• Isilon is a scale-out system, like Hadoop• HDFS on Isilon functions as a parallel

file system• Each compute node performs I/O on

every Isilon node in the rack• I/O bandwidth and storage capacity can

be increased linearly simply by adding Isilon nodes

• Compute can be increased or decreased on the fly and can easily be virtualized

• With a mesh network that is faster than the disks, data locality is irrelevant

IsilonNodes


Protocol SupportServers

Servers

Servers

Before

After

HDFS is not visible to Windows, Unix, Linux, Apple, or any other file system natively

Big Data is only used for Big Data

Inherent multi-protocol support in Isilon allows ubiquitous access to all file systems including Hadoop

Big Data is actual data!Servers


ACCESS FILES USING SMB AND HDFS!• With Isilon, you can use

SMB, NFS, and HDFS to access your files!

• Simply drag-and-drop input files to your HDFS root directory, analyze them using Hadoop, and drag-and-drop the results back to your desktop.


HDFS

SMB, NFS, HTTP, FTP,

HDFS

NodereplyNodereplyNodereplyNodereply

NameNodeData

Support for Multiple Hadoop Distributions

name node

name node

name node

name node data node

NFS

SMB

SMB

NFS

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

MAP Reduce

IBM

© Copyright 2015 EMC Corporation. All rights reserved.

HDFS protocol stack written in C++– Increased parallel processing– Greater scalability– Support for CloudPools and file filtering– Audit support on cluster

Easy web administration interface– Full configuration options

Extensive CLI options for scripting– isi hdfs controls HDFS settings

OneFS HDFS Protocol Advantages


CONFIGURE VIA WEB ADMIN INTERFACE

New HDFS configuration page in web

administration interface

Authentication type and root directory: Any

configuration previously done via CLI now done in web administration

interface

Can enable HDFS and change block size


PIVOTAL HDB (POWERED BY APACHE HAWK)


RECENT BETA TEST ENVIRONMENT


BETA TEST DETAILS…


BETA TEST DETAILS…Test runs through TPCDC Benchmark in regular and Kerberos clusters.


LOAD & ANALYZE RESULTS (UNOFFICIAL)…


HDB 2.0 – ONEFS V8.0 VS V7.2.1.1 (UNOFFICIAL)


HDB 2.0 – DAS VS ONEFS V8 (UNOFFICIAL)


5 USER CONCURRENCY RESULTS (UNOFFICIAL)…


TPCDS SCORES (UNOFFICIAL)…


ROLLING UPGRADE -> NON-DISRUPTIVE UPGRADE

8.0

8.0

8.0

8.0

8.x

8.x

8.x

8.x

8.08.x

8.0 8.x

Release Rollback7.2.1

7.2.1

7.2.1

7.2.1

7.2.1

Non-Disruptive Upgrade

INTERNAL USE ONLY. UNDER NDA. 40


FEATURESSeamless tiering of “frozen” data to CloudProvides OneFS with Cloud scale capacityChoice of public and private Cloud optionsOptional Encryption and compressionSeamless policy-based data placementUses the same SmartPools policy engineIntegrated with Backups and ReplicationTransparent to users and applicationsOptimized recall of portions of a fileOPEX options with Cloud provider while reducing CAPEX

WHAT IS CLOUDPOOLS

S-SeriesPerformance

HD-SeriesDeep archive

X-SeriesThroughput

NL-SeriesArchive

Capa

city

$/TB

CloudPoolsCold archive


High Low


S - Series X - Series

NL-Series

EXTENDING ISILON TO THE CLOUD

HD-Series


CloudCold archive


ISILON AND CLOUDPOOLS COMPARISON

Isilon Cloud vendors enabled by CloudPools

Capacity Up to 68 PB Virtually Limitless

Storage platforms S-, X-, NL-, HD-Series Public and private cloud providers

Tiering Cluster-wide using SmartPools Within data center and/or cloud

Management Same Same

Reporting Same Same


HADOOP RESPONSE WITH COTS INFRASTRUCTURE• TCP connection failure (failed request)• Multiple tasks waste time attempting to discover the failure

(failure information is not shared across tasks)• Task failure on a node can induce task failures in other

healthy nodes• Significant performance impact• System outage

KEY BENEFITS WITH ISILON• Network congestion on Isilon can be easily avoided via

Isilon’s SmartConnect IP load balancing software• Each node has four network interfaces which allows for

improved throughput and load balancing• Data Node traffic can be isolated from compute traffic due

to tiered architecture• Isilon provides monitoring tools for connectivity reporting

across the cluster

Hadoop Event


Failure Scenario:Overload condition such as network congestion or excessive end-host load.

Result:System Performance Degradation

Support Process:Network TeamServer TeamGreater BI Team/Leads


HADOOP RESPONSE WITH COTS INFRASTRUCTURE• System waits for non-responsive node for up to 10

minutes• Temporary overload conditions such as network

congestion or excessive end-host load can lead to TCP connection failures

• Completed map tasks whose output data is inaccessible is re-executed very conservatively

• Significant performance impact

KEY BENEFITS WITH ISILON• DataNode non-responsiveness due to network

contention is avoided via Isilon’s SmartConnect IP load balancing software

• Each node has four network interfaces which allows for improved throughput and load balancing

• Data Node traffic can be isolated from compute traffic due to tiered architecture

Hadoop Event


Failure Scenario:Non-responsiveness from Data Nodes / TaskTracker

Result:System Performance Degradation (5x delay)



HADOOP RESPONSE WITH COTS INFRASTRUCTURE• TCP connection failure (failed request)• Multiple tasks required to analyze and waste time

discovering the failure (failure information is not shared)• Since tasks do not share failure information, a task involving

multiple HDFS requests may encounter multiple CTO(connection timeout) errors

• DataNode considered underprotected and reprotection is initiated after 10 min.

• Significant performance impact

KEY BENEFITS WITH ISILON• Isilon is a combination of multiple nodes that all actively

participate in reads and writes and is fully redundant• Failures within Isilon are immediately discovered via the

OneFS OS and communicated on the Infiniband Network for millisecond resolution

• DataNode failures do not occur on Isilon due to Isilon’s high-availability and resiliency

Hadoop Event


Failure Scenario:Data Node Complete Failure

Result:Task FailureCTO ErrorsCluster Performance Impact



HADOOP RESPONSE WITH COTS INFRASTRUCTURE• Replicating data (3X mirroring - default) is required to

increase availability• Mirroring data across nodes can add massive amounts of IP

traffic over existing interfaces which can cause network congestion

• Network congestion caused by mirroring can cause failed tasks and delayed/failed processing

KEY BENEFITS WITH ISILON• Isilon utilizes erasure-encoding for efficient storage

utilization• All nodes in an Isilon cluster participate in reads and writes

for improved performance• All nodes in an Isilon cluster utilize in-memory and flash-

based caching strategies resulting in improved reads and writes

• Isilon utilizes a dedicated infiniband network (backplane), alleviating possible network contention scenarios between compute and storage nodes within a traditional hadoop environment

Hadoop Event


Failure Scenario:Slow reads and writes

Result:Storage InefficiencyUnused ResourcesNetwork Contention



HADOOP RESPONSE WITH COTS INFRASTRUCTURE

KEY BENEFITS WITH ISILON

Hadoop Event


Scalability/Growth

• Adding both compute and storage when only compute or storage is actually required (cost effectiveness?)

• Network infrastructure requirements grows exponentially over time

• 3x mirroring creates massive infrastructure growth as the environment matures and grows

• Lack of enterprise features for “plug and play” infrastructure, DR, multi-protocol, multi-tenancy, hardware abstraction, SEC-17A4 (WORM)

• Isilon node can be added to a production cluster in under 60 seconds

• Scale compute and storage independently• Minimize network requirements• Minimize data center footprint• Staging not required• Future proof, no downtime during refresh cycles

improving hadoop resiliency and operational efficiency with emc isilon

Technology