pre-con ed: better big data analytics with ca workload automation for hadoop

37
World ® ’1 6 Better Big Data Analytics with CA Workload Automation for Hadoop Simon Pearson, CA Technologies, Senior Engineering Services Architect MFX31E MAINFRAME AND WORKLOAD AUTOMATION CONTENT AREA

Upload: ca-technologies

Post on 07-Jan-2017

61 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

World®’16

BetterBigDataAnalyticswithCAWorkloadAutomationforHadoopSimonPearson,CATechnologies,SeniorEngineeringServicesArchitect

MFX31E

MAINFRAMEANDWORKLOADAUTOMATIONCONTENTAREA

Page 2: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation

Page 3: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Abstract

BigData.BigDeal.Theneedtominemassivesetsofinformationforuniqueinsightsintocustomerbehaviours,competitiveplaysandmarketfluctuationshastransformedbigdatainitiativesintoimperative,business-criticalpriorities.

Theopen-sourceApache™Hadoop®platformhasrapidlyemergedasthedominantmeansbywhichbusinessesprocess,analyseandextractinsightsfromtheirgrowingsetsofdata.

WhileHadoopdoesincludeabasicschedulerthatdeliverssomeautomation,itisfocusedprimarilyonjobsthatrunonHadoopclusters,anddoesn’tintegratewellwithotherworkloadautomationengines.

CAWorkloadAutomation(CAWA)AdvancedIntegration(AI)forHadoopmakesitpossibletointegrateHadoopwithtraditionaljobs.

SimonPearsonCATechnologiesSeniorEngineeringServicesArchitect

Page 4: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Agenda

INSTALLATION

CONFIGURATION

SUPPORTEDHADOOPJOBTYPES

EXAMPLES– DEFININGHADOOPJOBS

ANDFINALLY……

1

2

3

4

5

Page 5: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationDownloadCAWAAIforHadoopfromsupport.ca.com

Page 6: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationCheckthatyouhaveasupportedHadoopdistribution

Page 7: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationSupportingtheseCAWorkloadEngines

FormoreinformationaboutusingAdvancedIntegrationforHadoop,refertothedocumentationforyourschedulingengine:

•CAWorkloadAutomationAEandWorkloadControlCenter•CAWorkloadAutomationDE•CAWorkloadAutomationESPEdition

Page 8: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationHavethecorrectversionofthesystemagentinstalled

Page 9: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationOrelsethiswillhappen

Page 10: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationMounttheISOandruntheinstaller

Page 11: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

InstallationTheHadoopAIinstallerruns………

Page 12: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ConfigurationCreateagentdefinitionfortheHadoopplugin

Page 13: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ConfigurationCreateconnectionprofile

ThisidentifiestheDBIwanttoextractdatafrom

Page 14: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ConfigurationSecurityProfile– IDandPWforlogintosourceDBforSqoop jobs

Page 15: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ConfigurationSecurityProfile– AccessrighttoHadoopHDFSfilesystem

Page 16: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

ConfigurationSecurityProfile– IDwithrightstoexecuteHivejobs

Page 17: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

SupportedHadoopJobTypes• HDFS- FileSystemactions

• Hive- Scriptexecution

• Pig- Scriptexecution

• Sqoop – DatabaseImport/Exportoperations

• Oozie – ExecuteOozie Workflows

Page 18: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- HDFSOperationssupported– CreateDir,CreateFile,DeleteFile,DeleteDir

Page 19: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- HDFSExample– Deletefile

HadoopconnectionandHDFSsecurityprofileselectedfromthosedefinedinpreviousstep

Page 20: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- HDFSExample– MakeDirectory

ThefourHDSFoperationsSupportedcanbeselectedfromthedropdownlist

Page 21: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- HDFSViewtheresultsviathejoblog

Page 22: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- SqoopOperationssupported:

Page 23: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- SqoopSimpleexample

Page 24: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- SqoopSpoolFile

Page 25: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- PigJobDefinition

Page 26: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- PigScripttoexecuteandsampledata

Page 27: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

DefiningHadoopJobs- PigSpoolFile

Page 28: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

IntegratingHadoopwithExternalWorkflows

Page 29: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

AndFinally…..WhatifyouarerunningCAWAAE?

Page 30: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

AndFinally…..ThesameHadoopjobtypesareavailableintheQuickEditor

Page 31: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

31 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

AndFinally…..HadoopjobtypesalsoavailableinApplicationEditor

Page 32: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Questions?

Page 33: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

33 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

RecommendedSessions

SESSION# TITLE DATE/TIME

MFX20EBeyondtheCAWorkloadAutomationAECLI–LeveragingCAWLAAEWebServices

11/14/2016at5:00pm

JasmineC

MFX27EPlanningDisasterRecoveryforCAWorkloadAutomationAE

11/15/2016at9:00am

JasmineC

MFX37E YourVoiceCounts! 11/15/2016at1:00pm

MFX122S VisionandRoadmap:CAWorkloadAutomationAE11/16/2016at12:45pm

BanyanA/B

Page 34: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

You Are Cordially Invited……..CA Workload Automation & CA Process Automation

Where? South Seas E When? Tuesday, November 15th

Time? 4:30pm – 6:00pm

Customer & Partner Reception

Page 35: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

35 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

MustSeeDemos

ModernWorkloadAutomation

CAWLADEMFandWorkloadAutomation

SeamlessApplicationAutomationAgents&AdvancedIntegrationsMFandWorkloadAutomation

CAConversionasaService

CAWorkloadAutomationMFandWorkloadAutomation

Innovations:CrossEnterpriseWorkloadAutomationCAWLAAE&DEMFandWorkloadAutomation

Page 36: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.36 @CAWORLD#CAWORLD

MainframeandWorkloadAutomation

FormoreinformationonMainframeandWorkloadAutomation,pleasevisit:http://cainc.to/9GQ2JI

Page 37: Pre-Con Ed: Better Big Data Analytics with CA Workload Automation for Hadoop

37 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

Thankyou

Stayconnectedatcommunities.ca.com