pre-con ed: better big data analytics with ca workload automation for hadoop
TRANSCRIPT
World®’16
BetterBigDataAnalyticswithCAWorkloadAutomationforHadoopSimonPearson,CATechnologies,SeniorEngineeringServicesArchitect
MFX31E
MAINFRAMEANDWORKLOADAUTOMATIONCONTENTAREA
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
BigData.BigDeal.Theneedtominemassivesetsofinformationforuniqueinsightsintocustomerbehaviours,competitiveplaysandmarketfluctuationshastransformedbigdatainitiativesintoimperative,business-criticalpriorities.
Theopen-sourceApache™Hadoop®platformhasrapidlyemergedasthedominantmeansbywhichbusinessesprocess,analyseandextractinsightsfromtheirgrowingsetsofdata.
WhileHadoopdoesincludeabasicschedulerthatdeliverssomeautomation,itisfocusedprimarilyonjobsthatrunonHadoopclusters,anddoesn’tintegratewellwithotherworkloadautomationengines.
CAWorkloadAutomation(CAWA)AdvancedIntegration(AI)forHadoopmakesitpossibletointegrateHadoopwithtraditionaljobs.
SimonPearsonCATechnologiesSeniorEngineeringServicesArchitect
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
INSTALLATION
CONFIGURATION
SUPPORTEDHADOOPJOBTYPES
EXAMPLES– DEFININGHADOOPJOBS
ANDFINALLY……
1
2
3
4
5
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationDownloadCAWAAIforHadoopfromsupport.ca.com
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationCheckthatyouhaveasupportedHadoopdistribution
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationSupportingtheseCAWorkloadEngines
FormoreinformationaboutusingAdvancedIntegrationforHadoop,refertothedocumentationforyourschedulingengine:
•CAWorkloadAutomationAEandWorkloadControlCenter•CAWorkloadAutomationDE•CAWorkloadAutomationESPEdition
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationHavethecorrectversionofthesystemagentinstalled
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationOrelsethiswillhappen
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationMounttheISOandruntheinstaller
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
InstallationTheHadoopAIinstallerruns………
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateagentdefinitionfortheHadoopplugin
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationCreateconnectionprofile
ThisidentifiestheDBIwanttoextractdatafrom
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDandPWforlogintosourceDBforSqoop jobs
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– AccessrighttoHadoopHDFSfilesystem
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ConfigurationSecurityProfile– IDwithrightstoexecuteHivejobs
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SupportedHadoopJobTypes• HDFS- FileSystemactions
• Hive- Scriptexecution
• Pig- Scriptexecution
• Sqoop – DatabaseImport/Exportoperations
• Oozie – ExecuteOozie Workflows
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSOperationssupported– CreateDir,CreateFile,DeleteFile,DeleteDir
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– Deletefile
HadoopconnectionandHDFSsecurityprofileselectedfromthosedefinedinpreviousstep
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSExample– MakeDirectory
ThefourHDSFoperationsSupportedcanbeselectedfromthedropdownlist
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- HDFSViewtheresultsviathejoblog
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopOperationssupported:
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSimpleexample
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- SqoopSpoolFile
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigJobDefinition
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigScripttoexecuteandsampledata
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DefiningHadoopJobs- PigSpoolFile
28 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
IntegratingHadoopwithExternalWorkflows
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..WhatifyouarerunningCAWAAE?
30 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..ThesameHadoopjobtypesareavailableintheQuickEditor
31 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AndFinally…..HadoopjobtypesalsoavailableinApplicationEditor
32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Questions?
33 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
RecommendedSessions
SESSION# TITLE DATE/TIME
MFX20EBeyondtheCAWorkloadAutomationAECLI–LeveragingCAWLAAEWebServices
11/14/2016at5:00pm
JasmineC
MFX27EPlanningDisasterRecoveryforCAWorkloadAutomationAE
11/15/2016at9:00am
JasmineC
MFX37E YourVoiceCounts! 11/15/2016at1:00pm
MFX122S VisionandRoadmap:CAWorkloadAutomationAE11/16/2016at12:45pm
BanyanA/B
34 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
You Are Cordially Invited……..CA Workload Automation & CA Process Automation
Where? South Seas E When? Tuesday, November 15th
Time? 4:30pm – 6:00pm
Customer & Partner Reception
35 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
MustSeeDemos
ModernWorkloadAutomation
CAWLADEMFandWorkloadAutomation
SeamlessApplicationAutomationAgents&AdvancedIntegrationsMFandWorkloadAutomation
CAConversionasaService
CAWorkloadAutomationMFandWorkloadAutomation
Innovations:CrossEnterpriseWorkloadAutomationCAWLAAE&DEMFandWorkloadAutomation
@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.36 @CAWORLD#CAWORLD
MainframeandWorkloadAutomation
FormoreinformationonMainframeandWorkloadAutomation,pleasevisit:http://cainc.to/9GQ2JI
37 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Thankyou
Stayconnectedatcommunities.ca.com