ctd taking advantage of cloud elasticity and flexibility...
TRANSCRIPT
![Page 1: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/1.jpg)
1©Cloudera,Inc.Allrightsreserved.
FredKoopmansSr.DirectorofProductManagement
TakingAdvantageofCloudElasticityandFlexibility
1©Cloudera,Inc.Allrightsreserved.
![Page 2: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/2.jpg)
2©Cloudera,Inc.Allrightsreserved.
Publiccloudadoptionissurging
![Page 3: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/3.jpg)
3©Cloudera,Inc.Allrightsreserved.
Clouderacustomersareleadingtheway
![Page 4: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/4.jpg)
4©Cloudera,Inc.Allrightsreserved.
Speed Convenience Scale
Self-Service TCO
Hadoopwasbornforthecloud
![Page 5: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/5.jpg)
5©Cloudera,Inc.Allrightsreserved.
Performance BillShock ApplicationPortability Security
DataGovernance
DataSovereignty HybridCloud Lock-in
But,cloudcomeswithitsownsetofchallenges
![Page 6: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/6.jpg)
6©Cloudera,Inc.Allrightsreserved.
Liftandshifttheplatform
Optimizeeachapplicationindividually
ReconstructanEnterpriseData
Hub
Astepwiseapproach
![Page 7: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/7.jpg)
7©Cloudera,Inc.Allrightsreserved.©Cloudera,Inc.Allrightsreserved.
Liftandshifttheplatform
![Page 8: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/8.jpg)
8©Cloudera,Inc.Allrightsreserved.
OpenEnvironment
Runthesameplatformindifferentcloudsoron
baremetal,socustomerscanmoveasneededwithoutmigration
orretraining
OpenEcosystem
450+certifiedISV’sassuresbackwardcompatibilityacrossreleases,socustomers
canleveragetheirpre-existinginvestments
OpenSource
Avoidvendorlock-in,andleveragecomponentssupported
bythecommitterswhodrivethe
communityroadmap
Opennessisevenmoreimportantinthecloud
![Page 9: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/9.jpg)
9©Cloudera,Inc.Allrightsreserved.
Inon-premenvironments,manyapplicationstypicallyshareasingle,multi-tenantcluster
HDFS
![Page 10: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/10.jpg)
10©Cloudera,Inc.Allrightsreserved.
Thecloudcreatesmore&smallerclusters,specializedforeachapplication
S3 AzureDataLake GoogleStorage*
![Page 11: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/11.jpg)
11©Cloudera,Inc.Allrightsreserved.
• Differentdataconsistencymodels• Differentdirectorystructuresupport
Notadrop-inreplacementforHDFS
• Differentaccesscontrolmodels• Differentmaturitylevels
NotallObjectStorescreatedequal
• MostlyfinishedforS3• JustgettingstartedforADLS• NotyetstartedforGCS
NotyetuniversallysupportedbyCDH
Wheretostorethedata?
ObjectStoragegenerallybestchoice•Performanceoftengoodenough•GenerallycheaperperTBthanDAS•Scalesindependentlyfromcompute
![Page 12: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/12.jpg)
12©Cloudera,Inc.Allrightsreserved.
SeparationfromHDFS
•S3Aconnector•ADLSconnector
Fillingthegaps
•Performance•Consistency•Renames
ClouderaFunctionalEquivalence
•Security•Governance•Backup&Recovery
Cross-ClusterSharing
•Permissions•Catalogue•Lineage
ObjectStoragesupportisrapidlyreachingmaturity
S3 ADLS
MapReduce Y Y
Hive Y Y
HiveonSpark Y -
Spark Y Y
HBase -
Impala Y -
Hue Y -
SupportasofC5.11
![Page 13: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/13.jpg)
13©Cloudera,Inc.Allrightsreserved.
Howtoprovisionandmanagecloudinfrastructurecosteffectively?
Provisioningrequirements•Spinclustersup&downquickly•Grow&shrinkclustersdynamically•Selectrightinstancetypesforeachservice
•Leveragedemandbasedpricingwheneverpossible
Managementrequirements•Fullyautomatedandparallelizedinstallationandconfiguration
•Manageallaspectsofclustersecurityautomatically
•Retaindiagnosticandloginformationafterclusterisgone
•Supporttransientandlong-livedclusters
![Page 14: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/14.jpg)
14©Cloudera,Inc.Allrightsreserved.
Easy• Singlepaneofglassforallcloudinfrastructure• Createtemplatestorunapplicationsinapre-optimizedmanner
Flexible•Multi-cloud:AWS,Azure,GCP• Hourlypricingwithautobilling&metering• Spotinstance/blocksupport
Enterprise-grade• IntegrationacrossClouderaEnterprise•ManagementofCDHdeploymentsatscale• DeeplyintegratedwithClouderaManager
ClouderaDirectorautomatesclusterlifecyclemanagement
![Page 15: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/15.jpg)
15©Cloudera,Inc.Allrightsreserved.
Easyadministration• Spotinstanceresiliency• Automatedsecuritycredentialhandling
Transientclusteroperations• Optimizedclusterprovisioning• Automaticcollectionofdiagnosticsandlogs
Long-livedclusteroperations• Downtime-lessupgrade,patch,restart,andreconfiguration
•Monitoring,alerting,healthchecking,reporting,etc.
ClouderaManagerautomatesclusteroperations
ObjectStore
![Page 16: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/16.jpg)
16©Cloudera,Inc.Allrightsreserved.©Cloudera,Inc.Allrightsreserved.
Optimizeeachapplicationindependently
![Page 17: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/17.jpg)
17©Cloudera,Inc.Allrightsreserved.
Really,fourdiscreteapplicationsononeunifiedplatform
Moderndataprocessing(ETL)atscale
DataEngineering
Explore,analyze,andunderstandallyourdata
AnalyticDatabase
Data-drivenapplicationstodeliverreal-timeinsights
OperationalDatabase
Multi-Storage,Multi-Environment
Exploratorydatascienceandmachinelearningforthe
enterprise
DataScience
![Page 18: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/18.jpg)
18©Cloudera,Inc.Allrightsreserved.
DataScience&Engineering
AccessPatterns• Batch• Canbetransientorpersistent
PerformanceNeeds• Relativelyinsensitvetolatencyanddatalocality
Security• Securityoftennotrequiredformanyusecases
OperationalDatabase
AccessPatterns• Real-time• Typicallypersistent
PerformanceNeeds• Typicallyquitesensitivetolatencyanddatalocality
Security• Fine-grainedsecurityoftenrequired
AnalyticDatabase
AcessPatterns• Batchorinteractive• Canbetransientorpersisent
PerformanceNeeds• Relativelyinsensitvetolatencyanddatalocality
Security• Fine-grainedsecurityoftenrequired
Needsofeachapplicationcanvarygreatly
![Page 19: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/19.jpg)
19©Cloudera,Inc.Allrightsreserved.
DataScience&Engineering inthecloudThree architecturalpatternstooptimizeprice,convenience,performance
BatchCluster
TransientBatch(mostflexible)Spinupclustersasneeded● On-demand/spotinstances● Usage-basedpricing● Sizedforworkload● Clusterpertenant/user
BatchCluster
BatchCluster
PersistentBatch(mostcontrol)Persistentcluster(s)forfrequentETL● Reservedinstances● Node-basedpricing● Grow/shrink● Clusterpertenantgroup
PersistentClusterBatch
PersistentBatchonHDFS(fastest)TopperformanceforfrequentETL● Reservedinstances● Node-basedpricing● Grow/shrink● Sharedacrosstenantgroups
Batch Batch
PersistentClusterHDFS
Batch Batch
DefaultChoice
ObjectStorage
![Page 20: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/20.jpg)
20©Cloudera,Inc.Allrightsreserved.
AnalyticDBinthecloud
NewInsights,NewRevenue
BI/Analytics
Exploreandanalyzealldata,whereveritlives
▪ Long-runningclusters▪ Objectstorageorlocalstorage▪ Lift-and-shiftdeployment
Onlypayforwhatyouneed,whenyouneedit
▪ Transientclusters▪ Objectstoragecentric▪ Cloud-nativedeployment
ETL
ReduceOperatingCosts
RefertoDataScience&Engineeringguidelines
Presentsnewsetofchoices
![Page 21: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/21.jpg)
21©Cloudera,Inc.Allrightsreserved.
BI/AnalyticsinthecloudThree architecturalpatternstooptimizeprice,convenience,performance
ObjectStorage
TransientCluster
TransientBI(infrequentusage)Spinupclusterswhenneeded● On-demandinstances● Usage-basedpricing● Grow/shrink● Clusterpertenantoruser
PersistentBI(regularusage)PersistentclustersforBIanytime● Reservedinstances● Node-basedpricing● Grow/shrink● Clusterpertenantgroup
PersistentCluster
PersistentBIwithLocalStorage(fastest)Maxspeedformoreregularworkloads● Reservedinstances● Node-basedpricing● Lessfrequentgrow/shrink● Sharedclusterforsharedlocaldata
PersistentCluster HDFSand/orKudu
PersistentCluster
TransientCluster
DefaultChoice
![Page 22: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/22.jpg)
22©Cloudera,Inc.Allrightsreserved.
OperationalDBinthecloudNotaswellsuitedforcloud,buttargetedbenefitsarepossible
CostGoals
• Low-costbackupanddisasterrecovery• Developmentandtestingenvironmentseasytodeployanddecommission
ConvenienceGoals
• Elasticgrowthfortightlyprovisionedworkloadsmakesexpansioneasy,andenablesalower-coststeadystate
• Fastandeasyprovisioningofadditionalclustershelpsprojectsmovequickly
![Page 23: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/23.jpg)
23©Cloudera,Inc.Allrightsreserved.©Cloudera,Inc.Allrightsreserved.
ReconstructanEnterpriseDataHub
![Page 24: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/24.jpg)
24©Cloudera,Inc.Allrightsreserved.
ManyproblemsareacombinationofSQL&predictive,batch&online
EnterpriseDataWarehouse
ApplicationsDataSources OperationalDataStores
TraditionalArchitecture
EnterpriseDataWarehouse
ServeELT
Archive
BISystem
Modeling
Reporting
ETL
HPCGRID
Storage#2
Storage#1
Ingest
Process Load
Unstructured
FinancialLedgerP&L
RisksMarket,
Counterparty,Ratings
PaymentsCollectionsCharges
Ingest
Ingest
PortfolioContractsPortfolio
![Page 25: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/25.jpg)
25©Cloudera,Inc.Allrightsreserved.
CommonOperations
ObjectStore ObjectStore
DeveloperWorkbench
CommonGovernance
CommonSecurity
ReimaginingtheEnterpriseDataHubinthecloud
Common:Operations,Governance,Security,Schema,Catalog
SQLWorkbenchPartnerEcosystem
![Page 26: CTD Taking Advantage of Cloud Elasticity and Flexibility Finalcdn.govexec.com/media/ctd_taking_advantage_of_cloud...Operational DB in the cloud Not as well suited for cloud, but targeted](https://reader034.vdocuments.us/reader034/viewer/2022042404/5f1b56bea3f1563a5d5fd633/html5/thumbnails/26.jpg)
©Cloudera,Inc.Allrightsreserved. 26
ThankyouThankYouFredKoopmans