How to Automate Offloading ETL Processes to Hadoop

Download How to Automate Offloading ETL Processes to Hadoop

Post on 11-Apr-2017

818 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>Confidential</p><p>OPERATIONAL EXCELLENCE FOR BIG DATA APPS</p></li><li><p>Confidential2</p><p>TRUSTEDby over 10,000 </p><p>companies as their big data app platform</p><p>BACKEDby top Silicon Valley </p><p>investors True Ventures,Rembrandt VP, Bain </p><p>Capital</p><p>FOUNDEDin 2008, with </p><p>headquarters in San Francisco</p></li><li><p>Confidential</p><p>PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS</p><p>your big data apps</p><p>MONITORto resolve </p><p>issues fasterbig data apps </p><p>more effectively</p><p>MANAGECOLLABORATE</p></li><li><p>Confidential4</p><p>Java, Scala (Scalding), SQL SIMPLEEnsure best practices at any scale thanks to easy-to-learn design </p><p>principles</p><p>FLEXIBLELeverage existing Java, </p><p>Scala, and SQL skills and easily adapt to new </p><p>systems</p><p>WE ARE THE DEVELOPERS BEHIND CASCADING</p><p>RELIABLEAlways get optimal performance and </p><p>reliability for big data applications</p></li><li><p>Confidential</p><p> Use Hadoop for ETL / ELT Ensure quality and manageability </p><p>of our ELT / ELT applications Translate existing ETL work to </p><p>Hadoop GUI ETL tool for developers that </p><p>dont know Java, Scala, SQL</p><p>5</p><p>MIGRATING TO HADOOP FOR ETL AT ENTERPRISE SCALE.</p><p>Cascading</p><p>Driven</p><p>?</p><p>?</p></li><li><p>Confidential6</p><p>TODAYS SPEAKERS</p><p>Shahab KamalVice President at BitWise Inc.</p><p>Shahab is responsible for strategy, growth and client relations. Shahab works with client executives on ITStrategy for Business Intelligence, Big Data, Data Warehousing and Enterprise Applications. Shahab hasworked at Ford Motors, Aon Hewitt and Tribune Company on their PeopleSoft ERP implementation and support.His expertise has been around retrofitting data from legacy applications without loss of data integrity.</p><p>Mark CastilloDriven, Inc.</p><p>Mark is a Solutions Architect with 15+ years of software engineering background. He has worked in thefinance, security, healthcare, streaming music, marketing, and social networking industries. His technicalknowledge and skills are focused on distributed systems, data processing, networking, Linux appliances andBig Data.</p></li><li><p>DataMigrationSeamlessTransition toHadoopShahabKamal&amp;MarkCastillo</p></li><li><p>AboutBitwise</p><p>Founded</p><p>in1996withHQinChicago,IL</p><p>Located</p><p>InofficesinIndia&amp;Australia</p><p>ISO9001:2008&amp;ISO27001:2005Certified</p><p>Backed</p><p>ByFortune500customers</p><p>ProprietaryTechnology</p><p>suiteofAcceleratorsthatreducetheexpense,timeandcomplexityoflarge-scaledataprojects.</p></li><li><p>Reporting,Mining,Analytics</p><p>Analytics</p><p>Reporting,Mining,AnalyticsExploratoryDiscoverySearch</p><p>DATAMART</p><p>ReportingDataMining</p><p>STAGE TRANSFORM ARCHIVE</p><p>DataLake</p></li><li><p>BitwiseMigrationSolutionApproach</p><p>~70%EffortSaving~60%EffortSaving</p><p>Inventory DeepDive MigrationDesignMigration Validation</p><p>~30%EffortSaving</p><p>MigrationAutomationAssessmentAutomation TestAutomation</p><p>1 2 3</p></li><li><p>BigDataProcessingPlatform</p><p>OTHERCUSTOM</p><p>LocalIn-Memory MapReduce&amp;Tez</p><p>COMPUTATION FABRIC</p><p>CASCADINGEnterpriseDataApplication</p><p>BitWise BigDataProcessingPlatform</p><p>ETLMigration QualiDI</p><p>DataQualityFramework</p><p>ELTDevelopment</p><p>Development MigrationEngine Testing Checks&amp;Balances</p></li><li><p>CaseStudy</p><p>RECOVERYAPPLICATIONDATASOURCES</p><p>ANALYTICS</p><p>REPORTING</p><p>DeveloperUI</p><p>XMLCustomCode</p><p>ExecutionService</p><p>CascadingFramework</p><p>ETLApplication</p><p>RECOVERYAPPLICATIONDATASOURCES</p><p>ANALYTICS</p><p>REPORTING</p><p>AutomatedETL</p><p>Migration</p><p>RDBMS</p><p>RDBMS</p><p>DataQualityMonitoring</p><p>DataQua</p><p>lityMon</p><p>itorin</p><p>g</p><p>ETLTesting</p><p>OnExecution</p><p>GenerateCascadingFlow</p><p>LaunchMapReduce Jobs</p></li><li><p>BitwiseELTToolArchitecture</p><p>ETLMigration QualiDI</p><p>DataQualityFramework</p><p>DeveloperUI</p><p>XMLCustomCode</p><p>ExecutionService</p><p>CascadingFramework</p><p>DevelopmentEnvironment</p></li><li><p>KeyFeatures</p><p>IncreasesETLdeveloperproductivityonHadoopbyupto50%EASY</p><p>EFFECTIVE</p><p>ECONOMICAL</p><p>OPERATIONALVISIBILITY</p><p>PortsmajorityofexistingETLprocessestoHadoopwithlittletonochanges</p><p>OptimizesETLperformancebychoosingtherightcomputationfabric</p><p>ViewsETLprocessesinreal-timeforservicelevelmanagement</p></li><li><p>BenefitsofBitwiseMigrationSolutionUpto60%ReductionduringAssessmentPhasewithDarkDataDiscoveryFrameworkSAVESTIME</p><p>ECONOMICAL</p><p>INCREASESPRODUCTIVITY</p><p>QUICKERVALIDATION</p><p>Upto70%Touch-FreeMigration</p><p>Upto40%IncreaseinDeveloperProductivity</p><p>Upto30%EffortSavingsinDataValidation</p><p>SAVESEFFORT Upto75%90%EffortSavedforTestComplianceReports</p></li><li><p>AxesUI</p></li><li><p>AxesUI</p></li><li><p>AxesUI</p></li><li><p>AxesUI</p></li><li><p>AxesUI</p></li><li><p>Accelero Demo&amp;UI</p></li><li><p>Concurrent CascadingandDriven</p><p>OTHERCUSTOM</p><p>LocalIn-Memory MapReduce&amp;Tez</p><p>COMPUTATIONFABRIC</p><p>CASCADINGEnterpriseDataApplication</p></li><li><p>BitwisehelpedalargeFortune500companysavemillionsofdollarsandanestimated30-50%timeinETLdevelopment through utilizationof theBitwiseproprietaryETLmigrationaccelerator,offloading fromacostlylegacyplatformtoHadoop.Itbeganwhentheclientexpressedtheirinterestinmoving toHadoop/BigDatabymigrating theirexistingRecoveryAbInitioETLs.BitwisecameupwithaphasedapproachtoProof, ValidateandConverttheexistingETLs.</p><p>Takingthepartnership further, Bitwiseproposed aGUItotheELTtooltoactasadeveloper IDEbasedonEclipseasaNextStep.</p><p>ProofingtheTechnologyStack</p><p>ValidationoftheBitwiseHadoopELTStack</p><p>ETLMigrationusingAcceleroConversionEngine</p><p>PartnershipinAcceleroDevelopmentEngine</p><p>PartnershipinAcceleroGUIDevelopment</p><p>Stage1 Stage2 Stage3 Stage4 Stage5</p></li><li><p>Bitwisehasbeenworkingwithfortune500companytomovedatafromDatalaketoHadoopandidentify risksthatneedtobeaddressed.Theprimary focusondeveloping templatesandframeworkforDataIngestionandTestingafterthedataistransferredandbuild reportsontheoffloaded data.</p><p>PriorBitwisehashelped theclientwithDataIntegrationmigration throughutilizationoftheBitwiseproprietaryDataIntegrationmigrationexceleratorAccelero,offloading fromacostlylegacyplatformtoHadoop, saving30-50%timeinETLmigration.</p><p>Stage1 Stage2 Stage3</p><p>DataIngestionintoHadoopProofofConcept</p><p>TakingtheentireProofofConceptaheaddatalakemovingtoHive</p><p>BuildoptimizedreportsrunningoftheoffloadeddataonHadoop</p><p>Conversion ofproprietaryETLtoAcceleroELTusingCascadingandDriven</p><p>Stage4</p></li><li><p>ThankYou</p></li><li><p>Confidential</p><p> Bitwise website: http://www.bitwiseglobal.com/ Driven website: http://www.driven.io/</p><p> Speakers contact information:- Bob Taylor: bobt@driven.io- Shahab Kamel: Shahab.Kamal@bitwiseglobal.com- Mark Catillo: mark@drive.io</p><p>ADDITIONAL RESOURCES</p></li></ul>

Recommended

View more >