making big data work
TRANSCRIPT
![Page 1: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/1.jpg)
Making Big Data workLewis CrawfordPrincipal Architect @ the DataShed
thedatashed.co.uk
©theDataShedLimited 2015
![Page 2: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/2.jpg)
intro
![Page 3: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/3.jpg)
Who am I?
• Forthelast3years,theDataShed hasbeenprovidingconsultancyservicestoavastarrayoflargeclients.Ourprimaryfocusisensuringthattechnologyandanalyticalstrategiesaretrulyalignedsothatbusinessescanleveragethelatestandgreatestintechnologytomodel,mineanddescribetheirdataasset.
• WewereworkingwithBigDatatechnologybeforethetermwascoined,wehaveexperiencedeliveringanalyticalsystemsdrivenbyPetabytedatasets,andhavedesigned,implementedandsupportedoneofthelargestreal-timedataintegrationandpredictiveanalyticsplatformsintheaviationworld.
• Ourmodelisbasedonusingasmallnumberofexceptionallyhighlyskilledindividualstodeliverdisruptiveandinnovativesolutionsinanagileanddelivery-focusedmanner.
©theDataShedLimited 2015
![Page 4: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/4.jpg)
So what is ‘Big Data’?
©theDataShedLimited 2015
![Page 5: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/5.jpg)
![Page 6: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/6.jpg)
Why do Big Data projects fail?
ToomanypeoplethinkthatBigDatais:
“Thebeliefthatthemoredatayouhave,themoreinsightsandanswerswillriseautomaticallyfromthepoolofonesandzeros.”
GillPress,Forbes.com
©theDataShedLimited 2015
![Page 7: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/7.jpg)
How to make Big Data work?
1. Understandyourproblem
2. Applyappropriatetools
3. Automateeverything.
©theDataShedLimited 2015
![Page 8: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/8.jpg)
Real-time data
©theDataShedLimited 2015
![Page 9: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/9.jpg)
©theDataShedLimited 2015
![Page 10: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/10.jpg)
![Page 11: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/11.jpg)
©theDataShedLimited 2015
![Page 12: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/12.jpg)
Continuous Integration Demo
©theDataShedLimited 2015
![Page 13: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/13.jpg)
How to make Big Data work?
1. Understandyourproblem
2. Applyappropriatetools
3. Automateeverything.
©theDataShedLimited 2015
![Page 14: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/14.jpg)
Little Big Data
©theDataShedLimited 2015
![Page 15: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/15.jpg)
A problem closer to home…
• Everybusinessneedstounderstand:• Theirpotentialcustomersandmarket• Currentcustomers• Theirproductsandsales• Howandwhentheyengageprospectsandcustomers
• Analyticsanddataareexpensive• Manyofthemandatoryelementsareverysimilarforeveryone• TheDataShedisAnalyticsasaServiceandSingleCustomerViewasaService.
©theDataShedLimited 2015
![Page 16: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/16.jpg)
The deduplication problem…
• SMEhas250,000customers(twosystemsofrecord)• Toidentifyduplicatesbruteforceapproach: 31,249,875,000comparisons• Buildingasystemtoprocessaminimumof100clientsaday…• 3.1trillionrecordstocompareusing>10differentalgorithms
• Traditionalscaleupapproachwouldbeexpensive,andmakeslargeassumptionsaroundblockingandpartitioningrules• Asmalldataproblembutabigdatasolution?
Title FirstName Surname Address 1 Address2 Address3
Dr RJ Smith TwoOaks 112OldSt. CountyDurham
Mrs Robyn Smith 112OldStreet Durham DH15YJ
©theDataShedLimited 2015
![Page 17: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/17.jpg)
©theDataShedLimited 2015
![Page 18: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/18.jpg)
The Shed demo
©theDataShedLimited 2015
![Page 19: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/19.jpg)
How to make Big Data work?
1. Understandyourproblem
2. Applyappropriatetools
3. Automateeverything.
©theDataShedLimited 2015
![Page 20: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/20.jpg)
How to make Big Data work?1. Understandyourproblem
• ’BigData’challengesaren’tnecessarilynew,howevermuchofthetechnology is• Articulateandcommunicate– focusondistillingyourproblemdown• Incremental improvementnotwholesalereplacement
2. Applyappropriate tools• Understandtheeconomics aswellasthetechnology• Newtechnologiesneedtobeevaluatedwithinthecontextofyourproblemscope• Newtechnologiesareenablers notdeliverables(#datalake)• ’BigData’technologyshouldbeseenascomplementarytoexistingtechnology
3. Automateeverything• Continuousintegrationtoincludeall testing• Containerisewherepossible• Measureeverything
©theDataShedLimited 2015
![Page 21: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/21.jpg)
If you really want to get involved…
©theDataShedLimited 2015
![Page 22: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/22.jpg)
Get your hands dirty
Ifyou’reinterestedinlearningmore,we’llbehostingahands-onlabseventinthenearfuture.
Sendyourdetailsto:Email:[email protected]:@thedatashed
©theDataShedLimited 2015
![Page 23: Making Big Data Work](https://reader031.vdocuments.us/reader031/viewer/2022021921/58f1bcfe1a28ab8f458b45f5/html5/thumbnails/23.jpg)
Any questions?
©theDataShedLimited 2015
Lewis CrawfordPrincipal Architect @ the DataShed
thedatashed.co.uk