small big data - ipa conference 2017/small big data b… · small big data barry corish it...
TRANSCRIPT
![Page 1: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/1.jpg)
Institute of Public Administration
Institute of Public Administration | 57-61 Lansdowne Road | Dublin 4 | Ireland | Ph. +353 1 2403600 | www.ipa.ie
Small Big Data
Barry CorishIT Consultant
Associate Lecturer, IPA
![Page 2: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/2.jpg)
Me...
Barry Corish
– IT Architect
– Currently working in Financial Services
– Background in IT in Education, the Voluntary Sector and Logistics
© IPA 2
![Page 3: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/3.jpg)
What is Big Data?
Gathering of data?
Transformation of data into information?
Tools/software/hardware?
A set of skills?
An opportunity to drive change or progress?
60% of Big Data Initiatives in 2016 failed in some significant way.– Networkworld, 2017
© IPA 3
![Page 4: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/4.jpg)
What are the challenges?
Gartner Survey, 2015
© IPA 4
![Page 5: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/5.jpg)
Why do Big Data projects fail? Technology
Lacking the right skills
– Not just technology / statistics etc., but business understanding
Technology Problems beyond Big Data
– All the “old” technology problems
– Plus brand new ones relating to new products and systems
Svetlana Sicular, Gartner, 2016
© IPA 5
![Page 6: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/6.jpg)
Roles in a Data Analytics of Big Data Project:
Business Users/SME
Project Sponsor
Project Manager
Business Intelligence Analyst
DBA
Data Engineer
Data Scientist
© IPA 6
![Page 7: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/7.jpg)
Roles in a Data Analytics of Big Data Project:
Business Users/SME
Project Sponsor
Project Manager
Business Intelligence Analyst
DBA
Data Engineer
Data Scientist
© IPA 7
![Page 8: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/8.jpg)
Why do Big Data projects fail? Purpose
Management Inertia
– When we get results, what are we going to do?
Selecting wrong use cases
– Too advanced, or too simple
– Asking wrong questions
If we knew the answer to this, what would we do with it?
Svetlana Sicular, Gartner, 2016
© IPA 8
![Page 9: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/9.jpg)
What are we doing here?
IT projects tend to live or die based on good requirements and analysis
This is doubly true of big data projects.
What are the questions we are trying to answer?
© IPA 9
![Page 10: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/10.jpg)
Humans like patterns...
Search hard enough, you will find patterns.
– Numerology anyone?
Humans get fixed on patterns...
– ...particularly the first pattern they find
– ...or what they believe before they start.
© IPA 10
![Page 11: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/11.jpg)
The right questions are key...
Better Questions give Better Results
– BAD: Can you show that there is a relationship between X and Y?
– BETTER: Can you show if there is a relationship between X and Y?
– BEST: What are the strongest factors affecting Y?
© IPA 11
![Page 12: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/12.jpg)
Humans like simple answers...
– ...particularly if the confirm existing beliefs
– ...or align with other objectives
Presentation of results has to be done very carefully
– ...if you're explaining, you're losing
– ...and the local media always love a story
© IPA 12
![Page 13: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/13.jpg)
Why do Big Data projects fail? Politics
Disagreement on the enterprise strategy
– Lack of buy in from everyone who will be involved
Many parts of organisation will be involved
– Solution avoidance
Whose interests are served by the project?
– Whose toes are you about to stand on? Svetlana Sicular, Gartner, 2016
© IPA 13
![Page 14: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/14.jpg)
Governance and Compliance Questions
GDPR and Data Protection
Data Retention
Open Data initiatives have significant cost and risk
– Effective data anonymisation is hard
Extra Role in project: CIO/Compliance/Legal expertise
© IPA 14
![Page 15: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/15.jpg)
Don’t start with..
Hoovering up “all the data”
Buying in data
Buying new hardware, software or tools.
Don’t start by hiring technical skills
A Big Data project
© IPA 15
![Page 16: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/16.jpg)
Start with..
Start with the questions
– Work out and back from there
Start with what you have
– Data
– Tools
– Skills
Hire-in, train and buy
– When you know what is needed
– Ensure we can repeat in-house and/or in-budget© IPA 16
![Page 17: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/17.jpg)
Questions?
© IPA 17
![Page 18: Small Big Data - IPA Conference 2017/Small Big Data B… · Small Big Data Barry Corish IT Consultant Associate Lecturer, IPA. Me... Barry Corish ... 60% of Big Data Initiatives in](https://reader036.vdocuments.us/reader036/viewer/2022081323/5f03cdf27e708231d40adaca/html5/thumbnails/18.jpg)
Institute of Public Administration
Institute of Public Administration | 57-61 Lansdowne Road | Dublin 4 | Ireland | Ph. +353 1 2403600 | www.ipa.ie
Small Big Data
Barry CorishIT Consultant
Associate Lecturer, IPA