THE EVOLVING ROLE OF DATA (WAREHOUSING) DEPARTMENT ALMERE DATA CAPITAL/CIONET MEET 18 DEC 2013

Anurag Shrivastava

About Me

¨  Anurag Shrivastava ¨  Manager SODC (Customer &

Business Intelligence) ¨  At ING Retail Bank, Amsterdam ¨  Deliver solutions for Marketing,

Mortgages, KIM, Mobile, CRM etc.

¨  Deliver solutions for inbound and outbound marketing

¨  SODC (C&BI)

¨  Was set up in 2000 during Postbank era to support marketing and sales

¨  Information Analysts, ETL Developers, BI Specialists and Team Managers ~ (30 Int+20 ext.)

¨  Oracle/Business Objects based DWH platform, IBM Unica & SAS for marketing

¨  Development and Operations fall under separate line management

Transformation from CMM to Agile

¨  Pre 2012 ¨  Many roles and tollgates ¨  Release cycles of 3-9 months ¨  Focus upon processes in CMMI

¨  Post 2012 ¨  Implementation of Scrum

¨  Reduction in number of roles in IT organization

¨  Implementation of DevOps teams

¨  Introduction of new roles that require new skills and mindset

¨  Engineering and Craftsmanship Culture

ü  Customer Centricity

ü  Operational Excellence

ü  New Revenue Streams

The Way Forward

¨  Batch processes affect both stability and response times

¨  Solid skills in the present stack ¨  Business unprepared for agile ¨  Infra unprepared for agile ¨  Pressure from business and senior

management to deliver

¨  Change mindset and behaviour

¨  Improve skills and competence

¨  Speed up renewal of data platform

¨  Deliver faster and often ¨  Automate Automate

Automate

Challenges The Way forward

Big Bang?

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

New Roles are announced and selection started

First Hadoop cluster

Migration to GIT

Introductio of Dev+Ops Teams

First Predictive service built using R and Java

JIRA & Confluence

Test automated deployment with Nolio

IBM Netezza in Production

SAS is live

IBM Datastage Pilot

IBM Cognos in production

IBM Unica Interact is live

Capabilities contribute to our goals but: •  The link between the capabilities and goals is not direct and

obvious •  The change and too much and too fast

Old versus New World

Traditional DWH Stack •  A single or two vendor stack •  Built for durability •  Limited choices •  Proven Technology •  Knowledge retention •  Is this really a fun?

Challenges of Big Data World

u  Too many choices – Hadoop, Hive, Scoop, Flume, Ozzie, Pig etc.

u  No clear leader in the vendor space u  Open source and Java focused community u  New technology – first mover disadvantage u  Steep learning curve, no experienced people

available u  Attention and hype from CXO (read pressure to

deliver) u  Do we really have a big data problem? u  Many alternatives to Hadoop are challenging

Hadoop

Traditional BI to Big Data

u  Traditional BI and DWH will continue to be mainstream for some time but Big Data technologies may reach inflection point in 3 years from now

u  Customer centricity will be a key driver coupled with lower costs

u  Adoption among traditional DWH developers will resemble the technology adoption curve but acceleration is possible by injecting new team members who are early adopters

Source: http://setandbma.wordpress.com/2012/05/28/technology-adoption-shift/

Hadoop is Getting SQL Friendly

•  SQL or SQL like languages (HiveQL or CQL) are making fast inroads in Hadoop world

•  SQL is getting faster on Hadoop by bypassing the overhead of Map/Reduce •  Vendors are making learning Hadoop simpler to use by giving free Sandboxes •  Adming tools are still far behind and requires you to learn plethora of tools •  Knowledge of Java, Scripting and deployment tools becomes essential for Admin

people •  Learning Hadoop for application programmers is getting simpler but deployment

would still need different skillsets •  SQL skills will be useful but the pace of innovation will force developers to acquire

new skills other than SQL

Trends

¨  Hadoop will be made simpler to run and develop upon ¨  SQL is the way forward to ensure large scale adoption ¨  Support for enterprise admin tools ¨  Integration with other enterprise tools ¨  You do not have to be an open source geek to work on Hadoop ¨  Skills in data collection, cleansing and processing will be reusable

Waterfall to Scrum Transition

•  Scrum practices such as Sprint Planning, Short Iterations, Sprint Review and Planning Board get implemented quickly

•  Implementing engineering practices in traditional data warehousing world is hard. For example: TDD, Continuous Integration, Continuous Deployment, Automated Build

•  Agile coaching and mentorship is handy for managers as they might create major challenge in the way of Agile adoption due to their mindset

•  Empowerment of team to decide about designs and tools takes time before starts behaving like an empowered team

What we have tried and worked?

•  Start by visiting industry events for knowledge and inspiration •  Let people experiment in a small group •  Build a community of practices and attend meet-ups •  Start your first assignment with a combination of external and internal

people •  Train people through vendor’s certification programs or use platforms

such as Coursera •  Getting people out of their comfort zone is tough but worth a try

“Big Data is nothing but old DWH concepts in a new wrapper.”

"Traditional data warehousing professionals are fighting a losing battle with big data technologies"

Questions

Thank You