big data and analytics

Copyright 2013 Freeform Dynamics Ltd 1

Big Data, Analytics and the Future of Data CentresWhere are we and where are we going?

www.freeformdynamics.com

Tony Lock – Programme [email protected]

VMUG Meeting, Manchester February 12, 2013

mailto:[email protected]

http://www.freeformdynamics.com/


About Freeform Dynamics Industry analyst firm

Track IT industry developments and offerings Track the evolution of IT related activity and needs in business Advise both end user organisations and suppliers

Research approach IT vendor and service provider briefings Large scale studies - face to face, telephone and online

Community research programme Investigate strategy, business case, architecture, best practice Vendor patronage model allows free distribution Media partnerships for both input and output


Agenda

Big data and Analytics Where are we today?

The evolution of the data centre Visions of the future Will we ever reach Nirvana?

Closing thoughts


Big Data

What is it? Is anyone doing it? The only game in

town?


Defining Big Data?

Analogies Panning for gold Finding the needle in the hay stack Identifying a weak signal in a very noisy environment

Find valuable patterns, trends, correlations, etc. in noisy, unstructured, often complex, and high volume data sets

Doing analytics better / differently?


How much do you agree or disagree with the following statements?

The emergence of advanced storage, access and analytics solutions means the end of the traditional

RDBMS

Regardless of substance and reality of emerging technologies and techniques, the term ‘big data’ is

currently being over-hyped by IT vendors in an unhelpful way

I have a clear understanding of what the term ‘big data’ means

0% 20% 40% 60% 80% 100%

5-Totally agree 4 3 2 1-Totally disagree Unsure


The three Vs of Big Data

Volume

Variety

Velocity

Rule of thumb

Generally think of Big Data when two of these three apply

High physical volumes with low value density

Different sources and formats or information

Rapid rate of data movement, generation or acquisition


The concept of value density

Traditional search and document management

Traditional BI and data warehousing

High value density

High value density Low value density

Low value density

Structured

Unstructured

Structured

Unstructured

BIG DATA


What level of growth are you seeing in the following types of data within your organisation?

0% 20% 40% 60% 80% 100%

5 (Extremely high growth) 4 3 2 1 (No growth)

Structured data(e.g. tabular data in RDBMSs)

Unstructured data(e.g. documents, messages,

multimedia content, etc.)


In what form is your organisation’s most valuable or critical data held (i.e. your crown jewels in information terms)?

Exclusively structured

Mostly structured

Equal split

Mostly unstructured

Exclusively unstructured

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%


How is this changing?0% 20% 40% 60% 80% 100%

Steady shift(25%)

Steady shift(21%)

No change

(43%)

Rapid shift towards value in

unstructured data(4%)

Rapid shifttowards value in structured data

(7%)


To what degree does your organisation exploit its information assets for analysis and decision making?

0% 20% 40% 60% 80% 100%

5 (Fully) 4 3 2 1 (Very poorly)

Structured data(e.g. tabular data in RDBMSs)

Unstructured data(e.g. documents, messages,

multimedia content, etc)


Use of traditional and emerging technologies

Legacy databases and file systems

General purpose RDBMS servers

High performance RDBMS configurations

OLAP multi-dimensional database systems

Write once read many (WORM) databases

Rule-based stream processing engines

In memory databases

Scale-out storage architectures

Distributed indexing and search

Distributed data analytics engines

0% 20% 40% 60% 80% 100%

5 (Extensive use) 4 3 2 1 (Not used at all) Unsure

Series1

-60% -40% -20% 0% 20% 40% 60%

Less use More use

Current level of use Change over next 3 years


How much do you agree or disagree with the following statements?

Developments in advanced storage, access and analytics are allowing us to tackle problems today that were either too hard or too expensive to deal with in the past

Developments in advanced storage, access and analytics are allowing us to take different and better approaches to tackling some key business requirements

Vendors and consulting firms are well geared up to providing us with the support and services we need to take advanced storage, access and analytics on board effectively

We have a clear idea of the business benefits available to us through the use of big data technologies and solutions

We have a clear idea of the advanced data storage and big data analytic technologies that are becoming available

Database vendors are well geared up to support their customers with appropriate licensing and commercial arrangements as data related needs continue to evolve and become more demanding

0% 20% 40% 60% 80% 100%

5-Totally agree 4 3 2 1-Totally disagree Unsure


Elephants in the room Access to data – what data is there and where is it? Are there governance / regulatory / legal restrictions in play

concerning certain data sets? Skill shortage?

IT skills Numerical skills in user base

Just what questions could ‘Big Data’ help with? How do we exploit any results we generate?

Feedback into ‘mainstream systems’


Data Centre Evolution

Where are we now? ‘Perfect Visions’ Will we ever get to Nirvana?


Fragmentation and disjointsSystems

Rigid dedicated stacksTeams, processes & tools

Server, storage, networking, applications, security

Funding and governanceDepartmental budgeting,

ownership and accounting


How much have you virtualised the following elements of your IT landscape?

0% 20% 40% 60% 80% 100%

Your x86 server estate

Storage infrastructure

Your corporate network

Your desktop environment

Totally Extensively Partially A bit Not at all Unsure

Enterprise 481 respondents


How much have you architected your IT infrastructure in the form of shared resource pools such a private clouds?

Unsure6%

Not at all40%

A bit19%

Partially20%

Extensively11%

Totally4%



YOUR VISION FOR DATA CENTRE COMPUTING

Following charts based on this question:Putting all of the existing constraints and the current state of the industry to one side for a minute, how desirable would the following be as part of your perfect IT vision?


0% 20% 40% 60% 80% 100%

All/most of our ITrequirements would befulfilled via hosted cloud

services

All/most of our own ITinfrastructure would be

based on private clouds

Highly desirable-5 4 3 2 Not at all desirable-1 Unsure or N/A

THE CLOUD HOSTING THING (Perfect IT Vision)



THE WHOLE CLOUD THING

0% 20% 40% 60% 80% 100%

All/most of your IT requirements are fulfilled viahosted cloud services

All/most of your in-house (or co-located) ITinfrastructure is based on private cloud

architecture

Already there Within 1 yr Within 3 yrs Within 5 yrs

Within 10 yrs Later Never Don't know



0% 20% 40% 60% 80% 100%

We’d be able to provision workloads and new systems capacity on our privateclouds with a few clicks on a management console

Automation would mean that shared resource usage was continuallyoptimised as demands fluctuate, with no human intervention

We’d be able to migrate applications and workloads back and forth betweenpublic and private clouds with ease

Hybrid cloud management capability would allow us to take an agnostic viewof resources, mixing and matching internal and external capacity freely

We would have end-to-end visibility across on-premise and hosted systemsfor management and troubleshooting purposes

We would have a consistent/joined-up way of managing security and accessacross in-house and hosted systems

We would have a consistent/joined-up way of managing and protecting dataacross in-house and hosted systems

We’d be taking a unified approach to operations and management acrossservers, storage, networking, security, etc


OPERATIONS & MANAGEMENT (Perfect IT Vision)



OPERATIONS AND MANAGEMENT0% 20% 40% 60% 80% 100%

New workloads and systems capacity are generally provisioned with justa few clicks on a console

Use of key shared resources is continually optimised as demandsfluctuate, with no human intervention

You can quickly and easily move workloads back and forth betweenpublic and private clouds

An agnostic view of resourcing means choices between internal andexternal deployment are made purely on requirements and fitness for

purpose You have end-to-end operational visibility across on-premise and hosted

systems

You have a consistent/joined-up way of managing security across in-house and hosted systems

You have a consistent/joined-up way of managing data across in-houseand hosted systems

Already there Within 1 yr Within 3 yrs Within 5 yrs Within 10 yrs Later Never Don't know



0% 20% 40% 60% 80% 100%

Server, storage, networking, security and other specialists wouldbe working together seamlessly as part of a fully integrated ops

team

Other IT teams (developers, testers, support staff, departmentalIT, etc) would have self-service provisioning capability to obtain IT

resources

Non-technical users would have self-service provisioningcapability for new applications and services

IT activity and investment would revolve around the concept ofbusiness services rather than systems

We would be able to easily and accurately charge or report ITcosts back to the business based on activity or consumption


ORGANISATION & SERVICES (Perfect IT Vision)



ORGANISATION AND SERVICES

0% 20% 40% 60% 80% 100%

Self-service provisioning is in place for IT teams

Self-service provisioning is in place for end users

IT activity and investment revolves around the concept ofbusiness services rather than systems

You can easily and accurately charge or report IT costs backto the business based on consumption

Already there Within 1 yr Within 3 yrs Within 5 yrs

Within 10 yrs Later Never Don't know



How much are the following standing in the way of progressing towards the vision?

0% 20% 40% 60% 80% 100%

Inability of suppliers to deliver on visions and promises

Lack of interest/appreciation from senior management

The business not ready to upset the status quo

IT not ready to upset the status quo

Cultural impediments to investment in shared infrastructure

Ingrained IT funding models don’t support new ways of working

Historical under-investment means the mountain’s too high toclimb

Lack of a formally defined vision and strategy

We simply don’t have the time, resources or budget to focus onanything other than short term priorities

Big impediment Significant challenge Minor challenge Not a problem Unsure



When it comes to driving forwards, how is it playing out, or how do you think it’s going to play out in your organisation?

Other4%

Things will be left largely as they are for the foreseeable future

18%

Just let adoption of modern architectures, tools and

techniques creep along in an ad hoc manner

20%

Build a modern environment for new stuff, and migrate older systems

into it incrementally43%

Build a modern environment for new stuff, and leave older systems

where they are11%

Single big transformational initiative to modernise

things across the board4%



Summing Up

Can you have you cake (OK Apple) and eat it?


Moving forwards

• Understanding is needed• Communicate – often and in language business users can understand• Ensure governance processes are ‘big data aware’

Senior business awareness raising

• Proactive local involvement in planning/prioritisation • Don’t forget about getting ‘big data’ derived results back into business use• Clear policies/discipline around data usage

Minimisation of ad hoc adoption

• Monitor usage patterns to spot trends early• Don’t get hung up on transient fads and fashions• Embrace, substitute or block more persistent activity

Identify and deal with real business issues

• More of an orchestration approach to IT leadership• Architect systems with hard core and flexible edges• Virtualise the edge to handle diversity and personal use

Facilitate flexibility via core IT

Big Data

Data Centre


Adoption attitudes and tactics

Resist new ideas and solutions

Allow new stuff to creep in passively

Ad hoc opportunistic adoption

Draw a line, target new apps only

Establish beachhead, then expand

Big bang migration of everything

Recommended by most early movers

Recipe for disappointment

Rarely affordable in cost or risk


Available for download now:

Big Data and AnalyticsDazzling new solutions or irritating new hype?

Available from www.freeformdynamics.comhttp://www.freeformdynamics.com/fullarticle.asp?aid=1590

A Vision for the Data CentreAre you a Mover, Dreamer or Traditionalist?

Available from www.freeformdynamics.comhttp://www.freeformdynamics.com/fullarticle.asp?aid=1604

big data and analytics

Technology