getting more out of your big data
DESCRIPTION
A presentation we gave together with Microsoft at the latest inspirience days in Belgium.TRANSCRIPT
Getting More out ofYour Big DataRitchie HoutmeyersWesley BackelantMicrosoft
Geert Van Landeghem
Nathan Bijnensdatacrunchers
The Challenge of Data
10x increase every five years
85%
from new data
types
Dataexplosion
By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent.
– Gartner, Mark Beyer, “Information Management in the 21st Century”
Easy Accessibility of External Data
Hadoop
Cloud
Cheap, Distributed Storage &
Processing
Volume
Velocity
Variety
Creating New Business Opportunities
Revenue Growth
Increases ad revenue by processing 3.5 billion events per day
MassiveVolumes
Processes 464 billion rows per quarter, with average query time under 10 secs.
Business Innovation1
Measures and ranks online user influence by processing 3 billion signals per day
CloudConnectivity
Connects across 13 social networks via the cloud for data and API access
Operational Efficiencies
Uses sentiment analysis and web analytics for its internal cloud
GE
Real-TimeInsight
Improves operational decision making for IT managers and users
1. Klout Case Study: http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2012-Enterprise/Klout/Data-Services-Firm-Uses-Microsoft-BI-and-Hadoop-to-Boost-Insight-into-Big-Data/710000000129
Please Welcome
Geert Van LandeghemManaging Director
Big Data’s three drivers
Volume
Velocity
BigData
Variety
Gartner
McKinsey
Forrester Research
Big Data
Creating Transparency
Enabling Experimentation
to discover needs, expose variability and
improve performance
Segmenting populations to customize actions
Replacing/Supporting human decision making
with automated algorithms
Innovating new business models, products and services with big data
Big Data Transforms
Big Data defined
Big Data Technologies allow you to implement Use Cases which Legacy Technologies can’t.
Use Case: Truvo
HADOOP Data Repository
Internal Data through APIs
Fetcher & Parser to enrich & validate with external data
Data Silos
High Velocity changes
Cost of Changes
Busi
ness
C
halle
nges
Solu
tion
Solu
tion
Benefits
Faster Updates
Full dataset refresh possible every week instead of a few times per year
Cost Reduction
Significant reduction in validation phone calls
Solu
tion
Benefits
Use Case: Trimble
HADOOP Real-TimeArchitecture
Scaleable architecture to support current and future real-time insight needs
High Volume & High Velocity
Old solution not able to handle incoming volumes of data in timely mannerB
usi
ness
C
halle
nges
Solu
tion
Solu
tion
Benefits
Faster Insights
Realtime handling of the growing Volume & Velocity of the data. Adding at least 1TB per year.
Grow with Needs
Solution scales with business needs without upfront cost
Solu
tion
Benefits
Use Case: UZ Brussels
HADOOP Data & Processing Cluster
Scalable Image Library
Processing cluster to process images
High Variety &High Volume
Analyses of 30.000+ giant images of medical scans of Pancreas
Busi
ness
C
halle
nges
Solu
tion
Solu
tion B
enefits Faster Research & Diagnostical Insight
Diagnoses can be validated against previous diagnoses
New research ideas can be checked across full image set
Cost Friendly Reliability Improvement
Inexpensive data duplication over HADOOP storage nodes provides needed reliability improvements
Solu
tion B
enefits
Share your data with the world via Azure Marketplace
Enrich with social media data via Social Analytics
Advanced analytics with Hadoop
Connecting with the World’s Data
Analyze Big Data with familiar tools
Immersive insights from any data
JavaScript based simple programming
Immersive Insight, Wherever you are
Simplicity and manageability of Windows to Hadoop
Extended data warehousing with Hadoop
Scale & elasticity of cloud
Any Data, Any Size Anywhere
HDInsight - Microsoft’s approach to Big Data
Unlocking new insights from all data with familiar tools
Hive Excel Plugin, ODBC Driver integrates Hadoop to SQL Server Analysis Services, PowerPivot, and Power View
Familiar BI tools with structured and unstructured data
Benefits
Key
Featu
res
Extending your Enterprise Data Warehousewith hadoop
Integration with enterprise BI solutions
Microsoft SQL Server connector for Apache Hadoop with SQOOP (SQL to Hadoop)
Integration with Microsoft Enterprise Data Warehouses
SQL Server Parallel Data Warehouse connector for Apache Hadoop with SQOOP
Deeper insights from structured and unstructured data
Benefits
Key
Featu
res
Enhances your data through predictive analysis on Hadoop
Unlock rare patterns from bespoke data mining models
Support for open source predictive analytics tools such as R and Mahout
New business insights with predictive analytics from Microsoft
Hive ODBC Driver connects Hadoop to SQL Server Data Mining tools
Benefits
Key
Featu
res
Microsoft uniquely Connects Hadoop to the world via Windows Azure Marketplace
Mashing up of internal and public data sets via Data Explorer
Integration with third-party data and services
Sharing of data and insights through Windows Azure Marketplace
Integration with Windows Azure Marketplace
Benefits
Key
Featu
res
Enriches analysis with social media data via social analytics
Integration of social information with business applications
Social Analytics
Stronger customer relationships
Integration with social media sites
Models augmented with publicly available data from social media sites
Benefits
Key
Featu
res
Mic
roso
ft H
DIn
sight
HDInsight: Bring the simplicity and manageability of windows to Hadoop with Microsoft Support
Enterprise-class security
Integration with Microsoft System Center
Integration with Windows Server® Active Directory
Simplified management of Hadoop on Windows
Smart packaging of Hadoop on premises
Fast deployment of Hadoop on Azure
100% Microsoft Support
Easy setup on-premises and in the cloud
Benefits
Key
Featu
res
Mic
roso
ft H
DIn
sight
HDInsight: Choice of Deployment options
Elastic peta-scale analytics on Microsoft’s cloud platform
Hadoop-based Service on Windows Azure platform
Enterprise-class Big Data platform on-premises
Hadoop-based distribution on Windows Server
Benefits
Key
Featu
res
Demo: From Data to Insights
Wesley BackelantTechnology & Solution Advisor
Simplicity Analysis with familiar tools
Collaboration on insights
Nathan Bijnensdatacrunchers
1. Take a large problem and divide it into sub-
problems
2. Perform the same function on all sub-problems
3. Combine the output from all sub-problems
…
…
Output
MA
PR
ED
UC
E
MapReduce explained
DoWork() DoWork() DoWork()…
A holistic BIG DATA Solution from Microsoftspanning relational and non-relational Worlds
NON-RELATIONAL
100111
DATA MANAGEMENT
SHAREAND GOVERN
DISCOVERAND RECOMMEND
TRANSFORMAND CLEAN
INSIGHTS
DATA ENRICHMENT
OPERATIONAL
SELF-SERVICE MOBILE
PREDICTIVE
REAL-TIMECOLLABORATIVE
MA
RK
ETPLA
CE
Exte
rnal
Data
and
S
erv
ices
RELATIONAL MULTIDIMENSIONAL STREAMING
Microsoft Services Big Data Starter OfferObjectives
Starter Offer
Structured (+- 4/5weeks) engagement that demonstrates the capabilities of the Microsoft Big Data platform with a prototype using real customer data
Who Delivers
Microsoft Consulting Services & Industry Experts
Expected Outcome
Define Big Data Company Strategy
Implement Big Data Prototype solution
Customer Meeting to discuss Big Data Needs & Scoping for Starter Offer
Scoping
• Build customer confidence in Microsoft’s comprehensive Big Data platform
• Define Big Data Company Strategy• Showcase ease of use and implementation• Implement Big Data Prototype Solution
At your serviceScan: Invite Analytics Cloud Garden
Scan: Promo SQL Server
Have a one-to-one with a specialist, Visit Plug in to Experience
Visit our Partners in the expo area
eo
expo
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.