big data and advanced spatial analytics - aventri · projects contracts enterprise project...

1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Insert Information Protection Policy Classification from Slide 8 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Big Data and Advanced Spatial Analytics Eve Kleiman Director, APAC Technology Product Releases & Management Programs Oracle



Disappearing line between Geospatial Technologies and Information Technologies

Mapping Digital data file Spatial Information

Technology

SOA

Geographic Information Systems



Java, Databases, BI, Applications, Cloud

Location Infused Technology



Internet of Things

The New Infrastructure

http://www.oracle.com/ocom/groups/systemobject/@mktg_admin/documents/webcontent/videoplayer-ocom.html?bctid=2229087939001&playerType=single-social&size=c23�



Internet of Things| Open a World of Possibilities Total economic value add estimated at

$1.9 Trillion, Source:Gartner


Need for an Enterprise Platform

Standardization

Streamline how IoT applications are developed,

secured & deployed

1

Real-Time Analytics

Manage IoT data from collection,

storage, processing and

analysis

2

Integration

Connect intelligent devices

to existing enterprise

applications

3 Security

Manage security and identity of data, devices,

enterprise applications

4



Definition: What is Big Data Anyway? The 4 ‘V’s of Big Data

Volume – It’s about 100s Terabytes and Petabytes... BUT could be smaller volumes – size is in the eye of beholder

Value – At volume low value data can highlight high value patterns, trends and insights of significant commercial value

Variety – Highlights the new types of data from the ‘internet of things’ that can be stored and analysed

Velocity – Relates to streams of high frequency data arriving e.g. From a sensor, network, Imagery, UAV, or high volume event



• Mobile Sensing & Geocoding – Mobile tracking – Automated Data Capture / Geocoding – Filtering / Geocoding

• 3D data into mainstream – Apple, Google, Microsoft – 3D city models, terrains, imagery

• Enhancing Web and Enterprise apps – Business intelligence & operations

Spatial Data From Scarcity to Abundance

8



Conquering Big Data

If you could have all the data you wanted, what would you do differently?

The Business Opportunity 100 million daily 1 billion visitors / yr Web Transactions eCommerce

1 million daily Image uploads Claims analysis Insurance

10 billion daily Device syncs Consumer

10 million meters Hourly uploads Utility

20 million daily Monitoring Healthcare



MEDIA/ ENTERTAINMENT Viewers / advertising effectiveness Cross Sell

COMMUNICATIONS Location-based advertising

EDUCATION & RESEARCH Experiment sensor analysis

Retail / CPG Sentiment analysis Hot products Optimized Marketing

HEALTH CARE Patient sensors, monitoring, EHRs Quality of care

LIFE SCIENCES Clinical trials Genomics

HIGH TECHNOLOGY / INDUSTRIAL MFG. Mfg quality Warranty analysis

OIL & GAS Drilling exploration sensor analysis

FINANCIAL SERVICES Risk & portfolio analysis New products

AUTOMOTIVE Auto sensors reporting location, problems

Games Adjust to player behavior In-Game Ads

LAW ENFORCEMENT & DEFENSE Threat analysis - social media monitoring, photo analysis

TRAVEL & TRANSPORTATION Sensor analysis for optimal traffic flows Customer sentiment

UTILITIES Smart Meter analysis for network capacity,

Sample of Big Data Use Cases Today

ON-LINE SERVICES / SOCIAL MEDIA People & career matching Web-site optimization



What Does a Big Data World Look Like? Truck / Motor Manufacturer

What they collect •Monitors the complete car •Kilometers Per Litre, Driving techniques •Location information

How they use it •Better tailored servicing plans •Better targeted marketing •Offer better finance deals or related options •More data for R&D •Sell on to partners

Big Data means… •The ability to predict when components might fail •Create better service plans based on actual usage •Drivers can post their data to manufacturers website •Black box data can help with accidents



• Verizon Telematics provides a full service end-to-end solution

• Supports over 50 Telematics features (vehicle, driver, social, etc.)

• Verizon Telematics provides a modular platform

• Verizon Telematics supports a curated ecosystem

Examples

• The global first implementation of a city wide “smart” parking management system and technology to manage parking supply and demand more intelligently.

• Sensors at 8,200 of the city’s 27,000 metered parking spaces, to get information from the gate arms at the city’s 14 garages, which among them have about 13,000 spaces

Smart City: Parking Management



• Provide a composite identity for customers and cars

• Need for a reliable 24/7 IAM infrastructure

• Leverage Mobility/Web technologies to provide advanced Infotainment and Safety Services

Telematics

Examples (cont.) Subsidiary of GM that provides

subscription-based communications, in-vehicle security, hands free calling, turn-by-turn navigation, and remote diagnostics systems

OnStar relies on Oracle to deliver service to 5.5 million car owners

• Location Intelligence Company • GIS, Mobile, LBS products • Spatial Content & Platform Provider • InfoMobil vehicle tracking and

dispatching system • 800 Customers, 2 Operation Centers,

50,000+ Mobile Devices, 2000+ Users



What Does a Big Data World Look Like? Retailer / White Goods

What they collect •Food monitoring by RFID tags –Fridge monitors food usage and sell-by dates •Video Surveillance •Feedback / comments

How they use it •Automated food ordering •Better targeted marketing due to better data about how/when customer uses products

•More detailed information about customers •Better store based demand planning

Big Data means…


New ways sit along side existing ways to generate data Oil & Gas Industry

Business Users Data Warehouse

Data Marts

Enterprise Data Warehouse

Source Systems

EXTRACTS

REAL-TIME DATA EXTRACTS

PROCESS MANUFACTURING

PROJECTS CONTRACTS

ENTERPRISE PROJECT PORTFOLIO MANAGEMENT TELESERVICE

ENTERPRISE ASSET MANAGEMENT

FIINANCIALS

FEEDS

3C/4C Seismic Sensors

Micro-seismic monitoring

Time-lapse Seismic data

Monitoring Infrastructure

Borehole Data Acquisition

Environmental Monitoring

Asset Management



Telco Industry Big Data Use Cases

Location based marketing Real-time analytics combined with complex event processing to create analysis across the GPS data that a service provider can collect to understand the customers current location or look for patterns or relationships.

Network optimization Big Data is used to deliver real-time analytics to detect when a network is down, overloaded or reaching capacity.

Sentiment analysis Use of social media (Twitter, Facebook, etc.) - this is not just comms specific but a good example applicable across industry. Collect/stream data from social media sites into CRM and customer service applications to determine importance/”clout” of customer and to get a better overall picture of each customer’s behavior, likes and dislikes.

Weblog or clickstream analysis Applicable across industries; allowing service providers to better understand how their web site or online store is navigated.



Incoming Data

• Track outcomes • Track treatments, notes • Track social media, payments • Correlate Omics w/EMR data • Correlate biometrics data across

devices

• Model protocols • Model outcomes • Model normal behavior

• Visualize data • Discover unknowns • Highlight outliers • Determine risk • Populate metrics • Report

• Prevent • Predict • Diagnose • Treat • Measure

Pharmaceutical

Activity / Claims Cost

Clinical

Lifestyle/History

Metabolic

Biometrics

Healthcare Industry Complete secure solution from data collection to analysis to decision

EMR

Social

Stream Acquire Organize /Discover

Analyze, Visualize Decide



• Fuelband monitors active lifestyle of 8 million users on a daily basis

• Current data grid volume is approximately 150,000 request per minute with about 40 million objects at any given time on the grid

Healthcare

Nike



FSI - Bank Real-time Location Based Offers

Objectives

Customer profile enrichment with Big Data Capture credit card POS and merchant data with

event processor Determine geo location of POS and nearby bank

wholesale customers Leverage real-time decision engine to generate

offer to mobile device

Solution

Increase revenue through real-time, location based offers



The Challenge

• “Finding Useful Data” – Modeling, Statistics

– Econometrics

– Decision Science

– Psychology

Recognizing a Pattern – Predicting Behavior



Make Better Decisions Using Big Data

Big Data in Action

ANALYZE

DECIDE ACQUIRE

ORGANIZE



Acquire all available data

Big Data In Action

ANALYZE

DECIDE

ORGANIZE

ACQUIRE



Hadoop Architecture

Management/Monitoring

Hadoop Distributed File System (HDFS)

MapReduce

Distributed file system Map/Reduce programming paradigm Highly scalable data processing



NoSQL Databases

• A Distributed Key-Value Database

Nodes East

Nodes West

Nodes Central

NoSQL Driver

Application

NoSQL Driver

Application R

ead

Del

ete

Rea

d

Upd

ate

Simple data model

Scalability

High Availability

Transparent Load Balancing

Simple Administration



SMS Message

Immediate Automatic Responses

Real-Time Streaming Data

Workflow Initiation

Real-time Dashboards

Console Alerts

Aggregate, Correlate, Filter

Virtual Data Repository

Pattern Detection

Event capture

Enrichment

Real-time Data Streams

Complex Event Processing



Choose the RIGHT tool for the job Hadoop Distributed File System

(HDFS) NoSQL Databases Relational Databases

File System Key Value Store Relational Database

No inherent structure Simple data structure Complex data structures, rich SQL

High volume writes High volume random reads and writes High volume OLTP with 2-PC

Limited functionality, roll-your-own applications

Simple get/put high speed storage, flex configuration

Security, Backup/Restore, Data life cycle mgmt, XML, etc.

Batch Oriented Real-Time, web-scale specialized applications

General purpose SQL platform, multiple applications, ODBC, JDBC



Organize and distill big data using massive parallelism

Big Data in Action

ANALYZE

DECIDE ACQUIRE

ORGANIZE



Big Data Challenge – Aligning Data Types

• Semantic Graphs

Big Data: Decisions based on all your data

Machine-Generated Data Social Data

Documents

Video

Geospatial – Where is it Happening? In What Context?



RDF for Enterprise Integration

Index

Content Mgmt BI Server Data Warehouse

Machine Generated Data

RDF metadata layer (integrated graph metadata)

Transaction Systems

Big Data Appliance

Subscription Services Human Sourced

Information Social Media

Event Server

Data Servers

Data Sources / Types

Access & Presentation Layer



Graph Analysis

Recommendation

Purchase Record

customer items

• Create recommendations for a specific customer from its neighbors in a purchase graph

• Identify nodes that are critical to connectivity of an information flow graph

Influencer Identification

Communication Stream (e.g. tweets)

Pattern Matching

• Find sub-graphs that match a specified pattern

P1 P2 “ex”

P3 P4 friend friend friend

• Represent your data as a graph, analyze it and discover useful information • By considering relationships between data entities



Analyze all your data, at once

Big Data in Action

ANALYZE

DECIDE ACQUIRE

ORGANIZE ANALYZE



Navigation and Visualization of RDF graphs



Web Mapping with GeoSPARQL



Big Data Tools for Exploring & Analyzing

R Statistical Programming Language Open source language and environment Used for statistical computing and graphics Strength in easily producing publication-quality plots Highly extensible with open source community R packages

http://www.r-project.org/index.html�

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 35 35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Big Data Tools for Exploring & Analyzing Data Mining Tools for Guided Pattern Discovery & Statistical Analysis Problem

Classification Sample Problem

Classification Given demographic data about a set of customers, predict customer response to an affinity card program

Regression Given demographic and purchasing data about a set of customers, predict customers' age

Attribute Importance Given customer response to an affinity card program, find the most significant predictors

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 36

Big Data Tools for Exploring & Analyzing

Automated Discovery & Predictive Analysis Problem Classification Sample Problem

Anomaly Detection

Given demographic data about a set of customers, identify customer purchasing behavior that is significantly different from the norm

Association Rules

Find the items that tend to be purchased together and specify their relationship – market basket analysis

Clustering Segment demographic data into clusters and rank the probability that an individual will belong to a given cluster

Feature Extraction

Given demographic data about a set of customers, group the attributes into general characteristics of the customers

F1 F2 F3 F4



Text Analytics

2 miles

Spatial Analytics

Query and Reporting Data Mining

Statistics

Graph Analytics

Big Data Extends the Depth of Analytics



Decide based on real-time big data

Big Data in Action

ANALYZE

ACQUIRE

ORGANIZE

DECIDE



Summary



Q&A MORE INFORMATION

HTTP://WWW.ORACLE.COM/US/TECHNOLOGIES/BIG-DATA/INDEX.HTML

HTTP://WWW.ORACLE.COM/TECHNETWORK/TOPICS/BIGDATA/WHATSNEW/INDEX.HTML [email protected]

http://www.oracle.com/technetwork/topics/bigdata/whatsnew/index.html�