Sensing the world with Data of Things By:Sriskandarajah Suhothayan (Suho)

Technical Lead at WSO2@suhothayansuho@wso2.com

STRUCTURE DATA 2016MARCH 9 - 10 • SAN FRANCISCO

Any customer can have a car painted any colour that he wants so long as it is black

~ Henry Ford ~

Me Me Me !!!

Your customers want to have a personalized experience. We are in the time of ME!

What to do ?

Is IoT New ?

• source: http://community.arm.com/groups/internet-of-things/blog/2014/06

Internet of Things

http://na1.www.gartner.com/imagesrv/newsroom/images/HC_ET_2014.jpg;wadf79d1c8397a49a2

source : http://na1.www.gartner.com/imagesrv/newsroom/images/HC_ET_2014.jpg;wadf79d1c8397a49a2

IoT Ecosystem

WSO2 IoT Server M3 : https://goo.gl/nhbxnG

http://wso2.com/iot

Concepts of IoT Analytics

● Type of Data● Distributed Nature● Event-Drivenness ● Possible Type of Analytics● Scalability ● Edge Analytics● Uncertainty

Data Types of Things

● Time based data○ Continuous monitoring & reporting ○ Time series processing (e.g. Energy

consumption over time)○ Specialised DBs - OpenTSDB

● Location based data○ Things are allover the place & they move○ Tracked via GPS / iBeacons○ Geospatial processing (e.g Traffic planning,

better route suggestion for vehicles) ○ Geospatial optimised processing engines -

GeoTrellis

IoT is Distributed

● Constant changes ○ When components added and removed○ Data flows are modified or repurposed

● Data collection need to support ○ Weak 3G networks to Ad-hoc peer-to-peer networks. ○ Message Queuing Telemetry Transport (MQTT) ○ Common Open Source Publishing Platform (CoApp)○ ZigBee or Bluetooth low energy (BLE)

● Dynamic scaling ○ Hybrid cloud

IoT Analytics are Event-Driven

● Sensors report data as Event Streams ● Analysis on flowing (or perishable) data

● Realtime Analytics○ Detect temporal and logical patterns○ Identify KPIs and Thresholds○ Send out alerts immediately ○ E.g. Alert when temperature sensor hit a limit, notify in

car dashboard of low tire pressure ○ Systems : Apache Storm, Google Cloud DataFlow &

WSO2 CEP

History Repeats

● Present vs usual behavior ● Understand the history

● Batch Analytics○ Perform periodic summarisation/analytics ○ E.g. Average temperature in a room last month, total

power usage of the factory last year ○ Systems : Apache Hadoop, Apache Spark + Storage

● Ad-Hoc Queries

● Interactive Analytics○ Provides searchability ○ E.g. Identify fraud rings from simple fraud alerts ○ Systems : Apache Drill, indexed storage systems such

as Couchbase, Apache Lucene

Deep Investigations

Thinking Ahead

● When you don’t Know the equations ● Focusing conditions & preventing issues

● Predictive Analytics○ Incremental Learning ○ E.g. Proactive maintenance, fraud detection and health

warnings○ Systems : Apache Mahout, Apache Spark MLlib,

Microsoft Azure Machine Learning, WSO2 ML, Skytree

Technology we’ve chosen

Realtime Batch

Interactive Predictive

WSO2 Data Analytics Server

Plenty of Data

Scalable Data Processing

source : http://www.websitemagazine.com/content/blogs/posts/archive/2014/09/25/customer-service-in-2039.aspx

Scalable Realtime Deployment

More info : https://docs.wso2.com/display/CEP410/Creating+a+Storm+Based+Distributed+Execution+Plan

Scalable Deployment

Interactive

Batch Realtime & Predictive

● Publishing all events is not good!○ Hardware may not be scalable ○ Network getting flooded

● What we usually need ○ Aggregation over time ○ Trends that exceed thresholds○ Event matching a rare condition

● Results in○ Local optimisation○ Quick detection of issues○ Instant notification

Is Every Event Significant?

Edge Analytics

Analytics on the Edge with WSO2 Siddhi

Push

Outliers ...

● E.g. Anomaly detection, Fraud Analytics

● Alerts for known and unknown frauds and Deep Search Analyticshttps://goo.gl/TWV5C1

Outliers

● We used: Linear Regression, Markov Models & Credit Scoring

Uncertainty in Data of Things

Data can be ● Duplicated● Arrives out of order● Not arrive at all● Wrong readings

Events Duplicates & Out of Order …

● Due redundant sensors & network latency ● Difficult for temporal data processing

○ Time Windows ○ Temporal ordering

● Such as Fraud detection

define stream Purchase (price double, cardNo long,place string);

from every (a1 = Purchase[price < 10] ) -> a2 = Purchase[ price >10000 and a1.cardNo == a2.cardNo ]

within 1 dayselect a1.cardNo as cardNo, a2.price as price, a2.place as placeinsert into PotentialFraud ;

Events Arriving Out of Order

E.g. Realtime Soccer Analytics (DEBS 2013) https://goo.gl/c2gPrQ

● Identify ball kicks, ball possession, shot on goal & offside

● Solutions : K-Slack Based Algorithmshttps://www2.informatik.uni-erlangen.de/publication/download/IPDPS2013.pdf

Missing Data

● Due to network outages

● E.g. Smart Meters (DEBS 2014)○ Smart home electricity data: 2000 sensors,

40 houses, 4 Billion events in four months○ Processed 400K events/sec

● Solutions: ○ Approximate using complimenting

sensor reading ■ Electricity Monitoring

● Frequent Load readings● Occasional Work readings

○ Fault-tolerant data streams : Google Millwheel

Wrong Sensor Readings

● From GPS ● E.g.TFL Traffic Analysis

○ Using Transport for London open data feeds.

○ http://goo.gl/04tX6k, http://goo.gl/9xNiCm

○ Scales to 500,000 Events/Sec and more

● From iBcons at shops, ships and airport

● Solution: Kalman Filter

Visualisation

● Per-device & Summarization View● Ability to group by categories

● Solutions: Composable Dashboard with sampling & indexing

Communicate to Mobile & 3rd Party Apps

● Expose analytics Results as API○ Mobile Apps,

Third Party

● Provides ○ Security, Billing, ○ Throttling, Quotas

& SLA

● Solution ○ Write data to database ○ Expose them via secured APIs (E.g. WSO2 API Manager)

Reference Architecture for IoT Analytics

IoT Analytics

●○

●○

●○

http://wso2.com/analytics

http://wso2.com/iot

Thank You

Any Questions ?

Contact us !