View, Act, and React: Shaping Business Activity with Analytics,
BigData Queries, and Complex Event Processing
Srinath PereraDirector, Research
WSO2
Start
• 1942, Asimov wrote a book called Foundation, in which the character Hari Seldon use mathematical models to predict the future of civilization and then to save it.
• Paul Krugman,( the Nobel Laureate in Economics), said his interest in economic begin with foundation.
• We are entering that Era of our history where Mr. Asimov might have a point.
Image cedit, CC licence, http://ansem315.deviantart.com/art/Asimov-
Foundation-395188263
Consider a Day in your Life• What is the best road to take?• Would there be any bad
weather?• What is the best way to invest
the money?• Should I take that loan?• Is there a way to do this faster?• What others did in similar
cases?• Which product should I buy?
Bigdata Landscape
Big Data Architecture
Why it is hard?• System build of many
computers (1000 nodes to store 1PB with 1TB each)
• That handles lots of data (10Gb network => 83 days to copy 1PB)
• Running complex logic (models can be complex as the system)
• This pushes us to the frontier of Distributed Systems and Databases http://www.flickr.com/photos/mariachily/5250487136,
Licensed CC
Big Data Architecture with WSO2
Event Streams• We view the world as event
streams Event stream is series of events over time
• We use SQL like languages (Hive/ CEP) to process event streams and create new event streams
{'name':'PlayStream','version':'1.0.0', 'payloadData':[
'name':'sid', 'ts':'BIGINT','x':'DOUBLE',...
]}
Each stream has a name
Each event has attributes, that has
types
Select from PlayStream[x>2500 and .. ]İnsert into NearGoalStream
Demo Usecase (DEBS 2013)• Football game, players and ball
has sensors (DESB Challenge 2013)
• sid, ts, x,y,z, v,a• Use cases: Running analysis,
Ball Possession and Shots on Goal, Heatmap of Activity
• Siddhi did 100K+ on each usecase
• For this talk, we will look at user activity by region of the field.
Demo High-level Architecture
Data Collection• Can receive
events via SOAP, HTTP, JMS, ..
• WSO2 Events is highly optimized version (400K events TPS)
• Default Agents and you can write custom agents.
Agent agent = new Agent(agentConfiguration);publisher = new AsyncDataPublisher(
"tcp://localhost:7612", .. );
StreamDefinition definition = new
StreamDefinition(STREAM_NAME,VERSION);
definition.addPayloadData("sid", STRING);... publisher.addStreamDefinition(definition);... Event event = new Event();event.setPayloadData(eventData);publisher.publish(STREAM_NAME, VERSION, event);
Business Activity Monitor
BAM Hive QueryFind how much time spent in each cell.
CREATE EXTERNAL TABLE IF NOT EXISTS PlayStream …select sid,
ceiling((y+33000)*7/10000 + x/10000) as cell, count(sid) from PlayStream GROUP BY sid, ceiling((y+33000)*7/10000 + x/10000);
Complex Event Processor
CEP Querydefine partition sidPrt by PlayStream.sid, LocBySecStream.sid
from PlayStream#window.timeBatch(1sec) select sid, avg(x) as xMean, avg(y) as yMean, avg(z) as zMean insert into LocBySecStream partition by sidPrt
from every e1 = LocBySecStream -> e2 = LocBySecStream [e1.yMean + 10000 > yMean
or yMean + 10000 > e1.yMean] within 2sec select e1.sid insert into LongAdvStream partition by sidPrt ;
Calculate the mean location of each player every
second
Detect more than 10m run
Run Demo
Visualization
Conclusion
Thank You
