warp10: collect, store and manipulate sensor data - breizhcamp - 2016 03-24

@FinistSeb @LostInBrittany#BzhCmp #Warp10

Warp 10: Collect, store and manipulate sensor data

Horacio Gonzalez Sébastien Lambour


Horacio Gonzalez

@LostInBrittany

Cityzen Data

Spaniard lost in Brittany, developer, dreamer and all-around geek


Sébastien Lambour

@FinistSeb

Cityzen Data

Runner, 2 Kids, Geek, Handyman, Polyglot JVM Developer


Warp 10


IntroductionGeo-Time SeriesTM

Image: Spacetime distorsions

http://spacetimedistortions.blogspot.fr/2013/04/reflections-on-einsteins-space.html


Time Series

Image: Mike Bostock

https://bost.ocks.org/mike/cubism/intro/#12


Time series storage and analysis

Image: Hamza Fessi and ABC Bourse

Not suited for your vanilla SQL RDBMS

One simple example: moving averages...

https://fessih.wordpress.com/2014/02/24/moving_average_sql/

http://www.abcbourse.com/apprendre/11_lecon2.html


Geo-Time SeriesTM

Image: AIS Vessel Tracking

http://www.marinetraffic.com/


Geo-Time Series


Geo-Time Series and the IoT

Image: LinkedIn

https://www.linkedin.com/pulse/3-ways-internet-things-change-every-business-bernard-marr


IoT means talking thingHow fast are they talking?


Internet of very introverted Things

Long range transmissions


Internet of introverted Things

Personal Area Network


Internet of shy Things

Local Area NetworkCellular Networks


Lots of shy thing generate a huge lot of data

Image: Universal Studios

http://www.universalstudioshollywood.com/copyright-trademark/


Internet of chatty Things

10 000 Hz

670 000 sensors

20 000 metrics

per second


Internet of garrulous Things

Image: Google

Millions of metrics per second

https://google.com


Warp 10 : A software platform for IoT

Warp 10 is a software platform that● Ingests and stores data● Manipulates and analyzes data● Is dedicated to data from sensors, meters, IoT and any real or

virtual probe


Warp 10 General Synoptic

Stockage Architecture

Language, Functions, Algorithms

Application access

VizualisationRealTime


#collectHow do you get these metrics?

Image: Games Radar

http://www.gamesradar.com/the-8-best-game-watch-games/


Using our own Sensision agent


Using our own Sensision agent

With queue forwarder


Using plugins for other collecting systems


Or simply pushing data directly


Choosing an input format


XML? JSON?

139 bytes 108 bytes


Warp 10 GTS Input Format

57 bytes

But size isn't the most important reason

parsing time is way more important

XML or even JSON parsing is slow and costlyWarp 10 GTS input format isn't


timestamp (us by default)

latitude:longitude (WGS84)

elevation (millimeters)

classname*

labels (key=value)

value* (long, double, boolean or string)* mandatory fields

Warp 10 GTS Input Format


#storeFrom tiny to huge

Image: Games Radar

http://www.gamesradar.com/the-8-best-game-watch-games/


Warp 10 on Raspberry Pi B+

1 000 datapoints per second


Warp 10 on Raspberry Pi 2 B



Warp 10 on a modern server



Warp 10 on a cluster

3 millions of datapoints per second(our current record on input traffic)


#analyseFrom tiny to huge

Image: Amazon

http://www.amazon.com/Lego-Series-Detective-Mini-Figure/dp/B005JUV0ZU


Many time-series solutions

TSAR


But they are only stores...

Fetching data is only the tip of the iceberg


Analysing the data

High level analysis must be done elsewhere


Algorithms are resource hungry


Your computer is not a datacenter


Manipulating GTS

To be scalable, analysis must be done in Warp 10 platform, not in user's computer


A true GTS analysis toolbox○ Hundreds of functions○ Manipulation frameworks○ Analysis workflow

Manipulating GTS


GTS manipulation

Why not a simple REST API?● One endpoint by function?● How to chain a workflow analysis?

REST API not suitable for complex manipulations


GTS manipulation

Why not a SQL dialect?● How do you do a simple moving average in SQL?● How do you geo-time fencing in SQL?

SQL is not adapted to (G)TS analysis!


GTS manipulation language

Our solution: a GTS manipulation language

WarpScript


A stack based language


WarpScript

Non-compiled Optimized functions, fast execution


Basic operations


Five frameworks

●●●●●


More than 500 functions


Time series functions


Geo-Time Series functions

Geo mapping (WKT)

Horizontal & vertical speed

Horizontal & vertical distance

Haversine...


Quantum IDE


Enough teasing...


Fuel prices data

16 297 448 metrics

11 379 fuel stations

42 885 Geo Time Series


Basic analysis

Average diesel fuel prices in France since 2007

Image: LEGO Ideas

https://ideas.lego.com/projects/54995


First Fetch Data (SQL vs WarpScript )


FETCH gives us a GTS list



Timestamp (microseconds since epoch)



Location (latitude, longitude)



Value


Calculate the average

Using Groovy:


Calculate the average with WarpScript

1- Calculate the mean price by station



BUCKETIZE framework

Put the data of a GTS into regularly spaced buckets



2- Reduce to get the global average



REDUCE framework

Apply a function on a set of GTS tick by tick


Too verbose? Write it differently


Even more concise


Basic analysisMean of the last available

diesel fuel prices in France

Image: LEGO Ideas

https://ideas.lego.com/projects/54995


Fetching Data (SQL vs WarpScript )


Mean of those last prices

align ticks with BUCKETIZE framework

compute the average with REDUCE


Geo-time analysis

Find the cheapest fuel station near here

48.115434, -1.636877


WKT: Well-known text geometry


…

WKT in WarpScript


Geo-filtering points of GTS


Geo-filtering points of GTS

MAPPER framework

Apply a function on values of a GTS that fall into a sliding window


The stations near my position


There can only be one


And this is only the surfacePossibilities are endless


Think differently

Geo-Time Series are everywhere


Warp 10 platform and tools


Everything is on GitHub

https://github.com/cityzendata/


Thank you !

warp10: collect, store and manipulate sensor data - breizhcamp - 2016 03-24

Technology