mongodb europe 2016 - enabling the internet of things at proximus - belgium's largest telecoms...

ENABLING IOT ATPROXIMUS

HI, MY NAME IS DAVE.

Software Engineer Ordina Belgium

dave.degroote@ordina.be

IN THIS SESSIONHow to implement mongoDB in a IoT M2M use case.Data modelling is key when storing time series data.How to insert data into mongoDB using Rest interface.

INITIAL QUESTIONSHow to store data while keeping storage amount minimal?What storage engine to choose?How to store my data using a third party system?How can I easily deploy my mongoDB solution?

ESSENTIAL PROBLEMSHow do we model our data?De�ning storage engineSetting up your mongo InstancesHow do we insert our data in mongoDB?How do we query our data?

SETTING THE SCENEMythings Project at ProximusWhat is IoT?LoRa Technology

2 Front-end 3 Back-end

MYTHINGS DEVELOPMENT TEAM

Small team:

Mongo experience: 1,5 year

MYTHINGS AT PROXIMUS

MYTHINGS AT PROXIMUS

INTERNET OF THINGSVery broad, not contained to a single technology.

Not what but HOW !

Internet is the keyword

Collecting, analyzing and above all communicating data.

LORA TECHNOLOGY

LORA TECHNOLOGY VS ...

LORA ARCHITECTURE

SENSORS

Any type is possible as long as it is connected with a LoRa chip CO2, PIR, Humidity, Luminance, Temperature, Pressure, …

SENSORS

multiple containers per sensor possible

LORA ALLIANCE

https://www.lora-alliance.org

ESSENTIAL PROBLEMSHow do we model our data?De�ning storage engineSetting up your mongo InstancesHow do we insert our data in mongoDB?How do we query our data?

DATA MODELINGSENSOR HISTORY

company infosensor infocontainer infonoti�cation valuenoti�cation timestamp

SENSORHISTORY COLLECTION{ "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "data": { "type": "bool", "ts": 1444139821000, "value": "1" } }

Daily document size/sensor =

SENSORHISTORY COLLECTION SIZING

Document size = 496 BytesSensor will send 48 time a dayOn average 3 containers/sensor

496 B * 48 documents * 3 = 71,4 KB

71,4 KB

1 M sensors => 68.09 GB/day

Document with 3 additions:

BUCKETING TECHNIQUE

Data array (data values)Count metadata (Array size)Date �eld (day timestamp)

Use in combination with Upsert mechanism!

SENSORHISTORY COLLECTION (BUCKETING){ "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "date": 1444086000000, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "count": 48, "data": [ { "type": "bool", "ts": 1444139821000, "value": "1" }, { "type": "bool", "ts": 1444152754000, "value": "1" },... ]}

daily document size/sensor =

SENSORHISTORY COLLECTION SIZING(BUCKETING)

Document size = 3.98 KBSensor will send 48 time a dayOn average 3 containers/sensor

3.98 KB * 1 document * 3 = 11,9 KB

11,9 KB

1 M sensors => 11.34 GB/day

Use if possible (depending on use-case)

vs

DATA MODELINGBUCKETING

11GB/day 68GB/day

Concurrent updates on the bucket document may lead to queue

operations Do not overdue it!

DATA MODELINGBUCKETING

Advantages:Collection size will be smallerIndexes size will be considerable reducedReads will be faster

Remarks:

! Maximum document size 16MB !

DEFINING STORAGE ENGINEAPPLICATION REQUIREMENTS:

Write intensiveKeep storage volume to a minimumRead and write simultaniously to collection

DEFINING STORAGE ENGINEWIREDTIGER 3.2

Document level lock instead of collection levelData is stored in a compressed formatIndex size is smaller

Matched the application requirements

WiredTiger was recommended by MongoDB

SETTING UP MONGODB INSTANCESCLOUD MANAGER

Once automation agent is installed, smooth sailing!

SETTING UP YOUR MONGODBINSTANCES

CLOUD MANAGER

Easy setupEasy managementMetrics

INSERTING DATAMYTHINGS ARCHITECTURE

INSERTING DATAREST INTERFACE

Sleepy Mongoose (Python)https://github.com/mongodb-labs/sleepy.mongoose Rest Heart (Java)latest feature support http://restheart.org/ Advised to change to Rest Heart by MongoDB

Latest feature support

Uses PATCH method for updates (Upsert) => Not supported current release

Not mature enough: no bucketing for upsert!

INSERTING DATAREST HEART

Back to square one!

INSERTING DATASLEEPY MONGOOSE

Interchange Rest heart <> Sleepy Mongoose

Easy transition (1-2 days development)!

INSERTING DATASLEEPY MONGOOSE

Simple insert:

Advanced upsert:

INSERTING DATA

$ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_insert'

$ curl --data 'criteria= { "sensor":{"sensorId":1533,"name":"Binary mesurement","mac":"020000FFFF00B151"}, "company":{"companyId":134,"name":"Opinum"}, "container":{"containerId":1223,"name":"Door counter"}, "date":1456354800000 }, "count":{"$lt":100} &newobj={ "$push":{"data":{"ts":1456403435000,"value":"456","type":"int"}}, "$inc":{"count":1}, "$setOnInsert":{"location":{"latitude":"-1","longitude":"-1"}}} &upsert=true' 'http://localhost:27080/foo/bar/_insert'

QUERING DATAQuery frameworkAggregation framework

Retrieve sensorhistory for a week for all companies:

AGGREGATION FRAMEWORK

"op" : "command", "ns" : "sensoringlabs.sensorHistory", "command" : { "aggregate" : "sensorHistory", "pipeline" : [{ "$match" : { "company.companyId" : { $in" : [,54,61,76,77,79,80,81,82,83,86,87,90,122,123 ,124,125,126,127,128,129,130,131,132,133,134,135,136, 137,138,146,147,148,149,150,151,152,153,154] } } }, { "$match" : { "date" : {"$gte" : NumberLong("1457049600000")}, "$and" : [{"date" : {"$lte" : NumberLong("1457568000000")}}] } }, {"$skip" : NumberLong(0)}, {"$limit" : NumberLong(100)} ], "cursor" : {} } "responseLength" : 278888, "protocol" : "op_command", "millis" : 366

Retrieve sensorhistory for a week for all companies:

QUERY FRAMEWORK

"op" : "query", "ns" : "sensoringlabs.sensorHistory", "query" : { "find" : "sensorHistory", "filter" : { "company.companyId" : { "$in" : [2,54,61,76,77,79,80,81,82,83,86,87,90,122,123,124, 125,126,127,128,129,130,131,132,133,134,135,136,137, 138,146,147,148,149,150,151,152,153,154] }, "date" : { "$gte" : NumberLong("1457049600000"), "$lte" : NumberLong("1457568000000") } }, "limit" : 100, "singleBatch" : false } "nreturned" : 100, "responseLength" : 278888, "protocol" : "op_command", "millis" : 1

Same search:

Query framework :

Aggregation framework :

QUERING DATA

1ms

360 ms

-> fast for retrieving data

-> data aggregations that take more time

Read concern: read from secondary

QUERING DATAQuery framework

Aggregation framework

Query example:

QUERING DATASLEEPY MONGOOSE

$ curl -X GET 'http://localhost:27080/foo/bar/_find ?criteria={"_id":{"$oid":"4f8c6f05db61e2a72600001d"}} &sort={"date”:-1} &limit=1'

INDEX STRATEGYCOMPOUND INDEXES

Multitenant system: company idTime based data: date

Data retention policy:

-> Deterministic way to drop documents once expired

-> Fixed size collections

CC Not suitable because time range of documents will depend onthroughput!

(high throughput will decrease the time range stored).

CAPPED COLLECTION VS TTL INDEXESGETTING RID OF THE EXPIRED DATA

3 months

TTL index

Capped collection

PUTTING IT ALLTOGETHER

Bucketing

DATA MODELING

Wired tiger 3.2

STORAGE ENGINE

Sleepy Mongoose RestHeart (future) Java mongo Driver

DATA INSERTION

query framework aggregation framework

DATA EXTRACTION

compound indexes TTL index

INDEXES

SHARDINGWHEN

There is more data than one machine can hold on its drivesThe application is write heavy and you could experience too muchlatencyThe working set outgrows the memory can be allocated to a singlemachine

CONCLUSIONWHAT WE LEARNED

Take enough time to model new collections (bucketing)=> Can decrease the storage volume when done right

The use of cloud manager drastically decreases deployment andmaintenance time=> More time to think about your application

There are endless ways of getting your data into mongoDB=> Mongo supplies and support many di�erent solutions

THANKS FOR LISTENING!Questions?