cambada realtime database - tech united · cambada realtime database - the 2nd generation //...
TRANSCRIPT
![Page 1: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/1.jpg)
IRIS-Lab / IEETA / DETI / UAUniversity of Aveiro, Portugal
CAMBADA Realtime DatabaseThe 2nd Generation
Ricardo [email protected]
MSL Workshop 2017 – 16 November – Eindhoven, Netherlands
![Page 2: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/2.jpg)
Disclaimer
The RtDB 2 is a M.Sc. Thesis work by Diogo SilvaSupervised by professors Artur Pereira and Nuno Lau
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Disclaimer
2
![Page 3: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/3.jpg)
• Introduction– The CAMBADA RtDB– Motivation– Objectives
• The RtDB v2– Architecture– Design Choices– The new API
• Back-Compatibility With RtDB 1• Results• Conclusion and Future Work
Presentation Outline
3CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
![Page 4: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/4.jpg)
Introduction
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 4
![Page 5: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/5.jpg)
• RtDB is an an open-source module that handles inter-robot and inter-process communication in CAMBADA
• It allows the replication of data among different agents on the team in a key-value fashion
• Any agent can retrieve information about any teammate at any time
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
IntroductionThe CAMBADA RtDB v1
5
![Page 6: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/6.jpg)
• Blackboard logic
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
IntroductionThe CAMBADA RtDB v1
6
![Page 7: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/7.jpg)
int DB_init(void)
void DB_free(void)
int DB_put(int _id, void *_value)
int DB_get(int _from_agent, int _id, void *_value)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
IntroductionRtDB v1 API
7
![Page 8: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/8.jpg)
• RtDB v1 Limitations– Simple raw data structure binary storage in shared
memory• Inability to store dynamic items
– The user has to define a maximum size– Always store/broadcast all item capacity, although sometimes
only a portion is useful– Static Configuration Scheme
• A configuration file is used to define all local/shared structures
– Inability to dynamically (programmatically) create items in the RtDB
– Unreliable when agents are using different structures for the same key
– No compression on broadcast
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
IntroductionMotivation
8
![Page 9: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/9.jpg)
– Maintain blackboard logic– Dynamic data structures
• Allow dynamic containers (arrays) like C++ std::vector and alike– Dynamic Configuration Scheme
• Ability to add new items at will (by code)– Structure Changes Resilience
• Agents should be resilient to different schemes and data structures in the network
– Structures with new items coming should be parsed with the known items (new attributes should be ignored)
– Structures lacking attributes should also be parsed (missing attributes should retain a default value)
– No re-compilation when scheme changes• Time is precious during the competitions
– Smooth integration with the previous RtDB version• We wanted to validate the new version on the competition, but also an easy
way to fallback to the previous version if something went wrong– Duration of operations should be negligible
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
IntroductionRequirements
9
![Page 10: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/10.jpg)
The RtDB v2
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 10
![Page 11: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/11.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
11
Raw Data Structure
LocalStorage
![Page 12: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/12.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
12
Raw Data Structure
LocalStorage
ProblemsNot tailored for dynamic structuresNot resilient to differences in data structures between processes/robots
![Page 13: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/13.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
13
Raw Data Structure
Serialized Data
LocalStorage
![Page 14: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/14.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
14
Raw Data Structure
Serialized Data
LocalStorage
WiFi - Send/Receive
![Page 15: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/15.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
15
Raw Data Structure
Serialized Data
LocalStorage
WiFi - Send/Receive
ProblemGreater bandwidth usage
![Page 16: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/16.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
16
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/Receive
![Page 17: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/17.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The RtDB v2Storing and sending
17
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/Receive
![Page 18: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/18.jpg)
• We evaluated several key-value storage libraries
• Requirements:– Free– Non-copyleft license– Active project (receives regular updates)– In-memory storage (for better performance)– Support for multi-process and multi-thread access– C++ API
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
18
![Page 19: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/19.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
19
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
![Page 20: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/20.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
20
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
ExcludeCommercial
![Page 21: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/21.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
21
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACEExcludeNo in-memory
ExcludeNo in-memory
![Page 22: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/22.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
22
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
Exclude1 Writer process only
Exclude1 Writer process only
![Page 23: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/23.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
23
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
ExcludeNo multi-process writing
![Page 24: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/24.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
24
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
ExcludeNeeds additional ‘broker’ process
![Page 25: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/25.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
25
LMBDLevelDB
RedisRocksDB
BerkeleyDB
UnQLite
BangDB
Kyoto Cabinet
C-treeACE
ExcludeCopyleftlicense
![Page 26: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/26.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
26
LMBD
OpenLDAP 2.8 License (non-copyleft)
Active projectIn-memory storageMulti-processMulti-threadC++ API
![Page 27: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/27.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
27
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
![Page 28: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/28.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesSerialization
28
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
![Page 29: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/29.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesSerialization
29
CLCSchema Evolution
Binary Encoding
Text Encoding
Requires SchemaField
RenamedField
AddedField
RemovedType
Inheritance
Thrift Yes Yes Yes Yes No Yes Yes Yes
Protobuf Yes Yes Yes Yes Partial Yes Yes Yes
Avro Yes Yes Yes No Partial Yes No Yes
Msgpack Yes Yes Yes Yes Yes Yes No No
BSON Yes No Yes Yes No Yes No No
Flatbuffers Yes Yes Yes No No Yes Yes Yes
Cap’n Proto No Yes Yes No Partial Yes No Yes
Boost No Yes No No No Yes No No
Cereal No Yes No No No Yes Yes No
![Page 30: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/30.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesSerialization
30
CLCSchema Evolution
Binary Encoding
Text Encoding
Requires SchemaField
RenamedField
AddedField
RemovedType
Inheritance
Thrift Yes Yes Yes Yes No Yes Yes Yes
Protobuf Yes Yes Yes Yes Partial Yes Yes Yes
Avro Yes Yes Yes No Partial Yes No Yes
Msgpack Yes Yes Yes Yes Yes Yes No No
BSON Yes No Yes Yes No Yes No No
Flatbuffers Yes Yes Yes No No Yes Yes Yes
Cap’n Proto No Yes Yes No Partial Yes No Yes
Boost No Yes No No No Yes No No
Cereal No Yes No No No Yes Yes No
![Page 31: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/31.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesStorage
31
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
MsgPack
![Page 32: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/32.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
32
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
MsgPack
![Page 33: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/33.jpg)
• We want to compress MsgPack packets
• “It's like JSON. but fast and small.”
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
33
![Page 34: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/34.jpg)
{
“position”: [1.2, 4.5],
“velocity”: [0.2, 6.0],
“hasBall”: false,
“opponents”:
[
[4.3, 5.9],
[1.2, 2.0]
]
}
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
34
![Page 35: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/35.jpg)
{
“position”: [1.2, 4.5],“velocity”: [0.2, 6.0],“hasBall”: false,“opponents”:
[
[4.3, 5.9],[1.2, 2.0]
]
}
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
35
![Page 36: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/36.jpg)
{
“position”: [1.2, 4.5],“velocity”: [0.2, 6.0],“hasBall”: false,“opponents”:
[
[4.3, 5.9],[1.2, 2.0]
]
}
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
36
{
“position”: [???, ???],“velocity”: [???, ???],“hasBall”: ???,“opponents”:
[
[???, ???],[???, ???]
]
}
{
“position”: [???, ???],“velocity”: [???, ???],“hasBall”: ???,“opponents”:
[
[???, ???],[???, ???]
]
}
{
“position”: [???, ???],“velocity”: [???, ???],“hasBall”: ???,“opponents”:
[
[???, ???],[???, ???]
]
}
![Page 37: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/37.jpg)
• Train a dictionary for dictionary-based compression using these dummy data packets
• lzbench1 tool was used to benchmark different frameworks inside the CAMBADA system
• zstd was finally selected for the best performance
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
37
1 https://github.com/inikep/lzbench
![Page 38: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/38.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesCompression
38
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
MsgPack zstd
![Page 39: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/39.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Design ChoicesFinal Architecture
39
Raw Data Structure
Serialized Data
Compressed and Serialized
Data
LocalStorage
WiFi - Send/ReceiveLMDB
MsgPack zstd
![Page 40: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/40.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
RtDB v2 API
C
Global Functions
DB_put(int key …DB_get(int key …
DB_init()
DB_free()
![Page 41: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/41.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
RtDB v2 API
41
C C++
Global Functions
DB_put(int key …DB_get(int key …
DB_init()
DB_free()
![Page 42: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/42.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
RtDB v2 API
42
C C++
Global Functions Object Methods
DB_put(int key …DB_get(int key …
DB_init()
DB_free()
![Page 43: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/43.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
RtDB v2 API
43
C C++
Global Functions Object Methods
DB_put(int key …DB_get(int key …
DB_init() Constructor
DB_free() Destructor
![Page 44: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/44.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
RtDB v2 API
44
C C++
Global Functions Object Methods
DB_put(int key …DB_get(int key …
put(string key …get(string key …
DB_init() Constructor
DB_free() Destructor
![Page 45: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/45.jpg)
Backwards Compatibility With RtDB 1
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 45
![Page 46: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/46.jpg)
• By using a CMake custom option, we could easily revert to RtDB v1 at any time by just recompiling the code with an option– No additional actions on the code itself
• A set of macros were defined to maintain compatibility with the old API in the code
• An XML configuration file was used to ‘translate’ the old integer keys to strings
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Back-Compatibility with RtDB
46
![Page 47: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/47.jpg)
Results
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 47
![Page 48: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/48.jpg)
• We tried the RtDB v2 for the 1st time in a competition on Portuguese Open 2017 (Coimbra, Portugal)– It ran successfully without hiccups– We never had to actually use the revert function– Still using static arrays
• We continued using it on RoboCup 2017 (Nagoya, Japan)– We used dynamic arrays (std::vector instead of static
arrays)– Again, the solution proved to be effective
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Results
48
![Page 49: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/49.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ResultsTime – PUT Operation
49
RtDB 1 = 0.86 µs average
![Page 50: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/50.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ResultsTime – GET Operation
50
RtDB 1 = 0.94 µs average
![Page 51: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/51.jpg)
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ResultsCompression
51
RtDB 1 = N/A Compression: 45.2 µs avg. Decompression: 13.9 µs avg.
![Page 52: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/52.jpg)
Conclusion
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands 52
![Page 53: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/53.jpg)
• The implementation of the RtDB v2 was a success!– All requirements were met!
• It’s Fast– The performance impact is negligible
• It’s Stable– Used in the two competitions CAMBADA has
participated in 2017
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
Conclusion
53
![Page 54: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/54.jpg)
• Data can now be non-volatile– Database is stored in a memory-mapped file– You can reboot the computer and return to the
task right where you left it
• Cross-language compatibility• Data Safety• Sharing periodicity and phase
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ConclusionSweet Extras
54
![Page 55: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/55.jpg)
• More tools !!– Dictionary generator and RtDB realtime inspector
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ConclusionSweet Extras
55
![Page 56: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/56.jpg)
• More tools !!– Dictionary generator and RtDB realtime inspector
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
ConclusionSweet Extras
56
![Page 57: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/57.jpg)
Release it open-sourceSoon at github.com/CAMBADA
Integrate process synchronizationWake next process on the pipeline after a particular PUT operation is performed
Integrate LoggingSerialized data scales better
CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands
The Next Steps
57
![Page 58: CAMBADA Realtime Database - Tech United · CAMBADA Realtime Database - The 2nd Generation // Ricardo Dias // MSL Workshop 2017, Eindhoven, Netherlands. Introduction. Motivation. 8](https://reader035.vdocuments.us/reader035/viewer/2022070710/5ec67f0d0a2d8124256206bc/html5/thumbnails/58.jpg)
IRIS-Lab / IEETA / DETI / UAUniversity of Aveiro, Portugal
Thanks for your attention!/cambadamsl /cambadamsl robotica.ua.pt