maxim zaks: deep dive into data serialisation

MAXIM ZAKSFreelance Software Developer

Deep dive into data serialisation

Persisting State / User generated Content

Machine to Machine Communication

Representation of Configuration / Read only data

Why do we need data serialisation?

Custom binary representation

Language provided binary serialisation

Text based representation (CSV, XML, JSON, YAML)

Embedded SQL or NoSQL DB

Binary cross platform serialisation library (FlatBuffers, FlexBuffers Protocol Buffers, Cap’n Proto, SBE, Apache Thrift etc…)

How can we persist data on mobile?

Size “on disk”

Speed of read & write / partial read / memory consumption

Human readable and writable

Support of OO language type system

Data versioning / evolution / migration

Important criteria for persisting data

JSON vs.

Protocol Buffers vs.

FlatBuffers

Text based

Self describing

Weakly typed: Object, Array, String, Number, Bool, Null

JSON

Binary

IDL (interface definition language) based Schema

Strongly typed: Message, fix length Numbers, variable length

numbers, String, Bool, Enum, Bytes, Repeated values, Maps,

Oneof, Any

Evolution Strategy

Protocol Buffers

Binary

IDL (interface definition language) based Schema

Strongly typed: Table, Struct, fix length Numbers, String, Bool, Enum, Bytes, typed Vector, Union

Value / Reference type semantics

Random Value access

Evolution Strategy

Flat Buffers

Size “on disk”

Speed of read & write / partial read / memory consumption

Human readable and writable

Support of OO language type system

Data versioning / evolution / migration

Important criteria for persisting data

👎 JSON, bad for representing numbers, lots of repetition,

needs to be at least minified

👍 Protocol Buffers, stores only values and a bit of meta data,

thanks to VLQ very efficient for storing numbers

🤔 Flat Buffers, has some overhead compared to Protocol Buffers because of Ref semantics and random value access

Important criteria for persisting data: Size

👎 JSON, text need to be parsed and translated to

intermediate data representation

🤔 Protocol Buffers, no partial read, VLQ means you need to do

some operations before value is available

👍 Flat Buffers, support partial reading thanks to Ref semantics and random value access mechanism

Important criteria for persisting data: Speed of read & write / partial read / memory consumption

👍 JSON, text based and can be nicely formatted

🤔 Protocol Buffers, provides tools for binary to JSON conversion and vice versa

🤔 Flat Buffers, provides tools for binary to JSON conversion

and vice versa

Important criteria for persisting data: Human readable and writable

🤔 JSON, is generally weakly typed, there are ways to

transform/validate against OO types

👍 Protocol Buffers, code generator creates Accessor classes

which can be comfortably used for encoding and decoding

👍 Flat Buffers, code generator creates Accessor classes which can be comfortably used for decoding, performant encoding can

be a bit painful

Important criteria for persisting data: Support of OO language type system

🤔 JSON, is implicit because of self describing nature

👍 Protocol Buffers, provides a set of rule how a schema can be evolved

👍 Flat Buffers, provides a set of rule how a schema can be

evolved

Important criteria for persisting data: Data versioning / evolution / migration

Size Efficiency Human Readable Types Evolution

JSON 👎 👎 👍 🤔 🤔

Proto Buffers 👍 🤔 🤔 👍 👍

Flat Buffers 🤔 👍 🤔 👍 👍

Mobile Warsaw 2016: https://www.youtube.com/watch?

v=OUNpNo-oyQY

MobileTechCon 2016: https://medium.com/@icex33/beyond-

json-introduction-to-flatbuffers-fba1dfd0dcfe

AppBuilders 2017: https://www.youtube.com/watch?v=Ve9POqZJymw

BerlinBuzzWords 2017: https://www.youtube.com/watch?v=qF44UetsLsQ

Previous Talks

Protocol Buffers Encoding: https://developers.google.com/protocol-buffers/docs/encoding

FlatBuffers: https://google.github.io/flatbuffers/

FlatBuffersSwift: https://github.com/mzaks/FlatBuffersSwift

FlexBuffersSwift: https://github.com/mzaks/FlexBuffersSwift

Data Serialisation formats: https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

JVM Serialisers: https://github.com/eishay/jvm-serializers

More Links

WWW.MDEVTALK.CZ

mdevtalk