building a maintainable bi-directional cross platform protocol
DESCRIPTION
Our talk at JsConf EU 2014 http://2014.jsconf.eu/speakers/#/speakers/william-lewis-pavel-dovbush-building-a-maintainable-bi-directional-cross-platform-protocol Common web frameworks make the assumption you are going to build against a Restful API, but what if your use case doesn’t fit with the Restful principles. How might you go about systematically designing a protocol between client and server? In this talk we are going to discuss how you can design and build an RPC style protocol and service layer that is flexible and extenisble enough to serve multiple cross platform clients and servers, growing with application needs whilst letting developers focus on building features instead of maintaining API boilerplate.TRANSCRIPT
![Page 1: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/1.jpg)
Building a maintainable bi-directional cross platform protocol
Pavel DovbushWilliam Lewis@
![Page 2: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/2.jpg)
BackgroundSolution and ImplementationExamples
![Page 3: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/3.jpg)
● Application Programming Interface
● Operations● Inputs● Outputs● Type definitions
● Independent of implementation
API
API
![Page 4: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/4.jpg)
![Page 5: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/5.jpg)
RequirementsPerfect Client Server API:
● Just works – Magic! :)
● Flexible● Extensible● Testable● Maintainable● Platform and language neutral● Focused on features, not bytes over the wire
Requirements
![Page 6: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/6.jpg)
Overview● Encoding ● Message exchange● Data access
data access
Server
exchange data accessexchange
Client
serialise
deserialise
Overview
![Page 7: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/7.jpg)
REST+JSON
PHP array
Server
URI JS ObjectAjax
Client
JSON
URL
REST + JSON
![Page 8: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/8.jpg)
REST+JSON problems
Message exchange:● Client: HTTP request/response model● Server: URI-based, config on web- or app- server
Data access:● No canonical definition● No versioning● Duplicate implementations and configuration
REST + JSON
Encoding:● Server: JSON● Client: URLEncode
![Page 9: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/9.jpg)
Badoo APIs
Badoo APIs
Core API
Mobile API
???
![Page 10: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/10.jpg)
BackgroundSolution and ImplementationExamples
![Page 11: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/11.jpg)
Protocol implementations
SOAP
Protocol Values
![Page 12: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/12.jpg)
![Page 13: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/13.jpg)
Protocol values
● Protocol description● Encoding● Data access● Message exchange● Versioning
Protocol
![Page 14: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/14.jpg)
Protobuf + Own RPC
Generated class
Server
Events Generated classEvents
Client
Protobuf
Protobuf
Description, versioning
Usage
![Page 15: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/15.jpg)
Google Protocol Buffers
● Interface description language● Internal representation● Language support
○ v2.3 plugin support ● Encoding and network efficiency
Protcol values
![Page 16: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/16.jpg)
Interface description language
Protobuf is self-describing - descriptor.proto
Interface description language
● enum● message● field● service● option
![Page 17: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/17.jpg)
Interface description languageLabel Type Name Number
optionalrequiredrepeated
boolstring
messageenumfloatint32
+ more numeric
field_name = 1;
required string user_name = 1;optional uint32 age = 2;
Interface description language
![Page 18: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/18.jpg)
Interface description language
enum Role {ADMIN = 1;USER = 2;
}
message User {required string name = 1;repeated string nickname = 2;optional uint32 age = 3;required Role role = 4;
}
Interface description language
![Page 19: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/19.jpg)
Protobuf usage
Encoding:● Only binary
Message exchange:● Simple RPC
Data access:● No support for PHP● No support for JS
Protobuf compiler plugins
![Page 20: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/20.jpg)
Protobuf v2.3.0 compiler plugins
● Brilliant internal architecture● Very simple plugin system● Can generate any code in any language● IDL is completely separated from serialization part
● gist with example
Protobuf complier plugins
![Page 21: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/21.jpg)
Encoding
Encoding
Core API
Mobile APINative binary
serializationCustom JSON serialization
![Page 22: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/22.jpg)
Performance
Performance
![Page 23: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/23.jpg)
Code auto-generation
● WebSite DEVEL - On any request if ‘.proto’ file is newer than generated code - regenerate
● Mobile DEVEL - Grunt task - regenerate on file change
● PRODUCTION - generate before deploy
Code auto-generation
![Page 24: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/24.jpg)
Message exchange
service SearchService {rpc Search (SearchRequest) returns (SearchResponse);}
● Too simple for a complex application● We need a wrapper for every Request/Response● Anytime responses
Message exchange
![Page 25: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/25.jpg)
new RPC(request_type, parameter) .on(response_type, callback) .on([type1, type2], callback) .request();
Message exchange (two-way RPC)
RPC.any.on(type3, callback);
request_type & response_type are values of enum MessageTypeparameter & response are Protobuf messages
function callback(err, /** Type1 */ response1) {}
Message exchange
![Page 26: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/26.jpg)
Versioning
Old clients ignore:● new fields● unsupported commands
Migrating to new protocol version:● cause compilation error on field removal
Versioning
![Page 27: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/27.jpg)
Take aways● Protocol defined in one place● Code uses data-access classes● Validation● Encoding can vary● Flexible message exchange● Versioning
● Any part can be changed without affecting anything else
Summary
![Page 28: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/28.jpg)
BackgroundSolution and ImplementationExamples
![Page 29: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/29.jpg)
Building a service
Example
![Page 30: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/30.jpg)
Example
![Page 31: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/31.jpg)
Building a service
● Location search○ Client query: city name○ Server response: list of cities
(ID, name, lat, long)
● Client notification○ Anytime server response: user message
Example
![Page 32: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/32.jpg)
Protobuf definition
enum MessageType { // body: CityQuerySERVER_SEARCH_CITIES = 1;
// body: CitiesCLIENT_FOUND_CITIES = 2;
// body : ClientNotificationCLIENT_NOTIFICATION = 3;
}
message ClientNotification {required string id = 1;optional string title = 2;optional string message = 3;
}
message CityQuery {optional string name = 1;
}
message Cities {repeated City cities = 1;
}
message City {required int32 id = 1;required string name = 2;optional double longitude = 3;optional double latitude = 4;
}
Example protobuf
![Page 33: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/33.jpg)
Protobuf definition
message RPCMessage {required int32 version = 1;optional int32 message_id = 2;repeated MessageBody body = 3;
}
message MessageBody {required MessageType message_type = 1;optional CityQuery city_query = 2;optional Cities cities = 3;optional ClientNotification client_notification = 4;
}
Example protobuf
![Page 34: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/34.jpg)
Generated classes examplesdefine(['GPB/gpb2'], function(/** $gpb */$gpb) {var Protocol = $gpb.namespace('Demo');/** * CityQuery * @class {Protocol.CityQuery} * @extends {$gpb.Message} */var CityQuery = Protocol.CityQuery = function() {
$gpb.Message.apply(this, arguments);};$gpb.extend(CityQuery, $gpb.Message);CityQuery.prototype.$gpb = 'Demo.CityQuery';CityQuery.prototype._descriptor = {"fields": {"name": {"type": 9, "number": 1, "label": 1}}};
return CityQuery;
});
define(['GPB/gpb2'], function(/** $gpb */$gpb) {var Protocol = $gpb.namespace('Demo');/** * CityQuery * @class {Protocol.CityQuery} * @extends {$gpb.Message} */var CityQuery = Protocol.CityQuery = function() {
$gpb.Message.apply(this, arguments);};$gpb.extend(CityQuery, $gpb.Message);CityQuery.prototype.$gpb = 'Demo.CityQuery';CityQuery.prototype._descriptor = {"fields": {"name": {"type": 9, "number": 1, "label": 1}}};
return CityQuery;
});
define(['GPB/gpb2'], function(/** $gpb */$gpb) {var Protocol = $gpb.namespace('Demo');/** * CityQuery * @class {Protocol.CityQuery} * @extends {$gpb.Message} */var CityQuery = Protocol.CityQuery = function() {
$gpb.Message.apply(this, arguments);};$gpb.extend(CityQuery, $gpb.Message);CityQuery.prototype.$gpb = 'Demo.CityQuery';CityQuery.prototype._descriptor = {"fields": {"name": {"type": 9, "number": 1, "label": 1}}};
return CityQuery;
});
Generated class
![Page 35: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/35.jpg)
Generated classes examples<?php
namespace GPBJS\Demo;
class CityQuery extends \GPBJSBase\Message{ protected static $name = 'Demo.CityQuery'; protected static $fields = array( 'name' => array('type' => 'string', 'optional' => true, 'repeatable' => false, 'hash' =>
false, 'raw' => False, 'is_enum' => false, 'is_message' => false), );
public function setName($value) { $this->_setFieldValue('name', $value); return $this; } public function getName() { return $this->_getFieldValue('name'); }}
<?php
namespace GPBJS\Demo;
class CityQuery extends \GPBJSBase\Message{ protected static $name = 'Demo.CityQuery'; protected static $fields = array( 'name' => array('type' => 'string', 'optional' => true, 'repeatable' => false, 'hash' =>
false, 'raw' => False, 'is_enum' => false, 'is_message' => false), );
public function setName($value) { $this->_setFieldValue('name', $value); return $this; } public function getName() { return $this->_getFieldValue('name'); }}
<?php
namespace GPBJS\Demo;
class CityQuery extends \GPBJSBase\Message{ protected static $name = 'Demo.CityQuery'; protected static $fields = array( 'name' => array('type' => 'string', 'optional' => true, 'repeatable' => false,
'hash' => false, 'raw' => False, 'is_enum' => false, 'is_message' => false), );
public function setName($value) { $this->_setFieldValue('name', $value); return $this; } public function getName() { return $this->_getFieldValue('name'); }}
Generated class
![Page 36: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/36.jpg)
RPC city query example
new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery).on(Protocol.MessageType.CLIENT_CITIES, onCities).on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification).request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 37: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/37.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 38: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/38.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 39: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/39.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 40: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/40.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 41: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/41.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 42: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/42.jpg)
RPC city query examplevar cityQuery = new Protocol.CityQuery().setName(‘london’);new RPC(Protocol.MessageType.SERVER_SEARCH_CITIES, cityQuery)
.on(Protocol.MessageType.CLIENT_CITIES, onCities)
.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification)
.request();
function onCities (err, /** Protocol.Cities */ cities) { if (err) { /* error handling */ return; }
for (var city in cities.getCities()) { cityListView.update(city.getId(), city.getName()); }}
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}Example usage
![Page 43: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/43.jpg)
RPC anytime response example
RPC.any.on(Protocol.MessageType.CLIENT_NOTIFICATION, onNotification);
function onNotification ( err, /** Protocol.ClientNotification */ notification) {alert(notification.getTitle() + ‘\n’ + notification.getMessage());
}
Example usage
![Page 44: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/44.jpg)
Building a maintainable bi-directional cross platform protocol
● REST + JSON● Protocol values
○ Interface description language○ Encoding and performance○ Protobuf compiler plugins○ Code auto-generation○ Message exchange (two-way RPC)○ Versioning
● Examples
Summary
![Page 45: Building a maintainable bi-directional cross platform protocol](https://reader034.vdocuments.us/reader034/viewer/2022052315/555e5ca3d8b42a8e4c8b5558/html5/thumbnails/45.jpg)
Thanks! Questions?
Pavel Dovbush <[email protected]>
William Lewis <[email protected]>@netproteus
Slides: techblog.badoo.com
Thanks:Google team for Protobuf itselfAndrey Nigmatulin <[email protected]> for bringing GPB in Badoo in 2009