nosql now 2013 presentation

Service Entities Using MongoDB

NoSQL Now 2013, San Jose

Arjen Schoneveld

Friday, August 23, 13

Arjen Schoneveld

• co-Founder of Dexels, Amsterdam based software company

• Architect of Sportlink Sports ERP system

• PhD in Computer Science

• [email protected], http://nl.linkedin.com/in/aschoneveld/


mailto:[email protected]

mailto:[email protected]

http://nl.linkedin.com/in/aschoneveld/




Our Story

• How it started: Sportlink Sport Management System

• Navajo: a Service Oriented Application Framework

• Limitations of monolithic persistence

• MongoDB use case: Sportlink Digital Match Report

• Using Navajo Entity Services as a Schema

• RESTfull Entities


Sportlink Architecture (2003)Sport Club Application

FederationApplication

FederationWebsite

Database (Oracle) CrystalReports

Comp. Planning


Data Model

Object Model

Service Facade

Web Services

JSP/HTML/Javascript/CSS

ORM

JAX-WS

SOAP/WSDL

Service Proxy

A Service Oriented Application Stack

Object Model


How to simplify?


Services First

• Changes are “the rule” in many environments

• Most Services are CRUD and instances have very short live-span

• Many services do not need “OOP models”

• Use OOP for complex algorithms (e.g. match scheduling) only

• Build, not generate, Services


Usable Services• Loose Coupling: no client stub generation!

• Metadata is key ingredient: self describing

• Type, description, input validation, subsequent services (protocol)

• Self describing Abstract User Interface → enables automatic creation of usable first order UIs

• Use a Canonical Service Language as the “lingua franca” between the services and the client


What we came up with


Data Model

The Navajo Application FrameworkUIs

[Platform AgnosticDeclarative UI Language]

CanonicalService

Language

Services[Service Script Language]


How we implemented it


fd

[Resources]Database

Mail ServerFTP Server

SOAP Service

[Adapters]

[Tipi]

UI Widget

s

UI Events

Service Listeners

Call Se

rvice

s

UI A

ctio

ns

Layout + CSS

[Services]

Client State

Navajo

Expressions

[Logging][WorkFlows]

Use Adapters

[Authentication]

Valid

ation Service Links

JDBC

MongoDB

[Entities]

SOA

P

Call Service

Join message

Email Crystal

[Service Queue]


How Sportlink Evolved...


Sportlink Ecosystem (2013)Club

(Tipi/Swing)Association(Java/Swing)

Website(Third Party)

Oracle

CrystalFTP

~4000 services

Comp. Planning

E-Mail

Digital Match Report

(iOS/Android/Tipi)

Internet Payment(PHP)

Facility Mngtmt(GWT+G. Earth)

Course Registration

(GWT+G. Maps)

Official Portal(Tipi/VaadIn)

MongoDB

MySQL

Website Plugins(PHP)

Financial Admin

Service Workflow

EnterpriseDB


Growing Pains

• More end user applications

• More functionality

• More sport associations with different requirements

• More end users

• Single database has become the bottleneck


Oracle Limitations

• License Costs!

• Unpredictable query behavior

• Only partly/expensive High Availability

• Poor affordable(!) horizontal scalability in combination with good single node performance

• Service/Persistence impedance mismatch!


OracleMongo

Neo4j

MySQL

Dynamo

Cassandra

PostgreSQL

MatchReport

Web PluginMobile App

Club App

Association App

Association App

Courses App Facility Mngmnt

Polyglot Persistence


Our MongoDB Use Case• Digital Match Report (Mobile, Web and

Desktop)

• Player Roster for home- and away teams (role, captain, base player)

• Match Result, Match Events, Player Validation, Digital Players Pass

• 40.000 matches per weekend (fixed times: 11.00, 12:30, 14:00, 15:30, ...)

• typical: 3 users per match (official, two captains)


Why MongoDB

• MatchReport application is introduced gradually → Scalability → NoSQL

• MatchReport can be fetched and updated like a Data Aggregate → Document Based NoSQL


Home Away Official Mongo Oracle

Fetch Match Report

MatchReportReport Fetch Match Report

Report

Populate Home Players Report

Populate Away Players Report

Fetch Match Report

Report

Report Report

Fill In Report Report

Finalize Report Report Match

Players

Events

Players

Events


How do we integrate Navajo and MongoDB?


Navajo Services• A simple XML/HTTP/POST-based Web Service

• Uses a Canonical Service Language: Navajo Document

• Consumes and produces a Navajo Document (Everything is a Document)

• May contain additional (Service) links to next-step services (implementing an application protocol, like hyperlinks)

• Navascript consumes and produces a Navajo Document, using expressions and access (Java) adapters for: database access, calling other Navajo Services, calling SOAP, manipulating messages (e.g. joins), etc.

• JavaScript and JRuby are also supported Navajo Service implementation languages

• Navajo Services are (hot) deployed in a Navajo Server/OSGi runtime using realtime generation of OSGi bundles


Navajo Document

• Navajo Services use Navajo Documents for all message processing (Canonical Service Language)

• All data required by and delivered from a Navajo Service is contained in a Navajo Document

• A Navajo document consists of the following elements:

• messages, for grouping:

• properties and other messages

• methods (links to relevant subsequent services)


Properties• A property has metadata:

• name. Name must be unique within a message

• type. Natively supported types are: string, integer, float, boolean, date, binary, selections(!).

• value. The value of the property.

• description. Used to indicate the semantical meaning of the property

• subtype. May contain additional formatting hints for clients, like matching regular expressions for strings or the mime-type of a binary property

• direction. Indication for a client to render a property read-only or not.


Serializations• The default serialization of Navajo Document is XML

• Different other serialization are supported, e.g. JSON

• Navajo Document contains a lot of metadata automatic User Interface Generation is supported (an Eclipse Plugin is provided)

• Serialization of binary type properties is Base64. Navajo XML parser supports transparent streaming of binary properties


A Navajo Document<tml>

<message name=”Person”><property name=”Lastname” length=”100” description=”Lastname of the person” type=”string” subtype=”regexp=[A-z]*” direction=”out” value=”Kant”/><property name=”FirstName” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out” value=”Immanuel”/><property name=”DateOfBirth” type=”date” direction=”out” value=”1724-04-22”/><property name=”Photo” type=”binary” direction=”in”>

</property></message>

</tml>


Collections: Array Messages<tml> <message name=”Persons” type=”array”> <message name=” Persons” type=”definition”> <property name=”Lastname” type=”string” subtype=”regexp=[A-z]*” length=”100” direction=”out” description=”Last name of this Person”/>

<property name=”Firstname” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out” description=”First name of this Person”/>

</message> <message name=” Persons” index=”0”> <property name=”Lastname” value=”Kant”/>

<property name=”Firstname” value=”Immanuel”/> </message> <message name=” Persons” index=”1”> <property name=”Lastname” value=”Godel”/>

<property name=”Firstname” value=”Kurt”/> </message> <message name=” Persons” index=”2”> <property name=”Lastname” value=”Turing”/>

<property name=”Firstname” value=”Alan”/> </message> </message></tml>


Club Application

[Tipi]

QueryMatch

Oracle2. Request<tml rpc_name=”QueryMatch”>

<message name=”Match”><property name=”Id” value=”1”/>

</message></tml>4. Response

<tml rpc_name=”QueryMatch”><message name=”Match”>

<property name=”Id”/><property name=”Date”/><property name=”HomeTeam”/><property name=”AwayTeam”/>

</message><methods> <method name=”UpdateMatch”/> <method name=”DeleteMatch”/></methods>

</tml>

3. SELECT * FROM match WHERE matchid = ?

<navascript>[see next slide]

</navascript>

<tipi><button>

<onPush> (1) <callservice name=”QueryMatch”> </onPush>

</button>(5) <panel listener=”QueryMatch”> <label name=”Calendar date” value=”{navajo:/QueryMatch/MatchDate}”/></panel>

</tipi>


<navascript> <validations> <check code=”Empty Id”> [/Match/Id] != ‘’ </check> </validations> <map.sqladapter> <sqladapter.query>

SELECT * FROM MATCH WHERE matchid =?</sqladapter.query>

<sqladapter.parameter value=”[/Match/Id]”/> <message name=”Match”>

<property name=”FormattedMatchDate”> <expression value=”FormatDate($columnValue(‘Date’),’dd-MMM-yyy'”/></property>

</message> </map.sqladapter> <methods> <method name=”ProcessUpdateMatch”/> <method name=”ProcessDeleteMatch”/> </methods></navascript>


MongoDB

• MongoDB is a document-based NoSQL database

• MongoDB is schema-less, unlike SQL databases that rely on a predefined schema (or metadata) to define e.g. tables and table structures

• MongoDB serializes data as JSON objects

• MongoDB supports a sharded setup → scalability

• Advantage: ease of use due to lack of structure

• Disadvantage: unmaintainable due to lack of structure


Navajo and MongoDB

• Store Navajo Document directly in MongoDB

• Navajo Document can be easily transformed to JSON

• No need for writing “mapping code” (no impedance mismatch)


JSONify Navajo Document<message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”4321” type=” integer” description=”Unique id of the away team”/><property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers” type=”arrayelement”> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”selection” cardinality=”1” type=”selection”> <option name=”Defender” value=”DEF” selected=”1”/> <option name=”Keeper” value=”KPR” selected=”0”/> </property></message>

</message>[... same for AwayPlayers ... left for brevity ]

</message>

The Navajo Document XML above can be converted to JSON as follows:

{ “MatchId” : 1, “HomeTeamId” : “1234”, “AwayTeamId” : “4321”, “Date” : Date ( 23242873 ), “HomePlayers” : [ {“PersonId” : “CHGP12Y”, “Role” : “DEF”}] }


Query MongoDB• Entity in MongoDB can be queried using MongoDB query syntax

• Entity in MongoDB can be queried using a Navajo Document -> will be converted to a JSON query:

<message name=”MatchReport”> <property name=”MatchId” value=”1”/></message>

can be converted to Mongo query (message name maps to Collection name):

MatchReport.find({“MatchId” : 1})


<navascript> <message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/> </map.navajomap></navascript>

Mongo Query Service

Construct Query

Call Query Service and append result


Result of Query<message name=”MatchReport”>Unique Id <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer”/><property name=”AwayTeamId” value=”4321” type=”integer”/><property name=”Date” value=”2013-08-24” type=”date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>

</message></message>

What is Lost?What is Added?

Unique Id

Unique Id


How can we resolve this?


Query using Entity Service

• Define a Navajo Service for producing the Entity Navajo Document: Entity Service as a Schema

• Expects no input

• Simply returns Navajo Document describing the Entity

• Pass Entity Service as a “parameter” to the Navajo Service that queries or mutates Mongo

• Entity service is used to decorate the “raw” Mongo document


Example Entity Message<message name=”MatchReport”> <property name=”MatchId” type=”integer”/>

<property name=”HomeTeamId” type=”integer” description=”Unique id of the hometeam”/><property name=”AwayTeamId” type=”integer” description=”Unique id of the person”/><property name=”Date” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”>

<message name=” HomePlayers” type=”definition”> <property name=”PersonId” type=”string” value=”Unique person id”/> <property name=”Role” type=”selection” cardinality=”1” type=”selection”> <option name=”Defender” value=”DEF”/> <option name=”Keeper” value=”KPR”/> </property></message>


Navajo Document above describes the Entity type “MatchReport”


Mongo Query Service (revisited)

<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message> <message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/> </map.navajomap></navascript>


Result of Query (revisited)<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”string” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”4321” type=”string” description=” Unique id of the away team”/><property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”> <message name=” HomePlayers” type=”definition”>

<property name=”PersonId” type=”string” value=”Unique person id”/> <property name=”Role” type=”string” type=”selection”> <option name=”Defender” value=”DEF”/> <option name=”Keeper” value=”KPR”/> </property></message><message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>



Flexibility Harnessed

• Entity adds missing metadata

• Entity can be used for type validation

• Adding properties is easy, simply add it to the Entity, next time it will appear in the query

• Deleting properties is easy, simply delete it from the Entity, next time it won’t appear in the query.


Insert Example<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”MatchId” value=”2” type=”integer”/>

<property name=”HomeTeamId” value=”9787” type=”string” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”6543” type=”string” description=” Unique id of the away team”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”PersonId” type=”string” value=”BBFW63X”/> <property name=”Role” type=”string” value=”DEF”/></message>


<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessInsertMongo”/> </map.navajomap></navascript>

This will fail!


Update Example<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer”/><property name=”AwayTeamId” value=”4321” type=”integer”/><property name=”BogusAttribute” value=”FooBar” type=”string”/><property name=”Date” value=”2013-09-22” type=”date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>


<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessUpdateMongo”/> </map.navajomap></navascript>

This will be removed


Delete ExampleAssume that we accidentally inserted the following document:

<message name=”MatchReport”> <property name=”_id” value=”aaa666”/> <property name=”MatchId” value=”3” type=”integer”/>

<property name=”HomeTeamId” value=”809832” type=”integer” /><property name=”AwayTeamId” value=”321311” type=”integer” /><message name=”HomePlayers” type=”array”/>

</message>

This can be deleted using following script:

<navascript> <message name=”MatchReport”> <property name=”_id” value=”aaa666”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessDeleteMongo”/> </map.navajomap></navascript>


What about more fine grained mutations?


Simple Modify ExampleLet’s just change the date

<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”Date” value=”2013-09-22” type=”date”/></message>

<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessModifyMongo”/> </map.navajomap></navascript>


Inserting new Array Element

Insert a new player for the home team:

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”Insert” type=”boolean” value=”true”/> <property name=”PersonId” type=”string” value=”BBFW06E”/> <property name=”Role” type=”string” value=”DEF” /></message>


Note that the properties _id and Insert (set to true) are required

Id is not required,it will be assigned



<message name=”HomePlayers”> <property name=”Modify” type=”boolean” value=”true”/> <property name=”__id” value=”ffhggab3232b”/> <property name=”Role” type=”string” value=”KEP”/></message>


Id is requiredSet Modify flag

Modifying an Array Element

Change the role of a player:


Deleting an Array Element


<message name=”HomePlayers”> <property name=”Delete” type=”boolean” value=”true”/> <property name=”__id” value=”agb3f4g4468”/> </message>


Id is requiredSet Delete flag


What have we reached?

• A Flexible Schema to manage MongoDB document data

• Schema is managed outside Mongo

• Basically a metadata template to filter/validate both incoming and outgoing data

• No need to write any “Mapping Code”


Normal Services

Entity Services

... Symmetry at last!


What is missing?

• Notion of entity instances (to enable client push)

• Relations between entities

• Definition of key/required properties

• Data consistency

• Entity defined persistence


Entity First

• Thus far: Operation-Oriented Services: DoUpdateMatchReport, QueryMatchReport, FinalizeMatchReport, etc.

• Resource-Oriented, the Way of the Web

• RESTful resources can be viewed as First Class Citizens (everything is a resource)

• REST method semantics enable use of Cache


REST in 1 Slide

• A REST architecture relies fully on HTTP as an Application Protocol

• REST revolves around URI resources (nouns) and HTTP methods (actions) and ...

• Rely on hypermedia to drive the client(!) state machine defined by the application


RESTFul Navajo

• Navajo Entity == Resource

• Extend to support inheritance and referencing other Navajo Entities

• Add key properties

• Navajo Entity support one or more HTTP methods (GET, PUT, POST and DELETE)

• Entity Operations (HTTP methods) are implemented by Navajo Services


Entity Service<validations> <check code=”A MatchReport should have a date unless it is a DRAFT MatchReport”> [/Match/Status] != ‘DRAFT’ OR CheckDate([/MatchReport/Date)</check></validations><message name=”MatchReport” extends="navajo://Report"> <property name="_id" type="string" key="true,auto"/> <property name="MatchId" type="string" key="true" reference=”navajo://Match/MatchId”/> <property name="Status" type="string"/> <property name="HomeTeamId" type="integer" reference=”navajo://Team/TeamId”/> <property name="AwayTeamId" type="integer" reference=”navajo://Team/TeamId”/>

<message name=”HomePlayers” type=”array”><message name=” HomePlayers” type=”definition”> <property name=”PersonId” type=”string” reference=”navajo://Person”>

<property name=”Role” type=”string”/></message>



Entity Operations<navascript> <operations> <operation service=”mongo/ProcessQueryMongo” method=”GET”/> <operation service=”mongo/ProcessModifyMongo” method=”POST”/> </operations> <message name=”MatchReport” extends="navajo://Report"> [ .... ] </message></navascript>

[NOTE: operation == method]


Pure HTTP GET

http://dexels.example.org/MatchReport?MatchId=33


http://dexels.example.org/greatmind/

http://dexels.example.org/greatmind/

Backwards Compatibility

<navascript> <map.entity entity="'MatchReport'"> <entity.createproperty name="'MatchId'" value="3" /> <entity.get append="'/'" /> </map.entity></navascript>


Entity Insert<navascript> <message name="MatchReport"> <property name="MatchId"> <expression value="‘31232’"/> </property> <property name="HomeTeamId"> <expression value="‘1231232’"/> </property> <property name="AwayTeamId"> <expression value="‘345324’"/> </property> </message> <map.entity useCurrentOutDoc="true" entity="'MatchReport'"> <entity.insert/> </map.entity></navascript>


Caching and Consistency

• ETag can help us with

• Preventing that multiple GET operation return same data (send 304 instead)

• Preventing that Stale data is used for updating an Entity (check ETag on PUT,DELETE,POST) (send 409)


Polyglot Persistence

• Entity Operations can implement different persistence methods

• Some Entities can “live” inside MongoDB while other reside in e.g. Oracle

• Non-functional data requirements (e.g. consistency, availability) can drive persistence decision

• “Transparent” migration of Data to other datasources


Entity drives Service drives Entity drives Service drives Entity drives Service drives Entity drives Service drives Entity drives

Service

• Role of Normal Service and Entity Service are reversed

• Instead of QueryMatchForm, DoInsertMatchForm, DoDeleteMatchForm, we have MatchForm (GET, DELETE, INSERT)


Normal Services

Entity Services

... from process- to resource oriented


nosql now 2013 presentation

Technology

navajo entity services

sportlink architecture

sportlink ecosystem

service entities

sportlink digital match

usable services loose

canonical service language

sport associations