nosql now 2013 presentation

Service Entities Using MongoDB

NoSQL Now 2013, San Jose

Arjen Schoneveld

Friday, August 23, 13

Friday, August 23, 13

Arjen Schoneveld

• co-Founder of Dexels, Amsterdam based software company

• Architect of Sportlink Sports ERP system

• PhD in Computer Science

• arjen@dexels.com, http://nl.linkedin.com/in/aschoneveld/

Friday, August 23, 13

Our Story

• How it started: Sportlink Sport Management System

• Navajo: a Service Oriented Application Framework

• Limitations of monolithic persistence

• MongoDB use case: Sportlink Digital Match Report

• Using Navajo Entity Services as a Schema

• RESTfull Entities

Friday, August 23, 13

Sportlink Architecture (2003)Sport Club Application

FederationApplication

FederationWebsite

Database (Oracle) CrystalReports

Comp. Planning

Friday, August 23, 13

Data Model

Object Model

Service Facade

Web Services

JSP/HTML/Javascript/CSS

ORM

JAX-WS

SOAP/WSDL

Service Proxy

A Service Oriented Application Stack

Object Model

Friday, August 23, 13

How to simplify?

Friday, August 23, 13

Services First

• Changes are “the rule” in many environments

• Most Services are CRUD and instances have very short live-span

• Many services do not need “OOP models”

• Use OOP for complex algorithms (e.g. match scheduling) only

• Build, not generate, Services

Friday, August 23, 13

Usable Services• Loose Coupling: no client stub generation!

• Metadata is key ingredient: self describing

• Type, description, input validation, subsequent services (protocol)

• Self describing Abstract User Interface → enables automatic creation of usable first order UIs

• Use a Canonical Service Language as the “lingua franca” between the services and the client

Friday, August 23, 13

What we came up with

Friday, August 23, 13

Data Model

The Navajo Application FrameworkUIs

[Platform AgnosticDeclarative UI Language]

CanonicalService

Language

Services[Service Script Language]

Friday, August 23, 13

How we implemented it

Friday, August 23, 13

fd

[Resources]Database

Mail ServerFTP Server

SOAP Service

[Adapters]

[Tipi]

UI Widget

s

UI Events

Service Listeners

Call Se

rvice

s

UI A

ctio

ns

Layout + CSS

[Services]

Client State

Navajo

Expressions

[Logging][WorkFlows]

Use Adapters

[Authentication]

Valid

ation Service Links

JDBC

MongoDB

[Entities]

SOA

P

Call Service

Join message

Email Crystal

[Service Queue]

Friday, August 23, 13

How Sportlink Evolved...

Friday, August 23, 13

Sportlink Ecosystem (2013)Club

(Tipi/Swing)Association(Java/Swing)

Website(Third Party)

Oracle

CrystalFTP

~4000 services

Comp. Planning

E-Mail

Digital Match Report

(iOS/Android/Tipi)

Internet Payment(PHP)

Facility Mngtmt(GWT+G. Earth)

Course Registration

(GWT+G. Maps)

Official Portal(Tipi/VaadIn)

MongoDB

MySQL

Website Plugins(PHP)

Financial Admin

Service Workflow

EnterpriseDB

Friday, August 23, 13

Growing Pains

• More end user applications

• More functionality

• More sport associations with different requirements

• More end users

• Single database has become the bottleneck

Friday, August 23, 13

Oracle Limitations

• License Costs!

• Unpredictable query behavior

• Only partly/expensive High Availability

• Poor affordable(!) horizontal scalability in combination with good single node performance

• Service/Persistence impedance mismatch!

Friday, August 23, 13

OracleMongo

Neo4j

MySQL

Dynamo

Cassandra

PostgreSQL

MatchReport

Web PluginMobile App

Club App

Association App

Association App

Courses App Facility Mngmnt

Polyglot Persistence

Friday, August 23, 13

Our MongoDB Use Case• Digital Match Report (Mobile, Web and

Desktop)

• Player Roster for home- and away teams (role, captain, base player)

• Match Result, Match Events, Player Validation, Digital Players Pass

• 40.000 matches per weekend (fixed times: 11.00, 12:30, 14:00, 15:30, ...)

• typical: 3 users per match (official, two captains)

Friday, August 23, 13

Why MongoDB

• MatchReport application is introduced gradually → Scalability → NoSQL

• MatchReport can be fetched and updated like a Data Aggregate → Document Based NoSQL

Friday, August 23, 13

Home Away Official Mongo Oracle

Fetch Match Report

MatchReportReport Fetch Match Report

Report

Populate Home Players Report

Populate Away Players Report

Fetch Match Report

Report

Report Report

Fill In Report Report

Finalize Report Report Match

Players

Events

Players

Events

Friday, August 23, 13

How do we integrate Navajo and MongoDB?

Friday, August 23, 13

Navajo Services• A simple XML/HTTP/POST-based Web Service

• Uses a Canonical Service Language: Navajo Document

• Consumes and produces a Navajo Document (Everything is a Document)

• May contain additional (Service) links to next-step services (implementing an application protocol, like hyperlinks)

• Navascript consumes and produces a Navajo Document, using expressions and access (Java) adapters for: database access, calling other Navajo Services, calling SOAP, manipulating messages (e.g. joins), etc.

• JavaScript and JRuby are also supported Navajo Service implementation languages

• Navajo Services are (hot) deployed in a Navajo Server/OSGi runtime using realtime generation of OSGi bundles

Friday, August 23, 13

Navajo Document

• Navajo Services use Navajo Documents for all message processing (Canonical Service Language)

• All data required by and delivered from a Navajo Service is contained in a Navajo Document

• A Navajo document consists of the following elements:

• messages, for grouping:

• properties and other messages

• methods (links to relevant subsequent services)

Friday, August 23, 13

Properties• A property has metadata:

• name. Name must be unique within a message

• type. Natively supported types are: string, integer, float, boolean, date, binary, selections(!).

• value. The value of the property.

• description. Used to indicate the semantical meaning of the property

• subtype. May contain additional formatting hints for clients, like matching regular expressions for strings or the mime-type of a binary property

• direction. Indication for a client to render a property read-only or not.

Friday, August 23, 13

Serializations• The default serialization of Navajo Document is XML

• Different other serialization are supported, e.g. JSON

• Navajo Document contains a lot of metadata automatic User Interface Generation is supported (an Eclipse Plugin is provided)

• Serialization of binary type properties is Base64. Navajo XML parser supports transparent streaming of binary properties

Friday, August 23, 13

A Navajo Document<tml>

<message name=”Person”><property name=”Lastname” length=”100” description=”Lastname of the person” type=”string” subtype=”regexp=[A-z]*” direction=”out” value=”Kant”/><property name=”FirstName” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out” value=”Immanuel”/><property name=”DateOfBirth” type=”date” direction=”out” value=”1724-04-22”/><property name=”Photo” type=”binary” direction=”in”>

</property></message>

</tml>

Friday, August 23, 13

Collections: Array Messages<tml> <message name=”Persons” type=”array”> <message name=” Persons” type=”definition”> <property name=”Lastname” type=”string” subtype=”regexp=[A-z]*” length=”100” direction=”out” description=”Last name of this Person”/>

<property name=”Firstname” type=”string” subtype=”regexp=[A-z]*” length=”50” direction=”out” description=”First name of this Person”/>

</message> <message name=” Persons” index=”0”> <property name=”Lastname” value=”Kant”/>

<property name=”Firstname” value=”Immanuel”/> </message> <message name=” Persons” index=”1”> <property name=”Lastname” value=”Godel”/>

<property name=”Firstname” value=”Kurt”/> </message> <message name=” Persons” index=”2”> <property name=”Lastname” value=”Turing”/>

<property name=”Firstname” value=”Alan”/> </message> </message></tml>

Friday, August 23, 13

Club Application

[Tipi]

QueryMatch

Oracle2. Request<tml rpc_name=”QueryMatch”>

<message name=”Match”><property name=”Id” value=”1”/>

</message></tml>4. Response

<tml rpc_name=”QueryMatch”><message name=”Match”>

<property name=”Id”/><property name=”Date”/><property name=”HomeTeam”/><property name=”AwayTeam”/>

</message><methods> <method name=”UpdateMatch”/> <method name=”DeleteMatch”/></methods>

</tml>

3. SELECT * FROM match WHERE matchid = ?

<navascript>[see next slide]

</navascript>

<tipi><button>

<onPush> (1) <callservice name=”QueryMatch”> </onPush>

</button>(5) <panel listener=”QueryMatch”> <label name=”Calendar date” value=”{navajo:/QueryMatch/MatchDate}”/></panel>

</tipi>

Friday, August 23, 13

<navascript> <validations> <check code=”Empty Id”> [/Match/Id] != ‘’ </check> </validations> <map.sqladapter> <sqladapter.query>

SELECT * FROM MATCH WHERE matchid =?</sqladapter.query>

<sqladapter.parameter value=”[/Match/Id]”/> <message name=”Match”>

<property name=”FormattedMatchDate”> <expression value=”FormatDate($columnValue(‘Date’),’dd-MMM-yyy'”/></property>

</message> </map.sqladapter> <methods> <method name=”ProcessUpdateMatch”/> <method name=”ProcessDeleteMatch”/> </methods></navascript>

Friday, August 23, 13

Friday, August 23, 13

MongoDB

• MongoDB is a document-based NoSQL database

• MongoDB is schema-less, unlike SQL databases that rely on a predefined schema (or metadata) to define e.g. tables and table structures

• MongoDB serializes data as JSON objects

• MongoDB supports a sharded setup → scalability

• Advantage: ease of use due to lack of structure

• Disadvantage: unmaintainable due to lack of structure

Friday, August 23, 13

Navajo and MongoDB

• Store Navajo Document directly in MongoDB

• Navajo Document can be easily transformed to JSON

• No need for writing “mapping code” (no impedance mismatch)

Friday, August 23, 13

JSONify Navajo Document<message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”4321” type=” integer” description=”Unique id of the away team”/><property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers” type=”arrayelement”> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”selection” cardinality=”1” type=”selection”> <option name=”Defender” value=”DEF” selected=”1”/> <option name=”Keeper” value=”KPR” selected=”0”/> </property></message>

</message>[... same for AwayPlayers ... left for brevity ]

</message>

The Navajo Document XML above can be converted to JSON as follows:

{ “MatchId” : 1, “HomeTeamId” : “1234”, “AwayTeamId” : “4321”, “Date” : Date ( 23242873 ), “HomePlayers” : [ {“PersonId” : “CHGP12Y”, “Role” : “DEF”}] }

Friday, August 23, 13

Query MongoDB• Entity in MongoDB can be queried using MongoDB query syntax

• Entity in MongoDB can be queried using a Navajo Document -> will be converted to a JSON query:

<message name=”MatchReport”> <property name=”MatchId” value=”1”/></message>

can be converted to Mongo query (message name maps to Collection name):

MatchReport.find({“MatchId” : 1})

Friday, August 23, 13

<navascript> <message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/> </map.navajomap></navascript>

Mongo Query Service

Construct Query

Call Query Service and append result

Friday, August 23, 13

Result of Query<message name=”MatchReport”>Unique Id <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer”/><property name=”AwayTeamId” value=”4321” type=”integer”/><property name=”Date” value=”2013-08-24” type=”date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>

</message></message>

What is Lost?What is Added?

Unique Id

Unique Id

Friday, August 23, 13

How can we resolve this?

Friday, August 23, 13

Query using Entity Service

• Define a Navajo Service for producing the Entity Navajo Document: Entity Service as a Schema

• Expects no input

• Simply returns Navajo Document describing the Entity

• Pass Entity Service as a “parameter” to the Navajo Service that queries or mutates Mongo

• Entity service is used to decorate the “raw” Mongo document

Friday, August 23, 13

Example Entity Message<message name=”MatchReport”> <property name=”MatchId” type=”integer”/>

<property name=”HomeTeamId” type=”integer” description=”Unique id of the hometeam”/><property name=”AwayTeamId” type=”integer” description=”Unique id of the person”/><property name=”Date” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”>

<message name=” HomePlayers” type=”definition”> <property name=”PersonId” type=”string” value=”Unique person id”/> <property name=”Role” type=”selection” cardinality=”1” type=”selection”> <option name=”Defender” value=”DEF”/> <option name=”Keeper” value=”KPR”/> </property></message>

</message></message>

Navajo Document above describes the Entity type “MatchReport”

Friday, August 23, 13

Mongo Query Service (revisited)

<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message> <message name=”MatchReport”> <property name=”MatchId” value=”1” type=”integer”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessQueryMongo” append=”’/’”/> </map.navajomap></navascript>

Friday, August 23, 13

Result of Query (revisited)<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”string” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”4321” type=”string” description=” Unique id of the away team”/><property name=”Date” value=”2013-08-24” type=”date” description=”Match date”/><message name=”HomePlayers” type=”array”> <message name=” HomePlayers” type=”definition”>

<property name=”PersonId” type=”string” value=”Unique person id”/> <property name=”Role” type=”string” type=”selection”> <option name=”Defender” value=”DEF”/> <option name=”Keeper” value=”KPR”/> </property></message><message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>

</message></message>

Friday, August 23, 13

Flexibility Harnessed

• Entity adds missing metadata

• Entity can be used for type validation

• Adding properties is easy, simply add it to the Entity, next time it will appear in the query

• Deleting properties is easy, simply delete it from the Entity, next time it won’t appear in the query.

Friday, August 23, 13

Insert Example<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”MatchId” value=”2” type=”integer”/>

<property name=”HomeTeamId” value=”9787” type=”string” description=”Unique id of the home team”/><property name=”AwayTeamId” value=”6543” type=”string” description=” Unique id of the away team”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”PersonId” type=”string” value=”BBFW63X”/> <property name=”Role” type=”string” value=”DEF”/></message>

</message></message>

<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessInsertMongo”/> </map.navajomap></navascript>

This will fail!

Friday, August 23, 13

Update Example<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”MatchId” value=”1” type=”integer”/>

<property name=”HomeTeamId” value=”1234” type=”integer”/><property name=”AwayTeamId” value=”4321” type=”integer”/><property name=”BogusAttribute” value=”FooBar” type=”string”/><property name=”Date” value=”2013-09-22” type=”date”/><message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”__id” value=”abb344gf67”/> <property name=”PersonId” type=”string” value=”CHGP12Y”/> <property name=”Role” type=”string” value=”DEF”/></message><message name=”HomePlayers”> <property name=”__id” value=”ffhggab3232b”/> <property name=”PersonId” type=”string” value=”MJGP845”/> <property name=”Role” type=”string” value=”KEP”/></message>

</message></message>

<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessUpdateMongo”/> </map.navajomap></navascript>

This will be removed

Friday, August 23, 13

Delete ExampleAssume that we accidentally inserted the following document:

<message name=”MatchReport”> <property name=”_id” value=”aaa666”/> <property name=”MatchId” value=”3” type=”integer”/>

<property name=”HomeTeamId” value=”809832” type=”integer” /><property name=”AwayTeamId” value=”321311” type=”integer” /><message name=”HomePlayers” type=”array”/>

</message>

This can be deleted using following script:

<navascript> <message name=”MatchReport”> <property name=”_id” value=”aaa666”/> </message> <map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessDeleteMongo”/> </map.navajomap></navascript>

Friday, August 23, 13

What about more fine grained mutations?

Friday, August 23, 13

Simple Modify ExampleLet’s just change the date

<navascript> <message name=”__Mongo__”> <property name=”EntityService” value=”entity/MatchReport”/> </message>

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <property name=”Date” value=”2013-09-22” type=”date”/></message>

<map.navajomap useCurrentOutDoc=”true”> <navajomap.callwebservice name=”mongo/ProcessModifyMongo”/> </map.navajomap></navascript>

Friday, August 23, 13

Inserting new Array Element

Insert a new player for the home team:

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”Insert” type=”boolean” value=”true”/> <property name=”PersonId” type=”string” value=”BBFW06E”/> <property name=”Role” type=”string” value=”DEF” /></message>

</message></message>

Note that the properties _id and Insert (set to true) are required

Id is not required,it will be assigned

Friday, August 23, 13

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”Modify” type=”boolean” value=”true”/> <property name=”__id” value=”ffhggab3232b”/> <property name=”Role” type=”string” value=”KEP”/></message>

</message></message>

Id is requiredSet Modify flag

Modifying an Array Element

Change the role of a player:

Friday, August 23, 13

Deleting an Array Element

<message name=”MatchReport”> <property name=”_id” value=”aabf3fdde323g”/> <message name=”HomePlayers” type=”array”>

<message name=”HomePlayers”> <property name=”Delete” type=”boolean” value=”true”/> <property name=”__id” value=”agb3f4g4468”/> </message>

</message></message>

Id is requiredSet Delete flag

Friday, August 23, 13

What have we reached?

• A Flexible Schema to manage MongoDB document data

• Schema is managed outside Mongo

• Basically a metadata template to filter/validate both incoming and outgoing data

• No need to write any “Mapping Code”

Friday, August 23, 13

Normal Services

Entity Services

... Symmetry at last!

Friday, August 23, 13

What is missing?

• Notion of entity instances (to enable client push)

• Relations between entities

• Definition of key/required properties

• Data consistency

• Entity defined persistence

Friday, August 23, 13

Entity First

• Thus far: Operation-Oriented Services: DoUpdateMatchReport, QueryMatchReport, FinalizeMatchReport, etc.

• Resource-Oriented, the Way of the Web

• RESTful resources can be viewed as First Class Citizens (everything is a resource)

• REST method semantics enable use of Cache

Friday, August 23, 13

REST in 1 Slide

• A REST architecture relies fully on HTTP as an Application Protocol

• REST revolves around URI resources (nouns) and HTTP methods (actions) and ...

• Rely on hypermedia to drive the client(!) state machine defined by the application

Friday, August 23, 13

RESTFul Navajo

• Navajo Entity == Resource

• Extend to support inheritance and referencing other Navajo Entities

• Add key properties

• Navajo Entity support one or more HTTP methods (GET, PUT, POST and DELETE)

• Entity Operations (HTTP methods) are implemented by Navajo Services

Friday, August 23, 13

Entity Service<validations> <check code=”A MatchReport should have a date unless it is a DRAFT MatchReport”> [/Match/Status] != ‘DRAFT’ OR CheckDate([/MatchReport/Date)</check></validations><message name=”MatchReport” extends="navajo://Report"> <property name="_id" type="string" key="true,auto"/> <property name="MatchId" type="string" key="true" reference=”navajo://Match/MatchId”/> <property name="Status" type="string"/> <property name="HomeTeamId" type="integer" reference=”navajo://Team/TeamId”/> <property name="AwayTeamId" type="integer" reference=”navajo://Team/TeamId”/>

<message name=”HomePlayers” type=”array”><message name=” HomePlayers” type=”definition”> <property name=”PersonId” type=”string” reference=”navajo://Person”>

<property name=”Role” type=”string”/></message>

</message></message>

Friday, August 23, 13

Entity Operations<navascript> <operations> <operation service=”mongo/ProcessQueryMongo” method=”GET”/> <operation service=”mongo/ProcessModifyMongo” method=”POST”/> </operations> <message name=”MatchReport” extends="navajo://Report"> [ .... ] </message></navascript>

[NOTE: operation == method]

Friday, August 23, 13

Pure HTTP GET

http://dexels.example.org/MatchReport?MatchId=33

Friday, August 23, 13

Backwards Compatibility

<navascript> <map.entity entity="'MatchReport'"> <entity.createproperty name="'MatchId'" value="3" /> <entity.get append="'/'" /> </map.entity></navascript>

Friday, August 23, 13

Entity Insert<navascript> <message name="MatchReport"> <property name="MatchId"> <expression value="‘31232’"/> </property> <property name="HomeTeamId"> <expression value="‘1231232’"/> </property> <property name="AwayTeamId"> <expression value="‘345324’"/> </property> </message> <map.entity useCurrentOutDoc="true" entity="'MatchReport'"> <entity.insert/> </map.entity></navascript>

Friday, August 23, 13

Caching and Consistency

• ETag can help us with

• Preventing that multiple GET operation return same data (send 304 instead)

• Preventing that Stale data is used for updating an Entity (check ETag on PUT,DELETE,POST) (send 409)

Friday, August 23, 13

Polyglot Persistence

• Entity Operations can implement different persistence methods

• Some Entities can “live” inside MongoDB while other reside in e.g. Oracle

• Non-functional data requirements (e.g. consistency, availability) can drive persistence decision

• “Transparent” migration of Data to other datasources

Friday, August 23, 13

Entity drives Service drives Entity drives Service drives Entity drives Service drives Entity drives Service drives Entity drives

Service

• Role of Normal Service and Entity Service are reversed

• Instead of QueryMatchForm, DoInsertMatchForm, DoDeleteMatchForm, we have MatchForm (GET, DELETE, INSERT)

Friday, August 23, 13

Normal Services

Entity Services

... from process- to resource oriented

Friday, August 23, 13