breizhcamp 2013 - pimp my backend

By @FinistSeb and @LostInBrittany

Le développeur de l'Est le plus à l'Ouest !Java & C++ Developer, coding addict,

continuous intégration ayatollah

● Senior Software Engineer currently, Crédit Mutuel Arkea

● FinistJug actif member

BreizhBeans community and commiter

sebastien.lambour@gmail.com +sebastien.lambour

@FinistSeb

Sebastien Lambour

Horacio GonzalezSpaniard lost in Brittany, Java developer,

dreamer and all-around geek

● Senior Software Engineer at Crédit Mutuel Arkea○ Currently...

● Leader of the FinistJUG and the BreizhBeans community

http://lostinbrittany.org/ +Horacio.Gonzalez @LostInBrittany

BreizhBeans

Developer community at the far end of French Brittany

http://breizhbeans.org/ @BreizhBeans

I help to create a startup on social translation, would

you code with me ?

Great, I do the backend! How many members?

Some 30.000 - 50.000 will be fine!

Ouch, is it not too small? I like big toys!!!!

Me too!Let's think big then

OK, it will rock !

Nexus 4

Why ?

Backends They often look like that !

Transforming it is HARD !

Next challenge :Granite aircraft !!

A sexy backend?Let's groom it!

OK, go around...

Light weight Test driven

ScalableCloud Ready

Java

Data Driven

Following the beaten trackBrainless route

The usual suspects...

JSF 2.0Hibernate

JPACXF

SQL

Spring CRUD(Create, Read, Update,Delete)

Booooooring...And ineffective

Let's use it!

Design choicesBecause we need some solid roots

Core principlesWhat do we want our backend to be?

● Scalability ○ Because traffic growth should be a blessing, not a problem!

● Adaptability ○ Because your business evolves, and your backend need to evolve

with it!

● Testability○ Because you want to be sure that your code works!

● NoSQL (meaning Not Only SQL)○ Because you need not only tables and foreign keys, but non-SQL

stores, search indexes, caches...

● Productivity○ Because the other 4 doesn't matter if you need to spend 4

working days to modify a service...

A first choice : Apache Thrift

« Software framework for scalable and performant cross-language services development »

● Interface description language● Multi-language code generation● Serialisation/deserialisation● RPC Framework● High performance

○ On transport layer○ On data manipulation

Thrift, it's cool Thrift boots productivity!

A second choice : MongoDB

● We needed a NO-SQL store for documents

● We wanted to use Cassandra○ Cassandra & Thrift, a love story

● We wanted a PaaS-ready backend○ Neither Cloudbees nor Cloudfoundry supported Cassandra○ But they supported MongoDB

● We loved MongoDB○ Document DB are great for document oriented applications

ThriftMongoBridgeBecause we wanted both

Thrift and MongoDB

Thrift rocks and works

But...

How to put Thrift objects into MongoDB ?

First bad idea!

Use Spring Data or Morphia

Thrift object mapping is bad !

Damned!

First bad idea!

Second bad idea !

Use the Mongo Driver JSON parsing

Deserialisation will be a bit hard...

Damned !

Second bad idea!

Thrift isn't cool anymore ?

We need a new idea !

Thrift, it's cool !But we had work to do...

Write a TBSONProtocol !

The ThriftMongoBridge is born !

What does it do ?

struct AnotherThrift {1:string anotherString,2:i32

anotherInteger,}

Thrift IDL

POJO generated by the Thrift compiler

public class AnotherThrift implements org.apache.thrift.TBase... { ... public String anotherString; public int anotherInteger;

... public static final Map<_Fields, org.apache.thrift.meta_data.FieldMetaData> metaDataMap; .... public void read(org.apache.thrift.protocol.TProtocol iprot)... ... public void write(org.apache.thrift.protocol.TProtocol oprot) ......;}

Compile to

{anotherString:"value", anotherInteger:69}

Write To BSON Read from BSON

TBSonSerializerTBSonProtocol TBSonDeSerializer

Really simple to use Too simple ?

AnotherThrift anotherThrift = new AnotherThrift();

anotherThrift.setAnotherString("value");

anotherThrift.setAnotherInteger(69);

// serialize into DBObject

DBObject dbObject = ThriftMongoHelper.thrift2DBObject(anotherThrift);

{anotherString:"value", anotherInteger:69}

How it works

DBObject (BSON)

TBSONProtocol

TBaseTBSONSerializer

write(protocol)

getResult

Context Stack

sentense directives (begin / end)

Data (read / write)

Struct contextDBObject

push pop

Thrift object generated

MongoDB object

The BreizhCamp demoBecause you want to see some code,

don't you?

The BreizhCamp demoGiven this Thrift model & service

struct BreizhCampEvent { 1:i64 date, 2:string startTime, 3:string endTime, 4:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 5:string room, 6:string resume, 7:string details, 8:list<string> speakers 9:list<com.breizhbeans.backend.thrift.enums.model.Track> tracks,}

service ProgramService {

com.....GetProgramResponse getProgram(1:com.....GetProgramRequest request)

throws (1:com....TechnicalException texp, 2:com...FunctionalException fexp)

}

struct GetProgramRequest {1:com.breizhbeans.backend.thrift.enums.model.Track track,2:com.breizhbeans.backend.thrift.enums.model.EventTypes type,3:string room,4:list<string> speakers,

}struct GetProgramResponse {

1:list<com.breizhbeans.backend.thrift.model.BreizhCampEvent> events,}

The BreizhCamp demoFirst : Write a test

● One loader by collection● One goal : Have Human readable test ;-)● Manage automatically data insertion before test and

clean up after test @Test

public void testGetEvents() throws Exception {

List<BreizhCampEvent> actualEvents = programDao.getEvent();

List<BreizhCampEvent> expectedEvents = new ArrayList<BreizhCampEvent>();

expectedEvents.add(getEventPimpMyBackend());

Assert.assertEquals(expectedEvents, actualEvents);

}

Application code to test

Standard JUnit assertion

Expected data building(used by the test and the data loader)

The BreizhCamp demoWrite a DAO

public List<BreizhCampEvent> getEvent() throws TechnicalException {

DBCursor cursor = null;

try {

List<BreizhCampEvent> events = new ArrayList<BreizhCampEvent>();

DBCollection collection = db.getCollection(collectionName);

DBObject query = new BasicDBObject();

// execute the query

cursor = collection.find(query);

while(cursor.hasNext()) {

events.add( (BreizhCampEvent) ThriftMongoHelper.DBObject2Thrift(cursor.next()));

}

return events;

} catch (Exception e) {

throw new H4TTechnicalException("Unexpected error", e, null);

} finally {

cursor.close();

}

}

retrieve DB collection

Build and execute the Query

Deserialize Thrift objects

Error handling

Never forget it ;-)

@Override

public BreizhCampEvent updateRoom(final BreizhCampEvent event) throws TechnicalException {

try {

BasicDBObject updateQuery = new BasicDBObject();

updateQuery.append("$set", new BasicDBObject()

.append(getFields(BreizhCampEvent._Fields.ROOM), event.getRoom()));

db.getCollection(collectionName)

.update(getPrimaryKeyRequest(event), updateQuery)

.getLastError()

.throwOnError();

return event;

} catch (Exception e) {

throw new H4TTechnicalException("Unexpected error", e, ObjectUtils.toString(event));

}

}

The BreizhCamp demoCan I update thrift Object by code?

Mongo Update Query

Thrift structure mapping

Query execution

Error handling

The BreizhCamp demoCan I update thrift Object by script?

Mongo DB Bson writed : > db.program.find();

{ "_id" : "id=10", "date" : NumberLong("-56433978000000"), "startTime" : "14:45", "endTime" : "15:40", "type" : 4, "room" : "Bréhat", "resume" : "Pimp my backend!", "details" : "Le frontend, le frontend...... ...", "speakers" : [ "Sébastien Lambour", "Horacio Gonzalez" ], "tracks" : [ 7 ], "id" : 10, "classname" : "com.breizhbeans.backend.thrift.model.BreizhCampEvent" }

Update the field room :> db.program.update();

{ "id": 10}, { $set: { "room": "Ouessant" } }

And the perfs?Because speed matters!

Talking about perfsThe candidates are ...

Morphia

Spring Data

And... ThriftMongoBridge, of course

Spring Data vsThriftMongoBridge

● Write and Read 500 objects into MongoDB.

● Each document is composed with a List<> and a Set<> of 500 objects.

● The document have a JSON size of 91613 bytes

Who will win ?

CodeThriftMongoBridge

First// Get a collection

DBCollection collection = db.getCollection("dummyColl");

Write// Serialize the Thrift object

DBObject dbObject = ThriftMongoHelper.thrift2DBObject(thriftObject);

// Put the document

collection.insert(dbObject);

ReadDBCursor cursorDoc = collection.find();

while (cursorDoc.hasNext()) {

DBObject dbObject = cursorDoc.next();

ThriftObject thirftObject = (ThriftObject)ThriftMongoHelper.DBObject2Thrift(dbObject);

}

CodeSpring Data

First// Retrieve the MongoOperations bean

MongoOperations mongoOperation =

(MongoOperations) ctx.getBean("mongoTemplate");

Write// Serialize the Thrift object

mongoOperation.save(springObject, "springObject");

ReadList<SpringObject> listUser =

mongoOperation.findAll(SpringObject.class, "springObject");

OK. Spring Data is (a little bit) more compact...

Time to run !

Thrift over MongoDB rocks!

We need a logo for our OpenSource library...

Available now :

2.0

https://github.com/BreizhBeans/ThriftTools

A storage library a backend is not,young padawan

Because we need more things

At the beginning...Pretty layers, neatly isolated, like a Care Bears' birthday cake

Later in the projectLike a cake after an earthquake

And in the end...As pretty as a British pudding... and almost as tasty

Be Layer lessNow build the Backend !

● Each package have a goal

● NO package assembly rules

● Improve the complexity detection by dependencies analysis

Goal : Fight against the "God classes" anti-pattern

unreachable dream

Prevent protocol rupture, have a stable services signature.

com.h4t.CheckAccessResponse checkAccess (1:com.h4t.CheckAccessRequest request) throws (1:com.h4t.TechnicalException texp, 2:com.h4t.FunctionalException fexp)

stable method signature

Thrift services

Integrate this into the backend

Thrift IFace implementation

Request / Response decapsulation + log & monitoring

Core Business code

MongoOpérations

Thrift object

ThriftMongoHelper

Mongo Driver

DBObject

write read

Query

DBObject

Inversion of control

Keep simple too !

Prefer static dependencies injection.

Time to do the choice

or

Inversion of control

Why ?

Because and

○ binding of the MongoDB service by the cloud○ no production configuration required○ only one war

TestabilityBecause without tests

you need faith

Test strategy

No compromise!The rules are simple... no exception!

TDD is my religion1. Execute test without setup2. Execute test with your build tool3. Execute test into your IDE4. No injection into test framework5. Reuse only application injection6. Keep it simple (repeat it 7 times)7. Have a loose coupling (test the smallest parts)8. Test must be deterministic and reproducible9. Do pair review with the tests

10. If the complexity is growing up refactor now...

11. If a rule goes wrong, rewrite it !

From concept to reality

ThriftMongoTestToolsThrift DataSet

Collection Loader

Load thrift object with preset datasinto MongoDB

Get expected Thift object with preset datas

Get a Thrift object from MongoDB related to a unique key

DAO : Validate queries and set up the test data loaders

IFACE : Functional tests

SERVICE : Test a business centric service

TU : Unit test not related to the business (test first approach)

Tests have goals

@Before

public void setUp() throws Exception {

eventLoader.load(getEventPimpMyBackend());

}

private BreizhCampEvent getEventPimpMyBackend() throws Exception {

String details = "Le frontend, le frontend... .....";

BreizhCampEvent event = eventLoader.getExpected(10, "14/06/2013", "14:45", "15:40", EventTypes.CONF, "Bréhat", "Pimp my backend!", details);

event = eventLoader.addSpeakers(event, "Sébastien Lambour", "Horacio Gonzalez");

event = eventLoader.addTracks(event, Track.CLOUD);

return event;

}

Tests are Data Driven

Build test Data with java code, files, BDDor the MongoThrift Test tools (not opensourced yet ;-) )

Inject Data into MongoDB

What's next?Because our work is not completed... yet!

Why integrate our lib ?

#Pragmatism● Productivity oriented● Fully testable ● Container less (no servlet dependencies)● Open Source : Apache 2.0

The ThriftMongoBackend Core model will born soon !

With a name: TINAF (ThriftMongoBackend Is Not A Framework)

What's next ?

● monitoring Thrift services in the cloud

● load tests with vmware & 10gen

● contribute TINAF

● Develop a thrift service valve to ensure brandwitdth for critical services

... do a talk at Devoxx BE

Thank you !