atmosphere 2014: let's build a solid base for a scale. - krzysztof debski

@DebskiChris

#AtmosphereConf

Krzysztof Dębski Allegro Group

Let’s build a solid base for a scale

#AtmosphereConf @DebskiChris Allegro The New World Improvement WIIFY

Agenda

The Allegro Situation How to build a new World? The way to improve What’s in it for you?

Allegro The New World Improvement WIIFY #AtmosphereConf @DebskiChris

The Allegro Situation


Allegro


Allegro

6 million LOC


400 developers

Allegro


A New Hope


A New Hope?

Service

Oriented

Ambiguity


Domain Driven Design


(Micro)Services


(Micro)Services

Business needs Offer

User

Transaction


(Micro)Services

Offer

User

Transaction

Independent


(Micro)Services

API Offer

User

Transaction


(Micro)Services

Polyglot

Cassandra

MongoDB

Oracle

Offer

User

Transaction


(Micro)Services

Smart Endpoints

Cassandra

MongoDB

Oracle

Offer

User

Transaction


The first approach

Service

Auto deployable Monitored Auto scalable Auto healable Auto discoverable


How to build a new World?


The first project

Gradle Spring External Jetty server


Let’s REST

JAX-RS / JSR Compliant @Path("/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Let’s expose our methods

Swagger @Path("/users") @Api(value = "/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @ApiOperation(value=“Get all users”, response=UserCollectionResponse.class) @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Let’s expose our methods

Swagger


And make them discoverable

Offer

User

Discovery ZooKeeper

Register



Offer

User

Discovery ZooKeeper

Get User



Offer

User

Discovery ZooKeeper

Get User Get off

er for user


Multiple API versions

Header support public class UserMediaType {

public static final String V1_JSON = "application/vnd.allegro.user.v1+json” public static final String V2_JSON = "application/vnd.allegro.user.v2+json”

}

curl --dump-header - -H ”Accept: application/vnd.allegro.user.v2+json" -X GET http://localhost:8080/users


Multiple API versions

Content negotiation @Path("/users") @Consumes(AllegroUserMediaType.V1_JSON) @Produces(AllegroUserMediaType.V1_JSON) public class UsersEndpoint {

// [...] @GET @Produces(AllegroUserMediaType.V2_JSON) public UserCollectionResponse findAllUsers() { //[...] }

}


How to configure it?


How to configure it?

Zookeper Environment properties Command-line options


And it became slow


Lower startup time

External Jetty Embedded Jetty Embedded UnderTow


Easier deployment

Cargo deployment Built to zip package On immutable images


Tests


Do you test your tests?

Pitest


Integration tests

Test without mocks Run from IDE

public class UsersIntegrationTest {

private static Map<String, String> overrideConfiguration = Maps.newHashMap();

static { overrideConfiguration.put("property.name", "This was overwritten value"); }

@ClassRule public static final RestServiceStarted DEPLOYED_SERVICE = new RestServiceStarted(overrideConfiguration);

private WebTarget getUsersResourceWebTarget() { return DEPLOYED_SERVICE.getWebTarget().path("users"); }

}


Monitoring


Let’s see what happens

LogStash Kibana NxLog


Let’s see what happens

Graphite # Metrics metrics.reporters.graphite.enabled=true metrics.reporters.graphite.host=graphite.service metrics.reporters.graphite.port=2003 metrics.reporters.graphite.prefix=stats.Prod.service metrics.reporters.interval=30


Monolith Alert


Monolith Alert

Multi module project support


The way to improve


Authentication

OAuth2


Patch Support

Standardized by IETF https://tools.ietf.org/html/rfc6902

PATCH /user/1 HTTP/1.1 Host: user.service.local Content-Length: 312

Content-Type: application/json-patch+json [

{”op”: ”test”, ”path”: ”/firstname”, ”value”: ”Jane”}, {”op”: ”add”, ”path”: ”/maidenname”, ”value”: ”Smith”}, {”op”: ”replace”, ”path”: ”/lastname”, ”value”: ”Doe”},

{”op”: ”remove", ”path”: ”/meetings”}, {”op”: ”move”, ”from”: ”/balance”, ”path”: ”sharedbalance”}, {”op”: ”copy”, ”from”: ”/a”, ”path”: ”/b”}

]


Simplify annotations

Swagger? @Path("/users") @Api(value = "/users") @Consumes(CONTENT_TYPE_JSON) @Produces(CONTENT_TYPE_JSON) public class UsersEndpoint {

// [...] @ApiOperation(value=“Get all users”, response=UserCollectionResponse.class) @GET public UserCollectionResponse findAllUsers() { //[...] }

}


Deployment

Docker support


What’s in it for you?


Use the latest tools

GradleW issue in Continuous Integration Environment and parent POM issues. Don’t do DDOS yourself and your partners’ sites.


Change is the only constant

HTTP Server Service API provider Dependency injection engine Deployment tools


Focus on right metrics

import org.junit.Test;

import static net.trajano...

public class MediaTypesTest {

@Test public void mediaTypesShouldBeValidUtilityClasses()

throws Throwable { assertUtilityClassWellDefined(UserMediaType.class); } }


Tools also lie

Metrics tend to lie – Default PHP metrics in SonarQube – Tested file

•  4535 CLOC •  Whole code written using imperative programming

How many violations are there?


Tools also lie

How many violations are there?

4 At least according to Sonar


Learn to REST

REST is not so obvious

/api/bi/XYZ123/1

/recommendations/1/items

/offer/100?output=ESI/JSON


Community

Involve all developers in building the Bootstrap. Or they will build their own tools.

Łukasz Drumiński Mateusz Gajewski @wendigo


Q & A

atmosphere 2014: let's build a solid base for a scale. - krzysztof debski

Presentations & Public Speaking