from no services to microservices

From No Services to Microservices

A journey of a FinTech application into the cloud, starting as a monolith and growing to MicroServices. Why going functional is so good for finance

technology, and how Scala helped us build a better application.

João Cavalheiro - e.Nearjoao.cavalheiro@enear.co

Fintech in a Nutshell• Fintech is mostly about using technology to

make financial services more efficient.

• Fintech companies are in many cases startups, and have the purpose of disrupting existing financial systems.

• Popular areas are payments & lending, blockchain and market monitoring and tracking

Developing Fintech Software

• Complex business models

• Data security

• Code stability and adaptability

• Quality of service (QOS)

Your programming language, team skills and toolkit must be able to deal with all the above

Avoid• Non-Agile development processes

• Growing the team too quickly

• Start writing code before you understand the business model

Go Functional

• Intent over implementation

• Immutability - correct code by design

• Transparent concurrency

• Use conventions and patterns

Use Scala (you should)

• Focus on easiness of refactoring instead of elite-looking code

• Do not abuse implicits!

• Part of our team was moving from Java, so..

Leverage on Java Ecosystem

• Lots of useful Java libraries out there. You can use them in your Scala project

• Don’t feel guilty for using OO design in your first days in Scala

• All JVM concepts can be applied

Why Starting with a Monolith can be Good

• Fintech companies sometimes grow really fast

• The business model is a moving part

• Logical option is to invest time on a carefully designed monolith

• It would be simpler to move to microservices later

Monolith Design

• Logical option is to invest time on a carefully designed monolith

Monolith Design

• Should be adjusted to business and product requirements

• Sometimes is just better to focus initially on your monolith functionality and gradually peel off microservices at the edges

• You can also sacrifice your currently architecture and go for a complete system redesign

Our Application• Software-as-a-Service

• Core micro-finance system

• Combines administration, security, accounting, loan portfolio management and deposit tracking

• Currently in production with 20 paying customers (MFI banks) and over 400.000 end clients.

Initial Tech Stack• Using Scala and Play! framework

• PostgresSQL database

The Data Model

• Data model is event-based, every operation over an account is an event

• Repayment schedules are stored in the database, one row per instalment

• We also keep track of all non-transaction actions on the system

The Hit on The Wall• We reached 200.000 lines of code on our

monolith play! Application.

• It was getting hard to keep development under control

• Bug density was increasing with the product complexity

• And we had a few demanding requirements…

Business talks• Operations, including transactions, can be

backdated and reverted, any moment in time

• It is in fact, possible to change the past

• We could not rely on a simple snapshotting model. The snapshots in the past could change, and they would affect all the subsequent snapshots as well.

Headed for Microservices

• Services that are themselves very simple, focusing on doing one thing well

• Use different languages/frameworks and tools for each service

• Integrate new team members much easier

The First Microservice• Our immediate need was to find a dynamic

snapshot solution to accommodate transaction backdating and have the updated detailed status of each account, for at least the last 30 days

• This could never operate on top of current application stack and database

• We had a great candidate for the first microservice

Landscape

CQRS• The concept at the heart of it is to use a different

model to create/update information than the model you use to read information

• It adds complexity, but we were going to use it only on specific portions of the system

• It allowed us to define separate materialised query views that implement the aggregation strategy for account transactions

The outcome

• The view model It’s eventually consistent, but reports are in most cases statistical, and therefore “almost-real-time” is acceptable

• Denormalized = Efficiency lookup

• All seemed great!

The Dark Side

The doubts

• Can we share a single database?

• How do we manage dependencies?

• Multiple code bases, multiple git repos & branching models

The Next Service and experience with AKKA HTTP

• The need to add document management to our product and to do it with a scalable solution

• We needed something lighter than Play!

• Spray is no longer actively developed

• AKKA-http?

AKKA-http

• It was a moving target. Api stability was not proven

• It doesn’t work right out of the box

• Should we fall back to Play?

AKKA-http

• With Akka-http you can choose and control each component of your solution

• Add new components with a smaller risk

• Easier to find and fix problems

• But, if you are short on time, under pressure, and you don’t know Spray or Akka-http at all, consider Play

It worked in DEV…

• AKKA-http proved to be a good solution

• Microservices were helping

• Performance was improved

• But, someone was not happy…

“… OPS problem now”• Microservices come with operations overhead:

• Monitoring

• Deployment

• Security

• Seriously consider to have DEVOPS

What we Learned• Moving to microservices is not a silver bullet,

and the increased complexity means there are a lot of areas which need to be carefully considered

• All in all, microservices are not a free lunch, but in many cases it’s a worthwhile investment

• Scala made it easier!