open source databases as a swiss pocket knife polyglot ... persistence... · art van scheppingen -...

Art van Scheppingen - Senior (No)SQL DBA @ VidaXLBart Oleś - Senior Support Engineer @ Severalnines

Polyglot persistence: utilizing open source databases as a

Swiss pocket knife

Who are we?

VidaXL Company Stats

• Online retailer in (mostly) slow moving goods

• Founded 2008

• 350M turnover, 40% growth yearly

• 1500 employees (US, CN, AU, IN, RO, UA)

• HQ in the Netherlands

• 4 warehouses worldwide (NL, US and AU)

How does VidaXL sell its goods?

• Own webshop platform in EU, US and AU

• Warehouses in NL, US and AU

• Selling on other platforms, e.g. Amazon, eBay

• Allow selling on our own platform using Mirakl

• B2B drop-shipments

VidaXL Technical Foundations

• SAP as ERP system

• Genesys as CS system

• Webshop

• Open source web-based development strategy

• PHP / NodeJS

• Docker

• Cloudflare workers

VidaXL DevOps Datastores

• MySQL

• MariaDB (Galera) clusters

• MySQL replication

• PostgreSQL

• SOLR

• Elasticsearch• ELK

• MongoDB

• Couchbase

• (RabbitMQ)

• Prometheus

What is Polyglot Persistence?Using multiple specialized persistent stores rather than one single general-purpose database

Where does the term come from?

• The way we work is changing

• Enterprise applications are becoming more complex

• Separate (devops/agile) teams

• Ownership of applications

• (Micro)services

• Everyone has their preference

• Various programming languages

• Various storage systems

Where does the term come from?

• Monoglot Programming

• Only one programming language allowed

• Readability

• All code is in the same language

• Support

• One platform to support

• Knowledge

• Everybody is an expert

• Is there a jack-of-all-trades language?

Where does the term come from?

• Monoglot Programming

• Only one programming language allowed

• Readability

• All code is in the same language

• Support

• One platform to support

• Knowledge

• Everybody is an expert

• Is there a jack-of-all-trades language?

Monoglot programming•

Monoglot programming

Carpenters actually use a broad variety of tools

Polyglot Programming

• Polyglot Programming• Use programming languages for what they are good at• Flexibility

• Use Java for a secure API• Use Scala for real time stream processing• Use Python for text analysis• Tie everything together using AngularJS

• Knowledge• Everybody is expert at one or more languages

Polyglot Programming

Monoglot Persistence

Data storage landscape changes

• Relational data stores (RDBMS)• Key-Value data stores (“NoSQL”)• Columnar data stores (OLAP)• Document data stores (NoSQL)• Graph data stores (GDB)• Big Data

Data storage landscape changes

Software

RDBMS Oracle, MySQL, PostgreSQL

Key-Value Redis, Riak

Columnar InfiniDB, Clickhouse

Document MongoDB, Couchbase

Graph Neo4J, Janusgraph

Big Data Hadoop

Data storage landscape changes

Software AWS Google

RDBMS Oracle, MySQL, PostgreSQL

RDS, Aurora CloudSQL, Spanner

Key-Value Redis, Riak DynamoDB Datastore

Columnar InfiniDB, Clickhouse Redshift BigQuery

Document MongoDB, Couchbase

SimpleDB Bigtable

Graph Neo4J, Janusgraph Neptune

Big Data Hadoop EMR Cloud Dataproc

Even Hadoop has become a polyglot

Polyglot Persistence

• Complex problems require different storage systems• Use the right tool for the job, for example

• Use PostgreSQL for financial data• Use MySQL for website contents• Use MongoDB for user profiles• Use Cassandra for real time streams• Use Neo4J for recommendation analysis

Use the right tool for the right job

Document storage: MongoDB

Use the right tool for the right job

Columnar storage: Cassandra

Use the right tool for the right job

Graph storage: Neo4J

Polyglot Persistence

Polyglot Persistence at VidaXLYes we certainly are polyglots!

Quick recap on our data stores

• MySQL• MariaDB (Galera) clusters• MySQL replication• ProxySQL

• PostgreSQL

• SOLR

• Elasticsearch• ELK

• MongoDB

• Couchbase

• (RabbitMQ)

• Prometheus

How did this happen?

• Continuous growth

• Hardly any time to overhaul existing systems

• Transition from monolith to microservice architecture

• For each microservice the most optimal solution has been chosen

• Early adopters of new technology

• Gaining advantage over competition

From monolith to microservice

From monolith to microservice

From monolith to microservice

From monolith to microservice

From monolith to microservice

From monolith to microservice

What were the challenges?

• Automation

• Increased complexity

• Systems monitoring

• Multiple integrations

• Maintenance becomes more difficult

• Backups

• Scaling

• Software updates

• DevOps are not a DBA

What were the solutions?

• Invest in automation

• Never perform any (large) task thrice

• Increase tooling

• Build it ourselves costs time

• Buying/licensing tools costs money

• Keeping the headcount low saves money

• Focus on systems that matter most

• Get (exteneral) help

• Hire DBAs! ;)

ClusterControl by Severalnines<some subtitle here?>

Thank you