herd of containers · app container images pods aggregate images in one shared environment...
TRANSCRIPT
![Page 1: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/1.jpg)
Herd of Containers
![Page 2: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/2.jpg)
Saâd DIFDatabase Engineer
![Page 3: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/3.jpg)
Herd of Containers: PostgreSQL in containers at BlaBlaCar
pgDay Paris, Mar 15, 2018
![Page 4: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/4.jpg)
Today’s agenda
BlaBlaCar Overview
PostgreSQL usage at BlaBlaCar
Switching to a new implementation
![Page 5: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/5.jpg)
BlaBlaCar Overview
![Page 6: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/6.jpg)
60 million members
Foundedin 2006
1 million tonnesless CO
2
In the past year
30 million mobileapp downloadsIphone and Android
15 milliontravellers
Currently in22 countriesFrance, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey.
Facts and Figures
![Page 7: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/7.jpg)
Core Data Ecosystem
1 2 3
MySQL
Main DatabaseMariaDB 10.0+Galera Cluster
Cassandra
Column OrientedDistributed
Redis
In MemoryKey-ValueOptional durability
![Page 8: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/8.jpg)
Core Data Ecosystem
4 5
ElasticSearch
JSON documentsFullText searchDistributed
PostgreSQL
ORDBMSExtensibilityStability
![Page 9: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/9.jpg)
Why Containers ?
Resource allocationDeployment Speed
On premise
Skills already thereCost
Containers
![Page 10: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/10.jpg)
Rkt
Why Rkt over Docker ?
CoreOS Container Linux
Linux DistribSimple & Secure
Only run containers
Fleet
OrchestrationBy default with CoreOS
Containers
![Page 11: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/11.jpg)
GGN
Generate systemd units
Dgr
Build and configure App Container Images
Pods
Aggregate images in one shared environment
Containers
![Page 12: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/12.jpg)
bare-metal servers
1 type of hardware
3 disk profiles
fleet cluster
CoreOS
fleet etcd“Distributed init system”
Hardware
Container Registry
ggn
dgr
Service Codebase
rkt PODs
build
run
store
host
create pgsql
monitoring
nerve
pgsql-main1
php
nginx
nerve
monitoring
synapse
front1
synapse
nerve
zookeeper Service Discovery
Containers
![Page 13: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/13.jpg)
Get rid of DNS internallyAdapt to change
Why ?
1
Service Discovery
![Page 14: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/14.jpg)
Key-Value storeReliable, Fast, Scalable
Why ? Zookeeper
21
Service Discovery
![Page 15: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/15.jpg)
Go-NerveHealth ChecksEphemeral keysPresent on each pod
Why ? Zookeeper Report
2 31
Service Discovery
![Page 16: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/16.jpg)
Go-SynapseWatch ZookeeperUpdate HAProxy configuration
Why ? Zookeeper Report Discover
2 3 41
Service Discovery
![Page 17: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/17.jpg)
backend pod
client pod
Service Discovery
/database/node1
go-nerve does health checks and reports to zookeeper in
service keys
node1
/database
Applications hit their local haproxy to access backends
go-synapse watches zookeeper service keys and reloads haproxy if changes are
detected
HAProxy
go-nerve
Zookeeper
go-synapse
![Page 18: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/18.jpg)
PostgreSQL usage at BlaBlaCar
![Page 19: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/19.jpg)
Prerequisite
PostGIS
Third-party applications
Spatial
Confidence
Home Made tools
Usage
![Page 20: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/20.jpg)
Travel company
Corridoring
Point to Point
PostGIS
RambouilletParis
Lyon
Le Creusot
![Page 21: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/21.jpg)
3 685 1MRides passed by
Amiens last month
Number of
meeting points
50kRows reads per
minutes
![Page 22: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/22.jpg)
Change!Streaming Replication
Manual Interventions Not friendly
Painful failover recovery
Operate
![Page 23: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/23.jpg)
Target
Scale writes
Ease deployments
Maximum availability
Slaves
Failovers
Expandable resources
![Page 24: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/24.jpg)
Possibilities
Postgres-XC (x2)
Postgres-XL
PgLogical
Bucardo
Slony
Londiste
![Page 25: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/25.jpg)
Switching to a new implementation
![Page 26: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/26.jpg)
BDR
Bi-Directional Replication
OpenSource project by 2ndQuadrant
Multi Master Asynchronous Replication
2 to 48 nodes
Optimal for Geo Distributed databases
![Page 27: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/27.jpg)
BDR : The Confirmation
All nodes support reads and writes
No failovers
No other process / nodes needed
Partition tolerant
![Page 28: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/28.jpg)
BDR : Caveats
Modified version of PostgreSQL 9.4BDR 2.0 with PostgreSQL 9.6 for 2ndQuadrant support customers
DDL lock
Replication lag
Conflicts
Some statement not supported yet
Statement not replicated
![Page 29: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/29.jpg)
Check
InitCheck if node have
entries in the bdr_nodes table, if
yes : skip init
Implementation
Run
[~/build-tools/aci/aci-postgresql-bdr] $ tree.├── Jenkinsfile├── aci-manifest.yml├── attributes│ ├── base.yml│ └── postgresql.yml├── files│ └── tmp│ └── postgresql│ ├── environment│ ├── pg_ctl.conf│ ├── pg_ident.conf│ └── start.conf├── runlevels│ ├── build│ │ └── 00.install.sh│ └── build-late│ └── 00.clean.sh└── templates └── dgr └── runlevels └── prestart-late ├── 00.init-instance.sh.tmpl └── 01.init-database.sh.tmpl
![Page 30: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/30.jpg)
Implementation (init)
1 If no “donor” attributes : Init as new group
2
3
1
When the node have “donor” attributes :
Retrieve user definition on donor (pg_dumpall -g)
Join BDR group
Create minimum objects if not present
2
1 Part local node on donor
Delete entries on donor(bdr_nodes and bdr_connections)
New fresh node Node already referenced but changed host or have lost his data
![Page 31: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/31.jpg)
Pager DutyIncidents Manager
GrafanaBeautiful Visualizations
PrometheusSmart Monitoring
ExporterExpose metrics
Monitoring and Alerting
![Page 32: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/32.jpg)
Monitoring
Key principles:
Usage
Saturation
![Page 33: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/33.jpg)
BDR exporter specifics$ cat aci-prometheus-postgresql-exporter/templates/queries.tmpl.yaml
{{ if .use_bdr }}pg_replication_bdr_count: query: "select (select count(*) from bdr.bdr_nodes) as bdr_nodes, (select count(*) from bdr.bdr_connections) as bdr_connections;" metrics: - bdr_nodes: usage: "GAUGE" description: "Number of rows in the bdr_nodes table" - bdr_connections: usage: "GAUGE" description: "Number of rows in the bdr_connections table"{{ end }}
pg_replication_count: query: "select (select count(*) from pg_stat_replication) as stat_repli, (select count(*) from pg_replication_slots where active=true) as rep_slots;" metrics: - stat_repli: usage: "GAUGE" description: "Number of rows in the pg_stat_replication table" - rep_slots: usage: "GAUGE" description: "Number of rows in the pg_replication_slots table with the active status"
[...]
Template values for BDR specifics
Extend metrics to all PostgreSQL needs
![Page 34: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/34.jpg)
Backup and Recovery
1 Retrieve dumpspg_dump
2 Alter structure dump
3 Load structure and data dump
![Page 35: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/35.jpg)
Backup and Recovery
$ cat pod-mysql-backup/aci-backup/templates/opt/backup-main.tmpl.sh
function startbackup { begin_unixtime=$(date +%s) cat <<EOF | curl --data-binary @- http://prometheus-gw:9091/metrics/job/backup_{{.env}}/target/$node/service/$service/type/{{.backup.type}} # HELP backup_begin_unixtime # TYPE backup_begin_unixtime counter backup_begin_unixtime $begin_unixtimeEOF}
![Page 36: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/36.jpg)
$ cat prometheus-rules/alert.postgresql.rules
# Alert: There is less replication active than bdr nodesALERT BackupsTooOld
IF time() - backup_end_unixtime{exported_service=~".*postgresql.*"} ) > ( 3600 * 24 )
LABELS { severity="warning", stack="backups", team="data_infrastructure"}
ANNOTATIONS { summary="Backup {{ $labels.type }} on {{ $labels.exported_service }}.{{ $labels.target }} is too old.", dashboard=" https://grafana.blabla.car/dashboard/db/db-backups ",}
Alerting
PromQL to find out unhealthy services
Labeling for routing to Slack & Pager Duty
Annotations with templating to have clear descriptions, URL to dashboards and ops runbooks
![Page 37: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/37.jpg)
Feedback
Clearly satisfied with availability
Reactive community
Know what your needs are
Sanity checks
BDR 3.0 coming soon!
![Page 38: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/38.jpg)
What’s next?
![Page 39: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/39.jpg)
![Page 40: Herd of Containers · App Container Images Pods Aggregate images in one shared environment Containers. bare-metal servers 1 type of hardware 3 disk profiles fleet cluster CoreOS fleet](https://reader034.vdocuments.us/reader034/viewer/2022052013/6029cf8055e3ce301d001dd6/html5/thumbnails/40.jpg)