deployment and monitoring - cern
TRANSCRIPT
![Page 1: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/1.jpg)
Deployment and monitoring
Nicola Tarocco and our cool teamCERN
![Page 2: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/2.jpg)
Outline• Deployment configurations• Dimensions• Monitoring• Logging
2
![Page 3: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/3.jpg)
Deployment configurations
3
Prepare your infrastructure for production
https://invenio.readthedocs.io/en/latest/architecture/infrastructure.html
![Page 4: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/4.jpg)
Infrastructure overview
4
• Load balancers: HAProxy, Nginx or others.• Web servers: Nginx, Apache or others.• Application servers: uWSGI, Gunicorn or
mod_wsgi.• Distributed task queue: Celery• Database: PostgreSQL, MySQL or SQLite.• Search engine: Elasticsearch (v5 and v6).• Message queue: RabbitMQ, Redis or
Amazon SQS.• Cache system: Redis or Memcache.• Storage system: Local, S3, XRootD,
WebDAV and more.
![Page 5: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/5.jpg)
Load balancer: HAProxy
5
• Traffic distribution depending on its “type”• /static, /<path>
• URL Rewriting• SSL Termination• Downtime static website
• More than 1: redundancy
![Page 6: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/6.jpg)
HAProxy configuration
6
• Frontend• Backend• Headers: X-Forwarded-For
Go to documentation
![Page 7: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/7.jpg)
Web app: uWSGI (behind Nginx)
7
• PEP 333 (TL;DR)• Invenio UI and REST• processes and threads
![Page 8: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/8.jpg)
Web app: Nginx
8
• ngx_http_uwsgi_module: translate HTTP -> WSGI
• Reverse proxy to distribute traffic• ui, api, static, files
• protocols and ciphers• logging• headers: X-Request-ID
![Page 9: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/9.jpg)
DimensionsProcesses and Threads
9
How many servers do we need? How do I configure them?
![Page 10: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/10.jpg)
Let’s start with the database
10
• Depends on DB: Postgresql? MySQL? Servers?• General: configure max connections• DB specific: check documentation
Example: 200 max connections$ less /var/lib/pgsql/data/postgresql.confmax_connections = 200
$ mysql> SET GLOBAL max_connections = 200;
![Page 11: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/11.jpg)
Web nodes: uWSGI
11
• From the uWSGI documentation (interesting discussion here)“There is no magic rule for setting the number of processes or threads to use. It is very much application and system dependent. Simple math like processes = 2 * cpucores will not be enough. You need to experiment with various setups and be prepared to constantly monitor your apps. uwsgitop could be a great tool to find the best values.”
• Processes: • 1 process -> 1 Invenio app• start with 2 * CPU Cores, and then measure and adjust
• Threads: low, it really depends (take into account GIL, multithreading, etc...)
Example: 8 processes, 2 threads
![Page 12: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/12.jpg)
Max processes per web node
12
• SQLAlchemy: pool of db connections `pool_size` (default 5) per process
• Each uWSGI process -> 5 db connections• 8 process per web node -> 5 * 8 = 40 connections• 200 max connections on db / 40 per web node = ???
5 web nodes maxBUT
• You have to count all Invenio apps (your workers, celery apps)!
![Page 13: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/13.jpg)
DimensionsHTTP Connections
13
How many connections per web node? How do I distribute traffic?
![Page 14: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/14.jpg)
Measure!
14
![Page 15: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/15.jpg)
uWSGI: Start small
15
1. Predict/Estimate the n. of request/sec you need2. Take one web node3. Configure number of uWSGI processes4. Benchmark by increasing number of requests
locust --host=https://mydomain.org/ --no-web -c 1000 -r 100
![Page 16: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/16.jpg)
uWSGI: Iterate
16
Keep tuning processes/memory until your reach your target
![Page 17: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/17.jpg)
We found out how many connections per web nodes.
And now?
17
![Page 18: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/18.jpg)
HAProxy: distribute load
18
Since we now know how many connections per node, we can distribute the load
![Page 19: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/19.jpg)
Recap
19
1. Max db connections2. Processes/threads per web node3. Max connections per web node4. Load balancer config
![Page 20: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/20.jpg)
Monitoring
20
Health of your production infrastructure
![Page 22: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/22.jpg)
Web app: uWSGI
22
• uWSGI Stats Server• uwsgitop
![Page 26: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/26.jpg)
26
Monitoring
![Page 27: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/27.jpg)
Alerting: UpTimeRobot
27
![Page 28: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/28.jpg)
Logging
28
Understand what happened
![Page 29: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/29.jpg)
Request tracing
29
How do we find the traces of a specific request in each server?
How can we get the request IP?
![Page 30: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/30.jpg)
Load balancer: HAProxy
30
![Page 31: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/31.jpg)
Web app: Nginx
31
Infrastructure IPs
Nginx log config
Log example
![Page 32: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/32.jpg)
Web app: Nginx
32
Nginx passes params to the uWSGI server
![Page 33: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/33.jpg)
Web app: uWSGI
33
• invenio-app can read the header X-Forwarded-For and the request_id
• It will contain the list of all IPs (HAProxy, Nginx, etc…)
![Page 34: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/34.jpg)
Error reporting
34
• invenio-logging• Sentry
![Page 35: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/35.jpg)
Invenio Logging
35
• Log to file• Configure logging level• Configure file rotation• Configure Sentry integration
![Page 36: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/36.jpg)
Sentry
36
![Page 37: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/37.jpg)
37
Ready to GO LIVE?
![Page 38: Deployment and monitoring - CERN](https://reader033.vdocuments.us/reader033/viewer/2022061607/629fd681a4192d0a29386af6/html5/thumbnails/38.jpg)
Questions?