panorama des technologies web et devops · 2019-03-22 · criteo prestashop. etapes de devops (vii)...
TRANSCRIPT
![Page 1: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/1.jpg)
Panorama des Technologies Web et Devops
Didier DONSEZUniversité Grenoble AlpesPolytech’Grenoble & LIG
IUT de Valence / L3Pro SIL CASIR / Mars 2019
![Page 2: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/2.jpg)
Qui suis-je ?
Didier DONSEZUniversité Grenoble AlpesPolytech’Grenoble & LIG
![Page 3: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/3.jpg)
DevOps
● Dev(elopment) – Op(eration)s● Objectifs
– TTM– MVP (Lean Startup)– Rapid Production Deployment
● Moyens– Micro-services Architectures– Continuous Integration (CI)– Continuous Delivery (CD)– Application Performance Monitoring (APM)
![Page 4: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/4.jpg)
Le scénario DevOps
![Page 5: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/5.jpg)
Etapes de DevOps (o)
Express
Daphnée
Frontend developerBackend developerFull stack developer
![Page 6: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/6.jpg)
Etapes de DevOps (i)
Express
HTTP
Daphnée
![Page 7: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/7.jpg)
Etapes de DevOps (ii)
HAProxy Express
HTTPHTTPS
Daphnée Olivier
Backup for Crash Recovery
![Page 8: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/8.jpg)
Etapes de DevOps (iii)
Nginx
Spring
Grafana InfluxDB
metrics
HTTPHTTPSTelegraf
Agent
Momo
![Page 9: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/9.jpg)
Etapes de DevOps (iv)
Nginx
Nginx
Spring
Spring
MySQL
Grafana InfluxDB
HTTPHTTPS
metrics
TelegrafAgent
Rémi
![Page 10: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/10.jpg)
Etapes de DevOps (v)
Nginx
Nginx
Spring
Spring
MySQL
MySQL
Grafana InfluxDB
HTTPHTTPS
metrics
Smtp
Replication
Rémi
Wordpress
Wordpress
Smtp
![Page 11: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/11.jpg)
Etapes de DevOps (v)
Nginx
Nginx
Spring
Spring
MySQL
MySQL
Grafana InfluxDB
HTTPHTTPS
metrics
Stripe
Replication
Serge
![Page 12: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/12.jpg)
Etapes de DevOps (vi) SLM
HAProxy
HAProxy
Spring
Spring
MySQL
MySQL
Grafana InfluxDB
HTTPHTTPS
metrics
Stripe
Replication
Sophie
Criteo
Prestashop
![Page 13: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/13.jpg)
Etapes de DevOps (vii)
HAProxy
HAProxy
Spring
Spring
MySQL
MySQL
Grafana InfluxDB
HTTPHTTPS
metrics
Criteo
Replication
Pierre
Trello
Prestashop
![Page 14: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/14.jpg)
En conlusion…
• Des fonctonnalités simples : – Servir– Collecter,– Transporter,– Traiter,– Stocker… des données des diférentes sources
• … mais une architecture logicielle complexe !
![Page 15: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/15.jpg)
La complexité… au service des propriétés non fonctonnelles
• Sécurité• Haute disponibilité• Performances• Passage à l’échelle• Agilité
![Page 16: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/16.jpg)
Sécurité
• Objectfs : – Aucune corrupton de données– Aucun vol de données– Aucune ataque par déni de service
• Techniques mises en œuvre :– Chifrage des données – Protocoles de communicaton sécurisés – Pare-feux – Techniques logicielles contre les ataques par déni de
service– Audit/Test de sécurité
![Page 17: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/17.jpg)
Exemple de rançon pour une base Mongo non protégée
● $ db.PLEASE_READ_ME.find()● { "_id" : ObjectId("58a7287db7dc324adb249fdf"), "info" :
"Don't panic. Your DB is in safety and backed up (check logs). To restore send 0.1 BTC and email with your server ip or domain name. Each 48 hours we erase all the data.", "amount" : "0.1 BTC", "data_we_have" : { "local" : [ "startup_log" ], "first_database" : [ "users", "preferences" ], "MyAppXXX" : [ "emails" ] }, "Bitcoin Address" : "1NSz9TRBGKHKFdjdjH2Gme3LwDi5", "email" : "[email protected]" }
![Page 18: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/18.jpg)
Haute-disponibilité
• Objectf : – Aucune interrupton de service
• Techniques mises en œuvre :– Monitoring – Détecton de fautes – Protocoles de réplicaton – Mécanismes de redondance – Mécanismes de « fail-over »– Circuit Breakers– Equilibreur de charge– Testers de résilience
![Page 19: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/19.jpg)
Performances
• Objectf : – Réducton du coût d’infrastructure IaaS– Respect des SLA
• Techniques mises en œuvre :– Monitoring– Performance Debugging
Detecton des contentons sur les verroux N Copy Placement multicoeurs (Thread pinning)
– Equilibreur de charge– SLA Monitoring
![Page 20: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/20.jpg)
Passage à l’Echelle
• Objectf : – Supporter des variatons dans la charge– en minimisant les coûts d’infrastructure ou d’énergie
• Techniques mises en œuvre :– Monitoring– Sharding– N Copy – Elastcité dynamique– Cloud hybride (cloud burstng)
![Page 21: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/21.jpg)
Agilité
• Objectf – Time To Market– Philosophie « Minimal Viable Product » du Lean Startup
• Techniques mises en œuvre :– DevOps
Service immédiatement en producton Blue – Green Deployment
● Rolling Update● Fast rollback
● A/B Testng ou Split Testng Micro-services Serverless Architecture
![Page 22: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/22.jpg)
Les grands composants● Database Systems
– SQL does not fit all !– NoSQL is not unique !
● Backends– Servent des ressources REST
● Frontends– Implémente une SPA
● Tooling– IDE & Forge
![Page 23: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/23.jpg)
Data Models & Database SystemsOne does not fit all !
SQL● Oracle, MySQL/MariaDB,
Postgres, HSQL ...
NoSQL (Not 1 NF)● File Systems
– HDFS
● Table– Hbase (Big Table)
● Directories (LDAP)● Key-Value Stores
– Cassandra, Redis, Memcached, ...
● Document-oriented DB– MongoDB, CouchDB, ...
● ….
● Graph-oriented DB– Neo4J, ...
● Time-Series DB– OpenTSDB, InfluxDB, …
● Text Oriented– Lucene, OpenNLP, ElasticSearch– Geolocation
● GIS, Geo extensopns in MongoDB, Postgres, MySQL, ...
– Streams● Kafka, Flume
Performance● In-memory DB– MySQL Cluster, Redis, ...
![Page 24: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/24.jpg)
Database SystemsMultiple Data Models
![Page 25: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/25.jpg)
Gestionnaires de fonctions● Exemple
– CMS– Micro-Blogging– Ticketing / Issue Tracking– Shopping– ERP– Enterprise Social Network (ESN)– Newletter Campaign– …
● Notes– Utilise le propre systême de persistence– Expose une API (REST)– Disponible via un container Docker (Démo Wordpress)
![Page 26: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/26.jpg)
CAP Theorem (Brewer)
• Un SD ne peut garantr qu’au max. 2 propriétés
![Page 27: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/27.jpg)
RESTFul Services● Principles des services REST
– Hiérarchie d’URLs● /product● /product/12345● /user● /user/didier
– CRUD appliqué à HTTP● C(reate) → PUT● R(etrieve) → GET● U(pdate) → POST (or PUT for full update)● D(elete) → DELETE
● Description– OpenAPI (Swagger)
● Voir http://editor.swagger.io
![Page 28: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/28.jpg)
Frontends & Backends● Tendance architecture
– Frontend SPA (Single Page Application)● Angular 2+, React, VueJS …
– Backend RESTFul● Remark : services Pub-Sub (websocket, PubNub, Firebase …)
● Misc.– Full Stacks (Backend Node.js)
● MEAN, Sails, Meteor …– Voir le comparatif live http://todomvc.com/
– API Gateway– Application
● Electron, Cordova
![Page 29: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/29.jpg)
Thèmes Frontend Web● Thème Dashboard
– Organisation type tableau de bord– Regroupe des libs js dont Bootstrap– Boilerplate pour des canevas Full stack.
● Avantages– Permet de se passer d’un Web Designer (dans un premier
temps)– Gain de temps de dev.– Ergonomie connue.
● 20+ assez connus– Gentelella (11400 stars sur GitHub), SB Admin2, …
● https://colorlib.com/wp/free-bootstrap-admin-dashboard-templates/
![Page 30: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/30.jpg)
Thèmes Frontend Web
● Ergonomie Web– Bootstrap– Material Design– Ionic– ...
● Avantages– Web Responsive– Mobile interaction– Mobile sensor integration (camera, NFC, GPS ...)– …
![Page 31: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/31.jpg)
Dashboard Gentelella
![Page 34: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/34.jpg)
APMApplication Performance Monitoring
● Motivation– Supervision de l’infrastructure et des
applications– Recherche des « goulots d’étranglement »
● Techniques– Identification des événements– Centralisation des journaux d’événements– Visualisation (temps-réel) des journaux
![Page 35: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/35.jpg)
APM consoles
● Connexion temps réel à des TSDBs– Customisable, Snippets intégrables
Demo Datadoghttps://www.youtube.com/watch?v=mpuVItJSFMc
![Page 36: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/36.jpg)
Architecture Cloud● On-Demand Computing
– Scalabilité dynamique– Pay-per-use, Plan, Auction (AWS spot instance)
● Public, Privé, Hybride● Providers : FaaS, IaaS, PaaS, CaaS, SaaS● Gestionnaire de clusters privés (OpenStack, Mesos)● Gestionnaire de clusters containers (CaaS, Swarm,
Kubernetes)● Virtualisation vs Containerisation
![Page 37: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/37.jpg)
Virtual Machines and Containers
● Voir cours Docker
![Page 38: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/38.jpg)
Docker Modus operandi
![Page 39: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/39.jpg)
Tooling DevOps● Jenkins – Gitlab● Ansible – Puppet – Chef -Cloudify● Openshift (Redhat)● Tools : gulp, npm, grunt, bower, yeoman, webpack,
browserify, maven, graddle, ant, …● Netflix OSS (Simian Army)● IDE : Eclipse, IntelliJ IDEA, Netbeans, Atoms, Chrome
Devtools● Q&A : SonarQube
![Page 40: Panorama des Technologies Web et Devops · 2019-03-22 · Criteo Prestashop. Etapes de DevOps (vii) HAProxy HAProxy Spring Spring MySQL MySQL Grafana InfluxDB HTTP HTTPS ... – SLA](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5d6fa09d47022543be3e4/html5/thumbnails/40.jpg)
JHipster (Generator)
● Domain– Microservice(s) + Single Web Page Application
● Model : JDL (~UML with pagination pragmas)– JDL Studio
● Generator (and sub-generators with yeoman)– Frontend : Angular X, React, Vue, Ionic – Backend : Spring
● Monolithe ou Micro-Services (API Gateway, Swagger)● Authentication, Rate Limit, ...● Databases : SQLs, Cassandra, MongoDB, Elastic, Kafka
– Devops● Docker containers, CD/CI, Cloud deployment (Heroku,,GCE, AWS, Azure ….)● Load injection, APM (Metrics, ELK)● QA (SonarCube) ...