co-funded by the european union -...
TRANSCRIPT
Co-funded by the European Union
Slide! !
!Co-funded by
the European Union
Under the hood of the collaboratoryplatform
Fabrice Gaillard : DevOps @HBP
CodeJam #8 18 Sept 2017
1
Co-funded by the European Union
Slide! !
My Job : build and run infrastuctures
2
Co-funded by the European Union
Slide! !
the HBP infra @CSCS
3
Co-funded by the European Union
Slide! !
Common tools on every VM
All the platform is deployed and configured with Ansible
Some common roles to instrument all VMs
Monitoring
§ Zabbix Agent à Zabbix Server
Metrology
§ Telegraf à kafka à influxDB à (chronograf, graphana)
Logs
§ rsyslog à kafka à elastic search à kibana
4
Co-funded by the European Union
Slide! !
Zabbix : Jupyterhub workers CPU
5
Co-funded by the European Union
Slide! !
Zabbix : Jupyterhub workers memory
6
Co-funded by the European Union
Slide! !
Metrology : storage
7
Co-funded by the European Union
Slide! !
Kibana : search in the logs
8
Co-funded by the European Union
Slide! !
Status page
status.humanbrainproject.org
9
Co-funded by the European Union
Slide! !
« Code factory »GitLab : a lot of features in one platform
• Git repository management
• Issue tracking
• Code review
• Continuous Integration
• Continuous Deployment
• Integrated Docker registry
Still « work in progress » because it’s a very rich tool
10
Co-funded by the European Union
Slide! !
GitLab CI pipelines
11
Co-funded by the European Union
Slide! !
A lot of nice technologies
12
Co-funded by the European Union
Slide! !
Challenge : an increase of complexity
13
Co-funded by the European Union
Slide! !
Complexity : OpenID
Integration of various components with OpenID (oidc)
Most of the time "hard coded" for the big identityprovider (Google, GitHub, LinkedIn) : we need to « hack » the code.
But in many programming languages.
• Collab, jupyterhub : Python
• Forum, Gitlab : Ruby
• OsTicket : Php
• Education website : Java
14
Co-funded by the European Union
Slide! !
Complexity = nice "subtle" bugsSymptom
NMPI python client running in a jupyter notebooks
import requests
requests.get('https://nmpi.hbpneuromorphic.eu/')
The backtrace after the interrupt shows that the kernel is stuck in the ssl lib at:
result = _lib.SSL_do_handshake(self._ssl)
Root cause
The Jupyter notebooks run inside docker containers and the whole platform is hosted on OpenStack.
The OpenStack Software Defined Network use some voodoo encapsulation technologies, and as a consequence, the network interface have some non standard MTU = 1450.
The docker engine seems to not be aware of this and still uses the standard ethernet MTU= 1500.
And for very mysterious reason, the SSL code gets in trouble :
this is described here : https://github.com/moby/moby/issues/22297
15
Co-funded by the European Union
Slide! !
Thank You
www.humanbrainproject.eu @HumanBrainProj Human Brain Project
!Co-funded by
the European Union
16