python & devops: your own heroku
DESCRIPTION
Using python and uWSGI for agile deployment, effortless :)TRANSCRIPT
![Page 2: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/2.jpg)
Chakib Benziane @sp4ke
● Freelance Full Stack Developer / DevOps● CTO & CoFounder at Jib.li● Python Enthusiast
![Page 3: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/3.jpg)
Summary
1. Environment & Stack
2. Agile deployment with uWSGI
![Page 4: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/4.jpg)
1. Environment & Stack
Our needs:
● Web Application
● Deep linking with social networks
● Agile development
● Community and available packages
![Page 5: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/5.jpg)
1. Environment & Stack
Our needs:
● Web Application
● Deep linking with social networks
● Agile development
● Community and available packages
= Django + MongoDB + Github
![Page 6: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/6.jpg)
1. Environment & Stack
Our needs:
● Web Application
● Deep linking with social networks
● Agile development
● Community and available packages
= Django + MongoDB + Github
gevent-socketio, zmq, celery, AWS boto ...
![Page 7: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/7.jpg)
1. Environment & Stack
Local environment : The essentials
● Virtualenv
● PIP
![Page 8: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/8.jpg)
1. Environment & Stack
Local environment : The essentials
● Virtualenv
● PIP
● But they have RVM !
![Page 9: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/9.jpg)
1. Environment & Stack
Local environment : The essentials
● Virtualenv
● PIP
● But they have RVM !
● Pythonbrew: utahta/pythonbrew.git
![Page 10: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/10.jpg)
1. Environment & Stack
● Pythonbrew: utahta/pythonbrew.git
○ Compile system independent pythons
$ pythonbrew install 2.7.3$ pythonbrew use 2.7.3$ pythonbrew list && which python# pythonbrew pythons Python-2.7.3 (*)/home/spike/.pythonbrew/pythons/Python-2.7.3/bin/python
![Page 11: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/11.jpg)
1. Environment & Stack
● Pythonbrew: utahta/pythonbrew.git
○ Easy management of virtualenvs
$ pythonbrew venv create jibli$ pythonbrew venv use jibli && which python && which pip
# Using `jibli` environment# To leave an environment, simply run `deactivate`/home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/python/home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/pip
![Page 12: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/12.jpg)
1. Environment & Stack
Environment bootstrap:
git clone jibli/project && cd projectpythonbrew create venv jibli && pythonbrew activate jiblipip install -r dependencies.txt
![Page 13: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/13.jpg)
1. Environment & Stack
Environment bootstrap:
git clone jibli/project && cd projectpythonbrew create venv jibli && pythonbrew activate jiblipip install -r dependencies.txt --download-cache=CACHE
![Page 14: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/14.jpg)
1. Environment & Stack
pip & dependencies.txt:
package-foo don't package-bar==4.2 do
![Page 15: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/15.jpg)
1. Environment & Stack
pip & dependencies.txt:
package-foo package-bar==4.2
git+https://github.com/user/repo
![Page 16: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/16.jpg)
1. Environment & Stack
pip & dependencies.txt:
package-foo package-bar==4.2
git+https://github.com/user/repogit+https://github.com/user/repo#egg=mon-package
![Page 17: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/17.jpg)
1. Environment & Stack
pip & dependencies.txt:
package-foo package-bar==4.2
git+https://github.com/user/repogit+https://github.com/user/repo#egg=mon-packagegit+https://github.com/user/repo@branch
![Page 18: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/18.jpg)
1. Environment & Stack
pip & dependencies.txt:
● Quick dependencies updatepip freeze > dependencies.txt
package-foo package-bar==4.2
git+https://github.com/user/repogit+https://github.com/user/repo#egg=mon-packagegit+https://github.com/user/repo@branch
![Page 19: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/19.jpg)
1. Environment & Stack
MongoDB
● NoSQL, Schemaless, Document Oriented
● BSON data format
● Advantage: Python Dict -> JSON
● Good Python API pymongo
![Page 20: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/20.jpg)
1. Environment & Stack
MongoDB
● MongoDB Javascript Console$ mongo jibliMongoDB shell version: 2.0.6connecting to: jibli> db.users.find( {'profil.age': 10} );
![Page 21: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/21.jpg)
1. Environment & Stack
MongoDB
● MongoDB Javascript Console$ mongo jibliMongoDB shell version: 2.0.6connecting to: jibli> db.users.find( {'profil.age': 10} );
● PyMongo equivalentu = pymongo.Connection(host='localhost', port=27017)['jibli']['users']
u.find( {'profil.age': 10} )
![Page 22: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/22.jpg)
2. Agile deployment with uWSGI
Local development
● Git branch feature● Unit test● Implement● Test on local server (./manage.py runserver)● Commit and merge on master branch
![Page 23: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/23.jpg)
2. Agile deployment with uWSGI
Agile deployment
● Many features require a production like environment:○ OAuth Authentication and Social Networks○ Async tasks, Celery (notifications, crons ... )○ Push Notifications○ Hard to clone a production environment in local
![Page 24: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/24.jpg)
2. Agile deployment with uWSGI
● Development Scenario○ Start a new feature
$ git checkout -b feature
![Page 25: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/25.jpg)
2. Agile deployment with uWSGI
● Development Scenario○ Start a new feature
$ git checkout -b feature
○ Implement and push on dev server
$ fab push
![Page 26: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/26.jpg)
2. Agile deployment with uWSGI
● Development Scenario○ Start a new feature
$ git checkout -b feature
○ Implement and push on dev server
$ fab push
○ My branch is UP on feature.dev.com
![Page 27: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/27.jpg)
2. Agile deployment with uWSGI
● Development Scenario○ Start a new feature
$ git checkout -b feature
○ Implement and push on dev server
$ fab push
○ My branch is UP on feature.dev.com○ Remote access to the deployed app from local shell with a
production environment (ie. restart, upgrade, ipython, mongo shell ...)
![Page 28: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/28.jpg)
2. Agile deployment with uWSGI
● Development Scenario○ Start a new feature
$ git checkout -b feature
○ Implement and push on dev server
$ fab push
○ My branch is UP on feature.dev.com○ Remote access to the deployed app from local shell with a
production environment (ie. restart, upgrade, ipython, mongo shell ...)
○ Once satisfied merge on master and push on prod
![Page 29: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/29.jpg)
2. Agile deployment with uWSGI
Solution
● Nginx● Github● Fabric
+
● uWSGI
![Page 30: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/30.jpg)
2. Agile deployment with uWSGI
Solution
● Nginx● Github● Fabric
+
● uWSGI
= your heroku like solution
![Page 31: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/31.jpg)
2. Agile deployment with uWSGI
uWSGI
● Create developement stacks
● Host application clusters
![Page 32: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/32.jpg)
2. Agile deployment with uWSGI
uWSGI
● how ?
DjangoAppuWSGI
WSGI Interfacealso: FastCGI,CGI, PHP, Rack, ...
Web Server(nginx)
![Page 33: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/33.jpg)
2. Agile deployment with uWSGI
uWSGI
● how ?
DjangoAppuWSGI
DjangoApp
DjangoApp
Load balancing
Web Server(nginx)
![Page 34: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/34.jpg)
2. Agile deployment with uWSGI
uWSGI
● how ?
DjangoApp2uWSGI
DjangoApp1
FlaskApp3
RouterProxyapp1.dev.com
app2.dev.comapp3.dev.com
Web Server(nginx)
![Page 35: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/35.jpg)
2. Agile deployment with uWSGI
uWSGI
● how ?
DjangoApp2
uWSGI
DjangoApp1
FlaskApp3
EmperorMassive app deployment
app1.dev.comapp2.dev.comapp3.dev.com*.dev.com App n
Web Server(nginx)
![Page 36: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/36.jpg)
2. Agile deployment with uWSGI
uWSGI Emperor● Event based dynamic handling of applications
(Vassals)
Default:
● Scan for config files in directories (.ini, .xml, .yml, .json ... )
● dir:// & glob:// for conf files monitoring● Much more plugins available (mongodb, ampq, ldap
... )
![Page 37: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/37.jpg)
2. Agile deployment with uWSGI
glob:// plugin
uwsgi --emperor /opt/apps/*/*.ini
![Page 38: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/38.jpg)
2. Agile deployment with uWSGI
glob:// plugin
uwsgi --emperor "/opt/apps/*/*.ini"
One does not simply use glob patterns !
![Page 39: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/39.jpg)
2. Agile deployment with uWSGI
glob:// plugin
uwsgi --emperor "/opt/apps/*/*.ini"
Example:○ New file "/opt/apps/appn/uwsgi.ini"
● Spawn vassal○ File modified
● Restart vassal○ File removed
● Kill vassal○ Emperor dies
● All vassals die with him
![Page 40: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/40.jpg)
2. Agile deployment with uWSGI
● Create a conf file for each deployed app ?
/opt/apps/app1/uwsgi.ini/opt/apps/app2/uwsgi.ini/opt/apps/appn/uwsgi.ini
![Page 41: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/41.jpg)
2. Agile deployment with uWSGI
● Create a conf file for each deployed app ?
/opt/apps/app1/uwsgi.ini/opt/apps/app2/uwsgi.ini/opt/apps/appn/uwsgi.ini
● ln -s ○ Use template conf files/opt/apps/template
ln -s /opt/apps/template /opt/apps/app1/app1.ini
![Page 42: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/42.jpg)
2. Agile deployment with uWSGI
Template conf file (Django App)
[uwsgi]djangoproject = %d/app/home = %d/virtpythonpath = %d/env = DJANGO_SETTINGS_MODULE=app.settingschdir = %(djangoproject)module = uwsgi_appsocket = 127.0.0.1:0master = trueprocesses = 1idle = 300subscribe-to = 127.0.0.1:9999:%n.dev.comlogto = %d/log/uwsgi.log
![Page 43: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/43.jpg)
2. Agile deployment with uWSGI
Template conf file[uwsgi]
djangoproject = %d/app/
● Use variables like here djangoproject
● Magic variables :
○ %d - Absolute path to configuration file
○ %n - Name of configuration file without
extension
![Page 44: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/44.jpg)
2. Agile deployment with uWSGI
Template conf file[uwsgi]
djangoproject = %d/app/home = %d/virtpythonpath = %d/env = DJANGO_SETTINGS_MODULE=app.settings
● Define your app's virtualenv
![Page 45: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/45.jpg)
2. Agile deployment with uWSGI
Template conf file[uwsgi]
djangoproject = %d/app/home = %d/virtpythonpath = %d/env = DJANGO_SETTINGS_MODULE=app.settings
● Define your app's virtualenv
● Python search paths (You can repeat this one)
![Page 46: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/46.jpg)
2. Agile deployment with uWSGI
Template conf file[uwsgi]
djangoproject = %d/app/home = %d/virtpythonpath = %d/env = DJANGO_SETTINGS_MODULE=app.settings
● Define your app's virtualenv
● Python search paths (You can repeat this one)
● Custom environment variables
![Page 47: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/47.jpg)
2. Agile deployment with uWSGI
Template conf file (Django App)
[uwsgi]
chdir = %(djangoproject)module = uwsgi_app
● Which module to run when starting application ○ django.core.handlers.wsgi:WSGIHandler()
● Best spot to run your custom scripts and setup environment before launching application ○ ie. compile static, zMQ sockets, syncdb ...
![Page 48: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/48.jpg)
2. Agile deployment with uWSGI
Up until now we can:○ Git push origin feature○ Clone feature in remote /opt/apps/feature○ Prepare dirs structure○ Create venv & install dependencies○ Symllink to the uWSGI template file○ uWSGI emperor launches feature app
We still need to:
○ access our feature using subdomains● feature.dev.com
![Page 49: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/49.jpg)
2. Agile deployment with uWSGI
FastRouter
○ Proxy/Load Balancing/Router○ Speaks uWSGI protocol○ Unlimited setup possibilities○ Key/Value store
![Page 50: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/50.jpg)
2. Agile deployment with uWSGI
FastRouter
○ Proxy/Load Balancing/Router○ Speaks uWSGI protocol○ Unlimited setup possibilities○ Key/Value store
Example:
uwsgi --fastrouter /tmp/fastrouter.socket \ --fastrouter-subscription-server 127.0.0.1:9999
![Page 51: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/51.jpg)
2. Agile deployment with uWSGI
Always use unix sockets instead of localhost tcp
![Page 52: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/52.jpg)
2. Agile deployment with uWSGI
Nginx server { listen 80; server_name dev.com *.dev.com; location / { include /etc/nginx/uwsgi_params; uwsgi_param UWSGI_FASTROUTER_KEY $host; uwsgi_pass unix: /tmp/fastrouter.socket; }}
![Page 53: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/53.jpg)
2. Agile deployment with uWSGI
Template conf file (Django App)
[uwsgi]
...socket = 127.0.0.1:0
subscribe-to = 127.0.0.1:9999:%n.dev.com
![Page 54: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/54.jpg)
2. Agile deployment with uWSGI
Template conf file (Django App)
[uwsgi]
...socket = 127.0.0.1:0
subscribe-to = 127.0.0.1:9999:%n.dev.com
/opt/apps/feature1/feature1.ini
![Page 55: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/55.jpg)
2. Agile deployment with uWSGI
nginx
uWSGI emperor
Request on feature1.dev.com
Fastrouteur + Subscription server{ 'feature1.dev.com' : 127.0.0.1:n1, 'feautre2.dev.com' : 127.0.0.1:n2, ... }
feature2feature1
![Page 56: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/56.jpg)
2. Agile deployment with uWSGI
├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini│ │ ├─ /usr/local/bin/uwsgi │ │ └─ /usr/local/bin/uwsgi│ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini
#fastrouter#master
HTOP Deploying feature1
![Page 57: Python & Devops: Your own heroku](https://reader033.vdocuments.us/reader033/viewer/2022052410/55526f96b4c905d41d8b53cb/html5/thumbnails/57.jpg)
2. Agile deployment with uWSGI
├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini│ │ ├─ /usr/local/bin/uwsgi │ │ └─ /usr/local/bin/uwsgi│ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini│ │ └─uwsgi --ini /opt/apps/feature2/frature2.ini
#fastrouter#master
HTOP Deploying feature2