damn abstraction : development vs operational efficiency
DESCRIPTION
Abstraction, Abstraction, Abstraction.The presentation I wanted to give at GotoCon Copenhagen but some bugs decided otherwise.TRANSCRIPT
Patrick Debois
goto;conference
Copenhagen2011
Wednesday 11 May 2011
http://jedi.be/blog monitoring
http://devopsdays.orgWednesday 11 May 2011
Project TaskWe need performance!
1 million writes /5 seconds
Wednesday 11 May 2011
Wednesday 11 May 2011
“All problems in computer science can be solved by another level of abstraction”
Wednesday 11 May 2011
“All problems in computer science can be solved by another level of indirection”
Butler Lampson
http://zedshaw.com/essays/indirection_is_not_abstraction.html
Wednesday 11 May 2011
We love Ruby & Rails
Wednesday 11 May 2011
Performance Testing
JmeterApache
BenchmarkTsung
Rely much on OS tuningLimitations (tcp sessions, max fd, sockets,...)
Wednesday 11 May 2011
Rails Application
Rails
Rack
Ruby (threading)
OS (threading)
Active Record
NgnixApache, Passenger
Thin Unicorn,Rainbows
Event Machine
Threads, Fibers, PIL
FS, Network tuningWednesday 11 May 2011
Scaling
Web ServerApache/Nginx
MultiInstance
Loadbalancer
Database Pool
Wednesday 11 May 2011
“All problems in computer science can be solved by another level of abstraction”
Wednesday 11 May 2011
“... except for the problem of too many layers of indirection”
David Wheeler
Wednesday 11 May 2011
Magic?
Wednesday 11 May 2011
curl http://npmjs.org/
install.sh | sh
https://github.com/jordansissel/fpm/wiki/ConvertingNodeNPM
But Seriously??
Wednesday 11 May 2011
bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)
RVM
https://github.com/jordansissel/rvm-packaging
Wednesday 11 May 2011
Brew install
curl -LsSf https://github.com/mxcl/homebrew/tarball/master | sudo /usr/bin/tar xvz -C/usr/local --strip 1
Wednesday 11 May 2011
https://github.com/sstephenson/gosh
gosh: Security theater for piped shell scripts
Wednesday 11 May 2011
Select Database?
Wednesday 11 May 2011
Logging
http://logging.apache.org/log4j/http://log4r.rubyforge.org/
https://github.com/indexzero/winston
https://github.com/cloudera/flume
https://github.com/facebook/scribe
http://www.loggly.com/
Wednesday 11 May 2011
Dependency Mgmt
Capistrano
Warbler
BundlerMaven
Cargo Plugin
Fat Jar
Wednesday 11 May 2011
“All problems in computer science can be solved by another level of abstraction”
Wednesday 11 May 2011
“But that usually create another problem”
David Wheeler
http://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html
Wednesday 11 May 2011
“Every abstraction layer adds (a little) overhead
to the CPU
Wednesday 11 May 2011
but also to the poor human who has to read
that code
http://blog.billyang.me/2009/10/dark-side-of-abstraction.html
Dark Side of Abstraction
Wednesday 11 May 2011
VS
+++++++ +
Improvements
Wednesday 11 May 2011
Runtime Abstraction
Wednesday 11 May 2011
Database Abstraction
Wednesday 11 May 2011
Do compilers, Languages, and frames
make dumber programmers?
http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmers
Wednesday 11 May 2011
They don't make good programmers dumber, they
make them more productive;
http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmers
Wednesday 11 May 2011
But they make dumber people be able to program
and think they are great programmer
http://stackoverflow.com/questions/428571/do-smarter-compilers-languages-and-frameworks-make-dumber-programmers
Wednesday 11 May 2011
Sysadmin Land
Wednesday 11 May 2011
Hardware Abstraction(OS/Drivers)
Wednesday 11 May 2011
Disk Abstraction
Wednesday 11 May 2011
Filesystem Abstraction
VFAT
NTFS
Iscsi NAS
ext2
reiserfs
Wednesday 11 May 2011
VM/Storage Abstraction
http://libvirt.org/
Xen
Qemu
LXC
OpenVZ
Usermode Linux
Virtualbox
Vmware
Wednesday 11 May 2011
Disk Images
Wednesday 11 May 2011
Cloud Abstraction
https://github.com/geemus/fog https://github.com/jclouds/jclouds
Wednesday 11 May 2011
Abstraction Fusion
Wednesday 11 May 2011
Config ManagementAbstraction
Wednesday 11 May 2011
A model is an abstraction ... for a purpose
http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm
Wednesday 11 May 2011
http://docs.puppetlabs.com/guides/cloud_pack_getting_started.html
Puppet Cloudpack
https://github.com/opscode/knife-ec2
Knife ec2 Bootstrap
If all you have is a hammer
http://www.amusei.com/images/hammer-in-the-head1.jpg
Wednesday 11 May 2011
Orchestration
http://docs.puppetlabs.com/mcollective/ http://rundeck.org/
Wednesday 11 May 2011
http://www.iinteractive.com/yapc/slides.htmlThank you @jtimberman
Wednesday 11 May 2011
Vagrant
http://vagrantup.com/
https://github.com/jedi4ever/veewee
https://github.com/jedi4ever/sahara
http://vagrantbox.es/
https://github.com/jedi4ever/mccloud
http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/
Wednesday 11 May 2011
Packages Boundaries
gem
npm
jar
deb
rpm
tgzEffing Package Management
Wednesday 11 May 2011
Source/Binary
wget ...tar -xzvf ...
cd ..../configure
makemake install
package install
External Download => Local RepoCompiling => Do it only once
Wednesday 11 May 2011
Idempotent
Wednesday 11 May 2011
“Abstraction is
selective ignorance”
Andrew König
http://en.wikipedia.org/wiki/Andrew_Koenig_(programmer)
Wednesday 11 May 2011
IAAS
Wednesday 11 May 2011
PAAS
Google App Engine
Wednesday 11 May 2011
Cloud Stacks
Wednesday 11 May 2011
Cloud API Standardisation
Wednesday 11 May 2011
Result is suboptimization
http://www.michaelnygard.com/blog/2004/04/too_much_abstraction.html
Wednesday 11 May 2011
Too much abstraction(1)
Do the Simplest Thing that could possibly work
TDD
http://c2.com/cgi/wiki?TooMuchAbstraction
Wednesday 11 May 2011
Too much abstraction(2)
Abstraction should be based on Practical Experience
Premature Optimization
http://c2.com/cgi/wiki?TooMuchAbstraction
Wednesday 11 May 2011
Abstraction is deciding which aspects of a
problem to consider and which ones to ignore
http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm
Wednesday 11 May 2011
Sound a lot likeManagement
(without domain knowledge?)
Wednesday 11 May 2011
Organization Abstractions
Wednesday 11 May 2011
Something’s Fishy
http://weblog.raganwald.com/2007/12/somethings-fishy.html
Wednesday 11 May 2011
Devops
http://mdzlog.alcor.net/2010/06/08/devops-and-cloud/
Wednesday 11 May 2011
Devops
http://rationalautomation.wordpress.com/2010/02/04/dogs-and-cats-living-together/
Wednesday 11 May 2011
Devops
http://www.rajiv.com/blog/2009/03/17/technology-department/
Wednesday 11 May 2011
Generalist vs Specialist
Wednesday 11 May 2011
Consider Specialization First
... And Generalization Second
Wednesday 11 May 2011
http://www.noop.nl/2010/06/t-shaped-people.html
They have a principal skill that describes
the vertical leg of the T
T-shaped People
But they are so empathetic that they can branch out into other skills, such as anthropology, and do them as well.
Wednesday 11 May 2011
http://images.icanhascheezburger.com/completestore/2008/9/11/128656414813278195.jpg
Wednesday 11 May 2011
Wednesday 11 May 2011
NOOPS = Human Abstraction?
http://rlv.zcache.com/i_love_noops_t_shirt-p235186382091096346trlf_400.jpg
Wednesday 11 May 2011
abstraction is selective ignorance
http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm
Wednesday 11 May 2011
http://blog.rightscale.com/2011/04/25/amazon-ec2-outage-summary-and-lessons-learned/
Amazon.com’s real problem isn’t the outage, it’s the communication
http://aws.amazon.com/message/65648/
Wednesday 11 May 2011
The Chaos Monkey’s job is to randomly kill instances and services within our architecture
If we aren’t constantly tes0ng our ability to succeed despite failure, then it isn’t likely to work when it ma;ers most – in the event of an unexpected outage.
http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-monkey.html
Wednesday 11 May 2011
Chaos Monkey
https://github.com/simonmunro/ChaosMonkey
https://github.com/mattbasta/chaos-chimp
__ w c(..)o ( \__(-) __) /\ ( /(_)___) w /| | \ m m
Wednesday 11 May 2011
All non-trivial abstractions, to some
degree, are leaky
http://www.joelonsoftware.com/articles/LeakyAbstractions.html
Wednesday 11 May 2011
When you drive a car, thinking about howthe engine works is
a distraction
http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm
Wednesday 11 May 2011
When you repair the car, thinking about
how the engine works is essential.
http://www.cs.princeton.edu/courses/archive/spr98/cs333/lectures/19/sld002.htm
Wednesday 11 May 2011
Questions?
Wednesday 11 May 2011
Programming is a human activity;
forget that and all is lost
Meta-advicehttp://www.cs.princeton.edu/courses/archive/spr99/cs333/pdf/21.pdf
Wednesday 11 May 2011