KATELLO ON TORQUEBOXJava Loves Ruby

ABOUT MELukáš Zapletal

ABOUT ME@lzap

ABOUT ME@lzap_CZ @lzap80

THEME SELECTIONBlah blah blah:

- - - - - Sky Beige Simple Serif Night Default

KATELLO

TORQUEBOX

RUBY

JRUBY

JAVA

JAVA

CATS-FREE TALK

WHAT IS KATELLO

KATELLO IS

A OPEN-SOURCE

CONTENT AND SYSTEM

MANAGEMENT STACK

FOR DATACENTERS

AND CLOUD

IF YOU TAKE ...

AND CLOUD

FOR DATACENTERS

AND CLOUD

FOR DATACENTERS

AND CLOUD

ENOUGH FUN!What the cloud is?

NISTNational Institute of Standards and Technology

NIST DEFINITIONthe NIST definition of cloud computing

NIST DEFINITION

NIST DEFINITION

NIST DEFINITION

NIST DEFINITION

NIST DEFINITION

NIST DEFINITION

NIST DEFINITION

CLOUD DEFINITIONCloud computing is a model for enabling ubiquitous,

convenient, on-demand network access to a shared pool ofconfigurable computing resources.

CLOUD DEFINITIONIt can be rapidly provisioned and released with minimal

management effort or service provider interaction.

ESSENTIAL CHARACTERISTICSon-demand service

ESSENTIAL CHARACTERISTICSbroad network access

ESSENTIAL CHARACTERISTICSresource pooling

ESSENTIAL CHARACTERISTICSrapid elasticity

ESSENTIAL CHARACTERISTICSmeasured service

SERVICE MODELSSaaSPaaSIaas

DEPLOYMENT MODELSprivate cloudcommunity cloudhybrid cloudpublic cloud

WHAT YOU CAN DO WITH KATELLO?

RED HAT SUBSCRIPTIONwhat is it and how it works

SUBSCRIPTION MANAGEMENTimport Red Hat subscriptions from Portalcreate your own products and subscriptionsregister machines and consume themsee some statistics and graphs

CONTENT MANAGEMENTsync RPM content from CDNsync RPM content from other repositoriessync puppet content from Puppet Forgeseparate content into environments and content viewspromote contentconsume content using yum or puppetremote install/upgrade content

THERE IS FOREMAN

PROVISIONINGregister installation treesprepare provisioning templatesprovision bare-metal/virtual systemsmaintain registry of all systems

CONFIGURATION MANAGEMENTimport Puppet classes into Foreman databaseassign classes to hosts (existing or provisioned)assign parameters to classescollect info from Facter and Puppetcreate statistics and graphs

KATELLO UI

KATELLO UI

KATELLO UI

RED HAT PRODUCTSSubscription Asset Manager (SAM)CloudForms System Engine

WHAT IS TORQUEBOX AND JRUBY

JRUBYRuby 1.8/1.9 on JVMmature and stable projectJIT and AOTbidirectionalis in Fedora

TORQUEBOXapplication platform for Ruby on Rails, Sinatra...runs atop of JBoss ASoffers services like messaging, scheduling, cachingallows use of clustering, load-ballancing and HAuses standards where possible

WHY SHOULD WE CARE?why to port to JVM

WHY SHOULD WE CARE?memory :-)

WHY SHOULD WE CARE?performance (skipping for this talk)

WHY SHOULD WE CARE?

MEMORY IS THE ISSUEbefore that we need to cover threads

MRI RUBY 1.8green threads

MRI RUBY 1.9native threads with GIL

GLOBAL INTERPRETER LOCKany time one thread is running _Ruby_ codeno other thread can be running _Ruby_ code

GLOBAL INTERPRETER LOCK

GLOBAL INTERPRETER LOCKsignificant barrier to parallelismdoes _not_ limit I/O by the designbut many native rubygems also limits I/O

STATE OF THREADING IN MRI RUBYnot the best

FORKING SERVERS IN MRI RUBYthreads are not the only options for web concurencyprocess forking can do the thing tooLinux is good in forkingunfortunately MRI Ruby can't leverage COW memory

FORKING SERVERS IN MRI RUBYRuby Enterprise Edition solves this for 1.8Ruby 1.9 has many REE optimalizations (but not COW)Ruby 2.0 will finally deliver COW-friendly forking(bitmaps)

STATE OF FORKING IN MRI RUBYnot the best

DEPLOYMENT OPTIONS WITH RUBYThe Ruby community has always insisted that performance

is not an issue while constantly searching for higherperformance web servers and application stacks. -- Greg

Weber

DEPLOYMENT OPTIONS WITH RUBYevented programming (reactor pattern) brings some

parallelism

DEPLOYMENT OPTIONS WITH RUBYforking - phusion passenger, unicornevented - thin, goliath, vert.xthreaded - mongrel, torquebox

DEPLOYMENT OPTIONS WITH RUBYcombination of forking + eventedcombination of threaded + eventedcombination of threaded + forking

DEPLOYMENT OPTIONS WITH RUBYthe issue with evented servers (thin) is granularity

DEPLOYMENT OPTIONS WITH RUBYcontroller - sql* - render - response*controller - sql* - render - response*controller - sql* - render - response*

DEPLOYMENT OPTIONS WITH RUBYto unleash power of evented processing, you need to rewrite

your application (fibers, goliath, node.js, vert.x, asyncsinatra)

DEPLOYMENT OPTIONS WITH RUBYthere are not many options for threading setups

SO WHEN TO CONSIDER JRUBY?you have an app that is not build around evented patternyour app takes decent amount of memoryyour app also contains lots of I/O operations (SQL,messaging, REST calls)you want to scale up

WARNINGI did not cover Rubinius or REE which partially solves some

of these issues

BY THE WAYthe following languages have concurrency built in the

runtimeerlanghaskellgoogle go

AND WHAT'S JAVA

BEFORE YOU START

TRY WITH JRUBY FIRSTinstead of torquebox

SLOW STARTjruby start a little bit slower

OPTIMIZE JRUBY START# JAVA_OPTS="-client -Djruby.compile.mode=OFF" \bundle exec rails server

OPTIMIZE JRUBY START# JRUBY_OPTS="--1.9 -J-XX:+CMSClassUnloadingEnabled \-J-XX:+UseConcMarkSweepGC \-J-XX:MaxPermSize=256m -J-Xmx1800m" \bundle exec rails server

OPTIMIZE JRUBY START# jruby --ng-server &# bundle exec rails server

KATELLO START IN DEV$ time bundle exec rake environment

real 0m19.876suser 0m18.244ssys 0m0.764s

KATELLO START IN PROD$ time rake environment

real 0m13.322suser 0m9.979ssys 0m2.817s

RUBYGEMS ARE SLOWmultiple directories approachruby needs to walk the treemany stat/open calls with ENOENTbundler adds more dirsrvm/rbenv adds even more dirs

HOW RUBY HANDLES REQUIRE# strace rake environment 2>&1 | grep ENOENT...open("x/ldap_fluff-0.1.3/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/net-ldap-0.3.1/lib/singleton.rb", O_RDONLY) = -1 ENOENT open("x/jshintrb-0.2.1/lib/singleton.rb", O_RDONLY) = -1 ENOENT open("x/js-routes-0.6.2/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/jammit-0.6.5/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/yui-compressor-0.9.6/lib/singleton.rb", O_RDONLY) = -1 ENOENTopen("x/i18n_data-0.3.3/lib/singleton.rb", O_RDONLY) = -1 ENOENT...

HOW RUBY HANDLES REQUIREO(n^2)

HOW RUBY HANDLES REQUIREoptimized in ruby 2.0

HOW RUBY 2.0 HANDLES REQUIREO(n^2) - k

HOW RUBY HANDLES REQUIREenough theory!

RUBYGEMS IN KATELLO# bundle install | wc -l120

KATELLO STAT/OPEN MISSES IN PROD# strace bundle exec rake environment 2>&1 | grep ENOENT | wc -l4023

KATELLO STAT/OPEN MISSES IN DEV# strace bundle exec rake environment 2>&1 | grep ENOENT | wc -l172342

KATELLO STAT/OPEN MISSES IN DEV

RUBYGEMS ARE SLOWand it's not getting better

RUBYGEMS ARE SLOWavoid bundleravoid rvm/rbenvuse bundler_ext

PORTING ISSUES

BINARY FILESwriting to a binary file needs b-flag

BINARY FILESFile.open("thefile.bin", 'wb') do |f| f.write(stuff)end

ACTIVERECORDinstall proper gems

ACTIVERECORDif defined? JRUBY_VERSION gem 'activerecord-jdbc-adapter' gem 'jdbc-postgres', gem 'activerecord-jdbcpostgresql-adapter',else gem 'pg'end

ACTIVERECORDvarious versions (rails 3.0 vs new adapter)

ACTIVERECORDERROR undefined method collect' for "created_at DESC":String (NoMethodError).../activerecord-jdbc-adapter-1.2.6/lib/arjdbc/postgresql/adapter.rb:620:in distinct'...

OTHER ISSUESimproper rails namespace:-)

RUBY AND SYSTEMTAP

WHAT IS SYSTEMTAPfree software infrastructure to simplify the gathering of

information about the running Linux system

WHY SYSTEMTAP IS USEFULno need to modify your appno need to restart it

WHY SYSTEMTAP IS USEFULsteep learning curveC-like syntax

WHY SYSTEMTAP IS USEFULvery low-levelsupports high-level (JVM, Python, Ruby)

WHY SYSTEMTAP IS USEFULpart of RHEL and Fedorakernels are systemtap readyRuby extension part of RHEL 6.2 (RHSA-2011-1581)

WHY SYSTEMTAP IS USEFULproject documentation and wikiRHEL6 SystemTap Beginners Guide

INSTALL SYSTEMTAP# yum -y install \systemtap \systemtap-runtime \kernel-debuginfo-uname -r \kernel-debuginfo-common-uname -i -uname -r \kernel-devel-uname -r

UC1: HUNTING FILE CHANGE# touch /test

UC1: HUNTING FILE CHANGE# ls -i /test274

UC1: HUNTING FILE CHANGE# ll /dev/md-0brw-rw----. 1 root disk 253, 0 Apr 17 10:23 /dev/dm-0

UC1: HUNTING FILE CHANGE# cat filechange.stpglobal ATTR_MODE = 1probe kernel.function("setattr_copy")!, kernel.function("generic_setattr")!, kernel.function("inode_setattr") { dev_nr = $inode->i_sb->s_dev inode_nr = $inode->i_ino

if (dev_nr == MKDEV($1,$2) # major/minor device && inode_nr == $3 && $attr->ia_valid & ATTR_MODE) printf ("%d %s(%d) %s 0x%x/%u %o %d\n", gettimeofday_us(), execname(), pid(), probefunc(), dev_nr, inode_nr, $attr->ia_mode, uid())}

UC1: HUNTING FILE CHANGE# stap -v filechange.stp 253 0 274 &# chmod 600 /test1334676922011223 chmod(6157) generic_setattr 0xfd00000/274 100600 0

UC2: DOWN THE RUBY STACK# cat factorial.rbdef factorial n f = 1; for i in 1..n; f *= i; end; fendputs factorial 42

UC2: DOWN THE RUBY STACK# cat calls.stp probe ruby.function.entry{ printf("%s => %s.%s in %s:%d\n", thread_indent(1), classname, methodname, file, line);}probe ruby.function.return{ printf("%s <= %s.%s in %s:%d\n", thread_indent(-1), classname, methodname, file, line);}

UC2: DOWN THE RUBY STACK# stap calls.stp -c "ruby factorial.rb"1405006117752879898543142606244511569936384000000000 0 ruby(16160): => Module.method_added in factorial.rb:1 13 ruby(16160): <= Module.method_added in factorial.rb:1 0 ruby(16160): => Object.factorial in factorial.rb:5 25 ruby(16160): => Range.each in factorial.rb:2 61 ruby(16160): => Fixnum.* in factorial.rb:2 ... 705 ruby(16160): <= Bignum.* in factorial.rb:2 712 ruby(16160): <= Range.each in factorial.rb:2 718 ruby(16160): <= Object.factorial in factorial.rb:2 0 ruby(16160): => Object.puts in factorial.rb:5 20 ruby(16160): => Bignum.to_s in factorial.rb:5 38 ruby(16160): <= Bignum.to_s in factorial.rb:5 53 ruby(16160): => IO.write in factorial.rb:5 74 ruby(16160): <= IO.write in factorial.rb:5 81 ruby(16160): => IO.write in factorial.rb:5 99 ruby(16160): <= IO.write in factorial.rb:5 106 ruby(16160): <= Object.puts in factorial.rb:5

UC2: DOWN THE RUBY STACK# cat rubycount.stp global fn_calls;probe ruby.function.entry{ fn_calls[classname, methodname] <<< 1;}

probe end { foreach ([classname, methodname] in fn_calls- limit 30) { printf("%dx %s.%s\n", @count(fn_calls[classname, methodname]), classname, methodname); }

delete fn_calls;}

UC2: DOWN THE RUBY STACK# stap rubycount.stp -c "ruby factorial.rb"140500611775287989854314260624451156993638400000000021x Bignum.*21x Fixnum.*2x IO.write1x Module.method_added1x Range.each1x Bignum.to_s1x Object.puts1x Object.factorial

UC3: RUBY "TOP"# cat ./ruby-top-modified.stpglobal fn_calls[10240];probe ruby.function.entry { if (isinstr(file, "katello")) fn_calls[pid(), file, methodname, line] <<< 1;}probe timer.ms(4000) { ansi_clear_screen() printf("%6s %80s %6s %25s %6s\n", "PID", "FILENAME", "LINE", "FUNCTION", "CALLS") foreach ([pid,filename,funcname,lineno] in fn_calls- limit 15) { printf("%6d %80s %6d %25s %6d\n", pid, filename, lineno, funcname, @count(fn_calls[pid, filename, funcname, lineno])); } delete fn_calls;}

BUNDLER_EXT

BUNDLER_EXThttp://rubygems.org/gems/bundler_ext

https://github.com/aeolus-incubator/bundler_ext

BUNDLER_EXT# cat Gemfilegem 'rails', '3.0.10'gem 'json'gem 'rest-client', :require => 'rest_client'gem 'jammit', '>= 0.5.4'gem 'rails_warden', '>= 0.5.2'gem 'net-ldap'gem 'oauth'gem 'ldap_fluff'

BUNDLER_EXTif File.exist?(File.expand_path('../../Gemfile.in', __FILE__)) require 'bundler_ext' BundlerExt.system_require(File.expand_path('../../Gemfile.in', __FILE__), :group1, :group2, Rails.env)else Bundler.require :group1, :group2, Rails.envend

WE ARE DONE

CREDITSGreg Weber - http://blog.gregweber.info/posts/2011-06-16-high-performance-rb-part3Ilya Grigorik -http://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/inc.com - finish line picand world-famouse memegenerator.net