zenhacks & friends · builder.include ‘’ ... thingy() while condition do thingy() end end...
Post on 18-Jul-2020
9 Views
Preview:
TRANSCRIPT
Polishing RubyZenHacks & Friends
by: Ryan Davis, Seattle.rb
Setting ExpectationsThis Presentation is...
• ...58 slides in 45 minutes - very dense
• ...about the Shinies
• ...“Hard Core”, not for everyone
Driving Philosophy• Simplicity & Clarity
• Otherwise, you don’t fully understand.
• Make it Right
• Make it Work
• Make it Fast
• Speed comes last, and only objectively.
• Balance size and speed.
• Accelerate Maturity by splitting from parents.
• Have fun! Otherwise, what’s the point?
Standard Example
• I’ll be using factorial throughout most of the code in order to make things easier to track.
• Most of the time, it’ll be implemented as:
def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend
Overview
ZenHacks
ruby2ruby
sexp as method
auto-refactoring
obfuscator
profiler
zenoptimize
Advanced
Dependencies
ABC Metrics
Colored Sexp
Graphs
Overview
Current Status
Step by Step
Factorial
Future Plans
Intermediate
Architecture
& extensibility
+ A Sekrit
Overview
factorial
node printer
code2test
test2code
unit_diff
RubyInline ruby2c
Beginner
Overview
RI::C - hello world
factorial_c
ZenTestParseTree/
SexpProcessor
Newb
Overview
factorial
node printer
code2test
test2code
unit_diff
Beginner
Overview
RI::C - hello world
factorial_c
RubyInlineZenTestParseTree/
SexpProcessor
• Generates missing tests.
• Generates missing methods under test (MUT).
• Helps illuminate failing tests.
ZenTest
ZenTestGenerating Tests
class TestFactorial < Test::Unit::TestCase def test_factorial raise NotImplementedError, 'Need to write test_factorial' endend
class Factorial def factorial(n) f = 1 n.downto(2) { |x| f *= x } return f end end
ZenTestGenerating Tests
ZenTestGenerating MUTs
class Factorial def factorial raise NotImplementedError, ‘Need to write factorial’ endend
class TestFactorial < Test::Unit::TestCase def test_factorial @f = Factorial.new assert_equal(1, @f.factorial(1)) assert_equal(2, @f.factorial(2)) assert_equal(6, @f.factorial(3)) endend
ZenTestGenerating MUTs
& ./test.rb... 1) Failure:test_conditional4(TestTypeChecker) [./r2ctestcase.rb:1068:in `test_conditional4' ./r2ctestcase.rb:1055:in `test_conditional4']:<t(:if, t(:call, t(:lit, 42, Type.long), :==, t(:arglist, t(:lit, 0, Type.long)), Type.bool), t(:return, t(:lit, 2, Type.long), Type.void), t(:if, t(:call, t(:lit, 42, Type.long), :>, t(:arglist, t(:lit, 0, Type.long)), Type.bool), t(:return, t(:lit, 3, Type.long), Type.void), t(:return, t(:lit, 4, Type.long), Type.void), Type.void), Type.void)> expected but was<t(:if, t(:call,
ZenTestIlluminating Issues
Which would you rather read/debug???
This:
& ./test.rb | unit_diff... 1) Failure:test_conditional4(TestTypeChecker)12c10< :>,---> :<,
Or this:
ZenTestIlluminating Issues
• Allows you to write C functions within your ruby classes.
• No compiling/linking/packaging phase required.
• Only recompiles when needed.
• Incredibly accelerated development time.
• Write & Run. There is no step 3.
• You aren’t limited to C.
RubyInline
RubyInlineObligatory Hello World
class Hello inline do |builder| builder.include ‘<stdio.h>’ builder.c ‘ void hello() { puts(“hello world”); } ‘ endend
Hello.new.hello
% ./hello.rbhello world
RubyInline
RubyInlineFactorial
class FastMath inline do |builder| builder.include ‘<math.h>’ builder.c ‘ long factorial_c(int max) { int i=max, result=1; while (i >= 2) { result *= i--; } return result; }’ endend
math = FastMath.new10000.times do math.factorial(20); end
RubyInlineFactorial
ParseTree & SexpProcessor
“ParseTree is a little brown stinky ferret that digs down a hole and violently rips the AST away from the warm bosom of ruby”
ParseTree & SexpProcessor
• ParseTree:
• Extracts ruby’s AST internals and makes them digestible by mortals.
• SexpProcessor:
• The framework for digesting them.
Basic Vocabulary
• Sexp = s-expression = An S-expression (S stands for symbolic) is a convention for representing data structures in a text form. (wikipedia)
• AST = Abstract Syntax Tree = What the parser makes when it parses and what the interpreter runs.
ParseTree
• ParseTree is the library that converts ruby into sexps.
[:defn, :factorial, [:scope, [:block, [:args, :n], [:lasgn, :f, [:lit, 1]], [:iter, [:call, [:lvar, :n], :downto, [:array, [:lit, 2]]], [:dasgn_curr, :x], [:lasgn, :f, [:call, [:lvar, :f], :*, [:array, [:dvar, :x]]]]], [:lvar, :f]]]]
mmmmm, factorial…
ParseTreeparse_tree_show
SexpProcessorClass Catalog
class QuickPrinter < SexpProcessor def initialize super self.strict = false self.auto_shift_type = true end def process_class(exp) puts “class #{exp.shift}” exp.shift # superclass process exp.shift # body return s() end def process_defn(exp) puts “ def #{exp.shift}” exp.shift # body return s() endend
class QuickPrinter def initialize def process_class def process_defn
=>
SexpProcessor
Dependencies
ABC Metrics
Colored Sexp
Graphs
Overview
Current Status
Step by Step
Factorial
Future Plans
Intermediate
Architecture
& extensibility
+ A Sekrit
RubyInline ruby2cParseTree/
SexpProcessor
Intermediate
Eric Hodel’sCar!
It’s famous!Have you had Matz in your
car?
RubyInlineGeneral Architecture
• RubyInline has a totally open architecture making it easy to extend with other languages.
• Uses ducktyping, an RI extension only needs to implement:
• load_cache
• build
• load
builder = builder_class.new self
yield builder
unless options[:testing] then unless builder.load_cache then builder.build builder.load endend
• You thought jruby was wrong? HA!
• Ugly, yeah... but there is a LOT out there.
• Originally by Yoshida Masato for ruby 1.4!
• 20 minute conversion into RubyInline.
• Sooo easy. Goodbye setup.rb!
• Seems to run complex PERL just fine.
RubyInlineInline::PERL
I’m really sorry for this.
Honest.
RubyInlineInline::PERL
ParseTree & SexpProcessor
• Tools you easily make with SexpProcessor:
• Dependency Reporting
• Complexity Analysis
• Parser Visualization
• All of the Above
(and many more)
SexpProcessorDependency Analysis
def process_defn(exp) name = exp.shift @current_method = name return s(:defn, name, process(exp.shift), process(exp.shift)) end
def process_const(exp) name = exp.shift const = (defined?($c) ? @current_class.name : “#{@current_class}.#{@current_method}”) is_class = ! (Object.const_get(name) rescue nil).nil? @dependencies[name] << const if is_class return s(:const, name) end
SexpProcessorDependency Analysis
SexpProcessorComplexity Metrics
• ABC Metrics• # of Assignments• # of Branches• # of Calls
% parse_tree_abc printer.rb|ABC| = Math.sqrt(assignments^2 + branches^2 + calls^2)
1) QuickPrinter.process_class = 0 + 1 + 7 = 7.07 2) QuickPrinter.process_defn = 0 + 0 + 4 = 4.00 3) QuickPrinter.initialize = 2 + 0 + 0 = 2.00 4) Total = 2 + 1 + 11 = 13.07
SexpProcessorComplexity Metrics
:defn
:factorial :scope
:block
:args :lasgn :iter :return
:n :f :lit
1
:call :dasgn_curr :lasgn
:lvar :downto :array
:n :lit
2
:x :f :call
:lvar :* :array
:f :dvar
:x
:lvar
:f
SexpProcessorVisualization
SexpProcessorVisualization
Ruby2cThe Problem
Takahashi Method:
C Sucks
• Automatic translation of static ruby subset to C.
• Only translate if it has native support in C.
• Open architecture will allow it to translate to both Ruby internals C and generic ANSI C.
• Uses type inference to help with translation to generic C types.
Ruby2cThe Basics
Ruby2cBasic Design
SexpProcessor
ParseTree Rewriter TypeChecker Ruby2CCompositeSexp!
Processor
Sexpprocessesprocessors
ParseTree
Rewriter
TypeChecker
R2CRewriter
RubyToC
!P
ip
el
in
e!
Ruby2cStep-by-Step
• Ruby parses code
• ParseTree extracts AST
• Rewriter doesn’t touch it
• (SexpProcessor.process converts it to a Sexp)
• TypeChecker unifies it
• Ruby2C translates to C
Ruby
false and true
[:and, [:false], [:true]]
case NODE_AND: add_to_parse_tree(current, node->nd_1st); add_to_parse_tree(current, node->nd_2nd); break;
ParseTree
s(:and, s(:false), s(:true))
(no code for :and)
Rewriter
[:and, [:false, Type.bool], [:true,
Type.bool], Type.bool]
def process_and(exp) rhs = process exp.shift lhs = process exp.shift rhs.sexp_type.unify lhs.sexp_type rhs.sexp_type.unify Type.bool return t(:and, rhs, lhs, Type.bool)end
TypeChecker
Qfalse && Qtrue;
def process_and(exp) lhs = process exp.shift rhs = process exp.shift return "#{lhs} && #{rhs}"end
Ruby2C
Ruby2cFactorial
• For plain algorithmic code, ruby2c works quite well.
• Type inference determines the types of the args and automatically translates from Ruby types to C types.
• Ruby iterators are converted to equivalent while loops in C.
def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend
static VALUE factorial (VALUE self, VALUE _n) { long n = NUM2INT(_n); long f, x; f = 1; x = n; while (x >= 2) { f = f * x; x = x - 1; }; return INT2NUM(f);}
• For 1.0 final:
• Clean
• Document
• Differentiate between generic C and Ruby C
• More Ping-Pong against Metaruby
Ruby2cFuture Plans
Advanced
ruby2ruby -
factorial
sexp as method
auto-refactoring
obfuscator
profiler
zenoptimize
Advanced
ZenHacks
ZenHacksComing Full Circle
• A cornucopia of hackery: Toys, Tricks and Tools that have spawned out of our other projects but don't exactly fit there.
• Integrative: They almost always depend on multiple projects.
• Stuff that others might want to play with, but that I didn’t want to officially “support”.
• Anything I wrote that made me wince.
ZenHacksRubyToRuby - Building Blocks
• First SexpProcessor instance translates Sexps back to ruby.
• Provides a powerful foundation for a number of tools that follow a similar pattern:
• Extract Sexp
• Analyze and Modify Sexp
• Translate to Ruby
• Feed back to Ruby via eval
sexp = [:defn, :factorial, [:scope, [:block, [:args, :n], [:lasgn, :f, [:lit, 1]], [:iter, [:call, [:lvar, :n], :downto, [:array, [:lit, 2]]], [:dasgn_curr, :x], [:lasgn, :f, [:call, [:lvar, :f], :*, [:array, [:dvar, :x]]]]], [:lvar, :f]]]]
RubyToRuby.new.process sexp=> “def factorial(n) f = 1 n.downto(2) {|x| f = (f * x) } fend”
ZenHacksRubyToRuby - Building Blocks
ZenHacks• Back in June _why posted a blog entry about
having a ruby-lisp hybrid.
• It was just ruby w/ extra parens...
• ZenHacks lets you do nasty stuff like this:
require ‘sexp2ruby’
class Foo _ [:defn, :example, [:args], [:call, [:lit, 1], :+, [:array, [:lit, 1]]]]end
Foo.new.example #=> 2
...not that you’d want to.
RubyToRubyauto-refactoring examples
• User contribution by Rudi Cilibrasi.
• RubyToRuby allows sexps to be user friendly.
• We can analyze ruby mechanically, but then report what we find using ruby again.
• It allows us to write “lint-like” tools with nice reports like the following:
Suggest refactoring HastilyWritten#weirdfunc from:
def weirdfunc() thingy() while condition do thingy() endend
to:
def weirdfunc() begin thingy() end while conditionend
RubyToRubyauto-refactoring examples
Ruby Obfuscator
• ParseTree
• + SexpProcessor
• + ruby2c
• + a new tail-end
• = RubyObfuscator
def factorial(n) f = 1 n.downto(2) { |x| f *= x } return fend! ! ! ! ! becomes:
static VALUErrc_cF_factorial(VALUE __self, VALUE n) { VALUE f; VALUE x; f = LONG2FIX(1); x = n; while (RTEST(rb_funcall(x, rb_intern(“>=”), 1, LONG2FIX(2)))) { f = rb_funcall(f, rb_intern(“*”), 1, x); x = rb_funcall(x, rb_intern(“-”), 1, LONG2FIX(1)); }; return f;}
Ruby Obfuscator
zenprofile
• Standard profiler is 65 lines long, but dead slow.
• Shugo modified the event system to bypass set_trace_func.
• Massively fast, but 701 lines of C.
• zenprofile is 190 lines long, 114 are inlined C.
• 24x faster than ruby, ⅓rd slower than shugo’s.
• zenprofile’s goal is to balance size and speed.
0
187.5
375.0
562.5
750.0
original zenprofile shugo
16
24
576 701
190
65
zenprofilesize vs speedLoC Time (*8)
zenprofile
ZenOptimizeFull Circle... on steroids
• Take profiler’s architecture• Make a simple fast
method call counter• Use the profiler to trigger
a threshold event.• Event pulls method’s sexp.• Pass to ruby2c• Pass C to inline• Now you have
zenoptimize!• in 153 lines of code...
% time ruby factorial.rb 5000000Iter = 5000000, T = 67.23166600 sec, 0.00001345 sec / iterreal 1m7.310suser 0m55.980ssys 0m0.280s
% time ruby -reallyfast factorial.rb 5000000*** Optimizing Factorial.factorialIter = 5000000, T = 13.30087900 sec, 0.00000266 sec / iterreal 0m14.382suser 0m12.550ssys 0m0.290s
ZenOptimizeFull Circle... on steroids
Other Possibilities
• Automated Refactorings
• Idiomatic Optimizations (e.g. always use the most efficient iterator)
• Duplicate code checker
• Many kinds of lint-like tools
• Deprecation rewriter, etc.
Thank You
http://www.zenspider.com/seattle.rb
Presenters: Send me your
slides!
top related