scanning strings at supersonic speed (euruko 2011)

8/6/2019 Scanning Strings at Supersonic Speed (EuRuKo 2011)

http://slidepdf.com/reader/full/scanning-strings-at-supersonic-speed-euruko-2011 1/46



Scanning

Strings———— at ————

Supersonic

Speed

Scanning



@

murphy

(Kornelius Kalnbach)

[email protected]@murphy_karasu murfy

Everyththat can go

will go

sofatutor.com



Scanning"Lorem ipsum dolor sit amet, consectetur adielit, sed do eiusmod tempor incididunt ut la

dolore magna aliqua. Ut enim ad minim venianostrud exercitation ullamco laboris nisi ut al

ea commodo consequat. Duis aute irure doreprehenderit in voluptate velit esse cillum d

fugiat nulla pariatur. Excepteur sint occaecat cnon proident, sunt in culpa qui officia deseru

anim id est lakorum."



Strings"Lorem ipsum dolor sit amet, consectetur adielit, sed do eiusmod tempor incididunt ut la

dolore magna aliqua. Ut enim ad minim venianostrud exercitation ullamco laboris nisi ut al

ea commodo consequat. Duis aute irure doreprehenderit in voluptate velit esse cillum d

fugiat nulla pariatur. Excepteur sint occaecat cnon proident, sunt in culpa qui officia deserun

anim id est lakorum."



Strings"<!DOCTYPE html><html lang="en"><head><charset="utf-8"><title>Agenda –EuRuKo 20title><link rel="stylesheet" href="/styleshe

screen.css"><link rel="stylesheet" href="fanc

jquery.fancybox-1.3.4.css"><linkrel="altertype="application/atom+xml" title="ATO

feed"href="http://euruko2011.org/feed.atomrel="alternate"type="application/atom+x

title="github feed"…".scan

http://euruko2011.org/feed.atom

http://euruko2011.org/feed.atom



Strings"class Songdef initialize name, author@name = name@author = author

enddef to_s"this is :#{@name}----#{@author

end

end".scan



Supersonic

3KB

Mach 1.8 = 1900 km/h

2514 pages per second

7.5 MB/s



SpeedWhy?

• Syntax Highlighting

• Parsing (eg. HAML)

• Rite



Shootout

12.283

3.565

11.571



write a supersonic scannerwith pure Ruby code

Target



a fast machine

Resources

♥



big examples

5+ seconds

here: 160 MB

Resources

= 9 times ruby-head



rvm

Resources

rvm.beginr

♥

♥

♥



Time

Endurance

Craziness

Resources

← rvm.beginrescueend.com



General Ideas

C

• avoid convenient APIs

• write everything yourself

• write for the CPU



General Ideas

Ruby

• embrace the core libraries

• write less code

• write for the interpreter



General Ideas

Ruby


• write less code




String

s = "<head><title>EuRuKo 2011</title>"



String + RegExp


puts s.scan(/<(\w+)/)

headtitle



String + RegExp


s.scan(/<(\w+)/) do |tag|puts tag

end

headtitle



StringScanner

Why?

• avoid big RegExp

• control the scan process

• use patterns depending on state

• create patterns on the fly

h k



Benchmark


s *= 5_000_000

s.scan(/<(\w+)/) do |tag|tag

endsoni

1.81.9

jru

rb

ma

S i S



StringScanner

scanner = StringScanner.new(s)

until scanner.eos?if scanner.scan(/<(\w+)/)tag = scanner[1]

else scanner.getch

endend

soni

1.81.9

jru

rb

ma

St i S M R b



StringScanner on MacRuby

http://www.macruby.org/trac/ticket/938

St i S





StringScanner

scanner = StringScanner.new(s)

until scanner.eos?if scanner.scan(/<(\w+)/)tag = scanner[1]

else scanner.getch

endend

soni

1.81.9

jru

rb

ma

sonic™ 21.2

1.8.7 13.61.9.2 5.4

jruby 4.7

rbx 29.1

mac 34.7

G l Id



General Ideas

Ruby


• write less code


L C d



Less Code

class HTML

def initialize html @scanner = StringScanner.new(html)end

def scanuntil @scanner.eos?

if @scanner.scan(/<(\w+)/)yield @scanner[1]

...end

end

L C d



Less Code

class HTML

def initialize html @scanner = StringScanner.new(html)end

def scanuntil eos?

if scan(/<(\w+)/)yield self[1]

...end

end

L C d



Less Code

class HTML

def initialize html @scanner = StringScanner.new(html)enddelegate :eos?, :scan, :[]def scanuntil eos?


...end

end

Less Code



Less Code

class HTML

def initialize html @scanner = StringScanner.new(html)enddelegate :eos?, :scan, :[]def tokenizeuntil eos?


...end

end

Less Code



Less Code

class HTML < StringScanner

def initialize html superend

def tokenizeuntil eos?


...end

end

Less Code



Less Code

class HTML < StringScanner

def tokenizeuntil eos?if scan(/<(\w+)/)yield self[1]

...end

end

General Ideas



General Ideas

Ruby


• write less code




out =Encoder

.new.encodeScanner

.new(in)

• Scanner: simple, 9 rules

• Encoder: does nothing

Single Core



Single Core

out =Encoder

.new.encodeScanner

.new(in)

soni

jru

1.

• Scanner: simple, 9 rules

• Encoder: does nothing



threads = []

in.lines.each_slice 300_000 do |lines|threads << Thread.new do

chunk = lines.joinThread.current[:out] = Encoder.new.encode Scanner

endend

threads.each(&:join)

out = threads.map { |thread| thread[:out] }.joinsoni

jru

1.



threads = []chunk_offsets = [0]in.lines.each_slice slice_size do |lines|chunk_offsets << chunk_offsets.last + lines.join.by

end

chunk_offsets.each_cons(2) do |this_chunk, next_chunkthreads << Thread.new do

chunk = code[this_chunk...next_chunk] Thread.current[:out] = Encoder.new.encode Scannerend

end


out = threads.map { |thread| thread[:out] }.join

soni

jru

1.




end



end



inp

joi

offs




end



end



soni

2 co

4 co

Questions?



Questions?

not allowed:

• When will CodeRay 1.0 be released?

Thank you!



Thank you!

• @euruko

• @yukihiro_matz

• @bovensiepen

• @heinz_gies

• my girlfriend

scanning strings at supersonic speed (euruko 2011)

Documents