in the loop - lone star ruby conference
DESCRIPTION
The Reactor Pattern's present in a lot of production infrastructure (Nginx, Eventmachine, 0mq, Redis), yet not very well understood by developers and systems fellas alike. In this talk we'll have a look at what code is doing at a lower level and also how underlying subsystems affect your event driven services.Below the surface : system calls, file descriptor behavior, event loop internals and buffer managementEvented Patterns : handler types, deferrables and half sync / half async workAnti patterns : on CPU time, blocking system calls and protocol gotchasTRANSCRIPT
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
In the Loop
Lone StarRuby
Conference2011
5
6
78
9
10 R
R
R
W
W
W
Lourens Naudé
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Agenda
System callsFile descriptor semanticsBlocking, nonblocking and async I/OThe Reactor PatternPatterns and gotchas
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Caveats
Can’t run before you walkPatterns are framework agnosticKISSread, write and connect only
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Head count ?
Eventmachine (or any client libs)RedisNginxnode.jsZeroMQ
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Reactor Pattern sweet spot ?
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Soft realtime systems where throughput is more
important than processing
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
What does that mean ?
Lots of in flight I/OLittle CPU per request - proxiesImprove ON:OFF cpu timeNEVER about speed for a single client or request
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
"Upgraded to Thin, my app's flying!"
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
EDOINGITWRONG
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
To maintain acceptable response times for more clients,
with the same or less infrastructure
sábado, 13 de Agosto de 2011
Client 1
BrokerClient 2
Client 3
Client 4
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Example app
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
FIX
FIX clientFinancial Information eXchangeCompact protocol - encodes a lot of domain info
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
8=FIX.4.49=4535=049=AB56=CD34=352=20000426-12:05:0610=220
8=FIX.4.4 # FIX version 9=45 # body length 35=0 # Heartbeat msg 49=AB # sender 56=CD # receiver 34=3 # sequence number 52=20000426-12:05:06 # sent at 10=220 # checksum
FIX message
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Mediates access to system resources :
I/O, memory and CPU
The kernel
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Sandboxed execution for security and stability
requirements
User mode
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
System calls ( syscalls )
read(5, &buf, 4000)
How do we access resources ?
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Syscalls
Kernel
Ruby process
Ruby process
disk
read(5, &buf, 4000)
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
A protected function call from user to kernel space with the intent to interact with a system resource
Definition
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Characteristics
Uniform API on POSIX systemsDitto for behaviorLIES !Slow emulations
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Syscall performance
MUCH slower than function calls20x and moreDefinite context switchCalls can block - slow upstream peers
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Who doesn't know what file descriptors or file handles
are ?
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
fd = open(“/path/file”, O_READ)read(fd, &buf, 4000)
open syscall
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Examples
local filesocketdirectorypipeConsistent API, semantics differ
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Definition
Numeric handleReferences a kernel allocated resourceKernel bufferUser space buffer
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
A request to read a chunk of data from a descriptor may
block depending onbuffer state
Blocking I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Blocked buffer state
Kernel bufferUser buffer
2000 1000
read(5, &b, 4000)
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
fd = socket (PF_INET, SOCK_STREAM, 0); fcntl (fd, F_SETFL, O_NONBLOCK)
Nonblocking I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
A read request would only be initiated if the system
call won’t block
Nonblocking I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
EAGAIN or EWOULDBLOCK
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Not a silver bullet
Guideline onlyNot free - invokes a syscallNot supported by all devices
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Nonblocking I/O IS NOT asynchronous I/O
Async I/O myth
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Async I/O
Windows I/O Completion PortsAIO: POSIX Realtime ExtensionAsync I/O often skip double bufferingSupports file I/O only
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Recap
O_NONBLOCK is a guideline onlyInvoked in user spaceData transfer in user spaceBlocking and nonblocking I/O terms
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Remember, the primary use case is increased throughput
Reactor Pattern
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Challenges
Large number of file descriptorsRespond to descriptor state changesExecute domain logic
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
WHILE maintaining acceptable response times
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
while true do # shortened for brevity add_new_descriptors # attach new client descriptors modify_descriptors # update descriptor state break if terminate? # conditional loop breakend
The Reactor Loop
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Registered descriptors
Reactor
5
6
78
9
10 R
RR
W
W
W
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Who's familiar with select, poll, epoll or kqueue ?
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Multiplexed I/O
Nonblocking I/O is inefficientRetry on EAGAIN is pollingMultiplexed I/O: concurrent blockingNotified of state changes on any registered fd
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Multiplexed I/O
MultiplexerI/O bound
App
fd 5
User space Kernel space
fd 6fd 7
fd 8
fd 9fd 10fd 11
fd 12
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
1. Tell me when fds 1..200 are ready for a read or a
write
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
2. Do other work until one of them's ready
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
3. Get woken up by the multiplexer on state
changes
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
4. Which of fd set 1..200 are in a ready state ?
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
5. Handle all fds ready for I/O
Multiplexed I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
These notifications, or rather state changes in
readiness for reading or writing, that's the Events in
Event Driven I/O
Event driven I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
module MyFixApp def connection_completed end
def receive_data(buffer) endend
Event handlers / callbacks
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Callbacks
Reactor provides I/O concurrencyNO concurrent handling of usercodeFire on the reactor threadRuns within the reactor loop
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Ticks
A turn through the event loopThe reactor can do several thousand ticks per secondTime slice for doing workAim to use minimal CPU per tick
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Best practices, patterns and gotchas
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
#1: Coupling Layers
App handler
I/O Multiplexer
Dispatch handler
Kernel
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Dispatch Handler
Handles low level events from the multiplexed I/O frameworkeg. readiness for read or writeHide complexities from appsBufferingTransferConversion to and from Ruby strings
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class EventableDescriptor def read; end # Tell don't ask interface def write; end def heartbeat; end def select_for_read; end # I/O multiplexer def select_for_write; end # agnosticend
Dispatch Handler
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Application Handlers
Domain layerHigher level eventsConnection accepted, read, disconnectedEncode and decode protocolsPerform business logic
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(data); end # callbacks def connection_completed; end def unbind; endend
Application Handlers
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(buffer) # Encapsulate excess business logic in callbacks @queue.push *FIX::Parser.parse(buffer) endend
Glue
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
You don’t need a transport for testing
#2: Testing
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(data) @queue << data endend
def test_enqueue_on_receive conn.receive_data "stub data" assert_equal 1, conn.queue.sizeend
Testing
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Events occur asynchronously, but they're
handled synchronously
#3: Confusing sync and async
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
App handler
I/O Multiplexer
Dispatch handler
Kernel
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(buffer) # enqueue to a thread pool EM.defer{ FIX::Parser.parse(buffer) } endend
Defer work to a thread pool
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(buffer) # allow the reactor to service other clients also EM.schedule{ FIX::Parser.parse(buffer) } endend
Schedule on the reactor thread
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
#4: Too much ON cpu time
Reactor is single threadedSingle core on SMP systemsPegging the core blocks the event loopTime sharing
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
[1..n].each do |i| process(i) # expensive workend
Tight loops
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
work = [1..n]EM.tick_loop do if work.empty? :stop else process(work.shift) # expensive work endend
Prefer a tick loop
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Protocol parsers incur a cost for both encoding and decoding from buffers
Slow Protocol parsers
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Prefer a fast C extension / parser to native Ruby code
Negate encoding / decoding costs
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
EM.connect("slow-broker.net", 2000, FixConnection)
#5: Name resolution and connects
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
d = EM::DNS::Resolver.resolve "slow-broker.net"d.callback do |addrs| # connect to broker API when resolved EM.connect(addrs.first, 2000, FixConnection) end
Async resolver
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
conn = EM::HttpRequest.new('http://api.a-broker.net') req1 = conn.get :path => “x”, :keepalive => truereq1.callback { req2 = conn.get :path => “y” req2.callback { # same connection as req1 }}
Async HTTP APIs
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Not all file descriptors support O_NONBLOCK
#6: Blocking I/O
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
$ gem install eio
EIO.open(__FILE__) do |fd| EIO.read(fd, 1000) do |buf| p buf EIO.close(fd) endend
http://github.com/methodmissing/eio
Ruby wrapper for libeio
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
redis = EM::Protocols::Redis.connectredis.errback do |code| puts "Error code: #{code}"endredis.set "a", "foo" do |response| redis.get "a" do |response| puts response endend
#7: All libs have to be evented
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
client = Mysql2::EM::Client.newdefer = client.query "SELECT sleep(3) as query"defer.callback do |result| puts "Result: #{result.to_a.inspect}"end
Evented MySQL
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
The Garbage Collector could be invoked at any
time during request processing
#8: Memory
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Be careful with sloppy allocation patterns
Minimize GC pressure
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
module ProxyConnection def initialize(client, request) @client, @request = client, request end def post_init EM::enable_proxy(self, @client) end def connection_completed send_data @request end def proxy_target_unbound close_connection # end # # def unbind # @client.close_connection_after_writing # end # end
Connection level proxy
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(inbound) # inbound.size can be a single protocol # message or a batch - stream of data end
def send_data(outbound) # enqueued on a write queue endend
#9: Buffer sizes
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
class FixConnection def receive_data(buffer) # buffer packs multiple FIX messages Parser.parse(buffer).each do |msg| EM.next_tick { process(msg) } end endend
Scheduled parsing
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Takeaways
Time sharing for I/O bound servicesSystem call overheads and behaviorBlocking, nonblocking and async I/OSchedule through ticksOS, dispatch and app layersTransport last, interfaces first
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Questions ?
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
Wildfire Interactive, Inc. is hiring
sábado, 13 de Agosto de 2011
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929
http://wildfireapp.com/buzz/jobsfollow @methodmissing
fork github.com/methodmissing
Thanks !
sábado, 13 de Agosto de 2011