net::http async programming2011.rubyworld-conf.org/files/slides/a-5.pdf · features • keepalive...
TRANSCRIPT
Net::HTTP and Async Programming
Sunday, September 4, 11
I Like Net::HTTP
Sunday, September 4, 11
Features
• Keepalive
• Transfer-Encoding: Chunked
• Streaming Requests
• GZip and Deflate
• Threading Support
• Socket Management
• form-data and urlencoded
• SSL and SSPI
• Proxies
• Basic Auth
• Pure Ruby
Sunday, September 4, 11
Bugs :(
>> http = Net::HTTP.new("www.engineyard.com", 80)=> #<Net::HTTP www.engineyard.com:80 open=false>
>> http.get("/javascripts/application.min.js") { }TypeError: can't convert Net::ReadAdapter into String
Sunday, September 4, 11
Net::HTTP.new(host, port)
Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend
Sunday, September 4, 11
Net::HTTP.new(host, port)
Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend clean up response
Sunday, September 4, 11
Net::HTTP.new(host, port)
Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend
clean up socket
Sunday, September 4, 11
Keepalive
Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk end # keepalive if HTTP 1.1 and no # Connection: close http.get('/another') do |chunk| # do something with chunk endend
Sunday, September 4, 11
Keepalive
Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk end # keepalive if HTTP 1.1 and no # Connection: close http.get('/another') do |chunk| # do something with chunk endend
clean up socket
Sunday, September 4, 11
Manual
http = Net::HTTP.start(host, port) http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend
Sunday, September 4, 11
Manualhttp = Net::HTTP.start(host, port)
http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend
http.finish
Sunday, September 4, 11
Manualhttp = Net::HTTP.start(host, port)
http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend
http.finish clean up socket
Sunday, September 4, 11
Socket
http = Net::HTTP.start(host, port) http.get('/path') do |chunk| # do something with chunkend
# keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend
http.finish
socket = TCPSocket.new(host, port) socket.puts "GET / HTTP/1.1"socket.puts "Host: www.google.co.jp"socket.puts socket.read
socket.puts "GET / HTTP/1.1"socket.puts "Host: www.google.co.jp"socket.puts socket.read
socket.close
Sunday, September 4, 11
Why?なぜ?
Sunday, September 4, 11
HTTP Proxy
Client WEBrick Google
1 Thread
stream
Sunday, September 4, 11
Parallel Requests
Client WEBrick Google
1 Thread
stream
Client WEBrick Google
Client WEBrick Google
Sunday, September 4, 11
Blocking Read
Sunday, September 4, 11
get
http = Net::HTTP.start(host, port)http.get('/path') # block
Sunday, September 4, 11
GenericRequest
http = Net::HTTP.start(host, port)http.get('/path') # block
req = Net::HTTPGenericRequest.new( 'GET', false, true, '/path')http.request(req) # block
Sunday, September 4, 11
transport_request
def transport_request(req) begin_transport req res = catch(:response) { req.exec @socket, @curr_http_version, edit_path(req.path) begin res = HTTPResponse.read_new(@socket) end while res.kind_of?(HTTPContinue) res.reading_body(@socket, req.response_body_permitted?) { yield res if block_given? } res } end_transport req, res resrescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exceptionend
Sunday, September 4, 11
transport_request
req.exec @socket, @curr_http_version, edit_path(req.path)
begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue)
body = req.response_body_permitted?
res.reading_body(@socket, body) { yield res if block_given?}
Sunday, September 4, 11
transport_request
req.exec @socket, @curr_http_version, edit_path(req.path)
begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue)
body = req.response_body_permitted?
res.reading_body(@socket, body) { yield res if block_given?}
Sunday, September 4, 11
Socket Leak Impossible
Sunday, September 4, 11
Close Socket
def end_transport(req, res) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif !res.body && @close_on_empty_response D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' else D 'Conn close' @socket.close endend
Sunday, September 4, 11
Close Socket
def end_transport(req, res) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif !res.body && @close_on_empty_response D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' else D 'Conn close' @socket.close endend
Sunday, September 4, 11
Not Flexible
Sunday, September 4, 11
HTTP Proxy
Client WEBrick Google
1 Thread
stream
Sunday, September 4, 11
Net2::HTTP
Sunday, September 4, 11
Net2::HTTPdef transport_request(req) begin_transport req req.exec @socket, @curr_http_version, edit_path(req.path) begin res = HTTPResponse.read_new(@socket) end while res.kind_of?(HTTPContinue) res.request = req if block_given? yield res res.close end
end_transport req, res, block_given? @current_response = resrescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exceptionend
Sunday, September 4, 11
Net2::HTTPreq.exec @socket, @curr_http_version, edit_path(req.path)
begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue) res.request = req if block_given? yield res res.closeend
Sunday, September 4, 11
Net2::HTTPreq.exec @socket, @curr_http_version, edit_path(req.path)
begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue) res.request = req if block_given? yield res res.closeend
Sunday, September 4, 11
Block Form
def end_transport(req, res, block_form) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif @close_on_empty_response && !res.body D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' elsif block_form D 'Conn close' @socket.close endend
Sunday, September 4, 11
Block Form
def end_transport(req, res, block_form) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif @close_on_empty_response && !res.body D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' elsif block_form D 'Conn close' @socket.close endend
Sunday, September 4, 11
Parallel Requests
Client WEBrick Google
1 Thread
stream
Client WEBrick Google
Client WEBrick Google
Sunday, September 4, 11
"Async"
Sunday, September 4, 11
Blocking
foo = File.read("/foo")# blockbar = File.read("/bar")
Sunday, September 4, 11
Parallel I/O
foo, bar = nil
t1 = Thread.new do foo = File.read("/foo")end
t2 = Thread.new do bar = File.read("/bar")end
t1.joint2.join
Sunday, September 4, 11
One-Thread Parallel I/O
f1 = File.open("/usr/share/misc/operator", "r")f2 = File.open("/usr/share/misc/flowers", "r")foo, bar = "", "" until f1.eof? && f2.eof? begin IO.select [f1, f2] foo << f1.read_nonblock(1024) bar << f2.read_nonblock(1024) rescue IO::WaitReadable endend f1.closef2.close
Sunday, September 4, 11
IO.select read_ios
[ ]#<IO:fd 3>#<IO:fd 4>
until read_ios.any? { |io| i.ready? }
Sunday, September 4, 11
read_nonblock(size)
#<IO:fd 4>
bytes_available?
bytes[0...size] raise Errno::EWOULDBLOCK
yes? no?
Sunday, September 4, 11
General Purpose "Reactor"
f1 = File.open("/usr/share/misc/operator", "r")f2 = File.open("/usr/share/misc/flowers", "r")foo, bar = "", "" callbacks = {}
callbacks[f1] = proc { |chunk| foo << chunk }callbacks[f2] = proc { |chunk| bar << chunk }
Sunday, September 4, 11
General Purpose "Reactor"
until callbacks.empty? read, _ = IO.select callbacks.keys read.each do |io| begin callbacks[io].call io.read_nonblock(1024) rescue EOFError io.close callbacks.delete io end endend
Sunday, September 4, 11
Nonblocking Protocol
to_ioread_nonblock
closeeof?
Errno::EWOULDBLOCKEOFError
Sunday, September 4, 11
Nonblocking Protocol
to_ioread_nonblock
closeeof?
IO::WaitReadableEOFError
Sunday, September 4, 11
Demo
Sunday, September 4, 11
Proposal
Sunday, September 4, 11
Ruby Nonblocking Protocol
to_ioread_nonblock
closeeof?
Errno::EWOULDBLOCKErrno::EOFError
Sunday, September 4, 11
ありがとう!@wycats
profiles.google.com/wycatshttp://yehudakatz.com
Sunday, September 4, 11