really simple proxy

8/18/2019 Really Simple Proxy

1/39

Writing a reverse proxy/loadbalancer from the ground up in C, part 0:

introduction

We’re spending a lot of time on nginx conguration at Pythonny!here" We’re a platform#as#a#

service, and a lot of people host their !ebsites !ith us, so it’s important that !e have a reliable

load#balancer to receive all of the incoming !eb tra$c and appropriately distribute it around

bac%end !eb#server nodes"

nginx is a fantastic, possibly unbeatable tool for this" &t’s fast, reliable, and light!eight in terms of

CP' resources" We’re using the (pen)esty variant of it, !hich adds a number of useful modules *

most importantly for us, one for +ua scripting, !hich means that !e can dynamically !or% out

!here to send tra$c as the hits come in"

&t’s also uite simple to congure at a basic level" -ou !ant all incoming reuests for site . to go to

bac%end - ust !rite something li%e this:

server {

server_name X

listen 80;

location / {

proxy_set_header Host $host;

proxy_pass Y;

}

}

1imple enough" +ua scripting is pretty easy to add * you 2ust put an extra directive before

theproxy_pass that provides some +ua code to run, and then variables you set in the code can be

accessed from the proxy_pass"

3ut there are many more complicated

options" worker_connections, tcp_nopsh, send!ile,types_hash_max_si"e4 1ome are reasonably easy to

understand !ith a certain amount of reading, some are harder"

&’m a big believer that the best !ay to understand something complex is to try to build your o!n

simple version of it" 1o, in my copious free time, &’m going to start putting together a simple

loadbalancer in C" 5he aim isn’t to re!rite nginx or (pen)esty6 it’s to !rite enough euivalent

functionality that & can better understand !hat they are really doing under the hood, in the same

!ay as !riting a compiler for a toy language gives you a better understanding of ho! proper

compilers !or%" &’ll get a good grasp on some underlying (1 concepts that & have only a vague

appreciation of no!" &t’s also going to be uite fun coding in C again" &’ve not really !ritten any

since 7889"

ny!ay, &’ll document the steps & ta%e here on this blog6 partly because there’s a faint chance that

it might be interesting to other experienced Python programmers !hose C is rusty or nonexistent

and !ant to get a vie! under the hood, but mostly because the best !ay to be sure you really

understand it is to try to explain it to other people"

& hope it’ll be interesting

http://nginx.org/https://www.pythonanywhere.com/http://openresty.org/https://www.pythonanywhere.com/http://openresty.org/http://nginx.org/


2/39

Writing a reverse proxy/loadbalancer from the ground up in C, part 7: a trivial

single#threaded proxy

5his is the rst step along my road to building a simple C#based reverse proxy/loadbalancer so that &

can understand ho! nginx/(pen)esty !or%s * more explanation here" &t’s called rsp, for )eally

1imple Proxy" 5his version listens for connections on a particular port, specied on the command

line6 !hen one is made it sends the reuest do!n to a bac%end * another server !ith an associated

port, also specied on the command line * and sends !hatever comes bac% from the bac%end bac%

to the person !ho made the original connection" &t can only handle one connection at a time *

!hile it’s handling one, it 2ust ueues up others, and it handles them in turn" 5his !ill, of course,

change later"

&’m posting this in the hope that it might help people !ho %no! Python, and some basic C, but !ant

to learn more about ho! the (1#level net!or%ing stu; !or%s" &’m also vaguely hoping that any

readers !ho code in C day to day might ta%e a loo% and tell me !hat &’m doing !rong :#<

5he code that &’ll be describing is hosted on =it>ub as a pro2ect called rsp, for ?)eally 1imple Proxy@"

&t’s A&5 licensed, and the version of it &’ll be !al%ing through in this blog post is as of commit

fB7fDa" &’ll copy and paste the code that &’m describing into this post any!ay, so if you’re follo!ing

along there’s no need to do any %ind of complicated chec%out"

5he repository has this structure:

# %&'()&*md

# +,-&.&*md

# setpenv*sh # rn_interation_tests

# promote_to_live

# handle_interation_error

# *itinore

# !ts

1 test_can_proxy_http_re2est_to_3ackend*py

# src

1 )ake!ile

# rsp*c

%&'()&*md, +,-&.&*md and *itinore are pretty self#explanatory"

setpenv*sh contains a fe! shell commands that, !hen run on a fresh 'buntu machine, !ill install

all of the dependencies for compiling rsp"

!ts, short for Eunctional 5ests, contains one Python script6 this creates a simple Python !eb server

on a specic port on localhost, then starts rsp congured so that all reuests that come in on its

o!n port get for!arded to that bac%end" &t then sends a reuest to rsp and chec%s that the

response that comes bac% is the one from the bac%end, then does another to ma%e sure it can

handle multiple reuests" 5his is the most trivial test & could thin% of for a rst cut at rsp, and this

blog post contains an explanation of ho! the minimal C code to ma%e that test pass !or%s" (ver

time, &’ll add more test scripts to the repository to chec% each incremental improvement in rsp’s

http://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/f214f5a75e112311b90c81ad823bf86c3900b03dhttps://github.com/gpjt/rsp/commit/f214f5a75e112311b90c81ad823bf86c3900b03dhttp://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/f214f5a75e112311b90c81ad823bf86c3900b03dhttps://github.com/gpjt/rsp/commit/f214f5a75e112311b90c81ad823bf86c3900b03d


3/39

functionality" Faturally, &’m !riting all of this test#rst Gthough &’m too laHy to !rite unit tests right

no! * this may !ell come bac% to bite me laterub" -ou can probably ignore all of those, though if you do chec% out the

repository then rn_interation_tests is a convenient !ay to run the E5s in one go"

1o, that’s the structure" & !on’t go into a description of ho! the Python functional test !or%s, as &

expect most readers here !ill understand it pretty !ell" nd &’m assuming that you have at least a

nodding acuaintance !ith Aa%eles, so & !on’t explain that bit" 1o, on to the C code

rsp*c contains code for an incredibly basic proxy" &t’s actually best understood by !or%ing from the

top do!n, so let’s go" Eirst, some pretty standard header includes:

4inclde 5stdio*h6

4inclde 5stdli3*h6

4inclde 5strin*h6

4inclde 5sys/types*h6

4inclde 5sys/socket*h6

4inclde 5netinet/in*h6

4inclde 5netd3*h6

couple of constants for later use:

4de!ine )'X_+,7&._'-9+: <

4de!ine =>>&%_,?& @0AB

nd on to our rst function * one that, given a connection to a client Gie" bro!ser< that’s hit the

proxy, connects to a bac%end, sends the client’s reuest to it, then gets the response from the

bac%end and sends it bac% to the client" 5his is simpler than the code that handles the process of

listening for incoming connections, so it’s !orth running through it rst"

void handle_client_connectionCint client_socket_!dD

char E3ackend_hostD

char E3ackend_port_strF

{

5he rst parameter is an integer fle descriptor for the client soc%et connection G!hich has already

been established by the time !e get here


4/39

We also ta%e a string specifying the address of the bac%end !e’re going to connect to, and an

another specifying the port on the bac%end" We accept a string for the port Grather than an integer<

because one of the the (1 system calls !e’re going to use accepts string services rather than ports

* the most immediate advantage of !hich is that !e can 2ust specify GhttpG and let it !or% out that

means port I0" >ardly a huge !in, but neat enough"

)ight, next our local variable denitions * these need to go at the start of the function * an

annoying reuirement in C J'pdate: turns out that it hasn’t been a reuirement since the 7888 Cstandard, so later posts update thisK * but !e’ll bac%trac% to tal% about !hat each one’s used for as

!e use them"

strct addrin!o hints;

strct addrin!o Eaddrs;

strct addrin!o Eaddrs_iter;

int etaddrin!o_error;

int 3ackend_socket_!d;

char 3!!er=>>&%_,?&I;

int 3ytes_read;

1o no! it’s time to actually do something" (ur rst step is to convert the hostname/service

descriptor strings !e have into something that !e can use to ma%e a net!or% connection"

>istorically one might have used the ethost3yname system call, but that’s apparently fro!ned upon

these days6 it’s non#reentrant and ma%es it hard to support both &Pv and &PvL" 5he hip !ay to gethost information is by using etaddrin!o, so that’s !hat !e’ll do"

etaddrin!o needs three things6 the hostname and service !e’re connecting to, and

some hintstelling us !hat %ind of thing !e’re interested in hearing about * for example, in our case

!e only !ant to %no! about addresses of machines that can handle streaming soc%ets !hich !e

can read to and !rite from li%e les, rather than datagrams !here !e send lumps of data bac% and

forth, one lump at a time"

We already have the hostname and service passed in as arguments, so our rst step is to set up a

structure to represent these hints" We have the local variable that !as dened earlier as:


1o !e need to set some values on it" &n C code, a struct that’s allocated on the stac% as a local

variable li%e that has completely undened contents, !hich means that !e need to clear it out by

setting everything in it to Heros using memset, li%e this:

memsetCJhintsD 0D si"eo!Cstrct addrin!oFF;

…and once that’s done, !e can ll in the things !e’re interested in:

http://linux.die.net/man/3/gethostbynamehttp://en.wikipedia.org/wiki/Reentrancy_(computing)http://linux.die.net/man/3/getaddrinfohttp://linux.die.net/man/3/memsethttp://linux.die.net/man/3/gethostbynamehttp://en.wikipedia.org/wiki/Reentrancy_(computing)http://linux.die.net/man/3/getaddrinfohttp://linux.die.net/man/3/memset


5/39

hints*ai_!amily K '>_=.L&-;

hints*ai_socktype K :-9_7%&');

'>_=.L&- means that !e’re happy !ith either &Pv or &PvL results, and :-9_7%&') means that !e

!ant something that supports streaming soc%ets"

Fo! !e’ve set up our hints structure, !e can call etaddrin!o" &t returns Hero if it succeeds, or an

error code if it doesn’t, and the real address information results are returned in a parameter * !e

pass a pointer to a pointer to a strct addrin!o in, and it puts a pointer to the rst in a list of results

into the pointer that the pointer points to" +ovely, pointers to pointers and !e’re only a doHen lines

in4

etaddrin!o_error K etaddrin!oC3ackend_hostD 3ackend_port_strD JhintsD JaddrsF;

i! Cetaddrin!o_error MK 0F {

!print!CstderrD G-oldnNt !ind 3ackendO Ps1nGD ai_strerrorCetaddrin!o_errorFF;

exitC


6/39

addrs_iter6ai_addrlenF MK >&%_,?&FF {

writeCclient_socket_!dD 3!!erD 3ytes_readF;

}

5hen !e close the client soc%et, and that’s the total of our client#handling code"

http://linux.die.net/man/2/readhttp://linux.die.net/man/2/writehttp://linux.die.net/man/2/readhttp://linux.die.net/man/2/write


7/39

closeCclient_socket_!dF;

}

5he code !e use to create a soc%et to listen for incoming client connections and pass them o; to

the function !e’ve 2ust gone through is actually pretty similar, but !ith a fe! interesting t!ists" &t

lives Gas you might expect< in the program’s main function:

int mainCint arcD char EarvIF {

…!hich !e start o; !ith our local variables again:

char Eserver_port_str;

char E3ackend_addr;

char E3ackend_port_str;


strct addrin!o Eaddrs;

strct addrin!o Eaddr_iter;

int etaddrin!o_error;

int server_socket_!d;

int client_socket_!d;

int so_reseaddr;

5he rst step is 2ust to chec% that !e have the right number of command#line arguments and to put

them into some meaningfully#named variables:

i! Carc MK @F {

!print!CstderrD

G=saeO Ps 1nGD

arv0IF;

exitC


8/39

hints*ai_!amily K '>_=.L&-;

hints*ai_socktype K :-9_7%&');

hints*ai_!las K ',_L',S&;

etaddrin!o_error K etaddrin!oC.=++D server_port_strD JhintsD JaddrsF;

5he ai_!las structure member being set to ',_L',S&, combined !ith the .=++ rst parameter,

tells etaddrin!o that !e !ant to be able to run a server soc%et on this address * !e !ant to be

able to listen for incoming connections, accept them, and handle them"

(nce !e’ve got the list of appropriate addresses, !e iterate through them again, and for each one

!e create a soc%et li%e !e did before, but no! instead of trying to connect to them to ma%e an

outgoing connection, !e try to 3ind so that !e can accept incoming connections:

!or Caddr_iter K addrs; addr_iter MK .=++; addr_iter K addr_iter6ai_nextF {

server_socket_!d K socketCaddr_iter6ai_!amilyD

addr_iter6ai_socktypeD

addr_iter6ai_protocolF;

i! Cserver_socket_!d KK


9/39

share this soc%et !ith other people@, !hich mitigates this problem"

ny!ay, once !e’ve bound Gor if !e !ere unable to bind< then !e handle errors and tidy up 2ust as

!e did before:

i! Caddr_iter KK .=++F {

!print!CstderrD G-oldnNt 3ind1nGF;

exitC


10/39

nd o; !e go, around the loop again:

}

}

Phe!" 1o that !as a bit harder than it !ould have been in Python" 3ut not too scary" >opefully it !as

all reasonably clear * and if it !asn’t, please let me %no! in the comments" nd if any C experts

have been reading * than% you for putting up !ith the slo! pace, and if you have any suggestions

then &’d love to hear them

5he next step, & thin%, is to ma%e this a more useful proxy by ma%ing it no longer single#shot, and

instead accept multiple simultaneous client connections and proxy them bac% to the bac%end" We

can then add multiple bac%ends, and start loo%ing at selecting !hich one to proxy to based on

theHost >55P header" nd, as &’m aiming to produce a cut#do!n version of (pen)esty, then adding

some +ua scripting !ould help too"

3ut multiple connections rst" >ere’s ho! & handle them"

Writing a reverse proxy/loadbalancer from the ground up in C, part B: handling

multiple connections !ith epoll

5his is the second step along my road to building a simple C#based reverse proxy/loadbalancer so

that & can understand ho! nginx/(pen)esty !or%s * more bac%ground here" >ere’s a lin% to the

rst part, !here & sho!ed the basic net!or%ing code reuired to !rite a proxy that could handle one

incoming connection at a time and connect it !ith a single bac%end"

5his Grather long< post describes a version that uses +inux’s epoll P& to handle multiple

simultaneous connections * but it still 2ust sends all of them do!n to the same bac%end server" &’ve

tested it using the pache a3 server benchmar%ing tool, and over a million reuests, 700 running

concurrently, it adds about 0"7ms to the average reuest time as compared to a direct connection

to the !eb server, !hich is pretty good going at this early stage" &t also doesn’t appear to lea%

memory, !hich is doubly good going for someone !ho’s not coded in C since the late 80s" &’m

pretty sure it’s not totally stupid code, though obviously comments and corrections !ould be much

appreciated

J'PO5M: there’s denitely one bug in this version * it doesn’t gracefully handle cases !hen the !ecan’t send data to the client as fast as !e’re receiving it from the bac%end" Aore info here"K

ust li%e before, the code that &’ll be describing is hosted on =it>ub as a pro2ect called rsp, for

?)eally 1imple Proxy@" &t’s A&5 licensed, and the version of it &’ll be !al%ing through in this blog post

is as of commit fD78D0bB7" &’ll copy and paste the code that &’m describing into this post any!ay,

so if you’re follo!ing along there’s no need to do any %ind of complicated chec%out"

3efore !e dive into the code, though, it’s !orth tal%ing about epoll a bit"

-ou’ll remember that the code for the server in my last post !ent something li%e this:

while 7reO

http://www.gilesthomas.com/?p=727http://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=640http://linux.die.net/man/4/epollhttp://httpd.apache.org/docs/2.2/programs/ab.htmlhttp://httpd.apache.org/docs/2.2/programs/ab.htmlhttp://httpd.apache.org/docs/2.2/programs/ab.htmlhttp://httpd.apache.org/docs/2.2/programs/ab.htmlhttp://www.gilesthomas.com/?p=831https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/f51950b213c5ae4873d7c59e84f4a696b96859b5http://www.gilesthomas.com/?p=727http://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=640http://linux.die.net/man/4/epollhttp://httpd.apache.org/docs/2.2/programs/ab.htmlhttp://www.gilesthomas.com/?p=831https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/f51950b213c5ae4873d7c59e84f4a696b96859b5


11/39

wait !or a new incomin connection !rom a client

handle the client connection

…!here the code to handle the client connection !as basically:

connect to the 3ackend

read a 3lockNs worth o! st!! !rom the client

send it to the 3ackend

while thereNs still st!! to 3e read !rom the 3ackendO

send it to the client

Fo!, all of those steps to read stu;, or to !ait for incoming connections, !ere bloc%ing calls * !e

made the call, and !hen there !as data for us to process, the call returned" 1o handling multiple

connections !ould have been impossible, as Gsay< !hile !e !ere !aiting for data from one bac%end

!e !ould also have to be !aiting for ne! incoming connections, and perhaps reading from other

incoming connections or bac%ends" We’d be trying to do several things at once"

5hat sounds li%e the %ind of problem threads, or even cooperating processes, !ere made for" 5hat’s

a valid solution, and !as the normal !ay of doing it for a long time" 3ut that’s changed Gat least in

part


12/39

i! itNs an incomin client connectionO

et the !ile descriptor !or the client connectionD add it to the list

connect to the 3ackend

add the 3ackendNs !ile descriptor to the list

else i! itNs data !rom an client connection

send it to the associated 3ackend

else i! itNs data !rom a 3ackend

send it to the associated client connection

…!ith a bit of extra logic to handle closing connections"

It’s worth noting that this updated version can not only process multiple connections

with just a single thread — it can also handle bidirectional communication between the

client and the backend. The previous version read once rom the client, sent the result o

that read down to the backend, and rom then on only sent data rom the backend to the

client. The code above keeps sending stu in both directions, so i the client sends

something ater the initial block o data, while the backend is already replying, then it

gets sent to the backend. This isn’t super!useul or simple "TT# re$uests, but or

persistent connections %like &eb'ockets( it’s essential.

5here have been many system calls that been the ?!ait for an event on the list of things &’m

interested in@ call in 'nix/P(1&.#li%e environments over the years * select and poll, for example *

but they had poor performance as the number of le descriptors got large"

5he popular solution, in +inux at least, is epoll" &t can handle huge numbers of le descriptors !ith

minimal reduction in performance" G5he euivalent in Eree313 and Aac (1 . is %ueue, and

according to Wi%ipedia there’s something similar in Windo!s and 1olaris called ?&/( Completion

Ports?"<

5he rest of this post sho!s the code & !rote to use epoll in a !ay that Ga< ma%es sense to me and

feels li%e it !ill %eep ma%ing sense as & add more stu; to rsp, and Gb< !or%s pretty e$ciently"

ust as before, &’ll explain it by !or%ing through the code" 5here are a bunch of di;erent les no!,

but the main one is still rsp*c, !hich no! has a main routine that starts li%e this:

int mainCint arcD charE arvIF{

i! Carc MK @F {

!print!CstderrD

G=saeO Ps 1nGD

arv0IF;

exitC


13/39

meaningfully#named variables" G1harp#eyed readers !ill have noticed that &’ve updated my code

formatting * &’m no! putting the E to represent a pointer next to the type to !hich it points, !hich

ma%es more sense to me than splitting the type denition !ith a space, and &’ve also discovered

that the C88 standard allo!s you to declare variables any!here inside a function, !hich & thin%

ma%es the code much more readable"<

Fo!, the rst epoll#specic code:

int epoll_!d K epoll_create


14/39

event*data*ptr K handler;

event*events K event_mask;

i! Cepoll_ctlCepoll_!dD &L:++_-7+_'((D handler6!dD JeventF KK


15/39

% void* (, an integer, or one o two dierent types o specifcally!si+ed integers. &hen you

retrieve the value, you have to use the feld name appropriate to the type o thing you

put in there — or eample, i you were to store a -!bit integer in the data using

the u32name and then retrieve it using the ptr variable, the result would be undefned.

%&ell, on a -!bit machine it would probably be a pointer to whatever memory address

was represented by that -!bit integer, but that’s unlikely to be what you wanted.(

&n this case, !e’re using the data pointer inside the union, and !e’re setting it to a pointer toastrct epoll_event_handler" 5his is a structure &’ve created to provide callbac%#li%e functionality

from epoll" +et’s ta%e a loo% * it’s in epollinter!ace*h:

strct epoll_event_handler {

int !d;

void CEhandleFCstrct epoll_event_handlerED intRQ_tF;

voidE closre;

};

1o, an epoll_event_handler stores:

• 5he le descriptor it’s associated !ith

• callbac% function to handle an epoll event !hich ta%es a pointer to

a epoll_event_handlerstructure, and a intRQ_t !hich !ill hold the bitmas% representing the

events that need to be handled

•nd a pointer to something called closre6 basically, a place to store any data the callbac%

function needs to do its 2ob"

)ight" Fo! !e have a function called add_epoll_handler that %no!s ho! to add a le descriptor and

an associated structure to an epoll EO’s list of things it’s interested in so that it’s possible to do a

callbac% !ith data !hen an event happens on the epoll EO" +et’s go bac% to the code in rsp*cthat

!as calling this" >ere it is again:

strct epoll_event_handlerE server_socket_event_handler;

server_socket_event_handler K create_server_socket_handlerCepoll_!dD

server_port_strD

3ackend_addrD

3ackend_port_strF;

add_epoll_handlerCepoll_!dD server_socket_event_handlerD &L:++,.F;

5his presumably no! ma%es sense * !e’re creating a special handler to handle events on the

server soc%et Gthat is, the thing that listens for incoming client connections< and !e’re then adding

that to our epoll EO, !ith an event mas% that says that !e’re interested in hearing from it !hen

there’s something to read on it * that is, a client connection has come in"

+et’s put aside ho! that server soc%et handler !or%s for a moment, and nish !ith rsp*c" 5he next

lines loo% li%e this:


16/39

print!CGtarted* +istenin on port Ps*1nGD server_port_strF;

do_reactor_loopCepoll_!dF;

retrn 0;

}

Pretty simple" We print out our status, then call this do_reactor_loop function, then

return"do_reactor_loop is obviously the interesting bit6 it’s another part of the epoll abstraction layer,

and it basically does the ?!hile 5rue@ loop in the pseudocode above * it !aits for incoming events

on the epoll EO, and !hen they arrive it extracts the appropriate handler, and calls its callbac% !ith

its closure data" +et’s ta%e a loo%, bac% in epollinter!ace*c:

void do_reactor_loopCint epoll_!dF

{

strct epoll_event crrent_epoll_event;

while C


17/39

only accepting one event at a time has one advantage. Imagine that you’re processing

an event on a backend socket’s 67, which tells you that the backend has closed the

connection. 8ou then close the associated client socket, and you ree up the memory or

both the backend and the client socket’s handlers and closures. *losing the client socket

means that you’ll never get any more events on that client socket %it automatically

removes i rom any epoll 67s’ lists that it’s on(. 5ut what i there was already an event

or the client socket in the event array that was returned rom your last callto epoll_wait, and you’d just not got to it yet9 I that happened, then when you did try to

process it, you’d try to get its handler and closure data, which had already been reed.

This would almost certainly cause the server to crash. "andling this kind o situation

would make the code signifcantly more complicated, so I’ve dodged it or now,

especially given that it doesn’t seem to harm the proy’s speed.

1o that’s our reactor loop Gthe name ?reactor@ comes from 5!isted and &’ve doubtless completely

misused the !ord


18/39

!las K !cntlCsocket_!dD >_&7>+D 0F;

i! C!las KK +D !lasF KK


19/39

We create a closure of a special structure type that !ill contain all of the information that our

?there’s an incoming client connection@ callbac% !ill need to do its 2ob, and ll it in appropriately"

strct epoll_event_handlerE reslt K mallocCsi"eo!Cstrct epoll_event_handlerFF;

reslt6!d K server_socket_!d;

reslt6handle K handle_server_socket_event;

reslt6closre K closre;

retrn reslt;

}

…then !e create a strct epoll_event_handler !ith the EO, the handler function, and the closure,

and return it"

5hat’s ho! !e create a server soc%et that can listen for incoming client connections, !hich !hen

added to the event loop that the code in epollinter!ace*c dened, !ill call an appropriate function

!ith the appropriate data"

Fext, let’s loo% at that callbac%" &t’s called handle_server_socket_event, and it’s also

inserver_socket*c"

void handle_server_socket_eventCstrct epoll_event_handlerE sel!D intRQ_t eventsF

{

strct server_socket_event_dataE closre K Cstrct server_socket_event_dataEF sel!

6closre;

We need to be able to extract information from the closure !e set up originally for this handler, so

!e start o; by casting it to the appropriate type" Fext, !e need to accept any incoming

connections" We loop through all of them, accepting them one at a time6 !e don’t %no! up front

ho! many there !ill be to accept so !e 2ust do an innite loop that !e can brea% out of:

int client_socket_!d;

while C


20/39

program !ith an appropriate error message"

} else {

perrorCG-old not acceptGF;

exitC


21/39

add_epoll_handlerCepoll_!dD client_socket_event_handlerD &L:++,. T &L:++%(H=LF;

}

We 2ust create a ne! %ind of handler, one for handling events on client soc%ets, and register it !ith

our epoll loop saying that !e’re interested in events !hen data comes in, and !hen the remote end

of the connection is closed"

(n!ard to the client connection handler code, then create_client_socket_handler is dened

inclient_socket*c, and loo%s li%e this:

strct epoll_event_handlerE create_client_socket_handlerCint client_socket_!dD

int epoll_!dD

charE 3ackend_hostD

charE 3ackend_port_strF

{

make_socket_non_3lockinCclient_socket_!dF;

strct client_socket_event_dataE closre K mallocCsi"eo!Cstrct

client_socket_event_dataFF;


reslt6!d K client_socket_!d;

reslt6handle K handle_client_socket_event;


closre63ackend_handler K connect_to_3ackendCresltD epoll_!dD 3ackend_hostD

3ackend_port_strF;

retrn reslt;

}

5his code should be pretty clear by no!" We ma%e the client soc%et non#bloc%ing, create a closure to

store data for callbac%s relating to it Gin this case, the client handler needs to %no! about the

bac%end so that it can send data to it


22/39

3ackend_socket_event_handler K create_3ackend_socket_handlerC3ackend_socket_!dD

client_handlerF;

add_epoll_handlerCepoll_!dD 3ackend_socket_event_handlerD &L:++,. T &L:++%(H=LF;

5he same pattern as before * create a handler to loo% after that EO, passing in information for the

closure Gin this case, 2ust as the client handler needed to %no! about the bac%end, the bac%end

needs to %no! about this clientere it is:

void handle_client_socket_eventCstrct epoll_event_handlerE sel!D intRQ_t eventsF

{

strct client_socket_event_dataE closre K Cstrct client_socket_event_dataE F sel!

6closre;

char 3!!er=>>&%_,?&I;

int 3ytes_read;

fter our initial setup, !e !or% out !hat to do based on the event bitmas% that !e !ere provided"

Eirstly, if there’s some data coming in, !e read it:

i! Cevents J &L:++,.F {

3ytes_read K readCsel!6!dD 3!!erD =>>&%_,?&F;

5here are t!o possible error cases here" Eirstly, perhaps !e !ere misinformed and there’s no data"

&f that’s the case then !e do nothing"

i! C3ytes_read KK < JJ Cerrno KK &'',. TT errno KK &W:=+(+:-9FF {

retrn;

}

1econdly, perhaps the remote end has closed the connection" We don’t al!ays get an o$cial

?remote hung up@ if this happens, so !e explicitly close the connection if that happens, also closing

our connection to the bac%end"

i! C3ytes_read KK 0 TT 3ytes_read KK


23/39

writeCclosre63ackend_handler6!dD 3!!erD 3ytes_readF;

}

Fo!, the event bitmas% can Gof course< contain multiple events" 1o the follo!ing code might also be

executed for an event that triggered the above code:

i! CCevents J &L:++&%%F T Cevents J &L:++H=LF T Cevents J &L:++%(H=LFF {

close_3ackend_socketCclosre63ackend_handlerF;

close_client_socketCsel!F;

retrn;

}

…or in other !ords, if there’s been some %ind of error or the remote end hung up, !e

unceremoniously close the connection to the bac%end and the client connection itself"

}

nd that’s the sum of our event handling from client connections"

5here’s one interesting but perhaps non#obvious thing happening in that code" -ou’ll remember that

!hen !e !ere handling the ?incoming client connection@ event, !e had to carefully accept every

incoming connection because !e !eren’t going to be informed about it again" &n this handler,

ho!ever, !e read a maximum of =>>&%_,?& bytes Gcurrently 08L


24/39

must be something odd about my code, or deliberate dened behaviour that &’ve 2ust not found the

documentation for" Mither !ay, the thread continuing from that bug report has comments from

people saying that regardless of !hether you’re running edge# or level#triggered, if you !ant to

handle lots of connections then accepting them all in one go is a good idea" 1o &’ll stic% !ith that for

the time being, and if anyone more %no!ledgable than me !ants to clarify things in the comments

then &’d love to hear more

1o, !hat’s left Well, there’s the code to close the client soc%et handler:

void close_client_socketCstrct epoll_event_handlerE sel!F

{

closeCsel!6!dF;

!reeCsel!6closreF;

!reeCsel!F;

}

1imple enough * !e close the soc%et, then free the memory associated !ith the closure and the

handler"

5here’s a little bit of extra complexity here, in ho! !e call this close function from

thehandle_client_socket_event function" &t’s all to do !ith memory management, li%e most nasty

things in C programs" 3ut it’s !orth having a uic% loo% at the bac%end#handling code rst" s you’d

expect, it’s in 3ackend_socket*c, and it probably loo%s rather familiar" We have a function to create a

bac%end handler:

strct epoll_event_handlerE create_3ackend_socket_handlerCint 3ackend_socket_!dD strct epoll_event_handlerE

client_handlerF

{

make_socket_non_3lockinC3ackend_socket_!dF;

strct 3ackend_socket_event_dataE closre K mallocCsi"eo!Cstrct

3ackend_socket_event_dataFF;

closre6client_handler K client_handler;


reslt6!d K 3ackend_socket_!d;

reslt6handle K handle_3ackend_socket_event;


retrn reslt;

}

Mssentially the same as its client soc%et euivalent, but it doesn’t need to create a client handler for

its closure because one !as passed in"

5here’s a function to handle bac%end events, !hich also !on’t have many surprises:


25/39

void handle_3ackend_socket_eventCstrct epoll_event_handlerE sel!D intRQ_t eventsF

{

strct 3ackend_socket_event_dataE closre K Cstrct 3ackend_socket_event_dataEF sel!

6closre;

char 3!!er=>>&%_,?&I;

int 3ytes_read;




retrn;

}



26/39

mentioned in the close handling"

>ere’s the problem: !hen a connection is closed, !e need to free the memory allocated to

theepoll_event_handler structure and its associated closure" 1o

our handle_client_socket_eventfunction, !hich is the one notied !hen the remote end is closed,

needs to have access to the handler structure in order to close it" &f you !ere !ondering !hy the

epoll interface abstraction passes the handler structure into the callbac% function Ginstead of 2ust

the closure, !hich !ould be more traditional for a callbac% interface li%e this< then there’s theexplanation * so that it can be freed !hen the connection closes"

3ut, you might as%, !hy don’t !e 2ust put the memory management for the handler structure in the

epoll event loop, do_reactor_loop When an event comes in, !e could handle it as normal and then if

the event said that the connection had closed, !e could free the handler’s memory" &ndeed, !e

could even handle more obscure cases * perhaps the handler could returns a value saying ?&’m

done, you can free my handler@"

3ut it doesn’t !or%, because !e’re not only closing the connection for the EO the handler is

handling" When a client connection closes, !e need to close the bac%end, and vice versa" Fo!,!hen the remote end closes a connection, !e get an event from epoll" 3ut if !e close it ourselves,

then !e don’t" Eor most normal use, that doesn’t matter * after all, !e 2ust closed it, so !e should

%no! that !e’ve done so and tidy up appropriately"

3ut !hen in a client connection handler !e’re told that the remote end has disconnected, !e need

to not only free the client connection Gand thus free its handler and its closureere’s a lin% to the post"

Writing a reverse proxy/loadbalancer from the ground up in C, part : +ua#based

conguration

5his is the third step along my road to building a simple C#based reverse proxy/loadbalancer so that

& can understand ho! nginx/(pen)esty !or%s * more bac%ground here" >ere’s a lin% to the rst

part, !here & sho!ed the basic net!or%ing code reuired to !rite a proxy that could handle one

incoming connection at a time and connect it !ith a single bac%end, and to the second part, !here &

http://www.gilesthomas.com/?p=818http://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=727http://www.gilesthomas.com/?p=818http://wiki.nginx.org/Mainhttp://openresty.org/http://www.gilesthomas.com/?p=634http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=640http://www.gilesthomas.com/?p=727


27/39

added the code to handle multiple connections by using epoll"

5his post is much shorter than the last one" & !anted to ma%e the minimum changes to introduce

some +ua#based scripting * specically, & !anted to %eep the same proxy !ith the same behaviour,

and 2ust move the stu; that !as being congured via command#line parameters into a +ua script,

so that 2ust the name of that script !ould be specied on the command line" &t !as really easy :#< *

but obviously & may have got it !rong, so as ever, any comments and corrections !ould be much

appreciated"

ust li%e before, the code that &’ll be describing is hosted on =it>ub as a pro2ect called rsp, for

?)eally 1imple Proxy@" &t’s A&5 licensed, and the version of it &’ll be !al%ing through in this blog post

is as of commit L7DdB0d8a0" &’ll copy and paste the code that &’m describing into this post any!ay,

so if you’re follo!ing along there’s no need to do any %ind of complicated chec%out"

5he rst thing & should probably explain, though, is !hy & pic%ed +ua for this" &’m founder of a

company called Pythonny!here, so !hy not Python Well, partly it’s a %ind of cargo#cult thing"

nginx Gand particularly (pen)esty< use +ua for all of their scripting, so & probably should tooGespecially if that’s !hat &’m trying to emulate"

nother reason is that +ua is really, really easy to integrate into C programs * it !as one of the

design goals" Python is reasonably easy to embed, but as soon as you !ant to get ob2ects out, you

have to do a lot of memory management and it can get hairy G2ust scroll do!n that page a bit to see

!hat & mean+' OK WlDrpathD/sr/local/li3

+, OK llaitV*<

***

http://linux.die.net/man/4/epollhttp://www.lua.org/https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/615d20d9a07fc18e3d4af63bd8e7e69560347094http://www.pythonanywhere.com/http://docs.python.org/2/extending/embedding.htmlhttp://blog.gmarceau.qc.ca/2009/05/speed-size-and-dependability-of.htmlhttp://luajit.org/http://linux.die.net/man/4/epollhttp://www.lua.org/https://github.com/gpjt/rsphttps://github.com/gpjt/rsp/commit/615d20d9a07fc18e3d4af63bd8e7e69560347094http://www.pythonanywhere.com/http://docs.python.org/2/extending/embedding.htmlhttp://blog.gmarceau.qc.ca/2009/05/speed-size-and-dependability-of.htmlhttp://luajit.org/


28/39

P*oO P*c $CH&'(&%_>,+&F

$C--F c Wall stdKnAA $C,.-+=(&_(,%F $5

***

rspO $C:_>,+&F

$C--F o $Z $C:_>,+&F $C+(>+'F $C+,F

Fote the use of WlDrpathD/sr/local/li3 in the +(>+' instead of the more traditional

+/sr/local/li3" 5his ba%es the location of the library into the rsp executable, so that it %no!s to

loo% in /sr/local/li3 at runtime rather than relying on us al!ays setting +(_+,%'%Y_L'7H to tell it

!here to nd the li3laitV*


29/39

charE 3ackend_addr K et_con!i_optC+D G3ackend'ddressGF;

charE 3ackend_port_str K et_con!i_optC+D G3ackendLortGF;

"""and et_con!i_opt loo%s li%e this:

charE et_con!i_optCla_tateE +D charE nameF {

la_etlo3alC+D nameF;

i! CMla_isstrinC+D 55P headers so that !e can delegate incoming

client connections to bac%ends based on their host header Gand perhaps other things


30/39

void handle_3ackend_socket_eventCstrct epoll_event_handlerE sel!D intRQ_t eventsF

{

strct 3ackend_socket_event_dataE closre K Cstrct 3ackend_socket_event_dataEF sel!

6closre;

char 3!!er=>>&%_,?&I;

int 3ytes_read;




retrn;

}



31/39

1o that means that even for this simple example of an epoll#based proxy to !or% properly, !e need

to do some %ind of bu;ering in the server to handle cases !here !e’re getting stu; from the

bac%end faster than !e can send it to the client" nd possibly vice versa" &t’s possible to get epoll

events on an EO !hen it’s ready to accept output, so that’s probably the !ay to go * but it !ill

need a bit of restructuring" 1o the next step !ill be to implement that, rather than the multiple#

bac%end handling stu; & !as planning"

5his is excellent" Fo! & %no! a little more about !hy !riting something li%e nginx is hard, and havea vague idea of !hy & sometimes see stu; in its logs along the lines of an pstream response is

3!!ered to a temporary !ile" Which is entirely !hy & started !riting this stu; in the rst place :#<

>ere’s a run#through of the code & had to !rite to x the bug"

Writing a reverse proxy/loadbalancer from the ground up in C, pause to regroup:

xed it

&t too% a bit of !or%, but the bug is xed: rsp no! handles correctly the case !hen it can’t !rite as

much as it !ants to the client side" & think this is enough for it to properly !or% as a front#end for

this !ebsite, so it’s installed and running here" &f you’re reading this Gand &’ve not had to s!itch it o;

in the meantime< then the pages you’re reading !ere served over rsp" Which is very pleasing :#<

5he code needs a bit of refactoring before & can present it, and the same bug still exists on the

communicating#to#bac%ends side G!hich is one of the reasons it needs refactoring * this is

something & should have been able to x in one place only< so &’ll do that over the coming days, and

then do another post"

Writing a reverse proxy/loadbalancer from the ground up in C, part : Oealing

!ith slo! !rites to the net!or%

5his is the fourth step along my road to building a simple C#based reverse proxy/loadbalancer, rsp,

so that & can understand ho! nginx/(pen)esty !or%s * more bac%ground here" >ere are lin%s

to the rst part, !here & sho!ed the basic net!or%ing code reuired to !rite a proxy that could

handle one incoming connection at a time and connect it !ith a single bac%end, to the second part,

!here & added the code to handle multiple connections by using epoll, and to the third part, !here &

started using +ua to congure the proxy"

5his post !as !as unplanned6 it sho!s ho! & xed a bug that & discovered !hen & rst tried to use

rsp to act as a reverse proxy in front of this blog" 5he bug is xed, and you’re no! reading this via

rsp" 5he problem !as that !hen the connection from a bro!ser to the proxy !as slo!er than the

connection from the proxy to the bac%end Gthat is, most of the time


32/39

1o, the problem !as that !hen !e !rote to the le descriptor that represented the connection to

the client, !e ignored the return value:

writeCclosre6client_handler6!dD 3!!erD 3ytes_readF;

&f the connection to the client is bac%ed up, this call can:

•

)eturn a value !ith a number of bytes that’s less than the number !e as%ed it to !rite, saying?that’s all & !as able to !rite right no!@ or

•)eturn #7, !ith an error code in errno of &'',. or &W:=+(+:-9, meaning ?& !asn’t able to !rite

anything@

'nder those circumstances, all !e can really do is stash a!ay the data that !e’ve 2ust received

from the bac%end in a bu;er some!here, and !ait until the client is able to receive data again" 1o,

ho! do !e nd out !hen the client is ready for us We use epoll"

'ntil no!, all of our epoll event handlers have been listening for events related to reading

stu;:&L:++,., for !hen there’s data to read, and &L:++%(H=L, for !hen the connection has beenclosed" 3ut there’s also an &L:++:=7, !hich is emitted !hen the connection is ready for !riting"

1o in an ideal !orld, !e’d 2ust change the code to bu;er data !hen it’s unable to !rite stu;, and

!hen !e received an &L:++:=7 event !e’d 2ust send do!n !hatever’s in the bu;er, and continue"

3ut it’s not that simple"

)emember, until no! !e’ve been using epoll in level#triggered mode" Eor our reading#related

events, that means that if there’s something there for us to read, !e get an event" &f !e don’t read

all of it, that’s ne * because !hether or not !e get an event is based on the ?level@ of stu; that’s

!aiting, !e’ll get another event shortly to tell us that there’s still stu; to read"3ut if you as% for output#related events in level#triggered mode, you !ill get an event every time

you call epoll_wait if it’s possible to !rite to the le descriptor" nd for a le descriptor, being

!ritable is pretty much the default state * so almost every run around the loop triggers an event"

5hat happens tens of thousands of times a second, !hich is really !asteful6 your code is being told

?hey, you can !rite to this soc%et@, and it needs to chec% if there’s anything that needs !riting,

discover that there isn’t, and return" CP' usage goes to 700S for no good reason"

1o, to do pretty much anything sensible !ith &L:++:=7, !e have to s!itch a!ay from level#triggered

epoll, to edge#triggered" Mdge#triggering means that you only get told !hen something changes" &f

there’s data to read, you’re told once * and it’s up to you to read it all, because if you don’t you

!on’t be told again" 3ut on the other hand, if a soc%et that !as non#!ritable unbloc%s and you can

!rite to it again, you’ll be told about it once, !hich is !hat !e !ant"

Fo!, it !ould be great if !e could register t!o event handlers for our connection to the client * a

level#triggered one for reading, and an edge#triggered one for !riting" 3ut unfortunately that’s not

possible6 both reading and !riting happen to the same le descriptor" nd if you try to add the

same le descriptor to an epoll instance t!ice, you get an error" 1o either everything has to be

level#triggered, or everything has to be edge#triggered"

5o summarise:

•&n order to x the bug, !e need to detect !hen a client connection is temporarily un!ritable

because of net!or% speed, and start bu;ering the data" When the connection becomes


33/39

!ritable again, !e can send the contents of the bu;er to it, and then continue as normal"

•&n order to detect !hen the connection becomes !riteable again, !e need to listen

for&L:++:=7 events"

•&n order to !or% !ith &L:++:=7 events, !e need to s!itch to edge#triggered mode"

1o let’s ta%e a loo% at the changes for s!itching to edge#triggered mode rst" 5hey’re actually

pretty simple" Eirstly, !e need to add MP(++M5, the Nag that says ?this descriptor should be dealt

!ith in edge#triggered mode@, to the Nags !e pass do!n to our epoll_ctl function, !hich is !rapped

by add_epoll_wrapper" 1o, previously, !e did this to add the client connection:

add_epoll_handlerCepoll_!dD client_socket_event_handlerD &L:++,. T &L:++%(H=LF;

nd no! !e do this instead:

add_epoll_handlerCepoll_!dD client_socket_event_handlerD &L:++,. T &L:++%(H=L T

&L:++&7F;

5he other change is to ma%e sure that !e al!ays read everything !hen !e’re told that there’s stu;

to read" Previously, !e handled &L:++,. events li%e this:



retrn;

}

i! C3ytes_read KK 0 TT 3ytes_read KK >&%_,?& bytes, do some error handling, then !rite the results to the client"

Fo! that !e’re in edge#triggered mode, !e need to ma%e sure that !e read everything, so !e

update the code to use a simple loop:

while CC3ytes_read K readCsel!6!dD 3!!erD =>>&%_,?&FF MK < JJ 3ytes_read MK

0F {


retrn;

}



34/39

close_client_socketCclosre6client_handlerF;

close_3ackend_socketCsel!F;

retrn;

}


}

&t’s basically the same code, !ith a !hile loop !rapped around the outside" 1imple

Fo!, the code still has the bug !e’re trying to x * it still ignores the results of the call to write *

so let’s sort that out" >o! do !e do the bu;ering

5he rst thing is that !e !ant each of our connection handler ob2ects * the structures that

represent the connection to the bac%end and the connection to the client * to o!n its o!n bu;er"

>istorically, !hen !e !rote to the client connect in our event handler for the bac%end connection,

!e used its le descriptor directly:


Fo!, the client handler needs to ta%e a bit more control over the process, so it ma%es sense to

move this into a function that lives !ith it:

write_to_clientCclosre6client_handlerD 3!!erD 3ytes_readF;

1o !hat does that loo% li%e >ere’s ho! it starts:

void write_to_clientCstrct epoll_event_handlerE sel!D charE dataD int lenF

{


6closre;

int written K 0;

i! Cclosre6write_3!!er KK .=++F {

written K writeCsel!6!dD dataD lenF;

i! Cwritten KK lenF {

retrn;

}

}

1o, !e get hold of the data about our client soc%et, and then !e chec% if it has a !rite bu;er * that

is, is there anything to be !ritten ueued up" &f there’s not, !e try !riting our ne! data straight to

the connection * basically, !hat !e used to do" &f it !or%s, and !e’re told that all of stu; !e

!anted to be !ritten !as !ritten, then !e’re done" 5hat’s the simple case, covering the situation

!hen the client is accepting data as fast as !e’re sending it" Fo! things get a little more


35/39

complicated:

i! Cwritten KK ere, !e !or% out ho! much of our data !as un!ritten Gremember, the call to !rite !ill return the

number of bytes that !ere actually !ritten, so if things are clogged up then !e could potentially get

#7 and an appropriate error, !hich is the case !e’ve already handled, or 2ust a number less than the

number of bytes !e told it to !rite


36/39

data_3!!er_entryE new_entryF

{

strct data_3!!er_entryE last_3!!er_entry;


closre6write_3!!er K new_entry;

} else {

!or Clast_3!!er_entryKclosre6write_3!!er; last_3!!er_entry6next MK .=++;

last_3!!er_entryKlast_3!!er_entry6nextF

;

last_3!!er_entry6next K new_entry;

}

}

Eairly simple code, & !on’t go through it"

)ight, !hat about that is_close_messae eld in the bu;er entries Well, previously, !hen a bac%end

connection !as closed, !e 2ust closed the client connection right a!ay" 3ut no! !e can’t * there

might still be stu; bu;ered up that needs to be !ritten to the client" 1o !e simply regard the

reuest to close the soc%et as another thing !e should put into the bu;er, using that Nag" 5he old

code to close the client connection Gas called by the bac%end connection< loo%ed li%e this:


{

closeCsel!6!dF;

!reeCsel!6closreF;

!reeCsel!F;

}

Fo!, instead, !e do this:


{


6closre;


really_close_client_socketCsel!F;

1o, if there’s nothing in the bu;er !e call this

ne! really_close_client_socket function"really_close_client_socket 2ust has the code from the

original close_client_socket" &f there is something in the bu;er, !e do this:

} else { strct data_3!!er_entryE new_entry K mallocCsi"eo!Cstrct data_3!!er_entryFF;

new_entry6is_close_messae K


37/39

new_entry6next K .=++;

add_write_3!!er_entryCclosreD new_entryF;

}

}

We 2ust create a ne! bu;er entry to represent the soc%et close, and put it on the bu;er"

1o that’s ho! !e bu;er stu;" 5he next thing to loo% at is the code that, !hen the connection

becomes available for !riting, drains that bu;er so that !e can get bac% on trac%" (bviously, one

thing !e need to do is start listening for &L:++:=7 events, so


&L:++&7F;

becomes


&L:++&7 T &L:++:=7F;

3ut eually obviously, the interesting stu; is in handle_client_socket_event" We get a ne! bit of code

to handle our ne! event, in cases !here !e have something in the !rite bu;er"

i! CCevents J &L:++:=7F JJ Cclosre6write_3!!er MK .=++FF {

Eirst, a bit of setup:

int written;

int to_write;

strct data_3!!er_entryE temp;

nd no! !e try to iterate through the complete bu;er until it’s empty:

while Cclosre6write_3!!er MK .=++F {

Fo!, if !e encounter a message saying ?close the connection@, then !e do so Gusing the

samereally_close_client_socket function as !e did in the other function


38/39

retrn;

}

5here could, in theory, be other events that !e’ve been as%ed to handle at this point * but !e’ve

2ust shut do!n the soc%et !e’re managing, so there’s not much !e could do !ith them any!ay" We

could also, in theory, have other items on the write_3!!er list after the close, the memory for !hich

!ould be lea%ed by this code6 & don’t thin% that’s possible, though, so &’m not going to !orry about it

for no!4

1o, !e’ve handled soc%et closes" 5hat means that the bu;er entry !e’re dealing !ith contains

something to !rite to the client soc%et" Fo!, it’s possible that !e tried to !rite this bu;er item

before, and !ere only able to !rite part of it" 5hat’s !hat the crrent_o!!set eld in the bu;er

entries is for6 !e initialised it to Hero, but if a !rite fails to !rite everything !e use it to %eep trac% of

ho! far !e’ve got so far" 1o the number of bytes !e need to !rite is the total number of bytes in

this bu;er entry minus the number of bytes from the entry that have already been !ritten:

to_write K closre6write_3!!er6len closre6write_3!!er

6crrent_o!!set;

1o let’s try and !rite them:

written K writeCsel!6!dD closre6write_3!!er6data # closre

6write_3!!er6crrent_o!!setD to_writeF;

Fo!, our normal error handling to cover cases !here !e !eren’t able to !rite everything:

i! Cwritten MK to_writeF {

&f !e got an error bac%, !e either bomb out if it’s not something !e’re expecting, or !e set the

number of bytes !ritten to Hero if it’s 2ust another ?this !ould bloc%@ message:

i! Cwritten KK


39/39

closre6write_3!!er6crrent_o!!set #K written;

3reak;

1o no! !e’re in the code to handle the case !here all of the bu;er entry !as successfully !ritten" &f

that’s the case, !e 2ust need to free up this bu;er entry, and move on to the next one so that !e

can run through this loop again:

} else {

temp K closre6write_3!!er;

closre6write_3!!er K closre6write_3!!er6next;

!reeCtemp6dataF;

!reeCtempF;

}

}

}

nd that’s it 5hat is the total set of changes to the codebase reuired to handle bu;ering of data

!hen the client connection bac%s up and !e %eep receiving stu; from the bac%end"

5here are a couple of problems !ith the code as it stands no!: rstly, it’s getting a bit complicated6

the handle_client_socket_event method is getting a bit long" 1econdly, !e’re not handling the case

!hen the connection to the bac%end bac%s up and !e have stu; from the client * signicantly less

li%ely to happen, but potentially possible"

& thin% the next thing !e need to do is a thorough refactoring6 there’s far too much code duplicationbet!een the bac%end and the client soc%et handlers" Aa%ing those t!o more li%e each other so that

they can share this bu;ering code is !hat &’ll post about next"

really simple proxy

Documents