Reactor patternEvent handling pattern

Dejan PekterSoftware Development Engineer II

Nordeus and me

● Over 150 employees● Top Eleven

○ ~ 12 000 000 MAU○ ~ 5 000 000 DAU○ Cross-platform pioneer ○ Most popular online sports game

● Backend Software Development Engineer○ 250+ servers○ 10+ types○ 70.000 requests / minute○ Every fail is expected

Goal

● Share knowledge on Reactor design pattern used on our servers

● In example of server that is ○ Highly concurrent○ Highly responsive○ Well designed

● Consider different solutions○ Incrementally find solution○ Pros○ Cons

Prerequisites

● Design pattern ● Thread● Context switch● Request● Response

Easy solution

● Wait for clients to connect● For every accepted client

○ Create thread which will take responsibility over that client

Easy solution - conclusion

● Good:○ Intuitive○ Readable code○ Easy to understand

● Bad:○ 1000 users ~ 1000 threads○ Memory overhead○ Context switch overhead○ Create/kill overhead

Improved solution

● Create thread pool with N threads● Accept client

○ Pass client to thread pool ○ Thread pool assigns thread to that client

Improved solution - conclusion

● Good:○ Constrained number of threads○ Threads are reused

● Bad:○ Harder to implement○ Input/output is mixed with business logic○ Possibility of starvation

Basic nuclear reactor theory

NO! Re-act!

How would you react on good joke (event) that I can’t really tell?

FunnyThingHandler

How would you react onLOGIN_OR_REGISTER

You would not!Server would.

Event granulation

CLIENT MESSAGE

Reactor pattern

Reactor

Events

Callbacks

Event handler

EventHandler

+ handleEvent()+ getEventId()

LoginHandler

+ handleEvent()+ getEventId()

RegisterHandler

+ handleEvent()+ getEventId()

Reactor detailed design

EventHandler

+ handleEvent()+ getEventId()

LoginHandler

+ handleEvent()+ getEventId()

RegisterHandler

+ handleEvent()+ getEventId()

Server

Reactor

+ handleEvents()+ registerHandler()+ removeHandler()

dispatches 1 *

creates

Communicator

+ getEvents()

1

1

Reactor phases

● Initialize phase○ Registration of event handlers

● Event handling phase○ Gather all ready messages○ Process all messages

■ Get requests■ Identify event handler■ Execute callback

Reactor - Sequence Diagram

Main Program Event Handler Reactor

EventsregisterHandler()

instantiateHandler()

handleEvents()

handleEvent()

1) Initialize phase

2) Event handling phase

concreteHandler

event

Reactor solution - conclusion

● Good○ Logic is separated○ Easy to add new event handler○ Granulation to message level

● Bad○ Harder to debug○ Not intuitive○ Single threaded

Our improvements

● One reactor is not enough N reactors

● Separate logic even more○ Server - single thread

■ Accept clients ■ Dispatch to reactor by round robin

○ Reactor■ IO responsibility■ Build application level messages■ Find event handler■ send to dispatcher

○ Dispatcher - thread pool■ Execute callback

Our improvements - conclusion

● Good○ Responsibilities are fully separated○ 70.000 requests per minute in production○ Tested even for order of magnitude more○ Good design

● Bad○ Even harder debugging

Conclusion

● Reactor pattern○ Great maintainability○ Forget about how messages arrive, they just do○ Single threaded by definition○ Multithreaded in production

● Business○ Invest time and money in good solution○ You get new stuff almost for free

Remember...

● Don’t reinvent the wheel● Got the wheel, but you need a plane

○ think○ improve○ build it yourself

Don’t forget to...WORK AND PLAY

Thanks for your attention!Questions?