Optimizing your cloud for millions of connections: A Case StudyZbyněk Šlajchrt

Overall architecture

DeliveryRate vs. Cost

• Max(DeliveryRate)• Min(Cost)

• DeliveryRate ~ Servers*BandwithPerServer• Servers = ActiveClients/MaxConsPerServer• Cost = ServerCost + NetworkCost• ServerCost = CostPerServer*Servers• CostPerServer ~ 1/Servers• NetworkCost ~ MinimalDeliveryRate

Pull vs. Push

• Pull+ Existing distribution technology, simple

- Danger of SYN flood, minimal control over connected clients

• Push+ Better control over clients, kept-alive connections => less SYNs

- Unproven technology, more complex- How many connections can we keep on one

server? Less than 1M => Pull, otherwise Push

System Configuration

• CentOS 6• Max open file descriptors (startApp.sh)

ulimit -n 3000000

• System wide settings for file descriptors (/etc/sysctl.conf)fs.nr_open = 6291456

fs.file-max = 6291456

• System wide limit for open files per user (/etc/security/limits.conf)* hard nofile 6291456

Technologies

Netty

• Asynchronous event-driven network application framework written in Java by Trustin Lee

• Very good performance thanks to NIO and underlying epoll or kqueue kernel functions

• Easy programming model

Google Protocol Buffers

• Mechanism for serializing structured data• Language neutral• Platform neutral• Java, C++, Python• Very easy to grasp• Suitable for specifications (SRS)

Components

Key components

• Comet Session Handler• Distributor• Commands Queue Store• Bandwidth Limiter• Client module

Comet Session Handler

• Netty handler managing comet sessions• Creates, registers and destroys comet

sessions• Decodes command results from HTTP

requests• Encodes commands as HTTP responses

Distributor

• Responsible for continuous delivery of commands to clients (scheduled thread)

• Loops over all currently connected sessions

• For each session selects commands to be sent

• Uses client’s Queue Status held in session

Commands Queue Store

• Storage for commands to be distributed• Populated by the Viruslab• Commands are stored in one or more queues• Every queue has a priority ordinal (zero-based)• Client holds and sends the Queue Status

– List of last command IDs for every queue (used)– Serves for selecting command for a client by

Distributor

Command Scopes

• Priority 0 – request scope– Not kept in the client’s Queue Status =>

Command with the same ID can be delivered multiple times– Similar to JMS Topic (publish-subscribe) – commands are

distributed to the currently connected only• Priority 1 – session scope

– the Queue Status until the application is running => No command is delivered twice as long as the application runs

• Priority >1 – persistent scope– Last cmd IDs are kept persistently on the client– No command is sent twice unless the client is reinstalled

Bandwidth Limiter

• Protects kernel’s socket memory from exhaustion

• In case of big packets the distribution rate may exceed the bitrate of the NIC

• Limiter continuously computes the distribution rate and compares it with the predefined max bitrate

• Auto-regulating algorithm

Client side

• Command interpreter• DLL module written in C • Integrated with Avast AV• CURL library

Monitoring

JMX Aggregator

• From a defined set of servers – Aggregates read-only attributes into one

tabular JMX attribute– Corresponding read-write attributes on

servers are represented by one attribute on the aggregator

Aggregated read-only JMX attribute example:

HeapMemoryUsage.used[Node1]=18GHeapMemoryUsage.used[Node2]=16GHeapMemoryUsage.used[Node3]=19G

Aggregated read-write attributeexample:

MaxThroughput = 40000

Zenoss

• Network management platform• Based on Zope app server (Python)• GPL v2• Supports JMX• Nice web interface• Maintains history

SLOGAN

• Analyzer of log streams• Command line tool• SQL-like syntax (filtering, grouping,

sorting)• Events are POJOs• Can be represented in Google Protobuf

(very handy for Streaming Updates)• Events are serialized and appended to a

log file

SLOGAN - Examples

Usage:tail –f app.log | slogan <query>

Query examples:

tstamp etype

tstamp –f etype==‘NOP’

tstamp max(tstamp)-min(tstamp) –g reqID

H2 Database

• Used as the storage for the commands• Very fast in-memory database• Both embedded and server modes• JDBC API• Nice web-based console

Q&A

Thanks for your listening!

slajchrt@avast.com

Title

• Text

Title

• Text