Download - Tips on High Performance Server Programming
![Page 1: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/1.jpg)
Tips on High Performance Server Programming
Joshua Zhu
June 9, 2009
![Page 2: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/2.jpg)
Agenda
• Fundamentals
• Best practices and common tricks
• Design issues• Tools and resources
![Page 3: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/3.jpg)
Key Rules
• Do NOT block• Avoid excessive system calls• Cache/preprocess as much as possible• Use efficient algorithms and data structures• Threads are usually a bad idea• Separate the I/O code from the business-logic
code• Keep the most heavily used data near the CPU• Tune against the bottleneck
![Page 4: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/4.jpg)
I/O Models
• Blocking
• Non-blocking
• I/O multiplexing• Signal-driven I/O
• Asynchronous I/O
![Page 5: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/5.jpg)
I/O Multiplexing
• Select
• Poll
• /dev/poll• Epoll/kqueue
– Level triggered– Edge triggered
![Page 6: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/6.jpg)
I/O Strategies
• 1 thread, non-blocking, level-triggered
• 1 thread, non-blocking, edge-triggered
• 1 thread, AIO• 1 thread per client
• Build the server code into the kernel
![Page 7: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/7.jpg)
Let’s Face the C10K Problem
• 10000 connections– CPU/memory usage per connection
• 10000 hits/sec– 10 us per hit
• Instruction (ns)• System call (us)
![Page 8: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/8.jpg)
Agenda
• Fundamentals
• Best practices and common tricks
• Design issues• Tools and resources
![Page 9: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/9.jpg)
The Event-driven Modelwhile (true) {
for t in run_tasks:t.handler();
update_time(&now);timeout = ETERNITY;
for t in wait_tasks: /* sorted already */if (t.time <= now) {
t.timeout_handler();} else {
timeout = t.time - now;break;
}
nevents = poll_function(events, timeout);for i in nevents:
task t;
if (events[i].type == READ) {t.handler = read_handler;
} else (events[i].type == WRITE) {t.handler = write_handler;
}
run_tasks_add(t);}
![Page 10: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/10.jpg)
Scheduler
• Task scheduling– Run queue– Wait queue
• Timers– Time jumps– The next timeout
• Pass to select()/poll()/epoll_wait()...
– Data structures• Red-black tree • Min-heap• Timer wheel
![Page 11: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/11.jpg)
Unify Multiple Event Sources
• I/O events
• Timer events
• Signals/threads– Use a pipe or socketpair
• Register the read end in the event loop• Notify the event loop by writing to the write end
![Page 12: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/12.jpg)
Buffer Management
• Fixed size or not
• R/W pointers operation– Producer/consumer
• Circular buffer
• Single buffering
• Buffer chain
![Page 13: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/13.jpg)
Memory
• Memory fragment– Memory pool
• Fixed size– MRU
• Non-fixed size
– Slab allocator
• Memory operations are expensive– Allocation/free– Data copies
• Pre-allocation/static-allocation– Array
![Page 14: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/14.jpg)
Strings
• Gotchas– Strlen()
• Sizeof() - 1
– Strncpy()• Strlcpy()
– Vsnprintf()– …
• Algorithms– KMP/BM– AC/WM– …
![Page 15: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/15.jpg)
Caching
• Time– Reduce the number of gettimeofday() calls– Time resolution
• ms or sec?
• Content– Files
• In-memory buffering
– Database• Memcached
• Preprocessing
![Page 16: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/16.jpg)
Accept() Strategies
• Accept-limit– Multi-accept– Accept-mutex
• The thundering herd problem– Polling functions– Kernels
– The amount of processes/threads
![Page 17: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/17.jpg)
Concurrency
• Threads– Pros
• Utilize all the CPUs• True CPU concurrency
– Cons• Context switches• Locks hurt performance
– Deadlocks– Starvation– Race conditions
• Error prone and hard to debug• Hard limits of operating systems
![Page 18: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/18.jpg)
Concurrency (cont’d)
• Categorize your service– CPU-bound– I/O-bound
• Models– Thread pool– Process pool– SEDA– Master/workers
• CPU affinity
![Page 19: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/19.jpg)
Advanced I/O Functions
• Gather read, scatter write– Readv/writev
• Sendfile• Mmap
• Splice and tee
![Page 20: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/20.jpg)
Take Advantage of TCP/IP Options
• TCP/IP options– SO_REUSEADDR – SO_RCVBUF/SO_SNDBUF
– TCP_CORK– TCP_NODELAY– TCP_DEFER_ACCEPT
– …
![Page 21: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/21.jpg)
Misc.
• Byte order– Little-endian– Big-endian
• Max open file number– Ulimit/setrlimit
![Page 22: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/22.jpg)
Agenda
• Fundamentals
• Best practices and common tricks
• Design issues• Tools and resources
![Page 23: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/23.jpg)
Protocol
• Readability– Binary– Textual
• Compatibility• Security
– Heart beat– Invalid data
• The state machine (FSM)– Granularity– Traceability
![Page 24: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/24.jpg)
Security
• Timeouts
• Bad data
• Buffer overflow• DOS attack
• Encryption
• Privilege
• Working directory
![Page 25: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/25.jpg)
Flexibility
• Modularize– Modules– Plugins
• Scriptable– Lua
• Hot code update• HA
![Page 26: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/26.jpg)
Introspection
• Statistics– SNMP
• Debugable• Tunable
• Logs
![Page 27: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/27.jpg)
Patterns
• Reactor
• Proactor
• Half-Sync/Half-Async• Leader/Followers
• …
![Page 28: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/28.jpg)
Choose a Suitable Language
• C
• C++
• Java• Erlang
• …
![Page 29: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/29.jpg)
The Devil Hides in the Details
• Find out the shortest plank in the bucket– Profile/benchmark
• Take care of every line of your code!– Return values– Error code
• EINTR• SIGPIPE• …
• Tune your operating system if necessary– /proc
• Make sure hardware is not the bottleneck– Network cards– Disk– …
![Page 30: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/30.jpg)
Agenda
• Fundamentals
• Best practices and common tricks
• Design issues• Tools and resources
![Page 31: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/31.jpg)
Sharpen Your Tools
• Ping• Traceroute• Netstat• Lsof• Strace• Nc• Wireshark• Gprof• Valgrind• Vmstat/iostat/dstat• Dot/gnuplot
![Page 32: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/32.jpg)
Understand TCP/IP Well
• Three-way handshake
• Connection tear-down
• TCP state transition– TIME_WAIT
– CLOSE_WAIT– …
![Page 33: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/33.jpg)
Learn By Reading Code
• Frameworks– ACE– Libevent/libev– Boost::asio– …
• Servers– HAProxy– Nginx– Lighttpd– Memcached– MySQL Proxy– …
![Page 34: Tips on High Performance Server Programming](https://reader034.vdocuments.us/reader034/viewer/2022051412/54b79ef74a79591d4a8b48e7/html5/thumbnails/34.jpg)
Recommended Books
• Advanced Programming in the UNIX Environment
• UNIX Network Programming• TCP/IP Illustrated• Effective TCP/IP Programming• Pattern-Oriented Software Architecture• C++ Network Programming• Programming With POSIX Threads• Introduction to Algorithms