ac2003performance

8/2/2019 ac2003performance

1/82

Scalable Apache for Beginners

Aaron [email protected]


2/82


3/82

Measuring Performance

What is Performance?


4/82

How do we measure

performance? Benchmarks

Requests per Second

Bandwidth

Latency

Concurrency (Scalability)


5/82

Real-world Scenarios

Can benchmarks tell us how it will

perform in the real world?


6/82

What makes a good Web Server?

Correctness

Reliability

Scalability

Stability

Speed


7/82

Correctness

Does it conform to the HTTP specification?

Does it work with every browser?

Does it handle erroneous input gracefully?


8/82

Reliability

Can you sleep at night?

Are you being paged during dinner?

It is an appliance?


9/82

Scalability

Does it handle nominal load?

Have you been Slashdotted?

And did you survive?

What is your peak load?


10/82

Speed (Latency)

Does itfeel fast?

Do pages snap in quickly?

Do users often reload pages?


11/82

Apache the General PurposeWebserver

Apache developers strive for

correctness first, andspeed second.


12/82

Apache 1.3

Fast enough for most sites

Particularly on 1 and 2 CPU systems.


13/82

Apache 2.0

Adds more features

filters

threads

portability

(has excellent Windows support)

Scales to much higher loads.


14/82


15/82

Apache HTTP Server

Architecture Overview


16/82

Classic Prefork Model

Apache 1.3, and

Apache 2.0 Prefork

Many Children

Each child handles one

connection at a time. Child

Parent

ChildChild

(100s)


17/82

Multithreaded Worker Model

Apache 2.0 Worker

Few Children

Each child handles many

concurrent connections.Child

Parent

ChildChild

(10s)

10s of threads


18/82

Dynamic Content: Modules

Extensive API

Pluggable Interface

Dynamic or Static Linkage


19/82

In-process Modules

Run from inside the httpd process

CGI (mod_cgi)

mod_perl

mod_php

mod_python

mod_tcl


20/82

Out-of-process Modules

Processing happens

outside of httpd (eg.

Application Server)

Tomcat

mod_jk/jk2, mod_jserv

mod_proxy

mod_jrun

Parent

TomcatChild

Child

Child


21/82

Architecture: The Big Picture

Child

Parent

ChildChild (10s)

10s of threads Tomcat

DB

100s of threads

mod_jk

mod_rewrite

mod_php

mod_perl


22/82


23/82

Terms and Definitions

Terms from the Documentation

and the Configuration


24/82

HTTP

HyperText Transfer Protocol

A network protocol used to communicate

between web servers and web clients (eg. a

Web Browser).


25/82

Request and Response

Web browsers requestpages and webservers respondwith the result.

QuickTime a

Web Browser(Mosaic)

Web Server(Apache)

Request

Response


26/82

MPM

Multi-Processing Module

AnMPMdefines how the server will

receive and manage incoming requests.

Allows OS-specific optimizations.

Allows vastly different server models

(eg. threaded vs. multiprocess).


27/82

Child Process aka Server

Called a Server in

httpd.conf

A single httpd process.

May handle one or more

concurrent requests(depending on theMPM).

Child

Parent

ChildChild

(100s)

Servers


28/82

Parent Process

The main httpd

process.

Does not handleconnections itself.

Only creates and

destroys children.

Child

Parent

Child

Child

(100s)

OnlyonePare

nt


29/82

Client

Single HTTP connection (eg. web browser).

Note that many web browsers open up multiple

connections. Apache considers each connection

uniquely.

QuickTime a

Web Browser(Mosaic) Web Server(Apache)


30/82

Thread

In multi-threaded MPMs (eg. Worker).

Each thread handles a single connection.

Allows Children to handle many

connections at once.


31/82


32/82

Apache Configuration

httpd.conf walkthrough


33/82

Prefork MPM

Apache 1.3 and Apache 2.0 Prefork

Each child handles one connection at a time

Many children

High memory requirements

Youll run out of memory before CPU


34/82

Prefork Directives (Apache 2.0)

StartServers

MinSpareServers

MaxSpareServers

MaxClients

MaxRequestsPerChild


35/82

Worker MPM

Apache 2.0 and later

Multithreaded within each child

Dramatically reduced memory footprint

Only a few children (fewer than prefork)


36/82

Worker Directives

MinSpareThreads

MaxSpareThreads

ThreadsPerChild

MaxClients

MaxRequestsPerChild


37/82

KeepAlive Requests

Persistent connections

Multiple requests over one TCP socket

Directives:

KeepAlive

MaxKeepAliveRequests

KeepAliveTimeout


38/82


39/82

Apache 1.3 and 2.0

Performance Characteristics

Multi-process,

Multi-threaded,or Both?


40/82

Prefork

High memory usage

Highly tolerant of faulty modules

Highly tolerant of crashing children

Fast

Well-suited for 1 and 2-CPU systems

Tried-and-tested model from Apache 1.3

Youll run out of memory before CPU.


41/82

Worker

Low to moderate memory usage

Moderately tolerant to faulty modules

Faulty threads can affect all threads in child Highly-scalable

Well-suited for multiple processors

Requires a mature threading library(Solaris, AIX, Linux 2.6 and others work well)

Memory is no longer the bottleneck.


42/82

Important Performance

Considerations sendfile() support

DNS considerations

stat() calls

Unnecessary modules


43/82

sendfile() Support

No more double-copy

Zero-copy*

Dramatic improvement for static files Available on

Linux 2.4.x

Solaris 8+

FreeBSD/NetBSD/OpenBSD ...

* Zero-copy requires both OS support and NIC driver support.


44/82

DNS Considerations

HostNameLookups

DNS query for each incoming request

Use logresolve instead.

Name-based Allow/Deny clauses

Two DNS queries per request for each

allow/deny clause.


45/82

stat() for Symlinks

Options

FollowSymLinks

Symlinks are trusted.

SymLinksIfOwnersMatch

Must stat() and lstat() each symlink, yuck!


46/82

stat() for .htaccess files

AllowOverride

stat() for .htaccess in each path component of a

request Happens for any AllowOverride

Try to disable or limit to specific sub-dirs

Avoid use at the DocumentRoot


47/82

stat() for Content Negotiation

DirectoryIndex

Dont use wildcards like index

Use something like this instead

DirectoryIndex index.html index.php

index.shtml

mod_negotiation

Use a type-map instead of MultiViews if

possible


48/82

Remove Unused Modules

Saves Memory

Reduces code and data footprint

Reduces some processing (eg. filters)

Makes calls tofork() faster

Static modules are faster than dynamic


49/82


50/82

Testing Performance

Benchmarking Tools


51/82

Some Popular (Free) Tools

ab

flood

httperf

JMeter

...and many others


52/82

ab

Simple Load on a Single URL

Comes with Apache

Good for sanity check

Scales poorly


53/82

flood

Profile-driven load tester

Useful for generating real-world scenarios

I co-authored it

Part of the httpd-test project at the ASF

Built to be highly-scalable

Designed to be extremely flexible


54/82

JMeter

Has a graphical interface

Built on Java

Part of Apache Jakarta project

Depends heavily on JVM performance


55/82

Benchmarking Metrics

What are we interested in testing?

Recall that we want our web server to be

Correct Reliable

Scalable

Stable

Fast


56/82

Benchmarking Metrics:

Correctness No errors

No data corruption

Protocol compliant

Should not be an everyday concern for admins


57/82


Reliability MTBF - Mean Time Between Failures

Difficult to measure programmatically Easy to judge subjectively


58/82


Scalability Predicted concurrency

Maximum concurrent connections

Requests per Second (rps)

Concurrent Users


59/82


Stability Consistency, Predictability

Errors per Thousand

Correctness under Stress Never returns invalid information

Common problem with custom web-apps

Works well with 10 users, but chokes on 1000.


60/82


Speed Requests per Second (rps)

Latency

time until connected

time to first byte

time to last byte

time to close

Easy to test with current tools

Highly related to Scalability/Concurrency


61/82

Method

1. Define the problem

eg. Test Max Concurrency, Correctness,

etc...

1. Narrow the scope of the problem

Simplify the problem

1. Use tools to collect data

2. Come up with a hypothesis

3. Make minimal changes, retest


62/82


63/82

Troubleshooting

Common pitfalls

and their solutions


64/82

Check your error_log

The first place to look

Increase the LogLevel if needed

Make sure to turn it back down (but not off) inproduction


65/82

Check System Health

vmstat, systat, iostat, mpstat, lockstat, etc...

Check interrupt load

NIC might be overloaded

Are you swapping memory?

A web server should never swap

Check system logs

/var/log/message, /var/log/syslog, etc...


66/82

Check Apache Health

server-status

ExtendedStatus (see next slide)

Verify httpd -V

ps -elf | grep httpd | wc -l

How many httpd processes are running?


67/82

server-status Example


68/82

Other Possibilities

Set up a staging environment

Set up duplicate hardware

Check for known bugs

http://nagoya.apache.org/bugzilla/
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


69/82

Common Bottlenecks

No more File Descriptors

Sockets stuck in TIME_WAIT

High Memory Use (swapping)

CPU Overload

Interrupt (IRQ) Overload
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


70/82

File Descriptors

Symptoms

entry in error_log

new httpd children fail to start fork() failing across the system

Solutions Increase system-wide limits

Increase ulimit settings in apachectl
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


71/82

TIME_WAIT

Symptoms

Unable to accept new connections

CPU under-utilized, httpd processes sit idle

Not Swapping

netstat shows huge numbers of sockets in TIME_WAIT

Many TIME_WAIT are to be expected

Only when new connections are failing is it a problem

Decrease system-wide TCP/IP FIN timeout
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


72/82

Memory Overload, Swapping

Symptoms

Ignore system free memory, it is misleading!

Lots of Disk Activity

top/free show high swap usage Load gradually increasing

ps shows processes blocking on Disk I/O

Solutions

Add more memory

Use less dynamic content, cache as much as possible

Try the Worker MPM
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


73/82

How much free memory

do I really have? Output from top/free is misleading.

Kernels use buffers

File I/O uses cache Programs share memory

Explicit shared memory

Copy-On-Write after fork() The only time you can be sure is when it

starts swapping.
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


74/82

CPU Overload

Symptoms

top shows little or no idle CPU time

System is not Swapping

High system load

System feels sluggish

Much of the CPU time is spent in userspace

Solutions

Add another CPU, get a faster machine

Use less dynamic content, cache as much as possible
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


75/82

Interrupt (IRQ) Overload

Symptoms

Frequent on big machines (8-CPUs and above)

Not Swapping

One or two CPUs are busy, the rest are idle

Low overall system load

Solutions Add another NIC

bind it to the first or use two IP addresses in Apache

put NICs on different PCI busses if possible
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


76/82


77/82

Next Generation Improvements


78/82

Linux 2.6

NPTL and NGPT

Next-Gen Thread Libraries for Linux

Available in RedHat 9 already

O(1) scheduling patch

Preemptive Kernel patch

All improvements affect Apache, but the WorkerMPM will likely be the most affected.
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


79/82

Solaris 9

1:1 threads

Decreases thread library overhead

Improves CPU load sharing

sendfile()-like support (since late Solaris 7)

Zero-copy
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


80/82

64-bit Native Support

Sparc had it for a long time

G5s now have it (sort-of)

AMD64 (Opteron and Athlon64) have it

Noticeable improvement in Apache 2.0

Increased Requests-per-second

Faster 64-bit time calculations

Huge Virtual Memory Address-space

mmap/sendfile
http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/http://nagoya.apache.org/bugzilla/


81/82


82/82

The End

Thank You!

ac2003performance

Documents