copyright © 1998 alex chaffee building a robust multithreaded server in java alexander day chaffee...
TRANSCRIPT
Copyright © 1998 Alex Chaffee
Building a Robust Multithreaded Server in Java
Alexander Day ChaffeejGuru Training by the MageLang [email protected]
Copyright © 1998 Alex Chaffee
Abstract
• Mark Andreesen says, ”Client-side Java is dead." Fortunately, server-side Java is alive and kicking! This session will quickly introduce you to the concepts of threading and networking in Java. We then build a robust, multithreaded server (using a simple chat as our application). We examine several different threading models, specifically comparing the one-thread-per-connection model with the queuing model.
Copyright © 1998 Alex Chaffee
Introduction
• jGuru Training by the Magelang Institute– http://www.jguru.com– Java Training and Consulting
• Alex Chaffee– Creator of Gamelan– Cool Java Dude
Copyright © 1998 Alex Chaffee
Background
• Java language basics• Java networking• Java threading
Copyright © 1998 Alex Chaffee
Overview
• Start slow, end up really really fast• Part I: Networking and I/O in Java• Part II: Threads and concurrency• Part III: Multithreaded server design• Part IV: Implementations
– Design decisions– Object models– Problems and solutions
Copyright © 1998 Alex Chaffee
Part I: Networking and I/O in Java
Copyright © 1998 Alex Chaffee
Java and Networking
• Built into language• One of the 11 buzzwords• Network classloader• Java.Net API• Based on TCP/IP, the internet protocol
Copyright © 1998 Alex Chaffee
Client-server
• Difference between client and server is semantic
• It’s all just peers talking to each other• Protocol - roles, vocabulary, rules for
communication
Copyright © 1998 Alex Chaffee
TCP/IP: The Internet Protocol
Physical Network
Transport Layer (TCP, UDP)
Internet Layer (IP)
Application Layer (HTTP, FTP, SMTP)
Copyright © 1998 Alex Chaffee
Sockets and Ports
• Port: a meeting place on a host– One service per port– 1-1023 = well-known services– 1024+ = experimental services, temporary
• Socket: a two-way connection
Copyright © 1998 Alex Chaffee
Client
Sockets and Ports (Diagram)
port 13
port 80
Time Service
Web Service
Socket
Server
Socket
Copyright © 1997 Alex Chaffee
The Socket Class
• Socket(String host, int port)• InputStream getInputStream()• OutputStream getOutputStream()• void close()
Copyright © 1997 Alex Chaffee
Socket Code
Socket s = new Socket(“www.sigs.com”, 80);
• Simple, huh?• Creates the connection• We still need to get the data there and
back again
Copyright © 1997 Alex Chaffee
Streams
• A stream is a sequence of bytes• I/O from disk, network, memory, etc. is
handled in exactly the same way
Copyright © 1997 Alex Chaffee
InputStream and OutputStream
• InputStream:abstract int read()
• OutputStream:abstract void write(int b)
• Common:abstract void close()
Copyright © 1997 Alex Chaffee
Filter Streams
• Like photographic filters• Change the input (or output) on the
way through• E.g.: BufferedInputStream,
PrintStream, LineNumberInputStream
Copyright © 1997 Alex Chaffee
Filter Streams (Diagram)
Abc
FileInputStream AllCapsInputStreamAbc
ABC
NoVowelInputStreamBC
DataInputStream
readLine()
“BC” DataInputStream in = new DataInputStream(new NoVowelInputStream(
new AllCapsInputStream(new FileInputStream(“input.txt”))));
String line = in.readLine();
Copyright © 1997 Alex Chaffee
PrintStream
• print()– outputs an object, primitive, or String
• println()– same, but adds a newline
• System.out is a PrintStream
PrintStream ps =
new PrintStream(outputstream)
Copyright © 1997 Alex Chaffee
DataInputStream
• int readInt()• type readType()• String readLine()
DataInputStream in =
new DataInputStream(inputstream)
Copyright © 1997 Alex Chaffee
Browser.java (source)
import java.net.*;
import java.io.*;
public class Browser {
public static void main(String[] args)
{
String host = "www.stinky.com";
int port = 80;
String file = "/index.html";
Copyright © 1997 Alex Chaffee
Browser.java (source)
Socket s = new Socket(host, port);
InputStream strIn = s.getInputStream();
OutputStream strOut = s.getOutputStream();
PrintStream out =
new PrintStream(strOut);
DataInputStream in =
new DataInputStream(strIn);
Copyright © 1997 Alex Chaffee
Browser.java (source)
out.println("GET " + file + " HTTP/1.0");
out.println();
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
in.close();
out.close();
Copyright © 1998 Alex Chaffee
Java Servers
• A server listens on a port and accepts connections
• Must be applications (or signed applets)
• Only one service per port for the entire host machine– Java throws an exception if you try to open
more than one
Copyright © 1997 Alex Chaffee
ServerSocket
ServerSocket ss = new ServerSocket(1234);
Socket socket = ss.accept();
• accept() returns only after a client makes the connection
Copyright © 1997 Alex Chaffee
ServerSocket
• Now we have a socket, so we can call...
InputStream in = socket.getInputStream();
OutputStream out = socket.getOutputStream();
Copyright © 1997 Alex Chaffee
EchoServer.java
ServerSocket ss =
new ServerSocket(1234);
Socket s = ss.accept();
InputStream in = s.getInputStream();
OutputStream out = s.getOutputStream();
Copyright © 1997 Alex Chaffee
EchoServer.java
int ch;
while ((ch = in.read()) != -1) {
out.write(ch);
}
out.close();
in.close();
Copyright © 1998 Alex Chaffee
EchoServer Demo
• Connect• Type some stuff• Disconnect• Problem: Multiple simultaneous
connections– Connections are made, but server doesn’t
respond– Even if we loop, can still handle only one at a
time
Copyright © 1998 Alex Chaffee
Multiple Connections
• Solution:– Java threads!
Copyright © 1998 Alex Chaffee
Part II: Threads
Copyright © 1998 Alex Chaffee
Java Threads
• A thread is not an object• A thread is a flow of control• A thread is a series of executed
statements• A thread is a nested sequence of
method calls
Copyright © 1998 Alex Chaffee
The Thread Object
• A thread is not an object• A Thread is an object
void start()– Creates a new thread and makes it runnable
void run()– The new thread begins its life inside this
method
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Object A Object BThread (extends Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
Thread Creation DiagramObject A
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Object BThread (extends Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Thread t = new BThread();
t.start();
doMoreStuff();
BThread() {}
void start() {// create thread
}
void run() {doSomething();
}
Thread Creation DiagramObject A Object BThread (extends
Thread)
Copyright © 1998 Alex Chaffee
Runnable Interface
• A helper to the thread object• The Thread object’s run() method calls
the Runnable object’s run() method• Allows threads to run inside any
object, regardless of inheritance
Copyright © 1998 Alex Chaffee
Runnable Example
Talker talker = new Talker();
Thread t = new Thread(talker);
t.start();
---
class Talker implements Runnable {
public void run() {
while (true) {
System.out.println(“yakitty yak”);
}
}
}
Copyright © 1998 Alex Chaffee
Blocking Threads
• When reading from a stream, if input is not available, the thread will block
• Thread is suspended (“blocked”) until I/O is available
• Allows other threads to automatically activate• When I/O available, thread wakes back up
again– Becomes “runnable”– Not to be confused with the Runnable interface
Copyright © 1998 Alex Chaffee
Thread Scheduling
• In general, the runnable thread with the highest priority is active (running)
• Java is priority-preemptive– If a high-priority thread wakes up, and a low-
priority thread is running– Then the high-priority thread gets to run
immediately
• Allows on-demand processing– Efficient use of CPU
Copyright © 1998 Alex Chaffee
Thread Starvation
• If a high priority thread never blocks• Then all other threads will starve• Must be clever about thread priority
Copyright © 1998 Alex Chaffee
Thread Priorities: General Strategies
• Threads that have more to do should get lower priority
• Counterintuitive• Cut to head of line for short tasks• Give your I/O-bound threads high priority
– Wake up, immediately process data, go back to waiting for I/O
Copyright © 1998 Alex Chaffee
Race Conditions
• Two threads are simultaneously modifying a single object
• Both threads “race” to store their value
• In the end, the last one there “wins the race”
• (Actually, both lose)
Copyright © 1998 Alex Chaffee
Race Condition Example
class Account {
int balance;
public void deposit(int val)
{
int newBal;
newBal = balance + val;
balance = newBal;
}
}Looks good, right?
Copyright © 1998 Alex Chaffee
Race Condition Example
class Account {
int balance;
public void deposit(int val)
{
int newBal;
newBal = balance + val;
balance = newBal;
}
}What if we are swapped out right here?
Copyright © 1998 Alex Chaffee
Thread Synchronization
• Language keyword: synchronized
• Takes out a monitor lock on an object– Exclusive lock for that thread
• If lock is currently unavailable, thread will block
Copyright © 1998 Alex Chaffee
Thread Synchronization
• Protects access to code, not to data– Make data members private
– Synchronize accessor methods
• Puts a “force field” around the locked object so no other threads can enter
• Actually, it only blocks access to other synchronizing threads
• A rogue thread can "sneak in" and modify a variable it has access to
Copyright © 1998 Alex Chaffee
Synchronization Example
class Account { private int balance; synchronized public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; } synchronized public void withdraw(int val) { int newBal; newBal = balance - val; balance = newBal; }}
Copyright © 1998 Alex Chaffee
Thread Deadlock
• If two threads are competing for more than one lock
• Example: bank transfer between two accounts– Thread A has a lock on account 1 and wants
to lock account 2
– Thread B has a lock on account 2 and wants to lock account 1
Copyright © 1998 Alex Chaffee
Avoiding Deadlock
• No universal solution
• Ordered lock acquisition
• Encapsulation (“forcing directionality”)
• Spawn new threads
• Check and back off
• Timeout
• Minimize or remove synchronization
Copyright © 1998 Alex Chaffee
Wait and Notify
• Allows two threads to cooperate
• Based on a single shared lock object
Copyright © 1998 Alex Chaffee
Wait and Notify: Code
• Consumer:synchronized (lock) {
while (!resourceAvailable()) {
lock.wait();
}
consumeResource();
}
Copyright © 1998 Alex Chaffee
Wait and Notify: Code
• Producer:produceResource();
synchronized (lock) {
lock.notifyAll();
}
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
ConsumerThread
ProducerThread
1. synchronized(lock){
2. lock.wait();
3. produceResource()4. synchronized(lock) {5. lock.notify();6.}
7. Reacquire lock8. Return from wait()
9. consumeResource();10. }
Copyright © 1998 Alex Chaffee
Wait/Notify: Details
• Often the lock object is the resource itself
• Sometimes the lock object is the producer thread itself
Copyright © 1998 Alex Chaffee
Wait/Notify: Details
• Must loop on wait(), in case another thread grabs the resource...– After you are notified– Before you acquire the lock and return from
wait()
• Use lock.notifyAll() if there may be more than one waiting thread
Copyright © 1998 Alex Chaffee
Wait/Notify Example: Blocking Queue
class BlockingQueue extends Queue {
public synchronized Object remove() {
while (isEmpty()) {
wait(); // really this.wait()
}
return super.remove();
}
public synchronized void add(Object o) {
super.add(o);
notifyAll(); // this.notifyAll()
}
}
Copyright © 1998 Alex Chaffee
Part III: Multithreaded Server Design
Copyright © 1998 Alex Chaffee
Different Models of Information Flow
• Pull model• Push model: One thread per client• Buffer model: One thread per task• Single-thread model
Copyright © 1998 Alex Chaffee
Pull-based Flow
• Consumer asks producer for new information
• Demand-driven problems– Examples
• Not appropriate for chat/message-passing server
Copyright © 1998 Alex Chaffee
Push-based Flow
• Producer triggers sequence of events• One thread per client• Receives message, delivers it, waits
Copyright © 1998 Alex Chaffee
Push-based: Pro
• Simple• Efficient use of CPU
– Events occur only when message available - no polling
– No time wasted passing information from one stage to the next
Copyright © 1998 Alex Chaffee
Push-based: Con
• Much thread swapping• Slow receivers can stall upcoming
messages from sender
Copyright © 1998 Alex Chaffee
Buffer Model
• Independent threads• Communicate via buffers• Producer puts message in buffer,
consumer takes message out
Copyright © 1998 Alex Chaffee
Buffer Model: Pro
• Can separate tasks into separate threads
• Optimize and prioritize per task– e.g. Processing existing messages is more
important than accepting new connections
• Small number of threads means fewer chances for deadlock– Easier to model the system and avoid any
chance of deadlock
Copyright © 1998 Alex Chaffee
Buffer Model: Con
• More complex code
• Must deal with deadlock and starvation
Copyright © 1998 Alex Chaffee
The One-Thread Model
• All work done by a single thread
• Finishes processing one request before going to the next
Copyright © 1998 Alex Chaffee
The One-Thread Model (Cont.)
• Pro:– Suitable for low-traffic scenarios
– Quick to implement
– Avoids any possibility of deadlock or race conditions
• Con– Unsuitable for high-traffic server
– Hard to implement things like timers or callbacks
Copyright © 1998 Alex Chaffee
Part IV: Implementations
Copyright © 1998 Alex Chaffee
Implementation 1: Push Model Chat Server
• Listens on a port• Accepts connections• Spawns threads, one per connection• Receives a message from one client• Dispatches it to all clients
Copyright © 1998 Alex Chaffee
Chat Server Objects
• ChatServer– Main
• ClientHandler– One per connection– Threaded
• Dispatcher– Vector
Copyright © 1998 Alex Chaffee
Chat Server Objects (Diagram)
ChatServer
ClientHandlerClientHandler
Dispatcher
Socket Socket
Copyright © 1997 Alex Chaffee
ClientHandler
• Stores information for a single clientSocket incoming;
int id;
Dispatcher dispatcher;
Copyright © 1997 Alex Chaffee
ClientHandler.run()
• generate filter streams• read a line• send it to the dispatcher• loop
Copyright © 1997 Alex Chaffee
ClientHandler.send()
• sends a single message to the client• called by the dispatcher• uses the socket
Copyright © 1997 Alex Chaffee
Dispatcher
• Keeps a list of all client handlers• Uses java.util.Vector
– synchronized list structure
Copyright © 1997 Alex Chaffee
Dispatcher.dispatch()
• Sends a message to all clients• Accepts a String and an identifying
integer• Enumerates down all client handlers
and calls each one’s send() method
Copyright © 1997 Alex Chaffee
ChatServer
• One method: main()• Creates a new ServerSocket• Creates a new Dispatcher• Listens for connections• When connection received, creates a
new ClientHandler and starts a new thread running inside it
• Adds it to the Dispatcher
Copyright © 1998 Alex Chaffee
Push model: Pro and Con
• Pro– Straightforward object, thread models– Efficient CPU usage
• Threads unblock only when message available
• Con– Must send message to all clients before
receive new one– No upper limit on number of threads
• May thrash or starve
Copyright © 1998 Alex Chaffee
Implementation 2: Buffer Model Message Server
Copyright © 1998 Alex Chaffee
Queuing Model
• Producer and consumer communicate via a buffer
• Buffer is a queue• First in, first out
Copyright © 1998 Alex Chaffee
Blocking Queue
• If a thread tries to remove an item from an empty queue
• Then it blocks until an item is placed inside
• Uses wait/notify technique
Copyright © 1998 Alex Chaffee
JDK 1.2 Collections
• Set• List• Map• Queue
– Built on top of LinkedList
Copyright © 1998 Alex Chaffee
Basic Object Model
• Acceptor (thread)• Clients (set)• Receiver (thread)• Incoming (queue)• Processor (thread)
Copyright © 1998 Alex Chaffee
Acceptor
Receiver
Clients
Incoming
Processor
Thread
Data
write
read
read
write
create
Basic Object Model (Fig.)
Copyright © 1998 Alex Chaffee
Acceptor Thread
• Listens for incoming connections• Creates a client object• Adds it to clients set
Copyright © 1998 Alex Chaffee
Clients Set
• Set of all active clients• Synchronized methods
– Only one thread at a time can access
Copyright © 1998 Alex Chaffee
Receiver Thread
• Walks down clients set• For each client, checks if message is
(fully) available• Reads message and places it on
incoming queue• Uses non-blocking input streams
Copyright © 1998 Alex Chaffee
Receiver Thread: Alternative
• Alternative: one receiver thread per client
• Each receiver thread blocks until input available
• Con:– Reach number of threads bottleneck– Scheduling not determined - could starve or
do in weird order
Copyright © 1998 Alex Chaffee
Incoming Queue
• Queue of messages• Synchronized methods
– Only one thread at a time can access
• Blocking queue– If a thread tries to get a message from an
empty queue, it blocks until another thread adds a message
Copyright © 1998 Alex Chaffee
Processor Thread
• While incoming queue is not empty• Pops message off incoming queue• Delivers it
Copyright © 1998 Alex Chaffee
Full Object Model
• Checker (thread)• Problems (list)• Auditor (thread)
Copyright © 1998 Alex Chaffee
Acceptor
Receiver
Clients
Incoming
Processor
CheckerProblems
Auditor
Thread
Data
Full Object Model (Fig.)
Copyright © 1998 Alex Chaffee
Problems List
• List of clients who have problems• Receiver can add a client to problems
list if it gets a socket read error• Processor can add a client to problems
list if it gets a socket write error• Checker thread periodically
disconnects them all
Copyright © 1998 Alex Chaffee
Checker Thread
• Wakes up every N seconds• Walks list of clients• Move to problems list
– If N seconds has passed (“timeout”)– If socket has a problem
• Walk problems list and actively close each client
Copyright © 1998 Alex Chaffee
Auditor Thread
• Every N seconds, wakes up• Collects and prints statistics
Copyright © 1998 Alex Chaffee
Non-Blocking Input Stream
• Needed for receiver to do its job• If a full message is not available, it
does NOT block• Instead, it returns null• Tricky to implement
– Must use available(), mark() and reset() cleverly
Copyright © 1998 Alex Chaffee
Thread Safety Policies
• All data objects synchronized• All data objects independent• Therefore no possibility of deadlock in
the data objects themselves
Copyright © 1998 Alex Chaffee
Thread Safety Policies
• Thread objects must be very careful to avoid deadlock
• Current model: no multi-object transactions• Example
– Checker does clients.remove(c) then problems.add(c)
– NOT synchronized(clients) { synchronized(problems) { clients.remove(c); problems.add(c);}}
– It's OK if it's interrupted between remove and add
Copyright © 1998 Alex Chaffee
Priority: Why?
• In low-volume server, all threads are normally blocked
• When something needs to happen, it happens
• That's the beauty of threads!• So priorities are irrelevant
– If you don’t know what you’re doing, don’t mess with priorities
Copyright © 1998 Alex Chaffee
Priority: Why?
• In high-volume server, many threads always have something to do
• Important to do things in the right order
• Deal with starvation issue
Copyright © 1998 Alex Chaffee
Priorities: General Strategies
• Threads that have more to do get lower priority
• Counterintuitive• Cut to head of line for short tasks
Copyright © 1998 Alex Chaffee
Priorities: Queuing Strategies
• Most important: keep incoming queue empty
• So we don't get bogged down• Minimize latency for received
messages
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Don't want to add new messages if we're already busy– Therefore processor > receiver
• Don't want to accept new connections if we're already busy– Therefore processor > acceptor
• Implication: – Receiver and acceptor may starve?– No: queue will eventually get cleared
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Want to be able to accept new connections, even if a currently connected client has something to say– Therefore acceptor > receiver
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Checker's job is brief but important• Needs to wake up, check all clients,
and go back to sleep• Therefore checker > processor
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Auditor's job is brief but important• Need to get accurate statistics --
timing is crucial• Can't wait around for other threads to
finish• Therefore auditor > checker
Copyright © 1998 Alex Chaffee
Priorities: Conclusion
• Auditor - 10• Checker - 8• Processor - 6• Acceptor - 4• Receiver - 2• Goes by twos to accommodate
Windows threading model
Copyright © 1998 Alex Chaffee
Processor Thread Behavior
• Processor thread does:1. Pop message off incoming queue2. Deliver it3. Loop
• Very independent
Copyright © 1998 Alex Chaffee
Processor Thread Blocking
• Step 1 (pop message) may block• If queue is empty• This is good• Allows other threads to wake up and
fill queue
Copyright © 1998 Alex Chaffee
Processor Thread Blocking
• Step 2 (delivery) may block– If network is busy– If OS socket buffer is full
• Bad, because queue is still full• Increases latency (delivery time) for
already-queued messages – While new messages are arriving, old
messages should be delivered instead
Copyright © 1998 Alex Chaffee
Multiple Processor Threads
• Solution: have many processor threads working simultaneously
• If one blocks, another picks up the next message
Copyright © 1998 Alex Chaffee
Multiple Processors (Fig.)
Acceptor
Receiver
Clients
Incoming
Processor
Thread
Data
write
read
read
write
create
ProcessorProcessorProcessor
Copyright © 1998 Alex Chaffee
How Many Threads?
1. Constant, tuned per application2. Dynamic a/k/a thread pooling
Copyright © 1998 Alex Chaffee
Thread Pooling
• Thread pool keeps a certain number of threads alive
• Other threads ask the thread pool to perform a task
• If an existing thread is available, that thread performs the task
• Else a new thread is created, or the task blocks until one is available – (depending on policy)
Copyright © 1998 Alex Chaffee
Thread Pooling (Cont.)
• Removes overhead of creating threads• Allows modular task strategy
Copyright © 1998 Alex Chaffee
Information Flow
• Decoupling threads with buffers is a good idea– Maximizes concurrency– Smooths out bursty differences in rate
• Can cause problems with information flow– Producers outpace Consumers– Too many threads
Copyright © 1998 Alex Chaffee
Limiting Flow
• Bounded Buffers– Stalls producers if they outpace consumers
• Bounded Thread Pools– Prevents thrashing
• Rate-adjustment / back-pressure– Consumer asks Producer to slow down
Copyright © 1998 Alex Chaffee
Another Problem
• One message, multiple recipients, one processor thread
• If one recipient is in Finland, it will stall the remaining recipients for that message
Copyright © 1998 Alex Chaffee
Another Problem: Solutions
• Solution 1: one message queue per client; thread pool applied to these delivery queues
• Solution 2: split multi-message into several single-recipient messages– Called a Splitter (or Fork or Multicaster)– Also possible: Routers, Mergers, Collectors,
Combiners, Conduits -- see Lea’s Concurrent Programming in Java
Copyright © 1998 Alex Chaffee
Debugging Techniques• Make sure to trap all Throwables in
your Thread object’s run() method– Otherwise a stray OutOfMemoryError or
NullPointerException will kill your thread and it will look like deadlock
• Good logging procedure is vital– I use a global log method that outputs the
name of the thread and the time in addition to the message
– Name your threads at creation time
Copyright © 1998 Alex Chaffee
Thread Dump
• Control-backslash in Solaris• Control-break in Windows
– Unfortunately, it scrolls– There’s no way to redirect standard error– Oops
Copyright © 1998 Alex Chaffee
Thread Watcher
• Data object• All tasks (threads) have a pointer to the
master ThreadWatcher• Put debug code in your tasks to inform the
ThreadWatcher when they change state• watcher.setIdle();• Message m = incoming.remove();• watcher.set(“Processing message “ + m);
Copyright © 1998 Alex Chaffee
Thread Watcher (cont.)
• ThreadWatcher has its own thread– Periodically prints status of threads– Prints an error message if one thread is
blocked for too long
• Alternative to fancy GUI thread debugger– Symantec Café– Compuware DevPartner– et al.
Copyright © 1998 Alex Chaffee
Bots
• PingBot– periodically sends a message to itself,
measures the latency
• QuoteBot– sends random Zippy quotes to mimic a chat
room
Copyright © 1998 Alex Chaffee
Measurements
• Number of bots before sharp latency increase
• Average latency of single PingBot for N ZippyBots
Copyright © 1998 Alex Chaffee
Test Architecture
• Need at least three machines– Server– Zippy Bot Host– Ping Bot Host
• So they don’t interfere with one another• Should also have some clients on a far
away machine– If your server will be running on the Internet
Copyright © 1998 Alex Chaffee
Conclusion
Copyright © 1998 Alex Chaffee
Thanks to
• Eric Malmstrom• Carl Muckenhoupt• Gerry Seidman• Greg Travis• For helping prepare this talk
Copyright © 1998 Alex Chaffee
Where to Get More Information: Network Programming
• Cornell & Horstmann, Core Java (Sunsoft press)
• Harold, Java Network Programming (O’Reilly)
• Hughes et al., Java Network Programming, (Manning, an imprint of Prentice-Hall)
Copyright © 1998 Alex Chaffee
Where to Get More Information: Multithreading
– Cornell & Horstmann, Core Java (sunsoft press)
– Oaks and Wong, Java Threads (o’reilly)
– Lea, Concurrent Programming in Java (Addison Wesley)
– Travis, Using thread pools to increase threading efficiency, developer.com, http://www.Developer.Com/journal/techworkshop/060498_thread.html
– Travis, Working with the blocking queue, http://www.Developer.Com/journal/techworkshop/091098_blockq.html
Copyright © 1998 Alex Chaffee
Where to Get More Information:
• Web sites– http://www.jGuru.com/ (Java Training)
– http://www.Developer.com/ (Gamelan)– http://www.Javaworld.com/ (magazine)– http://www.Purpletech.com/ (author’s site)