a family of design patterns for application-level gatewaysschmidt/pdf/tapos-95.pdf ·...

A Family of Design Patterns for Application-Level Gateways

Douglas C. [email protected]

http://www.cs.wustl.edu/�schmidt/Department of Computer Science

Washington University, St. Louis 63130(TEL) 314-935-7538, (FAX) 314-935-7302

This paper appeared in the journal Theory and Practice ofObject Systems, special issue on Patterns and Pattern Lan-guages, Wiley & Sons, Vol. 2, No. 1, December 1996.

Abstract

Abstract

Developers of communication software must confront recur-ring design challenges involving robustness, efficiency, andextensibility. Many of these challenges are independent ofthe application-specific requirements. Successful developersresolve these challenges by applying appropriate design pat-terns. However, these patterns have traditionallybeen lockedin the minds of expert developers or buried within complexsystem source code. The primary contribution of this pa-per is to describe a family of design patterns that underlymany object-oriented communication software systems. Inaddition to describing each pattern separately, the paper il-lustrates how knowledge of the relationships and trade-offsamong patterns helps guide the construction of reusable com-munication software frameworks.

1 Introduction

Building, maintaining, and enhancing high quality communi-cation systems is hard. Developers must have a deep under-standing of many complex issues such as service initializa-tion and distribution, concurrency control, flow control, errorhanding, and event loop integration. Successful communi-cation software created by experienced software developersembodies solutions to these issues.

It is often difficult, however, to separate the essence ofsuccessful software solutions from the details of a particularimplementation. Even when software is written using well-structured object-oriented frameworks and components, itcan be hard to identify key roles and relationships. More-over, OS platform features (such as the absence or presenceof multi-threading) or requirements (such as best-effort vs.fault tolerance error handling) are often different. These dif-ferences can mask the underlying architectural commonality

among software solutions for different applications in thesame domain.

Capturing and articulating the essence and commonality ofsuccessful communication software is important for severalreasons:

� It helps guide the design choices of developers whoare building new communication systems – By under-standing the potential traps and pitfalls in their domain,developers can select suitable architectures, protocols,and platform features without wasting time and effortimplementing inefficient or error-prone solutions.

� It preserves important design information for program-mers who enhance and maintain existing software –Often, this information is locked in the minds of theoriginal developers. If this design information is notdocumented explicitly, however, it will be lost overtime, thereby increasing maintenance costs and decreas-ing software quality.

The purpose of this paper is to illustrate an effective way todocument the essence of successful communication softwareby describing key design patterns used to build application-level Gateways, which route messages between Peers dis-tributed throughout a communication system.

Design patterns represent successful solutions to problemsthat arise when building software [1]. Capturing and articu-lating key design patterns helps to enhance software qualityby addressing fundamental challenges in large-scale systemdevelopment. These challenges include communication ofarchitectural knowledge among developers; accommodatingnew design paradigms or architectural styles; resolving non-functional forces such as reusability, portability, and exten-sibility; and avoiding development traps and pitfalls that areusually learned only by costly trial and error.

This paper presents the object-oriented architecture anddesign of an application-level Gateway in terms ofthe design patterns used to guide the construction ofreusable and Gateway-specific frameworks and compo-nents. Application-level Gateways have stringent require-ments for reliability, performance, and extensibility. There-fore, they are excellent exemplars for presenting the struc-ture, participants, and consequences of key design patternsthat appear in many communication software systems.

1

PEERS

GATEWAY

PEERS

1: 1: sendsend__msg

msg()()

4: send_4: send_msgmsg()()

2: 2: recvrecv__msgmsg()()

3: 3: routeroute__msgmsg()()

5: recv_msg()5: recv_msg()

Figure 1: The Structure and Collaboration of Peers and theGateway

The patterns described in this paper were discovered whilebuilding a wide range of communication systems includingon-line transaction processing systems, telecommunicationswitch management systems [2], electronic medical imag-ing systems [3], and parallel communication subsystems [4].Although the specific application requirements in these sys-tems were quite different, the communication software designchallenges were very similar. Therefore, although the exam-ples in this paper focus on Gateways, the patterns embodydesign expertise that can be reused broadly in the communi-cation domain.

The remainder of this paper is organized as follows: Sec-tion 2 outlines an object-oriented software architecture forapplication-level Gateways; Section 3 examines the designpatterns that form the basis for reusable communication soft-ware, using application-level Gateways as an example; Sec-tion 4 compares these patterns with those described in relatedwork; and Section 5 presents concluding remarks.

2 An Object-Oriented Software Archi-tecture for Application-level Gate-ways

This paper examines framework components and design pat-terns that comprise and motivate the object-oriented architec-ture of application-level Gateways developed by the authorand his colleagues. A Gateway is a Mediator [1] that de-couples cooperating Peers throughout a network and allowsthem to interact without having direct dependencies on eachother [5]. As shown in Figure 1, messages routed throughtheGateway contain payloads encapsulated in routing mes-sages.

Figure 2 illustrates the structure, associations, and internaland external collaborations among objects within a software

CONNECTION

REQUEST

CONNECTION

REQUEST

OUTGOING

MESSAGES

: Output: OutputChannelChannel

: Message: MessageQueueQueue: SOCK: SOCK

StreamStream

INCOMING

MESSAGES

: Acceptor: Acceptor

: SOCK: SOCKAcceptorAcceptor

: Connector: Connector

: SOCK: SOCKConnectorConnector

: Map: MapManagerManager

: Input: InputChannelChannel

: SOCK: SOCKStreamStream

: Routing: RoutingTableTable

: Map: MapManagerManager


: Message: MessageQueueQueue: SOCK: SOCK

StreamStream


: SOCK: SOCKStreamStream

: Reactor: Reactor

GATEWAYGATEWAY

Figure 2: The Object-Oriented Gateway Software Archi-tecture

architecture for application-level Gateways.1 This architec-ture is based on extensive experience developing connection-oriented Gateways for various commercial and research com-munication systems. After building multiple Gateway sys-tems it became clear that the software architecture of thesesystems was largely independent of the protocols used toroute messages to Peers. This realization enabled the compo-nents depicted in Figure 2 to be reused for the communicationsoftware subsystems of many other projects. The ability toreuse these components so widely stems from two factors:

� Understanding the actions and interactions of key de-sign patterns within the domain of communication soft-ware: Patterns capture the structure and collaboration ofparticipants in a software architecture at a higher level thansource code and object-oriented design models that focus onindividual objects and classes. Some of the communicationsoftware patterns described in this paper have been docu-mented individually [7, 8, 9]. Although individual patterndescriptions capture valuable design expertise,complex com-munication software systems embody scores of patterns. Un-derstanding the relationships among these patterns is essen-tial to document, motivate, and resolve difficult challengesthat arise when building communication software. There-fore, Section 3 describes the interactions and relationshipsamong these patterns in terms of a family of design patternsfor communication software. These design patterns worktogether to solve complex problems within the domain ofcommunication software.

1Relationships between components are illustrated throughout this paperusing Booch notation [6]. In this figure solid clouds indicate objects; nestingindicates composition relationships between objects; and undirected edgesindicate an association exists between two objects.

2

� Developing an object-oriented framework that imple-ments these design patterns: Recognizing the patternsthat commonly occur in many communication software sys-tems helped shape the development of reusable frameworkcomponents. The Gateway systems this paper is basedupon were implemented with the ADAPTIVE Communica-tion Environment (ACE) software [10]. ACE provides anintegrated framework of reusable C++ wrappers and compo-nents that perform common communication software tasks.These tasks include event demultiplexing, event handler dis-patching, connection establishment, routing, dynamic con-figuration of application services, and concurrency control.In addition, the ACE framework contains implementations ofthe design patterns described in Section 3. However, the pat-terns are much richer than their implementation in ACE andhave been applied by many other communication systems, aswell.

This section describes how various ACE components havebeen reused and extended to implement the application-independent and application-specific components in the com-munication Gateway shown in Figure 2. Following thisoverview, Section 3 examines the family of design patternsthat underly the ACE components.

2.1 Application-independent Components

Most of the components in Figure 2 are based on ACEcomponents that can be reused in other communication sys-tems. The only components that are not widely reusableare the Input and Output Channels, which implementthe application-specific details related to message formatsand the routing protocol. The behavior of the application-independent components in theGateway is outlined below:

� Interprocess communication (IPC) components: TheSOCKStream, SOCKConnector, and SOCKAcceptorcomponents encapsulate the socket network programminginterface [11]. These components simplify the developmentof portable and correct communication software by shieldingdevelopers from low-level, tedious, and error-prone socket-level programming. In addition, they form the foundationfor the higher-level ACE components and patterns describedbelow.

� Service initialization components: The Connectorand Acceptor are factories [1] that implement activeand passive strategies for initializing network services,respectively.2 These components are based on the Connec-tor pattern described in Section 3.2 and Acceptor patterndescribed in Section 3.3. The Gateway uses these com-ponents to establish connections with Peers and produceinitialized Input and Output Channels.

2Establishing connections between endpoints involves two roles: thepassive role (which initializes an endpoint of communication at a particularaddress and waits passively for the other endpoint to connect with it) and theactive role (which actively initiates a connection to one or more endpointsthat are playing the passive role).

To increase system flexibility, connections can be estab-lished in two ways:

1. From the Gateway to the Peers – which is typicallydone whenever the Gateway first starts up to establishthe initial system configuration of Peers;

2. From aPeer to theGateway – which is typicallydoneonce the system is running whenever a new Peerwantsto send or receive routing messages.

In a large system, several scores of Peers may be con-nected to a single Gateway. Therefore, to expedite initial-ization, the Gateway’s Connector can initiate all con-nections asynchronously rather than synchronously. Asyn-chrony helps decrease connection latency over long delaypaths (such as wide-area networks (WANs) built over satel-lites or long-haul terrestrial links). The underlying SOCKConnector [11] contained within a Connector providesthe low-level asynchronous connection mechanism. When aSOCK Connector connects two socket endpoints via TCPit produces a SOCK Stream object, which is then used toexchange data between that Peer and the Gateway.

� Event demultiplexing components: The Reactor isan object-oriented event demultiplexing mechanism basedon the Reactor pattern [7] described in Section 3.1. It chan-nels all external stimuli in an event-driven application to asingle demultiplexing point. This permits single-threadedapplications to wait on event handles, demultiplex events,and dispatch event handlers efficiently. Events indicate thatsomething significant has occurred (e.g., the arrival of a newconnection or work request). The main source of events inthe Gateway is routing messages that encapsulate variouspayloads (such as commands, status messages, and bulk datatransmissions).

� Message demultiplexing components: The MapManager is a parameterized collection that efficiently mapsexternal ids (e.g., Peer routing addresses) onto internal ids(e.g., Output Channels). The Gateway uses a MapManager to implement a Routing Table that handlesthe demultiplexing and routing of messages internally to aGateway. The Routing Table maps addressing infor-mation contained in routing messages sent by Peers to theappropriate set of Output Channels.

� Message queueing components: The MessageQueue [10] provides a generic queueing mechanism. Thismechanism runs efficient and robustly in multi-threaded orsingle-threaded environments. Developers can select the de-sired concurrency strategy at the time a queue is instanti-ated, TheGateway usesMessage Queues to buffer mes-sages in Output Channels while they are being routedto Peers.

2.2 Application-specific Components

Only two of the components (Input and OutputChannels) in Figure 2 are specific to the Gateway ap-plication. These components implement the Router pattern

3

described in Section 3.4. Input and Output Channelsreside in the Gateway, where they serve as proxies for theoriginal source and the intended destination(s) of routingmessages sent to hosts across the network. The behavior ofthese two Gateway-specific components is described be-low:

� Input Channels: Input Channels are responsiblefor routing incoming messages to their destination(s). TheReactor notifies an Input Channel when it detects anevent on that connection’s communication endpoint. TheInput Channel then receives and frames a routing mes-sage from that endpoint, consults the Routing Table todetermine the set of Output Channel destinations for themessage, and requests the selected Output Channels toforward the message to the appropriate Peer destinations.

� Output Channels: An Output Channel is respon-sible for reliably delivering routing messages to their desti-nations. It implements a flow control mechanism to bufferbursts of routing messages that cannot be sent immediatelydue to transient network congestion or lack of buffer space ata receiver. Flow control is a transport layer mechanism thatensures a source Peer does not send data faster than a desti-nation Peer can buffer and process the data. For instance, ifa destination Peer runs out of buffer space, the underlyingTCP protocol instructs the associated Gateway’s OutputChannel to stop sending messages until the destinationPeer consumes its existing data.

A Gateway integrates the application-specific andapplication-independent components by inheriting from, in-stantiating, and composing the ACE components describedabove. As shown in Figure 33 Input and OutputChannels inherit from a common ancestor: the ACE SvcHandler class, which is produced by Connectors andAcceptors. The Svc Handler is a local Proxy [1] fora remotely connected Peer. It provides a SOCK Stream,which enables Peers to exchange messages via connectedChannels.

An Output Channel uses an ACE Message Queueto chain unsent messages in the order they must be deliv-ered when flow control mechanisms permit. Once a flowcontrolled connection opens up, the ACE framework notifiesitsOutput Channel, which starts draining theMessageQueue by sending messages to the Peer. If flow controloccurs again this sequence of steps is repeated until all mes-sages are delivered.

To improve reliability and performance, the Gatewaysdescribed in this paper utilize the Transmission Con-trol Protocol (TCP). TCP provides a reliable, in-order,non-duplicated bytestream service for application-levelGateways. Although TCP connections are inherently bi-directional, data sent from Peer to the Gateway use a

3This figure illustrates additional Booch notation. Dashed clouds indicateclasses; directed edges indicate inheritance relationships between classes;and an undirected edge with a small circle at one end indicates either acomposition relation between two classes.

SvcSvcHandlerHandler

InputInputChannelChannel

OutputOutputChannelChannel

SOCKSOCKStreamStream

MessageMessageQueueQueue

Figure 3: Channel Inheritance Hierarchy

different connection than data sent from the Gateway tothe Peer. There are several advantages to separating inputconnections from output connections in this manner:

� It simplifies the construction of Gateway RoutingTables;

� It allows more flexibility in connection configurationand concurrency strategies;

� It enhances reliability if errors occur on a connectionsince Input and Output Channels can be recon-nected independently.

3 A Family of Design Patterns forApplication-level Gateways

Section 2 illustrates the structure and functionality of anapplication-level Gateway. Although this architecturaloverview helps to clarify the behavior of key componentsin a Gateway, it does not reveal the deeper relationshipsand roles that underly the various software components. Inparticular, the architecture descriptions do not motivate whya Gateway is designed in this particular manner or whycertain components act and interact in certain ways. Under-standing these relationships and roles is crucial to develop,maintain, and enhance communication software.

An effective way to capture and articulate these relation-ships and roles is to describe the design patterns that reifythem. A design pattern is a recurring solution to a designproblem within a particular domain (such as business dataprocessing, telecommunications, graphical user interfaces,databases, or distributed communication software). Study-ing the patterns in Gateway software is important to:

1. Identify successful solutions to common design chal-lenges – The patterns underlying the Gateway archi-tecture transcend the particular details of the applicationand resolve common challenges faced by communica-tion software developers. A thorough understanding ofthe patterns presented below enables widespread reuseof Gateway software architecture in other systems,even when reuse of its algorithms, implementations, in-terfaces, or detailed designs is not feasible [12].

4

ConnectorConnectorActiveActiveObjectObject

ReactorReactor

BuilderBuilderIteratorIterator AdapterAdapterTemplateTemplateMethodMethod

TACTICALTACTICAL

PATTERNSPATTERNS

STRATEGIC

PATTERNS

AcceptorAcceptor

RouterRouter

ProxyProxy

Figure 4: The Family of Patterns for Application-levelGateways

2. Reduce the effort of maintaining and enhancingGateway software – The use of patterns helps to cap-ture and motivate the collaboration between multipleclasses and objects. This is important for developerswho must maintain and enhance a Gateway. Althoughthe roles and relationships in a Gateway design areembodied in the source code, extracting them from thesurrounding implementation details can be costly anderror-prone.

Figure 4 illustrates the following strategic patterns relatedto connection-oriented, application-level Gateways:

� Concurrency patterns:

� The Reactor pattern – which decouples event demul-tiplexing and event handler dispatching from servicesperformed in response to events;

� The Active Object pattern – which decouples methodexecution from method invocation.

� Service initialization patterns:

� The Connector pattern – which decouples active serviceinitialization from the tasks service performed once theservice is initialized;

� The Acceptor pattern – which decouples passive serviceinitialization from the tasks performed once the serviceis initialized.

� Application-specific patterns:

� The Router pattern – which decouples input mecha-nisms from output mechanisms to route data correctlywithout blocking a Gateway.

These five patterns are strategic because they significantlyinfluence the software architecture for applications in a par-ticular domain (in this case, the domain of communicationsoftware and Gateways). For example, the Router patterndescribed in Section 3.4 decouples input mechanisms fromoutput mechanisms to ensure that message processing is notdisrupted or postponed indefinitely when a Gateway expe-riences congestion or failure. This pattern helps to ensure aconsistently high quality of service for Gateways that usereliable transport protocols such as TCP/IP or IPX/SPX. Athorough understanding of the strategic communication pat-terns described in this paper is essential to develop robust,efficient, and extensible application-level Gateways.

Application-level Gateways also utilize many tactical pat-terns, such as the following:

� Builder pattern – which provides a factory for buildingcomplex objects incrementally. TheGatewayuses thispattern to create its Routing Table from a configu-ration file.

� Iterator pattern – which decouples sequential accessto a container from the representation of the container.TheGateway uses this pattern to connect and initializeInput and Output Channels with Peers.

� Template Method pattern – which specifies an algorithmwhere some steps are supplied by a derived class. TheGateway uses this pattern to selectively override cer-tain steps in the Connector and Acceptor in orderto restart failed connections automatically.

� Adapter pattern – which transforms a non-conforminginterface into one that can be used by a client. TheGateway uses this pattern to treat different types ofrouting messages (such as commands, status informa-tion, and bulk data) in a uniform manner.

� Proxy pattern – which provides a local surrogate objectthat acts in place of a remote object. The Gateway usesthis pattern to shield the main Gateway routing codefrom delays or errors caused by the fact that Peers arelocated on other host machines in the network.

Compared to strategic patterns (which are often domain-specific and have broad design implications), tactical pat-terns are domain-independent and have a relatively localizedimpact on a software design. For instance, Iterator is a tac-tical pattern used in the Gateway to process entries in theRouting Table sequentially without violating data en-capsulation. Although this pattern is domain-independentand thus widely applicable, the problem it addresses doesnot impact the application-level Gateway software design aspervasively as strategic patterns like the Router. A thoroughunderstanding of tactical patterns is essential to implementhighly flexible software that is resilient to changes in appli-cation requirements and platform environments.

Although there are various forms for describing patterns,they typically convey the following information [1]:

� The intent of the pattern;

5

Reactor

handle_events()register_handler(eh, type)remove_handler(eh, type) Event HandlerEvent Handler

handle_event(type)get_handle()

Ann

11

select (handles)foreach h in handles loop table[h]->handle_event (type)end loop

nn

11

APPLIC

ATIO

N

APPLIC

ATIO

N--SPECIFIC

SPECIFIC

APPLIC

ATIO

N

APPLIC

ATIO

N--IND

EPE

ND

EN

T

IND

EPE

ND

EN

T

HandlesHandles

11

11

ConcreteConcreteEventEvent

HandlerHandler

Figure 5: Structure and Participants in the Reactor Pattern

� The design forces that motivate and shape the pattern;

� The solution to these forces;

� The related classes and their roles in the solution;

� The responsibilities and dynamic collaborations amongclasses;

� The positive and negative consequences of using thepattern;

� Guidance for implementors of the pattern;

� Example source code illustrating how the pattern is ap-plied;

� References to related work.

It is important to recognize that the strategic patterns in thispaper are much more generally applicable than the specificuse cases for the Gateway described below. The references[7, 8, 9] provide additional use cases for these patterns, alongwith more detailed coverage of each pattern and sample im-plementations.

3.1 The Reactor Pattern

Intent: The Reactor pattern decouples event demultiplex-ing and event handler dispatching from the services per-formed in response to events.

Motivation and Forces: Single-threaded applicationsmust be able to handle events from multiple sources withoutblocking on any single source. The Reactor pattern resolvesthe following forces that impact the design of single-threaded,event-driven communication software:

1. The need to demultiplex multiple types of events frommultiple sources of events efficiently within a singlethread of control – A Reactor serializes the handlingof events from multiple sources within an applicationprocess at the level of event demultiplexing. By us-ing the Reactor pattern, the need for more complicated

mainmainprogramprogram

REGISTER HANDLER

DISPATCH HANDLER(S)

RUN EVENT LOOP

EXTRACT HANDLE

INITIALIZE

callback :Concrete

Event_Handler

handle_events()

handle_event(event_type)

reactor :Reactor

get_handle()

Reactor()

register_handler(callback)

select()

: Handles

WAIT FOR EVENTS

INIT

IAL

IZA

TIO

NIN

ITIA

LIZ

AT

ION

MO

DE

MO

DE

EV

EN

T H

AN

DL

ING

EV

EN

T H

AN

DL

ING

MO

DE

MO

DE

Figure 6: Object Interaction Diagram for the Reactor Pattern

threading, synchronization, or locking within an appli-cation is often eliminated.

2. The need to extend application behavior without requir-ing changes to the event dispatching framework – TheReactor factors out the demultiplexing and dispatchingmechanisms from the event handler processing poli-cies. The demultiplexing and dispatching mechanismsare generally independent of an application and are thusreusable. In contrast, the event handler policies are morespecific to an application. This separation of concernsallows application policies to change without affectingthe lower-level framework mechanisms.

Structure, Participants, and Implementation: Figure 5illustrates the structure and participants in the Reactor pat-tern. The Reactor defines an interface for registering,removing, and dispatching Concrete Event Handlerobjects (such as Input or Output Channels in theGateway). An implementation of this interface providesa set of application-independent mechanisms. These mech-anisms perform event demultiplexing and dispatching ofapplication-specific event handlers in response to events(such as input, output, signal, and timer events).

An Event Handler specifies an abstract interfaceused by the Reactor to dispatch callback methodsdefined by objects that register to events of interest.Each Concrete Event Handler selectively imple-ments callback method(s) to process events in an application-specific manner.

Collaborations: Figure 6 illustrates the collaborations be-tween participants in the Reactor pattern. These collabora-tions are divided into the following two modes:

1. Initialization mode – where Concrete EventHandler objects are registered with the Reactor;

2. Event handling mode – where the Reactor invokesupcalls on registered objects, which then handle eventsin an application-specific way.

Usage: The Reactor is used for the following event dis-patching operations in a Gateway:

6

:: Reactor Reactor

REGISTEREDREGISTERED

OBJECTSOBJECTS

FR

AM

EW

OR

KF

RA

ME

WO

RK

LE

VE

LL

EV

EL

KE

RN

EL

KE

RN

EL

LE

VE

LL

EV

EL

AP

PL

ICA

TIO

NA

PP

LIC

AT

ION

LE

VE

LL

EV

EL

OS EVENT DEMULTIPLEXING INTERFACE

:Timer:TimerQueueQueue

: Signal: SignalHandlersHandlers

: Handle: HandleTableTable

: Event: EventHandlerHandler






1: handle_event()1: handle_event()

4: send(msg)4: send(msg)

2: recv(msg)2: recv(msg)3: route(msg)3: route(msg)

Figure 7: Using the Reactor Pattern in the Gateway

� Input events – The Reactor dispatches each incomingrouting message to the Input Channel associatedwith its socket handle, at which point the message isrouted to the appropriate Output Channel(s). Thisuse-case is shown in Figure 7.

� Output events – The Reactor ensures that outgoingrouting messages are eventually delivered on flow con-trolled Output Channels described in Section 3.4and 3.5.

� Connection completion events – The Reactor dis-patches events that indicate the completion status ofconnections that are initiated asynchronously. Theseevents are used by the Connector described in Sec-tion 3.2.

� Connection request events – The Reactor also dis-patches events that indicate the arrival of passivelyinitiated connections. These events are used by theAcceptor described in Section 3.3.

The Reactor pattern has been used in many single-threadedevent-driven frameworks (such as the Motif, Interviews [13],System V STREAMS [14], the ACE object-oriented commu-nication framework [10], and implementations of DCE andCORBA). In addition, it forms the foundation for most of thestrategic patterns presented below.

3.2 The Connector Pattern

Intent: The Connector pattern decouples active service ini-tialization from the tasks performed once a service is initial-ized.

Motivation and Forces: Connection-orientedapplications(like a Gateway) and middleware (like CORBA or Dis-tributed COM) are often written using lower-level network

ReactorReactor11nn

EventEventHandlerHandler

ConnectorConnectorconnect_svc_handler()activate_svc_handler()handle_event()connect(sh, addr)

SVC_HANDLERSVC_HANDLER

PEER_CONNECTORPEER_CONNECTOR

ConcreteConcreteConnectorConnector

Concrete_Svc_HandlerConcrete_Svc_Handler

SOCK_ConnectorSOCK_Connector11

ConcreteConcreteSvc HandlerSvc Handler

SOCK_StreamSOCK_Stream

open()

nn

RE

AC

TIV

ER

EA

CT

IVE

LA

YE

RL

AY

ER

CO

NN

EC

TIO

NC

ON

NE

CT

ION

LA

YE

RL

AY

ER

AP

PL

ICA

TIO

NA

PP

LIC

AT

ION

LA

YE

RL

AY

ER

handle_event()

A

connect_svc_handlerconnect_svc_handler

(sh, addr); (sh, addr);1:1:

Svc HandlerSvc Handler

PEER_STREAMPEER_STREAM

open() AA

INITIALIZES

INITIALIZES

activate_svc_handleractivate_svc_handler

(sh); (sh);2:2:

nn

Figure 8: Structure and Participants in the Connector Pattern

programming interfaces (like sockets [15] and TLI [16]). TheConnector pattern resolves the following forces that impactthe active initialization of services written using these lower-level interfaces:

1. The need to reuse active connection establishment codefor each new service – The Connector pattern permitskey characteristics of services (such as the communi-cation protocol or the data format) to evolve indepen-dently and transparently from the mechanisms used toestablish the connections. Since service characteristicschange more frequently than connection establishmentmechanisms this separation of concerns reduces soft-ware coupling and increases code reuse.

2. The need to make the connection establishment codeportable across platforms that contain different networkprogramming interfaces – This is particularly importantfor asynchronous connection establishment, which ishard to program portably and correctly using lower-level network programming interfaces like sockets andTLI.

3. The need to enable flexible service concurrency policies– Once a connection is established, peer applications usethe connection to exchange data to perform some typeof service (e.g., remote login, WWW HTML documenttransfer, etc.). A service can run in a single-thread, inmultiple threads, or multiple processes, regardless ofhow the connection was established or how the serviceswere initialized.

4. The need to actively establish connections with largenumber of peers efficiently – The Connector pattern can

7

ClientClient

FOREACH CONNECTIONFOREACH CONNECTION

INITIATE CONNECTION INITIATE CONNECTION

ASYNC CONNECT ASYNC CONNECT

INSERT IN REACTOR INSERT IN REACTOR

START EVENT LOOPSTART EVENT LOOP

FOREACH EVENT DOFOREACH EVENT DO

handle_events()

select()

CONNECTION COMPLETECONNECTION COMPLETE

INSERT IN REACTORINSERT IN REACTOR

con :con :ConnectorConnector

handle_event()

reactor :reactor :ReactorReactor

sh:sh:Svc_HandlerSvc_Handler

handle_event()

register_handler(sh)

get_handle()EXTRACT HANDLEEXTRACT HANDLE

DATA ARRIVESDATA ARRIVES

svc()PROCESS DATAPROCESS DATA

connect(sh, addr)

connect()

ACTIVATE OBJECTACTIVATE OBJECT

register_handler(con)

: SOCK: SOCKConnectorConnector

CO

NN

EC

TIO

NC

ON

NE

CT

ION

INIT

IAT

ION

INIT

IAT

ION

PH

AS

EP

HA

SE

SE

RV

ICE

SE

RV

ICE

INIT

IAL

IZA

TIO

NIN

ITIA

LIZ

AT

ION

PH

AS

EP

HA

SE

SE

RV

ICE

SE

RV

ICE

PR

OC

ES

SIN

GP

RO

CE

SS

ING

PH

AS

EP

HA

SE

activate_svc_handler(sh)

connect_svc_handler(sh, addr)

open()

Figure 9: Object Interaction Diagram for the Connector Pat-tern

employ asynchrony to initiate and complete multipleconnections in non-blocking mode. By using asyn-chrony, the Connector pattern enables applications toactively establish connections with a large number ofpeers efficiently over long-delay WANs.

Structure, Participants, and Implementation: Figure 8illustrates the layering structure of participants in the Con-nector pattern.4 The Connector is a factory that assem-bles the resources necessary to connect and activate a SvcHandler, which is a Proxy that exchanges messages with itsPeer. The Connector’s initialization strategy can estab-lish connections with Peers either synchronously or asyn-chronously.

The participants in the Connection Layer of the Connec-tor pattern leverage off the Reactor pattern. For instance,the Connector’s asynchronous initialization strategy es-tablishes a connection after the Reactor notifies it that apreviously initiated connection request to a Peer has com-pleted. Using the Reactor pattern enables multiple SvcHandlers to be initialized actively within a single threadof control.

To increase flexibility, the implementation of aConnector can be parameterized by a particular typeof PEER CONNECTOR and SVC HANDLER. The PEERCONNECTOR supplies the underlying transport mechanism(such as C++ wrappers for sockets or TLI) used by theConnector to actively establish a connection. The SVCHANDLER specifies an abstract interface for defining a ser-vice that communicates with a connected Peer. A SvcHandler can be parameterized by a PEER STREAM end-point. The Connector associates this endpoint to its Peerwhen a connection is actively established.

4In this figure the dashed rectangles indicate template parameters and adashed directed edge indicates template instantiation.

By inheriting from Event Handler (shown in Fig-ure 5), a Svc Handler can register with a Reactor anduse the Reactor pattern to handle its I/O events within thesame thread of control as the Connector. Conversely, aSvc Handler can use the Active Object pattern and handleits I/O events within a separate thread. Section 3.5 evaluatesthe tradeoffs between these different patterns.

Parameterized types are used to decouple the Connectorpattern’s connection establishment strategy from the type ofservice and the type of connection mechanism. Develop-ers supply template arguments for these types to produceApplication Layer Connectors (such as the Connectorused by the Gateway to initialize its Input and OutputChannels). This enables the wholesale replacement of theSVC HANDLER and PEER CONNECTOR types, without af-fecting the Connector pattern’s service initialization strategy.

Note that a similar degree of decoupling could be achievedvia inheritance and dynamic binding by using the AbstractFactory or Factory Method patterns described in [1]. Pa-rameterized types were used to implement this pattern sincethey improve run-time efficiency. In general, templates tradecompile- and link-time overhead and space overhead for im-proved run-time performance.

Collaborations: The collaborations among participants inthe Connector pattern are divided into three phases:

1. Connection initiation phase – which actively con-nects one or more Svc Handlers with their peers.Connections can be initiated synchronously or asyn-chronously. The Connector’s connectmethod im-plements the strategy for actively establishing connec-tions.

2. Service initialization phase – which activates a SvcHandler by calling its open method when its con-nection completes successfully. The open method ofthe Svc Handler then performs service-specific ini-tialization.

3. Service processing phase – which performs theapplication-specific service processing using the dataexchanged between the Svc Handler and its con-nected Peer.

Figure 9 illustrates these three phases of collaboration us-ing asynchronous connection establishment. Note how theconnection initiation phase is temporally separated from theservice initialization phase. This enables multiple connec-tion initiations to proceed in parallel within a single thread ofcontrol. The collaboration for synchronous connection estab-lishment is similar. In this case, the Connector combinesthe connection initiationand service initializationphases intoa single blocking operation.

In general, synchronous connection establishment is usefulfor the following situations:

� If the latency for establishing a connection is very low(e.g., establishing a connection with a server on the samehost via the loopback device);

8

� If multiple threads of control are available and it isfeasible to use a different thread to connect each SvcHandler synchronously;

� If a client application cannot perform useful work untila connection is established.

In contrast, asynchronous connection establishment is usefulfor the following situations:

� If the connection latency is high and there are manypeers to connect with (e.g., establishing a large numberof connections over a high-latency WAN);

� If only a single thread of control is available (e.g., if theOS platform does not provide application-level threads);

� If the client application must perform additional work(such as refreshing a GUI) while the connection is in theprocess of being established.

It is often the case that network services like the Gatewaymust be developed without knowing if they will connectsynchronously or asynchronously. Therefore, componentsprovided by a general-purpose network programming frame-work must support multiple synchronous and asynchronoususe-cases.

The Connector pattern increases the flexibility and reuseof networking framework components by separating theconnection establishment logic from the service process-ing logic. The only coupling between a Connector anda Svc Handler occurs in the service initialization phase,when theConnector invokes theopenmethod of theSvcHandler. At this point, the Svc Handler can performits service-specific processing using any suitable application-level protocol or concurrency policy. For instance, whenmessages arrive at a Gateway, the Reactor can be usedto dispatch Input Channels to frame the messages, de-termine outgoing routes, and deliver the messages to theirOutput Channels. However, a different type of con-currency mechanism (such as Active Objects described inSection 3.5) can be used by the Output Channels tosend the data to the remote destinations.

Usage: The Connector pattern is used by the Gatewayto simplify the task of connecting to a large number ofPeers. Peer addresses are read from a configuration fileduring Gateway initialization. The Gateway uses theBuilder pattern [1] to bind these addresses to dynamicallyallocated Channels. Since Channels inherit from SvcHandler, all connections can be initiated asynchronouslyusing the Iterator pattern [1]. The connections are then com-pleted in parallel using the Connector.

Figure 10 illustrates the relationship between participantsin the Connector pattern after four connections have beenestablished. Three other connections that have not yetcompleted are owned by the Connector. As shown inthis figure, the Connector maintains a table of the threeChannels whose connections are pending completion. As

: Connector

: ReactorPENDING

CONNECTIONS

ACTIVE

CONNECTIONS

: SvcHandler

: Channel

: SvcHandler

: Channel

: SvcHandler

: Channel

: SvcHandler

: Channel

: SvcHandler

: Channel

: SvcHandler

: Channel: Svc

Handler

: Channel

Figure 10: Using the Connector Pattern in the Gateway

connections complete, the Connector removes each con-nectedChannel from its table and activates it. In the single-threaded implementationInput Channels register them-selves with the Reactor once they are activated. Hence-forth, when routing messages arrive, Input Channelsreceive and forward them to Output Channels, whichdeliver the messages to their destinations (these activities aredescribed in Section 3.4).

In addition to establishing connections, a Gateway canuse the Connector in conjunction with the Reactor toensure that connections are restarted if network errors occur.This enhances the Gateway’s fault tolerance by ensuringthat channels are automatically reinitiated when they discon-nect unexpectedly (e.g., if a Peer crashes or an excessiveamount of data is queued at an Output Channel dueto network congestion). If a connection fails unexpectedly,an exponential-backoff algorithm can be implemented usingthe timer-based dispatching capabilities of the Reactor torestart the connection efficiently.

3.3 The Acceptor Pattern

Intent: The Acceptor pattern decouples passive service ini-tialization from the tasks performed once the service is ini-tialized.

Motivationand Forces: The Acceptor pattern resolves thefollowing forces that impact the passive initialization of ser-vices written using lower-level network programming inter-faces like sockets and TLI:

1. The need to reuse passive connection establishment codefor each new service – The Acceptor pattern permits keycharacteristics of services (such as the communicationprotocol or the data format) to evolve independently andtransparently from the mechanisms used to establish theconnections. Since service characteristics change morefrequently than connection establishment mechanismsthis separation of concerns helps reduce software cou-pling and increases code reuse.

9

ReactorReactor11

AcceptorAcceptor

SVC_HANDLERSVC_HANDLER

PEER_ACCEPTORPEER_ACCEPTOR

ConcreteConcreteAcceptorAcceptor

Concrete_Svc_HandlerConcrete_Svc_Handler

SOCK_AcceptorSOCK_Acceptor11ConcreteConcrete

Svc HandlerSvc Handler

SOCK_StreamSOCK_Stream

open()

nnR

EA

CT

IVE

RE

AC

TIV

EL

AY

ER

LA

YE

RC

ON

NE

CT

ION

CO

NN

EC

TIO

NL

AY

ER

LA

YE

RA

PP

LIC

AT

ION

AP

PL

ICA

TIO

NL

AY

ER

LA

YE

R

INITIALIZES

INITIALIZES

sh = make_svc_handler();sh = make_svc_handler();

accept_svc_handler (sh);accept_svc_handler (sh);

activate_svc_handler (sh);activate_svc_handler (sh);

nn

EventEventHandlerHandler

handle_event()

AA

make_svc_handler()accept_svc_handler()activate_svc_handler()open()handle_event()

SvcSvcHandlerHandler

PEER_STREAMPEER_STREAM

open() AA

Figure 11: Structure and Participants in the Acceptor Pattern

2. The need to make the connection establishment codeportable across platforms that contain different networkprogramming interfaces – Parameterizing the Accep-tor’s mechanisms for accepting connections and per-forming services helps to improve portability by allow-ing the wholesale replacement of these mechanisms.This makes the connection establishment code portableacross platforms that contain different network program-ming interfaces (such as sockets but not TLI, or viceversa).

3. The need to enable flexible service concurrency policies– Once a connection is established, peer applications usethe connection to exchange data to perform some typeof service (e.g., remote login, WWW HTML documenttransfer, etc.). A service can run in a single-thread,in multiple threads, or multiple processes, regardlessregardless of how the connection was established orhow the services were initialized.

4. The need to ensure that a passive-mode I/O handle isnot accidentally used to read or write data – By stronglydecoupling the Acceptor from the Svc Handler,passive-mode listener endpoints cannot be used incor-rectly (e.g., to try to read or write data on a passive-mode listener socket used to accept connections). Thiseliminates an important class of network programmingerrors.

The Acceptor pattern is the “dual” of the Connector patterndescribed in Section 3.2. However, the Connector pattern es-tablishes connections actively, whereas the Acceptor patternestablishes connections passively. The consequences of this

ServerServer

REGISTER HANDLERREGISTER HANDLER

START EVENT LOOPSTART EVENT LOOP

CONNECTION EVENTCONNECTION EVENT

REGISTER HANDLERREGISTER HANDLER

FOR CLIENT FOR CLIENT I/OI/O

FOREACH EVENT DOFOREACH EVENT DO

EXTRACT HANDLEEXTRACT HANDLE

INITIALIZE PASSIVEINITIALIZE PASSIVE

ENDPOINTENDPOINT

acc :acc :AcceptorAcceptor

handle_event()

handle_close()

reactor :reactor :ReactorReactor

select()

sh:sh:Svc_HandlerSvc_Handler

handle_event()

register_handler(sh)

get_handle()EXTRACT HANDLEEXTRACT HANDLE

DATA EVENTDATA EVENT

CLIENT SHUTDOWNCLIENT SHUTDOWN

svc()PROCESS MSGPROCESS MSG

open()

CREATECREATE,, ACCEPT ACCEPT,,AND ACTIVATE OBJECTAND ACTIVATE OBJECT

SERVER SHUTDOWNSERVER SHUTDOWNhandle_close()

: SOCK: SOCKAcceptorAcceptor

handle_events()

get_handle()

register_handler(acc)

sh = make_svc_handler()accept_svc_handler (sh)activate_svc_handler (sh)

open()

EN

DP

OIN

TE

ND

PO

INT

INIT

IAL

IZA

TIO

NIN

ITIA

LIZ

AT

ION

PH

AS

EP

HA

SE

SE

RV

ICE

SE

RV

ICE

INIT

IAL

IZA

TIO

NIN

ITIA

LIZ

AT

ION

PH

AS

EP

HA

SE

SE

RV

ICE

SE

RV

ICE

PR

OC

ES

SIN

GP

RO

CE

SS

ING

PH

AS

EP

HA

SE

Figure 12: Object Interaction Diagram for the Acceptor Pat-tern

difference in connection roles is illustratedby the forces thesetwo patterns resolve. For instance, note that the first threeforces resolved by the Acceptor pattern are essentially thesame as for the Connector pattern, with only the passive andactive roles reversed. However, the final force resolved ineach pattern is different due to the inverse connection rolesplayed by each pattern.

Structure, Participants, and Implementation: Figure 11illustrates the layering structure of participants in the Ac-ceptor pattern, which is nearly identical to the Connectorlayering structure in Figure 8. The Acceptor is a factorythat assembles the resources necessary to create, accept, andactivate a Svc Handler. The Svc Handler in the Ac-ceptor pattern plays the same role as in the Connector pattern,i.e., it is a local Proxy for a remotely connected Peer.

The Connection Layer in the Acceptor pattern leverages offthe Reactor pattern. For instance, the Acceptor’s initial-ization strategy establishes a connection after the Reactornotifies it that a new connection request has arrived froma Peer. Using the Reactor pattern enables multiple SvcHandlers to be initialized passively within a single threadof control.

To increase flexibility, the implementation of anAcceptor can be parameterized by a particular typeof PEER CONNECTOR and SVC HANDLER. The PEERACCEPTOR supplies the underlying transport mechanism(such as C++ wrappers for sockets or TLI) used by theAcceptor to passively establish a connection. The SVCHANDLER specifies an abstract interface for defining a ser-vice that communicates with a connected Peer. A SVCHANDLER can be parameterized by a PEER STREAM end-point. The Acceptor associates this endpoint to its Peerwhen a connection is established passively.

As with the Connector pattern, a Svc Handler can useeither the Reactor pattern or Active Object pattern to handle

10

: Input: InputAcceptorAcceptor

:: Reactor Reactor

ACTIVE

CONNECTIONS

: SvcHandler


: Svc: SvcHandlerHandler






PASSIVE

LISTENERS: Output: OutputAcceptorAcceptor

Figure 13: Using the Acceptor Pattern in the Gateway

its I/O events. Likewise, the implementation of the Accep-tor pattern presented above also uses parameterized types.Parameterized types enhance portability since the Acceptorpattern’s connection establishment strategy is independent ofthe type of service and the type of IPC mechanism. De-velopers can supply concrete arguments for these types toproduce Application Layer Concrete Acceptor (suchas the Acceptor used by the Gateway and Peers topassively initialize Input and Output Channels).

Collaboration: Figure 12 illustrates the collaborationamong participants in the Acceptor pattern. These collab-orations are divided into three phases:

1. Endpoint initialization phase – which creates a passive-mode endpoint (encapsulated by PEER ACCEPTOR)that is bound to a network address (such as an IP addressand port number). The passive-mode endpoint listensfor connection requests from Peers. This endpoint isregistered with the Reactor, which drives the eventloop that waits on the endpoint for connection requeststo arrive from Peers.

2. Service activation phase – Since an Acceptor in-herits from an Event Handler the Reactor candispatch the Acceptor’s handle event methodwhen connection events arrive. This method performsthe Acceptor’s Svc Handler initialization strat-egy. This strategy assembles the resources necessary tocreate a new Concrete Svc Handler object, ac-cept the connection into this object, and activate theSvcHandler by calling its open method.

3. Service processing phase – once activated, the SvcHandler processes incoming event messages arrivingon thePEER STREAM. ASvc Handlerwill processincoming event messages using a concurrent event han-dling pattern such as the Reactor or the Active Object[9].

RoutingRoutingTableTable

find()

OutputOutputChannelChannel

send_msg()put()

InputInputChannelChannel

recv_msg()

1

nn

I/O

I/O

LA

YE

RL

AY

ER

RO

UT

ING

RO

UT

ING

LA

YE

RL

AY

ER

MessageMessageQueueQueue

EVENT SOURCE AND SINKEVENT SOURCE AND SINK

Figure 14: Structure and Participants in the Router Pattern

Usage: Figure 13 illustrates how the Acceptor pattern isused by the Gateway. The Gateway uses this patternwhen it plays the passive connection role. In this case,Peers connect to Gateway, which uses the Acceptor pat-tern to decouple the passive initialization of Input andOutput Channels from the routing tasks performed oncea Channel is initialized.

The intent and general architecture of the Acceptor patternis found in network server management tools likeinetd [15]andlisten [16]. These tools utilize a master Acceptor pro-cess that listens for connections on a set of communicationports. Each port is associated with a communication-relatedservice (such as the standard Internet services ftp, telnet,daytime, and echo). When a service request arrives ona monitored port, the Acceptor process accepts the requestand dispatches an appropriate pre-registered handler that per-forms the service.

3.4 The Router Pattern

Intent: The Router pattern decouples multiple sources ofinput from multiple sources of output to route data correctlywithout blocking a Gateway.

Motivation and Forces: Message routing in a Gatewaymust not be disrupted or postponed indefinitely when con-gestion or failure occurs on incoming and outgoing networklinks. The Router pattern resolves the following forces thatarise when building robust connection-orientedGateways:

1. The need to prevent misbehaving connections from dis-rupting the quality of service for well-behaved connec-tions – If outgoing connections can flow control as aresult of network congestion, or input connections canfail because Peers disconnect, theGatewaymust notperform blockingsend or recv operations on any sin-gle channel. Otherwise, messages on other channelscould not be sent or received and the quality of serviceprovided to Peers would degrade.

2. The need to allow different concurrency strategiesfor Input and Output Channels – Several concur-

11

rency strategies for processing Input and OutputChannels are described in this paper including (1)single-threaded processing using the Reactor patternand (2) multi-threaded processing using the Active Ob-ject pattern. Each strategy is appropriate under differ-ent situations, depending on factors such as the num-ber of CPUs, context switching overhead, and numberof Peers. By decoupling Input Channels fromOutput Channels the Router pattern allows cus-tomized concurrency strategies to be configured flexiblyinto a Gateway.

Structure, Participants, and Implementation: Figure 14illustrates the layer structuring of participants in the Routerpattern. The I/O Layer provides an event source for InputChannels and an event sink for Output Channels.An Input Channel uses a Routing Table to maprouting messages onto one or more Output Channels.If messages cannot be delivered to their destination Peersimmediately they are buffered in a Message Queue forsubsequent transmission.

Because Input Channels are decoupled fromOutput Channels their implementations can vary inde-pendently. This separation of concerns is important since itallows different concurrency strategies to be used for inputand output. The consequences of this decoupling is discussedfurther in Section 3.5.

Collaborations: Figure 15 illustrates the collaborationamong participants in the Router pattern. These collabo-rations can be divided into three phases:

1. Input processing phase – whereInput Channels re-assemble incoming TCP segments into complete routingmessages;

2. Route selection phase – where Input Channelsconsult a Routing Table to select the OutputChannels responsible for sending the routing mes-sages;

3. Output processing phase – where the selected OutputChannels transmit the routing messages to their des-tination(s) without blocking the process.

Usage: The other strategic patterns in this paper (i.e., Reac-tor, Connector, Acceptor, and Active Object) can be appliedto many other types of communication software. In con-trast, the Router pattern is tightly coupled with theGatewayapplication. A primary challenge of building a reliableconnection-oriented Gateway centers on avoiding block-ing I/O. This is necessary to reliably manage flow controlon Output Channels. If the Gateway blocked indefi-nitely when sending on a congested connection then incomingmessages could not be routed, even if those messages weredestined for non-flow controlled Output Channels.

The remainder of this section describes how the Router pat-tern can be implemented in a single-threaded, Reactor versionof the Gateway (Section 3.5 examines the multi-threaded,


recv_msg()

find ()

I/OI/OLayerLayer


FIND DESTINATIONSFIND DESTINATIONS

ROUTE MSGROUTE MSG

main()main()

SEND MSGSEND MSG

((QUEUE IF FLOWQUEUE IF FLOW

CONTROLLEDCONTROLLED))

put()

wakeup()FLOW CONTROLFLOW CONTROL

ABATESABATES

DEQUEUE AND SENDDEQUEUE AND SEND

MSG MSG ((REQUEUE IFREQUEUE IF

FLOW CONTROLLEDFLOW CONTROLLED))


RECV MSGRECV MSG

send_msg()

INP

UT

PR

OC

ES

SIN

GP

HA

SE

RO

UT

ES

EL

EC

TIO

NP

HA

SE

OU

TP

UT

PR

OC

ES

SIN

GP

HA

SE

dequeue()

enqueue()

send_msg()

schedule_wakeup()

Figure 15: Object Interaction Diagram for the Router Pattern

Active Object version of the Router pattern). The Router pat-tern uses a Reactor as a cooperative multi-tasking sched-uler forGateway I/O operations, just like the Connector andAcceptor patterns. The Reactor allows multiple events ondifferent connections to be demultiplexed within a singlethread of control. The use of single-threading eliminatesthe overhead of synchronization (since access to shared ob-jects like the Routing Table need not be serialized) andcontext switching (since message routing occurs in a singlethread).

In the Reactor version of the Router pattern, the InputChannels and Output Channels inherit indirectlyfrom Event Handler. This enables the Gatewayto route messages by having the Reactor dispatchthe handle event methods of Input and OutputChannels when messages arrive and flow control condi-tions subside, respectively.

Using the Reactor pattern to implement the Router patterninvolves the following steps:

1. Initialize non-blocking endpoints – The Input andOutput Channel handles are set into non-blockingmode after they are activated by an Acceptor orConnector. The use of non-blocking I/O is essen-tial to avoid subtle errors that can occur on faulty orcongested network links.

2. Input message reassembly and routing – Routing mes-sages are received in fragments byInput Channels.If an entire message is not immediately available, theInput Channelmust buffer the fragment and returncontrol to the event loop. This is essential to prevent“head of line” blocking on Input channels. Whenan Input Channel successfully receives and framesan entire message it uses the Routing Table to de-termine the appropriate set of Output Channelsthat will deliver the message.

3. Message delivery – The selected Output Channelstry to send the message to the destination Peer. Mes-

12



7: put (msg)7: put (msg)

1: handle_event()1: handle_event()2: recv_msg(msg)2: recv_msg(msg)

3: find()3: find()

:: MessageMessageQueueQueue


5: nonblk_put(msg)5: nonblk_put(msg)6: send_msg(msg)6: send_msg(msg)

ROUTEROUTEIDID

SubscriberSubscriberSetSet

4: p

ut (m

sg)

4: p

ut (m

sg)


8: nonblk_put(msg)8: nonblk_put(msg)9: send_msg(msg)9: send_msg(msg)10: enqueue(msg)10: enqueue(msg)11: schedule_wakeup()11: schedule_wakeup()------------------------------12: wakeup()12: wakeup()13: dequeue(msg)13: dequeue(msg)14: send_msg(msg)14: send_msg(msg)


Figure 16: Using the Router Pattern in a Single-threadedReactive Gateway

sages must be delivered reliably in “first-in, first-out”(FIFO) order. To avoid blocking, all send operationsin Output Channels must check to make sure thatthe network link is not flow controlled. If it is not, themessage can be sent successfully. This path is depictedby theOutput Channel in the upper right-hand cor-ner of Figure 16. If the link is flow controlled, however,the Router pattern must use a different strategy. Thispath is depicted by theOutput Channel in the lowerright-hand corner of Figure 16.

To handle flow controlled connections, the OutputChannel inserts the message it is trying to send intoits Message Queue. It then instructs the Reactorto call back to the Output Channel when the flowcontrol conditions abate, and returns to the main eventloop. When it is possible to try to send again, theReactor dispatches the handle event method onthe Output Channel, which then retries the opera-tion. This sequence of steps may be repeated multipletimes until the entire message is transmitted success-fully.

Note that the Gateway always returns control to its mainevent loop immediately after every I/O operation, regardlessof whether it sent or received an entire message. This isthe essence of the Router pattern – it correctly routes themessages to peers without blocking on any single I/O channel.

3.5 The Active Object Pattern

Intent: The Active Object pattern decouples method exe-cution from method invocation to enable concurrent execu-tion of methods.

ClientClientInterfaceInterface

ResultHandle m1()ResultHandle m2()ResultHandle m3()

ActivationActivationQueueQueueinsert()

remove()

SchedulerScheduler

dispatch()m1'()m2'()m3'()

ResourceResourceRepresentationRepresentation

MethodMethodObjectsObjects

loop { m = actQueue.remove() dispatch (m)}

INVISIBLEINVISIBLETOTO

CLIENTSCLIENTS

VISIBLEVISIBLETOTO

CLIENTSCLIENTS

nn

11

1111

11

11

Figure 17: Structure and Participants in the Active ObjectPattern

Motivation and Forces: All the strategic patterns used bythe single-threadedGateway in Section 3.4 are layered uponthe Reactor pattern. The Connector, Acceptor, and Routerpatterns all use the Reactor as a scheduler/dispatcher to ini-tialize and route messages within a single thread of control.In general, the Reactor pattern forms the central event loop insingle-threaded reactive systems. For example, in the single-threaded Gateway implementation, the Reactor providesa coarse-grained form of concurrency control that serializesthe invocation of event handlers at the level of event demul-tiplexing and dispatching within a process. This eliminatesthe need for additional synchronization mechanisms within aGateway and minimizes context switching.

The Reactor pattern is well-suited for applications that useshort-duration callbacks (such as passive connection estab-lishment in the Acceptor pattern). It is less appropriate, how-ever, for long-duration operations (such as blocking on flowcontrolled Output Channels during periods of networkcongestion). In fact, much of the complexity in the single-threaded Router pattern implementation stems from using theReactor pattern as a cooperative multi-tasking mechanism. Itis much easier, therefore, to implement the output portion ofthe Router pattern with the Active Object pattern. This patternallows Output Channels to block independently whensending messages to Peers.

The Active Object pattern resolves the following force thatimpacts the design of applications like a Gateway that mustcommunicate simultaneously with multiple Peers:

� The need to allow blocking read and write operationson one endpoint that do not detract from the quality ofservice of other endpoints. Network services are gener-ally easier to program if blocking I/O is used rather thanreactive non-blocking I/O [17]. The increased simplic-ity occurs since the execution state can be localized inthe activation records of a thread of control, rather thanbeing decentralized in a set of control blocks maintained

13

INVOKEINVOKE

INSERT ININSERT IN PRIORITY QUEUE PRIORITY QUEUE

cons(m1')

remove(m1')DEQUEUE NEXTDEQUEUE NEXT METHOD OBJECT METHOD OBJECT

RETURN RESULTRETURN RESULT

EXECUTEEXECUTE

clientclient: Client: Client

InterfaceInterface: Activation: Activation

QueueQueue

insert(m1')

dispatch(m1')

ME

TH

OD

OB

JE

CT

ME

TH

OD

OB

JE

CT

CO

NS

TR

UC

TIO

NC

ON

ST

RU

CT

ION

SC

HE

DU

LIN

G/

EX

EC

UT

ION

CO

MP

LE

TIO

N

m1()

: Represent-: Represent-ationation

: Scheduler: Scheduler

CREATE METHODOBJECT

reply_to_future()

future()RETURN RESULTRETURN RESULTHANDLEHANDLE

Figure 18: Object Interaction Diagram for the Active ObjectPattern

by application developers.

Structure, Participants, and Implementation: Figure 17illustrates the structure and participants in the Active Ob-ject pattern. The Client Interface presents the pub-lic methods available to clients. The Scheduler deter-mines next method to execute based on synchronizationand scheduling constraints. The Activation Queuemaintains a list of pending Method Objects. TheScheduler determines the order in which these MethodsObjects are executed (a FIFO scheduler is used in theGateway to maintain the order of message delivery). TheResource Representation maintains context infor-mation shared by the implementation methods.

Collaborations: Figure 18 illustrates the collaborationsamong participants in the Active Object pattern. These col-laborations are divided into the following phases:

1. Method Object construction – in this phase the clientapplication invokes a method defined by the ClientInterface. This triggers the creation of a MethodObject, which maintains the argument bindings tothe method, as well as any other bindings required toexecute the method and return a result. For example, abinding to a Result Handle object returned to thecaller of the method. A Result Handle is returnedto the client unless the method is “oneway,” in whichcase no Result Handle is returned.

2. Scheduling/execution – in this phase the Scheduleracquires a mutual exclusion lock, consults theActivation Queue to determine which MethodObject(s) meet the synchronization constraints.The Method Object is then bound to the currentResource Representation and the method is al-lowed to access/update this representation and create aResult Handle.

3. Return result – the final phase binds the ResultHandle value, if any, to a future [18, 19] object that





4: put (msg)4: put (msg)

1: handle_event ()1: handle_event ()2: recv_msg(msg)2: recv_msg(msg)

3: find()3: find()



5: send_msg(msg)5: send_msg(msg)

5: send_msg(msg)5: send_msg(msg)

ACTIVEACTIVE

ACTIVEACTIVE

ROUTEROUTEIDID

SubscriberSubscriberSetSet

Figure 19: Using the Router Pattern in a Multi-threadedActive Object Gateway

passes return values back to the caller when the methodfinishes executing. A future is a synchronization objectthat enforces “write-once, read-many” synchronization.Subsequently, any readers that rendezvous with the fu-ture will evaluate the future and obtain the result value.The future and the Method Object will be garbagecollected when they are no longer needed.

Usage: The Gateway implementation described in Sec-tion 3.4 is single-threaded. It uses the Reactor pattern imple-mentation of the Router Pattern as a cooperative multi-taskingscheduler that dispatches events of interest to a Gateway.After implementing a number of single-threadedGatewaysit became clear that using the Reactor pattern as the basis forall Gateway routing I/O operations was error-prone andhard to maintain. For example, maintenance programmersfrequently did not recognize the importance of returning con-trol to the Reactor’s event loop immediately when I/O op-erations cannot proceed. This misunderstanding became acommon source of errors in single-threaded Gateways.

To avoid these problems, a number of multi-threadedGateways were built using variations of the Active Ob-ject pattern. The remainder of this section describes howOutput Channels can be multi-threading using the Ac-tive Object pattern.5 This modification greatly simplifiedthe implementation of the Router pattern since OutputChannels can block in their own thread of control with-out affecting other Channels. Implementing the OutputChannels as Active Objects also eliminated the sub-tle and error-prone cooperative multi-tasking programmingtechniques required when using the Reactor to scheduleOutput Channels.

Figure 19 illustrates the Active Object version of theRouter pattern. Note how much simpler is it compared with

5While it is possible to apply the Active Object pattern to the InputChannels this has less impact on the Gateway design because theReactor already supports non-blocking input.

14

the Reactor solution in Figure 16. The simplification occursprimarily since the complex output scheduling logic movedinto the Active Objects, rather than being the responsibilityof the application programmer.

It is also possible to observe the difference in complexitybetween the single-threaded and multi-threaded Gatewaysby examining the source code that implements the Routerpattern in production Gateway systems. 6 However, usingsource code to identify the reasons behind this complexityis hard due to all the error handling and protocol-specificdetails that surround the implementation. These details tendto disguise the key insight: the main difference between thecomplexity of the single-threaded and multi-threaded solu-tions arise from the choice of the Reactor pattern vs. theActive Object pattern. This paper has explicitly focused onthe interactions and tradeoffs between these patterns in or-der to clarify the consequences of different design choices.In general, documenting the interactions and relationshipsbetween closely related patterns is a very challenging andunresolved topic that is currented be addressed by the pat-terns community.

4 Related Work

[1, 5, 20] identify, name, and catalog many fundamentalobject-oriented design patterns. This section examines howthe patterns described in this paper relate to other patterns inthe literature. Note that many of the tactical patterns formthe basis for implementing the strategic patterns presented inthis paper.

The Reactor pattern is related to the Observer pattern [1].In the Observer pattern, multiple dependents are updated au-tomatically when a subject changes. In the Reactor pattern,a handler is dispatched automatically when an event occurs.Thus, the Reactor dispatches a single handler for each event(though there can be multiple sources of events). The Reac-tor pattern also provides a Facade [1]. The Facade patternpresents an interface that shields applications from complexrelationshipswithina subsystem. The Reactor pattern shieldsapplications from complex mechanisms that perform eventdemultiplexing and event handler dispatching.

The mechanism the Reactor uses to dispatch EventHandlers is similar to the Factory Callback pattern [21].The intent of both patterns is to decoupling event receptionfrom event processing. The primary different is the purposeof the pattern – the Factory Callback is a creational pattern,whereas the Reactor dispatching is a behavioral pattern.

The Connector pattern is a variation of the TemplateMethod and Factory Method patterns [1]. In the TemplateMethod pattern, an algorithm is written such that some stepsare supplied by a derived class. In the Factory Method pat-tern, a method in a subclass creates an associate that performsa particular task, but the task is decoupled from the protocol

6An ACE-based example of single-threaded and multi-threadedGateways that illustrates all the patterns in this paper is freely availablevia the WWW at http://www.cs.wustl.edu/�schmidt.

used to create the task. The Connector pattern is a Factorythat use Template Methods to create, connect, and activatehandlers for communication channels. In the Connector pat-tern, theconnectmethod implements a standard algorithmfor initiating a connection and activating a handler when theconnection is established. The intent of the Connector patternis similar to the Client/Dispatcher/Server pattern described in[5]. They both are concerned with separating active connec-tion establishment from the subsequent service. The primarydifference is that the Connector pattern addresses both syn-chronous and asynchronous connection establishment.

The Acceptor pattern can also be viewed as a variation ofthe Strategy and Factory Method patterns [1]. The Acceptorpattern is a connection factory that embodies the strategy forcreate service handlers, accepting connections into servicehandlers, and activating service handles to process data ex-changed across communication channels. The Acceptorimplements the algorithm that passively listens for connec-tion requests, then creates and activates a handler when theconnection is established. The handler performs a serviceusing data exchanged on the connection. Thus, the service isdecoupled from the network programming interface and thetransport protocol used to establish the connection.

The Router pattern is a variant of the Mediator pattern [1],which decouples cooperating components of a software sys-tem and allows them to interact without having direct depen-dencies among each other. The Router pattern is specializedto resolve the forces associated with network communication.It decouples the mechanisms used to process input messagesfrom the mechanisms used to process output mechanisms toprevent blocking. In addition, the Router pattern allows theuse of different concurrency strategies for input and outputchannels.

5 Concluding Remarks

This paper illustrates how a family of patterns have beenapplied to facilitate widespread reuse of design exper-tise and software components in production communicationGateways. These patterns illustrate the structure of, andcollaboration between, objects that perform core communi-cation software tasks. The tasks addressed by these patternsinclude event demultiplexing and event handler dispatching,connection establishment and initialization of application ser-vices, concurrency control, and routing.

The family of design patterns and the ACE frameworkcomponents described in this paper have been reused by theauthor and his colleagues in many production communica-tion software systems ranging from telecommunication andelectronic medical imaging projects [12, 3] to academic re-search projects [10]. In general, patterns aid the developmentof components and frameworks in these systems by capturingthe structure and collaboration of participants in a softwarearchitecture at a higher level than (1) source code and (2)object-oriented design models that focus on individual ob-jects and classes.

15

Our experience applying a design pattern-based reuse strat-egy has been quite positive [2]. For instance, we’ve signifi-cantly reduced the software maintenance and training effortfor the production communication systems by documentingthe intent, structure, and behavior of ACE components interms of the patterns they reify. Focusing on patterns hasalso enabled us to reuse software architecture even whenreuse of algorithms, implementations, interfaces, or detaileddesigns was not feasible due to differences in OS platforms[12]. An in-depth discussion of our experiences and lessonslearned using patterns appeared in [2].

Acknowledgements

I would like to thank Steve Berczuk, Chris Cleeland, TimHarrison, Hans Rohnert, and the anonymous referees forcontributing valuable suggestions that helped improve thequality of this paper.

References[1] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design

Patterns: Elements of Reusable Object-Oriented Software.Reading, MA: Addison-Wesley, 1995.

[2] D. C. Schmidt, “Experience Using Design Patterns to DevelopReuseable Object-Oriented Communication Software,” Com-munications of the ACM (Special Issue on Object-OrientedExperiences), vol. 38, October 1995.

[3] I. Pyarali, T. H. Harrison, and D. C. Schmidt, “Design andPerformance of an Object-Oriented Framework for High-Performance Electronic Medical Imaging,” in Proceedingsof the 2nd Conference on Object-Oriented Technologies andSystems, (Toronto, Canada), USENIX, June 1996.

[4] D. C. Schmidt and T. Suda, “Measuring the Performance ofParallel Message-based Process Architectures,” in Proceed-ings of the Conference on Computer Communications (INFO-COM), (Boston, MA), pp. 624–633, IEEE, April 1995.

[5] F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, andM. Stal, Pattern-Oriented Software Architecture - A System ofPatterns. Wiley and Sons, 1996.

[6] G. Booch, Object Oriented Analysis and Design with Ap-plications (2nd Edition). Redwood City, California: Ben-jamin/Cummings, 1993.

[7] D. C. Schmidt, “Reactor: An Object Behavioral Pattern forConcurrent Event Demultiplexing and Event Handler Dis-patching,” in Pattern Languages of Program Design (J. O.Coplien and D. C. Schmidt, eds.), Reading, MA: Addison-Wesley, 1995.

[8] D. C. Schmidt, “Acceptor and Connector: Design Patterns forInitializing Communication Services,” in The 1st EuropeanPattern Languages of Programming Conference (WashingtonUniversity technical report #WUCS-97-07), July 1997.

[9] R. G. Lavender and D. C. Schmidt, “Active Object: an ObjectBehavioral Pattern for Concurrent Programming,” in Proceed-ings of the 2nd Annual Conference on the Pattern Languagesof Programs, (Monticello, Illinois), pp. 1–7, September 1995.

[10] D. C. Schmidt, “ACE: an Object-Oriented Framework forDeveloping Distributed Applications,” in Proceedings of the

6th USENIX C++ Technical Conference, (Cambridge, Mas-sachusetts), USENIX Association, April 1994.

[11] D. C. Schmidt, T. H. Harrison, and E. Al-Shaer, “Object-Oriented Components for High-speed Network Program-ming,” in Proceedings of the 1st Conference on Object-Oriented Technologies and Systems, (Monterey, CA),USENIX, June 1995.

[12] D. C. Schmidt and P. Stephenson, “Experiences Using De-sign Patterns to Evolve System Software Across Diverse OSPlatforms,” in Proceedings of the 9th European Conferenceon Object-Oriented Programming, (Aarhus, Denmark), ACM,August 1995.

[13] M. A. Linton, J. Vlissides, and P. Calder, “Composing UserInterfaces with InterViews,” IEEE Computer, vol. 22, pp. 8–22, February 1989.

[14] D. Ritchie, “A Stream Input–OutputSystem,” AT&T Bell LabsTechnical Journal, vol. 63, pp. 311–324, Oct. 1984.

[15] W. R. Stevens, UNIX Network Programming, First Edition.Englewood Cliffs, NJ: Prentice Hall, 1990.

[16] S. Rago, UNIX System V Network Programming. Reading,MA: Addison-Wesley, 1993.

[17] D. C. Schmidt and C. D. Cranor, “Half-Sync/Half-Async: anArchitectural Pattern for Efficient and Well-structured Con-current I/O,” in Pattern Languages of Program Design (J. O.Coplien, J. Vlissides, and N. Kerth, eds.), Reading, MA:Addison-Wesley, 1996.

[18] R. H. Halstead, Jr., “Multilisp: A Language for Concur-rent Symbolic Computation,” ACM Trans.ProgrammingLan-guages and Systems, vol. 7, pp. 501–538, Oct. 1985.

[19] B. Liskov and L. Shrira, “Promises: Linguistic Support forEfficient Asynchronous Procedure Calls in Distributed Sys-tems,” in Proceedingsof the SIGPLAN’88 Conferenceon Pro-gramming Language Design and Implementation, pp. 260–267, June 1988.

[20] J. O. Coplien and D. C. Schmidt, eds., Pattern Languages ofProgram Design. Reading, MA: Addison-Wesley, 1995.

[21] S. Berczuk, “A Pattern for Separating Assembly and Process-ing,” in Pattern Languages of Program Design (J. O. Coplienand D. C. Schmidt, eds.), Reading, MA: Addison-Wesley,1995.

16

a family of design patterns for application-level gatewaysschmidt/pdf/tapos-95.pdf ·...

Documents