microsoft journal2 (v2)€¦ · dear architect the advances in technology and industry standards...

Dear ArchitectThe advances in technology andindustry standards that enable serviceoriented architecture (SOA) are alsodriving a paradigm shift in clientarchitecture. Individually these changesmay be considered evolutionary, but inaggregate they unite to revolutionizeclient architecture. Historically, client-side applications were tightly boundto a specific backend system. Moreprecisely, the primary function of theclient-side application was to exposethe functionality and data of a specificbackend system to users. While it waspossible to build clients that couldinteract with multiple backend systems,doing so was extremely complex andseldom undertaken.

SOA has changed all of that. In aSOA, the functionality and data of

backend systems are exposed asservices (or fronted by mid-tier webservices), and those services areconsumed by client applications. Theclient application is no longer tightlybound to a single backend system.Businesses now have the freedom,and means, to efficiently buildradically new client applications thattruly optimize user productivity andstreamline workflow.

The design imperative for client sideapplications has shifted from system-centric models of connecting the userpopulation at large to specific systems,to user-centric models that provideusers with transparent access to all theservices they need for their job or task.If one extends this model by seamlesslyintegrating these services with the rich

capabilities of desktop applications,then one can achieve something whichredefines the standards for operationalexcellence. And furthermore, by givingequal treatment to both user andsystem models within a convergentservice-centric model, one cansignificantly enhance the integrationbetween them with seamlessfunctional interoperation.

In this edition of JOURNAL, we beginthe intriguing journey to exploreperspectives of these various models.

Enjoy!

Chris CaposselaVice President,Information Worker Business Group,Microsoft Corporation

JOURNAL4 MICROSOFT ARCHITECTS JOURNAL OCTOBER 2004 A NEW PUBLICATION FOR SOFTWARE ARCHITECTS

JOURNAL4

Choosing the RightPresentation LayerArchitectureDavid Hill, Microsoft Corporationpp 04 – 13

Information BridgeFrameworkRicard Roma i Dalfó,Microsoft Corporationpp 14 – 18

Benchmarking aTransaction EngineDesignRichard Drayton, FiS and Arvindra Sehmi, Microsoft EMEApp 19 – 33

Enterprise ArchitectureAlignment HeuristicsPedro Sousa, Carla Marques Pereiraand José Alves Marques,Link Consulting, SApp 34 – 39

RazorbillsMaarten Mullender,Microsoft Corporationpp 40 – 49

Next Generation Tools for OODOren Novotny,Goldman, Sachs & Copp 50 – 52

1The Nerd, The Suit and the FortuneTeller, Tech-Ed Europe 2004.

JOURNAL4 | Editorial 2

Dear ArchitectWelcome to the autumn issueof JOURNAL.

It has been several years now since.NET hit the streets and createdmomentum around Microsoft’s visionof connecting people and processestogether, anytime, anywhere and on anydevice. This vision was built on Webservices standards implementations on.NET and broad adoption in thetechnical community. Both prerequisiteshave been achieved, not only on .NETbut also on other vendor platformofferings. With this has come a betterunderstanding of the new possibilitiesfor application architectures, specificallySOA which, I would argue, is the firstexpression of that understanding.

We are seeing the architectural ideasbehind SOA being adopted in manyareas of the overall system solution.These are not restricted just to theapplication layer because of resultingproductivity and business benefits. Forinstance, integration, interoperability,management, operations, testing,security, data and user interfaceaspects of system solutions can each beviewed from the perspective of service-orientation. A so-called service orientedconvergence (SOC) phenomenon istaking place, at least conceptually,amongst the architectural thinkersI have been working with lately.1

In this issue of JOURNAL there arethree papers which will add somecredence to this way of thinkingespecially as it relates to the role ofservice consumers in SOC and

particularly the role of smart clientsand task-oriented user interfaces.

We start with a paper by David Hill,a member of the Microsoft ArchitectureStrategy team, who contrasts thin andsmart client architecture approachesand provides guidance on how to choosebetween them. David shows us that“smart clients are rich clients doneright”, leveraging new technology andtechniques to avoid the pitfalls oftraditional rich client applicationswhilst aligning neatly with theprinciples of convergent service-centricsystem models.

Ricard Roma i Dalfó from Microsoft’sOffice division discusses in his articlea new metadata-driven approach tobuilding task-oriented serviceconsumers directly within MicrosoftOffice applications. The InformationBridge Framework (IBF) goes beyonda service’s WSDL metadata byautomating the construction of user-interfaces from metadata descriptionsof the entities that constitute consumedservices. It does this by describingentity views, their relationships andidentity references, and operationswhich can be performed on them.

Performance is probably one of theleast understood and misquotedmetrics in the field of computing today.However, as a metric for systemevaluation, it is considered by mostusers to be one of the most importantand critical factors for assessing thesuitability of systems. Richard Draytonof FiS and Arvindra Sehmi from theEnterprise Architecture team in

Microsoft EMEA write a paper onbenchmarking a scalable transactionengine which has an architecturebased on loosely coupled, stateless,message processing componentsarranged in a queuing network. Thisarchitecture proved invaluable froma benchmarking perspective becauseit is backed up by sound mathematicaltechniques which enable an assessmentof the impact on performance of variousimplementation realization techniquesa-priori to deciding on any specificrealization technologies. Therefore,measured benchmark metrics inheritand benefit from these mathematicallycredible foundations.

Pedro Sousa, Carla Marques Pereiraand José Alves Marques, all from LinkConsulting, follow with their paperdiscussing how, within the context ofvarious Enterprise ArchitectureFrameworks each with their ownconcepts, components, andmethodologies, the most importantconcern for architects is alignment.This concern may lead to considerationof simpler architecture conceptsand simpler methodologies because thefocus is not to define developmentartifacts but to check their consistency.

Next, Maarten Mullender, SolutionsArchitect in Microsoft’s ArchitectureStrategy team introduces us toRazorbills which is a proposal fordescribing ‘what’ services do, therebydramatically improving the usefulnessof service consumption. There aresimilarities with the second article onIBF where the description is done interms of entities, views, actions,

Keep updated with additional informationat http://msdn.microsoft.com/architecture/journal

EditorialBy Arvindra Sehmi

JOURNAL4 | Editorial 3

references and relationships but theemphasis is on helping businessanalysts and users to better bridgethe gap between structured andunstructured, and formal and informalcontent and process. Maarten goesfurther to discuss the relevance ofthis approach in entity aggregationat informational and functional levels,user interaction and collaboration.

The final paper is a short piece on NextGeneration Tools by Oren Novotny ofGoldman, Sachs & Co. In this he arguesthat development tools must supportthe shift that languages have alreadymade and fully support model-drivendevelopment. He insists source codefiles should be eliminated as they nolonger serve a useful function in thiscontext and the current rationale forusing them is made irrelevant by betterand different means in these nextgeneration tools.

Please visithttp://msdn.microsoft.com/architectureto keep up-to-date on architecturethinking at Microsoft and toconveniently downloadJOURNAL articles.

As always, if you’re interested inwriting for this publication please sendme a brief outline of your topic andyour resume to [email protected].

Wishing you a great holiday season andgood reading!

Arvindra SehmiArchitect, D&PE,Microsoft EMEA

JOURNAL4 | Choosing the Right Presentation Layer Architecture 4

AbstractThe presentation layer is a vitallyimportant part of an application –an inappropriately architectedpresentation layer can lead to too muchcomplexity, a lack of flexibility and aninefficient and frustrating userexperience. Thin client applicationshave well known benefits overtraditional rich client applicationsin terms of deployment andmanageability and this has led to theirpopularity in recent years. However,with the advent of smart clients, thechoice of presentation layerarchitecture is no longerstraightforward. Rich clients haveevolved into smart clients that cancombine the central managementadvantages of thin clients with theflexibility, responsiveness, andperformance of rich clients. This articlediscusses the thin and smart clientapproaches and provides guidance onhow to choose between them.

The Importance of thePresentation LayerThe presentation layer of mostapplications is very often critical to theapplication’s success. After all, thepresentation layer represents theinterface between the user and the restof the application. It’s where the rubberhits the road, so to speak. If the usercan’t interact with the application ina way that lets them perform theirwork in an efficient and effectivemanner, then the overall success of theapplication will be severely impaired.

Personally, I think the termpresentation layer really does not do

justice to the function and importanceof this layer. It is rarely about justpresenting information to user – it’soften more about providing the userwith interactive access to theapplication. Perhaps a moreappropriate name for this layer is theuser interaction layer. For simplicity,though, for this article I’ll stickwith the commonly accepted name forthis layer.

In any case, you’ll want to design thislayer to provide the user with the rightuser experience, so that they caninteract with the application in aneffective and efficient way. Of courseyou also need to architect and thenimplement this layer in such a waythat it adequately takes into accountthe development, maintenance, andoperational needs of the business.Choosing the right architecture for thepresentation layer of the application isvitally important in order to achieve allof these goals.

The two commonly adopted approachesto presentation layer architecture anddesign are the thin client approach andthe smart client approach. Of course,many factors influence the decisionabout which approach is best for aparticular application – for example,client platform requirements,application deployment and update,user experience, performance, client-side integration, offline capabilities, etc.– and each has inherent strengths andweaknesses and naturally supports acertain style of application. You’ll find,however, that the distinction betweenthem can blur, and this can easily lead

to the wrong basic approach beingapplied leading to problems later on.

For example, it is possible to providea rich user interface with a browser-based presentation layer, just as it ispossible to provide a completelydynamic user interface with a smartclient. Neither would be very easy toachieve, and both would very likelyresult in unneeded complexity, a lackof flexibility and high development andmaintenance costs.

Many organizations choose a thinclient architecture by default withoutduly considering the alternatives.While they will not be appropriatefor all scenarios, a smart clientarchitecture can offer significantadvantages over a thin client approachwithout incurring the downsidestraditionally associated with richclients. Organizations should carefullyconsider each approach so that theyadopt the right approach from theoutset, minimizing the TCO over thelifetime of the application.

In the following sections, I’ll examinethe thin and smart client approachesand some of the technologies behindthem. For each I’ll describe the basicarchitecture and discuss some of thedesign options within each. After thatI’ll talk about the relative strengthsand weaknesses of each approach interms of a number of common factorsand requirements that you shouldtake into account when determiningthe most appropriate approach foryour application.

“A smart client architecture can offer significant advantages over a thin client approach withoutincurring the downsides traditionally associated with rich clients.”

Choosing the Right Presentation Layer ArchitectureBy David Hill, Microsoft Corporation


What Is a Thin Client?Many thin client technologies pertainto the server side and there are manyweb server platforms and frameworks(ASP, ASP.NET, JSP, and others) tochoose from. Each has particularfeatures that try to make it easier towrite thin client applications, but theyall deliver the user interface to abrowser on the client through a seriesof HTML pages. A thin clientapplication is pretty easily defined asone that uses a browser to provide theexecution environment for theapplication’s (HTML defined) userinterface.

In addition to rendering the userinterface and allowing the user tointeract with it, the browser alsoprovides generic security, state

management, and data handlingfacilities plus the execution environmentfor any client-side logic. For the latter,the browser typically provides a scriptengine and the ability to host otherexecutable components; such as JavaApplets, ActiveX and .NET controls,and so on, (though most definitionswould not consider these executablecomponents to be thin client technologies– see hybrid applications below).

An application architected to use a thinclient presentation layer is decomposedinto pages and each page is “deployed”to the client upon request. Each pagecontains the user interface descriptionand, typically, a small amount of client-side script logic and a small amount ofstate/data (view-state, cookies, XMLdata islands, etc.). Figure 1 shows a

schematic representation of a thinclient presentation layer architecture.

The browser has a limited ability tointeract with the client environment(hardware and other softwareapplications running on the client).It does provide a mechanism thatpermits storage of small amounts ofdata on the client (via cookies), andsometimes the ability to cache pages,but typically these facilities are oflimited use except as a way to providesimple session managementor tracking, and rudimentary read-onlyoffline capabilities, respectively.

The browser also provides the securityinfrastructure so that differentapplications (pages) can have more orfewer permissions assigned so thatthey can do different things aroundstate (such as cookies), they can hostcomponents, and execute scripts.Internet Explorer implements thesefacilities through different zones,trusted sites, ratings, etc.

In an attempt to provide a richer andmore responsive user interface, someweb applications have resorted toDHTML and similar technologies tofacilitate a richer user interface. Whilethese technologies are non-standard,in the sense that all browsers do notsupport them in the same way, they doprovide the ability to include moreadvanced user interface elements suchas drop down menus, drag and drop,and more with in a web page.

Other web applications have resortedto hosting complex components within

Page Deployment

Page BusinessLogic

Server Roundtrip

No Offline Support

BrowserSecuritySandbox

Browser

Page

UI

Data UI

Limited LocalData Support

Client Web Server

Figure 1: Schematic Overview of Thin Client Architecture


the page including Java Applets,ActiveX and .NET components.These components provide either amore responsive user interface orclient-side logic that cannot beimplemented in script for performanceor security reasons. This is where thethin client starts to overlap with smartclient leading to so-called hybridapplications.

While you can certainly use suchhybrid applications to leverage thestrengths and weaknesses of eachapproach, in this document, I’ll definethe term thin client to mean a genericweb application that does not rely onsuch components but just utilizes thebasic facilities provided by the browserenvironment. Since hybrid applicationsneed to rely on smart client capabilitiesto avoid management and operationalproblems, I’ll describe hybridapplications in a later section whenI discuss smart client applications.

What Is a Smart Client?Smart client applications are maybenot as easy to define as thin clientapplications because they can takemany different forms and are notlimited to the one-size-fits-all approachof thin client applications. Theessential difference between a smartclient and a thin client is that a smartclient does not rely on a browser toprovide the execution, security anduser interface environment for itsoperation. Also, smart clients, ratherthan HTML and Jscript, typicallyinvolve compiled code artifacts(components, assemblies, etc.) runningon the client machine to provide theapplication’s user interface and client-side logic.

How do smart clients relate to richclients? Rich client applications haveevolved into smart client applications.Rich clients offered many advantagesover thin client applications including

improved performance, responsivenessand flexibility, and the ability to workoffline, but rich clients suffered from anumber of operational problems whenit came to deploying and updating themin a robust way. Thin client solutionsof course excel in the deployment andupdate area and this is one of the mainreasons for their popularity.

However, smart client applicationsrepresent a best-of-both-worldsapproach by taking the manageabilityadvantages of thin client applicationsand combining them with the benefitsof rich client applications. Smart clientsare rich clients done right, leveragingnew technology and techniques toavoid the pitfalls of the traditional richclient applications.

For example, smart client applicationsbuilt on the .NET platform can takeadvantage of a number of fundamentaltechnologies that the .NET Frameworkprovides to solve many of the problemstraditionally associated with rich clientapplications. While it has always beenpossible to build rich client applicationsthat minimized or avoided thedeployment and security drawbacks,the facilities provided by the .NETFramework make it very much easierto do.

.NET provides the ability to deploy anapplication, or part of an application,from a web server. This technology,known as No-Touch Deployment, letsyou deploy applications via a URL. Thisallows you to release applications to acentral location (i.e. to a web server) sothat the applications can be deployedautomatically to the client on demand.All clients can be automatically kept upto date since the application checks forupdates automatically each time the

“Smart clients are rich clients done right, leveraging new technology and techniques to avoidthe pitfalls of traditional rich client applications.”

Deployment

SmartClient

BusinessLogic

Offline Support

CAS SecuritySandbox

Smart Client

User Interface

Business Logic

Local DataCaching

WebService

Client Web Server

Figure 2: Schematic Overview of a Smart Client Architecture


application runs and each clientapplication downloads the new codeif required.

.NET also provides the Code AccessSecurity (CAS) infrastructure. CASassigns .NET code-specific permissionsaccording to the evidence that itpresents. CAS serves much the samefunction as the browser does in a thinclient application, providing a sandboxenvironment in which the applicationoperates. No-Touch Deploymentintegrates with (CAS). By default,applications that are deployed usingNo-Touch Deployment will be granteda restricted set of permissionsaccording to the URL zone from whichthey were deployed. Networkadministrators can modify permissionsusing security policy so that theapplication can be granted, or denied,specific permissions according tothe requirements.

Creating a smart client applicationwith the .NET Framework providesless fragile applications. Traditionally,installing a rich client application couldbreak other applications as it replacedimportant components and DLLs thatwere shared by other applications..NET allows applications to be isolated,keeping all application artifacts in alocal directory so that all assembliesare kept separate. Furthermore, suchapplications do not require anyregistration process when they aredeployed, further reducing the riskof breaking other applications. Inaddition, the .NET Framework allowsmultiple versions of an assembly to bedeployed side by side. This ensuresthat when an application executes,it runs with the exact versions of theassemblies that it was built andtested with.

An application architected to use asmart client for its presentation layerwill typically provide a centraldeployment server, from which thesmart client artifacts can be deployedto the clients, and a number of webservices to provide access to the back-end business capabilities – businesslogic and data – and which areconsumed by the smart client. Sincethe smart client is running code onthe client, it can more cleanly separatethe user interface from the client-sidedata and logic. In addition, dependingon the permissions it has been granted,it can more freely interact with otherclient-side resources such as localhardware and other software runningon the client. Figure 2 shows aschematic overview of this architecture.

What does a smart client look like?Smart client applications can takemany forms and the architect of suchan application faces a number of designchoices. The first decision to be made isto choose the most appropriateapplication style – the way in which thesmart client is presented to the user.In general, there are three ways todesign a smart client application:

– Windows Applications. TraditionalWindows style applications, typicallybuilt using Windows Forms or mobileapplications built on the .NETCompact Framework.

– Office Applications. Microsoft Officeprograms that are extended toinclude smart client capabilities,connecting the user to line ofbusiness applications and businessprocesses.

– Hybrid Applications. Applicationsthat utilize a combination of thinand smart client technologies. Forexample, by hosting Windows Forms

controls within a browser page orby hosting the browser within aWindows Forms application.

Choosing the right application style iscrucial if you want to fully realize thebenefits of a smart client approach.Deployment, security, development, andoffline capabilities all affect the choiceof smart client application style butperhaps the biggest factor to consider isthe overall user experience. Each choicerepresents a different type of userexperience and choosing the right onewill give the user the right combinationof flexibility and performance they need.

Windows ApplicationsUsers associate smart clientapplications with traditional Windows-style applications because they providerich client functionality that includestoolbars, menubars, context menus,drag and drop support, context sensitivehelp, undo/redo, and more. Developerscan build these kinds of smart clientapplications on the .NET Frameworkor the .NET Compact Framework usingWindows Forms to provide these richuser interface features.

These developers can also takeadvantage of pre-built smart clientfunctionality by leveraging theApplication Blocks provided by theMicrosoft Patterns and Practices group.These blocks provide the application withcommon smart client capabilities such aslocal data caching, seamless deploymentand the ability to work offline.

Windows Forms applications providethe most control over the userexperience, allowing the developer tocraft the user interface and userinteraction model to suit their exactneeds. For applications which require


a specific user experience which cannotnaturally be provided by any of theOffice applications, this approach willbe the best fit.

Office 2003 Smart ClientApplicationsMicrosoft Office programs provide anextremely compelling platform forbuilding smart client solutions.Extending Office applications so thatthey form part of a distributed solution,connecting them to remote data sourcesand business services, not only benefitsthe users, it also brings efficiencies tothe developers that write theapplications, and to those that mustdeploy and manage them.

There are many users who are familiarwith Office and use it everyday in theirwork. Extending Office applications byconnecting them to remote data sourcesand business services means that thesolution can benefit from the user’sfamiliarity, obviating or drasticallyreducing the need to re-train the user.The user benefits too, since they cancontinue to use the application withwhich they are familiar.

Many organizations extensively useMicrosoft Office. Most business PCs –yours, your customers, your serviceproviders, and your suppliers – haveOffice applications installed. UsingOffice as the client for line of businesssystems can reduce the need to installand maintain incremental clientapplications to access backend datasources and services. And very often,data from line of business applicationsis copied into Office applications suchas Word or Excel for furthermanipulation, editing, analysis, andpresentation. Copying and pasting istime consuming and introduces thepotential for errors. More importantly,

the link to the data is lost so the userneeds constant refreshes, repeating thecopy and paste process, and possiblyintroducing concurrency problems.

Office applications can also provide alot of the functionality that is requiredto display and manipulate data,allowing the user to interact with thesolution using the full power of Office.This can save a huge amount of timeand effort allowing you to develop andrelease a solution must faster. Forexample, Excel provides powerfulcapabilities to sort, manipulate, anddisplay data. Reusing these capabilitiesin your smart client solution can bevery cost effective.

Of course, users have the ability tointegrate additional functionality intotheir Office applications for a while. Insome cases this has led to ad-hoc butbusiness-critical solutions that aredifficult to manage because they aren’tdeveloped or maintained by the ITdepartment. Building these solutionsusing smart client technologies allowsthem to be more easily deployed andupdated and represents a way to retainthe value of the solutions whilst solvingsome of the manageability issues.

Office 2003 provides support forintegrating smart client capabilitiesinto Office application and connectingthem to remote services that provideaccess to data and business processes.Some of the more importanttechnologies Office 2003 supports forcreating smart client solutions include:

– XML Support. Office 2003 providesa number of facilities that allowdevelopers to more easily connectOffice applications to remote datasources and business processthrough XML.

– Word, Excel and InfoPath can useXML to store the structure andcontents of a document in humanor machine readable XML form.Microsoft has released W3C-compliant XSD schemas for thesefile formats and these schemas arefreely available for everyone to usein their own solutions. Theseschemas allow Word and Exceldocuments and InfoPath forms tobe easily constructed on the serverand provided to the client throughXML web services and users canreadily display and edit thesedocuments. This technology canalso be used to provide documentcomposition, indexing or searchingfunctionality. And of course, sincethese documents are XML, theycan be exchanged with any othersystem or process, providinga means for data interchangeacross heterogeneous systems.This technology is ideally suitedto document-centric solutions.

– Word, Excel, and InfoPath can alsoconsume XML messages ordocuments that conform to customor user-defined schema. Users canuse their Office applications aspresentation layer services in data-centric solutions where thebusiness processes or servicesalready define the message schema.This type of smart clientapplication maps elements andattributes in the message to specificareas of the document so that theOffice application can display themappropriately and allow the user toedit the values whilst ensuring thatthe data entered by the userconforms to the underlying schema.Specific values can be queried, set,or referenced programmaticallyusing an XPath query statement.

“Extending Office applications so that they form part of a distributed solution, connecting them toremote data sources and business services, not only benefits the users, it also brings efficienciesto the developers that write the applications, and to those that must deploy and manage them.”


– Smart Documents. Smart documentsolutions help the user to interactwith a document by providingadditional data and guidance to theuser according to their currentlocation within the document. As theuser interacts with the document, itcan display relevant informationor guidance to the user using the taskpane, or it can automatically fill inmissing data according to the currenttask. Connecting this experienceto remote services to obtain live dataor to enable interaction with businessprocesses allows powerful andintegrated applications to be built.

– Information Bridge Framework (IBF).IBF is a declarative solution thatbuilds on smart document technologyto allow documents to be connected toservices through metadata. Smarttags within an Office applicationinteract with the generic IBFinfrastructure and the metadataassociated with the available webservices to provide access to relevantdata and business processes fromwithin the document according to thedocuments contents and the user’scurrent activity. For example, if auser receives a document that refersto a specific supplier, the IBFinfrastructure can access data aboutthat company and display it in thetask pane. It can also provide accessto available options, allowing thedocument to be connected to otherbusiness processes.

– Visual Studio Tools for Office (VSTO).VSTO provides access to the objectmodels for Word and Excel tomanaged code extensions. Developerscan build complex and comprehensiveOffice smart client solutions usingVSTO to not only provide access tothe full power of Word and Excel but

also to all of the features of the .NETFramework, such as Windows Forms,that enable rich and responsive userinterfaces to be easily integrated.VSTO also provides a superiordevelopment experience, allowing thedeveloper to easily create and debuga solution. VSTO essentially providesthe code behind the document to forma solution that leverages the facilitiesprovided by the “host” application.

Hybrid ApplicationsHybrid smart client applicationscombine the smart client and thinclient approaches. They can provide away to extend an existing thin clientapplication with smart clientcapabilities, or a way to integrate abrowser-based application into a smartclient application.

For example, a smart client applicationmay host an instance of a browser sothat certain content and applicationfunctionality can be provided using thethin client approach. This architecturecan be very useful when the applicationneeds to integrate an existing thinclient application, or when it needs toleverage a key benefit of the thin clientapproach to provide linked dynamiccontent provided by a web server. Ofcourse, such content and functionalitywill only be available when the user isonline but the smart client part of theapplication can be used to provideuseful functionality when offline andenhance the application with access tothe thin client functionality when online.

In some cases, the hybrid approachcan be used to extend an existing thinclient application by hosting smartclient controls or components withina web page. These components canprovide a rich and responsive userinterface and specific application

functionality (for example, renderingand visualizing data) while the rest ofthe application is delivered in a thinclient way. However, this architecture isnot suitable for providing offlinesupport since the hosting web page willnot be available without a connection,or for providing client-side integrationof software or hardware unless suitablesecurity policy changes are in place.

Choosing the Right InteractionLayer ArchitectureBoth the thin client and the smart clientapproaches clearly have their place. Eachhas its own strengths and weaknessesand the choice between them willdepend on the requirements of a specificapplication or business need. The correctapproach will provide the user withthe right user experience so that theycan interact with the application in aneffective and efficient way, whilstadequately taking into account thedevelopment, maintenance, andoperational aspects of the application.

Some organizations have a policy thatdictates a thin client approach for allapplications. Choosing a thin clientapproach by default can lead tosignificant technical problems for someapplications because the browserplatform is not able to easily supportthe requirements of moderatelycomplex applications. Developing athin client application to have the look,feel, and capabilities of a traditionalrich client application can be extremelychallenging and costly. Why? Thebrowser imposes severe limitationson the developer in terms of statemanagement, client-side logic, client-side data, and the provision of rich userinterface features such as drag anddrop, undo/redo, etc.


Conversely, choosing a smart clientapproach for all applications is notappropriate since it can result in overlycomplex solutions for applications thatjust present data dynamically andrequire the benefits of a highlydynamic user interface. Also, if yourapplication must support multipleclient operating systems, a smart clientapproach may not be appropriate dueto cross-platform restrictions.

Adopting a single approach for allapplications is therefore likely to resultin unnecessary cost, complexity, a lackof flexibility and reduced usability.Both approaches can likely co-existwithin the enterprise, according to therequirements of specific applicationsand the needs of the business. Choosingone approach over the other should bedecided on a per-application basis,although in some cases bothapproaches can be combined, eitherby integrating thin and smart clienttechnologies appropriately or byadopting a dual channel approachwhere some class of users can accessthe application using a thin client whileusers with more stringent requirementsuse a smart client. Either way, the keyis to leverage the appropriatetechnology at the appropriate time tofulfill the expectations of the users andthe overall needs of the business.

The thin client approach has well-known benefits in terms of reach andease of deployment and operation.However, with the advent of smartclient technology, smart clients arecatching up in this area and are nowa viable alternative to thin clients formany scenarios. In particular, smartclients do not suffer from thedeployment and management problemsthat rich client solutions suffered from,and smart client solutions add benefits

in terms of flexibility, responsiveness,and performance.

So, if deployment and manageabilityare no longer the dominant factors thatinfluence the decision between the twoapproaches, how does one choosebetween them? With the erosion ofthe relative benefits of the thin clientapproach in this area, the balance ischanging and the factors which haveto be considered have increased.Depending on the relative priority ofthe requirements, one approach willbe more suitable than the other, andchoosing the correct approach willresult in a faster and less complexdevelopment process, improved easeof use, and higher user satisfaction.

The features, advantages anddisadvantages of each approach aredescribed above but how do thesetranslate into a decision given therequirements of a specific application?Some of the more important factorsthat companies must consider include:

– Client platform requirements– Deployment and update

requirements– User experience requirements– Performance requirements– Client-side integration requirements– Offline requirements

This is not a comprehensive list offactors and your company’s ITdepartment may add other factors thatare critically important to specificapplications. In particular, this list offactors is focused on operational orfunctional requirements anddevelopment and design-time factorshave not been included here. Still,these factors may be of sufficientimportance to sway the balancebetween one approach and the other.

Deciding on the right approach is ajoint decision between the IT staff andbusiness owners. The adopted approachshould lead to a solution where bothgroups are happy: the IT from themanagement side, and the businessowners from the functionality side.

Client Platform Client platform flexibility can beimportant for a customer or partnerfacing applications where the principalusers are external to the organizationand for which a specific client platformcannot be mandated, or for applicationsthat have to be accessible from non-Windows operating systems.

Thin clients offer the ability to targetmultiple client platforms, though thisoften requires the application todetermine the exact type of targetplatform so that it can change itsoperation or behavior to accommodatethe differences between the variousbrowsers, especially when the targetplatform must include mobile devices.The thin client framework itself canhandle many of these differences. Forexample, ASP.NET on the server candetermine the target browser type andrender content for each browseraccordingly. Using some of the moreadvanced browser features, however,will likely result in having to developspecific code to handle the differencesbetween browser types.

The smart client approach does notoffer this capability althoughapplications that target only Windowsoperating systems can use the .NETFramework and/or the .NET CompactFramework (for mobile applications)to deliver a smart client solution on awide range of client devices, even forexternal users.

“The correct approach will provide the user with the right user experience so that they caninteract with the application in an effective and efficient way, whilst adequately taking intoaccount the development, maintenance, and operational aspects of the application.”


If your application must supportexternal users or clients running non-Windows operating systems, then thethin client approach should bestrongly considered.

Deployment and UpdateBoth the thin and smart clientapproaches involve deploying the userinterface, application logic, and data tothe client. In both cases, these artifactsare centrally located and managed anddeployed to the client on demand. Inthe thin client approach, these artifactsare not persistent on the client andneed to be “deployed” each time theuser runs the application. In the smartclient approach, the client can persistthese artifacts to enable offline usageor to optimize the deployment andupdate process.

Since both approaches allow thecompany to centrally locate theapplication’s artifacts, they bothprovide centralized managementwith respect to user authorization,application deployment, update, etc.Companies can use thin client andsmart client to provide solutions toensure that users only run the absolutelatest version of the application, thoughthe smart client allows for additionalflexibility, such as the ability fordifferent users to run different versionsof the application (for example, pilotgroups), or for the application to be runoffline. In order to realize thesebenefits, however, the solution mayrequire additional security policychanges and/or an update managercomponent to be deployed to the client.

If your scenario requires that theapplication runs offline then you shouldstrongly consider the smart clientapproach. On the other hand, if theapplication would not benefit from

having artifacts be persisted on theclient, then the thin client approachmay make more sense. In the lattercase, applications that primarilypresent dynamic data, or whereconcurrency problems (with respectto application logic or data) would besevere, tend to be better served usinga thin client.

User ExperienceThin and smart client applications areeach suited to specific user interfacestyles. Many thin client applicationstry to provide a rich user experiencebut they tend to fall short in certainimportant aspects due to the limitationsof the browser when compared to asmart client platform. For instance, basicrich client features such as drag/dropand undo/redo are very difficult todevelop in a thin client solution. Thecomplexity associated with providingthese features can be considerable andcan reduce the cross-platform benefitsof the thin client approach.

You should also consider how the userinteracts with the application. Someapplications are very linear becausethe user typically interacts with it ina pre-defined or similar way. Otherapplications are non-linear and the usermay start one task only to suspend it,complete another task, and then go backto the original task. Managing the staterequired to provide such functionalitycan be challenging in a thin clientsolution. For example, thin clientsolutions have to be designed to handlethe situation if the user presses the backbutton in the middle of an importanttransaction. Such a situation is easierto handle in a smart client application.

Smart clients can also take advantageof local resources to provide local datasearching, sorting, visualization, and

client-side validation to improve theusability of an application and enhancethe user experience. Such features canlead to an increase in data quality anduser satisfaction and productivity.

PerformanceA significant difference between thetwo approaches is that smart clientapplications provide superiorperformance compared to their thinclient counterparts.

At a basic level, a thin client typicallyuses script (which it must interpret onthe fly) as the means to deliver andexecute client-side application logic.In contrast, a smart client solution candeliver specific compiled code to theclient. Additionally, and perhaps moreimportantly, client-side logic in a smartclient application has fewer restricts inthe way that it can interact with theuser interface, local data storage, orwith services located on the network.For these reasons, good smart clientarchitecture allows the solutiondeveloper to more easily deliver a highperformance solution.

The user’s perception of performancedepends on how they use theapplication and how they expect it tobehave. Applications that are usedinfrequently or with which the userdoes not interact very much, forexample, applications that simplyobtain and display data, will notbenefit from raw higher client-sideperformance. Applications that areheavily used, however, will appear toperform poorly if there is even a smalldelay for frequently used features. In acall center application, for example, adelay of four or five seconds to retrievecustomer order details can easily addup to significant user dissatisfaction(and cost).


Of course, for functionality that sendsor retrieves data over the network, bothapproaches will exhibit the same rawperformance. However, a well-designedsmart client solution can perform itsnetwork communication on a separatethread enabling the application toremain responsive while it sends andreceived data over the network. Suchbackground work can be accomplishedpro-actively, say in response to anincoming customer call. In addition, itis easier for smart client solutions tocache data locally, which can reduce thenumber of network calls or reduce thebandwidth required to perform thesame function. These features can havea huge impact on the user’s perceivedperformance of an application.

Smart client solutions can providemore rigorous client-side datavalidation. For example, becausea smart client solution can cache dataand logic locally, it is possible to cacheread-only reference data that theapplication can use to provide field andcross-field validation. Applications thatuse such validation can provide earlyfeedback to the user, improving theperceived performance of theapplication, reduce the number of timesdata gets transferred over the network,and ensure higher data quality. Thinclient solutions may need to rely oncomplex scripts to provide the samelevel of functionality and may not beable to locally validate data withrespect to other data that isnot displayed on the current page.

A smart client solution can also takemaximum advantage of the localprocessing, storage, and displaycapabilities to allow the user to query,sort, and visualize data on the clientwithout the need to make a networkcall. This ability is especially evident

when using an Office application suchas Excel as the smart client hostenvironment. This can result in asignificant reduction in network calls toperform the same function for a thinclient application.

Since the smart client solution userinterface is typically provided by specificcode running on the client, it can providea more responsive user interface to theuser. Rich client user interface features,such as drag and drop, undo/redo,context sensitive help, keyboardshortcuts, etc., can all help to improvethe user experience, and with it theperceived performance of the application.

If performance is an important issueyou should consider a smart clientsolution. A user’s perceivedperformance of an application is oftenmore important that the actualperformance of individual operations.A good application’s ultimate goal is toensure that the user can perform theirwork effectively and efficiently in a waythat user satisfaction is maintained.

Client-Side IntegrationOften an application requires access toclient-side resources so that they canbe integrated into the overall solution.Sometimes the client-side resourcesinclude hardware (a printer, telephone,barcode reader, etc.) or software(integration of other line-of-businessor desktop applications).

Of course, both the thin client andsmart client approaches operate withina sandbox. In the thin client case, thebrowser provides the sandbox; fora smart client, the .NET Frameworkruntime provides the sandbox.Integrating client-side resources intoa thin client application typicallyrequires using a hybrid application

architecture to host a componentwithin a page (for example an ActiveXcontrol) to extend beyond the browsersandbox. This approach is not veryflexible and often depends on the userto make security decisions aboutdownloading components to run on theclient under the user’s login account.

The .NET Framework runtime utilizesa more flexible approach and grantsmanaged code permissions based uponthe evidence that it presents and thelocal security policy. By default, codedownloaded from a web server cannotinteract with the local resources exceptin very limited and specific ways.However, your application logic cangrant the code additional permissionsto access to specific resources such asspecific directories on the disk, accessto other applications, local databases, etc.

This managed approach represents amore granular and more flexiblemechanism for controlling the securityaspects of an application, allowing thesmart client to integrate other client-side resources without introducing asecurity risk. More importantly, thenetwork administrator uses securitypolicies to make security decisionsrather than individual users, so theapplication code cannot perform actionsor access resources for which it has notbeen granted permission.

Smart client applications often usecode access security to control thecaching of data and logic on the client.Such behavior is essential to providingoffline capabilities and so these kindsof applications typically requiresecurity policy changes to grantspecific permissions. Typically, thisinvolves granting the applicationpermission to cache the code and dataon the local disk.

“It is important that the correct approach is applied from the outset to avoid unnecessarycomplexity, cost, lack of flexibility, and a poor user experience. ”


If the solution requires access to client-side resources such as local hardwareor other applications installed locally,then the smart client approachprovides a secure and flexible solution.

Offline CapabilitiesAs organizations become increasinglydependent on their IT systems and thedata and services that they provide, itbecomes more important for users to beable to work offline. Providing supportfor offline access to data and services,using the same application whether theyare on or offline, enables the user toremain productive at all times and helpsto ensure consistency and data quality.

While network connectivity is becomingmore and more ubiquitous, it isimportant to note that having a networkconnection is typically not enough toguarantee access to an application andthe data and services that it represents.Line of business applications inside thefirewall may not be accessible to userswhen they are out of the office unlessthe organization invests in a VPNinfrastructure. Even in this case, forginga connection can be time consuming andexpensive. Ad-hoc or brief access to theapplication is often not appropriate orpossible leading to lost opportunities ordata inconsistencies.

Sometimes users can plan to be offline.For example, you might have a salesworker who is out of the office for aspecific period of time or a user whoworks from home. Sometimes, however,it is difficult to plan for an offlinescenario. For example, a user in awarehouse with a Tablet PC may havea wireless connection that periodicallydrops. Another consideration is the

quality of a user’s connection. Asorganizations become increasinglydistributed over the globe, networkconnectivity can suffer from highlatency or low bandwidth problems.

In each of these cases, a smart clientsolution can provide robust access tothe application so that the effect ofconnectivity changes can be minimizedor eliminated. By intelligently cachingdata and logic on the client, andautomatically deploying updates toboth when required, the applicationcan provide the user with a seamlessexperience independent of its connectedstate. In addition, a smart client canensure that all network calls arehandled on a background thread sothat the application never has to waitfor the network to respond, againallowing the user to carry on workingregardless of the state of the network.

Realizing these goals with a thin clientsolution is very difficult to achieve. Somesolutions try to solve this problem byproviding the web application, or asubset of it, on the client using a localweb server. Such solutions are difficult tomaintain and require a complexinfrastructure to ensure that updates tothe application and its data are handledappropriately. Such solutions reduce thecentralized management benefits thatare often cited as the main reason for theadoption of a thin client solution, andthey impose all of the other drawbacksinherent in a thin client solution.

ConclusionChoosing the right presentation layerarchitecture can be critical to theoverall success of an application. Theright architecture will provide the right

balance between the user experience,ease of development and testing, andthe operational requirements of theapplication. Users are increasinglydemanding that their part of thisequation is taken into account.

The thin and smart client approachesare both well-suited to particular stylesof applications. Recent advances intechnology have redressed some of theimbalance between these approaches sothat they need not be inappropriatelyapplied to situations for which they arenot suited. It is important that thecorrect approach is applied from theoutset to avoid unnecessary complexity,cost, lack of flexibility, and a pooruser experience.

Blanket corporate policies that favor oneapproach over the other are prone toincur these problems. An organizationmust carefully consider the overallneeds of the application and comparethese to the capabilities of eachapproach. The factors that can influencethis decision are many and varied andthis article has only touched on some ofthe more common ones. Invariably, thedecision will come down to acompromise between the various factors.Understanding these factors, and theirrelative priorities, can help ensure thatyour organization chooses the rightpresentation layer architecture.

Resources– Smart Client Architecture and

Design Guide – Microsoft Patternsand Practices.

– Overview of Office 2003 DeveloperTechnologies – MSDN.

– Overview of Office 2003 DeveloperTools and Programs – MSDN.

David HillMicrosoft [email protected]

David Hill is a Solution Architect onthe Microsoft Architecture Strategyteam. For the last two years, David hasbeen helping customers and partnersarchitect and build successful smart

client solutions on the .NET platform.He was a key contributor to thePatterns and Practices Smart ClientArchitecture and Design Guide, andthe Offline Application Block.

JOURNAL4 | Information Bridge Framework 14

IntroductionEnterprises today are moving towardsSOA as a way to expose theirapplications and data for consumption.By embracing SOA, enterprise assetslike line of business applications orback-end systems can be used by avariety of solutions/applications builton top of the services exposed by thoseassets. In this world you can look atan enterprise as a set of services thatexpose sets of data or functionalityand encapsulate the business logicbehind them.

Building solutions on top of theseservices is fairly easy today usingexisting development tools. Differentvendors provide tools for both exposingand developing on top of those servicesby using standards like SOAP or WSDL.

Once enterprises start developing a fewof these solutions problems start toarise. Here are some of the mostcommon problems:

a) Solutions are one off. They only talkto one or a set of predefined servicesand the solution itself is hard toreuse. Changes on the servicesrequire a rebuild/redeployment ofthe solution.

b) Knowledge about what the serviceexposes is in people’s heads ratherthan in the definition of the serviceitself. Current standards only coverthe how you get to those services.

c) It is hard to bring different servicestogether. There are no predefinedaggregation mechanisms and thereis no definition on how one servicerelates to another (services do notknow about each other).

d) Solutions UI is hard to do andusually poor (unless hugeinvestments are made) by mostcommon users standards. It is hard

to emulate current off the shelfapplications UIs in a one off solution.

e) Most users are fairly familiar withapplications like Office Suite (Word,Excel, Outlook, etc.) but they need tobe trained if a new application/solution is rolled out increasing thecosts of such deployment.

Because all of the above a bettermechanism for building solutions ontop of existing services is required.

The Metadata approachToday web services expose quite a bit ofinformation about how the service canbe consumed but offer very little helpin understanding what type ofinformation or functionality is offered.Web services usually expose WSDL sotools can easily discover what methodsand parameters the web serviceexposes but offer little clues about whatbusiness entities are defined behindthose methods or even if they affect theback-end systems at all (no way to sayif a method will update the back-endsystem for example). It seems thatWSDL is not enough for representingwhat today’s services expose.

We propose a new set of Metadata thatis associated with a service andexplains the kind of things that a userof the service (a solution developer) willrequire to know.

In this new Metadata we will exposeconcepts like:

a) Entities – Abstract business or userdefinitions that will encapsulate a setof data or functionality. For example,we can have a Customer entity.

b) Views – A schema associated with anentity that describes a subset of dataabout it. For example, for theCustomer entity we may

have several views like CustomerContact Information or CustomerFinancial Information. Each viewcomplies to a particular schema andis a representation of the entity for agiven context.

c) Relationships – Entities/Views canbe related to others and theserelationships should be describedin this Metadata. For example,Customer entity may be related toan Orders entity. Relationships allowfor navigations between the entitiesby just executing the Metadatadescription. A relationship then willdescribe how to get from one entityinto another.

d) References – A reference is acommon way to point to a set ofinformation. It is a schema andrepresents the minimum set ofinformation needed to retrieve apiece of data, e.g. Customer Id forretrieving a Customer. There can bemultiple ways you can retrieve a pieceof information, e.g. a customer couldbe retrieved by name, Id, SSN, etc.

e) Operations – These are the methodsthat are available for a givenentity/view to operate on. You couldthink of GetCustomer orUpdateCustomer or ReleaseOrder asexamples of such operations.

Describing Metadata for existingServices only solves half of theequation. The other half (the solutionsdeveloped on top of these services) alsorequires a Metadata description. Wethink that you can build most solutionsby thinking in terms of Actions that areexecutable by the end user. TheseActions are constructed on top of theServices Entities/Views and provideactionability on top of them. A Customeraction will certainly have an Action todisplay its data and maybe anotheraction to update it. The action

“Today web services expose quite a bit of information about how the service can be consumedbut offer very little help in understanding what type of information or functionality is offered.”

Information Bridge Framework: Bringing SOA to the desktop in Office applicationsBy Ricard Roma i Dalfó, Microsoft Corporation


description should link the dataretrieved from the service into theUI or solution functionality thatwill use it.

Information Bridge FrameworkInformation Bridge Framework (IBF)is the Microsoft response to the abovechallenges and Metadata approach.IBF allows connecting LOBs and back-end systems with Office applicationsand creating solutions on top of webservices via a Metadata approach.

IBF accomplishes several things:

a) Create Metadata description forservices

b) Create Metadata infrastructure forbuilding solutions/applications ontop of services

c) High level of reusability acrosssolutions

d) Easy maintenance and deploymentof solutions

e) High level of integration with Officeapplications

f) Very low learning curve for existingOffice users

Information Bridge ArchitectureIBF architecture (as seen in Figure 1)includes the following components:

a) An IBF compliant web service thatencapsulates the LOB or back-endsystem. We discuss the complianceissue in the next section (Designingand developing an IBF solution).

b) A Metadata repository (MetadataService) that includes both theService and the Solution Metadata.The repository is exposed itself asa web service that provides access tothe Metadata. There is one centralrepository where all Services andSolutions are described. Clients willdownload subsets of this Metadataon as needed basis for executionbased on their permissions.

c) IBF Client Engine. This last piecehas two distinct components:a. The Engine which downloads the

Metadata from the MetadataService when needed and keeps alocal cache of it. It also understandsMetadata and executes it based onthe current context. It performs allthe non UI related operations likeSOAP calls, transformations, etc.This component is UI agnostic.

b. The UI Engine which is the partthat understands about theapplication where it’s being hosted(Word, Excel, etc.) and will renderthe UI and provide servicesspecific to the host application.It creates an abstraction layer ontop of the hosting application soSolutions built with IBF don’tneed to know about differencesbetween hosting apps.

d) Metadata Designer is a VisualStudio based tool that allows forediting/importing of Metadata tothe Metadata Service.

“We propose a new set of Metadata that is associated with a service and explains the kind ofthings that a user of the service (a solution developer) will require to know.”

MetadataDesigner

Metadata

MetadataService

InformationBridge Engine

OfficeApplication

Application Integrationand Rendering

AttachedSchema

Smart Tag

Task Pane Manager

Host Integration

Rendering

Client

Server

Metadata

CRM

CustomerInformation

Invoices

Orders

InformationBridge

CompliantWeb Service

InformationBridge


InformationBridge


Web Services

SAP LegacyApplication

AggregationService

SOAP

Figure 1: IBF Architecture


Designing and developing anInformation Bridge solutionWhen designing an IBF (InformationBridge Framework) solution you mustseparate it into three distinct blocks(see Figure 2). On one end you need todescribe an IBF compliant web servicethat encapsulates the functionality ofyour back-end application that youwant to offer your end users. On theother end you need to design the UIand experience that you want to offerto the users of your solution. The finalstep is to link both your Service andthe UI solution you have built by usingIBF metadata. By separating thesethree stages you can allocate differentresources to each one of them and thencan operate in an independent mannerand only agreeing on the interfaces (orcommon schemas) that they will share.

Creating an IBF compliant ServiceIBF requires Services that will providethe data and interaction with the dataneeded for your solution. IBF currentlysupports two kinds of Services: WebServices and CLR Components. WebServices are the most common way toexpose back-end data, and most of theIBF examples use them for the Servicedescription. If you require offlining ofdata or caching (for performancereasons) a CLR implementation isalso possible.

When designing a Service for IBF youshould keep in mind that you arebuilding a Service for userconsumption, so you want to exposedata and methods that are meaningfulto the user.

There are also a few concepts youneed to be aware of when buildingthese Services:

– Entities – You can think of an entityas a business object that has aparticular meaning to the user andthat the user will be able to act upon.An example of an entity can beCustomer, Order or Opportunity. All

of those have some data associatedwith them and are actionable fromthe user point of view. For example,the customer entity might have dataassociated with a particular customer(name, address, location, etc.) as well

“IBF allows connecting LOBs and back-end systems with Office applications and creatingsolutions on top of web services via a Metadata approach.”

Web ServicesDevelopment

VisualStudio .NET

InformationBridge-compliant

Web Service


Web Service


Web Service

LOBApp.

LOBApp.

LOBApp.

.NET WebServicesDeveloper

Developer /BusinessAnalyst

UI Controls and Bindings

.NETDeveloper

VisualStudio .NET

XSL orWindowsForums

Active Directory

Solution Metadata

AuthenticationManager

VisualStudio .NET

MetadataDesignerPlug-in

MicrosoftOffice System

Metadata Library

Context &ActionService

MicrosoftOfficeSystem

Developer

XMLAuthoring

Tools

SmartTags SDK

MicrosoftOffice System

Schema-AttachedDocument

EmbeddedSmart Tags

Document Solutions

Metadata Configuration

Figure 2: The three distinct blocks of an IBF solution


as methods that allow the userto act on the entity, such asUpdateCustomerInformation orSendEmailToCustomer. It might alsobe a starting point to other entitiesvia relationships likeCustomerOrders orCustomerOpportunities.

– Views – IBF partitions entities indifferent views. A view is a subset ofinformation related to the entity. Fora customer you might have aCustomer Contact Information viewand a Customer Financial view.

– References – A reference in IBF worldis a piece of information thatuniquely identifies an instance of anentity/view. For the previous examplea reference could be Customer Id orCustomer Name if that allows you touniquely identify the customer.

– Relationships – Some of theentity/views will have relationshipsbetween them and the Metadata webuild should describe those. Anexample would be Customer andOrders since you can relate aCustomer with its orders andan Order with its Customer.

Based on the previous concepts whenyou build a Service you will identifythree different kinds of methods:

– Get – A get method is one that allowsyou to retrieve the data for anentity/view by passing a Reference.An example would be a method calledGetCustomerContactInformationthat would accept a Customer IdReference parameter.

– Put – This is a method that allowsyou to modify the content of anentity/view by updating the back-endsystem. It accepts two inputs, theReference to the entity/view toupdate and the data to be updated.

– Act – This kind of method allows fordoing things that are not related togetting/updating an entity/view oracross multiple entities.

When you understand these concepts,you can build a Service around them.The service will expose a collection ofmethods of type Get/Put/Act and bydoing so will define the schemas for thereferences and the views (data returnby Get operations).

For the Service to be complete it has toexpose IBF Metadata that describesthe previously explained concepts.IBF provides tools that automaticallygenerate Metadata from a web serviceand you can then increment theMetadata by annotating the methodsexposed in there around Entities/Viewsand map them to the right References.

Creating UI componentsIBF allows your documents to containlive links to back-end data. The waythese documents contain informationabout what back-end data to obtainis via SmartTags or by having anattached schema to the document.The SmartTag or element node in theschema will store information aboutwhat back-end piece of informationis pointing to. As discussed in theprevious topic on how to create anIBF Service, these are References.SmartTags, for example, will containa Reference to the back-end piece ofinformation. Your solution will have todefine how it wants these SmartTagsto get into the document and IBFprovides/recommends several ways fordoing so. You can automatically generatea document with the SmartTagsembedded (this would be useful if theemails/documents are dynamicallygenerated by some process); you can

use a SmartTag recognizer to detectpieces of text based on a regularexpression or by doing a look up anddynamically insert a SmartTag inthem; and you can also use the built inSearch capability in IBF for the user tofind the instance of information theyare interested in and allowing themto paste it into the document.

The remaining UI pieces are what willget displayed to the user. IBF providesa Window Pane approach that hostsregions that are fully definable by thesolution provider. IBF supports .NETCLR controls and HTML regions (andmenus for those regions). Creating apiece of UI is just a matter of creating acontrol and implementing one interfacethat will get the data into the control.The control itself does not need to knowhow or where the data is coming from.The control only needs to know thetype of data that will be provided. IBFwill dynamically instantiate the controlat run time and will pass the right datato the control. This allows for aseparation of displaying of data fromhow you get to that data. Following theprevious example you could create acontrol that knows how to renderCustomer information (it knows aboutthe schema of a Customer and that itcontains its name, address and so on).

Creating Solution MetadataThe final step for creating an IBFsolution is to create the Metadata thatwill link the Service description withthe UI elements that have been definedfor it. IBF provides a few concepts thatallow for easy creation of theseMetadata based solutions:

– Actions – These are the executableunits from a user point of view andcan contain both Service and UI


methods/operations. In the previousexample you would have aDisplayInformation action that woulduse the Service entity/view onCustomerContactInformation andwould link it to the user control wecreated for displaying customerinformation.

– Transformations – Because data fromthe Service and the data required bythe UI elements might not be thesame, IBF allows you to transformthe data. XSL transformations,regular expressions or calling CLRcomponents are all supported ways totransform data.

– Relationships – Your solution mayhave relationships beyond thoseprovided by the Service. It may alsoknow about relationships acrossServices. As an example I may be ableto relate a Customer in one of mylegacy apps with my Customer in myCRM system.

Deployment and SecurityYou can think of IBF as a centralrepository of Metadata, Servicedescription and UI elements that will

be deployed dynamically as solutionsare used by the IBF client components.No code/Metadata other than the IBFclient needs to be installed on clientmachines. The IBF client componentconnects to the appropriate MetadataService to obtain all Metadata and UIelements needed for a given context.After it has the Metadata descriptionand UI elements, it will execute themalong with the Service method calls,and it will construct the UI and userexperience as needed.

Because IBF uses CLR components forUI rendering it builds on top of .NETsecurity, all components aredynamically downloaded and cachedlocally, and are executed in asandboxed environment so they cannotharm the client machine. If you needyour controls to have a higher level ofcontrol, you can sign those controls andincrease their privileges by usingstandard .NET Security Policies.

This provides for a robust anddeployment free environment for yourenterprise solutions.

ConclusionIBF, by separating the Service layerfrom the UI layer and linking them viaMetadata, allows for a high level ofabstraction and reusability of both yourServices and your UI components. Thisprovides a very powerful platform forspecifying the back-end assets in anenterprise and creating solutionsaround them that can be linked orcombined without coding. ThisMetadata approach adds a lot offlexibility and allows for furtherrefinement of solutions aroundcustomer scenarios in a Metadatadriven approach. IBF provides powerfulUI constructs to help build a completeUI experience and integration withOffice applications. It also provides fora secure and deployment freeenvironment of new solutions bybuilding on top of .NET technologies.

ResourcesFor more information about IBF go tothe site:http://msdn.microsoft.com/ibframework

Ricard Roma i DalfóMicrosoft [email protected]

Ricard Roma i Dalfó is the developmentlead for the Information BridgeFramework project. He is working onnext versions of IBF and solvingconnectivity problems with Line ofBusiness applications. He had beenpreviously in the Office team andhelped release Office 2000, XP and2003 in various development roles. Heholds a M.S. in Computer Science fromPolytechnic University of Catalunya.

JOURNAL4 | Benchmarking a Transaction Engine Design 19

Performance is probably one of theleast understood and misquotedmetrics in the field of computing today.It is common practise amongsttechnologists and application vendorsto regard performance as an issue thatcan safely be left to a tuning exerciseperformed at the end of a project orsystem implementation. In contrast,performance, as a metric for systemevaluation, is considered by most usersto be one of the most important andindeed critical factors for assessing thesuitability of a system for a particularpurpose. This paper describes abenchmarking exercise undertakenduring October 2002 at Microsoft’s ISVLaboratories (ISV Labs) in Redmond,Washington as part of joint project

between the Capital Markets Company(Capco) and Microsoft.

The project was initiated when Capcowas commissioned by the SingaporeExchange Limited (SGX) to provide abusiness assessment and to develop atechnical architecture for the exchange’scentralised transaction processingutility. The utility was to providematching services for post-trade, pre-settlement interactions of participantsin the equities and fixed incometrading areas of the Singapore market.The design of the main processingengine was done following a processknown as Software PerformanceEngineering (SPE) [SMITH90,SMWIL02] in which the entire design

and validation exercise is modelledfrom a performance perspective ratherthan from a traditional object-orienteddesign perspective.

Two subsystems were created byCapco, known respectively as STPBridge (a communicationsinfrastructure and exchange gateway)and STE (a scalable transactionengine). Both were used in thebenchmarking exercise.

The Processing ModelThe architecture of STE was based onloosely coupled, stateless, messageprocessing components arranged in aqueuing network for high scalability,high performance, and extremely high

Benchmarking a Transaction Engine DesignBy Richard Drayton, FiS and Arvindra Sehmi, Microsoft EMEA

“Unrealisable performance is a common characteristic of benchmarks”

Front EndPAM

Component

MessageRouting

Component

Workflow Processor(Exception Management)

Database Management

NotificationProcessor

MessageProcessor

MessageProcessor

MessageProcessor

MessageProcessor

MessageProcessor

MessageProcessor

MessageProcessor

MessageProcessor

Participant Connection(VPN in-bound messages)

Connection to ExternalDatabase Services

Connection to Participants(VPN out-bound messages)and Settlement Services

Connection to ExternalException Managementand Workflow Services

Figure 1: Processing Engine Model


transaction throughputs. For SGX, aworst case transaction processing loadof approximately 600 messages persecond was estimated by analysis oftheir trading history over the previousfew years; which turned out to be thehighest processing level requiredduring the previous Asian financialcrisis in 1998 when exchange dealingswere considered abnormally high. Thisvalue was used as the baseline/targetprocessing load. A stretch load target of2000 messages per second was set toensure the architecture, if successful,would have adequate headroom to copewith future expected trading volumegrowth and volatility.

The decision to use loosely coupledcomponents communicating throughmessage queues rather than throughmore traditional component interfaces(API’s) requires information to bepassed from component to componenteither directly through the messagesthemselves (persistent message flow) orby enriching the messages as they passthrough various processing steps(temporal message flow). Theprocessing components themselves arelargely independent of each other andstateless. This results in benefits suchas lower software development riskand realisation costs for individualcomponents together with higherscalability and flexibilitycharacteristics of the processing engineas a whole when compared totraditional monolithic applicationdevelopment approaches. Mostconventional designs support onlyone of the two possible scalabilitydimensions: scale-up – increasedprocessing power through increasedprocessor resources (memory, CPU,etc.), or scale-out – increasedprocessing power through increasednumber of processing nodes. This

architecture supports both typesof scaling.

The overall architecture of the STEprocessing engine is shown in Figure 1.

This consists of a number of STEcomponents which have responsibilityfor processing sub areas of theunderlying trading business activities.The business process support providedby the engine is realised by breakingdown the entire trade lifecycle into aset of related atomic messages. Eachmarket participant generates andreceives various subsets of these atomicmessages during execution of the trade.

An implication of the architecture isthat the business process itself must becapable of being represented by fullyasynchronous commutative set ofoperations, that is, it must be able toprocess messages in any order. Thisremoves the necessity to synchronisemessage processing and businessoperations throughout the engine,a task which would result in anincredibly slow and complexapplication. Note that synchronizationis different to correlation of messageswhich is applied in normal processing.Several other “autonomous computing”requirements are catered for in theprocessing model. Amongst these arethe notions of distrusting systems,idempotent operations, statemanagement, message correlation,context management, tentativeoperations, message cancellation, andtransaction compensation1.

Business-level message flow throughthe processing engine is based, in part,on the execution model proposed by theGlobal Straight Through ProcessingAssociation (GSTPA) which proposed asimilar centralised utility model for

cross-border trade settlement. The SGXprocessing model was likely to requireoperational links to other centralisedutilities like the GSTPA, so themessage sets used were based, in part,on those used by the GSTPA to helpwith inter-operability in the future2.

The business process for the Singaporemarket reduced to a set of four principalmessage types, namely, Notice ofExecution (NOE), Trade Allocation(ALC), Net Proceeds Allocation (NPR)and Settlement Instructions (SET).The process surrounding these messagetypes involved a number of interactionsleading to a set of some 35 messagevariants that formed the entire businessoperating model. As recent events in thefinancial services industry illustrated,the range of processing volumes thatcould be experienced by the utilitywould be comparatively large. For thisreason a heavy focus was placed on theability of the architecture to supporthigh scalability requirements.

Modelling PerformanceA valid benchmarking exercise allowsother institutions to repeat thebenchmark for themselves and to beable to achieve similar results. It wasalso important to support the

1 Pat Helland’s “Autonomous computing:Fiefdoms and Emissaries” [PHEL02]webcast gives more details on theautonomous computing model.

2 The GSTPA is now defunct but thisunfortunate event has no impact onthe substance of this study.Additionally, the original proposalchampioned by the GSTPA could not beused directly in the Singapore market,so specific processing schemes werecreated for the SGX processing engine.


arguments for the architecture modelthrough sound mathematicaltechniques which would enable anassessment of the impact onperformance of various implementationrealisation techniques a-priorito deciding on any specific realisationtechnologies. Figure 2 shows thequeuing network model used for theperformance analysis in thebenchmark. Note, this works hand-in-hand with a client-side ParticipantAccess Module (PAM).

By conducting the benchmark in thisway it was felt that the results of theexercise, good or bad, would be credibleand valid and that the results wouldserve as the basis for a case study intothe application of queuing networkmodels for the processing of Post Trade,Pre-Settlement information as part ofan overall high performance straightthrough processing (STP) initiative inthe financial services industry.

Measuring PerformanceOne of the most frustrating aspects ofperformance engineering occurs when aperformance unit which permits validcomparisons to be made betweensimilar systems is not identified.

Performance is a subjective quantitywhose absolute value is oftendetermined solely by the user of asystem and not by any systematicalgorithm or process. This nebulousaspect of performance gives rise tovariations in the perceived value ofperformance characteristics for aspecific system even withoutaccompanying physical changes to theunderlying technology whatsoever.

The results of any benchmark aresubject to wide interpretation and thiscan jeopardise any comparativeanalysis of software systems.Unrealisable performance is a commoncharacteristic of benchmarks that haslittle relevance to a user’s perspectiveof performance. The existence ofunrealisable performance can be shownin numerous examples of publishedbenchmarks where results oftenindicate high levels of performance thatare simply unobtainable in the realworld. For example, an ADSLconnection offers a theoreticaldownload speed of 512 K, but in realityis limited by contention with otherusers on the same exchange. So, a morerealistic way to do performancecomparisons between systems is to

establish metrics for an operationalsituation which can be readilyreproduced outside the testenvironment and is meaningful fromthe system user’s perspective.

To avoid these sorts of frustrationswith the benchmarking process, apractical technique was required whichwould provide a supportable andreproducible figure for the performanceof the realised system. The techniquechosen was based on Buzen andDenning’s work [BDEN78] allowingcredible performance figures to beobtained based upon soundmathematical principles.

The Operational BenchmarkThe purpose of the benchmark was tovalidate the architectural design onMicrosoft platform technologies andto establish the credibility of thebenchmark in an environment highlycorrelated with the operationalrequirements that would be presentedto potential system users. This goalcould only be achieved if the benchmarktests were firstly performed usingvolumes and process loading levelsconsistent with those that would beexperienced in the final realised utility,and secondly performed in a mannerthat could be reproduced on-site aswell as in the laboratory.

For the laboratory exercise a set oftest management components werecreated which allowed an end-to-endanalysis to be performed. Theseconsisted of a scalable message driverand corresponding scalable messagesink with performance measurementsbeing taken from these two points. Theperformance figures were calculatedon the time taken to fully process aknown number of messages forming aknown number of financial trades.

Figure 2: Queuing Network Model

In-bound Message Queueand STE side PAM and

routing server

Matching processqueues and

matching servers

Confirmation andout-bound message queuewith processing server

Message handlerprocessing queuesand message servers


The Buzen & Denning MethodBuzen and Denning [BDEN78]described a practical technique forevaluating the performance of queuingnetwork software systems. In essence,they argued that a simple and easilyobtainable measurement of queuelength over time could be used toestablish all the performance metricswhich they had defined for queuingnetworks. A realisation of the Buzenand Denning technique produces anexecution graph for the network modelsimilar to that shown in Figure 3.

A simple queuing model can be viewedas a queue together with an associatedserver, as shown in Figure 3 wheremessages arrive (arrivals) on an in-bound queue on the left hand side,are processed by the server, and exit(completions) on the right hand side.The Buzen and Denning techniqueinvolves sampling the queue lengthof the server at regular intervals andrecording the number of messageremaining in the queue andcontinuing the observations over aperiod of time.

The resulting graph of observations isknown the execution graph and clearlyshows the arrivals (i.e., increases in theheight of the graph) and completions(i.e., decreases in the height of thegraph) of messages. Using thesefigures, Buzen and Denning derived aset of formulae from an original resultestablished by J.D.C. Little (known asLittle’s Law [LITTL61]) which enabledall of the required performance metricsto be determined. A summary of theformulae is shown in Table 1.

Measuring PerformanceTo complete the performancemeasurement a standard set ofbusiness messages was neededrepresenting the mean businessprocess life cycle for the utility. Astandard “trade” was created using anNOE (notice of execution) message, twoALC (trade allocation) messages, twoNPR (net proceeds) messages and twoSET (settlement instruction) messages.A standard trade therefore consisted of7 base messages which together with

“One of the most frustrating aspects of performance engineering occurs when a performanceunit which permits valid comparisons to be made between similar systems is not identified”

CompletionsArrivals

Queuing Component

Number of Jobs(Tasks)

Time

Figure 3: Simple Execution Graph

Metric Symbol Definition

Length of Observation T Total number of time units overwhich the observation has beenmade.

Arrivals N Total number of Arrivals over thelength of observation.

Completions C Total number of Completions overthe length of the observation.

Busy Time B The number of time units wherethe number of messages in thesystem exceeds zero.

Utilisation UThe calculated value:

Throughput XThe calculated value:

Mean Service Time SThe calculated value:

Execution Distribution AThe calculated value:

Mean Queue Length LThe calculated value:

Residence Time RTThe calculated value:

Queuing Time Q The calculated value: RT – S

Table 1: Buzen and Denning Formulae

U = BT

X = CT

S = BC

A =

L = AT

RT = AC

∑=

T

ttMessages

0

)(


the requisite acknowledgement andconfirmation messages betweenparticipants comprised the entireinformation set for the tests. Thecompletion of a trade’s processing bythe system generated one furthermessage which was designated the CLS(clearing and settlement) message.

In the GSTPA specification acombination of up to seventeendifferent elements in the messageswere required to match before theentire trade could be considered valid.The requirements differed dependingon the particular message types beingmatched but at some point allseventeen matching criteria had to beapplied for the trade to be processed.The full set of matching criteria isshown in Table 2 for reference. Thematching process entailed scanningthe message database matching tablefor all elements of the trade specifiedin each message as each message wasreceived. When the scan yielded therequired seven matching atomicmessages a valid CLS message wassent to the out-bound queue of thetest system.

In addition to the matching process,the business process for the Singaporemarket also requires validation of thecontents of the messages for items suchas currency and country designations,security reference codes, payment andsettlement dates and participantidentification information. A collectionof static data was created to form thebase reference data for the utility andis shown in Table 3 for reference.

To establish the benchmark result testmessage sets were generated using thestatic data. A message buildercomponent was used to randomly selectappropriate static data and then

Criteria Definition

Trade Reference The trade reference identification information mustmatch for all elements of the trade.

Buyer Reference The Banking Identification Code (BIC) for the buyerinvolved in the trade for all elements must match.

Seller Reference The Banking Identification Code (BIC) for the sellerinvolved in the trade for all elements must match.

Security Reference The security reference code (ISIN, QSIP, SEDOL etc.)must match for all messages in the trade wheresecurity designation is included.

Security Type The security reference type (ISIN) must match.

Trade Date The specified date of the trade must match on allmessages containing a trade date. Additionally, thetrade date must be a valid trading day for thelocations specified in the messages.

Settlement Date The specified settlement data of the trade must matchon all messages containing a settlement date.Additionally, the settlement date must be a validtrading day for the locations specified in the messages.

Trade Currency The trade currency designation of the trade must matchon all messages where the trade currency is specified.

Settlement Currency The settlement currency designation of the trade must match on all messages where the trade currencyis specified.

Trade Quantity The quantity of the security or fixed income trademust match in all relevant messages. This criteriarequired also that the sum of the allocations for atrade matched the execution quantity.

Trade Type The type of the business trade being performed mustmatch on all relevant messages.

Trade Direction The buy/sell status of the trade must match on allrelevant messages.

Unit Price The price per unit traded must match on all relevant messages.

Trade Fees Fees and commissions contained within the trademust match on all relevant messages.

Country Reference Base country designations must match on all relevant messages.

Gross Amount The calculated value of quantity * price must matchthe value contained in the relevant messages.

Rate The fixed income interest rate must match on allrelevant messages.

Table 2: Message Matching Criteria


combine it into the standard trade ofseven independent messages. Two coremessage sets were created; the firstcontained blocks of 250,000 trades(1,750,000 messages) and the secondcontained 1,000,000 trades (7,000,000messages). A normal business cyclewould require all matched orunmatched trades existing in thesystem after 3 days to be cleared fromthe system database; however, it wasdecided to keep all processedtradeinformation in to investigate thedegradation in database performanceas volumes increased. The message setswere stored in standard text files inXML format ready for transmissioninto the processing engine.

To monitor the benchmark and obtainthe required performance metrics aset of management components wasconstructed. These were a messagedriver component, a message sinkcomponent, and an additionalcomponent to monitor the queuelengths of both the in-bound and out-bound message queues as well as theinternal queue lengths at regularintervals. The message drivercomponent processes the test messageset files sequentially and applies adigital signature prior to sendingthem to the in-bound queue of theprocessing engine. Each messagedriver running on its own withoutcontention with other processes in thesystem can achieve in the region of8000 messages per second. This figureis very close to the benchmark resultsprovided by Microsoft for the MSMQproduct. A message sink componentreads the information destined for theCLS queue and monitors the timetaken to process a given number ofmessages through the system. Themonitoring components are shownin Figure 4.

“The results of any benchmark are subject to wide interpretation and this can jeopardise any comparative analysis of software systems”

Machine Operating Sys. Support Software

PARC4603 PARC4602 PARC4504 MSMQ 3.0; Capco STE 3.0 PARC4503 Windows Server 2003 (C/C++) with Microsoft PARC4502 build 3680.main LibrariesPARC4501 PARC4202 PARC4201

PARC4208 PARC4107 Windows Server 2003 MSMQ 3.0; MSSQL 2000 sp2PARC4507 build 3680.mainPARC4508

Table 5: Software Operating Environment

Data Item Records Comment

Participant 38 Reference data designating the validparticipant codes available to the system.

Security 20,000 Reference data designating the valid securitycodes available to the system. For theSingapore market only ISIN codes are used.

Currency 172 The ISO 4217 standard codes for currencies.

Country 240 The ISO 3166 standard codes for countries.

Profile 38 The profile data for active participants. Theprofile static data contained the X.509 publickey data for the signed messages to bereturned to participants and the private keydata for the participant designed as theutility. All messages were digitally signedand check for each message transferred.

Routing Data 44 Internal and external queue routinginformation. Each participant was designatedan out-bound reference queue to whichrelevant messages were routed. Internalqueue references for the grid computing partof the processing utility were also containedin this data block.

Queue Data 44 Additional queue information relating to thehost machine IP address and TCP/IP postnumber for data connections (used on theMQSeries version of the software only).

Table 3: Static Data Parameters


Hardware EnvironmentThe nature of the architectureproposed for the processing utilitylends itself to higher performancewhere multiple machines areconcerned. To remove resourcecontention issues within the overallarchitecture, it is better to havemultiple single CPU machines than asingle multiple CPU machine. Thehardware used was 4 instances of theset of machines shown in Table 4.

To scale-out processing during thebenchmark we simply deployedmultiple copies of this basic hardwareenvironment.

In addition to the hardware listed inTable 4, a further 8 single CPUmachines hosted the message drivers,monitor and message sink componentsused to record the benchmark results.

The database was put on Compaq MSA1000 RAID (Redundant Array ofInexpensive Disks) storage deviceconfigured for level 0 support(maximum throughput, minimumrecoverability). Since the businessscenario for the trading exchangeutility required local (i.e. client-side)database support for each participantconnected to the utility, disasterrecovery sites, and on-line dual systemredundancy, the loss of RAIDrecoverability was considered a smallprice to pay compared to the gain in

performance provided by a RAID level0 configuration.

We initially thought a single databasestorage device could manage the fulltrade transaction volumes. But it soonbecame apparent during execution ofthe benchmark tests that internal diskqueuing reached excessive levels. We’llsee later how this problem was avoided.

Software EnvironmentThe software operating environmentused for the various hardwareplatforms and support softwarecomponents is listed in Table 5.Although the RTM release ofWindows™ Server 2003 had notoccurred at the time of the benchmark,the release candidate (RC1) was

available and was considered to besufficiently stable and complete tobe a valid part of the benchmark. Allapplicable service packs were appliedto the operating environment includingany third party drivers used forperipheral devices.

Scaling the ArchitectureFor the benchmark tests the scale-upmodel involved executing the softwarecomponents on 4 CPU machines (anincrease of 2 over the basic processingnode) and also executing the code onthe 64-bit Itanium processors. Althoughthe Itanium machines contained only2 processors, the available bus andI/O bandwidth was considerablyhigher than the standard 32-bitplatforms and the results obtained

Machine Manufacturer CPU RAM Disk Network Ext. Storage

PARCXXXX Dell 1550 2 2GB 36GB Gigabit

PARCXXXX Dell 1550 2 2GB 36GB Gigabit

PARCXXXX IBM X370 8 8GB 2 x 73GB Gigabit Compaq MSA 1000

Table 4: Basic Hardware Environment (x4)

ProcessingEngine

QueueMonitor

MessageSink

MessageDriver

Figure 4: Benchmark Monitoring Arrangement


were certainly encouraging. We did nothave time to investigate the scale-upmodel thoroughly.

The scale-out model employed was toincrease the number of processingnodes from 1 through 8 processingengines and from 1 through 4 databaseprocessors. We spent the bulk of ourtime investigating the scale-out model.

Software ProcessingThe STE engine components werewritten in C/C++. They process XMLmessages using an exception basedheuristic in which messages areassumed to be correct for both contentand format until such time as an erroris detected (missing or incorrect data).On exception, the message in questionis directed to a nominated exceptionqueue for further processing. Thesubsequent exception resolutionprocesses were not included in thescope of the benchmark tests.

Validation of the content of eachmessage was carried out against staticdata which had been pre-loaded intomemory-based arrays. The validationprocess involved organising therequired static data into sorted arrays,using C/C++ qsort function, andvalidating the existence of XMLelements by locating an entry withinthe array, using C/C++ bsearchfunction. Data elements in the XMLmessages are accessed using thestandard C/C++ strstr function.

Benchmark ResultsThe benchmark tests produced someinteresting results. Some validated theapplication design, while others led toarchitectural changes to addressidentified performance issues. Thebasic lesson we learned was that yougenerally need to “tune for percentagesand re-architect for orders of magnitudeimprovements in performance.”

Message Queue ProcessingThe benchmark test was conductedat two basic levels, the first having250,000 trades (1.75 million messages)and the second having 1,000,000 trades(7 million messages). All processedinformation was left in the databaseas an added processing load on thesystem. The arrangement shown inFigure 4 was used as the basis for thebenchmark evaluation.

A significant latency time was notedwhen the message driver componentswere first started due mainly tocontention issues within the individualqueue managers processing themessage streams. The insert processrate was so high that the individualqueue manager processes hadinsufficient time during the start ofeach run to transfer messages acrossthe network to the remote processingmachines. This gave the effect of theinitial transfers taking several secondsbefore the performance monitors pickedup any activity on the processing nodesof the system. Figure 5 explains thisqueue manager contention process.In practise it is extremely unlikely thatseveral million messages would arriveas a single input into the utility andtherefore the latency effect could beignored for the purposes of thebenchmark evaluation. This effect wouldbe common to all asynchronous messagetransfer systems, not just MSMQ whichwas used for the benchmark.

A similar effect could be observed inthe processing components when largenumbers of messages would arrive intheir in-bound queue (remote hostmessage buffer) as a single input. Thismeant that the processing componentswould be pushed to approximately100% loading during the initial start ofa run and would stabilise to a lowerFigure 5: Single Queue Manager Contention Process

Memory Buffer(4MB increments, MSMQ 2.0)

In-bound queue process tofill the local message buffer

Out-bound queue process totransfer the local messagequeue to the remote machine

Contention between the in-bound and out-bound threadscauses delayed throughput

Buffer emptythread process

Buffer fillthread process

Memory based, non-persistent queuebuffer. Buffer is allocated in

steps of 4MB (MSMQ 2.0) depending on the demands of the input

message size and rate


figure once flow through the machineshad evened out. A typical performancemonitor output for a single processingnode is shown in Figure 6. The effectof the initial message burst can beclearly seen in the response curve forthe combined dual processor machine.

Furthermore, the effect of processingcompletion of the in-bound messagesflowing into the queue managermessage buffer can be seen in the laterpart of the response graph for the hostmachine. Here, increasing resources (ormore accurately reducing contention)would allow an increased use ofprocessor power in the latter stagesof the processing cycle.

To counter both these effects of hittingmessage queue buffer limits too quicklyand uneven processor utilizationduring the message injection phase, weadded a larger number of in-boundqueue processes and in-bound queues.

This enabled the in-bound messageload to be aggregated over moreresources thus reducing the latencytimes involved. Perhaps the biggestdifference was made by using MSMQ3.0 instead of MSMQ 2.0. The formerhas a 4GB memory buffer size limitbefore a new buffer allocation is made,which is three orders of magnitude abovethe 4MB buffer size in MSMQ 2.03.

Multi-node ProcessingIn the original designs of STE a singledatabase was used. As the number ofprocessing nodes (dual processormachines running the componentsoftware) increased, a distinct drop inthe overall processing rate was noticed.This drop in processing throughput isshown in Figure 7 and was caused bycontention within the databasecomponent.

The cause of the contention was notdue to operating system or component

software problems but due to excessivedisk queuing occurring in the RAIDarray. This meant that availablebandwidth for transferring informationonto the disks of the RAID array wasinsufficient to meet the demands madeby STE’s software elements. This effectis most easily seen when examining theinsertion rate of information into thedatabase. The performance graph forthe single database server is shown inFigure 8. Here, the correspondingperformance graph to the one shown inFigure 7 shows the dramatic reductionin inserts into the database cause bydisk queuing in the RAID array as thenumber of processing nodes increases.The contention for available resourcescaused by this queuing means that thesystem couldn’t reasonably cope withmore than 2 processing nodes in theoriginal design.

The next section discusses how thisissue was overcome.

Figure 6: Processor Time for a Typical Component

Initial recovery timefrom “step function”

input Mean processing levelfor sustained messagethroughput (in-bound

messages beingreceived)

Mean processing levelfor sustained messagethroughput (in-bound

message flowcomplete)

3 The individual message size islimited to 4MB in both MSMQ 3.0and MSMQ 2.0, but this was not anissue in our scenario.

Process Rate per Node

(single database server)

0

500

1000

1500

2000

2500

1 2 3 4

Processing Nodes

Inse

rtR

ate

(mes

sag

esp

erse

con

d)

Theoretical linear increasein performance on a pernode basis

Recorded process rate into the database as the number ofnodes increases

Figure 8: Process Rate per Node (Single DatabaseEngine)

Process Rate per Node

0.00

100.00

200.00

300.00

400.00

500.00

600.00

1 2 3 4

Number of Nodes (Processors)

Mes

sag

eT

hro

ug

hp

ut

(mes

sag

esp

erse

con

d)

Figure 7: Process Rate per Node


Server Hashing AlgorithmAfter discussions with the MicrosoftSQL Server team regarding techniquesfor improving the available bandwidthfor disk operations the use of a hashingalgorithm was proposed which wouldenable multiple database serves to beincorporated into the overall solution.The purpose of the hashing algorithmis to use a unique key on the tradederived from the message data, and acorresponding hash function whichproduces a single numeric valueresolving to a unique instance of adatabase server. A hashing algorithmwas chosen to reflect our business needto always direct constituent messagesof specific trades to the same databaseserver. For the benchmark the keyconsisted of a subset of the matchingcriteria defined in Table 2. The keywas constructed by concatenating achosen subset of matching criteriavalues and converted to a single, verylong integer number. This number wasthen divided by a binary value

representing the number of proposeddatabase servers (or instances) asshown in the following formula:

Using this formula4 we very effectivelyimproved performance by federatingmultiple database servers (orinstances) with a hash.

The infrastructure architecture for theproposed utility was revised to reflectthe inclusion of the multiple databasesolution using the hashing algorithmas shown in Figure 9. For ongoing

benchmark testing support for up to 16database servers was made within thesoftware components, however, thesystem was tested to a maximum of4 such servers.

Repeating the tests using the hashingalgorithm to distribute the databaseload across four database serversyielded very impressive results.

The Little’s Law CurveThe performance metrics determinedby Buzen and Denning are based on afundamental result produced by J.D.C.Little [LITTL61]. The generalisedperformance characteristic discoveredby Little is shown in Figure 10.

=

N

KremainderS

2

4 S denotes the server number (base 0)in the range 0-(N-1); K denotes thevery large integer value determined bythe key for the message and N denotesthe proposed number of databaseservers (clearly N must be greaterthan 0 and less than K). Thecalculation of the modulus functionmay seem complex, however, bychoosing a binary multiple for thedivisor the calculation reduces to rightshifting the value of K by N bitposition for N > 0. For example if Nwere set to 3 then the divisor would be8 and shifting the value of K right by3 bit positions would accomplish therequired division. The remainder ondivision by 8 is therefore simply thevalue of the least significant 3 bits of K(an integer in the range 0 through 7). Figure 9: Final Hardware Infrastructure Arrangement

“The impressive results of the benchmark largely speak for themselves with the overallperformance, scalability and flexibility well established.”


For any queuing model, whenprocessing a specific number of tasksthe response time increases asthe arrival rate (and for a balancedqueuing model the completion rate)increases. The curve is characterisedby having an almost linear portionin the early stages gettingprogressively more asymptotic as theinput (and completion rate) increases.

The first observable results, Figure 11,showed that the increase of 400% inthe available database bandwidthplaced the system in the linear portionof the of the performance graphproducing an almost linear responsecharacteristic when the processingcomponents are scaled from 1 to4 nodes.

In fact the measured results showed anextremely linear scaling between 1 and4 processing nodes with only a minimaldivergence from the linear model beingobservable. However, if the input rate is

increased (in this case by increasingthe number of processing nodes to 8) adivergence from the linear scaling casecan be observed. This measured effectis shown in Figure 12.

Using measured results the Little’sLaw curve can be drawn for the testqueuing network model as shown inFigure 13. The result shows that theoperational performance of the queuingmodel will suffer increasing reductionin performance as the number ofprocessing nodes increases past 4components with a significantreduction at 8 processing components.At this point it is worth noting thescale on the left hand side of the graphin Figure 12 showing the throughputrate of the entire STE queuing modelmeasured at some 7734 messagesper second.

Clearly the next scaling option to usewould be to increase the number ofdatabase servers to 8 (i.e., the next

available binary multiple). With thiswe would reasonably expect to see themessage processing throughput ratereach in excess of 15,000 messages persecond given a suitable increase in thenumber of processing nodes used.

Degradation with Stored VolumeAt the measured processing rate thequeuing network would achieve asustained rate in excess of 27,842,400messages per hour or 222,739,200 peroperating day. It is reasonable to ask ifsuch a high processing rate measuredover several minutes could besustained over time. To determine thecharacteristics of the model as storedvolumes increase, the processing loadof some 2,000,000 messages was usedas a base figure and a subsequentprocessing run of 7,000,000 messageswas used to determine the effect on theoverall performance as the databasevolumes increased. The measureddegradation in message throughputwith volume is shown in Figure 14.

Database Degradation with Volume

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Message Volume(millions of messages)

1 2 3 4 5 6 7

Mes

sag

eR

ate

(mes

sag

esp

erse

con

d)

Figure 14: Database Degradation with Volume

The Generalised Little's Law Curve

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 6 7 8 9 1 11 1 1 1 15 1 17

Completion Rate (jobs/unit time)

Res

po

nse

Tim

e

Deviation from the Linearmodel as the completionrate increases

Message Process Rate

0

1000

2000

3000

4000

5000

6000

Pro

cess

Rate

(mess

ag

ep

er

se

co

nd

)

1 2 3 4

Processing Nodes

Message Process Rate

0.00

1000.00

2000.00

3000.00

4000.00

5000.00

6000.00

7000.00

8000.00

9000.00

10000.00

1 2 3 4 5 6 7 8

Processing Nodes

Pro

cess

Rat

e(m

essa

ges

per

seco

nd

)

Figure 10: Generalised Little's Law Curve

Figure 11: Linear Scaling with Multiple DatabaseInstances (Single Database Engine)

Figure 12: Scalability with Eight Processing Nodes

Performance CurveIncreasing Processing Nodes

0100200300400500600700800900

1 2 3

Number of Processing Nodes

Res

po

nse

Tim

e

Figure 13: The Measured Little's Law Curve


Here the processing rate dropped tosome 5,500 messages per second as thevolume processed reached the7,000,000 messages target. Even at thisextreme level, the queuing model wasachieving some 19,800,000 message perhour or 158,400,000 per operating day.The granularity of the result did notpermit a more accurate measurementof the degradation effect other than thelinear approximation shown here. If afiner granularity observation had beenmade the degradation rate would havebeen seen as a curve rather than astraight line indicating that a reduceddegradation effect would be experiencedas volumes increased further (possiblya characteristic of the paging schemasused for the B-Tree structure ofmodern RDBMS’s). The 7,000,000messages processed during this testrepresented 1,000,000 trades processedin a relatively short period of time.

It is worth noting that there are manyexamples of existing transactionengines in the financial servicesindustry that have failed to reach thisoperating level using technology ratedto a higher performance level than theWindows and Intel based machinesused in these tests.

The Buzen & Denning ResultsTo determine the performance of theindividual components the Buzen andDenning metrics need to be determined.The monitoring process measured thelength of each of the processing queuesused for the queuing network modeland the performance metrics werecalculated. A sample result from thecalculation process is shown in Table 6.

The sample shows the processing ofapproximately 398,573 messagesthrough the queuing model (taken as

a sample from one of two processingnodes). The host machine supportingthe 7 software components (one NOEand two each of ALC, NPR and SETsoftware modules) reached an average93% utilisation during the monitoringinterval (U) according to the Windowsperformance monitoring tools. For thethroughput calculation it must beborne in mind that there were twoprocessing components running foreach of the ALC, NPR and SETmessage types. The processing networktherefore achieved a mean throughputrate of approximately 1,162 messagesper time unit during the test with alatency time of approximately 14.97seconds. Latency, in this case refers tothe time difference between input andoutput message processing. A messageentering the network will appear onthe output side approximately 14.97seconds later at the measured

Figure 15: Measured Performance Curve

Figure 16: Expanded Performance Curve

Parameter NOE ALC NPR SET

Number of Processors 1 2 2 2

Observation Time - T units 343 343 343 343

Arrivals – N 53476 59040.5 56659 56849

Completions – C 53476 59040.5 56659 56849

Busy Time – B 315 329 308 322

Utilisation – U 0.92 0.96 0.90 0.94

Throughput – X 155.9067 172.1297 165.1866 165.7405

Mean Service Time - S 0.00589 0.005572 0.005436 0.005664

Execution Distribution - A 817,385 887,046.5 818,094 860,110

Mean Queue Length - L 2383.047 2586.141 2385.114 2507.609

Residence Time - RT 15.28508 15.02437 14.43891 15.12973

Queuing Time 15.27919 15.0188 14.43347 15.12407

Little's Law Curve –Buzen and Denning Results

0.00

100.00

200.00

300.00

400.00

500.00

600.00

700.00

100 200 300 400 500 600 700 800 900 1000 11001200 1300 1400 15001600 17001800

Processing Rate (messages per second)

Res

po

nse

Tim

e(s

eco

nd

s)

Response curve for 500,000message load

Little's Law Curve - Expanded

0.00

50.00

100.00

150.00

200.00

250.00

300.00

350.00

400.00

Processing Rate (messages per second)

Res

po

nse

Tim

e(s

eco

nd

s)

Performance Operating Region

NO

EM

essa

ge

Pro

cess

or

NPR

Mes

sag

ePr

oce

sso

r

SET

Mes

sag

ePr

oce

sso

r

ALC

Mes

sag

ePr

oce

sso

r

Table 6: Sample Buzen and Denning Results Calculation


processing levels. The sample rate wasset to a 7 second interval (owing to theuse of an un-calibrated tick interval)and the sample data set was 500,000messages (two messages drivers eachwith 250,000 messages from thestandard test data set). In this casethere were two process engines (or hostmachines) and the results shown weretaken from one of those processingengines (note that the results for theALC, NPR and SET components areaggregated across two componentssince the original data was measuredfrom the message queue from whichboth components were being fed).

The marginally higher service time(S) for the NOE messages reflects thehigher processing level required ofthis component because of audit trailpersistence and validation processing.The performance curve for each of theprocessing components is shown inFigure 15 with the overall performancepoint marked for clarity. This isfundamentally the generalised Little’sLaw curve for the STE processingengine; however, this generalised viewdoes not give all of the detail necessaryto accurately predict the operatingperformance of the engine. It isapparent that, for the individualcomponents there are differentcompletion rates and thereforedepending on which view of theperformance metrics you wish to takethere will be a different correspondingperformance values.

This gives rise to the PerformanceOperating Region (POR) for thenetwork as shown in the shaded areaof the graph in Figure 16. In thisparticular instance the results arereasonably close and therefore thecorresponding performance operatingregion is narrow. This is not always the

case, however, and there are examplesof systems where the POR covers aregion exceeding 400% of the meanperformance level. The prediction ofthe POR requires some complicatedmathematics that were beyond thescope of this benchmark exercise,however, the effect of the POR isincluded here to explain the variationin the measured results duringrepetitive tests.

Comments and ConclusionsThe impressive results of the benchmarklargely speak for themselves with theoverall performance, scalability andflexibility well established. Thethroughput rate for the overall enginecertainly places it amongst some of thelargest transaction engines for posttrade processing infrastructure in thefinancial services industry. The targetand stretch performance levels wereexceeded with comfortable margins,and there are strong indications thatthe overall architectural approach willsupport even greater messagethroughputs. It is certainly worthstating that, with the achievedperformance levels, current Microsofttechnology offerings are capable ofoperating within the enterprise layer ofany financial institution. Some aspectsof operation within the enterpriselayer, like resilience and reliability werenot tested during this benchmark andremain to be proven for this design.However, networked or grid computingbased architectures like this one haveinherent characteristics to supportextremely high levels of resilience andreliability. The use, therefore, of efficientgrid based processing machines andlow cost software technology wouldseem to be a winning combination.

Low Cost and Efficient RealisationProbably one of the more significantresults of the entire benchmark processis the now proven ability of Microsofttechnology to perform and scale toenterprise levels. The processing ratesachieved with the queuing architecturecertainly place the Microsoft operatingsystem in the upper band for capabilityand scalability in networkedcomputing. The second most significantresult of the testing was the relativelylow cost of implementation of thesystem in the Microsoft environment.

Potential Improvement AreasIn addition to the monitoring toolsused to detail the benchmark results,Microsoft was able to provideadditional process monitoring tools(which will be available in VisualStudio 2005) that gave a detailed viewof the execution of the softwareelements. The Microsoft analysis toolsindicated that the software componentswere spending on average 30%-35% oftheir time performing functions relatedto data extraction from the XMLmessages. This was not an overlysurprising result since the mainfunction of the software componentswas to validate and process stringtypes. To access required informationthe C/C++ strstr search function wasused and we treated the entire messageas one complex string. (Note: for ourproblem domain this was faster thandirectly using an XML parser andDOM objects with XSLT.) Althoughin general circumstances the use ofstrstr produces adequate performancelevels, there are more efficienttechniques that can be employed toextract information from string-basedmessage structures.

R.S. Boyer and J.S. Moore [BYMR77]described a very efficient mechanism


for searching well structured strings.The algorithm works well where thestructure of the string is known inadvance and is used predominantly inapplications where searching large textstrings is required, such as in digitallibrary applications, editors or wordprocessors. For the queuing networkthe use of the algorithm would at firstsight seem inappropriate since we haveno way of determining the nature orstructure of the next received messagewithin the network. However, for theprocessing components the structure ofthe message is known since we routethe message according to type forfurther processing. The use of theBoyer-Moore algorithm could yieldimproved results over the existingimplementation of the network,however, the relatively small size of theXML messages (an average of 1,500bytes per message) might be too smallfor the Boyer-Moore algorithm to yieldresults that would justify the workrequired to implement the algorithm.

Itanium and 64-Bit ProcessingThe operation of the queuing networkmodel was tested using a beta versionof SQL Server running on new (at leastit was then) Itanium 64-bit hardwareand the Windows operating system.Although this was not acceptable as aproduction benchmarking environment(because of using beta software) theresults would be a useful indicator of

future performance gains that couldbe obtained using this future Intel/Microsoft technology.

On this hardware the measuredthroughput was an average of 872messages per second which wasconsidered extremely high consideringthe environment in which the test tookplace. Firstly, this result was obtainedusing a standard SCSI disk unit asopposed to the RAID arrays used in themain benchmark exercise. StandardSCSI performance rates would havebeen considerably slower than theRAID performance rates. Secondly,the Itanium database server had onlytwo processors installed against the8 processors used for the databaseengines in the benchmark. Theopportunity to perform a fullbenchmark test within a 64-bitenvironment is eagerly awaited.

C# and Managed CodeThe software components were alsogenerated for use in the Microsoft C#Managed Code environment where adirect comparison could be madebetween the C/C++ and C# versions.As a simple test the operation of themessage drivers was comparedbetween the operating models.The process involved was fairly simpleso that the effects of inefficient codingcould be ignored (the actual numberof active lines of code was very small).The process was to take a preparedmessage file and stream the data intoa code loop. Processing would continueuntil a message separation characterwas received. The resultant messagewas then wrapped in a standardGSTPA header and a digital signatureapplied to the message block. Themessage was then written to themessage queue for the queuingnetwork model for processing. This

process continued until the entireprepared file was read.

The parameter of interest to us wasthe throughput rate at which messagecould be read off the data file andqueued. The results of the test areshown in Figure 17.

[These results clearly show theenhanced performance of C/C++ overthe managed code environment (.NETFramework 1.1). It is also fair to pointout that the results also include thecomparison of efficiency of theinteroperability layer between C/C++and C# which is crossed for accessingMSMQ.

At first sight it could be argued thatfrom a performance perspective, themanaged code environment shouldnever be implemented in place of aC/C++ installation. This view, however,would be misleading since all systemsolutions are a compromise betweencost, performance and reliability. Theoverall performance results for themanaged code environment reflectedthe test performed on the (simple)message driver component producing athroughput rate of approximately 2000messages per second. Although thisrate is around 25% of the base C/C++level, there are definitely compensatingfactors that must be considered. Theproduction of C# code is significantlymore efficient that that obtainable withC/C++ code, in fact the rate at whichoperational code could be developed inC# was extremely impressive. Note, alower rate of 2000 messages per second(which is 7,200,000 messages per houror 57,600,000 messages per day) is stillconsidered in the upper bracket oftransaction engine benchmarks and itwill only get better as managed codegets faster!

Comparison of C# (managed) and C/C++ (unmanaged) code

01000200030004000500060007000

1 2

Module Realisation

Mes

sag

eT

hro

ug

hp

ut

C+

#C

Figure 17: Comparison of Managed andUnmanaged Code


Furthermore, care must be taken whencomparing managed and unmanagedcode environments. The use of virtualmachine environments like the CommonLanguage Runtime (CLR) used in the.NET Framework and indeed Java/ J2EEVM based environments can producebenchmark figures comparable with thatof C/C++ code where memory-basedoperation are concerned. Unfortunately,such benchmark comparisons give afalse picture of the overall performancelevels that can be expected because mostapplications include elements of localand remote I/O operations and dynamicobject creation and deletion. Conversely,the ease of implementation and potentialimprovements in reliability andmanageability may well allow managedcode environment to out-perform C/C++in the creation of applications sacrificingperformance for lower costs and fasterimplementation times.

Given that we have significantunrealisable performance (see theabove discussion on this topic) with ourunmanaged code implementation of theSTE system – because the architecturescaled-up so remarkably well, weactually find ourselves in an opportuneposition to trade-off this unrealisableperformance for the benefits of usingmanaged code in future implementationsof this generalised architecture. Forsome insights into this statementsee [SEVA04].

References[BDEN78] Buzen J.P. and Denning P.J.,“The Operational Analysis of QueuingNetwork Models”, ACM Computingsurveys, 10, 3, Sept. 1978, 225-261.

[BYMR77] Boyer R.S, Moore J.S,1977, A fast string searchingalgorithm. Communications of theACM. 20:762-772.

[LITTL61] Little J.D.C, “A Proof of theQueuing Formula L = λW”, OperationsResearch, 9, 1961.

[PHEL02] Pat Helland, 2002,“Autonomous computing: Fiefdoms andEmissaries”, Microsoft Webcast,http://microsoft.com/usa/Webcasts/ondemand/892.asp

[SMITH90] Connie U. Smith,“Performance Engineering of SoftwareSystems”, Addison-Wesley, ISBN 0-201-53769-9, 1990.

[SEVA04] Arvindra Sehmi andClemens Vasters, FABRIQ GotDotNetWorkspace:http://workspaces.gotdotnet.com/fabriq

[SMWIL02] Connie U. Smith andLloyd Williams, “PerformanceSolutions: A Practical Guide toCreating Responsive, ScalableSoftware”, Addison-Wesley,ISBN 0-201-72229-1, 2002.

[UASDS01] Susan D. Urban, AkashSaxena, Suzanne W. Dietrich, and AmySundermier, “An Evaluation ofDistributed Computing Options for aRule-Based Approach to Black-BoxSoftware Component Integration”,Proceedings of the Third Int’l WorkshopAdvanced Issues of E-Commerce andWeb-Based Information Systems(WECWIS’01), 2001.

[ZHES01] Huaxin Zhang and EleniStroulia, “Babel: RepresentingBusiness Rules in XML for ApplicationIntegration”, Proceedings of the 23rdInternational Conference on SoftwareEngineering (ICSE’01), 2001.

Richard DraytonFiS [email protected] Drayton has been activelyinvolved in technology architecture anddesign for 30 years. Recognized for hiswork in the field of software performanceengineering, he is a practicing memberof the ACM's SIGMETRICS group. Forthe past 15 years Richard has beenactive in the design and development ofFinancial Trading and Processingsystems for many of the worlds leadinginvestment banks. Previously Richardwas the head of Front Office Technologyfor Deutsche Bank in Frankfurt, GlobalHead of Architecture for CommerzFinancial Products and Global Head ofArchitecture for Dresdner Bank. Aspart of his work with the IEEE andACM, he has focused on the adaptationof high performance computing solutionsto the financial services industry withparticular focus on the implementationof end to end STP market infrastructuresolutions. He is the Executive Directorresponsible for technology solutionswith the Financial InfrastructureSolutions Group. He holds an M.Sc.in Electronic System Design as wellas degrees in Pure Mathematics andCommunications Technology.

Arvindra SehmiMicrosoft [email protected] Sehmi is an Architect inMicrosoft EMEA Developer andPlatform Evangelism Group. He focuseson enterprise software-engineering bestpractice adoption throughout the EMEAdeveloper and architect communityand leads Architecture Evangelism inEMEA for the Financial ServicesIndustry where his current interest isin high performance asynchronouscomputing architectures. Arvindra isthe executive editor of JOURNAL. Heholds a Ph.D. in Bio-medical Engineeringand a Masters degree in Business.

JOURNAL4 | Enterprise Architecture Alignment Heuristics 34

IntroductionThe alignment between BusinessProcesses (BP) and InformationTechnologies (IT) is a major issue inmost organizations, as it directlyimpacts on the organization’s agilityand flexibility to change according tobusiness needs. The concepts uponwhich alignment is perceived areaddressed in what is called today the“Enterprise Architecture”, gatheringbusiness and IT together.

Many Enterprise ArchitectureFrameworks have been proposed,focusing on different concerns and withdifferent approaches for guiding thedevelopment of an IT infra-structurewell suited for the organization. EachEnterprise Architecture Frameworkhas its own concepts, components,and methodologies to derive thecomponent all the required artifact.However, when the main concern isalignment, we may consider simplerarchitecture concepts and simplermethodologies because the focus is notto define development artifacts but onlyto check their consistency.

The focus of this paper is to show howalignment between Business and ITcan be stated in terms of thecomponents found in most EnterpriseArchitectures.

In the next section, we briefly introducethree well known EnterpriseArchitecture Frameworks, namely: TheZachman Framework, Capgemini’sIntegrated Architecture Frameworkand the Microsoft EnterpriseArchitecture. We do not intend to fullydescribe them, but solely presentthe main aspects.

Next, we present the basic conceptscommon to these frameworks, focusing

on their generic properties and leavingout specificities of each framework. Wewill consider four basic components ofan Enterprise Architecture: BusinessArchitecture, Information Architecture,Application Architecture andTechnical Architecture.

Finally, we show how alignmentbetween Business and IT can bedisaggregated into alignment betweenthese basic components, we presentgeneral heuristics defined in terms ofthe architectural components andpresent the work in progress. We willnot address the Technical Architecture;the main reason being that technicalalignment is mostly dependent on thetechnology itself.

Enterprise ArchitectureFrameworksThe Zachman FrameworkThe Zachman Framework forEnterprise Architecture (www.zifa.com)proposes a logical structure forclassifying and organizing thedescriptive representations of anenterprise. It considers six dimensions,which can be analyzed in different

perspectives, as presented in Figure 1;the rows represent the perspectivesand the columns the dimensions withina perspective.

The Framework is structured aroundthe views of different users involved inplanning, designing, building andmaintaining an enterprise’sInformation Systems:– Scope (Planner’s Perspective) –

The planner is concerned with thestrategic aspects of the organization,thus addressing the context of itsenvironment and scope.

– Enterprise Model (Owner’sPerspective) – The owner is

“We developed Alignment’s Heuristics as a common sense rule (or a set of rules) to increasethe probability of finding an easier way to achieve Business, Information and ApplicationArchitectures alignment.”

Enterprise Architecture Alignment HeuristicsBy Pedro Sousa, Carla Marques Pereira and José Alves Marques, Link Consulting, SA

Contextual

Why?

Security

Governance

Conceptual

what?

Logical

how?

Physical

with what?

Business Information Information

systems

Technology

infra-structure

Contextual answers the question 'why do we need an architecture?' and 'what is the overall context?'

Conceptual answers the question 'what are the requirements and what is the vision of a solution?'

Logical answers the question 'how are these requirements to be met?'

Physical answers the question 'with what is the solution built?'

Figure 2 – The Integrated Architecture Framework

Figure 1 – The Zachman Framework, © John A.Zachman International


interested in the business perspectiveof the organization, how it deliversand how it will be used.

– System Model (Designer’sPerspective) – The designer isconcerned with the systems of theorganization to ensure that they will,in fact, fulfill the owner’s expectations.

– Technology Model (Builder’sPerspective) – The builder isconcerned with the technology usedto support the systems and thebusiness in the organization.

– Detailed Representations(Subcontractor’s Perspective) – Thisperspective addresses the builder’sspecifications of system componentsto be subcontracted to third parties.

While the rows describe theorganization users’ views, the columnsallow focusing on each dimension:– Data (What?) – Each of the cells in

this column addresses theinformation of the organization. Eachof the above perspectives should havean understanding of enterprise’s dataand how it is used.

– Function (How?) – The cells in thefunction column describe the processof translating the mission of theorganization into the business andinto successively more detaileddefinitions of its operations.

– Network (Where?) – This column isconcerned with the geographicaldistribution of the organization’sactivities and artifacts, and how theyrelate with each perspective of theorganization.

– People (Who?) – This columndescribes who is related with eachartifact of the organization, namelyBusiness processes, information andIT. At higher level cells, the “who”refers to organizational units,whereas in lower cells it refers tosystem users and usernames.

– Time (When?) – This columndescribes how each artifact of theorganizations relates to a timeline, ineach perspective.

– Motivation (Why?) – This column isconcerned with the translation ofgoals in each row into actions andobjectives in lower rows.

Capgemini’s IntegratedArchitecture FrameworkCapgemini has developed an approachto the analysis and development ofenterprise and project-levelarchitectures know as the IntegratedArchitecture Framework (IAF) shownin Figure 2 [AM04].

IAF breaks down the overall probleminto a number of the related areascovering Business (people andprocesses), Information (includingknowledge), Information Systems, andTechnology Infrastructure, with twospecial areas addressing theGovernance and Security aspectsacross all of these. Analysis of each ofthese areas is structured into fourlevels of abstraction: Contextual,Conceptual, Logical and Physical.

The Contextual view presents theoverall justification for the organizationand describes the contextual

environment. It corresponds largely toZachman’s Planner’s Perspective row.

The Conceptual view describes whatthe requirements are and what thevision for the solution is. The Logicalview describes how these requirementsand vision are met. Finally, thePhysical view describes the artifacts ofthe solution.

These views have no direct relation toZachman’s perspectives because in IAF,Business, Information, InformationSystems and Technology Infra-structure are the artifacts of thearchitecture whereas in Zachman,Business, Information Systems andTechnology are views (perspectives).

Microsoft Enterprise ArchitectureMicrosoft Enterprise Architectureshown in Figure 3, is a two dimensionalframework, that considers four basicperspectives (business, application,information, and technology), andfour different levels of detail(conceptual, logical and physicaland implementation).

The business perspective describes howa business works. It includes broadbusiness strategies along with plans formoving the organization from its currentstate to an envisaged future state.

“The issue of alignment is based on the coherency between elements of Business Architecture,elements of Information Architecture and elements of Application Architecture.”

Microsoft Architecture

Business Perspective

Application Perspective

Information Perspective

Technology Perspective

Figure 3 – Microsoft Enterprise ArchitecturePerspectives

BusinessArchitecture

InformationArchitecture

ApplicationArchitecture

TechnicalArchitecture

Alignment

Figure 4 – Decomposing Business and IT Alignmentinto Architectural Components


The application perspective defines theenterprise’s application portfolio and isapplication-centered. The applicationperspective may represent cross-organization services, information, andfunctionality, linking users of differentskills and job functions in order toachieve common business objectives.

The information perspective describeswhat the organization needs to know torun its business processes andoperations. The information perspectivealso describes how data is boundinto the business processes, includingstructured data stores such asdatabases, and unstructured datastores such as documents,spreadsheets, and presentations thatexist throughout the organization.

The technology perspective provides alogical, vendor-independent descriptionof infrastructure and system componentsthat are necessary to support theapplication and informationperspectives. It defines the set oftechnology standards and servicesneeded to execute the business mission.

Each of these perspectives has aconceptual view, a logical view and aphysical view. Elements of the physicalview also have an implementation view.Microsoft Enterprise Architectureis described in detail atmsdn.microsoft.com/architecture/enterprise/

Making a parallel between theMicrosoft’s and Zachman’s EnterpriseFramework, the Business perspectivecorresponds to Planner’s and Owner’sperspectives; Application perspective toZachman Designer’s perspective;Technology perspective to Builder’s andSubcontractor’s perspectives

and finally, the Microsoft Informationperspective corresponds to the Datacolumn in the Zachman framework.

Identifying EnterpriseArchitecture Components froman Alignment perspectiveAs we could see in previous sections,different Enterprise Frameworks havedifferent ways to model artifacts ofthe Enterprise, their perspectives andthe different levels at which they canbe described.

The Enterprise Frameworks addressa large number of problems andtherefore have a degree of complexityfar larger than needed if the soleproblem is the alignment of thebusiness and IT architecture. Thus wecan simplify these models and justconsider the sub-architectures whichhave a certain commonality. It is outof the scope of this paper to fully studyand justify similar concepts in theseEnterprise Architecture Frameworks.If alignment is the main concern, anEnterprise Architecture has fourfundamentals components: BusinessArchitecture, Information Architecture,Application Architecture and TechnicalArchitecture. This is not new, and it hasbeen long accepted in the EnterpriseArchitect Community (for instance seewww.eacommunity.com).

We will address the issue of alignmentbased on the coherency betweenelements of Business Architecture,elements of Information Architectureand elements of ApplicationArchitecture. The more elements eachof these Architectures has, the morerich and complex is the concept ofalignment, because more rules andheuristics need to be stated to governthe relation between these elements.

So, in order to build up alignment,one must first clarify the elements ofeach architecture (see Figure 4).

In what concerns the TechnicalArchitecture, its alignment is mostlydependent on the technology itself. Weare currently investigating how ServiceOriented Architecture (SOA) conceptsoverlap with previous architecturesand how alignment could be formulatedin its model. This is ongoing work andis beyond the scope of this paper.

Business ArchitectureThe Business Architecture is the resultof defining business strategies,processes, and functional requirements.It is the base for identifying therequirements for the informationsystems that support business activities.It typically includes the following: theenterprise’s high-level objectives andgoals; the business processes carriedout by the enterprise as a whole, or atleast a significant part; the businessfunctions performed; majororganizational structures; and therelationships between these elements.

In this paper, we consider a simplercase where the Business Architectureincludes only Business Processes, eachbusiness process is composed by a flowof business activities and each activityis associated with information entities,time, and people. Business Processeshave attributes such as criticalness,security level, delayed execution(batch), on-line, and so on.

Information ArchitectureThe Information Architecture describeswhat the organization needs to know torun its processes and operations, asdescribed in the Business Architecture.It provides a view of the business

“Applications that manage information entities should provide means to make the entityinformation distributable across the organization using agreed-on protocols and formats.”


information independent of the IT viewof databases. In the InformationArchitecture, business information isstructured in Information Entities,each having a business responsible forits management and performingoperations like: Acquisition,Classification, Quality Control,Presentation, Distribution, Assessment,and so on.

Information Entities must have anidentifier, defined from a businessperspective, a description, and a set ofattributes. Attributes are related tobusiness processes (that use or producethem) and to applications (that create,read, update and delete them).Attributes are classified according todifferent properties such as Security,Availability and so on.

As an example, Client and Employee aretypical Information Entities. Employeehas attributes such as “courses taken”,“competences”, “labor accidents”, and“career”. Each of these attributes canbe physically supported by a complexdatabase schema in different databasesused by several applications.

Application ArchitectureThe Application Architecture describesthe applications required to fulfill twomajor goals: (i) support the businessrequirements, and (ii) allow efficientmanagement of Information Entities.Application Architecture is normallyderived from the analyses of bothBusiness and Information Architectures.

Application Architecture typicallyinclude: descriptions of automatedservices that support the businessprocesses; descriptions of theinteraction and interdependencies(interfaces) of the organization’s

application systems; plans fordeveloping new applications andrevision of old applications based onthe enterprises objectives, goals, andevolving technology platforms.

Applications also have requiredattributes, such as availability (uptime), scalability (ability to scale upperform), profile based accesses (abilityto identify who does each tasks).

Alignment and ArchitectureComponents After identifying the major architecturalcomponents from an alignment point ofview, we are now in position to addressthe relations between these componentsin terms of alignment.

Alignment between Business andApplications In a fully aligned Business andApplications scenario, the time andeffort business people spent to run thebusiness should be only devoted to“reasoning” functions. On the contrary,misalignments force business peopleto perform extra and mechanic worksuch as:

– Inserting the same data multipletimes in different applications.

– Logging in multiple times, once foreach application they need to access.

– Recovering from a failed operationacross multiple systems, requiringcareful human analyses to rollback toa coherent state.

– Overcoming inappropriate applicationfunctionality. For example, printinginvoices one by one becauseapplications do not have an interfacefor multiple printing.

Notice that alignment betweenBusiness and Applications in the above

context, does not imply a flexible andagile IT Architecture, in fact, a measureof a flexible and agile IT Architecture isthe effort IT people make to keep theBusiness and Applications alignedwhen Business is changing. This topicis addressed next.

Alignment between Informationand ApplicationsIn fully aligned Information andApplications Architectures, IT peopleonly spent effort and time codingbusiness functions and logics. On thecontrary, misalignments betweeninformation and application require ITpeople to do extra coding for:– Keeping multiple replicas of the same

data coherent, because they areupdated by multiple applications.

– Assuring coherency from multipletransactions, because a singlebusiness process crosses multipleapplications.

– Gathering information from multiplesystems and coding rules to produce acoherent view of the organization’sbusiness information.

– Transforming data structures whendata migrates between applications.

The extra coding is required toconsistently modify both architectures.However, since the information criticalto run the business (InformationArchitecture) is far more stablethan the applications to support it(Applications Architecture), mosteffort really is put into changing inthe Applications.

Alignment between Business andInformation Information and BusinessArchitectures are aligned whenbusiness people have the informationthey need to run the business. This


means accurate, with the right level ofdetail, and on time information. Unlikethe previous misalignments, here theimpact is neither time nor effort, butthe impossibility of getting theadequate piece of information relevantfor the business.

Examples are abundant; a CEO asksfor some report, where sales figuresneed to be disaggregated by type ofservices. Assuming the reportrequested by the CEO has either actualor foreseen business relevance, thepossibility/ impossibility to producesuch report is an evidence of thealignment/misalignment betweeninformation and BusinessArchitectures. To produce the report wemust have the adequate basic data anddata exploring applications, and thusthis is an issue that should be dealt bythe previous Alignments (Information/Applications and Business/Applications).

Alignment HeuristicsWe developed Alignment’s Heuristicsas a common sense rule (or a set ofrules) to increase the probability offinding an easier way to achieveBusiness, Information and ApplicationArchitectures alignment.

Heuristics presented result frommapping our experience, both asacademic teaching at university andprofessional consultancy services, intothe context of the Business,Information and ApplicationArchitectures presented in this paper.We present the heuristics that weconsider to have a greater value, givenits simplicity and its results.

The main heuristics to consider whenchecking alignment between Businessand Application Architectures are:

– Each business process should besupported by a minimum number ofapplications. This simplifies userinterfaces among applications,reduces the need for applicationintegration, and also minimizes thenumber of applications that must bemodified when the businessprocess changes.

– Business activities should besupported by a single application.This reduces the need for distributedtransactions among applications.

– Critical business processes should besupported by scalable and highlyavailable applications.

– Critical business processes/activitiesshould be supported by differentapplications than the non criticalbusiness processes/activities. Thishelps to keep critical hardware andpermanent maintenance teams assmall as possible.

– Each application’s functionalityshould support at least one businessprocess activity. Otherwise, it playsno role in supporting the business.

– Information required for criticalprocesses should be also supportedby scalable and highly availablesystems.

– Business processes activitiesrequiring on-line/batch supportshould be supported by applicationsrunning on different infra-structures,making easier the tuning of thesystems for operating window.

The main heuristics to check alignmentbetween Application and InformationArchitectures are:

– An information entity is managed byonly one application. This means thatentities are identified, created andreused by a single application.

– Information entities are createdwhen identifiers are assigned tothem, even if at that time noattributes are known. For example, ifthe Client information entity may becreated before its name and addressand other attributes are known. Evenso, the application that managesClient information entity must be theapplication that manages their IDs.

– Applications that manageinformation entities should providemeans to make the entity informationdistributable across the organizationusing agreed-on protocols and formats.

– Exporting and distributinginformation entities acrossorganization applications shouldmake use of a “data store”, ratherthan a point to point Applicationintegration. Applications managinga given information entity shouldexport its contents to the data storewhen its contents have changed.Applications requiring a giveninformation entity should inquire thedata store for up-to-date information.This allows for computationalindependence between applications,and make possible to size the HWrequired to run an applicationwithout knowing the rate that otherapplications demand informationfrom it. Further, if the applicationgoes down, it allows other to continueoperation using best possible data.


– Whenever possible, applicationsshould manage information entitiesof the same security level. Thissimplifies the implementation ofcontrols and procedures inaccordance with the security policy.

Finally the main heuristics to apply forBusiness and Information alignment are:

– All business processes activity create,update and/or delete at least oneinformation entity.

– All information entities attributesare read at least by one businessprocess activity.

– All information entities havean identifier understood bybusiness people.

– All information entities must havea mean of being transformed forpresentation to appropriateaudiences using enterprise-standardapplications and tools.

– All information entities must derivefrom known sources, and must havea business people responsible for itscoherency, accuracy, relevance andquality control.

– All information entities must beclassified and named within theInformation Architecture.

– For each information entity, Businesspeople should be responsible forassessing the usefulness andcost/benefits of information andsustain its continued use.

Final RemarksThe heuristics presented have beenvalidated and tested in real projects.In some cases, different heuristicsproduce opposite recommendations.This means that, a compromise solutionmust be reached. In other cases,heuristics do not favor optimal solutions

from an engineering point of view,because optimal solutions do notnormally take into account flexibilityand ease of change.

Another remark is the heuristicspresented intend to validate alignmentamong architectures, but assume thateach architecture is “aligned withinitself”. This means that we are notchecking if, for example, the BusinessArchitecture presents a good and acoherent schema of the businessprocesses and activities. Likewise,we are not checking if ApplicationsArchitecture makes sense or not.This requires more complex models,such as the ones initially proposed inoriginal Frameworks (Zachman, IAFand Microsoft).

The work presented was developed usingthe Zachman Framework as the generalapproach, but is strongly focused in thealignment issues. We have coded a largepercentage of the heuristics presentedin a modeling tool (System Architectfrom Popkin Software) and we are ableto derive a measurement for alignmentgiven a Business, Application andInformation Architecture. We havestarted work to include these heuristicsin Microsoft Visio.

We consider the heuristics to bevaluable because they force architectsto think about the justification of theirdecisions, leaving better documentedand solid architectures.

References[AM04] Andrew Macaulay, EnterpriseArchitecture Design and the IntegratedArchitecture Framework, JOURNAL1:Microsoft Architects Journal, Issue 1,January 2004.

Pedro [email protected] Sousa is responsible forEnterprise Architecture professionalservices at Link Consulting SA, wherehe has been involved in severalEnterprise Architecture projects forthe past six years. He is an AssociateProfessor at the Technical Universityof Lisbon (IST), where he teaches ontheses subjects in Masters of ComputerScience courses. He has published aseries of papers about Business andIT alignment.

Carla Marques PereiraCarla Marques Pereira has a MSc inComputer Science at the TechnicalUniversity of Lisbon (IST). She isreading for a PhD on the subject of“Business and IT alignment”. Carla is a member of Link Consulting SA’sEnterprise Architects team and aresearcher in the Center forOrganizational Engineering(ceo.inesc.pt).

José Alves MarquesJosé Alves Marques is CEO of LinkConsulting SA and a full Professor atTechnical University of Lisbon (IST).Technically, his main focus isDistributed Systems Architecturesand Services Oriented Architectures.He has a long track of publishedpapers and projects on these domains.

JOURNAL4 | Razorbills 40

Why did the razorbill raise her bill? So that the sea urchin would seeher chin!

IntroductionAlmost exactly seven years ago, I reada Microsoft internal memo called TheInternet Applications Manifesto, Take IIor The Sea Urchins Memo. This paperpushed for using XML as thecommunication mechanism betweenservices. To quote:

I call this model the sea urchin modelbecause I think of sites already as seaurchins with many spikes sticking outof them (URLs) each dispensing UI(HTML). These URLs as most havealready realized are the “methods”of the site. In this brave new world,I expect sites to also routinely becomeInformation dispensers with the“methods” of the site returning not UIbut information. Thus one can think ofthe Web as a sea filled with spiky seaurchins, all willing to emit informationof various kinds if one but knew to ask.Initially, one will have to know to ask.The evolution of an infrastructure thatcan automatically find suchinformation will depend upon theemergence of what I call “Schema”,namely universally agreed upon typesof semantics that allow multipleindependent consumers and producersto agree on how to share informationin specific domains. See opportunitiesbelow for more on this.

Since the Sea Urchins memo, we havemade great progress in describing thehow of services. With “Razorbills”, Ipropose putting a similar effort behindproviding a description of the ‘what’ ofservices. Describing the ‘what’ will adddramatically to the usefulness ofservices and “raise her bill”.

I propose standardizing the ‘what’description using Entities, Views,Actions, References and Relationships.

Service Oriented Architecture (SOA) isas much, if not more, about consumingservices as it is about building services.To enable better consumption I believewe need infrastructure that allowspeople that may be less technical butdo have domain expertise (the so calledbusiness analysts) to design and buildsolutions. We need infrastructure thatallows users of any application,structured forms as well as free formdocuments, to really use thefunctionality offered by services.Standardization on a way of describingservices in business terminology willhelp do that. It will help businessanalysts and users to better bridge thegap between the structured and theunstructured, the formal and informal,both in content and in process.

What and HowCurrent service descriptions such asWSDL and the WS-Standards describehow to access functionality. Once thecaller has decided which functionalityto invoke, the WSDL describes howto encode the message, how to routethe message, how to secure themessage, how to authenticate the callerand everything else needed to describehow to send the message. It does notdescribe what should be called. Forany dedicated client or consumer of aservice that already knows what to call,this is great and since the infrastructureand the description are based onstandards, the code is reusable andmany technologies can interoperate.This means that you can createcommunications between many parts,based on a single infrastructure.

However, this implies that the logic ofwhat is being called is coded into theapplication. When organizations wantto be flexible, they may want to accessservices to get information, aggregateinformation, and reorder information,or they may want to change processflows. Currently, to do this, they requiredevelopers. This may make the desiredchanges expensive – in many cases,too expensive. Organizations facing thisproblem would rather have businessanalysts, and even business users,make these changes. This would makechange cheaper, shorten the changecycle, and make them more agile.

If our software could communicatethis what to the business people, thenthey could define new aggregation ofinformation, and change or createthe flow of processes without the helpof developers.

If we describe what services have tooffer and we describe this in termsunderstood by both business peopleand software, we would not only havea description of the how, but also ofthe ‘what’.

This information is available today, butit is in the heads of the designers anddevelopers – at worst – and in somedesign documents at best. Theconsuming applications are beingwritten using this knowledge. However,it is generally not available to tools andother software that could be used byanalysts and users to solve theirchallenges. We need to find a standardway of describing this information.

In the following I will explain thefundamental idea of such a descriptionof what services expose and then I willhighlight possible use of such

“With ‘Razorbills’, I propose putting a similar effort behind providing a description of the what of services.”

Razorbills:What and How of Service ConsumptionBy Maarten Mullender, Microsoft Corporation


description. I believe and I hope thatI will be incomplete; going down thisroad will undoubtedly provide us withmore insight and more ideas.

WhatServices expose business functionalityto their consumers. Business analystsand business users often prefer tothink in terms of the entities they use.They understand the relationshipsbetween these entities and the actionsthat can be taken on those entities.

Entities Most services encapsulate entities.However concrete these entities werefor the designers of the service, for theconsumer of the service they areabstract. The consumer cannot touchthem or manipulate them; theconsumer does not know whether theyare stored in a single table or in a setof files. The consumer, however, doesunderstand what they represent:Customer, Order, Problem report, etc.The entity is made up of thecombination of all of its state, itsbehavior as well as its relations toother entities.

Views The service exposes views on theseentities. The consumer may obtain anaddress view, or a financial view, or ahistorical view on the customer. Theseviews are snapshots of the entity,taken from a specific viewpoint. Theviews are concrete. They have a well-defined schema exposing attributesor properties of the entity. It must benoted however, that views do notnecessarily present a subset of theentity information. They may verywell contain information from relatedentities and overlap with views on thoseentities. The view on an order may

contain the customer’s name and addressas well as the product descriptions sothat it may be used in the userinterface or to populate a document.

Actions The service also exposes methods oractions. Many of the actions arerelated to an entity. For example,“release order” or “upgrade customer”are clearly related to specific entities.And, if these actions require inputinformation, this information oftencomes from views on other entities.For example, a request to a supplier tocreate a sales order is built using theinformation from a purchase proposal.A view on the purchase proposal isconverted into the sales order request.

Entities, views and actions are artifactsthat we can describe, that a user canunderstand and that may help usdescribe services in ways that can beunderstood by business analysts.

I want to mention other artifacts thatare less often used.

References Many entities can be identified usingkey attributes. However, they can oftenalso be uniquely identified using otherattributes. Customers can be identifiedusing their social security number, taxnumber, or DUNS number. A list ofcustomers can be retrieved via the zipcode, the region, or the accountmanager. We can formalize thisthrough the concept of reference.A reference defines either a uniqueinstance or a list of instances. Areference, like a view, has a schema.

The actions exposed by the serviceaccept references to identify theentities they operate on. The references

are a more formal way of definingpossible input and a more genericway as well.

References formulate queries toservices. Such a query may be assimple as the list of key attributes oras complex as the formulation of acomplete set of conditions. Often areference will consist of a combinationof required and optional attributesallowing clients that have differentinformation to obtain the same viewsand use the same actions through thesame interfaces. Design of referenceschemas is just as important and just ascomplex as the design of view schemas.

RelationshipsNow, if we have a view of one entity, anorder say, it should not be too hard tobuild a reference to a related entity, e.g.the customer for that order. In otherwords, we can transform the order-viewinto a customer-reference which refersto the customer entity. This effectivelygives us a relationship between theorder view and the customer entity.

A relationship uses the data in oneview to construct a reference to anotherentity. Such a relationship can bedefined using the schemas of a viewand a reference. Consequently,relationships are not constrainedto a single service; they can definerelationships between entitiesencapsulated in different services.

If we are able to describe theseentities, the views on these entities,the relationships between theseentities and the actions on theseentities in terms that are understoodboth by the business people and thesoftware, we would have a descriptionof the ‘what’. This description should


not only describe the artifacts and therelation between them, but also theirbehavior, what people can do withthem and what software can do withthem. This description can easily belinked to, and in fact has to build on,the existing description of the how.

Even though much of this informationcan be provided by the designer of theservice interface similar to how WSDLis currently provided, an extension toWSDL cannot solve all of our needs.Relationships between entities offeredby different services should not bestored in either, but would requirestorage external to these services.Instance specific information, such asthe current availability of relationshipsand actions, as well as informationcreated at run-time, such as tasks ina collaborative workflow, require run-time interrogation of the service andthus a run-time description. This isvery much related to service contractsand the required infrastructure forproviding run-time information aboutthe state of a specific conversation.

Since views are what the designerexposes of the entities, much of theinformation is available in the systemstoday. Every service that was designedaround entities provides most of theinformation needed and in many othersystems the analysis has been doneand captured in some formof metadata.

The values in these references camefrom the view on the order. The namesof the elements and attributes may bedifferent between the view and thereference. Sometimes it is even useful tomerge in values that did not come fromthe original, if you want to restrict theresult set to a specific department forinstance, or, if you need a combination

“It is only a logical evolution that leads us from how to access services to what services offer.”

ExampleLet me give a simple example. Suppose we retrieve an order – i.e. a view on anorder – from a Web service. The order view may come in XML that, after muchsimplification, looks something like:

<ERP:Order OrderID=”8245” >

<Customer CustomerID=”4711” >

<CustomerAddress

Name=”Microsoft Corporation”

Country=”USA”

...

/>

</Customer>

<OrderLines>

...

</OrderLines>

</ERP:Order>

Table 1: Simplified Order View

The view is not restricted to what is stored in the order table or in the order object.This view on an order includes some customer information and possibly someproduct information, so that it can be displayed or printed.

From such an order, we may derive references to the customer for the order:

<ERP:CustomerReference CustomerID=”4711” Such a reference is useful /> to access services that share

the same customer numbers

<CRM:CustomerReference This reference would be CustomerName=”Microsoft Corporation” useful to access services State=”WA” that do not share the ... unique key.

/>

<My:CustomerReference Of course you can add CustomerID=”4711” redundant information,

so that the reference canCustomerName=”Microsoft Corporation” be used for all kinds of State=”WA” services. Ultimately there ... is no reason why the view

/> itself could not be used as a reference and it maysometimes be very useful.

Table 2: Examples of Customer References


of views as the source (e.g.: allcustomers that bought a product xyzfrom account manager abc). The recipethat gets you from the view or the setof views to the reference is calleda relationship. The relationship can bedefined as an XSLT or as a piece ofcode, it doesn’t really matter as longas the consumer understands how tointerpret the metadata and constructthe reference. The relationshipdescribes how the consumer ofthe services can construct a referencefrom a view.

References may refer to entities, thusallowing the consumer (the user or theclient-side code) to decide which view toretrieve, or they may refer to a specificview on an entity. The referenceschema alone is not sufficient for theclient to retrieve a view. The clientneeds a description of which serviceprovides access to the service, whichmethod needs to be called, and howto call that method using the givenreference. For any given view, theremay be one or more references that canbe used to obtain that view and foreach valid combination we need the

description of which method to call.Not all customer references can be usedto retrieve all customer views. Forexample, an ERP service may notprovide a method to retrieve customerinformation using the Social SecurityNumber (SSN), but the CRM servicemay. So, you may go to the CRM serviceto retrieve enough customer informationto build a reference to the ERP customer.

Lists have views too: a customeraddress list is completely different froma customer balance list. Both are viewson a list of customers.

The descriptions of Entities, Views,Actions, References and Relationshipswould not be limited to schemas andrecipes. These descriptions wouldencompass information about thoseartifacts. The descriptions need to beuseful both to humans and to software,including describing:

– Which method needs to be invoked toretrieve a specific view from areference.

– How long a view is guaranteed toremain valid, or how long it may be

used to formulate requests. Theprices in the catalog from Ikea arevalid for a year, but what informationcan I use from a production schedule,and, for how long? What can wedescribe about these different typesof information contained in views?

– Caching behavior and how tooptimally retrieve and manage theoffline views.

– Access rights on references and thuson relationships.

– Relevance of relationships for certainroles, allowing the user interface todisplay the most relevantrelationships.

– Types of relationships, such as one-to-one relationships, one-to-many,one-way, hierarchical, sibling, ‘hard’and ‘soft’, etc.

– How an aggregated view is composedand what the caching policy for theconstituent views and thus for theaggregated view is.

– When an action should be enabled ordisabled in the user interface.

– The action’s impact on the service’sstate. Does it update state, is theupdate transactional, does it throwexceptions or offer retries, etc.

Entities and Views in SOAThe differentiation between entitiesand views is important in a distributedenvironment. Data is copied betweenthe systems and whenever theconsumer of a service retrieves data,it will get a copy. When I requestinformation about an entity, I willreceive a copy. This copy is a view on,or a snapshot of, the state of the entityfrom a particular angle. I may receivethe customer’s financial view, or thecustomer’s satisfaction view, or thecustomer’s contact view. These viewsdepend on what I want to do and whatI am allowed to see. When I receivethat information only a few

<ERP:OrderListReference CustomerID=”4711” A reference to find all orders/> for a specific customer.

<ERP:OrderListReference A reference to a list of ProductID=”B-747” orders requiring a more

/> complex query over the order items.

<ERP:CustomerReference There is no problem CustomerID=”4711” offering a method that

/> retrieves orders using acustomer reference.

Table 3: Examples of OrderList References

References do not always return a single instance. It is equally possible to havereferences to lists, like for instance:


milliseconds later, the service mayalready have moved on and changed itsinternal state. The view may no longerreflect the internal state of the serviceeven when I receive it, and the chanceof my view being out of sync increasesthe longer I hold on to it. Thedifferentiation between view and entityis an important recognition of the factthat the consumer has an outside viewon the entity and that there is a timedifference between creating that viewand receiving it. It’s like taking apicture; the picture is frozen in time,whereas the subject of the picture maymove on. If you take a picture of abuilding, you will not see the entirebuilding; you will only see it from oneparticular angle. The building will notchange much after the picture has beentaken, but the people on the picture,the clouds on the picture and even thetrees on the picture will move. Thesnapshot is taken from a particularangle or viewpoint and frozen in time.

Because views are copies of data andmay be or may get out of sync, theconcept of references is important.They provide access to the source of theinformation, to the entity itself. Theyallow us to convey and store informationand let the service that encapsulatesthe entity deal with the concurrencyissues. References are less likely toget out of sync. Normally, that will onlyhappen if the entity ceases to exist. Toavoid this from happening, I wouldrecommend that the entity be marked asbeing obsolete rather than to delete it.

As I said above, the entity is made upof the combination of all of its state,its behavior and its relations to otherentities. Understanding the mechanismsthat cover the relationships betweenentities and their views, their behaviorand other entities should form one of

the fundamentals of any distributedarchitecture and thus of any SOA. Itwould be a huge step to capture thesemechanisms in ways that can be usedby the business people, the designers,and developers as well as by the systems.

These concepts are very similar towhat we know from object orientation.However, there are differences. Themost important one, I find, is the notionof distance between consumer andsupplier of the functionality. The datahas to travel from within the service tothe consumer. One of the consequencesof this is that, since the data will notbe locked on behalf of the consumer,the consumer has to be prepared forchanges to the service’s statei. Thisleads to the concept of views. Thisnotion of physical distance and theconsequential temporal distancebetween requesting information,providing information and using thatinformation leads to a whole new fieldrequiring explorationii. For instance,what is the behavior of the servicewhen it is receives a request that usesolder data? What are the guarantees theservice will give about that data? WhenI issue a price list with prices validthrough January 1, I guarantee thatI will honor those prices. When I requesta stock price, there is no guaranteewhatsoever. What are the possibleguarantees or service level agreementsand how can we describe these?

How Can What Be Used?Exposing all this information is extrawork for the designers and developersof the service. This extra work will onlypay off if these services are consumedusing this new description. We don’tonly need infrastructure to capture andexpose the functionality offered by theservice, we also need the infrastructureto consume that information and offerit to business analysts and users sothat they can do useful things. In thefollowing, I present only a fewexamples of infrastructure to providebetter service to users and thus to offerbetter human interaction.

Entity AggregationFormalizing the concepts of entities,views, and references provides a goodstarting point for a model for entityaggregation and for informationintegration in general and thus for aninformation architecture. By providinginfrastructure and tools for informationintegration, building on a description ofservices in terms a domain expertunderstands, a business analyst willhave more influence on the way thebusiness is organized.

Let me revisit entities. Earlier I wrotethat a service encapsulates entities. Ifwe look at the customer entity, manyorganizations will have multipleservices defining customer entities.There could be an ERP system with an

“Relationships are not constrained to a single service; they can define relationships betweenentities encapsulated in different services.”

i See for instance my article: “Dealingwith Concurrency: DesigningInteraction Between Services andTheir Agents” athttp://www.msdn.microsoft.com/library/en-us/dnbda/html/concurev4M.asp.

ii Pat Helland has an interesting articleon this subject “Data on the Outsidevs. Data on the Inside” athttp://www.msdn.microsoft.com/library/en-us/dnbda/html/dataoutsideinside.asp.


ERP customer entity, a CRM systemwith a CRM customer entity and manyother systems as well. Users mightwant to see an aggregated customerentity that has aggregated views aswell as the original ones and that hasaggregated actions as well as theoriginal ones and that has all therelationships from the original onesand possibly some new ones. We wantthe business analyst to define theseaggregations and reduce the requiredamount of support from the developers.

In entity aggregation, entities arecombined to form new entities. Theseentities may live in the same serviceor may come from different services.Entity aggregation, or informationintegration, has two main aspects:

– Aggregating information– Aggregating functionality

I will discuss these separately.

Aggregating informationAn aggregated entity is defined by theviews on that aggregated entity and byspecifying how these views are builtout of the underlying views. A businessanalyst can build new schemas usingthe existing ones, taking into accountconstraints such as whether the accessrights to these underlying views matchthe intended audience for theaggregated view.

A tool should be able to see how theinformation can be retrieved from thespecification of the composition. Forinstance the tool could decide thatwhen presented with a CRM customerreference, the CRM view should beretrieved first and then a relationshipfrom that view to an ERP customerreference could be used to obtain theERP information. It could define a

different path when presented withan ERP customer reference.

When the information needs to bereplicated, the tool could look at thecaching policies of the constituentviews and provide replication for thoseand offer aggregation based on thecached views. In other cases it may tryto minimize the number of constituentviews. For example, by retrieving anaggregate view of the ERP customer,retrieve and cache that and thenderive the smaller views from thecached aggregate.

Whichever the best algorithm for thespecific problem, a tool should be ableto offer working solutions using theinformation given by the description ofwhat the services have to offer. Themetadata should provide informationthat helps the business analyst defineuseful combinations and makeappropriate policy choices and it shouldprovide information for tools and run-time to support and execute both thedecision process and the outcome.

Aggregating functionalityActions on such an aggregated entitywill often have to touch the underlyingentities. In some cases this has to be an“all or nothing” action and in othercases it has to be made certain that allindividual steps are executed. Both areexamples of what I call short runningprocesses or activities. A tool couldconcatenate actions based on theirinputs and outputs. Moreover, given adescription of the underlying actions ormethods, including a description oftheir transactional behavior such aswhether they update state, throwexceptions or offer retries, a tool couldconcatenate the underlying actions in alogical sequence to minimize chance offailure and inconsistency and even

raise an event or send an email in casesomething does go wrong after all. Ofcourse there is no reason to restrict thisto entity aggregation, these types ofactivities happen all the time. This isgenerally applicable in the area ofservice orchestration.

Again, a description in userunderstandable terms allows a user oran analyst to combine and order a setof actions and build larger actions.A description of behaviors and sideeffects of actions allows a tool to makesure that such aggregated actionsbehave predictably and can becontrolled and managed.

Even with good tool support, defininggood schemas for views on andreferences to entities remains difficult.We should put more effort in providingguidance in this area. One of the majorchallenges in this area is designing forchange, where the different artifactshave a different rate of change. Dataobviously changes much faster thanschemas do and some types of datachange faster than others. Thedefinition of the data may change whilethe definition of the processes remainsunchanged. The reverse of this mayhappen too. Views, references, actions,code and process definitions will oftenchange at different rates and atdifferent times. A discussion on thisgoes beyond the scope of this document.Nonetheless, it must be clear that agood design of all of these artifactsmust include a concept of versioningeven in the first version.

CollaborationReferences play an important rolewhen crossing the boundary oforganizations where one wants to makesure that both parties collaborate onthe same subject. Often this is


complicated by the fact that not allinformation can or may be shared.For instance when one physician sendsinformation about a patient to anotherit is important to be able to send anexact reference to the other party, tounambiguously identify the patient,but both parties will have access totheir own private additionalinformation. The other party will havedifferent information available and willaccess different services to obtain thatinformation, but the reference may stillbe standardized for the domain and becommunicated. The key attributes andtherefore the data model do not haveto be the same on both sides. It sufficesto agree on an interchange format forthe references.

On the other side, the work ofcollaborating users that do haveaccess to the same services could becoordinated by those services if theusers communicate references insteadof chunks of information. Multipleusers can request changes from theservice and the service will resolve theconcurrency issues. The single truth isclear to all in the coordinating service.

Microsoft Office Information BridgeFramework (IBF) uses references indocuments to communicate informationand to allow the user to add businesscontext to a document. References canbe used to convey not only a limitedpiece of information, such as anexpense report, but also to provideaccess to a wealth of information, suchas the trip report, the customeropportunities, and other informationrelated to such an expense report. Ifthe recipient of the information has adescription of the references and ofrelationships used in the document,this user can then do more than just

consume the information compiledby the sender. Related relevantinformation will automatically beavailable and offered as well.

Many collaborations or workflowsdesigned today store the data with theflow. The flow then just runs by itselfbased on that data. It breaks as soon assomeone does anything in the backendsystem that influences the flow. Forexample, when designing a flow fordoing a stock trade, the progress of thetrade is stored in the workflow systemand the flow tries to just go ahead.If someone blocks that trade in thebackend system, the flow runs into anexception at some next step. In otherwords the designer should havesynchronized the state of the flowwith the state of the backend system.

References are also a way to avoidunneeded duplication or evenmultiplication of state – state that may,and often will, become out of sync.Formalizing references will reduce, notcompletely solve, the problem.

This is true for workflows and fordocument collaboration. There is nodifference between using references inworkflows, documents, e-mails or ininstant messenger. In all cases I havethe choice between sending the dataitself or a reference to that data andin all cases I have to manage changebetween many copies when I send thedata or I have a service manage it forme. If I send a document to ten peoplefor review; each person gets a copy andeveryone makes changes. If insteadI send everyone a reference to thedocument, each person can edit thedocument and the changes aresynchronized by the service. Using thereferences avoids making many copies

and having to manage those copies.This can now be delegated to a servicethat was already designed to deal withconcurrent updates and that servicedoes not care whether the referenceswere sent by email, instant messengeror as part of a workflow.

The main points here are that indesigning for collaboration andprocesses in general:

1. Avoid building a parallel datastructure, but use references insteadto make it easier to develop robustflows and collaborations

2. Eliminating different types ofpayload makes designing the systemeasier. References have behavior:the actions defined by the servicedescription of the entity. Theworkflow can use this behavior.

The concept of references helps reducesynchronization issues and this helpsreduce the need for conflict resolution.It also helps convey uniquelyidentifiable information without theneed to align the systems, which thencan be used to identify relatedinformation by following therelationships. Capturing these conceptsmakes them available to tools,infrastructure, and thus to humans.

User InteractionThe description of views and actionseffectively exports this functionality tothe consumer of the service. A consumercan use this to provide presentation forviews on entities in the user interface.The step to portals seems a naturalcontinuation. For instance a portal candefine parts to show views on entities. Apart showing the content of such a viewto the user can provide access to theactions, using buttons, or smart tags,


thus exposing these actions to endusers. The metadata can provide hintsto the user interface on how to exposethe actions and the run-time descriptioncan provide hints as to whether certainactions are available at this time for theinstance that is currently displayed.

If these parts use portal infrastructurebased in the proposed metadata, thenthat infrastructure would know whichview on which entity instance is beingshown. It could then offer the user achoice of relevant related instancesthat the user could navigate to. IBFwas the first instantiation of this ideaand offered a portal in the task pane.

Not only presentation, but also offlineavailability would be aided by adescription of the views and theirrelationships. Service agents manageoffline and cached information onbehalf of their consumers. I could see adesign where if such a service agent istriggered to take an instance offline,it uses the relationships to see whatfurther relevant information should betaken offline. It uses the relationshipsto find the relevant instances and toprune the offline tree by weighing thatrelevance. For instance, if the calendartakes an appointment offline for acustomer visit, by following thereference in the appointment thecustomer information can be takenoffline, and by following furtherrelationships, the problem reports canbe taken offline as well.

The concepts of portal parts and serviceagent infrastructure form the basis for asmart portal infrastructure. The additionof the extended service descriptionprovides user interface with the abilityto generically provide actions to theuser and enable or disable these actions

as appropriate. Moreover the descriptionof relationships between the viewsprovides navigation between the parts.The description also offers service agentsthe opportunity for more intelligentoffline behavior. All in all, the descriptionof services offers much to makea smarter portal.

What Should Happen Next?It is only a logical evolution that leadsus from how to access services to whatservices offer. Many organizations andvendors already describe thefunctionality of their services inmetadata. It is only a reasonable toexpect that the industry will standardizeon the way to describe these services. Itis equally reasonable to expect that itwill take time, a lot of time.

The Sea Urchins memo was about howto communicate. Razorbills is aboutwhat to communicate; until she can“raise her bill” the sea urchin will never“see her chin.” Just like the original SeaUrchins memo discussed the need forboth a method and tools for sharingdata, for functionality, we will needinfrastructure to describe and exposewhat functionality services have to offer.But such infrastructure alone doesn’tbring about change. So, we will needapplications and applicationinfrastructure to use these descriptions.One will not be available before theother, they will have to grow up togetherand in the mean time, we will want touse some of the concepts already. Wedon’t want to wait for this entireinfrastructure to be available, but we dowant to use it as it becomes available.

Raise Her BillStandardization on exposing whatservices have to offer requires anindustry effort in the following areas:

Extensible model to describe whatservices offerThis, like the current WS standards,should offer a fixed basis and beextensible in most areas. For instance,multiple standards describing cachingbehavior or the guarantees on datavalidity might evolve and each ofthese needs to be plugged in to theoverall description.

Extensible infrastructure for thatmodelIt is important to provide a model that,like WSDL, can be queried by bothtools and service consumers. However,it is equally important to not justprovide static information. The effortneeds to align with our thinking onservice contracts and should includedynamic definition and redefinition ofviews, actions, references, relationshipsand even entities. Such as, providingthe current availability of actions andrelationships on a particular instanceor provide information about aninstance of an ad-hoc workflow.Building on the extensible SOAPinfrastructure, this should become partof the .NET Framework.

Shared infrastructure for storinginter-service descriptionThe description of everything thattranscends a single service needs aplace to go. This includes relationshipsbetween entities in different servicesas well as the description of userinterface or hints for user interface.I am hesitant to call for a directoryinfrastructure on top of UDDI.Distributed directory infrastructuresare gaining popularity and seem tooffer good alternatives. Important inthis context is only that consumersof these services can store and query adescription of inter-service relationships.


After the Sea Urchins memo, it took usa few years to be convinced that XMLwas the way to communicate and we’restill struggling to define all of thestandards and required infrastructure.It will take time before the above startsbeing realized and the razorbill willhave proudly raised her bill. However,would this be enough for the sea urchinto see her chin?

See Her ChinFor the above infrastructure to bevisible and usable, the industry wouldhave to provide tools, applications, andhigher level infrastructure that allowsthe users, the business analysts, andeven developers to build on thatinfrastructure. I will suggest a few hereand others will be able to extend thisshort list with many other usefulexamples. Not all of the aboveinfrastructure needs to be in place.Some of these can start using moreproprietary solutions. When solutionslike these become available, everybody,including the sea urchin, will be able tosee the razorbill’s chin.

Build tools for Entity Aggregationand Service OrchestrationBring aggregation and orchestration tothe domain of business analysts whereit belongs. In Microsoft, BizTalk wouldform an ideal basis to build toolsfor this.

Provide User Interfaceinfrastructure to browse the businessBuild a Smarter Portal defining defaultpresentation for views, thus allowingnavigation from view to view, acrossentities and across services.

Provide a Framework forService AgentsSuch a framework would enable smartclients to deal with intermittentconnectivity and caching and queuing.Using the relationships, service agentscould not only take views offline,but could also provision relevantrelated information.

What Should We Do in theMean Time?Putting all this infrastructure in placewill be a multi-year effort. To preparefor this future, I recommend thosebuilding services:

Differentiate between entitiesand viewsDistributed computing and thereforeService Oriented Architecture andDesign introduce the necessity to thinkabout the distance between theservices and between the service andits consumer. Differentiate betweenthe entity information contained inthe service and the copies ofinformation defined by the views. Putthese views into schemas and assemblea collection of reusable schemas. Buildon existing industry specific schemaswhen available.

Define and describe the properties ofthese views. For example, manage accessto information by means of views andexamine how the service honors theinformation it handed out in acceptingrequests that use that information.

Formalize the concept of referencesWhen defining an information model,model references, build services to usethese references, abstract away the keyattributes, think about the definingattributes of the entities and schematizethis to allow other services andhumans to interact with these services.


Define your own metadataRegardless of where the metadatastandardization effort is going, you willbe able to reuse the effort put into theanalysis of service interaction. Makingthe parts of your software that consumeservices more generic, providing moreconsistent user interaction andproviding more relevant information tothe users will be easier if you have thismetadata in place.

ConclusionThe information architecture is themodel for interaction between servicesat the semantic level. It is the modelfor interaction about what. Much likethe programming model describes howto design services for programming theircommunication and coordination, theinformation architecture describes howto design services for human interactions.

Capturing and exposing what serviceshave to offer, in such a way that thefunctionality can be used is an importantand crucial part of the informationarchitecture. Doing this in businessterms will provide business analysts andusers access to more, and more relevant,information and functionality. Opening

up services to business analysts andusers allows them to create flexiblecontent and to create flexible processes,and it provides them with informationand functionality. This not only drivesdown the cost of business and the costof change, it also gives them the chanceto seize opportunities that were noteven visible before.

Addressing these challenges will helpour industry address another set ofbusiness requirements: Informationshould not just be available; peopleshould be able to find it, be able toidentify the relevant parts, act on it,convey it and interact with it.

The sea urchin not only lives, it startedswimming. We use XML as thecommunication mechanism betweenservices, and the industry as a whole isworking hard to describe even betterhow to communicate between services.

Now the razorbill should raise her billto show what services can offer so theycan be used by other software andhumans alike. If the industry takes onthis challenge, the razorbill will notonly swim, but fly as well.

Maarten MullenderMicrosoft [email protected]

Maarten Mullender is a solutionsarchitect for the .NET EnterpriseArchitecture Team; this team isresponsible for architectural guidanceto enterprise architects. During the lastseven years, Maarten has been workingat Microsoft with a select group ofenterprise partners and customerson projects that were exploring theboundaries of Microsoft’s productofferings. The main objectives of theseprojects were to solve a real customerneed and, in doing so, to generatefeedback to the product groups.Maarten has been working for morethan twenty years as a programmanager, as a product managerand as an architect on a wide rangeof products.

JOURNAL4 | Next Generation Tools for OOD 50

BackgroundThe file has been the central containerof work in software development forover thirty years. All of a project’sstructure and logic is ultimatelyreduced down to files and directories.The tools that surround softwaredevelopment are built around thisconcept, too; compilers, linkers, andeven language features rely on sourcecode files as input. Version controlsystems mirror the file system’sstructure, maintaining a copy of everyversion of each file monitored. Fordecades this has been acceptable andhas even become standard. Languagessuch as C are built upon filereferences in the source code viainclude directives.

The ProblemIn the mid-1990s, the idea of Object-Oriented development finally began togain momentum. Languages like C++,once formerly constrained to smallresearch projects, now became themainstream. With this fundamentalshift, the file-based system ofdevelopment was dragged along as arelic of old. C++, like its predecessor,also uses file includes to resolvedependencies. Despite new techniquesand languages for modelling anddesigning software, such as UML, inthe end source files still needed to becreated and linked to each other. Thislinkage leads to a duplication of effortas models must then be translated intofiles and source code. Complex buildscripts must also be written to ensurethe proper building of the application.Should a model change, the sourceneeds to be updated to reflect thechange; conversely, should the sourcecode change, the models need to beupdated. A number of differentsoftware solutions have been created tosmooth the synchronization of these

disparate files, but they’re stilldeveloped under the fundamentalnotion that the file is the central unitof output.

The file also proves to be a difficultchallenge when working in a teamenvironment. If multiple developerscheck a file out from source control,then the version control system mustensure that conflicting changes do notoccur. If they do occur, then manualintervention is required to resolve thedifferences. The source control systemsare limited since they rely on the file asthe unit to control. The source controlsystems are completely unaware of thelanguage structure contained withinthe file. It seems clear that the file is arelic that has outlived its usefulness forsoftware development.

Model Driven ArchitectureIn late 2001, a new methodology forsoftware development began to emerge:model driven architecture (MDA). Thefundamental principal behind MDA isthat the model, not the code, should bethe central focus. Platform independentmodels are created in UML and thenundergo various levels of transformationto eventually wind up as source code fora specific platform. It stands to reasonthen, that the model, not the file, shouldbecome the new unit of output. A modelhas many different views at differentlevels of abstraction. At the highestlevel, platform independent componentscan be specified in analysis; at thelowest level there is a platform specificimplementation that reduces to a set ofclasses in code. The available modellingnotation allows for the definition of allnecessary features needed to replacefiles; specifically, dependencies betweenclasses can easily be marked by meansof an association arrow with adependency stereotype.

Tools that support the creation ofmodels should ideally allow for versioncontrol on each individual modelelement. Each element should have theability to be checked in and out of arepository. Similarly, the modelling toolshould work with the repository toprovide branching and merging supportat the element level. For example,a developer should be able to connectto the repository and obtain a list ofelements that meet a given need. Thedeveloper should then be able to branchthe repository element for inclusion ina new model. At a later date, anotherdeveloper might wish to reconciledifferences between branched elementsin different models. The developmentenvironment should support this.One common example of when thisfunctionality is needed is for bug fixes.If a bug in a model element is fixed inone project, then other projects that areusing the same model element mightwish to incorporate those changes; thisshould be an easy task to accomplish.

The SolutionThe solution to these problems isnothing short of a complete paradigmshift. The development tools mustsupport the shift that languages havealready made and fully support MDAdevelopment. It is easy to argue thatthe source code is, in reality, justanother model. The argument isexemplified by the existence of alanguage-independent Code DocumentObject Model (CodeDOM). When youget down to the basics, the source codeand the UML classes are simplydifferent views of the same underlyingdata. Any separation between them isan artificial construct that needs to beeliminated. A developer should be ableto create classes as UML, completewith methods and attributes,and toggle between the UML and the

“Development tools must support the shift that languages have already made and fully supportMDA development.”

Next Generation Tools for Object-Oriented Development By Oren Novotny, Goldman, Sachs & Co


source code. The source code should beavailable in many different languages –simultaneously. This is largely possibletoday with CodeDOM, which isessentially an object-oriented meta-model that can be easily mapped toUML. The code structure is easilyportable between most .NETcompatible languages and the toolsshould support this.

One artifact that must be eliminatedis the source code file. It serves nofunction in MDA that cannot be betterdone by different means. In place ofsource files, an IDE would insteadconnect directly to a project repository.The project would then list the objectsdirectly, bypassing the need for files.This new functionality would worksimilarly to the “Class View” currentlyavailable in Visual Studio. The viewwould be organized by nestednamespaces rather than directoriesand by classes, structs, and resourcesinstead of files. When a developerdouble-clicks the class, the code isopened in the editor.

The Class View would also containUML diagrams. A diagram would beable to be created at any place in thehierarchy, and the developer shouldbe able to drag the classes onto thedesigner surface. Each namespacewould have a default diagram and anynumber of supplementary diagramsshowing different parts of the system.The tool would have a graphical view ofthe classes and their relationships inaddition to a code view.

Ideally, the code should not be stored inany single language; it should be storedas a model with additional CodeDOMstructures. Minor additions to theCodeDOM classes could enable therepresentation of all language features,

eliminating the need for language-specific code snippets. Each methodin the model can contain language-independent CodeDOM constructswhich can then be translated into anysupported language for easy editing.Just like there can be a split-view ofdesign and HTML view in FrontPage,a development environment shouldprovide a split-view of model and code.

When file-based development isreplaced with model-baseddevelopment, a new world ofpossibilities is opened. Rather thandesigning websites by creating webpages directly, Activity Diagrams canbe constructed to model the user’sinteraction with the site. Uponcompilation, the tools would generatethe necessary output files to supportthe diagrams. Each activity on thediagram that requires a user interfacewould contain a UI design. Forexample, for any given activity, a webUI design and Windows UI designcould be a part of the activity. The toolscould allow the import/export of HTMLfiles to allow designers the ability touse their own tools rather than thedefault IDE. After a designer is done,the HTML file would then be importedback into the activity. This wholeconcept falls squarely in line with thenew “Whitehorse” technologiesMicrosoft is including in VisualStudio 2005.

Visual Studio 2005 will include supportfor Microsoft’s Dynamic SystemsInitiative (DSI) by providing a tool ableto create a System Definition Model(SDM) linking components tohardware. Whitehorse includes a setof new designers in support of MDAcalled Software Factories [JG04];a UML-like class designer is availablefor source code and logical application

and infrastructure and designers allowdevelopers to declare the applicationcomponent structure, configuration anddeployment settings early on in thedesign process. Like code files are now,these models are currently stored asfiles within the solution. All modelsshould be stored within the repositorydirectly with APIs created to access themodels from the repository.

The source control system should alsobe cognizant of the inherent structurepresent in the model and code. Itshould support the check-in and check-out of model elements down to thefunction level. That way, twodevelopers could work on differentparts of a class without the potentialfor later reconciliation.

The build tools also need to supportthis new environment. Rather thancompile files, what they really need todo is compile a model. By way of adeployment model, classes would beassociated with assemblies andexecutables. The compilers would lookto the deployment model to determinethe physical separation of the project’sclasses and resources, and to the classdependencies to resolve symbols. Thebuild process would output manydifferent files depending on needs; for aweb project, the ASPX or ASMX pageswould be created alongside the binaryassemblies. The build process wouldalso read the SDM and generate theappropriate setup files and export theSDM for import into a deployment tool.

For compatibility purposes, the toolsshould allow for the extraction andimport of source code files in anysupported language. A developer couldselect a given set of classes to exportand the tool would generate therequired files. A developer could then


modify the files and then import themback into the project. The tool wouldautomatically create the modelelements and prompt for reconciliationchanges if needed.

ConclusionThe modern development environmenthas not yet fully caught up to theobject-oriented shift. All of the toolsstill rely on a file to be the containerof source code. While some containmodelling capabilities as well, themodels exist as different entities thanthe file and its code. The tools needto evolve to support modern softwaredevelopment; they need to merge themodel view and the code view into asingle entity.

While this may seem like a radicalshift, and it is, the new tools will benatural to developers. Developersalready expect their classes to beorganized by namespace; they do notreally care about which file or directorycontains what code. If a developerchanges a method signature, then hewill want, and expect, to find allreferences to the method to fix them.Architects already construct models ofsoftware and systems so they canunderstand them.

References[JG04] Jack Greenfield, The Case forSoftware Factories, JOURNAL3:Microsoft Architects Journal, Issue 3,July 2004.

Oren NovotnyGoldman, Sachs & [email protected]

Oren Novotny is a software developerwith Goldman, Sachs & Co with aninterest in all technical things.Previously he was a solutions architectwith Unisys Corporation. Oren is also afreelance consultant frequently helpingin various open source projects.

Microsoft is a registered trademark of Microsoft Corporation

JOURNAL4

Executive Editor & Program ManagerArvindra SehmiArchitect, Developer and PlatformEvangelism Group, Microsoft EMEAwww.thearchitectexchange.com/asehmi

Managing EditorGraeme MalcolmPrincipal Technologist, ContentMaster Ltd.

Editorial BoardChristopher Baldwin Architect, Developer and PlatformEvangelism Group, Microsoft EMEAFelipe CabreraArchitect, Advanced Web Services,Microsoft CorporationGianpaolo CarraroArchitect, Developer and PlatformEvangelism Group, WindowsEvangelism,Microsoft CorporationMark GliksonProgram Manager,Developer and Platform EvangelismGroup, Architecture Strategy, MicrosoftCorporationSimon Guest Program Manager, Developer andPlatform Evangelism Group,Architecture Strategy,Microsoft Corporationwww.simonguest.com

Neil Hutson Director of Windows Evangelism,Developer and Platform EvangelismGroup, Microsoft CorporationTerry LeeperDirector, Developer and PlatformEvangelism Group, Microsoft EMEAEugenio PaceProduct Manager, PAG, MicrosoftCorporationHarry Pierson Architect, Developer and PlatformEvangelism Group, ArchitectureStrategy,Microsoft Corporationdevhawk.comMichael PlattIT Professional Manager, Developerand Platform Evangelism Group,Microsoft Ltd.blogs.msdn.com/michael_plattBeat Schwegler Architect, Developer and PlatformEvangelism Group, Microsoft EMEAweblogs.asp.net/beatschPhilip TealePartner Strategy Consultant, MCS,Microsoft Ltd.

Project ManagementContent Master Ltd.

Design Directionventurethree, Londonwww.venturethree.comOrb Solutions, Londonwww.orb-solutions.co.uk

OrchestrationRichard HughesProgram Manager, Developer andPlatform Evangelism Group,Architecture Strategy,Microsoft Corporation

Foreword ContributorChris CaposselaVice President, Information WorkerBusiness Group, Microsoft Corporation

The information contained in this Microsoft® Architects Journal (‘Journal’) is for information purposes only. The material in the Journal does not constitute the opinion of Microsoft or Microsoft’s advice and you should not relyon any material in this Journal without seeking independent advice. Microsoft does not make any warranty or representation as to the accuracy or fitness for purpose of any material in this Journal and in no event does

Microsoft accept liability of any description, including liability for negligence (except for personal injury or death), for any damages or losses (including, without limitation, loss of business, revenue, profits, or consequential loss)whatsoever resulting from use of this Journal. The Journal may contain technical inaccuracies and typographical errors. The Journal may be updated from time to time and may at times be out of date. Microsoft accepts no

responsibility for keeping the information in this Journal up to date or liability for any failure to do so. This Journal contains material submitted and created by third parties. To the maximum extent permitted by applicable law,Microsoft excludes all liability for any illegality arising from or error, omission or inaccuracy in this Journal and Microsoft takes no responsibility for such third party material.

All copyright, trade marks and other intellectual property rights in the material contained in the Journal belong, or are licenced to, Microsoft Corporation. Copyright © 2003 All rights reserved. You may not copy, reproduce,transmit, store, adapt or modify the layout or content of this Journal without the prior written consent of Microsoft Corporation and the individual authors. Unless otherwise specified, the authors of the literary and artistic

works in this Journal have asserted their moral right pursuant to Section 77 of the Copyright Designs and Patents Act 1988 to be identified as the author of those works.

Part No: 098-101154