Development of theibm.com InteractiveSolution Marketplace(ISM): A SystemsEngineering Case StudyEirik Hole,1 Dinesh Verma,1, * Rashmi Jain,1 Vito Vitale,2 and Paul Popick2

1Systems Integration Laboratory, Stevens Institute of Technology, Castle Point on Hudson, Hoboken, NJ 07030

2IBM Global Services, 6710 Rockledge Drive, Bethesda, MD 20817

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE

Received 3 December 2003; Accepted 24 November 2004, after one or more revisionsPublished online XXX in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/sys.20022

ABSTRACT

IBM’s ibm.com business unit contracted with IBM Global Services (IGS), another business unitof IBM to develop their Interactive Solution Marketplace (ISM) application. ISM is a singlepoint of entry on the ibm.com website for browsing and searching for a suite of solutions asopposed to individual software and hardware items. The first release of ISM experiencedschedule and release content challenges. Development of the second release of ISM (ISM2.0) became one of the first projects where IBM Global Services implemented their formalSystems Engineering and Architecture (SE&A) process. During the SE&A process implemen-tation, particular emphasis was placed on requirements analysis and management, and onstructured and scored reviews. This technical paper presents this implementation and theresults in the form of a systems engineering case study. Implementation of the SE&A processduring the development of ISM 2.0 was successful. ISM 2.0 was delivered on schedule, at 5%below projected cost, and it constituted the functionality and features expected and desired

Regular Paper

Contract grant sponsors: IBM Global Services; Stevens Institute ofTechnology

*Author to whom all correspondence should be addressed (e-mail: dverma@stevens.edu

Systems Engineering, Vol. 8, No. 1, 2005© 2005 Wiley Periodicals, Inc.

1

by stakeholders. Consensus on core stakeholder requirements was achieved early duringproject development, and the technical reviews allowed the team to identify and resolve keyissues before proceeding into subsequent phases. This discipline, along with the expenditureof resources to develop quality requirements, greatly reduced project rework. As such, onlyone-tenth of anticipated defects were actually found and reported during testing. The qualityof the deployed application was further validated by the minimal number of post-launchdefects reported. © 2005 Wiley Periodicals, Inc. Syst Eng 8: 000–000, 2005

Key words: PLEASE PROVIDE

1. INTRODUCTION

1.1. Scope and Methodology

This case study investigates and illustrates the formalimplementation of the SE&A process and principles ona troubled medium-sized $3M Web-application devel-opment project. The case study demonstrates that sig-nificant benefits can be achieved through appropriateapplication of SE&A1 principles. These benefits in-clude the completion of a development project on time,under the projected budget, and with fewer defects atdelivery. The SE&A principles described in this casestudy are similar to those articulated in the US Depart-ment of Defense Acquisition Policy.2 As such, this casestudy is also a demonstration of the applicability ofDoD SE&A principles in improving system acquisitionand development.

Section 2 discusses some of the deficiencies of theexisting development methodology, prior to its integra-tion with the SE&A methodology, and the relevantchanges realized through this integration on the ISM 2.0project. Section 3 describes how a project team, inex-perienced with the practical application of formal sys-tem engineering practices, applied these practicessuccessfully on ISM 2.0. Significant findings are sum-marized in the final section of this paper, and someconcluding remarks are proposed.

The research presented in this paper is based on theFriedman and Sage [2004] framework for systems en-gineering case study research. Specifically, all aspectsproposed within this framework are addressed with theexception of “Life Cycle Support.” Since the develop-ment team on this project was getting introduced toformal systems engineering methods and practices, em-phasis of this case study research is on the “ConceptDomains” of “Requirements Definition and Manage-

ment” and “System and Program Management.” Fur-ther, in the responsibility domain, this research replacedthe concept of “Government Responsibility” with“Customer Responsibility” given the nature of the con-tractual agreement for this development [Friedman andSage, 2004].

Authors of research methods have discussed thestrengths and limitations of case study method of re-search [Graziano and Raulin, 2002; MacNealy, 1997;Friedman and Sage, 2004]. Case studies provide richdescription of events, behaviors, and situations whichhave not been studied and researched. However, theyare weak in generalizability, and are susceptible tothreats of reliability and validity. Friedman and Sage[2004] discuss the threat of researcher bias and high-light the importance of “sufficient validity.” In thisregard, the researchers had access to key project stake-holders and leaders, and while each stakeholder hadselected biases, there was remarkable agreement amongthem on the significant aspects of the case study. Thiswas also evident in the data collected from the develop-ment and customer team members. In addition, thisconcurrence was confirmed through the project-relatedtechnical documentation, and relevant project manage-ment information.

1.2. Background of the ISM Project

Two trends in the IT industry in late 1990s led to theinitiation of the Interactive Solution Marketplace (ISM)development project. The first was the exponentialgrowth of Internet usage and its increasing importanceas a marketing and sales channel [Odlyzko, 1999;Roberts, 2000; Teresko, 1999]. Second, IT customersincreasingly requested complete solutions to solve theirbusiness needs rather than discrete software, hardwareand service elements that had to be subsequently inte-grated [MacLachian, 1998].

IBM was an early adopter of the Internet as a com-munication channel, but their web presence was prod-uct oriented and did not provide an integrated solutionsapproach. As an example, potential customers had to

1A glossary for the acronyms used in this paper is included at the endof the article text.2DoD Directive 5000.1, The Defense Acquisition System, USD(AT&L), May 12, 2003; and DoD Instruction 5000.2, Operation ofthe Defense Acquisition System, USD (AT&L), May 12, 2003.

2 HOLE ET AL.

navigate the ibm.com site specifically and separatelyfor server technologies, application software, and IT-services. IBM and its business partners realized thelimitations and business implications of this productcentric approach.

The ibm.com business unit was chartered to initiatethe ISM development program to specifically addressthe business limitations of a product centric web pres-ence. The primary mission of ISM is to provide a singlepoint-of-entry for solutions-oriented browsing andsearching, allowing potential customers to search forintegrated services, technologies, and products to ad-dress their business needs. The goal was not to replacetraditional face-to-face sales strategies, but to enhancethem by providing initial product information in anintegrated fashion to potential customers. Further, ISMwould capture valuable customer and product leads andprovide the sales organization with relevant informa-tion for direct engagement.

According to internal IBM documentation, ISM wasto “become the vehicle to help customers identify busi-ness problems, involve decision-makers from the be-ginning of the purchase process, and line up thecapability to ensure IBM is actively engaged through-out the solutions sales purchase process. ISM will be astand-alone portal for demand generation activities. Itwill also serve as a publishing tool (common service)that supports display of solution content in the contextof a brand or audience site. ISM revolves around theconcept of selling ‘solutions’ to customers and as suchmust become a desirable and sought-out resource forLine-of-Business (LOB) decision-makers and IT buy-ers wanting to find, evaluate, and connect with solutionproviders. By providing a positive user experience,offering search and research tools and a robust reposi-tory of solution information from IBM and BusinessPartners, ISM can advance IBM’s objective.” [IBMInternal, 2002zaq;2]

Given the “scope” of the ISM application, and theneed to present integrated solutions to potential custom-ers, ISM impacted most business units within IBM. Tomake this “single point of contact” sales and marketingapproach work, business processes of different businessunits had to be realigned and diverse needs of thesebusiness units had to be satisfied. Accordingly, it was achallenge for the ISM team to obtain scope and intentconcurrence from numerous stakeholders on the oftenoverlapping and conflicting requirements. As one mem-ber of the SE&A team put it, “Everyone wanted every-thing.”

This was especially evident during the developmentof the first release of ISM. While there was a tangibleframework of processes and methods for project man-agement and software development, formal processes

for stakeholder requirements and system requirementswere largely absent. Application requirements keptchanging, and the deployment at the end of 2001seemed in jeopardy. As a result significant compro-mises had to be made with the functionality and scopeof the first release of ISM (ISM 1.0) in order to meetthe deployment schedule. Much of the capability origi-nally envisioned for ISM 1.0 had to be implemented inthe second release of ISM (ISM 2.0). At the same time,the business imperative to present IBM as an integratedsolutions provider over the Internet resulted in in-creased visibility and senior management attention forthe project. Changes were also made by the ibm.combusiness unit to their project management team for ISM2.0 development project.

IBM Global Services (IGS) was contracted byibm.com to develop and deliver the technical solutionfor ISM. The IGS project manager saw an opportunityto deploy the recently developed and formalized Sys-tems Engineering and Architecture (SE&A) methodol-ogy. This methodology was soon to become mandatoryon all IGS projects with a development budget greaterthan $500,000 as part of their CMMI®3 [2001] initia-tives. The ibm.com program manager was receptive tothe SE&A approach, and ISM 2.0 became one of thefirst projects to embrace the complete SE&A method-ology.

2. IGS’ SE&A APPROACH

IBM Global Services has a framework of processes andmethods governing any internal or external customerengagement. This framework outlines necessary activi-ties, roles, and life-cycle models for typical businessengagement scenarios and domains. Further, over 280standard work products are defined within this frame-work to provide a common reference for project plan-ning and documentation. In its existing state(pre-SE&A) this framework did not address require-ments, architecture development, integration, and veri-fication as part of a coherent Systems Engineeringmethodology. Existing descriptions of these SE&Apractices were general and open to interpretations, andoften “hidden” in broader activities and work products.

Formal descriptions of SE&A activities, practices,roles, and work products were added to the IGS meth-ods framework in mid-2001. These SE&A artifacts areconsistent with the methods discussed in the SE litera-ture [Kossiakoff, 2002; Sage, 1995; Sage and Arm-strong, 2000; Verma and Fabrycky, 1997; INCOSE,

3Capability Maturity Model® Integration from the Software Engi-neering Institute at Carnegie Mellon, IBM Global Services has re-cently attained Level 5.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 3

2004]. After the formal definition and integration of SEartifacts into the IGS methods framework, an SE edu-cation and training initiative was launched and con-ducted within IGS from mid-2001 on a continuous basisto provide practitioners with the necessary support tofacilitate application of SE&A practices to IT develop-ment and integration projects. To provide a necessarycontext for this case study research, a brief discussionof the SE&A related additions to the IGS methodsframework is included herein.

Figure 1 shows a mapping of the IGS SE&A modelto the existing models in the IGS methods framework.The existing models were largely focused on programand project management and software development.The SE&A model provided a coherent, multilevel (sys-tem, subsystem, application, and subapplication) andrecursive emphasis on requirements, architecture devel-opment, design, verification and validation, and inte-gration. Selected and relevant aspects of the SE&Alife-cycle model are discussed in the following subsec-tions.

2.1. Systems Engineering Reviews andTechnical Baselines

As shown in Figure 1, a gating concept was present inthe existing IGS methods framework in the form ofDecision Check Points (DCPs). DCPs are not technicalreviews, but rather executive assessments of projectstatus and its readiness to progress to subsequent phasesof the framework. As an example, the “Plan DCP” iscritical since this is the milestone where final commit-ment is made to project scope, budget, and schedule.Technical reviews, their scope, and rigor were left to thediscretion of the project manager in the existing IGSmethods framework. SE&A introduced a set of techni-cal reviews to the IGS methods framework as shown inFigure 2. These technical reviews are keyed to projectbaselines. For example, the Customer Baseline is fixedat the Business Requirements Review (BRR).

These reviews had a dual purpose. They provided amore detailed assessment of a project’s technical matu-ration to support the executive decisions made at DCPs,while providing the formality of a systems approach tothe development of key SE artifacts. The latter is im-portant when transitioning to a systems approach sincea correlating set of SE artifacts provide a pragmaticbasis for the systems approach. Technical baselinesprovide a reference for the expected maturity of aproject at different development milestones. They arealso the basis for change control and management.

To promote consistency in the SE&A technical re-views, a scoring scheme was introduced. For each tech-nical review a set of entry and exit criteria was defined.Each of these criteria was weighted on a scale of 1 to 4,and scored by the reviewers. An example is shown inFigure 3. A “red”, “yellow”, or “green” score is used toassess the quality of the work products reviewed. Issuescontributing to a “red” score (below 70) must be re-solved before a phase can be exited. Issues contributingto a “yellow” score (between 70 and 85) must be re-solved by the next SE&A technical review.

2.2. Problem vs. Solution Domains: Impacton Requirements QualityWhile the importance of “good” requirements is undis-puted in the systems (SE) and software (SW) engineer-ing communities [Sage and Armstrong, 2000;Blanchard and Fabrycky, 1998; Standish Group 1994–2001], there is a divergence on characteristics of “good”requirements. Furthermore, there is general acceptancewithin the SE community of the value of clearly sepa-rating the problem domain (WHAT) from the solutiondomain (HOW) [Sage and Armstrong, 2000; Blanchardand Fabrycky, 1998; Alexander, 2002; Alexander andStevens, 2002; Hooks and Farry, 2001]. This is empha-sized in the IGS SE&A model.

The IGS SE&A model defines three categories ofrequirements: stakeholder requirements (or businessrequirements, or business process requirements, or cus-

Figure 1. SE&A life cycle model mapped into the existing IGS life cycle models.

4 HOLE ET AL.

tomer requirements), system requirements (or designrequirements), and component requirements. Compo-nent requirements can exist at multiple levels as afunction of system complexity. Templates are providedin the IGS SE&A framework to support the develop-ment of “good” stakeholder, system, and componentrequirements.

2.3. Traceability of Requirements

The existing IGS methods framework lacked a consis-tent emphasis on requirements traceability. This wasaddressed by the SE&A methodology. Requirementstraceability and verification matrices were added aswork products to the existing IGS methods framework

to ensure traceability from business requirements tosystem requirements, to component requirements, andfinally to test and verification cases. This provided thedevelopment team a method to ensure consistencythrough the different phases of development, test, andintegration.

2.4. SE&A Roles and Responsibilities in aProject Organization

While a project manager was and is ultimately respon-sible for project success, the SE&A enhanced IGSmethods framework facilitates the allocation of rolesand responsibilities between project management andsystems engineering.

Figure 2. Overview of technical reviews and baselines.

Figure 3. A sample scorecard for the review of system requirements at the SRR. [Color figure can be viewed in the online issue,which is available at www. interscience.wiley.com.]

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 5

IGS SE&A methodology requires that there be aLead Systems Engineer (LSE) on a project, optionallyassisted by a Lead Systems Architect and other systemsengineers depending on project size and complexity.The LSE has overall responsibility for the technicalsolution and serves as the project manager’s technicalfocal point. The LSE assumes responsibility for inter-disciplinary collaboration, interfacing with the projectstakeholders, the development team, and the test andverification team during project development, test, in-tegration, and deployment.

3. SE&A DEPLOYMENT ON ISM 2.0

3.1. ISM 2.0 Project Context andCharacteristics

ISM 2.0 was the second release of the ISM applicationfor the ibm.com website. Given the limited scope andfunctionality of ISM 1.0, ISM 2.0 included the primaryfunctionality needed for solutions based browsing. Thecritical nature of the second release attracted significantexecutive management oversight and attention. TheISM 2.0 concept phase was launched in late January2002 with a development budget of $3 million. Theplanned release date was mid-December 2002.

While the development and integration of ISM 2.0was executed internally at IBM, a formal customer–contractor relationship existed between the two IBMorganizations. ibm.com was the internal customer or-ganization and IBM Global Services was the internalcontractor. A Project Development Team Lead (PDTL)within ibm.com was assigned to ISM. He had overallresponsibility for the definition and planning of featuresand functionality across the multiple ISM releases. TheISM PDTL was supported by two project managers.One managed the maintenance and support for ISM 1.0while the second project manager managed the ISM 2.0acquisition and deployment. A Requirements Managertogether with a team from Business Analysis and Mar-keting rounded off the ISM team within ibm.com. Ad-ditional stakeholders and subject matter experts fromother IBM business units were engaged on an as-neededbasis.

The IBM Global Services team constituted a projectmanager, supported by three project teams. These threeteams were the software development team, the testteam, and (for the first time) the SE&A team led by aLead Systems Engineer with overall responsibility forthe technical solution of ISM 2.0.

It is worth noting that the ibm.com and IBM GlobalServices teams were geographically spread over fourcontinents. This is the norm rather than the exceptionwithin IBM, and a companywide infrastructure of e-

collaboration tools and virtual “Team Rooms” accom-modate the necessary communication and collaborationbetween the project team members. Apart from a singleISM 2.0 workshop that caused the two (customer andcontractor) teams to meet for live discussions, the pro-ject team members did not meet in person. The entireproject, including the technical reviews, was conductedand completed using IBM’s virtual project environ-ment. Only the software development team within IBMGlobal Services was physically collocated.

3.2. Controlling the ISM 2.0 Scope

A Requirements Manager joined the ibm.com teamsoon after the ISM 2.0 project start. Her initial task wasto consolidate and document the “unrealized” require-ments from ISM 1.0. In the absence of formal require-ments management and traceability, this was a tedioustask. In addition to formal documents such as the origi-nal Request for Information, Business and FunctionalRequirements document, she had to glean requirementsfrom informal and ad-hoc sources such as e-mails andmeeting minutes. Once consolidated, these unrealizedrequirements numbered approximately 100.

ISM 2.0 was approaching the end of concept phaseby the time the development team (IBM Global Serv-ices—IGS) had the charter and resources to formallydeploy the SE&A methodology. The Lead SystemsEngineer (LSE) joined the project 10 days before thescheduled System Requirements Review (SRR). Con-solidation efforts by the Requirements Engineer on thecustomer team contributed to the successful conduct ofthis review. A number of pre-SSR sessions were organ-ized by the LSE to generate “review ready” stakeholderand system requirements compliant with SE&A stand-ards and practices. These sessions involved the SE&Ateam along with all relevant stakeholders and subjectmatter experts from ibm.com and other IBM businessunits. The software development and test teams werealso represented on a limited basis. The focus of thesepre-SRR sessions was on the following aspects of therequirements:

Clear distinction between stakeholder and systemrequirements: The consolidated list of originalrequirements was reviewed and reformulated toexpress stakeholder needs and expectations (Thisrepresented the business or problem domain). Anumber of stakeholder requirements were reclas-sified as system or component requirements,since they represented the solution or implemen-tation domain. This was the first time team mem-bers on the ISM 2.0 project had experienced such

6 HOLE ET AL.

rigor in separating the problem and solution do-mains.

System requirements were developed in the contextof the ISM 2.0 architecture. System requirements weredeveloped to a level of detail and completeness neces-sary to ensure that the stakeholder requirements andexpectations were adequately addressed and that thearchitecture was feasible for the ibm.com environment.

Understandable and unambiguous requirements:Considerable time was invested during the pre-SRR sessions on the understandability and clar-ity of requirements. The LSE insisted that everyrelevant stakeholder and development teammember understand and agree with the meaningand implication of each requirement (reflectedthrough the definition of acceptance criteria).This became their pragmatic definition of a“good” requirement, avoiding the more theoreti-cal discussions in the literature.

Acceptance criteria: Each critical system require-ment was linked to a quantifiable set of accep-tance criteria. This was agreed to at the SRR. Thispractice enhanced the understanding and clarityof system requirements while also facilitating the

definition of test cases during the ISM 2.0 devel-opment. Further, this practice increased the levelof commitment and trust between the customerand contractor teams, thereby reducing the prob-ability and risk of dispute during system delivery.

Stakeholder ownership: Every stakeholder require-ment was “owned” by a specific stakeholder. This“owner” stakeholder was involved in any deci-sions or trade studies that impacted this require-ment during the project.

Requirements-database with discrete and uniquelyidentified requirements: The ISM 2.0 projectused the IBM “Team Room” infrastructure. Thisserved as a project data repository while alsoproviding a suite of online collaboration tools.This environment was enhanced with basic re-quirements management capabilities such as at-tribution, sorting, filtering, and reporting toovercome the limitations of document based re-quirements management and to facilitate easyaccess for team members.

Figure 4 shows an example ISM 2.0 stakeholderrequirement with corresponding system requirements.

The ISM 2.0 SRR was conducted as a three-dayonline collaboration session. This event passed with a

Figure 4. Sample of a stakeholder requirement with derived system requirements.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 7

“green” score per the scoring scheme discussed earlier.Leading up to the SRR, the approximately 100 stake-holder requirements from the original list were reducedto 46. Eighty-nine corresponding system requirementswere formulated. Rather than signifying scope reduc-tion, the reduction in stakeholder requirements re-flected the team effort to reduce redundancy, overlap,while providing the necessary development focus andconsistency on the ISM 2.0 project.

Team discussions to finalize stakeholder and systemrequirements were often difficult to resolve. Converg-ing on the functional scope of ISM 2.0 was a challeng-ing task. After SRR the SE&A team felt they had a setof “good” requirements. They were not, however, con-fident that they had the “right” requirements for thescope of this release. The concept of project or missionobjectives has been discussed in the systems engineer-ing literature [Hooks and Farry, 2001; Sage and Arm-strong, 2000; Sage, 1995]. This practice was used tofacilitate convergence on functional scope. The teamdefined five nonnegotiable project objectives. Theseobjectives were called Critical Mission Requirementsand reflected the core functionality and features of ISM2.0 desired by the stakeholders. One of these five Criti-cal Mission Requirements was the release date. Thisreflected the criticality of getting the core functionalityreleased on time.

3.3. High Level Design and DetailedDevelopment

As shown in Figure 1, a completed SRR marked the endof the ISM 2.0 Concept Phase. The project then enteredand exited the Plan Phase with definitive commitmentsto scope, cost, and schedule. The project focus was nowon finalizing the ISM 2.0 architecture, its high-leveldesign and component requirements, and preparing forthe Preliminary Design Review (PDR). Schedule andcost estimates were refined and final commitmentswere made with acceptable risk.

As the ISM 2.0 high-level design evolved and thedetailed development schedule and budget was devel-oped, it became obvious that all stakeholder require-ments agreed to during SRR could not be implementedin the designated timeframe. The project team had to goback to the stakeholders and prioritize each stakeholderrequirement. Additionally, they developed an estimateof the implementation effort necessary for each stake-holder requirement. To assess priority, each stakeholderrequirement was examined in terms of its contributionto the Critical Mission Requirements. This proved criti-cal to defining and scoping the capabilities of the ISM2.0 release. A member from the customer team recalled:“It killed discussions about this whistle or that bell and

focused us on the core functionality to be imple-mented.”zaq;2 At the end of these discussions, thenumber of stakeholder requirements for ISM 2.0 wasreduced from 46 to 35. This was not viewed by thecustomer as a functionality reduction exercise, butrather as an exercise to identify the necessary andfeasible implementation scope for the ISM 2.0 releaseversus what could be left for subsequent releases. Thefive critical mission requirements provided a common-ness of purpose and helped converge the discussionsbetween the customer and contractor teams.

Change management was given particular attentionduring the Plan Phase. Good traceability of require-ments from stakeholders to the architecture allowed forincreased confidence during the (many) impact assess-ments and trade studies. The project team could quicklygain visibility into the components impacted by a pro-posed stakeholder requirement change and provide acredible schedule and cost impact assessment. Addi-tionally, impact of technical issues could be assessedand resolved in collaboration with the owning stake-holder(s). All change requests were justified and re-viewed vis-à-vis the Critical Mission Requirements.

The level of customer involvement on ISM 2.0 wasunprecedented according to the development and cus-tomer team members interviewed. One team memberfrom ibm.com stated: “We used to give our require-ments to the software developers and they would disap-pear behind ‘a curtain,’ and we did not see them againbefore they returned with the code that might or mightnot have done what we needed.”zaq;2 An enhancedlevel of customer involvement assured that customerexpectations were aligned with the technical solutionbeing developed.

The charter of the SE&A team made them responsi-ble for the system and component architectures. Tradi-tionally this responsibility was allocated to the softwaredevelopment team. The ISM 2.0 approach constituteda significant change from their perspective. Further,resource constraints prevented the desired level of in-volvement from members of the software developmentteam to support the SE&A activities during the planphase. This impacted quality during the formulation ofthe high-level design and became critical to the overallproject success.

The software development team was critical to thesuccess of the ISM 2.0 Preliminary Design Review(PDR) conducted in mid-July 2002. Several issues anddeficiencies with the high level design were identifiedduring the PDR. These had to be resolved before theproject could enter the component development phase.The SE&A team and the software development team,along with representatives from a third party vendor,were involved in a week-long workshop to resolve

8 HOLE ET AL.

critical technical issues. Given the traceability betweenthe stakeholder, system, and component requirementsin the context of the ISM 2.0 architecture, the workshopwas structured into focused breakout sessions to ad-dress and resolve specific technical issues and deficien-cies.

The resulting high-level design had the necessaryquality to enable the development team to complete thedetailed design and conduct the Critical Design Review(CDR) in 3 weeks. Thereafter, the coding and compo-nent level testing was completed in 5 weeks to keep theproject on schedule. Well-defined components andcomponent interfaces made it possible to add resourcesto the development team without significant coordina-tion challenges.

The CDR resulted in some contention between theSE&A and software development teams. While thesoftware development team was used to internal peerreviews, they were skeptical of the value of using pre-cious development time for conducting a high levelreview with the stakeholders. The program managerand the LSE insisted on this review since the CDR wasa formally required review by the SE&A methodology.Further, the stakeholders’ involvement on the ISM 2.0project thus far had been very strong and constructive.They did not want to lose the final opportunity toidentify any remaining issues before full scale codingwas initialized, while also reassuring stakeholders thattheir expectations would be met.

3.4. Verification and Delivery

Preparations for system level integration, and func-tional, usability, and performance testing started longbefore the Test Readiness Review. The test organizationwas involved at the SRR when the acceptance criteria

were defined and agreed to with the stakeholders. TheSE&A team also involved the test organization duringthe Plan Phase (Figure 1) to define the test architecture.The test team collaborated with the SE&A team todefine the test data requirements, and the requirementsfor monitoring performance and volume during theperformance tests. After PDR, once the architecture andhigh-level design stabilized, the SE&A team engagedthe test team to define and review test cases. Misunder-standings were identified and resolved and the test teamgained a deeper understanding of the ISM 2.0 architec-ture and functionality. The result, the test lead com-mented, was “flawless test cases.”zaq;2 In hisexperience, misunderstood requirements and poorlydefined test cases could cause up to 10–20% of reporteddefects during system testing.

The test team had expected and planned for a defectrate of approximately 20%. This was based on experi-ence with other development projects of similar sizeand complexity. The functional verification test, withapproximately 1350 test cases, actually reported onlyone tenth of the estimated number of defects. This isshown in Figure 5. System level testing began after 5weeks, 2 weeks ahead of the planned schedule. Theusability test team from ibm.com had a similar experi-ence. Their lead would comment that usability testingwas a matter of hours rather than the usual practice ofdays or weeks.

Performance testing in the target environment en-countered some issues. This caused unplanned trou-bleshooting before a configuration discrepancy in thetest and target environments was identified and re-solved. This activity was not on the project critical pathand did not cause any schedule delays.

ISM 2.0 went live in December 2002 per the sched-ule. At completion, the ISM 2.0 project had spent $2.85

Figure 5. Anticipated vs. actual defects found during the functional verification test.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 9

million out of the $3 million development and integra-tion budget.

In the first 3 months, following the deployment ofISM 2.0, the operations unit at ibm.com reported anaverage of 12 defects per month. Only two of these wereof a severity that required an immediate fix. Accordingto ibm.com, this was a remarkably low defect rate for aWeb application of this complexity.

3.5. Project Management Aspects

It is interesting to highlight the reflections of the cus-tomer and contractor project managers with regard tousing the SE&A methodology. The contractor projectmanager (PM) from IBM Global Services highlightedthe following aspects:

Ability to focus on project management and cus-tomer relationship at the business level: Used toworking in a constant “firefighting mode,” en-gaged in technical scope discussions, and as me-diator between software development andcustomer, the PM could instead focus on priori-tizing conflicting tasks, managing resources andengaging with the customer team on business andmanagement issues. On the ISM 2.0 project, thePM delegated “technical mediation” to the LSEand his team.

Ability to assess technical status and compliancethroughout the project: The quality of the tech-

nical solution and its compliance with stake-holder expectations is often not revealed untilverification or deployment. Any critical devia-tions discovered this late often lead to projectdelays and expensive rework. With the conceptof scored reviews and stakeholder involvement,the SE&A team could quantify the quality andmaturity of work products early in the project,along with ensuring compliance between thetechnical solution and stakeholder expectations.The PM especially identified the PDR as a valu-able milestone in that the project would not haverecovered in time if they had not been able tounderstand and isolate the gap between the ISM2.0 requirements and the ISM 2.0 architecture.Good traceability between requirements and thearchitecture enabled risk and impact assessmentswith fidelity and confidence, and provided a de-sired level of confidence in the PM’s communi-cations with the customer.

Management of third party subcontractor: Well-documented requirements and a clean compo-nent s tructure were invaluable in thedevelopment of subcontract work packages. Thissignificantly lowered the potential for conflictwith the third party supplier. Project rework wasalso significantly minimized because of this andthe project required four man-weeks less workthan originally planned.

Figure 6. ISM 2.0 defects detected by phase vs. average of 19 similar IT projects.

10 HOLE ET AL.

The PM on the customer team (ibm.com) also madesimilar comments. He highlighted the project docu-mentation and the structured interaction between stake-holders and developers as key contributors to thesuccess of ISM 2.0. He also confirmed the credibilityof the ISM 2.0 status reports, and the associated risk andimpact assessments.

The Project Development Team Lead at ibm.comwas convinced early of the utility of the SE&A meth-odology. In specific, enhanced risk assessment andmanagement was cited as a key advantage of the SE&Amethodology. In his opinion, SE&A reduced overallrisk, while also enabling better understanding of risksand their mitigation. “For once,” he said, “I felt incontrol of what was being delivered.”zaq;2

3.6. Deployment of the SE&A Methodologywithin IBM Global Services

IBM Global Systems is currently sponsoring researchto quantify the impact of formal SE&A methodologyon the effectiveness and efficiency of IT developmentand integration projects [Barker and Verma, 2003]. Oneaspect of this study involves investigating the distribu-tion of identified requirements and design defects overproject phases. Figure 6 reflects preliminary resultsfrom this research where the ISM 2.0 project is com-pared with an average across 19 similar IT projects notusing the formal SE&A methodology. A more detailedpublication on this subject is forthcoming. Profile of thecurves suggests a “cause and effect” relationship be-tween early attention to requirements and architecture,and defect prevention in earlier versus later projectphases when the correction cost is significantly higher[Boehm, 1981; Clark and Fujimoto, 1991; Fricke et al.,2000].

4. SUMMARY AND LESSONS LEARNED

Application of the SE&A methodology on ISM 2.0 wassuccessful both from the customer and contractor per-spectives. They acknowledged certain core features ofthis methodology that contributed to its successful im-plementation. These are listed below:

Definition of critical mission requirements: Defin-ing the critical mission requirements was instru-mental in providing focus along with facilitatingthe priorities and trade studies during the project.It also helped in clarifying the project scope.

Making schedule a critical mission requirement:Both ibm.com and IGS point to this as beingcritical to limit scope creep. Supportive of thisapproach, Boehm et al. [2004] report on a num-

ber of successful IT development projects that areon schedule, on budget, and meeting customerexpectations using Schedule as an IndependentVariable (SAIV).

Intimate stakeholder involvement throughout: Closeinvolvement of the stakeholders helped developtrust between customer and contractor teams.The problem and solution domains were alignedwith each other throughout the project, and theinevitable trade studies and change managementwere conducted relatively smoothly.

Traceability: Rigorous management of traceabilityprovided the contractor project manager with an“architectural understanding” of risk and impactof proposed changes. Both the customer andcontractor acknowledged the increased credibil-ity of the communication as a result of goodtraceability management. The test organizationwas a key beneficiary of this practice, and conse-quently the quality of test cases was significantlyenhanced.

Structured baseline change control: After an agreedto baseline, every change request was evaluatedfor its impact and its relevance to critical missionrequirements. This promoted structured changecontrol and discipline. ibm.com was cognizantthat every change had consequences on projectschedule and budget, and IGS recognized thatevery technical change with system level impactshad to be resolved with the owning stakeholder.

Scored reviews: Numerical and graded scoring ofreviews brought consistency to the review proc-ess along with quantifying the maturity and qual-ity of work products. This enabled better riskassessment and mitigation.

Important lessons were learned during the ISM 2.0project. These are listed below:

Involve key software development personnel in theearly SE&A activities: This is a critical lessonlearned. While the SE&A team included experi-enced software architects, they did not have thenecessary time or resources to develop a highlevel software design of sufficient quality. Theconsequences of not prioritizing the necessaryresources from software development to supportthe SE&A team in the architecting and high-leveldesign activities were underestimated. By thetime the software team got involved (at PDR)they a steep learning curve. They had to getfamiliar with the evolving ISM 2.0 architectureand requirements along with adapting to theSE&A methodology.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 11

Scheduling of technical reviews relative to projectdecision check points: Based upon the ISM 2.0experience, particularly the PDR, additional timewas introduced in the framework between thereviews and the decision check points. This pro-vided time for critical issue identification, assess-ment, and resolution before a DCP.

Allow sufficient time for requirements analysis be-fore SRR: This can go a long way in reducing“flux” and the possibility of a scope creep overthe life of the project.

5. CONCLUSION

The implementation of SE&A methodology on the ISM2.0 project demonstrated that a motivated customer andcontractor team could very rapidly adopt and applysystems engineering practices on a time critical andcomplex project. The advantages of the SE&A ap-proach from the perspectives of the involved partiescould briefly be summarized as follows:

The stakeholder perspective: The stakeholders feltthey had an unprecedented transparency in howtheir requirements would be implemented. TheSE&A approach also helped them to focus andto prioritize the scope according to the overallobjective for this particular release. The projectmanagement on the customer side expressedtheir confidence with regard to having good con-trol over all project risks. The high quality ofproject documentation augmented their confi-dence in the IGS team. The final product hadfewer defects during initial operation than ex-pected for a new application of this size andcomplexity

The development team perspective: SE&A provideda structured way to interact with the customerregarding the technical scope and project pro-gress. For the project manager this meant thatmore attention could be devoted to managing theproject team and interacting with the customer ata business level. The SE&A team especiallyhighlighted the value of a disciplined changecontrol and technical reviews. The focus pro-vided by a rigorous requirements baseline andtraceability between requirements and high-levelsoftware design facilitated scope control on theproject. The requirements driven high-level de-sign provided the software developers with abetter contextual understanding. Good require-ments and early involvement allowed the testteam to define test cases based on a good under-

standing of the requirements and the applicationarchitecture.

It would, however, be incorrect to conclude that theapplication of the SE&A methodology was the solereason for the success of the ISM 2.0 project. Everyoneinterviewed explicitly acknowledged leadership andmanagement support, and the highly motivated cus-tomer and contractor teams as being critical to projectsuccess. The authors believe that systems and projectmanagement, and leadership are mutually supportive.It is difficult to introduce a new way of project executionwithout leadership, motivation and management sup-port. On the other hand, good project leadership andmanagement depend upon a measurable referenceframework and structured baseline control. The SE&Amethodology provided this framework and the associ-ated baselines. As appropriately pointed out by a teammember, “the requirements and the architecture mighthave been the glue of the project, but it was the teamthat made it stick.”zaq;2

ACRONYMS

BRR Business Requirements ReviewCDR Critical Design ReviewCMMI® Capability Maturity Model® IntegrationDCP Decision Check PointIGS IBM Global ServicesISM Interactive Solution MarketplaceLSE Lead Systems EngineerPDR Preliminary Design ReviewPDTL Project Development Team LeadPRR Production Readiness Review (Production in

terms of an IT system like ISM means performing itsintended task in its operational environment)

RSD Requirement Specification DocumentSAIV Schedule As an Independent VariableSE Systems Engineering / Systems EngineerSE&A Systems Engineering and ArchitectingSME Subject Matter ExpertSRR System Requirements ReviewSW SoftwareTRR Test Readiness Review

ACKNOWLEDGMENTS

This Case Study has been developed as a collaborativeactivity between IBM Global Services (IGS) and theSystems Integration Laboratory at Stevens Institute ofTechnology. IGS provided the primary funding for thiseffort. The authors are grateful for the valuable inputfrom Jan Van Hoomissen, Keith Instone, Avinash Ko-

12 HOLE ET AL.

hirkar and Jim De Piante from ibm.com, and fromPierre Carlson, Jeannette Decker, Ted Hamilton andKelvin Wong from IBM Global Services.

REFERENCES

I.F. Alexander, Being clear: Requirements are EITHER needsOR specifications, Requirements Quart, RequirementsEng Specialist Group Br Comput Soc 25 (January 2002),10–12.

I.F. Alexander and R. Stevens, Writing better requirements,Addison-Wesley, London, 2002.

B.G. Barker and D. Verma, System engineering effectiveness:A complexity point paradigm for software intensive sys-tems in the information technology sector, Eng Manage-ment J 15(3) (2003), 29–38.

B.S. Blanchard and W.J. Fabrycky, Systems engineering andanalysis, 3rd edition, Prentice Hall, Englewood Cliffs, NJ,1998.

B. Boehm, Software engineering economics, Prentice Hall,Englewood Cliffs, NJ, 1981.

B. Boehm, W. Brown, L. Huang, and D.N. Port, Schedule asindependent variable process for acquisition of software-intensive systems, INCOSE 2004, 14th Int Symp Proc,Toulouse, 2004.zaq;3

K.B. Clark and T. Fujimoto, Product development perform-ance strategy, organization and management in the worldauto industries, Harvard Business School, Boston, MA,1991.

Capability Maturity Model® Integration (CMMISM), Ver-sion 1.1, CMMI-SE/SW/IPPD, V1.1 Staged Repre-sentation, Software Engineering Institute, Pittsburgh,2001, http://www.sei.cmu.edu/cmmi/.

E. Fricke, B. Gebhard, H. Negele, and E. Igenbergs, Copingwith changes: Causes, findings and strategies, Syst Eng3(4) (2000), 169–179.

G. Friedman and A.P. Sage, Case studies of systems engineer-ing and management in systems acquisition, Syst Eng 7(1)(2004), 84–97.

A.M. Graziano and M.L. Raulin, Research methods—a proc-ess of inquiry, 5th edition, Allyn & Bacon, Boston, MA,2002.

I.F. Hooks and F.A. Farry, Customer-centered products,AMACOM, New York, 2001.

IBM (IBM Internal), ISM 2.0 Project Proposal, IBM GlobalServices (internal document), Bethesda, MD, 2002.

INCOSE, Systems engineering handbook, INCOSE-TP-2003-016-02, Version 2a, June 2004.zaq;4

A. Kossiakoff and W.N. Sweet, Systems engineering princi-ples and practice, Wiley, New York, 2002.

R. MacLachian, Regeneration X, People Management 4(7)(1998), 34–40.

M.S. MacNealy, Towards better case study research, IEEETrans Professional Commun 40(3) (1997), 182–196.

A. Odlyzko, Current state and likely evolution of the Internet,Global Telecommun Conf (GLOBECOM), 1999, Vol. 3,pp. 1869–1876.

L.G. Roberts, Beyond Moore’s law: Internet growth trends,Computer 33(1) (2000), 117–119.zaq;5

A.P. Sage, Systems management for information technologyand software engineering, Wiley, New York, 1995.

A.P. Sage and A.E. Armstrong, Jr., Introduction to systemsengineering, Wiley, New York, 2000.

Standish Group, CHAOS reports, http://www.standishgroup.com/sample_research/index.php, 1994, 1999, 2001.

J. Teresko, Driving success at New Blue, Industry Week(December 1999), 56–67.zaq;6

D. Verma and W.J. Fabrycky, Systematically identifying sys-tem engineering practices and methods, IEEE Trans Aero-space Electron Syst 33(2) (1997), 587–595.

Eirik Hole is currently a Lecturer and Ph.D. student in Systems Engineering at Stevens Institute ofTechnology in Hoboken, NJ, USA. He received his Dipl.Ing. Degree (~Msc) in Aerospace Engineeringfrom the University of Stuttgart in Germany in 1995. Mr. Hole has 8 years of experience working in varioussystems engineering related positions. He started his career as a Systems Engineer on a new generationmissile system program being involved in system specification, overall system design and the planningof major design reviews. He has also worked as a Systems Engineer on other European military programs.Mr. Hole has lately worked as a consultant in the area of applied systems engineering and requirementsmanagement as well as on the practical application of SE and RM tools. His consulting experience includescustomers from defense as well as the automotive and consumer goods industry in Scandinavia and CentralEurope.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 13

Dinesh Verma received the Ph.D. and the M.S. in Industrial and Systems Engineering from Virginia Tech.He is currently serving as the Associate Dean for Outreach and Executive Education, and Professor inSystems Engineering at Stevens Institute of Technology. Prior to this role, he served as Technical Directorat Lockheed Martin Undersea Systems, in Manassas, Virginia, in the area of adapted systems andsupportability engineering processes, methods and tools for complex system development and integration.Verma continues to serve numerous companies in a consulting capacity, to include Eastman Kodak,Lockheed Martin Corporation, L3 Communications, United Defense, Raytheon, IBM Corporation, SunMicrosystems, SAIC, VOLVO Car Corporation (Sweden), NOKIA (Finland), RAMSE (Finland), JohnsonControls, Ericsson-SAAB Avionics (Sweden), and Motorola. Dr. Verma has authored over 75 technicalpapers, book reviews, technical monographs, and co-authored two textbooks: Maintainability: A Key toEffective Serviceability and Maintenance Management (Wiley, 1995), and Economic Decision Analysis(Prentice Hall, 1998). In addition to his publications, Verma has received one patent and has two pendingin the areas of life-cycle costing and fuzzy logic techniques for evaluating design concepts. He is a Fellowof the International Council on Systems Engineering (INCOSE), a senior member of SOLE, and waselected to Sigma Xi, the honorary research society of America.

Rashmi Jain is Associate Professor of Systems Engineering at Stevens Institute of Technology. Dr. Jainhas over 15 years of experience of working on socio-economic and information technology (IT) systems.Over the course of her career she has been involved in leading the implementation of large and complexsystems engineering and integration projects involving substantially the entire life cycle from businessrequirements, systems requirements, systems architecture, high level design, detailed designs, develop-ment, testing, prototyping to production. Dr. Jain joined the Accenture (formerly known as AndersenConsulting) New York Office in 1997 and has since then consulted for clients ranging from Fortune 100companies to new dotcom startups in industry sectors like energy, telecom, utilities, and financial services(foreign exchange, investment banking, and capital markets). Many of her consulting projects involvedbusiness process reengineering. Dr. Jain left Accenture to join academia in early 2003. Dr. Jain has chairedsessions, presented papers, and published on systems engineering topics. She teaches systems integrationand agile systems engineering. Her research interests include network-centric systems, systems integra-tion, robust systems architecting, agile systems engineering, and applying Systems Engineering tohomeland security issues. Dr. Jain is a member of the International Council on Systems Engineering andis on its Education and Research Committee, a member of the American Society for EngineeringManagement, and Agile Alliance. She holds Ph.D. and M.S. degrees in Technology Management fromStevens Institute of Technology.

Vito Vitale has 20 years experience as a Systems Engineer, IT Architect, Project Manager, and SoftwareDeveloper. He is a certified IBM IT/Architect and is currently performing the role of lead Systems Engineerassigned to internal IBM projects. Mr. Vitale served as lead architect for IBM on numerous commercialprojects including The New York Times, Wall Street Journal, and Universal Music Group. Mr. Vitaleworked as a consultant in the New York Wall Street area for 5 years prior to joining IBM managing projectsand architecting collaborative oriented solutions for American Express, Merrill Lynch, Prudential Secu-rities, Philip Morris, and UNICEF. In AT&T, NCR and Olivetti (10 years) Mr. Vitale was softwaredeveloper and systems engineer dedicated to notebook computer diagnostics/utilities and communicationssoftware.

14 HOLE ET AL.

Paul Popick has more than 30 years experience as a program manager, systems engineering manager,software development manager, and software developer, predominantly for the public sector and commu-nications industries. Currently, he is a Systems Engineering and Architecture (SEA) Business AreaManager. As a systems engineering business area manager, he is responsible for the definition, education,and improvement of the systems engineering process as well as systems engineering for commercialprojects. Formerly a Project Executive responsible for a project portfolio of software implementations forstate and local government. His responsibilities included project oversight, project staffing, projectmanager selection, project reviews, and the project implementation methodology. Mr. Popick is also anAdjunct Professor in systems engineering at Stevens Institute of Technology and a former part timeinstructor of System Engineering for the Whiting Engineering Graduate School at the Johns HopkinsUniversity. Mr. Popick received a Master of Science in Applied Mathematics from New York Universityin 1973.

<enote>AQ1: Please list key words.<enote>AQ2: Please give page number of quota-

tion.<enote>AQ3: Please give page range.

<enote>AQ4: Please give INCOSE city.<enote>AQ5: Graziano and Rubin 1993 reference

deletion OK? Not cited in text and you have alreadylisted and cited the 5th edition.

<enote>AQ6: Weekly? If so, please give exact date.

DEVELOPMENT OF THE ibm.com INTERACTIVE SOLUTION MARKETPLACE 15