gao-01-962 hud information systems: immature software

Report to the Ranking MinorityMember, Subcommittee on Housing,and Transportation, Committee onBanking, Housing and Urban Affairs,U.S. Senate

United States General Accounting Office

GAO

September 2001 HUD INFORMATIONSYSTEMS

Immature SoftwareAcquisition CapabilityIncreases ProjectRisks

GAO-01-962

Page i GAO-01-962 HUD Information Systems

Letter 1

Executive Summary 2

Chapter 1 Introduction 7

HUD Relies on Information Systems 7The Software Acquisition Capability Maturity Model Provides a

Means of Assessing an Organization’s Ability to ManageSoftware Acquisitions 9

Objective, Scope, and Methodology 12

Chapter 2 HUD’s Requirements Development and Management

Processes Are Not Repeatable 15

Chapter 3 HUD’s Project Management Processes Are Not

Repeatable 28

Chapter 4 HUD’s Contract Tracking and Oversight Processes

Are Not Repeatable 41

Chapter 5 HUD’s Software Evaluation Processes Are Not

Repeatable 54

Chapter 6 Conclusions, Recommendations, and Agency

Comments 67

Appendix I Comments From the Department of Housing and

Urban Development 69

Contents

Page ii GAO-01-962 HUD Information Systems

Appendix II GAO Contact and Staff Acknowledgments 71

Tables

Table 1: Total Number of Key Process Area Strengths, Weaknesses,and Observations for the Five Projects 5

Table 2: Key Process Areas for SA-CMM Level 2 11Table 3: Requirements Development and Management Findings for

the Public and Indian Housing Information Center System 18Table 4: Requirements Development and Management Findings for

the Real Estate Management System 20Table 5: Requirements Development and Management Findings for

the Resident Assessment Subsystem 22Table 6: Requirements Development and Management Findings for

HUD’s Central Accounting and Program System 24Table 7: Requirements Development and Management Findings for

the Empowerment Information System 26Table 8: Project Management Findings for the Public and Indian

Housing Information Center System 31Table 9: Project Management Findings for the Real Estate

Management System 33Table 10: Project Management Findings for the Resident

Assessment Subsystem 35Table 11: Project Management Findings for HUD’s Central

Accounting and Program System 37Table 12: Project Management Findings for the Empowerment

Information System 39Table 13: Contract Tracking and Oversight Findings for the Public

and Indian Housing Information Center System 44Table 14: Contract Tracking and Oversight Findings for the Real

Estate Management System 46Table 15: Contract Tracking and Oversight Findings for the

Resident Assessment Subsystem 48Table 16: Contract Tracking and Oversight Findings for HUD’s

Central Accounting and Program System 50Table 17: Contract Tracking and Oversight Findings for the

Empowerment Information System 52Table 18: Software Evaluation Findings for the Public and Indian

Housing Information Center System 57Table 19: Software Evaluation Findings for the Real Estate

Management System 59

Page iii GAO-01-962 HUD Information Systems

Table 20: Software Evaluation Findings for the ResidentAssessment Subsystem 61

Table 21: Software Evaluation Findings for HUD’s CentralAccounting and Program System 63

Table 22: Software Evaluation Findings for the EmpowermentInformation System 65

Figures

Figure 1: SA-CMM Levels and Descriptions 10Figure 2: Requirements Development and Management Summary 17Figure 3: Project Management Summary 30Figure 4: Contract Tracking and Oversight Summary 43Figure 5: Software Evaluation Summary 56

Abbreviations

CIO Chief Information OfficerEIS Empowerment Information SystemGAO General Accounting OfficeHUD Department of Housing and Urban DevelopmentHUDCAPS HUD Central Accounting and Program SystemPIC Public and Indian Housing Information Center systemRASS Resident Assessment SubsystemREMS Real Estate Management SystemSA-CMM Software Acquisition Capability Maturity ModelSEI Software Engineering Institute

GAO-01-962 HUD Information Systems

September 14, 2001

The Honorable Wayne AllardRanking Minority MemberSubcommittee on Housing and TransportationCommittee on Banking, Housing, and Urban AffairsUnited States Senate

Dear Senator Allard:

The accompanying report is the first in a series of reports responding toyour February 2001 request that we examine a range of managementissues at the Department of Housing and Urban Development. This reportdiscusses our assessment of the department’s capability to acquiresoftware. We are making recommendations to the Secretary of Housingand Urban Development to assist the department in improving thiscapability.

We are sending copies of this report to the Secretary of Housing andUrban Development, the Director of the Office of Management andBudget, and other congressional committees. We will also make copiesavailable to other interested parties upon request.

If you have questions or wish to discuss the issues in this report, pleasecontact me at (202) 512-6240 or by email at [email protected]. Anadditional GAO contact and staff acknowledgements are listed inappendix II.

Sincerely yours,

Linda D. KoontzDirector, Information Management Issues

United States General Accounting Office

Washington, DC 20548

mailto:[email protected]

Executive Summary

Page 2 GAO-01-962 HUD Information Systems

The Department of Housing and Urban Development (HUD) routinelyacquires new information systems and enhancements to manage andsupport its various programs and operations. GAO has designated HUD’smajor program areas as high risk, in part because the department’sinformation and financial management systems were poorly integrated,ineffective, and generally unreliable.1 HUD has been endeavoring toimprove its systems to better support its missions and managementreforms.

Because of the importance of information and financial managementsystems and related improvement efforts to meeting HUD’s mission, theRanking Minority Member, Subcommittee on Housing and Transportation,Senate Committee on Banking, Housing, and Urban Affairs, requested thatGAO review the maturity of HUD’s software acquisition processes. Moremature processes increase the likelihood that the software acquired willmeet an organization’s needs.

HUD is the principal federal agency responsible for housing andcommunity development programs. HUD’s mission includes makinghousing affordable (through mortgage insurance for multifamily housingand through rental assistance), helping to revitalize localities, andencouraging home ownership. In fiscal year 2001, the department’s budgetwas about $32 billion, including $360 million for information technologyinvestments.

High-quality software is essential for HUD’s information systems toprovide reliable management, financial, and administrative informationand support the department’s many programs. The quality of software isgoverned largely by the quality of the processes involved in developing oracquiring it and in maintaining it. Carnegie Mellon University’s SoftwareEngineering Institute (SEI), recognized for its expertise in softwareprocesses, has developed models and methods that define and determineorganizations’ software process maturity. Together, these models and

1GAO High-Risk Program (GAO/AIMD-94-72R, January 27, 1994); High-Risk Series:

Department of Housing and Urban Development (GAO/HR-95-11, February 1995); High-

Risk Series: Department of Housing and Urban Development (GAO/HR-97-12, February1997); Major Management Challenges and Program Risks: Department of Housing and

Urban Development (GAO/OCG-99-8, January 1999); Major Management Challenges and

Program Risks: Department of Housing and Urban Development (GAO-01-248, January2001).

Executive Summary

Purpose

Background

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

Executive Summary


methods provide a logical framework for baselining an organization’scurrent process capabilities (i.e., determining what practices areeffectively implemented (strengths), not effectively implemented(weaknesses), or contain mixed or inconclusive evidence (observations))and providing a structured plan for incremental process improvement.

Using SEI’s software acquisition capability maturity modelSM and SEI’ssoftware capability evaluation method,2 GAO analysts trained at SEIevaluated HUD’s software acquisition maturity in four of seven keyprocess areas3 that are necessary to attain a “repeatable” level of processmaturity. The “repeatable” level of process maturity is the second level onSEI’s five-level scale. An organization at the repeatable level of processmaturity has the necessary process discipline in place to repeat earliersuccesses on similar projects. Organizations that do not satisfy therequirements for the repeatable level are by default judged to be at the“initial” level of maturity. This means that their processes are immature, adhoc, and sometimes even chaotic, with few of the processes defined andsuccess dependent mainly on the heroic efforts of individuals.

In this evaluation, GAO examined five ongoing projects selected by HUDas being representative of the department’s current software acquisitionpractices.4 These were the Public and Indian Housing Information Centersystem, the Real Estate Management System, the Resident AssessmentSubsystem, HUD’s Central Accounting and Program System, and theEmpowerment Information System.

HUD did not fully satisfy the requirements for any of the “repeatable” keyprocess areas we reviewed. While HUD has several software acquisition

2 Capability Maturity ModelSM is the service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and Trademark Office. GAO used Software Acquisition

Capability Maturity Model (SA-CMM) Version 1.2 (CMU/SEI-99-TR-002, April 1999), thelatest version of the model.

3 The four key process areas GAO evaluated are requirements development andmanagement, project management, contract tracking and oversight, and softwareevaluation. GAO did not evaluate the remaining three key process areas—softwareacquisition planning, solicitation, and transition to support—because HUD’s project teamshad not recently performed activities in these areas.

4 GAO also asked that the five projects be (1) major efforts with large software acquisitioncomponents, (2) managed by integrated project teams, (3) at different stages of the lifecycle, and (4) among HUD’s best managed projects.

Results in Brief

Executive Summary


process strengths, GAO found a large number of software processweaknesses in all key process areas evaluated: requirements developmentand management, project management, contract tracking and oversight,and software evaluation. As a result, its processes for acquiring softwareare immature and can be characterized as ad hoc, sometimes chaotic, andnot repeatable across projects. These weaknesses can lead to systems thatdo not meet the information needs of management and staff, do notprovide support for needed programs and operations, and cost more andtake longer than expected to complete. HUD is aware that it has softwareacquisition weaknesses, has stated its commitment to improving itssoftware and system acquisition processes, and will begin a processimprovement effort in the near future.

HUD did not meet the requirements for the repeatable level of maturity inthe four key process areas we reviewed. Of the 310 key practices GAOevaluated, 50 percent constituted strengths (i.e., key practice waseffectively implemented), 39 percent were weaknesses (i.e., key practicewas not effectively implemented), 10 percent were rated as observations,and 1 percent were not rated.

Certain weaknesses were systemic, recurring in most or all of the keyprocess areas. For example, HUD has no overall policy for the acquisitionof software; the majority of project teams did not develop plans to conductvarious activities; none of the teams measured the status of activities; themajority of the teams did not report regularly to management on progressand problems; and most project managers did not require regular reportson progress and problems. These weaknesses can lead to systems that donot meet HUD’s information needs, do not effectively support HUD’sprograms and services, and cost more and take longer to complete.

To reach the repeatable level of maturity, HUD must overcome the keypractice weaknesses identified in this report. Table 1 summarizes thestrengths and weaknesses for the five projects.

Principal Finding:HUD’s SoftwareAcquisition ProcessesAre Immature

Executive Summary


Table 1: Total Number of Key Process Area Strengths, Weaknesses, andObservations for the Five Projects

Key practice ratingsKey process area Strengths Weaknesses Observations Not ratedRequirementsdevelopment andmanagement 27 31 12 0Project management 45 25 10 0Contract tracking andoversight 45 33 4 3Evaluation 37 33 5 0Total 154 122 31 3

HUD acknowledged that it has software acquisition weaknesses and statedits commitment to improving its software and system acquisitionprocesses. The department has prepared a statement of work to obtaincontractor support to begin a software process improvement effort. One ofthe tasks the contract will include is development of a software processimprovement plan.

To successfully complete such an effort, HUD’s plan must address severalimportant issues. To be comprehensive, this plan should include theresults of this review, measurable goals and time frames, estimates ofresource requirements, and time frames to reach the repeatable level. Inaddition, it should address the systemic weaknesses that GAO found.Finally, the plan should include steps to address the key process areasGAO did not review, as well as steps to ensure that all ongoing and newsoftware acquisition projects adopt processes that meet the requirementsfor the repeatable level.

To improve HUD’s software acquisition capabilities, GAO recommendsthat the Secretary of Housing and Urban Development direct the HUDChief Information Officer to develop and implement a comprehensive planfor software acquisition process improvement that is based on thesoftware capability results in this report and specifies measurable goalsand time frames, sets priorities for initiatives, estimates resourcerequirements (for trained staff and funding), and defines a processimprovement management structure.

Also, to address the systemic weaknesses mentioned above, the planshould contain steps to

Recommendations forExecutive Action

Executive Summary


• develop a comprehensive policy for the acquisition of software,• require plans for specific acquisition activities,• measure and track the status of activities performed by the software

acquisition teams, and• report regularly to management on progress and problems.

In addition, to ensure that all aspects of software acquisition areaddressed, we recommend that the Secretary direct the CIO to

• assess HUD’s maturity in the three key process areas that could not beevaluated by GAO and include any needed improvement actions in thecomprehensive plan for software acquisition process improvement;

• ensure that all new software acquisition projects in HUD adopt processesthat satisfy at least SA-CMM level 2 requirements; and

• ensure that process improvement activities are initiated for all ongoingsoftware acquisition projects.

In providing written comments on a draft of our report, HUD agreed withour assessment of its software acquisition processes and ourrecommendations for strengthening them, and stated that our assessmentand recommendations will be addressed as part of its processimprovement effort. This effort should help the department improve itsability to acquire systems that meet management and staff informationneeds and efficiently support programs and operations.

Agency Comments

Chapter 1: Introduction


The Department of Housing and Urban Development (HUD), establishedin 1965, has as its primary mission ensuring a decent, safe, and sanitaryhome and suitable living environment for every American. HUD’s missionincludes making housing affordable for about 4 million low-incomehouseholds by insuring loans for multifamily rental housing and byproviding rental assistance. Its mission also includes helping to revitalizeover 4,000 localities through community development programs. Thedepartment encourages homeownership by providing, through its FederalHousing Administration, mortgage insurance for about 7 millionhomeowners who otherwise might not have qualified for loans. HUD isone of the nation’s largest financial institutions, responsible for managingabout $508 billion in insured mortgages and $570 billion in guarantees ofmortgage-backed securities. HUD’s budget in fiscal year 2001 was about$32 billion, including $360 million for information technology investments.

Like many federal agencies, HUD relies on its information and financialmanagement systems to help carry out its missions. However, past reportsby us and others showed that HUD was plagued by poorly integrated,ineffective, and generally unreliable information systems that did notsatisfy management needs or provide adequate controls.1 We designatedHUD’s program areas as high risk in part because of widespread problemswith information and financial management systems. In 1997, HUD begana management reform effort designed to transform its culture, manage itsprograms and staff more effectively, ultimately restore public trust in thedepartment, and modernize it for the 21st century.

In January 2001, we reported that HUD had been making credible progresstowards addressing its management reform goals.2 However, we alsoreported that information and financial systems remained an area ofmanagement challenge because of unresolved issues. In addition, in itsMarch 1, 2001, financial statement audit report, the Office of the HUDInspector General listed two material internal control weaknesses relatedto systems. The report stated that HUD needs to (1) completeimprovements to its financial management systems to meet its programand financial management needs and (2) enhance the Federal Housing

1 GAO-01-248, January 2001; GAO/OCG-99-8, January 1999; GAO/HR-97-12, February 1997;GAO/HR-95-11, February 1995; GAO/AIMD-94-72R, January 27, 1994.

2 GAO-01-248, January 2001.


HUD Relies onInformation Systems

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov

htt://www.gao.gov



Administration’s information systems to more effectively support itsbusiness practices.

HUD is aware that information systems remain an area of challenge. Tohelp address some of the challenges, HUD plans to (1) institute moredisciplined processes in its information technology capital investmentplanning reviews (quarterly reviews that assess how well projects arefunctioning and attempt to alert management to potential problems),(2) continue training its project managers in new project managementtechniques, (3) complete efforts to bring all software products understandard change control and configuration management3 and designate aseparate organization to address configuration management issues,(4) improve HUD’s end-to-end testing4 capabilities, and (5) obtain anddeploy new software for project-level status reporting.

To manage specific software acquisitions, HUD uses integrated projectteams. These teams include representatives from the businessorganization(s) that will use the acquired software to do their work;information technology staff to oversee the contractor and reviewcontractor work products; and contract and procurement staff to assistthe team in contract management. Policy and procedures for softwareacquisition are the responsibility of HUD’s Chief Information Officer.

3 Configuration management is a discipline applying technical and administrative directionand surveillance to identify and document the functional and physical characteristics ofhardware or software, control changes to these characteristics, record and report changeprocessing, and verify compliance of hardware or software with specified requirements.

4 End-to-end testing is done to verify that a defined set of interrelated systems, whichcollectively support an organizational core business area or function, interoperate asintended in an operational environment. This includes not only those systems owned andmanaged by the organization but also those external systems with which they interface.



The software acquisition capability maturity model (SA-CMM),5 developedby Carnegie Mellon’s Software Engineering Institute (SEI), is used tomeasure an organization’s capability to manage the acquisition ofsoftware. SEI’s expertise in and methods for software process assessmentare recognized and accepted worldwide throughout the industry. Themodel defines five levels of software acquisition maturity. Each level ofmaturity (except level 1) indicates process capability and identifies keyprocess areas. For a maturity level to be achieved, all key process areasrelated to that level must be implemented effectively.

The first level of process capability is level 2 (referred to as the repeatablelevel), where basic management processes are established to trackperformance, cost, and schedule, and the necessary discipline is in placeto repeat earlier successes on similar projects. Organizations that do noteffectively implement all key process areas for the repeatable level are, bydefault, at level 1 or the initial level of maturity. Level 1 processes can bedescribed as immature, ad hoc, and sometimes chaotic; success insoftware acquisition for these organizations is usually dependent upon theability and commitment of the staff involved. Figure 1 explains further thefive-level software acquisition model.

5 Software Acquisition Capability Maturity Model (SA-CMM), Version 1.2, SoftwareEngineering Institute, CMU/SEI-99-TR-002 (April 1999).

The SoftwareAcquisition CapabilityMaturity ModelProvides a Means ofAssessing anOrganization’s Abilityto Manage SoftwareAcquisitions



Figure 1: SA-CMM Levels and Descriptions

Table 2 identifies each of the seven key process areas associated with therepeatable level of maturity.

The software acquisition process is characterized asad hoc, and occasionally even chaotic. Few processesare defined and success depends on individual effort.

Level 1: Initial

Basic project management processes are establishedto track performance, cost, and schedule. Thenecessary process discipline is in place to repeatearlier successes on projects in similar domains.

Level 2: Repeatable

The acquisition organization’s software acquisition processis documented, standardized, and established as thestandard software acquisition process. All projects use anapproved, tailored version of the organization’s standardsoftware acquisition process for acquiring their softwareproducts and services.

Level 3: Defined

Level 5: Optimizing

Level 4: Managed

Continuous process improvement is empowered byquantitative feedback from the process and from pilotinginnovative ideas and technologies.

Detailed measures of quality of the software acquisitionprocesses, products, and services are collected. Thesoftware processes, products, and services arequantitatively understood and controlled.

Disciplinedprocess

Standardconsistentprocess

Predictableprocess

Continuouslyimprovingprocess



Table 2: Key Process Areas for SA-CMM Level 2

SA-CMM level 2 keyprocess areas DescriptionSoftware acquisitionplanning

Ensuring that reasonable planning for the softwareacquisition is conducted and that all elements of theproject are included.

Solicitation Ensuring that award is made to the contractor mostcapable of satisfying the specified requirements.

Requirements developmentand management

Establishing a common and unambiguous definition ofsoftware acquisition requirements that is understood bythe acquisition team, system users, and contractor(s).

Project management Managing the activities of the project office andsupporting contractor(s) to ensure a timely, efficient, andeffective software acquisition.

Contract tracking andoversight

Ensuring that the software activities under contract arebeing performed in accordance with contractrequirements and that products and services will satisfycontract requirements.

Evaluation Determining that the acquired software products andservices satisfy contract requirements before acceptance.

Transition to support Providing for the transition of the software products beingacquired to their eventual support organization.

As established by the model, each key process area contains five commonfeatures—commitment to perform, ability to perform, activitiesperformed, measurement and analysis, and verifying implementation—thatindicate whether the implementation and institutionalization of the keyprocess area can be effective, repeatable, and lasting. The common featuredefinitions are as follows:

• Commitment to perform: This feature describes the actions that theorganization takes to establish the process and ensure that it can endure.Key practices typically involve establishing organizational policies andsponsorship.

• Ability to perform: This feature describes the preconditions that mustexist in the project or organization to competently implement the softwareacquisition process. Key practices typically include assigningresponsibility and providing training.

• Activities performed: This feature describes the roles and proceduresnecessary to implement a key process area. Key practices typically involveestablishing plans and procedures, performing the work, tracking it, andtaking appropriate management actions.

• Measurement and analysis: This feature describes the activitiesnecessary to measure progress and analyze the measurements. Keypractices typically involve defining the measurements to be taken and the



analyses to be conducted to determine the status and effectiveness of theactivities performed.

• Verifying implementation: This feature describes the steps theorganization must take to ensure that project activities are performed inaccordance with established processes. Key practices typically involveregular reviews by management.

Each common feature consists of a number of key practices—specificactivities such as developing an organizational policy for softwareacquisition, developing various plans for software acquisition activities,and tracking a contractor’s progress. When an organization is evaluatedagainst the SA-CMM, comparisons of actual performance against a keypractice can result in one of four possible outcomes or ratings:

• Strength: The key practice involved was effectively implemented.• Weakness: The key practice was ineffectively implemented or was not

implemented.• Observation: The key practice was evaluated, but cannot be characterized

as a strength because the (1) project team did not provide sufficientevidence to support a strength rating or (2) key practice was only partiallyperformed.

• Not rated: The key practice is not relevant to the project and was thereforenot rated.

To achieve the repeatable level, HUD would have to demonstrate that thekey practices related to that level were implemented effectively in itssoftware acquisition projects.

The Ranking Minority Member of the Subcommittee on Housing andTransportation, Senate Committee on Banking, Housing, and UrbanAffairs, requested that we review HUD’s software acquisition capabilities.Our objective for this review was to determine the maturity of HUD’ssoftware acquisition processes.

To determine HUD’s software acquisition process maturity, we applied theSoftware Engineering Institute’s SA-CMM using our SEI-trained analysts.We focused on the key process areas necessary to obtain a repeatablelevel of maturity, the second level of SEI’s five-level model. We met withproject managers and project team members to determine whether and to

Objective, Scope, andMethodology



what extent they implemented each key practice and to obtain relevantdocumentation. In accordance with the SEI model, for each key processarea we reviewed,6 we evaluated HUD’s institutional policies and practicesand compared project-specific guidance and practices against the requiredkey practices.

More specifically, for each key practice we reviewed, we comparedproject-specific documentation and practices against the criteria in thesoftware acquisition model. If the project met the criteria for the keypractice reviewed, we rated it as a strength. If the project did not meet thecriteria for the key practice reviewed, we rated it as a weakness. If theevidence did not support a rating of a strength or a weakness, we rated itas an observation. If the key practice was not relevant to a project, westated that the key practice was not rated.

We evaluated five HUD projects, each of which is described below. Weasked HUD to select five system acquisition projects that wererepresentative of HUD software acquisition practices and met thefollowing criteria: all were (1) major efforts with large software acquisitioncomponents, (2) managed by integrated project teams, (3) at differentstages of the life cycle, and (4) among the best-managed acquisitions. HUDselected the following projects:

• Public and Indian Housing Information Center system (PIC)—an Internet-based system that collects information about the public housing inventorymanaged by HUD and automates the processes for allocating programfunds to public housing authorities. It is a central repository forinformation about public and Indian housing events and program areas aswell as for the housing inventory.

• Real Estate Management System (REMS)—a system that providesautomated support to collect and maintain data on multifamily insuredand assisted properties and their ownership/management entities andprovides access to data collected by HUD on physical and financialassessments.

6We evaluated HUD on four of the seven key process areas—requirements developmentand management, project management, contract tracking and oversight, and softwareevaluation. We did not evaluate HUD on the three other key process areas—softwareacquisition planning, solicitation, and transition to support—because these were notrecently performed for the selected projects.



• Resident Assessment Subsystem (RASS)—a system that collectsinformation on public housing from residents and manages the residentassessment process.

• HUD Central Accounting and Program System (HUDCAPS)—HUD’s coreaccounting and general ledger system. It serves as the standard accountingsystem for all program areas.

• Empowerment Information System (EIS)—an enterprisewide datawarehouse project that is to provide HUD users and business partnerswith access to reports and analytical information across a large variety ofprogram data sources.

We performed our work from March 2001 through August 2001 inaccordance with generally accepted government auditing standards. Weprovided a draft of our report to HUD for review, and the department’scomments are addressed in chapter 6 and reprinted as appendix I.

Chapter 2: HUD’s Requirements Development

and Management Processes Are Not

Repeatable


HUD did not satisfy the criteria for the repeatable level of maturity in thiskey process area because of the number and severity of key practiceweaknesses found. As a result of these weaknesses, HUD is at greater riskof acquiring software products that do not meet mission needs and thatexceed cost, schedule, and performance goals.

The purpose of requirements development and management is to establishand maintain a common and unambiguous definition of softwarerequirements1 among the acquisition team, system users, and softwaredevelopment contractor. Within this key process area, the SA-CMMspecifies key practices for each of the five common features that anorganization must implement effectively to achieve the repeatable level ofmaturity. Generally, these practices include having a writtenorganizational policy for establishing and managing requirementsallocated to software, documenting plans for the development andmanagement of requirements, having documented processes forrequirements development, including elicitation, analysis, and verification,measuring and reporting on the status of activities to management,appraising the impact of system-level requirements changes on thesoftware, and having a mechanism to ensure that contractor-deliveredwork products meet the specified requirements.

Of the 70 key practices for this key process area, HUD had 27 strengths, 31weaknesses and 12 observations. For the

• Commitment to perform feature, there were two strengths, fiveweaknesses, and three observations.

• Ability to perform feature, there were nine strengths, five weaknesses, andone observation.

• Activities performed feature, there were ten strengths, 14 weaknesses, andsix observations.

• Measurement and analysis feature, there were one strength, threeweaknesses, and one observation.

• Verifying implementation feature, there were five strengths, fourweaknesses, and one observation.

In addition, none of the projects had strengths in all the key practices.

1 Requirements describe the functions that the software will perform to meet user needsand provide support for business processes.

Chapter 2: HUD’s Requirements Developmentand Management Processes Are NotRepeatable



Repeatable


As a result of these weaknesses, HUD is exposed to increased risks thatacquired software will not meet its needs and that projects will exceedcost, schedule, and performance goals. HUD acknowledges the need toimprove its requirements development and management processes and iscommitted to improving its software and systems acquisition capability.The department has prepared a statement of work to obtain contractorsupport to begin a software process improvement effort. This documentspecifies that the contractor would review HUD software acquisitionprocesses, prioritize the key processes and practices to address, anddevelop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.Figure 2 provides a comprehensive listing of the strengths, weaknesses,and observations for the requirements development and management keyprocess area. The specific findings supporting the practice ratings cited infigure 2 are in tables 3 through 7.



Repeatable


Figure 2: Requirements Development and Management Summary

ProjectsCommonfeatures Key practices PIC REMS RASS HUDCAPS EISCommitment toperform 1

The acquisition organization has a written policy forestablishing and managing the software-relatedcontractual requirements.

Weakness Weakness Observation Observation Weakness

Commitment toperform 2

Responsibility has been designated for requirementsdevelopment and management activities. Weakness Strength Strength Observation Weakness

Ability toperform 1

A group is responsible for performing requirementsdevelopment and management activities. Weakness Strength Strength Observation Strength

Ability toperform 2

Adequate resources are provided for requirementsdevelopment and management activities. Weakness Weakness Strength Weakness Weakness

Ability toperform 3

Individuals performing requirements development andmanagement activities have experience or receivetraining.

Strength Strength Strength Strength Strength

Activityperformed 1

The project team performs its activities in accordancewith its documented requirements development andmanagement plans.

Weakness Observation Strength Weakness Weakness

Activityperformed 2

The project team develops, baselines, and maintainssoftware-related contractual requirements and placesthem under change control early in the project, but notlater than the release of the solicitation package.

Observation Weakness Strength Strength Observation

Activityperformed 3

The project team appraises system requirementschange requests for their impact on the software beingacquired.

Weakness Weakness Strength Strength Weakness

Activityperformed 4

The project team appraises changes to the software-related contractual requirements for their impact onperformance, architecture, supportability, systemresource utilization, and contract schedule and cost.

Weakness Weakness Strength Weakness Weakness

Activityperformed 5

Bidirectional traceability between the contractualrequirements and the contractor team’s softwareproducts and services is maintained throughout theeffort.

Observation Weakness Strength Strength Weakness

Activityperformed 6

The end user and other affected groups are involved inthe development of all software-related contractualrequirements and any subsequent change activity.

Observation Strength Strength Observation Weakness

Measurementand analysis 1

Measurements are made and used to determine thestatus of the requirements development andmanagement activities and resultant products.

Weakness Weakness Observation Weakness Strength

Verifyingimplementation 1

Requirements development and management activitiesare reviewed by the acquisition organization on aperiodic basis.

Strength Observation Weakness Strength Strength


Requirements development and management activitiesare reviewed by the project manager on both a periodicand an event-driven basis.

Weakness Strength Weakness Strength Weakness

Key Strength Strength: Key practice effectively implemented.Weakness Weakness: Key practice not effectively implemented.

Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed.



Repeatable


Table 3: Requirements Development and Management Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a written

policy for establishing and managing thesoftware-related contractualrequirements.

HUD has no written policy for establishingand managing software-relatedcontractual requirements.

Weakness

Commitment to perform 2 Responsibility has been designated forrequirements development andmanagement activities.

Responsibility for requirementsdevelopment and management has notbeen designated.

Weakness

Ability to perform 1 A group is responsible for performingrequirements development andmanagement activities.

No group is assigned responsibility forperforming requirements developmentand management.

Weakness

Ability to perform 2 Adequate resources are provided forrequirements development andmanagement activities.

There is no mechanism to identifyresource needs and ensure that they areprovided to the project. Team membersindicated that they do not have adequateresources for performing these activities.

Weakness

Ability to perform 3 Individuals performing requirementsdevelopment and management activitieshave experience or receive training.

Project team members performingrequirements development andmanagement activities have experience.

Strength

Activity performed 1 The project team performs its activities inaccordance with its documentedrequirements development andmanagement plans.

There is no requirements developmentand management plan, and so theactivities are not performed in accordancewith it.

Weakness

Activity performed 2 The project team develops, baselines,and maintains software-relatedcontractual requirements and places themunder change control early in the project,but not later than the release of thesolicitation package.

The project team developed andbaselined software-related requirementsbefore the solicitation package wasreleased. The project team provided noevidence of change control.

Observation

Activity performed 3 The project team appraises systemrequirements change requests for theirimpact on the software being acquired.

The project team does not assess systemrequirements change requests for theirimpact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changes tothe software-related contractualrequirements for their impact onperformance, architecture, supportability,system resource utilization, and contractschedule and cost.

The project team does not assess theimpact of software-related contractualrequirements changes on performance,architecture, supportability, and resourceutilization. Cost and schedule impacts areassessed.

Weakness

Activity performed 5 Bidirectional traceability between thecontractual requirements and thecontractor team’s software products andservices is maintained throughout theeffort.

The project team maintains traceabilitybetween the contract deliverables and thecontractor’s task/subtask specificationnumbering. Traceability to HUD’s originalrequirements, however, is not maintained.

Observation

Activity performed 6 The end user and other affected groupsare involved in the development of allsoftware-related contractual requirementsand any subsequent change activity.

End users are involved in identifyingsoftware-related requirements fordevelopment but are not involved insubsequent change activities.

Observation



Repeatable


Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingMeasurement and analysis 1 Measurements are made and used to

determine the status of the requirementsdevelopment and management activitiesand resultant products.

No measurements are made of HUD’srequirements development andmanagement activities and resultantproducts.

Weakness

Verifying implementation 1 Requirements development andmanagement activities are reviewed bythe acquisition organization on a periodicbasis.

Project status, including requirementsdevelopment and management activities,is reviewed quarterly by acquisitionmanagement.

Strength

Verifying implementation 2 Requirements development andmanagement activities are reviewed bythe project manager on both a periodicand an event-driven basis.

The project manager is not briefed onproject activities and the projectmanager’s reviews are not documented.

Weakness



Repeatable


Table 4: Requirements Development and Management Findings for the Real Estate Management System

Real Estate Management SystemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a

written policy for establishing andmanaging the software-relatedcontractual requirements.

HUD has no written policy for establishingand managing software-related contractualrequirements.

Weakness


Responsibility for requirementsdevelopment and management activities isdesignated in the project plan.

Strength


End users and project staff are responsiblefor performing requirements developmentand management activities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project manager said thatthe team does not have adequateresources for performing these activities.

Weakness

Ability to perform 3 Individuals performing requirementsdevelopment and managementactivities have experience or receivetraining.

Project team members performingrequirements development andmanagement activities have experience.

Strength

Activity performed 1 The project team performs its activitiesin accordance with its documentedrequirements development andmanagement plans.

There is no requirements development andmanagement plan, and so the activities arenot performed in accordance with it. Theteam does follow a process forrequirements gathering.

Observation

Activity performed 2 The project team develops, baselines,and maintains software-relatedcontractual requirements and placesthem under change control early in theproject, but not later than the releaseof the solicitation package.

The project team did not baselinerequirements before solicitation. Taskorders do not show traceability to software-related requirements.

Weakness


The project team does not appraise systemrequirements change requests for theirimpact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changesto the software-related contractualrequirements for their impact onperformance, architecture, support-ability, system resource utilization, andcontract schedule and cost.

The project team does not assess theimpact of software-related contractualrequirements changes on performance,architecture, supportability, and resourceutilization. Cost and schedule areassessed.

Weakness

Activity performed 5 Bidirectional traceability between thecontractual requirements and thecontractor team’s software productsand services is maintained throughoutthe effort.

The project team does not maintainbidirectional traceability between thecontractual requirements and contractorwork products.

Weakness



Repeatable


Real Estate Management SystemCommon feature Key practice Finding RatingActivity performed 6 The end user and other affected

groups are involved in thedevelopment of all software-relatedcontractual requirements and anysubsequent change activity.

End users are involved in the developmentof and changes to software-relatedcontractual requirements.

Strength

Measurement and analysis 1 Measurements are made and used todetermine the status of therequirements development andmanagement activities and resultantproducts.

No measurements are made of HUD’srequirements development andmanagement activities and resultantproducts.

Weakness

Verifying implementation 1 Requirements development andmanagement activities are reviewedby the acquisition organization on aperiodic basis.

Requirements development andmanagement activities are not routinelyreviewed by the acquisition organization.However, the project manager is a seniorofficial of the acquisition organization.

Observation

Verifying implementation 2 Requirements development andmanagement activities are reviewedby the project manager on both aperiodic and an event-driven basis.

The project manager told us that heparticipates in biweekly status meetingsthat include a review of requirementsdevelopment and management issues. Thestatus meetings are documented.

Strength



Repeatable


Table 5: Requirements Development and Management Findings for the Resident Assessment Subsystem

Resident Assessment SubsystemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a


HUD has no written policy for establishing andmanaging software-related contractualrequirements. The project team used HUD’ssoftware development methodology as itsstandard for establishing and managingsoftware-related contractual requirements.

Observation

Commitment to perform 2 Responsibility has been designatedfor requirements development andmanagement activities.

Responsibility for requirements developmentand management is designated in the projectplan.

Strength


The project team is responsible for performingrequirements development and managementactivities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are provided to theproject. However, the project manager statedthat they do have adequate resources forperforming requirements development andmanagement activities.

Strength


Project team members performingrequirements development and managementactivities have experience and have receivedtraining.

Strength

Activity performed 1 The project team performs itsactivities in accordance with itsdocumented requirementsdevelopment and managementplans.

The project team performs its activities inaccordance with its requirements developmentand management plan.

Strength

Activity performed 2 The project team develops,baselines, and maintains software-related contractual requirements andplaces them under change controlearly in the project, but not later thanthe release of the solicitationpackage.

The project team developed and baselinedcontract requirements in its functionalrequirements document and requirementstraceability matrix, and placed therequirements under change control.

Strength

Activity performed 3 The project team appraises systemrequirements change requests fortheir impact on the software beingacquired.

The project team assesses systemrequirements change requests for their impacton the software being acquired.

Strength

Activity performed 4 The project team appraises changesto the software-related contractualrequirements for their impact onperformance, architecture, support-ability, system resource utilization,and contract schedule and cost.

The project team appraises changes to thesoftware-related contractual requirements forimpact on performance, architecture, support-ability, system resource utilization, andcontract schedule and cost.

Strength



Repeatable


Resident Assessment SubsystemCommon feature Key practice Finding RatingActivity performed 5 Bidirectional traceability between the

contractual requirements and thecontractor team’s software productsand services is maintainedthroughout the effort.

The project team maintains bidirectionaltraceability between contractual requirementsand contractor work products.

Strength

Activity performed 6 The end user and other affectedgroups are involved in thedevelopment of all software-relatedcontractual requirements and anysubsequent change activity.

End users are involved in the development ofcontractual requirements, generation of systemchange requests, and testing.

Strength


Masurements are only made of HUD’srequirements development and managementproducts.

Observation

Verifying implementation 1 Requirements development andmanagement activities are reviewedby the acquisition organization on aperiodic basis.

Requirements development and managementactivities are not routinely reviewed by theacquisition organization. However, cost andschedule performance is reviewed periodicallyby the acquisition organization.

Weakness

Verifying implementation 2 Requirements development andmanagement activities are reviewedby the project manager on both aperiodic and an event-driven basis.

The project manager is not briefed onrequirements development and managementactivities, and the project manager’s reviewsare not documented.

Weakness



Repeatable


Table 6: Requirements Development and Management Findings for HUD’s Central Accounting and Program System

HUD’s Central Accounting and Program SystemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a


HUD has no written policy for establishingand managing the software-relatedcontractual requirements. The project teamused HUD’s software developmentmethodology as its standard for establishingand managing software-related contractualrequirements.

Observation


The project team has been designatedresponsible for requirements developmentand management, with no specificdesignation of what group or person withinthe team.

Observation


The project manager told us that the projectteam is responsible for requirementsdevelopment and management activities.

Observation


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project team indicated thatthey do not have adequate resources forperforming requirements development andmanagement activities.

Weakness


Project team members performingrequirements development andmanagement activities have experience andhave received training.

Strength


There is no requirements development andmanagement plan, and so the activities arenot performed in accordance with it.

Weakness

Activity performed 2 The project team develops, baselines,and maintains software-relatedcontractual requirements and placesthem under change control early in theproject, but not later than the release ofthe solicitation package.

The project team developed and baselinedsoftware-related contractual requirementsbefore the contractor began work, and hasmaintained the requirements underconfiguration board control.

Strength


The project team assesses all systemrequirements change requests for theirimpact on the software being acquired.

Strength

Activity performed 4 The project team appraises changes tothe software-related contractualrequirements for their impact onperformance, architecture, support-ability, system resource utilization, andcontract schedule and cost.

The project team does not assess theimpact of software-related contractualrequirements changes on performance,architecture, supportability, and systemresource utilization. Cost is appraised.

Weakness



Repeatable


HUD’s Central Accounting and Program SystemCommon feature Key practice Finding RatingActivity performed 5 Bidirectional traceability between the

contractual requirements and thecontractor team’s software productsand services is maintained throughoutthe effort.

The project team maintains bidirectionaltraceability between the contractualrequirements and the contractor team’ssoftware products and services.

Strength

Activity performed 6 The end user and other affected groupsare involved in the development of allsoftware-related contractualrequirements and any subsequentchange activity.

End users and an independent verificationand validation team were involved in testingof changes. No evidence was provided todocument user activity in the developmentof requirements.

Observation


No measurements are made of HUD’srequirements development andmanagement activities or resultant products.

Weakness

Verifying implementation 1 Requirements development andmanagement activities are reviewed bythe acquisition organization on aperiodic basis.

Acquisition management is regularlyupdated on project status, includingrequirements development andmaintenance activities.

Strength


Project manager has regular, documentedmeetings to review requirementsdevelopment and management activities.

Strength



Repeatable


Table 7: Requirements Development and Management Findings for the Empowerment Information System

Empowerment Information SystemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a


HUD has no written policy forestablishing and managing software-related contractual requirements.

Weakness


Responsibility for requirementsdevelopment and management has notbeen designated.

Weakness


The project team is responsible forperforming these activities.

Strength


There is no mechanism to identifyresource needs and ensure that theyare provided to the project. The projectteam stated that they do not haveadequate resources for performingrequirements development andmanagement activities.

Weakness

Ability to perform 3 Individuals performing requirementsdevelopment and management activitieshave experience or receive training.

Project team members have training inIT project management, includingrequirements management.

Strength


There is no requirements developmentand management plan, and so theactivities are not performed inaccordance with it.

Weakness

Activity performed 2 The project team develops, baselines,and maintains software-relatedcontractual requirements and placesthem under change control early in theproject, but not later than the release ofthe solicitation package.

The project team developed high-levelrequirements. Detailed requirementswere developed by the contractor. Noevidence was provided regardingchange control.

Observation


The project team does not appraisesystem requirements changes for theirimpact on the software being acquired.

Weakness

Activity performed 4 The project team appraises changes tothe software-related contractualrequirements for their impact onperformance, architecture, supportability,system resource utilization, and contractschedule and cost.

The project team does not assess theimpact of software-related contractualrequirements changes on performance,architecture, supportability, systemresource utilization, and contractschedule and cost.

Weakness

Activity performed 5 Bidirectional traceability between thecontractual requirements and thecontractor team’s software products andservices is maintained throughout theeffort.

The project team does not maintainbidirectional traceability between thecontractual requirements and contractorwork products.

Weakness



Repeatable


Empowerment Information SystemCommon feature Key practice Finding RatingActivity performed 6 The end user and other affected groups

are involved in the development of allsoftware-related contractualrequirements and any subsequentchange activity.

No end users were involved inrequirements development orsubsequent change activities.

Weakness

Measurement and analysis 1 Measurements are made and used todetermine the status of the requirementsdevelopment and management activitiesand resultant products.

Measurements are made that identifythe status of requirements developmentand management activities.

Strength

Verifying implementation 1 Requirements development andmanagement activities are reviewed bythe acquisition organization on a periodicbasis.

Requirements development andmanagement activities are reviewedperiodically by the acquisitionorganization.

Strength


The project manager told us that he isnot briefed on activities, and the projectmanager’s reviews are not documented.

Weakness

Chapter 3: HUD’s Project Management

Processes Are Not Repeatable


HUD did not meet the criteria for the repeatable level in the projectmanagement key process area because of the number and severity of keypractice weaknesses found. These weaknesses increase the risk that thesoftware acquisition project, the project team, and supporting contractorswill not be adequately managed, resulting in software that does not meetmission needs or acquisitions that do not meet cost, schedule, andperformance goals.

The purpose of project management is to direct and oversee the activitiesof the project team and supporting contractors to ensure a timely,efficient, and effective software acquisition. Within this key process area,the SA-CMM specifies key practices for each of the five common featuresthat an organization must implement effectively to achieve the repeatablelevel of maturity. Generally, these practices include having a project teamorganized to accomplish the project’s objective; having a written policy forthe management of the project; documenting plans for the activities of theproject team; measuring and controlling the cost, schedule, andperformance objectives throughout the software acquisition; and reportingto management periodically on the status of project managementactivities.

Of the 80 key practices for this key process area, HUD had 45 strengths, 25weaknesses, and 10 observations. For the

• Commitment to perform feature, there were four strengths and sixweaknesses.

• Ability to perform feature, there were 14 strengths, four weaknesses, andtwo observations.

• Activities performed feature, there were 22 strengths, seven weaknesses,and six observations.

• Measurement and analysis feature, there were no strengths, fourweaknesses, and one observation.

• Verifying implementation feature, there were five strengths, fourweaknesses, and one observation.


As a result of these weaknesses, HUD is exposed to increased risks thatsoftware acquisition projects will exceed cost, schedule, and performancegoals, and that acquired software will not meet mission needs. HUDacknowledges the need to improve its project management processes andis committed to improving its software and system acquisition capability.The department has prepared a statement of work to obtain contract

Chapter 3: HUD’s Project ManagementProcesses Are Not Repeatable




support to begin a software process improvement effort. This documentspecifies that the contractor would review HUD software acquisitionprocesses, prioritize the key processes and practices to address, anddevelop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.Figure 3 provides a comprehensive listing of the strengths, weaknesses,and observations for the project management key process area. Thespecific findings supporting the practice ratings cited in figure 3 are intables 8 through 12.




Figure 3: Project Management Summary


The acquisition organization has a written policy formanagement of the software acquisition project. Weakness Weakness Weakness Weakness Weakness


Responsibility has been designated for projectmanagement activities. Weakness Strength Strength Strength Strength

Ability toperform 1

A group is responsible for managing softwareacquisition activities. Observation Strength Strength Strength Strength

Ability toperform 2

Adequate resources for the project team are providedfor the duration of the project. Weakness Weakness Strength Weakness Weakness

Ability toperform 3

When project trade-offs are necessary, the projectmanager is permitted to alter the software acquisitioncost, schedule, or performance.


Ability toperform 4

Individuals performing software acquisitionmanagement activities have experience or receivetraining.

Strength Observation Strength Strength Strength

Activityperformed 1

The project team performs its activities in accordancewith its documented software acquisition plans. Observation Strength Strength Strength Strength

Activityperformed 2

The roles, responsibilities, and authority of the projectfunction are documented, maintained, andcommunicated to affected groups.

Weakness Observation Strength Strength Strength

Activityperformed 3

The project team’s commitments, and changes tocommitments, are communicated to affected groups. Observation Strength Strength Strength Strength

Activityperformed 4

The project team tracks the cost, schedule, resource,and technical risks of the project. Observation Weakness Strength Weakness Strength

Activityperformed 5

The project team tracks project issues, status,execution, funding, and expenditures against projectplans and takes action.

Observation Strength Strength Strength Strength

Activityperformed 6

The project team implements a corrective action systemfor the identification, recording, tracking, and correctionof problems discovered during the software acquisition.

Weakness Observation Strength Strength Strength

Activityperformed 7

The project team keeps its plans current during the lifeof the project as replanning occurs, issues are resolved,and new risks are discovered.

Weakness Weakness Strength Weakness Strength


Measurements are made and used to determine thestatus of the project management activities andresultant products.

Weakness Weakness Observation Weakness Weakness


Project management activities are reviewed by theacquisition organization on a periodic basis. Strength Weakness Strength Strength Strength


Project management activities are reviewed by theproject manager on both a periodic and an event-drivenbasis.

Weakness Observation Weakness Strength Weakness






Table 8: Project Management Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a written

policy for management of the softwareacquisition project.

HUD has no written policy for managingthe acquisition of software products andservices.

Weakness

Commitment to perform 2 Responsibility has been designated forproject management activities.

The project team told us that projectmanagement responsibilities had beenassigned, but no documentation wasprovided to support this statement.

Weakness

Ability to perform 1 A group is responsible for managingsoftware acquisition activities.

The project team told us that IT andbusiness project leads and support staffare responsible for managing softwareacquisition activities, but no documentationwas provided to support this statement.

Observation

Ability to perform 2 Adequate resources for the project teamare provided for the duration of theproject.

There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. Team members indicated thatthe team does not have adequateresources for conducting projectmanagement activities.

Weakness

Ability to perform 3 When project trade-offs are necessary,the project manager is permitted to alterthe software acquisition cost, schedule, orperformance.

The project manager is permitted to alterschedule or performance to ensure thatannual project budgets are not exceeded.

Strength

Ability to perform 4 Individuals performing softwareacquisition management activities haveexperience or receive training.

Project team members performing projectmanagement activities have experienceand project management training.

Strength

Activity performed 1 The project team performs its activities inaccordance with its documented softwareacquisition plans.

The project team told us they did developproject management plans, but nodocumentation was provided to supportthis statement.

Observation

Activity performed 2 The roles, responsibilities, and authorityof the project function are documented,maintained, and communicated toaffected groups.

The project team has not documented theroles, responsibilities, and authority of itsmembers.

Weakness

Activity performed 3 The project team’s commitments, andchanges to commitments, arecommunicated to affected groups.

The commitments and changes tocommitments are communicated duringsmall informal group meetings, but they arenot documented and no records are kept.

Observation

Activity performed 4 The project team tracks the cost,schedule, resource, and technical risks ofthe project.

Cost and schedule are tracked, butresource and technical risks of the projectare not tracked.

Observation

Activity performed 5 The project team tracks project issues,status, execution, funding, andexpenditures against project plans andtakes action.

The project team only tracks and takesaction on changes to the project schedule.

Observation




Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingActivity performed 6 The project team implements a corrective

action system for the identification,recording, tracking, and correction ofproblems discovered during the softwareacquisition.

The project team does not maintain acorrective action system for theidentification, recording, tracking, andcorrection of problems discovered duringthe software acquisition.

Weakness

Activity performed 7 The project team keeps its plans currentduring the life of the project as replanningoccurs, issues are resolved, and newrisks are discovered.

The project team does not keep its plansup to date over the life of the project.

Weakness

Measurement and analysis 1 Measurements are made and used todetermine the status of the projectmanagement activities and resultantproducts.

No measurements are made of HUD’sproject management activities or resultantproducts.

Weakness

Verifying implementation 1 Project management activities arereviewed by the acquisition organizationon a periodic basis.

Acquisition management reviews projectactivities quarterly.

Strength

Verifying implementation 2 Project management activities arereviewed by the project manager on botha periodic and an event-driven basis.

The project manager stated that while hemeets with members of his staff daily todiscuss the status of the project, meetingresults and action items are notdocumented.

Weakness




Table 9: Project Management Findings for the Real Estate Management System


written policy for management of thesoftware acquisition project.

HUD has no written policy for managing theacquisition of software products andservices.

Weakness


Responsibility for project managementactivities has been assigned to the projectmanager in the project management plan.

Strength


The project manager, along with businessand IT support staff, is responsible forproject management activities.

Strength

Ability to perform 2 Adequate resources for the projectteam are provided for the duration ofthe project.

There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project manager stated thatthe team does not have adequate resourcesfor conducting project managementactivities.

Weakness

Ability to perform 3 When project trade-offs arenecessary, the project manager ispermitted to alter the softwareacquisition cost, schedule, orperformance.


Strength

Ability to perform 4 Individuals performing softwareacquisition management activitieshave experience or receive training.

The project team told us that individualsperforming project management activitieshave experience and have had training.However, the team did not providedocumentation to support this statement.

Observation

Activity performed 1 The project team performs its activitiesin accordance with its documentedsoftware acquisition plans.

Software acquisition project managementplans are used to guide the activities of theproject team.

Strength

Activity performed 2 The roles, responsibilities, andauthority of the project function aredocumented, maintained, andcommunicated to affected groups.

Roles and responsibilities are defined in theproject plan; however, responsible teammembers are not named, which hinderseffective communication of this information.

Observation


The commitments, and changes tocommitments, are communicated to affectedgroups through weekly status meetings,emails, and telephone conversations.

Strength

Activity performed 4 The project team tracks the cost,schedule, resource, and technicalrisks of the project.

The project team does not track risksrelated to cost, schedule, resource, andtechnical aspects of the project.

Weakness


The project manager tracks project status,execution, funding, and expenditures, andchanges the scope or schedule ifnecessary.

Strength




Real Estate Management SystemCommon feature Key practice Finding RatingActivity performed 6 The project team implements a

corrective action system for theidentification, recording, tracking, andcorrection of problems discoveredduring the software acquisition.

The project team does not maintain acorrective action system for theidentification, recording, tracking, andcorrection of problems discovered duringthe software acquisition. Minutes ofbiweekly status meetings identify actionitems and who is responsible for each item.

Observation

Activity performed 7 The project team keeps its planscurrent during the life of the project asreplanning occurs, issues areresolved, and new risks arediscovered.

The project team does not keep its plans upto date over the life of the project.

Weakness


No measurements are made of HUD’sproject management activities or resultantproducts.

Weakness

Verifying implementation 1 Project management activities arereviewed by the acquisitionorganization on a periodic basis.

Acquisition management does not regularlyreview project management activities.

Weakness

Verifying implementation 2 Project management activities arereviewed by the project manager onboth a periodic and an event-drivenbasis.

The project manager is involved with theproject, and status meetings are heldregularly; however, the project manager’sreviews are not documented.

Observation




Table 10: Project Management Findings for the Resident Assessment Subsystem



HUD has no written policy for managing theacquisition of software products andservices. The project team uses HUD’ssoftware development methodology whereapplicable.

Weakness


Responsibility for project managementactivities has been assigned to the IT andbusiness project leads.

Strength


The IT and business project leads andsupport staff are responsible for projectmanagement activities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project manager stated thatthe team has adequate resources forconducting project management activities.

Strength

Ability to perform 3 When project trade-offs are necessary,the project manager is permitted to alterthe software acquisition cost, schedule,or performance.


Strength

Ability to perform 4 Individuals performing softwareacquisition management activities haveexperience or receive training.

Project team members performing projectmanagement activities have experience andhave had training.

Strength

Activity performed 1 The project team performs its activitiesin accordance with its documentedsoftware acquisition plans.

Software acquisition project managementplans are used to guide the activities of theproject team.

Strength

Activity performed 2 The roles, responsibilities, and authorityof the project function are documented,maintained, and communicated toaffected groups.

Roles, responsibilities, and authority aredefined in the project plan and in theConfiguration Control Board’s (CCB) rolematrix.

Strength


The project team’s commitments andchanges to commitments regarding thisproject are communicated to affectedgroups.

Strength

Activity performed 4 The project team tracks the cost,schedule, resource, and technical risksof the project.

A risk management plan is maintained thattracks risks associated with costs, schedule,resource, and technical risks.

Strength


The project team uses contractor weeklyand monthly status reports to trackcontractor costs and schedule issues. Also,a quarterly project control review is done toreview cost and schedule status.

Strength

Activity performed 6 The project team implements acorrective action system for theidentification, recording, tracking, andcorrection of problems discoveredduring the software acquisition.

A corrective action system has beenimplemented for the identification, recording,tracking, and correction of problemsdiscovered during the software acquisition.

Strength




Resident Assessment SubsystemCommon feature Key practice Finding RatingActivity performed 7 The project team keeps its plans

current during the life of the project asreplanning occurs, issues are resolved,and new risks are discovered.

The project plans are kept up to date asreplanning occurs, issues are resolved, andnew risks are discovered. Monthly reportsare provided that show changes toscheduled software releases resulting fromreplanning.

Strength


Measurements are only made of HUD’sproject management products.

Observation


The acquisition organization reviews projectmanagement activities on a quarterly basis.

Strength


The project manager stated that while hemeets with members of his staff daily todiscuss the status of the project, meetingresults and action items are notdocumented.

Weakness




Table 11: Project Management Findings for HUD’s Central Accounting and Program System



HUD has no written policy for managing theacquisition of software products and services.

Weakness

Commitment to perform 2 Responsibility has been designatedfor project management activities.

Responsibility for project management activitieshas been assigned to the project lead.

Strength


The IT and business project leads areresponsible for managing software acquisitionactivities.

Strength


There is no mechanism to identify resourceneeds and ensure they were made available tothe project. The project manager stated that theteam does not have adequate resources forperforming project management activities.

Weakness


The project manager is permitted to alter theschedule or performance to ensure that annualproject budgets are not exceeded.

Strength


Project team members performing projectmanagement activities have experience andhave had training.

Strength

Activity performed 1 The project team performs itsactivities in accordance with itsdocumented software acquisitionplans.

Software acquisition project management plansare used to guide the activities of the projectteam.

Strength


The roles, responsibilities, and authority of theproject team are defined in the softwareacquisition project plan.

Strength


The commitments and changes tocommitments are communicated to affectedgroups through weekly status meetings, emails,and telephone conversations.

Strength


The project plans and exhibits identify risksassociated with the project and definemitigation plans for these risks. However, theproject team does not track the risks.

Weakness

Activity performed 5 The project team tracks projectissues, status, execution, funding,and expenditures against projectplans and takes action.

The project team tracks project issues, status,execution, and funding through weeklyfunctional and technical status meetings.Funding is tracked and actual costs comparedto projected costs.

Strength

Activity performed 6 The project team implements acorrective action system for theidentification, recording, tracking, andcorrection of problems discoveredduring the software acquisition.

A corrective action system has beenimplemented for the identification, recording,tracking, and correction of problems discoveredduring the software acquisition.

Strength




HUD’s Central Accounting and Program SystemCommon feature Key practice Finding RatingActivity performed 7 The project team keeps its plans

current during the life of the projectas replanning occurs, issues areresolved, and new risks arediscovered.

The project team does have a risk assessmentand mitigation strategy. However, the risks arenot tracked, and plans are not kept current.

Weakness


No measurements are made of HUD’s projectmanagement activities or resultant products.

Weakness


The acquisition organization reviews projectmanagement activities on a regular basisthrough meetings with the project manager.

Strength


The project manager reviews projectmanagement activities regularly throughbriefings and document review.

Strength




Table 12: Project Management Findings for the Empowerment Information System



HUD has no written policy for managing theacquisition of software products and services.

Weakness

Commitment to perform 2 Responsibility has been designatedfor project management activities.

Responsibility for project management activitieshas been assigned to the project technical lead.

Strength


The project lead and assigned staff areresponsible for performing project managementactivities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are made availableto the project. The project manager stated thatthe team does not have adequate resources forperforming project management activities.

Weakness


The project manager is allowed to alterschedule or performance to ensure that annualproject budgets are not exceeded.

Strength


Project team members performing projectmanagement activities have experience andproject management training.

Strength

Activity performed 1 The project team performs itsactivities in accordance with itsdocumented software acquisitionplans.

Software acquisition project management plansare used to guide the activities of the projectteam.

Strength


Roles, responsibilities, and authority of theproject team are defined in the projectorganization chart.

Strength

Activity performed 3 The project team’s commitments,and changes to commitments, arecommunicated to affected groups.

The project manager uses contractor statusreports to communicate commitments andchanges to commitments to the project teamand other affected groups in meetings.

Strength


The project team does assess and track risksfor several areas, including contract, funding,schedule, management, and technical areas.

Strength

Activity performed 5 The project team tracks projectissues, status, execution, funding,and expenditures against projectplans and takes action.

The project team did conduct a review of thesoftware developed by the contractor. As aresult of issues found during that review,modifications were made to the contract.

Strength

Activity performed 6 The project team implements acorrective action system for theidentification, recording, tracking,and correction of problemsdiscovered during the softwareacquisition.

A remedial action plan was developed and acorrective action system implemented torecord, track, and correct problems foundduring the software acquisition.

Strength




Empowerment Information SystemCommon feature Key practice Finding RatingActivity performed 7 The project team keeps its plans

current during the life of the projectas replanning occurs, issues areresolved, and new risks arediscovered.

The replanning is documented in the remedialaction plan.

Strength

Measurement and analysis 1 Measurements are made and usedto determine the status of theproject management activities andresultant products.

No measurements are made of HUD’s projectmanagement activities and resultant products.

Weakness


The acquisition organization reviews projectmanagement activities quarterly.

Strength


The project manager is not briefed by othermembers on either a regular or event-drivenbasis, and project manager reviews are notdocumented.

Weakness

Chapter 4: HUD’s Contract Tracking and

Oversight Processes Are Not Repeatable


HUD’s contract tracking and oversight processes do not satisfy the criteriafor the repeatable level of maturity because of the number and severity ofkey practice weaknesses found. These weaknesses increase the risk thatsoftware activities will not be performed in accordance with contractualrequirements, resulting in software acquisitions being late, costing morethan intended, and not performing as intended.

The purpose of contract tracking and oversight is to ensure that thesoftware activities under contract are being performed in accordance withcontractual requirements. Within this key process area, the SA-CMMspecifies a number of key practices across the five common features thatan organization must implement effectively to achieve the repeatable levelof maturity. Generally, these practices include having a writtenorganizational policy for contract tracking and oversight, documentingplans for contract tracking and oversight activities, reviewing contractorsoftware planning documents and using those documents to oversee thecontractor’s efforts, comparing actual cost and schedule to plannedbudgets and schedules, and having a mechanism to ensure that contractor-delivered work products meet contract requirements.

For the 85 key practices in this key process area, HUD had 45 strengths, 33weaknesses, four observations, and three practices not rated. For the

• Commitment to perform feature, there were 12 strengths and threeweaknesses.

• Ability to perform feature, there were ten strengths and five weaknesses.• Activities performed feature, there were 20 strengths, 15 weaknesses, two

observations, and three practices not rated.• Measurement and analysis feature, there were no strengths and five

weaknesses.• Verifying implementation feature, there were three strengths, five

weaknesses, and two observations.


As a result of these weaknesses, HUD is exposed to increased risk thatsoftware activities will not be performed in accordance with contractualrequirements, resulting in software acquisitions being late, costing morethan planned, and not performing as intended. HUD acknowledges theneed to improve its contract tracking and oversight processes and iscommitted to improving its software and system acquisition capability.The department has prepared a statement of work to obtain contractorsupport to begin a software process improvement effort. This document

Chapter 4: HUD’s Contract Tracking andOversight Processes Are Not Repeatable




specifies that the contractor would review HUD software acquisitionprocesses, prioritize the key processes and practices to address, anddevelop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.Figure 4 provides a comprehensive listing of the strengths, weaknesses,and observations for the contract tracking and oversight key process area.The specific findings supporting the practice ratings cited in figure 4 are intables 13 through 17.




Figure 4: Contract Tracking and Oversight Summary


The acquisition organization has a written policy formanaging the contract tracking and oversight of thesoftware acquisition project.

Weakness Strength Strength Strength Strength


Responsibility has been designated for contract trackingand oversight activities. Weakness Strength Strength Strength Strength


The project team includes contracting specialists in themanagement of the contract. Weakness Strength Strength Strength Strength

Ability toperform 1

A group is responsible for managing contract trackingand oversight activities. Weakness Strength Strength Strength Strength

Ability toperform 2

Adequate resources are provided for contract trackingand oversight activities. Weakness Weakness Strength Weakness Weakness

Ability toperform 3

Individuals performing contract tracking and oversightactivities have experience or receive training. Strength Strength Strength Strength Strength

Activityperformed 1

The project team performs its activities in accordancewith documented contract tracking and oversight plans. Weakness Weakness Strength Weakness Weakness

Activityperformed 2

The project team reviews required contractor softwareplanning documents which, when satisfactory, are usedto oversee the contractor team’s software effort.

Weakness Weakness Weakness Weakness Weakness

Activityperformed 3

The project team conducts periodic reviews andinterchanges with the contractor team. Strength Strength Strength Strength Strength

Activityperformed 4

The actual cost and schedule of the contractor’ssoftware effort are compared to planned budgets andschedules, and issues are identified.

Strength Strength Strength Observation Strength

Activityperformed 5

The size, critical computer resources, and technicalactivities associated with the contractor team’s workproducts are tracked, and issues are identified.

Weakness Weakness Strength Strength Observation

Activityperformed 6

The project team reviews and tracks the development ofthe software engineering environment required toprovide life-cycle support for the acquired software, andissues are identified.


Activityperformed 7

Any issues found by the project team during contracttracking and oversight activities are recorded in theappropriate corrective action system, action is taken,and issues are tracked to closure.

Weakness Weakness Strength Weakness Strength

Activityperformed 8

The project team ensures that changes to the software-related contractual requirements are coordinated withaffected groups and individuals, such as the contractingofficial, contractor, and end user.

Strength Not rated Not rated Not rated Weakness


Measurements are made and used to determine thestatus of the contract tracking and oversight activitiesand resultant products.

Weakness Weakness Weakness Weakness Weakness


Contract tracking and oversight activities are reviewedby acquisition management on a periodic basis. Weakness Weakness Weakness Strength Strength


Contract tracking and oversight activities are reviewedby the project manager on both a periodic and an event-driven basis.

Weakness Observation Weakness Strength Observation






Table 13: Contract Tracking and Oversight Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a

written policy for managing the contracttracking and oversight of the softwareacquisition project.

HUD has no written policy for contracttracking and oversight of the acquisition ofsoftware products and services, and theproject team did not establish one.

Weakness

Commitment to perform 2 Responsibility has been designated forcontract tracking and oversightactivities.

Responsibility for contract tracking andoversight activities is not designated.

Weakness

Commitment to perform 3 The project team includes contractingspecialists in the management of thecontract.

The project team told us they did includecontracting specialists in the managementof the contract, but no documentation wasprovided to support this statement.

Weakness

Ability to perform 1 A group is responsible for managingcontract tracking and oversightactivities.

The project team told us there was a groupassigned responsibility for contract trackingand oversight activities, but nodocumentation was provided to supportthis statement.

Weakness

Ability to perform 2 Adequate resources are provided forcontract tracking and oversightactivities.

There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. Team members indicated thatthe resources for performing contracttracking and oversight activities are notadequate.

Weakness

Ability to perform 3 Individuals performing contract trackingand oversight activities haveexperience or receive training.

Project team members performing contracttracking and oversight activities haveexperience.

Strength

Activity performed 1 The project team performs its activitiesin accordance with its documentedcontract tracking and oversight plans.

There is no contract tracking and oversightplan; therefore, the activities are notperformed in accordance with it.

Weakness

Activity performed 2 The project team reviews requiredcontractor software planningdocuments which, when satisfactory,are used to oversee the contractorteam’s software effort.

The contractor did not prepare softwareplanning documents that the project teamcould use to oversee the contractor’ssoftware development effort.

Weakness

Activity performed 3 The project team conducts periodicreviews and interchanges with thecontractor team.

The project team does have periodicreviews and discussions with thecontractor team that are documented.

Strength

Activity performed 4 The actual cost and schedule of thecontractor’s software effort arecompared to planned budgets andschedules, and issues are identified.

The project team does compare the actualcost and schedule of the contractor’ssoftware engineering effort to plannedschedules and budgets, identifies issues,and documents results.

Strength

Activity performed 5 The size, critical computer resources,and technical activities associated withthe contractor team’s work products aretracked, and issues are identified.

The project team does not track the size,critical computer resources, and technicalactivities associated with the contractorteam’s work products or identify issues.

Weakness




Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingActivity performed 6 The project team reviews and tracks

the development of the softwareengineering environment required toprovide life-cycle support for theacquired software, and issues areidentified.

The project team does review and trackdevelopment of the software engineeringenvironment required to provide life-cyclesupport for the acquired software, andissues are identified and documented.

Strength

Activity performed 7 Any issues found by the project teamduring contract tracking and oversightactivities are recorded in theappropriate corrective action system,action is taken, and issues are trackedto closure.

The project team does not have acorrective action system to record or trackaction taken on issues found by the projectteam during contract tracking and oversightactivities.

Weakness

Activity performed 8 The project team ensures that changesto the software-related contractualrequirements are coordinated withaffected groups and individuals, suchas the contracting official, contractor,and end user.

The project team does ensure thatchanges to contractual requirements arecoordinated with all affected groups andindividuals.

Strength

Measurement and analysis 1 Measurements are made and used todetermine the status of the contracttracking and oversight activities andresultant products.

No measurements are made of HUD’scontract tracking and oversight activities orresultant products.

Weakness

Verifying implementation 1 Contract tracking and oversightactivities are reviewed by acquisitionmanagement on a periodic basis.

The acquisition organization does notreview contract tracking and oversightactivities on a periodic basis.

Weakness

Verifying implementation 2 Contract tracking and oversightactivities are reviewed by the projectmanager on both a periodic and anevent-driven basis.

The project manager stated that while hediscusses the status of the project withteam members, meeting results and actionitems are not documented.

Weakness




Table 14: Contract Tracking and Oversight Findings for the Real Estate Management System


written policy for managing thecontract tracking and oversight of thesoftware acquisition project.

HUD has no written policy for contracttracking and oversight of the acquisition ofsoftware products and services. Theproject team is following the policy inHUD’s procurement manual for contracttracking and oversight.

Strength


Responsibility for contract tracking andoversight activities has been designated tocontract specialists on the team.

Strength


The project team includes contractspecialists to help manage the contract.

Strength


A group is assigned responsibility forcontract tracking and oversight activities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. Team members indicated thatthe resources for performing contracttracking and oversight activities are notadequate.

Weakness

Ability to perform 3 Individuals performing contracttracking and oversight activities haveexperience or receive training.

Project team members performing contracttracking and oversight activities haveexperience and have received training.

Strength



Weakness


The contractor did prepare softwareplanning documents that could be used tooversee the contractor team’s softwaredevelopment effort, but no evidence wasprovided that these were reviewed and/orapproved by the project team.

Weakness



Strength


The project team does compare the actualcost and schedule of the contractor’ssoftware engineering effort to plannedschedules and budgets, identify issues,and document results.

Strength

Activity performed 5 The size, critical computer resources,and technical activities associated withthe contractor team’s work productsare tracked, and issues are identified.

The project team told us they track thesize, critical computer resources, andtechnical activities associated with thecontractor team’s work products, but nodocumentation was provided to supportthis statement.

Weakness




Real Estate Management SystemCommon feature Key practice Finding RatingActivity performed 6 The project team reviews and tracks


The project team does review and track thedevelopment of the software engineeringenvironment required to provide life-cyclesupport for the acquired software, andissues are identified and documented.

Strength



Weakness

Activity performed 8 The project team ensures thatchanges to the software-relatedcontractual requirements arecoordinated with affected groups andindividuals, such as the contractingofficial, contractor, and end user.

The project team stated that no changes tothe software-related contractualrequirements have taken place becausetheir contracts are broad enough to coverchanges to the contractual requirements.

Not rated



Weakness



Weakness


The project manager informally reviews thework of the project team; however, theresults of these reviews are notdocumented. Minutes of biweekly statusmeetings with contractors are maintained.

Observation




Table 15: Contract Tracking and Oversight Findings for the Resident Assessment Subsystem



HUD has no written policy for contracttracking and oversight of the acquisition ofsoftware products and services. Theproject team is following the policy inHUD’s Procurement Handbook for contracttracking and oversight.

Strength


Responsibility for contract tracking andoversight activities is designated in theproject plan.

Strength



Strength



Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project manager statedthat the team does have adequateresources for performing contract trackingand oversight activities.

Strength



Strength


The project team performs its activities inaccordance with its documented contracttracking and oversight plan.

Strength



Weakness



Strength


The project team does compare the actualcost and schedule of the contractor’ssoftware engineering effort to plannedschedules and budgets, identify issues,and document results.

Strength


The project team does track the size,critical computer resources, and technicalactivities associated with the contractorteam’s work products and identifies anddocuments issues.

Strength




Resident Assessment SubsystemCommon feature Key practice Finding RatingActivity performed 6 The project team reviews and tracks



Strength


The project team records issues related tocontract tracking and oversight in acorrective action system, and trackscorrective actions to closure.

Strength


The project team stated that no changes tothe software-related contractualrequirements have taken place.

Not rated


No measurements are made of HUD’scontract tracking and oversight activitiesand resultant products.

Weakness



Weakness


The project manager informally reviews thework of the project team. However, theresults of these reviews are notdocumented.

Weakness




Table 16: Contract Tracking and Oversight Findings for HUD’s Central Accounting and Program System



HUD has no written policy for contracttracking and oversight of the acquisition ofsoftware products and services. Theproject team is following the policy inHUD’s Procurement Handbook for contracttracking and oversight.

Strength


Responsibility for contract tracking andoversight activities is designated in theorganization chart.

Strength



Strength



Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. The project manager statedthat the team does not have adequateresources for performing contract trackingand oversight activities.

Weakness



Strength



Weakness



Weakness



Strength


The project team told us they do comparethe actual cost and schedule of thecontractor’s software engineering effort toplanned schedules and budgets, but nodocumentation was provided to supportthis statement.

Observation


The project team does track the size,critical computer resources, and technicalactivities associated with the contractorteam’s work products and identifies anddocuments issues.

Strength




HUD’s Central Accounting and Program SystemCommon feature Key practice Finding RatingActivity performed 6 The project team reviews and tracks



Strength



Weakness


The project team stated that no changes tothe software-related contractualrequirements have taken place becausetheir contracts are broad enough to coverchanges to contractual requirements.

Not rated



Weakness


The acquisition organization regularlyreviews contract tracking and oversightactivities.

Strength


The project manager is briefed weekly onproject status, including contract trackingand oversight activities.

Strength




Table 17: Contract Tracking and Oversight Findings for the Empowerment Information System



HUD has no written policy for contracttracking and oversight of the acquisition ofsoftware products and services. Theproject team is following the policy inHUD’s Procurement Handbook forcontract tracking and oversight.

Strength


Responsibility for contract tracking andoversight activities has been designatedin the organization chart.

Strength



Strength


A group is assigned responsibility formanaging contract tracking and oversightactivities.

Strength


There is no mechanism to identifyresource needs and ensure that they areprovided to the project. The projectmanager indicated that the team does nothave adequate resources for performingcontract tracking and oversight activities.

Weakness


Project team members performingcontract tracking and oversight activitieshave experience and have receivedtraining.

Strength


There is no contract tracking andoversight plan; therefore, the activities arenot performed in accordance with it.

Weakness



Weakness



Strength


The project team does compare theactual cost and schedule of thecontractor’s software engineering effort toplanned schedules and budgets, identifyissues, and document results.

Strength


The project team reviewed the size,critical computer resources, and technicalactivities associated with the contractorteam’s work products only during pre-release testing.

Observation




Empowerment Information SystemCommon feature Key practice Finding RatingActivity performed 6 The project team reviews and tracks


The project does review and trackdevelopment of the software engineeringenvironment required to provide life-cyclesupport for the acquired software, andissues are identified and documented.

Strength


The project team records issues related tocontract tracking and oversight in acorrective action system, and trackscorrective actions to closure.

Strength


The project team stated that changes tosoftware-related contractual requirementswere coordinated with all affected groupsand individuals, but they provided nodocumentation to support this statement.

Weakness


No measurements are made of HUD’scontract tracking and oversight activitiesor resultant products.

Weakness


The acquisition organization periodicallyreviews contract tracking and oversightactivities.

Strength


The project manager works closely withand informally reviews the work of theproject team. However, the results ofthese reviews are not documented.

Observation

Chapter 5: HUD’s Software Evaluation



HUD did not satisfy the criteria for the repeatable level of maturity in thiskey process area because of the number and severity of key practiceweaknesses found. These weaknesses result in increased risk that HUD’ssoftware acquisition projects will not be evaluated effectively, softwareproducts will not work effectively or meet mission needs, and cost,schedule, and other performance goals will not be met.

The purpose of evaluation is to determine that the acquired softwareproducts and services satisfy contract requirements before acceptance.Within this key process area, the SA-CMM specifies key practices for eachof the five common features that an organization must implementeffectively to achieve the repeatable level of maturity. Generally, thesepractices include having a written organizational policy for evaluatingacquired software products and services, documenting plans forevaluating software products and services, developing and managingevaluation requirements in conjunction with software technicalrequirements, tracking contractor evaluation activities for compliancewith the contract, and measuring and reporting on the status and results ofevaluation activities to management.

For the 75 key practices in this key process area, HUD had 37 strengths, 33weaknesses, and five observations. For the

• Commitment to perform feature, there were four strengths, fourweaknesses, and two observations.

• Ability to perform feature, there were 11 strengths and nine weaknesses.• Activities performed feature, there were 19 strengths, nine weaknesses,

and two observations.• Measurement and analysis feature, there were no strengths, four

weaknesses, and one observation.• Verifying implementation feature, there were three strengths and seven

weaknesses.


As a result of these weaknesses, HUD is exposed to increased risk thatsoftware acquisition projects will not be evaluated effectively, thatacquired software will not work efficiently or meet mission needs, and thatcost, schedule, and other performance goals will not be met. HUDacknowledges the need to improve its software evaluation processes andis committed to improving its software and system acquisition capability.The department has prepared a statement of work to obtain contractsupport to begin a software process improvement effort. This document

Chapter 5: HUD’s Software EvaluationProcesses Are Not Repeatable




specifies that the contractor would review HUD software acquisitionprocesses, prioritize the key processes and practices to address, anddevelop a plan to improve HUD’s acquisition processes.

Details of our evaluation are provided in the following figure and tables.Figure 5 provides a comprehensive listing of the strengths, weaknesses,and observations in the evaluation key process area. The specific findingssupporting the practice ratings cited in figure 5 are in tables 18 through 22.




Figure 5: Software Evaluation Summary


The acquisition organization has a written policy formanaging the evaluation of acquired software productsand services.

Weakness Weakness Observation Observation Weakness


Responsibility has been designated for evaluationactivities. Weakness Strength Strength Strength Strength

Ability toperform 1

A group is responsible for planning, managing, andperforming evaluation activities. Weakness Strength Strength Strength Strength

Ability toperform 2

Adequate resources are provided for evaluationactivities. Weakness Weakness Strength Weakness Weakness

Ability toperform 3

Individuals performing evaluation activities haveexperience or receive training. Strength Strength Strength Strength Strength

Ability toperform 4

Members of the project team and groups supporting thesoftware acquisition receive orientation on theobjectives of the evaluation approach.

Weakness Weakness Weakness Weakness Strength

Activityperformed 1

The project team performs its activities in accordancewith its documented evaluation plans. Strength Strength Strength Weakness Weakness

Activityperformed 2

The project’s evaluation requirements are developed inconjunction with the development of the system orsoftware technical requirements.

Strength Observation Strength Strength Weakness

Activityperformed 3

The project’s evaluation activities are planned tominimize duplication and take advantage of allevaluation results, where appropriate.

Strength Strength Strength Strength Weakness

Activityperformed 4

The project team appraises the contractor team’sperformance over the total period of the contract forcompliance with requirements.

Weakness Observation Strength Strength Weakness

Activityperformed 5

Planned evaluations are performed on the evolvingsoftware products and services before acceptance foroperational use.

Strength Strength Strength Strength Weakness

Activityperformed 6

Results of the evaluations are analyzed and comparedto the contract’s requirements to establish an objectivebasis to support the decision to accept the product orservices or to take further action.

Weakness Strength Strength Strength Weakness


Measurements are made and used to determine thestatus of the evaluation activities and resultant products. Weakness Weakness Weakness Weakness Observation


Evaluation activities are reviewed by acquisitionorganization management on a periodic basis. Strength Strength Weakness Weakness Weakness


Evaluation activities are reviewed by the projectmanager on both a periodic and an event-driven basis. Weakness Strength Weakness Weakness Weakness






Table 18: Software Evaluation Findings for the Public and Indian Housing Information Center System

Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingCommitment to perform 1 The acquisition organization has a

written policy for managing theevaluation of acquired softwareproducts and services.

HUD has no written policy for managing theevaluation of acquired software productsand services.

Weakness

Commitment to perform 2 Responsibility has been designatedfor evaluation activities.

The project team told us that responsibilityfor evaluation activities has been assignedto the lead technical person, but nodocumentation was provided to support thisstatement.

Weakness

Ability to perform 1 A group is responsible for planning,managing, and performing evaluationactivities.

The project team told us that evaluationactivities are managed by the lead technicalperson, but no documentation was providedto support this statement.

Weakness

Ability to perform 2 Adequate resources are provided forevaluation activities.

There is no mechanism to identify neededresources and ensure that they are providedto the project. The lead technical personindicated that the team does not haveadequate resources for performingevaluation activities.

Weakness

Ability to perform 3 Individuals performing evaluationactivities have experience or receivetraining.

Project team members performingevaluation activities have experience.

Strength

Ability to perform 4 Members of the project team andgroups supporting the softwareacquisition receive orientation on theobjectives of the evaluationapproach.

Members of the project team or otherssupporting the software acquisition did notreceive orientation on the objectives of theevaluation approach.

Weakness

Activity performed 1 The project team performs itsactivities in accordance with itsdocumented evaluation plans.

The project team performs its evaluationactivities in accordance with its documentedevaluation plans.

Strength

Activity performed 2 The project’s evaluation requirementsare developed in conjunction with thedevelopment of the system orsoftware technical requirements.

Project evaluation requirements weredeveloped in conjunction with softwaretechnical requirements and are designatedin the test plans and test scripts.

Strength

Activity performed 3 The project’s evaluation activities areplanned to minimize duplication andtake advantage of all evaluationresults, where appropriate.

There appears to be no duplication ofevaluation activities.

Strength

Activity performed 4 The project team appraises thecontractor team’s performance overthe total period of the contract forcompliance with requirements.

The project team stated that quarterlyreports of contractor deliverables were usedto appraise contractor performance, but nodocumentation was provided to support thisstatement.

Weakness

Activity performed 5 Planned evaluations are performedon the evolving software productsand services before acceptance foroperational use.

Several evaluations are performed onevolving software products and servicesbefore acceptance for operational use.

Strength




Public and Indian Housing Information Center systemCommon feature Key practice Finding RatingActivity performed 6 Results of the evaluations are

analyzed and compared to thecontract’s requirements to establishan objective basis to support thedecision to accept the product orservices or to take further action.

There is no analysis or comparison ofevaluation results to contract requirementsto establish an objective basis supportingthe decision to accept the products andservices or to take further action.

Weakness

Measurement and analysis 1 Measurements are made and used todetermine the status of the evaluationactivities and resultant products.

No measurements are made of HUD’sevaluation activities and resultant products.

Weakness

Verifying implementation 1 Evaluation activities are reviewed byacquisition organization managementon a periodic basis.

Acquisition organization managementregularly reviews evaluation activities.

Strength

Verifying implementation 2 Evaluation activities are reviewed bythe project manager on both aperiodic and an event-driven basis.

The project manager does not formallyreview evaluation activities, and the projectmanager’s reviews are not documented.

Weakness




Table 19: Software Evaluation Findings for the Real Estate Management System



HUD has no written policy for managing theevaluation of acquired software productsand services.

Weakness

Commitment to perform 2 Responsibility has been designated forevaluation activities.

Contractors and users have been assignedresponsibility for evaluation activities.

Strength


The contractor and end users areresponsible for evaluation activities.

Strength


There is no mechanism to identify neededresources and ensure that they are providedto the project. The project manager statedthat the team does not have adequateresources for performing evaluationactivities.

Weakness


Project team members performingevaluation activities have experience.

Strength

Ability to perform 4 Members of the project team andgroups supporting the softwareacquisition receive orientation on theobjectives of the evaluation approach.


Weakness

Activity performed 1 The project team performs its activitiesin accordance with its documentedevaluation plans.

The team performs its evaluation activities inaccordance with its documented evaluationplans.

Strength

Activity performed 2 The project’s evaluation requirementsare developed in conjunction with thedevelopment of the system or softwaretechnical requirements.

Project evaluation requirements aredocumented in the project plan, but nodocumentation was provided showing thatthey were developed in conjunction withdevelopment of the software technicalrequirements.

Observation

Activity performed 3 The project’s evaluation activities areplanned to minimize duplication andtake advantage of all evaluation results,where appropriate.


Strength

Activity performed 4 The project team appraises thecontractor team’s performance over thetotal period of the contract forcompliance with requirements.

The project team assesses contractor teamperformance for compliance withrequirements only during user acceptancetesting.

Observation

Activity performed 5 Planned evaluations are performed onthe evolving software products andservices before acceptance foroperational use.

Planned evaluations are performed beforesoftware releases are accepted foroperational use.

Strength




Real Estate Management SystemCommon feature Key practice Finding RatingActivity performed 6 Results of the evaluations are analyzed

and compared to the contract’srequirements to establish an objectivebasis to support the decision to acceptthe product or services or to takefurther action.

The results of evaluations are analyzed andcompared to the contract’s requirements toestablish an objective basis supporting thedecision to accept the products and servicesor to take further action.

Strength

Measurement and analysis 1 Measurements are made and used todetermine the status of the evaluationactivities and resultant products.

No measurements are made of HUD’sevaluation activities or resultant products.

Weakness


Acquisition organization managementregularly reviews evaluation activities.

Strength

Verifying implementation 2 Evaluation activities are reviewed bythe project manager on both a periodicand an event-driven basis.

Project reviews conducted by the projectmanager cover evaluation activities.

Strength




Table 20: Software Evaluation Findings for the Resident Assessment Subsystem



HUD has no written policy for managing theevaluation of acquired software products andservices. The project is following theevaluation policies in HUD’s softwaredevelopment methodology.

Observation

Commitment to perform 2 Responsibility has been designatedfor evaluation activities.

Responsibility for evaluation activities hasbeen assigned to the IT project lead.

Strength


The IT project lead and support staff areresponsible for evaluation activities.

Strength


There is no mechanism to identify resourceneeds and ensure that they are provided tothe project. According to the project manager,the project team has adequate resources forconducting evaluation activities.

Strength


Project team members performing evaluationactivities have experience and have receivedtraining.

Strength

Ability to perform 4 Members of the project team andgroups supporting the softwareacquisition receive orientation on theobjectives of the evaluationapproach.


Weakness

Activity performed 1 The project team performs itsactivities in accordance with itsdocumented evaluation plans.

The project team performs its evaluationactivities in accordance with its documentedevaluation plan.

Strength

Activity performed 2 The project’s evaluation requirementsare developed in conjunction with thedevelopment of the system orsoftware technical requirements.

Project evaluation requirements are developedin conjunction with software technicalrequirements.

Strength



Strength


The project team appraises the contractorteam’s performance over the total period ofthe contract for compliance with requirements.

Strength

Activity performed 5 Planned evaluations are performedon the evolving software productsand services before acceptance foroperational use.

Planned evaluation activities are performed onthe evolving software products and servicesbefore they are accepted for operational use.

Strength

Activity performed 6 Results of the evaluations areanalyzed and compared to thecontract’s requirements to establishan objective basis to support thedecision to accept the product orservices or to take further action.

Results of the evaluations are analyzed andcompared to the contract’s requirements toestablish an objective basis supporting thedecision to accept the software products or totake further action.

Strength




Resident Assessment SubsystemCommon feature Key practice Finding RatingMeasurement and analysis 1 Measurements are made and used to

determine the status of the evaluationactivities and resultant products.

No measurements are made of HUD’sevaluation activities and resultant products.

Weakness


Acquisition organization management doesnot review evaluation activities on a regularbasis.

Weakness

Verifying implementation 2 Evaluation activities are reviewed bythe project manager on both aperiodic and an event-driven basis.

The project manager stated that while hemeets with his staff daily to discuss the statusof the project, meeting results and action itemsare not documented.

Weakness




Table 21: Software Evaluation Findings for HUD’s Central Accounting and Program System



HUD has no written policy for managing theevaluation of acquired software productsand services. The project is following theevaluation policy in HUD’s softwaredevelopment methodology.

Observation


Responsibility for evaluation activities hasbeen assigned to the project lead.

Strength


The project lead and support staff areresponsible for evaluation activities.

Strength


There is no mechanism to identify neededresources and ensure that they are providedto the project. The project manager statedthat the team does not have adequateresources for performing evaluationactivities.

Weakness

Ability to perform 3 Individuals performing evaluationactivities have experience or receivetraining

Project team members performingevaluation activities have experience andhave received training.

Strength


Members of the project team and groupssupporting the software acquisition did notreceive orientation on the objectives of theevaluation approach.

Weakness


There is no evaluation plan; therefore, theactivities are not performed in accordancewith it.

Weakness


Project evaluation requirements weredeveloped in conjunction with softwaretechnical requirements and are designatedin the statement of work.

Strength

Activity performed 3 The project’s evaluation activities areplanned to minimize duplication andtake advantage of all evaluation results,where appropriate.


Strength

Activity performed 4 The project team appraises thecontractor team’s performance over thetotal period of the contract forcompliance with requirements.

The project team appraises the contractorteam’s performance over the total period ofthe contract for compliance withrequirements.

Strength


Planned evaluation activities are performedon the evolving software products andservices before they are accepted foroperational use.

Strength

Activity performed 6 Results of the evaluations are analyzedand compared to the contract’srequirements to establish an objectivebasis to support the decision to acceptthe product or services or to takefurther action.

Results of the evaluations are analyzed andcompared to the contract’s requirements toestablish an objective basis to support thedecision to accept the software products orto take further action.

Strength




HUD’s Central Accounting and Program SystemCommon feature Key practice Finding RatingMeasurement and analysis 1 Measurements are made and used to


No measurements are made of HUD’sevaluation activities or resultant products.

Weakness



Weakness


The project manager stated that while hediscusses the status of the project with teammembers, meeting results and action itemsare not documented.

Weakness




Table 22: Software Evaluation Findings for the Empowerment Information System



HUD has no written policy for managing theevaluation of acquired software products andservices.

Weakness


Responsibility for evaluation activities hasbeen assigned to the project lead.

Strength


The project lead and team members areresponsible for evaluation activities.

Strength


There is no mechanism to identify neededresources and ensure that they are providedto the project. The project manager statedthat the team does not have adequateresources for performing evaluation activities.

Weakness


Project team members performing evaluationactivities have experience and have receivedtraining.

Strength


Members of the project team or otherssupporting the software acquisition didreceive orientation on the objectives of theevaluation approach.

Strength


There is no evaluation plan; therefore, theactivities were not performed in accordancewith it.

Weakness


The project’s evaluation requirements werenot developed in conjunction with thedevelopment of the software technicalrequirements.

Weakness


There is no evidence of evaluation activitiesbeing performed.

Weakness


The project team assessed contractor teamperformance for compliance withrequirements only at the end of the effort.

Weakness


No planned evaluations were performed onthe software. A user survey was conducted,and the software was rejected on the basis ofthe survey.

Weakness

Activity performed 6 Results of the evaluations areanalyzed and compared to thecontract’s requirements to establish anobjective basis to support the decisionto accept the product or services or totake further action.

No planned evaluations were conducted. Thesoftware was rejected based on the results ofthe user survey.

Weakness




Empowerment Information SystemCommon feature Key practice Finding RatingMeasurement and analysis 1 Measurements are made and used to


No measurements are made of HUD’sevaluation activities or resultant products.The cost of the user survey was reported.

Observation



Weakness


The project manager does not formallyreview evaluation activities, and the projectmanager’s reviews are not documented.

Weakness

hapter 6: Conclusions, Recommendations, and

Agency Comments


HUD’s software acquisition processes are immature and are not repeatableon a project-by-project basis because of the many weaknesses it has in thekey practices. Strong performance of the key practices is essential forachieving effective, repeatable, and lasting implementation andinstitutionalization of the four key process areas we reviewed. Currently,HUD’s success or failure in acquiring software depends largely on specificindividuals, rather than on well-defined and disciplined softwareacquisition management practices. As a result, HUD is exposed to higherrisks that software-intensive acquisition projects will not consistently meetmission requirements, perform as intended, or be delivered on scheduleand within budget.

HUD acknowledged that it has software acquisition weaknesses and iscommitted to improving its software and system acquisition processes. Weagree that HUD should begin a software process improvement effort.However, for HUD to successfully complete such an effort, its softwareprocess improvement plan must address several important issues. To becomprehensive, this plan should include the results of our review andinclude measurable goals and time frames, estimates of resourcerequirements, and time frames to reach the repeatable level. It should alsocontain steps to address the systemic weaknesses we found. Finally, theplan should include steps to address the areas we did not review and toensure that all ongoing as well as new software acquisition projects adoptprocesses and meet the requirements for the repeatable level of maturity.

To improve HUD’s software acquisition capabilities, GAO recommendsthat the Secretary of Housing and Urban Development direct the HUDChief Information Officer to develop and implement a comprehensive planfor software acquisition process improvement that is based on thesoftware capability results in this report and specifies measurable goalsand time frames, sets priorities for initiatives, estimates resourcerequirements (for trained staff and funding), and defines a processimprovement management structure.

Also, to address the systemic weaknesses mentioned above, the planshould contain steps to

• develop a comprehensive policy for the acquisition of software,• require plans for specific acquisition activities,• measure and track the status of activities performed by the software

acquisition teams, and• report regularly to management on progress and problems.

hapter 6: Conclusions, Recommendations,and Agency Comments

hapter 6: Conclusions, Recommendations, and

Agency Comments


In addition, to ensure that all aspects of software acquisition areaddressed, we recommend that the Secretary direct the CIO to

• assess HUD’s maturity in the three key process areas that could not beevaluated by GAO and include any needed improvement actions in thecomprehensive plan for software acquisition process improvement;

• ensure that all new software acquisition projects in HUD adopt processesthat satisfy at least SA-CMM level 2 requirements; and

• ensure that process improvement activities are initiated for all ongoingsoftware acquisition projects.

In providing written comments on a draft of this report, the ChiefInformation Officer (CIO) stated that HUD concurred with ourrecommendations to strengthen its software acquisition processes. TheCIO acknowledged that its software acquisition processes haveweaknesses and stated that it has an improvement project under way. Thisproject includes plans to seek contractor assistance to address theweaknesses we identified and help improve HUD’s software acquisitionprocesses. Finally, the CIO stated that the department generally agreedwith our assessment of its software acquisition practices. HUD alsoprovided additional documentation on its practices for the ResidentAssessment Subsystem. We reviewed this information and made revisionswhere appropriate and supported by the additional documentation. Thedepartment’s comments are reprinted as appendix I.

Agency Commentsand Our Evaluation

Appendix I: Comments From the Department of Housing and Urban Development


Appendix I: Comments From the Departmentof Housing and Urban Development

Appendix I: Comments From the Department of Housing and Urban Development


Appendix II: GAO Contact and Staff Acknowledgments


David G. Gill (202) 512-6250

In addition to those named above, Naba Barkakati, Suzanne M. Burns,John Christian, Barbara Collier, Tonia Johnson, Barbara Oliver, andMadhav Panwar made key contributions to this report.

Appendix II: GAO Contact and StaffAcknowledgments

GAO Contact

StaffAcknowledgments

(310306)

The first copy of each GAO report is free. Additional copies of reports are$2 each. A check or money order should be made out to theSuperintendent of Documents. VISA and MasterCard credit cards are alsoaccepted.

Orders for 100 or more copies to be mailed to a single address arediscounted 25 percent.

Orders by mail:

U.S. General Accounting OfficeP.O. Box 37050Washington, DC 20013

Orders by visiting:

Room 1100700 4th St., NW (corner of 4th and G Sts. NW)Washington, DC 20013

Orders by phone:

(202) 512-6000fax: (202) 512-6061TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. Toreceive facsimile copies of the daily list or any list from the past 30 days,please call (202) 512-6000 using a touchtone phone. A recorded menu willprovide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an e-mail message with “info” in the body to

[email protected]

or visit GAO’s World Wide Web home page at

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm• E-mail: [email protected]• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,Waste, and Abuse inFederal Programs

mailto:[email protected]

http://www.gao.gov/

http://www.gao.gov/fraudnet/fraudnet.htm