Information Systems Overview

Organizing Complex Projects Around Risks Arising From System Dynamic Behavior

Neil SiegelSector Vice-President & Chief Engineer

Northrop Grumman

9 March 2011

Agenda

• Scope• Hypothesis • The design-based technique• Research results• Summary

2

Scope of proposed study

This study aims to assess the efficacy of one particular method of improving the outcomes on

complicated, large-scale, software-intensive system development projects

3

“Bottom-line up front”

4

Systems are important to society . . .

Hypothesis: centralizing control of

system dynamic behavior while applying critical skills will lead to superior results

. . . but system development efforts often

fail

Assess in 1 domain

Assess in 4 additional domains

Formulate conclusions

Select scope / characteristics of systems of

interest

Systems are important to society,but many system development efforts fail

5

• What do we mean by “fail”:– Do not produce a product that meets the original specifications– Produce such a product only after taking significantly more time

and/or money than originally expected. – In the extreme, many such projects are cancelled before

completion• Failure is apparently common:

– (Glass 2001) cites data indicating that only about 16% of the system development projects that he surveyed were listed as successful by their own developers

– (Johnson 2006) cites data from the Standish Group, which describes results from a 2004 survey: just 29% of development projects succeeded

Scope of the systems of interest

• Complex emergent behavior, as described by (Rechtin 1991)

• Interactions with physical devices (physically-moving mechanisms, other time-sensitive mechanical devices, etc.)

• Stressing asynchronous stimuli (such as extraordinary high data-ingest rates, or highly-stressed communications structures)

• Extraordinarily high availability / reliability requirements• Development efforts of large size • Systems that need to display much early progress

through prototyping and re-use

6

Hypotheses

During the development phase of a large-scale, complex computer-based system, the use of a specific design-based technique that

centralizes the control of the dynamic behavior of a system will lower the density of

those defects that are attributable to unplanned adverse dynamic system behavior

7

Partitioning the work

• I propose using the design process to partition the work into different “skill bins”, so as to provide better ways to assign people to tasks.

– This allows a particular difficult task (control and management of the system’s dynamic behavior) to be partitioned away from most of the development team.

• Under current methods, the “hard” parts of the work can be disseminated into a large portion of the tasks

8

Describe system’s desired dynamic behavior, to the level of every independently-schedulable entity

Take a design decision to centralize control of system dynamic behavior into a small portion of the design and implementation (“control kernel”)

Recognize every mechanism available to a developer that could create concurrency and other forms of dynamic behavior

Designate a small team with suitable expertise as the implementers of the control kernel

Train the developers outside of those implementing the control kernel that they are not to use these mechanisms, and create ways to enforce that proscription

Provide a mechanism for specification & implementation of the threads and allowable interconnects / interactions amongst the components

Provide a mechanism that allows for a small portion of the implementation to specify and control the dynamic behavior of the system, e.g., implement the control kernel

Provide for a mechanism that supports the instrumented exercising and analysis of the threads under realistic stimuli, and do so in a way that allows the dynamic behavior to be observed and adjusted

separately from the specific functionality of the system, and can do so even prior to the availability of actual system components

Allow for the entities within the system that are designed and implemented without reference to dynamic behavior in some fashion to “inherit” a set of behavioral controls in accordance with the

parameters and algorithms implemented within the control kernel

Automatically translate the specification of the threads and allowable interconnects / interactions amongst the components (whether hardware or software or people) into an executable mechanism

How I accomplish that partitioningThe System Architecture Skeleton

9

Results

10

All 6 periods / all 4 cases

11

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11

Project YYYY period II

Project AAAA

Project BBBB

Project YYYY period I

Project YYYY period IIII

Project ZZZZ

Did not use the design-based

technique

Did use the design-based

technique

Defect density“Use of the design-based technique will reduce the density of defects attributable to errors in managing system dynamic

12

Periods I-III contractor test on a single plot – FBCB2 – problem reports attributable to unplanned dynamic system behavior – opened per month

(adjusted for peer-review and unit-test procedure changes)(time is discontinuous between periods)

Initial version 31 December 2010This version 31 December 2010

1998 2007 2009Den

sity

–op

ened

thi

s m

onth

/ 1

M S

LOC

-5

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Monthly results

UNPL

LNPL

2-s upper

2-s lower

1-s upper

1-s lower

average

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

MR

MR limit

Period II behavior is materially different – o(4x) more problem reports attributable to this cause

Project YYYY, periods I-III contractor tests, attributable problem reports by month.

behavior”

13

Periods I-III contractor test on a single plot – FBCB2 –actual cost per month divided by budgeted cost per month

(time is discontinuous between periods)

1998 2007 2009

Actu

al c

ost

per

mon

th /

bud

get

per

mon

th

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

1.25

1.3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Monthly results

UNPL

LNPL

2-s upper

2-s lower

1-s upper

1-s lower

average

Initial version 2 February 2011This version 2 February 2011

Cost performance also tracksthe dependent variable

14

Summary & interpretations

15

• The data indicate that the proposed design-based technique may in fact lead to better outcomes.

• Indicated by a materially-lower density of defects (on the order of four times lower) that were attributable to errors in controlling system dynamic behavior

Summary & interpretations

• The design-based technique considered herein only applies to projects where such centralization is possible

– The study did demonstrate that this set of projects is not vanishingly small

• This specific technique, however, is only one possible technique for creating a partitioning of a project into easy and difficult portions

– Future studies could propose and assess other such techniques.

16

Summary & interpretations

Implications for practice

• Control structures may be more important than generally recognized

• New goal for the design process

17

Questions?

18

NGIS clearance approval number: ISHQ-2011-0010

19