contents 1 description of 1 description of initiative initiative 3 results: 3 results:...

ContentsContentsContentsContents

Slide 1

1 Description of 1 Description of InitiativeInitiative

3 Results:3 Results: CharacterizationCharacterization

2 Description of 2 Description of Lessons LearnedLessons Learned StudyStudy

4 Results: Lessons4 Results: Lessons LearnedLearned

5 Implications &5 Implications & Next StepsNext Steps

Forrest Shull, Fraunhofer Center for Experimental Software Engineering - Maryland

Judith Bachman, Computer Sciences Corporation

John Van Voorhis, Fraunhofer Center for Experimental Software Engineering – Maryland

Mike Stark, GSFC

John Kelly, JPL

Software Inspections at NASA:

Lessons Learned Report on

State of the Practice


Slide 2






Project Information

• This initiative

– a collaboration between JPL, GSFC, CSC, and FC-MD

– funded by the Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program

• Major objectives include:

– A lessons learned report about current state-of-the-practice in inspections at NASA

– Running of experiments to investigate tailoring and training of new inspection techniques

– Running and support of pilot studies to provide long-term support for piloting new inspection techniques


Slide 3






Lessons Learned: Process

“Inspection” defined broadly: Any technical examination process during which a product is examined for defect detection by more than just the author.

1. Analyzed existing process recommendations.

2. Contacted key personnel at centers for potential participants, then participants directly.

3. Distributed characterization questionnaire.

4. Performed semi-directed interview

- elicit processes, attitudes, experiences.

5. Analysis of qualitative data for lessons learned.


Slide 4






Lessons Learned: Subjects

• 17 subjects

– 9 GSFC & Wallops; 4 JPL; 2 GRC; 1 JSC; 1 LRC

– 7 years on average at current center

– Majority participated in >10 inspections at that center

– Projects: Flight SW; data capture; control centers; mission

planning Team size mostly <10 people; 2/3 of projects last longer

than 1 year.

– ¾ of respondents currently doing inspections.

• Not a comprehensive or random sample…

• But experienced people and representative environments.


Slide 5






Characterization Results (1)

• What do people inspect?

– Most often: requirements, high-level design, low-level design (14/17)

– Also test plans (12/17) and code (13/17) Only 1 picked code inspections as most beneficial: took

over management in mid-development. Other respondents who inspect code

develop many small projects with little mission criticality

don’t have chance to influence requirements.

– Teams tend to start inspections in the first phase where: They have input; The document is sufficiently stable and complicated

Characterization of Development Errors

Top Ranked Sources of Errors

0

2

4

6

8

10

12

14

16

1 2 3

Ranking

Co

un

t

other factors

environment problems

coding errors

interface problems

design errors

changing requirements

missing requirements

misinterpreted requirements


Slide 7







• Why do people inspect?– “Changing/missing/misinterpreted requirements”: on average

most important, most often mentioned as source of errors. “Unstable requirements” also a major development

problem.– “Coding errors” were mentioned by everyone but consistently

ranked least important.– Most important thing to look for in any phase is consistency

with the previous documents.

• How do people inspect?– Almost all knew of or trained on a process. (JPL, SSDM, RA)– 9 used a process. – Very few use checklists. – Only 8 ever collected metrics.

No correlation with size/mission criticality. Those who still do use relatively simple metrics. Simple tools for those who do: websites, Excel, Access…


Slide 8







• Do people want to inspect?

– Not at first.

– Many were convinced by being involved in effective inspections.

• No failing projects with inspections

• Some used inspections to repair projects

• Several subjects felt that all successful development required inspections.


Slide 9






Lessons Learned Results (1)

• Communication Benefit - Most important– Team cohesion: Understanding who to go to with a question.– Communication to the customer

Metrics “White box reviews” / adaptability

– Getting the right technical info to the right people.– Can survive without inspections if communication happens

anyway– Effective practice: combine inspections with other meetings

• Training Benefit– Many projects required significant time for learning, had

staffing issues– Team building: Getting on board– Cross-training, novice training

• Defect Reduction Benefit – It happens - mainly by more communication and better

trained staff.


Slide 10







• Using Perspectives during Review

– Perspective: Point-of-view and existing expertise against which a reviewer inspects a software product.

– [Mars Climate Orbiter, Mishap Investigation Board Phase I Report: had inspections but critical staff were missing]

– Everybody who had a choice looked to optimize the set of perspectives. Even teams with low formality otherwise.

– Some would reschedule if a critical perspective was missing

– Typically, no checklists but different perspectives implicitly assumed to look for different things.

– When checklists are used: “Don’t use them as a replacement for thinking.”


Slide 11







• Staffing / Mgmt.

– Rare that initial plans include process/metrics support. “Inadequate staffing” second-largest development

problem. Understaffed projects, even if they want inspection

help, have more pressing concerns.

– Functional support lacking, especially for metrics collection.

– Losing follow-up: surest way to demotivate people. Chief benefit of a more formal process is action item

tracking with suitable rigor.

– Support is necessary…

– … and worth paying for.


Slide 12






Implications and Next Steps

• Candidates for further study:

– Perspective-Based Reading Focus on consistency in requirements/design; individual

review Communication/Training: Novice and cross-training;

explicit responsibilities and expertise Perspectives: Make important perspectives explicit; go

beyond checklists

– JPL Formal Inspections Positive feedback; often formed the basis for processes Communication: Formal assignment of roles Staffing/Mgmt: Very strong on follow-up tracking

– DaimlerChrysler: inspection “sampling” techniques Staffing/Mgmt: Seems well-suited to software

acquisition and projects on tight schedule


Slide 13






Request for Participation

• We are looking for:– Civil servants to directly participate– Civil servants who are overseeing contracts to

allow/encourage contractors to participate

• Controlled experiments: Participants spend 1 day for:– Receiving training in state-of-the-art techniques that can

be taken away and used on real projects.– Receiving feedback on types of defects detected and

effectiveness of the training, and some comparison to their usual approach

• Pilot studies: Participants work with us on actual projects and:– Receive training in state-of-the-art techniques tailored for

their environment & project– Receive extended support for inspections including

data collection consultation analysis & feedback


Slide 14






Contact Info

Forrest Shull

[email protected]

301-403-8970

contents 1 description of 1 description of initiative initiative 3 results: 3 results:...

Documents