Automated Requirements TraceabilityStudy of the Analyst

Presented by Jeff Holden

Advisor Alex Dekhtyar

What is requirements traceability?

“The ability to describe and follow the life of a requirement, in both a forwards and backwards direction”. [gotel]

Requirements process

Output of tracing generates Requirements Traceability Matrix (RTM)

Specifies connections between low and high level elements

Why care about tracing?

Verification & Validation (V&V/IV&V) Required for mission & safety critical

systems Test coverage analysis Change impact analysis Reverse engineering

Typical tracing process

Manual tracing Norm for industry Laborious & error-prone

Automated systems Use information retrieval methods Quick, can produce good results Mission critical systems need verified

Semi-Automated tracing

Tracing tool generates candidate RTM Analyst validates the RTM to produce a final

RTM Quicker, analyst validates rather than

creates.

Typical view on tracing quality

Precision Percent of links found that are true links.

Recall Percent of true links found.

F-# measure Harmonic mean between precision & recall Use F-2: weights recall heavier than precision

Easier for analyst to resolve errors of commission than omission.

Does better candidate RTM lead to better final RTMs?

Proposed in 2005 Initial study: 4 users

Not statistically significant Showed an interesting finding, better may

not be better.

Pilot study findings

Is high quality good?

Initial experiment David Cuddeback 35 responses Old RETRO Showed “region” trends

My additions

Expanded automated study to new RETRO Simpler, more user-friendly UI Enhanced logging capabilities

MORE DATA!!!

Conducted manual tracing study Utilized the same data set

RTM locations

RTM submissions

Region trends – low recall, low precision Low precision, low recall

Improvement of precision & recall Maintain ~same RTM size

Region trends – high recall, low precision Low precision, high recall

Focus on removing links

Improve precision, some time at cost of recall

Region trends – low recall, high precision High precision, low recall

Opposite trend, focus on adding links Increase recall, normally at cost of precision

Region trends – high recall, high precision High precision, high recall

Almost all decrease quality of final RTM

Preliminary results!!!

Good initial != Good final No consensus on “true RTM” Final RTM converge on “hotspot” Automated tools may assist in finding errors of

omission better than manual! Its hard to get good precision + recall!

Contributions (so far)

Improved experimental RETRO.NET Expanded upon experimental framework to

work with other tools & other tracing methods MORE DATA!!! (52 more data points)

Up to ~90 data points total Currently writing up & submitting early findings

Planned next steps

Work with existing IR methods, filters, and feedback mechanisms.

Determine if real methods can get “good” results

Validate findings on real IR methods in similar experimental setup

Conduct usability study on RETRO.NET

Thesis goal

Create a tracing tool that analysts can use to reliably generate quality final RTM in a efficient manner.

Questions?!?