software engineering g22.2440-001€¦ · software engineering g22.2440-001 session 12 - main theme...
TRANSCRIPT
1
1
Software EngineeringG22.2440-001
Session 12 - Main ThemeSoftware Verification, Validation, and Quality
Dr. Jean-Claude Franchitti
New York UniversityComputer Science Department
Courant Institute of Mathematical Sciences
2
AgendaReviewTesting, Verification, and Validaton
Integration and System TestingStatic ConfirmationDynamic TestingTraceability MatricesAutomated Testing
Configuration ManagementSoftware Quality
Software Quality Assurance (SQA)Project MeasurementsSoftware Quality and Agile MethodsProject Management MetricsQuality and Process Standards and Guidelines
SummaryProject (Part 4) - ongoing
2
3
Summary of Previous Session
ReviewSoftware Engineering TeamsPair ProgrammingPair Programming EffectsRefactoringTest Driven Development (TDD)Distributed Development and Agile Methods Scalability Languages, Platforms, and Component InfrastructuresSummary
ReadingsIndividual Assignment #5 (ongoing)Project Part 3 (ongoing)
4
Part I
Software Testing, Verification and Validation
3
5
Testing
6
Why Test?
• No matter how well software has been designed and coded, it will inevitably still contain defects
• Testing is the process of executing a program with the intent of finding faults
• A “successful” test is one that finds errors, not one that doesn’t find errors
• Testing can “prove” the presence of faults (bugs), but can not “prove” their absence
4
7
What to Test?
• Unit test – test of small code unit written by a single programmer (or programming pair) – file, class, subroutine
• Integration test – test of several units combined to form a (sub)system, preferably adding one unit at a time
• System (alpha) test – test of a finished program (a system release) by system testers
• Acceptance (beta) test – test of a finished program by end-users or their representatives
• The larger the unit of testing when defects are detected, the harder and more expensive it is to find and correct the defects (ignoring XP flattened curve…)
8
When to Test?
Early• XPers write unit test cases before coding• Many software processes involve writing
system/acceptance tests in parallel with development
Often• Regression testing: rerun unit, integration and
system/acceptance tests– After refactoring and each iteration– Throughout integration– Before each release
5
9
Who should Test?
• Software authors should not test their own code because– Testers who don’t believe they will find faults generally
don’t find many faults (cognitive dissonance)– Testers who have to fix any faults they find don’t tend
to find very many (avoidance behavior)– Coders want code to be fault free, but effective testers
must want to find faults (conflict of interest)• However, usually authors do unit tests and often
integration tests• Separate team, preferably “independent”, usually
does system tests and/or acceptance tests
10
Development Testing Overview
Unit Testing Integration Testing System Testing
Class or ModuleTesting Package Testing Component
Testing Subsystem Testing
Positive Testing
Negative Testing
Requirements-based Testing
Alpha Testing Beta Testing
AcceptanceTesting
Types of Test Cases
Types of Testing
White Box Testing Black Box Testing
6
11
Classification of Software Testing
Software Testing
Static Testing Dynamic Testing
Black-Box Testing White-Box Testing
12
Economics of Testing
Costs associated with testing software• Planning testing process• Writing test cases• Setting up equipment• Systematically executing test cases• Follow-up on problems identified• Removing [most] defects
7
13
Economics of Testing
Costs associated with shipping software defects to customers
• Customers lose faith in developers and get very angry
• Customers can lose a great deal of money if their system goes down because of defects
• Expensive to obtain specific information about customer problems and to find and fix their bugs, possibly on-site
14
Economics of Testing
Also need to consider the loss potential of each individual project developers work on
• Quality is much more important for safety-or mission-critical software than for consumer software (e.g., video games)
• Balance the cost of testing versus the cost of software failures
8
15
Economics of Testing
To minimize costs, one goal of testing must be to uncover as many defects as possible with as little testing as possible
• It is simply impossible to test every possible input-output combination of the system– There are simply too many permutations and
combinations • Define test cases that have a high likelihood
of uncovering defects
16
Testing: The Process
Software to be tested
Test Case
Output
Test CaseGeneration
Verification
9
17
When to Stop?
Software to be tested
Test Case
Output
Enough?No
Yes
Test Coverage
18
Determining Adequacy
• Statement coverage– Statements
• Branch coverage– Both IF and ELSE
• Path coverage• Example
– Stop when 90% of statements covered
10
19
Defining a Test
• Goal – the aspect of the system being tested• Input and system state – specify the actions
and conditions that lead up to the test as well as the input (state of the world, not just parameters) that actually constitutes the test
• Expected behavior – specify how the system should respond or what it should compute, according to its requirements
20
Running a Test
• Degree of scaffolding (addition software needed just to test) depends on whether black box or white box
• Need to record each test case and its results in a testing database (include date, tester, etc.)
• Preferably automate testing
11
21
Automating Testing
• Scripts that setup and run a test suite• Compare actual and expected outputs• Difficult for GUI testing and large external
files/databases
22
Unit Testing Overview
• Unit testing is testing some program unit in isolation from the rest of the system (which often doesn’t exist yet)
• Usually the programmer is responsible for testing a unit during its implementation (even though this violates the rule about a programmer not testing own software)
12
23
Test Harness (Scaffolding)
• A test driver is supporting code and data used to provide an environment for invoking part of a system in isolation
• A stub is a dummy procedure, module or unit that stands in for another portion of a system, intended to be invoked by that isolated part of the system – May consist of nothing more than a function
header with no body– If a stub needs to return values for testing
purposes, it may read and return test data from a file, return hard-coded values, or obtain data from a user (the tester) and return it
24
Unit Testing Strategies
• Black box (specification-based) testing• White box (program-based) testing• Normally perform both (not alternatives!)
13
25
White Box Testing• Test suite constructed by inspecting the
program (code)• Look at specification only to determine
what is an error• Tests control and/or data flow paths through
unit– Complexity usually too high to extend across
unit boundaries– Sometimes applied only to individual
subroutines• Attempt to exercise all statements, all
branches, or all paths
26
14
27
White-box Testing• Example
INPUT-FROM-USER(x);
If (x <= 300) {
INPUT-FROM-FILE(BALANCE);
If (x <= BALANCE)
GiveMoney x;
else Print “You don’t have $x in your account!!”}
else
Print “You cannot withdraw more than $300”;
Eject Card;
1
2
3
4
5
6
x: 1..1000;
28
public class Math {private int number1; private int number2;public Math(int num1, int num2) {
number1 = num1; number2 = num2;}public int add() {
int number3 = number1 + number2;return number3;
}public int subtract() {
int number3 = number1 - number2;return number3;
}public int multiply() {
int number3 = number1 * number2;return number3;
}public double divide() {
double number3 = number1 / number2;return number3;
}
15
29
public class Math {private int number1; private int number2;public Math(int num1, int num2) {
number1 = num1; number2 = num2;}public int add() {
int number3 = number1 + number2;return number3;
}public int subtract() {
int number3 = number1 - number2;return number3;
}public int multiply() {
int number3 = number1 * number2;return number3;
}public double divide() {
double number3 = number1 / number2;return number3;
}
testVar1 = new Math(20, 10);
public void testadd() {// {20 + 10} == {30}int expected= 30;assertEquals(expected,testVar1.add());
}
30
public class Math {private int number1; private int number2;public Math(int num1, int num2) {
number1 = num1; number2 = num2;}public int add() {
int number3 = number1 + number2;return number3;
}public int subtract() {
int number3 = number1 - number2;return number3;
}public int multiply() {
int number3 = number1 * number2;return number3;
}public double divide() {
double number3 = number1 / number2;return number3;
}
testVar1 = new Math(20, 10);
public void testsubtract() {// {20 - 10} == {10}int expected= 10;assertEquals(expected,testVar1.subtract());
}
16
31
public class Math {private int number1; private int number2;public Math(int num1, int num2) {
number1 = num1; number2 = num2;}public int add() {
int number3 = number1 + number2;return number3;
}public int subtract() {
int number3 = number1 - number2;return number3;
}public int multiply() {
int number3 = number1 * number2;return number3;
}public double divide() {
double number3 = number1 / number2;return number3;
}
testVar2 = new Math(10, 20);
public void testdivide() {// {10 / 20} == {0.5}double expected= 0.5;assertEquals(new Double(expected),
new Double(testVar2.divide())); }
32
Black Box Testing
• Test suite constructed by inspecting the specification (requirements)
• Ideally, no access to source code being tested
• Tests unit against functional and, sometimes, extra-functional specifications
• Attempts to force behavior that doesn't match specification
17
33
34
Based on Requirements
• Could either be based on the “pure”requirements in the eyes of customer
• Or on a requirements analysis or requirements specification document
• The act of writing test cases can drive the requirements gathering process to be more complete and specific
18
35
Blackbox Approaches to System Testing
• Functional tests corresponding to customer-level requirements
• Extra-functional tests corresponding to customer-level constraints
36
Black-box Testing Example
• Discussion: MAC/ATM machine example– Specs
• Cannot withdraw more than $300• Cannot withdraw more than your account balance
Softwarex
Balance
19
37
Blackbox Approaches to System Testing
• Performance tests – evaluate response time, memory usage, throughput, device utilization, CPU time
• Stress tests – push system to or beyond its specified limits to evaluate its robustness and error handling capabilities
• Reliability tests – monitor response to representative user input, counting failures over time to measure or certify reliability
38
Blackbox Approaches to Unit Testing
• Functional testing – exercise code with valid or nearly valid input for which the expected output is known
• Exhaustive testing usually infeasible, so need way(s) to select test cases and determine when “done” testing
• Choose test cases to attempt to find different faults– Equivalence partitioning– Boundary value analysis
20
39
Equivalence Partitioning
• Assume similar inputs will evoke similar responses
• Equivalence class is related set of valid or invalid values or states
• Only one or a few inputs are selected to represent an entire equivalence class
40
Equivalence Partitioning
• Divide input domain into equivalence classes
• Might also divide output domain into equivalence classes – Need to determine inputs to cover each
output equivalence class – Attempt to cover classes of errors
21
41
Range equivalence classes
1. Arbitrary value below range 2. Arbitrary value within range 3. Arbitrary value above range
42
Set equivalence classes
• Member of set • Non-member of set
22
43
Examples
• negative, positive, and zero inputs or outputs• input or output strings within size limits, beyond
size limits, and the empty string• input or output numeric values within size limits
and beyond size limits • input files that exist or not, are readable or not, are
correct format or not• output files that exist or not, are writeable or not,
are correct format or not (if being modified)• combinations of input values that are consistent,
and combinations that are not consistent
44
Boundary Value Analysis
• Extends equivalence partitioning • In practice, more errors found at or near
equivalence class boundaries than within the classes
• Divide input domain into equivalence classes
• Again may also divide output domain into equivalence classes
23
45
Range equivalence classes
1. Value immediately below range 2. First value of range 3. Second value of range 4. Next to last value of range 5. Last value of range 6. Value immediately above range 7. Minimum and maximum values of range's
basic type
46
Scalar set equivalence classes
• Member of set • Values immediately above and below each
member of set • Minimum and maximum values of set
element's basic type
24
47
Non-scalar set equivalence classes
• Member of set • Null value (if not in set) • Other non-member value(s) of correct type
("above" and "below" each set member, "minimum" and "maximum" of type, if such concepts are applicable)
• Value of incorrect type, structurally equivalent (if not detected by compiler)
• Value of incorrect type, not structurally equivalent (if not detected by compiler)
48
Examples
• maximum negative, maximum positive, and zero inputs or outputs
• input or output strings at size limits, one character beyond size limits, the empty string, and strings of one character
• input or output numeric values at size limits and one beyond size limits
• empty input files, files with one character in them, files at maximum file size (if any)
25
49
Whitebox Approaches to Unit Testing
1. Execute all (reachable) statements 2. Execute all branches of logical decisions, including
boundaries of loops 3. Execute all (feasible) control flow paths in
combination 4. Execute all data flow paths (from each variable
definition to all its uses) • Usually applied only to individual subroutines rather
than larger unit • May be combined with black box testing if tool
available to track coverage
50
Statement Coverage
• Requires each statement in unit (or entire program) to be executed at least once
• 85% easy, 100% hard– Unreachable code– Dead code– Complex sequence
26
51
Path Coverage
• Requires each control flow path in unit to be executed at least once– Rarely applied across program units
• May be impossibleread N;SUM := 0;for I = 1 to N do
read X;SUM := SUM + X;
endfor
52
27
53
Basis Path Testing
• Cover all independent control flow paths• Generate flow chart corresponding to subroutine
logic • Each test case should traverse one edge not
covered by another test case • Guarantees execution of every statement at least
once • Set of independent paths is not unique • May have difficulty in finding parameters (and
global variable values) to force a test case - then test in combination with another path
54
Problems with Basis Path Testing
• Expensive to generate flow charts and find test suite
• Tedious and error-prone (unless automated) • Usually reserved for subroutines with high
cyclomatic complexity
28
55
Cyclomatic Complexity
• Upper bound on number of test cases needed (for basis path testing)
• Flow chart must be expanded to distinguish every simple condition (single conditional operator)
• The complexity metric is number of simple logical tests plus 1
56
29
57
58
Loop Testing
• Isolate independent paths within loop • Apply special tests to each loop type • Detect initialization, indexing,
incrementing, and bounding errors at loop limits
30
59
Simple Loops
1. Skip loop entirely 2. Only one pass through loop 3. Two passes through loop 4. m passes through loop, where m = n-1, n,
n+1 passes through loop 5. Cover branching paths within loop
60
31
61
Nested Loops
1. Too expensive to use simple loop approach 2. Apply simple loop tests on innermost loop
while holding outer loops at minimum iteration parameter
3. Apply simple loop tests to next innermost loop while holding outer loops at minimum values and inner loops at typical values
4. Repeat outward through all levels of nesting
62
Other Loops
• Concatenated loops– Treat as either simple or nested depending on
degree of dependency between the loops • Unstructured loops
– Recode the loops to be structured
32
63
Data Flow Paths
• Reconsider control flow paths • Cover path from each parameter or variable
assignment to each of its uses (before reassignment)
64
33
65
Mutation Testing
• Develop specification-based test suite • Program executes test suite correctly • Make small change in a condition • Retain syntactic correctness! • Run test suite and see if there are any
changes in results • Repeat with next condition
66
Mutations
• Change sense of comparison operator • Change arithmetic operator• Change a constant • Change which variable or array index is
used• Delete statement• Delete/insert return statement
34
67
Intuition
• If a mutation doesn't matter, then something is wrong!– Either mutated code is unnecessary – Or there's a problem with the test suite itself
• Expensive: n^2 mutants for n statements, but automatable
• Often basis for validation suites
68
Integration Testing
Integration Strategies• “Big Bang” vs. Incremental• Top-down vs. Bottom-up vs. Sandwich
35
69
“Big Bang” Approach
1. Unit test all modules 2. Link everything together and run
– Difficult to track down errors because bug could be anywhere
70
A
DCB
FE G
36
71
TestA
TestA,B,C,D,
E,F,G
TestB
TestD
TestE
TestC
TestF
TestG
A
DCB
FE G
72
Incremental Approaches
• Emphasis on testing interfaces• Put a few units together at a time
– Ideally add only one unit at a time• Limits scope of errors
37
73
Top-down integration
1. Start with top-level module 2. Move downwards through control
hierarchy 3. Whenever add new modules, perform
regression tests on old modules to make sure no new errors introduced
74
Top-down integration
• Depth-first integrates all modules in a major vertical branch (wrt caller-callee module breakdown structure)
• Breadth-first integrates all modules directly subordinate at each level (or entire layer)
• Intuition in either case is to test program in relatively natural execution mode
• Difficult when stubs must be relatively complicated to fully test other modules
38
75
TestA
TestA,B,C,D
TestA,B,C,D,
E,F,G
A
DCB
FE G
76
TestA
TestA,B,C,D
TestA,B,C,D,
E,F,G
TestB
TestC
TestD
TestE
TestC
TestD
TestF
TestG
A
DCB
FE G
39
77
Bottom-up integration
1. Start with leaf modules, so no stubs required
2. Combine into larger and larger subsystems– Move up caller-callee hierarchy to full logical
groupings– And, eventually, across logical grouping
boundaries
78
Bottom-up integration
• Drivers needed to control and handle test case input/output
• Intuition is to construct program from separately tested pieces
• Tests control aspects of program relatively late
40
79
TestE
TestF
TestG
TestB,E,F
TestC
TestD,G
TestA,B,C,D,
E,F,G
A
DCB
FE G
80
Sandwich integration
• Hybrid• Top-down for highest levels of control
hierarchy (within logical groupings and across system topology)
• Bottom-up for lower levels • Reduces need for drivers and stubs
41
81
TestA
TestB,E,F
TestA,B,C,D,
E,F,G
TestE
TestD,G
TestF
TestG
A
DCB
FE G
82
TestA
TestB,E,F
TestA,B,C,D,
E,F,G
TestB
TestD
TestE
TestC
TestD,G
TestF
TestG
A
DCB
FE G
42
83
Microsoft “Synch-and-Stabilize”
• Working product as soon as possible• Many small parallel teams work on “feature
sets”• Iterate among designing, building and
testing (and involve customers in testing)• Most important features developed and
integrated first• Integrate frequently – daily builds
(analogous to “code and fix”)
84
Milestone 1: Most critical featuresand shared componentsDesign, code, prototype
Usability testingDaily builds
Feature integrationEliminate severe faults
Milestone 2: Desirable featuresDesign, code, prototype
Usability testingDaily builds
Feature integration
Milestone 3: Least critical featuresDesign, code, prototype
Usability testingDaily builds
Feature integration and completionRelease to manufacturing
43
85
Testing Database
• What program unit was tested • Actual inputs • Expected outputs • Actual outputs • Include global variables, files, other state
(when relevant) • Who performed tests • Who was informed of errors
86
Links to Other Materials
• Configuration information (e.g., module versions, system release)
• Test scripts or programs (if any) • Scaffolding used (if any) • Bug report (if any) • Error analysis (if any) • Change request (if any) • Anything else...
44
87
Why Keep Records?
• Some parts are necessary for regression testing
• Same bug may be reported multiple times by different customers
• Track staff responsibility - who did what • Process measurement and improvement -
relate quality/quantity of tests to quality of program
88
Regression Testing After Changes
• Change is successful, and no new errors are introduced
• Change does not work as intended, and no new errors are introduced
• Change is successful, but at least one new error is introduced
• Change does not work, and at least one new error is introduced
45
89
Verification and Validation
90
Verification vs. Validation
• Verification: Are we building the product right?
• Validation: Are we building the rightproduct?
46
91
Verifying and Validating Results
• Static confirmation– Reviews, walkthroughs and inspections – Static confirmation processes improve software quality
• Dynamic testing– Based on various types of testing – Must determine who performs testing
• Traceability matrices– Traceability matrices are used throughout the life cycle – Assumes creation and maintenance of the matrices
92
Event-Driven Software Testing
47
93
Event-driven Software
Events
E1
E2
E3
E4
E5
E6
E7
E8
•Graphical-user Interfaces•Web applications•Device drivers•Network protocol implementations•Object-oriented applications•Component-based applications
94
Event-driven Software
E1
E2
E3
E4
E5
E6
E7
E8
•Ordering not predetermined•Number of permutations•Source code not always available•Environment affects behavior
48
95
CaptureTester MANUALLY performs events on the softwareTool records all user inputs and application response
INPUT/RESPONSE
TESTER INPUT
RESPONSE
APPLICATION UNDER TEST
DATABASE
TESTER
Capture/replay Tools
96
ReplayTool replays users actionsVerify application response against expected response
INPUT/RESPONSE
REPLAY INPUT
RESPONSE
APPLICATION UNDER TEST
DATABASE
COMPARE
EXPECTEDRESPONSE
Capture/replay ToolsModifications made to enhance the software
49
97
Automated Testing
98
Automated Testing
Software to be tested
Test Case
Output
Enough?No
Yes
Test Coverage
50
99
Automated Testing
Software to be tested
Test Case
Output
CoverageEvaluator
Test CaseGenerator
VerifierOR
Test Oracle
TestSpecs
100
Test CaseGenerator
RegressionTester
TestCoverageEvaluator
Test Oracle
GUI Representation
Sample Automated Testing Framework
ExecutingSoftware
Implementation:Tools (Languages/Toolkits)
GUI Implementer
INFORMALSpecifications
Test Executor
51
101
Automated Testing Sample Capabilities
• Automatic Analysis of Software– 1-2 minutes
• Automatic Test Case Generation– 1000s in minutes– Invalid inputs (outside of specified ranges)
• Automatic Verification• Automatic Replay• Automatic Instrumentation• Automatic Coverage Evaluation
102
Sample System: DART
• Daily Automated Regression Testing– ICSM 2003
• Nightly/Daily Smoke Testing• Create GUI “smoke” tests• Every night, checkout source from CVS repository• Build• And run smoke tests• E-mail Bug Reports and successful/unsuccessful
logs
52
103
GUI Ripping
Representation of GUI
AUTOMATED GENERATION
XML EFG
TEST ORACLE TEST CASES
DART PROCESS
FIX BUGS
DEVELOPERSAUTOMATED REPORTING
SETUPITERATIVE
NIGHTLY PROCESS
APPLICATION UNDER TEST - Version (i)
APPLICATION UNDER TEST – Version (i + 1)
DEVELOPMENT
AUTOMATED EXECUTION
Run Test
COVERAGE REPORT
ERROR REPORT
104
Traceability
53
105
Traceability• Traceability information is information which
helps you assess the impact of requirements change. It links related requirements and the requirements and other system representations
• Types of traceability information– Backward-from traceability Links requirements to their sources
in other documents or people– Forward-from traceability Links requirements to the design
and implementation components– Backward-to traceability Links design and implementation
components backs to requirements– Forward-to traceability Links other documents (which may
have preceded the requirements document) to relevant requirements.
106
Backwards/Forwards Traceability
Business plan Design SpecificationRequirements Document
Forward-to traceability Forward-from traceability
Backward-from traceability Backward-to traceability
54
107
Types of Traceability
• Requirements-sources traceability– Links the requirement and the people or
documents which specified the requirement• Requirements-rationale traceability
– Links the requirement with a description of why that requirement has been specified.
• Requirements-requirements traceability– Links requirements with other requirements
which are, in some way, dependent on them. This should be a two-way link (dependants and is-dependent on).
108
Types of Traceability• Requirements-architecture traceability
– Links requirements with the sub-systems where these requirements are implemented. This is particularly important where sub-systems are being developed by different sub-contractors
• Requirements-design traceability– Links requirements with specific hardware or
software components in the system which are used to implement the requirement
• Requirements-interface traceability– Links requirements with the interfaces of external
systems which are used in the provision of the requirements
55
109
Traceability Tables• Traceability tables show the relationships
between requirements or between requirements and design components
• Requirements are listed along the horizontal and vertical axes and relationships between requirements are marked in the table cells
• Traceability tables for showing requirements dependencies should be defined with requirement numbers used to label the rows and columns of the table
110
A Traceability Table
Depends-onR1 R2 R3 R4 R5 R6
R1 * *R2 * *R3 * *R4 *R5 *R6
56
111
Traceability Lists
• If a relatively small number of requirements have to be managed (up to 250, say), traceability tables can be implemented using a spreadsheet
• Traceability tables become more of a problem when there are hundreds or thousands of requirements as the tables become large and sparsely populated
• A simplified form of traceability table may be used where, along with each requirement description, one or more lists of the identifiers of related requirements are maintained.
• Traceability lists are simple lists of relationships which can be implemented as text or as simple tables
112
A Traceability List
Requirement Depends-onR1 R3, R4R2 R5, R6R3 R4, R5R4 R2R5 R6
57
113
Traceability Policies
• Traceability policies define what and how traceability information should be maintained.
• Traceability policies may include– The traceability information which should be maintained. – Techniques, such as traceability matrices, which should be used
for maintaining traceability. – A description of when the traceability information should be
collected during the requirements engineering and system development processes.
– The roles of the people, such as the traceability manager, who are responsible for maintaining the traceability information should also be defined.
– A description of how to handle and document policy exceptions– The process of managing traceability information
114
Factors Influencing Traceability Policies
• Number of requirements – The greater the number of requirements, the
more the need for formal traceability policies.• Estimated system lifetime
– More comprehensive traceability policies should be defined for systems which have a long lifetime.
• Level of organizational maturity– Detailed traceability policies are most likely
to be cost-effective in organizations which have a higher level of process maturity
58
115
Factors Influencing Traceability Policies
• Project team size and composition – With a small team, it may be possible to assess the impact
of proposed informally without structured traceability information. With larger teams, however, you need more formal traceability policies.
• Type of system– Critical systems such as hard real-time control systems or
safety-critical systems need more comprehensive traceability policies than non-critical systems.
• Specific customer requirements– Some customers may specify that specific traceability
information should be delivered as part of the system documentation.
116
Part II
Configuration Management
59
117
The process of identifying, organizing, and controlling changes to the software during development and maintenance.
• SCM is a support activity that makes technical and managerial activities more effective
•SCM operates throughout the SW life-cycle
Software Configuration Management (SCM)
118
Problems of Change
Which component ? Which version ?
• Double (or multiple) maintenance• Updates to shared data• Simultaneous update
60
119
SCM Functions
• Identification of software items and products• Definition of Baselines• Access controls• Progressing defect reports• Progressing change requests• Recording item status• Controlling releases (versions and variants)• Reporting
120
SCM Tasks
• Configuration identification• Configuration control• Status accounting• Configuration audit
61
121
Some Definitions ...
Development item not yet approved, canitem be informally changed
Configuration an approved and accepteditem (CI) deliverable, changes done
through formal changecontrol procedures
122
Typical SW Configuration Items (CIs)
• Management plans• Specifications (requirements, design)• User documentation• Test design, case and procedure specifications• Test data and test generation procedures• Data dictionaries and databases• Source code, executable code• Libraries• Maintenance documentation • Support software
62
123
Milestones and Baselines
MilestoneA milestone is the end of a stage or phase of a project at which one or more deliverables are actually delivered.
BaselinesA baseline is that collection of items which when complete indicates that a milestone in the development process has been reached.
124
Typical Baselines
Phase BaselineFeasibility studyRequirements defn. Functional baseline
SRS, Interface spec. Allocated baseline
Detailed design Design baseline
Source and Object codeUser manualsTest documents Product baseline
Installation Operational baseline
63
125
Baselines
• Baselines serve as the basis for further development
• Baselines can be changed only through formal change control procedures
• Only items that have been approved and obtained through a formal technical review are accepted into the baseline.
126
Configuration Identification
• Identify what the different baselines will consist of
• Set labelling and identification conventions for the CIs
64
127
Basic CI information
• Item identity• Baseline to which it belongs• Relationships to other items• Version• Variant
128
SCM TerminologyVersion
A SW CI having a defined set of functional capabilities.
Revisionschanges to a version to correct only errors in design logic but does not affect documented functional capabilities since none of the requirements have changed.
Variantsa variation of a version developed to run on different types of HW, or to provide slightly different facilities for different users.
65
129
Examples
1.1 1.2 1.3 1.4
successive versions
branching versions (variants)
1.1 1.2 1.3 1.4
1.3.1.1 1.3.1.2
130
Merging
• Two diverging versions may be merged to create a single new version combining both set of change requests.
• Merge operations are typically done interactively with tool assistance
66
131
SCM Terminology
Promotion of a CIA CI may be promoted from one developmental baseline to another to signify a change in a CI’s internal development state.
ReleaseA Release is used to designate certain promotions of CI’s that are distributed outside the development organization.
132
Configuration Control
• Enforces a rigorous change control mechanism
• Requires formal procedures to– request changes– carry out impact analysis– approve changes– carry out changes
67
133
Change Management Methodology• Submission of Change Request (CR)• Technical and business evaluation and
impact analysis• Approval by Change Control Board• Engineering Change Order (ECO) is
generated stating– changes to be made– criteria for reviewing the changed CI
• CI’s checked out• Changes made and reviewed• CI’s checked in
134
Change Control Board
• A group consisting of representatives of user, customer, producer.
• Responsibilities:– to approve, monitor and control baselines– to approve, monitor, and control changes– to authorise changes
• CCB concerns in change approval– technically ok solution, cost, schedule,
configuration of the whole system, user satisfaction
68
135
Software Libraries
• SW libraries provide the means for implementing SCM
• The number and kind of libraries will vary from project to project . It depends on the levels of control needed.
136
Three Kinds of Software Libraries
Dynamic library (programmer’s library)– programmer’s workspace
Controlled library (master library)– used for managing the current baseline(s)
and for controlling changes made to them
Static library (software repository)– used to archive various baselines released
for general use
69
137
Techniques for Storing Versions
• Full files• Forward Delta files• Reverse Delta files• The set of differences between two
versions is called a delta.
138
Forward Delta Files
foward delta files
+ firstversion
Vnversion
Vnversion
User
Vn+1version
Vn+1version
Vnversion
CM System
changes
70
139
Reverse Delta Files
reverse delta files
+Vn+1
new recentversion
Vn+1version
User
Vnversion
Vnversion
CM System
changes
recent version
140
Status Accounting• provides a mechanism for administrative
tracking and reporting of all SW items identified and controlled.
• Examples of Status reports:– the status of proposed changes– the status of approved changes– the baselines and the approved changes
associated with each baseline– the date when each revision of each CI was
recorded– deficiencies identified by configuration
audit
71
141
Configuration Audit
• A configuration audit establishes that product integrity has been maintained and that changes have taken place in an orderly and controlled manner.
• Audit of the SW product• Audit of SCM activities
142
Physical Configuration Audit
– Consists of determining that all items identified as being part of the configuration are present in the Product baseline
– It must also establish that the correct version and revision of each part are included in the product baseline and that they correspond to information contained in the baseline’s configuration status report.
72
143
Functional Configuration Audit
– It verifies that each CI in the product has been tested to determine that it satisfies the functions defined in the specifications or contract(s) for which it was developed.
144
Organizing for SCM
Roles:• Configuration manager
• Change Control Boardincludes representatives of
- user- customer- developer
73
145
SCM Plan
The SCM Plan is prepared in Project Initiation phase. It documents
- what SCM activities are to be done- how they are to be done- who is responsible for doing specific activities
- when they are to happen- what resources are required
146
SCM Tools
Common features of popular PC-based tools (PVCS, MS Visual SourceSafe, Clearcase, etc.):
• Support for controlling all types of files (source code as well as binary)
• Managing changes as deltas• Supporting branching and merging• Identifying and re-creating releases• Providing a project view
74
147
Intersolv PVCS Version Manager
• One of the oldest PC-based version control products
• Large installed base• A fairly rich feature set• Interfaces with other third party tools• Gateways to mainframe-based library
management systems• Comprehensive security
148
Microsoft Visual SourceSafe
• Project support• File sharing• Intuitive GUI interface• Good repository architecture • Powerful security features• Tight integration with Visual Studio
development tools
75
149
SCM Tools for Unix
• SCCS (Source Code Control System)– manages changes to text files– uses a single file (s-file) to store first version
and successive forward deltas
• RCS (Revision Control System)– manages changes to text files– uses reverse deltas to store versions
150
SW Configuration Management Plan
-- IEEE Standard 828-1990 for SCM Plan
1.Introduction
2.SCM Management2.1 Organization2.2 SCM Responsibilities2.3 Applicable policies, directives and procedures
76
151
SW Configuration Management Plan
-- IEEE Standard 828-1990 for SCM Plan
3.SCM Activities3.1 Configuration identification
3.1.1 Identifying configuration items3.1.2 Naming configuration items3.1.3 Acquiring configuration items
3.2 Configuration control3.2.1 Requesting changes3.2.2 Evaluating changes3.2.3 Approving or disapproving changes3.2.4 Implementing changes
152
SW Configuration Management Plan
-- IEEE Standard 828-1990 for SCM Plan
(3. SCM Activities)3.3 Configuration Status Accounting3.4 Configuration Audits and Reviews3.5 Interface control3.6 Subcontractor/Vendor control
4.SCM Schedules5.SCM Resources6.SCM Plan maintenance
77
153
Part III
Software Quality
154
Quality Is Hard to Pin Down
• Concise, clear definition is elusive • Often not quantifiable • Many things to many people • “You'll know it when you see it”• Most often defined as set of “ilities”
(attributes)
78
155
Good quality software has …
• Reliability • Human engineering • Understandability • Modifiability • Efficiency • Testability • Portability Notice: all related to maintenance
156
Understandability
• The ability of a reader of the software to understand its function
• Critical for maintenance
79
157
Modifiability
• The ability of the software to be changed by that reader
• Almost defines "maintainability"
158
Reliability
• The ability of the software to perform as intended without failure
• If it isn't reliable, the maintainer must fix it
80
159
Efficiency
• The ability of the software to operate with minimal use of time and space resources
• If it isn't efficient, the maintainer must improve it
160
Testability
• The ability of the software to be tested easily
• Finding/fixing bugs is part of maintenance • Enhancements/additions must also be tested
81
161
Human Engineering
• The ability of the software to be easily used (usability)
• Not easily used implies more support calls, enhancements, corrections
162
Portability
• The ease with which the software can be made useful in another environment
• Porting is usually done by the maintainer
82
163
Quality Assurance Techniques
• Reviews (walkthroughs, inspections)• Standards• Metrics • Testing
164
Inspection
• Formally review the product and identify defects
• Applied to all project documents including but not limited to source code
• Used to increase software quality and improve productivity and manageability of the development process
83
165
Inspection Roles
• The minimum roles required for an inspection are moderator/recorder, author/reader, and one other inspector (3 participants)
• Customer may or may not be present• Moderator - Ensures that the inspection
procedures are followed, and that the other inspectors perform their responsibilities for each step of the inspection process
• Recorder - Records and classifies all the defects detected at the inspection meeting, and assists the moderator in preparing inspection reports
166
Inspection Roles, cont.
• Reader - Leads the team through the product in a comprehensive and logical manner
• Author - Addresses specific questions that the reader is not able to answer, and detects defects based upon their special understanding of the product
• Inspector - All of the inspection team members are inspectors, regardless of their other roles
84
167
Inspection Process
• Define what will be inspected, • when it will be inspected, • who will do the inspection, • what data will be collected, and • what follow-up actions must be taken.
168
Moderator/AuthorFollow-up
ModeratorInspection
ModeratorPreparation
Author/ModeratorPlanning
ResponsibilityActions
85
169
Planning Phase
• Author assembles all software elements necessary for the inspection to allow easy access for the entire inspection team
• Author notifies the Project Coordinator that the product is ready for inspection
170
Planning Phase, cont.
• Minimum criteria, if applicable, includes:– Spelling and grammar checked – Code should compile clean– Compliant with standards– Each inspection item is line numbered for
easy reference by inspectors• Moderator certifies that the product to
be inspected is complete, the team is selected and notified, and all scheduling actions are accomplished
86
171
ModeratorSchedule inspection
PCAssign roles
PCSelect and notify inspection team
ModeratorReview inspection items for completeness
Project Coordinator (PC)
Select moderator from SQA group
AuthorAssemble items to be inspected
ResponsibilityActions
172
Preparation Phase
• Each member of the inspection team is provided access to the product to be inspected
• Each inspector uses the appropriate checklist and standards to review the inspection items
• Defects identified by each inspector are documented in a Defect Log
87
173
Preparation Phase, cont.
• If more than a few spelling and/or grammatical mistakes are found, then inspection may be postponed
• A redlined copy of the inspection item may also be given to the author at the end of the inspection to aid in defect location
174
ModeratorDisperse the combined Defect Log to all participants for review 24 hours prior to the meeting
ModeratorCombine individual Defect Logs in a logical order
InspectorsSend individual Defect Log to Moderator 48 hours prior to inspection meeting
InspectorsRecord identified defects on individual Defect Log
InspectorsReview all items to be inspectedResponsibilityActions
88
175
Inspection Meeting
• Formally review (walkthrough) the product, indicating previously identified defects
• Moderator, with input from the inspection team, determines the inspection disposition (Accepted, Conditional, or Re-inspect)
• PC approves re-inspections and rework efforts recommended by the inspection team
• New combined Defects Log results from the inspection meeting
176
Rules for Inspections
• Maximum time scheduled for an inspection meeting should be about 2 hours
• Start meeting on time and move it along so that it finishes on schedule
• Defer arguments over style or technique• Solutions or alternatives may be mentioned,
but avoid spending too much time on group rework
• Address the product directly, do not attack the author
89
177
Inspection Disposition
• Accepted - The product is complete, without any further verification of rework. Only a very few trivial defects that can be left to the Author to correct.
• Conditional - Conditionally accept the product, with verification of the rework by the Moderator. In this case there are a few major defects, and rework is not expected to create any substantial changes in the product
178
Inspection Disposition
• Re-inspect - Re-inspect the Author’s rework. This disposition requires that the rework be examined by the Moderator, the Author, and at least one other member of the inspection team in a reiteration of the inspection meeting. There are either a substantial number of major defects, or rework that will change the original product
90
179
ModeratorInform PC of inspection disposition
AuthorDetermine estimated rework effort and completion date
ModeratorDetermine results of the inspection
RecorderNote status of each defect and categorize valid defects
Author, InspectorsPresent and clarify defects identified
ReaderSystematically review (walkthrough) the product being inspected
ModeratorConfirm inspection preparationModeratorReview rules for inspectionsResponsibilityActions
180
Follow-up
• Ensure inspection results and decisions are properly recorded and resolved
• All inspectors must agree that the list of defects is accurate and complete
• Author corrects defects• Moderator certifies correction of all defects
(conditional disposition)• Inspection Report created and stored
91
181
ModeratorRecord metrics (e.g., Size of materials inspected; Defects by severity, class, and type; Preparation times prior to inspection; Estimates of rework effort)
ModeratorPost Inspection Report to project repository
Moderator/RecorderComplete Inspection Report
ModeratorValidate corrections made by author (conditional disposition)
AuthorCorrect or resolve all noted defectsResponsibilityActions
182
Project Measurements(Traditional SDLC)
• Team Productivity Measurement– Measure over several releases– Number of people– Number of features – Number of calendar months– Lines of code produced– Defects pre-release– Defect post-release
92
183
Project Measurements(Agile Projects)
• In an agile project, you can determine each iteration's velocity (number of user stories per iteration) and the number of outstanding defects.
• Because people on agile projects tend to perform test-first development the sheer number of defects tends to be fewer.– If the testers and the customers write user acceptance tests in advance of the
first line of code, agile projects tend to have fewer egregious defects.– Problem is obviously to keep the customer involved.
• So if you're measuring productivity, try measuring size, time, number of people, and defects.– Once you've calculated some sort of team productivity, see if you can figure
out personal productivity. Just don't forget that a person's productivity is based on their entire contribution to the product, not just the number of lines of code (or tests or whatever) they contribute.
184
Software Quality and Agile Methods• Acceptance testing• Mock objects
– Mock objects are usually regarded as a programming technique that merely supports existing methods of unit testing
– Fundamentally, mock objects enable an iterative, top-down development process that drives the creation of well designed object-oriented software.
– http://www.xprogramming.com/xpmag/virtualMockObjects.htm
– http://www.xprogramming.com/xpmag/virtualMockObjects.htm#N78
• User interface testing (HTTPUnit, Canoo)• Performance testing• Agile Project Management
– Incorporate an adaptive and highly responsive development approach with a framework for effective project and risk management
93
185
Quality and Process Standards and Guidelines
• ISO 9000• SWEBOK• ISO 15504• SEI’s Capability Maturity Model (CMM)• CMM Integration (CMMI)
186
Part IV
Conclusion
94
187
Course AssignmentsIndividual Assignments
Reports based on case studies or exercises
Project-Related AssignmentsAll assignments (other than the individual assessments) will correspond to milestones in the team project.As the course progresses, students will be applying various methodologies to a project of their choice. The project and related software system should relate to a real-world scenario chosen by each team. The project will consists inter-related deliverables which are due on a (bi-) weekly basis.There will be only one submission per team per deliverable and all teams must demonstrate their projects to the course instructor.A sample project description and additional details will be available under handouts on the course Web site.
188
Course ProjectProject Logistics
Teams will pick their own projects, within certain constraints: for instance, all projects should involve multiple distributed subsystems (e.g., web-based electronic services projects including client, application server, and database tiers). Students will need to come up to speed on whatever programming languages and/or software technologies they choose for their projects - which will not necessarily be covered in class.Students will be required to form themselves into "pairs" of exactly two (2) members each; if there is an odd number of students in the class, then one (1) team of three (3) members will be permitted. There may not be any "pairs" of only one member! The instructor and TA(s) will then assist the pairs in forming "teams", ideally each consisting of two (2) "pairs", possibly three (3) pairs if necessary due to enrollment, but students are encouraged to form their own 2-pair teams in advance. If some students drop the course, any remaining pair or team members may be arbitrarily reassigned to other pairs/teams at the discretion of the instructor (but are strongly encouraged to reform pairs/teams on their own). Students will develop and test their project code together with the other member of their programming pair.
95
189
Very eXtreme Programming (VXP)
• After teams formed, 1/2 week to Project Concept
• 1/2 week to Revised Project Concept• 2 to 3 iterations• For each iteration:
– 1/2 week to plan– 1 week to iteration report and demo
190
Very eXtreme Programming (VXP)(continued)
• Requirements: Your project focuses on two application services
• Planning: User stories and work breakdown• Doing: Pair programming, write test cases before coding,
automate testing• Demoing: 5 minute presentation plus 15 minute demo• Reporting: What got done, what didn’t, what tests show• 1st iteration: Any• 2nd iteration: Use some component model framework• 3rd iteration: Refactoring, do it right this time
96
191
Revised Project Concept (Tips)
1. Cover page (max 1 page)2. Basic concept (max 3 pages): Briefly
describe the system your team proposes to build. Write this description in the form of either user stories or use cases (your choice). Illustrations do not count towards page limits.
3. Controversies (max 1 page)
192
First Iteration Plan (Tips)• Requirements (max 2 pages):• Select user stories or use cases to implement in
your first iteration, to produce a demo by the last week of class
• Assign priorities and points to each unit - A point should correspond to the amount of work you expect one pair to be able to accomplish within one week
• You may optionally include additional medium priority points to do “if you have time”
• It is acceptable to include fewer, more or different use cases or user stories than actually appeared in your Revised Project Concept
97
193
First Iteration Plan (Tips)
• Work Breakdown (max 3 pages): • Refine as engineering tasks and assign to pairs• Describe specifically what will need to be coded
in order to complete each task• Also describe what unit and integration tests will
be implemented and performed• You may need additional engineering tasks that do
not match one-to-one with your user stories/use cases
• Map out a schedule for the next weeks• Be realistic – demo has to been shown before the
end of the semester
194
2nd Iteration Plan (Tips): Requirements
• Max 3 pages• Redesign/reengineer your system to use a
component framework (e.g., COM+, EJB, CCM, .NET or Web Services)
• Select the user stories to include in the new system– Could be identical to those completed for your 1st
Iteration– Could be brand new (but explain how they fit)
• Aim to maintain project velocity from 1st iteration• Consider what will require new coding vs. major
rework vs. minor rework vs. can be reused “as is”
98
195
2nd Iteration Plan (Tips): Breakdown
• Max 4 pages• Define engineering tasks, again try to maintain
project velocity• Describe new unit and integration testing• Describe regression testing
– Can you reuse tests from 1st iteration?– If not, how will you know you didn’t break something
that previously worked?• 2nd iteration report and demo to be presented
before the end of the semester
196
2nd Iteration Report (Tips): Requirements
• Max 2 pages• For each engineering task from your 2nd Iteration
Plan, indicate whether it succeeded, partially succeeded (and to what extent), failed (and how so?), or was not attempted
• Estimate how many user story points were actually completed (these might be fractional)
• Discuss specifically your success, or lack thereof, in porting to or reengineering for your chosen component model framework(s)
99
197
2nd Iteration Report (Tips): Testing
• Max 3 pages• Describe the general strategy you followed for
unit testing, integration testing and regression testing
• Were you able to reuse unit and/or integration tests, with little or no change, from your 1st
Iteration as regression tests?• What was most difficult to test?• Did using a component model framework help or
hinder your testing?
198
Project Presentation and Demo
• All Iterations Due• Presentation slides (optional)
100
199
Readings
ReadingsSlides and Handouts posted on the course web siteDocumentation provided with software engineering toolsTextbook chapters 13-15, 20, 26-27
Project Frameworks Setup (ongoing)As per references provided on the course Web site
AssignmentsSee Handouts: “Project (Part 3)”
200
Next Session:Quantifying Specifications Using Formal
MethodsReviewVerifying Requirements Mathematically
Using Set Theory and Logic Evaluating Agile Software EngineeringSoftware Engineering EthicsWeb Engineering TechniquesMeasuring User SatisfactionRefactoring and Anti-Patterns