mtat.03.243 software engineering management · 2014-03-19 · • homework 2 – solution sketch...

MTAT.03.243 / Lecture 09 / © Dietmar Pfahl 2014

MTAT.03.243

Software Engineering Management

Lecture 09:

SPI & Measurement –

Part B

Dietmar Pfahl

email: [email protected] Spring 2014


Structure of Lecture 9

• Homework 2 – Solution Sketch

• Objective vs. Subjective Measurement

• Exercise

• Standards

– Measurement Process

– Product Quality

– Process Capability/Maturity

• GQM (Goal/Question/Metric) Process

• KPI (Key Performance Indicator)

• Balanced Scorecard

• Homework 3


Homework 2: Software Process Modeling

Background information:

• Recall Exercise 2 from the lecture where you modeled a process using one of the

(equivalent) notations shown in Figure 1 below. Then perform the tasks described on

the next page.



Task T1 [3 marks]:

• Have a look at the description of ‘Scrum’ published in the Eclipse Process

Framework (EPF) Wiki at http://epf.eclipse.org/wikis/scrum/index.htm.

• Create a process model using the table notation shown in Figure 1.

• Make a note/comment, if you think, information is missing (incomplete) or

inconsistent in the process description provided by the EPF Wiki.

• Note: Marking will consider completeness and correctness of the model as

well as appropriateness of your notes/comments

http://epf.eclipse.org/wikis/scrum/index.htm


Homework 2 –

Task 1:

Solution

Sketch



Task T2 [2 marks]:

• In the model you created in Task T1, select at least three activities that are

directly connected via product-flow (i.e., one activity consumes at least one

artifact produced by another activity). Then create a graphical representation

of this subset of the Scrum process model. You can choose one of the

following three notations (and tools):

– Notation from Figure 1

– Spearmint (you find information on how to download and install

Spearmint on the course wiki – Lecture 4)

– Bizagi (you find information on how to download and install Bizagi

Process Modeler on the course wiki – Lecture 4)

• Note: Marking will consider completeness and correctness of the model and

consistency with the model developed in Task T1.


Homework 2 – Task 2: Solution Sketch



Task T3 [1 mark]:

• Explain your choice of the notation/tool in Task T2. What was the reason

why you chose it and what are the advantages and disadvantages of the

notation you chose? To get full marks, you need to define and discuss at

least two criteria that guided your decision. For example, you may use a

table notation as shown below:


Homework 2 – Task 3: Solution Sketch





• Exercise

• Standards


– Product Quality





• Homework 3


Objective vs. Subjective Measurement

• Objective Measurement

– Usually, the measurement

process can be automated

– (Almost) no random

measurement error, i.e., the

process is perfectly reliable

• Subjective Measurement

– Human involvement in the

measurement process

– If we repeat the

measurement of the same

object(s) several times, we

might not get exactly the

same measured value every

time, i.e., the measurement

process is not perfectly

reliable

Rule of Thumb:

• Subjective measures have proven to be useful – but if an objective measure is available, then it is (usually) preferable


Procedures for Subjective Measurement

• Subjective Measures usually entail a well-defined

Measurement Procedure that precisely

describes:

–How to collect the data (usually via

questionnaires on paper or online)

–How to conduct interviews

–How to review documents (software artifacts)

–In which order to assess the dimensions/items

of the data collection instrument, etc.

• Examples: ISO9000 Audit, CMMI/SPICE

Assessment, Function Points


Objective vs. Subjective Measurement

Examples:

• Subjective Measurement

– Classification of defects into severity classes

– Function Points (when counted manually)

– Software Process Assessments

• Objective Measurement

– Lines of Code

– Cyclomatic Complexity

– Memory Size

– Test Coverage


Basic Concepts in Subjective Measurement

• Construct: A conceptual object that cannot be

directly observed and therefore cannot be directly

measured (i.e., we estimate the quantity we are

interested in rather than directly measure it); for

example:

–User Satisfaction

–Competence of a Software Engineer

–Efficiency of a Process

–Maturity of an Organization

• Item: A subjective measurement scale that is used to

measure a construct

–A question on a questionnaire is an item

Construct

Item1

Itemn

.

.

.

Measurement

Instrument


Dimensionality of Constructs

• Constructs can be one-dimensional or multi-dimensional

• If a construct is multidimensional, then each dimension covers a

different and distinct aspect of the construct

–e.g., the different dimensions of customer satisfaction

Construct

Item1

Itemn

.

.

.

One-Dimensional


Likert Type Scales

• Evaluation-type

Example:

“Familiarity with and

comprehension of the

software development

environment”

Little

Unsatisfactory

Satisfactory

Excellent

• Frequency-type

Example:

“Customers provide

information to the

project team about the

requirements”

Never

Rarely

Occasionally

Most of the

time

• Agreement-type

Example:

“The tasks supported

by the software at the

customer site change

frequently”

Strongly Agree

Agree

Disagree

Strongly

Disagree


Semantic Differential Scale

• Items which include semantic opposites

• Example:

“Processing of change requests to existing systems or services:

the time that MIS staff takes until responding to change requests

received from users of existing computer-based information

systems or services.”

Slow □ □ □ □ □ □ □ Fast

Timely □ □ □ □ □ □ □ Untimely


Assigning numbers to scale responses

• Likert-Type Scales:

Strongly Agree -> 1

Agree -> 2

Disagree -> 3

Strongly Disagree -> 4

• Ordinal Scale

• But:

Often the distances between the four response categories are approximately (conceptually) equidistant and thus are treated like approximate interval scales.

• Semantic Differential Scale:

Slow □ □ □ □ □ □ □ Fast

1 2 3 4 5 6 7

• Ordinal scale, but again, often treated

as interval scales


Reliability versus Validity

Assume you measure several times the same attribute of an entity (say, complexity of a code module) and the centre point is the true (but unknown) value.

http://www.uni.edu/chfasoa/reliabilityandvalidity.htm


Reliability versus Validity

Assume you measure several times the same attribute of an entity (say, complexity of a code module) and the centre point is the true (but unknown) value.

Not reliable: too much random bias (noise) Not valid: too much systematic bias


Reliability Estimation Techniques – Classes

• Number of

administrations is

the number of times

that the same object is

measured (per

observer)

• Number of

instruments is the

number of different

but equivalent

instruments that would

need to be

administered

Number of Instruments

One Two

Number of

Administrations

(per Observer /

Rater)

One Inter-Rater

Internal

Consistency

Parallel Forms

(immediate)

Two Test-Retest Parallel Forms

(delayed)

http://www.socialresearchmethods.net/kb/reltypes.php

http://www.socialresearchmethods.net/kb/reltypes.php


Inter-Rater Agreement vs.

Internal Consistency

• Example

Book 1 Book 2

Book 4 Book 3

4 Books 2 Reviewers 1 Instrument – 4 Items

Quality

Readability

Weight

Suspense

Length

bad

little

good

much

short

light

long

heavy

1 2 3 4 5 R1

R2


Inter-Rater Agreement vs.

Internal Consistency

Example – Data Average Inter-Item Correlation

R1:

R2: Inter-rater Agreement (Readability):

R1: R: 2 – 4 – 2 – 4

R2: R: 3 – 3 – 2 – 4

Fleiss’ Kappa = 0.33

(fair agreement)

Book 1: Q: - R: 2 - S: 3 - L: 3 - W: 3

Book 2: Q: - R: 4 - S: 3 - L: 2 - W: 2

Book 3: Q: - R: 2 - S: 3 - L: 1 - W: 2

Book 4: Q: - R: 4 - S: 5 - L: 4 - W: 3

Book 1: Q: - R: 3 - S: 3 - L: 3 - W: 3

Book 2: Q: - R: 3 - S: 4 - L: 3 - W: 2

Book 3: Q: - R: 2 - S: 1 - L: 2 - W: 2

Book 4: Q: - R: 4 - S: 4 - L: 3 - W: 3

R: S: L: W: R: 1 S: 0.66 1 L: 0.51 0.64 1 W: 0.29 0.46 0.73 1

Avg = 0.55

Quality rating: 1-2-3-4 Book 1: 0 1 1 0 Book 2: 0 0 1 1 Book 3: 0 0 2 0 Book 4: 0 0 0 2





• Exercise

• Standards


– Product Quality





• Homework 3


Process Measure – Avg. Sprint Productivity

Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)

• Name: Avg. Sprint Prod. ???????? ...

• Entity: Software Sprint

• Attribute: Size Effort

• Unit:

• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)

[0, ∞)

• [ph] = TM [p] * TD [d] * WHD [h/d]

Productivity = Output / Input




• Name: Avg. Sprint Prod. ???????? ???????? ????????

• Entity: are Sprint

• Attribute: Size Effort

• Unit:

• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)

[0, ∞)

• [ph] = TM [p] * TD [d] * WHD [h/d]

Productivity = Output / Input




• Name: Avg. Sprint Prod. Sprint Count Sprint Output Sprint Input

• Entity: Dev. Process

• Attribute: Efficiency Sprint count Size

• Unit: LOC/person-hour dimensionless person-hour

• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)




• Name: Avg. Sprint Prod. Sprint Count Sprint Output Sprint Input

• Entity: Dev. Process Dev. Process Software Sprint

• Attribute: Efficiency Sprint Number Size Effort

• Unit: LOC/person-hour dimensionless LOC person-hour

• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞) [0, ∞)

• Formula (Model): ASP [loc/ph] = (SUMsprint (SO [loc] / SI [ph])) / SC [-]

– Question: Does this formula make sense?

• Note: Since a sprint has a fixed time in days (TD) and a fixed number of fulltime team-members

(TM) with a fixed number of work-hours per day (WHD), one could ’Sprint Input’ (SI) consider a

derived measure, calculated as SI [ph] = TM [p] * TD [d] * WHD [h/d]





• Exercise

• Standards


– Product Quality





• Homework 3


Measurement Process -> ISO 15939


SW Product Quality -> ISO 9126

Software Quality Model (ISO 9126 – Part 1)


SW Product Quality -> ISO 9126 (cont’d)

External and Internal Measures (ISO 9126 – Parts 2 & 3)

Proposals for measures in ISO9126 - Appendix C (-> reading materials)

Example: Maturity – internal: actual defect detection/correction during development, test adequacy, ... Maturity – external: freedom of the SW system from failures caused by defects in the SW


ISO 9126 – Example Application

Measures how many defects (faults, bugs) remain in the SW system that may emerge as future failure.

Formula: X = (abs(A1 - A2)) / B

with: A1 is the total number of predicted latent defects in the system. A2 is the total number of actually occurring failures. B is the system size.

Note: A1 is an internal measure derived from a reliability growth model.


Internal vs. External Measures


ISO 9126 –

Software Product Quality

Effectiveness relates to how capable is the software in enabling users to achieve specific goals with accuracy and completeness. Productivity relates to how capable is the software in enabling users to reach effective goals with appropriate amounts of resources. Safety relates to the software ability to operate within acceptable level of risk of harm to people, business, property or the environment. Satisfaction relates to the ability of the software product to satisfy the needs of users.

Quality in Use Measures (ISO 9126 – Part 4)


ISO 9126 – Future Developments: ISO 25000

• New series of standards currently under development

– Name: Software Product Quality Requirements and Evaluation (SQuaRE - ISO 25000)

• This series of standards will replace the current ISO 9126 (and ISO 14598) series of standards

– Note: the new standard will replace the word ”metric” by “measure”

• Link: http://maisqual.squoring.com/wiki/index.php/ISO/IEC_SQuaRE

http://maisqual.squoring.com/wiki/index.php/ISO/IEC_SQuaRE


Software Process & Product Quality

process

quality

Process

Process measures

(ISO 15504)

internal

quality

attributes

external

quality

attributes

quality

in use

attributes

influences

depends on

influences

depends on

influences

Software products Effect of software

product

Internal

measures

External

measures

Quality in use

measures Context

of use


Software Process Assessment

CMMI / ISO 15504 (SPICE)

Staged

ML 1

ML2

ML3

ML4

ML5

Defines 5 maturity levels (MLs); in order to

achieve a maturity level all process areas

associated to this level, plus all process areas

associated with levels below must have a certain

minimal capability.

Continuous

A maturity profile is established

based on the capabilities of

individual process areas

PA PA PA


Causal Analysis and Resolution Organizational Innovation and Deployment

5 Optimizing

4 Quantitatively Managed

3 Defined

2 Managed

Quantitative Project Management Organizational Process Performance

Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Risk Management Integrated Project Management (for IPPD*) Integrated Teaming* Integrated Supplier Management** Decision Analysis and Resolution Organizational Environment for Integration*

Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management

1 Performed

Process Areas Level CMMI Levels and

Process Areas

(staged)

* Integrated Product/Process

Development (IPPD) –

add-on to the Engineering

processes

** Acquisition – add-on to the

Engineering processes


CMM Assessment Results (continuous)





• Exercise

• Standards


– Product Quality





• Homework 3


GQM Core Elements

GQM has three elements:

• Goals

• Questions (and associated Models)

• Measures

Goal

Q1 Q2 Q3 Q4

M1 M2 M3 ...

Implicit

models


GQM Core Elements: Goals

• GQM goal (or: Measurement

Goals) are derived from

business or improvement

goals

• A GQM goal defines

–which object is

measured,

–for which purpose,

–with respect to which

quality focus (aspect),

–from which viewpoint,

–and in which context

(environment).

GQM Goal Template


GQM Goal – Example

Analyze test process

for the purpose of characterization (understanding)

with respect to (quality aspect) effectiveness

from the viewpoint of the test team

in the environment of project X, organization Y.


GQM Question – Examples

• Goal: Analyze the test process for the

purpose of characterization with respect

to (quality aspect) effectiveness from the

viewpoint of the test team in the

environment of project X, organization Y.

• Question 1: How many failures are

detected during testing?

• Question 2: When are failures detected

(time)?

• Question 3: What types of failures are

detected?

• Question 4: How much testing effort is

spent?

• Question 5: Which test techniques/tools

are applied?

• Etc.

Test

Effectiveness,


Developing the GQM Hierarchy

Example GQM Hierarchy (incomplete):

• Question 3: What is the distribution of failures by criticality?

• Model: D = F(x, y) = x[y]/x[all], x = Measure 1.1, y = Measure 3.1,

where D: distribution of # failures per criticality class

• Measure 1.1: Failure count (ST: absolute; U: n/a; S: positive

integer; O: product version 1.0)

–Hypothesis: 120 failures

• Measure 3.1: Failure criticality (ST: nominal; U: n/a; S: {critical =

complete breakdown of system, uncritical = unable to perform

one or more of the functions F1, ..., F6, other}, O: failure report)

–Hypothesis: 5% critical failures, 15% major failures, 80%

minor failures

Q3

Goal1

M3.1 M1.1


GQM Plan

• The models and measures are

defined by answering ”What kind of

information do we need in order to

answer the questions?"

• The GQM-tree is documented in

tabular form

• Each measure is defined by:

–Name, ID

–Scale, unit, etc.

–Hypotheses


Measurement Plan – Example

• Table for tracing Measurement Plan entries to GQM Plan, Project Plan

(based on Process Model) and Data Collection Forms

GQM Plan Project Plan Data Collection Forms


The GQM Process Business Goals

Measurement Goals

(Why ?)

GQM Plan (What ?)

Measurement Data

Measurement Plan

(How ?)

Initiate Measurment Project / Set Goals

Define Measures

Define Measurement (and Analysis)

Execute Measurement (Collect data)

Analysis &

Feedback

Development Process





• Exercise

• Standards


– Product Quality





• Homework 3


KPI (Key Performance Indicator)

A set of quantifiable measures that a company or

industry uses to gauge or compare performance in

terms of meeting their strategic and operational goals.

Similar to GQM – but:

1. Stronger focus on traceability of

Goals to Business Objectives

2. Explicit targets

3. Lack of guidance on how to find

the right measures (no ’Question’

step)

4. Lack of explicit baselining (where

are we today?)

Example:


KPI – A Software Example

Context:

• Ericsson R&D, Zagreb

• Projects on CPP platform

Objective:

• Improve project performance

Key areas of concern:

• Schedule and Progress

• Resources and Cost

• Product Size and Stability

• Product Quality

• Process Performance

• Technology Effectiveness

• Customer Satisfaction

Example KPIs:

Cost Adherence: [1 – (ECost - CCost) / CCost] x 100% measured monthly ECost: expected cost (actual + forecast of remaining cost) CCost: commited cost

Fault Slip Through: [1 – FT Faults / All Faults] x 100% measured monthly FT: Function Test I&V: Intgration & Validation Test All = FT + I&V

Source: An Example of Using Key Performance Indicators for Software Development Process Efficiency Evaluation by Ž. Antolić, R&D Center Ericsson Nikola tesla d.d. (available in reading materials)


KPI – A Software Example (cont’d)

Project Benchmarking

Projects:

D1, ..., D6

DFU1, DFU2

M3





• Exercise

• Standards


– Product Quality





• Homework 3


Balanced Scorecard – 4-Box Model

• A strategic performance

management tool

• A mixture of financial and non-

financial measures each compared

to a 'target' value within a single

concise report

• Four strategic dimensions

• Financial

• Customer

• Internal Business

• Learning and Growth

More info in materials on course-web: Cobbold, I. and

Lawrie, G. (2002a). "The Development of the Balanced

Scorecard as a Strategic Management Tool".

Performance Measurement Association 2002

See also: http://www.balancedscorecard.org/ http://www.ap-institute.com/Balanced%20Scorecard.html


Balanced

Scorecard –

Strategy Map

Cause-and-Effect Logic:

• A Strategy Map highlights that

delivering the right performance

in the one perspective (e.g.

financial success) can only be

achieved by delivering the

objectives in the other

perspectives (e.g. delivering

what customers want).

• You basically create a map of

interlinked objectives.

See also: http://www.balancedscorecard.org/ http://www.ap-institute.com/Balanced%20Scorecard.html





• Exercise

• Standards


– Product Quality





• Homework 3


Homework 3: SW Measurement

Administrative information:

• This homework has to be done individually. Maximum number of marks: 6

• Submission deadline is Monday, 31-Mar-2014, at 17:00 sharp. If you don’t submit in time, you

receive a penalty as follows:

– Late delivery until Tuesday, 01-Apr-2014, at 17:00 2 marks penalty (-33%)

– Late delivery after Tuesday, 01-Apr-2014, at 17:00 6 marks penalty (-100%)

• No exceptions from the penalty-rules will be made!

• Submit your homework using the ‘Submission’ function provided on the course web-page:

https://courses.cs.ut.ee/2014/SE_Man/spring/Main/Submission

– Make sure to select the correct task, i.e., ‘Homework 3’, when you submit your file.

• IMPORTANT: Only files in PDF format will be accepted! I won’t look at files that are not PDF.

– Please name the file as follows: HW3-LastnameFirstname.pdf (of course, replacing

‘Firstname’ and ‘Lastname’ with your own names)

– Please make sure that you have put your name somewhere at the top of the first page of

the file you are submitting.


Homework 3: SW Measurement

• Task T1 [1 mark]

– Understand

measurement

definitions

• Task T2 [1.5 marks]

– Develop a measure


• Task T4 [1.5 marks]


– Subjective Meas. &

Reliability

Tasks 3 and 4 are

about ’Abstraction

Sheets’


Next Lecture

• Topic:

– Lean and Flow-based (KANBAN) Principles and

Processes - Part A

• For you to do:

– Start working on Homework 3 – Deadline:

– Continue working on Project

– Optional: Read the articles posted on course wiki (Lecture 9)

mtat.03.243 software engineering management · 2014-03-19 · • homework 2 – solution sketch...

Documents