measuring and managing performance in organizations

Measuring and Managing Performance in Organizations

EECS 811: Software Project ManagementPresenter: Molly Coplen

March 3, 2009

2

Contents

• Introduction– terminology– why we measure activity, why we care

• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study• Conclusion - lessons learned

3

Management and Measurement

• Why do we care? – each organization has a purpose or a goal– we measure <parameter> in hopes of

improving <goal>.

4

Terminology

A measure is a way to appraise or determine by comparing to a standard or unit of measurement. The act or process of measuring is referred to as a measurement.

A metric is a quantitative measure of the degree to which a component, system or process posses a given characteristic or attribute.

5

Why We Measure Activity • Focus workers’ attention on the task for a

desired outcome (motivate workers).

• Provide an objective basis for decision making – “You can’t control what you can’t measure.” Tom DeMarco

– resource allocation– hold workers accountable for

performance.

6

Contents

• Introduction• Types of measures

– motivational– informational

• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Conclusion

7

Types of Measures

Motivational measurements - intended to affect the people being measured, to provoke greater expenditure of effort in pursuit of organizational goals.

Informational measurements – valued for the logistical, statistical and research information they convey, which allows better short-term management and long-term improvement of organizational processes.

8

Informational Measurement

Process refinement measurement – provides information that reveals the detailed understanding of a process are useful in redesigning the process.

Coordination measure – provides information that allows short-term (real-time) management of organizational flows and schedules.

9

Informational Measures

• More accurate information, advanced warning of upcoming problems.

• Dysfunction can arise when rewards are tied to performance.

• Aggregating measures makes motivational measurement impossible.

• Self-assessment tool, not diagnostic/ analysis tool.

• Works well when workers are internally motivated.

10

Informational MeasuresE. Nichols and G. Peterson (2007)

• Design-time metrics – identify eaknesses early in the application’s life cycle

• Deployment-time metrics –identify the amount of change, uncover patterns over time, and establish baselines for anomaly detection

• Run-time metrics – determine the application’s behavior in production environment

11

Contents• Introduction• Types of measures• Basic problems with measuring

activity– economic costs– difficult to measure some things– measurement dysfunction

• Economic models and motivation theory• Software metrics• Case-study• Conclusion – lessons learned

12

Problems with Measurement Programs

• Costs– establishing/maintaining measurement

systems– workers take short-cuts which impact the

quality, require re-work

• Some activities are difficult to measure (i.e., are we using the correct measure?)

• Potential for dysfunction

13

Metrics cost a ton of money. It costs a lot to collect them badly and a lot more to collect them well. Sure, measurement costs money, but it does have the potential to help us work more effectively. At its best, the use of software metrics can inform and guide developers, and help organizations to improve. At its worst, it can do actual harm. And there is an entire range between the two extremes, varying all the way from function to dysfunction. --- Tom DeMarco

14

Measurement Costs - Factors

• Repetitiveness of task₋ offer more opportunities for observing all phases of

agent’s work₋ permit use of statistical sampling techniques₋ inexpensive measurement enabling full supervision

• Complexity of task₋ overtaxes principal’s cognitive capabilities

• Newness of task– principal more likely to be familiar with older tasks– older tasks provide more measurement opportunity over

time

15

Measurement Costs - Factors

• Specialized knowledge required by the task

₋ difficult/expensive to find someone qualified to do measurement & interpret results

₋ typically include tasks that are pushing technology beyond boundary

• Interdependence and separability of effort₋ tasks of numerous workers tightly coupled₋ all have ability to hinder others tasks₋ impossible to distinguish how much effort each

member is contributing

16

Measurement Dysfunction“Consequences of organizational actions that interfere with the spirit of the stated intentions of the organization.”

TimeTrue performance

Measurement Indicators

Level of performance

17

Characteristics of Measurement Dysfunction

• We measure too few things or the wrong things - we reward “A” for hoping for “B”.

– key measures do not provide clear account of performance

– measured standards interfere with achievement of intended effect

– employee rewards not associated with performance of organization

18

Problems with Measurement Deffers,C and McQuaid, P . (2002)

• Basic misunderstanding what is being measured

• Incorrect use of measurement data –leads to additional costs (morale, quality, lost revenues)

• Disregard for the human factor – how the cultural change of taking measurements will affect people – and the knowledge intensity of software development

19

Problems with Measurement Ordonez, M. and Haddad, H . (2008)

• Upfront commitment to gather necessary data for useful metrics, including time, cost, and resources factors

• Developers reluctant to collect and archive data that could be (mis)used against the developer and project stakeholders

20

Contents• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation

theory– internal and external motivation– theory X and theory Y– R-H and R-M models

• Software metrics• Case-study • Conclusion – lessons learned

21

Motivation

External motivation – a tendency to react in response to promises or rewards for performance according to mutually known criteria set in advance

Internal motivation – a tendency to act as compelled by private expectations of performance based on own personal criteria

22

Theory X

Managers believe people:• Dislike work and attempt to avoid it• Must be coerced, controlled, directed,

threatened• Inherently unambitious and want to avoid

responsibility• External motivation is a stronger and more

reliable way of motivating employees•http://www.youtube.com/watch?v=nXN5s8fQvHY

http://www.youtube.com/watch?v=nXN5s8fQvHY

23

Theory Y

Managers believe people:• Expenditure of effort as natural as rest or

play• Exercise self-direction and self-control• Seek responsibility, exercise responsibility

in solving problems

Managers concerned with inspiring internal motivation and clearly communicating direction to employees

24

Internal and External Motivation

Internal motivation – Theory Y

External motivation – Theory X

Expected payback - feelings of fulfillment, gratification organizational success, identify with the organization (family)

Expected payback – promise (expectation) of reward

Considers effort on work as natural as play and rest; seek responsibility and exercise creativity in solving problems

Workers dislike work and are basically lazy

Self direction, self control to achieve objectives

Coerced, directed, threatened to achieve organizational objectives

Managers use communication, inspiration to motivate

Managers concerned with design & implementation of external motivation schemes

25

Ross – Holmstrom (R-H) Model

• Principal – manager/supervisor – motivated by profits

• Agent– worker– motivated by fondness for money, dislikes

work– possible internal motivation– effort gets more expensive as its level of

activity (output) increases

26


• Principal cannot determine the value of the agents production directly– Use measurements as a proxy of value

• Agent & principal have opposite interests– principle wants maximum value (effort) for

least cost– agent wants maximum value (money) for

minimum effort

27


• The optimal payment schedule –payment to the agent increases as revenue increases– justification for merit-pay, pay-for-performance

• Assumptions valid?– employees care about more than making

money and avoiding work– managers care about more than profit– effort/mix problem

28

Holmstrom and Milgroom

• Effort must be devoted to each task

• Internal motivation – finds pleasure to provide value to the customer– might agents be persuaded to enjoy doing

more work that is beneficial to the principal without monetary reward?

– work towards goals of the company by evoking feelings of identification, belonging

29

Holmstrom and Milgroom

• Contributions– abandoned the assumption that what is

measured is a perfect indicator of the true value produced by the agent

– true output depends jointly on effort devoted to all tasks

– value is provided in a non-additive manner as agents devote effort to tasks

– use multiple measures – have all key areas been measured

30

Designing Measurement Systems

• Comprehensiveness is desirable – must measure all critical dimensions– essential to prevent dysfunction– difficult to verify performance of knowledge workers– quality is notoriously difficult to measure

• Motivational uses of measurement– agent can not be prevented from reacting to measure– results in competitive dynamics that distorts measure– causes warning signs to be concealed– extremely difficult to achieve informational only use

31

Designing Measurement Systems

• Use measurements for information only – convince agents that measure is informational

only• promote people longer time• no sudden rising stars• reduces competition within employees

– aggregation of measurement• individual see measures at their or higher level • need procedural methods to enforce aggregation• gives manager a self assessment• managerial job shifts from evaluation to inspiration

and assistance to subordinates

32

How to Make Jobs More Measurable• Standardization

₋ almost all processes repetitive at some level₋ identify phases and establish standard

methods of execution

• Specification₋ construct detailed model of the process₋ standardize each step in a process₋ identify variances from specification

33

How to Make Jobs More Measurable• Subdivison

– decompose jobs into subtasks and group similar tasks

– grouping makes repetition & self similarity visible making measurement easier

– people doing similar activities can be assigned overseers that have specialized knowledge as workers

34

Delegatory Management

• Promote organizational intimacy– make hierarchy flatter & work groups smaller– reconfigure work spaces to promote casual

contacts between managers and workers in same team

• Promote trust– facilitate mutual trust - employer and

employees– investments that suggest permanence of

employee base such as investing heavily in training

35

Delegatory Management

• Provide better leadership– skilful expression of vision– management styles that emphasize inspiring

and trusting employees rather than analyzing and coercing

36

Internal Motivation vs External Reward

• Offer of an external reward for that which would be provided through internal motivation may be insulting and lower internal motivation

• “It’s demeaning... People are motivated by intrinsics and extrinsics and the intrinsics are much more powerful – the pride, the workmanship, the enjoyment of doing things with their colleagues....But the extrinsics shove the intrinsics aside. Say you give me an extrinsic reason to do this. I lose track of whether I’m having a good time working with my colleagues on this goal.” – DeMarco on performance measurement

37

Contents• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics

– common software metrics– factors to consider when choosing a metric– characteristics of successful programs– human factor – the programmers

• Case-study • Conclusion – lessons learned

38

What is a Software Metric?

A software metric is defined as a method of quantitatively determining the extent to which software processes, product, or project possess a certain attribute. (Daskalantonakis, 1992)

Rationale - a metrics program should lead the software organization to more disciplined processes through an efficient feedback mechanism. (Gopal, Krishnan, Mukhopadhyay, and Goldenson, 2002)

39

Common Software Metrics (1/2)

Activity Description

Cost and Effort Estimation models help plan and execute projects. Mathematical models such as Boehm’s COCOMO, Putnam’s SLIM, and Albrecht’s Function Points

Productivity Measures

Human side - team structure, tools, skills, backgrounds, environment.

Data Collection Leads to perception of being “under pressure” and “at risk” as negative data can be used against the developer.

Quality Assessment Efficiency, reliability, flexibility, portability, usability, correctness, etc.

Reliability Models Related to software failures

40

Common Software Metrics (2/2)

Metric Description

Process Focus on software development and maintenance and are used to assess people’s productivity (private metrics) or the productivity of the entire organization (public metrics).

Project Tactical and related to project characteristics and execution. Used by project managers and software developers to adjust product workflow and technical activities.

Product Measure key characteristics of the software product.

41

Common Software MetricsList of Project Metrics (1/2)

• Order of growth• Lines of code• Cyclomatic

complexity• Function points• Code coverage• Coupling• Cohesion

requirements size

• Application size• Cost• Schedule• Productivity• Number of software

developers

42

Common Software MetricsList of Product Metrics (2/2)

• Specification quality metrics• System size metrics• Architectural metrics• Length metrics• Complexity metrics• Testing effectiveness

43

Cem Kaner’s Factors to Consider When Choosing a

Metric (1/2)1. What is the purpose of the metric? 2. What is the scope of the measure?3. What attribute are you trying to are you

trying to measure?4. What is the attribute’s natural scale?5. What it the attribute’s natural variability?6. What instrument are you using to

measure the attribute, and what reading do you take from the instrument?

44

Cem Kaner’s Factors to Consider When Choosing a

Metric (2/2)7. What is the instrument’s natural scale?8. What is the reading’s natural variability

(measurement error)?9. What is the attribute’s relationship to the

instrument? 10. What are the natural and foreseeable side

effects of using this instrument?

- Cem Kaner (2002) as cited by Dekkers, C.A. and McQuaid, P.A. (2002)

45

Basic Guideline – Selecting a Metric

Dekkers and McQuaid• Research and analyze your organization’s

measurement goals• Decide what questions will tell you

whether you are progressing towards the goals.

• Select the appropriate measures to answer the questions.

• Recognize intentional and unintentional side effects of the chosen measures and metrics.

46

Goal Questioning Metric (GQM)Victor Basili and David Weiss

1. Goals – define the objectives (scope and purpose) to be addressed by measurement

2. Question – defines what quantifiable information is required to determine whether there is progress toward the goal(s)

3. Metric – defines the particular attributes, definitions, observation (collection) frequency for measurement data, measurement theory, statistical design, and applicability of measurement data.

47

Characteristics of Highly Successful Measurement

Programs (1/4)

1. Set solid measurement objectives and plans

– aim to achieve firm objectives– implement as a development project

2. Make measurement part of the process– measure key success factors– integrate data collection

48


Programs (2/4)3. Gain a thorough understanding of

measurement – use metrics to:– implement corrective actions– measure requirements, not individual

performance

4. Focus on cultural issues– involve developers in the measurement

program’s development– recognize cultural changes affects how people

view their work

49


Programs (3/4)

5. Create a safe environment to collect and report true data.

₋ “..understand the data that your people take pride in reporting. Don’t ever use it against them. Don’t even hint that you might.” Robert Grady – Hewlett Packard

₋ give people access to the measurement data₋ collect data using consistent measurement

definitions

50


Programs (4/4)6. Cultivate a predisposition to change

₋ respond to the measurement data7. Develop a complementary suite

(dashboard) of measures

-- Dekkers, C.A. and McQuaid, P.A. (2002)

51

Hewlett-Packard (HP) (1/3)

• Goal - to improve software project management, team productivity, and software quality– achieved in the short term for individual

development projects

• Metrics categorized– primitive – directly observed (e.g., total

development time for a project, lines of code)– computed – mathematical aggregations of two

or more primitive metrics

52


• Metrics for project scheduling, cost of defects, work load, and project control– average fixed defects/working day– average engineering hours/fixed defect– average reported defects/working day– defects/testing time– percent overtime: average overtime per week– phase: engineering months/total engineering

months

53


• End product quality metrics– defects/KNCSS (thousand non-comment source

statements)– defects/lines of documentation not included in

the program source code

• Testing effectiveness metrics– defects/testing time

• Testing coverage metrics– branches covered/total branches

54

Motorola (1/2)

Goal Metric Program

Improve project planning

Schedule and Effort Estimation ActivitySEA = Actual project duration/estimated project durationEEA = Actual project effort/estimated project effort

Increase defect containment

Total Defect Containment EffectivenessTDCE = # of prerelease defects /( # of prerelease defects + number of post release defects)Phase i Containment EffectivenessPCEi = # of phase i errors/# of phase I errors + # pf phase i defects

Increased reliability

Failure Rate (FR)FR = # of failures /execution time

55

Motorola (2/2)

Goal Metric Program

Decrease software defect density

In Process FaultsIPF = in process faults caused by incremental software and development/assembly – equivalent delta source size

Total Released Defects TRD total = # of released defects/assembly – equivalent total source size

Customer Found Defects (CFD) totalCFD Total = # of Customer found defects / Assembly – equivalent source size

Improve customer service

NOP = Total new post release problems opened during the month

TOP = Total post release problems that remain open at the end of the month.

56

Why Programmers Avoid Metrics

• Problems with data collection– suppress partial information– scripting– entering false data– illusion of compliance

• Reasons for push-back– maintaining image– resistance to standardization– begrudging compliance– metrics are not representative of effort– product and process compliance

57

Why Programmers Avoid Metrics

• Metrics Acceptance Model – social influences – organizational usefulness– personal usefulness– available resources– self-efficacy– compatibility– ease of use– attitude

58

Contents

• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study

– Financial Software Solutions• Conclusion - lessons learned

59

Case: Financial Software Solution

Financial Group

Finance Bank Financial Software Solutions

60


• Primary business of FSS is the development of IT systems and services for Financial Group– also sells IT systems to other financial

institutions in Europe– expertise in developing banking, insurance,

mortgage and financing applications• 850 employees • Projects – small and short-term, 3-5 people

of the project team, 6-12 months

61

Case: Financial Software Solution• Projects

– small and short-term, 3-5 people, duration of 6-12 months

– major projects, 10-20 people, duration of years

• 4 development divisions– headed by a Senior Vice President– divisions composed of departments headed by

Vice Presidents, staffed by 20-25 people divided into 5 projects

• Project managers manage regular projects, VP manage high-profile projects

62


• FSS an independent subsidiary delivering IT services for Financial Group

• FSS wanted to stay competitive with others – Software Process Improvement (SPI) strategy for staying competitive– project given high profile – VP as a PM and

other VPs were team members– idea was each division have a representative –

few were actually active

63


• Key stakeholders– CEO – sponsor - goal – improve efficiency 10%– Vice Presidents– Project managers – provide data on their

project to the metrics program– SPI project manager – 4 during the life of the

project– Finley – VP and member of SPI improvement

group – key to defining the metrics program– Linda – VP and member of SPI group

• Who is not included?

64


• CMM level 1 - Initial (chaotic, ad hoc, heroic) the starting point for use of a new process

• Approached this project as a project• Goal – improve efficiency 10%

– how measured ? undefined– how is data to be analyzed? unanswered

65


Indicators of the Metrics Program

Factor Definitions

Project productivity Resources used to develop the system relative to its size in function points

Quality Number of error reports both absolute and relative to size in function points

Adherence to schedule

Variation from agreed time of delivery both absolute and relative to the size in function points

Adherence to budget Variation in estimated use of resources

Customer satisfaction Satisfaction with the development process and the implemented solution (questionnaire)

Employee satisfaction Satisfaction with the development process (questionnaire)

66


• Principles the metrics program should adhere to:– Measurements should be relevant in relation to

process improvement and quality– Measurements should be made automatic

whenever possible– Minimal cost of performing the measurement– Use of questionnaires limited– Data collected on finished projects – Results published quarterly

67


• Timeline– March 1997 – decision to implement– September 1997 – first report

• data on 3 of the 6 factors • 13 of 56 projects completed in September• report withheld from wide distribution due to

‘adherence to budget” measure• parts disseminated to developers via a “road show”

to raise awareness of the project

68


– March 1998 – second report • reported on 65% of completed projects• October – December 1997 (time lag)• data considered to be sufficiently unreliable for wide

distribution• concern with how the data will be used

• Public disclosure• One member of the team had promised

development staff that information would not be disclosed

• concerns about the customer satisfaction survey• resulted in a project to improve the metrics program

69


– Project established – August 1998 • 18 months after kick-off• signed contract to improve the metrics • goal: improve the quality of the measurement report

so much that management would publish the report• success: published report April 1999 with all 6

indicators, all projects completed 1Q 1999• function points dismissed due to problems of

inaccurate count• decided to not include a size metric and rely on the

rest of the measure to provide information on the effect

70


– Third report – April 1999• 10 projects reported complete data and data was not

reliable• misunderstanding of the definition of some fields

and other things entered• data not collected• data not maintained if changes need to be made

• no measurement report, produce a memo to management describing the problems

• bonus system introduced for divisions collecting accurate data

71


– Decision to publish – September 1999• 2 ½ years after project initiated• SVP made corrections they wanted made• released to VPs• 64% projects reported on• “commitment from management is required to reach

a better result”

72

Contents

• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study• Conclusion - lessons learned

73

Lessons Learned

• Adequate resources available• Set solid objectives and plans• Approach as a software engineering

project– apply Goal – Question – Metric model

• characterize the environment• identify management goals and develop

measurement plans• define data collection procedures• collect, analyze, and interpret data• perform postmortem analysis and interpret data• package experience

74

Lessons Learned

• Adequate resources available• Set solid objectives and plans• Approach as a software engineering

project– apply Goal – Question – Metric model

• characterize the environment• identify management goals and develop

measurement plans• define data collection procedures• collect, analyze, and interpret data• perform postmortem analysis and interpret data• package experience

75

Lessons Learned

• Start with a few indicators – stepwise approach

• Use a complementary suite of metrics• Establish incentives for compliance• Create a safe-environment for collecting

and withholding information• performance measures of an individual should be kept to

the individual• don’t withhold information

76

Lessons Learned

• Recognize the culture shift decisions are based on relevant and accurate information, not intuition

• Discuss metrics and what makes sense for the organization

• Take corrective action based on the measurement

77

Lessons Learned

• Learning cycle of a metrics program

Interpretation

Data Collection

Intervention

Inte

rven

tion

Results Knowledge

Object of Study Object

78

References• Austin, R.D. (1996), Measuring and Managing Performance

in Organizations,” Dorset House Publishing.• Basili, V. and Weiss, D. (1984), “A Methodology for

Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, 10 (11), pp 728-738.

• Daskalantonakis, M. (1992), “A Practical View of Software Measurement and Implementation Experiences with Motorola,” IEEE Transactions on Software Engineering, 18 (11), pp. 998-1010.

• Dekkers, C.A. and McQuaid, P.A. (2002), “The Dangers of Using Software Metrics to (Mis)Manage, IT Pro, 4(2), pp. 24-30.

79

References• Gopal, A., Krishnan, M.S., Mukhopadhyay, T., Goldenson, D.

(2002), “Measurement Programs in Software Development: Determinants of Success,” IEEE Transactions on Software Engineering, 28 (9), pp. 863-875.

• Iversen, J. and Mathiassen, L. (2000) “Lessons from Implementing a Software Metrics Program,” Proceedings of the 33rd Hawaii International Conference on System Sciences.

• Nichols, E. and Peterson, G. (2007), “A Metrics Framework to Drive Application Security Improvement,” IEEE Security and Privacy, March/April 2007, pp. 88-91.

• Ordonez, M. and Haddadm, H.M. (2008), ‘The State of Metrics in Software Industry,” Fifth International Conference on Information Technology: New Generations, April 2008, pp 453-458.

• Umarji, M and Seaman, C. (2008), Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement , Kaiserslautern, Germany , pp. 129-138.

measuring and managing performance in organizations

Documents