four mega-challenges facing the commercial “software quality movement”: definition, composition,...

Four Mega-Challenges Facing The Four Mega-Challenges Facing The Commercial “Software Quality Movement”:Commercial “Software Quality Movement”: Definition, Composition, Certification, and Commercialization and Definition, Composition, Certification, and Commercialization and Return-On-InvestmentReturn-On-Investment

Jeffrey M. VoasJeffrey M. VoasChief ScientistChief Scientist

Definitional ProblemDefinitional Problem

Ask the CIO

A: What do you mean by that?A: What do you mean by that?

Q: Are you interested in quality software?Q: Are you interested in quality software?

Q: Are you interested in software quality?Q: Are you interested in software quality?

A: Yes.A: Yes.

What is Software Quality?

Subjective term that produces confusion among Subjective term that produces confusion among most most software engineering professionalssoftware engineering professionals

IEEE

• ““Totality of features of a software product that Totality of features of a software product that bears on its ability to satisfy given needsbears on its ability to satisfy given needs.” .” [Source: IEEE-STD-729][Source: IEEE-STD-729]

• ““Composite characteristics of software that Composite characteristics of software that determine the degree to which the software in determine the degree to which the software in use will meet the expectations of the use will meet the expectations of the customercustomer.”.” [Source: IEEE-STD-729][Source: IEEE-STD-729]

Three High-Level AttributesThree High-Level Attributes

Quality Software

Reliable/Accurate(integrity)

Secure/private

Timeliness

High-Level AttributesHigh-Level Attributes

Reliable/Accurate(integrity)

Secure/private

Timeliness

Problem: Intuitive, but not formal

Quality Software

Lower-Level AttributesLower-Level Attributes

Reliable/accurate

Secure/private

Timeliness

reliability security performanceavailabilityprivacy

fault tolerance fault tolerance

confidentiality

intrusion tolerancetestability

Non-functional attributes (“ilities”)

Quality Software

Functional attributes

Software Quality (or Quality Software)Software Quality (or Quality Software) must be must be viewed/defined as some combination of: (1) the viewed/defined as some combination of: (1) the degree to which the degree to which the functionalfunctional requirements are met, requirements are met, as well as,as well as, (2) the degree to which the (2) the degree to which the non-functionalnon-functional requirements are met.requirements are met.

Position StatementPosition Statement



confidentiality



+

Functional attributes

Software QualitySoftware Quality then is then is somesome combination of the following functional and non- combination of the following functional and non-functional attributes:functional attributes:

Reliability [R], Performance [P], Safety [Sa]Reliability [R], Performance [P], Safety [Sa]Fault Tolerance [F], Security [Se], Availability [A]Fault Tolerance [F], Security [Se], Availability [A]Testability [T], and Maintainability [M]Testability [T], and Maintainability [M]

Position StatementPosition Statement

Software QualitySoftware Quality can also be viewed as some combination of the can also be viewed as some combination of the previous attributes PLUS:previous attributes PLUS:

Scalability, Usability, Sustainability, Survivability,Scalability, Usability, Sustainability, Survivability,Interoperability, Extensibility, Reusability, Interoperability, Extensibility, Reusability, Readability, etc.Readability, etc.

However ….However ….

QQ = = aaR + R + bbP + P + ccF + F + ddSa + Sa + eeSe + Se + ffA + A + ggT + T + hhMM

where where a, b, c, d, e, f, g,a, b, c, d, e, f, g, and and hh are units of quantitative are units of quantitative or qualitativeor qualitative measures of a particular attribute. measures of a particular attribute.

Eight in an Equation?Eight in an Equation?

Key ProblemsKey Problems

• The equation cannot be linear, since the The equation cannot be linear, since the

units of measure for each attribute units of measure for each attribute cannot be standardized (the apples and cannot be standardized (the apples and oranges problem).oranges problem).

• ddSa = Sa = QQ – ( – (aaR + R + bbP + P + ccF + F + eeSe + Se + ffA + A + ggT + T + hhM)M)

• Most “ilities” are not quantifiably Most “ilities” are not quantifiably measurable.measurable.

• Reliability, Availability, and Performance are Reliability, Availability, and Performance are measurable (via testing).measurable (via testing).

For Example …MaintainabilityFor Example …Maintainability

• Size, defect density, amount of testing, T, Size, defect density, amount of testing, T,

R, cohesion, coupling, documentation, R, cohesion, coupling, documentation, complexity, depth of inheritance, number complexity, depth of inheritance, number of objects, testing infrastructure, mean-of objects, testing infrastructure, mean-time-to-repair, experience of time-to-repair, experience of maintenance personnel as well as their maintenance personnel as well as their domain knowledge, existence of impact domain knowledge, existence of impact analysis tools, etc., all impact M.analysis tools, etc., all impact M.

• Q: So how can you assign a single Q: So how can you assign a single numerical score for M?numerical score for M?

SecuritySecurity

• The level of security of an information system is a The level of security of an information system is a

function of the partially unknown threat space, that function of the partially unknown threat space, that changes by the minute.changes by the minute.

• Q: So how can you assign a single numerical score for Se?Q: So how can you assign a single numerical score for Se?

• A: You can assess, for a bounded set of anticipated A: You can assess, for a bounded set of anticipated threats, how the system will respond to those, e.g., 100 threats, how the system will respond to those, e.g., 100 known threats, 50 mitigated.known threats, 50 mitigated.

• A: Or you could measure the percentage of patches that A: Or you could measure the percentage of patches that are installed based on the number that need to be, and are installed based on the number that need to be, and then test to make sure those installed work. Such then test to make sure those installed work. Such information could also be used to give security “a score” information could also be used to give security “a score” but once again, that is only a score based on known but once again, that is only a score based on known threats and available patches.threats and available patches.

It is more difficult to directly measure

the quality of software than to achieve quality.

It is more difficult to directly measure

the quality of software than to achieve quality.

The “Culprit” PhenomenonThe “Culprit” Phenomenon

Without a Numerical Quality Equation, But With a Way to

Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define

Quality Goals

Without a Numerical Quality Equation, But With a Way to

Discuss the Attributes of Quality Software,and Therefore With a Means for Industry to Define

Quality Goals

So Where Does That Leave Us?So Where Does That Leave Us?

Composition ProblemComposition Problem

Two Software Components

has the following properties:

((aaR, R, bbP, P, ccF, F, ddSa, Sa, eeSe, Se, f fA, A, ggT, T, hhM)M) has the following properties:

(i(iR, R, jjP, P, kkF, F, llSa, Sa, mmSe,Se, n nA, A, ooT, T, ppM)M)

With Attributes

Then f( ) will inherit some level of Quality from the individual components. Is that level of quality an integer? Probability? An n-tuple of values? Color coded (green red yellow)?

Key Point: The Composite Quality must represent something from which predictions of future behavior can be made.

What Have You Got?

Key ProblemsKey Problems

• It is hard enough to know, with any It is hard enough to know, with any preciseprecise accuracy, accuracy,

what the composite what the composite reliabilityreliability score will be as a result of score will be as a result of the the aa and and ii values (let alone for the non-functional values (let alone for the non-functional attributes). attributes).

• But an even greater challenge exists here. For example, But an even greater challenge exists here. For example, the the securitysecurity mechanisms in component mechanisms in component could thwart the could thwart the

performanceperformance that is built into component that is built into component ..

• Attributes are only reasonable to talk about within the Attributes are only reasonable to talk about within the context of a context of a systemsystem, i.e., it is not reasonable to talk about , i.e., it is not reasonable to talk about them and attempt to measure them as standalone them and attempt to measure them as standalone component properties. Their component properties. Their eventual target eventual target environmentsenvironments must weighed into their individual must weighed into their individual assessments.assessments.

EnvironmentEnvironment

Reliable/accurate

Secure/private

Timeliness



confidentiality



QualityOperational environment

In Search of a Calculus or Calculi for Predicting How a Composite

System Will Behave in the Future in a Specific Environment

In Search of a Calculus or Calculi for Predicting How a Composite

System Will Behave in the Future in a Specific Environment


Product Certification and Product Certification and Software Engineering StandardsSoftware Engineering StandardsTo Aide the Composition ProblemTo Aide the Composition Problem

Standardized Parts?Standardized Parts?

Ideally, it is a line in the sand from which a certificate of compliance can be written.

Ideally, it is a line in the sand from which a certificate of compliance can be written.

What is a Standard? What is a Standard?

ProsPros

Any bar or hurdle is better than no bar or Any bar or hurdle is better than no bar or hurdlehurdle

ConsCons

Possibly the developers would have done Possibly the developers would have done moremore to improve quality but now feel they to improve quality but now feel they

have a license to do have a license to do lessless..

Premise for SW Product CertificationPremise for SW Product Certification

Commercially built software should be Commercially built software should be tagged tagged

with some guarantee (or with some guarantee (or at leastat least a “warm a “warm fuzzy”) as to how good the software fuzzy”) as to how good the software

should be.should be.

Problem: Software Of Unknown Pedigree Problem: Software Of Unknown Pedigree ((SOUPSOUP))

Goal of Product Certification: Goal of Product Certification: SOSO((KKnown)nown)PP

ProcessesProcesses

ProductsProducts

PeoplePeopleAll SE standards incorporateone or more of these perspectives

All SE standards incorporateone or more of these perspectives

Three Schools of ThoughtThree Schools of Thought

1. Process: Clean Pipes, Dirty Water?1. Process: Clean Pipes, Dirty Water?

Certifying that you know how to do things correctly

does not mean that you do them

correctly!

Certifying that you know how to do things correctly

does not mean that you do them

correctly!

The IEEE Computer Society has developed a The IEEE Computer Society has developed a program to certify software engineering program to certify software engineering

professionals. This program provides professionals. This program provides formal recognition of professionals who formal recognition of professionals who

have successfully achieved a level of have successfully achieved a level of proficiency commonly accepted and valued proficiency commonly accepted and valued

by the industry.by the industry.

2. People2. People

3. Product: The Software Itself3. Product: The Software Itself

Spectrum of possibilities as to what a certificate Spectrum of possibilities as to what a certificate proclaiming that some “quantified” level of quality has proclaiming that some “quantified” level of quality has been built in could state --- it could say anything in the been built in could state --- it could say anything in the range between “Nothing” (range between “Nothing” (e.g.e.g., “here is a piece of , “here is a piece of software”, etc.) to “This software will always work perfectly software”, etc.) to “This software will always work perfectly under all conditions” (i.e., a 100% guarantee of perfection).under all conditions” (i.e., a 100% guarantee of perfection).

0% 0% confidencconfidenc

ee

100% 100% confidenceconfidence

But Problems Exist With StandardsBut Problems Exist With Standards

– Vague: Vague: Develop software that only does "good" thingsDevelop software that only does "good" things• Common sense "dos" and "don'ts" - Very watered done by voting Common sense "dos" and "don'ts" - Very watered done by voting

timetime– Disclaimers by publishing organizationsDisclaimers by publishing organizations

• Profitable to organization that publishes themProfitable to organization that publishes them– Used only if mandatedUsed only if mandated– Return-on-investment is unknownReturn-on-investment is unknown– Thwart intellectual creativity Thwart intellectual creativity

• "Protectionist" legislation"Protectionist" legislation– PaperworkPaperwork

• 2167A: 2167A: ~400~400 English words per Ada code statement English words per Ada code statement– "Old news" before being ratified"Old news" before being ratified– Relating one to another is very hardRelating one to another is very hard

• Hundreds in existenceHundreds in existence– Cannot be easily tested for complianceCannot be easily tested for compliance

• Mis-certifications are possibleMis-certifications are possible

– Different interpretationsDifferent interpretations

– Lack of fairness during certification judgment Lack of fairness during certification judgment

– So much legacy code exists that complies with So much legacy code exists that complies with no standards and therefore get excluded in no standards and therefore get excluded in heterogeneous systems, making it’s impact to heterogeneous systems, making it’s impact to the system unknown.the system unknown.

Suppose you have the following logical expression:

(A and B) or (B and C) or (A and C)

where A, B, and C are Boolean variables

To meet verification requirements for Level A software in RTCA DO178-B, you need to know the number of conditions in this statement

Condition: A Boolean expression containing no Boolean operations

How many conditions are there? 3, 4, 6, or 9

Example of “Standards” ConfusionExample of “Standards” Confusion

[Source: “Challenges in Software Aspects of Aerospace Systems”, K. Hayhurst & C.M. Holloway, Presented at the 26th Software Engineering Workshop, Greenbelt, MD, November 28, 2001]

The FAA Says …The FAA Says …

Distribution of Responses from 39 FAA Certification Authorities

0

5

10

15

20

25

30

35

40

45

3 4 6 9

% ofResponses

35.9

17.9

41.0

5.1


And the Answer is …And the Answer is …

6


(A and B) or (B and C) or (A and C) has 6 conditions

The full definition for condition is not contained in the glossary entry for that term

Part of the definition is given in the entry for decision

Decision: A Boolean expression composed of conditions and zero or more Boolean operators. A decision without a Boolean operator is a condition. If a condition appears more than once in a decision, each occurrence is a distinct condition.

ExplanationExplanation


In Need of More Precise, Less Vague, and Repeatable

Processes, for Grading The Quality of Software

In Need of More Precise, Less Vague, and Repeatable

Processes, for Grading The Quality of Software


Commercialization and Commercialization and

ROI IssuesROI Issues

Commercialization IssuesCommercialization Issues

• Proven technology? (empirical vs. anecdotal) Proven technology? (empirical vs. anecdotal)

• Prototypes? Are they Maintainable/Extensible or Prototypes? Are they Maintainable/Extensible or Trashware?Trashware?

• Scalable? Theoretical or Practical? Maturity?Scalable? Theoretical or Practical? Maturity?

• Automated? Is it a solution or standalone?Automated? Is it a solution or standalone?

• What languages/architectures does it support? What languages/architectures does it support? Fad/Lifetime?Fad/Lifetime?

• Difficult to learn? Ease of use? Time-to-market enabler Difficult to learn? Ease of use? Time-to-market enabler or disabler?or disabler?

• Client base: commercial or government? Number of Client base: commercial or government? Number of site?site?

• Evolutionary (leap frog-able) vs. Revolutionary?Evolutionary (leap frog-able) vs. Revolutionary?

• Compatible with existing technology (Compatible with existing technology (e.ge.g., Microsoft)?., Microsoft)?

• Point of Origination: University? Small business?, Large Point of Origination: University? Small business?, Large Corporation?, Government?Corporation?, Government?

• Competing foreign technologies?Competing foreign technologies?

• Process, People, and Product - oriented?Process, People, and Product - oriented?

• Which attribute(s) does it address? Measurement or Which attribute(s) does it address? Measurement or design? If design, how much of that attribute can it design? If design, how much of that attribute can it offer?offer?

• SOUP or SOKP?SOUP or SOKP?

• Does this technology self-certify Quality after use?Does this technology self-certify Quality after use?

Final ThoughtsFinal Thoughts

1. Quality is a Recipe1. Quality is a Recipe

As with food, ingredients of different types (liquids, As with food, ingredients of different types (liquids, powders, vegetables, meats, etc.) can all be mixed together. powders, vegetables, meats, etc.) can all be mixed together.

What food tastes like is a function of the ingredients and What food tastes like is a function of the ingredients and their proportions.their proportions.

Quality Software can be viewed/defined in a similar manner. Quality Software can be viewed/defined in a similar manner.

And certain ingredients overpower others.And certain ingredients overpower others.

• Software engineering standards are completely necessary Software engineering standards are completely necessary despite their limitations. They usually are a “good rule of despite their limitations. They usually are a “good rule of thumb”, but not an absolute process for achieving thumb”, but not an absolute process for achieving perfection.perfection.

• Virtually any standard beats development chaosVirtually any standard beats development chaos

• What is Missing in SE Standards?What is Missing in SE Standards?• Not technology!Not technology!• How to Implement, How to Implement, • How to gain regulatory approval given uncertain How to gain regulatory approval given uncertain

knowledge as to how judgment will be rendered,knowledge as to how judgment will be rendered,• Fairness in the certification processes,Fairness in the certification processes,• And ROI (most are anecdotes, not statistical studies)And ROI (most are anecdotes, not statistical studies)

2. Standards Beat Chaos2. Standards Beat Chaos

• Until the non-functional attributes of software components Until the non-functional attributes of software components can be graded, and assumptions about the target can be graded, and assumptions about the target environments of those components can be nailed down environments of those components can be nailed down (HUGE PROBLEM), (HUGE PROBLEM), a prioria priori certification of the quality of any certification of the quality of any software component is suspect.software component is suspect.

• Research is needed into how to compose both functional Research is needed into how to compose both functional and non-functional attributes.and non-functional attributes.

3. Only Product Certification Can Address The Composition Problem

3. Only Product Certification Can Address The Composition Problem

4. Attributes Need to Be Pre-Defined4. Attributes Need to Be Pre-Defined

• Requirements should prescribe at some level of Requirements should prescribe at some level of

granularity as to what the weights are for various “ilities”, granularity as to what the weights are for various “ilities”, as well as how much of each “ility” is desired. as well as how much of each “ility” is desired.

• But HOW?But HOW?

• Ignoring the non-functional attributes is not an option for Ignoring the non-functional attributes is not an option for high assurance and trustworthy systems! Make an high assurance and trustworthy systems! Make an attempt to discuss them with the client even if attempt to discuss them with the client even if quantification is not possible. Just get the issue on the quantification is not possible. Just get the issue on the table!table!

ww11R R ww22P P ww33F F ww44Sa Sa

ww55Se Se ww66A A ww77T T ww88MM

in order to not in order to not over-designover-design any attribute into the system. any attribute into the system.

For example, for an e-commerce application, For example, for an e-commerce application, ww44 would would

probably equal 0.0 and probably equal 0.0 and ww7 7 would also be less than something would also be less than something

like like ww22

5. Weighting is Important5. Weighting is Important

How much will you spend for increased reliability knowing that doing so will take needed, financialresources away from security or performance or …?

6. Tradeoffs6. Tradeoffs

• Security vs. PerformanceSecurity vs. Performance

• Fault tolerance vs. TestabilityFault tolerance vs. Testability

• Fault tolerance vs. PerformanceFault tolerance vs. Performance

• etc.etc.

Counterintuitive RealitiesCounterintuitive Realities

• 100% safety and 0% reliability100% safety and 0% reliability

• 100% reliability and 0% safety100% reliability and 0% safety

• 0% functionality/reliability and 100% 0% functionality/reliability and 100% securitysecurity

• 100% availability and 0% reliability100% availability and 0% reliability

• 100% availability and 0% performance100% availability and 0% performance

• 0% performance and 100% safety0% performance and 100% safety

…. is alive and well. There are still many interesting andfascinating software engineering research challenges to pursue that not only will benefit NASA, but industry at-large.

7. The Software Quality Movement …7. The Software Quality Movement …

21351 Ridgetop Circle21351 Ridgetop CircleDulles, VA 20166 USA Dulles, VA 20166 USA www.cigital.comwww.cigital.com

Jeffrey VoasJeffrey Voas

phone: 703.404.9293phone: 703.404.9293

e-mail: [email protected]: [email protected]

four mega-challenges facing the commercial “software quality movement”: definition, composition,...

Documents

formal quality software

software product

quality issues

nonfunctional requirements

ingredients measurable

definitional problem

ieee std

voas chief scientist