portugal space - ece.cmu.edukoopman/ifip_wg_10_4_sigdeb/external/01... · henrique madeira,...

Summary of SIG activity anddependability benchmarking

space

Henrique Madeira

Vice-Chair of the Special Interest Group on Dependability Benchmarking

DEI/CISUCUniversity of Coimbra

Portugal

Henrique Madeira, University of Coimbra, Portugal 39th Meeting of IFIP Working Group 10.4, Parati, Brazil, March 1-3, 2001 2

Outline

• The Special Interest Group on DependabilityBenchmarking

• Summary of SIG activities

• Dependability benchmarking space: discussion of someproblems


IFIP’s Special Interest Group (SIG) on Dependability Benchmarking

• The SIG on Dependability Benchmarking was created inJune 1999 under IFIP’s Working Group 10.4

Chair: Philip Koopman, Carnegie Mellon University, USAVice-Chair: Henrique Madeira, University of Coimbra, Portugal

• Aim: promote the research, practice , adoption, anddissemination of benchmarks for computer-relateddependability;

• Membership: participants from universities, companies,and government agencies (about 45 participants; major computer andsoftware vendors represented).


Performance Benchmarking

Typical benchmark (performance)Test that measures the performance of a system or subsystemon a well-defined task or set of tasks:

• Different measures of performance;

• The workload represents the task;

• Procedures and rules to run the workload and to interpret the results


Dependability Benchmarking

Dependability benchmark (a working definition)

A test (or set of tests) to assess measures related to the behaviorof a computer system in the presence of faults (e.g., failuremodes, error detection coverage, error latency, diagnosisefficiency, recovery time, recovery losses, etc.), supporting theevaluation of dependability attributes (reliability, availability,safety).

Workload Systemunder test

Faultload

Measurements

Otherparameters

Models

DependabilityAttributes

Direct DependabilityBenchmark Measures


Why Dependability Benchmarks?

• There is no practical way to quantify or characterize dependabilityattributes of computers or computers components;

We have analytical modeling, simulation, fault injection,robustness testing, etc.

However, dependability benchmarking must provide a practical,uniform, comparable, repeatable way of evaluating dependabilityattributes.


Why Dependability Benchmarks?

• There is no practical way to quantify or characterize dependabilityattributes of computers or computers components;

• More and more activities in our society rely on the correct service ofcomputers (dependable computing is not a niche market);

• COTS-based systems are being used in application areas requiringhigh dependability (e.g., e-comerce), which demands practical waysto evaluate dependability of these systems/components;

• Dependability benchmarks must contribute to improve thedependability of computer systems for real world applications andenvironments (not a marketing tool).


SIG goals and means• Goals

! Exchange ideas about dependability benchmarking and promote cross-pollination among researchers and practitioners;

! Document state of the art on dependability evaluation and define newterms and concepts required by the dependability benchmarking effort;

! Create lists of issues that must be resolved to advance dependabilitybenchmarking;

! Propose a path (and maybe an agenda) to a group to createdependability benchmarks.

• Means! Technical discussions among SIG members in the SIG email lists;

! Organize, meetings and workshops on Dependability Bencharking.


SIG meetings

• Boca Raton, Fl, USA, November 1, 1999(17 participants)

• San Jose, CA, USA, April 3-4, 2000(16 participants)

• Pittsburgh, PA, USA, November 9-10, 2000(12 participants)

• Next meeting: Poughkeepsie (IBM), NY, USA, May 2001.


Boca Raton meeting (Nov. 1999)

• Was the founding meeting; ambitious agenda:! Solidify charter;! Exchange position statements;! Precise SIG goals and scope;! Devise mechnisms and actions;! Define long-term agenda.

• Intense brainstorming; many, many issues addressed;

• Convergence points:

! transactional systems should be the first application area tobe addressed (although, this is a very large area...)

! Need of another meeting to converge on technical aspects.


San Jose meeting (April 2000)

Agreement on (assuming transactional systems as main target):

! The main SIG goal should be to provide a framework to evaluateand characterize the dependability of computer systems;

! Concrete problem: create a taxonomy of classes of dependability,based on idea that systems fall into different general classes withrespect to dependability requirements (Don Wilson);

! Four more areas were identifyed as prioritary:– Mine existing vendor data for dependability assessment (George Candea);– Define measures/metrics for dependability assessment (Karama);– Survey of customer needs for dependability (Ken Henderson);– Method for mining data on what dependability problems really matter in

industry (Wayne Bowers);

! Organize periodical meetings/workshops on identified areas.


Pittsburg meeting (Nov. 2000)

• Theme of the meeting: dependability classes;

• Don’s proposal was discussed and improved (LAAS) andreached general consensus;

• Other subjects discussed:! Dependability benchmarking concepts and definitions;

! Potential collaboration with the Open Group QoS Task Force

• Definition of new sub-tasks to define (in progress):! The role of fault injection (Lisa Spainhower);

! Dependability benchmarking concepts and definitions (H. Madeira)

! Problem space description (Phil Koopman).


What do we have after one and a halfyear?

• Good! We have made progress on technical aspects (depend. classes);! Technical subjects to work on are well defined (and work is

going on);! Meetings have been interesting and productive from the

technical point of view;! Active set of SIG members (about 15 active members), including

members from major vendors;

• Need to be improved:! We still are very far from a dependability benchmark framework;! Meetings and email discussions are not enough;! The image of the SIG to the outside should be stronger:! What “business” model for the dependability benchmarks?


• Direct product measures

• Indirect product measures

• Process measures

Dependability benchmarking space


Faultload

Measurements

Otherparameters

Models



• Product vs process


• Design (Design correctness; design robustness to component failure)

• Manufacturing/implementation (Manufacturing defects; coding errors)

• Deployment (Are features likely to be actually installed / used / operated properly?)

• Operations (Correct usage of system)

• Maintenance (Appropriate system maintenance)



Faultload

Measurements

Otherparameters

Models



• Product vs process • Life cycle phase


• Transactional, embedded, etc

• What is the grain?



Faultload

Measurements

Otherparameters

Models



• Application area

• Product vs process • Life cycle phase




Faultload

Measurements

Otherparameters

Models




• Product vs process • Life cycle phase • Operating environment

• How to take into account environmental features?

• Environmental stress (physical)

• People– Operation– Maintenance– Interface– Security attacks




Faultload

Measurements

Otherparameters

Models





• Target • Component

• System




Faultload

Measurements

Otherparameters

Models





• Target • User perspective• Developer

• System integrator

• End user




Faultload

Measurements

Otherparameters

Models





• Target • User perspective

• Abstraction of the real world for a given application area

• Well established experience from performance benchmarks

• Could be described by functional specification + dataprofile

• Workload




Faultload

Measurements

Otherparameters

Models






• Workload • Measures• Which direct measures?

• What about performance and price?

• Which models to assess dependability attributes?

• Which parameters (assuming parametric models)?




Faultload

Measurements

Otherparameters

Models






• Workload • Measures

• Faultload

• Probably the most difficult one.

• Which classes of faults?

• Which mixture?

• How to inject them?




Faultload

Measurements

Otherparameters

Models






• Workload • Measures

• Faultload • and more...

Benchmark representativeness (workload, faultload, measures, etc);


Conclusion (benchmark space)

Defining a framework means to make choices:

! Pick up the right application area to start;

! Use mature technologies to start (from dependabilityevaluation area);

! Representativeness is the issue;

! Improving benchmark scores must means betterproducts?

portugal space - ece.cmu.edukoopman/ifip_wg_10_4_sigdeb/external/01... · henrique madeira,...

Documents