analyzing goal models – different approaches and how to choose among them jennifer horkoff 1 eric...

Analyzing Goal Models – Different Approaches and How

to Choose Among Them

Jennifer Horkoff1

Eric Yu2

1Department of Computer Science2Faculty of Information

University of Toronto, Canada

SAC’11 RE Track

Analyzing Goal Models, Horkoff & Yu 2

Goal-Oriented Requirements Engineering (GORE)

GORE has received much attention in RE research as a means of: Understanding the motivations for system requirements Helping to ensure that the right system is built

Generally, GORE frameworks allow for: Representation of stakeholder goals Goals may be assigned to an agent (stakeholder or

system) Goals may have relationships to other goals, often

describing achievement Several goal modeling frameworks

KAOS, GBRAM, AGORA, NFR, i*, Tropos, GRL, …


Goal-Oriented Requirements Engineering (GORE)

Example: Counseling Organization i* Model (Horkoff & Yu, 2009)

We can analyze the contents of goal models systematically…


Example: Qualitative, Interactive Forward Satisfaction Analysis of Goal-Oriented Models

Horkoff, Yu: Evaluating Goal Achievement in Enterprise Modeling

What is the effect of using a Cybercafe/ Portal/ Chat Room?”


Models and Analysis become Complex


Goal Model Analysis Many different analysis techniques for goal

models have been introduced: Propagate satisfaction values through the model Measure metrics over the model Apply planning techniques Run simulations Perform checks over models

Abundance of approaches is encouraging from a research perspective, but…

From a user or practitioner perspective can be confusing What are the differences? When would I use one and not another?

Limits adoption


Motivating Questions Survey of methods

What methods are available? What types of analysis questions can these methods

answer? What types of goal modeling constructs do the

procedures support? What information is needed in order to use the methods?

Analysis benefits What are some of the potential benefits of goal model

analysis in the requirements process? Mapping and Selection

What available methods can be applied to achieve which kinds of usage objectives?

How can we use this information to advise on selection?


Survey of Goal Model Analysis ProceduresApproach Analysis Results Additional Notation

SupportedPaper Satisf

ForwdsSatisf

BackwdsHuman Interv

Metrics Plan-ning

Simu-lation

Model Check

Qual Quant Binary Depend-encies

Soft-goals

Contribut-ion Links

Chung et al. [9] Y N Y N N N N Y N Y N Y YGiorgini et al. [21] Y N N N N N N Y Y Y N M YGiorgini et al.[22] Y Y N N N N N Y N Y N M YGiorgini et al. [23] Y Y N N N N N Y N Y M Y YHorkoff & Yu [26] Y N Y N N N N Y N Y Y Y YMaiden et al. [33] Y N Y N N N N N N Y Y Y YAmyot et al. [1] Y N N N N N N Y Y Y Y Y YAsnar & Giorgini [3] Y Y N N N N N Y M Y N M YLetier & vLams. [31] Y Y N N N N N N Y Y M N NHorkoff & Yu [27] Y Y Y N N N N Y N Y Y Y YWang et al. [35] Y Y N N N N N N N Y N M YBryl et al. [6] N N N Y Y N N N Y Y Y N NBryl et al. [7] N N Y Y Y N Y M Y Y Y N NAsnar et al. [4] Y Y Y Y Y N N Y N Y Y M YGans et al [18] N N Y N N Y Y N N Y Y N NWang & Lesper. [34] N N N N N Y N N N Y N N NGans et al. [16] [18] N N Y N Y Y Y N Y Y Y Y YGans et al. [17] N N N Y N Y M N Y Y Y N NFuxman et al. [14] [15] N N M N N N Y N N Y Y Y NGiorgini et al.[20] N N N N N N Y N N Y Y N NBryl et al.[8] N N N N Y N Y N N Y Y N N

Procedures

Summary Dimensions


A Survey of GORE Analysis Techniques Summarize results over the following points:

Algorithm Approach Satisfaction Forwards, Backwards, Human Intervention,

Metrics, Planning, Simulation, and Model Checking Format of analysis results

Qualitative ( ), quantitative (0.37), binary (T/F) Goal-oriented concepts supported (beyond AND/OR)

Dependencies, softgoals, contribution links

Additional information required beyond typical goal model constructs

e.g., priority, probabilities, events, delegations, and trust.

Approach Analysis Results Additional Notation Supported

Paper Satisf Forwds

Satisf Backwds

Human Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Smith et al. [1] Y N Y N N N N Y N Y N Y Y


Satisfaction Analysis Example Analysis Questions:

What is the effect of this alternative? Can this goal be satisfied?

Evaluates the satisfaction or denial of goals given a functional or design alternative

Values are propagated forward or backward throughout the model

Qualitative or quantitative approaches Techniques take different approaches to

resolving multiple values for incoming goals: Adding evidence, combine using probabilistic rules,

separate evidence, fixed rules, human judgment


Satisfaction AnalysisApproach Analysis Results Additional Notation


ForwdsSatisf

BackwdsHuman Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Chung et al. [9] Y N Y N N N N Y N Y N Y YGiorgini et al. [21] Y N N N N N N Y Y Y N M YGiorgini et al.[22] Y Y N N N N N Y N Y N M YGiorgini et al. [23] Y Y N N N N N Y N Y M Y YHorkoff & Yu [26] Y N Y N N N N Y N Y Y Y YMaiden et al. [33] Y N Y N N N N N N Y Y Y YAmyot et al. [1] Y N N N N N N Y Y Y Y Y YAsnar & Giorgini [3] Y Y N N N N N Y M Y N M YLetier & vLams. [31] Y Y N N N N N N Y Y M N NHorkoff & Yu [27] Y Y Y N N N N Y N Y Y Y YWang et al. [35] Y Y N N N N N N N Y N M Y


Metrics Example Analysis Questions:

How secure is the system represented by the model? How risky is a particular alternative for a stakeholder?

Structural properties of the model and construct classifications are used to calculate metrics Example: counts of dependency classifications (instance,

model, duplicate, hidden) in a Strategic Dependency (SD) Metrics often represent non-functional requirements

Examples: predictability, security, privacy, accuracy, etc. They can also represent model properties:

Examples: completeness, consistency and correctness Metrics can be local or global


Metrics


Paper Satisf Forwds

Satisf Backwds

Human Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Franch & Maiden [12] N N N Y N N N N Y N Y Y NFranch et al. [13] N N N Y N N N M Y N Y Y N

Franch [11] N N Y Y N N N Y Y N Y Y YKaiya et al. [30] N N N Y N N N N Y Y N N M


Planning Example Analysis Questions:

What actions must be taken to satisfy goals? What are the best plans according to certain criteria?

Work has applied AI-type planning to find satisfactory sequences of actions in models

Requires definition of axioms that express possible goal decompositions and delegations Expresses the capabilities of actors in a model

A planner finds a delegation of goals to actors which fulfills model goals

Plans are evaluated by some criteria


Planning


Paper Satisf Forwds

Satisf Backwds

Human Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Bryl et al. [6] N N N Y Y N N N Y Y Y N NBryl et al. [7] N N Y Y Y N Y M Y Y Y N NAsnar et al. [4] Y Y Y Y Y N N Y N Y Y M Y


Simulation Example Analysis Questions:

What happens when an alternative is selected? Are there unexpected properties in a simulation?

Adds temporal information including pre- and post- conditions to models Translated to ConGolog (situation calculus) programs

for simulation Extensions simulate confidence, trust and

distrust


Simulation


Paper Satisf Forwds

Satisf Backwds

Human Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Gans et al [18] N N Y N N Y Y N N Y Y N NWang & Lesper. [34] N N N N N Y N N N Y N N NGans et al. [16] [18] N N Y N Y Y Y N Y Y Y Y YGans et al. [17] N N N Y N Y M N Y Y Y N N


Model Checking Example Analysis Questions:

Is it possible to achieve a particular goal? Is the model consistent?

Models are expanded/converted to a temporal formalism Includes expressions of creation, fulfillment and

invariant properties First order temporal logic statements are used to

represent desired constraints Model checker is used to validate properties and

check for consistency Further work adds in checks for security and

trust


Model Checking


Paper Satisf Forwds

Satisf Backwds

Human Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Fuxman et al. [14] [15] N N M N N N Y N N Y Y Y N

Giorgini et al.[20] N N N N N N Y N N Y Y N N

Bryl et al.[8] N N N N Y N Y N N Y Y N N


Tabular SummaryApproach Analysis Results Additional Notation


ForwdsSatisf

BackwdsHuman Interv

Metrics Plan-ning

Simu-lation

Model Check


Soft-goals

Contribut-ion Links

Chung et al. [9] Y N Y N N N N Y N Y N Y YGiorgini et al. [21] Y N N N N N N Y Y Y N M YGiorgini et al.[22] Y Y N N N N N Y N Y N M YGiorgini et al. [23] Y Y N N N N N Y N Y M Y YHorkoff & Yu [26] Y N Y N N N N Y N Y Y Y YMaiden et al. [33] Y N Y N N N N N N Y Y Y YAmyot et al. [1] Y N N N N N N Y Y Y Y Y YAsnar & Giorgini [3] Y Y N N N N N Y M Y N M YLetier & vLams. [31] Y Y N N N N N N Y Y M N NHorkoff & Yu [27] Y Y Y N N N N Y N Y Y Y YWang et al. [35] Y Y N N N N N N N Y N M YBryl et al. [6] N N N Y Y N N N Y Y Y N NBryl et al. [7] N N Y Y Y N Y M Y Y Y N NAsnar et al. [4] Y Y Y Y Y N N Y N Y Y M YGans et al [18] N N Y N N Y Y N N Y Y N NWang & Lesper. [34] N N N N N Y N N N Y N N NGans et al. [16] [18] N N Y N Y Y Y N Y Y Y Y YGans et al. [17] N N N Y N Y M N Y Y Y N NFuxman et al. [14] [15] N N M N N N Y N N Y Y Y NGiorgini et al.[20] N N N N N N Y N N Y Y N NBryl et al.[8] N N N N Y N Y N N Y Y N N


Information Required by each ProcedureAdditional Information Required by

1 Goal Cost Satisfaction Analysis: [23][4][22][3], Planning: [6]2 Risk Satisfaction Analysis: [3], Planning: [4]3 Textual Arguments Satisfaction Analysis:[33],

Metrics, Model Checking: [30]4 Probabilistic Information Satisfaction Analysis: [23] [31]5 Events and Treatments Satisfaction Analysis: [3]6 Importance/Priority Satisfaction Analysis: [1],

Metrics: [13] [1], Simulation: [34]7 Actor Capabilities Planning: [6] [7] [4], Model Checking: [8]8 (Pre/Post) Conditions/ Temporal

InformationSimulation: [34] [18] [18] [16] [17],

Model Checking: [15] [14]9 Delegation/Ownership Model Checking: [19] [8]

10 Trust Planning: [4], Simulation: [17], Model Checking: [20][8]

11 Speech Acts Simulation: [17]12 Confidence and Distrust Simulation: [17]13 Preferences Model Checking: [30]14 Cardinalities Simulation:[34], Model Checking: [14]


Goal Model Analysis Objectives Using capabilities of techniques in our survey, as

well as our own experience in modeling and analysis, we list categories of objectives for goal model analysis List is likely not complete

Objective Categories (goal model analysis can help…): Understand the domain Communicate Improve the model Make scoping decisions Prompt requirements elicitation Improve requirements Design a system


Mapping Procedures to Objectives We have made suggestions concerning what procedures

may map to what objectives Each mapping can be considered as a hypothesis

We have included guiding questions with each objective to help motivate the mapping and guide users

Example mappings:

Downloadable interactive mapping table: www.cs.utoronto.ca/~jenhork/GOREAnalysisSelectionTable.zip

Category Guidelines Recommended ProceduresDomain Understanding

QU1. Does the domain contain a high degree of social interaction, have many stakeholders with differing goals, or involve many interacting systems?

Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26][27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET([16][17][18])

Requirements Improvement

QR1. Are you working with a system where safety/security/ privacy/risks or other specific properties are critical considerations?

Yes. Try: Analysis over Specific Constructs or Metric Approaches: KAOS([31]) i* Metrics([11][12][13]) AGORA([30])

Tropos Risk, Trust, and Security([3][4] [8][19]) SNET Trust([17])

http://www.cs.utoronto.ca/~jenhork/GOREAnalysisSelectionTable.zip

http://www.cs.utoronto.ca/~jenhork/GOREAnalysisSelectionTable.zip


Category Guidelines Recommended ProceduresDomain Understanding

QU1. Does the domain contain a high degree of social interaction, have many stakeholders with differing goals, or involve many interacting systems?

Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26][27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET([16][17][18])

QU2. Do you need to understand details of the system at this point? Do you have access to detailed information such as cost, probabilities, and conditions? Can you express necessary or desired domain properties?

Yes. Try: Quantitative or Detailed Information: Tropos Probabilistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]), or Model Checking: Tropos ([8][14][15][19]) SNET([16][18])

Communication QC1. Do you need to communicate with stakeholders? Validate requirements in the model? Justify recommendations?

Yes. Try: Forward Satisfaction Approaches: NFR([9]) Tropos([3][21][22][23]) KAOS([31]) i*([26][33]) GRL([1])

Model Improvement

QM1. Are you confident in the accuracy, structure, and completeness of domain knowledge and models?

No. Try: Interactive Approaches: NFR([9]) i*([26][27][33]) Tropos([4][7]) SNET([16][18]) i* Metrics([11])

QM2. Would you like to verify critical properties over the model? Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18])Scoping QS1. Do you need to determine system scope? Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26] [27][33]) i* Metrics ([11]

[12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET ([16][18])

Requirements Elicitation

QE1. Do you need to find more high-level requirements? Are you looking for ways to prompt further elicitation?

Yes. Try: Interactive Approaches: NFR([9]) i*([27][27][33]) Tropos([4][7]) SNET([16][18]) i* Metrics([11])

QE2. Do you need to find detailed system requirements? Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34])

QE3. Do you need to consider non-functional requirements difficult to quantify? Yes. Try: Approaches supporting softgoals or contributions: NFR([9]) i* Satisfaction Analysis ([26][27][33]) Tropos Satisfaction Analysis ([3][21][22][23]) Tropos Model Checking([14][15]) GRL([1]) i* Metrics([11][12][13]) SNET([16][17][18])

QE4. Do you need to capture domain assumptions? Yes. Try: Approaches using Satisfaction Arguments: i* Satisfaction Arguments [33]

Requirements Improvement

QR1. Are you working with a system where safety/security/ privacy/risks or other specific properties are critical considerations?

Yes. Try: Analysis over Specific Constructs or Metric Approaches: KAOS([31]) i* Metrics([11][12][13]) AGORA([30]) Tropos Risk, Trust, and Security([3][4] [8][19]) SNET Trust([17])

QR2. Do you need to find errors and inconsistencies in requirements? Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18])Design QD1. Are you aware of a sufficient number of high-level design alternatives? No. Try: Agent, Planning, Forward and Backward Satisfaction Approaches: NFR([9]) i*

Satisfaction Analysis ([26][27][33]) Tropos Planning([4][6][7][8]) KAOS([31]) GRL Forward Satisfaction Analysis([1]) SNET Planning([16][18])

QD2. Are you aware of a sufficient number of detailed design alternatives? No. Try: Quantitative Planning, Forward and Backward Satisfaction Approaches: KAOS Satisfaction Analysis ([31]) GRL Forward Satisfaction Analysis([1]) Tropos Planning([6][7]) SNET Planning([16][18])

QD3. Do you need to evaluate and choose between high-level design alternatives? Yes. Try: Satisfaction Analysis, Metrics and Agent Approaches: KAOS Satisfaction Analysis([31]) i* Forward Satisfaction([26][33]) GRL Satisfaction Analysis([1]) i* Metrics([11][12][13]) Tropos Risk([4])

QD4. Do you need to evaluate and choose between detailed design alternatives? Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34])

QD5. Do you need to find acceptable processes? Yes. Try: Planning Approaches: Tropos Planning([4][6][7][8]) SNET Planning([16][18])

QD6. Do you need to test run-time operation before implementation? Yes. Try: Simulation Approaches: SNET([16][17][18]) i* Simulation([34])

Mapping of Procedures to Objectives

Objectives

Procedures

25

Guideline Usage Examples Example: Online Counseling Domain (Horkoff & Yu, 2009)

Online counseling alternatives: text messaging or chat room?

Apply guiding questions… High degree of social interaction (QU1) Do no yet understand details, not yet confident in the

accuracy and completeness of models (Qu2, QM1) Communication is important, scoping is challenging (QS1,

QU2) Etc…

Recommendations: Interactive, agent-oriented techniques for forward

satisfaction analysis supporting softgoals Analysis for anonymity or privacy with the same

techniques or with GRL Satisfaction Analysis, and/or i* Metrics

If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu


Conclusions First step towards making goal model analysis

techniques more accessible to modelers Enable potential users to user their knowledge of

the domain and analysis objectives to select one or more procedures

We have attempted to be neutral in our analysis Each procedure has unique abilities

Future work is needed to undertake studies to validate or refute the claims made by our guidelines Hope that guidelines will be expanded and refined as

more application experiences are available


Thank you www.cs.utoronto.ca/~jenhork

[email protected]

www.cs.utoronto.ca/~eric [email protected]

http://www.cs.utoronto.ca/~jenhork

mailto:[email protected]

http://www.cs.utoronto.ca/~eric

mailto:[email protected]


Outline

Goal Models Goal Model Analysis Motivation: Abundance of Approaches Survey of Goal Model Analysis Approaches Survey Results Summary Objectives of Goal Model Analysis Mapping of Procedures to Objectives Example Selection Conclusions & Future Work


Survey of GORE Analysis Techniques: Selection Article selection:

Started with a set of known relevant papers Linked work through references Stopped with picture of breadth was captured (24 papers)

Alternative selection methods: Search for specific key words… … in specific journals, conferences, portals … during specific time periods

Challenge: work in goal model analysis appears in a range of venues with a range of keywords

Venues: Books, RE, REJ, Agent-related conferences, CAiSE, AI related journal, FSE, PoEM, Journal of Information Systems, Trust-related Conference, ASE, etc…

Keywords: agent-oriented software development, goal-oriented requirements analysis, early requirements analysis, multi-agent systems, agent-oriented software engineering, agent-oriented methodologies, risk analysis, countermeasure identification, goal modeling, goal-oriented analysis, quality metrics, etc….


Guideline Usage Examples Example: Wireless service from Amyot et al. 2010

New wireless service added to existing network Where should the data and service be located? Apply guiding questions…

Domain contains some interacting systems (QU1), no emphasis on communication (QC1)

Aware of alternatives, but need to select one (QD1, QD3) Do not yet understand details, detailed alternatives, don’t have

access to specific information, don’t want to find processes or perform simulations (QU2, QR2, QD2, QD5, QD6)

Domain is well understood, scope is clear, models are sufficiently complete (QE1, QS1, QM1)

Must consider non-functional requirements (QE3), data privacy (QR1)

Recommendations: Agent-oriented approaches supporting softgoals to

consider social and non-functional nature of the problem Satisfaction analysis or metrics to chose between

alternatives

31

Guideline Usage Examples Example: Online Counseling Domain from Horkoff & Yu,

2009 Online counseling alternatives: text messaging or chat room? Apply guiding questions…

High degree of social interaction (QU1) Do no yet understand details, not yet confident in the accuracy

and completeness of models (Qu2, QM1) Communication is important, scoping is challenging (QS1, QU2) Consider many non-functional requirements, privacy and security

especially, capture assumptions (QE3, QE4, QR1) Need to find and evaluate alternatives (QD1, QD3) Could be useful to find the most successful process (plan) for

counseling or simulate throughput (QD4, QD5) Recommendations:

Interactive, agent-oriented techniques for forward satisfaction analysis supporting softgoals

Analysis for anonymity or privacy with the same techniques or with GRL Satisfaction Analysis, and/or i* Metrics

If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu

analyzing goal models – different approaches and how to choose among them jennifer horkoff 1 eric...

Documents