patterns in business analysis and enterprise modeling: how to evaluate their value? by kurt...

1

Patterns in Business Analysis andEnterprise Modeling: How to evaluate their value?

Kurt Sandkuhl

The University of Rostock, Germany

2nd Business and System Conference 2013Riga (Latvia), November 2013

Overview

Motivation

Background

• Patterns for Capturing Organizational Knowledge

• Evaluating the Business Value of IT

Evaluation Strategy for Patterns

Pattern Examples

• Information Demand Patterns

• Task Patterns

Pattern Use in Business Analysis

Summary and Future Work

12.05.2011 © 2011 The UNIVERSITY OF ROSTOCK | FACULTY FOR COMPUTER SCIENCE AND ELECTRICAL ENGINEERING 2

2

Motivation

Business analysis and solution design projects

• are highly specialized for the specific enterprise under consideration, i.e. project is unique

• different enterprises in the same area often show similarities

• potential to reuse at least some of the knowledge from earlier businessanalysis projects in later projects

Different techniques for supporting reuse in analysis and solution design

• high abstraction level: reference models/frameworks (COBIT, TOGAF, ITIL)

• Lower abstraction level: patterns

• Methods: means of capturing proven knowledge as support for action(ARIS, OneMethodology, 4EM)

How to evaluated the “value” of patterns for business analysis process and thesolution developed?

Overview

Motivation

Background




Pattern Examples


• Task Patterns




3

Patterns

Patterns in Computer Science and Enterprise Modeling:

• Initial definition: Christopher Alexander: “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice”

5

• Pattern type examples:

– Analysis Patterns (Fowler)

– Design Patterns (Gamma et al.)

– Architecture Patterns (Buschmann et al.)

– Workflow Patterns (van der Aalst et al.)

– Knowledge Patterns (Clark et al.)

– and many more …

Organizational Knowledge Patterns

An organizational knowledge pattern is a formalization of knowledge for a recurringorganizational task abstracting from organization-specific aspects, which is of value for an organizational actor and an asset for an organization.

Characteristics

• OKP address recurring organizational tasks and at the same time abstracting from a specific organization. They represent organizational knowledge, not individual knowledge

• OKP are expressed in a formalized way, which requires a formal language or at least a structured representation

• OKP are an asset of the organization, i.e. are not only a resource as such but capture knowledge about the resource’s use

• An OKP is of value for an organizational actor in its original form and / or its adaptation for a specific organization.

4

Business Value of IT

Kinds of approaches for evaluating the business value of IT

• Process-oriented approaches

• Perceived value approaches

• Project-focused approaches

• Multi-perspective approaches based on performance indicators

Process Oriented approaches (Mooney et al., 1995)

Focus: value created by process improvements in different dimensions

Potential IT Business Value Metrics (Mooney et al. 1995)

5

Perceived Values – DeLone and McLean (1992 + 2006)

Information can be captured at different levels

• the technical level,

• the semantic level and

• the effectiveness level

Identification of six dimensions of IS success:

Project-based Approach - Parker (1998)

(CSC 1997)

• Identification of stakeholders and risk factors• Financial and strategic objectives of the business; add the technical dimensions of a project• Definition of evaluation scales• Negotiation of the importance of each evaluation criterion via a weighting factor

6

Multi-perspective approach - Zee (2002)

A ‘dashboard’ of relevant performance indicators for IT

Three levels

Business Value of IT (business management level)

Effectiveness of IT (IT management level)

Effectiveness and Efficiency of IT supply (IT supply management level)

BVIT is measurable on three dimensions

Financial performance

Business performance

Strategic performance

Overview

Motivation

Background




Pattern Examples


• Task Patterns




7

Evaluation Strategy for Patterns: Origin

Evaluation strategy: integrating two different lines of work

13

1. pattern use as part of solutions

– evaluation of the solution, not the elicitation process of the pattern type and the actual pattern

– patterns were evaluated from the economic perspective using the balanced scorecard approach

2. pattern type elicitation and actual pattern development

– evaluation strategies from naturalistic inquiry

• alternating phases of evaluation in theory and practice

• Internal and external validation

Evaluation Strategy: Characteristics

Four evaluation stages

• Assessment of the patterns by experts from the field

• Evaluation by the pattern users: perception of the pattern utility

• Case studies. Observation of real-world cases by researchers

• Measurement using scorecards: long-term way of assessing

14

Four perspectives

• the pattern concept from the perspective of an individual

• the pattern concept from the perspective of an organization

• the actual pattern from the perspective of an individual

• the actual pattern from the perspective of an organization

8

Overview

Motivation

Background




Pattern Examples


• Task Patterns




Evaluation Strategy: Status for Example Patterns

16

Pattern:Individualperspective

Pattern:organizationalperspective

Concept:Individualperspective

Concept:Organizational

perspective

Assessment:Expert Reviews

IDP:[25] and [26]

IDP: [24] forcollaborativeengineeringTP: [32]

IDP: [25] and [26]

IDP: [27] for changemanagementTP: [30]

Perception:User Evaluation

IDP: subject of section 4.2

Observation:Case Studies

IDP: [24] and [27] IDP: [28TP: [31]

Measurement:Scorecard

IDP: Not performed yetTP: summarized in section 5.2

IDP: Information Demand Patterns TP: TaskPatterns

9

Information Demand Patterns

Information Demand Patterns (Excerpt)

• Change Administrator

• Process Verifier

• Material Specification Responsible

• Responsible for Quote Preparation

• Responsible for Proposal Writing

• Branding Responsible

18

Example: Proposal Writing Responsible

Context

• Research and development projects at universities or research institutes to a large extent are based on project funding. Preparing successful proposals for funding is a task that requires experience and accurate information about the published call, funding conditions, economic parameters, etc., which usually originate from different information sources and actors. The pattern describes the information demand typically experienced by the role responsible for preparing a proposal for funding.

• The pattern is supposed to be useful for any type of organization aiming at submitting proposals for project funding to funding agencies.

19

10

Problem The pattern addresses the general problem of submitting proposals of unnecessary low quality, which basically is wasting resources at the preparing organization or reducing the probability and volume of grants. This includes the following problems, which were often observed when preparing proposals for funding:

• Submitted proposals fail to meet the formal requirements published in the call, which usually leads to a rejection. Examples are missing attachments, missing signature, wrong number or different kind of partners.

• Submitted proposals contain an invalid or incorrect budget. Typical mistakes are to include non-eligible costs, use the wrong cost rates or don’t include applicable surcharges

• Proposals ask for funding of activities, which already are being funded in other projects of the same fund giver

• Proposals are submitted to late, either due to deadline changes or misunderstandings

20

Tasks and Responsibility

The tasks of the role responsible for a proposal include

• The preparation of the a competitive, complete and high quality technical content of the proposal

• The preparation of an accurate, complete and consistent formal part of the proposal

• Partner management, i.e. coordination of all contributions from participating partners

• On-time submission of the proposal including all attachments, forms and signatures

21

11

Information Demand

The information demand of the role responsible for a proposal consists of:

• The work program or policy documents guiding the funding from the fund giver

• General conditions from the fund giver stating eligibility of costs, preconditions to apply for funding or evaluation procedure

• The complete call information, including supplementary material like the guide for proposers, schedule of information meetings, forms to be submitted, etc.

• Economic parameters from the own organization, including personnel cost rates, surcharges, administrative overheads, investment and facility costs to be included, VAT rates, etc.

• Earlier funded projects at the fund giver in the subject field of the call under consideration

• Plans of the competitors regarding submissions in the same call to the fund giver

22

Quality Criteria

General importance

Accurate as early as possible

complete

Work Program high high high high

Call Content decisive decisive high decisive

General Conditions high high Nice to have

high

Economic Parameter high decisive Nice to have

high

Earlier Funded Projects high Nice to have

Nice to have

high

Competitor plans Nice to have

Nice to have

Nice to have

Nice to have

12

Effects

24

Economiceffect

Timeefficiency

Qualityeffect

Motivation LearningExperience

Customer

Work Program low Very high Very high moderate moderate moderateCall Content low Very high Very high moderate moderate moderateGeneral Conditions

moderate Very high moderate low low moderate

Economic Parameter

Very high low Very high low low low

Earlier Funded Projects

low moderate Very high moderate moderate low

Competitor plans low moderate moderate low moderate low

Perception – Organisational Perspective

• Method:

– Questionnaire, 5 point Likert scale

– Demand pattern ”proposal responsible”

• Participants

– 4 different organisations from 3 different countries

– 2 research institutes, 2 universities

– Vice-director / Assoc. Dean for Research

– 25 – 100 proposals / year

– 20+ employees involved

25

13

Selected Results - Organisational Perspective

• All respondents agree

– Very much / much with the tasks / responsibilities

– Very much / much with the quality criteria

– that the described effects are valuable / very valuable(exception: timeline, motivation)

– that all elements of the pattern are needed

– Could imagine to use the demand pattern for

• training of new employees

• Training of current employees

• Basis or best practice description

• Improvement of best practices

26

Perception – Individual Perspective

• Method:

– Questionnaire, 5 point Likert scale

– Demand pattern ”proposal responsible”

• Participants

– 3 PhD students (less than 2 proposals)

– 4 researchers (4-6 proposals)

– 1 Head of research group (10-12 proposals)

– 2 senior researchers (more than 20 proposals)

– from

• 4 different organisations from 3 different countries

• 2 research institutes, 2 universities

27

14

Selected Results – Individual Perspective

• The respondents agree

– All agree very much / much with the tasks / responsibilities

– 70% find that the information demand, quality criteria, effects and task & responsibility descriptions are valuable / very valuable

– 90% find that all elements of the pattern are needed

– Difficult to estimate how much effort the overall pattern or the different parts of the information demand pattern would save

– 3 propose the same additional part: ”background information from program manager at the fund giver”

28

30

TaskPatternExample

15

31

Balanced Scorecard

Approach: Balanced Scorecard (Kaplan & Norton):A measurement-based strategic management system, which provides a method of aligning business activities to the strategy, and monitoring performance of strategic goals over time.

Implementation of a scorecard in each MAPPER use case and on project level

Financial Customer

Internal BusinessProcess

Learning andGrowth

32

Evaluation Results – Use and Quality of Task Patterns

• Number of Best Practices

– number of best practice descriptions increased significantly

– C1: 7; C2: 5; C3: 6; C4: 9• Quality of Best Practices

– average level of detail of the best practice descriptions improved

– intensity of use of best practices at significantly C2 and C4,

– accuracy of the best practice descriptions

– training new employees within C4• Updating best practice descriptions

– The time needed to update best practice descriptions decreased

– at C3 22 days to 14 days; at C4 6 to 4 months; stable at C1 and C2

• Conclusion: clearly positive evaluation of task patterns

16

33

Evaluation Results (2)

• Quality of Product Documentation– Number of design rules:

C3: from 95 to 160; C2 from 25 to 27; C4: no changes– refinement level improved at C4 from 3 refinements to 4;

• Cycle Time, Time to market– indicators provided by C4

• material specification task: from 4 months to 3 months (reduced by 25%)

– indicators provided by C1• average length of the target setting process: from 14 to 12

months • average time for assessing solutions for a customer problem:

reduced by 50%.

Overview

Motivation

Background




Pattern Examples


• Task Patterns




17

Pattern Use for Business Analysis

Knowledge areas according to BABOK:

• Business Analysis Planning and Monitoring

• Enterprise Analysis

• Elicitation

• Requirements Analysis

• Solution Assessment and Validation

• Requirements Management and Communication

35

Pattern Use for Business Analysis (2)

IDP and TP do not offer support:

• Business Analysis Planning and Monitoring


• Elicitation

• Requirements Analysis


• Requirements Management and Communication

36

Patterns do not offer method support, but provide solution elements!

18

Pattern Use for Business Analysis (3)

Knowledge areas with potential pattern use:


– identify the potential source for shortcomings andoperational problems in an enterprise

• Elicitation

– patterns propose a solution which has to be configured orinstantiated for the company under consideration


– knowledge patterns can be used as a best practice reference

37

Summary & Future Work

The intention of our work is to contribute to evaluating the value of patterns and improve patterns for use in enterprise modeling and business analysis

Contributions so far• (1) a proposal for how to evaluate the maturity and value of patterns• (2) two actual pattern types and results regarding their evaluation• (3) a discussion of pattern use in the context of business analysis

Future work• Continue evaluation of more pattern types• Further investigate the use in business analysis

19

39

Thank you for your attention!

Questions?