design science as nested problem solving · financial evaluation? st, ph, go, cr k build taxonomy...

30th April 2009 Univ. Rey Juan Carlos 1

Design Science as

Nested Problem Solving

Roel Wieringa

University of twente

The Netherlands


How to do design science research?

• Guidelines for doing a Master’s project

• and for doing a PhD project

• State of the practice of research:

– Master’s and PhD students tend to confuse

design with research

– Methodological mistakes


1. Knowledge problems versus practical problems

2. Engineering cycle

3. Problem nesting

4. Design science


Problems

• Difference between what stakeholder wants and what stakeholder experiences

• What is the problem (stakeholder, experience, desire)?– Build a communication infrastructure between

hospital A and insurance company B• Problem: there exists no such structure, stakeholder wants

one

– Improve communication infrastructure between …• Problem: Current infrastructure does not satisfy stakeholder

– Design a communication infrastructure between …• Problem: there is no blueprint, stakeholder wants a blueprint


Practical problems versus

knowledge problems

• Practical problem– Difference between current state of the world

and what a stakeholder would like it to be

– To solve it we need to change the world

• Knowledge problem– Difference between what current stakeholder

knows and what the stakeholder wants to know

– To solve it stakeholder needs to change theirknowledge of the world


What kind of problem?

• What is the architecture of the communicationinfrastructure between A and B?– K Problem: infrastructure exists, stakeholder does not know what

its architecture is

• Build a communication infrastructure between hospital A and insurance company B– P Problem: there exists no such structure, stakeholder wants

one

• Improve communication infrastructure between …– P Problem: Current infrastructure must be changed

• Design a communication infrastructure between …– P Problem: A blueprint must be made

• What is a communication infrastructure between …– P Problem: A blueprint must be made

Misleading


Design problems are practical problems

• Customer: “Design a communication infrastructure formeˮ

• An architect designs one

• Architect: “Here is an architecture!ˮ

• Customer: “Thank you! Now I know what to do! I’ll do itthis way.ˮ

This means:•“ Now I have made up my mind what to do. I’ll

build the infrastructure this way. ˮ

•To design something is to decide what to do

• = Practical problem


Heuristics• Practical problems

– Are solved by

changing the state of the world

– Solution criterion is utility

• Problem-dependent: stakeholders and goals

• Several solutions; but trade-offs

• Knowledge questions

– Are solved by changing

the knowledge of stakeholders.

– Solution criterion is truth

• Problem-independent: no stakeholders

• One solution; but approximations

Doing

Changing the world

Future-oriented

Thinking

Changing our mind

Past-oriented


But …

• Solving a practical problem may produce

knowledge

– That is not what makes it a practical problem!

– Solving a practical problemmay also fail to teach usanything

– We solved a practical problem = we satisfiedstakeholder goals

• Knowledge may be usefultoo– Useless knowledge is still

knowledge– A useful false proposition is

not knowledge• “There is no speed limitˮ

– We solved the knowledgequestion = we have a trueproposition that answersthe question.




3. Problem nesting

4. Design science


The engineering cycle

• Solving practical problems rationally

– problem investigation

– solution design

– specification validation

– implementation

– implementation evaluation

•Stakeholders•Goals, criteria•Phenomena, diagnosis

•Specify a possible solution

•Will it work?•Will it satisfy criteria?•Trade-offs?•Sensitivity?

•Does it work?•Does it satisfy criteria?


Design

• To design is to make a decision about a solution– Software design

– House design

– Business design

– Job role design

– …

– Functional design

• A specification is a documentation of those decisions– Software architecture

– House blueprint

– Job role descriptions

– …

– Functional specification




3. Problem nesting

4. Design science


Subproblems

• Engineering cycle

– problem investigation

– solution design

– specification validation

– implementation

– implementation evaluation

•K Stakeholders?•K Goals, criteria?•K Phenomena, diagnosis?

•P Specify a possible solution

•K Will it work?•K Will it satisfy criteria?•K Trade-offs?•K Sensitivity?

•K Does it work?•K Does it satisfy criteria?

P


How can we improve financial evaluation of process-aware information systems?

K Current problems

with evaluation?

K Current approaches to

financial evaluation?

St, Ph, Go, Cr

K Build taxonomy

of approaches

K Classify approaches

K Validate classification

K Criteria for taxonomies?

K Collect taxonomies

K Evaluate

P Design new one

KValidate against criteria

K Evaluate them

P Develop new approach:

Causal loop modelsK Make causal loop models

of cost factors of PAISK Collect modeling

guidelines

P Acquire modeling

toolsK Validate it K Check design

argument P Experiment to test one model

P Pilot study using another model

K Reflection: lessons learned

Problemdecomposition

Problemsequence


Bulleted list form• Improving the financial evaluation of PAIS• Problem investigation

– Current problems with financial evaluation• Current approaches

– Taxonomies of approaches

– Our taxonomy

• Evaluation of approaches

• Solution approach– Causal loop models– CLDs of cost factors

• Validation– The engineering argument– Experiment– Pilot study

• Discussion and lessons learned

• Appendices– Modeling guidelines for CL modeling– Tools for CL modeling

Very good PhD thesis outline


P How to solve conflicts between and agencies standards

for data exchange between HRM depts?

K Conflict taxonomy?

K Conflicts between X and Y? K Make CM of subject domain of X and Y

K Problems caused by this? K Stakeholders, phenomena, goals, criteria

K What causes the conflicts? K Describe the causes

P Specify solutions K Inventarize existing ones

P Compose new ones

K Validate the solutions

•K Internal: C & S → criteria?

•properties of S

•Assumptions about C

•Interaction

•K External: Sensitivity?

•K Trade-offsK Reflection:

Lessons learned


Bulleted list formHow to solve conflicts between standards X and Y?• Kinds of conflicts

– A conflict taxonomy

– Conflicts between X and Y

• Problem analysis– Stakeholders, goals, criteria, problems

– Causes of conflicts

– Impact of conflicts

• Solution proposal– Inventory of existing solutions

– Solution for standards X and Y

• Solution validation– Effectiveness: properties, certainty, value of solution

– Trade-offs

– Sensitivity: Future scenarios and behavior of solution in those scenarios

– Stakeholder analysis

• Summary and recommendations• Lessons learned: reflection on the project itself

Would have beenan excellentMaster’s thesis outline


• Find a modelling method for embedded software

– What is the problem?

– Design an improved method

– Validate the method

• What happens if used?

• Does this satisy goals?

• Trade-offs?

• Sensitivity?

• Reflect and identify lessons learned about finding animproved modeling method

•What is the modeling problem in this case?

•Customize my method to this problem

•Validate

•Use it

•Evaluate it

�stakeholder’s goals

�researcher’s goals

Technical action research

•Find a stakeholder with a problem that

you think you can solve using your

technique

•Use your technique

•Repeat for other stakeholder:

•trade-offs and sensitivity

PhD project

Problems

1. Modelling not repeatable

2. Formal models not validated

3. Implicit assumptions about plant

Solution specification

1. Design argument Plant & SW → Goals

2. List of assumptions about plant




3. Problem nesting

4. Design science issues


Problem investigation• Stakeholders

– Stand to lose or gain by solving the problem

• Goals

– What stakeholders want

– Need to be analyzed and agreed on:

• Usually inconsistent

• Usually not feasible

• Usually unmeasurable

• Criteria

– Operationalized goals

• Phenomena

– What is happening?

– Diagnosis: what is the cause of the problems?

– Prediction: What would happen if we do not solve the problem?


Reasons for problem investigation

• Problem driven• Stakeholders currently experience problem

• Goal-driven• Goals have changed

• Solution-driven• New technology available

• New technology can be developed

• Impact driven• Evaluate past implementation


Design validation

• Internal validity– Solution & Context produce Effects

– Effects satisfy criteria to some extent

• Trade-off analysis– Effect of slightly different solutions

– Score differently on same criteria

• Sensitivity analysis– Effects in different contexts?

– This is external validity

The design argument


Levels of theory

• Nomothetic– N=∞– Universal statement, for all times and places

– E.g. Theory of gravity

• Practice-oriented– N=k– E.g. Statement about trade-offs in ERP

implementation

• Case theory– N=1– E.g. Claim about trade-offs in this company

Design theories are usually of this kind


Refinement of the framework of

Hevner et al.


Design theories

• Must be practice-oriented– Effort estimation models for IS implementation

– Claim about effectiveness of a modeling method• E.g. It helps mechanical engineers communicate with

software engineers

– Claim about HRM data exchange standard• E.g. it is compatible with existing standards

• General form– Context & Solution → Effects

Possible goal variables

Something stakeholders can do

Some class of relevant contexts


Conditions of practice

• Laboratory versus field

– Engineers must work under conditions of

practice

– Scientists can abstract

• Design theories in practice:– Under not fully known conditions of practice,

– Context & Solution → Effects


Kinds of theory

• What kind of knowledge does a theory give you

– conceptual

• taxonomies

• ontologies

– empirical

• explanatory

• predictive

– design.

– analytical

• mathematical

• logical


Research methods

• Everything is allowed

– lab experiments

– field experiments

– surveys

– case studies

– technical action research

– …

• Do not claim more than you can justify


Take-home messages

• Distinguish practical problems fromknowledge questions

– Changing the world versus changing our

knowledge

• Solve practical problems with the engineering cycle

• Problems are mutually nested


Knowledge problems

Practical problems

Every knowledge problem

hides a practical problem

Every practical problem

hides a knowledge problem

design science as nested problem solving · financial evaluation? st, ph, go, cr k build taxonomy...

Documents