conceptual model evaluation. towards more epistemological rigor presentation at emmsad ’05 jan...

Conceptual Model Evaluation. Towards more Epistemological Rigor

Presentation at EMMSAD ’05

Jan ReckerQueensland University of Technology

June 14, 2005

© Jan Recker, Queensland University of Technology 2

Why Philosophy?

[a] good part of the answer to the question “why philosophy?” is that the alternative to philosophy is not no philosophy but bad philosophy. The ‘unphilosophical’ person has an unconscious philosophy, which they apply in their practice – whether of science or politics or daily life.

Collier (1994), p. 17


Agenda

• Motivation

• Philosophy in IS Research

• Conceptual Model Evaluation

• A Paradigmatic Discussion Framework

• Sample Application

• Conclusions


The Need for Evaluation

“No problem-solving process can be considered complete until evaluation has been carried out. It is the evaluation which helps us to measure the effectiveness of the problem-solving process and the problem solver in the 'problem situation' – unless this element is considered there is no way of establishing that the 'problems' have been successfully resolved”

Jayaratna (1994), p. 108

MotivationPhilosophy in ISModel EvaluationDiscussion FrameworkApplicationConclusions


Scope, Idea, and Objective

• Scope:Evaluation of conceptual models

• Idea:Transfer Insights from Philosophy to the field of Evaluation: Paradigmatic Analysis

• Objective:Explicate the implications of paradigmatic presuppositions onto the evaluation of conceptual models

• Research method:Critique: Explicating and Discussing paradigmatic presuppositions onto the evaluation of conceptual models to determine possibilities, scope, and limits.



Philosophy and IS Research

• IS discipline at the intersection of multiple research fields

• “State of diversity”

• Diverse approaches towards cognition, reality, truth etc. are being used



Paradigms in IS Research

• Paradigm:a specific way of thinking about problems based on a set of achievements that are acknowledged as a foundation of further research practice

Avison & Fitzgerald (1995)

• Predominant paradigms in IS:– Positivism– Interpretivism


Chen & Hirschheim (2004)


Paradigmatic Framework

Philosophical aspect Positivist paradigm Interpretivist paradigm

Ontological realism.There is an objective reality existent independently from subjective cognition, i.e. independent from thought and speech (Bunge, 1977).

[I]What is the object of

cognition? (Ontological aspect)

Ontological idealism.Reality is not a priori given, concepts are socially constructed, dependant on subjective perception, cognition and language (Berger and Luckmann, 1966).


[II]What is the relationship

between cognition and the object of cognition?

(Epistemological aspect)

Epistemological realism.Reality can be perceived objectively without subjective distortions by a cognitive subject (Losee, 2001).

Epistemological constructivism.Reality perception is always “private“, hence susceptible to a (predominantly linguistic) (re-) construction of the cognition of reality (von Glasersfeld, 1987).


Models in IS Research

• Conceptual Models– facilitate, systemize, and aid the process

of information system engineering.– describe object systems (e.g. an

information system) of some domain in semantic terms, using an abstract yet formalized language.

– Core (if not the) artefact(s) of the discipline



Selected Evaluation Approaches(empirical)


SurveyThis technique refers to the use of questionnaires to gather human attitudes, opinions, and impressions on methods. It mainly focuses on the perception of users.

Laboratory ExperimentThis technique is used to assess the internal validity of modelling methods. In a laboratory experiment, independent variables, such as different modelling methods, are manipulated in order to measure the effect on dependent variables like accuracy or time costs of modelling.

Field ExperimentThis technique is similar to laboratory experiments but the experiment takes place in natural settings. Researchers perform the investigation in concrete business organisations while trying to maintain control over the most significant independent variables.

Case StudyThis technique focuses the systematic observation of a particular group or subject that utilises modelling methods. The investigation is conducted without intervening in the modelling process; instead focus is spent on gathering real life data on the usage of modelling methods.

Action ResearchAction research is the application and testing of ideas developed in an academic environment in real world situations under participation of the researching individuals. By attending the modelling process, information about modelling experience through the participating researcher can be gathered.

Cf. Siau & Rossi (1998)


Selected Evaluation Approaches(non-empirical)


Feature ComparisonThis technique attempts to compare existing modelling methods by modelling the same domain with different methods and investigate how the various methods represent the same problem. The evaluation is then conducted based on a checklist of features that should ideally be included in the modelling methods.

MetamodellingThis technique uses meta models of methods as an analysis basis. It is attempted to evaluate methods by structurally investigating analogies and dissimilarities of their meta models.

Metrics ApproachThis technique aims at comparing methods based on a predefined set of method metrics. Metric values are compared to reference values which address complexity and appropriateness of a method.

Paradigmatic AnalysesThis technique refers to the analysis of underlying assumptions of methods, e. g., the view of IS development intention, the view on language functions, the definition of IS etc.

Contingency IdentificationThis technique aims at identifying the contingencies of the project in which a method is utilised. Therefore, criteria, such as the problem to be solved or the project team, are investigated to provide heuristics for the selection of an appropriate method.

Ontological EvaluationThis technique uses ontological concepts to evaluate modelling methods. The idea is to map existing modelling language constructs to the constructs provided by the ontology to assess the modelling language’s capability to represent reality.

Cf. Siau & Rossi (1998)


Preliminary Conclusions

• Plethora of model evaluation methods available

• Evaluation design is coined by philosophical presuppositions of designing and applying researchers

• Discussion of presuppositions aids explicating evaluation approaches



Construction (1/2)

• Investigation of existing model evaluation techniques based on– Philosophical aspects

• Ontological aspect• Epistemological aspect

– Paradigmatic approach upon aspects• Positivist approach• Interpretivist approach

– Paradigmatically coined perception of• Models• Evaluation• Quality



Construction (2/2)MotivationPhilosophy in ISModel EvaluationDiscussion FrameworkApplicationConclusions

Philosophical aspect

IS r

ese

arc

h s

cho

ols

quality perception

Ontology [I] Epistemology [II]

evaluation perception

model perception


The BWW Ontology

• Wand and Weber applied an ontology based on Bunge’s work to the field of conceptual modeling

• The BWW Ontology specifies “reality” constructs that a conceptual modeling language should be able to depict

• The BWW ontology serves as a reference point in evaluation

• Evaluation is conducted by mapping language constructs against ontology constructs and thereby assessing ontological completeness and ontological deficiency within the modeling language


Cf. Wand & Weber (1993)

Cf. Bunge (1977)


Applying the framework

Consideration aspect Explanation and Reasoning

Following the positivist research approach taken (a both ontological and epistemological realistic position), the universe of discourse (UoD) comprises immutable objects and object structures that exist as empirical entities. Consequently, models of the UoD exist independently from any observer’s perception. Therefore, a conceptual ISAD model is a descriptive representation of the UoD. The model perception is more that of a reproduction or representation than that of a (re-) construction. A conceptual model is, in this perception, regarded to as an objective perspective upon reality through which observers can perceive reality (unbiased).

[a]Model perception

[b]Evaluation perception

According to the positivist research approach followed, the evaluation of ISAD models refers to an investigation of how well the model represents reality. The ontology hereby serves as a reference point for evaluation. Following the correspondence theory of truth, the BWW ontology is understood as a fact statement that is assumed to be objectively true. Evaluation in the BWW approach following this perception hence refers to a structural analysis of analogies and dissimilarities between ISAD modeling language constructs and ontology constructs, thereby determining statements in the model under observation which do or do not correspond to the "true" statements of the BWW ontology.


[c]Quality perception

In the tradition of the BWW approach, the quality of a model is determined through its compliance to reality. Specifically, the quality of an ISAD model is expressed through the degree of ontological completeness and ontological clarity of its conceptual modeling language. This quality perception does not include any reference to model purpose, developer, or addressee and is thus solely focused on the formal semantics, leaving out aspects of the symbolic interaction or pragmatic quality aspects. Quality of a model is in this belief expressible through the number of 1:1 mapping relationships of language constructs to ontology constructs and does not address the question of ist effectiveness for communication amongst model addressee communities.


Findings: Some Implications

• Application of BWW approach restricted to contexts adhering to same paradigm

• Positivist stance of BWW approach problematic“Positivism” should no longer even be mentioned in discussion of theory or epistemology in information systems as a defensible position.

Gregor (2004), p. 4

• Reconsideration and modificaiton of the BWW ontology?



Conclusions

• Evaluation is problematic due to its dependancy on paradigmatic viewpoints onto– Object of evaluation (models)– Evaluation methodology (obtainment of truth)– Evaluation target objective (quality)

• Multi-paradigm research during artefact design and evaluation is proscribed

• Need for more paradigmatic rigor in IS evaluation research



Thank you for listening


Selected References (1/2)MotivationPhilosophy in ISModel EvaluationDiscussion FrameworkApplicationConclusions

• Avison, D.E. and Fitzgerald, G. (1995) Information Systems Development: Methodologies, Techniques and Tools, McGraw-Hill Companies, London.

• Berger, P.L. and Luckmann, T. (1966) The Social Construction of Reality : A Treatise in the Sociology of Knowledge, Doubleday, Garden City.

• Bunge, M.A. (1977) Treatise on Basic Philosophy Volume 3: Ontology I - The Furniture of the World, Kluwer Academic Publishers, Dordrecht.

• Collier, A. (1994) Critical Realism: An Introduction to Roy Bhaskar's Philosophy, Verso, London.

• Gregor, S. (2004) "The Struggle Towards an Understanding of Theory in Information Systems" in D. Hart and S. Gregor (eds.) Information Systems Workshop: Constructing and Criticising, School of Business and Information Management, Canberra, pp. 1-11.

• Gruber, T.R. (1993) What is an Ontology?, Retrieved from http://www.ksl.stanford.edu/kst/what-is-an-ontology.html.


Selected References (2/2)

• Jayaratna, N. (1994) Understanding and Evaluating Methodologies: Nimsad, a Systematic Framework, McGraw Hill, New York et al.

• Losee, J. (2001) A Historical Introduction to the Philosophy of Science, Oxford University Press, Oxford.

• Siau, K. and Rossi, M. (1998) "Evaluation of Information Modeling Methods -- A Review" in D. Dolk (eds.) 31st Hawaii International Conference on System Sciences, Computer Society Press, Big Island, pp. 314-322.

• Stufflebeam, D.L. (2001) Evaluation Models. New Directions for Evaluation, Jossey-Bass, San Francisco.

• von Glasersfeld, E. (1987) The Construction of Knowledge. Contributions to Conceptual Semantics, Intersystems Publications, Seaside.

• Wand, Y. and Weber, R. (1993) On the Ontological Expressiveness of Information Systems Analysis and Design Grammars, Journal of Information Systems, 3, 217-237.


conceptual model evaluation. towards more epistemological rigor presentation at emmsad ’05 jan...

Documents

field of evaluation

unconscious philosophy

bad philosophy

agenda motivation philosophy

research conceptual

research paradigm

objective scope

problemsolving process