hans akkermansis research methodology (06sep2010)1 notes on scientific method hans akkermans

Hans Akkermans IS Research Methodology (06Sep2010) 1

Notes on Scientific Method

Hans Akkermans

Issues of Research Methodology

If so, what are these rules actually in our science(s)?Hans Akkermans IS Research Methodology (06Sep2010) 2

Do we do it this way? … or that way? …

Or do we follow The Rules (of Scientific Method)


What is Science? (MyDef)

Originally, term for systematic inquiry (“broad church”, inclusive formulation)

[ what I personally still continue to prefer]

A social – institutionalized – practice focused on the production of claims to knowledge through a process of inquiry, in a way that is: Relevant: produced knowledge is about something of

interest/importance [Q1 → what does that mean?] Systematic: process of inquiry is carried out in a well-

organized, critical and rigorous fashion [Q2 → i.e?] Transparent: claims are produced and argued for

such that they are clear and open to critical scrutiny for/by others [Q3 → who are they?]


Many different scientific research goals exist [as a matter of fact] Description Understanding Explanation (e.g. causal) Prediction Guidance for action

Design (e.g. informatics, engineering, architecture, urban planning) Intervention (e.g. therapy, medicine, organizational change) Professional practice (e.g. law); decision making (e.g. management, policy),

Combinations possible and quite common Note1: different disciplines strongly differ in view on what are “proper”

goals of science and research Note2: also linked to different philosophical positions on science

Reference vs. coherence vs. interpretivist vs. pragmatic theories of truth Realism vs. positivism vs. constructivism vs. postmodernism vs.

pragmatism (cf. reflective practitioner) Hence: varying positions on what science is or ought to be


Typical academic misunderstandings

By other disciplines about I/CS It’s engineering (i.e. useful but

not really science) Where is the theory? Results: Any universal laws? Does I/CS have any empirical

nature/content? Isn’t experiment absent? Big difficulty in understanding

I/CS-style conceptualization Idem: what counts as a proper

research claim or contribution Generally, no clue about the

computational paradigm as scientific method

Design as a proper research goal? How can that be??

By I/CS researchers about others How key notions such as theory,

hypothesis, empirical research, experiment, validation/validity are seen in other disciplines

Knowledge about quantitative and qualitative research methodology Although commonly used in I/CS Methodology often left quite (too)

implicit [you wouldn’t get away with this elsewhere!], OR: naïve import

Ignorance about thinking and results from e.g. social and other sciences

Generally, big cultural gap wrt. nature and esp. justification of research claims

Method as the toolbox of the researcher Formal methods F

Logic(s) Math Formal specs

Inductive-statistical (“quantitative”) methods S

Qualitative methods Q Interview Field observation Case study Action research

Computational methods C

Modelling and simulation

Demo programs and architectures

Design methods and rules

Protocols, algorithms

Experimental methods E



Experimental Method(s)

The “classic” laboratory experiment [physics etc.] Randomized Controlled Trials (RCT) [medicine etc.] Field test [e.g. engineering, innovation] Quasi-experiment [psychology, cf. Donald Campbell] Thought experiment [e.g. Einstein, Bohr]

One may view modelling & simulation as a further developed, formalized and computerized version of this

Campbell (2003, discussing case study vs. experiment) Core of the scientific method is strategy to deal with “plausible

rival hypotheses” (and not experimentation per se) Not (positivistic) corroboration of context-independent

hypotheses, but evaluation of “extended networks of implications” of rival hypotheses [“ramification extinction”]

Research is a 3-place relation (not 2) Research = make [argued] claims to knowledge:

ABOUT the world W (some part of reality) TO a community H (some part of humankind)


W (world)

H (audience, “your peers”) S (you as researcher)

REPRESENTATION

APPEALEXPRESSION

Cf. Buhler’s linguistics/communication diagram (1930’s)

Note1: ContextW

importance

Note2: ContextH

importance


Argument and “Informal Logic”

D + T R C - Core idea of scientific argument Data plus Theory produce Conclusions by means of Reasoning

Toulmin (1958, 2000): Reasoning is field-dependent: each domain has its own knowledge and forms of reasoning (non-universal logic)

Test questions for argument acceptability (Fisher, 2004) [Note: argument can be represented as directed and/or graph]

AR (argument reconstruction): what is the claim, and what reasons are presented to justify the conclusion?

Principle of Charity: construe argument favourably, by adding unstated, implicit but necessary, reasons or assumptions

TQ1: Are the reasons (premisses) true? TQ2: Are there situations in which all premisses can be true and

conclusion false? AQ (assertability question): What evidence/reasons would justify

one to assert the conclusion?


Validity and the Structure of Argument

(1) Descriptive validity (D) (2) Theoretical validity (T) (3) Interpretive validity [D + T] (I) (4) Validity of Reasoning [consistency, coherence] (R) (5) Internal validity [+ qualifiers …] (CINT) (6) External validity (CEXT) = Generalizability

(Toulmin, 1958)

Freedom & the Principle of Methodological Accountability I/CS is methodologically relatively liberal (good) but

also often relatively unreflective (not so good) Either: you follow standardized, accepted method

in a field (e.g. quantitative, economics) Safe research route (mainstream, but danger: maybe boring)

Or research freedom: construct your own method But then: you are accountable and have to argue for it

Science historical note: ground-breaking research often invents its own (new, so contested) method

E.g. Leibniz/Newton: differential calculus (etc. etc.) I/CS: computational paradigm (!)


How/where I/CS can become (methodologically) better What is your research

claim precisely? Context!: What difference

does your contribution make (the “delta”)

And how did you get there? Argument A → B → C

→…. Claim Transparent =

scrutinizable for others Note: overall argument is

“informal” (even in math/logic!)

Typical mistakes: Claim too implicit NOT: the work you did (e.g.

simulations; but: what do they tell us)

Claim too strong or too weak (qualifications etc.)

Overall argument too implicit, or components of argument weak / dubious

= do not pass test questions of argument validity


hans akkermansis research methodology (06sep2010)1 notes on scientific method hans akkermans

Documents

issues of research methodology

research note2

empirical research

ics methodology

proper research claim

proper research goal

scientific method design

rules of scientific