methods 2 - faculdade de engenharia da universidade do...

11/5/2009

1

��

��

��

�� ! "��# $ %&�� ! "��# $ %&�� ' (�� # ) �(��* �� ' ��(��+�� ' ��

,-) �(��# ) �� # �� -$ �

� �� # (��* � ��. "/��. ��. ��

' �� !# 0 ��1 ��# 0 ��1 $ ! "��(��

Eugénio Oliveira- 2009 MSR- Methodologies for Scientific ResearchMSR- Methodologies for Scientific ResearchEugénio Oliveira- 2009

' �� !# 0 ��1 ��# 0 ��1 $ ! "��(�� !#! "� �2�� $ %�� !# $ %�� "�� ' ��(��* ��

� ��! � � (��* � � ��! ��

Research Techniques

1.6 Techniques associated to Research


11/5/2009

2

I. Non empírical Techniques

Research Techniques

B- Techniques

II. Scientific Research positivist Techniques

III. Scientific Research Interpretative Techniques

IV. Scientific Research Techniques at the positivist/interpretative boundery

V. Scientific Research Engineering-based Techniques


I. Non empirical Techniques depend on:

• Artificially created data, or• Conceptual thinking about abstractions

Research Techniques

• Conceptual thinking about abstractions

• Includes:• conceptual research based on opinion and speculation.

( argumentative and dialectic analysis)• theorem proof: This applies to mathematical abstractions:

• formal methods application, • induction, • mathematical abstractions

Eugénio Oliveira- 2009 MSR- Methodologies for Scientific Research

• mathematical abstractions• Models verification• Simulation:

• Mapping a complex environment in a simplified Model.

MSR- Methodologies for Scientific ResearchEugénio Oliveira- 2009

11/5/2009

3

• Other non empiric Techniques:

Research Techniques

• Using hypothetic future scenarios• Mapping a real scenario into Games• ...

• Reviewing the State-of-the-art and doing Meta-analysis


��

��

Eugénio Oliveira- 2009 MSR- Methodologies for Scientific ResearchMarço 2007 FEUP-LIACC / Eugénio Oliveira

11/5/2009

4

II. Scientific Research positivist Techniques(applicable to IS and other Engineering disciplines):

• Techniques for forecasting applying

Research Techniques

• Techniques for forecasting applying • regression algorithms �' ��(��+��

�+�� • time-series based algorithms to extrapolate on historic

data• Field Experiments :

• Possibly isolating and controlling a set of phenomenato be looked at


• Laboratory Experiments:• Creating artificial setups in which phenomena, factors

and variables are isolated and controlled.


• Research approach is Descriptive/Interpretative:

III. Scientific Research Interpretativist Techniques:

Research Techniques

• Research approach is Descriptive/Interpretative: Going from empirical observations to a limited rigour analysis.Need for controlling the researcher’s perspective:

- Critical self- Examination - Revising all the postulates and biases- Varying Observations- Submission to peer reviewing


• Group Research: • frequent discussion with a group of people including

who may be affected by the application (or the technology)• Use Collaborative Work Tools.


11/5/2009

5

• Action-based Research:

- Researcher is like an “Agent” changing existent

Research Techniques

- Researcher is like an “Agent” changing existent conditions to measure the respective reaction

• Other Methods for Social Sciences(Ethnographic..)

In http://www.qual.auckland.ac.nz/ Michael Myers Ed. Ass.Inf.Sys.


In http://www.qual.auckland.ac.nz/ Michael Myers Ed. Ass.Inf.Sys.It is advocated qualitative research, for example, for IS.

We also can do (but it is not recommended) R&D just through Case-study analysis


IV. Scientific Research Techniques at the positivist/interpretativistboundery

Including:

Research Techniques

Including:- Field Work : Data related to the object of study is directly observed by the Researcher in the original context

- Questionary-based : Structured Data collection comming from interviewees

- Case Study: - Isolate and describe a relevant real-world


- Case Study: - Isolate and describe a relevant real-worldsituation,

- Gather a collection of data from multiple sources


11/5/2009

6

• Secondary Research:

Research Techniques

• Analysis starting from pre-existent documents (texts, papers, descriptions, memoirs).

• Analysis is done under new perspective and using different techniques or procedures.


v. Scientific Research Engineering-based Techniques(Informatics Engineering, IS, Electronic Engineering, Telecomm...)

• Divided into two categories:

Research Techniques

• Divided into two categories:

1- Construction Technique: • Includes Design, Project, Implementation

(or prototyping) of a system(ex: Computer program System, or physical device)

- objective:-- explicitly testing an hypothesis or


-- explicitly testing an hypothesis or -- to solve a class of problems.

- it implies:-- Existence of either a theory or a modelfor explaining the results (testing the hypothesis)


11/5/2009

7

Research Techniques

Materials and Methods

• in exact sciences materials include definitions,

• theoretical model is defined (preferably a mathematical model),including boundary conditions (define the environment)and initial conditions (define the initial values of the

• in exact sciences materials include definitions,model, and related assumptions


and initial conditions (define the initial values of theparameters of the model) [Losee01]

J. Losee, A Historical Introduction to the Philosophy of Science,4th ed. Oxford Univ. Press, 2001.


Research Techniques

• methods include:rules of analysis and rules of verification and validation

• rules of analysis follow deductive reasoning,statistical analysis and approximations (analysismust often be replaced by simulations due tomathematical tractability problems)

• the theoretical model is verified by comparing the results

rules of verification and validation


• the theoretical model is verified by comparing the results with reality (measurements with a prototype, known as hypothetical-deductive method)


11/5/2009

8

Research Techniques

• Results

• results can be analytical (deductive), from simulation, or

• use analytical results in simple limited cases to obtain referencevalues

• results can be analytical (deductive), from simulation, ormeasurement results

• it is important to present numerical results thatverify (or not) your own thesis,


values

• the value of the results depend on how well they can be generalized (bottom up approach will help you)


Conclusions (constructive technique report)

(1) “We have shown that [some] algorithms proved right…”

(2) “Our approach is simpler to implement than previous algorithms and is

Research Techniques

(2) “Our approach is simpler to implement than previous algorithms and is practically feasible …. On the theoretical side, the greatest deficiency in our results is …”

(3) “In a practical situation the advantages and limitations of our approach are…”

(4) “These results can be extended in a number of directions…”

Eugénio Oliveira- 2009 MSR- Methodologies for Scientific ResearchMSR- Methodologies for Scientific Research

1. What is shown by this work and its significance.2. Limitations and advantages.3. Applications of the results.4. Recommendations for further work.

Eugénio Oliveira- 2009

11/5/2009

9

• Destructive Techniques:

Research Techniques

• To analyse a system, or a set of methods to gather new information about those technology procedures under analysis.

- Recognized information and characteristics are then reused may be in a different way


- Example: Reverse Engineering


• There are philosophies permitting the Classification of Research Methods through Paradigms or basic Epistemologies:

Phylosophy Underlying Research

I. PositivistII. Interpretative (post-positivist)III. CriticalIV. (Constructivism)

At Harvard Business School they use just the three categories


At Harvard Business School they use just the three categories


11/5/2009

10

• Positivist Method:

• Assumes that reality is objectively accessible, it can be

Philosophy Underlying Research

• Assumes that reality is objectively accessible, it can be measured and described by the observer and measured by the same observer through their instruments and devices.Tests are made making predictions possible over future phenomena

• IS research (also Telecomm research?) can be of the positivist kind (for IS, Orlikowski and Baroudi,1995)- Use math/logic formalisms


- Use math/logic formalisms- use quantifiable and measurable variables- Test an Hypothesis and make inferences from what is known.


• Interpretativist Method

• Assumes that accessing to reality is only possible through social constructions like language, consciousness and


social constructions like language, consciousness and ontologies (sharing of concepts’ meaning).

• Hermeneutics-based philosophy (based on the human understanding and text interpretation) as well as in phenomenology(describing, understanding and interpretation perceived phenomena)


phenomena)

Proposes to abolish separation between "subject“ and "object", It opposes to XIX century positivist thinking.


11/5/2009

11

• Can IS and Telecomm studies influence or be influenced by


the context?

• Here, the focus is not as much on the variables, measurements and processes as it is on interpretation of what is being the results of the test, the measurement or implementation


• Can IS and Telecomm studies influence or be influenced by


the context?

• Here, the focus is not as much on the variables, measurements and processes as it is on interpretation of what is being the results of the test, the measurement or implementation


11/5/2009

12

• Research Criticism:

• It is a radical enterprise once it tries to identify what constraintsresearch:


research:• socially, culturally,• Historically, politically, • economically, physicallytrying to emancipate science by eliminating causes for ignorance,alienation etc...

Ex: Why is it not possible to do research on X?


Ex; Why cannot we implement systems of type Y?

Habermas from Frankfurt School following the way of Adorno, Lukacs, Marcuse


• Junk Science (?) Stories of the Past Decade

Dial “F” for Fear.


Dial “F” for Fear.

Since the 1993 Larry King Live broadcast featuring a man suing a cell phone maker claiming his wife died from a cell phone-induced brain cancer, many cell phone users have worried about phone safety.

But studies failed to identify any risk.


11/5/2009

13

• Junk Science (?) Stories of the Past Decade


Powerline scare unplugged.

Fears that electric and magnetic fields (EMFs) created by power lines and appliances caused cancer started in 1978. Parents worried about power lines over schools.

Power companies worried about burying power lines. The National


Power companies worried about burying power lines. The National Academy of Sciences finally unplugged the scare in October 1996, concluding that no evidence showed EMFs presented a health hazard


• Junk Science Stories of the Past Decade

The Lone-Tree Theory.


The Lone-Tree Theory.

It nearly took an act of Congress to get the researcher behind the notorious “hockey stick” graph, which purports to show a steep rise in global temperature in the 20th century following a millennium of stable temperatures, to release his publicly funded data and computer code.


Among other dubious presumptions, the graph is derived from data that bases climate estimates for the entire 15th century on the tree ring measurements of a single tree.


11/5/2009

14

Quality of Research

1.7 Quality of Research


Quality of Research

• Quality depends on the Objectives to reach1. To increase available (and also general) Knowledge

• Objectives and Quality criteria are internal to the discipline• Objectives and Quality criteria are internal to the discipline• RIGOUR is very critical

2. Developing Computer Systems– (Instrumentalisme)• Objectives and Criteria are external (regarding R&D)• Relevance can be measured by the impact and is of the

most importance


• Relevance Versus Rigour• Not antagonic but taken with different emphasis


11/5/2009

15

• Computer Science (and other Engineering-like areas) tendency:

• Until the 80s � conceptual (not empirical) papers

Quality of Research

• Case-studies based research is mainly descriptive concerningsimple instances and then less scientific

• Until the 80s � conceptual (not empirical) papers dominated

• After the 80 s� Less theoretical papers, more instantiated, less general and Knowledge less cumulative


• Empirical research is now dominated by the descriptive approaches

• Specific Models have limited capabilities for explaining aphenomen as well as little predictive power.


• Informatics and Computer Science Research:

• Services, Platforms and Frameworks for Electronic

Quality of Research

• Include support to:

• Logistics, procurement, contracting, trading

• Services, Platforms and Frameworks for Electronicinter-operability between Organizations (enterprises, schools, institutions…) inside themselves and interfacing to society (markets)


• Logistics, procurement, contracting, trading• Reorganization and reconfiguration


11/5/2009

16

• Informatics and Computer Science Research:

• Algorithms and Systems for Information management:

Quality of Research

• Supporting Sciences:

• Biotechnology

• Algorithms and Systems for Information management:• Visual (image, graphic…)• Textual (Corpus, data in DB)• Signals (traffic, biologic, environmental….)


• Biotechnology• Graphic Computing• Medicine• Law• Astronomy, Phisics, History, Management….• Reorganization and reconfiguration


Quality of Research

• Requirements for the Research in CS and TC

• Choose an appropriate Research Method:• Choose an appropriate Research Method:

• Make explicit a Body of Theory sometimes coming fromother adjacent disciplines (ex: Computer Science, Organization

• Traditional/positivist, “interpretativiste” or Engineering-oriented


other adjacent disciplines (ex: Computer Science, OrganizationTheory, Information Theory, Physics, Statistics…)

• Extend the Theory and Infer predictions to be checked(at least in the case of positivist/traditional research)


11/5/2009

17

(following Björn Lisper, Datorteknik, Mälardalens högskolaVästerås, Sweeden)http://www.idt.mdh.se/kurser/ct3340/ht03/

Scientific Methods, General

Theoretical methods:

Create formal models (mathematics, logic)

Define concepts within these

Prove properties of the concepts


Prove properties of the concepts

Abstraction, hide details to make the whole more understandable(and to make it possible to prove properties of it)

Proofs of properties by deductive methods


Empirical methods:

Perform experiments

See how it turned out

Scientific Methods, General

See how it turned out

Draw conclusions

Simulation:Start with a formal model at some "easy-to-understand" level

Make "artificial experiments" in your computer


Collect statistics and draw conclusions


11/5/2009

18

In Physics:Make hypotheses about the surrounding world (theory), observe it (experiment)Relate the result of experiment to the theory

Scientific Methods

Adjust the theory if it doesn’t predict the reality well enoughTheory is used to predict the future (e.g., if a bridge will hold for acertain load, or an asteroid fall down on our heads)

Common pattern in Computing Science:The system is constructed to behave according to some theoretical model


theoretical modelDeviations are seen as construction errors rather than deficiencies in the theory (hardware error, bug in OS, : : :)

In both cases: the theory helps us understand and predict, but indifferent ways!


Scientific MethodsTheoretical vs. Empirical Methods in Computing Science

Computing Science really has a “spectrum”, from “extremeconstructivism” to a use of theory close to the one in physics:constructivism” to a use of theory close to the one in physics:

• “Extreme constructivism”: (ideal) programming language design:

Formal semantics for the language, pure construction of modeldefining the mathematical meaning of each program

Abstraction of details to make the meaning of the language


Abstraction of details to make the meaning of the languagesimpler (for instance, assume that data structures can growarbitrarily big)Implement the language according to the semantics


11/5/2009

19


One can prove formally within the model that a program is correct –valuable!

But the model does not cover all kinds of failures. E.g., hardware errors, or stack overflow (or an asteroid falling down on the computer)


asteroid falling down on the computer)



Extreme “physics” approach: performance modeling of complex computer-and communication systemsand communication systems

Extremely hard to make analytical calculationsSimplified performance models, tested against experiments (e.g., long suites of benchmarks)

Discrepancy leads to a modified theory, as in physics


Discrepancy leads to a modified theory, as in physicsOften simulation (desire to evaluate systems before building them)


11/5/2009

20


In-between: algorithm analysis

Build on some form of formal model for how the algorithm executs (meta-language with formal semantics), and some performance model (how long does a step in the algorithm take, how much memory is needed to store an entity)



In-between: algorithm analysis

Given that the performance model is correct, one provesmathematically that the algorithm needs certain resources (time, memory) to be carried outBut the performance model is often very approximate.

Sometimes is possible to refine the performance model,


Sometimes is possible to refine the performance model, but this can make it impossible to calculate the resource needs of the algorithm


11/5/2009

21

Quality of Research

• Characteristics of the Research in CS and TC(?)

• Report the S-o-A. • Report the S-o-A. • Prove you made an advancement in Knowledge• Combination of Techniques trying to compensate one’s

weaknesses with other one strengths

• Report about the object of study and the nature of the domain• Validate Data• Analysis using statistics• Extract relationships between variables


• Praticable according to the resources• Relevant results (at different levels: community,society...)• Publish• Ambitious


• Characterístics of the research in CS:

• Research may have as an objective:

Quality of Research

• Research may have as an objective:• Building a Theory• Testing a Theory• Extending a Theory

• The practical research objectives may include:• Exploring a specific domain• Descreving a domain


• Descreving a domain• Explaining a domain• Forecasting over a domain• (Testing formalized hipothesis)

11/5/2009

22

• Other characteristics:

• Concerning Data:• Extensive

Quality of Research

• Extensive• Representative• What is Data validity (ex. temporal, precision…)• Granularity:

• Simple or agregate• Specific or large generic experiments

• Temporal Horizon:


• Temporal Horizon:

• “Snapshot” of the domain at a specific point in time• “Longitudinal cut section” (múltiple points in time)

• Methods for colecting and analising data; quantification...


Quality of Research

1.8 Challenges


11/5/2009

23

Research in CS and TC Challenges

• Challenges:• There exists shortage of theory. Mainly theories in related disciplines• Phenomena are unstable:

• Data may get old quickly, usually a snapshot that may varyPay attention to the validity period

• Organizations and Markets are always moving

• Significant cultural variations:• Different national/regional cultures imply different speed/rhythm

• When you are doing research (requirements, interaction,questionnaires, demos) you influence the domain


questionnaires, demos) you influence the domain

• Pressure for publishing during the research action


• Pressure for technical instantiation:

• Prototyping without a theoretical ground or well defined objective


objective• Economics and business models either not known or consolidate.

• Impact of the automation and rationalization:• Depends on time dynamics concerns• Depends on who are directly affected


• Adopting a new technology may take long time• Impact on other partners, markets, standards, tools…


11/5/2009

24

• Conclusions on Research actions in CS and TC

• Consider tradoff Rigour Versus Relevance• Identify challenges BEFORE starting doing research


• Be clear about conventions related with researchto be done

• Select a significative sub-domain • Define questions to be answered through that research• Select the apropriate Research Method and Techniques

(possibly a good combination of some) and justify the choice.

Whenever the selected aproach is instrumentalist, you need


Whenever the selected aproach is instrumentalist, you need to be sure about the external quality, data validation and rigouras well as:Relevance


Other Bibliography

• D. Sternberg, How to Complete and Survive a Doctoral Thesis. • D. Sternberg, How to Complete and Survive a Doctoral Thesis. St. Martin’s Press, 1981.


methods 2 - faculdade de engenharia da universidade do...

Documents