HUMAN-COMPUTER INTERACTIONMETHODOLOGY MATTERS

Professor Bilge MutluComputer Sciences, Psychology, & Industrial and Systems Engineering

University of Wisconsin–Madison

CS/Psych-770 Human-Computer Interaction

Doing research is the systematic use of some set of theoretical and empirical tools to try to increase our understanding of

some set of phenomena or events.

BASIC FEATURES

All research brings together:

Content

Ideas

Techniques and procedures

THREE DOMAINS OF RESEARCH

Substantive

Actors and context

Conceptual domain

Behavior or relations

Methodological domain

Modes and techniques

TECHNIQUES

Techniques for measurement

Techniques for manipulation

Techniques for controlling impact

MEASUREMENT

Measuring some feature of a research situation

MANIPULATION

Systematically varying components of a system

Giving instruction

Imposing constraints

Selecting materials

Giving feedback

Using confederates

CONTROL

Controlling the impact of various extraneous features of the situation

Experimental control

Statistical control

Distributing impact (e.g., randomization)

COMPARISON

Independent variables

Dependent variables

The research process, like a three-legged stool, always depends on

materials from all three domains—content, ideas, and techniques.

LIMITATIONS

Methods pose opportunities and limitations;

Might have weaknesses that limit evidence

Can offset weaknesses by using multiple methods

CHOOSING A SETTING

Three key considerations

Generalizability

Precision of measurement

Realism of the situation

Seek to maximize all three

Not attainable but we do our best

RESEARCH STRATEGIES

Field strategies

Experimental strategies

Respondent strategies

Theoretical strategies

oftheothertwo.Forexample:

Thethings

youcandototrytoincreasetheprecision

i:O'"'i"!:l

....

.t;:so 0';'

U1

with

which

youcanmeasure

behavior

andcontrolrelatedvariables

(B)(forexample,

"$:

...~~en

conductingacarefullyconLl'olledlaboratoryexperiment)willintrudeuponthesituation

om>

;::~a

w~"

'.andreduce

its"naturalness"

orrealism

(thatis,reduce

C),andwill

also

reduce

the

IIIIII

~JJ

(:)~

3:rangeofactors

(A)towhom

thefmdings

canbe

generalized.

Conversely,

thethings

you

JJ~(j)

3'CD

:1C'\

01/1

91u

CD~CD

ID"'C

G)

ID::J

o0

candototry

tokeephigh

realism

ofcontext(C)(forexample,conductingafieldstudy

=_.~

0'::JI:

...~!e.III

...C,

IDII)

inanaturalsituation)willreduceboththerangeofpopulationstowhichyourresults

0=~!!!

~~--

.-+::IN

!!!.

Ro~ID

-ID

;J'canbeapplied(A)andtheprecisionoftheinformationyougenerate(B).Asathird

g;;:r

m::J

~~~0

::;--

example,thethingsyoucandototrytoestablish

ahighdegreeofgeneralizabilityover

::I:

ID=

~CD

~-a

actors

(A)(forexample,

conducting

awell-designed

sample

survey)will

reduce

real-

::J.

0iii'

~~1/1

ism(C)by

obtaining

themeasures

outofcontext,

andwill

reduce

precision

(B)both

byhaving

measures

ofonly

alimited

number

ofbehaviors,

andby

failing

tocontroloroth-

erwise

take

into

accountextraneous

factorsthatmay

affect

results.

me..

Youcanappreciate

thisdilemmabetterby

examiningsomeofthemajor

research

-I: Cc,

c'(Q

strategiesusedinthebehavioral

andsocialsciences.Figure2showsasetofeightalter-

'<3

mm

CDnativeresearch

strategies,orsettingsforgatheringresearch

information.

Inthatfigure,

I:D!

a<3

theeightstrategiesareshownaslyinginacirculararrangementinrelationtotwo

CD"'C

'<(jj'

underlyingdimensions:thedegree

towhich

thesetting

usedinthestrategyisuniversal

mr

orabstractvs.particular

orconcrete;andthedegree

towhich

thestrategyinvolvespro-

)(& -go

ceduresthatareobtrusive,vs.proceduresthatareunobtrusive,

with

respecttotheongo-

::J.iiI

ier

inghuman

systems(theactor-behavior-context

units)thataretobetheobjectofstudy.

a-<

ThefOUTstrategies

ontherightsideofthecircleinvolvefairlyconcrete

orparticularis-

ticsettings;thefourontheleftsideusefairlyuniversalorabstractsettings.Theproce-

duresusedinthefourstrategiesinthelowerhalfofthecirclecanbefairlyunobtrusive.

Thefourstrategiesinthetophalfofthecirclenecessarily

useprocedures

thatarefair-

mo

lyobtrusive,

thatis,theydisturb

theongoinghuman

systems(theactor-behavior-con-

~r ~E."'C

textunits)thatarebeingstudied.

!!!.C

ci"CD

Figure2alsoshowswhere,amongthestrategies,eachofthethreedesired

features,

::J'"

orcriteriaisatitsmaximum.CriterionA,generalizabilitywithrespecttothepopula-

m-n

[~tionofActors,ispotentially

maximizedinthesamplesurveyandinformaltheory.

,<c,

Criteriona,precisionwithrespecttomeasurementandcontrolofbehaviors,ispoten-

m"

tiallyatitsmaximum

inthelaboratoryexperimentandinjudgmentstudies.Criterion

)(-.

-g~

C,realism

ofcontext,ispotentiallyatitsmaximum

inthefieldstudy;Thegeometryof

::I.

.3

>figure2-1emphasizes

thedilemmajustdiscussed,

namely:

strategiesthatmaximize

CD ar::r !4.

oneofthesearefarfromthemaximum

pointfortheothertwo.Theverysamechanges

iiI~in researchproceduresthatwouldletyoumovetowardthemaximum

ofanyone

of"

thesecriteria-A,a,orC-atthesametimewouldmoveyouawayfromthemaxi-

...~o

Ii0

c'cmum

pointoftheothertwo.

Itisnotpossible,

inprinciple,

tomaximize

allthree

crite-

0mID

::J_c,

();i1

riasim

ultaneously.Thus,anyone

researchstrategyislimitedinwhatitcanachieve.

iii-::J

;-CD_

(Q-

Research

done

byanysingle

strategy

isflawed,

although

thevarious

strate!";"s

are

iii"

1/1

flawed

indifferent

ways.

QUESTIONS?

COMPARISON TECHNIQUES

Correlation

Systematic covariation in the values of two properties of a system

Linear or nonlinear

Difference

Variance in a variable across different levels of another variable

RANDOMIZATION

Random assignment of cases to conditions

A “true” experiment

Sampling technique plays an important role

VALIDITY

Internal validity

Construct validity

External validity

MEASUREMENTS

Self-reports

Observations

By visible or hidden observers

Archival records

Private or public

Trace records

MANIPULATIONSelection

Varying the population across conditions

Direct intervention

Varying the structure of or processes in a system

Indirect inductions

Evoking varied responses

SUMMARY

Methods dictate the results the researcher will obtain

This is why it is extremely important to report all details of your method

It is impossible to maximize all desirable features of a method

Hence we have “limitations” sections in our papers

You need to interpret your results in the light of other related results

This is why we include relevant background in our papers and interpret our results in the light of the results from this background

QUESTIONS?

THANKS!Professor Bilge Mutlu

Computer Sciences, Psychology, & Industrial and Systems EngineeringUniversity of Wisconsin–Madison

CS/Psych-770 Human-Computer Interaction | Fall 2010