comparing interactive online and face-to-face repertory grid interviews in terms of construct...
TRANSCRIPT
Comparing interactive online and
face-to-face repertory grid interviews
in terms of construct quality and
practicability
Matthias Kaulartz & Mark Heckmann University of Bremen, Germany
EPCA, Brno, June 21, 2014
movie&pilot&&300&par.cipants&
h"p://www.intanges.com/relaunch/index.php?id=8&L=1=
Tourism&in&EU&&1,000&par.cipants&
Research&on&automated&grid&administra.on&is&sparse&
• Research&by&Gaines&&&Shaw&in&the&80s&(1980;&e.g.&PEGASUS),&but&no&systema.c&research&on&outcome&quality&
• Van&de&Kerkhof&et&al.&(2009,&p.&430):&– “Quite&some&constructs&men.oned&were&unclear&to&the&analysts&of&the&research&team.&Many&were&nonVbipolar&....&Moreover,&in&a&faceVtoVface&interview&the&interviewer&can&check&if&two&constructs&are&similar&to&the&respondent.”&
Current&situa.on&
Automated&grid&administra.on&is$used$by&commercial&companies&and&par.ally&in&research,&
but&there&is&virtually&no$research$on&outcome&quality&of&online&grid&administra.on&
Research&ques.ons&• How&can&we&do&“good”&online&repertory&grid&administra.on?&• Is&it&possible&to&get&results&comparable&to&faceVtoVface&interviews?&
&
Online&grid&situa.on&
No&chance&to&ask&for&clarifica.on&
Quality&of&verbal&labels&crucial&to&infer&meaning&
Here:&Quality&of&constructs&refers&to&quality&of&verbaliza.on&
nonV&bipolarity& repeated&
constructs&
poor&verbaliza.ons&
mortality&
Virtual&interviewer&as&
harness&
unintended&ra.ng&
behaviour&
Study&Design&• n&=&80&• Two&groups&(faceVtoVface&&&online)&• 31&+&49&subjects&&
• Average&age:&24,1&years&• Average&number&of&shoes:&21,6&• !&516&constructs&in&total&• Three&expert&raters&rate&construct&on&four&dimensions&of&construct&“quality”&
1.&Understandability&(of&verbaliza.on)&
Is&the&seman.c&verbaliza.on&of&the&constructs&clear&to&understand&for&a&third&person?&
&Example:&Good:&“This&shoe&makes&me&feel&comfortable&–&makes&me&feel&uncomfortable”&Bad:&“This&shoe&is&so&‘shoeVish’&–&this&one&isn’t”&
2.&OneVdimensionality&
Has&the&construct&just&one&dimension&or&does&one&of&the&poles&include&more&than&one&aspect?&
Example:&Good:&& &“This&shoe&is&for&daily&use&–&only&for&& &&&special&occasions”&
Bad:&& &“Feels&comfortable,&looks&good&and&is&& &&&cheap&–&…”&
3.&Appropriate&level&of&abstrac.on&(with&respect&to&aim&of&study)=
&&Highly&abstract&
Very&concrete&
$
„I&like&it&–&I&don‘t&like&it“&
$
„Shoe&–&Gumboot“&&
“Makes&me&friendly&and&selfVconfident&V&…”&
4.&Overall&ra.ng&of&construct&quality&
“Intui.ve”&score&by&expert&raters&with&experience&in&grid&studies&using&content&analysis&
Examples:&Good:&&“Makes&me&feel&happyVgoVlucky&–&&& &Makes&me&feel&weighted”&
Bad:&&“High&or&low&heels&–&Medium&height&heels”&
Dimensionality:&Is&the&construct&oneVdimensional?&
***&***&Δ=33,5%&
***&Δ=29,3%&Δ=27,0%&Δ=31,8%&
***&
Understandability:&How&understandable&is&the&construct&for&the&rater&on&a&scale&from&1&to&5?&
*&*&d=.28&
*&d=.26&
d=.27&
Reverse&effect&(!)&
Results&in&a&nutshell&• Online&grid&performs&significantly&worse&on&oneVdimensionality&and&overall&quality&• No&significant&differences&for&abstrac.on&• Reverse&effect&for&understandability&(?)&
!&While&faceVtoVface&interviews&achieve&bever&results,&online&administra.on&is&able&to&generate&quality&verbaliza.on&of&constructs&
What’s&next?&• Which&design&elements&help&to&…&
&…&generate&oneVdimensionality?&&…&systema.cally&steer&the&abstrac.on&level?&
• Differences&with®ard&to&other&grid&measures&of&interest&(e.g.&complexity)&• How&can&we&make&online&grid&administra.on&(more)&fun?&
Gaines,&B.&R.,&&&Shaw,&M.&L.&G.&(1980).&New&direc.ons&in&the&analysis&and&interac.ve&elicita.on&of&personal&construct&systems.&Interna?onal=Journal=of=ManCMachine=Studies,&13(1),&81–116.&
Honey,&P.&(1979).&The&repertory&grid&in&ac.on:&How&to&use&it&to&conduct&an&awtude&survey.&Industrial=and=Commercial=Training,&11(11),&452–459.&doi:10.1108/eb003756.&
Russell,&C.&G.,&&&Cox,&D.&N.&(2003).&A&computerised&adapta.on&of&the&repertory&grid&methodology&as&a&useful&tool&to&elicit&older&consumers’&percep.ons&of&foods.&Food=Quality=and=Preference,&14(8),&681–691.&
Van&de&Kerkhof,&M.,&Cuppen,&E.,&&&Hisschemoller,&M.&(2009).&The&repertory&grid&to&unfold&conflic.ng&posi.ons:&The&case&of&a&stakeholder&dialogue&on&prospects&for&hydrogen.&Technological=Forecas?ng=and=Social=Change,&76(3),&422–432.&
&&&
References&