an empirical evaluation of 2d and interactive 3d terrain ... · an empirical evaluation of 2d and...
TRANSCRIPT
■ Myles Cook
MSc Thesis Defence
Cartography MSc
15th May 2017
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1 Introduction
2 Context & Background
3 Research Questions
4 Methodology
5 Results & Discussion
6 Conclusions
7 Limitations & Future Research
P R E S E N TAT I O N
OUTLINE
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1 Introduction
•Howcanwebestshow the hillsonacyclingroute?
•Whichvisualisationsofterrainaremost usableforacyclist?
•Whichvisualisationsaremoreusableforcycleroute-planning–2D or 3D?
What problems stimulated the research?
To assess the relative usability of 2D and 3D elevation visualisations for
cycle route planners
Broad Aim
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1 Introduction
•Growthintheubiquityof3Dandinteractivecartographies,butdotheyofferanyrealbenefits?
•Researchinto3Dcartographyusabilityhasdrawndiffering conclusions,andsuffersfrommethodological flaws.
•Forcyclists,terrain is important.Bettercommunicationofterrainshouldmeanbettercyclingmaps.
•Thereisaneedforbetter design guidelinesforcyclingmaps.
Justification – why do we care?
2D
vs.
3D
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
2 Context and BackgroundWhat other work has been done in this field?
Expand on Brügger’s research by looking at 3D visualisations, while addressing that study’s flaws.
CyclingMap
Research
•Maps to make cyclists ‘happy’ (Dickinson, 2012), and ‘universal’ maps to meet the needs of all users (Wessel & Widener, 2015).
•Most relevant research – Brügger et al (2016):
▪︎ Study of the usability of 2D terrain depictions for cycle planners.
▪︎ Results showed that arrow symbolisation was most efficient & effective for determining height, and colour for determining slope.
▪︎ User preference = elevation profile.
▪︎ BUT! Methodological flaws (potential learning effects) & a-typical visualisations...
THIS THESIS:
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
2 Context and BackgroundWhat other work has been done in this field?
So, the need for further research into 2D and 3D terrain visualisations for cycling maps is still there!
2Dvs.3Dcarto
research
•A larger body of research has focussed on the usability of 2D and 3D cartographies in general.
•But, it has issues...
▪︎ Results can’t necessarily be applied to cycling maps/route planners.
▪︎ Some research has suggested 3D is more usable than 2D, other research the opposite.
▪︎ Focusses mainly on 3D terrain models.
▪︎ Majority use static rather than interactive 3D depictions.
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
3 Specific Research Questions
Toassesstherelativeusabilityof2Dand3Delevationvisualisationsforcyclerouteplanners.
1.Isthereasignificantdifferenceintherelative efficiencyof2Dand3Delevationvisualisationsforcycleroute-planners?
2.Isthereasignificantdifferenceintherelative effectivenessof2Dand3Delevationvisualisationsforcycleroute-planners?
3.Dousersprefer2Dor3Delevationdepictionswhenperformingcycleroute-planningtasks?
Research Questions
Broad Aim
What are we trying to find out?
Efficiency + Effectiveness + User Preference = Usability (ISO 9241-11)
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
4 Methodology•Conductedanempirical user-study(repeatedmeasures,36participants),undercontrolledexperimentalconditions.
•Testedsix different visualisation types:
Terrain
Por terBr ook
BlackbrookWood
Nor
thf i
eld
Road
F ulwo od Roa d
Tom Lan e
O ak br oo k Roa d
Ly dg a te Lan e
Tof t s L ane
S lay l e ig hLa ne
Gr a h am R oad
Car si ck
H i llR o ad
Net her g reen Roa d
San dy gateRo ad
Rust l ings Roa d
G l adston eRoa d
Red mir es R oad
Hag
gLa
ne
Hal l am sh ir e Roa d
T ink er
L ane
B ol eHi ll
Roa dH opw oo
dLa
ne
Boleh ill L ane
Bing ley L an e
Lo ng L aneRos coe
Ba nk
Han
gin
gw
ate r
Ro a
d
Cold
well
L a ne
Cr o ok es
Crim
ic arL an e
Ivy Par kRoa d
Ste p he n
Hil l
Ha llam
G
range
Road
Back
L a
ne
Manch ester R oad
Manche ste r Road
Manche ster Road
Rivelin Valley Road
A57
A625
A57
A610 1
Rivelin Glen
WindleHou se Farm
Samehill Farm Hagg LaneCott ages
BingleyVilla
EndcliffeCrescent
Ranmoor
Endcliffe Edge
Carsick
Tapton Hill
Crosspool
Stephen Hill
Lydgate
Den Bank
Crookes
Carsick Hill
Hallam Head
Sandygate
Rivelin Glen
Fulwood
Stumperlowe
Path Slope
Slight Uphill Slope
Moderate Uphill Slope
Steep Uphill Slope
Por terBr ook
BlackbrookWood
Nor
thf i
eld
Road
F ulwo od Roa d
Tom Lan e
O ak br oo k Roa d
Ly dg a te Lan e
Tof t s L ane
S lay l e ig hLa ne
Gr a h am R oad
Car si ck
H i llR o ad
Net her g reen Roa d
San dy gateRo ad
Rust l ings Roa d
G l adston eRoa d
Red mir es R oad
Hag
gLa
ne
Hal l am sh ir e Roa d
T ink er
L ane
B ol eHi ll
Roa dH opw oo
dLa
ne
Boleh ill L ane
Bing ley L an e
Lo ng L aneRos coe
Ba nk
Han
gin
gw
ate r
Ro a
d
Cold
well
L a ne
Cr o ok es
Crim
ic arL an e
Ivy Par kRoa d
Ste p he n
Hil l
Ha llam
G
range
Road
Back
L a
ne
Manch ester R oad
Manche ste r Road
Manche ster Road
Rivelin Valley Road
A57
A625
A57
A610 1
Rivelin Glen
WindleHou se Farm
Samehill Farm Hagg LaneCott ages
BingleyVilla
EndcliffeCrescent
Ranmoor
Endcliffe Edge
Carsick
Tapton Hill
Crosspool
Stephen Hill
Lydgate
Den Bank
Crookes
Carsick Hill
Hallam Head
Sandygate
Rivelin Glen
Fulwood
Stumperlowe
ORANGE
PURPLE
GREY
Path Slope
Por terBr ook
BlackbrookWood
Nor
thf i
eld
Road
F ulwo od Roa d
Tom Lan e
O ak br oo k Roa d
Ly dg a te Lan e
Tof t s L ane
S lay l e ig hLa ne
Gr a h am R oad
Car si ck
H i llR o ad
Net her g reen Roa d
San dy gateRo ad
Rust l ings Roa d
G l adston eRoa d
Red mir es R oad
Hag
gLa
ne
Hal l am sh ir e Roa d
T ink er
L ane
B ol eHi ll
Roa dH opw oo
dLa
ne
Boleh ill L ane
Bing ley L an e
Lo ng L aneRos coe
Ba nk
Han
gin
gw
ate r
Ro a
d
Cold
well
L a ne
Cr o ok es
Crim
ic arL an e
Ivy Par kRoa d
Ste p he n
Hil l
Ha llam
G
range
Road
Back
L a
ne
Manch ester R oad
Manche ste r Road
Manche ster Road
Rivelin Valley Road
A57
A625
A57
A610 1
Rivelin Glen
WindleHou se Farm
Samehill Farm Hagg LaneCott ages
BingleyVilla
EndcliffeCrescent
Ranmoor
Endcliffe Edge
Carsick
Tapton Hill
Crosspool
Stephen Hill
Lydgate
Den Bank
Crookes
Carsick Hill
Hallam Head
Sandygate
Rivelin Glen
Fulwood
Stumperlowe
270m155m160m
Elevation (m)
Por terBr ook
BlackbrookWood
Nor
thf i
eld
Road
F ulwo od Roa d
Tom Lan e
O ak br oo k Roa d
Ly dg a te Lan e
Tof t s L ane
S lay l e ig hLa ne
Gr a h am R oad
Car si ck
H i llR o ad
Net her g reen Roa d
San dy gateRo ad
Rust l ings Roa d
G l adston eRoa d
Red mir es R oad
Hag
gLa
ne
Hal l am sh ir e Roa d
T ink er
L ane
B ol eHi ll
Roa dH opw oo
dLa
ne
Boleh ill L ane
Bing ley L an e
Lo ng L aneRos coe
Ba nk
Han
gin
gw
ate r
Ro a
d
Cold
well
L a ne
Cr o ok es
Crim
ic arL an e
Ivy Par kRoa d
Ste p he n
Hil l
Ha llam
G
range
Road
Back
L a
ne
Manch ester R oad
Manche ste r Road
Manche ster Road
Rivelin Valley Road
A57
A625
A57
A610 1
Rivelin Glen
WindleHou se Farm
Samehill Farm Hagg LaneCott ages
BingleyVilla
EndcliffeCrescent
Ranmoor
Endcliffe Edge
Carsick
Tapton Hill
Crosspool
Stephen Hill
Lydgate
Den Bank
Crookes
Carsick Hill
Hallam Head
Sandygate
Rivelin Glen
Fulwood
Stumperlowe
Arrow Line-Width
Colour ElevationProfile ElevationProfile
2D 3D
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
4 Methodology
•Foreachvisualisationtype,userswereaskedtofind:
1.Thehighestofthreepoints.2.Thesteepestofthreeslopes.3.Whichofthreeroutescontainedthehighesttotalamountofclimbing?
•Usedaweb-hostedsurvey(basedonLimeSurveyopensourcesoftware),allowing:
1.Automatedrecordingofresponses(tocalculateeffectiveness).2.Automatedrecordingofresponsetimes(tocalculateefficiency).3.Submissionofuserfeedback(todetermineuserpreference).
•Visualisationandstatisticalanalysisofresultsdatathenidentifiedsignificantdifferencesbetweenvisualisationtypes,foreachofthethreeusabilitycriteria.
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
4 Methodology
Exampleuser-studyquestion(heightdetection).
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
5 ResultsEfficiency
Box-whiskerplotsshowingresponsetimedifferencesforeachvisualisation(forallquestiontypes).
•Relativeefficiencydependentuponroute-planningtask(height/slope/climbdetection).
•Line-widthwasmostefficientforthemajorityoftasks.
•Forslopetasks,arrowandcolourvisualisationsweresignificantlymoreefficientthanallothers.
•3Dterrainmodel=consistentlyinefficient.
•Nosingledimensionality(i.e.2Dor3D)wasconsistentlymoreefficientthantheother.
Measured using user response
time
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
5 ResultsEffectiveness
Visualisation Type
LineColourArrow3D Terrain3D Profile2D Profile
Cor
rect
Ans
wer
s (p
erce
nt)
100
80
60
40
20
0
Page 1
Differencesinthepercentageofcorrectanswersbetweendifferentvisualisations(usingdatapooledfromallquestiontypes)
•Relativeeffectivenessalsodependentontask(height/slope/climbdetection).
•Nosingledimensionality(i.e.2Dor3D)wasconsistentlymoreeffectivethantheother.
•Thelowerthecognitiveloaddemandedbythevisualisation,themoreeffectiveitappearedtobe.
•e.g.pointmatchingorjudginglineangles=highcognitiveload=ineffectiveMeasured
using the proportion of correct answers
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
5 ResultsUser Preference
Question Type
Slope TasksHeight TasksClimb Tasks
Par
tici
pant
Cou
nt20
15
10
5
0
Line Width
Colour
Arrow
3D Terrain
3D Profile
2D Profile
Page 1
VisualisationType
Thevariationinuserpreferencefordifferentroute-planningtasks.
•Visualisationpreferencedependedonthetaskathand.
•However,themajorityofuserspreferred2Dvisualisations,foralltasks.
•The2Dprofilewasespeciallypopular.
•Dislikeof3Dvisualisationsmainlystemmed(accordingtomajorityofusercomments)fromtheneedforinteraction.
Solid fill – 2D
Line fill – 3D
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
6 ConclusionsWhat can we conclude from these results?
1. The diversity of different 2D and 3D visualisation types, means we cannot state that one dimensionality is ‘more usable’ or ‘better’ than the other.
2. The usability of each visualisation type is inherently tied to the problem it is being used to solve.
3. The greatest impact on usability appears not to be visualisation dimensionality, but the cognitive loadplacedupontheuser.
4. 3D visualisations which presented targeted, abstracted versions of reality appear more useful than those which simply attempt to imitate reality.
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
7 Limitations & Future Research
LimitationsFuture
Research
Artificial test environment - indoors, quiet, constant lighting.
Repeat the study in an outdoor setting using mobile devices.
Inconsistent data visualisation: slope/elevation/both.
Only compare like for like, e.g. 2D profile with 3D profile
Participants were all highly educated, and mostly skilled in Cartography.
Include a wider range of participants & use a study design that allows testing of educational influence.
Lack of eye tracking (due to interactivity) = hard to explain usability differences.
Develop/apply eye-tracking techniques that can be used with interactive systems.
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
1. Introduction
2. Context & Background
3. Research
Questions
4. Methodology
5. Results & Discussion
6. Conclusions
7. Limitations & Future Research
An Empirical Evaluation of 2D and Interactive 3D Terrain Visualisations for Cycling Maps – MSc Thesis Defence
Thankyou!
Any questions?