themis ia testing: lessons learnt from performing study scott rippon, user experience consultant its...
TRANSCRIPT
Themis IA testing:Lessons learnt from performing study
Scott Rippon, User Experience ConsultantITS > Web Services > User Experience Design team
2
Table of contents
• IA testing background• Lessons:
– L1: Creating Confidence Interval (CI) for discrete data
– L2: Cleaning continuous data & calculating mean and CI
– L3: Free text fields are evil!– L4: Adding straight line to column chart– L5: Analysing for particular audience
groups was manual & time consuming
3
IA testing background
• Themis is the UoM HR system.• IA testing:
– Goal: evaluate the usability of the Information Architecture (IA) of the present Themis system. Does the grouping/labeling make it easy for staff to find content?
– Use OptimalWorkshop’s TreeJack tool to perform study online.
– Performed in Dec 2012.
4
L1: Creating Confidence Interval (CI) for discrete data
• Definitions:– Discrete data = Data that can only take certain values (eg.
task success: Pass = 1; Fail = 0).– Confidence interval = Provides a value that we can ± to our
mean/average. If we repeat the study with a different sample we can be X% certain their results will fall between this range.
• Essential reading:Albert, W., Tullis, T., Tedesco, D. (2010), Beyond the Usability Lab, Morgan Kaufmann.
• Used formula from Calculating a Confidence Interval for Task Completion (Measuring the UX)
5
L1: Creating Confidence Interval (CI) for discrete data cont.
• Used CI to add error bars to graph.
• Steps:– Right clicked Scores bar,
selected ‘Format Data Series…’
– Select ‘Error Bars’ from LHS menu.
– In ‘Display’ group select ‘Both’.– In ‘Error amount’ group click
[Specify Value] button.– Select CI values from CI
column for both the ‘Positive (& Negative) Error Value’ fields.
* For Mac Excel 2008.
6
L2: Cleaning continuous data & calculating mean and CI
• Definition:– Continuous data = Data that can take any value within a range (eg. time, height, 1 to 7
scores).
• Pre-test questionnaire contained number of questions asking participants to rate whether they agreed (1 to 7).
• Excel cells contained same text as form (eg. ‘1 - Strongly disagree’). Cannot use this label to calculate mean/average.
• Solution:– Created new column for calculation.– Multiplied rating (C57) by responses (E57). See below figure.– Calculated average by dividing total of these multiplication (SUM(F57:F63))by total
responses (SUM(C57:C63)).
• See L1 for creating & graphing CI.
7
L3: Free text fields are evil!
• Study had 2 free text fields:– Pre-test questionnaire: training completed
• Had to manually recode all the responses.• Time consuming and had to make a number of
assumptions.• Should have present participants with set list.
– Post-test questionnaire: other feedback• Created individual cards and performed card
sort (AKA Affinity Diagramming) exercise to group responses and find themes in the data.
8
L4: Adding straight line to column chart
• Added success goal benchmark to graph (Guy’s idea).
• Hair pulling experience!
1 2 3 4 5 6 7 8 9 1011121314151617180%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Task success(Themis IA testing 2011)Error bars represent 95%
confidence interval
Score
Goal
9
L4: Adding straight line to column chart cont.
• Steps:– Created new column with 75%.– Highlighted scores and goal columns &
created column graph:‘Charts’ tab, [Column] button, [Clustered Column] button
– In the graph clicked the ‘Goal’ bars then changed it to a line graph:‘Charts’ tab, [Line] button, [Line] button
* For Mac Excel 2008.
10
L5: Analysing for particular audience groups was manual & time consuming
• TreeJack doesn’t allow you to filter the results for a particular audience group.
• To calculate scores for a particular audience group had to be done manually by manipulating Excel.
• Pain!!!
11
Thank you…
• If anyone would like copy of the Excel spreadsheet please contact me...
© Copyright The University of Melbourne 2011