user centred evaluation · –according to iso 9241-210 from 2010 •this can include evaluating...
TRANSCRIPT
Presentation at the Twintide meeting in Copenhagen,
13th of October 2012
User Centred Evaluation
in ‚the Wild‘
Marta Kristín Lárusdóttir
Overview for the Presentation
• Introduction to the topic
• My research focus
• The 3studies I have made on this topic
• Conclusion
• Future work
What do I mean by ‚in the Wild‘?
What I mean is Software Development Practice
Scrum is popular in Iceland and Sweden
Scrum is feature oriented
• It is a process for planning software projects
• Usually there is 2 – 4 weeks between delivery
Requirements analysis
Design
Implementation
Testing/Evaluation
Functionality of the system
Tra
ditio
nal - T
ime
Scrum - Time
The main rules in Scrum*
1. Work in iterations of no more than a month long
2. Be “done” with something
– by the end of each iteration
– to some pre-agreed upon definition of done and solicit
feedback from your key stakeholders on it
3. Get together and figure out what you’re doing
– at the start of an iteration
4. Reflect on how well you did during the iteration
at the end of the iteration
5. Talk a lot during the iteration
* According to Mike Cohn (2012)
User Centred Evaluation
• Evaluation based on the user's perspective
– According to ISO 9241-210 from 2010
• This can include evaluating Usability and UX
• User-centred evaluation can be used to
– a) collect new information about user needs,
– b) provide feedback on strengths and weaknesses of the
design solution from the user's perspective
– c) assess whether user requirements have been achieved
– d) establish baselines or make comparisons between
designs.
Evaluation Methods
• Analytical evaluation
1. An expert analyses according to
supporting material
• Gathering lists of usability problems
• Empirical evaluation
2. Measuring user’s performance
• Quantitatively gathering data on time
on task, user satisfaction, etc.
3. Qualitatively by observing users
and asking users their opinion
• Informal feedback from users
User Centred Evaluation ‚in the Wild‘
Usability and User
Experience evaluation
In software development
using Scrum for project
management
My Research Focus
• My main research questions:
1. How do IT professionals conduct user centred evaluation in the
Scrum development process?
2. How does it affect their evaluation that they are using Scrum?
4 papers accepted on this subject
1. Larusdottir, M. K., Bjarnadottir, E., Gulliksen, J.: The Focus on Usability
in Testing Practices in Industry, – Proceedings of the Human Computer Interaction Symposium at the World
Computer Congress 2010, Brisbane, Australia, September 2010.
2. Larusdottir, M. K., Cajander, A, Gulliksen, J: Informal Feedback
Rather Than Performance Measurements – User Centred Evaluation in
Scrum Projects
– Accepted to the journal Behavior and Information Technology, October 2012.
3. Larusdottir, M. K., Cajander, A, Gulliksen, J: The Big Picture of UX is
missing in Scrum Projects
– Proceedings of the Ix-Used workshop at NordiCHI, October, 2012
4. Jia, Y., Larusdottir, M. K., Cajander, A: The Usage of Usability
Techniques in Scrum Projects
– Proceedings of the HCSE 2012 conference in Toulouse, October, 2012
State-of-the-Art
• The complexity of evaluation should be studied – Results from research studies fail the practitioner, Wixon (2003)
• Asking IT professionals mainly in surveys
– The usage of evaluation methods
• Bygstad et al. (2008) Vukelja et al. (2007), Rosenbaum (2000)
– Frequency of evaluation
• Bak, et al. (2008), Ardito, et. al. (2011)
– Purpose of the evaluation
• Vermeeren, et al. (2011), Venturi, et. al. (2006), Monahan et al.
(2008), Bark et al. (2006)
• No study on evaluation according to the process used
• Conducted in 2009
• 25 participants from 18 companies
Study 1: Survey study in Iceland
Results – Survey on Scrum Projects
Testing technique Yes, a
lot
Yes, some So and so Little No, not at
all
Unit/component testing 22% 35% 26% 13% 4%
Integration testing 17% 35% 31% 13% 4%
System testing 39% 30% 22% 9% 0%
Acceptance testing 30% 44% 13% 13% 0%
Usability testing 4% 22% 35% 35% 4%
Alpha testing 4% 13% 17% 17% 48%
Beta testing 9% 22% 9% 17% 44%
Performance/load testing 0% 26% 26% 35% 13%
Security testing 4% 22% 8% 39% 26%
Testing technique Lack of
training/
knowledge
Lack of
budget
Lack
of
time
Other N/A N
Unit/component testing 36% 0% 32% 5% 27% 22
Integration testing 11% 0% 42% 0% 47% 19
System testing 7% 0% 47% 0% 47% 15
Acceptance testing 7% 0% 27% 7% 60% 15
Usability testing 20% 15% 35% 10% 20% 20
Alpha testing 0% 11% 11% 10% 68% 19
Beta testing 0% 11% 17% 11% 61% 18
Performance/load testing 26% 11% 32% 0% 32% 19
Security testing 47% 5% 16% 0% 32% 19
Results - Survey on Scrum Projects
Study 2: Interview study in Sweden
• Conducted in 2010
• 21 participants working for 14 companies using Scrum
Results:
Interviews
Type of evaluation
Empirical
Quantitative
Evaluation Empirical Qualitative Evaluation
Analytical
Qualitative
Evaluation
Evaluation
Method
Professional Role
N Measuring
user
performance
and surveys
Observing
Users
Asking
user their
opinions
Feedback
from user
surrogates
Inspection
evaluation
Usability experts 5 2 5 4 1 5
Interaction designers 7 1 5 6 3 3
Business analysts 4 0 1 4 0 1
Developers 2 0 0 1 1 2
Scrum managers 3 1 2 3 2 1
Total 21 4 13 18 7 12
Study 2 Results: Interviews in Sweden
Purpose of the
Evaluation
Professional Role
N Feedback on
Context of Use
Feedback on User
Requirements
Feedback on
Design
Usability experts 5 3 4 5
Interaction designers 7 5 6 6
Business analysts 4 3 4 3
Developers 2 1 0 2
Scrum managers 3 0 2 3
Total 21 12 16 19
Study 2: Two success stories
• The big picture for UX is Missing in Scrum projects
– A vital activity is visioning in the pre-phase
• There is a lack of responsibility for UX
• The Scrum context affects the Usability
professionals work
– Scrum is feature oriented
– Hard to find time for the User Centred Evaluation
• Informal cooperation emphasised in Scrum
– Much informal evaluation is conducted
Study 3: Usability techniques in Scrum
• Conducted in 2011 in Sweden
• 49 participants
Study 3 Results: Survey in Sweden
Study 3 Results: Survey in Sweden
Main conclusions
• Formal user centred evaluation not that common
– But it get high rating from the partictioners
• Informal evaluation is conducted by many
– Especially evaluating in the design phase
– Especially in the results from the interview study
• Heuristic evaluation not commonly used
– But half of the participants in the interviews do expert reviews
• Participants do not name the methods/approaches
they are using
•
Future work
• How is the deployment managed in Scrum projects?
• How is UCD integrated in Scrum projects?
• How is UCD conducted in the games industry?
• How are user requirements before the production
starts specified?