lecture #8 special methods of testing y39tur spring 2011 tvorba uživatelského rozhraní
TRANSCRIPT
![Page 1: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/1.jpg)
Lecture #8SPECIAL METHODS OF TESTING
Y39TUR Spring 2011
Tvorba uživatelského rozhraní
![Page 2: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/2.jpg)
Remote Testing
![Page 3: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/3.jpg)
(3)
Software for X
Example: A software needs to be tested for the market in the country of X.
Possibilities:– Invite 10 people from X to the Czech Republic
• Air tickets, accommodation, visa• Not their own environment
– Go to X• Use a local recruitment agency• Rent a usability lab• Vaccination
– Is it always necessary?– Use the remote testing
![Page 4: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/4.jpg)
(4)
Traditional Methods vs. Remote Testing
• Traditional methods• Participants sit in the lab• Testers physically observe & record
• Remote testing• Participants sit in their office/home• Testers observe their screen via a cable & record
![Page 5: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/5.jpg)
(5)
Hierarchy of remote testing methods
Same time Different time
Sa
me
pla
ce
Diffe
ren
t pla
ce
Classic usability testing(Up until now)
(Not included in the course)
Remote testingTeleconferencing
SurveysOn-line evaluation tools
…
![Page 6: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/6.jpg)
Same time, different place
![Page 7: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/7.jpg)
(7)
Same time, different place
Synchronous– People connected via teleconferencing
MODERATOR
PARTICIPANT
STAKEHOLDERS
![Page 8: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/8.jpg)
(8)
Remote Testing
The testers observe the participants remotely– Via telephone– Via videoconferencing– Via screen capturing and streaming software
• Could be a combination of a remote desktop (VNC, …) + a screen grabber (Camtasia, …)
Methodology similar to the one of the classic usability tests– Certain differences
![Page 9: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/9.jpg)
(9)
Selvaraj & Houck-Whitaker:– Remote tests have at least the same effectiveness as traditional
Benefits– Time and costs savings
• You and your participants don’t need to spend time traveling
– Realistic context of use• You reach people in their own environment
– Geographic representation• Different portions of the globe can be covered
– Access to professionals• It’s easier to ask a $500/hr professional to take part in this test because
it will claim less of their time
Quality Comparison
![Page 10: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/10.jpg)
(10)
Quality Comparison
Limitations– Lack of nonverbal signs
• Communication delay
• Low resolution of the video, or perhaps no video link at all
– No control over the participant’s conditions• To check the software is well installed
• To make sure the participant is not being disturbed
– The moderator can’t assist the users on-site• The users are on their own using the system
– Higher level of the user IT literacy is expected• Can not test with the novice users
![Page 11: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/11.jpg)
Quality Comparison
Limitations– Will the users trust our application?
• People afraid of spyware
• Privately owned vs. corporate computers
– Will the stakeholders believe that it’s not fake?
![Page 12: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/12.jpg)
Costs Comparison
Traditional TestsLab
Equipment
Recruitment
Travel costs
User incentives –
physical presents,
money
Remote TestsOnline Meeting
User’s equipment
Recruitment
No travel costs
User incentives –
electronic coupons,
money
![Page 13: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/13.jpg)
(13)
Remote Testing Overview
Very similar to the “classic usability testing”– Define Objectives & Target Audience– Set up Test Scenario– Recruit Test Users– Carry out Tests– Analyze Findings– Design Report & Brief Stakeholders
![Page 14: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/14.jpg)
(14)
Remote Testing Overview
What are you testing? Who are you testing?
Representative Tasks– Within time-limits & user capabilities– In line with test objectives
Methods of data collection– Screen capture– Questionnaires, interviews
![Page 15: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/15.jpg)
(15)
Remote Testing: Test preparation
Consult the objectives with the project stakeholders
Develop instructions for the participants Run pilot test with home users Apply changes suggested by the results of the pilot
test
![Page 16: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/16.jpg)
(16)
Remote Testing: Recruitment
Define user profile & recruitment criteria Set up recruitment screener
– Screener can be filled out on the web• Questionnaires Database of potential participants
• Selection from the database
– Telephone screener• Very low success rate (telephone marketing failure)
Decide on incentives
![Page 17: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/17.jpg)
(17)
Remote Testing: Recruitment
Recruitment channels– Web
• Social networks, mailing lists, job portals
– Traditional: Newspapers, ads• With a URL to enter
– Recruitment agency• May be important when testing in an unknown market
• Perhaps better targeted participants
– Web – advanced services• ethnio.com
• clicktale.com
![Page 18: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/18.jpg)
(18)
ethnio.com – Example of Screener
![Page 19: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/19.jpg)
(19)
ethnio.com
Recruiting people directly from a website Procedure:
– Set up a screener at your ethnio.com profile– Set up your website to display the screener– A website visitor will see the screener– If responds, you will be notified immediately– You contact the person by telephone / e-mail
![Page 20: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/20.jpg)
(20)
Remote Testing: Recruitment
Specific requirement– The users must be able to install:
• The software that is to be tested
• The tools used for the test
– The task sheet for the participants must be more specific• There is no moderator in their place
Consent solicitation– By voice, saying “Yes, I agree.”– By clicking “I agree” on the screener form
![Page 21: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/21.jpg)
(21)
Remote Testing: Technology to use
Teleconferencing– Skype– Screen capture and streaming
• VNC• Remote Desktop in MS Windows
![Page 22: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/22.jpg)
(22)
Carry out the Test
During the test:– confirm user profile eligibility– ask for permission to record session– limit moderator intrusion– encourage thinking aloud– take notes– deliver incentive/payment– have fun
![Page 23: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/23.jpg)
(23)
Analysis & Reports
• During tests: track all usability issues• After each test: compare notes & analyze• After all tests: summarize patterns & major
problems• Set up report & sample videos• Communicate to all stakeholders
![Page 24: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/24.jpg)
Same place, different time
![Page 25: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/25.jpg)
Same place, different time
Data are physically acquired Data are picked up later on
Examples:– Customer satisfaction surveys– Elections– Geocashing
![Page 26: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/26.jpg)
Different time, different place
![Page 27: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/27.jpg)
(27)
Different time, different place
Asynchronous– Passing messages between the testers and the participants– The whole test can take a considerable amount of time due to
delays of communication between the testers and the participants
– Testers provide instructions • Through a website / e-mail message
– Participants provide data• answering a questionnaire
• by monitored interaction with the product
– The data are aggregated automatically
![Page 28: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/28.jpg)
(28)
Different time, different place
Features– Can be done automatically– Good for quantitative data collection– Good when there are lot of participants (25 – 100)
Drawback– We can’t control the conditions well
![Page 29: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/29.jpg)
(29)
Questionnaire-based Testing
Questionnaire– A set of questions
• With defined responses ([yes][no], [1][2][3][4][5], …)
• Open ended questions
– The same questionnaire administered to all participants– Easy to administer
• Point to a web form
• Send a structured e-mail
– Easy to process• Automatic processing of the web forms
• Automatic processing of returned e-mails
![Page 30: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/30.jpg)
(30)
Questionnaire-based Testing
Not many people respond to questionnaires– Need to “market” the study well
How to aim for specific target group?– Questionnaire should contain some screening questions
• Questionnaire contains Screener
Danger of …– … self-selection!
![Page 31: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/31.jpg)
(31)
SUMI
Software Usability Measurement Inventory Measuring software quality from the user’s point of view
– “Quality of Use”
Input:– The software or its prototype must exist– 10 users minimum
Output:– Five “grades”: Efficiency, Affect, Helpfulness, Control, Learnability– Based on existing database of gathered questionnaires
• Kept by the authors of SUMI
![Page 32: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/32.jpg)
(32)
SUMI: Use
How can be used:– Assess new products during product evaluation– Make comparisons between products or versions of products– Set targets for future application developments
Able to test verifiable goals for quality of use Track achievement of targets during product development In a quantitative manner
Source: http://www.ucc.ie/hfrg/questionnaires/sumi/whatis.html
![Page 33: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/33.jpg)
(33)
SUMI: Scales
Efficiency– Tasks are completed by the user in a direct and timely manner
Affect– How much the product captures emotional responses
Helpfulness– The product seems to assist the user
Control– Users feels that they set the pace, not the product (they are in
control)
Learnability– Ease with which the user can learn using the software and/or new
features
![Page 34: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/34.jpg)
(34)
SUMI: Questionnaire
50 fixed and predefined questions, such as:– “This software responds too slowly to inputs”– “I would recommend this software to my colleagues”– “The instructions and prompts are helpful”– “I sometimes wonder if I am using the right command”– “I think this software is consistent”
Responses to these questions:– “Yes”– “No”– “Undecided”
![Page 35: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/35.jpg)
(35)
SUMI: Processing
The assignment between questions and scales is not disclosed.– SUMI is a commercial service– Know-how of the authors
Procedure:– Participants try the tested system– SUMI questionnaires administered to the participants by testers– Responses to the questionnaires sent to the authors of SUMI
• e-mail, web form, …
– Testers receive the grades from SUMI• A nominal fee (hundreds USD)
![Page 36: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/36.jpg)
(36)
SUMI: Example Evaluation
Reference Value
![Page 37: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/37.jpg)
(37)
SUMI: Evaluation
The data provided are with respect to the corpus of previously gathered data– The values show the usability of the system compared
to the reference score (50 in each scale)
The data can be used to compare two different systems– Better score vs. worse score
![Page 38: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/38.jpg)
SUMI: Evaluation
Enough to provide an unbiased and objective results?– YES
Enough to give insights into particular problems?– NO … we only have 5 numbers as an output– We know nothing about the sources of errors
![Page 39: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/39.jpg)
Automated User Testing
![Page 40: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/40.jpg)
(40)Federico M. Facca 40
Automating Usability Testing
Usability Testing– a prototype or the final application is provided to a set of
users and the evaluator collect and analyze usage data– can be based on a set of predetermined tasks
What can be automated in such method?– capture of usage data– analysis based on predefined metrics or a model
Usability Evaluation of:– navigation– functionalities
![Page 41: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/41.jpg)
(41)Federico M. Facca 41
Capturing Data
Information– easy to record but difficult to interpret (e.g., keystrokes)– meaningful but difficult to label correctly (e.g., when a
task can be considered completed?)
Method Type:– Performance logging (e.g. events and time of
occurrence, no evaluator)– Remote testing (e.g. assigned task performed by user
and monitored by evaluators)
![Page 42: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/42.jpg)
(42)Federico M. Facca 42
Capturing Data – the Web – Server-side Logging
Web Server commonly log each user request to the server
Available information is: – IP address, request time, requested page, referrer
We can derive:– Number of visitors– Breakdown by countries– Coverage by robots …
![Page 43: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/43.jpg)
Server-side Logging
Pro– huge quantity of easily available data– do not require “ideal” users
Typical questions that we can answer:– “Which contents is interesting?”– “Do people reach all contents?”
• “Is all contents necessary?” … which is not the same as:
• “Is the navigation good?”
– “Does the new design keep people longer on site?”– “Does the new design make people buy more?”
![Page 44: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/44.jpg)
Server-side Loggin
Disadvantages:– Highly quantitative method– Almost no data of exact user interaction with the
interface
![Page 45: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/45.jpg)
(45)Federico M. Facca 45
Client Side Logging
Dedicated tools and settings– The web client must be enhanced to log information on interaction– The client pushes information into a repository on the testers’
server Available information is:
– IP address, request time, requested page, referring page, mouse position on the screen, clicked links, back button…
Pro– actual data of exact user interaction with the interface– session are automatically reconstructed
Against:– The participant must use this enhanced browser.
![Page 46: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/46.jpg)
(46)Federico M. Facca 46
Tools
“Formal” Client Side User Tracking/Analysis– Commercial tools
• ETHNIO (http://www.ethnio.com/)
• Ulog/Observer (http://www.noldus.com)
• UserZoom (http://www.userzoom.com)
• ClickTale (http://www.clicktale.com/)
• Usabilla (http://www.usabilla.com)
• Nielsen Eye Tracking (example in the next slides)
– Other tools (some are a bit old)• WebQuilt (http://guir.berkeley.edu/projects/webquilt/)
• SCONE/TEA (http://www.scone.de/docus.html)
• NIST WebMetrics (http://zing.ncsl.nist.gov/WebTools/, not only for tracking and relative analysis)
![Page 47: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/47.jpg)
(47)Federico M. Facca 47
Tools
“Informal” Client Side Tracking/Analysis– Commercial Tools
• Google Analytics (http://www.google.com/analytics/)• Fireclick (http://www.fireclick.com/)• SiteCatalyst (http://www.omniture.com/products/web_analytics)• Hitslink (http://os.hitslink.com/)• Crazy Egg (http://crazyegg.com) nice example• Usabilla (http://usabilla.com)• …. tons really
– Free Tools• Search with Google you can find some
Server side analysis– Again tons of solutions!
![Page 48: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/48.jpg)
(48)
usabilla.com
A Web 2.0 application
Main principle– Testers present a website screenshot– Participants mark points on the screenshots according
to tasks, e.g.:• “Click on the element that you would remove from the page.”
– Comments can be added– The testers can see the results in an aggregate form
• Individual points and comments are anonymous
![Page 49: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/49.jpg)
TUR 2010
(49)
Task: “Click on the element that you found most interesting.”
Responsesfrom the user
![Page 50: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/50.jpg)
(50)
usabilla.com: Example
Object of the test– Czech version of the home page of the DCGI website
Participants– CTU students
Tasks– Click on the element that you found most interesting– Click on the elements that you like most– Click on the elements that you would remove from the
page– Where would you look for contact information?– Where would you look for CVs of the faculty members?
![Page 51: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/51.jpg)
(51)
usabilla.com: Example
Recruitment– 18 people asked to participate– People who were online at the moment– Via ICQ and Skype– 10 users participated
Data acquired– 95 points (for all tasks)– 3 comments
Total time to carry out this test: 1 hour
![Page 52: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/52.jpg)
usabilla.com: Example – All points (for all tasks)
![Page 53: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/53.jpg)
usabilla.com: Example – Most interesting
![Page 54: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/54.jpg)
usabilla.com: Example – What to remove
Comments
![Page 55: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/55.jpg)
(55)
usabilla.com: Example – What to remove – Comments
The users clicked almost all elements to be removed– Can we trust such data?– We can not assume that all the users preferred to
remove all the elements– We need to interpret this as a possible dissatisfaction
with the layout of the website• This needs to be verified and concretized
• Separate test (different method) needs to be applied
![Page 56: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/56.jpg)
(56)
usabilla.com: Example – What to remove – Comments
Actual responses from the participants:– "osklive menu"
• (the menu is ugly)
– "mno nevim jestli odstranil, ale mail to moc nepripomina"• (well, probably not really remove but it does not look like an e-mail)
– "no tohle je asi z nejakeho publikacniho systemu, me to trochu rozciluje"
• (this is probably from some CMS, makes me angry a bit)
How to interpret these?– Only suggestions for further testing– (Not enough data for conclusive statements)
![Page 57: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/57.jpg)
usabilla.com: Example – Where to find faculty CVs?
Very niceand conclusive
cluster.
![Page 58: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/58.jpg)
usabilla.com: Example – Where to find contact info?
Most people wouldlook here
But someassumed there
were other ways
![Page 59: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/59.jpg)
(59)
usabilla.com:
General rule:– We should trust a point only as long as it is verified by
multiple instances
Benefits:– A rapid method of testing– Very easy analysis of data
Drawbacks:– No protection from “malevolent participants”
• “I’ll click you to death!”
– Motivation for placing the points not always understood
![Page 60: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/60.jpg)
(60)
By Federico M. Facca
60
SCONE / TEA
Support for Formal User Testing:– task specification– browser control– user tracking according to task specification
![Page 61: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/61.jpg)
(61)Federico M. Facca 61
Specific user behavior testing - Eye Tracking
Records eye movements Originally designed for mobility impaired users Relies on the “eye-mind” hypothesis Used in neuroscience, cognitive psychology,
advertising, and now … usability
![Page 62: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/62.jpg)
(62)Federico M. Facca
Eye Tracking
Equipment– Head-mounted systems– Remote systems (ERICA)– Computer monitor camera
systems
![Page 63: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/63.jpg)
(63)Federico M. Facca 63
Eye Tracking
Eye movements collected– Fixations – where the eye stops
long enough to absorb information– Saccades – move the eye from one
fixation to the next
Visual representation– Scanpath – the temporal sequence
of fixations and saccades
Other representations– Numerical– Real-time
![Page 64: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/64.jpg)
(64)Federico M. Facca 64
Eye Tracking
Provides a higher level of granularity than other data collection methods & quantitative measures of user behavior
Reveals behavior not evident in concurrent think-aloud protocol– Scanning continues when people are silent or using verbal
fillers (ums and ahs)– Eye movement occurs faster than verbalization
Shows parts of a user interface/web page that receive user attention and how search is visually distributed.
![Page 65: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/65.jpg)
(65)Federico M. Facca 65
Eye Tracking
![Page 66: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/66.jpg)
(66)Federico M. Facca 66
Eye Tracking
![Page 67: Lecture #8 SPECIAL METHODS OF TESTING Y39TUR Spring 2011 Tvorba uživatelského rozhraní](https://reader035.vdocuments.us/reader035/viewer/2022070306/5517ed52550346c1568b4a16/html5/thumbnails/67.jpg)
Eye Tracking – Heat Map