©2003 regents of the university of minnesota. web usability evaluation at the university of...

©2003 Regents of the University of Minnesota.

Web Usability Web Usability EvaluationEvaluation

at theat theUniversity of University of MinnesotaMinnesota

©2003 Regents of the University of Minnesota. 2

Web Usability EvaluationWeb Usability Evaluationat the at the

University of MinnesotaUniversity of Minnesota

Copyright 2003 by the Regents of the University of Minnesota. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.


PresentersPresenters

Kari BranjordDirector, Enterprise Application and Web DevelopmentUniversity of [email protected]

Alice de la CovaUsability Services ManagerEnterprise Application and Web DevelopmentUniversity of [email protected]


CreditsCredits

Design assistance: Kristin Kinzler-Deal

Lab demo assistance: Nicole Tollefson Charlotte Tschider


Facts about the Facts about the University of MinnesotaUniversity of Minnesota

• 4 campuses with over 63,000 students total

Twin Cities: 49,474 Undergraduate: 40,324

Crookston: 2,320 Graduate: 13,382

Duluth: 10,114 Professional: 2,923

Morris: 1,861 Non-degree: 7,140

• 28 colleges with over 500 degree programs

201 undergraduate degree programs

245 Master’s degree programs

140 Ph.D. degree programs



• Ranking among top 25 public and private research universities – Total research: 12th

– Federal research: 15th

– Endowment: 24th

– Annual private giving: 15th

– National Academy Members: 25th

– Faculty awards: 17th

– Ph.D. degrees awarded: 5th

– Post-doctoral students: 15th



• Over 20,000 applications for admission– First-time freshman applicants: 14,700 – Transfer applicants: 5,600

• Over 16,000 applications for financial aid

• Over 31,000 employees– Tenure or Tenure Track Faculty: 2,767– Other Academic: 5,459– Civil Service/Bargaining Unit: 10,062– Student Employees: 12,955



• Use of Information Technology– Web page hits: 40,000,000

transactions per day

– Web pages under umn.edu domain: 774,000

– Domain rank of umn.edu among all universities in pages viewed: 8th

– E-mails to faculty, staff, students: 647,000 per day

– One-Stop Web site static page count: 1,404

– Users of enterprise Web applications: over 80,000


Enterprise Application and Enterprise Application and Web DevelopmentWeb Development

• Application development and support– Student system– Human Resources system– Financial systems– Library system

• Web-enabled processes

• Usability services


• A team in Application and Web Development

• Usability assessment activities– Usability evaluations– Category sorting assessments (card sort)– Focus groups– Field studies

• Accessibility reviews

• Enterprise-level Web applications and sites

• Assistance to other departments doing their own Web design


Web Development’s Web Development’s interest in usabilityinterest in usability

With over 80,000 internal users of Web applications, Web usability is very important to the University of Minnesota.

Example:

A self-service Web application to register online for a course should be easy to use successfully without training.



President Bruininks’ 2002 Enhanced Service and Productivity Initiative

• Improve service quality in delivering high volume transactions and services to students

• Leverage the University’s enterprise-wide technology investment

• Find ways to reduce costs of providing services (or increase revenue)

• Improve the quality, efficiency, and level of service for services delivered by non-academic service and support units



Why self-service?• A fast easy way for users to get services

• No time waiting in line for services

• No long turnaround times for paper forms being

processed• Services on demand, even at 3:00 a.m.

• Increase user satisfaction– Improve student experience and help develop loyal alumni for the

future– Decrease job-related frustration for staff and faculty



Why self-service?• Reduce cost of providing services• Reduce amount of paperwork• Provide a consistent delivery of services• Increase hours services are available• Deliver services to/from enterprise systems with a

Web user interface


Admission and housing Web Admission and housing Web applications and sites designed for applications and sites designed for

usabilityusability• Admissions Web site

• Application for admission

• Online catalogs

• Housing Web site

• Application for housing

• Housing roommates matching


Other student-oriented Web Other student-oriented Web applications and sites designed for applications and sites designed for

usabilityusability• Student One Stop Web site• Course registration, cancellation, swap, and

waitlist• Degree audit report for students and advisors• Financial aid steps• Financial aid award notice and response• Electronic promissory notes• Electronic student billing and electronic payment• Portfolio application for students, faculty, and

staff


Faculty and staff Web Faculty and staff Web applications and sites applications and sites designed for usabilitydesigned for usability

• Midterm alerts from instructors for students and their advisors

• Final grade entry• Electronic course authorization• Electronic course scheduling• HIPAA training on the Web• Effort reporting for time spent on sponsored projects• Financial One Stop Web site about University business

processes• Information Management Systems online reporting for

departments• Data Warehouse Web site and query tool


Employee self-service Web Employee self-service Web applications and sites designed for applications and sites designed for

usabilityusability• Employee view/update of W-4 info, with

online paycheck calculator• Pay statements for employees who use direct

deposit• Direct deposit setup for employees• Statements for employees with flexible

spending accounts• Employee registration for training• Notices of appointment for faculty


Other Web applications and Other Web applications and sites designed for usabilitysites designed for usability

• University of Minnesota home page• University Foundation Web site • Central authentication login page• “My U” Portal


Video excerpt from “Browsing the Web”Video excerpt from “Browsing the Web”episode of “Tech Talk”, a series producedepisode of “Tech Talk”, a series produced

by the Digital Media Center at theby the Digital Media Center at theUniversity of Minnesota for public University of Minnesota for public

televisiontelevision

We’d like to express our appreciation for the video segment and still photos to the "Tech Talk” producers and crew, especially Susan Tade, J.B. Eckert, Rich Reardon, Steve Barbo, Nicole Wilson, Paul Pecilunas, Laura Cervin, Richard Stachow, Bob Hanson, Jonathan Kranzler, and Nancy Johnson.


The usability video excerpt is available on the Web at the Usability Services site:

http://web.umn.edu/Webteam/usability/usabilityVideo.html

Tech Talk episodes can be seen on the Web:

http://www.techtalk.umn.edu/


What is usability?What is usability?• Usability is a product’s capability of being used easily and

intuitively by the intended users.

• It is not sufficient for a product to be “possible to use”.

• Users want to use a product without …– training– thinking hard– trying again and again– getting frustrated – being insulted

• Users want a product to work …– the way they expect it to work– the same way as similar products they’ve seen– the first time they try it


““Patience is a minor form of Patience is a minor form of despair disguised as a despair disguised as a virtue.”virtue.”

Ambrose Ambrose BierceBierce


Pull or Push?Pull or Push?


Financial Aid Award Notice and Financial Aid Award Notice and Acceptance Web application before Acceptance Web application before

usability evaluationusability evaluation


After design changes:After design changes:This is how it looks nowThis is how it looks now


The case for user-centered The case for user-centered designdesign

The product must permit the user to accomplish the user’s own objectives in using it

The product must be quick and easy to learn … and quick and easy to use …

and easy to remember between uses

The product must appeal to the users subjectively

“Everything must be made as simple as possible, but not one bit simpler.”

Albert Einstein


The Web-based self-service The Web-based self-service model model

depends on 3 groupsdepends on 3 groups


The emphasis is on the The emphasis is on the user in user in

user-centered design!user-centered design!“We are inclined to see things not as they are, but as we are.”

The user has a different perspective than the business experts or the development team.

Usability allows the user’s voice to be heard at the design phase, before the product is developed and delivered.

(proverb)


Web Application Web Application Development Life CycleDevelopment Life Cycle

• Kickoff

• Analysis

• Design

• Usability Evaluation

• Development

• Testing

• Implementation


Why do a usability Why do a usability evaluation?evaluation?

• Get the project team to see the product “through the user’s eyes”

• Provide a hard and fast deadline in the middle of the project


Benefits of a usability Benefits of a usability evaluationevaluation

• Increase user satisfaction with the design

• Lower cost by avoiding re-work: discover usability issues before development or implementation

• Decrease the number of Help Desk calls after implementation

• Provide a more objective basis for making a decision on a design

• Lead the team in an effective decision-making process for the final decision on the design

• Create a “bonding experience” opportunity for the project team


What a usability evaluation What a usability evaluation isn’t …isn’t …

• A test of the evaluator’s intelligence or abilities

• A critique of the designer’s ability or talent

• Research

• Analysis

• System testing

• Stress testing • Accessibility evaluation to determine if software

meets federal accessibility standards for people with disabilities


Task-based methodTask-based method

• Recruit 6-8 people from the target audience• Evaluators try out the application one at a time

while the project team watches

• Evaluators are given 3-5 typical tasks to complete

• Evaluators use a “think out loud” protocol• Usability consultant debriefs evaluator after task

completion• Team watches and listens for usability issues• Usability consultant debriefs team and documents

issues


Scenario exampleScenario exampleYou’ve always been a great fan of horses and you ride frequently, so you’ve decided to make an immediate gift online of $500 to support University research on the health and treatment of horses.

Make an online gift to the Equine Research Center by using a credit card.

Use the Visa credit card number: 1234 5678 1234 5678

Expiration: January 2004

Your name on the credit card is John (or Jane) Doe.

The evaluator’s tasks are to navigate to the correct page to make an online gift, enter donor and gift information, designate the gift to the right department, complete the credit card fields, submit the donation online, and receive the gift confirmation.


Usability LogUsability LogTime Stamp Observer Comment12:23:00 PM Tape Timer Started12:23:16 PM Evaluator and consultant enter room12:24:07 PM Evaluator briefed with Bill of Rights and intro to THINK OUT LOUD protocol12:24:35 PM Evaluator is familiar with web site.12:24:42 PM No questions.12:24:53 PM Consultant leaves room.12:25:04 PM Reading task 1.12:25:10 PM Opens browser12:25:31 PM Looks at Student One Stop page.12:27:44 PM Mouses over all the links12:27:57 PM Clicks on Services Index Page link12:28:21 PM Cursor over the Housing links12:28:56 PM Says she hadn't known that Housing would be under Services, but it makes sense

that Dining is under Housing.12:29:30 PM Asks why Housing wasn't important enough to put on the home page.12:29:37 PM Clicks on Dining link.12:30:14 PM Sees message that the page being looked for is not on the Dining site.12:30:39 PM Reads link to Residence Hall Dining12:30:57 PM "Looks like I can use this page anyway"12:31:38 PM Residence Halls Dining Centers page12:31:44 PM Scrolls down.12:31:49 PM Sees Middlebrook Hall.12:32:18 PM Finds hours for Middlebrook dining center.12:32:28 PM OK -- I'm done.12:32:31 PM End of Task 1


Sample Student One Stop Usability Results Spreadsheet (from Usability Demo)

# Page name Issue Severity1 = Showstopper2 = Major3 = Minor4 = Cosmetic+ = GoodC =Comment

Problem Identification 1 2 3 4 5 6 7 8

Fre

qu

en

cy

1 UCard Web Page User never found the UCard page using Student One Stop.

1 Referred to designer. Ideas: Include on Help Page.

x x x 3

2 Registration Page, Help Page

User thought a UCard link ought to be on the Registration Page or the Help Page.

1 See Item #1. x x x x x 5

3 Computer and Internet Information Page

Selecting 'Computer Store' did not lead to any information about buying computers.

2 Referred to designer. Ideas: Remove link to University Bookstores.

x x x x x x 6

4 Computer and Internet Information Page

User did not recognize TechMart as a relevant link for buying a computer.

2 Referred to designer. Ideas: Change wording of link.

x x x x

5 New and International Student Information Page

User tried to look for international student registration procedures first under "New and International Student Information", but couldn't find it there. Eventually found it under 'Registration'.

2 Referred to designer. Ideas: add link under "New and International Student Information".

x x x x x x x 7

6 OneStop Search User tried to search for "dining hall hours", but did not receive any helpful results.

2 Referred to designer. Ideas: set up tags so that search will pick up page on dining hours.

x x x x x 5

7 Tuition and Fees User did not recognize '2002-03 Tuition Rates' as a link.

2 Referred to designer. x 1

8 Final Exam Schedule Page

User took a long time before finding exam schedule timetable.

3 Referred to designer. Ideas: Change layout to make table more obvious. Or make a link to the table from the top of page.

x 1

9 Final Exam Schedule Page

Asked for a link directly to the timetable.

3 See Item #8 x x 2


Usability Evaluation Usability Evaluation DeliverablesDeliverables

• 2-4 page executive summary report

• Usability results spreadsheet is attached

• Report sections:

– Brief description of application or Web site

– Evaluation purpose or usability goals

– Description of evaluation methodology

– Short table of the most significant issues and what steps will be taken to address them

– List of the usability team members


Ideal ScheduleIdeal Schedule

Project StartKickoffMeeting

Preparefor

Evaluations

EvaluationSessions

Analysisand

ReportingProject End

Week 1 Weeks 1-4 Week 5 Week 5

The usability schedule may be compressed to accommodate software projects with a tight deadline.


What makes our process work What makes our process work well?well?

• Team approach

Objective is team buy-in to the outcomes.

• Quick delivery of results

Objective is for project team members to resolve the issues quickly and finish the project on time.

• Fun experience for the team

Objective is for team members want to do usability evaluations again on their next project.


5 years of Web usability at 5 years of Web usability at the University of the University of

MinnesotaMinnesota1996 Formed Web Development team

1998 Started doing usability evaluations with IBM user-centered design practice consultant

1999 Developed usability process methodology

1999 Developed Web Development application life cycle

1999 Made first attempt at building a usability lab

2001 Started usability lab project again

2002 Implemented usability lab

2003 Expansion of services beyond Web Development


Do you need a usability Do you need a usability lab?lab?

Evaluator RoomObserver Room

Mic

roph

oneS

pea

ker

Rec

ord

er

On

e-w

ay

mir

ror


A quick tour of our lab . . A quick tour of our lab . . ..


Our lab is located in Walter Library, Our lab is located in Walter Library, on the East Bank of the Minneapolis on the East Bank of the Minneapolis

campuscampus


Usability Services Lab at Usability Services Lab at the University of the University of

MinnesotaMinnesota

21' - 6

"

6' - 0"

5' - 0

"

Pilla

r

3' - 0

"

Pho

ne

EvaluatorRoom

Ceiling

-m

ountedC

amera

Ceilin

g-m

ounted

Cam

era

White b

oard

Coat

Rack

White

board

MiniFridge

MaterialsTable

5' - 0

"

Clo

set

Clo

se

t

Sto

rage L

ocke

rs

Shelves for Supplies

Sto

rag

eC

abin

et

32' - 6"

Help

Desk

Cam

era/ V

ideo

Narrator

Logger

Phone

Lo

gg

ing

Co

mp

ute

rC

om

pu

terM

on

itor

Co

mp

uter

Mo

nito

r

21"

3' - 0"

Printer

Vid

eoC

on

trol &

Rec

ord

ing

Ca

mera

Co

ntro

l

Co

mp

uter

Mo

nito

r

Wall-mounted plasmamonitor for observers

Materials TableMaterials Table Refreshments Table

ObserverRoom

Imag

eC

on

trol

Co

mp

ute

r

Co

mp

ute

rM

on

itor

Coat Rack

Com

pute

rC

art

Sto

rage

Cabin

et

Printer


Participant reception room is Participant reception room is located down the hall from the lablocated down the hall from the lab

13' - 0"17' - 0"

Re

freshment T

able

Magazines

Ph

one

ParticipantReception Room

Coa

tR

ack

Pillar

Pillar

Re

ceptio

nistD

esk

Lockin

g File

Cab

inet


The Lab PartnershipThe Lab Partnership

• Investor roles

– Office of Information Technology

• Provided leadership and project management, funding for remodeling and equipment, lab operations and support

– Digital Technology Center

• Provided space, furniture, phones, network connections


The Lab PartnershipThe Lab Partnership• Stakeholders

– Usability Services team in Web Development

– Digital Technology Center in the Office of the V.P. for Research

– Dept. of Computer Science & Engineering

– Dept. of Rhetoric (Scientific & Technical Communication Program)

– School of Journalism & Mass Communication (Institute of New Media Studies)

– School of Kinesiology (Human Factors Program)


Benefits to Web Benefits to Web Development from the lab Development from the lab

partnershippartnership• Space for a lab

• Academic legitimacy for use of space

• Convenient campus location

• Reduced cost of having own lab

• Ability to use lab to capacity (under-used space can get cut)


Benefits to Web Benefits to Web Development from the lab Development from the lab

partnershippartnership• Good relationship between administrative and

academic units

• Supporters for lab from stakeholder departments

• Source of student employees from stakeholder departments

• Ability to make a greater contribution to the University mission by providing a very low-cost lab facility for researchers and for instructors


Quantification of benefits the lab Quantification of benefits the lab has provided to the University has provided to the University• Number of usability projects increased 57% in 1 year

Previous year: 21 projects After lab built: 33 projects

• Number of usability projects for other units increased 129% in 1 year

Previous year: 7 projects After lab built: 16 projects

• 75 hours of research projects by faculty and students

• 65 hours of lab tours and outreach events, with over 500 people touring the lab in 1 year

• 12 hours of presentations to visiting classes brought by instructors teaching related courses


How much is the lab used?How much is the lab used?

• Estimated 1300 – 1400 hours/year

• 750 hours of usage in the first year

• Usage increased from an average of 11.1 hr/week during the first 3 months to an average of 24.2 hr/week in September 2003


Breakdown of Lab UsageBreakdown of Lab Usage74%: Usability projects

10%: Academic stakeholder projects

9%: Lab tours, open houses, and

other outreach events

5%: Lab orientation and training sessions

2%: Class visits

<1%: Stakeholder meetings


Leading by ExampleLeading by ExampleFor years Enterprise Web Development has recommended a user-centered design approach to other departments.

Having a usability lab has enabled us to offer our usability services to other University units that do their own design work.

Academic Health Center

College of Continuing Education

Controller’s Office

Information Management Systems

School of Public Health

Sponsored Projects Administration

University of Minnesota Foundation

University Relations


Return on Investment in Return on Investment in LabLab

• Web Development’s return on investment was better than expected.

– 39% ROI after 1st year

– Expenses included accumulated costs for construction, equipment, computers, furniture, common supplies, and lab support staff.

– Return on investment calculation based on lab revenues and avoided costs


How can I get the How can I get the functional experts and functional experts and development team on development team on board with usability?board with usability?


Building Support for Building Support for UsabilityUsability

• Success begets success

• Don’t wait for the opportunity to build a lab

• Look at your competition’s products

• Visit a usability lab and watch an

evaluation



• As an experiment, do a usability assessment on an existing Web site

– Find usability issues you didn’t know about– Find out what users think is missing from

the site– Rent a lab or do in-room usability

evaluations– Get a benchmark of what user-centered

design could do for you



• Do a usability assessment after a prototype design is created on your next development project.

• Start gathering statistics

• Publicize successes



• The most persuasive arguments for usability will come from your customers

• When you have strong support for usability, do the cost-justification for a lab

• Once you have a lab, you have an easy way to build more support by giving lab tours and demos


Questions?Questions?

Kari Branjord <[email protected]>Alice de la Cova <[email protected]> http://web.umn.edu/…(url)

Kari Branjord <[email protected]>Alice de la Cova <[email protected]> http://web.umn.edu/Webteam/usability/index.html

©2003 regents of the university of minnesota. web usability evaluation at the university of...

Documents

university of minnesota

th slide

web usability evaluation

day web pages

th endowment

th emails

th federal research

th faculty awards