Usability Study of theApply TexasApplication

including Interviews with High School Counselors

Sandra Martinez, Ph.D.Apply Texas Team

Performed Spring 2010 by Garrett Stettler as his Master’s project for UT School of Information

Garrett observed, interviewed, and videotaped 15 participants

He interviewed 5 high school counselors

Usability Study

Getting a third party to perform a usability study is more effective than using developers because:◦ They have less personal involvement in the

website, which produces less emotional response.

◦ They are not biased by over-familiarity with the website

◦ They have different expectations than developers

Usability Study – the 3rd party

They evaluate a product by testing it on actual users.

They discover errors and areas for improvement by observing people using the product.

They measure efficiency, accuracy, recall, and emotional responses of test subjects.

How are usability studies helpful?

The Apply Texas Committee, in authorizing the study, charged the Apply Texas team with four goals:◦ Fix simple errors

◦ Fix big, urgent problems

◦ Create list of usability issues for the Apply Texas committee to prioritize

◦ Create list of suggested changes for the Apply Texas committee

How the study will be used

8 postsecondary-bound Texas high school students

2 postsecondary-bound high school students from other states

2 current college students (transfer)

1 prospective graduate student

1 “non-traditional” adult returning to school

1 pilot participant

Participants

1) Navigation 2) Copy process 3) Help 4) Confusion and frustration 5) Accuracy and completion

Areas of research for Apply Texas usability study

Tasks by Group Group A (“new”

users)1. Create account, input

profile information

2. Start and complete a new application

3. Enter essay information

4. Enter scholarship information

5. Logout

Group B (“returning” users)

1. Retrieve forgotten password

2. View/edit a saved profile

3. View/edit a saved application

4. View/edit saved essays

5. Copy an existing application

Critical concerns

Major concerns

Moderate concerns

Minor concerns

Levels of Concern

Description of critical concerns:

Critical data may be lost, or

The user may not be able to complete the task, or

The user may not want to continue using the application

Critical concerns (None found)

Description of major concerns: Users can accomplish the task but only with

considerable frustration and/or unnecessary steps Non-critical data may be lost The user will have great difficulty in circumventing

the problem Users can overcome the problem only after they

have been shown how to perform the task

Major concerns (2 found)

Error handling

◦ Error messages are grouped at the top of the page.

◦ Errors are not highlighted within the page itself

◦ Participants tended to skim over error messages and used a “trial-and-error” approach to correcting the error

Major concerns (1 of 2)

Major concerns (1 of 2)

Error handling recommendations

◦ Errors should be highlighted at the top of the page and at the location in the page where the error has occurred.

◦ If possible, identify the error as the user is typing

Major concerns (1 of 2)

Home / My Account / Logout

◦ Navigating to the main account page was confusing

◦ Attempting to log out could be confusing

Major concerns (2 of 2)

Major concerns (2 of 2)Current view of top toolbar

Currently links to logout page

Currently links to logout page

Currently links to application listing page

Currently links to help page

Major concerns (2 of 2)

For new cycle, links to logout page

For new cycle, links to application listing page

Links to application listing page (no change)

For new cycle, name changed to Logoff from Home/Logoff

Changes for 2011-2012 cycle

Description of moderate concerns:

The user will be able to complete the task in most cases but will undertake some moderate effort in getting around the problem

The user may need to investigate several links or pathways through the system to determine which option will allow them to accomplish the intended task

Users will most likely remember how to perform the task on subsequent encounters with the system

Moderate concerns (5 found)

Helper text and other indicators◦ Every participant failed to notice help text at

some point during the test.

Moderate concerns (1 of 5)

Helper Text

Moderate concerns (1 of 5)

Color of helper text changed from gray to blue

Buttons or important information appear ‘below the fold’

Some participants had difficulty completing a task if the continue button required scrolling

Moderate concerns (2 of 5)

Moderate concerns (2 of 5)

Continue button off the screen

Moderate concerns (2 of 5)

For new cycle, continue button moved to top of page

Confusing terminology on essay page

Participants were unsure about clicking ‘Save’ if they had not entered an essay.

Some participants were unsure about the difference between save and submit.

Moderate concerns (3 of 5)

Moderate concerns (3 of 5)

Use of the term “save” before anything is entered

Moderate concerns (3 of 5)

Changed to Enter Topic A

Moderate concerns (3 of 5)

Changed to Edit after first save

Difficulty using Perform High School/College Search function

◦ Over one-third of participants clicked the “perform search” button before entering any text

◦ Two participants failed to select the radio button

◦ Several participants did not initially use the search feature

Moderate concerns (4 of 5)

Moderate concerns (4 of 5)

The Apply Texas team recommends reworking this page as follows:

1. Make the “perform high school name search” button more prominent

2. Ask applicants to enter the name of the school in a new window when the button is clicked

3. When the applicant chooses a radio button, automatically close the window and fill in information on the application

Moderate concerns (4 or 5)

Top Menu and Quick Links

◦ Some participants were confused about the Menu bar and the Quick Links menu, and how they differed

◦ Participants generally did not use the navigation bar at the top for moving between Admissions/Scholarships/Essays/Submit

Moderate concerns (5 of 5)

Moderate concerns (5 of 5)

The Apply Texas team recommends reworking the navigation and links:

◦ Make the navigation links as a side navbar

◦ Add quick links to the top menu bar with a “mouseover” dropdown.

Moderate concerns (5 of 5)

Moderate concerns (5 of 5)

Example of a page with a side navigation bar

Description of minor concerns:

The concern is an irritant for the user

There is a cosmetic problem

There is a typographical error

Minor concerns ( 4 found)

Confusing/inconsistent terminology

◦ 1) Applicants were unsure what was being asked for “Mini or short” semester

Minor concerns (1 of 4)

Confusing/inconsistent terminology◦ 2) Participants were unsure about the difference

between ‘community and volunteer service’ and ‘extracurricular activities’

◦ 3) Under extracurricular activities, participants were unclear what ‘activity level’ meant

◦ 4) One participant was unclear whether ‘household size’ included only children, or all persons in household.

Minor concerns (1 of 4)

Minor concerns (1 of 4)For mini/short semester, the following paragraph was added for the new cycle as clarification:

Constraining employment form

◦ Two participants felt the employment form would be difficult to use, especially for applicants who have recurring summer jobs.

Minor concerns (2 of 4)

Custom Questions◦ Many participants found the custom question

page to be irritating, particularly for institutions with many custom questions.

◦ The page did not cause confusion, just user irritation as it requires scrolling down the page for each question in order to click the ‘save, acknowledge question button.’

Minor concerns (3 of 4)

Minor concerns (3 of 4)

Suggestion: Schools with many custom questions may want to consider moving some of their questions to other modules instead of grouping them all on the custom question page.

Minor concerns – 3 of 4

Yes/No/Reset Answer

◦ Several participants were confused by the Yes/No/Reset Answer response.

Minor concerns – 4 of 4

One suggestion would be to change the button labels from Reset Answer – Not Applicable to No Answer – Not Applicable

Overall, Apply Texas is a fairly usable product, with no critical concerns

Many improvements to reduce frustration could be done fairly easily

Some improvements will require reworking certain modules or processes

And now, let’s hear what the high school counselors had to say --

Usability Study

Suggestions for improving Apply Texas:◦ Give applicants a warning that if they select to send a paper

resume, schools may not process their application until the paper resume is received.

◦ Allow more space in online resumes, or provide an upload option into Apply Texas

◦ Provide another mechanism for retrieval of lost usernames, passwords, and application id numbers.

◦ Provide an online application deadline table equivalent to the one available in the paper copy of the Apply Texas application.

High School Counselor interviews

High School Counselor interviews

Application deadline table

Suggestions for improving the High School Counselor System

◦ Different navigation options

◦ Different documentation on what data is available

◦ The ability to separate four year and two year information

High School Counselor interviews

The Apply Texas team would like to thank Garrett Stettler for his time, effort, and excellent results.

We would also like to thank the Apply Texas committee for authorizing this study.

We are confident that implementing his results will make the application easier and less frustrating for our applicants.

Usability Study

Questions?Usability Study