usability test report for queso

Usability Test Report: Queso

Usability Test Conducted on April 11, 2016Report prepared for Apr 18, 2016CIM 622 UX Research Methods

Executive SummaryQueso is a learning management system. Queso provides students and instructors with a platform to share content, class materials, instructor feedback and point-of-semester grade level.

The usability test type is problem identification usability test. We conducted this test with the objective of identifying usability problems on the desktop version of Queso from the student profile. The 6 participants(4 expert users and 2 novices) completed5 tasks.

Moderators found and categorized 22 problems after testing primary functionality and 5 positive attributes of Queso to keep. The usability metrics measured were satisfaction scores, efficiency and effectiveness rates. The mean value of satisfaction score is 47.5 and the mean value of completion rate across tasks is 67.8%, mean value of task time is 126.2 seconds. As a group, the team aggregated the findings and made redesign suggestions, improving the usability of the site. The main issues Queso faces are as follows:

● The language used in Queso system was not understandable to users● Poor information architecture impacts navigation and content organization● Missing affordances to accomplish goals● Visual design is unclear and does not show placement of user is in the system

The next steps are to give results to the developer and improve usability through further research and redesign the site

2

IntroductionFull Product Description

Queso is a learning management system (LMS). Queso’s main competitor in the market space is Blackboard, one of the most used LMS systems available.

Queso is used in Universities and college in both undergraduate and graduate level classrooms . Intended users of this learning management system are both instructors and students.

Queso provides students and instructors with a platform to share content, class materials, instructor feedback and point-of-semester grade level, for students. Queso’s design and user interface’s goal was to have a game-like, playful, experience for its intended audience.

3

IntroductionTerminology and Language used in Queso

Quest: Assignments with guidelines and deadlines created by instructors for students to submit.

Projected grade: The visualization of the current grade in course from total points available, the grade the student will likely get if percentage of points earned continues

Current Level: Grade earned in class during the semester from total assignments available

Feedback: Comments from the instructor about assignments submitted.

Course content: This function allows instructors to post syllabus, slides, videos and articles on a feed for all students to view and access.

Available assignment(Available Quest): This feature allows assignments to be posted by instructors for students to submit online. Students can see the due date and points worth for each assignment.

Completed assignment(Completed Quest): This function shows feedback from instructors on graded assignments. Students see their assignment grade, optional feedback and are given the option to revise and re-submit.

Current Grade (Progress): This function allows students to see their current grade earned from all assignments of the course. They can look up for current grade and total grade for individual assignments.

IntroductionTest Objectives

Gather data about the overall usability of Queso. The goals of this study are:

● Assess the overall usability of Queso for target-users performing most frequently used and most troublesome tasks. Identify obstacles and usability problems with:

○ Assignment submission○ Revising a submitted assignment○ Reviewing assignment grade○ Viewing course progress ○ Reviewing instructor's’ feedback on assignments.

● Provide recommendations to redesign the site5

METHODS

6

MethodsParticipants

7

Characteristic Number of participants

Total 6

Product experienceNovice (have used Queso for more than 6 months)Expert (never used Queso)

24

Key characteristicsUM student 6

Representative of groups with special needsYoung, old or physically disabled

0

Paid/compensated No

MethodsParticipants Demographic Data

8

Age Range:

Employment status: 6 (100%) participants are Full time students and employed

Language: 4 (66.7%) Native English Speaker 2 (33.3%) Non-Native English Speaker

Education status: 3 (50%) participants completed Undergraduate degree 3 (50%) participants completed Graduate degree

Gender: Female: 4 (66.7%) Male: 2 (33.3%)

Methods

9

6 (100%) participants access internet Multiple times a day

Participants Demographic Data

Current computer skills: Scale from Poorly Excellent

1 7

6 participants (100%) all have experience of using Learning management system.

Types of Learning management system:

3 (50%) participants chose scale 5 1 (16.7%) participants chose scale 62 (33.3%) participants chose scale 7

Methods

10

Participants Demographic Data

How familiar the participants are with the "Queso"Scale from Unfamiliar Very Familiar

1 7

Devices access to Queso: 4 (66.7%) participants use Desktop/ Laptop

to access Queso2 (33.3%) participants haven’t used Queso

2 (33.3%) participants chose scale 1 2 (33.3%) participants chose scale 51 (16.7%) participants chose scale 61 (16.7%) participants chose scale 7

Methods

11

Participants Demographic Questionnaire DataParticipants’ Comments about Queso before usability test (4 participants used Queso before)

+ “It was a very useful tool with some opportunities for improvement. I already used blackboard and Queso is more friendly user.”

+/- “I found it good but slightly confusing sometimes. I never liked that the commentsand reviews disappeared once you watched them. It needs more interaction

optionslike Blackboard. I think it's too simple yet confusing sometimes. As far as I knowQueso aims to emulate the flexibility and fun of video games and bring that to theclassroom, but it needs to be more fun, exciting and visually appealing to reach

that.”

+/- “Good platform, needs to be reorganized.”

- “It's confusing and not sure how to edit an assignment.The navigation is in severaldifferent places which is confusing.”

“+” : Positive Comment “+/-” Both positive and negative comment “-”: Negative Comment

12

Objective To find usability problemsReason chosen Most frequently used and most troublesome task to be performedSource of task Team explored website and identified primary and secondary tasks to be tested

● Task 1 Submit an Assignment(Quest)● Task 2 Look up class grade (progress)● Task 3 Look up Points from an assignment(Quest)● Task 4 Revise a Submitted assignment(Quest)● Task 5 Access material posted from professor

*All task scenarios and success criteria can be found in the appendix

Tasks

Methods

MethodsPackets before usability test

● Moderator Guide● Participant Guide● Observer Guide● Test Plan

FacilityClassroom setting

13

Test Tools & EquipmentParticipant’s Computing Environment

● Display Devices○ Wireless connection to projector with Crestron

AirMedia and have HDMI cable as backup○ Dedicated test desk with 13-inch Retina Display

Macintosh Powerbook (2013) computer with OS X El Capitan installed open on Google Chrome and signed into Queso for each participant

● No Audio Devices were used● Manual Input Devices

○ Keyboard on Macintosh Powerbook○ Printed guides on standard 8.5 x 11 inch paper + Pens○ Apple Magic Bluetooth Wireless Mouse connected to

the computerTest Administrator Tools

● Devices: Two iPhones for time on task- Timing for test duration (overall)- Timing for task completion duration

Methods

Before Test

1. Created usability test plan, moderator, participant and observer guides2. Created individual accounts for each participant to perform usability test3. Ran 2 pilot studies4. Revised scenarios based on pilot studies5. Setup testing environment

14

Moderator Procedure

Methods

During Test

1. Greeted participant, read introductions and informed consent with participant2. Instructed participant to think aloud and say start and end time for each task3. Test session should not last longer than 25 minutes. 4. Recorded start time 5. Moderated task: take notes of observations and record problem identification6. Recorded total task time for each task7. Determined completion of task (pass/gray pass/fail) 8. Asked participant to take the Single Ease Question(SEQ) 9. Asked follow-up questions

After Test

10. Asked participants to take the System Usability Scale(SUS) questionnaire 11. Asked follow up questions after each task

15

Moderator Procedure

MethodsParticipant General Instructions

1. Arrive to the location at confirmed time2. Listened to Introductions3. Read and signed informed consent4. Filled out demographic survey5. Stated start time 6. Performed 5 tasks and followed think aloud protocol during each task7. Stated when finished task8. Answered the Single Ease Question (SEQ) 9. Answered the System Usability Scale (SUS) questions10. Answered follow up questions after each task11. Debrief

16

*One moderator interacted with one participant during the test and 5 observers observed the process.*Participant Task Instructions are in appendix session.

MethodsUsability Metrics

● Effectiveness Metrics○ Completion Rate- the completion rate is calculated by assigning a ‘1’ if the

test participant manages to complete a task and ‘0’ if he/she does not. A 0.5 is assignment to a task completed with an assist, or a “gray pass”.

○ Errors- Errors can be unintended actions, slips, mistakes or omissions that a user makes while attempting a task.

○ Assists - Moderator steps in to help users avoid and correct mistakes;

● Efficiency Metrics○ Task Time - Total time user/participant spent attempting to complete a task.

17

MethodsUsability Metrics

● Satisfaction Metrics○ SEQ- The Single Ease Question (SEQ) with a 1-7 rating scale to assess how difficult

users find a task. It's administered immediately after a user attempts a task.○ SUS- The System Usability Scale (SUS) with a 1-5 rating scale measures the usability. It consists

of a 10 item questionnaire with five response options for respondents from Strongly agree to Strongly disagree.

● Usability Keepers: Aspects, functions participants identified as positive attributes.

● Improvement Opportunities: Targets for redesign

● Problem Severity Metrics: Severity Score = Impact + Frequency○ Impact is the effect on the user experience, measured on a 1~4 scale○ Frequency of problem occurrence is converted into a 1~4 ranking○ So Severity Score is on a 2~8 scale. Problems with the greatest severity score are of

greatest priority18*Measuring standards for Impact and Frequency ranking are in Appendix

RESULTS

19

ResultsPerformance and Satisfaction Results Efficiency Metrics(Task Time: seconds )

20

Participant # Task 1 Task 2 Task 3 Task 4 Task 5

1 182 183 138 72 101

2 124 84 35 47 86

3 296 202 111 110 83

4 145 290 128 56 37

5 145 160 87 51 76

6 247 175 192 N/A N/A

Geometric Mean

(seconds)181 171 102 64 73

ResultsPerformance and Satisfaction Results Efficiency Metrics(Task Time: seconds )

21

The chart shows the Geometric Mean of task time for each task, including the Confidence Intervals(95%).

Participants spent more time on Task 1 and 2 than the other three. Task 1 is “Submit an Assignment” and Task 2 is “Look up class grade”.

ResultsPerformance and Satisfaction ResultsEffectiveness Metrics (Completion Rate by Task: Pass-1; Fail-0; Gray Pass-0.5 )

22


1 1 1 1 1 1

2 0.5 1 1 1 1

3 1 0.5 0 1 1

4 0 0.5 1 0 1

5 0 0 1 1 1

6 0 0 0 N/A N/A

Average 42% 50% 67% 80% 100%

ResultsPerformance and Satisfaction ResultsEffectiveness Metrics (Completion Rate by Task: Pass-1; Fail-0; Gray Pass-0.5 )Confidence IntervalConfidence level: 95%

24


Sub-TaskFind the

deadline

Upload a file

Make link

clickable

Find the

current grade

Find the

current points

Find the

grade

Find feedba

ck

Find revised

file

Resubmit assignme

nt

Find contact information

Check Presentation Schedule

1 1 1 1 1 1 1 1 1 1 1 1

2 0.5 1 1 0 1 0 1 1 1 1 1

3 1 1 0 0.5 0.5 1 1 1 1 1 1

4 1 1 0 0.5 0.5 1 1 1 0 1 1

5 0 1 1 0 1 1 1 1 1 1 1

6 0 1 1 0 1 0 0 N/A N/A N/A N/A

Average 58.33% 100% 66.67% 33.33% 83.33% 66.67% 83.33% 100% 80% 100% 100%

Performance and Satisfaction ResultsEffectiveness Metrics (Completion Rate by Task: Pass-1; Fail-0; Gray Pass-0.5 )

Performance and Satisfaction ResultsEffectiveness Metrics (Completion Rate by Task: Pass-1; Fail-0; Gray Pass-0.5 )

Confidence IntervalConfidence level: 95%

ResultsPerformance and Satisfaction Results Satisfaction Metrics(Single Ease Question with a 1 to 7 scale, higher means easier)

26


1 7 3 3 6 72 7 4 4 5 53 4 5 5 5 54 3 1 5 7 75 4 5 5 2 36 5 3 1 N/A N/A

Average 5 3.5 3.83 5 5.4

ResultsPerformance and Satisfaction Results Satisfaction Metrics(Single Ease Question with a 1 to 7 scale, larger number means easier)

27

Task 1Submit an Assignment(Quest)

Task 2Look up class grade (progress)

Task 3Look up Points from an assignment(Quest)

Task 4Revise a Submitted assignment(Quest)

Task 5Access material posted from professor

ResultsPerformance and Satisfaction Results Satisfaction Metrics(System Usability Scale with a 1 to 5 scale)

28

Participant # SUS Score

1 - Expert 37.5

2 - Expert 52.5

3 - Expert 72.5

4 - Novice 45

5 - Expert 55

6 - Novice 20

Average 47.9*Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average

ResultsIssue-Based Results Issues By Task

29

Task 1Submit an Assignment(Quest)

Task 2Look up class grade (progress)

Task 3Look up Points from an assignment(Quest)

Task 4Revise a Submitted assignment(Quest)

Task 5Access material posted from professor

ResultsIssue-Based Results Issues By Category

30

ResultsIssue-Based Results Issues By Participant- Usability Problem Matrix

31

Participants

Problems Title

FINDINGS

32

Positive Attributes(Keeper)

Task 1 Submit a QuestKeeper Proper use of language and location of important informationLocation Sidebar in Available Quests page, Completed Quests page and Class homepage

Description“Available” and “Completed” labels are self-explanatory for assignments, and the location in side navigation is constant.

33

Voices“Was helped by the “available” in available quests; Knew to assume that meant something still pending.”

Task 1 Submit a QuestKeeper Intuitive blue affordance for linkLocation Revise page, Attempt page

DescriptionWhen creating a hyperlink for submission, the url turns blue once it is successfully made a hyperlink. During usability testing this was noted as a positive affordance that informed of an updated system status.

34

Task 2 Look Up Your Class ProgressKeeper Finding total points earned in the Completed page was intuitiveLocation Completed Quests page

DescriptionHaving total points for all assignments on the completed page of a course was an intuitive flow for users to find their graded information. Keeping similar content together made finding and accomplishing task during usability testing efficient.

35

Task 4 Revise a Submitted QuestKeeper Naming of ‘Revise’ buttonLocation Completed Quests Page

DescriptionNaming of button matched the function it performed, so user knew that by clicking the ‘revise’ button they would navigate to the revise page.

36

Voices“Naming of ‘Revise’ button makes sense to me”.

Task 4 Revise a Submitted QuestKeeper Flow to revise an assignment matched the mental model of usersLocation Revise page

DescriptionDuring usability testing, a user noted finding previous assignments were easy and the properly labeled revise button made completing the task "easy"

37

FINDINGS

38

Problems

TASK #1Submit an Assignment (Quest)

39

Task 1 Submit an Assignment(Quest)Problem Making link clickable is inefficient and not intuitiveLocation Attempt page

DescriptionTo add a clickable link, users need to type some text first, then highlight the text, click the "Link" icon, and enter the URL before hyperlink will work. These required steps proved difficult for users to complete during testing

Problem Type InteractionElement Persistence Local

RecommendationAutomatically convert URLs to be clickable by auto detecting web address format or at least once hyperlink button is pressed; remove the need to retype link after pressing hyperlink button

40

Severity 2 3 4 5 6 7 8

Task 1 Submit an Assignment(Quest)Problem Due date disappears after submission of assignment Location View page

DescriptionDeadline date for assignments disappears from all Queso assignment descriptions after a user submits an assignment. All other content concerning assignment remains. This inconsistency did not match user expectations

Problem Type ContentElement Persistence Local

RecommendationCreate conventions and standards for Queso platform. No information concerning an assignment should be invisible at any point in the grading process

41

Severity 2 3 4 5 6 7 8

Task 1 Submit an Assignment(Quest)Problem Inefficient flow for uploading fileLocation Attempt page

DescriptionMultiple steps to upload a file make process inefficient: Opens My Computer, but then have to press Choose a File button to access any content.


RecommendationWhen clicking on "My Computer", show up the window for choosing files. Remove "Choose File" button.

42

Severity 2 3 4 5 6 7 8

Voice“I don’t see grades. It would be easier if it was all in

one place.” “I thought when I clicked “My computer” it would give me a list of my local files. I assumed it would be clickable.”

Task 1 Submit an Assignment(Quest)Problem Unnatural language and terminologyLocation Class Homepage

DescriptionFor novices, there is no explanation that "Quest" means "assignment." "Attempt" and "Expires" were not seen as a match between user mental model during testing.

Problem Type Naming conventionsElement Persistence Global

RecommendationChange "Quest" to "assignment", "Attempt" to "Submit", "Expires" to "Deadline" to match expectations of users during testing and create a dedicated about page that introduces course and philosophy behind gameful learning.

43

Severity 2 3 4 5 6 7 8

Task 1 Submit an Assignment(Quest)Problem Lack of confirmation of completing file uploadLocation Class Homepage

DescriptionThere is no clear notification when files are already uploaded. Lack of system status proved confusing to users unsure if they had actually triggered uploading of a file to turn in. The lack of system status proved inefficient in accomplishing tasks as users sat waiting for a file to upload that was already complete


RecommendationQueso should keep users informed about what is going on in the system with consistent update of placement in the site and feedback from actions. Create an alert once file is uploaded so user can submit assignment.

44

Severity 2 3 4 5 6 7 8

TASK #2Look up class grade (current level)

45

Task 2 Look Up Class ProgressProblem Not intuitive naming for “grade” and “points” Location Progress page

DescriptionIn Queso, "grade" is called "level" and "points" are called "skills", which is not intuitive. Users didn't understand how level corresponds to points range in the usability test. The way of showing points, "number + skill name", such as "5 Physical Computing" leads to difficulty accomplishing task goals

Problem Type Naming conventionsElement Persistence Local

RecommendationChange "current level" to "grade", "skills" to "points". Get rid of skills, and for showing points, use the format of "points earned / total points". For example, "8/10". Naming should match user expectations and standard conventions should be created for all courses in Queso.

46

Voice“I don’t see grades. It would be easier if it was all in

one place.”

Severity 2 3 4 5 6 7 8

Task 2 Look Up Class ProgressProblem Uninformative “Totals” in Completed Assignments pageLocation Completed Quest pageDescriptionIn the page of Completed Quests, the card of total points is called "Totals", which is not self-explanatory enough. Users can see nothing other than points earned, making it inefficient to find information about grade/points.

Problem Type Visual design and naming conventionsElement Persistence Local

RecommendationRename "Totals" based on an open card sort done with additional users and provide information like current grade. Display all information concerning grades in one location that is always visible and accessible to users.

47

Voice“There is no bar so I don’t know if it’s 40 out of 40”

Severity 2 3 4 5 6 7 8

Task 2 Look Up Class ProgressProblem “Totals” in Completed assignments page is not clickableLocation Completed Quest page

DescriptionThe assignment container that provides total information is not clickable, while users expected it to be in the usability test, to show information about their grade since none is visible from the completed assignments page

Problem Type Interaction and visual designElement Persistence Local

RecommendationWhile "Totals" is not clickable, we will redesign the completed grade display, removing the "Totals" section, so all information concerning a graded assignment is visible

48

Severity 2 3 4 5 6 7 8

Task 2 Look Up Class ProgressProblem Progress bars don’t match grade earnedLocation Completed Quest page

DescriptionLack of title for progress bar in side navigation that depicts current grade in a course caused confusion for users attempting to find their progress. This increases the memory load of users without explanatory text of what progress they're viewing.

Problem Type Visual designElement Persistence Local

RecommendationMake clear to users what information they are seeing by labeling features and creating standards in content shown. All progress bars should have points earned out of total points and a title referencing what points the progress bar is referring to.

49

Severity 2 3 4 5 6 7 8

Task 2 Look Up Class ProgressProblem Progress bar in left-hand navigationLocation Class homepage

DescriptionLack of title for progress bar in side navigation that depicts current grade in a course caused confusion for users attempting to find their progress. This increases the memory load of users without explanatory text of what progress they're viewing.


RecommendationMake clear to users what information they are seeing by labeling features and creating standards in content shown. All progress bars should have points earned out of total points and a title referencing what points the progress bar is referring to.

50

Severity 2 3 4 5 6 7 8

Task 2 Look Up Class ProgressProblem Progress bars are not clickable for more informationLocation Class homepage

DescriptionWhen trying to view grade information, two participants could not find what they were looking for and started clicking the progress bar. As information wasn't visible to user, they thought the progress bar would display more content.


RecommendationMove primary content concerning grading to one place and display it without the need of clickable content. Make progress bars clickable and show additional information about grading.

51

Severity 2 3 4 5 6 7 8

TASK #3Look up grade (points) on an assignment (Quest)

52

Task 3 Look up grade on an assignment (Quest)Problem Notifications of feedback disappear after viewing onceLocation Completed Quests Page

DescriptionThe feedback from the professor appears in the main page but disappears right after leaving the page. After that it is only visible in the revise page. There is no way to look back at previous notifications. There is lack of consistency and standards in the presentation of all material concerning grading.


RecommendationCreate a new completed assignment page that displays all information concerning graded work without removing any content. Standards in documentation available to user should mark feedback next to grade and always visible.

53

Severity 2 3 4 5 6 7 8

Task 3 Look up grade on an assignment (Quest)Problem Professor feedback is presented under the Revise page Location Revise Quest Page

DescriptionWhen looking for feedback from professor on a graded assignment during the usability test, participants clicked around all possible options, finally found information in the "Revise" page as its organization does not match mental models and expectations


RecommendationPut feedback with graded information so that similar content is in organized by one point of use.

54

Voice“I was able to click around and find the information, but it was not clear where professor notes were”

Severity 2 3 4 5 6 7 8

Task 3 Look up grade on an assignment (Quest)Problem Non-Intuitive visualization of grade earned on a completedLocation Completed Quests Page

DescriptionVisualization of grade (points) earned on a completed assignment is not intuitive and lacks title of what content is displaying.

Problem Type Visual DesignElement Persistence Local

RecommendationUsers could not see how many points they earned out of total points. The bar which displays the earned points is blank and lacks title to explain what it refers to which caused users to misread points earned vs. total points.

55

Voice“I see “5” but the progress bar is empty. It seems I haven’t submitted this assignment.”

Severity 2 3 4 5 6 7 8

Task 3 Look up grade on an assignment (Quest)Problem “View” button mismatches users‘ expectation for feedbackLocation Completed Quests Page

DescriptionUsers do not understand the meaning of "View" button. They feel the functionality of "View" button is not related to getting feedback; feedback from instructors appears in both "Revise" page and "View" page, which is overlapped.

Problem Type Icon IntuitivenessElement Persistence Local

RecommendationChange the naming from "View" to "Feedback". Make the feedback only appear in the "Feedback" page.

56

Severity 2 3 4 5 6 7 8

Task 3 Look up grade on an assignment (Quest)Problem Dropdown menu hides grade informationLocation Completed Quests Page

DescriptionCurrent page for graded assignments does not show any points users earned unless they click on a dropdown button in the top left corner for each completed assignment.

Problem Type Layout, hidden functionalityElement Persistence Local

57

RecommendationRemove dropdown button from all assignment information. This content is important and should be immediately visible on the page. Redesign the visual layout to use the available space to provide value to user by consistently showing grade information and feedback.

Voice“Points should appear without need to click.”

Severity 2 3 4 5 6 7 8

TASK #4Revise a submitted assignment (Quest)

58

Task 4 Revise a submitted assignment (Quest)Problem Revise page has redundant contentLocation Revise Page

DescriptionIn ‘Revise’ page, there is redundant content that is also include in the original submission page like an assignment to-do-list, information and feedback from instructors. This made users confused about the main functionality of this page.


RecommendationCreate an alert with the message confirming for the user that the file has been uploaded.

59

Severity 2 3 4 5 6 7 8

Task 4 Revise a submitted assignment (Quest)Problem Lack of feedback that a file was successfully uploadedLocation Revise Page

DescriptionWhen a user completed uploading a file, there is no feedback provided to user that signals that assignment was successfully submitted. This led to users feeling unsure of whether they had successfully submitted their file.


RecommendationCreate an alert with the message confirming for the user that the file has been uploaded.

60

Severity 2 3 4 5 6 7 8

Task 4 Revise a submitted assignment (Quest)Problem Previous submission information maintained in

working area of "Revise" pageLocation Revise Page

DescriptionNo separation of previous content submitted and work area for submitting revision. This led to confusion among users over what is being submitted as part of revision and what was part of previous submissions.

Problem Type Visual DesignElement Persistence Local

RecommendationMove content that is part of previous submission to 'original submission page', which can be accessed through the dropdown menu on the revise page; Work area to submit revision should be blank.

61

Severity 2 3 4 5 6 7 8

TASK #5Access material posted from professor

62

Task 5 Access material posted from professorProblem Posts on class homepage are not well organizedLocation Class Homepage

DescriptionPosts are displayed chronologically, so users became frustrated that important information is not organized by content category.

Problem Type NavigationElement Persistence Local

RecommendationOrganize content on the class homepage and create tabs for posts of different categories or importance. This will foster recognition over recollection by displaying sections of posts like readings, schedules and more.

63

Severity 2 3 4 5 6 7 8

Task 5 Access material posted from professorProblem No static location for contact informationLocation Class Homepage

DescriptionLocation of professor contact information was difficult to find for users because it is classified as a post and not as static content. The location isn't consistent. Users have to scroll down and find new placement once new content is posted.

Problem Type LayoutElement Persistence Local

RecommendationSet a dedicated, consistent and static area to provide contact information, and organize posts for consistent placement that users can recognize easily

64

Severity 2 3 4 5 6 7 8

Task 5 Access material posted from professorProblem Lack of clarity in title of professor contact informationLocation Class Homepage

DescriptionUsers had difficulty finding contact information for the professor due to unclear naming standards in Queso system


RecommendationCreate naming standards and documentation for general course posts and organize the posts by categories matching user mental model

65

Voice“It needs to be more obvious.”

Severity 2 3 4 5 6 7 8

RECOMMENDATIONS

66

Re-Design

67

Redesign Recommendations Main Points● Improve website visual style with affordances and visual hierarchy

○ Design Focus needs to be changed from a “game-like” style to a more focused and easy-of-use style. Similar functions grouped together at point of use.

● Clarify Language used by creating consistent and understandable naming conventions

○ Change site language to something more familiar to users. For example “Quests” should be assignments, “levels” should be grades.”Expires on date” should be deadline.

● Change navigation by adding secondary navigation focused on serving primary functionality

○ Navigation throughout site needs to be reconcepted. Build in a way to move forwards and backwards. This traditional navigation is not available in the current site.

● Improve information structure○ Better placement (sidebar) for contact info, syllabus, class, professor feedback alerts and

assignments due.

68

Redesign Recommendations Main PointsRedesigned site map

Current site map

Redesign Recommendation - User’s Home page Description of problems

1. Current homepage is unorganized and doesn’t clearly label different segments and added functionality of Queso

Recommendations from test results

2. Have all students enrolled classes together on the home page. Labeled “current.”

3. Access to “Past Classes”

4. Other home page content such as “settings” and “about” should be located here.

Redesign Recommendation - Available Assignments PageDescription of problems

1. Current course homepage houses unorganized posts without categorization or clear navigation to interact with information

2. Sidebar navigation wastes space with unexplained progress bar and only two main functions of the classpage

3. Primary uses of the site are submitting an assignment, checking current grade and viewing assignments that are still due. The current information architecture is inefficient

4. Uploading file button is too small and doesn’t provide system status feedback once a file has been uploaded

5. Lack of back button limits user freedom


6. Have everything at point-of-use for available and completed assignments

7. Submit drag-and-drop feature. A one stop directive to upload assignments rather than the current user interface that provides three options and causes confusion. Red direction type would change from red to blue so user knows file was uploaded by the system.

8. Top line information like “current grade” is always current and visible.

Redesign Recommendation - Completed Assignments Page

Description of problems1. Assignment information is segmented

across the current website causing users to recall placement instead of read organized navigation

2. When viewing a graded assignment, the grade earned and possible grade for an assignment are unclear due to lack of descriptive text and poor naming conventions

3. Graded assignments have a progress bar that doesn’t show any information and show points without saying whether they are grade earned from a graded assignment or the possible grade


4. Color coding visually shows user “assignment 01” has been completed and uploaded.

5. Revise assignment upload is found here.

6. Assignment grade, shows progress and success rate. Deletes current progress bar function.

7. Line graph shows progress and week through the semester

Redesign Recommendation - Reading Page

Description of problems1. ‘Posts’ on current homepage of

individual courses house important information, but during usability testing lack of clear naming conventions and organization lead to confusion and uncertainty

2. Contact information for professors and other important information like the syllabus were requested on separate pages and organized during usability testing


3. Readings provides up-to-date materials such as articles for the user. Student additions would also be available here. For example, class projects or inspiration, to share with fellow classmates.

Redesign Recommendation - Notifications Page

Description of problems1. Feedback from professor on a

graded assignment is found in the "Revise" page. This does not match mental models and expectations

2. The feedback from the professor appears in the main page but disappears. There is no way to look back work or notification to help guide users.

3. There is lack of consistency and standards in the presentation of all material concerning grading.


4. Notifications were not visible within the current queso system. Notifications are now seen as soon as user is logged onto the Homepage.

5. Under notifications user will now be able to find all feedback in one place.

74

ConclusionThrough the usability test run, 6 moderators identified 5 positive attributes of the current site to leave unchanged. The overview of parts to keep are the naming of certain functionality like the ‘Revise’ button on completed assignments page, the correct blue, underlined affordance for hyperlinks and the flow to access both the total points for a course and to revise an assignment.

However, there were 22 usability problems identified and 8 with a severity score over 6. Overall, opportunities for improvement include restructuring the information architecture of Queso’s environment, clarifying the visual design to help users know their place in the system in order to find the content they’re looking for. Also, improving naming conventions and standards would significantly reduce confusion heard from participants as they used the think aloud method during testing.

The next steps include running an open card sort to clarify language and terminology used in Queso. The redesign implemented new language but remains untested. An open card sort would identify any further problems or better naming opportunities. Furthermore, Queso could continue adding visual design suggestions for improvement of site functionality based on user testing of the redesign.

APPENDIX

75

1. Demographic Questionnaire2. Participant Task Instructions3. Post-task and Post-study Questionnaire4. Rubin’s problem impact ranking5. Rubin’s frequency ranking6. Raw Data7. Task Flows Outliers By Participants

Demographic Questionnaire How old are you?

◯ Under 18 ◯ 19 - 24 ◯ 25-30 ◯ 31 - 36 ◯ 37 - 42 ◯ Over 42

What gender do you identify with ?

Male Female

Employment and enrollment status?

▢ Part-time student and unemployed ▢ Part-time student and employed

▢ Full-time student and employed ▢ Full-time student and unemployed

▢ Non-student and employed ▢ Non-student and unemployed

76

Demographic Questionnaire ContinuedAre you a native English speaker?

◯ Native English speaker ◯ Non-native English speaker

Other: ________________________

What is your highest level of education earned?

◯ High school diploma ◯ Undergraduate degree

◯ Graduate degree ◯Post-graduate degree

◯ Other: ________________________

Are you a student from the University of Miami?

◯ Yes ◯ No 77

Demographic Questionnaire ContinuedHow often do you access the internet?

◯ Multiple times a day ◯ Several Times a Week ◯ Several Times a Month ◯ About once a month ◯ Less than once a month

How would you rate your current computer skills?

Have you ever used a learning management system for classrooms ?

◯ Yes ◯ No78

Demographic Questionnaire ContinuedWhat type of management system for classrooms have you used? Please write their names:

▢ Blackboard ▢ Schoology ▢ Queso

▢ Axis ▢ Moodle ▢ Absorb

How familiar are you with the Management system for classrooms "Queso"?

If you've used Queso, what device do you use to access it?

◯ Mobile device ◯ Desktop or laptop ◯ Tablet ◯ Haven't used Queso Other: __________

If you've used Queso, please write comments about the platform from your experience?

79

Task 1: Submit an AssignmentScenario

You are signed onto Queso in a class called “Test Physical Computing.” You’ve completed the assignment and it’s saved on your desktop

1. Find the deadline for the assignment 2.1 More in Serial Processing. 2. Turn in the assignment with a file "sketch.ino" 3. Add a link in a clickable format with this url: "http://bit.ly"

Success CriteriaFind the assignment 2.1 More in Serial Processing and write down the deadline: April 12th. Put the video link in the text editor, make clickable and click “Submit”.

Optimal Task flow

80

http://bit.ly/

Task 2: Look up your class progressScenario

You are signed onto Queso in a class called “Test Physical Computing.” 1. Find the current grade you’ve earned in the class thus far.2. Find the current points you’ve earned in the class thus far.

Success CriteriaFind the current grade is Level 1. The total points is 40.

Optimal Task Flow

81

Task 3: Look up Points from a QuestScenario

You are signed onto Queso in a class called “Test Physical Computing.” You receive an email from your professor that an assignment, 1.1 Blink LED, has been graded.

1. Find the grade you earned on the assignment 2. See if the professor gave feedback

Success CriteriaFind the grade is 5 points and the note is “Very Good.”

Optimal Task Flow

82

Task 4: Revise a Submitted QuestScenario

You are signed onto Queso in a class called “Test Physical Computing.” 1.1 Blink LED is an assignment you’ve already submitted.

1. Find the “newSketch.ino” file that is on the desktop of this computer2. Resubmit your assignment with the ”newSketch.ino” file

Success CriteriaFind assignment 1.1 Blink LED and upload the revised file “newSketch.ino” on the desktop and submit.

Optimal Task Flow

83

Task 5: Access material posted from professorScenario

You are signed onto Queso in a class called “Test Physical Computing.” 1. Find the contact information for this course’s instructor 2. Check the “Inspiration Presentation Schedule” to see when Lauren Kett is

presenting.

ObjectivesTo see whether the participant could easily find information from instructor’s posts.

Success CriteriaFind the instructor’s email address is [email protected], mobile phone number is 757-810-1913. Lauren Kett’s presentation date is Mar 31.

Optimal Task Flow

84

mailto:[email protected]

Overall, how difficult or easy did you complete this task?

System Usability Scale

1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated

After Task and Post Study Questionnaires

85

6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system

Severity Score Metrics

Impact Ranking

Impact Description

Impact Definition

4 Unusable The user is not able to or will not want to use a particular part of the product because of the way that the product has been designed and implemented.

3 Severe The user will probably use or attempt to use the product here, but will be severely limited in his or her ability to do so. The user will have great difficulty in circumventing the problem.

2 Moderate The user will be able to use the product in most cases, but will have to undertake some moderate effort in getting around the problem.

1 Irritant The problem occurs only intermittently, can be circumvented easily, or is dependent on a standard that is outside the product's boundaries. Could also be a cosmetic problem.

Rubin’s Problem Impact Ranking

Severity Score MetricsRubin’s Problem Frequency Ranking

Raw Data

Raw DataTask Flows Outliers By Participant 1

Task 1Optimal task flow.






Task 1Took a detour, didn't understand the how to create a link instruction.

Task 2She got to the progress page, but she was very confused with the labeling and start looking somewhere else.

Task 3She didn’t notice the points in the quest drop down area.


Task 5Participant didn´t scroll down right away in order to find the information

needed.


Task 1Optimal for submitting file; failed to also add clickable link.

Task 2Alternate flow (only see total points).



Task 5Optimal task flow, but initially attempted task by going off site.

Raw DataTask Flows By Participant 4

Task 1Start - Queso Homepage - Test Physical Computing Page - Go to "Available" - Find "2.1 More in Serial Processing" - Click "Attempt" - Upload the file(Add file) - Add video link in the text editor - Click"Submit" - Stop

Task 2Start - Queso Homepage - Test Physical Computing Page - Click username menu in Navi Bar - Go to "Completed" Quests - Check "Total Points" box - Click down arrow inside each assignment - Go back to Test Physical Computing Page - Click username menu in Navi Bar - Click "Progress" - Scroll down - Check total points and current grade - Stop

Task 3Start - Queso Homepage - Test Physical Computing Page - Go to "Completed" Quests - Find "1.1Blink Led" - Click down arrow - Click "View" - Go back to "Completed" Quests - Click "Revise" - Check "Notes on Quest" - "Stop"

Task 4Start - Queso Homepage - Test Physical Computing Page - Go to "Completed" Quests - Find "1.1Blink Led" - Click "Revise" - Upload files - Click "Submit" - Stop



Task 2Alternate to see total points, viewed alternate to see level but was unsure of answer and chose wrong content.





Task 1Optimal task flow - But

1. Clicked dropdown icon before "Attempt"2. Clicked the link icon twice.3. Uploaded file first.4. Clicked "My Computer" several times.5. Went back to "Available" to look for deadline.

Task 2Optimal task flow - But went back to "Completed" after "Progress", to look for Current Grade.

Task 3Optimal task flow - But continued to Class Homepage - Queso - Progress -

View.

usability test report for queso

Technology