Round 1 UX Research Cycle

I.Design: Making ID Real A User Experience Design Challenge

EDIT 752

Spring 2018 Dr. Bannan

Jordana Anderson • Youliduzi Niyazi • Mary-Leigh Phillips • Brett Sparrgrove • Sonya

Wein

Background 2

Goals 2

Methodology 2 Research Questions 2 Participants 3 Think Aloud Approach and UX Inspection 3 Brief Questionnaire 5

Round One Data Collection 6 Think-Aloud Results 6 Questionnaire Results 6

Analysis 9 Task- Based Analysis 10 UX Inspection Analysis 11

Conclusions 12

Appendix I: Questionnaire Detailed Responses 13 Participant 1. 13 Participant 2. 15 Participant 3. 17

Appendix II: Round 1 Testing User Instructions 19 I.Design: Making ID Real 19 Recording 20 Informed Consent 20 Testing Process 20 Informed Consent Form 21 Think Aloud Overview 23 Series of Tasks 24

Appendix III: Think-Aloud Data 25

1

Background The group designed a technology resource that enhances the learning experience of instructional design and technology (IDT) students who are pursuing degrees and certificates at George Mason University (GMU).

Specifically, the technology resource curates concise, accurate, and accepted IDT information. It also provides practice scenarios that 1) show how IDT principles are applied in real-life design projects and 2) poses design problems for users to reflect on/solve.

Goals The goal of the research and testing plan is to determine how best to refine, enhance, and re-design the I.Design application. The research phase will include two rounds of testing. Each round will include task-based and user experience inspection methods. The prototype will be redesigned to incorporate key user feedback after each round. The research and testing process will answer the following overarching questions:

● Are efficiency and flexibility balanced within the application workflows? ● What is the emotional impact of the application? ● Is the design intuitive and consistent? What is the impact of the design on the

user’s memory load?

Our objective is to improve the application in each of these areas. The following section outlines additional research questions falling under the overarching themes and objective.

Methodology Research Questions Building from our overarching goals for the test and evaluation phase, additional research questions can be grouped into the following categories, 1) task-based and 2) user experience inspection. These questions are listed below.

Task- Based

● Do the users understand all that is available to them in the mobile application from the first page?

2

● Where might there be redundancy (e.g. unnecessary clicking, superfluous content)? Where do we need to build in more support?

● Does the user understand the program’s feedback and understand what to do with mentor interaction in the scenarios or how to favorite a page?

UX Inspection

● Are the favorites features (i.e. playlist) organized efficiently for effective UX?

● Does a user find making an account helpful in order to use the app as it was intended to be used?

● Can the user easily navigate the scenarios and find all its features useful?

Participants Due to low rates of detection of design flaw rates by individual inspectors, we selected 3 individuals with this dual expertise to inspect our design using a heuristic approach rather than a single dual expert/inspector. We did not want to select more than 5 participants as the standard operation model in the UX field shows that more than 5 dual experts will produce diminishing returns. Participants were peers recruited from current IDT classes at GMU; they are all pursuing master’s degrees. The participants have dual expertise as domain experts in instructional design as well as expertise in user experience design. The three people who tested the app had instructional design processes and methodologies fresh in their minds and were assessing the app as the ideal user for this app. Think Aloud Approach and UX Inspection The participants inspected the application using heuristic evaluation and corresponding forms to complete key tasks outlined in Appendix II. The team has iterated the key tasks from earlier high-fidelity design tests due to the high quality test results many of the initial tasks produced in that phase. The inspection also included a think-aloud component and captured participants’ performance using Snagit a web application that allows for screen capture as seen below in Image 1. Screenshot of a Participant’s Think-Aloud Recording.

3

Image 1. Screenshot of a Participant’s Think-Aloud Recording

Appendix III Thematic Map Data includes a table of the participants’ qualitative think-aloud feedback as well as their heuristic rating forms. The ability to capture tone of voice and potential frustrations allowed us to extract initial emotional impact information during this phase. The Think-Aloud participant tasks (see Appendix II) led each participant through the working features of the prototype version 1 looking at the following features:

● Home Screen: ○ Tutorial ○ Login/account ○ Search

● Glossary: ○ Glossary video ○ Favorite feature

● Practice: ○ Filter for scenarios ○ Selecting a Scenario ○ Scenario Navigation and Icons

■ Scaffolding ■ Mentor Feature ■ Bookmarking your spot

4

Throughout testing, the participants did not comment on every feature listed above. The lack of comments seemed to indicate that the feature was clear, there was ease of use and the feature met user expectation. The following are features that fall into this category:

● Playing the glossary video ● Selecting a scenario

Brief Questionnaire Following the heuristic inspection sessions, the team sent out a brief questionnaire to the participants to capture more data on the application’s emotional impact. The questionnaire will include targeted questions on their thoughts and feelings while using the application.

5

Round One Data Collection

Think-Aloud Results After watching the recordings of the Think-Aloud and screen-share video recording, we collected the qualitative data using a thematic map approach. Chart 1. Think-Aloud Thematic Map

The above chart aggregates the feedback from the participants’ recorded experiences as they followed as series of tasks outlined in Appendix II. The x-axis defines the sections of the I.Design app on which participant provided feedback or noteable reactions.

Questionnaire Results Following the Think-Aloud, participants further evaluated their experience by answering ten questions. The heuristic evaluation forms were aggregated below:

6

Table 1. Questionnaire Responses

Question Participant 1 Participant 2 Participant 3

1.Ease of Navigation

5 4 Overall intuitive except document icon (looks like notes)

5

2. Usefulness 5 5- would be great in professional sense/ onboarding.

5 - “I do not feel lost”. Great for students.

3. Overall Aesthetic 4 5- Change blues in tutorial, add arrows in tutorial, add search to all screens not just homepage

5- Initially thought app tried to do too much, eventually got comfortable. Make glossary text color black.

4. Amount of Multimedia

Positive Yes Yes- good balance

5. Interactivity Positive Yes but prefer results list in search

Yes

6. What do you Like?

Concept, layout, design

Yes - especially the practice area

Liked the initial pop up explaining the app, liked the flow-encouraged outside exploration, overall good flow.

7. What was confusing or difficult?

Nothing, app intuitive

Mentor feedback is confusing - need to make clear not chatting directly with mentor.

Department wouldn’t fund development of such an app

8. Is it helpful for school/work?

Yes, as a student, especially with filters

Yes-especially for newer designers. Like ADDIE format.

Yes - current professor content is all over blackboard, this would streamline it.

7

9. Suggestions Some crowded screens, especially first screen

Add a place for collaboration from experienced designers to be mentors, add notes or comments feature, make bookmark versus favorites more clear or remove bookmarks.

Add back button or “I design practice” clickable so not having to go through home each time

10. How does it compare to other resources?

Nothing like this currently

Nothing else like it in a “one stop shop”, especially like practice feature.

Playlist feature is “pure brilliance”. Nothing else like this especially practice pages.

The comments above were consistent with the feedback in the video. Overall these comments seemed more positive. Many of the suggestions were features that I.Design will eventually include.

8

Analysis

Below is a comparison of the results. We will incorporate the most pressing improvements, as resources allow, into the Round 1 Design Prototype before beginning the second round of testing. Chart 2. Think-Aloud Feedback

The chart above shows the balance of positive and negative feedback from the Think-Aloud video recordings. There are four data points that were considered positive feedback because the task was completed successfully during the Think-Aloud. However, due to hesitation, underutilization of the feature, or inflections in tone of voice of the participant, two data points are marked as unclear and two as underutilized in Chart 1. Think-Aloud Thematic Map. Below are the areas that may need improvement. There were several helpful comments that were repeated among the participants. These (see Table 1 below) will be considered for updates prior to Round 2 testing.

9

Table 2. Areas that Need Work

Category App Location Implementation: Prototype 2.0

Unclear/ Poor navigation

Initial Tutorial Streamline Tutorial

Account/Login Navigation Remove initial Login screen Other edits will be implemented in further buildouts

Practice Mentor Think-Aloud Prompt from “Solicit Mentor” to “View Mentor”

Practice Icons Remove/Replace Icons

Favorites vs Bookmarks Move “Favorite” icon from the top navigation to bottom left near info “i” to declutter top navigation and indicate optional use.

Accessing Bookmarks Remove/Alter Bookmark Feature

Cognitive Overload/ Dissonance

Initial Tutorial

Simplify information by removing the obvious

Practice Scenario description

Test further and Consider future formatting options

Underutilized Favorite Tagging Test further

Practice Scaffolding Icon Consider future user testing for alternative icons/titles for this functionality

Task- Based Analysis

Do the users understand all that is available to them in the mobile application from the first page?

The tutorial helped to address this, however it seemed to be too much information. Prototype improvements will be made to improve this for Round 2.

Where might there be redundancy (e.g. unnecessary clicking, superfluous content)? Where do we need to build in more support?

10

One feature of redundancy mentioned was the login avatar head. A participant mentioned that click was unnecessary to get to the create account/login feature. It seems overall there was too much support and the next prototype needs to simplify features (see Table 2. Areas that Need Work)

Does the user understand the program’s feedback and understand what to do with mentor interaction in the scenarios or how to favorite a page?

The users seemed to understand the Stop and Think and Mentor features, however, their expectation for user engagement with these features were not entirely met. Most users expected live interactivity with other users, an area to take or give notes and the ability to connect with a live mentor. These features functionalities have not been completely built out and may include those capabilities in future models. Round 1 and 2 will not include this development. Nevertheless, the original design intends to have a connect feature and pre-recorded video for mentor engagement.

UX Inspection Analysis

Are the favorites features (i.e. playlist) organized efficiently for effective UX?

This Round of testing did not include extensive favorites feature functionality. Nevertheless, the tag feature seems unfamiliar to users and created hesitation. Other apps commonly used today use the heart icon without a tag feature. As a result, the action of saving a tag created hesitation. This is something we will continue to consider.

Does a user find making an account helpful in order to use the app as it was intended to be used?

Yes, at least one participants had a notable positive reaction to the account/login features. She appreciated that the app could be personalized, even with a photo. Moreover, she recognized that the account gave the user the ability to personalize the experience with favorites and bookmarks.

Can the user easily navigate the scenarios and find all its features useful?

Yes, user seemed to find their way through the main content of the practice scenarios. Nevertheless, there were improvements to be made with the use and necessity of the icons. Users mentioned the icons in the scenario made the screen “cluttered”. We will re-evaluate the icons and seek to simplify the navigation. The features like “mentor” and “stop and think” were understood but did not meet user expectation.

11

Conclusions Overall, round 1 testing successfully highlighted functions of the application that met user expectation as well as areas that needed to be altered. The research and testing process sought to answer the following questions: Are efficiency and flexibility balanced within the application workflows?

Participant feedback reveals that the app overall is very intuitive and efficient. However, there were specific functionalities or features like the tutorial and the practice scenario icons that could be more efficient. The flexibility of the account login feature is an area that we did not spend much time designing for this prototype as it was not unique to our application. Nevertheless, participants brought the feature to our attention, revealing the heart of User Experience design: usability. For this and the next implementation, only small changes will be made to address this feature until the app is developed further.

What is the emotional impact of the application?

The questionnaire following the Think-Aloud clearly reveals that the participants were very positive and hopeful to be able to use the app upon its completion. Consistent comments like “fantastic!” “I love it!” and “I would definitely use this as an an Instructional Designer!” confirms an overwhelmingly positive emotional impact.

Is the design intuitive and consistent? What is the impact of the design on the user’s memory load?

The participants commented directly that the app was intuitive. There were suggestions to make navigation more consistent, specifically in the practice feature. The practice section is the heart of the application, and therefore includes much of the information for the user. Cognitive overload seemed to be an issue in the tutorial and the practice section’s initial screen. Other than that, the design was considered easy to use, clean and clear. The information in the glossary and within the scenario itself received positive feedback. As a result, small improvements will be made to the prototype as listed in Table 1 above. Research testing for round 2 testing will focus on the same goals and questions but a different group of participants. Participants who are not instructional designers or experts in user experience design will be asked to give their feedback. This will allow us to see if I.Design is intuitive and easy to navigate regardless of one’s level of expertise in the field.

12

Appendix I: Questionnaire Detailed Responses Participant 1.

13

14

Participant 2.

15

16

Participant 3.

17

18

Appendix II: Round 1 Testing User Instructions  

 

I.Design: Making ID Real Thank you for helping us test our prototype! We are very interested in your feedback. We have created this information packet to clarify the testing procedures for you. Please review it before we meet, and do not hesitate to contact us if you have any questions!  

Jordana Anderson • Youliduzi Niyazi • Mary-Leigh Phillips • Brett Sparrgrove • Sonya Wein  

 

19

Recording You can use whatever recording software you have available to capture and send us the test. If you would like one of our designers to manage the recording, he/she will use Zoom. Zoom is a video conferencing platform that allows participants to see and hear each other and share what’s on their computer screens. In order to use Zoom, you will need to sign up for a free account at https://zoom.us/.  Informed Consent Because we will be recording the testing session, we request your permission. Please review and sign the informed consent form on page 3.  Testing Process 

1. If you chose to use Zoom, first, you will receive an email with a Zoom link, which will take you to the virtual space wherein you will do the pilot testing. One of our designers will email you this link once you agree on a day and time for the test. 

2. Once in the virtual space, the designer will greet you and begin recording the session.  

3. The pilot testing involves exploring our app by trying to accomplish a series of tasks. As you move through the app, we would like you to vocalize your thoughts. This is called a think aloud. The designer will run through both the tasks and the think aloud protocol with you before beginning the pilot testing. You can also review them on pages 5 and 6 of this packet. 

4. The designer will provide you with a link to the I. Design prototype. 

5. In Zoom, you will share your screen with the designer so he or she can see what you see. Then, you will explore the app and think aloud. 

6. At the conclusion of the pilot testing, the designer will thank you for your participation and provide you with a follow-up questionnaire link. Please complete the questionnaire as soon as possible following the pilot testing. 

20

 

Informed Consent Form I.Design Learning Application

RESEARCH PROCEDURES This research is being conducted to test to usability, functionality, and effectiveness of the I.Design Mobile Learning Application prototype. If you agree to participate, you will be asked to test the application by completing specific tasks, exploring the application on your own, and providing feedback. RISKS There are no foreseeable risks for participating in this research. Participants are requested to wear glasses or contact lenses as they ordinarily would while using a laptop or tablet to prevent eye strain. BENEFITS There are no benefits to you as a participant other than to further research into the usefulness of a learning application for instructional designers. CONFIDENTIALITY The data in this study will be confidential. Names and other identifiers will not be placed on surveys or other research data. This study involves the audio or video recording of your interview with George Mason students. Neither your name nor any other identifying information will be associated with the audio or video transcript. Only the George Mason university students in 752-002 will be able to listen (view) to the video recordings. Transcripts of your interview may be reproduced in whole or in part for use in presentations or written products that result from this study. PARTICIPATION Your participation is voluntary, and you may withdraw from the study at any time and for any reason. If you decide not to participate or if you withdraw from the study, there is no

21

penalty or loss of benefits to which you are otherwise entitled. There are no costs to you or any other party. CONTACT This research is being conducted by as a part of a course on research and design methods, EDIT 752-002 at George Mason University. The research team may be reached by contacting Mary-Leigh Phillips by email at maryleighp@gmail.com for questions or to report a research-related problem. This research is being conducted under the guidance of Dr. Brenda Bannan, bbannan@gmu.edu. You may contact the George Mason University Office of Research Integrity & Assurance at 703-993-4121 if you have questions or comments regarding your rights as a participant in the research. CONSENT I have read this form, all my questions have been answered by the research staff, and I agree to participate in this study. _______ I agree to audio (video-) recording. _______ I do not agree to audio (video-) taping.

__________________________

Name

__________________________

Signature

__________________________

Date 

22

Think Aloud Overview 

 

Thank you for participating in this test! While you use I.Design, we would like you to vocalize your thoughts so that we can better understand your experience. This is called a think aloud. The following video demonstrates the think aloud process: https://www.youtube.com/watch?v=g34tOmyKaMM.  

WHAT DO YOU THINK ALOUD ABOUT? 

Think aloud as you’re wondering or trying figure something out, for instance. Talk about what catches your eye or confuses you. It is important for you to say what you’re thinking not describe what you’re doing.  

WHAT IF I DON’T KNOW WHAT TO SAY? 

We might prompt you to think aloud by asking, “What are you thinking?”, “What do you think you should do next?”, or “What do you/did you expect will/would happen?” 

SHOULD I FILTER MYSELF? 

Be honest in your thoughts; we’re not judging you, and you won’t hurt our feelings! 

   

23

Series of Tasks Please complete the following tasks as you move through the app. Remember to think aloud as you navigate. You can also leave written comments on the right side of the screen. 

1. Navigate the pop-up tutorial. 

2. Login. 

3. Create an account (you don’t actually have to enter any info… just click create 

account) 

4. Return to the home screen. 

5. Access the glossary and perform a search for the term “Multimedia Principle”. 

a. Play the video. 

b. Favorite the page. 

6. Return to the home screen. 

7. Go to practice. 

a. Explore the variety of filter options (ADDIE Process & Skill level). 

b. Expand the details of the scenario ‘campus safety’. 

i. Select this scenario to experience 

8. Located the ADDIE menu. Under the ‘analysis’ part of the ADDIE process, select 

‘interviewing audience’. 

a. Use scaffolding icon to support your learning. 

i. Exit out of the scaffolding assistance. 

b. Select the stop and think icon.  

i. Solicit the mentor’s feedback. 

ii. Exit stop and think. 

c. Bookmark this scenario. 

9. Return to the home screen. 

24

Appendix III: Think-Aloud Data

TESTER 1 Feedback Tutorial Account/Login Favorite Practice

Ease of Use Mentor

Questions Suggestions

Navigate the pop up tutorial

tone indicated one box to many

cognitive overload

how do i go back to tutorial if needed? last box i button for info was not comprehended

cognitive overload

where can i find the tutorial again?

favorite a page

skips favorite tagging and just presses the heart

underutilized

Scaffolding button

scaffolding still unclear, she used the A dropdown first and then the i for information to review the icons

underutilized

the play button leads to an intuitive click expecting a video intuitive intuitive

scrolls down in scaffolding pretty intuitively intuitive

TESTER 2 Feedback Tutorial Account/Login Favorite Practice

Ease of Use Mentor

Questions Suggestions

Navigate the pop up tutorial

she would normally skip tutorials

did not find home button on the screen other than in the instructions

cognitive dissonance

noticed a spelling error in tutorial

spelling error in tutorial

change in text color give the impression that it is a link mentioned 2 times

Title text color false impression

25

this tester is the only tester who noticed the boxes are associate with the tabs on the homepage clear

suggests an arrow that point to the location of the icon mentioned in the tutorial

arrow to aid in text-image association

Create an account

positive response to example completed account with picture

positive impression

mentions login is not the same as create an account. Suggest to save a click by having the login and create an account options up front unclear

save a click by having the login and create an account options up front

suggests using tab to move through the input boxes

tab as a shortcut to navigate on desktop

Search for multimedia principle

seeking a list of results related to search

Expects live version provides a list of search results

Practice: Explore the filters

expresses that she likes the filters

positive impression

skill level filter is unclear at first. suggests a definition for skill level (based on where you are at from an ID experience standpoint) and complexity (how complex the senario is) theory (the different theories like constructivist...) helpful

Choose a scenario

appreciates the details provided under the scenario helpful

26

Locate the Analysis part of ADDIE

The ADDIE buttons were not discussed in the tutorial and it is not clear at first

Use scaffolding icon

tester found the scaffolding icon however didn't understand it intuitively but found the info provided in the section helpful unclear

tester chooses to favorite (calls it a save button) without being provoked intuitive

scrolls down in scaffolding pretty intuitively intuitive

****

wonders how to access favorites and figures it out by going to home page but has trouble returning to where she left off in practive

poor navigation

****

would like to add notes about the content for personal use- not available

add a section for notetaking

Stop and Think

the stop and think icon/section is helpful helpful

Stop and Think Video is unclear: is it a chat? conference session? provide options to solicit mentor feedback. What is the intent? "or is it just a video of the mentor talking about the reflection excercise." unclear

Bookmark

What's the difference between Bookmark and Favorites unclear

How to access bookmarks? tester unclear

27

goes to favorites to search

explores the settings and wonders about the suggestion box. Tester unable to identify it. unclear

Returns to practice and suggests a search bar within practice

search bar in practice

does the search within the glossary alone or everything in the app?

does the search within the glossary alone or everything in the app?

loves the colors of the app- trendy

clear design

accessibility button is awesome helpful

easy to navigate clear design

extra icon with hand is confusing perhaps unneccesary unclear

took 26 minutes to find the i info button

cognitive overload

TESTER 3 Feedback Tutorial Account/Login Favorite Practice

Ease of Use Mentor

Questions Suggestions

Navigate the tutorial

boxes seem to jump around is offputting but pretty self explanatory/some unnecessary but prefers to have too much info than not enough unclear

design is sharp and clean

clear design

28

color combinations is great

clear design

home instruction seems unnecessary because it is a universal symbol

cognitive overload

remove home button instruction

search is pretty intuitive intuitive

quickly identifies the heart as a "favorite page" feature but is thrown off by the word "tag" and the save icon but the text in the box helps to clarify

underutilized

familiar heart icon but unfamiliar input for tag

loves the filters "fantastic" to be able to filter by the ADDIE process helpful

The text under scenario is "hard to read" but the design is intuitive/universal

cognitive overload

scenario info busy

Use scaffolding icon

identifies the scaffolding icon quickly clear

ADDIE navigation in Practice

finds the A dropdown but hesitates unclear

finds the navigation in the practice "a little too crowded"

cognitive overload

Stop and Think

wonders if person is supposed to stop and think questions and get feedback from mentor or if user can input answers unclear

suggests a live back and forth conversation

converse with mentor real mentor

29