usability test report - michigan technological university · test plan 2.1 usability test scenario...

Usability Test Report For CS4760 & HU4628 Group 3 -- BAER

Can Li ([email protected])

04/27/2015

Background: The primary mission of BAER Teams is to protect lives, property and sensitive habitats from post-fire effects. Once the danger posed by an active wildfire has passed, land managers must rapidly assess the threat from post-fire runoff and erosion due to the loss of surface cover and fire-induced changes in soil properties. Increased runoff and sediment delivery are of great concern to both resource managers and the public. Post-fire assessments and proposals to mitigate these threats are typically undertaken by interdisciplinary BAER teams. These teams are under very tight deadlines, so they often begin their analysis while the fire is still burning and typically must complete their plans within a couple of weeks. One of the primary information sources for making decisions is a burn severity map (derived from earth observations) that reflects fire induced changes in vegetative cover and soil properties. Burn severity maps can be improved upon with field observations of ground cover and measurements of soil hydrophobicity. This app will allow BAER Teams to gather, compile and share observations in an electronic format while they are in the field. Time is limited therefore an app that saves time would be very useful.

TABLE OF CONTENTS

1. Introduction ...................................................................................................................................... 3 1.1 Usability Test Background .................................................................................................................... 3 1.2 Usability Test Purpose ........................................................................................................................... 3

2. Test Plan ............................................................................................................................................. 4 2.1 Usability Test Scenario ........................................................................................................................... 4 2.1.1 Scenario 1: Create Form ..................................................................................................................................... 4 2.1.2 Scenario 2: Edit Form .......................................................................................................................................... 4

2.2 Common Usability Test Questions ..................................................................................................... 5 2.2.1 Pre-‐test Questions ................................................................................................................................................ 5 2.2.2 Post-‐test Questions .............................................................................................................................................. 5

2.3 Test Set-‐up Details ................................................................................................................................... 7 2.3.1 Schedule .................................................................................................................................................................... 7 2.3.2 Procedure ................................................................................................................................................................. 8 2.3.3 Role ............................................................................................................................................................................. 8

3. Results ................................................................................................................................................. 9 3.1 Participants Background ....................................................................................................................... 9 3.2 Quantitative Data ..................................................................................................................................... 9 3.2.1 Test Time Consuming ......................................................................................................................................... 9 3.2.2 Number of Times of Using Help ...................................................................................................................... 9 3.2.3 Wrong Input ............................................................................................................................................................ 9 3.2.4 Easy to Navigate Between Screens ................................................................................................................ 9 3.2.5 Easy to Find Fields ............................................................................................................................................. 10

3.3 Users’ Comments .................................................................................................................................... 10

4 Conclusion ......................................................................................................................................... 11

5. Appendix A: Participants Attendance 6. Appendix B: Bug Reports ............................ 12

1. Introduction

1.1 Usability Test Background The primary mission of BAER (Burned Area Emergency Response) Teams is to protect lives, property and sensitive habitats from post-fire effects. Once the danger posed by an active wildfire has passed, land managers must rapidly assess the threat from post-fire runoff and erosion due to the loss of surface cover and fire-induced changes in soil properties. Increased runoff and sediment delivery are of great concern to both resource managers and the public. Post-fire assessments and proposals to mitigate these threats are typically undertaken by interdisciplinary BAER teams. These teams are under very tight deadlines, so they often begin their analysis while the fire is still burning and typically must complete their plans within a couple of weeks. One of the primary information sources for making decisions is a burn severity map (derived from earth observations) that reflects fire induced changes in vegetative cover and soil properties. Burn severity maps can be improved upon with field observations of ground cover and measurements of soil hydrophobicity. BAER App is developed for BAER Team to gather, compile and share observation data in an electronic format collected from post-fire areas.

1.2 Usability Test Purpose The purpose of this usability test is to evaluate the efficiency of users inputting data using BAER App. Also, potential bugs may be exposed. Thus, the developers can make further improvement and modification based on this test.

2. Test Plan

2.1 Usability Test Scenario Considering there are two major functionality provided by the app -- creating the new form and editing the existed form, we design two test scenarios correspondingly.

2.1.1 Scenario 1: Create Form

2.1.1.1 Test Goal This test scenario is to evaluate how efficiency the user can create the form using the app. Note that it’s impossible to let the user input the data in the real post-fire area, we will provide the data on a paper form for the user to create the digital form. The time limit is 30 minutes.

2.1.1.2 Quantitative Measurement List ● The amount of time for one user to create the form ● The number of users who completes the task within 30 minutes ● The number of times when the user misses one or more required entries ● The number of places where the user input wrong data ● The number of times when the user hits the Help ● The number of times when the user gets stuck and has to ask developers for help

2.1.1.3 Scenario Description Assuming you are one of the members in BAER Team. You are at a post-fire area to collect data. Based on what you observe, you need to input all the data into your IPAD with BAER App.

2.1.1.4 Task List Hit the New Form and input data according to the paper form provided.

2.1.1.5 Qualitative Measurement List ● Facial expressions ● Verbal comments

2.1.1.6 Bug Report Form See Attached 1.

2.1.2 Scenario 2: Edit Form

2.1.2.1 Test Goal This test scenario is to evaluate how efficiency the user can edit the form just created using the app. We will specify three places which need the user to modify using the app. The time limit is 10 minutes.

2.1.2.2 Quantitative Measurement List ● The amount of time for one user to edit the form ● The number of users who completes the task within 10 minutes ● The number of times when the user hits the Help ● The number of times when the user gets stuck and has to ask developers for help

2.1.2.3 Scenario Description Assuming you are one of the members in the BAER Team. You finished the data collection from the post-fire field and stored those data in your IPAD with the app. Now you need to open the existed form you just created and modify several data.

2.1.2.4 Task List Hit the Edit/View Form, find the form just created and edit the data specified.

2.1.2.5 Qualitative Measurement List ● Facial expressions ● Verbal comments

2.1.2.6 Bug Report Form See Attached 1

2.2 Common Usability Test Questions

2.2.1 Pre-test Questions 1). Name and Major 2). How many years have you used a smartphone/smart-device? 3). Please indicate your level of agreement to the follow statement: I am very interest in the testing of this android application.

● Strongly agree ● Agree ● Neutral ● Disagree ● Strongly disagree

2.2.2 Post-test Questions 1). Please indicate your level of agreement to the follow statement: Overall, this application was easy to perform the task.


2). Please indicate your level of agreement to the follow statement:

I enjoy using this application. ● Very much ● A little bit ● Neutral ● Not very much ● Not at all

3). Please indicate your level of agreement to the follow statement: I would use this application again.


4). Please indicate your level of agreement to the follow statement: I feel the time (30 minutes) for creating a form is sufficient.


5). Please indicate your level of agreement to the follow statement: I feel the time (10 minutes) for editing three places in a form is sufficient.


6). Please indicate your level of agreement to the follow statement: I feel the paper form provided is helpful for inputting the data.


7). Please indicate your level of agreement to the follow statement The terminology used throughout the app consistent and clear

● Strongly agree ● Agree ● Neutral

● Disagree ● Strongly disagree ●

8). Please indicate your level of agreement to the follow statement I could easily find the fields in the app that correspond to those in the paper form


9). Please indicate your level of agreement to the follow statement I feel it is easy to navigate between the screens of the app? (Navigating to the Home screen, Sync screen, Edit screen or Help screen)


10). Please indicate your level of agreement to the follow statement I used the Help screen while performing the task and it’s helpful.


11). Describe any difficulties you had during the whole test. 12). Do you have any suggestions or comments for this app or this test procedure?

2.3 Test Set-up Details

2.3.1 Schedule

Time Location

April 13 (Monday) 5 -- 6pm Rekhi 218

April 14 (Tuesday) 11:30 -- 12:30pm MEEM 402

April 14 (Tuesday) 5 -- 6pm Rekhi 218

April 15 (Wednesday) 5 -- 6pm Rekhi 218

April 16 (Thursday) 11:30 -- 12:30pm MEEM 402

April 16 (Thursday) 5 -- 6pm Rekhi 218

April 17 (Friday) 5 -- 6pm Rekhi 218

2.3.2 Procedure ● Preparing devices and other paper materials such as the consent form, the paper form

providing the data, the introduction page and the questionnaire page ● Briefly introducing our test to the participant, such as the background, the goal and

scenarios we are going to use ● Answering the pre-test questions ● Performing the test scenario 1 and 2 ● Answering the post-test questions

2.3.3 Role The graduate student, as the test administrator and three undergraduate students should be present at each session. One undergrad is responsible for introduction and the other two will record quantitative data and bugs.

3. Results

3.1 Participants Background ● Major

○ 1 Computer Science ○ 3 Electrical and Computer Engineering ○ 1 Math ○ 1 Electrical Engineering

● Experience of using smart device ○ 4: more than 4 years ○ 2: less than 2 years

● All showed interest in testing the App

3.2 Quantitative Data

3.2.1 Test Time Consuming ● Create Form

○ 5 people: less than 10 minutes ○ 1 people: 15 minutes

● Edit Form ○ 5 people: 1 minute ○ 1 people: 2 minutes

3.2.2 Number of Times of Using Help No one used Help functionality, which proves that the design of the app is pretty straightforward. On the other hand, it may due to the test was over-specified, especially in the Edit scenario.

3.2.3 Wrong Input This data recorded the number of times when students input wrong data or missed one or more entries.

● 2 people: 2 mistakes ● 3 people: 1 mistakes ● 1 people: 0 mistakes

It is safe to say that people can use this app to input data accurately even if he is not a BAER team member.

3.2.4 Easy to Navigate Between Screens This question asks users if they feel easy to navigate between different screens such as Home, Edit and Help.

● 2 people: strongly agree ● 3 people: agree ● 1 people: neutral

Most students feel it’s very easy to navigate between different screens in the app, which proves that the layout design is reasonable.

3.2.5 Easy to Find Fields This question asks users if they feel easy to find the corresponding fields in the app according to the paper form.

● 1 people: strongly agree ● 2 people: agree ● 3 people: neutral

Half of the students felt neutral, which means they might have a difficulty to find the fields. I suppose this is acceptable since students never have experience of using the field form.

3.3 Users’ Comments “Water drop test was unclear” “...units were not consistent” “Have the app auto save when I go to the observation page” “...app was well laid out, data entry was easy to follow”

4 Conclusion BAER App is very easy to use, even if the users is entirely new for the field data. Navigation between different screens is very convenient. The layout design is very reasonable. Basically, BAER App successfully converts the paper form into the digital one, which will allow BAER Team collect and share data much more easily. Some improvements include:

● Any bugs appeared during tests need to be fixed ● Units were not consistent ● Water drop test is confusing

5. Appendix A: Participants Attendance

Usability Test Attendance Sheet BAER App – Can Li

Session Time Location Participant Course Attendance

1 4/13, 5pm -- 6pm Rekhi 218 Christopher Coppernoll

CS2321 ✔

2 4/14, 11:30am -- 12:30pm

MEEM 402 Nopparuj Saipong CS2311 ✔

3 4/14, 5pm -- 6pm Rekhi 218 Adam Kausch CS2311 ✔

4 4/15, 5pm -- 6pm Rekhi 218 James Koole CS2311 ✔

5 4/16, 11:30am -- 12:30pm

MEEM 402 Ethan Hoffman CS1121 ✔

6 4/16, 5pm -- 6pm Rekhi 218 Ethan Koeppe CS2311 ✔

7 4/17, 5pm -- 6pm Rekhi 218 Brett Kolz CS2311 --

6. Appendix B: Bug Reports

usability test report - michigan technological university · test plan 2.1 usability test scenario...

Documents