1

1. Introduction

The Wide Area Video Communication Application is a mobile video call

application that offers features namely Video Call, Video Chat, and File Transfer.

1.1 Purpose

The primary purpose of the test was to test the functionality of the application and

investigate whether users are comfortable with the applications features and

functionalities.

Executive Summary

The researchers conducted a usability test to selected students from Cor Jesu

College on November 8, 2014. The students were provided with the application’s

Android Project Kit (APK) and Wireless Fedility (Wi-Fi) connection to test the

functionality of the application.

All the selected students are from the Computer Studies Division (CSD). There

are only 10 selected CSD students to perform the test due to some of the students don’t

have android devices or smart phones.

The test identified some of the problems stated below:

Stability of sessions

The sessions are unstable due to the availability of the web

server. The website where the sessions are made is not always

available. There are times when the website is timed out without

certain reason.

Quality Output

The video and voice quality is sometimes poor due to the

internet connection’s performance. The quality output depends on

how fast and reliable the internet connection is.

Incompatibility of the application to any Android mobile Operating

System (OS).

2

Methodology

Sessions

The researchers recruited 10 CSD students to perform the test. The researcher

gives a brief introduction to what the application is about and the functionality of the

application. Then, the researchers gives proper instruction on how to use the application.

The students were gathered before giving the instructions. The session lasted for

15 minutes including the giving of the instructions and the test questionnaires for

evaluation.

After each task, the administrator asked the participant to rate the interface on a 2-point

Likert Scale with measures ranging from Disagree and Agree. Post-task scenario subjective

measures included (see Attachment B):

the application is easy to use;

the application is modernize and offers friendly user interface;

the application is functional due to new technologies involved namely Android;

and

the application is useful to the society.

After the Post-task scenario subjective measures, the students were asked to rate the

application overall by using a 5-point Likert scale (Strongly Disagree to Strongly Agree) by the

eight subjective measures mentioned below:

ease of use;

frequency of use;

learning ability - How quick would it be for the users to learn the application

and its features;

graphical Appeal – User likes the User Interface of the application;

3

application content – Applications contains sufficient functions for the needs of

the user; and

organized Application.

Lastly, the researchers asked the students the following overall application

questions listed below:

what the participants likes the most in the application?;

what the participants likes the least in the application?; and

comments and Recommendations for improvement.

Refer to Attachment C for the subjective and overall questionnaires.

Participants

The participants were selected students from Computer Studies Division. The

participants completed the test on November 8, 2014 at ITR1 class room. Among the

selected participants, each has an android phone.

Table 1: The table below shows the registration information needed from each users to

login and use the application.

Username Input user’s desired username

Password

Email Address

Session ID and Token

API key

Input user’s desired password

Input user’s valid Email Address

Auto-generated by the server upon

successive registration

Given by the server upon successive

registration

4

Evaluation Tasks/Scenarios

The selected students attempt to complete the eight tasks given by the researchers

listed below:

fill up the registration fields to complete the registration;

after successive registration, Login using the registered accounts;

the application now directs users to the main interface of the application to

provide selection of contacts;

after selecting the desired contact, the application directs users to the feature

options the application provides;

the application offers features namely video call and video chat;

after selecting the desired option, the application will now connect to the other

pair of user to perform the specific feature selected;

wait for the applications response; and

after the application responded, the two pair of users are now connected and ready

to perform the specific feature.

5

Results

Task Completion Success Rate

All the selected students completed all the given tasks. Task Completion

Rates Table is given below.

Table 2: Task Completion Success Rates

Participant Task1

Task2

Task 3

Task 4

Task 5

Task6

Task7

Task8

1

2

3

4

5

6

7

8

Success

Completion Rates

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

√

√

√

√

√

√

√

√

7

100%

Task Ratings

The selected students completed each tasks given and rated the ease and difficulty

of completing the tasks given the factors listed below:

i was able to do the register and login function with ease and without

further questions;

6

as I go on with the application, I was able to understand each feature and

completed the task given; and

i was able to understand the whole function of the application.

The five-point rating scale ranged from 1 (Strongly disagree) to 5 (Strongly

agree). Agree ratings are combined with the strongly agreed (5) and agreed (4) with a

mean agreement ratings of 5.0 considering that all the participants agrees that they were

able to do the register and login functions without further questions, they were able to

understand each features the application provides and understand the whole functionality

of the application.

Ease of completing the Register and Login requirements

All the selected participants agreed that it was easy to perform the Register and

Login Functions

Understands the provided features the application offers

The selected participants agreed that the features were easy to understand and

they were able to complete the task with ease.

Understands the whole function of the Application

The selected participants agreed that they understand the whole application and

how it works.

7

Table 3: Mean Task Ratings & Agree Percentage

Task Ease of Completing

Register and Login

Understands the

Features

Understands the

whole

Functionality

Overall

1

2

3

4

5

6

7

8

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0 (100%)

5.0

5.0

5.0

5.0

5.0

5.0

5.0

5.0

*Percent Agree (%) = Agree & Strongly Agree Responses combined

Time on Task

The testing of the application and performing the tasks were recorded for each

participant. All the tasks given were completed without difficulties.

The selected participants were given task to fill up the information asked from

register and login, test the features the application offers, and judge if the application was

easy to use and whether they understand the flow of the application. Completion of the

tasks given ranged from 2 minutes minimum time to 10 minutes maximum time.

8

Table 4: Time on Task

P1 P2 P3 P4 P5 P6 P7 P8 Avg. Total

(mins)

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Task 7

Task 8

4

5

10

7

3

2

5

4

5

6

5

6

3

3

5

4

5

4

4

7

6

3

5

6

7

4

2

8

7

3

5

7

6

7

3

8

6

5

6

4

4

6

2

7

8

4

5

7

3

9

8

9

9

4

7

6

3

10

8

6

5

4

6

4

4.63

6.38

5.25

7.25

5.88

3.5

5.5

5.25

9

Summary of Data

The table below displays a summary of the test data. Low completion rates and

satisfaction ratings and high errors and time on tasks.

Table 5: Summary of Completion, Errors, Time on Task, Mean Satisfaction

Task Task Completion Errors Time on Task Satisfaction*

1

2

3

4

5

6

7

8

8

8

8

8

8

8

8

8

0

0

0

0

0

0

0

0

4

6

5

7

5

3

5

5

4.7

4.7

4.7

4.7

4.7

4.7

4.7

4.7

* Satisfaction = Mean combined rating across three post-task measures: ease of finding the information, ability to keep track of location in site, and site information prediction accuracy.

Overall Metrics

Overall Ratings

After task session completion, participants rated the application for eight

overall measures. These measures include:

10

Table 6: Post-Task Overall Questionnaire

Strongly

Disagree

Disagree Neutral Agree Strongly

Agree

Mean

Rating

Percent

Agree

Thought

application was

easy to use

0 0 0 0 8 3.9 100%

Would use

application

frequently

0 0 4 2 2 3.9 92%

Found it fast

and easy to

learn the

application and

its features

0 0 0 0 8 3.9 100%

Graphical User

Interface makes

me want to

explore more of

the application

0 1 4 1 2 3.9 85%

Application

content would

keep me using it

0 0 4 3 1 3.9 90%

Application is

well organized

0 0 2 5 1 3.9 95%

Likes and Dislikes

Upon completion of all the tasks, selected participants were asked about their

comments and provided the researchers feedback of what they like the most and what

they like the least in the application.

11

Liked Most

The participants were asked what they liked the most and give the following

replies:

i like doing the video call and voice call feature;

i liked graphical output of the video call and chat feature;

Liked Least

The participants were asked what they liked the least and give the following

replies:

it takes several seconds to display the video stream;

Conclusion

Most of the selected participants found that the application is very useful to the

school and to the work groups within the society which lessen their time contacting

others specially when it’s an emergency due to the application is available anytime as

long provided with fast and reliable internet connection to process the video feeding.

12

ATTACHMENTS

Attachment A – Background Questionnaire

Name: _____________________________________________________

School: ____________________________________________________

Age: ______

Gender: __ Male

__ Female

Status: __ Single

__ Married

__ Separated

__ Widowed

Occupation: _________________

Attachment B – Post-Task Questionnaire

Directions: In answering the questions below, use this scale (From standard form for evaluation):

5 = Strongly Agree/Always 4 = Agree 3 = Neutral/Often

2 = Disagree 1 = strongly Disagree/Never

13

For each question, please circle the number that best represents your opinion.

Table 7. Post-task scenario subjective measures

A. Post-task scenario subjective measures

The application is easy to use. 5 4 3 2 1

The application is modernize and offers friendly

user interface.5 4 3 2 1

The application is functional due to new

technologies involved namely Android.5 4 3 2 1

The application is useful to the society. 5 4 3 2 1

Table 8. Task Ratings

Task Ratings

It was easy to do the register and login

function.5 4 3 2 1

The features are easy to use and understand.

5 4 3 2 1

The application’s whole functionality is very

understandable.5 4 3 2 1

14

Attachment C

Table 9. Post-session Overall Subjective Questionnaire

Overall Application Questions:

What you liked most?

What you liked least?

Any recommendations for

improvement of the application?

Attachment D – Task Scenarios

Directions: In answering the questions below, use this scale (From standard form for evaluation):

5 = Strongly Agree/Always 4 = Agree 3 = Neutral/Often

2 = Disagree 1 = strongly Disagree/Never

15

For each question, please circle the number that best represents your opinion.

Table 10. Post-Task Overall Questionnaire

Post-Task Overall Questionnaire

1. Thought application was easy to use 5 4 3 2 1

2. Would use application frequently 5 4 3 2 13. Found it fast and easy to learn the application and

its features.5 4 3 2 1

4. Graphical User Interface makes me want to explore

more of the application5 4 3 2 1

5. Application content would keep me using it 5 4 3 2 16. Application is well organized 5 4 3 2 1