This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
MoLE Testing & Evaluation Results
1
11 – 12 September 2012Testing & Evaluation Team
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
PRIMARY OBJECTIVES
The Testing & Evaluation Working Group goals include:
• Conduct an operational evaluation of mobile learning utility, usability and suitability in a multinational environment
• Ensure the guidelines regarding the Human Research Protection Program are implemented
• Develop operational measures of effectiveness and performance
• Develop data collection and analysis plans to ensure the data is collected sufficiently to fully evaluate the project’s technical components
• Make recommendations on how the project’s technical lessons learned may be incorporated into future multinational mobile learning initiatives
2
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
FY11 ACTIVITIES
• Human Research Protection Program• Established a small group to focus on US, EU and International Research
Ethics Requirements • Developed an initial draft of the MoLE Proof of Concept (PoC) Informed
Consent based on post-MoLE Kick-Off Workshop discussion
• Questionnaire/Survey Development• Drafted Terms of Reference (TOR) for utility, usability, suitability and self-
efficacy• Sent an email to each international delegate seeking input regarding
questions to ask for each TOR
• Testing & Evaluation Workshop (September 2011)• Conducted a final review/edit on the MoLE Proof of Concept (PoC)
Informed Consent• Formed two subgroups (i.e., Questionnaire Development & Data Analysis)
3
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
FY12 ACTIVITIES
• Human Research Protection Program• Submitted the final MoLE PoC Informed Consent to international
delegates for review/comment• MoLE PoC Informed Consent translated into French and Spanish
• Questionnaire/Data Analysis• Finalized the questions based on their TOR• Finalized the data analysis approach
• Testing & Evaluation Workshop (December 2011)• Conducted a final review/edit on the MoLE PoC Questions • Outlined the process for integrating the questions into the MoLE MedAid
App• Finalized the Demographic Questions for the PoC
4
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
PROOF OF CONCEPT APPROACH
5
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
TESTING & EVALUATION PROCESS
6
Email announcement
Online registration App install Activate Try it out Evaluation
Your personal pin number is 00000
Your unique start page: http://mole.tribalhosting.net/register/00000/35400ee9d41f252e3e2d7e3a41708847aace01475375ee272f56363a1e565da2/8abf02dbe063858fc1dd7adb20277addd28cf375f6a2ae45e4fb4871a2ecf16b
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
WELCOME EMAIL
7
• A “Welcome Email” was initially sent to
approximately 600 volunteer on the first
day of the MoLE Proof of Concept
• The announcement email provided
additional information on installation, the
user guide and an introduction video.
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
REMINDER EMAIL
8
• A “Welcome Reminder Email”
was initially bi-weekly to remind
the volunteers.
• The reminder email also
provided additional information
on installation, the user guide
and an introduction video.
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
INFORMED CONSENT
9
• The Informed Consent was provided in three language, which requires the consent process to be conducted in their native language if they have English proficiency concerns;
• Before participating in the MoLE Proof of Concept, each volunteer acknowledged they understood the informed consent in its entirety and understood their rights.
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
DEMOGRAPHICS
10
Question Responses
Age less than 20, 20-29, 30-39, 40-49, 50+, no answer
Gender Male, Female
How proficient are you in English Beginner . . . Advanced
Are you using your own personal smartphone for the purpose of this trial?
Yes, No, No Answer
How comfortable are you with using the mobile device that's running the MoLE app? [Beginner to Advanced]
Beginner . . . Advanced
Have you previously been involved in humanitarian assistance or disaster relief operations?
Yes, No, No Answer
What is your professional expertise? Medical, Rescue, Training, E-learning, Other
Have you taken the Trafficking in Persons (CTIP ) course within the last two years?
Yes, No, No Answer
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
INSTITUTIONAL REVIEW BOARD (IRB)
11
• An independent review was conducted at the
request of the CNE-CNA-C6F Medical Officer
• Review confirmed MoLE Research Protocol
complied with US/EU/International Research Ethics
Requirements
• Email to SPAWAR SSC Pacific confirmed SPAWAR
SSC Pacific (DoN Research Protocol Office)
confirmed Independent IRB Findings – except for
Tribal’s Usability Study
• SPAWAR SSC Pacific stated they would provide a
letter; however, subsequent follow-up requires
Letters of Support from key US organizations that
identified US Government/Contractors as
volunteers.
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
PROOF OF CONCEPT DATA ANALYSIS
12
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
PARTICIPANT TOTALS
13
• Launched: 268• Started Proof of Concept: 177 (66.4%)• Completed Proof of Concept: 137 (51.12%)
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
AGE GROUP
14
Age Group Breakdown< 20 years 5 1.87%20+ years 51 19.03%30+ years 83 30.97%40+ years 75 27.99%50+ years 52 19.40%No Response 2 0.75%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
GENDER
15
Gender BreakdownMale 181 67.54%Female 87 32.46%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
ENGLISH PROFICIENCY
16
English Proficiency LevelsLevel Response Percentage
Level 1 81 30.22%Level 2 11 4.10%Level 3 10 3.73%Level 4 38 14.18%Level 5 35 13.06%Level 6 38 14.18%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
English Proficiency Levels Defined
Level 1—Beginning/Preproduction [WIDA level = Entering]:• Does not understand or speak English with the exception of a few isolated words or expressions.
Level 2—Beginning/Production [WIDA level = Beginning]:• Speaks conversational and academic English with hesitancy and difficulty• Understands parts of lessons and simple directions• At a pre-emergent or emergent level of reading and writing in English, significantly below grade level
Level 3—Intermediate [WIDA level = Developing]:• Understands and speaks conversational and academic English with decreasing hesitancy and
difficulty• Is post-emergent, developing reading comprehension and writing skills in English.• English literacy skills allow the student to demonstrate academic knowledge in content areas with
assistance
17
The WIDA Consortium (World-Class Instructional Design and Assessment) is a not-for-profit educational consortium of state departments of education. WIDA designs and implements proficiency standards and
equitable educational opportunities for grade K-12 students who are English language learners.
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
English Proficiency Levels Defined
Level 4—Advanced Intermediate [WIDA level = Expanding]:• Understands and speaks conversational English without apparent difficulty, but understands and
speaks academic English with some hesitancy• The pupil continues to acquire reading and writing skills in content areas needed to achieve grade
level expectations with assistance.
Level 5—Advanced [WIDA level = Bridging]:• Understands and speaks conversational and academic English well• Near proficient in reading, writing, and content area skills needed to meet grade level expectations• The pupil requires occasional support
Level 6—Formerly Limited-English Proficient/Now Fully-English Proficient:• The pupil was formerly limited-English proficient and is now fully English Proficient• Reads, writes, speaks and comprehends English within academic classroom settings
Level 7—Fully-English Proficient/Never Limited-English Proficient:• Never classified as limited-English proficient and does not fit the definition of a limited-English
proficient student outlined in either state of federal law
18
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
ENGLISH PROFICIENCY (LEVEL 1)
19
Unusually high numbers of Level-1 Proficiency for English speaking countries (US/UK) can be attributed some users not answering the question (within their right under the research ethics
guidelines) and accepting the default value of ‘1'
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
“BRING YOUR OWN DEVICE (BYOD)”
20
Bring Your Own Device (BYOD) Analysis
No 78 29.1%Yes 190 70.9%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
MOBILE DEVICE PROFICIENCY
21
Mobile Device Proficiency AnalysisLevel Response Percentage
Level 1 81 30.22%Level 2 11 4.10%Level 3 10 3.73%Level 4 38 14.18%Level 5 35 13.06%Level 6 38 14.18%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
MOBILE DEVICE PROFICIENCY (LEVEL 1)
22
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
HUMANITARIAN ASSISTANCE PROFICIENCY
23
Humanitarian AssistanceResponses Number PercentageNo 77 28.73%Yes 191 71.27%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
PROFESSIONAL EXPERTISE
24
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
TRAFFICKING IN PERSONS (CTIP) COURSE
25
CTIP Course AnalysisYes 45 16.79%No 223 83.21%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
DEVICE TOTALS
26
Mobile Device Analysis
Phone Type Number Percentage
iPhone 4 23 8.58%
iPhone 5 146 54.48%
android 2 32 11.94%
android 2.3 61 22.76%
android 3 or 4 6 2.24%
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
TESTING & EVALUATION APPROACH
27
EVALUATION QUESTIONS Category1.1 Rate your confidence in using the checklist. Self-efficacy2.1 How useful are mobile devices for this type of training? Utility2.2 How easy was the CTIP content to navigate? Accessibility3.1 How useful is access to digital reference materials on a mobile device? Utility4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy
5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness6.1 How helpful are collaboratively updated contact details? Self-efficacy7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility
7.5 How useful are collaborative contact pages to access real time information? Utility7.6 How easy was the Global MedAid app to navigate? Accessibility
7.7 Please write up to five single words which best describe your overall experience of using the mobile device as a tool for learning.
7.8 What did you like most about the mobile device as a tool for learning?7.9 What did you like least about the mobile device as a tool for learning?8 Any additional comments
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
OVERALL MEDAID APP RATING
28
Likert Rating Question All-Users Android iPhoneN/A or Skipped 8 3 5
1 Not useful at all 9 1 82 Not useful 9 4 53 Not very useful 20 14 64 Neither useful nor not useful 18 6 125 Quite useful 33 15 186 Useful 60 23 377 Very useful 22 5 17
n: 179Minimum: 0Maximum: 7Mean: 4.7 +/-0.3Median 5.4Mode: 6Standard Deviation: 1.9Variance: 3.5
0102030405060
0
1
2
3
4
5
6
7
All Users Responses
05
10152025
0
1
2
3
4
5
6
7
Android Users Responses
010203040
0
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
QUESTIONS & LIKERT SCALE
29
EVALUATION QUESTIONS Category1.1 Rate your confidence in using the checklist. Self-efficacy2.1 How useful are mobile devices for this type of training? Utility2.2 How easy was the CTIP content to navigate? Accessibility3.1 How useful is access to digital reference materials on a mobile device? Utility4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness6.1 How helpful are collaboratively updated contact details? Self-efficacy7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility7.5 How useful are collaborative contact pages to access real time information? Utility7.6 How easy was the Global MedAid app to navigate? Accessibility
Rating 1 Rating 2 Rating 3 Rating 4 Rating 5 Rating 6 Rating 7Not confident at all Not confident Not very confident
Neither confident nor unconfident Quite confident Confident
Very confident
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
1.1: RATE YOUR CONFIDENCE IN USING THE CHECKLIST
30
Likert Rating Question All-Users Android iPhoneN/A or Skipped 18 5 13
1 Not useful at all 2 0 22 Not useful 4 2 23 Not very useful 6 2 44 Neither useful nor not useful 9 2 75 Quite useful 42 17 256 Useful 47 26 217 Very useful 51 17 34
n: 179Minimum: 0Maximum: 7Mean: 5.1 +/-0.3Median 6.0Mode: 7Standard Deviation: 2.1Variance: 4.5
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
2.1: HOW USEFUL ARE MOBILE DEVICES FOR TRAINING?
31
Likert Rating Question All-Users Android iPhoneN/A or Skipped 25 9 16
1 Not useful at all 1 0 12 Not useful 2 0 23 Not very useful 3 0 34 Neither useful nor not useful 9 2 75 Quite useful 35 19 166 Useful 45 17 287 Very useful 59 24 35
n: 179Minimum: 1Maximum: 7Mean: 5.1 +/-0.3Median 6.0Mode: 7Standard Deviation: 2.3Variance: 5.4
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
2.2: HOW EASY WAS THE CTIP CONTENT TO NAVIGATE?
32
Likert Rating Question All-Users Android iPhoneN/A or Skipped 24 9 15
1 Not useful at all 0 0 02 Not useful 5 1 43 Not very useful 9 1 84 Neither useful nor not useful 17 5 125 Quite useful 27 15 126 Useful 54 30 247 Very useful 43 10 33
n: 179Minimum: 2Maximum: 7.0Mean: 4.8 +/-0.3Median 6.0Mode: 6Standard Deviation: 2.3Variance: 5.2
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
3.1: HOW USEFUL IS ACCESS TO DIGITAL REFERENCE MATERIALS ON A MOBILE DEVICE?
33
Likert Rating Question All-Users Android iPhoneN/A or Skipped 23 10 13
1 Not useful at all 2 0 22 Not useful 2 0 23 Not very useful 8 4 44 Neither useful nor not useful 14 7 75 Quite useful 34 9 256 Useful 46 25 217 Very useful 50 16 34
n: 179Minimum: 1Maximum: 7Mean: 5.0 +/-0.3Median 6.0Mode: 7Standard Deviation: 2.3Variance: 5.2
01020304050
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
4.1: HOW EASILY COULD YOU EXPLAIN TO A COLLEAGUE HOW TO LIIK UP NGO LOGOS?
34
Likert Rating Question All-Users Android iPhoneN/A or Skipped 30 10 20
1 Not useful at all 1 1 02 Not useful 5 0 53 Not very useful 16 6 104 Neither useful nor not useful 14 4 105 Quite useful 29 13 166 Useful 46 27 197 Very useful 38 10 28
n: 179Minimum: 1Maximum: 7Mean: 4.5 +/-0.4Median 5.0Mode: 6Standard Deviation: 2.4Variance: 5.8
01020304050
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
5.1: HOW USEFUL ARE VIDEOS AS A TOOL TO ENHANCE YOUR UNDERSTANDING OF USAID?
35
Likert Rating Question All-Users Android iPhoneN/A or Skipped 33 12 21
1 Not useful at all 0 0 02 Not useful 2 0 23 Not very useful 7 3 44 Neither useful nor not useful 11 3 85 Quite useful 28 15 136 Useful 45 17 287 Very useful 53 21 32
n: 179Minimum: 2Maximum: 7Mean: 4.7 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.5Variance: 6.3
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
6.1 HOW HELPFUL ARE COLLABORATIVELY UPDATED CONTACT DETAILS?
36
Likert Rating Question All-Users Android iPhoneN/A or Skipped 30 11 19
1 Not useful at all 0 0 02 Not useful 2 0 23 Not very useful 6 1 54 Neither useful nor not useful 12 4 85 Quite useful 20 10 106 Useful 53 27 267 Very useful 56 18 38
n: 179Minimum: 2Maximum: 7Mean: 4.9 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.5Variance: 6.1
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.1 HOW USEFUL ARE MOBILE DEVICES FOR TRAINING?
37
Likert Rating Question All-Users Android iPhoneN/A or Skipped 43 20 23
1 Not useful at all 0 0 02 Not useful 0 0 03 Not very useful 7 1 64 Neither useful nor not useful 11 5 65 Quite useful 22 15 76 Useful 54 21 337 Very useful 42 9 33
n: 179Minimum: 3Maximum: 7Mean: 4.4 +/-0.4Median 6.0Mode: 6Standard Deviation: 2.7Variance: 7.2
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.2 HOW USEFUL ARE MOBILE DEVICES FOR REFRESHER TRAINING?
38
Likert Rating Question All-Users Android iPhoneN/A or Skipped 41 18 23
1 Not useful at all 0 0 02 Not useful 0 0 03 Not very useful 4 2 24 Neither useful nor not useful 8 2 65 Quite useful 15 7 86 Useful 47 21 267 Very useful 64 21 43
n: 179Minimum: 3Maximum: 7Mean: 4.7 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.7Variance: 7.5
020406080
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
01020304050
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.3 HOW USEFUL ARE MOBILE VIDEOS TO ENHANCE UNDERSTANDING?
39
Likert Rating Question All-Users Android iPhoneN/A or Skipped 42 19 23
1 Not useful at all 0 0 02 Not useful 0 0 03 Not very useful 5 3 24 Neither useful nor not useful 16 7 95 Quite useful 27 15 126 Useful 34 11 237 Very useful 55 16 39
n: 179Minimum: 3Maximum: 7Mean: 4.5 +/-0.4Median 5.0Mode: 7Standard Deviation: 2.7Variance: 7.3
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
101520
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.4 HOW USEFUL ARE DIGITAL REFERENCE MATERIALS ON MOBILE DEVICES?
40
Likert Rating Question All-Users Android iPhoneN/A or Skipped 43 20 23
1 Not useful at all 0 0 02 Not useful 0 0 03 Not very useful 4 3 14 Neither useful nor not useful 10 3 75 Quite useful 22 9 136 Useful 46 24 227 Very useful 54 12 42
n: 179Minimum: 3Maximum: 7Mean: 4.6 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.7Variance: 7.5
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
01020304050
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.5 HOW USEFUL ARE COLLABORATIVE CONTACT PAGES TO ACCESS REAL-TIME INFORMATION?
41
Likert Rating Question All-Users Android iPhoneN/A or Skipped 44 21 23
1 Not useful at all 1 0 12 Not useful 2 1 13 Not very useful 4 2 24 Neither useful nor not useful 13 5 85 Quite useful 25 11 146 Useful 35 15 207 Very useful 55 16 39
n: 179Minimum: 1Maximum: 7Mean: 4.4 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.8Variance: 7.6
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.6 HOW EASY WAS THE GLOBAL MEDAID APP TO NAVIGATE?
42
Likert Rating Question All-Users Android iPhoneN/A or Skipped 46 22 24
1 Not useful at all 1 0 12 Not useful 3 2 13 Not very useful 12 3 94 Neither useful nor not useful 22 11 115 Quite useful 21 7 146 Useful 44 23 217 Very useful 30 3 27
n: 179Minimum: 0Maximum: 7Mean: 4.0 +/-0.4Median 5.0Mode: N/A or
SkippedStandard Deviation: 2.6Variance: 6.9
01020304050
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
05
1015202530
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
CATEGORY ANALYSIS
43
EVALUATION QUESTIONS Category2.2 How easy was the CTIP content to navigate? Accessibility7.6 How easy was the Global MedAid app to navigate? Accessibility1.1 Rate your confidence in using the checklist. Self-efficacy
4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy
6.1 How helpful are collaboratively updated contact details? Self-efficacy
5.1 How useful are videos as a tool to enhance your understanding of USAID? Usefulness
2.1 How useful are mobile devices for this type of training? Utility
3.1 How useful is access to digital reference materials on a mobile device? Utility
7.1 How useful are mobile devices for training? Utility7.2 How useful are mobile devices for refresher training? Utility7.3 How useful are mobile videos to enhance understanding? Utility7.4 How useful are digital reference materials on mobile devices? Utility
7.5 How useful are collaborative contact pages to access real time information? Utility
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
ACCESSIBILITY
44
Likert Rating Question All-Users Android iPhoneN/A or Skipped 14 4 10
1 Not useful at all 2 1 12 Not useful 11 4 73 Not very useful 27 16 114 Neither useful nor not useful 23 6 175 Quite useful 26 14 126 Useful 38 20 187 Very useful 38 6 32
n: 179Minimum: 0Maximum: 7Mean: 4.4 +/-0.3Median 5.0Mode: 6, 7Standard Deviation: 2.0Variance: 4.2
010203040
0
1
2
3
4
5
6
7
All Users Responses
05
101520
0
1
2
3
4
5
6
7
Android Users Responses
010203040
0
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
SELF-EFFICACY
45
Likert Rating Question All-Users Android iPhoneN/A or Skipped 5 1 4
1 Not useful at all 5 2 32 Not useful 22 8 143 Not very useful 6 3 34 Neither useful nor not useful 18 3 155 Quite useful 37 16 216 Useful 58 29 297 Very useful 28 9 19
n: 179Minimum: 0Maximum: 7Mean: 4.8 +/-0.3Median 5.3Mode: 6Standard Deviation: 1.8Variance: 3.4
0102030405060
0
1
2
3
4
5
6
7
All Users Responses
05
1015202530
0
1
2
3
4
5
6
7
Android Users Responses
05
1015202530
0
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
USEFULNESS
46
Likert Rating Question All-Users Android iPhoneN/A or Skipped 33 12 21
1 Not useful at all 0 0 02 Not useful 2 0 23 Not very useful 7 3 44 Neither useful nor not useful 11 3 85 Quite useful 28 15 136 Useful 45 17 287 Very useful 53 21 32
n: 179Minimum: 2Maximum: 7Mean: 4.7 +/-0.4Median 6.0Mode: 7Standard Deviation: 2.5Variance: 6.3
0102030405060
Skipped orN/A
1
2
3
4
5
6
7
All Users Responses
010203040
Skipped orN/A
1
2
3
4
5
6
7
iPhone Users Responses
05
10152025
Skipped orN/A
1
2
3
4
5
6
7
Android Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
UTILITY
47
Likert Rating Question All-Users Android iPhoneN/A or Skipped 18 5 13
1 Not useful at all 2 0 22 Not useful 4 2 23 Not very useful 6 2 44 Neither useful nor not useful 9 2 75 Quite useful 42 17 256 Useful 47 26 217 Very useful 51 17 34
n: 179Minimum: 0Maximum: 7Mean: 4.7 +/-0.3Median 5.6Mode: 7Standard Deviation: 2.2Variance: 4.6
0102030405060
0
1
2
3
4
5
6
7
All Users Responses
05
1015202530
0
1
2
3
4
5
6
7
Android Users Responses
010203040
0
1
2
3
4
5
6
7
iPhone Users Responses
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.7 WRITE FIVE SINGLE WORDS THAT DESCRIBE YOUR OVERALL EXPERIENCE?
48
9971
51 4329 23
2
31.1%
53.5%
69.5%
83.0%92.1%
99.4%
0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%100.0%
0
50
100
150
200
250
300
Num
ber o
f Res
pons
es
Categories
Descriptive Terms
This chart is based on the frequency of the words using synonym analysis
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.7 WRITE FIVE SINGLE WORDS THAT DESCRIBE YOUR OVERALL EXPERIENCE?
49
This chart is based on pairwise high frequency analysis
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.8 WHAT DID YOU LIKE MOST ABOUT THE MOBILE DEVICE FOR LEARNING?
50
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.8 WHAT DID YOU LIKE MOST ABOUT THE MOBILE DEVICE FOR LEARNING?
51
This chart is based on pairwise high frequency analysis
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.9 WHAT DID YOU LIKE LEAST ABOUT THE MOBILE DEVICE FOR TRAINING?.
52
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
7.9 WHAT DID YOU LIKE LEAST ABOUT THE MOBILE DEVICE FOR TRAINING?.
53
This chart is based on pairwise high frequency analysis
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
8. ADDITIONAL COMMENTS
54
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
8. ADDITIONAL COMMENTS
55
This chart is based on pairwise high frequency analysis
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
USER-FEEDBACK
• A very good and innovating idea
• I love how you enable on/offline and the graphs are really nice. I also like the in-app evaluation layer. An interesting combination of action and feedback
• Great to access all this resources in a mobile device
• Quick access to byte sized and chunked information for just in time learning
• This was interesting. I learned a lot about emergency medical care and some things about NGOs
• Overall I thought the information in the tool was very well chunked. I didn't have to dive too deep into anything or scroll too much
• The training modules were very well put together• I tried several ways to take the survey but most of the selections were greyed out in
which only the “prefer not to say” was answered• The mission packs did not seem to work. I can open it and it gives me a list but nothing
opens• Mission tools should be tailored for different regions vs. a generic list; especially, in light
of the immunizations section56
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
USER-FEEDBACK
• Information is at my fingertips and a tool that is always there when I need it
• I envision the mobile device will be a big part of education in the very near future
• I think the application and the contents are very useful for the purposes that have been developed
• I am a big believer in the potential for mobile devices for military training, keep up the good work
• I just want to say that I am impressed! The videos are clear, as are some of the medical procedures!
57
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
USER-SUGGESTIONS
• Mission tools should be tailored for different regions vs. a generic list; especially, in light of the immunizations section
• Some users had difficulties submitting evaluation forms
• Request for more information in app
• More off-line video (vs faster install time)
• Use GPS to better target content
• App still needs a little work; great concept
• Would like more interactivity
• A combination of smart phone and larger screen apps might be ideal with real learning on the larger screen and look-up and refresher learning on the smartphone
58
This work relates to the US Department of Defence (DoD) Contract # N68171-12-P-9000 which was awarded as part of the Coalition Warfare Program (CWP) Project entitled Mobile Learning Environment (MoLE).
CONCLUSIONS
59
• A suitable representation of the population (e.g., professional expertise, countries, etc)
• “BYOD” did not present any obstacles in the evaluation
• Post-MoLE Questionnaire confirms international partners intend on developing mobile capabilities
• Data worksheet will be cleansed to ensure no identification among volunteers and nations
• Technology works – where is the data to show effectiveness?