Usability evaluation in a multiphase, exploratory design-based research study of

an online community for the practice of special education in Bulgaria

Rob Peterson, PhDUniversity of Wollongong

Faculty of Educationrobpeterson@gmx.comwww.robertrpeterson.com

Ian Olney, EdDUniversity of Western Sydney Teaching Development Uniti.olney@uws.edu.au

June 2009, ED-MEDIA, Hawaii

June 2009ED-MEDIA

Hawaii

2

OverviewBackground of studyWebsite development cycleTypes of usability evaluationsParticipant selectionUsability tasks and scenariosData collectionEquipmentData analysisResults

June 2009ED-MEDIA

Hawaii

3

Background

Establishment of the Special Education Bulgaria online community 3-year PhD project Design-based research Exploratory research design Design principles for virtual communities of

practice Moodle and Mahara www.SpecialEducationBulgaria.com

June 2009ED-MEDIA

Hawaii

4

Data Sources

Research Participants…………………..76 Personal interviews 19 Usability interviews...……………………….18 Expert consultations 4 Web-based questionnaires 18 Email-based questionnaires 17

Website Participants…………………….242 Registered website users Discussion forum and other posts

Website Usage Logs

Website

Development

Cycle by

Research

Phase

Аdapted from “Product

development life cycle”

(Rubin,1994)

1. User andusage needsanalysis

2. Specificationof requirements

3. Preliminarydesign

4. Detaileddesign

5. Product build

6. Productrelease

Phase 1: Needs assessmentExploratory usability evaluations

Phase 2: Formative evaluationAssessment usability evaluations

Phase 3: Effectiveness evaluationValidation usability evaluations

Prototype

Alpha

Beta

Final

Website Versions

Prototype

Alpha Beta/Final

June 2009ED-MEDIA

Hawaii

7

Usability Evaluations

Inquiries, inspections, and tests

Exploratory, assessment, and validation usability evaluations

Phase 1, Phase 2, Phase 3

Qualitative Quantitative

June 2009ED-MEDIA

Hawaii

8

Participant Selection

Purposeful-quota sampling

Purposeful-intensity sampling

June 2009ED-MEDIA

Hawaii

9

Tasks and ScenariosTask: Find a definition for the word dyslexia.

Task: Send a message to the website administrator.

Scenario 1: Registration

Scenario 2: Chat

Scenario 3: Discussion forums

Scenario 4: Lost password and username

Scenario 5: Upload

Post-Test Questions

June 2009ED-MEDIA

Hawaii

10

Data Collection

Observation Sheet

Design Element Importance (A-C)Repairability (1-3)

Problem /Recommendation

Setup and Equipment for Usability Evaluation at Remote Location

Problem/Solution Matrix forAnalysis of Usability Data

# yyyy-mm-dd hh:mm:ss Participant

A g e

Se x

E x p

Version Task (1-22) Scene (A-E)

Design Element

1 2005-09-15

00:50:38 Name 40+ F H Proto 6 Navigation bar

2 2005-09-15

00:48:23 Name 23-30 F M Proto 9 Registration

3 2006-05-15

00:10:17 Name 17-22 M L Alpha C Forums

4 2006-05-15

00:22:14 Name 40+ M H Alpha E Internet resources

Imp. (A-C) Rep. (1-3)

Problem / Recommendation

Solution See Also

Re test

Fixed / or X

B/3 - Problem description - Recommendation

- Solution enacted 2 Y /

A/2 - Problem description - Recommendation

- Solution enacted 1 Y X

C/2 - Problem description - Recommendation

A/1 - Problem description - Recommendation

- Solution enacted N X

N

Usability Findings by Importance (A-C) and Repairability (1-3)

0

5

10

15

20

25

30

A1 B1 C1 A2 B2 C2 A3 B3 C3

Total number offindings: 142

Number offindingsrepaired: 108

June 2009ED-MEDIA

Hawaii

14

Common Findings (Repaired)

Superfluous or confusing data required in registration process.

Confusion regarding the type of characters to use, Cyrillic or Latin.

Cannot see other participant’s full profile information such as email address or city.

June 2009ED-MEDIA

Hawaii

15

Common Findings (Not Repaired)

Cyrillic characters not supported by Moodle, e.g., Cyrillic characters do not alphabetize correctly in lists.

Only 50% of the RSS feed’s Bulgarian news items returned actually had to do with education and Bulgaria.

June 2009ED-MEDIA

Hawaii

16

Key Points

Special Education Bulgaria project

Exploratory and design-based research

Usability inquires versus tests

Usability evaluations at remote locations

Data analysis in terms of importance and repairability

Challenges and solutions … see the paper

Rob Peterson, PhDUniversity of Wollongong Faculty of Educationwww.robertrpeterson.com

A/Prof Krassen StefanovSofia University, Faculty of Mathematics and Informatics

A/Prof Mira Tzvetkova-ArsovaSofia University, Faculty of Primary and Preschool Education

A/Prof Deslea KonzaEdith Cowan UniversityFaculty Education and Arts

Prof Jan HerringtonMurdoch UniversitySchool of Education

17

Ian Olney, EdDUniversity of Western Sydney Teaching Development Uniti.olney@uws.edu.auSupervisors

Authors

Ian Olney, PhDUniversity of Western Sydney Teaching Development Unit

Building the European Network for Lifelong Competence Development

June 2009ED-MEDIA

Hawaii

18

Key ReferencesBarnum, C. M. (2002). Usability testing and research. New York, NY: Longman Publishers.

Barnum, C. M., Mukherjee, A., Mielke, N., Porter, M., Schumann, M., Speaks, B., et al. (1999). Hotmail usability test plan. Retrieved January 12, 2006, from http://www.ablongman.com/barnum

Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications.

Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66). London: Routledge.

Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York, NY: John Wiley & Sons, Inc.

Rubin, J., Chisnell, D., & Spool, J. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing, Inc.

Addendum

MultiphaseDesignBased Research

Adapted from “Predictive and design research approaches in educationaltechnology research” (Reeves, 2006)

Stage 1Iterative cycle of testing and refinement of solutions in practice

Stage 2 Development of solutions informed by existing design principles and technologicalinnovations

Phase 2 Formative Evaluation

Stage 1Iterative cycle of testing and refinement of solutions in practice

Stage 2 Development of solutions informed by existing design principles and technologicalinnovations

Stage 3 Reflection to produce “design principles” and enhance solution implementation

Phase 3 Effectiveness

Evaluation

Future impactand maintenance evaluations

Stage 1 Analysis of practical problems by researchers in collaboration

Stage 2 Development of solutions informed by existing design principles and technological innovations

Phase 1 Needs

Assessment

Stage 3 Reflection to produce “design principles” and enhance solution implementation

Stage 3 Reflection to produce “design principles” and enhance solution implementation

Stage 1Analysis of practical problems by researchers in collaboration

Stage 2Development of solutions informed by existing design principles and technological innovations

Stage 3Iterative cycles of testing and refinement of solutions in practice

Stage 4Reflection to produce “design principles” and enhance solution implementation