using an assessment engine for creating flexible educational games

8th European Conference on Games Based Learning An Assessment Engine Educators as Editors of their Serious Games Assessment

UWS Research Student Conference 2015

Using an Assessment Engine for Creating Flexible Educational GamesYaelle [email protected]

1

OutlineEngAGe: An Engine for Assessment in Games MethodologyPreliminary results on Usability and UsefulnessConclusion

29 April 20152/14UWS Annual Research Student ConferenceEngAGeMethodologyResultsConclusion

2

Engage An Engine for Assessment in Games:29 April 2015UWS Annual Research Student Conference

3/14EngAGeMethodologyResultsConclusion

3

Story of a serious game29 April 2015UWS Annual Research Student Conference4/14EngAGeMethodologyResultsConclusion

4

29 April 2015UWS Annual Research Student Conference

In-game Assessment and FeedbackTime consumingCommunication (Teacher - Developer)Black Box

Problems to addressTeacherLittle trust in a games assessmentNeed to feel in controlImprovisation and AdaptationLearning analyticsLittle documentationBeyond reach of most teachers


5

An Assessment Engine: EngAGe29 April 2015Domain Specific Language for configuration of the assessment

Learning Analytics dashboard

Authoring Tool for teachers

UWS Annual Research Student Conference


6

EVALUATIONMethodology and results29 April 2015UWS Annual Research Student Conference7/14

EngAGeMethodologyResultsConclusion

7

MethodologyTarget users:

Aspects to evaluate:

Generalisation UsabilityUsefulness

29 April 2015UWS Annual Research Student Conference8/14

DevelopersEducators(Players)


8

Methodology29 April 2015UWS Annual Research Student Conference9/14TutorialConfiguration file of the gameCalls to web services

Post questionnaireUsefulness (Likert scales)Usability (SUS)


9

Preliminary resultsUsefulness29 April 2015UWS Annual Research Student Conference10/14


10

Preliminary resultsUsability: System Usability Scale (SUS) by Brooke (1996)10 questions to be graded between Strongly Agree and Strongly disagree.

29 April 2015UWS Annual Research Student Conference11/14pilot study tutorial study overallI think I would like to use this system in the future when developing serious games71.410085.7I found the system unnecessarily complex28.6014.3I thought the system was easy to use14.371.442.9I think that I would need support to be able to use this system57.1028.6I found the various functions in this system were well integrated28.671.450I thought there was too much inconsistency in this system000I would imagine that most serious game developers would learn to use this system very quickly42.971.457.1I found the system very cumbersome to use28.6014.3I felt very confident using the system14.385.750I needed to learn a lot of things before I could get going with this system57.1028.6

Percentage of participants agreeing or strongly agreeing with the statementsEngAGeMethodologyResultsConclusion

11

Preliminary resultsPilot study (n=7) OKMean = 55Median = 50

Tutorial study (n=7) GoodMean = 73.5Median = 72.5

Total score (n=14) GoodMean : 64.2Median: 7029 April 2015UWS Annual Research Student Conference12/14

Bangor, A., Kortum, P. T., and Miller, J. T. (2008) An empirical evaluation of the system usability scale. Intl. Journal of HumanComputer Interaction. Vol.24(6), pp.574-594.EngAGeMethodologyResultsConclusion

12

ConclusionsEngAGe for developers:Guidelines for assessment in gamesEfficient and Effective toolFairly easy to use

EngAGe for educators:No more Black Box issueAccess to gameplay data (Learning Analytics)

Future workMore participants for tutorial Include EngAGe in various on-going projectsExperiment with educators

29 April 2015UWS Annual Research Student Conference13/14EngAGeMethodologyResultsConclusion

13

Any Question?Thank you!29 April 2015UWS Annual Research Student Conference14

Interested?Email me: [email protected]

14