A case-study in compliance training and education for UK Export ControlsVenkat SastryTechnology Enhanced Learning Laboratory (TELL)v.v.s.s.sastry@cranfield.ac.uk

Word cloud

Word cloud - focussed

Word cloud – structured

Background What was done? Data Analysis Results and Conclusions

Word cloud – structured

Background What was done? Data Analysis Results and Conclusions

Strategic Export Controls

focus

through to4

onlinesupported by and

Initial pilot

and evaluation of current assessment

Various techniques used including

ing

analysis and principle

analysis

and

Questions with clear deficiencies or anomalies identified indicating issues with question item

What was done?

• Why was the pilot question item analysis undertaken?

• Description of the data analysed- 115 question items- 9740 question responses- 974 users

• Expected results – what did we hope to find

Data analysis

• Exploratory data analysis consisted of:- 4 attributes observed:

1) Exposure count2) % of correct answers3) ratio of question stem length to distractor length4) presence of a negative term

- Principle Component Analysis- Clustering Analysis

EDA = direction

Results – Observation of attributes

Question IDs69, 71, 78, 86,176, 84, 81,105,106, 96, 104

100%

50%/50%

Majority incorrect

Results – Clustering Analysis

Perfect cluster shape

Items that may not belong in the cluster

Cluster 1 – Low exposure counts

Cluster 2 – High exposure counts. High % correct

Cluster 3 – High ratio for question stem to distractor

Cluster 5 – Negative present

Cluster 6 – % correct is 100%

Cluster 4 – High ratio for q stem to distractor. High % correct

Question item

Results – Chernoff faces

KeyEach colour represents a cluster.

Attributes mappingExposure count = size% correct = elongationRatio = forehead shapeNegative = jaw shape

Conclusions

• Exploratory data analysis has acted as a guide and highlighted both healthy and weak question items

• Identifies weaker question items

• Gives direction for improvement

• Item design and consistency issues identified (+ve and –ve)

• Results broadly inline with expectations

Recommendations

• Development of visual dashboard based on the data analysis algorithms

• Validation rules should be applied to the question item authoring tool

• Aligning of learning objectives with question items at topic level to improve validity and reliability

• Adoption of a systems approach e.g. extended ADDIE process model

• Identify roles and activities required to manage current Level 1 design

Questions?

QUESTIONS?

Contact details

Peter Jolliffe - p.m.jolliffe@cranfield.ac.uk

Piers MacLean - p.j.maclean@cranfield.ac.uk

Venkat Sastry - v.v.s.s.sastry@cranfield.ac.uk

SEC project website:http://www.strategicexportcontrols.org