streamlined orchestration: investigating the impact of organised orchestration on teaching
TRANSCRIPT
Streamlined OrchestrationInvestigating the Impact of Organised Orchestration on Teaching
Lighton PhiriSupervisors:A/Prof. Hussein SulemanProf. Dr. Christoph Meinel
March 15, 2016
Introduction
■ Exploring technology-driven orchestration◻ Understand whether and/or how organisation of learning activities
influences educators’ effectiveness■ Thesis goal
◻ Investigate the impact of organised orchestration on teaching◻ Investigate successful use of an orchestration workbench
2
Motivation
Supporting educators considered one the of most effective ways of improving the quality of education
However, contemporary orchestration is:1) Challenging
◻ Heterogenous activities—instructional and non-instructional◻ Timing constraints—typical lecture session lasts ~45 minutes
2) Ad hoc◻ Specialised tools used for managing activities◻ Generic tools appropriated to achieve desired goal
3
Thesis statement
Streamlined orchestration—attainable through explicit organisation of enactment activities using an orchestration workbench—could potentially make educators more effective.
Research questions:1) Does an orchestration workbench enable educators to
become more effective?2) What is the impact of an orchestration workbench on
educators’ teaching experience?
4
Contributions
5
Organised orchestration: streamlined technology-driven orchestration.
■ Prototype UIs and workflows◻ Workbench UIs and workflows for attaining organised
orchestration■ Case study results
◻ Case studies demonstrating approach in authentic educational settings
■ Experimental studies◻ Analysis and results of experiments conducted to demonstrate
the effect on organised orchestration on teaching
Evaluation plan (1)
‘5+3’ Conceptual Orchestration Framework
■ Management◻ Coordination of learning
activities◻ Regulation of activities to
guide learning activities■ Flexibility
◻ Flexibility of tool■ Assessment
◻ Learning experience
6
Evaluation plan (2)
7
# Evaluation Technique
1. Case studies
2. Experiment in the large
3. Targeted group
4. Survey of existing tools
5. Captive audience
6. General random trial
7. Survey of teaching staff
8. UCT Lecture recording analysis
Study 1: Expert reviews
■ Expert review sessions held with potential EdTech early adopters◻ Individuals integrating technology
in the classroom◻ Understand potential study
environment; recruit participants■ Eight teaching staff interviewed■ Main outcomes
◻ Scale of technology integration◻ Varying learning models
8
Department #
Architecture & Planning 2
CILT 1
Chemical Engineering 1
Computer Science 3
Mechanical Engineering 1
8
Study 2: Flipped class case study
■ Computer Architecture second year course◻ 175 registered students◻ Three core-activities conducted during lecture sessions
■ Study aimed at assessing feasibility and potential◻ Extent of prototype tool use◻ Tool effect on learners’ learning experience
9
Activity Orchestration Orchestration Level
Demonstration Live demonstration Class
Discussion Interactive discussions Class/Individual
Timed quiz Timed quiz session Individual
Orchestration tool usage pattern
■ Nine orchestration tools used during duration of course ◻ Most tools used to render content◻ Specialised tools once-off tasks
■ On average prototype orchestration tool used most of the time◻ Used 66.72% on average◻ Context switching occurred an
average of two times with noticeable time during switchover
10
Tool Freq. Duration
Prototype 10 00:30:31
VideoGlide 8 00:07:56
Firefox 1 00:21:29
Impress 1 00:38:26
Evince 1 00:00:50
QtSpim 1 00:12:47
Robotic Arm 1 00:01:24
TextEditor 1 00:02:07
VirtualBox 1 00:00:58
Learner experience
■ Tool helped organise activities
■ Static sequencing (activity listing) found helpful
■ Timer useful to average performers
11
■ ”I did not really notice the tool”
■ ”I found the classroom experience fun”
Study findings
Feasibility of organised orchestration■ Facilitated neutral flow of activities■ Results from learner survey indicate tool was impact
neutral
Potential to facilitate improved learning outcomes■ Learner survey suggests orchestration workbench has
the potential to positively impact learning experience
12
Study 3: Ad hoc vs. organised orchestration
13
PortableApps—ad hoc orchestrationWorkbench prototype interface—organised orchestration
PortableApps Prototype Workbench
Study objectives
14
How does organised orchestration, using an orchestration workbench, compare with ad hoc orchestration?
■ Hypothesis #1: Workbench is more effective◻ H0: Workbench effectiveness is comparable to ad hoc approach◻ Ha: Workbench is more effective than ad hoc approach
■ Hypothesis #2: Workbench results in more positive effect teaching experience◻ H0: Effect of workbench on teaching experience is comparable to
ad hoc approach◻ Ha: Workbench has a positive effect on teaching experience
Experimental variables & conditions
15
■ Within group laboratory study◻ 29 participants from General Education and Training department
at Cape Peninsula University of Technology■ IV: orchestration technique■ DV: time on task; AttrakDiff 2 dimensions■ Control variables: ISP level; Teaching experience &
Computing experience
RQ AH Factor Variable Scale Description
RQ1 H1Task speed Tt Min Task time
Success PQ [-3 – 3] AttrakDiff 2
RQ2 H2 User experience HQ-I; HQ-S; ATT [-3 – 3] AttrakDiff 2
Experimental procedure
16
Experimental procedure involved random blocks■ Learning scenario
◻ CAPS Grade 5 Teacher Guide■ Orchestration techniques
◻ PortableApps: Ad hoc approach◻ Prototype: Workbench approach
■ Data collection◻ Time on tasks (effectiveness)◻ AttrakDiff 2 ‘Comparison A–B’
investigation (effectiveness and user experience)
◻ General comments
Experimental results: Time on tasks
■ Time of tasks (paired student t-test)◻ p-value = 0.1349
■ Orchestration 14.7% faster using workbench
17
p > 0.05—no statistically significant difference for all four dimensions
Grp 1 Grp 2 ISP Teaching Computing
Workbench 5.14 4.33 ISP 2, 4; N/A 2; 3; 4+ times 2; 3+ years
PortableApps 7.00 4.11 — — —
Experimental results: AttrakDiff 2 (2)
AttrakDiff 2 portfolio-presentation
■ Both orchestration approaches fall within the same character region
■ Workbench approach falls closer to desired character region
18
NOTE: Missing confidence rectangles ***
Experimental results: AttrakDiff 2 (3)
AttrakDiff 2 dimensions
■ PQ (Wilcoxon test)◻ p-value = 0.5465
■ HQ-I (paired student t-test)◻ p-value = 0.08498
■ HQ-S (paired student t-test)◻ p-value = 0.8886
■ ATT (Wilcoxon test)◻ p-value = 0.1634
19
p > 0.05—no statistically significant difference for all four dimensions
Experimental results: AttrakDiff 2 (4)
20
Ad hoc analysis of AttrakDiff 2 dimensions performed by analysing dimension word-pairs.
Experimental results: AttrakDiff 2 (5)
21
Ad hoc analysis of AttrakDiff 2 dimensions performed by analysing dimension word-pairs.
Study findings
Effectiveness■ Learning activities orchestrated faster using workbench■ Participants’ perceived success more pronounced with
workbench
User experience■ Higher mean score for HQ-I and ATT dimensions
Validity of results■ No statistical significance
◻ Only 29 of planned 52 participants took part in study◻ Study to be replicated with additional participants
22
Future directions
23
Further explore the correlation between the use of an orchestration workbench and positive teaching and learning experience
■ Dynamic pre-session management◻ IMS Simple Sequencing standard
■ Evaluate approach in authentic educational settings
Planned activities
25
Milestone Milestone description Timescale
1. Experiment #2 Replicate study May 2016
2. Experiment #3 CSC1010H case study May 2016
3. Experiment #4 Large-scale experiment August 2016
4. : : :
5. : : :
6. Thesis draft Draft manuscript February 2017
7. Thesis manuscript Final manuscript April 2017
Bibliography
[1] Jeremy Roschelle et al. “Classroom Orchestration: Synthesis”. Computers & Education, 6a9:523–526, 2013.
[2] Pierre Dillenbourg. “Design for Classroom Orchestration”. Computers & Education, 69:485–492, 2013.
[3] Pierre Dillenbourg and Patrick Jermann. “Technology for classroom orchestration”. New Science of Learning. 525–552, 2011.
[4] Luis P. Prieto et al. “Orchestrating technology enhanced learning: a literature review and a conceptual framework”. International Journal of Technology Enhanced Learning, 3(6):583, 2011.
[5] Juan A. Munoz-Cristobal et al. “Supporting Teacher Orchestration in Ubiquitous Learning Environments: A Study in Primary Education”. TLT, 83–97, 2014.
26