Two make a network:

using graphs to assess the quality of collaboration of dyads

Irene-Angelica Chounta 1, Tobias Hecking 1, H. Ulrich Hoppe 1, Nikolaos Avouris 2

1Collide, University of Duisburg-Essen, Germany

2HCI Group, University of Patras, Greece

{chounta, giemza, hoppe}@collide.info

avouris@upatras.gr

Collaborative Activities: Analysis and Evaluation

Analysis & Evaluation:

� To map collaborative mechanisms

� To improve the learning outcome

� To support teachers in class orchestration

Learning Analytics: „the measurement, collection, analysis and reporting of data about learners and their contexts” (LAK 2011)

Collaborative Activities: Analysis and Evaluation

Methods &Tools:

� Logfile Analysisactivity metrics retrieved from logfiles

� Interaction Analysismetrics of interaction among users while working together

� Social Network Analysis and Graph Theoryuser activity and interaction is represented through graph representations

Objectives of the study

� To represent the collaborative activities of dyads using networks

� To use network metrics in order to assess qualitative aspects of collaboration

Research hypothesis:

The properties and characteristics of networks that represent collaborative activities, reflect the quality of collaboration

Method of the study

� Network generation from logfiles of previously evaluated collaborative activities

� Visual inspection of networks representing good vs. bad collaboration quality

� Study of the relation of human ratings and network properties in a systematic way (correlation analysis)

Collaborative Activities of Dyads

� Team: two students working over a shared-spaces (Synergo)

� Task: the construction of an algorithmic flowchart

� Time: synchronous communication for 90 minutes

• Dataset of 228 Collaborative Sessions

• Pre-evaluated with qualitative and quantitative methods with respect to collaboration quality

(a) Two human experts evaluated the dataset [1]

� Quality of Collaboration (CQA) was evaluated on a 5-point Likert scale [-2, +2]

� Well-established results for inter-rater reliability and consistency

(b) The dataset was used to validate an automatic rater (time-series classification) [2]

� Meaningful Activity takes place in time frames of

15 – 30 seconds

� Time series depict the quality of collaboration

Related Work

[1] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)[2] Chounta, I.-A.,Avouris, N.: Time Series Analysis of Collaborative Activities. CRIWG2012 (2012)

Quality of Collaboration

Rating scheme[3] for the assessment of the quality of collaboration (CQA):

General aspects of collaboration

Collaborative Dimensions

CommunicationCollaboration flow (CF)

Sustaining mutual understanding (SMU)

Joint information processingKnowledge exchange (KE)

Argumentation (Ar)

CoordinationStructuring the problem solving process

(SPSP)

Interpersonal relationship Cooperative orientation (CO)

CQA = average(CF, SMU, KE, Ar, SPSP, CO)[3] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)

Network generation from logfiles

� Network maps generated from logfiles of collaborative activities:

- Nodes represent user actions

- Edges represent dependencies among actions

• To track dependencies:

(a) The time distance between actions ranges from

10 to 30 seconds

(b) Relevance on temporal and spatial proximity

(c) The identity of the actor should differ

Logfile example:

Resulting Network:

Network generation from logfiles

Visual inspection of networks

� The SiSOB Workbench[4] was used for the visualization and analysis of the network graphs

Good Collaboration Quality Bad Collaboration Quality

[4] Göhnert, T., Harrer, A., Hecking, T., Hoppe, H. U.: A workbench to construct and re-use network analysis workflows: concept, implementation, and example case. (2013)

Network metrics

• The number of nodes (N) and the number of edges (E)

• The diameter of the network (d)

• The average path length (APL)

• The density of the network (D) = (��

�������

• The Power Law Fit (PLF)

k

P(k)

Hubs

Results

� Comparison of Network metrics and Ratings of human experts for Quality of Collaboration (CQA)

� Most metrics correlate (p<.05) to CQA and to individual collaborative dimensions

#Nodes #Edges Density Diameter (PLF) (APL)

Quality of Collaboration (CQA)

0.446 0.18 -0.394 0.294 -0.233 0.243

Results

� #Nodes correlates highly with collaboration quality

� Sessions with intense activity point to bettercollaboration

� ...In particular to successful argumentation and knowledge exchange � efficient communication

� Good collaboration produces to larger event networks

CF SMU KE Ar SPSP CO CQA

#Nodes (N) 0.358 0.351 0.4 0.416 0.339 0.41 0.446

#Edges (E) 0.179 0.19 0.169 0.156 0.195 0.18

Results

� Diameter and Average path length correlate positively with collaboration quality

� Good quality leads to :

- Bigger network maps

- Longer paths, thus long uptake chains

- Faster unfolding networks

CF SMU KE Ar SPSP CO CQA

Diameter (d) 0.267 0.246 0.228 0.273 0.185 0.295 0.294

(APL) 0.226 0.205 0.191 0.223 0.154 0.257 0.243

Results

� PLF (Power Law Degree Distribution): a smaller value indicates a better fit

� Correlates negatively to the Quality of Collaboration

� Activities of good quality appear to lead to Scale-free networks

� Good collaborations contain key events that cause activity bursts

CF SMU KE Ar SPSP CO CQA

(PLF) -0.229 -0.192 -0.225 -0.208 -0.196 -0.233

Results

� Density is a negatively correlated to collaboration quality

� Dense Networks point to poor collaboration quality

But why?

� Density is anti-proportional to number of nodes in scale-free networks (

�_� �

������[5]

� Thus, a low density indicates a bigger network ☺

CF SMU KE Ar SPSP CO CQA

Density (D) -0.405 -0.275 -0.34 -0.36 -0.268 -0.323 -0.394

[5] Hoppe, H. U., Engler, J., Weinbrenner, S.: The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement. (2012)

Results – Solution Quality

� The diagram was graded for the correctness of the solution from 0 to 10

� Detailed solutions (big networks) are evaluated as good

� no correlation with power law fit or network density… But a correct solution does not presuppose good collaboration and vice versa

#Nodes (N)

#Edges (E)

Diameter (d)

(APL)

solutiongrade

0.319 0.305 0.202 0.189

Discussion

� Efficient collaboration results in biggernetworks.

� Good practices unfold faster in time and form longer chains of actions.

� Dense networks do not portray good collaboration

� Good collaboration results in scale-free networks : a key action leads to reciprocal interplay

� The size of the network is a good indicator for the solution

Conclusion

� Dyads interaction as represented by networks can indicate the meaningful interplay and successful collaborative practice

� Certain properties of networks reflect the quality of collaboration

& Future work

� Content analysis to refine the relations and connections between user actions

� Network metrics as a tool for automaticassessment of collaborative activities

Questions.

Thank you ☺

chounta@collide.info