strategy and tools to evaluate learning analytics ... · strategy and tools to evaluate learning...

Strategy and tools to evaluate learninganalytics interventions

in the transition from secondary to higher education

STELA Erasmus+ project (562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD)

"The European Commission support for the production of this publication does not constitute anendorsement of the contents which reflects the views only of the authors, and the Commission cannotbe held responsible for any use which may be made of the information contained therein."

2

Contents

Contents

1 Introduction 4

2 Connecting to academic achievement 52.1 MOOC for future students: Learning Tracker . . . . . . . . . . . . . . . . . . . . . . . 52.2 Dashboards for prospective and future students in regular higher education: LASSI,

POS, and REX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Use of the learning analytics intervention 93.1 Dashboard response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Dashboard use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2.1 Log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2.2 Qualitative approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 Perceived usefulness and usability 134.1 Microinteractions for fast feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Quantitative instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 Qualitative approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5 Impact on overall first-year experience 21

6 Learning engagement and learning behavior 236.1 MOOC for furture students: Learning Tracker . . . . . . . . . . . . . . . . . . . . . . . 23

7 Return-on-investment 24

8 Conclusion 25

Authors 27

Bibliography 29

3

Chapter 1. Introduction

Chapter 1

Introduction

This chapter elaborates on the strategies and tools used to evaluate learning analytics interventionssupporting the transition from secondary to higher education. The evaluation strategies and tools areorganized according to six themes depending on the dimension on which the evaluation is focusing:

• Academic achievement (Chapter 2): where the learning analytics interventions are connectedto academic achievement;

• Use of the learning analytics intervention (Chapter 3): the first step towards generatingimpact of learning analytics interventions is acceptance. This chapter shows how dashboard usecan be used to evaluate learning analytics interventions;

• Perceived usefulness and usability (Chapter 4): where the learning analytics interventionsare evaluated using the perceived usefulness and usability of the dashboards;

• First-year experience (Chapter 5): where the dashboards are evaluated using the overallfirst-year experience; and

• Learning engagement and learning behavior (Chapter 6): where the learning analyticsinterventions are evaluated based on the impact of the interventions on learning behavior.

• Return-on-investment (Chapter 7): where learning analytics interventions are assessed basedon their investments but also on their scalability to larger populations and transferability toother contexts.

4

Chapter 2. Connecting to academic achievement

Chapter 2

Connecting to academic achievement

This chapter elaborates on how the learning analytics interventions supporting the transition fromsecondary to higher education can be evaluated by connecting the intervention to academic achieve-ment.

2.1 MOOC for future students: Learning Tracker

(The description of the Learning Tracker below is based on [11]).

The Learning Tracker case study explored to what extent the insights from social comparison canbe translated to support learners in Massive Open Online Courses (MOOCs). To this end a “LearningTracker” was developed that provides feedback based on social comparison in online courses. ThisLearning Trackers is a personalized and scalable feedback system that presents MOOC learners witha visual comparison of their behavior to that of their successful peers who successfully completed apast iteration of the course. Figure 2.1 presents an example of the Learning Tracker.

Evaluation strategy For evaluation the Learning Tracker within MOOCs a test setup (A-B testing)with a treatment group (receiving the MOOC with the learning tracker) and a control group (receivingthe MOOC without the learning tracker) was used. By random assignment learners were assigned toeither the control or a treatment condition and remained in this condition throughout the study.

The Learning Tracker was released for the treatment group from the third course week. Afterwards,the learning tracker was updated week-by-week using the learning tracker activity up to that week.In the control condition, learners did not receive the Learning Tracker. However, the edX platformoffers by default a very basic form of learner feedback: a learner can visit the progress page and viewthe number of points scored so far in the course. This progress page is available to all learners, bothin the treatment and in the control group. In the treatment condition, learners received the LearningTracker in addition to edX’s progress page.

The impact of the learning tracker on the successful course completion was evaluated. Alearner is considered to have successfully completed the MOOC if the learner earned the requiredminimum passing score based on the summative quiz questions. Student that complete the courseget a successful completion certificate. While individual student intentions may vary throughout thelearner population, the achievement of earning a certificate is an appropriate outcome measure, asit demonstrates sustained commitment to the course and mastery over the course material. Morespecifically, the hypothesis was that the Learning Tracker will increase learner achievement in terms ofcourse completion. To this end the significant difference between the completion rate of the treatmentgroup and control group is assessed.

Particular case-study setup The case study of the STELA project the Learning Tracker focusedon the Pre-University Calculus MOOC developed by TU Delft1. This self-paced MOOC aims

1Pre-University Calculus MOOC: https://www.edx.org/course/pre-university-calculus

5

https://www.edx.org/course/pre-university-calculus


Figure 2.1: Example of Learning Tracker visualizing the learning activity of a MOOC learner incomparison to past users who successfully completed the MOOC [11].

at preparing the learners for the introductory calculus courses in the first year of higher educationby revising four important mathematical subjects that the programs mostly assume to be masteredby beginning bachelor students: functions, equations, differentiation, and integration. The MOOCconsists of six modules of one week each and one final test. The modules consist of a collectionof three to five minute lecture videos, inspirational videos on the use of mathematics in Science,Engineering, and Technology, (interactive) exercises, homework, and tests. The MOOC is offered forfree, although a verified certificate can be obtained for $50 USD.

The Learning Tracker of the Pre-University Calculus MOOC provided feedback on (1) quiz submis-sion timeliness, (2) number of quiz questions attempted, (3) average time on the platform per week (inhours), (4) number of revisited video lectures, (5) number of forum contributions, and (6) percentageof time spent on quizzes.

Evaluation of Learning Tracker in MOOC

The actual evaluation of the Learning Tracker within the Pre-University Calculus MOOC can befound in Output 8: “Evaluation report of the learning analytics interventions”, Chapter 2.

6


2.2 Dashboards for prospective and future students in regular highereducation: LASSI, POS, and REX

Evaluating the impact of the learning dashboards deployed to provide first-year students feedbackon their learning skills (LASSI dashboard [5, 8]) or academic achievement (REX [6]) or prospectivestudents after participation to the positioning test (POS [7]) on academic achievement is challenging.The two main underlying problems are:

1. The dashboards are deployed in real university settings. The use of an evaluation strategy with acontrol group and treatment group (A-B testing) was not possible due to ethical restrictions.As based on literature one could expect positive outcomes of additional feedback, it was ethical-wise not acceptable to not offer the learning analytics intervention with additional feedback tothe control group.

2. There is no immediate academic achievement connected to the intervention of theLASSI, POS, and REX learning dashboards. The three dashboard are targeting students’ self-reflection and learning and studying skills.

3. A real university setting is no well-controlled environment where all conditions, except a learninganalytics intervention, can be kept unchanged. Even more, often (and preferably) learninganalytics interventions are part of a larger strategy of the programs or universities to improvelearning or student support.

The three points below provide a small comparison of the context of the Learning Tracker case study(MOOC), for which connection to academic achievement using a setup with treatment and controlgroup was possible, and the LASSI, POS, and REX dashboard for which this was not feasible.

1. For the case study of the MOOC, learners from all over the world participate, which are oftennot connected to the MOOC builders (TU Delft). In contrast, for the LASSI, POS, and REXlearning dashboards the learners are subscribed to the university (KU Leuven and TU Delft) orare aspiring students about to subscribe at the university (KU Leuven). In the latter case, theuniversity’s strong ethical standards and existing cultural practices apply.

2. For the Learning Tracker in the MOOC, the learners are, beside joint participation in a MOOC,not connected to each other. For the LASSI, POS, and REX dashboards the students are studentsin the same physical class or program or will be soon. As such, providing these groups with adifferent ‘treatment’ will not go unobserved and can be considered unfair, especially consideringthe expected positive outcome of providing additional feedback.

3. The MOOC has a clear immediate and short-term academic achievement connected to it: thesuccessful completion of the MOOC itself. For the LASSI, POS, and REX dashboards, noacademic achievement is the immediate result of the activity on the dashboard itself. The‘closest’ academic achievement is the academic achievement in the first year of higher education,which is influenced by many factors beside the learning dashboards.

For the LASSI, POS, and REX dashboards another kind of connection was made to academicachievement. The variables to which these learning dashboard interventions were providing feedbackon were selected based on their relation with later academic achievement, i.e. they are predictive forstudent success.

• For the REX dashboard, providing feedback on early academic achievement, it is clear that earlyacademic achievement is the best predictor for later academic achievement, explicitly shown forthe case of KU Leuven engineering bachelor in [17].

• For the LASSI dashboard, providing feedback on learning and studying skills, these skills havebeen shown to have added predictive value in predicting first-year academic achievement, overprior education for the case of KU Leuven STEM education in [13].

7


• For the POS dashboard, both the positioning test, prior education and learning and studyingskills have been proven to have predictive power for first-year academic achievement in STEM [13,17, 18].

Evaluation of the LASSI dashboard

The actual evaluation of the LASSI dashboard for feedback on learning and studying skills canbe found in Output 8: “Evaluation report of the learning analytics interventions”, Chapter 3.

Evaluation of the REX dashboard

The actual evaluation of the REX dashboard for feedback on academic achievement can be foundin Output 8: “Evaluation report of the learning analytics interventions”, Chapter 4.

Evaluation of the POS dashboard

The actual evaluation of the POS dashboard for feedback to aspiring students after participationto the positioning test can be found in Output 8: “Evaluation report of the learning analyticsinterventions”, Chapter 5.

8

Chapter 3. Use of the learning analytics intervention

Chapter 3

Use of the learning analytics intervention

This chapter elaborates on how the use of learning analytics interventions supporting the transitionfrom secondary to higher education, i.e. the buy-in to the interventions, can be used to evaluate them.This is an important step, which is important not to take for granted.

Rec. 1: Acceptance precedes impact.

Before impact can be realized with learning analytics dashboards, dashboards have to find theirway to practice. Therefore, these dashboards have to be accepted first by the stakeholders. Animportant, and often first, step in evaluating learning analytics interventions is to assess thedashboards acceptance.

3.1 Dashboard response

When dashboards are offered to users, the first step in the evaluation is to check if the users actuallyuse the dashboard.In case of the LASSI [5, 8], REX [6], and POS [7] dashboards the users (students or prospective users)were invited to the dashboard through email.

• LASSI dashboard: The first-year students that filled in the LASSI survey received an invitationto the dashboard around two weeks after completion through their official student email address.This process was used both in the KU Leuven case study [5] and in the TU Delft case study.

• REX dashboard: The students received an invitation to the REX dashboard [6], providingfeedback on academic achievement, through their official student email address one day laterthan the official announcement to their grades.

• POS dashboard: The participants to the positioning test [18] received an invitation to thefeedback POS dashboard within a week after participation, through the email address they usedfor registered to the test.

After the users are invited, the response to the invitation can be monitored by logging the entranceto the learning dashboards. Different outcomes can be recorded:

• if the users respond to the invitation and enter the dashboard or not (binary variable),

• how long after the invitation the users respond to the invitation (continuous variable),

• when the users respond to the invitation (the time of the day, week, etc),

• which device they use to go to the dashboard, and

• if they visit the dashboard for the first time or if they return.

9


For each of the above outcome variables it is possible to research how they differ depending onunderlying characteristics and background variables of the users such as study program, gender, learn-ing and studying skills, prior education, academic achievement, etc. This allows to assess if students’response regarding entering the dashboard differs between different subgroups.This evaluation approach was used for evaluation of the LASSI, REX, and POS dashboards.







3.2 Dashboard use

This section elaborates on two possible ways to evaluate the use of a dashboard that were used withinthe STELA project: log files (Section 3.2.1) or observations (Section 3.2.2).

3.2.1 Log files

The interaction with the learning dashboards obviously goes beyond ‘entering’ the dashboard (re-sponse). Once the user enters the dashboard the interaction with the dashboard can be logged inmore detail. As a consequence this behavior can then be reported as an outcome measure. Within theSTELA project one particular use of the dashboard was monitored directly related to the ‘actionablefeedback’ objective: the interaction of the user with the explicit advice provided in the dashboards(Figure 3.1).Again, this outcome variable related to dashboard use can be related to underlying characteristics andbackground variables of the user.

This evaluation approach was used for evaluating the use of the LASSI dashboard and to relate itto the learning skills and background variables (gender and study program) of the student [8].



3.2.2 Qualitative approach

On top of the previous, quantitative approach, a more qualitative approach can be taken to evaluatethe use of the dashboard. Beside the more traditional focus groups (typically occurring either beforeor after using the dashboard), an actual observation of the use of the dashboard can be done. While

10


Figure 3.1: Screenshot of the LASSI dashboard1 [5, 8] showing how users can interact with furtheractionable advice in the dashboard (by clicking the “How to improve?” region).

11


the observations can assess the usability or usefulness of the dashboard (see Chapter 4), they can alsofocus on other impact that the dashboard has on the user.

Within the STELA project scenario-based observations were used, using the concurrent think-aloud protocol. The scenario-based approach means that the testers were not the real end-users ofthe dashboards, but were users who were asked to imagine being in a particular situation, which isrepresentative for the real end-user. Subsequently, a think-aloud protocol was used where users arerequested to explain what they are thinking while performing their tasks.

One way to evaluate the quality of the dashboard use is by recording which insights occur withthe users when they use the dashboard: factual, interpretative, or reflective insights [10].

1. factual insights: simple, objective statements or questions that are triggered by the dashboard,e.g. “I obtained a total score of 14/20.”;

2. interpretative insights: interpretation of the displayed data, relying on the participant’s knowl-edge and experiences, e.g. “I mainly score well on the easier questions.”; and

3. reflective insights: subjective, emotional, and personal connotations triggered by the dashboard,leading to further awareness and reflection, e.g. “I feel like I did not do well enough at this test,making me doubt about whether I should go for another study program.”.

Such observations can either be done in an authentic setting [9] or in mock-up settings wherethe users are provided with particular tasks [12]. This approach was used within the project forthe evaluation of the more visual dashboard providing feedback after the positioning test [12] incomparison to the original POS dashboard [7]. Here, 48 in-depth interviews (16 per dashboard), eachlasting between 40 minutes and 1 hour were performed. Each interview consists of four stages. The firstphase of the interview is scenario-based, using the concurrent think-aloud protocol. End-users have toimagine having participated in the positioning test and now getting their result. Three scenarios arepossible. Either they get a score in which they belong to group A (total score of 15/20), either groupB (12/20) or group C (6/20). Each test user says out loud the insights they obtain upon visualizationof the dashboard. Each insight is categorized into one of the three categories levels. The test usercan also mention when something in the dashboard is unclear, but the monitor of the test does notintervene and only writes down all statements made by the test person.



12

Chapter 4. Perceived usefulness and usability

Chapter 4

Perceived usefulness and usability

This chapter elaborates on approaches to evaluate the perceived usefulness and usability of learninganalytics interventions used within the STELA project [1]. It elaborates on the use microinteractionsfor fast feedback of the users (Section 4.1), the use of quantitative instruments (Section 4.2), andqualitative approaches (Section 4.3).

4.1 Microinteractions for fast feedback

While feedback of users is often gathered using questionnaires, sent to users after the learning analyticsintervention, current technology allows for fast and “light” querying of the users, using so-called micro-interactions.Microinteractions are defined as “short-time interruptions of primary tasks” [2]. Facebook’s thumbs upand swiping left (dislike) or right (like) in the Tinder dating app are examples.

Using microinteractions for evaluating learning analytics interventions has some advantages:

• they are less obtrusive that traditional questionnaires and

• they allow the user to evaluate the learning analytics intervention while being part of it.

On the negative side:

• the microinteractions can disturb the use of the actual learning analytics intervention and

• the microinteraction can be shown before the user had the full experience of the learning analyticsintervention, hereby only offering a partial evaluation of the intervention.

For the learning dashboards of the STELA project, microinteractions were used to collect fastfeedback from the dashboards users during the use of the dashboard. This procedure was used for theLASSI [5, 8] and REX [6].

Over the different iterations of the dashboard, different questions were used in the microinterac-tions.

• How CLEAR is the provided information? (LASSI, REX)

• How USEFUL is the provided information? (LASSI, REX)

• Does this information tells you MORE than your grades alone? (REX)

• Any further comments ... (open question) (LASSI, REX)

Typically, and shown in Figure 4.1), users provided their answer using stars (1-5 stars) or indicatethey have no opinion (“Geen mening”). Moreover, if users don’t want to reply to the questions theycan just click anywhere outside the pop-up box to continue their interaction with the dashboard.

13


Figure 4.1: Microinteraction in the REX dashboard [6], popping up from the bottom of the screen,to request for feedback (“HOW CLEAR is the provided information? ”). To focus the user to themicrointeraction, the dashboard itself is shortly shaded.

14



The actual evaluation of the LASSI dashboard for feedback on academic skills can be found inOutput 8: “Evaluation report of the learning analytics interventions”, Chapter 3.



4.2 Quantitative instruments

Quantitative instruments (surveys and questionnaires) are a common method to evaluate the usabilityand useflness of learning analytics interventions.

Within the STELA project three quantitative instruments were used:

1. Evaluation Framework for Learing Analaytics (EFLA) [14, 15, 16],

2. System Usability Scale (SUS) [3, 4], and

3. custom-made questionnaires.

EFLA and SUS are instruments that offer a standardized evaluation. Hereby they compare differ-ent instruments and applications.

EFLA EFLA was especially designed for the purpose of evaluation learning analytics inteventions.EFLA “makes use of the subjective assessments about learning analytics tools by their users in orderto obtain a general indication of the overall quality of a tool in a quick and simple, yet thoroughlydeveloped, validated and reliable way” [15]. EFLA evaluates three dimensions: data (two questions),awareness & reflection (four questions), and impact (two questions).

There are two versions of EFLA, one focused on learners and one in teachers. Figure 4.2 shows theoriginal EFLA for learners. Within EFLA the learners rate the eight items of EFLA on a scale from1 for ‘strongly disagree’ to 10 for ‘strongly agree’. From the answers of the users, an overall EFLAscore can be calculated which ranges between 0 and 100.

As the STELA dashboards for which the evaluation was used are student-facing dashboards, thelearners version of EFLA was used (version 4). Several adaptations were made:

• the questions were translated to Dutch,

• the questions were adapted minimally to reflect the topic of the dashboard, and

• in consultation with the student advisers, the data-related questions D1 and D2 were placed atthe bottom of the questionnaire.

Figure 4.4 shows the questions as asked within the POS dashboard. The questions were, translatedto English:

• A1: This dashboard makes me more aware of my current study situation.

• A2: This dashboard makes me forecast my possible study situation, with/without change in mylearning behavior.

• A3: This dashboard stimulates me to think about my past learning behavior.

15


Figure 4.2: Original Evaluation Framework for Learning Analytics (EFLA) for learners [15] retrievedfrom http://www.laceproject.eu/evaluation-framework-for-la/.

16

http://www.laceproject.eu/evaluation-framework-for-la/


Figure 4.3: EFLA [14, 15, 16] questionnaire integrated within the POS dashboard [7]. The yellow baron the left (“JOUW MENING” = “YOUR OPINION”) contains the feedback questions and is openedupon hovering or clicking.

• A4: This dashboard stimulates me to change my learning behavior or study trajectory, and/orstrengthens me in my current learning behavior or study trajectory.

• I1: This dashboard stimulates me to study more efficiently (as in: making sure to work in theright way).

• I2: This dashboard stimulates me to study more effectively (as in: making sure to reach thetarget, in any way).

• D1: It is clear to me which data are being collected to assemble this dashboard.

• D2: It is clear to me why the data shown in this dashboard are being collected.

In the POS dashboard, the questions were supplemented with two open questions (Figure 4.4): thefirst one requests for a contact email-address in case the users wants to be contacted regarding theirperception of the feedback and the second one requests for further remarks or suggestions.

In the case study comparing the original POS dashboard with a more visual version [12] the eightadapted EFLA questions were used in the third phase of the 48 in-depth interviews (see Section 3.2.2for the first two phases).

SUS SUS is a simple usability scale that covers “the effectiveness (the ability of users to completetasks using the system, and the quality of the output of those tasks), the efficiency (the level of resourceconsumed in performing tasks), and satisfaction of the use of an application (users’ subjective reactionsto using the system)” [3]. It was designed to allow for a fast but only global view of the subjectiveassessments of usability.

SUS consists of ten items scale, which users respond to using a 5-point Likert scale. The ten itemsare [3]:

1. I think that I would like to use this system frequently.

17


Figure 4.4: EFLA questions [14, 15, 16] as used in the POS dashboard [7]. These questions appearafter clicking the yellow bar (“JOUW MENING”) in the dashboard as shown in Figure 4.3.

18


2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

The SUS questionnaire was used to evaluate the usability of the more visual version of the POSdashboard [12] and the original POS dashboard [7]. In particular it was used in the third phase of the48 in-depth interviews (see Section 3.2.2 for the first two phases).

Custom-made questionnaires Custom-made questionnaire can be used when no standardizedquestionnaires are available or when the context requires a specific approach.

The microinteractions of Section 4.1 are one example of such short custom-made questionnaires.A custom-made questionnaire was used to evaluate the usability of the more visual version of the

POS dashboard [12] and the original POS dashboard [7]. In particular it was used, together with theEFLA and the SUS, during the third phase of the 48 in-depth interviews (see Section 3.2.2 for thefirst two phases). The custom-made questionnaire was used to evaluate how the dashboard satisfy theoriginal design requirements. It consisted of 21 statements, to which the user could “Strongly disagree”or “Strongly agree”, using a 5-point Likert scale. The 21 items, translated from Dutch to English, are:

1. The feedback helps to assess whether my current starting competences match with the requiredstarting competences of the program.

2. The feedback triggers reflection.

3. I can use the feedback system independently (I do not need extra aid, neither technical orpedagogic, to understand the feedback).

4. The feedback provides me with a detailed overview of my results on the positioning test.

5. The feedback helps me to assess my strengths and points of improvement.

6. The feedback allows to position myself with respect to the other participants.

7. The feedback allows to explain and interpret my score.

8. The feedback provides me with a definite advice (i.e. an advice stating that it would be betterto (not) start the program).

9. The feedback only shows factual information and does not provide interpretations or conclusions.

10. The feedback only contains information that is relevant.

11. The feedback is personalized.

12. The feedback is confronting.

13. I feel stigmatized based on the feedback.

14. The feedback provides a good overview and an immediate view on the most relevant information.

15. The dashboard is not complex but contains sufficient detail.

16. The dashboard provides sufficient context.

17. I understand where the information is coming from.

18. I understand the meaning of the information.

19. There is no ambiguity in the dashboard (no alternative interpretations are possible).

19


20. There is a good balance between textual and visual elements.

21. The dashboard is visually appealing.

22. (open question) Any additional remarks or suggestions?


The actual evaluation of the POS dashboard for feedback to aspiring students after the positioningtest can be found in Output 8: “Evaluation report of the learning analytics interventions”, Chapter5.

4.3 Qualitative approaches

Beside the quantitative approach, more qualitative methods using focus-groups or interviews can beused to assess the perceived usefulness and usability of learning analytics interventions

In the case study comparing the more visual dashboard providing feedback after the positioningtest [12] with the original POS dashboard [7], the first evaluation phase was related to the use of thedashboard (See Section 3.2.2). The third phase was used for a quantitative evaluation of the perceivedusefulness and usability using the EFLA, SUS, and a custom-made questionnaire (see Section 4.2).In the second phase a qualitative evaluation was used. In this phase the interviewer switched to atask-based interview with active intervention. The interviewer gives the test persons tasks based onthe information or insights they missed during the first phase and finds out why these parts andinsights have been missed. This phase tries to examine whether the dashboard is intuitive and hasany shortcomings.The final and fourth phase of the evaluation is a short phase where the user can express their preferencebetween the different versions of the dashboard.


The actual evaluation of the POS dashboard for feedback to aspiring students after the positioningtest can be found in Output 8: “Evaluation report of the learning analytics interventions”, Chapter5.

20

Chapter 5. Impact on overall first-year experience

Chapter 5

Impact on overall first-year experience

The STELA learning analytics interventions were targeting the transition from secondary to highereducation. As two dashboard (LASSI [5, 8] and REX [6]) were deployed within a traditional first-yearuniversity context, an important evaluation measure is to assess the impact of the dashboards on theoverall first-year experience.

To this end a questionnaire was developed that asks former first-year students regarding theirexperiences in the first-year. Obviously this questionnaire should be deployed shortly after the firstyear, at latest in the beginning of the second year.

The custom-made questionnaire targets different dimensions:

1. Beliefs regarding the behaviour supporting first-year study success.

2. How easy it was to position the own academic engagement and activities, academic performance,and academic well-being with respect to their peers (Social comparison, one of the three mainprinciples from the STELA interventions (see Output 6)) and with respect to the expectationsof the program.

3. The emphasis of the program regarding study techniques, studying hard, attending classes,regularly checking student email, using the virtual learning environment, well-being, and theavailable support options.

4. The actual behavior during the first year regarding: studying hard, attending classes, preparingclasses, following student email, and using the virtual learning environment.

5. If the provided feedback was sufficient regarding academic engagement and activities, academicachievement, and academic well-being.

6. Who provided advice regarding their studies (staff members, teachers, friends, other students,or family)

7. If the received feedback was informative, was motivating, induced reflection, induced changein study-related behavior, supported social comparison to peers, clarified expectations, showedimpact of academic engagement or achievement on the future study career, supported academicdecision-making.

8. Global satisfaction regarding the feedback.

9. Global satisfaction with the provided support.

10. Availability of staff and teachers for additional feedback.

11. Whether faculty or program made sure that support was available, people were available toanswer questions, information was provided regarding the available support.

12. Final specific questions regarding feeling lost, feeling uncertain regarding academic engagementor academic achievement, and the participation in additional support or training.

21

Chapter 5. Impact on overall first-year experience

13. Open question asking for the most valuable support during the first year.

14. Open question for additional remarks or suggestions.

To measure the impact of the learning analytics interventions of the STELA project the ques-tionnaire was deployed with second-year students during the first week of the academic year. It wasdeployed in two consecutive years. In the first year, the second-year students did not have accessto the dashboards yet, hereby providing a baseline measurement. The next year, the new cohorts ofsecond-year students did have access to the dashboard. Comparing the outcome of the questionnaireover the two years allows to assess the impact of the learning analytics interventions on the overallfirst-year experience. Additionally, as it is very likely that other changes took place between the dif-ferent years, a baseline program was used where the dashboards were not implemented. This allowsto asses changes over academic years that are not related to the deployment of the learning analyticsinterventions.

Measured impact on the first-year experience

The impact of the REX and LASSI dashboard on the overall first-year experience can be foundin Output 8: “Evaluation report of the learning analytics interventions”, Chapter 6.

22

Chapter 6. Learning engagement and learning behavior

Chapter 6

Learning engagement and learningbehavior

This chapter elaborates on the evaluation of learning analytics interventions on learning behaviorwithin the STELA project [1].

6.1 MOOC for furture students: Learning Tracker

(The description of the Learning Tracker below is based on [11]).

Within the case study of the Learning Tracker (for a short description see Section 2.1), beside theimpact of the Learning Tracker on MOOC completion (Section 2.1) the impact on learning engagementand learning activities within the MOOC was assessed. Two research question were used in the casestudy:

1. Learners subject to the Learning Tracker will increase learning engagement.

2. Learners subject to the Learning Tracker will change the aspects of their behavior that theLearning Tracker makes them aware of.

Learning engagement was operationalized as the overall activity of the learner within the onlinecourse, and was measured based on the log files. For the specific behaviors, the impact on the specificdimensions visualized in the Learning Tracker was checked (also derived from the log files).

As elaborated on in the earlier Section, the case study of the Learning Tracker within the STELAproject [1] used A-B testing with a treatment group (receiving the MOOC with the learning tracker)and a control group (receiving the MOOC without the learning tracker). The dimensions evaluatedfor the learning behaviour were therefore: (1) quiz submission timeliness, (2) number of quiz questionsattempted, (3) average time on the platform per week (in hours), (4) number of revisited video lectures,(5) number of forum contributions, (6) percentage of time spent on quizzes.

Evaluation of Learning Tracker in MOOC

The actual evaluation of the Learning Tracker withing the Pre-University Calculus MOOC canbe found in Output 8: “Evaluation report of the learning analytics interventions”, Chapter 2.

23

Chapter 7. Return-on-investment

Chapter 7

Return-on-investment

One of the three main principles for learning analytics interventions within the STELA project [1] wasgood return-on-investment. As discussed in Output 6, the interventions in the project aimed, at ameta-level, at obtaining good return-on-investment by means of scalability and transferability to othercontexts. An important evaluation metric is therefore if the learning analytics interventions proved tobe:

1. scalable: if the interventions can be extended to larger scales with reasonable investments,

2. transferable: if the interventions be be transferred from one module, course, program or insti-tute to another.

The particular evaluation of scalability and transferability strongly depends on the particular contextof the learning analytics intervention.

Evaluation of return-on-investement

The actual evaluation of the return on investment can be found in Output 8: “Evaluation reportof the learning analytics interventions”,

• Learning Tracker, Chapter 2

• LASSI dashbaord, Chapter 3,

• REX dashboard, Chapter 4.

• POS dashboard, Chapter 5.

• SPOC dashboard for teachers, Chapter 7.

24

Chapter 8. Conclusion

Chapter 8

Conclusion

This chapter elaborated on the strategies and tools used to evaluate learning analytics interventionssupporting the transition from secondary to higher education inside the STELA project [1].“ The evaluation strategies and tools used within the project were organized according to six themesdepending on the dimension on which the evaluation is focusing:

• Academic achievement (Chapter 2): where the learning analytics interventions are connectedto academic achievement;

• Use of the learning analytics intervention (Chapter 3): the first step towards generatingimpact of learning analytics interventions is acceptance. This chapter shows how dashboard usecan be used to evaluate learning analytics interventions;

• Perceived usefulness and usability (Chapter 4): where the learning analytics interventionsare evaluated using the perceived usefulness and usability of the dashboards;

• First-year experience (Chapter 5): where the dashboards are evaluated using the overallfirst-year experience; and

• Learning engagement and learning behavior (Chapter 6): where the learning analyticsinterventions are evaluated based on the impact of the interventions on learning behavior.

• Return-on-investment (Chapter 7): where learning analytics interventions are assessed basedon their investments but also on their scalability to larger populations and transferability toother contexts.

The first dimension, academic achievement, is one of the first dimensions that management typicallyaims to impact with learning analytics. Experiences from the STELA project show that evaluatingimpact on academic achievement is possible within a particular isolated context. In the project suchcontext occurred in Massive Open and Online Courses (Chapter 2). The impact of the LearningTarcker, a learning analytics intervention targeting MOOC learners, on successful MOOC completioncould be evaluated using A-B testing with a treatment group and a control group. In the isolatedMOOC contexts it was more manageable to keep all other conditions fixed beside the learning analyt-ics interventions.Evaluating the impact of the learning dashboards providing first-year students feedback on their learn-ing skills (LASSI dashboard [5, 8]) or academic achievement (REX [6]) or prospective students afterparticipation to the positioning test (POS [7]) proved to be challenging.

1. The dashboards are deployed in real university settings. The use of an evaluation strategy with acontrol group and treatment group (A-B testing) was not possible due to ethical restrictions.As based on literature one could expect positive outcomes of additional feedback, it was ethical-wise not acceptable to not offer the learning analytics intervention with additional feedback tothe control group.

25


2. There is no immediate academic achievement connected to the intervention of theLASSI, POS, and REX learning dashboards. The three dashboard are targeting students’ self-reflection and learning and studying skills.

3. A real university setting is no well-controlled environment where all conditions, except a learninganalytics intervention, can be kept unchanged. Even more, often (and preferably) learninganalytics interventions are part of a larger strategy of the programs or universities to improvelearning or student support.

This chapter has however shown different dimensions, and strategies connected to it, to evaluatelearning analytics interventions. The most important dimension to evaluate learning analytics inter-ventions first, before impact on academic achievement and retention could be considered, is actual‘use’ of the learning analytics intervention (Chapter 3). Before learning analytics interventionscan have impact they have to be used. Therefore, it is key to evaluate the the buy-in to the interven-tions.Next, when learning analytics interventions are used, the perceived usefulness and usability (Chap-ter 4), and the impact on for instance academic experience (Chapter 5) and learning engagement orbehavior (Chapter 6) can be assessed.This chapter furthermore discussed different evaluation methods ranging from quantitative approaches(impact on academic achievement, standardized questionnaires, custom-made questionnaires) andqualitative approaches (task-based interviews and focus groups).

Based on the findings in the report the following two recommendations regarding strategies andtools to evaluate learning analytics interventions can be made:

Rec. 1: Acceptance precedes impact.

Before impact can be realized with learning analytics dashboards, dashboards have to find theirway to practice. Therefore, these dashboards have to be accepted first by the stakeholders. Animportant, and often first, step in evaluating learning analytics interventions is to assess thedashboards acceptance.

Rec. 2: Impact is hard to measure

Impact of learning analytics in academic achievement or retention can be hard to measure. Whileparticular isolated context such as Massive Open and Online Course can provide opportunities forassessing the impact of learning analytics interventions on academic achievement and retention,similar application to higher education context can prove to be more challenging.First, often learning analytics interventions do not have an immediate academic evaluation, andthus assessment, connected to it.Secondly, real higher education contexts are no well-controlled environment where all conditions,except a learning analytics intervention, can kept unchanged. Even more, often (and prerably)learning analytics interventions are part of a larger strategy to improve learning or student sup-port.Thirdly, ethical restrictions apply within a real university context which for instance can preventtesting with a treatment and control group.

Impact of learning analytics interventions should however be considered morebroadly than academic achievement and retention. Learning analytics interventions can alsoimpact learning engagement and behavior, or overall academic experience.

26


Authors

Jan-Paul van StaalduinenJ. P. vanStaalduinen@ tudelft. nlTU Delft, Netherlands

Tinne De Laettinne. delaet@ kuleuven. beKU Leuven, Belgium

Tom Broostom. broos@ kuleuven. beKU Leuven, Belgium

Philipp Leitnerphilipp. leitner@ tugraz. atTU Graz, Austria

Martin Ebnermartin. ebner@ tugraz. atTU Graz, Austria

Rebecca Siddlerebecca. siddle@ ntu. ac. ukNottingham Trent University, United Kingdom

Ed Fostered. foster@ ntu. ac. ukNottingham Trent University, United Kingdom

27

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

Bibliography

Bibliography

[1] STELA project, 2017.

[2] Daniel L Ashbrook. Enabling mobile microinteractions. Georgia Institute of Technology, 2010.

[3] John Brooke. SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194):4–7, 1996.

[4] John Brooke. SUS: a retrospective. Journal of Usability Studies, 8(2):29–40, 2013.

[5] Tom Broos, Lauri Peeters, Katrien Verbert, Carolien Van Soom, Greet Langie, and Tinne De Laet.Dashboard for actionable feedback on learning skills: Scalability and usefulness, volume 10296LNCS. 2017.

[6] Tom Broos, Katrien Verbert, Greet Langie, Carolien Van Soom, and Tinne De Laet. Small dataas a conversation starter for learning analytics. Journal of Research in Innovative Teaching &Learning, 10(2):94–106, 7 2017.

[7] Tom Broos, Katrien Verbert, Greet Langie, Carolien Van Soom, and Tinne De Laet. Multi-Institutional positioning test feedback dashboard for aspiring students lessons learnt from a casestudy in Flanders. In ACM International Conference Proceeding Series, pages 51–55, 2018.

[8] Tom Broos, Katrien Verbert, Carolien Van Soom, Greet Langie, and Tinne De Laet. Dashboardfor actionable feedback on learning skills: how learner prole affects use. In Springer LectureNotes in Computer Science (LNCS) series. (Proceedings of the ECTEL 2017 conference; ARTELworkshop), pages 1–15, Tallinn, Estonia, 2017.

[9] Sven Charleer, Andrew Vande Moere, Joris Klerkx, Katrien Verbert, and Tinne De Laet. LearningAnalytics Dashboards to Support Adviser-Student Dialogue. IEEE Transactions on LearningTechnologies, 11(3):389 – 399, 2017.

[10] Sandy Claes, Niels Wouters, Karin Slegers, and Andrew Vande Moere. Controlling In-the-WildEvaluation Studies of Public Displays. In Proceedings of the 33rd Annual ACM Conference onHuman Factors in Computing Systems - CHI ’15, pages 81–84, New York, New York, USA, 2015.ACM Press.

[11] Dan Davis, Ioana Jivet, René F. Kizilcec, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben.Follow the successful crowd. In Proceedings of the Seventh International Learning Analytics &Knowledge Conference on - LAK ’17, pages 454–463, Vancouver, British Columbia, Canada, 2017.ACM Press.

[12] Nicolas Hoppenbrouwers, Tom Broos, Katrien Verbert, and Tinne De Laet. Less (context) ismore? Evaluation of a positioning test feedback dashboard for aspiring students. In Sumitted tothe Learning Analytics Conference, Tempe, Arizona, 2019.

[13] Maarten Pinxten, Carolien van Soom, Christine Peeters, Tinne De Laet, and Greet Langie. At-risk at the gate: prediction of study success of first-year science and engineering students in anopen-admission university in Flanders—any incremental validity of study strategies? EuropeanJournal of Psychology of Education, 2017.

28

Bibliography

[14] Maren Scheffel. Evaluation Framework for Learning Analytics, 2017.

[15] Maren Scheffel. The evaluation framework for learning analytics. PhD thesis, 9 2017.

[16] Maren Scheffel, Hendrik Drachsler, and Marcus Specht. Developing an Evaluation Framework ofQuality Indicators for Learning Analytics. In Proceedings of the Fifth International Conferenceon Learning Analytics And Knowledge, pages 16–20, Poughkeepsie, New York, 2015. ACM, NewYork, NY, USA.

[17] Jef Vanderoost, Riet Callens, Joos Vandewalle, and Tinne De Laet. Engineering positioning testin Flanders : a powerful predictor for study success? In Proceedings of the 42nd Annual SEFIconference, Birmingham UK, 2014.

[18] Jef Vanderoost, Carolien Van Soom, Greet Langie, Johan Van Den Bossche, Riet Callens, JoosVandewalle, and Tinne De Laet. Engineering and science positioning tests in Flanders : powerfulpredictors for study success ? In Proceedings of the 43rd Annual SEFI Conference, pages 1–8,2015.

29

strategy and tools to evaluate learning analytics ... · strategy and tools to evaluate learning...

Documents