a long-term evaluation of educational animations of functional programs

7/29/2019 A Long-Term Evaluation of Educational Animations of Functional Programs

1/5

A Long-Term Evaluation of Educational Animations of Functional Programs

Jaime Urquiza-Fuentes, J.Angel Velazquez-Iturbide

Laboratory of Information Technology in Education LITERey Juan Carlos University, Madrid, Spain

{jaime.urquiza,angel.velazquez}@urjc.es

AbstractIn this paper we study two different approachesto using program animations with educational aims: theirconstruction by students a constructivist and active approachand their vision a less active approach. In addition, we com-pare both approaches to a traditional teaching methodologywhere animations are not used. We have conducted a long-term evaluation with functional program animations usingan existing IDE with visualization features called WinHIPE.Our results are mixed. Students have a positive opinion aboutthe animations, either viewing or constructing them. While

the viewing approach improves some aspect of knowledgeacquisition, both approaches decrease the fail-rate. Finally, theconstruction approach improves the students attitude towardsthe subject. Therefore, both uses of program animations shouldbe integrated in the teaching methodology.

Keywords-program visualization; engagement; evaluation;

I. INTRODUCTION

Learning to program is one of the most difficult issues in

computer science curricula. Program animations, animations

for the rest of the paper, are one of the existing approaches

to overcome this problem. But despite of its potential, it is

not widely used by computer science educators. One of themain reasons, highlighted in [1], is the lack of evidence of

their educational benefits.

One of the most significant contributions in field recom-

mends a more constructivist approach in their educational

use [2]. Other studies propose to increase students en-

gagement with animations [1]. But despite of the literature

supporting the educational use of animations, results are not

clear regarding either the way of using them or their key

features to be effective educational tools [3].

In this work we investigate the benefits of two different

approaches: students construction of animations (an active,

constructivist and highly engaging educational task) and

students vision of animations (a less active approach).In addition, we compare the previous approaches with a

traditional approach where animations are not used (the

typical approach for the rest of the paper). To this end

we use a previously developed tool called WinHIPE [4],

a programming environment with visualization capabilities.

The rest of the paper is structured as follows. In section

II we survey the educational experiences about viewing and

constructing animations. The section III explains both, vision

and construction approaches. The sections IV and V detail

the experiment and its results. Finally in the section VI we

draw our conclusions.

I I . RELATED WOR K

Program animation is not a new research field. It is

partially driven by a firm belief from researchers on that

animations can help students in learning programming con-

cepts. However, existing experiments do not clearly support

this intuition. Hundhausen et al. [2] conclude that the moreactive the educational use the better the learning outcomes,

but they also highlight that there are areas that need further

investigation, e.g. the use of narratives and textual contents

within animations [5].

The Engagement Taxonomy [1] classifies different ed-

ucational uses of animations based on students engage-

ment. It identifies six different engagement levels: Non-

viewing, Viewing, Responding, Changing, Constructing and

Presenting. These levels are sorted by students engagement

with animations, being Non-viewing the lowest level and

Presenting the highest one. This taxonomy proposes that

the more the engagement the better the learning outcomes.

Here we briefly describe those levels involved in ourstudy. The Non-viewing level represents the typical scenario

without the use of animations. In the Viewing level the

students can play animations, changing their direction, pace

or abstraction level. If the animation provides multiple

views, switching among them is also considered in this

level. Finally, the Constructing level requires from students

to construct explanatory animations of program/algorithms

behavior. Students need neither special construction tools

nor to program and execute the program/algorithm.

This section surveys related studies that evaluate the

educational impact of constructing and viewing animations.

A. Experiments with Animation ConstructionWe have found five studies related to animation con-

struction. Stasko [6] designed assignments where students

had to construct their own animations. He detected that

students dedicated more time to study the algorithms for

which they had constructed animations. Urquiza-Fuentes et

al. [7] made a comparative study with vision tasks. Students

who constructed animations used the WinHIPE IDE, while

the others just viewed the animations produced by the

instructor with the same IDE. Both groups of animations

2012 12th IEEE International Conference on Advanced Learning Technologies

978-0-7695-4702-2/12 $26.00 2012 IEEE

DOI 10.1109/ICALT.2012.50

26


2/5

had additional textual explanations. The results showed that

students who constructed animations were longer involved

with the algorithm. As a consequence, they also improved

their knowledge acquisition.

The following three experiments studied animation con-

struction and presentation tasks. Hundhausen [8] compared

two different tools: a well-known algorithm visualization

tool, and artifacts selected by the students, ranging from

slides to crafts. This observational study detected improve-

ments in the students attitude who used their own artifacts.

Based on these results, a tool to construct algorithm anima-

tions was developed and compared again with the construc-

tion artifacts selected by students [9]. In this experiment,

improvements in programming performance were detected

on students who worked with the new tool.

Finally, Hubscher-Younger et al. [10] made a comparative

study with vision tasks. In this study, a number of volunteers

developed and presented animations for a set of algorithms.

They were allowed to choose the construction utilities. Then,

all the students, animation developers included, had to eval-uate those animations this can be considered as the vision

task. The results showed that the students who developed

and evaluated the animations acquired more knowledge than

those who just evaluated the animations.

The construction utilities play an important role in con-

struction tasks. Thus, most studies allow students to choose

the construction utilities or provide them with carefully de-

signed interfaces for this task. In addition, the use of textual

contents is present in most of the studies. We must also

remark that we have only found two comparative studies.

Although both studies delivered positive results, their rep-

resentativeness can be questioned due to their experimental

design. One of them involved only fifteen subjects [7]. Theother [10] seems to use groups of participants biased in

terms of students motivation prior to the experiment. While

the evaluation (viewing) tasks were compulsory, construction

and presenting tasks were voluntary, therefore students who

constructed animations were probably more motivated than

those who just evaluated others animations.

B. Experiments with Animation Vision

We have found five experiments for this approach. All

of them detected educational improvements in terms of

knowledge acquisition. Most of them compare vision tasks

with, at least, the non-viewing approach. Lawrence [11]s

study is the only one without a comparison: it just detectedimprovements when animations were used with textual la-

bels. Kann et al. [12] made a comparative study among

different educational uses of animations. However, they

only detected significant improvements between the viewing

and non-viewing approaches. Crosby et al. [13] detected

improvements when using multimedia materials made up of

visualizations and textual contents. Kehoe et al. [14] detected

improvements when using animations in an environment that

simulated homework. Thus, students used animations with

narrative contents to complete assignments without time

limit.

Finally, Kumar [15] showed that using visualizations

within the feedback provided by a tutoring system improves

knowledge acquisition. Furthermore, students who received

feedback with animations and textual contents obtained

better results than those who received feedback with just

animations.

In summary, narrative contents and textual explanations

are used in all the experiments but one. Therefore, they seem

to be features of educational tools that make animations

effective, even though they deserve more research [2].

III. EDUCATIONAL ANIMATIONS WITH WIN HIPE

This section summarizes the features of educational ani-

mations produced with WinHIPE [4] and their construction

process; further details can be found elsewhere [16].

A. Design and Contents of the Educational Animations

Our educational animations are web-based materials. Con-sequently, they can be easily shared, reused and published.

In addition, they can be used either in theoretical sessions,

lab sessions or at home. These animations consist of four

components: a description of the problem, an explanation of

an algorithm that solves the problem, the source code that

implements the algorithm, and the program animation. The

pace and direction of program animations can be controlled

with a typical VCR interface. Besides, animations can si-

multaneously display one or two consecutive steps for better

understanding. Fig. 1 shows an example of an educational

animation about simple recursive data types (computing the

sum of a list of integers), with all its contents visible and

showing two consecutive steps in the program animation.We provide four different web formats to view our educa-

tional animations, the user can change among them anytime.

Figure 1. An educational animation of the SumList program

27


3/5

Two of them, called Framed1 and Framed2, are inspired by

general principles of web pages usability [17]. They allow

the user to see, respectivelly one or simultaneously two of

the four contents of the educational animation selected by

the user. The third web format, called Plain, is an HTML

page navigable by means of a local index and the scroll

bar, where all the contents are available. Finally, the fourth

format, called Star, is intended to enhance flexibility of user

interaction. Here the user selects the contents that will be

simultaneously visible and distributes them in the screen,

Fig. 1 is an example of this web format as well.

B. Producing Educational Animations with WinHIPE

We have paid special attention to achieve an effortless

animation production process in WinHIPE. An effortless

approach allows the students to produce animations focusing

on educational tasks rather than on technical issues. Thus,

animation construction can be used as an active educational

task which could improve students learning outcomes.

The construction of our educational animations is divided

into two phases: generation of the program animation (the

graphical representation of the execution of a program)

and generation of the educational animation (the web-based

materials).

The phase of program animation generation starts with the

user typing the source code of the program and ends with a

sequence of discrete visualizations of some execution steps

of the program.

The execution model of the functional programming para-

digm is term rewriting. In this model, running a program

consists in a process of rewriting a starting term, the

main expression, step by step, until no rewriting can be

done. The basis of program animations in WinHIPE are

the graphical representation of these execution steps, also

called visualizations. The contents of visualizations can be

pretty-printed text and/or graphical representations of data

structures, i.e. binary trees and lists. These visualizations are

automatically produced by WinHIPE. Then, the user selects

which visualizations will form part of the final program

animation. She/he can easily complete this task with a

specially designed interface, also integrated within the IDE.

This interface provides simultaneously the user with a global

view of the execution process and a detailed view of a

specific execution step selected by the user. Finally, the

program animation is displayed in WinHIPE as a sequence

of discrete visualizations. Therefore, the generation of theprogram animation is quite similar to the program develop-

ment process (edit-compile-execute), where additional tasks

are assisted by specially designed interfaces.

The educational animation is also generated through the

user interface provided by WinHIPE. Thus, the user must

provide the information corresponding to textual contents

(title, problem description, and algorithm explanation) and

look contents (appearance on the web).

Notice that user interaction is limited to building the

program animation, explaining the problem and its solution,

and specifying the web look of the animation. Any additional

processing is automatic. A final, automatic step generates the

textual contents (HTML code) and the web look information

(CSS code) and adds them to the program animation (images

corresponding to the visualizations). Therefore we have an

effortless educational animation construction process totally

integrated into the development environment.

IV. EVALUATION

In this evaluation we investigate the long-term effects of

the previously described treatments.

A. Subjects and Experimental Design

The participants where 132 novel students enrolled in a

Programming Languages Foundations course placed in the

first year of CS degrees. The evaluation took place during

the second half of the course, which was focused on the

functional programming paradigm. Students participation

was almost voluntary, and incentive based. Students can

slightly improve their grades (0.25 out of 10) only if they

pass the course exam. We have discarded the students who

have previous knowledge about the functional paradigm.

Our study has two treatment conditions for the viewing

(VG, n=50) and constructing tasks (CG, n=40) and one

control condition for the typical approach tasks (TG, n=42).

Students were randomly assigned to the treatments/control

conditions. Thus, the independent variable is the engagement

level used in each group. The dependent variables are:

knowledge acquisition, measured with the term exam grades,

attitude towards the subject, measured with the exam taking

rate, and opinion about the animations, measured with aquestionnaire.

B. Tasks and Protocol

At the beginning of the course, students were informed

about the evaluation. Previously to the treatments appli-

cation, students got familiar with the WinHIPE program-

ming environment. Also they were trained in the facilities

provided by WinHIPE related with the tasks they have to

complete, view or construct animations.

The treatments were applied during the sessions corre-

sponding to the topics Prefix and infix operators, User

defined data types and Recursive data types. For each

topic, the protocol consisted in two parts: theoretical ses-sions, and a lab session. Theoretical sessions of all groups

followed the typical teaching methodology, but sessions of

VG and CG were enhanced with the use of animations for

examples and simple exercises. The lab sessions of each

group consisted different types of tasks but using the same

problems/cases, all of them selected from the course book.

The students in the control group (TG) worked on solving

problems proposed by the teacher. They were allowed to use

28


4/5

WinHIPE without its visualization facilities. The students

attention were focused on understanding the description of

the problem and coding the solution. The main task of the

vision group (VG) was viewing animations constructed by

the teacher. For each topic they have a list of animations

in a web page. They had to view them ensuring that they

understood them. The students attention were focused on

understanding the description of the problem and its solution

using the animation and the textual explanations. Finally, the

main task of the construction group (CG) was constructing

animations following the construction treatment described in

the previous section. They had available through the web

all needed materials: the description of the problem and

the source code. Construction tasks consisted in studying

the program so they can select appropriate input data,

execute the program and generate the animation using the

visualization facilities provided by WinHIPE. Actually, the

students effort were focused on producing a representative

animation choosing suitable input data and selecting the

important execution steps and writing a correct explanation

of the solution. The three groups ended their lab sessions

working on an exercise sheet, equal for the three groups,

with theoretical questions and problems.

In summary, the students participated in three evaluative

sessions, completing three groups of tasks and three exercise

sheets. At the end of the last lab session, students from all

the groups were asked to fill in the opinion questionnaire.

V. RESULTS OF THE EVALUATION

In this section we detail the results of the experiment. We

have studied the existing differences among the three groups.

We have used ANOVA and t-student tests for normal data

sets and, Kruskal-Wallis X

2

and Mann-Whitney U for non-normal data sets.

A. Knowledge Acquisition

Knowledge acquisition was measured in terms of the

grades of the term exam. We have analyzed these data

from two points of view, scalar and categorical. From a

scalar point of view, the grades were standardized to the

range [0,1]. The term exam grade (TEG) was divided into

two parts, labs exercises (L, 25%) and theoretical exercises

(75%). Theoretical exercises consisted of a theoretical ques-

tion (T, 33.3%) and a problem (P, 66.6%). The following

formula was used to compute TEG(Term Exam Grade):

TEG(L,T,P) = 0, 25 L + 0, 75 ((T+ 2 P)/3)No significant differences among the groups have beendetected, see table I. However, differences for the theoretical

question seem to be almost significant p = .057 and nearto significant for the term exam grade p = .084. A detailedanalysis of both cases shows two significant differences be-

tween the VG and TG groups, both in favour of VG, regard-

ing the theoretical question (20.39%, U = 565.5, p = .028)and the term exam grade (12.72%, t(61) = 2.028, p = .046).

Table ISCALAR ANALYSIS OF TERM EXAM GRADES

Average grades for Significant differences testCG VG TG among groups

L .5486 .65 .525 2(2, 95) = 1.77, p = .412T .7122 .7233 .5194 2(2, 95) = 5.74, p = .057P .3954 .3707 .2808 2(2, 95) = 3.92, p = .141

TEG .5129 .5287 .4015 F(2, 92) = 2.528, p = .084

Scalar grades were mapped to four categorical grades:

failure, fair, good and excellent. Table II shows their dis-

tribution. We have detected significant differences of these

grades, 2(6, 95) = 13, 408, p = .037. The most importantdifference is the number of failures, less than 47% in the

CG and VG groups and more than 61% in the TG group.

B. Students Attitude towards the Subject

We measured this variable with the exam-taking rate

for each group. Table III shows the exam-taking rates for

all the students enrolled in the subject and for those whoparticipated in the evaluation. While the rates for all the

students are similar in the three groups, i.e. the rates are

independent from the groups (p = .649), the rates for par-ticipants are almost dependent from the groups (p = .057). Adetailed study shows that there exist significant differences

between CG and TG (U = 1223, p = .022) of 20.71%,and almost significant differences between CG and VG

(U= 1348.5, p = .054) of 17%, always in favor of CG.

C. Students Opinion about the Animations

We collected students opinion about animations with a

questionnaire. This questionnaire was only applicable to the

groups that used animations. It consisted in five statementsfollowing a Likert scale of five values (being 5 the best

and 1 the worst). The statements were: (1) animations have

helped me in understanding the concepts, (2) animations are

easy to construct, (3) animations are easy to use (play/view)

and (4) animations are useful. Note that statement (2) was

only answered by students from CG while statement (3) was

Table IICATEGORICAL ANALYSIS OF THE TERM EXAM GRADES

Failure Fair Good ExcellentTEG [0.0, 0.5) [0.5, 0.7) [0.7, 0.9) [0.9, 1.0]

CG 43.20% 35. 15% 21. 65% 0.0%

VG 46.40% 25.60% 14.00% 14.00%TG 61.10% 22.20% 16.70% 0.0%

Table IIIEXA M-TAKING RATE ANALYSIS

All the students Just participants

CG 65.5% (78/119) 85.00% (34/40)

VG 60.8% (73/120) 68.00% (34/50)

TG 60.4% (90/149) 64.28% (27/42)

2(2, 388) = .865, p = .649 2(2, 132) = 5.73, p = .057

29


5/5

Table IVANSWERS TO THE QUESTIONNAIRE

% of answers in [4-5] CG VG Total (CG & VG)

Help understanding 86,2% 1 00% 93%

Ease of construction 100% NA 100%

Ease of use NA 100% 100%

Usefulness 83,4% 100% 91,5%

only answered by students from VG. Table IV summarizes

the students answers to these questions.

V I . CONCLUSION

In this study we have analyzed the educational impact

of two uses of functional program animations, vision and

construction tasks. Both of them were compared to a typical

teaching methodology where animations are not used.

Taking into account the Learning by Doing theory, Hund-

hausen et al.s [2] conclusions and the Engagement Theory

[1], construction tasks should have overcome the other two

approaches. Our results only support partially this assump-tion.

We have considered knowledge acquisition from two

points of view: scalar and categorical. From the scalar

point of view, only the vision approach outperformed the

typical approach, although grades from the construction

approach were quite similar to the vieweing one. From

the categorical point of view, the use of animations, either

viewing or constructing, decreased the fail-rate more than

14% with respect to the typical approach. Students from the

construction approach showed to have better attitude towards

the subject than the rest of students. Their exam-taking rate

(85%) is significantly higher than the one from the typical

aproach (64.28%). In addition, the students exam-takingrate from the viewing approach (68%) was quite similar to

the typical approach. Finally, the students opinion about

the animations is clearly positive. Most of the students

(between 83.4% and 100%) who used animations think that:

animations helped them to understand concepts, animations

are easy to use and construct and animations are useful.

All of them are desirable effects on students learning pro-

cess, therefore both approaches, viewing and constructing,

should be integrated in the teaching methodology.

ACKNOWLEDGMENT

This work was supported by the II Teaching Innovation

Projects programme of the Rey Juan Carlos Universityand project TIN2008-04103/TSI of the Spanish Ministry of

Science and Innovation.

REFERENCES

[1] T. Naps, G. Roling, V. Almstrum, W. Dann, R. Fleis-cher, C. Hundhausen, A. Korhonen, L. Malmi, M. McNally,S. Rodger, and J. Velzquez-Iturbide, Exploring the role ofvisualization and engagement in computer science education,SIGCSE Bull., vol. 35, no. 2, pp. 131152, june 2002.

[2] C. Hundhausen, S. Douglas, and J. Stasko, A meta-studyof algorithm visualization effectiveness, J. Visual Lang.Comput., vol. 13, no. 3, pp. 259290, 2002.

[3] J. Urquiza-Fuentes and J. A. Velazquez-Iturbide, A survey ofsuccessful evaluations of program visualization and algorithmanimation systems, ACM TOCE, vol. 9, pp. 9:19:21, 2009.

[4] C. Pareja-Flores, J. Urquiza-Fuentes, and J. A. Velazquez-Iturbide, WinHIPE: an IDE for functional programmingbased on rewriting and visualization, ACM SIGPLAN Not.,vol. 42, no. 3, pp. 1423, 2007.

[5] R. Mayer, Multimedia Learning. Cambrige Univ. Press, 2008.

[6] J. Stasko, Using student-built algorithm animations as learn-ing aids, SIGCSE Bull., vol. 29, no. 1, pp. 2529, 1997.

[7] J. Urquiza-Fuentes and J. Velazquez-Iturbide, An evaluationof the effortless approach to build algorithm animationswith WinHIPE, Electronic Notes in Theoretical ComputerScience, vol. 178, pp. 313, 2007.

[8] C. Hundhausen, Integrating algorithm visualization technol-ogy into an undergraduate algorithms course: Ethnographicstudies of a social constructivist approach, Comp. Educ.,vol. 39, no. 3, pp. 237260, 2002.

[9] C. Hundhausen and J. Brown, Designing, visualizing, anddiscussing algorithms within CS1 studio experience: Empiri-cal study, Comp. Educ., vol. 50, no. 1, pp. 301326, 2008.

[10] T. Hubscher-Younger and N. Narayanan, Dancing hamstersand marble statues: characterizing student visualizations ofalgorithms, in Proc. of the 2003 ACM Symposium on Soft-ware Visualization. New York, NY, USA: ACM Press, 2003,pp. 95104.

[11] A. Lawrence, Empirical studies of the value of algorithm

animation in algorithm understanding, Ph.D. dissertation,Dep. of Computer Science, Georgia Inst. of Technology, 1993.

[12] C. Kann, R. Lindeman, and R. Heller, Integrating algorithmanimation into a learning environment, Comp. Educ., vol. 28,no. 4, pp. 223228, 1997.

[13] M. Crosby and J. Stelovsky, From multimedia instructionto multimedia evaluation, J. Educational Multimedia and

Hypermedia, vol. 4, pp. 147162, 1995.

[14] C. Kehoe, J. Stasko, and A. Taylor, Rethinking the evaluationof algorithm animations as learning aids: an observationalstudy, Int. J. Hum.-Comput. St., vol. 54, no. 2, pp. 265284,2001.

[15] A. Kumar, Results from the evaluation of the effectivenessof an online tutor on expression evaluation, SIGCSE Bull.,vol. 37, no. 1, pp. 216220, 2005.

[16] J. Velazquez-Iturbide, C. Pareja-Flores, and J. Urquiza-Fuentes, An approach to effortless construction of programanimations, Comp. Educ., vol. 50, no. 1, pp. 179192, 2008.

[17] J. Nielsen, Designing Web Usability. New Riders Pub., 1999.

30

a long-term evaluation of educational animations of functional programs

Documents