exploring differential levels of feedback in digital ...€¦ · digital learning objects. while...
TRANSCRIPT
EXPLORING DIFFERENTIAL LEVELS OF FEEDBACK IN
DIGITAL LEARNING OBJECTS
By
Todd Michael Sumner
A thesis submitted in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
in
Psychological Studies in Education
Department of Educational Psychology
University of Alberta
© Todd Michael Sumner, 2017
Running Head: FEEDBACK IN DIGITAL LEARNING OBJECTS ii
Abstract
The paper explores the effect of embedding differential levels of elaborated feedback into digital
learning objects. The effect of the embedded feedback is also considered in relation to computer
experience and learner characteristics. Three digital learning objects were developed for this
study and were based on three Calculus topics that were common to all of the participants in
their post-secondary Engineering courses. Three experiments were conducted using the three
separate digital learning objects and participants within each of the digital learning object groups,
were randomly assigned to one of three different treatment conditions; simple feedback, positive
feedback and negative feedback. The participants used the digital learning objects as regularly
scheduled activities in their classroom activities with the learning objects and subsequent post-
tests delivered through the Moodle Learning Management System. Results were analysed using a
two-way analysis of variance (ANOVA) and indicated that there was no significant difference
between simple, positive and negative feedback directly, however, when analyzing the results in
relation to computer experience, it was found that test score results for participants receiving
positive and negative feedback were significantly higher for participants with high computer
attitude. This study is expands on research on feedback on the use of feedback in the learning
context of digital learning objects and in relation to the learner characteristic of computer
attitude. As Hatziapostolou and Paraskakis (2010) have noted, feedback is an important
component in all learning contexts, and as Van der Kleij, Feskens and Eggen (2015) have
suggested, there has been little further experiments in feedback should also account for the
learning context. Furthermore, they also suggest that additional research in feedback should
include learner characteristics.
Running Head: FEEDBACK IN DIGITAL LEARNING OBJECTS iii
Preface
This thesis is an original work by John Doe. The research project, of which this thesis is a
part, received research ethics approval from the University of Alberta Research Ethics Board,
Project Name “EXPLORING THE EFFECTIVENESS OF DIFFERENTIAL LEVELS OF
FEEDBACK IN DIGITAL LEARNING OBJECTS”, No. Pro00045923, January 2,2015. The
research project, also received research ethics approval from the Northern Alberta Institute of
Technology Research Ethics Board, No. 2015-02, January 12, 2015.
This thesis is an original work by Todd Sumner. No part of this thesis has been
previously published.
Running Head: FEEDBACK IN DIGITAL LEARNING OBJECTS iv
Acknowledgements
Doctoral supervisor: Dr. Patricia Boechler
Supervising committee members: Dr. Florence Glanfield, Dr. Mike Carbonaro
FEEDBACK IN DIGITAL LEARNING OBJECTS v
Table of Contents
Abstract ........................................................................................................................................... ii
Preface............................................................................................................................................ iii
Acknowledgements ........................................................................................................................ iv
Table of Contents ............................................................................................................................ v
Figures............................................................................................................................................ vi
Tables ............................................................................................................................................ vii
CHAPTER ONE – INTRODUCTION ........................................................................................... 1
Background ................................................................................................................................. 1
Problem ....................................................................................................................................... 4
Purpose ........................................................................................................................................ 5
Research Questions ..................................................................................................................... 6
CHAPTER TWO – REVIEW OF THE LITERATURE ................................................................ 7
Introduction ................................................................................................................................. 7
Feedback Defined ....................................................................................................................... 9
Feedback Research ................................................................................................................... 11
Feedback in Online Courses ..................................................................................................... 21
Feedback and Digital Learning Objects .................................................................................... 34
Digital Learning Objects Defined ............................................................................................. 35
Previous Research on Digital Learning Objects ....................................................................... 37
Learning and digital learning objects ........................................................................................ 38
Conclusion ................................................................................................................................ 47
CHAPTER THREE – METHODOLOGY ................................................................................... 49
Introduction ............................................................................................................................... 49
Research Hypotheses ................................................................................................................ 49
Variables ................................................................................................................................... 49
Participants ................................................................................................................................ 52
Setting ....................................................................................................................................... 53
Research .................................................................................................................................... 54
Procedures ................................................................................................................................. 55
Data Analysis ............................................................................................................................ 75
Assumptions and Limitations ................................................................................................... 75
CHAPTER FOUR – FINDINGS .................................................................................................. 77
Introduction ............................................................................................................................... 77
Demographic Data .................................................................................................................... 77
Tests and Data Collection Methods .......................................................................................... 79
Missing Data ............................................................................................................................. 83
Finding – Independent Variable 1: Difference in learning based on the levels of feedback .... 84
Finding – Independent Variable 2: Difference in learning with differential feedback based on
participants’ computer experience ............................................................................................ 87
FEEDBACK IN DIGITAL LEARNING OBJECTS vi
Other Analyses ........................................................................................................................ 100
Levels of Feedback and Guided Practice Time on Task ........................................................ 105
Participant comments .............................................................................................................. 105
Validity and Reliability ........................................................................................................... 108
Conclusion .............................................................................................................................. 111
CHAPTER FIVE – DISCUSSION ............................................................................................. 112
Findings and Interpretations based on the levels of feedback ................................................ 114
Difference in learning with differential levels of feedback based on participant’s computer
experience ............................................................................................................................... 117
Difference in learning with differential feedback based on other factors ............................... 118
Participant Comments ............................................................................................................. 119
Recommendations ................................................................................................................... 120
Researcher’s Reflections ......................................................................................................... 122
Suggestions for Further Research ........................................................................................... 124
Summary and Conclusions ..................................................................................................... 124
REFERENCES ........................................................................................................................... 126
APPENDIX 1 .............................................................................................................................. 134
Course Instructor: INFORMATION LETTER and CONSENT FORM ................................ 134
APPENDIX 2 .............................................................................................................................. 137
Student Participant: INFORMATION LETTER and CONSENT FORM ............................. 137
APPENDIX 3 .............................................................................................................................. 140
Pre-study Survey ..................................................................................................................... 140
APPENDIX 4 .............................................................................................................................. 142
Post-Test Questions ................................................................................................................ 142
APPENDIX 5 .............................................................................................................................. 148
Participant Comments ............................................................................................................. 148
APPENDIX 6 .............................................................................................................................. 151
Summary Review and Full Disclosure ................................................................................... 151
Figures
Figure 1 : A Model of Formative Assessment and Feedback ....................................................... 14
Figure 2: Lesson Time/Screen Time Screen Shot......................................................................... 60
Figure 3: Differentials Interactive Object - Part 1 ........................................................................ 63
Figure 4: Differentials Interactive Object – Part 2........................................................................ 63
Figure 5: Areas by Integration example - part 1 ........................................................................... 64
Figure 6: Areas by Integration example - part 2 ........................................................................... 65
Figure 7 Areas by Integration example - part 3 ............................................................................ 65
Figure 8: Areas by Integration example - part 4 ........................................................................... 66
Figure 9: Simple Feedback example ............................................................................................. 68
Figure 10: Positive feedback example .......................................................................................... 68
FEEDBACK IN DIGITAL LEARNING OBJECTS vii
Figure 11: Negative feedback example......................................................................................... 69
Figure 12: Pre-Study Questionnaire module................................................................................. 70
Figure 13: Digital learning object module ................................................................................... 70
Figure 14: Post - study questionnaire module............................................................................... 70
Figure 15: Structure of Moodle Module ....................................................................................... 74
Figure 16: Age Distribution .......................................................................................................... 79
Figure 17: Marginal Means for the Application DLO .................................................................. 85
Figure 18: Marginal Means for the Evaluation DLO .................................................................... 86
Figure 19: Marginal Means for the Analysis DLO ....................................................................... 87
Figure 20: Frequency Distribution of Video Game Usage ........................................................... 90
Figure 21: Video Game Usage Correlations ................................................................................. 91
Figure 22: Application DLO (negative feedback condition) Scatter plot – Video Game Usage . 91
Figure 23: Evaluation DLO (positive feedback condition) Scatter plot – Video Game Usage .... 92
Figure 24: Frequency Distribution of Social Networking Usage ................................................. 94
Figure 25: Social Media Usage Correlations ................................................................................ 94
Figure 26: Frequency Distribution of Computer Attitude ............................................................ 96
Figure 27: Frequency Distribution of Age .................................................................................. 102
Figure 28: Application DLO – Marginal Means for Feedback ................................................... 116
Figure 29: Evaluation DLO – Marginal Means for Feedback .................................................. 116
Figure 30: Analysis DLO – Marginal Means for Feedback ....................................................... 117
Tables
Table 1: Sample Size Calculations ............................................................................................... 53
Table 2: Age Distribution ............................................................................................................. 78
Table 3: Gender............................................................................................................................. 79
Table 4: Post test summary analysis ............................................................................................ 82
Table 5: Post test question analysis .............................................................................................. 82
Table 6: Age Correlations ........................................................................................................... 103
Table 7: Lesson Time Correlations ............................................................................................. 104
Table 8: Guided Practice Time Correlations .............................................................................. 104
Table 9: Themed Participant Comments..................................................................................... 108
FEEDBACK IN DIGITAL LEARNING OBJECTS 1
CHAPTER ONE – INTRODUCTION
Background
There are many challenges involved in designing effective online courses with some of
these challenges arising from the fact that designers must create an environment that responds to
the students’ needs in terms of providing effective feedback. In a face-to-face classroom setting,
feedback is typically instructor driven in that the instructor controls the timing, type, frequency,
and learning context of the feedback. An instructor in a face-to-face classroom can use feedback
to adapt a lesson to the needs of a class or an individual. In an online learning environment,
however, the timing, type and frequency of feedback can present both challenges and advantages
when developing online courses. Bonnel (2008) has noted that rich and rapid feedback is
essential for online courses given that students often feel abandoned by instructors without
adequate feedback, and students and instructors typically do not often communicate in real time,
that is, they often communicate asynchronously. Race (2001) has outlined a number of types of
instructor driven feedback in both face-to-face and online classes and describes e-mail,
discussion groups and online conferencing as ways for instructors to provide feedback to
students in online settings. Race also outlines a number of significant challenges to instructor
driven feedback in online courses in terms of timing and frequency since online instructor driven
feedback needs to be delivered, initiated and mediated by the instructor across a distance and
often asynchronously. There are, however, more opportunities for student driven feedback in
online courses where students, rather than the instructor, control the timing, frequency, type and
learning context of the feedback. Van der Kleij, Feskens and Eggen (2015) offer that in a
computer environment, standardized feedback can be generated to the students’ responses. This
allows instruction to be continuously adapted to the learners and make the feedback timelier.
FEEDBACK IN DIGITAL LEARNING OBJECTS 2
Brown, Lovett, Bajzek and Burnette (2006) outline a number of types of student driven feedback
such as interactive animations, simulations, low stake online quizzes, and mini tutors. In
general, they offer that student driven feedback could best be defined as feedback that is initiated
by actions students take while interacting with software or web pages and where the instructor
may or may not provide additional feedback. Race (2001) adds that online student driven
feedback can be designed so that instructors can create detailed feedback that is geared towards
the most likely mistakes or misconceptions that students have.
While feedback has consistently been shown to be invaluable in online courses (Bonnel,
2008; Miller, Doering, & Scharber, 2010, Van der Kleij, Feskens & Eggen, 2015), there is still a
lack of good empirical studies exploring the number factors that influence the effectiveness of
online feedback (Miller, Doering & Scharber, 2010; Van der Kleij, Feskens & Eggen, 2015). It
has been shown that online instructional designers often place feedback in the last phase of their
designs, if they do at all, with feedback typically being used for online quizzes and tests (Miller,
Hooper, Rose, & Montalto-Rook, 2008). Also, designers opt for instructor-based feedback such
as e-mail and conferencing since it is easier and less expensive to implement since most students
have access to e-mail, and Learning Management Systems (LMS) typically have conferencing
utilities built into the system. Developing effective, student-driven, online feedback also faces
significant obstacles such as lack of time, limited technical skills, fear of technology, insufficient
access, and a lack of understanding about how to integrate technology into an educational setting
(Miller, Hooper, Rose, & Montalto-Rook, 2008). Additionally, the rapidly evolving nature of
computers and learning systems often makes it difficult to implement and keep online feedback
up to date.
FEEDBACK IN DIGITAL LEARNING OBJECTS 3
One online learning mechanism that can potentially help to overcome some of the
obstacles of including student driven feedback into the design of online courses is the use of
digital learning objects. While there isn’t a clear definition of digital learning objects that is
universally accepted (Gadanidis & Schindler, 2006; McGreal, 2004), most authors do agree upon
certain characteristics that are common to digital learning objects such as interactivity,
reusability, they must be self-contained, must be digital, and must be adaptable (Chitwood, May,
Bunnow, & Langan, 2002; Gadanidis & Schindler, 2006; Kay, 2007; Kay & Knaack, 2008a;
Muller, Buteau & Mgombelo, 2009; Nurmi & Jaakkola, 2006; Reece, 2016; Sims, 2000;Yacci,
2000). These characteristics are useful in describing the use of digital learning objects, however,
they offer little in terms of how to best design digital learning objects, and do not describe how
to best incorporate student driven feedback into digital learning objects to increase learning. One
of the reasons for this is that even though many authors see digital learning objects as effective
online learning tools, there is a lack of empirical evidence with respect to the design and
effectiveness of digital learning objects to increase learning (Kay, 2007; Kay & Knaack, 2008a ;
Nurmi & Jaakkola, 2006). Kay (2007) and Kay and Knaack (2008b) found few studies related to
the effectiveness and design of digital learning objects with most of the studies being qualitative.
Only a handful of the studies had a quantitative design, however, they noted concerns in the
design of the studies that affected the reliability and validity of many of them (Kay, 2007; Kay &
Knaack, 2008b). Additionally, many studies had small sample sizes which made it difficult to
generalize the results to a larger population. In terms of interactivity and feedback, only one
study examined different levels of interactivity, with no studies looking at the effectiveness of
incorporating feedback into the design of digital learning objects (Kay, 2007; Kay & Knaack,
2008b).
FEEDBACK IN DIGITAL LEARNING OBJECTS 4
As Kay and Knaack (2008b) and Kay (2014) have noted, there has been little research in
regards to learner characteristics and the effectiveness of the use of digital learning objects. The
learner characteristics that have been examined are gender, age, subject ability, and computer
self-efficacy, however, these learner characteristics were not studied in relation to feedback.
With respect to gender, studies by Kay and Knaack (2008b) and Van Zele, Vandaele,
Botteldooren, and Lenaerts (2003) have shown that there is no observable difference between
males and females in regards to attitude when using digital learning objects. Kay and Knaack
(2008b) did note, however, that the difference in attitude and performance for computer related
tasks between males and females was significant in the early ‘90’s, but they also indicate that
recent research has shown that it has significantly decreased. Lastly, in terms of the effect of
computer self efficacy and performance using digital learning objects, no current research was
found. Kay (2007) does note, however, that this could be supported by the substantial body of
research done in the area of computer self-efficacy and computer related behaviors. One study
by Lim et al (2006) did find that students who were not comfortable with computers did tend to
use digital learning objects less.
Problem
Van der Kleij, Feskens and Eggen (2015) have identified that there remains a number of
gaps in the literature in the use of feedback in online settings, and specifically in the areas of
feedback type, learner characteristics and learner context. When looking at digital learning
objects as the learning context, Nurmi and Jaakkola (2006) have noted that there continues be
enthusiasm towards designing and using digital learning objects to support online learning.
However, as already noted, there is a lack of empirical studies that have investigated the design
of digital learning objects in relation to increasing learning with most studies focusing on the
FEEDBACK IN DIGITAL LEARNING OBJECTS 5
technological aspects of the design of learning objects (Watson, 2010). Kay (2007) adds that
digital learning objects have the potential to revolutionize online learning, but that it will not
occur until instructional use and pedagogy in relation to digital learning objects are explored and
evaluated. Kay further notes that research must look at how digital learning objects can be used
to create high quality instruction and that a detailed analysis of the features of digital learning
objects needs to take place. One potential area of research into digital learning objects would
involve embedding feedback into digital learning objects. With feedback being shown to be one
of the most effective ways teachers can increase learning, designing digital learning objects with
feedback that mimics a student-instructor interaction would seem logical. Embedding feedback
into digital learning objects has the potential to allow designers to create online learning
environments that allow student driven feedback to be incorporated directly into online learning,
and in fact, designers often build feedback into their digital learning objects. Additionally, of
further interest for this study would be to look at the interaction of learner characteristics and
levels of feedback in digital learning objects.
Purpose
The purpose of this study is to investigate the effect on student achievement scores of
student driven feedback embedded in digital learning objects designed for post-secondary
calculus students. Additionally, this study will investigate the influence of the learner
characteristic of computer experience in relation to student achievement of differential levels of
student driven feedback embedded in digital learning objects. This study will contribute to a
better understanding of the practices in the design of interactive digital learning objects and will
help to guide instructional designers in how to incorporate student driven feedback into digital
learning objects. Additionally, this study will contribute to the understanding of how feedback
FEEDBACK IN DIGITAL LEARNING OBJECTS 6
can be used in online environment to increase learning specifically within the context of digital
learning objects.
Research Questions
1. Is there a difference in post-secondary students’ learning based on the type of feedback they
encounter while using an online digital learning object as a basis of instruction in an
introductory calculus class?
Types of feedback in this study will be feedback that addresses faulty interpretations and not
a lack of understanding in the form of either positive or negative feedback as described by
Black and Wiliam (1998) and Hattie and Timperley (2007).
2. Is there a difference in post-secondary students’ learning based on the type of feedback they
encounter using an online digital learning object in relation to learner characteristics of
computer experience?
FEEDBACK IN DIGITAL LEARNING OBJECTS 7
CHAPTER TWO – REVIEW OF THE LITERATURE
Introduction
Since the advent of the personal computer in the late 1970’s, there has been a consistent
effort to increase the use of computers in education (McRobbie, Ginns & Stein, 2000). This is
supported by a report on students, computers and learning by the organization for economic
cooperation and development (OECD) which shows that computer use in education has
consistently increased between 2009 and 2012 (OECD, 2015). Kay (2007) and the OECD
(2015), however, report that technology has had a minor impact on student learning, and notes
that the problem with the use of technology in the classroom is due to a number of obstacles that
have prevented the successful implementation of technology. These include a lack of time,
limited technical skills, fear of technology, insufficient access, and a lack of understanding about
how to integrate technology into an educational setting for both learners and teachers. This is
further supported by the OECD report that argues that schools must encourage and support
changes in policies, curriculum and training for integrating computers into schools (OECD,
2015). They further argue that teachers must create new resources for integrating computers into
learning. One area of computer integration into education that faces these obstacles while
continuing to grow at an incredible pace is online education. That is, courses delivered online
using Learning Management Systems (LMS). There has been a significant amount of research
into the implementation of online courses with practices developed to address a number of these
obstacles (Kay, 2007; Miller, Doering & Scharber, 2010); however, one area that is often
overlooked in the development of online courses is feedback that is given to students about their
learning (Diane & Richards, 2004; Hattie, Biggs & Purdie, 1996). Hatziapostolou and
Paraskakis (2010) and Blair, Wyburn-Powell, Goodwin, and Shields (2014) note that feedback is
FEEDBACK IN DIGITAL LEARNING OBJECTS 8
important in all learning contexts, online included, but Diane and Richards (2004) and Miller,
Doering and Scharber (2010) point out that instructional developers usually fail to include
feedback options in the design of online learning environments. They further note that online
courses are typically designed with summative assessments primarily with little or no formative
assessment which could allow for effective feedback.
This review will focus specifically on the research and best practices in feedback in
online education and is divided into three sections. The first section examines feedback in
learning focussing specifically on face-face settings and will explore a definition of feedback in
the literature followed by a general overview of feedback in relation to timing of feedback, type
of feedback, learning context, usefulness of feedback and the ability of feedback to motivate.
The second section of the literature review will focus on feedback in online learning and looks at
instructor driven feedback versus student driven feedback. A number of factors relating to
student driven feedback are more fully explored with feedback type being the focus. The third
section of the literature review examines feedback in relation to the specific learning context of
digital learning objects. This section begins with a general overview of feedback and digital
learning objects followed by a section that explores what digital learning objects are and a
general review of the literature of digital learning objects. The final portion of this section on
digital learning objects focuses on factors that impact learning with digital learning objects such
as attitude, design and learner characteristics. This also includes a comparative section on
computer assisted instruction. The literature review ends with a conclusion that includes
identified gaps in online feedback with an overview of the factors that will be explored in this
study to help fill in the identified gaps.
FEEDBACK IN DIGITAL LEARNING OBJECTS 9
Feedback Defined
When looking at the effectiveness of feedback one must first look at the definition of
feedback since how it is defined will affect how findings are interpreted. While the term
feedback is used and defined differently in various disciplines, a number of authors have defined
feedback focussing on various aspects of feedback with the definitions typically being
categorized in terms of gaining understanding or in terms of effect on learning.
Gaining Understanding
Hattie and Timberley (2007) define feedback very broadly as a consequence of
performance and conceptualized as information provided by a variety of sources (teachers, peers,
books, parents, self, and experience) in response to performance. Consequently, feedback could
be interpreted as any type of information that informs the student in response to his/her
performance. Other authors have a narrower definition of feedback and describe it as simply as a
way to gauge whether the understanding of a topic is correct (Diane & Richards, 2004; Fleming
and Levine, 1978)
Effect on Learning
A number of authors define feedback in terms of its effect. Rather than providing only
information to the student about formative assessments, the feedback must also lead to student
learning. That is, the student must reflect on and use the information in a way that increases
learning to be considered feedback (Bonnel, 2008; Sadler, 1989; Turmond & Wambach, 2004).
FEEDBACK IN DIGITAL LEARNING OBJECTS 10
For the purpose of this study, the following definition of feedback by Bonnel (2008)
describes feedback in terms of its effect and can readily be applied to feedback in online courses
and digital learning objects.
Feedback is defined as communication of information to the student (based on
assessment of a learning task) that helps the student reflect on the information, construct
self-knowledge relevant to learning and set further learning goals. Feedback, building on
assessment, allows students to gauge their progress, consider alternate learning strategies,
and project their own continued learning needs. For feedback to be successful, students
must reflect on and interact with the communicated information, thus taking an active
role in their own learning. (p. 290)
The emphasis of this definition on the effect on learning of feedback is more in line with the
concept of feedback being used as formative assessment which closely aligns with online
feedback in that it typically is used for formative purposes to increase the learning in an online
environment. Using this definition, there is little distinction between formative assessment and
feedback with respect to purpose. The main distinction between them is that formative
assessment is typically seen as more formalized purposeful assessment that helps students realign
their learning, while feedback can encompass formative feedback as well as any other type of
communication of information to the student that the student can use to help them take an active
role in their learning by enhancing learning and achievement (Nichol & MacFarlane-Dick,
2004). Consequently, this review will distinguish between formative assessment and feedback
by referring to feedback as any communication of information to the student that helps the
student take an active role in their own learning, while formative assessment is a more
FEEDBACK IN DIGITAL LEARNING OBJECTS 11
formalized assessment such as quizzes and assignments and can be thought of as a subset of
feedback.
Feedback Research
Feedback has traditionally been recognized as playing an important and significant role in
student learning (Black & Wiliam, 1998; Blair, A, Wyburn-Powell, Goodwin, & Shields, 2014;
Diane & Richards, 2004; Ghilay & Ghilay, 2015, Hattie, 2012; Hattie & Timperley, 2007;
Hatziapostolou & Paraskakis, 2010; Sadler, 1989; Yuliang, 2009). Hatziapostolou and
Paraskakis (2010) note that feedback is an important component in all learning contexts, and
offer that it serves a variety of purposes such as assessment of achievement, development of
competencies and understanding, and elevation of motivation and confidence. Essentially, as the
definition by Bonnel (2008) shows, feedback must help the student take an active role in their
learning. The definition, however, gives no insight into what constitutes effective feedback or
what feedback strategies should be used. As Van der Kleij, Feskens, and Eggen (2015) add, there
is “no accepted model of how feedback leads to learning” (p.476). When examining the current
literature on feedback, there continues to be a significant amount of research into formative
assessment, which is a form of feedback. However, Hattie and Timperley (2007) note that there
has been little systematic research into feedback in classrooms, even though the importance of
feedback is a recurring theme in many educational articles. Black and Wiliam (1998) in a meta-
analysis of a number of quantitative studies that included feedback and various forms of
formative assessment have reported large effect sizes (.4 to .7). However, these results have been
criticized by a number of authors because the diversity of the studies reviewed makes compiling
the results from the studies meaningless (Bennett, 2011; Van der Kleij, Feskens, & Eggen,
2015). Black and Wiliam (1998) also indicated that the number of quantitative studies with
FEEDBACK IN DIGITAL LEARNING OBJECTS 12
significant rigour to perform a meta-analysis is small and that the differences in the studies
would make any combination of the results meaningless. Black and Wiliam (1998) do contend,
however, that these quantitative studies are significantly rigorous and can stand on their own
within the purposes of the studies. Another meta-analysis on formative assessment and feedback
was conducted by Kingston and Nash (2011) which has also been criticized for its methodology
(Van der Kleij, F. M., Feskens, R. W., & Eggen, 2015). While the research into feedback does
not appear to be thoroughly explored, a reasonable understanding of what constitutes effective
feedback can be developed while acknowledging that there is still a need for further research
(Van der Kleij, F. M., Feskens, R. W., & Eggen, 2015).
Much of the research into formative assessment and feedback is based on work by Black
and Wiliam (1998) and Sadler (1989) who state that the “learner has to (a) possess a concept of
the standard (or goal, or reference level) being aimed for, (b) compare the actual level of
performance with the standard, and (c) engage in appropriate action which leads to some closure
of the gap” (p. 121). While Sadler (1989) was commenting primarily on formative assessment,
this would also apply to all forms of feedback. The important point that Sadler makes is that for
the feedback to be effective they must be aware of their standard for their learning and be able to
assess their performance when compared to the standard and then change their action
accordingly. Essentially, Sadler introduces the concepts of internal and external feedback with
internal feedback being the internal dialogue that the student has when comparing his/her
performance to a standard. Sadler notes that while many teachers give their students feedback
information in regards to how they compare to the standard, there is often a difficulty when the
students try to use the feedback because the student may not be able to understand the feedback
or the standard may not have been fully assimilated. That is, there may be something in how the
FEEDBACK IN DIGITAL LEARNING OBJECTS 13
feedback was delivered or in the understanding of the standard that prevents that student from
processing the feedback internally. This is reiterated by Black and Wiliam (1998), and they add
that the beliefs about one’s capacity to respond to feedback and the perceived risks involved in
responding in different ways also contribute to how students respond to feedback. Higgins
(2000) adds that students may simply be unable to understand feedback comments and correctly
interpret them because they are generally delivered in academic language which the students do
not understand. This is supported by Nicol and Macfarlane-Dick (2006). Blair, Wyburn-Powell,
Goodwin, and Shields (2014) further add that students also are unaware that feedback is intended
to help their development and learning and therefore do not use it or act on it.
Nicol and MacFarlane (2004) offer the following conceptual model in Figure 1 : A Model
of Formative Assessment and Feedback that synthesizes the ideas by Sadler (2004), Black and
Wiliam (1998) as well as others to help explain the feedback process. The primary focus of this
is that the student is actively involved in the feedback process. The model begins with the
teacher setting the task and then the students set their own goals based on their conceptions
which may or may not align with the teacher’s goals. The students then monitor their own
progress internally by comparing their progress to their goals and they may re-interpret the task,
adjust goals or adjust domain knowledge depending on the internal feedback. Additionally,
external feedback on performance might agree with, disagree or augment the student’s internal
goals and how they interpret the task leading them to re-evaluate how they interpret the task.
Nicol and MacFarlane (2004), argue that this model demonstrates that teachers should not just
focus on good feedback methods, but also on helping students develop strong and effective skills
of self assessment. This is also supported by York (2003) and Sadler (2004). In fact, Sadler
FEEDBACK IN DIGITAL LEARNING OBJECTS 14
(2004) contends that if students are to be able to compare their performance with a standard and
then take action, they must already possess some of the same evaluation skills as their teacher.
Figure 1 : A Model of Formative Assessment and Feedback
(Nicol & MacFarlane, 2004). p. 2
While a conceptual model of the feedback process is important in understanding how students
use feedback to increase learning, it is important to examine how feedback can best be used in a
classroom or in an online setting to increase learning. Feedback research was found that looked
at a number of factors that could impact the effectiveness of feedback such as timing of
feedback, type of feedback, learning context, and usefulness of the feedback. Each of these is
explored below.
Timing of Feedback
While there are a number of factors that have been investigated in regards to the effects of
feedback on learning, timing of the feedback has consistently been shown to be important if
feedback is to be effective (Brown, Lovett, Bajzek, & Burnette, 2006; Carless, 2006; Hattie &
FEEDBACK IN DIGITAL LEARNING OBJECTS 15
Timperley, 2007; Hatziapostolou & Paraskakis, 2010; Miller, Doering, & Scharber, 2010).
Hatziapostolou and Paraskakis (2010) argue that it is because timely feedback allows students to
be able to recall how they assessed tasks and allows them to apply it to future learning and
assessments. In regards to when feedback should occur, most research has shown that feedback
is more effective if it occurs immediately after the students’ responses (Brown et al., 2006;
Hattie, & Timperley, 2007; Kulik & Kulik, 1998). However, a number of studies have shown
that although immediate feedback is generally effective, it is not always necessarily so (Douglas,
Salter, Iglesias, Dowlman and Eri (2016). They have found that timing of feedback is also
dependent on the type of learning. They note that if the learning is for task acquisition then
immediate feedback is very effective, but if feedback is for learning that represents fluency
building then some delay of the feedback is beneficial (Hatziapostolou & Paraskakis, 2010;
Kulik and Kulik, 1998). Corbett and Anderson (2001) also add that immediate feedback is also
effective for procedure skills such as mathematics and programming. This is also supported by a
meta-analysis of feedback literature in computer based instruction by Azevedo and Bernard
(1995) who contend that the timing of feedback should be aligned with the outcome.
Specifically, if students are learning a new task that is difficult such as mathematics, procedural
or conceptual tasks, then immediate feedback is more appropriate. For simple tasks, delayed
feedback is suggested as immediate feedback could be seen as an intrusion or annoyance when
learning simple tasks.
Type of Feedback
Type of feedback has consistently also been shown to influence the effectiveness of
feedback, specifically in terms of whether the type of feedback is either positive or negative
FEEDBACK IN DIGITAL LEARNING OBJECTS 16
feedback (Black & Wiliam, 1998; Davis & McGowen, 2007; Hattie & Timperley, 2007).
However, the effectiveness of positive or negative feedback is contingent upon a number of
factors, but first, it is important to distinguish feedback from reinforcement and punishment.
Hattie and Timperley (2007) and Hattie (2012) have argued that praise, punishment, and other
rewards are not effective forms of feedback and, in fact, Deci, Koestner and Ryan (1999)
contend that rewards (praise) and punishments should not be thought of as feedback at all, but
instead, they should be thought of as contingencies to activities. Hattie and Timperley (2007)
support this position and argue that feedback should not be thought of as reinforcement since,
unlike reinforcement, it can be accepted, modified, or rejected. They further add that feedback is
different from reinforcement or punishment in that it contains information about the task while
reinforcement does not. Hattie (2012) adds that rewards combined with praise can impede the
effectiveness of feedback because it can prevent a student from hearing feedback because they
focus on the reward. Given that feedback should not be thought of as either reinforcement or
punishment, how then does positive or negative feedback influence learning? While most
studies show that positive feedback is generally more effective that negative feedback (Black &
Wiliam, 1998; Davis & McGowen, 2007; Hattie & Timperley, 2007), some authors have found
there are additional factors that may determine when one is more effective than the other. Van-
Dijk and Kluger (2001) argue that positive feedback is more motivating for tasks that people
want to do (i.e, they are committed to a goal) while negative feedback may better support
learning when people are not committed to it. They warn though, that increased learning in
response to negative feedback is often short lived and often leads to avoidance behaviour. Marie
(2016) did find that students reported that negative feedback was important for work that was
marked because it provided information on where they had made mistakes or gone wrong.
FEEDBACK IN DIGITAL LEARNING OBJECTS 17
However, positive feedback is likely to increase persistence and increase interest in an activity
(as cited in Hattie & Timperley, 2007). Black and Wiliam supports this and adds that not all
feedback in a classroom is beneficial, especially in discussions and questions as teachers often
are looking for specific responses to questions and may inhibit learning by impeding thoughtful
unanticipated responses. Another interesting finding regarding the effectiveness of positive
feedback is that studies have shown that positive feedback supports processes that are critical to
developing intelligence such as stimulating self-regulation (Davis & McGowen, 2007).
Shute (2008) expanded the discussion of types of feedback by identifying two broad
types of feedback as verification and elaboration. Verification feedback is simply confirming to
the student that an answer is correct or incorrect. This can include various forms of verification
feedback in what Van der Kleij, Feskens and Eggen (2015) summarize as knowledge of results
(KR) and knowledge of correct response (KCR). Knowledge of results (KR) is a form of
verification feedback back that simply confirms to the student whether an answer is correct or
incorrect, but can also include feedback such as error flagging which points out to the student
where the error was made. Primary to knowledge of results feedback is that no additional
information is provided to the student. Knowledge of correct response (KCR) feedback is
similar to knowledge of response feedback, however, its purpose is to provide the correct
answer, but it too does not provide additional information to the student. Elaboration, however,
is more varied, but primarily has the purpose of providing more information to the student by
explaining the response is correct or not. Elaborative feedback can include feedback such as
hints, extra information or explanation of the correct answer, explanation of misconceptions to
name a few. In relation to positive and negative feedback, Shute (2008) describes response
contingent feedback as a type of elaborative feedback “that focuses on the learner’s specific
FEEDBACK IN DIGITAL LEARNING OBJECTS 18
response. It may describe why the incorrect answer is wrong and why the correct answer is
correct” (p.160). Shute (2008) also identifies additional need for research in this area and
specifically with feedback that elaborates typical errors, or negative feedback.
Learning Context
In learning contexts, Diane and Richards (2004) contend that most faculty and
instructional staff often do not consider feedback during the curriculum design process and, that
during instruction, feedback is typically associated primarily with marking and assessment. This
results in feedback being used primarily as summative feedback and it therefore is delivered too
late in the learning process to help students improve their understanding. Hattie, Biggs and
Purdie (1996), however, in a meta-analysis of studies involving learning skills interventions, note
that feedback is most effective when it is designed into the learning process specifically if it
provides learning cues or reinforcement, and primarily if it is related to the learning goals.
Furthermore, Hattie and Timperley (2007) add that the learning context is also important when
embedding feedback into the learning process. They note that feedback should address faulty
interpretations and not a lack of understanding; in fact, they suggest that feedback that addresses
a lack of understanding can be threatening to a student.
Usefulness of Feedback
Another significant factor of feedback is that it must be useful to the students if they are
to use it to improve learning. While this may seem self evident, Black and Wiliam (2010) note
that most feedback and assessment emphasizes marks and grading, and not on giving useful
advice to help in the learning process. They suggest that when marking and grading is
FEEDBACK IN DIGITAL LEARNING OBJECTS 19
emphasized, it tends to foster competition which may have a negative impact on low achieving
students potentially leading them to believe that they lack ability. Additionally, Carless (2006)
notes that students often report that they are not satisfied with the feedback they receive and that
the feedback typically lacks information to help them improve, or is difficult to interpret. Black
and Wiliam (2010) instead recommend that feedback should focus on the usefulness of the
feedback to an individual student by focusing on the individual students work and how he/she
can improve rather than focussing on competition. This is also supported by Hatziapostolou and
Paraskakis (2006) who add that feedback must fit to each student’s achievement and tailored to
their individual strengths and weaknesses, but caution that overly detailed and too much
feedback can result in the feedback being too confusing and overwhelming to be useful.
Hepplestone and Chikwa (2014) also found that students’ use of feedback was increased when
connections were made between feedback they have previously received and the learning they
were involved in currently. They also found that students used and valued the feedback they
received, internalizing it for the future. This is supported by Marie (2016) who found that
students found feedback the most valuable when it was useful for future learning which is
supported by Small and Atree (2015) who note that students found the most useful feedback to
be feedback that closed the gap between what they submitted and what was expected of them.
Lastly, Brown et al. (2006) add that for feedback to be useful, it should also lead the students to
revisit the activity.
Motivational
A number of authors have suggested that feedback also serves as a significant motivating
factor in the learning process (Carless, 2006; Hattie & Timperley, 2007; Hatziapostolou &
FEEDBACK IN DIGITAL LEARNING OBJECTS 20
Paraskakis, 2010; Marie, 2016). While the other factors mentioned here are factors that
influence the effectiveness of feedback, motivation instead should be seen as both a factor that
influences its effectiveness and as a by-product of effective feedback. That is, motivation can be
seen as a quality of effective feedback given that feedback is motivational and will encourage
student engagement and persistence in a task (Hattie & Timperley, 2007). Hatziapostolou and
Paraskakis (2010) add that feedback can have either a positive or a negative effect on student
emotion and self-esteem. They recommend that feedback should be both empowering and
constructive to better increase student motivation on a task. As already noted, Van-Dijk and
Kluger (2001) have shown that the motivational aspect of positive feedback is significant for
tasks that students want to do and less motivational for tasks that students have to do. Thus the
motivational aspect of feedback is tied to how committed we are to a goal. If we are highly
committed to a goal then positive feedback is shown to be much more motivational, and will also
increase the likelihood we will persist in the activity and also that likelihood that there will be an
increase in measures of self efficacy. Carless (2006) offers an explanation for the influence of
positive feedback on student motivation. Carless notes that assessment is an emotional process
in which students make an emotional investment on an assignment or exam and expect some
return on the investment. If feedback is negative, it can be threatening to a student’s self-
perception and self efficacy. This is further supported by Hatziapostolou and Paraskakis (2010)
who have noted that a significant number of students do not collect feedback for a number of
reasons, one of which is avoidance in case of bad performance which may affect their self
perception. This is in contrast to what Marie (2016) found. Marie notes that students reported
that feedback that showed them where they were wrong were more motivated and less worried
about the course because they knew what was required.
FEEDBACK IN DIGITAL LEARNING OBJECTS 21
Feedback in Online Courses
While feedback has been shown to be an effective method of improving student learning,
the introduction of the computer and internet into educational settings has allowed for significant
opportunities for increases in the frequency, timing and types of feedback. Online course
designers have many options for feedback available to them with many more being made
available all the time. When looking at feedback in a face-to-face classroom setting, feedback
was typically instructor driven in that the instructor determined the timing, type of feedback,
frequency of feedback and the learning context of the feedback. As shown, to use feedback
effectively, the instructor had to be mindful of these factors, and the instructor had to be mindful
of the needs of the students. Little research seems to have been executed around student driven
feedback in face-to-face classrooms (Miller, Doering & Scharber, 2010). In online learning
environments, instructor driven feedback is also widely used, however, there is much more
opportunity for student driven feedback where the student determines the timing, frequency,
type, and learning context especially when the feedback is delivered through Learning
Management Systems that may also include adaptive feedback (Brown, Lovett, Bajzekand &
Burnette, 2006).
Instructor Driven Feedback in Online Courses.
Instructor driven feedback in online courses typically is very similar to feedback in a
face-to-face instructional setting. In both online and face-to-face instructional settings, feedback
is created by the instructor and dependent upon the instructor for timing and frequency, and both
share many of the same advantages and disadvantages. The difference between the two settings
is that online feedback is offered at a distance and can be delivered either synchronously or
FEEDBACK IN DIGITAL LEARNING OBJECTS 22
asynchronously. What this means for the student is that they are separated from their classmates
and instructor by space and, for asynchronous classes, by time. This separation offers unique
challenges and advantages for the students and instructors in online classes. These challenges
and advantages can apply to factors that are similar to those of a face-to-face classroom which
include timing and frequency of feedback, type of feedback, and message type. In addition to
these challenges and advantages, Small and Attree (2015) note that online students have a
limited relationship to the instructor and have a higher belief in the expertise of the instructor
which may positively impact how they receive feedback.
Timing and frequency of instructor driven feedback in online courses can present both
challenges and advantages (Leibold & Schwartz, 2015). Leibold and Schwartz (2015) have noted
that effective online, instructor driven feedback, is important for student learning and an essential
skill for instructors to develop. Additionally, Bonnel (2008) and Leibold and Schwartz (2015)
have noted that rich, timely and rapid feedback is considered essential for online courses and
Bonnel (2008) states that those students who do not receive frequent feedback often feel
abandoned in courses which may lead to attrition. Bonnel further notes, however, that faculty
often express concern over the amount of time needed to provide feedback to students in courses
while also questioning the effectiveness of the feedback. Jarvenpaaa and Leidner (1998) note
that students report that timely and frequent feedback from instructors is essential to help build
trust with the student and to give the student the feeling that someone is there. This is also
supported by Billings (2000) who notes that timely and frequent feedback helps overcome
feelings of isolation. Thurmond (2003) adds that it helps ensure students they are meeting the
course expectations and maintaining the correct pace and schedule. One of the biggest
difficulties in providing timely feedback in asynchronous courses is that students and instructors
FEEDBACK IN DIGITAL LEARNING OBJECTS 23
do not communicate in real time. However, feedback in asynchronous courses such as e-mail
and discussion forums can allow for more frequent and personalized feedback from the
instructor, and can allow for timely and frequent communication between the student and other
students (Bonnel, 2008). Cobb, Billings, Mays, and Canty-Mitchell (2001) suggest that feedback
such as e-mail and discussion forums in an online course may be beneficial because more
frequent and timely feedback allows students who are typically reluctant to ask questions or to
seek guidance to do so. Additionally, the timing of online feedback may allow students more
time to reflect on responses. However, they also suggest that it is important for instructors to let
students know when and how often to expect feedback from instructors.
Another important consideration in instructor driven feedback in online courses is how
the effectiveness of the feedback is affected by the type of feedback. When discussing feedback
in general, the question of the type of feedback centers on whether feedback is positive or
negative in nature. While this consideration is equally important in online education, the
discussion and implications are essentially the same as for a face-to-face classroom. When
looking at type of feedback in online education, many authors also focus on how the delivery
method might affect learning since the feedback is delivered at a distance. Hatziapostolou and
Paraskakis (2010) note there are numerous electronic and traditional ways for providing
feedback to students in an online environment. Race (2001) outlines a complete list of the types
of feedback in online and face-to-face instruction and lists advantages and disadvantages for all.
For online classes, Race outlines the two most common types of instructor driven electronic
feedback. These include e-mailed comments on students work in which the instructor provided
individual feedback on student’s work that was delivered to them by e-mail or submitted
electronically in a Learning Management System (LMS) or computer conference. The
FEEDBACK IN DIGITAL LEARNING OBJECTS 24
advantages Race lists for e-mailed feedback can be summarized as: a) it is convenient for both
the instructor and the student, it is private, b) students can refer back to the feedback again and
again, c) instructors can keep track of the feedback, d) students are free and more likely to reply
to the feedback, e) feedback can be tailored to the individual, and f) the instructor can take the
time to edit feedback before sending it. Disadvantages that Race lists can be summarized as: a)
the students may not have access to e-mail, b) they may not take electronic feedback as seriously
as face to face feedback, and c) there may be a delay between when the instructor sends the
feedback and when students read it. Another concern may also be that the interactive nature of a
face to face conversation is lost in an e-mail which means that the tone and quality of the
message may be lost (Bonnel, 2008). Bonnel suggests that there also needs to be thoughtful
consideration of the content of the message when sending any text-based feedback to students.
This idea is supported by Diekelmann and Mendias (2005) who argue that teachers need to
reflect on the meaning and significance of the electronic feedback that they deliver to their
students. Muirhead (2001) adds that the literature on social interaction in online education
would suggest that giving consideration to affective approaches such as using a student’s name,
building community and providing encouragement would help students be more receptive of
text-based electronic feedback.
Race (2001) also lists computer conferencing as another common type of instructor
driven electronic feedback and outlines a number of advantages and disadvantages in using it.
Computer conferencing can either be a synchronous method of interacting with students in a real
time environment that more closely approximates instructor/student interaction in a face-to-face
classroom or it might be an online discussion board that is not in real time where students post
and reply to feedback from the instructor and from each other. Race suggests that many of the
FEEDBACK IN DIGITAL LEARNING OBJECTS 25
advantages of this type of feedback are the same as for e-mail and suggests that e-mail should
still be used in conjunction with conferences when individualized or private feedback is
necessary, however, individual feedback can also be accomplished in conferencing as well. In
terms of the advantages unique to conferencing, Race offers that the instructor can provide more
general feedback to a large number of students at once, or individually so everyone can see each
others feedback, if appropriate. Students can then learn from this feedback and choose to reply
to each other, the class or to the instructor or choose to continue discussions through e-mail.
Disadvantages are that students may be less inclined to search through a large discussion board
for responses that apply to them. If the conference is done synchronously, then it may be
difficult to find times that all of the students are available especially if they are in geographically
diverse locations. Also, students may be less inclined to reply to feedback in an online
conference, however, e-mail is also an option for these students or a private discussion if the
conferencing system supports it.
Blignaut and Trollip (2003) have a broader definition of feedback which goes beyond
Bonnel’s (2008) definition of feedback to include all communication to students in an online
course and not just information provided to the student based on assessment of a learning task.
They suggest that feedback in online courses be categorized into six distinct message types.
While these are not necessarily factors in effective feedback of a learning task, it is essential for
online instructors to be mindful of the response types when providing feedback to help the
instructors better interact with the students. Blignaut and Trollip suggest that these types all
serve different purposes and that the integration of all six types is important in a successful
online course. The six types are grouped as either messages without academic content or
messages with academic content and are as follows:
FEEDBACK IN DIGITAL LEARNING OBJECTS 26
1) Messages without academic content.
a) Administrative: Feedback that provided general administrative content, such as
dates, profiles, formats, functionality of software and other organizational
aspects..
b) Affective: Messages that acknowledge learner participation and offer affective
support.
c) Other type: These include forums and message boards that offer open discussion.
2) Messages with Academic Content
a) Corrective: These are messages that correct a users content or response to a
posting
b) Informative: These postings that the instructor users to provide information to the
student on course content and provide individual feedback.
c) Socratic: Posting that ask the student reflective questions (p. 154)
Student Driven Feedback in Online Courses.
Instructor driven feedback in online courses is typically delivered, initiated and/or
mediated by the instructor and can be characterized as feedback that typically involves a
dialogue between a student and an instructor or between students mediated by an instructor.
Student driven feedback, however, can be classified as feedback generated by a website or
computer software such as Learning Management Systems (LMS) or Computer Assisted
Instruction (CAI) and can include pre-programmed computer feedback or adaptive feedback as
well as computer based formative assignments and quizzes. Hepplestone and Chikwa (2014) add
that technology can support feedback in student – driven feedback scenarios because it provides
quick feedback, increases access to feedback and gives easy storage of feedback. Brown, Lovett,
Bajzek and Burnette (2006) list a number of types of student driven feedback such as interactive
animations, simulations, low stake quizzes, and mini-tutors. While the options for student driven
feedback are significant, in each case they can be characterized as feedback that is initiated by
actions students take while interacting with software or web pages and the instructor may or may
not provide additional feedback.
FEEDBACK IN DIGITAL LEARNING OBJECTS 27
Race (2001) offers a list of advantages and disadvantages to using student driven
feedback, but the definition for computer based feedback Race offers is fairly narrow and applies
to pre-prepared feedback responses to structured self assessment questions in computer based
learning. The advantages Race suggests are:
• Students can work through computer-based learning materials at their own
pace, and within limits at their own choice of time and place.
• Feedback to pre-designed tasks can be received almost instantly by students,
at the point of entering their decision or choice into the system.
• Computer-based feedback legitimizes learning by trial and error, and allows
students to learn from mistakes in the comfort of privacy.
• Instructors can prepare detailed feedback in anticipation of the most likely
mistakes or misconceptions which they know will be common amongst their
students.
• Students can view the feedback as often as they need it as they work through
the package. (p. 11)
It should be further noted that each of these advantages is student driven or focused and the
instructor does not need to be involved in the feedback process, however, this does not preclude
the instructor from offering additional feedback. This is important for the reasons outlined
earlier in terms of helping the students feel that someone is “out there” and is available to offer
extra support. Race also offers the following disadvantages to using computer based feedback
which can also apply to any student driven feedback:
• The instructor cannot easily tell to what extent individual students are
benefiting from the feedback he/she has designed.
• Students who don't understand the feedback responses the instructor designed
may not be able to question the instructor further at the time, in the ways they
could have used with emailed or computer-conference-based feedback.
• The 'now you see it, now it's gone' syndrome can affect students' retention of
the instructor’s feedback messages, as students move quickly from one screen
of information to another in the package. (p. 11)
FEEDBACK IN DIGITAL LEARNING OBJECTS 28
To overcome these disadvantages of student driven feedback in online courses, it is necessary for
the instructor to also be involved in the monitoring of the students progress and to also offer
instructor driven feedback.
Research into the effectiveness of student driven online feedback has not been fully
explored in the literature (Miller, Doering & Scharber, 2010; Van der Kleij, Feskens & Eggen,
2015) which may be due to the fact that designing student driven feedback requires significantly
more time, cost, and expertise to develop. Regardless, there has been some research in a
number of areas that are unique to student driven online feedback which include preferred form
of feedback, frequency of feedback, adaptive feedback, and feedback type. Additionally there
are a number of authors who have suggested best practices for incorporating feedback (student
driven and instructor driven) into online course design (Azevedo & Bernard, 1995; Van der
Kleij, Feskens & Eggen, 2015).
Self-Regulated Learning and Feedback
In relation to self-regulated learning in an online setting, Wang (2011) states that “the
main advantage of e-Learning is that it overcomes the limits of time and space and provides
learners opportunities to perform self-directed learning” (p. 1802). This means that online
learning allows an opportunity for student driven feedback to support self-directed learning that
is more difficult to achieve in a face to face environment. This is further supported by Johnson
and Davies (2014) who state that online learning has “the capacity to promote the cyclical phases
of self-regulated learning including task comprehension and then planning, strategizing and
evaluating movement toward completion of the required task” (p. 5). While there is extensive
research in self-regulated learning, Nicol and Mcfarlane (2006) looked specifically at the role
FEEDBACK IN DIGITAL LEARNING OBJECTS 29
that formative assessment and feedback play in self regulated learning. They argue that the
purpose of formative assessment and feedback should be to allow students to become self-
regulated learners. That is, “students can regulate aspects of their thinking, motivation and
behaviour during learning” (p. 200). This is in contrast to what has traditionally been seen as the
purpose of formative assessment and feedback where it has usually been seen as instructor driven
where the instructor controls the feedback and consequently not self regulated learning (Nicol
and McFarlane, 2006). This is also problematic because it has been shown that when instructors
provide the feedback to the students, it is often complex and difficult for the student to interpret
(Hatziapostolou & Paraskakis, 2006; Nicol & McFarlane, 2006). Another issue that Nicol and
McFarlane note about instructor driven feedback is that feedback from instructors can regulate
the beliefs the students feel about themselves, motivation and how they learn. They argue that
self regulated feedback through a student driven or internal feedback process helps to overcome
these shortcomings by allowing students to have an active role in the feedback. Furthermore,
there is evidence that “learners who are more self-regulated are more effective learners; they are
more persistent, resourceful, confident and higher achievers (Nicol & McFarlane, 2006).
Pereira, Flores, Simão, and Barros (2016) and Zimmerman (2000) note that external feedback
can also be beneficial in self-regulated learning because it provides feedback about the
performance of the student which can help support students’ internal feedback by providing them
a mechanism to reflect on the learning and competencies. The study by Pereira, Flores, Simão,
and Barros (2016) further found that feedback provided during the learning process is perceived
by students to be more effective and more relevant than feedback found in the assessment tests.
They argue that feedback that is given during the learning process better enables self-regulation
of the learning by the students compared to feedback during assessments.
FEEDBACK IN DIGITAL LEARNING OBJECTS 30
Preferred form of feedback
Bower (2005) investigated whether allowing students to choose their preferred form of
feedback has a significant effect on learning. Bower allowed students to work through an online
module and then gave them access to an online practice facility with the feedback being one of
two forms, competitive or individualistic. Students were assigned to groups with one third
receiving their preferred method (either competitive or individualistic), one third receiving their
non-preferred method and one third receiving simple feedback. Bower found that there was a
significant improvement in test scores for students who received their preferred method of
feedback, while students receiving their non-preferred method actually had a negative impact on
their performance. While this study only examined preferred method in respect to one type of
student driven online feedback (low stake quizzes), it would be interesting to examine preferred
method for other types of student driven feedback such as simulations, interactive animations
and digital learning objects. It would also be interesting to examine the effect of preferred type
in which numerous types of student driven feedback was available to the student.
Frequency of feedback
Increased frequency of feedback has been shown to be an important factor in both face-
to-face and instructor driven feedback in online courses. In respect to student driven feedback, a
number of authors have shown that a high frequency of feedback also leads to a significant
increase in performance (Butler, Pyzdrowski, Goodykoontz, & Walker, 2008; Cassady, Budenz,
Anders, Pavlechko, and Mock, 2001; Klecker, 2007; Sonak, Suen, Zappe, & Hunter, 2002).
Bower (2005) suggests that online student driven feedback, though time consuming to
implement, can help lead to student mastery which might account for increases in performance.
FEEDBACK IN DIGITAL LEARNING OBJECTS 31
Adaptive or Tailored feedback
One type of student driven feedback that has consistently shown a significant positive
impact on student learning is adaptive or tailored feedback (Bower, 2005; Nguyen, Hsieh &
Allen, 2006). Adaptive or tailored feedback is implemented in LMS or computer assisted
instruction systems as well as digital learning objects when the computer gives feedback that is
directly tailored to a student’s response to a question or to an interaction with software. While
adaptive feedback has been shown to be very effective, it is often the most difficult to implement
by instructors due to constraints in time, cost and expertise, and it is also the most sensitive to
changes in technology and software.
Course design
When designing online courses, it is important to ensure that feedback is included in the
structure of the course (Bonnel, 2008; Miller, Doering, & Scharber, 2010), but often instructional
designers place feedback as the last phase of instructional design with feedback playing a
minimal role other than traditional quizzes and tests (Miller, Hooper, Rose, Montalto-Rook,
2008). Bonnel (2008) suggests that when designing a course, keeping the course goals in mind
allows for the natural integration of feedback, and once goals are defined, numerous ways to
provide feedback can be implemented. Miller, Doering and Scharber (2010) also suggest that the
designer should understand and consider the intricate details of when to use and when to avoid
different types of feedback.
While feedback has consistently shown to be invaluable in all types of education, it is
often poorly implemented in online education (Miller, Doering & Scharber, 2010; Miller,
FEEDBACK IN DIGITAL LEARNING OBJECTS 32
Hooper, Rose, Montalto-Rook, 2008). Instructor driven feedback is the type of feedback that is
most commonly used in online education likely because it is similar to the feedback in face-to-
face classrooms, and e-mail and conferencing are therefore easy for an instructor to use and
understand. Another reason that instructor driven feedback is more commonly used is because it
is typically very easy to implement given that most students have access to e-mail and most LMS
systems have conferencing utilities built into the system that an instructor can easily set up and
use (Race, 2001). In regards to student driven feedback, it is much more difficult for course
designers to implement. As already suggested, to create effective student driven feedback, a
significant investment in time and money, along with a lack of expertise makes implementation
difficult (Miller, Doering & Scharber, 2010). Many LMS systems have built in testing facilities
that offer numerous options for creating low stake quizzes, however, they too take a significant
amount of time and money to implement. Other types of student driven feedback take such a
large commitment to implement that individuals or even smaller institutions cannot afford to
implement them as the return on the investment for a small number of students is not practical.
From my observations, one encouraging trend is that software companies and textbook
companies are incorporating student driven feedback into their learning systems. Another reason
that instructors do not use feedback effectively in online environments is that they often do not
have the experience in, or knowledge of alternative forms of student or instructor driven
feedback. Instead, they tend to incorporate feedback methods made available to them by their
institution, or methods that are built into the LMS system they are using.
Feedback has shown to be important in all learning contexts, from face-to-face
instruction to the online learning environment. Feedback in face-to-face education requires
FEEDBACK IN DIGITAL LEARNING OBJECTS 33
significantly less effort and preparation to implement, as it can be as simple as a face to face
conversation with a student. With online feedback, it is typically more difficult to implement
effective feedback strategies because the complexities and time to incorporate effective
feedback. Development costs and lack of expertise by developers also contribute significantly to
the difficulty in effectively implementing online feedback.
Feedback type
As already discussed, feedback type has consistently shown to positively impact learning
in a face to face environment (Black & Wiliam, 1998; Davis & McGowen, 2007; Hattie &
Timperley, 2007). Specifically, Van der Kleij, Feskens and Eggen (2015) have shown that
elaborated feedback (EF) seems to be the most effective in terms of increasing learning whereas
knowledge of results (KR) and knowledge of correct response (KCR) typically serve corrective
functions and is “not very effective in student learning because it does not inform the learner
about how to improve” (p.478). This is supported by Adams and Strickland (2012) who note
that knowledge of results (KR) and knowledge of correct response (KCR) feedback is not any
more effective that no feedback (NR). When looking specifically at feedback in an online
environment, Van der Kleij, Feskens and Eggen (2015) conducted a meta-analysis of the effects
of feedback in computer based learning Van der Kleij, Feskens and Eggen (2015) and have
shown that elaborated feedback is generally more effective in an online environment than simple
feedback. This was especially true when looking at higher order learning outcomes. However,
they have indicated that previous studies and meta-analyses in the area of feedback in online
courses, and specifically feedback type, “do not shed sufficient light on the complex interplay of
feedback type, feedback timing, and the level of learning outcomes and do not give usable
FEEDBACK IN DIGITAL LEARNING OBJECTS 34
insight into the magnitude of the effects” (p.481). Van der Kleij, Feskens and Eggen (2015)
concluded that more research is needed in the area of elaborated feedback in an online learning
environment. They suggest future studies looking at feedback type, and specifically elaborated
feedback, and should have larger sample sizes and take into account feedback characteristics, the
task, the context in which the learning takes place, and the characteristics of the learner. They
also indicate that the research in elaborated feedback in online settings take place in an
educational setting to “help inform practice as more and more computer-based learning
environments that include feedback are being developed” (p.505).
Feedback and Digital Learning Objects
As already noted, instructor and online course developers typically implement instructor
driven feedback such as e-mail and online conferencing instead of student driven feedback since
student driven feedback takes such a large commitment to implement. These obstacles of time,
limited technical skill, fear of technology are easily overcome using instructor driven feedback
strategies as most LMS systems have many conferencing systems built in and e-mail is
ubiquitous. However, these obstacles are significant with student driven feedback. Most LMS
systems do have a student driven feedback mechanism which are sophisticated testing functions
which do incorporate some of the important aspects of effective feedback. These testing
functions allow the designer to incorporate feedback that gives comments based on a student’s
response to a question which allows for immediate feedback to the students. However, this type
of feedback still occurs after the student has completed online learning materials. It doesn’t
allow for feedback while the student is engaged in the learning material such as in computer
assisted instruction. As already discussed, simple feedback that emphasizes marks and grading,
instead of elaborated feedback is not as effective (Black & Wiliam, 2010; Hattie & Timperley,
FEEDBACK IN DIGITAL LEARNING OBJECTS 35
2007; Van der Kleij, Feskens & Eggen, 2015). Additionally, as Hattie, Biggs and Purdie (1996)
have noted, feedback is most effective when it is designed as part of the learning process.
One type of online learning mechanism that helps to overcome some of the obstacles of
feedback in online learning environments are digital learning objects. Digital learning objects
are an effective method of implementing technology in an educational setting (Kay, 2014).
While digital learning objects are still time consuming and expensive to produce, once created,
they can easily be incorporated into any Learning Management System (Kay, 2007; Kay &
Knaack, 2008a; Kay, 2014; Nurmi & Jaakkola, 2006; Reece, 2016). Additionally, digital
learning objects can be programmed to incorporate student driven feedback during or after the
learning process, can control timing of the feedback, and can be used to incorporate different
types of feedback such as simple or elaborated feedback (Kay, 2014).
Digital Learning Objects Defined
When looking at digital learning objects and how to use feedback in learning objects, one
must first look at the definition of a digital learning object since how they are defined will affect
how we interpret any findings. This is because there is far from agreement on what a learning
object is (Gadanidis & Schindler, 2006; McGreal, 2004) with many authors not distinguishing
between learning objects and digital learning objects. Many authors have attempted to define
what a learning object is and there are a broad range of definitions. Kopp and Crichton (2007)
note that definitions of learning objects range from people as resources to digital content such as
pictures. Also, learning objects are seen to some as primarily digital content while others include
non-digital content in their definitions. Kopp and Chrichton (2007) have noted that many
definitions of learning objects also include instructional context. These definitions offer a very
FEEDBACK IN DIGITAL LEARNING OBJECTS 36
simple view of learning objects as solitary media which include graphics, animation, and video
clips. It is assumed that teachers will provide the instructional context for the learning objects by
incorporating them in a lesson that is useful, meaningful and effective. Still other definitions
consider learning objects primarily as web-based learning tools (Gadanidis & Schindler, 2006;
Kay, 2007; Kay and Knaack, 2008a).
There are, however, a number of characteristics that most definitions have in common.
These include interactivity (Gadanidis & Schindler, 2006; Hui, 2012; Kay, 2007; Kay 2014; Kay
& Knaack, 2008a; Muller, Buteau & Mgombelo, 2009; Reece, 2016, Sims, 2000; Yacci, 2000),
reusability (Chitwood, May, Bunnow, & Langan, 2002; Nurmi & Jaakkola, 2006; Kay, 2007;
Kay 2014), digital (Chitwood et al., 2002; Gadanidis & Schindler, 2006; Nurmi & Jaakkola,
2006; Kay, 2007; Kay and Knaack, 2008a; Muller et al., 2009), self-contained (Chitwood et al.,
2002; Gadanidis & Schindler, 2006; Kay, 2007), and adaptability (Chitwood et al., 2002; Kay,
2007). The characteristics of interactivity, reusability, digital, self-contained, and adaptability
offer a fairly concise definition of learning objects that would best be described as digital
learning objects. However, these characteristics primarily define how digital learning objects are
to be used, but not how they are designed to impact learning. Someone who wishes to design an
effective digital learning object would get little help from such a definition as they do not
describe the elements that make a digital learning object effective at increasing learning. Also,
many investigations into the effectiveness of digital learning objects typically define learning
objects that they use in their investigations in terms of these characteristics, but make little effort
to describe them beyond these characteristics. This makes interpretation of the results very
difficult as there is significant variability in the design of the digital learning objects used in the
investigations.
FEEDBACK IN DIGITAL LEARNING OBJECTS 37
Previous Research on Digital Learning Objects
Digital learning objects are seen by many authors as an effective learning tool with
possibilities to significantly change online learning by overcoming the obstacles to online
learning, however, there is a lack of empirical evidence with respect to how to effectively design
digital learning objects (Kay, 2007; Kay, 2014; Kay & Knaack, 2008a; Nurmi & Jaakkola,
2006). Kay (2007) in an extensive review of the literature found 58 articles relating to digital
learning objects. Of these, 50 of the papers were based on informal qualitative methods and
although they examined a number of topics in digital learning objects such as design, they did
not offer any empirical evidence to support their claims. The topics included design (n = 24),
metadata (n = 17), learning (n = 17), reusability (n = 12), development (n = 11), assessment (n =
11), definition (n = 9), repositories and searching (n = 9), use (n = 7) and standards (n = 5). Kay
and Knaack (2008b) examined 18 articles and found that although the articles addressed topics
such as faculty attitudes (n =5), student attitudes (n = 11) and student performance (n = 7) the
majority of the studies were also based on informal qualitative methods. Another study by Kay
and Knaack (2008c) examined 22 articles and found that the majority of studies had targeted
higher education (n =18) with four studies targeting secondary schools. Kay and Knaack
(2008a) identified the following challenges. First, only four studies examined how teachers react
to and use digital learning objects. Second, the majority of the studies only relied on one digital
learning object, which made it difficult to generalize the results to other digital learning objects.
Third, sample sizes in the studies were typically very small and poorly defined which also makes
generalizing the results to a larger population difficult. Fourth, the majority of the studies
examined only student attitude, teacher attitude or student performance. Only four of the studies
examined more than one of these variables and none examined all three. Fifth, while the
FEEDBACK IN DIGITAL LEARNING OBJECTS 38
majority of the studies reported on the effectiveness of using digital learning objects, they mostly
had poorly designed assessment tools with little reliability or validity. Only two of the studies
provided estimates of reliability while only one provided data validity. The study by Lim et al.
(2006) was the only study found that investigated different levels of interactivity. This was true
even given that interactivity is consistently identified as one of the primary characteristics of
digital learning objects.
Despite the clear lack of good empirical studies that have investigated the effectiveness
and usefulness of digital learning objects, Nurmi and Jaakkola (2006) have noted that there
continues to be enthusiasm towards digital learning objects and many believe that digital
learning objects have the potential to transform online education. Kay (2007) notes that this
digital learning object revolution will not take place until instructional use and pedagogy are
explored and evaluated. Furthermore, research must look at how digital learning objects can be
used to create high quality instruction and that a detailed analysis of the features of digital
learning objects needs to take place.
Learning and digital learning objects
Digital learning objects offer many benefits to teachers and provide solutions to the many
problems that teachers face with respect to implementing technology into their curriculum. Kay
and Knaack (2008c) have identified the following benefits. First, digital learning objects are
easy to use and require little time to learn how to use. Even teachers with low computer skills do
not need to spend considerable time to learn how to use them. Second, good digital learning
objects have well defined objectives and are focused on very specific topics. This allows
teachers to easily integrate them into lesson plans or online instruction. Third, digital learning
FEEDBACK IN DIGITAL LEARNING OBJECTS 39
objects are easily accessible. Over 90% of all public schools in Europe and North America have
internet access with the majority now having high speed internet. Finally, digital learning
objects are reusable which allows the development of a single learning object to be available to a
wide audience and be applied in a variety of instructional settings.
The question about what factors contribute to the effective design to increase learning
remains. It must be reiterated at this point that research into the effective use and design of
digital learning objects has mostly been qualitative and few studies have addressed the factors
that contribute to learning from digital learning objects. Nevertheless, the studies do give insight
into some of the various factors and at the very least help to identify areas where further
empirical research is needed. The factors that studies have most consistently investigated are
teacher and student attitude, design, and learner characteristics such as computer self efficacy,
academic ability, age and gender. No studies on digital learning objects have been identified that
specifically address the effect of feedback. However, previous studies on computer assisted
instruction (CAI) have looked at the effect of feedback and can provide some insight into the
effective use of feedback in digital learning objects. Computer assisted instruction research
along with research on feedback in online learning offer a good comparison to digital learning
objects and can further help to identify areas for further research.
Teacher attitude
Although teacher attitude is not directly a factor in the effectiveness of digital learning
objects, teacher attitude about digital learning objects indirectly gives insight into their
effectiveness. This is primarily because if teachers do not see digital learning objects as effective
and easy to use, they will be less inclined to integrate them into their lessons. This is highlighted
FEEDBACK IN DIGITAL LEARNING OBJECTS 40
by Collis and Stijker (2003) who note three obstacles with respect to teachers who have no
experience with digital learning objects and their attitudes toward digital learning objects. First,
teachers did not have sufficient awareness and understanding of digital learning objects to
effectively make decisions about their educational advantages. Second, teachers anticipated that
it would take significant time to integrate digital learning objects into existing courses. Third,
teachers anticipated that it would also take significant time to find good digital learning objects
that fit well into their curriculum. Gadanidis and Schindler (2006) offer two other reasons why
teachers may hesitate to incorporate investigative type digital learning objects. First, are the
teacher’s personal beliefs about what the students can do and how they learn. Second, teachers
have concerns about classroom management issues with digital learning objects that are
investigative or exploratory.
Studies that investigated attitudes of teachers who actually used digital learning objects in
a classroom setting reported findings that contradict the obstacles that teachers identified above
and found that teachers typically had a positive reaction to digital learning objects (Kay and
Knaack, 2008a; Kay, 2014). More specifically, teachers who used digital learning objects
consistently reported that they believed that students were more engaged (De Salas & Ellis,
2006), digital learning objects were beneficial to the students (Kay and Knaack, 2008a), and
searching for and planning lessons with digital learning objects did not take an inordinate amount
of time (Kay and Knaack, 2008c; Kay 2014). It should be noted, however, that a possible
explanation for positive attitudes by teachers who were actively engaged in using digital learning
objects might be that the teachers had already “bought into” the effectiveness of digital learning
objects (Kay, 2014).
FEEDBACK IN DIGITAL LEARNING OBJECTS 41
Student Attitude
Student attitude about the use of digital learning objects was dependent on the education
level of the student with a greater percentage of students in post-secondary reporting a positive
attitude towards digital learning objects as compared to secondary students. Kay and Knaack
(2008a) looked at a number of studies that investigated student attitude towards learning objects
with post-secondary students and found that eight studies reported positive student attitude, one
study reported neutral attitude and one study reported a negative attitude. Post-secondary
students typically gave positive comments about characteristics such as animations, self-
assessment, attractiveness, control over learning, ease of use, feedback, scaffolding or support,
interactivity, navigation, and self efficacy. Negative comments focussed on navigation
problems, technology, and workload. Typically, students reported that they preferred digital
learning objects that were interactive, recreation based, or collaborative.
There have been only a few studies that have looked at student attitude at the secondary
level. Kay and Knaack (2008a) found that there were mixed results with student attitude at the
secondary level reporting that students were only moderately positive about the quality of the
digital learning objects and neutral in regards to the learning value. An interesting point is made
by Kay and Knaack (2008c) with regards to student attitude at the secondary level. They note
that although students were relatively neutral about digital learning objects, a significant number
of students reported that they were an improvement over face-to-face teaching methods.
Further study is definitely needed in the area of student attitude given that most of the
studies were qualitative and failed to control for a number of factors such as computer self
efficacy, quality and type of digital learning objects, and design of the digital learning object.
FEEDBACK IN DIGITAL LEARNING OBJECTS 42
Learner Characteristics
Kay and Knaack (2008b) note that although digital learning objects offer positive
educational opportunities, there is very little research in regards to learner characteristics.
Learner characteristics that have been examined are gender, age, subject ability, and computer
self-efficacy.
In regards to gender, there were only two studies found that directly investigated the
differences between males and females with respect to digital learning objects (Kay & Knaack,
2008b; Van Zele, Vandaele, Botteldooren, & Lenaerts, 2003). Both studies indicate that there
was no observed difference between males and females with respect to student attitude and
performance. This is reflected in the recent research into gender and attitudes about computer
use. In the 1990’s, males tended to have much more positive attitudes and higher ability in
respect to computer use. However, current research shows that this difference has significantly
lessened (Kay & Knaack, 2008b). One study did find that females tended to emphasize the
quality of the help features in the digital learning objects more than males (Kay, 2007).
Kay and Knaack (2008b) found that age is a significant factor in both attitude and
performance in regards to digital learning objects. They found that post-secondary students
performed significantly better and had a more positive attitude than secondary students.
Furthermore, grade 12 students performed better and had a more positive attitude than grade 9 or
10 students. Again no attempt was used to distinguish different types of digital learning objects
in terms of interactivity. It would be beneficial to examine age differences with different types
of learning objects since younger students may not perform well with exploratory digital learning
objects, while older students may prefer them.
FEEDBACK IN DIGITAL LEARNING OBJECTS 43
Only one study was found that measured subjects’ prior knowledge as a factor in
performance with digital learning objects. Nurmi and Jaakola (2006) found that there was no
statistically significant difference between groups with higher and lower prior knowledge about a
subject with the exception of using simulations and mixed (simulation and face-to-face labs)
versus a face-to-face lab. They found that the higher knowledge students were not dependent on
the learning condition, but they did find that lower prior knowledge students benefited from
using the simulation.
Kay (2007) indicates that no research has been completed that examines computer self
efficacy in terms of performance and digital learning objects, but anticipates that students who
are more comfortable with computers would benefit more from using digital learning objects and
would have a more positive attitude towards them. Kay notes that this could best be supported
by the substantial amount of research done in the area of self efficacy and computer related
behaviours. This research has consistently shown that positive computer attitudes are correlated
with higher levels of ability. Lim et al. (2006) did report that students who were not comfortable
with computers did tend to use digital learning objects less.
Design
Given the interest in digital learning objects and the large amount of research in
instructional design and on multimedia elements in design, it is surprising that there is very little
research on the design elements specific to digital learning objects and which design elements
make learning objects most effective. Of course specific characteristics as outlined in the
definition of digital learning objects are important considerations when designing digital learning
objects. These include interactivity, reusability, being self-contained, and adaptability.
FEEDBACK IN DIGITAL LEARNING OBJECTS 44
However, as noted previously all of the characteristics but interactivity describe the conditions
for using learning objects and do not address the specific design elements that would make a
digital learning object effective at increasing learning. There is a noticeable lack of research
specifically in the area of the design of effective learning objects, however, design principles of
multimedia learning have been investigated and are well represented in the literature. Mayer
(2001) describes a number of strategies for designing multimedia learning and these strategies
would certainly translate well to the design of digital learning objects.
One study by Rieber, Tzeng and Tribble (2004) showed that computer based learning
with embedded explanations and graphical representations were more effective than those
without. Another study by Lim et al. (2006) found that the level of interactivity in the digital
learning objects was an important factor to consider in the design. Finally, Kopp and Crichton
(2007) have shown that embedding digital learning objects into an online learning resource was
significantly more effective than simply presenting the digital learning objects as links in the
online resource.
Some authors have attempted to describe some of the effective design elements that are
specific to digital learning objects. Lim et al. (2006) list a number of these effective design
elements. These include chunking the content in a meaningful way, targeting the digital learning
objects to specific learner types (this may impact the reusability of the learning object), and level
of interactivity and control. Ally, Cleveland-Innes, Boskic and Larwill (2006) add that digital
learning objects should also provide hands-on activities and examples, and that digital learning
objects should be designed in a way that the learners can see the relevance of the content.
Computer Assisted Instruction and Digital Learning Objects
FEEDBACK IN DIGITAL LEARNING OBJECTS 45
As noted, there has been little empirical research on how digital learning objects can be
best designed to increase learning (Kay, 2007; Kay & Knaack, 2008a; Nurmi & Jaakkola, 2006)
and no studies were found on the effect of integrating feedback into learning objects in terms of
increasing learning. Watson (2010) contends that there has been an emphasis on the
technological aspects of digital learning objects such as reusability as well as the visual
appearance of the digital learning objects, but there has been a lack of research into the
pedagogical basis for designing digital learning objects. Chikh (2012) adds that previous
research has focussed on targeting learners as consumers of digital learning objects or targeted
instructors and designers and focussed on the reusability of digital learning objects. While
research on digital learning objects has not focussed on design elements that increase learning,
there has been significant research in the early days of computer based learning in the area of
Computer Assisted Instruction (CAI) that did focus on increasing learning. Computer Assisted
Instruction is analogous to digital learning objects in many ways with the exception that digital
learning objects are specifically designed to be self contained, reusable and adaptable or
integrated easily into multiple learning environments. It could easily be argued that learning
objects are a subset of computer assisted instruction that have developed in response to the
internet and availability of learning management systems. Consequently, previous research on
computer assisted instruction and specifically, research on feedback in computer assisted
instruction, can inform research on feedback in digital learning objects.
Feedback research in computer assisted instruction is fairly consistent with the current
research in feedback in online courses. Feedback research in computer assisted instruction has
focused primarily on what Shute (2008) describes as verification or simple feedback, and Van
der Kleij, Feskens & Eggen (2015) describe as knowledge of results feedback (KR). In
FEEDBACK IN DIGITAL LEARNING OBJECTS 46
computer assisted instruction research KR feedback is typically described as non-corrective and
corrective feedback which is analogous to knowledge of results feedback (KR). Additional
research also looked at elaborated feedback, but was limited to providing elaborated feedback to
correct answers (positive feedback) and no studies were found that looked at feedback that
provided elaborated feedback for incorrect responses (negative feedback). Furthermore,
categories of elaborative feedback such as hints, error flagging, providing of explanation of
correct answer and response contingent feedback were not defined in these earlier studies. That
being said, the results of the studies can still be categorized in relation to the feedback types
currently used to help understand the implications for feedback in digital learning objects and
this study.
A number of authors exploring feedback in computer assisted instruction looked at
simple and elaborated feedback. Hodes (1984) looked as simple feedback and found that there
was no significant difference between corrective (KCR) and non-corrective (KR) feedback.
However, Hodes did not have a control group in the study so conclusions could not be made
about the overall effect of simple feedback compared to no feedback or other forms of feedback.
Hodes did find that there was a significant effect for gender and reported that females receiving
non-corrective feedback had significantly lower scores than males. Other researchers looked at
simple non-corrective feedback (KR) and compared it to elaborated feedback and found that
elaborated feedback was significantly better than no feedback and simple feedback (Collins,
Carnine, & Gersten, 1987; Gilman, 1968; Lysakowski & Walberg, 1982). Additional results
include an effect found for ability level and feedback as well as competitive feedback versus
individual feedback. Collins, Carnine and Gersten (1987) compared simple feedback to
elaborated feedback for low ability students and found a significant effect for increased learning
FEEDBACK IN DIGITAL LEARNING OBJECTS 47
with elaborated feedback over simple feedback. Lewis and Cooney (1986) compared a no
feedback control with individual feedback and competitive feedback and found that students
receiving competitive had in increase in what they describe as attributes for ability to succeed.
They found no significant effect for individual feedback.
Conclusion
Current and previous studies have looked closely at feedback in both face to face and
online settings with a number of meta-analyses (Azevedo & Bernard, 1995; Van der Kleij,
Feskens & Eggen, 2015) showing elaborated feedback (EF) to be significantly better than
knowledge of results feedback (KR) and knowledge of correct response feedback (KCR). While
elaborated feedback has been shown to be more effective than simple feedback (KR) and (KCR),
Van der Kleij, Feskens and Eggen (2015) identified a number of gaps in the research on online
feedback that need to be further explored. They advise that more research needs to take place on
the specific types of elaborated feedback. They also identify that many previous quantitative
studies did not have sufficient sample sizes and that the psychometric properties of the
instruments need to be reported. They specifically suggest that “future investigations should
account for and describe characteristics of the feedback, the task, the learning context and the
learners in order to advance the research field” (p. 505). This study seeked to extend some of
these research gaps in online feedback identified by Van der Kleij, Feskens and Eggen (2015).
Specifically, this study will look at two areas. First, it will look at a specific type of elaborated
feedback (EF) that has not been thoroughly researched which is positive and negative feedback
which Shute (2008) describes as response contingent feedback. It is the type of elaborative
feedback “that describes why the incorrect answer is wrong and why the correct answer is
correct” (p. 106). Furthermore, positive or negative feedback has been selected as one of the
FEEDBACK IN DIGITAL LEARNING OBJECTS 48
independent variables in this study since it has consistently also been shown to influence the
effectiveness of feedback in face to face environments (Black & Wiliam, 1998; Davis, &
McGowen, 2007). Second, this study looked at the feedback type in relation to a specific
characteristic of the learner, that being, computer attitude and experience. There has been little
recent research on learner characteristics in relation to the effect of feedback in digital learning
objects, and the research that has been done suggests that learner characteristics do not have a
significant effect on learning (Kay & Knaack (2008b) with the exception of computer self
efficacy where no research has been completed Kay (2007).
The reason for selecting feedback to investigate as an effective design element in digital
learning objects is because, as Hatziapostolou and Paraskakis (2010) have noted, feedback is an
important component in all learning contexts. Since digital learning objects are becoming
prevalent in online learning it is worth investigating whether the effect of feedback can be
extended to digital learning objects as a learning context. Additionally, Van der Kleij, Feskens
and Eggen (2015) have suggested that further experiments should also account for the learning
context. By conducting this research specifically for digital learning objects, this study can
control for learning context while giving some specific insight into the design of learning objects
as well as extending the previous research on computer assisted instruction.
FEEDBACK IN DIGITAL LEARNING OBJECTS 49
CHAPTER THREE – METHODOLOGY
Introduction
The purpose of this study was to investigate the effect on student achievement of
different levels of student driven simple feedback and elaborated feedback embedded in digital
learning objects designed for post-secondary calculus students. Specifically it looked at response
contingent feedback which is an elaborated feedback type that describes why the incorrect
answer is wrong (negative feedback) and why the correct answer is correct (positive feedback)
(Shute, 2008). Additionally, computer experience was also considered in relation to student
achievement of different levels of student driven feedback. In the design of this study,
participants were selected to maximize the power of the study and to increase its validity.
Research Hypotheses
1. Ho: There is no difference in post-secondary students’ learning based on the type of
feedback, in online digital learning objects.
2. Ho: There is no difference in post-secondary students’ learning from digital learning objects
with differential levels of feedback based on the students’ computer experience.
Variables
Independent Variables
1. The first independent variable (IV) is defined as the type of feedback. Type of feedback is
operationalized as differential levels of feedback embedded into a digital learning object
designed to instruct an introductory post-secondary calculus course. The levels of feedback
were designed specifically around the learning content and were simple feedback, positive
feedback and negative feedback. Three digital learning objects were created for each of the
FEEDBACK IN DIGITAL LEARNING OBJECTS 50
three topics and each of the three learning objects was embedded with one of the three types
of feedback. The digital learning objects embedded with simple feedback provided only
simple correct or incorrect responses to the participants’ answers to questions in the guided
practice portion of the digital learning objects. The digital learning objects embedded with
positive feedback provided elaborated feedback for only correct responses which described
why the response was correct. The digital learning objects embedded with negative feedback
provided elaborated feedback for only incorrect responses which described why the response
was incorrect.
2. The second independent variable (IV) is defined as the computer experience of the
participant. Computer experience is operationalized as the comfort level the participants
have with using a computer and their attitude about computers. Computer experience was
determined by a series of questions on a questionnaire given at the beginning of the study
(Appendix 3), and the computer experience questions were delivered in two parts. First,
computer experience was determined by a series of questions about the participant’s
experience with online courses, social media use, video game use, and self reported ability
with using computers. The second measure was a series of questions on computer attitude
using the Computer Attitude Scale developed by Nickell and Pinto (1986). The Computer
Attitude Scale is a series of 20 psychometrically tested questions that returns an overall
computer attitude score between 20 and 100 with 20 indicating a strong negative attitude
towards computers and 100 indicating a strong positive attitude towards computers.
FEEDBACK IN DIGITAL LEARNING OBJECTS 51
Dependent Variable
The dependent variables (DVs) are defined as the participant’s scores on post tests consisting of
multiple choice questions for each of the three digital learning objects. Essentially, three separate
experiments were conducted using three separate learning objects each with their own associated
post questions unique to that learning object. Consequently, the test scores for each of the post
tests represents a different dependent variable and were therefore subsequently analyzed
independently using ANOVA. The topics selected for each of the learning objects were common
application topics taught in the first year Calculus courses that the participants were enrolled in.
Each of the three topics represented a distinct topic that were determined to be independent of
one another and are commonly taught in any order. That is, none of the topics requires the other
two be completed as a pre-requisite. Additionally, each of the topics was based on a unique
cognitive function as described by Bloom, Engelhart, Furst, Hill, and Krathwohl (1956) which
made them independent from one another. The first digital learning object was on related rates
in calculus and represented application as the cognitive function. The second digital learning
object was on differentials and represented evaluating as the cognitive function. The last digital
learning object was on areas by integration and represented analysis as the cognitive function.
Throughout this paper the first digital learning object will be referred to as the Application DLO,
the second digital learning object will be referred to as the Evaluation DLO and the third digital
learning object will be referred to as the Analysis DLO. Each of these three digital learning
objects represents a separate experiment within the overall study.
FEEDBACK IN DIGITAL LEARNING OBJECTS 52
Participants
Digital learning objects can be used in online courses across all grade levels and all
subjects which makes defining a target population difficult. However, if consideration is given
to the need to create digital learning objects for a subject that better lends itself to embedding
differential levels of feedback, then a defined population can be considered. For this study, the
population was post-secondary engineering technology students who have a program entrance
requirement of grade 12 mathematics and grade 12 physics or equivalent.
When determining the participants for this study, it was important to reduce the threat to
internal validity and to increase power by selecting participants in a learning setting where there
would be the least amount of diversity among the participants, that is, to ensure the sample is
homogeneous in terms of entrance requirements. Consequently, this study examined a defined
group of learners enrolled at a post-secondary institute. To minimize diversity in terms of
entrance requirements among the participants, the participants were selected from first year
students enrolled in introductory calculus courses as a requirement for engineering technology
diploma programs. The participants were selected from several engineering programs that have
the same program entrance requirements (grade 12 math and physics) and similar first year
calculus courses. These programs included Chemical Engineering, Civil Engineering,
Engineering Design and Drafting, Wireless Systems Engineering and Petroleum Engineering.
Participants for this study were selected through convenience sampling from these programs
using established groups in first year calculus classes. There were seven different engineering
calculus classes that agreed to take part in the study with a total of 192 students. Of the 192
students in the seven classes, 145 consented to be participants in the study.
FEEDBACK IN DIGITAL LEARNING OBJECTS 53
This study contains two independent variables, and is a two-way randomized groups,
fixed effects design with sampling for the study being determined to maximize the power of the
study. The first independent variable was levels of feedback. The differential levels of feedback
were analyzed using a two-way randomized groups, fixed effects design with the independent
variable and computer experience. The total sample size for this study was calculated using the
G*Power statistical analysis software, and was determined to be 96 with a size of approximately
30 per treatment group (Faul, Erdfelder, Lang, & Buchner, 2007). The sample calculation was
based on a postulated minimum detectable difference of 2% on post test scores with the level
set to a value of 0.05 and desired power (1 - ) set to 0.80 (Table 1). A 2% mean detectable
difference was selected as it provides a conservative measure of detectable mean differences
requiring a larger sample size.
Table 1: Sample Size Calculations F tests - ANOVA: Fixed effects, omnibus, one-way
Analysis: A priori: Compute required sample size
Input: Effect size f = 0.3265986
α err prob = 0.05
Power (1-β err prob) = .8
Number of groups = 3
Output: Noncentrality parameter λ = 10.2399980
Critical F = 3.0943374
Numerator df = 2
Denominator df = 93
Total sample size = 96
Actual power = 0.8118639
Setting
This study was conducted at a Polytechnic Institute, in face to face calculus courses
where all courses were first year calculus courses delivered in a 15 week semester. A Polytechnic
is Post-Secondary Institute specializing in applied research and instruction in technical education
and trades. While the regularly scheduled classes were face to face classes, participants were
FEEDBACK IN DIGITAL LEARNING OBJECTS 54
asked to participate in three one hour online lessons using the digital learning objects on three
separate introductory calculus concepts. An important aspect to note is that the samples within
each digital learning object group were unique to each group as few students participated in all
three digital learning object groups. These lessons were delivered in a computer lab at the
institute using digital learning objects designed specifically for this study by the researcher. The
digital learning objects were integrated into the regularly scheduled activities of the class through
an online Learning Management System. All students in the class were asked to complete the
digital learning objects regardless of whether they consented to take part in the study. However,
data was only collected for students who consented to take part in the study.
Research
This two-way randomized group, fixed effects design was selected to determine if
learning is significantly different depending on the type of feedback embedded into a digital
learning object with the second dimension being the learner characteristic of computer
experience. In this study, the type of feedback was operationalized as differential levels of
feedback with the levels of feedback designed specifically around the learning content. The first
level only provided simple feedback to the participants’ responses with only a correct or
incorrect response as they worked through the learning material in the digital learning object.
The second and third levels of feedback were elaborated feedback that are adaptive, or response
contingent. The second level of feedback only gave positive feedback. Meaning that the
adaptive feedback provided was only given for the correct response which described why the
response was correct. The third level of feedback was also adaptive and addressed faulty
interpretations. However, the third level of feedback only gave negative feedback. Meaning that
FEEDBACK IN DIGITAL LEARNING OBJECTS 55
the adaptive feedback provided was only given for incorrect responses which described why the
response was incorrect.
This research design was chosen because it first allowed for the exploration of the
relationship between a single categorical independent variable (level of feedback) and a single
dependent variable for each of the three digital learning objects. The addition of the second
categorical variable for learner characteristics also allowed for further exploration of the effect of
differential levels of feedback in digital learning objects and whether the achievement on these
levels is also affected by computer experience. In addition, a fixed effects design allows for
selection of the levels of the Independent variables. To ensure that the groups were randomized,
participants were selected from a number of post-secondary, first-year, engineering calculus
classes using convenience sampling and then randomly placed into feedback conditions. Again,
an important aspect to note is that the samples within each digital learning object group were
unique to each group as few students completed all three digital learning objects. In placing
participants within each of the feedback conditions, no regard was given to the type of
engineering technology to control for the type of engineering program. In addition, to further
control for the type of engineering program, participants were selected from engineering
programs that have similar first year calculus courses and have similar program entrance
requirements.
Procedures
The procedure for this study involved a number of steps. First, necessary approvals
needed to be obtained from a number of stakeholders. Second, instructors needed to be asked to
have students in their classes participate in the study and to have the digital learning objects to be
FEEDBACK IN DIGITAL LEARNING OBJECTS 56
used as part of the activities of the class for all students. Third, students needed to be asked to
participate in the study and to use the results in the study (informed consent). Fourth, the study
needed to be conducted with the participants. Fifth, the participants in the study had to provide
re-consent for their data to be used in the study. Lastly, an analysis of the results was conducted
and the final report completed.
Approvals
There were a number of stakeholders to gain approval from prior to conducting this
research. The first was get approval from necessary research Ethics boards at the University
where the researcher was completing the study and at the Polytechnic Institute where the data
was being collected. The second set of approvals was from the leadership of the programs in
which the research was being conducted.
Consent
Once approvals for the research was secured, the instructors were approached to provide
consent to have students in their classes participate in the study by allowing the results from the
participants’ use of the digital learning objects to be used in the study. An Information and
Consent letter outlining the background, purpose, procedures, risks, benefits of the study was
made available to the instructors which included the timeline of the study, and the
responsibilities of the instructors in the study (Appendix 1). The instructor was also asked to
instruct the concepts that were covered in the digital learning objects. This instruction occurred
after each digital learning object was presented to ensure that students were not disadvantaged by
FEEDBACK IN DIGITAL LEARNING OBJECTS 57
participation in the study. Based on the approvals and the participation of the instructors, seven
classes of first year calculus students participated in the study.
Next, the researcher met with the students in each class and reviewed the consent forms
to ask for their consent to use the data generated from their use of the digital learning objects and
to sign the consent form (Appendix 2). Students were made aware that the use of the digital
learning objects were a required elements of the course, and that they were only giving consent
to use their data as part of the research study. During the meeting with the students, the
procedures and timeline of the study were clearly outlined. Students were informed that there
was no monetary or other reward for participating in the study, however, the benefits of using
their data in the study to inform research into the effective design of online learning objects was
described. Also, students were only informed of the independent variable of computer
experience, but were not informed of the independent variable of differential levels of feedback
embedded into the design of the digital learning objects since knowledge of the independent
variable of levels of feedback would have potentially confounded the results of the study.
Additionally, students were informed that any scores that they received on the assessment
portion of the digital learning object would not be used in the final determination of their course
mark and that their instructor would not have access to the results of study. To ensure that no
student was disadvantaged, all students in the class were expected to complete the digital
learning objects as a required element of the course, and the instructor was expected to re-teach
all of the concepts presented in the study in a face to face setting. To protect privacy, students
were informed that their anonymity would be protected throughout the study and that only the
researcher would have access to the results. Furthermore, the students were informed that they
FEEDBACK IN DIGITAL LEARNING OBJECTS 58
could choose to withdraw from the study at any time up to the point where the data was
anonymized at the conclusion of the data collection.
Re-consent
During the initial meeting with the students in the classes participating in the study, the
students were asked to consent to have their data used in the study. Prior to consenting to
participate in the study, the students were made aware that the study was designed to investigate
the design of digital learning object and that computer experience was the factor being
investigated. However, the students were not made aware of the Independent variable of
differential levels of feedback in the digital learning objects before consenting to participate and
were not made aware of the independent variable throughout the course of the study. This was to
ensure that prior knowledge of the variable did not bias the results. Not disclosing the
independent variable of differential levels of feedback meant that deception was used in the
study. Consequently, at the end of the study, the researcher met with the participants in their
classrooms and made them aware of the independent variable not previously disclosed and asked
the participants to sign a re-consent form indicating that they were fully aware of all of the
variables and wished to allow their data to remain in the study (Appendix 6). Some students were
not in attendance during the classroom visit and were contacted through email. A total of four
students who had originally consented to take part in the study could not be contacted for re-
consent and their data was withdrawn from the study leaving a total of 141 participants in the
student who had re-consented.
FEEDBACK IN DIGITAL LEARNING OBJECTS 59
Data Gathering
Data from the study came from a number of sources, a pre-study survey, the digital
learning objects (including post tests delivered in Moodle), and a post study survey.
The pre-study survey was a survey that only students who agreed to participate in the
study completed. The survey (Appendix 3) included demographic questions, computer
experience, and a questionnaire on computer attitude that was adapted from a standardized
computer attitude survey by Nickell and Pinto (1986).
The digital learning objects provided the second source of data. There were three separate
digital learning objects created for three different independent calculus topics. As stated earlier,
the three digital learning objects were Application DLO which was on the topic of related rates,
Evaluation DLO which was on the topic of differentials, and Analysis DLO which was on the
topic of areas by integration. Participants completed each one hour session in a computer lab
monitored by their class instructor. Each digital learning object captured two sets of data points
that were collected by way of a Learning Management System as described in the delivery
method section below. The first data point for the digital learning objects determined the time
spent on task on each of the two parts of the digital learning object which was the lesson and
guided practice. The digital learning objects were programmed to record the time that the
participant took to complete the lesson and the guided practice portions of the digital learning
objects. When the participants completed the digital learning objects, they were presented with
the times they took to complete the lesson and the guided practice portions and asked to write
down the times and record them in the first two questions in the post-test (Figure 2).
FEEDBACK IN DIGITAL LEARNING OBJECTS 60
Figure 2: Lesson Time/Screen Time Screen Shot
The second set of data points for each of the digital learning objects was captured by way
of a multiple choice test at the end of each of the digital learning objects which measured
learning as the dependent variable for this study. Each multiple choice test was specific to the
content of the digital learning object it was associated with, and were developed based on the
question presented in the lesson and guided practice portions of the digital learning objects.
The addition data source was the post-study survey which was a survey that only
participants in the study completed. The post-study survey was an open ended question on the
participants’ perceived usefulness of the learning objects.
Design of the Digital Learning Objects
In total, three digital learning objects were created for the study and they were each
designed with a lesson and guided practice portion. Essentially, three separate experiments were
conducted using three separate learning objects each with their own associated post questions
FEEDBACK IN DIGITAL LEARNING OBJECTS 61
unique to that learning object. To determine which topics to select for the study, each instructor
was asked to provide a list of topics and their class timeline for each of the seven classes
participating in the study. From the list, 12 potential topics were selected that were common to
all three courses and were independent of one another. That is, no learning in one object was
dependent on the learning found in another and represented a different cognitive function
(Bloom et al., 1956). Eight of the topics were to be taught in the first month of the course which
did not give sufficient time to develop the digital learning objects and pre and post tests. Of the
four remaining topics, three were identified by the instructors as the best choices for the study in
terms of timing with their class schedules. These topics were related rates, differentials, and
areas by integration. These three topics were determined to be independent of one another and
are commonly taught in any order and one, two or all three can be taught to students.
The design of the digital learning objects for each topic was identical in all aspects other
than the treatment of differential levels of feedback. Each digital learning object was designed
using Adobe Flash software and was delivered via the internet using the Moodle Learning
Management system (LMS). Five modules were created for each participant in Moodle. The
first module contained the pre-study survey, the second, third and fourth modules contained the
digital learning objects (including post tests) and the fifth module contained the post-study
survey. Both the pre and post study surveys were identical for all participants.
The design of each digital learning object was developed with two parts, the lesson and
the guided practice. The lesson part of the digital learning objects was identical for all
participants. However, the guided practice parts varied by participant depending upon which
FEEDBACK IN DIGITAL LEARNING OBJECTS 62
treatment condition they were part of. That is, the guided practice provided simple feedback,
positive feedback or negative feedback for all questions depending on the treatment condition.
The lesson portion of the digital learning objects presented a brief explanation or lesson
which included animations or interactive objects as necessary to teach the topic. Additionally,
each lesson also included two or three worked examples of problems with explanations for each
step as appropriate. For example, the following two graphics show the interactive object portion
of the lesson on differentials from the Evaluation DLO where the participant was able explore
the relationship between dy and y as the value of x is manipulated by the participant (Figure
3and Figure 4). The results are shown to the participant both graphically and numerically. As the
participant manipulated the value for x, the graph changed accordingly and the new calculated
values of dy and y were displayed. Explanations were also dynamically changed depending on
the value of x.
FEEDBACK IN DIGITAL LEARNING OBJECTS 63
Figure 3: Differentials Interactive Object - Part 1
Figure 4: Differentials Interactive Object – Part 2
FEEDBACK IN DIGITAL LEARNING OBJECTS 64
The lesson portion of the digital learning objects also included worked examples with
explanations and, where appropriate, the steps to solve, illustrations and animations were
included. Figure 5, Figure 6, Figure 7, and Figure 8 show four parts of a worked example in the
lesson on Area by integration which was the Analysis DLO.
Figure 5: Areas by Integration example - part 1
FEEDBACK IN DIGITAL LEARNING OBJECTS 65
Figure 6: Areas by Integration example - part 2
Figure 7 Areas by Integration example - part 3
FEEDBACK IN DIGITAL LEARNING OBJECTS 66
Figure 8: Areas by Integration example - part 4
The second part in each of the digital learning objects was the guided practice section. In
the guided practice section, the participants were presented with a problem based on the learning
from the lesson. As the participants worked through the problem in the guided practice, they
were asked to answer multiple choice questions at each step through the problem. For each
digital learning object, the guided practice section was identical for all participants except for the
feedback they received when they answered the multiple choice questions. Depending on the
treatment condition the participants were assigned to, they received simple feedback, positive
feedback or negative feedback. The digital learning objects with simple feedback did not provide
feedback to the participants when they answered the questions other than a simple “correct” or
“incorrect.” The digital learning objects with positive feedback provided feedback which
explained why their answer was correct, and the digital learning objects with negative feedback
provided feedback which explained why their answer was incorrect. In designing the feedback
FEEDBACK IN DIGITAL LEARNING OBJECTS 67
for the digital learning objects with negative and positive feedback, it was important to ensure
that the feedback provided did not provide new information to the participant that was not
presented in the lesson portion of the digital learning object. Otherwise, it would be impossible
to determine whether any significant difference in learning was the result of the new learning
presented in the feedback or from the presence or absence of positive and negative feedback.
Figure 9, Figure 10, and Figure 11 show an example guided practice question for the Analysis
DLO which was on the topic of area by integration. The correct response to the question in the
learning object was “vertical.” Figure 9 is a screen shot from the digital learning object with
simple feedback embedded. When the correct answer is selected, the response “correct” is
displayed, and when an incorrect response is selected, the response “incorrect” is displayed.
Figure 10 is a screen shot from the digital learning object with positive feedback. When the
correct answer is selected, feedback that explains why the answer was correct is displayed. If
any incorrect response was selected then the response “incorrect” is displayed. Figure 11 is a
screen shot from the digital learning object with negative feedback. When the incorrect answer
is selected, feedback that explains why the answer was incorrect is displayed. If any correct
response was selected then the response “correct” is displayed.
FEEDBACK IN DIGITAL LEARNING OBJECTS 68
Figure 9: Simple Feedback example
Figure 10: Positive feedback example
FEEDBACK IN DIGITAL LEARNING OBJECTS 69
Figure 11: Negative feedback example
Delivery method
Moodle was selected as the delivery platform for the study since all participants in the
study used Moodle in every course in their program of studies and were highly familiar with the
Moodle Learning Management System which helped to reduce any affect that varying degrees of
familiarity with the delivery mode may have on reliability. Additionally, the Moodle Learning
Management System allowed for ease of set up and delivery of the digital learning objects,
quizzes and surveys which also included functionality to support the set up and delivery of the
learning objects by treatment condition.
In total, five modules were created for each class in Moodle. The first module delivered the pre-
study questionnaire (Figure 12), the second, third and fourth modules delivered the three digital
FEEDBACK IN DIGITAL LEARNING OBJECTS 70
learning objects along with the pre-tests and quizzes (
Figure 13), and the fifth module delivered the post-study survey (Figure 14).
Figure 12: Pre-Study Questionnaire module
Figure 13: Digital learning object module
Figure 14: Post - study questionnaire module
The three digital learning modules (
FEEDBACK IN DIGITAL LEARNING OBJECTS 71
Figure 13) contained not only the post-tests, but also all four versions of the digital
learning objects (non-participants, simple feedback positive feedback, and negative feedback).
Group functionality within the Moodle Learning Management System allowed for the placement
of participants into a group that represented one of the treatment conditions or a non-participant
group. The digital learning object that was made available to the participant was dependent
upon which group in Moodle the participant was placed into. However, the form of the pre-test
and post-test were identical for all digital learning object groups. Participants were not able to
determine which treatment condition they were in since names of all the versions of the digital
learning objects were the same, and in fact, the participants were not aware that there were
different treatment conditions.
The researcher determined the best date and time for each of the seven classes
participating in the study to complete the digital learning objects in collaboration with the course
instructors. A computer lab was scheduled for a one hour time period and all of the class
participants in the class participated in using the digital learning object during the scheduled
period. Non-participants in the study still completed the learning material, but they were put into
a different group with a different digital learning object and a note was displayed on the Moodle
page reminding them that their data was not being collected. It was expected that the
participants completed the digital learning objects as part of the normal part of their classroom
activities. As such, the researcher was only available for the beginning of the class period, to
troubleshoot as needed. The Moodle Learning Management System also allowed the researcher
to activate or deactivate the modules as needed. For each of the learning sessions, the researcher
activated the modules just prior to the start of the class and deactivated them at the end of the
class. It should also be noted that not all participants were able to complete all three digital
FEEDBACK IN DIGITAL LEARNING OBJECTS 72
learning objects since the learning objects were three distinct modules presented at different
times. Consequently, most of participants were absent for one or more of the sessions and were
unable to complete. This meant that the sample varied from learning object to learning object.
However, as already stated this study considered each learning object as a distinct case with
different post tests. With no dependencies between the learning object content and differing
samples, the results from each learning object were analyzed independently, as a separate
sample, using an Analysis of Variance (ANOVA) for each.
Summary of the Procedure
For this study, there was one Moodle site created for each of the seven classes
participating in the study. Five modules were created in Moodle for the participants to complete
through the course of the study. The first Moodle module delivered the pre-study survey and the
fifth Moodle module delivered the post-study survey. Moodle modules two through four
represented the three different Moodle Modules that contained the digital learning objects and
the post-tests designed to test the content of the digital learning objects. Module two contained
the digital learning object on related rates in Calculus along with a post test for related rates,
module three contained the learning object on differentials along with a post test for differentials,
and module four contained a digital learning object for areas by integration along with a post-
test for area by integration. All students who agreed to participate in the study were randomly
assigned into one of the three feedback conditions of simple feedback, positive feedback and
negative feedback. For each of the learning objects, participants were presented with a learning
object that was dependent upon the feedback condition they were in. For example, for Moodle
module two, the Evaluation DLO was on the topic of related rates and three learning objects
FEEDBACK IN DIGITAL LEARNING OBJECTS 73
were created for related rates that were identical except for the type of feedback embedded into
the learning object. The learning object that the participant received was dependent upon the
treatment condition the participant was placed in. A participant was assigned to the same
feedback condition for all of the digital learning objects he/she completed. Figure 15 shows the
structure of Moodle module two for the Application DLO (related rates).
FEEDBACK IN DIGITAL LEARNING OBJECTS 74
Figure 15: Structure of Moodle Module
Not all participants were able to complete modules two through four which contained the
digital learning objects since the modules were delivered on separate dates. Consequently,
participants may have completed one, two, or three of the modules containing the digital learning
objects dependent on their availability.
Participants who did not consent to participate in the study were given the digital
learning object with simple feedback to complete and were not presented with the pre-study
survey or the post study survey and no data was collected. All participants had the same pre-
study survey and post-study survey regardless of the treatment condition they were part of.
The actual digital learning objects were structured in two parts. The first part was a lesson
that was a demonstration of the theory with no interaction other than allowing the participant to
advance through the lesson at their own pace. The second part was guided practice which
allowed the participants to work through the problem and provide answers to questions at each
step. Different types of feedback were provided to the participants during the guided practice
part of the digital learning object with the feedback determined by the assigned feedback
condition. The lesson itself was the same for all participants. Only the guided practice varied
depending on the feedback condition as the guided practice part contained the different levels of
feedback.
Application – DLO (Related Rates)
Simple Feedback Condition
Application – DLO (Related Rates)
Positive Feedback Condition
Application – DLO (Related Rates)
Negative Feedback Condition
Related Rates
Post Test
DLO assigned is
dependent on the
treatment condition
Application – DLO (Related Rates) – Moodle
Module 2
FEEDBACK IN DIGITAL LEARNING OBJECTS 75
Data Analysis
A one-way analysis of variance (ANOVA) was used for each digital learning object
separately to initially analyze the results for the levels of feedback using a p-value with a level of
significance of <0.05. Further two-way univariate analyses of variance (ANOVA) were used to
analyze the results for the computer experience independent variable. A two way analysis of
variance was chosen as all of the independent variables in the study were categorical.
Consideration was given to using an analysis of covariance (ANCOVA) instead of a two-way
ANOVA design, however, an ANCOVA design requires one of the independent variables to be
categorical and one of the independent variables to be continuous. This is because ANCOVA is a
linear model blending both ANOVA and regression. Without a continuous independent variable
a regression model cannot be run and ANCOVA is not a possible method of analysis.
The two-way analyses were only run if the distribution of the sample sizes for the
independent variables allowed for statistical analysis. If sample sizes warranted, a two-way
ANOVA was used to compare differential levels of feedback with each of the individual
different independent variables and post-hoc tests were conducted as appropriate.
Assumptions and Limitations
One of the assumptions in this study is that all of the participants who have the same
level of experience with the topics will have the same current knowledge. That is, participants
whose prior knowledge is identified as having completed grade 12 math and physics courses may
not necessarily be at the same current level of ability. This is because the time between
completing their grade 12 math and physics courses and when the participant begins post-
FEEDBACK IN DIGITAL LEARNING OBJECTS 76
secondary studies varies significantly. Additionally, simply completing a high school math and
physics course may not always be an indicator of success in a post-secondary calculus class.
Another potential limitation of the study is that there many not be a homogenous sample
in terms of computer experience. As Kay and Knaack (2008b) note, the differences in computer
experience that existed in the past are currently not as pronounced since computers are now
consistently used at a secondary school level. Consequently, depending on the distribution of the
participants’ computer experience, statistical analysis may not be possible as sample sizes may
be insufficient.
FEEDBACK IN DIGITAL LEARNING OBJECTS 77
CHAPTER FOUR – FINDINGS
Introduction
In this study, the purpose was to investigate the effect on student achievement scores of
student driven feedback embedded in digital learning objects designed for post-secondary
calculus students. The effectiveness of the feedback was determined by examining the effect of
differential levels of elaborated feedback embedded into the digital learning objects. Effect of the
feedback was determined by comparing post-test scores of the different feedback conditions to
determine if there was a statistically significant difference in learning between the feedback
conditions. Additionally, the effect of embedded feedback was investigated in relation to learner
characteristics. Learner characteristics were measured in a number of ways, primarily in terms
of computer experience. Other measures of learner characteristics were also determined which
included demographic data as well as time on task while completing the lesson and guided
practice sections of the digital learning object. While these additional measures of learner
characteristics such as age and gender were not included in the research question, the data was
readily accessible and collected. These additional characteristics were easily analyzed for
potential opportunities for further research. Data collection methods and analysis of the data is
presented in this chapter along with a discussion of the reliability and validity of the findings.
Additionally, results from the additional comments question at the end of the post study survey
are summarized and presented.
Demographic Data
A total of 145 engineering students, enrolled in seven different 1st year calculus classes
consented to participate in the study, however, only 141 participants re-consented to participate
as four participants could not be contacted for re-consent. The demographic data was collected
FEEDBACK IN DIGITAL LEARNING OBJECTS 78
in the pre-study questionnaire, and of the 141 students participating in the study, only 105
participants chose to complete the pre-study questionnaire which was comprised of demographic
questions, questions on previous math experience, and questions on computer experience. Two
demographic questions were also asked in the questionnaire which were age and gender. While
age and gender were not directly identified as variables in the study, age and gender were
identified as possible considerations in the literature in terms of computer attitude. Kay and
Knaack (2008b) noted that males tended to have much more positive attitudes and higher ability
in respect to computer use, and age is a significant factor in both attitude and performance in
regards to digital learning objects. They also found that post-secondary students performed
significantly better and had a more positive attitude than secondary students.
In terms of age, the majority of the participants who provided demographic data were age
26 or less with only 11 participants (10%) 27 years of age or older. Participants 27 or older
ranged in age from 27 – 44. Of the participants who were age 26 or less, 52 of the participants
(50%) were in the range of 18-20, 21 participants (20%) were in the range 21-23, and 21
participants (20%) were in the range 24-26 (Error! Reference source not found.). The full d
istribution of ages is shown in the graph in Figure 16: Age Distribution.
Table 2: Age Distribution
Age Count % 18-20 52 50% 21-23 21 20% 24-26 21 20% 27-29 4 4% 30-32 3 3% 33-35 1 1% 36-38 2 2% >38 1 1%
Total 105 100%
FEEDBACK IN DIGITAL LEARNING OBJECTS 79
Figure 16: Age Distribution
Gender was also determined with males being over-represented in the sample comprising
75% of the total (Table 3: Gender ).
Table 3: Gender
Gender Total Male 79
Female 26 Total 105
Tests and Data Collection Methods
All of the data in the study was collected using surveys and tests in the Moodle Learning
Management System. As already noted, Moodle was selected because it was the Learning
Management System used by the Institute for all course delivery and all of the participants were
familiar with the use and functionality of Moodle. The Moodle Learning Management System
was also selected because it included functionality that supported the delivery of the digital
learning objects by treatment condition as well as easy setup and administration of the surveys
and tests.
02468
1012141618202224
18 20 22 24 26 28 30 32 34 36 38 40 42 44
Co
un
t
Age
Age Distribution
FEEDBACK IN DIGITAL LEARNING OBJECTS 80
The study involved two independent variables which was determined using the pre-study
survey, the task matching questions, and the post-test.
The first independent variable was defined as the type of feedback which was embedded
into each of three digital learning objects. The levels of feedback were defined as simple
feedback, positive feedback and negative feedback with positive and negative feedback being
response contingent elaborative feedback (Shute, 2008; Van der Kleij, Feskens & Eggen, 2015).
The other independent variable was computer experience and was determined by way of
a pre-study survey. The pre-study survey (Appendix 3) was a brief survey that the participants
answered at the beginning of the study that was used to determine the participant’s age, gender,
and computer experience. Computer experience was determined by a series of questions on the
survey that looked at the participants past and current use of computers in the areas of social
media use, experience with online courses, video game uses and self reported level of computer
experience questions. Additionally, computer attitude was determined using the Computer
Attitude Scale developed by Nickell & Pinto (1986). The Computer Attitude Scale is a series of
20 psychometrically tested questions that returns an overall computer attitude score between 20
and 100 with 20 indicating a strong negative attitude towards computers and 100 indicating a
strong positive attitude towards computers.
The dependent variable was measured as the participants’ scores on the post-test. The
post-test (Appendix 4) consisted of multiple choice questions developed for each digital learning
object to measure learning. Each test was also set up and delivered using the Moodle Learning
Management System which allowed for easy delivery of the questions to all of the participants as
well as providing statistical analysis of the questions. All of the questions were designed to test
FEEDBACK IN DIGITAL LEARNING OBJECTS 81
the learning of the concepts presented in the digital learning objects. Additionally, the first two
questions in each post test were not scored, and were instead used to capture the time on task by
asking the participants to record the times taken to complete the lesson and guided practice
portions of the digital learning object.
The Moodle Learning Management System also allowed for easy export of the data into
excel while also calculating statistics for reliability of the tests. Error! Reference source not f
ound. below represents the summary analysis of the three post-tests provided by Moodle.
Table 5 is a summary of the statistical analysis provided at the question level for the post-
test questions for each of the digital learning objects. The discriminative index was calculated by
Moodle which determines a correlation between the test question and the participants score on
the quiz. For both digital learning object one and digital learning object two the discriminative
index was at reasonable levels for the majority of the questions with the exception of question
four for digital learning object two which had a discriminative index of 0.09. For the third DLO
which was the Analysis DLO, there were a number of questions with a low discriminative index
which may indicate that the test questions were potentially too easy or too difficult which
contributed to a low level of correlation between the question score and the test score. This is
supported by examining the facility index which determines the percent of participants who
correctly answered the question. The questions with the lowest discriminative indices were also
the questions that had the higher or lowest facility index scores. Additionally, the questions
themselves may be poorly worded which may have led to the misinterpretations or
misunderstanding. While a low discriminative index score doesn’t necessarily mean that the test
or questions are poorly written, it does suggest that the questions should be examined to ensure
that they are well written and not ambiguous. In many cases the questions may be purposefully
FEEDBACK IN DIGITAL LEARNING OBJECTS 82
written so that they are either easy or difficult. Question six for the Analysis DLO had a
negative discriminative index. Although the value is low, the negative correlation indicates that
those who did well on the test did poorly on this question. A close examination of the question
indicates that there was nothing that would indicate that this question was poorly written or
didn’t test what was intended. In fact, the wording on the question was identical to two other
questions (5 and 7) except for the diagram. This may also indicate that the explanations in the
digital learning objects may have led to a misunderstanding.
Table 4: Post test summary analysis
Digital Learning Object
Number of
attempts
Average
grade
Median
grade
Standard
deviation
Application DLO 148 76.2% 83.0% 24.9%
Evaluation DLO 117 72.7% 78.5% 23.8%
Analysis DLO 120 62.0% 67.4% 19.2%
Table 5: Post test question analysis
Digital
Learning
Object
Question
number
Facility
index
Intended
weight
Effective
weight
Discrimination
index
Application
DLO
1 89.9% 10.0% 8.0% 0.3335
2 89.2% 10.0% 7.2% 0.2698
3 87.2% 10.0% 9.5% 0.3740
4 66.2% 20.0% 22.2% 0.4945
5 78.4% 10.0% 12.5% 0.3984
6 77.0% 20.0% 20.3% 0.4598
7 65.5% 20.0% 20.3% 0.3287
Evaluation
DLO
1 78.6% 11.1% 10.8% 0.4569
2 71.8% 11.1% 11.7% 0.4339
3 70.9% 11.1% 10.6% 0.3529
4 88.9% 11.1% 6.0% 0.0977
5 73.5% 11.1% 10.5% 0.3803
6 60.7% 11.1% 12.6% 0.4941
7 76.1% 11.1% 10.8% 0.3831
8 66.7% 11.1% 13.7% 0.6143
9 67.5% 11.1% 13.1% 0.5270
Analysis
DLO
1 81.7% 14.3% 15.1% 0.3212
2 84.2% 14.3% 15.3% 0.3516
3 42.5% 14.3% 17.2% 0.1147
4 90.8% 14.3% 8.9% 0.0542
5 57.5% 14.3% 18.1% 0.2021
6 22.5% 14.3% 6.3% -0.1920
7 55.0% 14.3% 19.1% 0.2928
FEEDBACK IN DIGITAL LEARNING OBJECTS 83
At the conclusion of the study, each participant was asked to complete a short survey
comprising of a single open-ended comment box. The survey question was designed to
determine the experience the participants had using the digital learning objects.
Missing Data
In total, there were 141 students who participated in the study. There were 48 students in
the simple feedback condition, 45 students in the negative feedback condition and 48 students in
the positive feedback condition. All of the participants were asked to complete the pre-test and
post-test surveys. However, the participants were not required to complete the surveys so a
number of participants chose not to complete the surveys. Additionally, when the pre-test
survey was administered to the first group of participants, it was determined that the labelling on
the likert scales for the survey were incorrectly labelled, consequently the entire survey for that
group had to be removed from the data set. This accounted for 14 incomplete pre-study surveys.
Consequently, 105 of the 141 participants completed the pre-test surveys. The post-test surveys
had a much lower completion rate with only 56 participants completing the survey. This is
possibly because the last digital learning objects were delivered in the final two weeks of class
with just enough time for the participants to complete the digital learning objects and the test.
The participants were instructed by their teacher to complete the post-test survey on their own
time, however, many did not do so.
For the post-tests for the digital learning objects, there was also missing data with a
number of factors contributing to the missing data. First, participants were not required to
complete the test after they completed the digital learning objects; however, the number who
selected not to complete the test was very low as almost all of the participants who were present
FEEDBACK IN DIGITAL LEARNING OBJECTS 84
in class to complete the digital learning objects also completed the tests. The majority of the
participants who did not complete the post-tests did not attend the sessions for the digital
learning objects and so did not complete the digital learning objects. While a number of factors
may have contributed to the reasons for the participants not attending for one or more of the
three sessions for the digital learning objects; the primary reason was provided by the instructors.
That is, for a number of the scheduled sessions for the digital learning objects, the participants
had assignments due or major exams on the same day in the days following. This resulted in a
number of the participants electing not to participate in one or more of the sessions because of
conflicting priorities. At the end of the study, 119 participants completed the post-test for the
Application DLO, 98 participants completed the post-test for the Evaluation DLO, and 103
participants completed the post-test for the Analysis DLO.
Finding – Independent Variable 1: Difference in learning based on the levels of
feedback
The results of the post-test for each of the digital learning objects were first analyzed
using only the type of feedback as the independent variable and the scores on the post tests as the
dependent variable. For this first analysis, no consideration was given to computer experience.
The results for type of feedback were analyzed using a fixed effect, one-way randomized groups,
analysis of variance (ANOVA). Omnibus tests were completed for each digital learning object
separately followed by post hoc pair wise comparisons. To determine if pair wise comparisons
were appropriate to conduct, p values of <0.1 were used to as determining factor. This is in
alignment with Simon (2006) who suggests that p values of >= 0.05 and <0.1 indicate that there
is suggestive evidence against the null hypothesis. Suggestive evidence is not used as an
FEEDBACK IN DIGITAL LEARNING OBJECTS 85
indication of statistically significant results; instead it is used as a determining factor to conduct
further analysis.
Application DLO (Related Rates)
An analysis of variance for the results of the Application DLO found no significant main
effect for the type of feedback (simple feedback, positive feedback, negative feedback) for the
participants score in a post-test at a p <0.05 level for the three levels of feedback F(2,115) =
.072, p = .93, p2= .001. Additionally, Levene’s test of homogeneity of variance indicated
unequal variances (F = 6.03, p = 0.003) so post hoc pair wise comparisons were conducted using
the Dunnett T3 test (Tabachnick & Fidell, 2007). Results of Dunnett’s T3 indicated that test
scores for the simple feedback condition did not differ significantly from test scores with positive
feedback condition (p = .98) and from test scores with negative feedback condition (p = .99).
Test scores from the positive feedback condition also did not differ significantly from the test
scores from the negative feedback condition (p = 1.00). A means plot below represents the
marginal means of the test scores for the Application DLO.
Figure 17: Marginal Means for the Application DLO
FEEDBACK IN DIGITAL LEARNING OBJECTS 86
Evaluation DLO (Differentials)
An analysis of variance for the results of the Evaluation DLO found no significant main
effect for the type of feedback (simple feedback, positive feedback, negative feedback) for the
participants score in a post-test at a p <0.05 level for the three levels of feedback F(2,95) = 2.00,
p = .14, p2= .040. Levene’s test of homogeneity of variance indicated equal variances (F =
1.05, p = 0.35) so post hoc pair wise comparisons were conducted using the Tukey HSD test
(Tabachnick & Fidell, 2007). Results of Tukey HSD indicated that test scores for the simple
feedback condition did not differ significantly from test scores with positive feedback condition
(p = .25) and from test scores with negative feedback condition (p = .17). Test scores from the
positive feedback condition also did not differ significantly from the test scores from the
negative feedback condition (p = .954). A means plot below represents the marginal means of
the test scores for the Evaluation DLO.
Figure 18: Marginal Means for the Evaluation DLO
Analysis DLO (Areas by Integration)
FEEDBACK IN DIGITAL LEARNING OBJECTS 87
An analysis of variance for the results of the Analysis DLO found no significant main
effect for the type of feedback (simple feedback, positive feedback, negative feedback) for the
participants score in a post-test at a p <0.05 level for the three levels of feedback F(2,100) = .45,
p = .64, p2= .009. Levene’s test of homogeneity of variance indicated equal variances (F =
2.64, p = 0.076) so post hoc pair wise comparisons were conducted using the Tukey HSD test.
Results of Tukey HSD indicated that test scores for the simple feedback condition did not differ
significantly from test scores with positive feedback condition (p = .68) and from test scores with
negative feedback condition (p = .89). Test scores from the positive feedback condition also did
not differ significantly from the test scores from the negative feedback condition (p = .99). A
means plot below represents the marginal means of the test scores for the Analysis DLO.
Figure 19: Marginal Means for the Analysis DLO
Finding – Independent Variable 2: Difference in learning with differential feedback
based on participants’ computer experience
The results of the post tests for each of the digital learning objects were next analyzed
based on differential feedback and computer experience (Independent variable two) using
factorial randomized-groups, fixed effects design and Pearson correlation where appropriate.
FEEDBACK IN DIGITAL LEARNING OBJECTS 88
The first independent variable for all of the analyses was levels of feedback and the second
independent variable was computer experience. Computer experience was measured five ways
using the pre-study survey (appendix 3). The first measure of computer experience determined
how many online courses the participant had previously completed. The second measure of
computer experience determined the frequency the participant played video games both in the
past and in the present. The third measure of computer experience determined the frequency the
participant used social media in the past and the present. The fourth measure of computer
experience, which could be better described as computer attitude, was determined using the
Computer Attitude Scale developed by Nickell and Pinto (1986). The last measure of computer
experience was determined from the participants’ self rating of their level of computer
experience on a likert scale. Post hoc pair wise comparisons were conducted where appropriate.
Levels of Feedback and Number of Online Courses.
The first measure of computer experience was a question on the pre-study survey that
determined that the number of online course the participant completed. 63 participants had not
completed an online course, 18 participants had completed one online course, six participants
had completed 2 online courses, one participant had completed three online courses, seven
participants had completed four online courses, and four participants had completed five online
courses. Sample sizes were only large enough to complete analyses for participants who had not
completed an online course or had completed one online course. 3x2 ANOVA’s were conducted
for each of the three digital learning objects. Analyses used feedback and number of online
courses (0, 1) as between-subjects factors, and post hoc analyses were conducted as appropriate.
FEEDBACK IN DIGITAL LEARNING OBJECTS 89
The results for the Application DLO revealed no significant main effect for levels of
feedback, F(2,68) = .81, p =.45, p2= .023, and no main effect for number of online courses
F(1,68) = .54, p = .45, p2= .008. Additionally, the interaction between levels of feedback and
number of online courses was not found to be significant F(2,68) = 2.86, p = .064, p2= .078.
Since no main effect was found for levels of feedback, post hoc analyses were not conducted.
The results for the Evaluation DLO also revealed no significant main effect for levels of
feedback, F(2,53) = .63, p =.54, p2= .023, and no main effect for number of online courses
F(1,53) = 1.49, p = .23, p2= .027. Additionally, the interaction between levels of feedback and
number of online courses was not found to be significant F(2,53) = .39, p = .68, p2= .015. Since
no main effect was found for levels of feedback, post hoc analyses were not conducted.
As with the results for the Application DLO and the Evaluation DLO, the results for the
Analysis DLO also revealed no significant main effect for levels of feedback, F(2,53) = .39, p
=.68, p2= .015, and a no main effect for number of online courses F(1,53) = .004, p = .95, p
2<
.001. Additionally, the interaction between levels of feedback and number of online courses was
not found to be significant F(2,53) = 1.91, p = .16, p2= .067. Since no main effect was found for
levels of feedback, post hoc analyses were not conducted.
Levels of Feedback and Video Game Usage
The second measure of computer experience was determined from a question on the pre-
study survey that asked the participant to indicate their frequency of video game usage. The
question asked the participants to indicate how much time they spent playing video games, on
average, per week, during the following:
FEEDBACK IN DIGITAL LEARNING OBJECTS 90
Not at all 1 – 3 hours
4 – 6 hours
7 -10 hours
>10 hours
In Recent Weeks
While in High School
While in Junior High
While in Elementary
A total video games usage score was calculated for each participant by assigning values
for each category. The category “not at all” was assigned a score of one and “>10 hours” was
assigned a score of five in order to quantify the results in order to determine comparative scores.
The other three categories were assigned scores of two through four respectively. Potential video
game usage scores could range from four through twenty and a chart of the frequency
distribution of the scores are shown in Figure 20: Frequency Distribution of Video Game Usage.
The results for the video game usage and the post-tests were analyzed to determine the level of
correlation between the video game usage score and the post-tests scores. Additionally, 3x4
ANOVA’s were conducted for each of the three digital learning objects. Analyses used feedback
and video game usage scores as between-subjects factors, and post hoc analyses were conducted
as appropriate.
Figure 20: Frequency Distribution of Video Game Usage
Correlations were completed for all of the participants for each digital learning object.
Additionally, the data for each digital learning object was sorted by feedback type and
FEEDBACK IN DIGITAL LEARNING OBJECTS 91
correlations were completed to determine if there was any difference in the correlations between
the video game usage scores and the post-test scores based on levels of feedback. Figure 21
below shows the correlation between the video game usage scores and post-test scores.
Significant correlations (p < .05) were only found for the negative feedback condition for the
Application DLO, Pearson’s r(24) = .48, p = .048 and for the positive feedback condition for the
Evaluation DLO, Pearson’s r(22) = .505, p = .016. The correlations were small as indicated in
the scatter plots for both as shown in Figure 222 and Figure 233 below.
Figure 21: Video Game Usage Correlations
Application DLO
Evaluation DLO
Analysis DLO
All Pearson Correlation 0.124 0.181 0.171 Sig. 2-tailed 0.291 0.159 0.179 N 74 62 63
simple feedback Pearson Correlation -0.05 0.006 0.344 Sig. 2-tailed 0.82 0.978 0.138 N 23 22 20
positive feedback Pearson Correlation 0.127 0.505 0.002 Sig. 2-tailed 0.527 0.016 0.992 N 27 22 22
negative feedback Pearson Correlation 0.408 0.031 0.298 Sig. 2-tailed 0.048 0.903 0.189 N 24 18 21
Figure 22: Application DLO (negative feedback condition) Scatter plot – Video Game Usage
FEEDBACK IN DIGITAL LEARNING OBJECTS 92
Figure 23: Evaluation DLO (positive feedback condition) Scatter plot – Video Game Usage
The results of the ANOVA for the Application DLO revealed no significant main effect
for levels of feedback, F(2,62) = .11, p =.90, p2= .004, and no main effect for video game usage
F(3,62) = 1.19, p = .32, p2= .054. Additionally, the interaction between levels of feedback and
video game usage was not found to be significant F(6,62) = .59, p = .74, p2= .054. Since no
main effect was found for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Evaluation DLO also revealed no significant main
effect for levels of feedback, F(2,50) = .65, p =.52, p2= .025, and no main effect for video game
usage F(3,50) = .99, p = .40, p2= .056. Additionally, the interaction between levels of feedback
and video game usage was not found to be significant F(6,50) = 1.08, p = .39, p2= .12. Since no
main effect was found for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Analysis DLO also revealed no significant main effect
for levels of feedback, F(2,51) = .44, p =.65, p2 = .017, and no main effect for video game
usage F(3,51) = 1.36, p = .27, p2 = .074. Additionally, the interaction between levels of
feedback and video game usage was not found to be significant F(6,51) = .83, p = .56, p2 =
FEEDBACK IN DIGITAL LEARNING OBJECTS 93
.089. Since no main effect was found for levels of feedback, post hoc analyses were not
conducted.
Levels of Feedback and Social Networking
The third measure of computer experience was determined from a question on the pre-
study survey that asked the participant to indicate their frequency of social networking usage
recently and historically. The question asked the participants to indicate how much time they
spent using social networking (ie, Facebook, Twitter, Instagram, LinkedIn, Pinterest), on
average, per week, during the following:
Not at all
1 – 3 hours
4 – 6 hours
7 -10 hours
>10 hours
In Recent Weeks
While in High School
While in Junior High
While in Elementary
As with video game usage, social networking usage score was calculated for each
participant by assigning the same scores for each category. The category “not at all” was
assigned a score of one and “>10 hours” was assigned a score of five in order to determine an
overall video usage score. The other three categories were assigned scores of two through four
respectively. Potential social networking usage scores could range from four through twenty and
a chart of the frequency distribution of the scores are shown in Figure 2424. The results for the
social networking usage and the post-tests were analyzed to determine the level of correlation
between the social networking usage score and the post-tests scores. Additionally, 3x4
ANOVA’s were conducted for each of the three digital learning objects. Analyses used feedback
and social networking usage scores (4-7, 8-11, 12-15, 16-20) as between-subjects factors, and
post hoc analyses were conducted as appropriate.
FEEDBACK IN DIGITAL LEARNING OBJECTS 94
Figure 24: Frequency Distribution of Social Networking Usage
Correlations were completed for all of the participants for each digital learning object.
Additionally, the data for each digital learning object was sorted by feedback type and
correlations were completed to determine if there was any difference in the correlations between
the social networking usage scores and the post-test scores based on levels of feedback. Figure
2525 below shows the correlation between the social networking usage scores and post-test
scores. No significant correlations (p < .05) were found between social networking usage scores
and post-test scores.
Figure 25: Social Media Usage Correlations
Application DLO
Evaluation DLO
Analysis DLO
All Pearson Correlation 0.037 -0.051 -0.068 Sig. 2-tailed 0.723 0.655 0.551 N 94 78 78
simple feedback Pearson Correlation 0.04 -0.215 -0.042 Sig. 2-tailed 0.827 0.271 0.841 N 32 28 25
positive feedback Pearson Correlation -0.066 0.166 -0.108 Sig. 2-tailed 0.723 0.39 0.593 N 31 29 27
negative feedback Pearson Correlation 0.287 -0.034 0.007 Sig. 2-tailed 117 0.883 0.972 N 31 21 26
The results of the ANOVA for Application DLO revealed no significant main effect for
levels of feedback, F(2,83) = .98, p =.38, p2= .023, and no main effect for social networking
0
5
10
15
20
25
30
35
4 6 8 10 12 14 16 18 20
Fre
qu
en
cy
Soical Networking Usage Score
Frequency of Social Networking Usage
FEEDBACK IN DIGITAL LEARNING OBJECTS 95
usage F(3,83) = .83, p = .48, p2= .029. Additionally, the interaction between levels of feedback
and social networking usage was not found to be significant F(5,83) = .68, p = .64, p2= .039.
Since no main effect was found for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Evaluation DLO revealed no significant main effect
for levels of feedback, F(2,68) = .72, p =.49, p2= .021, and no main effect for social networking
usage F(3,68) = .50, p = .69, p2= .022. Additionally, the interaction between levels of feedback
and social networking usage was not found to be significant F(4,68) = .78, p = .54, p2= .044.
Since no main effect was found for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Analysis DLO also revealed no significant main effect
for levels of feedback, F(2,68) = .29, p =.75, p2 = .008, and no main effect for social
networking usage F(3,68) = .29, p = .83, p2 = .013. Additionally, the interaction between levels
of feedback and social networking usage was not found to be significant F(4,68) = .22, p = .93,
p2 = .013. Since no main effect was found for levels of feedback, post hoc analyses were not
conducted.
Levels of Feedback and Computer Attitude
The next measure of computer experience was determined from a series of questions on
the pre-study survey which were from The Computer Attitude Scale developed by Nickell &
Pinto (1986). The computer attitude scale is a series of 20 questions measured on a likert scale
that determines the participant’s computer attitude score between 20 and 100 with a score of 20
representing an extremely low attitude towards computers and a score of 100 representing an
extremely high attitude towards computers. The computer attitude scores were assigned to four
FEEDBACK IN DIGITAL LEARNING OBJECTS 96
categories. The first category, “extremely low attitude” was assigned for scores ranging from
20-39, the second category, “low attitude” was assigned for scores ranging from 40-59, the third
category, “high attitude” was assigned for scores ranging from 60-79, and the last category, “
extremely high attitude” was assigned for scores ranging from 80-100. There were 102 responses
for computer attitude with 23 participant’s scores in the “extremely low attitude” category, 63
participants in the low attitude category, 16 in the high category, and no scores in the “extremely
high attitude” category. The frequency distribution for overall computer scores on computer
attitude is shown in Figure 26: Frequency Distribution of Computer Attitude below.
Figure 26: Frequency Distribution of Computer Attitude
A 3x2 ANOVA was conducted for the data from each of the three digital learning objects
with the independent variables of feedback (no feedback, positive feedback, negative feedback)
and computer attitude (extremely low attitude, low attitude, high attitude ) as between-subjects
factors.
The results for the Application DLO revealed no significant main effect for levels of
feedback, F(2,86) = .42, p =.66, p2= .010, and computer attitude, F(2,86) = 1.26, p = .66, p
2=
.010. Additionally, the interaction between levels of feedback and computer attitude was not
0
10
20
30
40
50
60
70
80
90
100
110
extremely low low high extremely high
Computer Attitude Score
Computer Attitude Frequency
FEEDBACK IN DIGITAL LEARNING OBJECTS 97
found to be significant F(4,86) = .71, p = .59, p2= .032. Since no main effect was found for
levels of feedback, post hoc analyses were not conducted.
The results for the Evaluation DLO revealed a significant main effect for levels of
feedback, F(2,69) = 3.27, p =.044, p2= 0.087 and no significant main effect for computer
attitude, F(2,69) = .46, p = .63, p2= .013. Additionally, the interaction between levels of
feedback and computer attitude showed a suggestive significant main effect F(4,69) = 2.45, p =
.055, p2= .12. Since a main effect was found for levels of feedback, post hoc analyses were
conducted. Separate one way ANOVAs were conducted separately for the levels of feedback for
three levels of computer attitude (extremely low attitude, low attitude, high attitude). Extremely
high levels of computer attitude were omitted in the analysis since there were no participants
with scores in the extremely high level category.
The first post hoc, one way ANOVA for the Evaluation DLO was conducted on the test
scores for only participants who had an extremely low attitude towards computers. The ANOVA
used feedback as the independent variable and indicated that there was a significant main effect
for feedback F(2,14) = 4.43, p =.032, p2= 0.39. Levene’s test of homogeneity of variance
indicated equal variances (F = .95, p = .41) so post hoc pair wise comparisons were conducted
using the Tukey HSD test. Results of the Tukey HSD test indicated that test scores for the
simple feedback condition did differ significantly from test scores with positive feedback
condition (p = .033), and there was a suggestive effect from test scores with negative feedback
condition (p = .067). Additionally, test scores from the positive feedback condition did not differ
significantly from the test scores from the negative feedback condition (p = .860). The mean
score for participants in the simple feedback group was 3.80, the mean score for participants in
FEEDBACK IN DIGITAL LEARNING OBJECTS 98
the positive feedback group was 8.50 and the mean score for the negative feedback group was
7.80. While the results were found to be significant, the sample size for this group was very low
with N=15.
The second post hoc one way ANOVA for the Evaluation DLO was conducted on the test
scores for only participants who had low computer attitude. The ANOVA used the feedback
independent variable and indicated that there was no significant main effect for feedback
F(2,844) =.72, p =.49, p2= 0.032. Consequently, post hoc tests comparisons were not conducted
for the test scores from the participants who had low computer attitude.
The third post hoc one way ANOVA for the Evaluation DLO could not be conducted on
the test scores for only participants who had high computer attitude because the sample size was
too small with N=14 and only one participant was in the negative feedback condition.
The results for the Analysis DLO revealed no significant main effect for levels of
feedback, F(2,68) = 1.36, p =.26, p2= 0.038, and for computer attitude, F(2,68) = 2.23, p = .12,
p2= .061. Additionally, the interaction between levels of feedback and computer attitude was
not found to be significant F(4,68) = 1.68, p = .17, p2= .090. Since no main effect was found for
levels of feedback, post hoc analyses were not conducted.
Levels of Feedback and Computer Experience
The last measure of computer experience was determined from a question on the pre-
study survey that asked the participant to self rate their level of computer experience as no
experience, little experience, average experience or high experience. Two participants rated
themselves as having little experience, 59 participants rated themselves as having average
FEEDBACK IN DIGITAL LEARNING OBJECTS 99
experience, and 39 participants rated themselves as having high experience. No participants
rated themselves as having no experience. Given that no participants rated themselves with no
experience and only two participants rated themselves as having little experience, analyses were
conducted for only participants with average or high experience. 3x2 ANOVAs were conducted
for each of the three digital learning. Analyses used feedback and computer experience (average
experience, high experience) as between-subjects factors, and post hoc analyses were conducted
as appropriate.
The results of the ANOVA for the Application DLO revealed no significant main effect
for levels of feedback, F(2,85) = .31, p =.74, p2= .007, and no main effect for computer
experience F(1,85) = .76, p = .39, p2= .009. Additionally, the interaction between levels of
feedback and computer experience was not found to be significant F(2,85) = .23, p = .80, p2=
.005. Since no main effect was found for levels of feedback, post hoc analyses were not
conducted.
The results of the ANOVA for the Evaluation DLO also revealed no significant main
effect for levels of feedback, F(2,68) = .17, p =.84, p2= .005, and no main effect for computer
experience F(1,68) = .94, p = .34, p2= .014. Additionally, the interaction between levels of
feedback and computer experience was not found to be significant F(2,68) = .64, p = .53, p2=
.019. Since no main effect was found for levels of feedback, post hoc analyses were not
conducted.
The results of the ANOVA for the Analysis DLO also revealed no significant main effect
for levels of feedback, F(2,69) = .53, p =.59, p2 = .015, and no main effect for computer
FEEDBACK IN DIGITAL LEARNING OBJECTS 100
experience F(1,69) = .37, p = .55, p2 = .005. Additionally, the interaction between levels of
feedback and computer experience was not found to be significant F(2,69) = .85, p = .43, p2 =
.024. Since no main effect was found for levels of feedback, post hoc analyses were not
conducted.
Other Analyses
In addition to measures of computer experience, additional data was collected throughout
the course of the study. Two additional sets of data were collected using the pre-study survey.
These were gender and age, and the post tests for each of the digital learning objects were
analyzed based on differential feedback and these two factors. While age and gender were not
directly identified as variables in the study, age and gender were identified as possible
considerations in the literature in terms of differences in learning and computer attitude when
using digital learning objects. Kay and Knaack (2008b) noted that males tended to have much
more positive attitudes and higher ability in respect to computer use, and age is a significant
factor in both attitude and performance in regards to digital learning objects. They also found
that post-secondary students performed significantly better and had a more positive attitude than
secondary students.
The data for gender was analyzed using an ANOVA and the data for age was analyzed
using a Pearson correlation. Additionally, while the participants were completing the digital
learning objects, the time on task was recorded and an analysis completed below to determine if
there was a correlation between time on task and their score on the post test. There were two
separate measures for time on task. The first measured the time the participant took to complete
the lesson portion of the digital learning object and the second measured the time the participant
FEEDBACK IN DIGITAL LEARNING OBJECTS 101
took to complete the guided practice portion of the digital learning objects. The data was
analyzed separately for each digital learning object. Post hoc pair wise comparisons were
conducted where appropriate.
Levels of Feedback and Gender
Of the participants who completed the pre-study survey, 77 participants were male and 26
were female. 3x2 ANOVAs were conducted for each of the three digital learning objects as well
as for the combined data from all of the learning objects. Analyses used feedback and gender
(male, female) as between-subjects factors, and post hoc analyses were conducted as appropriate.
The results of the ANOVA for the Application DLO revealed no significant main effect
for levels of feedback, F(2,90) = .94, p =.34, p2= .020, and no main effect for gender F(1,90) =
1.88, p = .17, p2= .020. Additionally, the interaction between levels of feedback and gender
was not found to be significant F(2,90) = .30, p = .74, p2= .007. Since no main effect was found
for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Evaluation DLO also revealed no significant main
effect for levels of feedback, F(2,73) = .58, p =.56, p2= .016, and no main effect for gender
F(1,73 ) = .78, p = .38, p2= .011. Additionally, the interaction between levels of feedback and
gender was not found to be significant F(2,73) = .12, p = .89, p2= .003. Since no main effect
was found for levels of feedback, post hoc analyses were not conducted.
The results of the ANOVA for the Analysis DLO also revealed no significant main effect
for levels of feedback, F(2,72) = 1.70, p =.19, p2 = .045, and no main effect for gender F(1,72)
= .60, p = .44, p2 = .008. Additionally, the interaction between levels of feedback and gender
FEEDBACK IN DIGITAL LEARNING OBJECTS 102
was not found to be significant F(2,72) = 1.33, p = .27, p2 = .036. Since no main effect was
found for levels of feedback, post hoc analyses were not conducted.
Levels of Feedback and Age
Age was determined from the pre-study survey and an analysis was conducted using
Pearson correlation to determine if there was a correlation between the age and the post-test
scores. Correlations were completed for all of the participants for each digital learning object,
and the data for each digital learning object was sorted by feedback type. Correlations were
completed to determine if there were any significant differences in the correlations between the
participants’ age and their post-test scores based on levels of feedback. Figure 27: Frequency
Distribution of Age below shows the frequency distribution of the ages of the participants and
Table 66 shows correlations between the participants’ ages and their post-test scores. No
significant correlations (p < .05) were found between age and post-tests scores.
Figure 27: Frequency Distribution of Age
0
5
10
15
20
25
30
35
40
Fre
qu
en
cy
Age (years)
Frequency - Age
FEEDBACK IN DIGITAL LEARNING OBJECTS 103
Table 6: Age Correlations
Application DLO
Evaluation DLO
Analysis DLO
All feedback Pearson Correlation -.089 .076 .123 combined Sig. 2-tailed 0.386 0.504 .283
N 96 79 78
simple feedback Pearson Correlation -.012 -0.039 .111 Sig. 2-tailed 0.949 0.844 0.597 N 32 28 25
positive feedback Pearson Correlation -0.191 0.109 .048 Sig. 2-tailed 0.295 0.575 .813 N 32 29 27
negative feedback Pearson Correlation -.046 .055 .107 Sig. 2-tailed .804 0.808 .612 N 32 22 26
Levels of Feedback and Time on Task
The digital learning objects the participants completed were organized into two parts. The
first part was a lesson that participants completed where the participants were not asked to
complete any questions during the lesson portion of the digital learning objects and simple
feedback was given. The second section of the digital learning objects was a guided practice part
where the participants were guided step by step through a problem and asked multiple choice
questions during each step of the problem. Feedback for each answer was either, simple
feedback, positive feedback or negative feedback depending on which treatment condition the
participant was placed in. Both the lesson and the guided practice parts were timed to determine
the total time on task in seconds. Pearson correlations were completed for the data for both the
guided practice part and the lesson part to determine if there were significant correlations
between time on task and post-test scores.
Correlations were completed for the time on task for both the lesson part and the guided
practice part of the digital learning objects. The data for each digital learning object was sorted
by feedback type, and correlations were completed using the participants’ post-test scores and
both the time on task on the lesson part and the time on task on the guided practice part of the
digital learning objects. Table 7 shows correlations between the participants’ time on task for the
FEEDBACK IN DIGITAL LEARNING OBJECTS 104
lesson part and their post-test scores, and Table 8shows correlations between the participants’
time on task for the guided practice part and their post-test scores. Significant correlations (p <
.05) for the lesson time and the guided practice time compared to the post-test scores were only
found for the results of the Evaluation DLO. The lesson time for Evaluation DLO had
significant correlation with the post-test scores for the simple feedback condition, Pearson’s r
(30) = .43, p = .019 and the negative feedback condition, Pearson’s r (26) = .49, p = .011. The
guided practice time for Evaluation DLO had significant correlation with the post-test scores for
the simple feedback condition, Pearson’s r(30) = .46, p = .011 and the positive feedback
condition, Pearson’s r (33) = -.32, p = .067.
Table 7: Lesson Time Correlations
Application DLO
Evaluation DLO
Analysis DLO
All Pearson Correlation -0.175 0.149 0.05 Sig. 2-tailed 0.065 0.164 0.635 N 112 89 94
simple feedback Pearson Correlation -0.336 0.426 0.258 Sig. 2-tailed 0.034 0.019 0.176 N 40 30 29
positive feedback Pearson Correlation -0.128 -0.124 -0.052 Sig. 2-tailed 0.463 0.491 0.762 N 35 33 36
negative feedback Pearson Correlation 0.091 0.491 0.058 Sig. 2-tailed 0.593 0.011 0.766 N 37 26 29
Table 8: Guided Practice Time Correlations
Application DLO
Evaluation DLO
Analysis DLO
All Pearson Correlation -0.129 -0.045 0.006 Sig. 2-tailed 0.174 0.674 0.955 N 112 89 94
simple feedback Pearson Correlation -0.128 0.459 .171. Sig. 2-tailed 0.431 0.011 0.375 N 40 30 29
positive feedback Pearson Correlation -0.195 -0.322 -0.153 Sig. 2-tailed 0.262 0.067 0.373 N 35 33 36
negative feedback Pearson Correlation -0.035 0.032 0.197 Sig. 2-tailed 0.836 0.878 0.306 N 37 26 29
FEEDBACK IN DIGITAL LEARNING OBJECTS 105
Levels of Feedback and Guided Practice Time on Task
ANOVAs for each digital learning object were also conducted with the time on task for
the guided practice part of the digital learning objects as the dependent variable and feedback
(simple feedback, positive feedback, negative feedback) as the between subject-factor. The
ANOVAs were used to determine if there was a significant effect of feedback on the time on task
for the guided practice part of each digital learning object. Each of the ANOVAs was conducted
separately for the results of each digital learning object.
The results of the ANOVAs indicated that there were no significant main effects for
levels of feedback with guide practice time as the dependent variable for any of the digital
learning objects as well as the combined data from all of the digital learning objects. The
ANOVA results for each of the application, evaluation and analysis digital learning objects were
F(2,109) = .45, p =.64, p2= .008, F(2,86) = .94, p =.40, p
2= .021, F(2,91) = 1.41, p =.25, p2=
.030 respectively. Since no main effect was found for levels of feedback, post hoc analyses were
not conducted.
Participant comments
Participants were asked to provide additional open-ended comments about the digital
learning objects at the end of the post-study survey (appendix 5). While the comments were not
designed to address a particular research question or to be used for explanatory purposes as
described by Creswell and Plano Clark (2011), it did offer some insight into the experience the
participants had using the digital learning objects and are summarized and discussed below in
relation to the research questions. Additionally, Kay and Knaack (2008a) have shown student
attitude about the use of digital learning objects was dependent on the education level of the
FEEDBACK IN DIGITAL LEARNING OBJECTS 106
student with a greater percentage of students in post-secondary reporting a positive attitude
towards digital learning objects as compared to secondary students. Kay and Knaack (2008a)
looked at a number of studies that investigated student attitude towards learning objects with
post-secondary students and found that eight studies reported positive student attitude, one study
reported neutral attitude and one study reported a negative attitude. Post-secondary students
typically gave positive comments about characteristics such as animations, self-assessment,
attractiveness, control over learning, ease of use, feedback, scaffolding or support, interactivity,
navigation and self efficacy. Consequently, collecting open-ended comments about student
attitude was seen to be of some value as student attitude about the use of digital learning objects
was related to the independent variables of levels of feedback, and previous knowledge.
The comments were analyzed by coding and grouping into themes. Three major themes
emerged from the analysis which were positive comments, suggestions for improvement and
negative comments. The first theme represented positive comments totalling 59 which is
consistent with what Kay (2014) reported about student attitude and using digital learning
objects. Other themes were suggestions for improvements totalling 5 and negative comments
which totalled to 3. Each of the themes were then sub themed with 9 sub themes emerging for
the positive codes with two additional codes that did not fit into one of the sub themes. There
weren’t sufficient numbers of codes that were suggestions for improvements or negative
comments to create sub themes.
FEEDBACK IN DIGITAL LEARNING OBJECTS 107
Table 9 below represents the themes, sub themes and count of each.
FEEDBACK IN DIGITAL LEARNING OBJECTS 108
Table 9: Themed Participant Comments
Theme Sub Themes/codes Count
Positive comments Good experience 16
DLOs are a good supplement to classroom
teaching 12
DLOs are a good learning tool 7
The DLOs are easy to understand 7
The DLOs were helpful 5
Having steps was good 4
Please implement ASAP 2
Liked that they were self paced 2
Had good visuals 2
Superior to MyMath Lab 1
Liked that it started at a beginner level 1
Suggestions for Improvement Needs more feedback 1
Needs more questions at the end 1
Would like more examples 1
Would like voice over 1
Would like more interaction 1
Negative comments Animations were distracting 1
Eyes hurt working on the DLOs for so long 1
Too many words to read 1
Validity and Reliability
Internal Validity
The design of the study protected against threats to internal validity in a number of ways.
One of the ways the design of the study protected against threats to internal validity was to
measure computer experience through multiple measures. This allowed for multiple analyses on
the same independent variable. Additionally, computer experience was not only determined by
multiple measures, but computer experience was also determined through a number of measures
that determined computer experience and a single measure that determined computer attitude.
Another way that the design of the study protected against threats to internal validity was by
having the participants complete multiple digital learning objects on topics that were unrelated
and none of the concepts of any of the topics presented prerequisite knowledge for any of the
others.
FEEDBACK IN DIGITAL LEARNING OBJECTS 109
External Validity
The design of the digital learning objects, the demographics of the participants, and the
topics chosen for the digital learning objects had an impact on the external validity of the results
of the study. The design of the learning objects impacted external validity in that the learning
objects were designed to be completed in a linear fashion in which the participants followed a
predetermined path of a lesson and guided practice. While this design helps to increase the
reliability of the results by ensuring all participants complete all parts of the digital learning
objects, to some degree it limits the external validity of the results as it may be difficult to
generalize the results to other designs. Additionally, the question types of the digital learning
objects were multiple choice questions. Multiple choice questions helped create consistency in
questioning and how feedback was given, thus increasing reliability, but potentially they
decreased the ability to generalize the results to digital learning objects with other question types.
Another threat to external validity in the study was the demographics of the participants.
All of the participants were post-secondary students enrolled in a Calculus class in the first year
of an Engineering Technology program. Other groups in terms of education program and age
may respond differently to different levels of embedded feedback.
The topics chosen for the study may have also posed a threat to external validity in terms
of the topic, the complexity of the topic, and type of problems. The topics selected were for the
digital learning objects were all topics that typically make up an introductory post-secondary
calculus course, which could impact whether the results could be generalized to other
mathematics topics or non-math topics. Also, the complexity and the type of problems may have
an impact on the external validity of the results. Two of the topics were more complex than the
FEEDBACK IN DIGITAL LEARNING OBJECTS 110
other in that the questions were word problems and involved multiple steps to learn and required
higher level thinking skills. The topic of the Evaluation DLO was not as complex and the
questions asked were not word problems but were instead problems that were solved
algebraically.
Reliability
There were a number of issues in regards to the reliability of the data which mostly
centered on the collection of the data. The first issue was that not every participant completed
the pre-study questionnaire. While the pre-study questionnaire was completely voluntary, there
is a risk that those participants who chose not to complete the pre-study questionnaire may have
been a homogeneous group and consequently impacted the random distribution of the
participants. The second issue was that that not all of the participants were able to complete
every digital learning object. The data collection occurred during the participants’ regularly
scheduled calculus classes; however, for various reasons not all were able to attend. The
instructors for the classes indicated that some participants did not attend some of the sessions
because there was a scheduled midterm examination in another course that the participants were
preparing for. This could have impacted the distribution of the student as there is a possibility
that the participants studying for other classes could have been similar in their ability or
experience which could have impacted the makeup of the sample. The third issue with
reliability is that there were two occasions where technical difficulties disrupted some of the
participants’ ability to complete the digital learning objects during the scheduled class period. In
both cases, the participants were encouraged to complete the digital learning objects at a later
time and provisions were made to extend the time that the digital learning objects and post-test
FEEDBACK IN DIGITAL LEARNING OBJECTS 111
were available to the participants. The threat to the reliability by doing this is that the
participants were not monitored when they completed the digital learning objects on their own.
However, clear directions were provided on the Moodle page and by the course instructor to only
rely on the learning from the digital learning objects to complete the post-tests. That last threat to
reliability was that it was not possible to control whether the participants reviewed the concept
presented in the digital learning objects prior to completing the digital learning objects. This
could have impacted the analyses for prior knowledge as their results on the post-tests may not
have been consistent with their reported prior knowledge in the pre-study survey.
Conclusion
This chapter analyzed the data from the various learning objects examining whether there
was a significant effect on learning by embedding feedback into digital learning objects.
Analyses were further completed by examining feedback in relation to various measures of
computer attitude in addition to gender and age which have been shown to be related to computer
attitude. Significant main effects were found for a number of analyses, most notably in relation
to computer experience and feedback. These are further discussed in the following chapter.
FEEDBACK IN DIGITAL LEARNING OBJECTS 112
CHAPTER FIVE – DISCUSSION
Van der Kleij, Feskens and Eggen (2015) identified a number of gaps in the literature in
the area of the effect of feedback in online learning.They recommend that future research
investigate the characteristics of the feedback as well as the task, the learning context and the
characteristics of the learners. This study extends the research in feedback in online research
identified by Van der Kleij, Feskens and Eggen (2015) by exploring levels of feedback as a
characteristic of online learning in the learning context of digital learning objects. Characteristics
of the learner were also explored in relation to levels of feedback.
The learning context selected for this study was digital learning objects. This is because
as Kay (2012) observes, that “ a comprehensive examination of factors that might influence the
effectiveness of learning objects including student characteristics, design of learning objects, and
teaching strategy has yet to be conducted” (p. 351). Additionally, Watson (2010) contends that
there has been an emphasis on the technological aspects of digital learning objects such as
reusability as well as the visual appearance of the digital learning objects, but there has been a
lack of research into the pedagogical basis for designing digital learning objects. This is also
supported by Chikh (2012) who adds that previous research has focussed on targeting learners as
consumers of digital learning objects or targeted instructors and designers and focussed on the
reusability of digital learning objects. By conducting this research specifically for digital
learning objects and the effectiveness of feedback, this study focuses on feedback as a design
element of digital learning objects that can increase learning rather than the technological aspects
of digital learning objects. Additionally, by selecting learning objects, this study controls for
learning context while extending the previous research on computer assisted instruction.
FEEDBACK IN DIGITAL LEARNING OBJECTS 113
The type of feedback selected for this study was a type of elaborated feedback that Shute
(2008) describes as response contingent feedback that describes why a correct response is correct
and why an incorrect response is incorrect. For this study these were identified as positive and
negative feedback respectively. Black and Wiliam (1998) and Davis and McGowen (2007) have
shown that in an face to face context, positive and negative feedback have a positive influence on
learning. This study also explores these findings in an online context using digital learning
objects and seeks to extend what had previously explored in computer assisted instruction. The
study was also designed to investigate the effect on achievement of a number of learner
characteristics in relation to embedded feedback, specifically, computer experience.
Additionally, age and gender were also examined as they have been shown to be related to
computer attitude (Kay and Knaack, 2008a).
It is hoped that this study can be the catalyst for further empirical studies that investigate
the factors that influence the design of digital learning objects. The findings and interpretations
presented in this section will discuss the results of the study in relation to the research questions
of study. This will be followed by recommendations as well as suggestions for further research.
Research Questions
1. Is there a difference in post-secondary students’ learning based on the type of feedback they
encounter while using an online digital learning object as a basis of instruction in an
introductory calculus class? Types of feedback in this study will be feedback that addresses
faulty interpretations and not a lack of understanding in the form of either positive or
negative feedback as described by Black and Wiliam (1998) and Hattie and Timperley
(2007).
FEEDBACK IN DIGITAL LEARNING OBJECTS 114
2. Is there a difference in post-secondary students’ learning based on the type of feedback they
encounter using an online digital learning object in relation to learner characteristics of
computer experience?
Findings and Interpretations based on the levels of feedback
The results of the 3x3 fixed effects ANOVA in this study did not find any statistically
significant effects for levels of feedback and consequently there was no evidence to reject the
first null hypothesis in the study which is: there is no difference in post-secondary students’
learning based on the type of feedback in online digital learning objects. However, a close
examination of the results from each of the three digital learning objects does show a consistent
pattern with all three digital learning objects; learning for the digital learning objects with
embedded feedback being greater than for the learning objects without feedback which suggests
further investigation of the results and an examination of the methods used
When examining each of the digital learning objects, the structure of all three learning
objects was identical in that the learning objects were all made up of two parts with a lesson
section first which was followed by a guided practice section with the feedback embedded into
the guided practice section of the digital learning object. The only identifiable difference
between the three digital learning objects was that the Application DLO and Analysis DLO were
both complex word problems whereas the Evaluation DLO was a simpler topic that identified the
algebraic steps to solve a problem. This suggests that complexity may also be a factor that
impacts the effect of the feedback in regards to learning when the feedback is embedded into the
digital learning objects.
FEEDBACK IN DIGITAL LEARNING OBJECTS 115
When examining the plots of all three digital learning objects and the plot of the
combined data (Figure 28,
Figure 29, Figure 3030) it can be seen that a pattern is emerging where the simple feedback
condition is consistently lower than the positive and negative feedback conditions whereas the
positive and negative feedback conditions are consistently higher than simple feedback. It should
be noted that the comparisons of the plots are comparing plots with different scales.
Additionally, the plots are from the post-test results from three different digital learning objects
and three different post-tests. Consequently comparisons are for examining trends only and not
absolute differences in values. While there wasn’t a significant difference shown for type of
feedback for any of the digital learning objects, the pattern suggest that embedding feedback into
digital learning objects could be a factor to further explore in the design in digital learning
objects. In addition, the complexity of the topic should also be considered in the design of the
digital learning objects when embedding feedback and that feedback may be more effective with
less complex concepts. In this case, significant results were found in relation to computer attitude
for the less complex, evaluation DLO which was only based on determining and evaluating
algebraic step. Significant results were not found for the synthesis and application DLOs which
were complex word problems. A possible explanation is that feedback with complex concepts
requires more feedback and more frequent feedback which may become confusing. This supports
Hatziapostolou and Paraskakis (2006) who contend that overly detailed and too much feedback
can result in the feedback being too confusing and overwhelming to be useful. Regardless, the
pattern shown in the plots suggest that feedback may be of some use regardless of the
complexity. This also aligns with Hatziapostolou and Paraskakis (2010) who note that feedback
FEEDBACK IN DIGITAL LEARNING OBJECTS 116
is important in all learning contexts. However, concepts that are less complex may also benefit
more from embedded feedback.
Figure 28: Application DLO – Marginal Means for Feedback
Figure 29: Evaluation DLO – Marginal Means for Feedback
FEEDBACK IN DIGITAL LEARNING OBJECTS 117
Figure 30: Analysis DLO – Marginal Means for Feedback
Difference in learning with differential levels of feedback based on participant’s
computer experience
The second set of analyses in this study was based on differential levels of feedback and
computer experience. Computer experience was measured five ways using categorical data. The
first four measures of computer experience were how many online courses the participant had
previously completed, the frequency the participant played video games, the frequency the
participant used social media, and a self rated level of computer experience measure on a likert
scale. The last measure of computer experience could instead be defined as a measure of
computer attitude and was determined using the Computer Attitude Scale developed by Nickell
& Pinto (1986). The first four measures of computer experience did not reveal significant effects
for levels of feedback and did not indicate and significant correlations. However, significant
main effects were found with levels of feedback and computer attitude as between subject
factors.
The separate analyses indicated that significant results were found in the Evaluation
DLO with p = 0.044. These results from the ANOVA of the Evaluation DLO show that
FEEDBACK IN DIGITAL LEARNING OBJECTS 118
computer attitude was a significant factor in determining whether participants benefited from
embedded feedback, but the results from the ANOVAs from the Application DLO and the
Analysis DLO did not indicate that computer attitude was a significant factor. The Evaluation
DLO was designed around a topic that is less complex than the topics for the Application DLO
and the Analysis DLO. This suggests that complexity should be further examined as an
additional factor in determining the effectiveness of embedding feedback in digital learning
objects.
These results suggest that it may be appropriate to reject the second null hypothesis of:
Ho: There is no difference in post-secondary students’ learning in digital learning objects with
differential levels of feedback based on the students’ computer experience. However, it was
found the there was a distinction between computer experience and computer attitude.
Consequently, while there is evidence to reject the null hypothesis, it would only be in relation to
computer attitude and not with computer experience. These results support Kay (2007) and Kay
(2012) who suggested that students who are more comfortable with computers would likely
benefit more from using digital learning objects as evidenced by the substantial amount of
research on self-efficacy and computer related behaviours. However, as Kay (2007) noted, little
research had been completed in regards to computer attitude and self efficacy in regards to the
use of digital learning objects.
Difference in learning with differential feedback based on other factors
Further analyses were conducted based on levels of feedback and a number of other
factors which include age, gender and time on task. An ANOVA for each DLO was used to
analyze the results for gender and time on task and no significant main effect was found with
FEEDBACK IN DIGITAL LEARNING OBJECTS 119
feedback and gender or time on task as between subject factors. Additionally, within each DLO,
there was no significant difference between the results between males and females. This is
consistent with what has been reported in the literature in regards to learner characteristics in
which there have been no observable significant differences between males and females or with
regard to age (Kay & Knaack, 2008b; Van Zele, Vandaele, Botteldooren, & Lenaerts, 2003).
Not surprisingly then, results of this study did not find that there was a significant difference
between the results of males and females on the post-study test, and there was no significant
main effect of feedback and gender as between subject factors. Age and time on task were also
examined to see if there was a correlation between levels of feedback and age and levels of
feedback and time on task. Only small correlations were found <0.5 and with no significant
evidence to suggest that there was a correlation between the different levels of feedback and the
factors of age and time on task.
Participant Comments
At the end of the study, all participants were provided the opportunity to provided open
ended comments about the experience in using the digital learning objects. The participant
comments from the post study survey were primarily positive comments and the sub themes
show that the majority of the participants liked using the digital learning objects with some of the
participants preferring to use the digital learning objects exclusively. While the comments were
overwhelmingly positive and the participants indicated they liked using the digital learning
objects, the participants were not able to comment on the efficacy of the embedded feedback as
they were unaware that there were different groups in the study who had received different
feedback conditions. One notable exception is a comment from a participant who was placed in
the simple feedback condition and who commented that more feedback is needed.
FEEDBACK IN DIGITAL LEARNING OBJECTS 120
All of the comments were consistent with what Kay and Knaack (2008a) and Kay (2014)
found in regards to student attitude with using digital learning objects. They found that students
typically gave positive comments about the characteristics of the digital learning objects such as
animations, self-assessment, attractiveness, control over learning, ease of use, feedback,
scaffolding or support, interactivity, navigation and self efficacy. Also, they found that negative
comments were focussed on the navigation, the technology and the workload which is consistent
with what the participants in this study reported. Another aspect of the design of the learning
objects that the participants reported finding valuable in this study is that they were interactive,
with explanations and visuals. This supports the study by Rieber, Tzeng and Tribble (2004) who
indicated the computer based learning with embedded explanations and graphical representations
were more effective than those without. Additionally, Lim et al. (2006) suggests that the level of
interactivity found in digital learning objects was also an important design element to consider.
A final factor to consider is that a number of the participants (n=12) did comment that the digital
learning objects were good, but would work best as supplement to classroom instruction. This is
supported by Nurmi and Jaakkola (2006) who found that learning of students using digital
learning objects alone was not significantly different from students in a blended setting (digital
learning objects and face to face) or to face to face only. They did find that the learning in a
blended (digital learning objects and face to face) setting was significantly better than learning in
a face to face setting only. This supports what many of the participants in this study suggested in
that they believe the digital learning objects would be a good supplement to a face to face setting.
Recommendations
This study explored a number of gaps in the literature on the effect of feedback in online
learning identified by Van der Kleij, Feskens and Eggen (2015). Type of feedback using digital
FEEDBACK IN DIGITAL LEARNING OBJECTS 121
learning objects as the learning context was explored as the primary focus of this study, where
levels of feedback were simple feedback and elaborated feedback which was in the forms of
positive or negative feedback as described by Black and Wiliam (1998) and Timberley (2007).
Additionally, learner characteristics such as computer experience and attitude, gender and age
were explored in relation to embedded feedback. In terms of the learning context of the study, as
Kay (2007) and Kay (2014) noted, digital learning objects could potentially revolutionize online
learning, but there is a lack of empirical studies focussing on the effective use and design of
digital learning objects. This does not mean that there is not an interest in developing and using
digital learning objects. Nurmi and Jaakkola (2006) noted that there continues to be enthusiasm
towards using and designing digital learning objects. These statements are now eight and nine
years old, and while the development of digital learning objects continues to grow with many
publishers and developers creating learning objects as part of their online supports for textbooks
on online courses, the state of research into the design of digital learning objects has changed
little. As Kay (2013) observed recently, “a comprehensive examination of factors that might
influence the effectiveness of learning objects including student characteristics, design of
learning objects, and teaching strategy has yet to be conducted” (p. 351).
Results from this study suggest to the developer of digital learning objects that all users
of digital learning objects could benefit to some degree from feedback in the learning objects as
evidenced by the consistent pattern when examining the marginal means of the results.
However, findings show that users with high computer attitude benefit significantly from
embedded feedback while users with low or extremely low computer attitude do not. However,
comments from participants were overwhelming positive regarding the use of digital learning
objects with most rating the experience good, useful, and easy to use which suggests that the
FEEDBACK IN DIGITAL LEARNING OBJECTS 122
learning objects themselves offered an extra level of motivation to learn. Also, participants
suggested that the digital learning objects would be best used in a blended learning environment
with both face to face and online modes of delivery available. While participants in the study
were not aware of the independent variable of feedback, the results would suggest that well
designed learning objects with embedded feedback would be beneficial to users particularly in a
blended learning setting and most useful for users with high computer attitude.
Researcher’s Reflections
When planning and designing this study, the researcher borrowed from 25 years of
experience of working in online education and teaching mathematics and science to both
secondary and post-secondary students. Over the past 15 years the researcher has been designing
and using digital learning objects as a part of his self-paced online courses and blended online
courses. Anecdotal evidence suggested to the researcher that his students enjoyed using the
digital learning objects and in some cases used them as their primary source for learning the
topic. All of the digital learning objects that the researcher had designed in the past were similar
to the lesson portion of the digital learning objects in this study; however, they did not include
guided practice with embedded feedback. Whenever students had a difficult time with a specific
concept presented in the digital learning objects, they would have to wait to contact the instructor
for extra help. While this was sufficient in a blended learning setting where a student has
regularly planned opportunities to interact with the instructor, the self-paced setting often
provided less opportunity for the students to meet or discuss the concepts with the instructor, and
any discussions were typically delayed. This suggested to the researcher that embedding
feedback into digital learning objects that was similar to the feedback that the instructor would
FEEDBACK IN DIGITAL LEARNING OBJECTS 123
provide could be beneficial which ultimately helped to frame the problem and purpose of the
study.
While the researcher had a keen interest in the results of the study in terms of the effect of
embedded feedback, the opportunity allowed for exploring various other factors that could
influence the effect of embedded feedback. The literature review also helped make evident that
not only was elaborated feedback, response contingent feedback a factor to consider in designing
digital learning objects, but that there were other factors to consider in relation to feedback for
which few empirical studies had been completed. This study fully explored a factor in the
effective design of digital learning objects using a controlled experiment and was the largest
found in terms of participants in relation to digital learning objects. In order to complete this
study, the researcher had to create three separate learning objects for each of three concepts and
then ensure that the objects were made available to the participants as part of their normal
activities in their classrooms while ensuring the timing of the delivery of the learning objects
occurred just before the face to face delivery of the topic. This necessitated that the researcher
explore and learn a delivery mechanism that was online and could deliver the correct learning
object to the appropriate randomly selected group without the participants being aware that there
were three treatment conditions. Also, the delivery of the learning objects had to be in a
platform that the participants were familiar with and was easy to use and set up so it was intuitive
for the participants to use. In consultation with a Moodle expert, the researcher was able to
employ Moodle as the delivery platform to meet the requirements for delivery and to provide the
testing capabilities necessary to conduct the study. As the researcher planned the study,
questions remained about how to best present the learning objects to the participants. The
researcher found that Moodle was not only appropriate, but has now demonstrated the
FEEDBACK IN DIGITAL LEARNING OBJECTS 124
functionality that was used for this study to other researchers who have similar methodological
needs.
Suggestions for Further Research
In analyzing the data, a number of topics for further research emerged. One of these
potential topics is exploring how the complexity of the concepts in the digital learning object
may interact with the effectiveness of the embedded feedback in digital learning objects. The
results of the study indicated that the complexity of the digital learning object content may have
a played a role in the overall effectiveness of the digital learning objects. The topics presented in
the digital learning objects in this study were word problems which represented analysis,
application and evaluation as cognitive functions which typically are more complex and require
higher level thinking skills as described by Bloom, Engelhart, Furst, Hill, and Krathwohl (1956).
Potential research topics would include exploring the appropriate level of complexity or amount
of information covered by the digital learning objects.
Summary and Conclusions
This quantitative study explored the effect of differentiating levels of feedback (simple,
positive and negative) as a type of elaborated feedback using digital learning object as the
learning context. A research hypothesis guided the study to explore whether computer
experience influenced the effectiveness of embedded feedback in digital learning objects. The
literature suggested that feedback is a significant factor in learning in a face to face setting and in
online setting. This implied that feedback may also be a significant factor in the design of digital
learning objects. This study with 141 participants randomly selected from post-secondary
engineering first calculus classes found that feedback is a significant factor for learning for
FEEDBACK IN DIGITAL LEARNING OBJECTS 125
participants with high computer experience. Comments from participants suggested that the
digital learning objects were useful as a learning tool both on their own or as a supplement to
learning in a face to face setting.
FEEDBACK IN DIGITAL LEARNING OBJECTS 126
REFERENCES
Ally, M., Cleveland-Innes, M., Boskic, N., & Larwill, S. (2006). Learners' Use of Learning
Objects. Journal of Distance Education, 21(2), 44-57.
Azevedo, R., & Bernard, R. M. (1995). The Effects of Computer-Presented Feedback on
Learning from Computer-Based Instruction: A Meta-Analysis. Journal of Educational
Computing Research, 13(2), 111-127.
Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education:
Principles, Policy & Practice, 18, 5–25.
Bennett, R., Braswell, J., Oranje, A., Sandene, B., Kaplan, B., & Yan, F. (2008). Does It Matter
if I Take My Mathematics Test on Computer? A Second Empirical Study of Mode
Effects in NAEP. Journal of Technology, Learning, and Assessment, 6(9),
Billings, D. (2000). A framework for assessing outcomes and practices in Web-based courses in
nursing. Journal of Nursing Education, 39(2), 60-67.
Black, P. & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education. 5
(1), 7-74.
Black, P., & Wiliam, D. (2010). Inside the Black Box: Raising Standards Through Classroom
Assessment. Phi Delta Kappan, 92(1), 81.
Blair, A., Wyburn-Powell, A., Goodwin, M., & Shields, S. (2014). Can dialogue help to improve
feedback on examinations? Studies In Higher Education, 39(6), 1039-1054.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (1956). Taxonomy
of educational objectives: The classification of educational goals. Handbook I: Cognitive
domain. New York: David McKay Company.
Blignaut, S., & Trollip, S. R. (2003). Developing a Taxonomy of Faculty Participation in
Asynchronous Learning Environments--An Exploratory Investigation. Computers &
Education, 41(2), 149-172.
Bonnel, W. (2008). Improving feedback to students in online courses. Nursing Education
Perspectives, 29(5), 290-294.
Bower, M. (2005). Online Assessment Feedback: Competitive, Individualistic, or… Preferred
Form! Journal of Computers in Mathematics and Science Teaching, 24(2), 121 – 147.
FEEDBACK IN DIGITAL LEARNING OBJECTS 127
Brown, W.E., Lovett, M., Bajzek, D. & Burnette, J.M. (2006). Improving the Feedback Cycle to
Improve Learning in Introductory Biology Using the Digital Dashboard. In T. Reeves &
S. Yamashita (Eds.), Proceedings of World Conference on E-Learning in Corporate,
Government, Healthcare, and Higher Education 2006 (pp. 1030-1035).
Butler, M., Pyzdrowski, L., Goodykoontz, A., & Walker, V. (2008). The Effects of Feedback on
Online Quizzes. International Journal for Technology in Mathematics Education, 15(4),
131-136.
Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education,
31(2), 219-233.
Cassady, J. C., Budenz-Anders, J., Pavlechko, G., & Mock, W. (2001). The Effects of Internet-
Based Formative and Summative Assessment on Test Anxiety, Perceptions of Threat,
and Achievement.
Chikh, A. (2014). Original Article: A general model of learning design objects. Journal Of King
Saud University - Computer And Information Sciences, 2629-40.
Chitwood, K., May, C., Bunnow, D., & Langan, T. (2002). Battle stories from the field:
Wisconsin Online Resource Center Learning Object Project. In D. Wiley (Ed.), The
instructional use of learning objects (pp. 203–222). Bloomington, IN: Agency for
Instructional Technology.
Cobb, K., Billings, D., Mays, R., & Canty-Mitchell, J. (2001). Peer review of teaching in Web-
based courses in nursing. Nurse Educator, 26(6), 274-279.
Collis, B., & Strijker, A. (2003) Re-usable learning objects in context. International Journal on
E-Learning 2, 5–16.
Collis, B., & Strijker, A. (2004, May). Technology and human issues in reusing learning objects.
Journal of Interactive Media in Education, 4. Special Issue on the Educational Semantic
Web.
Corbett. A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring:
Impact on Learning rate, achievement and attitudes. In Proceedings of ACM CHI 2001
Conference on Human Factors in Computing Systems. (pp.245-252). New York:
Association for Computing Machinery Press.
Creswell, J., & Plano Clark, V. (2011). Designing and Conducting Mixed Methods Research.
Los Angeles. CA: Sage.
Crichton, S. & Li, Q. (2004). Expert practice and the development of learning objects.
Educational Technology, 44(2), 37–42.
FEEDBACK IN DIGITAL LEARNING OBJECTS 128
De Salas K. & Ellis L. (2006) The development and implementation of learning objects in a
higher education. Interdisciplinary Journal of E-Learning and Learning Objects, 2, 1–22.
Davis, G. E., & McGowen, M. A. (2007). Formative Feedback and the Mindful Teaching of
Mathematics. Australian Senior Mathematics Journal, 21(1), 19-29.
Deci, E. L., & Ryan, R. (1985). Intrinsic motivation and self-determination in human behaviour.
New York: Plenum.
Diane, S. & Richards, L. (2004). Online Feedback Strategies for Learning and Teaching. In J.
Nall & R. Robson (Eds.), Proceedings of World Conference on E-Learning in Corporate,
Government, Healthcare, and Higher Education 2004 (pp. 1174-1180).
Diekelmann, N., & Mendias, E. P. (2005). Being a Supportive Presence in Online Courses:
Attending to Students' Online Presence with Each Other. Journal of Nursing Education,
44(9), 393-395.
Douglas, T., Salter, S., Iglesias, M., Dowlman, M., & Eri, R. (2016). The Feedback Process:
Perspectives of First and Second Year Undergraduate Students in the Disciplines of
Education, Health Science and Nursing. Journal Of University Teaching And Learning
Practice, 13(1).
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical
power analysis program for the social, behavioral, and biomedical sciences.
Fleming, M., & Levine, H.W (1978). Instructional message design: Principles from the
behavioural sciences. Englewood Cliffs, NJ: Educational Technology Publications.
Fyfe, E. R. (2016). Providing feedback on computer-based algebra homework in middle-school
classrooms. Computers In Human Behavior, 63,568-574
Gadanidis, G., & Schindler, K. (2006). Learning Objects, Type II Applications, and Embedded
Pedagogical Models. Computers in the Schools, 23(1/2), 19-32.
Ghilay, Y., & Ghilay, R. (2015). FBL: Feedback Based Learning in Higher Education. Higher
Education Studies, 5(5), 1-10.
Hattie, J. (2012). Know Thy Impact The effects of feedback, although positive overall, are
remarkably variable. Educational Leadership, 70(1), 18-23.
Hattie, J. & Timperley, H. (2007). The power of feedback, J. Review of Educational Research,
87 (1), 81–112.
Hattie, J. A., Biggs, J., & Purdie, N. (1996). Effects of learning skills intervention on student
learning: A meta-analysis. Review of Research in Education, 66, 99–136.
FEEDBACK IN DIGITAL LEARNING OBJECTS 129
Hatziapostolou, T., & Paraskakis, I. (2010). Enhancing the Impact of Formative Feedback on
Student Learning through an Online Feedback System. Electronic Journal of e-Learning,
8(2), 111-122.
Hepplestone, S., & Chikwa, G. (2014). Understanding how students process and use feedback to
support their learning. Practitioner Research in Higher Education, 8(1), 41-53.
Higgins, R. (2000). ‘Be more critical’: rethinking assessment feedback, paper presented to
British Educational Research Association Conference, Cardiff University, 7–10
September. Available online at:
http://www.leeds.ac.uk/educol/documents/00001548.htm.
Hui, Y. (2012). What is a digital object? Metaphilosophy, 43(4), 380-395.
Jarvenpaa, S. L., & Leidner, D. E. (1998). Communication and Trust in Global Virtual Teams.
Journal of Computer-Mediated Communication, 3(4), 0.
Johnson, G. M., & Davies, S. M. (2014). Self-regulated learning in digital environments: Theory,
research,praxis. British Journal of Research, 1, 1–14.
Juwah, C., Macfarlane-Dick, D., Matthew, B., Nicol, D., and Smith, B. (2004). Enhancing
Student Learning though Effective Formative Feedback, The Higher Education Academy,
[Online],
http://www.heacademy.ac.uk/assets/York/documents/resources/resourcedatabase/id353_s
enlef_guide.pdf
Kay, R. (2007). A Systematic Evaluation of Learning Objects for Secondary School Students.
Journal of Educational Technology Systems, 35(4), 411-448.
Kay, R., & Knaack, L. (2008a). An examination of the impact of learning objects in secondary
school. Journal of Computer Assisted Learning, 24(6), 447-461.
Kay, R., & Knaack, L. (2008b). A formative analysis of individual differences in the
effectiveness of learning objects in secondary school. Computers & Education, 51(3),
1304-1320.
Kay, R., & Knaack, L. (2008c). Investigating the Use of Learning Objects for Secondary School
Mathematics. Interdisciplinary Journal of Knowledge & Learning Objects, 269-289.
Kay, R. (2007). A Systematic Evaluation of Learning Objects for Secondary School Students.
Journal of Educational Technology Systems, 35(4), 411-448.
FEEDBACK IN DIGITAL LEARNING OBJECTS 130
Kay, R. H. (2012). Examining Factors that Influence the Effectiveness of Learning Objects in
Mathematics Classrooms. Canadian Journal Of Science, Mathematics And Technology
Education, 12(4), 350-366.
Kay, R. (2014). Exploring the use of web-based learning tools in secondary school
classrooms. Interactive Learning Environments, 22(1), 67-83.
Kingston, N., & Nash, B. (2011). Formative assessment: A meta-analysis and a call for research.
Educational Measurement: Issues and Practice, 30, 28–37.
Klecker, B. M. (2007). The Impact of Formative Feedback on Student Learning in an Online
Classroom. Journal of Instructional Psychology, 34(3), 161.
Kopp, G., & Crichton, S. (2007). Embedded or linked learning objects: Implications for content
development, course design and classroom use. Canadian Journal of Learning &
Technology, 33(2), 1-20.
Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of
Educational Research, 58(1), 79–97.
Lefevre, D., & Cox, B. (2016). Feedback in Technology-Based Instruction: Learner Preferences.
British Journal Of Educational Technology, 47(2), 248-256.
Leibold, N., & Schwarz, L. M. (2015). The Art of Giving Online Feedback. Journal of Effective
Teaching, 15(1), 34-46.
Marie, J. A. (2016). Student Views on the Value of Feedback. Journal of Education and
Training Studies, 4(6), 207-213.
McGreal, R. (2004). Learning objects: A practical definition. The International Journal of
Instructional Technology and Distance Learning, 1(9).
McRobbie, C. J., Ginns, I. S., & Stein, S. J. (2000). Preservice primary teachers’ thinking about
technology and technology education. International Journal of Technology and Design
Education, 10, 81-101.
Mayer, R. E. (2001). Multimedia Learning. New York, NY: Cambridge University Press.
Miller, C., Doering, A. & Scharber, C. (2010). No such thing as failure, only feedback:
Designing innovative opportunities for e-assessment and technology-mediated feedback.
Journal of Interactive Learning Research, 21(1), 65-92.
FEEDBACK IN DIGITAL LEARNING OBJECTS 131
Miller, C., Hooper, S., Rose, S., Montalto-Rook, M. (2008). Transforming e-assessment in
American Sign Language: Pedagogical and technological enhancements in online
language learning and performance assessment. Learning, Media and Technology, 33(3),
155-168.
Muirhead, B. (2001). Enhancing Social Interaction in Computer-Mediated Distance Education.
Ed at a Distance Journal, 15(4), [online]
http://ifets.ieee.org.login.ezproxy.library.ualberta.ca/discussions/discuss_sept2000.html
Muller, E., Buteau, C., Ralph, B., & Mgombelo, J. (2009). Learning Mathematics through the
Design and Implementation of Exploratory and Learning Objects. International Journal
for Technology in Mathematics Education, 16(2), 63-73.
Nicol, D. & MacFarlane-Dick, D. (2004) Rethinking Formative Assessment in HE: A
Theoretical Model and Seven Principles of Good Feedback Practice, in Enhancing
Student Learning Through Effective Formative Feedback, York, HEA.
Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: a
model and seven principles of good feedback practice. Studies In Higher
Education, 31(2), 199-218.
Nickell, G. & Pinto, J. (1986) The Computer Attitude Scale, Computers in Human Behavior,2,
301-306.
Nguyen, D., Hsieh, Y., & Allen, D. (2006). The Effect of Receiving the Preferred Form of
Online Assessment Feedback upon Middle School Mathematics Students. Journal of
Computers in Mathematics and Science Teaching. 25(3), 251 - 279.
Nurmi, S., & Jaakkola, T. (2006). Effectiveness of Learning Objects in Various Instructional
Settings. Learning, Media & Technology, 31(3), 233-247.
OECD. (2015). Students, Computers and Learning: Making the Connection, OECD Publishing,
Paris. DOI: http://dx.doi.org/10.1787/9789264239555-en
Peat, M., & Franklin, S. (2003). Has Student Learning been Improved by the Use of Online and
Offline Formative Assessment Opportunities? Australian Journal of Educational
Technology, 19 (1), 87-99.
Pereira, D., Flores, M. A., Simão, A. V., & Barros, A. (2016). Effectiveness and relevance of
feedback in Higher Education: A study of undergraduate students. Studies in Educational
Evaluation, 497-14.
FEEDBACK IN DIGITAL LEARNING OBJECTS 132
Race, P. (2001). Using Feedback to Help Students Learn, The Higher Education Academy,
[Online],
http://www.heacademy.ac.uk/assets/York/documents/resources/resourcedatabase/id432_u
sing_feedback.pdf
Reece, A. A. (2016). Best Practices in Digital Object Development for Education: Promoting
Excellence and Innovation in Instructional Quality and Assessment. Journal Of Learning
Design, 9(1), 80-86.
Rieber, L.P., Tzeng, S., &Tribble, K. (2004) Discovery learning, representation, and explanation
within a computer-based simulation: finding the right mix. Learning and Instruction 14,
307–323.
Sadler, D. (1989). Formative Assessment and the Design of Instructional Systems. Instructional
Science, 18(2), 119-44.
Shute, V. J. (2008). Focus on Formative Feedback. Review of Educational Research, (1). 153.
Simon, M. (2006). Dissertation and Scholarly Research: Recipes for Success. Dubuque, IA:
Kendall Hunt Pub Co.
Sims, R. (2000). An interactive conundrum: Constructs of interactivity and learning theory.
Australian Journal of Educational Technology, 16(1), 45-57.
Small, F., & Attree, K. (2016). Undergraduate student responses to feedback: expectations and
experiences. Studies In Higher Education, 41(11), 2078-2094.
Sonak, B. C., Suen, H. K., Zappe, S. M., & Hunter, M. W. (2002). The Efforts of a Web-Based
Academic Record and Feedback System on Student Achievement at the Junior High
School Level.
Tabachnick, B. & Fidell, L. (2007). Experimental Design Using ANOVA. Belmont, CA:
Thomson Higher Education.
Thiele, J. (2003). Learning patterns of online students. Journal of Nursing Education, 42(8), 364-
366
Thurmond,V, & Wambach, K. (2004). Understanding interactions in distance education: A
review of the literature. International Journal of Instructional Technology and Distance
Learning, 1(1), 9-26.
Thurmond, V. (2003). Defining interaction and strategies to enhance interactions in Web-based
courses. Nurse Educator, 28(5), 237-241.
FEEDBACK IN DIGITAL LEARNING OBJECTS 133
Van der Kleij, F. M., Feskens, R. W., & Eggen, T. M. (2015). Effects of Feedback in a
Computer-Based Learning Environment on Students’ Learning Outcomes. Review Of
Educational Research, 85(4), 475.
Van-Dijk, D., & Kluger, A.N. (2001). Goal orientation versus self-regulation:Different labels or
different constructs? Paper presented at the 16th annual convention of the Society for
Industrial and Organizational Psychology, San Diego, CA.
Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementation and
Evaluation of a Course Concept Based on Reusable Learning Objects. Journal of
Educational Computing Research, 28(4), 355-372.
Watson, J. (2010). A Case Study: Developing Learning Objects with an Explicit Learning
Design. Electronic Journal of E-Learning, 8(1), 41-50.
Wang, T.-H. (2011). Developing web-based assessment strategies for facilitating junior high
school students to perform self-regulated learning in an e-learning environment.
Computers & Education, 57, 1801–1812.
Yacci, M. (2000). Interactivity Demystified: A Structural Definition for Distance Education and
Intelligent Computer-Based Instruction. Educational Technology, 40(4), 5-16.
Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the
enhancement of pedagogic practice. Higher Education, 45 (4), 477-501.
Yuliang, L. (2009). Providing Formative Feedback to Maximize Student Satisfaction and
Learning in Online Courses. International Journal of the Humanities, 7(4), 41-51.
Zimmerman, B. (2000). Self-Efficacy: an essential motive to learn. Contemporary Educational
Psychology, 25, 82–91.
FEEDBACK IN DIGITAL LEARNING OBJECTS 134
APPENDIX 1
Course Instructor: INFORMATION LETTER and CONSENT FORM
Study Title: Exploring Digital Learning Objects
Background You are being asked to participate in this study because you are the instructor of a Post-
Secondary Calculus course which comprises the sample identified for this study. The results of this study will be used in support of the researcher’s dissertation for
completion of a PhD in Educational Psychology. Purpose The purpose of this research study is to investigate the design of digital learning objects.
Digital learning objects are computer based learning tools that are designed to teach or support the learning of a specific concept. Digital learning objects can range from digital images, videos, simulations to interactive computer programs designed to teach a specific concept. In this study, the digital learning objects being investigated are interactive computer programs.
Research Questions: 1. Is there a difference in students’ learning based on the design of an online digital
learning object as a basis of instruction in an introductory calculus class? 2. Is there a difference in students’ learning based on the design on an online digital
learning object in relation to learner characteristic of the students’ comfort level in using a computer?
Study Procedures This study will involve five modules that will be accessed through a Moodle course which is
the Learning Management System (LMS) that all students at the Northern Alberta Institute of Technology (NAIT) use for the courses. Each module in Moodle will represent one part of the study.
o The first module will be an online survey that will ask some questions on basic demographic information such as age and gender. The survey will also include a set of questions on your students’ experience using a computer.
o The second, third and fourth modules will all be modules that will contain the digital learning objects that will be used in the study. Each digital learning object will contain three parts. The first part will be a lesson on a specific concept in calculus. The second part will be guided practice where your students will answer questions on the concept presented in the lesson. The third part of the digital learning object will be a short quiz on the concept presented in the digital learning object.
FEEDBACK IN DIGITAL LEARNING OBJECTS 135
o The fifth module in the study will be a short post-study survey designed to gather feedback from your students on their thoughts and experiences while using the digital learning objects.
Data to be collected.
o Demographic data collected in the first module - Gender, age, highest level of math course taken, time since last math course
taken, has the student completed a calculus course - A questionnaire which will include questions on a students’ prior experience
with using a computer and online learning. o Post-study survey. After your students have completed all of the digital learning
objects they will be asked to complete a survey on their thoughts and experiences while using the digital learning objects.
o Test scores will be collected from the quizzes taken after completing each of the three digital learning objects. All test scores from the quizzes will be collected during three 1 hour classroom periods. These one hour classroom periods will be part of the regularly scheduled activities of the class, and as such all students in your class will be expected to complete the digital learning activities. However, data for the study will only be collected from your students who have agreed to participate in the study. Students who have not agreed to participate in the study will access the digital learning objects through a guest Moodle account and will not be required to complete the pre and post study surveys.
o There will be no extra time commitment for your students other than the time needed to complete the pre and post study surveys which will take approximately 10 minutes each to complete.
Benefits
We hope that the information we get from doing this study will help us better understand how to better design digital learning objects to increase learning.
All students in your class will be expected to complete the digital learning objects as part of the regularly scheduled activities of the class. Your students will benefit from using the digital learning objects by having access to an alternative delivery of the concepts. However, it is expected that after your students have completed each concept using the digital learning objects in the study, you as the instructor will also teach the students the same concept as you would all other concepts in your course.
At the conclusion of the study, you as the instructor will need to make all of the digital learning objects available on your course Moodle page so that your students can benefit from them by reviewing them as needed.
There is no cost to you for your participation in the study and you will not receive any compensation.
Risk There are no foreseeable risks to being involved in this study.
FEEDBACK IN DIGITAL LEARNING OBJECTS 136
All data gathered in this study are to be used for the purpose of the study only. The results of the quizzes or surveys will not be made available to you as the instructor and cannot be used to form part of your students’ course evaluation.
Voluntary Participation You are under no obligation to participate in this study. Participation is completely voluntary,
and you are not obliged to answer any specific questions even if participating in the study. Even if you agree to be in the study you can change your mind and withdraw at any time.
Your students may also choose to opt out of the study at any time, and they will be given access to a guest Moodle account so that they would still be able to access the digital learning objects and not be disadvantaged by opting out. Students who withdraw from the study may request to have their data withdrawn from the study as well.
Confidentiality & Anonymity The results of this research will be used primarily for completing the dissertation
requirements for the researcher. However, the results may also be used in presentations and research articles. Neither you nor your students would be identified in any of these.
All data gathered for this study will be kept confidential with only the researcher and the supervisor able to access your students’ personal data. Only summary data will be reported in any publications.
You and your students will remain anonymous through the study. Each student will be randomly assigned a numbered Moodle account and only the researcher will be able to access the data. Students will be assigned to a random Moodle account by you as the instructor.
Research data will be kept electronically by the researcher for a period of five years following the completion of the study. All electronic data will be password protected and encrypted and stored on the researcher’s computer on a password protected account. No data will be kept that includes student names.
At the conclusion of the study, as the instructor, you will be asked to gather contact information from your students who wish to view a copy of the final report of the study.
There is a possibility that the data collected in this study may get used in future research, but if so, it will have to be approved by a Research Ethics Board.
Further Information If you have any further questions regarding this study, please do not hesitate to contact:
The plan for this study has been reviewed for its adherence to ethical guidelines by a
Research Ethics Board at the Researcher’s University and the Research Ethics Board at the Polytechnic Institute where the data was collected).
FEEDBACK IN DIGITAL LEARNING OBJECTS 137
APPENDIX 2
Student Participant: INFORMATION LETTER and CONSENT FORM
Study Title: Exploring Digital Learning Objects Background You are being asked to participate in this study because you are the student in a Post-
Secondary Calculus course which comprises the sample identified for this study. The results of this study will be used in support of the researcher’s dissertation for
completion of a PhD in Educational Psychology.
Purpose The purpose of this research study is to investigate the design of digital learning objects.
Digital learning objects are computer based learning tools that are designed to teach or support the learning of a specific concept. Digital learning objects can range from digital images, videos, simulations to interactive computer programs designed to teach a specific concept. In this study, the digital learning objects being investigated are interactive computer programs.
Research Questions: 1. Is there a difference in students’ learning based on the design of an online digital
learning object as a basis of instruction in an introductory calculus class? 2. Is there a difference in students’ learning based on the design on an online digital
learning object in relation to the students’ comfort level in using a computer? Study Procedures This study will involve five modules that will be accessed through a Moodle course which is
the Learning Management System (LMS) that all students at the Northern Alberta Institute of Technology (NAIT) use for the courses. Each module in Moodle will represent one part of the study.
o The first module will be an online survey that will ask some questions on basic demographic information such as age and gender. The survey will also include a set of questions on about your experience using a computer.
o The second, third and fourth modules will all be modules that will contain the digital learning objects that will be used in the study. Each digital learning object will contain three parts. The first part will be a lesson on a specific concept in calculus. The second part will be guided practice where you will answer questions on the concept presented in the lesson. The third part of the digital learning object will be a short quiz on the concept presented in the digital learning object.
o The fifth module in the study will be a short post-study survey designed to gather feedback on your thoughts and experiences while using the digital learning objects.
Data to be collected. o Demographic data collected in the first module
FEEDBACK IN DIGITAL LEARNING OBJECTS 138
- Gender, age, Citizenship, highest level of math course taken, time since last math course taken, have you completed a calculus course
- A questionnaire which will include questions on your prior experience with using a computer and online learning.
o Post-study survey. After you have completed all of the digital learning objects you will be asked to complete a survey your thoughts and experiences while using the digital learning objects.
o Test scores will be collected from the quizzes taken after completing each of the three digital learning objects. All test scores from the quizzes will be collected during three 1 hour classroom periods. These one hour classroom periods will be part of the regularly scheduled activities of the class, and as such you will be expected to complete the digital learning activities whether they choose to participate in the study. However, data for the study will only be collected about you if you agree to participate in the study. If you choose not to participate in the study you will access the digital learning objects through a guest Moodle account and will not be required to complete the pre and post study surveys.
o There will be no extra time commitment to participate in the study other than the time needed to complete the pre and post study surveys which will take approximately 10 minutes each to complete.
Benefits
We hope that the information we get from doing this study will help us better understand how to better design digital learning objects to increase learning.
You will be expected to complete the digital learning objects as part of the regularly scheduled activities of the class whether you participate in the study, and you will benefit from using the digital learning objects by having access to an alternative delivery of the concepts. However, after you have completed each concept using the digital learning objects in the study, your instructor will also teach the students the same concept as he/she would all other concepts in your course.
At the conclusion of the study, your instructor make all of the digital learning objects available on your course Moodle page so that you can benefit from them by reviewing them as needed.
There is no cost to you for your participation in the study and you will not receive any compensation.
Risk There are no foreseeable risks to being involved in this study. All data gathered in this study are to be used for the purpose of the study only. The results
of the quizzes or surveys will not be made available to your instructor and will not be used to form part of your course evaluation.
Voluntary Participation You are under no obligation to participate in this study. Participation is completely voluntary,
and you are not obliged to answer any specific questions even if participating in the study.
FEEDBACK IN DIGITAL LEARNING OBJECTS 139
Even if you agree to be in the study you can change your mind and withdraw at any time. You may also choose to opt out of the study at any time, and you will be given access to a guest Moodle account so that you would still be able to access the digital learning objects and not be disadvantaged by opting out. If you choose to withdraw from the study you may request to have your data withdrawn from the study as well.
Confidentiality & Anonymity The results of this research will be used primarily for completing the dissertation
requirements for the researcher. However, the results may also be used in presentations and research articles. There will not be any data presented in any of these formats that would allow individuals to be identifiable.
All data gathered for this study will be kept confidential with only the researcher and the supervisor able to access your personal data. Only summary data will be reported in any publications.
You will remain anonymous through the study. You will be randomly assigned a numbered Moodle account and only the researcher will be able to access the data. You will be assigned to a random Moodle account by your instructor.
Research data will be kept electronically by the researcher for a period of five years following the completion of the study. All electronic data will be password protected and encrypted and stored on the researcher’s computer on a password protected account. No data will be kept that includes your name or other identifiable information.
At the conclusion of the study your will be given an opportunity to add your name to a list of participants who would like a copy of the final report of the study when it is completed.
There is a possibility that the data collected in this study may get used in future research, but if so, it will have to be approved by a Research Ethics Board.
Further Information If you have any further questions regarding this study, please do not hesitate to contact:
Todd Sumner: E-mail: [email protected] Phone: 780.491.1348
The plan for this study has been reviewed for its adherence to ethical guidelines by a Research Ethics Board at the Researcher’s University and the Research Ethics Board at the Polytechnic Institute where the data was collected).
FEEDBACK IN DIGITAL LEARNING OBJECTS 140
APPENDIX 3
Pre-study Survey
The following are the pre-study survey questions for this study along with the number of
response categories in brackets. The pre-study survey consists of two parts, demographic
questions and computer experience questions.
Demographic Questions
1. What is your age
2. What is your gender
Male
Female
Computer Experience
3. How many online courses have you taken?
4. Please indicate how much time you spent PLAYING VIDEO GAMES, on average, PER
WEEK, during the following:
Not at all 1 – 3 hours
4 – 6 hours
7 -10 hours
>10 hours
In Recent Weeks
While in High School
While in Junior High
While in Elementary
5. Please indicate how much time you spent USING SOCIAL NETWORKING (ie, Facebook,
Twitter, Instagram, LinkedIn, Pinterest), on average, PER WEEK, during the following:
Not at all 1 – 3 hours
4 – 6 hours
7 -10 hours
>10 hours
In Recent Weeks
While in High School
While in Junior High
While in Elementary
6. How would you describe your experience using computers?
No Experience Little Experience Average Experience High Experience
Computer Attitude: Used with permission (Nickell & Pinto, 1986)
7. Please answer the following questions.
Strongly Agree Agree Neutral Disagree
Strongly Disagree
Computers will never replace human life.
Computers make me uncomfortable because I don't understand them.
People are becoming slaves to computers.
Computers are responsible for many of the good things we enjoy.
FEEDBACK IN DIGITAL LEARNING OBJECTS 141
Soon our lives will be controlled by computers.
I feel intimidated by computers.
There are unlimited possibilities of computer applications that haven't even been thought of yet.
Computers are lessening the importance of too many jobs now done by humans.
Computers are dehumanizing to society.
Computers turn people into just another number.
Computers are a fast and efficient means of gaining information.
The overuse of computers may be harmful and damaging to humans.
Computers can eliminate a lot of tedious work for people.
The use of computers is enhancing our standard of living.
Computers intimidate me because they seem so complex.
Computers will replace the need for working human beings.
Computers are bringing us into a bright new era.
Soon our world will be completely run by computers.
Life will be easier and faster with computers.
Computers are difficult to understand and frustrating to work with.
FEEDBACK IN DIGITAL LEARNING OBJECTS 142
APPENDIX 4
Post-Test Questions
Digital Learning Object 1 questions:
Please use the following question for the next 4 answers.
A rock is thrown into a pond and creates a circular wave radiating out from where the rock
entered the water. The radius (r) of the wave is increasing at a rate of 0.25 m/sec. How fast is the
Area (A) of the wave changing when the radius is 20 m?
1) What is the rate you are given in the question?
a) dr/dt = 0.25m/sec
b) dA/dt = 0.25 m/sec
c) A/t = 20 m/sec
d) r/t = 20 m
2) What is the rate you are asked to find?
a) dr/dt
b) dA/dt
c) A/t
d) r/t
3) Given the equation for an area of a circle is A = π r2. Find the derivatives with respect to
time.
a) dA/dt = 2 π r
b) A/t = 2 π r/t
c) dA/dt = 2 π r (dr/dt)
d) A = 2 π r (dr/dt)
4) How fast is the Area of the wave changing with respect to time when the
radius = 20 m?
a) 1.57 m2/sec
b) 125.6 m2/sec
c) 31.4 m2/sec
d) 1256 m2/sec
FEEDBACK IN DIGITAL LEARNING OBJECTS 143
Please use the following question for the next 2 questions.
The electric resistance (R) (in Ω) of a resistor as a function of temperature (T) (in C) is
R=4.000+0.003T 2. If the temperature is increasing at the rate of 0.1000 C/sec, find how fast the
resistance changes when T = 150C
5) Find the derivatives with respect to time.
a) dR/dt = 0.003T(dT/dt)
b) dR/dt = 4 + 0.006 (dT/dt)
c) dR/dt = (4 + 0.006T) (dT/dt)
d) dR/dt = 0.006T (dT/dt)
6) How fast is the resistance changing when T = 150C?
a) 4.090 Ω/sec
b) 0.0900 Ω/sec
c) 71.50 Ω/sec
d) 0.0450 Ω/sec
7) A metal cube dissolves in acid such that an edge of the cube decreases by 0.50 mm/min.
How fast is the volume of the cube changing when the edge is 8.20 mm?
a) -202 mm3/min
b) -6.15 mm3/min
c) -101 mm3/min
d) -12.3 mm3/min
Digital Learning Object 2 questions:
1) What is the formula for finding Δy?
a) Δy = f '(x) dx
b) Δy = f(x+Δx)
c) Δy = x2 - x1
d) Δy = f(x+Δx) - f(x)
2) The differential is defined as:
a) dy = f '(x) dx
b) dy/dx = f '(x)
c) dy = f(x+Δx) - f(x)
d) dy = f(x+Δx) dx
FEEDBACK IN DIGITAL LEARNING OBJECTS 144
3) Find the differential of the following function. y = 4x3 - 3x2 + 4
a) dy = 12x2 – (6x) dx
b) dy/dx = 12x2 - 6x
c) dy = (12x2 - 6x) dx
d) dy = (4x3 - 3x2 + 4) dx
4) Find the differential of the following function. y = 4(x2 - 3)3
a) dy = 24x(x2 - 3)2 dx
b) dy = 12(x2-3)2 dx
c) dy/dx = 24x(x2 - 3)2
d) dy = 4(x2 - 3)3 dx
Please use the following information to answer the next 3 questions
y = 6x2 - 3x, x = 6, Δx = 0.2
5) What is the differential of the function?
a) dy = 12x - 3 dx
b) dy = 12x – 3
c) dy = (12x - 3) dx
d) dy/dx = 12x – 3
6) Calculate Δy
a) 212.04
b) 198
c) 212.4
d) 14.04
7) Calculate dy
a) 13.8
b) 71.4
c) 54
d) -3.6
FEEDBACK IN DIGITAL LEARNING OBJECTS 145
Use the following information to answer the next 2 questions
y = (4x2 - 10)2 x = 2, Δx = 0.01
8) What is Δy? (rounded to two decimal places)
a) 37.95
b) 36.00
c) 1.95
d) 1.92
9) What is dy? (Rounded to two decimal places)
a) 1.95
b) 1.92
c) 192.00
d) -3.20
Digital Learning Object 3 questions:
1) What is the equation you would use to find the area between two curves using horizontal
elements of area?
a)
b)
c)
d)
2) What is the equation you would use to find the area between two curves using vertical
elements of area?
a)
b)
c)
d)
FEEDBACK IN DIGITAL LEARNING OBJECTS 146
3) Why could you not use vertical rectangular elements of area for the following?
a) The left boundary is unknown
b) Vertical elements cannot be used for parabolas with this orientation
c) There is not one single curve representing the bottom boundary
d) The lower boundary is negative
4) Integrate the following.
a)
b)
c)
d)
5) Which equation would you use to find the area by integration?
a)
b)
c)
d)
FEEDBACK IN DIGITAL LEARNING OBJECTS 147
6) Which equation would you use to find the area by integration?
a)
b)
c)
d)
7) Which equation would you use to find the area by integration?
a)
b)
c)
d)
FEEDBACK IN DIGITAL LEARNING OBJECTS 148
APPENDIX 5
Participant Comments
The following represent the comments submitted by the participants on the last question of the
post-study survey. The last question was a field entitled “Additional Comments.” The comments
have not been edited for grammar, punctuation or spelling.
Very good tool to be used along with learning from your instructor for people who need that
extra help.
The digital learning objects helped me to understand differentials and area integration by
providing visual representations of the steps involved when computing integrals and
differentials. I feel that the digital learning objects presented will greatly aid in my final
exam for this class. Thank you for providing the opportunity to be involved in this study.
Prior to this, I had never used a computer based learning object at this level of school/math,
so it was a cool and different way to learn. I can see why you're completing your doctorate in
this field. Well structured, good job!
Great experience.
It was a nice experience.
possibly add more interaction in the learning lesson portion
Because Digital learning objects, application of deriavaties, integrating and limits are very
easy to understand.
Good.
The DLOS was well presented to students but more questions could be added to lengthen the
time spent on the DLOS and to ensure the student is understanding all aspects of the topics.
There are a lot of concepts covered in each DLOS and a few more questions would allow for
greater understating.
Provide notes package (fill in the blank) in order for students to review once they leave the
classroom. There is no motivation to take notes when using the computer.
I feel that in order to increase the effectiveness of DLO, assignments should be given in order
to encourage the students to develop their techniques otherwise the students will neglect
these short lectures in time but overall, DLO is very helpful in teaching the basics.
More feedback would be useful.
Sometime I feel that there are too many words to read.
Great if it can mix with classroom instructor.
Please implement this ASAP.
I found the material helpful. I liked the individually paced aspect.
I personally do not learn well from the online learning objects. Could be used as an
additional learning supplement for students but not the same as a teacher.
It was a good thing to learn from because it was easy to understand the procedures.
FEEDBACK IN DIGITAL LEARNING OBJECTS 149
It was a good system to use to learn.
It was quick and easy.
It was a good experience, in future if we are getting opportunity to learn from digital module,
that would be great.
The instructor in classroom is better than digital learning, but digital helps to get extra
practice!!
It was really nice experience but if possible add voice guidance for DLO that would be great
experience.
The digital learning objects are very good for learning how to do a certain skill or process
(such as solving for areas using integration). Because of this I feel like they would be useful
when used along side classroom learning - as something extra to help people understand a
process. I do not feel that digital learning objects are the most efficient way to teach the
theory behind a process (such as why the integral can be used to solve for area). For things
such as theory or understanding of concepts and why the process works, I feel like the
classroom is a superior learning environment.
Easy to follow.
There are some people who are old fashion and can only accept learning trough pencil and
paper. as the technology advances I think we should also take advantage of it. digital learning
is an other tool that can go into student tool box. when a job is assign, if you don't have the
right tools, you might not be precise or will take longer to get the end. I think that digital
learning is an essential tool student can use to learn. it allow us to repeat the steps and proses
as many times as we need to understand what is going on. if we missed a class or were sick
for a week, it it a simple way to catch up. I find that NAIT needs to adapt to digital learning.
They were good.
It should display the result of the post study questions.
I have experience with using My Math Lab and had struggled with its use far more than this
on line learning supplement.
The digital learning objects were helpful for giving a visual representation of the problems
presented and showed a clear way to solve the problems.
The digital learning objects were well organized and I like how they outlined the steps for
each question.
Its aight.
The DLOS does not assume the student know anything of the topic. It begin teaching the
lesson from a beginner level.
Would be better if we got more attempts on quiz . maybe indicate what section you can reefer
back to if answered question wrong.
DLO was good as an additional tool to use outside of the classroom but would not want to
learn a whole course based purely on DLO.
It was an easy way to learn. Should be implemented to teach online courses.
It was a decent experience overall.
FEEDBACK IN DIGITAL LEARNING OBJECTS 150
It was great, but any questions i had were unable to be answered without an instructor.
Thanks for opportunity. I personally prefer Digital Learning mixed with some hours lecturer
to ask any question or need any explanation. It might work perfectly fine for MATH but not
for other courses which need more explanation to understand. However if someone has
passed some math courses before it might help perfectly but I believe we still need instructor
to understand 100 percent of contents.
It was a helpful substitute to classroom lectures.
My eyes kind hurt for watching computer too long then i can not think.
It was okay.
Good Stuff.
Very useful and easy to learn.
More examples!!!
Very neat and very easy to understand the topic that has being discuss.
Digital Learning Objects are really helpful for students and easy to understand.
The only point I would add is that the animations ie. arrows moving and things moving on
the page, made it hard for me to keep concentrated on reading and not be distracted. (In short
they made it hard to read)
i don't think this could ever replace a traditional teacher.
Should be able to do the digital learning object pre test after the lesson as well as before to
see how much you learned.
The digital learning object is useful for extra information or review but I prefer being taught
by an instructor as the main method of learning.
It's really helpful but in class learning is still the way to go.
Cool course. I really enjoyed my class it was very helpful.
FEEDBACK IN DIGITAL LEARNING OBJECTS 151
APPENDIX 6
Summary Review and Full Disclosure
Study Title: Exploring Digital Learning Objects
Thank-you for participating in the study: Exploring Digital learning objects. When you were asked to participate in the study you were made aware of the purpose of the study which was to investigate the design of digital learning objects (DLOs). However, when you were asked to participate, you were not told what design elements of DLOs were being investigated. This was because in order to obtain scientifically valid findings it is important to not fully disclose the details of the study. In particular, if you had knowledge of the variables being investigated it may have influenced your responses and invalidated the data collected. Full Disclosure It is important to fully disclose all aspects of the study and make all participants in the study aware of all of the variables being investigated. This is to ensure that all participants are aware of the purpose of the study and to clear up any misconceptions or misunderstandings that may have arisen during the course of the study. Furthermore, full disclosure allows for an opportunity to fully share purpose of the study and how the results will be used to further the understanding the design of digital learning objects. Study Variables In this study there were two variables being investigated that participants were not made aware of prior to, or during the study. 1) The first variable was whether embedding feedback into the digital learning objects made a
difference in learning. In the study, all of the participants were given identical learning objects except for one difference. That is there were different types of feedback embedded into the learning objects. Each participant was randomly assigned to one three groups with each group getting access to a digital learning object with different types of feedback embedded into the learning object. The types of feedback were
Simple feedback – this feedback was simply a response of “correct” or “incorrect” for each question asked in the digital learning object.
Positive feedback – this feedback was feedback that gave an explanation of why a correct response was correct. In other words, it explained why you got the answer right.
Negative feedback – this feedback was feedback that gave an explanation of why an incorrect response was incorrect. In other words, it explained why you got the question wrong.
2) The second variable looked at whether computer experience impacted how well students
learned from digital learning objects. Whenever using technology for learning, it is important to know whether a participants experience and/or comfort level with using technology has an impact on the results.
FEEDBACK IN DIGITAL LEARNING OBJECTS 152
To ensure that you were not disadvantaged by participating in the study, the digital learning objects will be made available to all students in the classes who participated in the study. The learning objects will be provided to the class instructor and posted on the class Moodle page for all students to use. The digital learning objects that will be posted on the website will include both the positive and negative feedback from the study. Re-Consent Complete this section if, after reading the above full disclosure, you wish to provide re-consent to allow your data to be used in the study. If you do not give re-consent, all data collected about you will be removed from the data set. I _______________________________ (First and last name) have read the above full disclosure agreement and give consent for the data collected about me during the course of the study to be used in the study. Participant’s Name: (please print) _____________________________________________ Participant’s Signature ________________________________Date: _______________ Study Results Would you like a copy of the study sent to you once all results have been compiled and the final report completed. Circle one: Yes No If yes, please provide an e-mail address where an electronic copy of the study may be sent. e-mail: _____________________________