a test of strategies for enhanced learing of descriptive...
TRANSCRIPT
A TEST OF STRATEGIES FOR ENHANCED LEARING
OF DESCRIPTIVE CHEMISTRY
by
Suhasini Kotcherlakota
A DISSERTATION
Presented to the Faculty of
The Graduate College at the University of Nebraska
In Partial Fulfillment of Requirements
For the Degree of Doctor of Philosophy
Major: Educational Studies
Under the Supervision of Professor David W. Brooks
Lincoln, Nebraska
January, 2007
A TEST OF STRATEGIES FOR ENHANCED LEARING
OF DESCRIPTIVE CHEMISTRY
Suhasini Kotcherlakota, M.S.
University of Nebraska, 2007
Adviser: David W. Brooks
The Advanced Placement (AP) Descriptive Chemistry Website allows students to
repeatedly practice chemistry problems. The current study involves the redesign of the
AP Descriptive Chemistry Website using worked examples to enhance learner
performance. The population sample for the study includes students interested in learning
descriptive chemistry materials. Students’ usage patterns with the Website were analyzed
to assess learner performance using the Independent Samples t-test. No important
differences were found in learning between the worked example and more conventional
strategies.
i
Dedication
To my parents and grand parents
ii
Acknowledgements
I wish to express my gratitude and sincere thanks to each one of you for your support,
encouragement and guidance throughout my doctoral studies.
Dr. David Brooks (mentor)
Faculty, Staff & Colleagues, UNL
Family members and friends
iii
TABLE OF CONTENTS
I. Introduction .................................................................................................................. 1
Statement of the Problem........................................................................................ 2 Research study in 1999 ............................................................................... 2 Research study in 2000 ............................................................................... 4 Research study in 2005 ............................................................................... 5
Purpose of the Study ............................................................................................... 6 Research Questions................................................................................................. 6 Significance of the Study ........................................................................................ 7 Limitations and Advantages ................................................................................... 7 Definition of Terms................................................................................................. 9
II. Review of Literature.................................................................................................... 10 Introduction........................................................................................................ ...10 Cognitive Load Theory ......................................................................................... 10 View of Memory................................................................................................... 11 Human Cognitive Architecture ............................................................................. 12 Categories of Cognitive Load ............................................................................... 12
Intrinsic Cognitive Load (ICL) ................................................................. 12 Extrinsic Cognitive Load (ECL)............................................................... 13 Germane Cognitive Load (GCL) .............................................................. 13
Reducing Cognitive Load and Enhancing Learning............................................. 14 Feedback ............................................................................................................... 14 Worked examples.................................................................................................. 16
III. Methodology............................................................................................................... 19 Procedural Steps.................................................................................................... 20 Population and Sample ......................................................................................... 20 Chemistry Study Items.......................................................................................... 25 Eight-item Sample Tests ....................................................................................... 27 Feedback ............................................................................................................... 28 Review .................................................................................................................. 29 Surprise Quizzes ................................................................................................... 30 Design of Website................................................................................................. 32 Data File Structure ................................................................................................ 33 The Use Pattern..................................................................................................... 33 Use Analysis ......................................................................................................... 34
IV.Results.......................................................................................................................... 36 Usage Pattern ........................................................................................................ 37 Date of First Access .............................................................................................. 37 Serious Users ........................................................................................................ 38 Total Time of Practice .......................................................................................... 39 The Number of Interactions .................................................................................. 41
iv
Chemistry Quiz Items ........................................................................................... 43 Tutor Access ......................................................................................................... 45 Worked Examples................................................................................................. 47 Eight Item Practice Quizzes.................................................................................. 48 Review .................................................................................................................. 49 Surprise Tests........................................................................................................ 50
V. Discussion ................................................................................................................... 53 Usage Pattern ........................................................................................................ 53 Chemistry Quiz Items ........................................................................................... 55 Tutor Access ......................................................................................................... 55 Eight Item Practice Quizzes.................................................................................. 56 Worked Examples................................................................................................. 56 Surprise Tests........................................................................................................ 56
VI.Summary and Recommendations ................................................................................ 59 References......................................................................................................................... 61
Appendix A....................................................................................................................... 64
Informed Consent Form........................................................................................ 65
Appendix B ....................................................................................................................... 68 Surprise test 1........................................................................................................ 69 Surprise test 2........................................................................................................ 70 Surprise test 3........................................................................................................ 71 Surprise test 4........................................................................................................ 72 Surprise test 5........................................................................................................ 73
v
LIST OF TABLES
TABLE PAGE
31 Chemistry study items and number of quiz categories 25
32 Access to correct answer 35
41 Users from the experimental and control group 36
42 Users Website transaction 37
43 Group Statistics for user transaction time elapsed rates 41
44 Independent Sample t-test for user transaction time
elapsed rates
41
45 Independent Sample t-test for number of user interactions
with the Website
43
46 Independent Sample t-test for number of user interactions
with the Website
43
47 Group Statistics for correctly answered practice items 44
48 Independent Sample t-test for correctly answered
practice items
44
49 Group Statistics for Incorrectly answered practice items 45
410 Independent Sample t-test for incorrectly answered
practice items
45
411 Group Statistics for tutor access 46
412 Independent Sample t-test for tutor access 46
413 Descriptive Statistics for worked example-1 and worked
example-2
48
vi
414 Group Statistics for eight item practice quizzes 48
415 Independent Sample t-test for eight item practice quizzes 49
416 Group Statistics for surprise quizzes 51
417 Independent Sample t-test for surprise quizzes 52
vii
LIST OF FIGURES
FIGURE PAGE
31 Descriptive Chemistry Website log-in page 21
32 Informed consent 23
33 Descriptive chemistry study site showing three
entry areas
24
34 A chemistry quiz question item 26
35 Worked example 28
36 A Chemistry quiz question item feedback answer 29
37 A chemistry quiz question item feedback answer
with reported errors and accepted answers
information
29
38 Student review of transaction record 30
39 Sample Surprise Quiz 31
310 Student transaction record XML file 32
41 Month-wise user first logins 38
42 Month-wise serious user registrations 39
43 Box plot of user time elapsed rates 40
44 Users vs Number of interactions with the
chemistry Website
42
45 Number of worked examples accessed by users 47
46 Number of times users reviewed their records 50
51 Surprise test items 58
1
I. INTRODUCTION
The College Board, a not-for-profit organization, has offered the Advanced
Placement Program® (AP) to millions of students in United States and other countries
since 1955 allowing them to take college-level courses and exams for earning college
credit or placement while still in high school (College Board, 2006). According to the
College Board (2006), 68 percent of United States public schools now participate in AP.
Since 2000, students from all 50 states and the District of Columbia have succeeded on
the AP exam. In 2005, The College Board reported a national total of 57,102 exam takers
who were qualified for receiving chemistry college credit or advanced placement.
The AP chemistry course is broken down into five major topic areas indicated by
percentage of approximate proportion of multiple choice questions that pertain to each
topic: structure of matter (20%), states of matter (20%), reactions (35-40%), descriptive
chemistry (10-15%), and laboratory (5-10%) (College Board, 2006).
Crippen (2000) states: “Anecdotal evidence suggests that the descriptive
chemistry section of the AP chemistry exam is traditionally difficult for high school
students. The exam's difficulty can be attributed to the nature of the material and the
current structure of the AP curriculum. Descriptive chemistry is difficult to teach because
it requires either a large amount of memorization or experience; it tends to be disjointed
within the traditional curriculum.” The College Board outlines descriptive chemistry as
follows (College Board, 2006):
“Knowledge of specific facts of chemistry is essential for an
understanding of principles and concepts. These descriptive facts, including the
2
chemistry involved in environmental and societal issues, should not be isolated
from the principles being studied but should be taught throughout the course to
illustrate and illuminate the principles. The following areas should be covered:
1. Chemical reactivity and products of chemical reactions
2. Relationships in the periodic table: horizontal, vertical, and diagonal
with examples from alkali metals, alkaline earth metals, halogens, and
the first series of transition elements
3. Introduction to organic chemistry: hydrocarbons and functional groups
(structure, nomenclature, chemical properties). Physical and chemical
properties of simple organic compounds should also be included as
exemplary material for the study of other areas such as bonding,
equilibria involving weak acids, kinetics, colligative properties, and
stoichiometric determinations of empirical and molecular formulas.”
Statement of the Problem
This study was based on the review of three research studies carried out in 1999
and 2000, and 2005 on the Descriptive Chemistry Website.
Research study in 1999
The chemistry Website was developed in 1997 and made accessible since then as
a learning aid to serve users all over the world. The Website was dedicated solely to the
descriptive portion of the advanced placement chemistry exam. About 200 descriptive
chemistry questions in the form of quiz items were devised. These were intended to help
3
high school students prepare for the AP exam, and for chemistry teachers to use as a
learning or teaching resource. The questions were stored in a database. They were served
to the user upon request. Students were allowed to practice chemistry quizzes repeatedly.
Feedback was provided for student’s errors in responding to the quiz questions. The
following is an example of a descriptive chemistry quiz question:
A solution of tin (II) chloride is added to a solution of iron (III) chloride.
A correct answer response to the above question would be:
Sn2+ + Fe3+ ---> Sn4+ + Fe2+ (Not balanced)
The 1999 study is reported by Crippen (2000) who notes:
“An Analysis of the Web server log for 1999 suggests the following conclusions:
• The maximum number of hits in a single day (n = 1,336) occurred on Sunday,
May 16, 1999, the day before the 1999 AP exam.
• While the site shows consistent use, its use tended to be cyclic around the AP
exams. For the 1999 term, site use reached a peak within the week before the
AP exam. Use culminated the day before the exam, and dropped off
significantly after the exam.
• 77.7 percent of the users requested to have hints sent with their quizzes. An
almost equal number of hits were for exams as for grading responses, including
answer keys; 94.2% of the exams were graded.
• The site had consistent use between the hours of 10:00 a.m. and 11:00 p.m.;
Sunday and Wednesday were peak days, though use is consistent throughout
the week. These two statistics suggest strongly that the site was used in
classrooms at schools.
4
• Self-reported AP chemistry students make up the largest proportion of the
users. AP students are using the site between 10:00 a.m. and 11:00 p.m. with
heaviest use closer to 10:00 a.m. High school teachers are using the site at the
same time as the AP students, yet their use is heavier towards the later hours of
the school day.”
Research study in 2000
One of the major conclusions from the 1999 data suggests that AP chemistry
students and/or teachers used the chemistry Website extensively as a learning tool. In the
2000 study, Crippen questions that the 1999 study did not provide documentation about
how the Website was used as a learning tool and what learning occured. Another
conclusion from the 1999 study suggests that the Website provided tutoring components
and feedback to the users. Crippen (2000) states that not only the effectiveness of the
tutorial components but also the implications for the use of tutoring components by users
for chemistry learning were not documented in the 1999 study.
Hence, in the 2000 study, user patterns including user access to tutorials and
feedback were tracked and stored in the database. This was carried out in order to
understand the users’ perspective in learning descriptive chemistry. The Website operated
as follows. Once a user logged in and requested a quiz, a randomly generated set of eight-
items was drawn from 14 arbitrary categories defined by the researcher. Each quiz item
came from a different category, and quiz items were not duplicated. The user might or
might not opt for tutoring when requesting quizzes. The tutoring included detailed text
5
instruction, sometimes with images, and the category from which the quiz item was
posed to the user.
Further, user perceptions were analyzed by conducting a post-AP-examination
survey. Comparisons were made between individual users’ perception about the
effectiveness of the site and site usage. The study demonstrated that effective teaching
and learning could be conducted over the Web through repetitive practice and feedback.
However, in this study survey, respondents indicated problems with formula typing and
bias in the scoring due to formula entry problems. Further, the quiz design lacked
flexibility in providing tutoring and item-specific feedback.
Crippen (2000) recommended enhancements in three areas for the AP
Descriptive Chemistry Website for maximizing student learning. First was the redesign of
the user interface to provide appropriate options for users to access chemistry Website
learning materials. The options include menus that help in constructing formulas or
formula typing for quizzes and eliminating bias in graded quizzes. Second, enhanced
feedback and tutorial components were developed for each quiz item. This included
providing appropriate tutoring material related to each quiz item and feedback given to
students based on their responses to the quizzes. Third, users were not permitted to
resubmit graded quiz items.
Research study in 2005
A more recent study at the modified Website, Crippen and Brooks (2005)
analyzed over 250,000 incorrect responses made by students. Student errors were
recorded and categorized. It was found that the highest percentages of errors were
6
misconceptions (28%) and typographical errors (28%). Next highest percentages of errors
included inappropriate "chemical" dissociations (27%) and/or writing incorrect formulas.
The rest of the errors included not recognizing weak electrolytes and including spectator
ions.
Purpose of the Study
The current study is based on the conclusions and recommendations from the past
studies. The rationale for conducting this study is to minimize student errors and
misconceptions in order to improve learning and overall performance of students
preparing for the AP exam. The redesigned Website focuses on providing tutoring
components for quiz categories and specific performance-related feedback for each quiz
item. The Descriptive Chemistry Website was redesigned by adopting effective
instructional design methods based on the cognitive design principles for fostering
learning. While providing worked examples and feedback was the main target of this
study, tutoring and a redesign also were included.
Research Questions
1. Does a student’s performance in descriptive chemistry improve with descriptive
practicing chemistry problems?
2. Does the use of worked examples decrease the rate of errors when solving
descriptive chemistry problems?
7
Significance of the Study
This study is important for two primary reasons. One is the notion of using a
Web-based tool and the application of technology to enhance teaching and learning. The
other is the development of teaching and learning strategies that help in improving the
performance of students solving chemistry problems.
In this study, three major arguments are hypothesized.
First, it is hypothesized that the repetitive practicing with worked examples will
enhance learning in users relative to practicing quiz items alone.
Second, it is hypothesized that tutoring components in the study will help students
in learning the chemistry material.
Third, it is hypothesized that specific feedback provided in response to the
submitted answers will improve learners’ performance.
The definition of learning used in the study by Crippen (2000) will be used here:
“Learning is defined as an increase in the quiz score, or item score as a function of time
or the number of graded quizzes returned.” Data analysis is performed from the
automatically recorded transaction records of the users.
Limitations and Advantages
The current design of the study has limitations and advantages. The redesigned
Descriptive Chemistry Website attempts to retain the advantages but to minimize the
limitations raised in the past studies.
The Website allows anonymous user log-ins with valid or invalid email address
and password, and the researcher is unable to verify such user information. As mentioned
8
by Crippen (2000), “the advantage of covertly tracking a self-motivated user's actions in
an environment designed to produce learning makes this study appealing.”
“Hardware and software have the potential to limit any study where they are
significant components. Computers crash, hard drives fail, and software programs do not
function as advertised. All of these hardware/software issues have the potential to limit
the study” (Crippen, 2000). The current Website is served to the World Wide Web via a
computer server for increasing the speed of information transfer through the Internet.
This was made possible by utilizing greater bandwidth and transmission speed modem
lines. Unlike in the previous studies, where the Websites were developed on
HyperCard™-based programming techniques, the current study was redesigned
completely by utilizing the scripting programming methods of Runtime Revolution™.
Similar to the 2000 study, this design strategy provided a powerful environment for
automatically tracking user information, generating and grading quizzes, and capturing
users’ actions while interacting with the site. It allowed the researcher to remain removed
from the users and the data until the completion of the study.
Other similar aspects retained from the previous studies were:
• the unique characteristics of the users that include accessing the Internet and
having basic computing skills for learning online material.
• the researcher remains un-associated with the Website.
The current study did not collect user’s perspectives of the Website. That is, no follow-up
surveys were undertaken.
9
Definition of Terms
The following defined terms are required the purpose of this study:
Feedback: “Any message generated in response to a learner’s action. The
outcomes of feedback include helping learners to identify errors and become aware of
misconceptions” (Mason & Bruning, 2000).
Redesigned user interface: Enhancement in the appearance of the Website
features to make a more user-friendly Website as well as to provide more user options.
Runtime Revolution™: A Scripting language and developmental tool used to
develop the current AP Descriptive Chemistry Website.
Server Request: Commonly referred to as hit on the Web server. A server request
involves a user asking the Web server to do something. This includes sending/grading a
quiz or tutoring. Al user-initiated interactions with the Web server are defined as server
requests.
Tutoring: Tutoring is a detailed instruction of the category from which the
chemistry question was posed to the user.
Worked Example: “A worked example is a step-by-step demonstration of how to
solve a problem or perform a task” (Clark & Mayer, 2003).
10
II. REVIEW OF LITERATURE
Introduction
Research studies in 1999, 2000 and 2005 provided extensive literature reviews
supporting the effectiveness of Web instruction. The studies conclude that the Web is a
powerful teaching and learning tool for delivering instruction through repetitive testing
with feedback. This review does not re-emphasize the supporting views of effective Web
instruction and repetitive testing with feedback mentioned in the previous studies, but
instead provides grounded literature that focuses on adopting a design theory approach
that helps in reducing cognitive load and enhancing students’ learning performance.
The following literature review provides an overview of the cognitive load theory,
view of memory, cognitive architecture, and instructional design techniques to reduce
cognitive load, feedback and worked-examples.
Cognitive Load Theory
There has been an increased focus on the role of education and training, and
effectiveness of various instructional design strategies among many researchers. Some of
the most important breakthroughs in this regard have come from cognitive science, which
deals with the mental processes of learning, memory and problem solving (Cooper,
1998). Cognitive load theory (CLT) is a major theory that has undergone substantial
development over the past three decades and has gained importance in both traditional
and technology-based instruction. The human brain is considered to be the center for
human thought process and memory. Cognitive psychologists have aimed at a deeper
11
understanding exploring the most potent concepts of how humans perceive, store, and
retrieve information.
View of Memory
Researchers divided memory processes into stages of acquisition, storage, and
retrieval of information. Sweller’s (1999) ‘modal model’ of memory consists of three
types of memory: sensory memory, working memory, and long-term memory. Sensory
memory refers to the perception of things by the incoming stimuli from our senses.
Information is initially processed in sensory memory and then passed to working
memory. Working memory refers to the conscious cognitive processing of information
that occurs and has a very limited capacity in terms of the amount or complexity of
information that can be retained or processed at once. According to Miller (1956),
humans can process only seven plus or minus two elements at a time. If the limit
exceeded, learners experience what is called "cognitive overload" and no learning takes
place. However, in the recent literature, Sweller points out that working memory can
handle only a very limited number (possibly no more than two or three) novel interacting
elements (Paas et al., 2003). They further note that working memory can process simple
information very easily when carrying out cognitive activities. Long term memory stores
general world knowledge. The information that is present can be very sophisticated and
enable learners to perceive, think, and problem-solve. It is more than a mere
memorization of learned facts. The mental structures formed in memory are called
schema or schemata. They compose the knowledge base which plays a crucial role in the
human thinking process.
12
Human Cognitive Architecture
According to Sweller (1999), cognitive load theory assumes that some learning
environments impose greater demand than others and, as a consequence, impose a higher
information processing load on limited cognitive resources in working memory.
Cognitive load depends upon the degree to which one efficiently rehearses tasks and
skills over various domains and gains knowledge over a period of time. Human cognitive
architecture interacts with instructional material in various ways. Information present in
human memory varies on a continuum of low-high interactivity. "Each element of low
interactivity material can be understood and learned individually without consideration of
any other elements. The elements of high interactivity material can be learned
individually, but they cannot be understood until all of the elements and their interactions
are processed simultaneously. As a consequence, high interactivity materials are difficult
to understand" (Paas et al., 2003).
Categories of Cognitive Load
Based on different sources of cognitive load, Sweller (1999) distinguished three
types of load: Intrinsic, extraneous, and germane load.
Intrinsic cognitive load (ICL)
Element interactivity is the driver of this category because demands on working
memory capacity imposed by element interactivity are intrinsic to the material being
learned. Intrinsic cognitive load cannot be altered by instructional manipulations. A
simpler learning task that omits some interacting elements can be chosen to reduce ICL.
13
Extrinsic Cognitive Load (ECL)
The manner in which information is presented to the learners and the learning
activities required of learners can also impose cognitive load. When that load is
unnecessary (ECL), it may interfere with schema acquisition and automation.
Instructional procedures developed without any consideration or knowledge of the
structure of information or cognitive architecture impose heavy ECL because working
memory resources must be used for activities that are irrelevant to schema acquisition
and automation. Instructional designs intended to reduce cognitive load are most effective
when the ICL is high. When ICL is low, instructional designs intended to reduce
cognitive load have little or no effect.
Germane or Effective Cognitive Load (GCL)
The manner in which information is presented to the learners and the learning
activities required of learners can also impose cognitive load. While ECL interferes with
learning, GCL may enhance learning. Instead of working memory resources being used
to engage in searching, for example when dealing with ECL, GCL results in those being
devoted to schema acquisition and automation. Increases in effort or motivation can
increase the cognitive resources devoted to a task. In a sense, GCL amounts to what
generally is called self-regulation in the educational psychology literature.
In summary, Swellers’ view of learning is as follows:
Learning can occur when [(Intrinsic Cognitive load + Extraneous Cognitive load +
Germane Cognitive load) = Total Cognitive Load] < Working Memory capacity.
14
With respect to multimedia learning, Mayer and Moreno (2003) explain that,
when processing demands evoked by the learning task exceed the processing capacity of
the cognitive system or working memory capacity per-se, the result is cognitive overload.
Reducing Cognitive Load and Enhancing Learning
Cooper (1998) illustrates how extraneous load occurs in instruction: "When
intrinsic cognitive load is low (simple content) sufficient mental resources may remain to
enable a learner to learn from ‘any’ type of instructional material, even that which
imposes a high level of extraneous cognitive load. If the intrinsic cognitive load is high
(difficult content) and the extraneous cognitive load is also high, then total cognitive load
will exceed mental resources and learning may fail to occur. Modifying the instructional
materials to engineer a lower level of extraneous cognitive load will facilitate learning if
the resulting total cognitive load falls to a level that is within the bounds of mental
resources."
Feedback
Mason and Bruning (2000) explain that, “Feedback is any message generated in
response to a learner’s action. The outcomes of feedback include helping learners to
identify errors and become aware of misconceptions.” Research shows that, in everyday
classrooms and online learning environments, feedback is provided in one or several
methods of instruction to improve student learning and achievement. Brooks et al.,
(2005) cited methods that provide feedback including repetitive testing with immediate
feedback, encouraging in-class pair discussions where students evaluate and provide
15
feedback to each other, promoting cooperative learning in team-led groups, just-in-time
dynamic expert feedback provided in computer assisted instructional settings,
scaffolding, and self-explanation.
Further, Brooks et al. (2005) strongly suggest that “performance-related feedback
is the hallmark of efficient instruction” and teachers can utilize one or more forms of
feedback in their instruction to improve learning. Immediate and quick feedback without
delays is effective in increasing student comprehension on the learning materials.
(Bangert-Drowns, Kulik, Kulik, & Morgan, 1991).
Student learning increases when feedback is corrective in nature (Walberg, 1999).
Asking students to work repeatedly on a task until they are successful enhances student
achievement (Marzano, Pickering, & Pollock, 2001). Crippen et al. (2000) assert that
“Feedback from practice need not be confined to the correctness of an answer. Since
feedback points out missing or misunderstood knowledge, it provides an ideal teaching
moment. Thus, any feedback that further explains what students do not know adds to the
value of feedback.”
Instructional strategies and course design play an important role in achieving
higher learner outcomes for content, especially that taught over the Web (Clark & Mayer,
2003; Dick et al., 2001; Gagne et al., 1992; Sweller, 1999). A good instructional design
accommodates complex information, reduces cognitive load on working memory, and
enables learners to form effective schema and automate their learning processes. Recent
developments in instructional design have investigated ways to reduce cognitive load
(Paas et al., 2003; Mayer & Moreno, 2003). Cooper (1998) stresses that “The quality of
16
instructional design will be raised if greater consideration is given to the role and
limitations of working memory.”
Worked Examples
Worked examples are among the earliest and probably the best-known cognitive
load reducing techniques (Paas et al. 2003). Traditional ways of learning by solving lots
of problems load working memory. Worked examples help learners to reduce this load by
freeing working memory resources and building new knowledge (Clark & Mayer, 2003).
The effectiveness of worked examples depends on the previous domain
knowledge of students. When students cannot form schemas within a disciplinary area of
study, it is difficult for teachers to find suitable aspects of the area for them to explore
(Sweller & Tuovinen, 1999). Further research on worked examples is based on the
level(s) of cognitive skill acquisition. Cognitive skills are acquired by learners in three
phases. In the early phase, learners attempt to gain a basic understanding of the domain
by studying principles in instructional materials. Worked examples are most effective in
this early phase. During the intermediate phase, learners begin solving problems with the
knowledge acquired previously from the early phase. As the learners practice and solve
problems, they self-explain concepts (VanLehn, 1996). In the late phase, learners practice
striving for speed and accuracy. In this phase, skills are becoming automated. It is
uncertain from an instructional point of view of how best learners should structure the
transition from an example-based initial phase learning to problem solving in the
intermediate phase (Renkl & Atkinson, 2003). For learners who are already familiar with
the skills of problem-solving, interpreting a worked example may be redundant and
17
impose a greater cognitive load than simply providing a solution to the problem (Kalyuga
et al., 2003). One way to circumvent this problem is to design worked examples as
“completion” problems. van Merrienboer et al. (2003) suggest that the intrinsic cognitive
load can be decreased by practicing the simplest version of the whole task encountered by
experts in the real world and progress towards increasingly more complex versions.
The second approach is used to decrease extraneous load by scaffolding worked
examples followed by completion problems and then full problems. Sweller (1999)
suggests that worked examples are effective only under conditions where students do not
have to mentally integrate disparate sources of mutually referring information as well as
to eliminate redundant information during problem-solving. As learners acquire cognitive
skills and gain experience, using worked examples will cause redundancy. Devoting
working memory to redundant information leads to allocating limited cognitive capacity
to the redundancy and results in little or no learning. Redundant information may even
interfere with the schemas constructed by experienced learners and may also have
negative consequences. This phenomenon is known as the expertise reversal effect
(Kalyuga et al., 2003).
Renkl and Atkinson (2003) suggest that, in the earlier stages of learning when
intrinsic cognitive load is high because of problem-solving demands and fewer schemas,
learners should study instructions. During intermediate stages, when schema information
has freed some working memory capacity, they should study worked examples and
increase germane load by using self-explanations. In the final phase, there should be
sufficient working memory capacity to permit more problem-solving. This is called the
“fading procedure” of using worked examples for cognitive skill acquisition in solving
18
problems and learning. The fading technique facilitates transition from initial through
intermediate to final phases where complete worked examples are faded by successively
eliminating sections of the worked example until eventually only a full problem remains.
Research studies on the use of full or partial worked examples along with the
emphasis on other cognitive load reducing techniques and strategies are on going and
experimental. Based on what is known by research so far and to test the criterion lays the
foundation for this study.
In summary, this study conducts a test of instructional design strategies and
evaluates the results for enhancing learning.
19
III. METHODOLOGY
The AP Descriptive Chemistry Website was initially designed in 1997 and was
redesigned in 2000. This Website was visited by AP chemistry users extensively over the
past years for AP chemistry exam preparation (Crippen et al., 2000).
Crippen (2000) in his literature states that “The data set for 1997-99 is limited due
to the design of the site. Those data are limited because it does not allow for tracking any
one individual's use of the site. In addition to quizzing, the site offered a tutoring
component. The tutoring component is used but is not correlated to users and their
scores.” In order to overcome the problems in 1999 study, the 2000 study redesigned the
HyperCard Website and for tracking of an individual's use and surveying user
perceptions.
The current Website was once again redesigned for understanding the
effectiveness of the site as a learning tool. The Website was reconstructed by adopting
instructional design methods based on the cognitive design principles for fostering
learning. Worked examples with feedback and feedback only are the two main
approaches administered in the current chemistry Website. The rationale for identifying
these approaches in this study was to minimize student errors and misconceptions in
order to improve learning and overall performance.
20
Procedural Steps
1. Obtain Institutional Review Board (IRB) approval.
2. Redesign the HyperCard database using Runtime Revolution.
3. Announce the site to the AP chemistry community.
4. Gather user access data.
5. Perform data analysis.
Population and Sample
The research sample for this study represents individuals interested in descriptive
chemistry, especially as it applies to the AP chemistry examination. A total number of
1373 subjects were recruited for this study. Subjects presumably were AP chemistry
students and teachers.
The sample was recruited by word of mouth, conference presentations, email, and
newsgroup postings (misc. education. science, K-12.education.science). The URL for the
chemistry Website http://dwb2.unl.edu/apchem/main.html was made available for access
for the users worldwide. The homepage of the Descriptive Chemistry Website indicates
to the users that the site is a teaching site for chemistry and that research is conducted on
learning.
As shown in Figure 3.1, users logged-in to the chemistry Website using their
email address and password. If the user was logging in for the first time, they were
requested to provide an email and password that they wished to use for future access to
the Website. Consent was obtained from the site registrant after their first log-in attempt
to the Website.
21
Figure 3.1. Descriptive Chemistry Website log-in page.
Once informed consent was obtained (see Figure 3.2), each new user had access
to the chemistry quizzes and all the content in the Website. Each time a user logged in, all
of his/her transactions were recorded together with the e-mail login. The access time,
access address (computer IP number), specific identity of the chemistry items accessed in
the Website, and responses made all were recorded automatically.
22
23
Figure 3.2. Informed consent.
In this study, users were randomly assigned to one of two groups. One group had
access to worked examples and the other group did not. The group that received the
worked examples was the experimental group and the group that did not was the control
group. Presenting worked examples was the independent variable while the learning
performance of the students was the dependent variable in this study. The current
chemistry Website allowed students to practice chemistry problems repetitively. Based
upon the students’ performance (correct or incorrect responses), immediate feedback was
given. Immediate feedback was provided to students about their errors in answering the
24
quiz questions. Students in the control group were provided with correct answers and
asked to enter correct responses until they demonstrated proficiency. Students in the
experimental group were provided with completed worked examples in addition to the
explanation components for the first few problem sets in each chemistry category. After
three pairs of such worked examples (one pair for each of the three initial quiz questions),
the students were treated identically in both the experimental and control groups.
Tutoring (access to related, appropriate text material) was accessible to both groups.
As shown in Figure 3.3, the main Webpage contained three entry areas.
Figure 3.3. Descriptive chemistry study site showing three entry areas.
25
The first area is the Study section where users can access Chemistry Study Items.
The second area provides access to Eight-item sample tests (a model AP question), and
the third area provides access to Review their transactions with the Website.
Chemistry Study Items
The Chemistry study items were drawn from 14 arbitrary categories defined by
David Brooks and based upon his review of the AP questions over nearly four decades.
They are displayed in Table 3.1 along with the number of items available.
Chemistry Study Category Number of quiz items
Acid-Base General 30
Acid-Base anhydrides 20
Acid-Base Lewis 5
Acid –Base Hydrolysis 16
Redox Aqueous 35
Redox Metal Aqueous 5
Metals/Other 5
Redox/Other 10
Organic 12
Combustions 10
Precipitations 26
Complex Ion 21
Thermal decomposition 8
Mixed 5
Table 3.1. Chemistry study items and number of quiz categories.
26
A user may select any one of the categories or select the random option from the
study item area drop-down list. Once the user makes a selection of a category, he/she is
presented with a chemistry quiz item along with a tutoring component (see Figure 3.4).
Figure 3.4. A chemistry quiz question item
Each chemistry category contained a varied number of quiz items (see Table 3.1).
A chemistry quiz item consisted of the question number, question, three text field boxes
each for reactants and products, and a submission link for submitting answers. Students
entered their answers in the text field boxes and submitted them for evaluation.
Instructions for entering the answer formulas and a graphic of the periodic table were
provided to students in the same Webpage for reference. Students could choose to study
the material in the tutoring before answering the quiz question. Specific tutoring was
27
provided to users based on the quiz item category. Users could access tutoring any
number of times and return back for answering the same quiz question.
When users submitted the answers correctly, both the experimental and control
group received new quiz items based on their selection from the Chemistry Study Items
area.
However, when users from the worked examples group submitted an answer
incorrectly, they were provided with a completed worked example for the first two
attempts and a correct answer thereafter (see Figure 3.5). Control group users were
provided with a correct answer if they failed to answer a quiz item (See Table 3.1).
Eight-Item Sample Tests
Eight-item Sample Tests is the second entry area in the main chemistry Webpage.
This item format has been used up until 2006 for the AP descriptive chemistry question.
(The AP is changing this format beginning in 2007.) When the user selects this option,
s/he is provided with a set of eight quiz items. These items are randomly generated from
the database when the user requests them. Hence, each user may get a different set of
items each time they access. Users answer all eight-items and submit their answers for
evaluation. Unlike the Chemistry study items, users do not have access to the tutoring
component.
28
Figure 3.5. Worked example
Feedback
Feedback was provided for submitted answers for both the study item quiz
questions and the eight-item sample tests. Feedback included indicating the number of
wrong responses submitted and the number of expected correct responses (see Figures
3.6 & 3.7).
29
Figure 3.6. A Chemistry quiz question item feedback answer
Figure 3.7. A chemistry quiz question item feedback answer with reported errors and
accepted answers information.
Review
Review is another entry area in the main chemistry Webpage. Users can view all
transactions they made in the Website by selecting this option (see Figure 3.8). Review
contains the most recent transaction down to the first transaction with the Website.
30
Figure 3.8 . Student review of transaction record
Surprise Quizzes
Besides the eigh-item quizzes, both the experimental and control group users
received the same 3-item “surprise” quizzes.
Five such quizzes were developed, and these are presented in sequence after the
20th, 40th, 60th, 80th, and 100th interactions. Unlike other items such as the chemistry study
quizzes or eight-item tests whose selection is based upon either user choice or random
choice, all users saw the same “surprise quizzes” in the same order (see Figure 3.9).
31
Figure 3.9. Sample Surprise Quiz
32
Design of Website
The AP Descriptive Chemistry Website was re-designed using Runtime
Revolution™, JavaScript, cgi and XML technologies. Runtime Revolution™ and
Javascript were the main two scripting languages used for rewriting the code to redevelop
the old HyperCard-based Website. cgi is the engine used for interfacing connections with
the Website program and the Web server. XML was the file format used for storing the
user transactions with the Website. Each user has an XML-like record, which contains
transaction information (see Figure 3.10).
Figure 3.10 . Student transaction record XML file
33
MySQL was used during data analysis after the collection of the XML files. That
is, data from the XML files were aggregated into a MySQL database.
Data File Structure
The transaction data were stored in a folder named apchem in the Webserver
folder under the Documents folder. The apchem folder contained two folders named
Adata and Bdata. Adata folder contained transaction record files of all the users assigned
to the worked-example, and the Bdata folder contained transaction record files of all the
users assigned to the no-worked-example group. Each transaction record is an XML-like
file and contained information about the time of access, date of access, IP address, and
information about the user transaction within the three entry areas along with the quiz
items accessed and answers submitted (see Figure 3.10).
The Use Pattern
“A user’s use pattern is defined by a compilation of statistics from the database”
(Crippen, 2000). Similar to the 2000 study, descriptive statistics for each user were
compiled based upon the use patterns. The average usage patterns were then used to
understand the site’s usage and the effectiveness of the site as a learning tool.
Statistics for creating a use pattern for each user of the Descriptive Chemistry Website.
1. Usage pattern
a. Use time, dates and number of log-in’s
2. Usage pattern of chemistry quiz items
a. Average number of chemistry quiz items taken
34
b. Average number of chemistry quiz items graded
c. Average quiz item scores
d. Average tutoring request by subtopic
e. Average number of worked examples accessed
3. Usage pattern of eight-item sample tests
a. Average number of correct answers
b. Average number of eight-item sample tests taken
c. Average scores of eight-item sample tests.
4. Usage pattern of surprise quizzes
a. Average number of correct answers
b. Average number of surprise quiz items taken
c. Average scores of surprise quiz items
Use Analysis
The variables under analysis from the transaction records will be usage patterns of
practice items, usage patterns of the eight-item sample tests, and usage patterns of
surprise quizzes. Each usage pattern involves descriptive statistics of the subject such as
the times and dates the Website was accessed. In particular, statistics for practice items
including average number of practice items taken, average number of practice items
graded, average practice scores and average request of practice items by topic were
collected. For the experimental group, the average number of worked examples accessed
for each practice item was measured. Similarly, statistics for an eight-item sample test
included the number of correct answers, the average number of eight-item sample tests
35
taken, and the average scores of the eight-item sample tests. Further, statistics for surprise
quizzes were the average number of surprise quizzes taken and average scores on the
surprise quizzes.
The results obtained from the experimental variables (transaction record data)
were gathered in a database and analyzed using SPSS software. The final results are
discussed using ANOVA and regression analysis methods on the data.
Failed to answer quiz question correctly in the first attempt.
Failed to answer quiz question correctly in the second attempt and/or need more help or tutoring.
Failed to answer the quiz question correctly in the third attempt.
Worked Example Group
Provide first worked example
Provide second worked example
Provide correct answer
No Worked example group
Provide correct answer
Provide correct answer Provide correct answer
Table 3.2. Access to correct answer
36
IV. RESULTS
The AP Descriptive Chemistry Website was designed based on the
recommendations from the previous studies conducted in 1999, 2000 and 2005. The
current redesign is consistent with the literature on worked examples, feedback and
repetitive practice. The Website recorded everything users did while accessing the AP
descriptive chemistry Website during the 2006 academic term. Users were randomly
assigned to one of two groups. The worked-example group (WE) is the experimental
group and no-Worked-example group (NWE) is the control group. Analyses were
performed based on the user transactions and comparisons were made between the
experimental and the control groups.
A total of 1,373 unique users have registered and used the Website for practicing
chemistry problems. Each unique user has been identified with a user e-mail address that
they have chosen to provide during the initial registration with the Website. All the
Website interactions of the users were tracked and saved in the database for data analysis.
The selection of users for this study has been based on the completion of five “surprise
tests.”
Users who did not complete all 'five'
surprise tests
Users who completed all five surprise tests
Total number of users
Worked- example 637 50 687 No-Worked-example 620 66 686 Table 4.1. Users from the experimental and control group.
37
Table 4.1 illustrates that a total 50 out of 687 users from worked-example group
and a total of 66 out of 686 from the no-worked-example group completed all five
surprise tests. Approximately 93% of the students in the experimental group and 90% in
the control group did not complete all five surprise tests.
Usage Pattern
The usage pattern includes the dates users accessed the Website, the total amount
of time spent using the Website for practicing chemistry problems, and the number of
user logins to the descriptive chemistry Website.
Date of First Access
User transactions were collected from January until August, 2006. Students used
the Website increasingly up through the month of May. Very minimal usage occurred
thereafter during the months of June, July and August (Table 4.2 and Figure 4.1).
20-Jan-06 3-Aug-06 Jan Feb Mar Apr May Jun Jul Aug
Worked- example 94 124 119 166 148 17 15 4 687
No-Worked-example 95 124 117 166 147 17 16 4 686
Total 189 248 236 332 295 34 31 8 1373 Table 4.2. Users Website transaction.
38
0
20
40
60
80
100
120
140
160
180
jan Feb mar apr may jun jul aug
Month
Num
ber o
f use
rs re
gist
ered
with
AP
desc
riptiv
e ch
emis
try w
ebsi
te
Worked-ExampleNo-Worked-Example
Figure 4.1. Month-wise user first logins.
Serious users
The Website for this research study was designed to provide chemistry materials
to users interested in learning descriptive chemistry. The selection of (experimental [WE]
and control group [NWE]) users in the study was based on users who consistently used
the Website and reached the point of attempting all the five sets of surprise tests. This
group of 116 users is called the serious users. Although data analysis shows that a
number of users interested in learning chemistry materials had registered to the AP
descriptive chemistry Website, the final results indicate that only a small fraction of the
total number of users met the selection criterion of "serious user.” That is, of the 1373
users for whom some data were available, only 116 were serious enough to persist in
using the Website. Usage patterns and user associations such as the dates accessed the
39
total amount of time spent, and number of each user logins were considered to understand
the users’ improvement in practicing chemistry problems and thus the performance.
0
5
10
15
20
25
30
Jan Feb Mar Apr May May+
Month
Num
ber o
f ser
ious
use
rs re
gist
ered
with
AP
desc
riptiv
e ch
emis
try w
ebsi
te
Worked-example No-workedexample
Figure 4.2. Month-wise serious user registrations.
Figure 4.2 shows the number of serious users registered with the AP descriptive
chemistry website from the beginning of the study in the month of January until the day
of the actual AP exam in the month of May.
Total Time of Practice
The box plot (Figure 4.3) shows that the elapsed time variability between the
worked example and the no-worked-example group. The worked example group has
slightly greater variability than the no-worked example group. The figure also indicates
that there are some outliers and extreme variables. The median time elapsed rates for
worked example and no-worked example group are 5.72 and 4.52 hours respectively. The
40
maximum (177.37 and 193.85) and minimum elapsed rates (0.71 and 0.94) respectively
indicate the variability in the time spent for practicing chemistry problems in the Website.
Both the groups are skewed to the right (i.e., toward shorter time intervals)
2.001.00
(1.00) Worked Example and (2.00) No-Worked Example Group
200.00
150.00
100.00
50.00
0.00
Elap
sed
Tim
e ra
tes
Figure 4.3. Box plot of user time elapsed rates.
Overall, no statistical significant difference was found between the groups for
elapsed time (mean values MWE = 12.12; MNWE =13.05).
41
Time Elapsed - Group Statistics
50 12.1202 27.78812 3.9298366 13.0539 24.91376 3.06667
Group1.002.00
ValueN Mean
Std.Deviation
Std. ErrorMean
Table 4.3. Group Statistics for user transaction time elapsed rates
Time Elapsed - Independent Samples Test
.342 .560 -.190 114 .850 -.93374 4.90990 -10.66022 8.79274
-.187 99.135 .852 -.93374 4.98478 -10.82447 8.95699
Equal variancesassumedEqual variancesnot assumed
ValueF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.4. Independent Sample t-test for user transaction time elapsed rates.
The group statistics indicate the sample sizes (N), means, and standard deviations
of both the control and experimental groups. Independent Samples t-test Table 4.4 shows
the Levene’s Test for the Equality of Variances in the two groups for elapsed time rates.
The significance column shows that assumption is not violated (p = 0.56) is not
significant. Because homogeneity can be assumed, the two-tailed significance, p = 0.85
indicates that the observed difference in the means between the WE and NWE is not
significant. The output also indicates that the observed difference in the means is not
significant, t(114) = -0.19, p = 0.85.
The Number of Interactions
The total number of transactions each user made with the descriptive chemistry
Website was measured. Users practiced on chemistry practice quizzes, eight-item
quizzes, surprise tests, and viewed their usage transaction record. Each such interaction
42
was counted towards the total number of transactions (Figure 4.4). The group statistics
indicate that the mean value of interactions for the WE group is 259.48 and NWE is
294.05.
Independent Samples t-test Table 4.5 shows the Levene’s Test for the Equality of
Variances in the two groups for number of interactions with the website. The significance
column shows that assumption is not violated because the significance (p = 0.07) is not
significant.
64615855524946434037343128252219161310741
User
1400.00
1300.00
1200.00
1100.00
1000.00
900.00
800.00
700.00
600.00
500.00
400.00
300.00
200.00
100.00
0.00
Num
ber o
f int
erac
tions
with
the
chem
istr
y w
ebsi
te
No Worked Example GroupWorked Example group
Figure 4.4. Users vs. Number of interactions with the chemistry Website.
43
User Interactions - Group Statistics
50 259.4800 156.88736 22.1872266 294.0455 218.22913 26.86215
Group1.002.00
InteractionsN Mean
Std.Deviation
Std. ErrorMean
Table 4.5. Independent Sample t-test for number of user interactions with the Website
Because homogeneity can be assumed, the two-tailed significance, p =0.34
indicates that the observed difference in the means between the WE and NWE is not
significant. The output also indicates that the observed difference in the means is not
significant, t(114) = -0.95, p = 0.35.
Independent Samples Test
3.386 .068 -.949 114 .345 -34.56545 36.41966 -106.713 37.58161
-.992 113.726 .323 -34.56545 34.84032 -103.586 34.45473
Equal variancesassumedEqual variancesnot assumed
InteractionsF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.6. Independent Sample t-test for number of user interactions with the Website
Chemistry Quiz Items
Users practiced chemistry quiz items of their choice from a set of the descriptive
chemistry materials and then submitted the quiz item for grading. Each quiz item carried
a maximum of three possible reactant answers and three possible product answers. A
final “score” for each practice quiz item was calculated based on correctly answering the
chemistry quiz item, i.e. submitting all the correct reactant and product answers. Credit
was not given to the user if one or more incorrect answers were submitted. The answer
was then considered incorrect.
44
The chemistry practice quiz items do not include the eight-item quiz questions
practiced or the surprise quiz items. The group statistics of the chemistry practice quiz
items indicate that the mean value of correctly answered practice quiz items for the WE
group is 26.72 and NWE is 35.17 (Table 4.7).
Practice Items Correct Answered - Group Statistics
50 26.7200 23.97111 3.3900366 35.1667 41.15135 5.06538
Group1.002.00
PracticeItemsN Mean
Std.Deviation
Std. ErrorMean
Table 4.7. Group Statistics for correctly answered practice items
The Independent Samples t-test Table 4.8 shows the Levene’s Test for the
Equality of Variances in the two groups for correctly answered practice items. The
significance column shows that the assumption is violated (p= 0.00 being less than 0.05)
was significant. Because homogeneity cannot be assumed, the two-tailed significance,
p =0.17 indicates that the observed difference in the means between the WE and NWE is
not significant. The output also indicates that the observed difference in the means is not
significant, t(107.63) = -1.39, p = 0.17.
Practice Items Correct Answered - Independent Samples Test
11.241 .001 -1.294 114 .198 -8.44667 6.52859 -21.37976 4.48643
-1.386 107.626 .169 -8.44667 6.09511 -20.52871 3.63537
Equal variancesassumedEqual variancesnot assumed
PracticeItemsF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.8. Independent Sample t-test for correctly answered practice items.
The group statistics indicate that the mean value of incorrectly answered practice
items for the WE group is 40.44 and NWE is 56.83 (Table 4.9).
45
Group Statistics
50 40.4400 28.88836 4.0854366 56.8333 48.80466 6.00744
Group1.002.00
PiIncorrectN Mean
Std.Deviation
Std. ErrorMean
Table 4.9. Group Statistics for Incorrectly answered practice items
Independent Samples t-test Table 4.10 shows the Levene’s Test for the Equality
of Variances in the two groups for incorrectly answered practice items. The significance
column shows that assumption was violated, (p = 0.03) was significant. Because the
assumption of homogeneity cannot be assumed, the two-tailed significance, p = 0.03
indicates that the observed difference in the means between the WE and NWE was
significant. The output also indicates that the observed difference in the means was
significant, t(108.30) = -2.26, p = 0.03.
Independent Samples Test
4.788 .031 -2.110 114 .037 -16.39333 7.76841 -31.78250 -1.00417
-2.256 108.298 .026 -16.39333 7.26499 -30.79335 -1.99331
Equal variancesassumedEqual variancesnot assumed
PiIncorrectF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.10. Independent Sample t-test for incorrectly answered practice items.
Tutor Access
Tutoring was provided for each descriptive chemistry quiz item category.
Students in both the WE and NWE groups can access tutors when practicing the
chemistry quiz items.
The group statistics indicate that the mean value of tutors accessed for the WE
group is 4.26 and NWE is 3.73 (Table 4.11).
46
Tutor Access Group Statistics
38 4.2632 3.65901 .59357
52 3.7308 3.83030 .53117
groupWorked Example TutorNo Worked ExampleTutor
valueN Mean
Std.Deviation
Std. ErrorMean
Table 4.11. Group Statistics for tutor access
The Independent Samples t-test Table 4.12 shows the Levene’s Test for the
Equality of Variances in the two groups for tutor access. The significance column shows
that assumption was not violated, (p = 0.92) was not significant. Because homogeneity
can be assumed, the two-tailed significance, p =0.51 indicates that the observed
difference in the means between the WE and NWE was not significant. The output also
indicates that the observed difference in the means was not significant, t(88) = 0.66,
p = 0.51.
Tutor Access Independent Samples Test
.009 .924 .664 88 .509 .53239 .80228 -1.06198 2.12675
.668 81.888 .506 .53239 .79653 -1.05220 2.11698
Equal variancesassumedEqual variancesnot assumed
valueF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.12. Independent Sample t-test for tutor access
47
Worked Examples
Two worked examples (worked example-1, WE1 and worked example-2, WE2)
were provided for first few sets of chemistry quiz items. The worked example group
alone had access to these examples when practicing chemistry quiz items. The mean
value for worked-example 1 is 16.28 and for worked-example 2 is 6.74 (Figure 4.5 and
Table 4.13).
494745434139373533312927252321191715131197531
Users
60.00
50.00
40.00
30.00
20.00
10.00
0.00
Num
ber o
f wor
ked
exam
ples
acc
esse
d
Example2Example1
Figure 4.5. Number of worked examples accessed by users.
48
Descriptive Statistics
50 2.00 56.00 16.2800 11.4785350 .00 28.00 6.7400 5.9618250
Example1Example2Valid N (listwise)
N Minimum Maximum MeanStd.
Deviation
Table 4.13. Descriptive Statistics for worked example-1 and worked example-2
Eight-item Practice Quizzes
The eight practice questions are a set of eight practice quiz items. The final score
for the practice quiz item is calculated based on the total number of reactant and product
answers submitted correctly out of the maximum possible correct answers for each quiz
item. Total score of the each set of practice item answered is considered for the analysis.
The group statistics indicate that the mean value of eight-item practice quiz items for the
WE group is 3.37 and NWE is 3.78 (Table 4.14).
8 Item Practice Quiz - Group Statistics
25 3.3692 1.54697 .3093932 3.7756 1.40723 .24877
value1.002.00
groupsN Mean
Std.Deviation
Std. ErrorMean
Table 4.14. Group Statistics for eight-item practice quizzes
The Independent Samples t-test Table 4.15 shows the Levene’s Test for the
Equality of Variances in the two groups for eight-item practice quizzes. The significance
column shows that assumption was not violated, (p= 0.70) was not significant. Because
homogeneity can be assumed, the two-tailed significance, p = 0.31 indicates that the
observed difference in the means between the WE and NWE was not significant. The
49
output also indicates that the observed difference in the means was not significant
t(55) = -1.04, p = 0.30.
8 Item Practice Quiz - Independent Samples Test
.152 .698 -1.036 55 .305 -.40643 .39234 -1.19269 .37984
-1.024 49.156 .311 -.40643 .39700 -1.20416 .39131
Equal variancesassumedEqual variancesnot assumed
groupsF Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.15. Independent Sample t-test for eight-item practice quizzes
Review
A total of 24 out of 50 users from the worked-example group and 33 out of 66
users from the no-worked-example group reviewed their transactions with the descriptive
chemistry website. Figure 4.6 shows the total number of times each user reviewed their
record.
The mean values for number of times record reviewed for worked-example group
and no-worked-example group are 1.74 and 2.36 respectively. These mean values are
extremely low compared to the mean values of total interactions (worked-example group,
259 and no-worked example group, 294) with the website indicating that the review area
in the website was not utilized often by the users.
50
0
5
10
15
20
25
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
Users
Nu
mb
er
of
tim
es
reco
rd r
evie
wed
Worked-Example No-Worked-Example
Figure 4.6. Number of times users reviewed their records.
Surprise Tests
A total of 15 surprise test items were analyzed. The users had no control over the
appearance of surprise tests, and the surprise test items appeared only once (in the context
of a surprise test). Thus the surprise test items were the measurable features that all users
of the Website saw in common and at the same relative time during the site usage. The
group statistics of the surprise tests were calculated as shown in the Table 4.16.
The Independent Samples t-test Table 4.17 shows the Levene’s Test for the
Equality of Variances in the two groups for surprise quiz items. The significance column
is considered to determine whether the assumption is violated or not violated. Based on
the homogeneity of the assumption and the two-tailed significance value, the significance
51
of the difference between the groups is determined. All the 15 Independent sample t-tests
indicated that there was no significant difference between the WE and NWE groups.
Table 4.16. Group Statistics for surprise tests
Surprise Tests Group Statistics
50 .4550 .34140 .04828 66 .4811 .32907 .04051 50 .4450 .42043 .05946 66 .4545 .36927 .04545 50 .3730 .45504 .06435 66 .3936 .44893 .05526 50 .2500 .23690 .03350 66 .2500 .23205 .02856 50 .4650 .41958 .05934 66 .4773 .41074 .05056 50 .2900 .33640 .04757 66 .3295 .34851 .04290 50 .4600 .28067 .03969 66 .4606 .34368 .04230 50 .4700 .35942 .05083 66 .4356 .40266 .04956 50 .6050 .41061 .05807 66 .4962 .44179 .05438 50 .3500 .29881 .04226 66 .2992 .30134 .03709 50 .6500 .37457 .05297 66 .5606 .41385 .05094 50 .2730 .35470 .05016 66 .2521 .33139 .04079 50 .0200 .06061 .00857 66 .0273 .06916 .00851 50 .2920 .21077 .02981 66 .2424 .23407 .02881 50 .4398 .45912 .06493 66 .4041 .47697 .05871
group Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example Worked Example No Worked Example
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T15
N MeanStd.
DeviationStd. Error
Mean
52
Surprise tests Independent Samples Test
.134 .715 -.416 114 .678 -.02606 .06270 -.15027 .09815
-.414 103.572 .680 -.02606 .06302 -.15104 .09892
3.838 .053 -.130 114 .897 -.00955 .07351 -.15517 .13608
-.128 97.823 .899 -.00955 .07484 -.15807 .13898
.004 .948 -.244 114 .808 -.02064 .08466 -.18835 .14708
-.243 104.906 .808 -.02064 .08482 -.18882 .14755
.234 .629 .000 114 1.000 .00000 .04390 -.08696 .08696
.000 104.500 1.000 .00000 .04403 -.08730 .08730
.009 .924 -.158 114 .875 -.01227 .07773 -.16625 .14170
-.157 104.466 .875 -.01227 .07796 -.16685 .14231
.103 .748 -.614 114 .540 -.03955 .06438 -.16707 .08798
-.617 107.502 .538 -.03955 .06406 -.16653 .08744
5.287 .023 -.010 114 .992 -.00061 .05965 -.11876 .11755
-.010 113.320 .992 -.00061 .05801 -.11553 .11432
2.363 .127 .477 114 .634 .03439 .07212 -.10848 .17727
.484 110.899 .629 .03439 .07099 -.10629 .17508
2.803 .097 1.354 114 .179 .10879 .08037 -.05042 .26800
1.367 109.275 .174 .10879 .07956 -.04889 .26646
.280 .598 .902 114 .369 .05076 .05629 -.06076 .16228
.903 106.110 .369 .05076 .05623 -.06072 .16223
3.749 .055 1.200 114 .233 .08939 .07452 -.05822 .23701
1.216 110.375 .226 .08939 .07349 -.05625 .23503
.321 .572 .326 114 .745 .02088 .06405 -.10600 .14775
.323 101.705 .747 .02088 .06465 -.10737 .14913
1.431 .234 -.591 114 .556 -.00727 .01230 -.03165 .01710
-.602 111.535 .548 -.00727 .01208 -.03121 .01666
.969 .327 1.179 114 .241 .04958 .04206 -.03375 .13290
1.196 110.573 .234 .04958 .04146 -.03258 .13173
1.140 .288 .406 114 .686 .03571 .08800 -.13862 .21004
.408 107.639 .684 .03571 .08754 -.13781 .20923
Equal variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumedEqual variancesassumedEqual variancesnot assumed
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T15
F Sig.
Levene's Test forEquality of Variances
t df Sig. (2-tailed)Mean
DifferenceStd. ErrorDifference Lower Upper
95% ConfidenceInterval of the
Difference
t-test for Equality of Means
Table 4.17. Independent Sample t-test for surprise tests
53
V. DISCUSSION
This chapter discusses and addresses the research questions based on the results
obtained from the data analysis of this study.
The Website for this research study was designed to provide chemistry materials
to users interested in learning descriptive chemistry. The selection of (experimental [WE]
and control group [NWE]) users in the study was based on users who consistently used
the Website and reached the point of attempting all the five sets of surprise tests. This
group of 116 users is called the serious users. Although data analysis shows that a
number of users interested in learning chemistry materials had registered to the AP
descriptive chemistry Website, the final results indicate that only a small fraction of the
total number of users met the selection criteria of "serious user" and utilized the Website.
That is, of the 1373 users for whom some data are available, only 116 were serious
enough to persist in using the Website. Usage patterns and user associations such as the
dates accessed, the total amount of time spent, and number of each user logins were
considered to understand the users’ improvement in practicing chemistry problems and
thus the performance.
Usage Pattern
During the 2006 academic term, the dates of access to the Website show that
many users registered increasingly at the Website each month until the date of the AP
chemistry exam (May 2006). After the AP examination, new registrations declined
precipitously. This pattern of increase and decrease in the number of registrations was
similar in both the experimental and control groups. That is, even for serious users, the
54
date of the first Website visit was skewed toward the time of the AP examination with
more users making their first visit at a time nearer that of the examination. More users
were prone to practice chemistry problems just before the approaching AP chemistry
examination rather than practicing them at an earlier period of time.
The results of the total amount of time spent, which are indicated through the
elapsed time rates, suggest that a majority of all users spent little time practicing
descriptive chemistry problems. Also, it was found that both the WE and NWE group
followed an overall similar pattern of spending brief Website transaction times. That is,
there were no significant differences between the groups suggesting that the use of
worked examples did not make a difference in having users spend more or less time when
practicing chemistry problems (MWE = 12.12 hours, MNWE = 13.05 hours, p = 0.85, Tables
4.3 and 4.4). The extreme variables and outliers in the analyzed results data indicate that
there were a few cases where users appeared to spend either very long or very short times
using the Website. This is a case where having a laboratory situation rather than an
anonymous Website access approach would have given insights as to the reasons for
outliers. For example, for very long times, the users were likely to be off task due to
interruptions.
The total number of user interactions with the Website was analyzed. No
statistically significant difference between the experimental and control group means was
found with respect to average number of interactions (MWE = 259.48 interactions,
MNWE = 294.05 interactions, p = 0.35, Tables 4.5 and 4.6). Serious users in both groups
accessed the chemistry quiz items, eight-item quizzes, tutoring, and feedback. In addition,
users in the worked-example group received worked examples for practice.
55
Chemistry Quiz Items
The final scores of correctly and incorrectly answered chemistry quiz items were
analyzed to distinguish the differences between the experimental and control groups.
Quiz items were chosen by the user from one of the descriptive chemistry categories, and
the submitted quiz item was considered to be answered correctly when all the product and
reactant answers matched with the answers in the database for the particular quiz item.
Both the experimental and control groups accessed various quiz items and submitted their
quizzes for analysis. The total number of quiz items accessed varied within the groups,
however. Statistically significant differences were not found for correctly answered
practice items between the two groups (MWE = 26.72 correct, MNWE = 35.17 correct,
p = 0.17, Tables 4.7 and 4.8). A significant difference was found for incorrectly
answered quiz items. The worked-example group reported fewer incorrect answers than
did the no-worked-example group (MWE = 40.44 incorrect, MNWE = 56.83 correct,
p = 0.04, Tables 4.9 and 4.10).
Tutor Access
Users accessed tutoring resource components for each chemistry category of
interest. Some tutoring resources were accessed repetitively suggesting that the users
reviewed the tutoring more than one time. During the data analysis, the repetitive tutoring
was eliminated from calculations and only the unique tutoring accessed by users was
considered. However, significant differences were not found between the groups; both
groups accessed a similar number of tutoring resources (MWE = 4.26 tutors, MNWE = 3.73
tutors, p = 0.51, Tables 4.11 and 4.12).
56
Eight-item Practice Quizzes
Any user could access an eight-item quiz reminiscent of an actual AP descriptive
chemistry examination question. Significant differences were not found between the two
groups for accessing eight-item quizzes. Users from each group would receive a set of
eight quiz items. Each set was analyzed based on the number of correctly answered
products and reactants. In analyzing the eight-item quizzes, therefore, users were given
credit for partially correctly answered questions (MWE = 3.37 eight-item correct, MNWE =
3.78 eight-item correct, p = 0.31, Tables 4.14 and 4.15).
Worked Examples
Only users in the WE group had access to worked examples. Two worked
examples were possible for each of the first three practice items for each of the 15
practice group categories. The mean values for both the worked-example-1 and worked-
example-2 were considered when analyzing the worked example group user pattern in
utilizing the examples when practicing chemistry problems. It was found that worked-
example-1 was accessed more than twice as often as worked-example-2 (MWE-1 = 16.28
accesses; MWE-2 = 6.74 accesses; Table 4.13).
Surprise Tests
The principal part of this study was based upon the use of surprise tests. In the
many previous studies of this Website, there were no differential treatments nor were all
users assessed in some consistent way. The surprise tests were considered to be the
principal indicators for analyzing the performance of users when practicing chemistry
57
quiz items. Both groups received the same set of questions after the same number of
interactions with the Website. As closely as possible, the twentieth, fortieth, sixtieth,
eightieth, and one hundredth accesses to the site confronted the user with a surprise 3-
item quiz. The results are displayed in Table 4.16.
Based upon previous studies, it was thought that using worked examples would
lead to improved learning. Statistically significant differences were not found in any of
the five sets of surprise tests, however (Table 4.17). Each surprise test included 3 items,
and each of these items (15 total) was studied separately. There was also no discernable
pattern of improvement of scores from the first surprise test to the fifth surprise test
(Figure 5.1). Because the intrinsic difficulty of items cannot be controlled in the context
of realistic practice for the AP examination, variations when only three items are included
can be large. The first item of surprise-test-5 ("excess aqueous 1.0 M sodium iodide is
added to acidified 1.0 M sodium iodate") was perceived of by users as importantly more
difficult than item 2 of surprise-test-4 ("a few drops of methanol are burned in an excess
of air"). Most experienced high school or college chemistry teachers likely would have
guessed this outcome. However, item 3 of surprise-test-1 ("aqueous sodium sulfate is
added to aqueous strontium chloride") is really quite similar to item 3 of surprise-test-5
("aqueous barium chloride is mixed with aqueous potassium sulfate") but the average
scores (MWE = 0.37, p= 0.44 and MNWE = 0.39, p = 0.40) are not dramatically improved.
Eighty interactions took place with the Website between the first and last of these
“surprise test” items.
58
Surprise test items
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Test item number
Scor
e
Worked-example No-worked-example
Figure 5.1. Surprise test items
59
VI. SUMMARY AND RECOMMENDATIONS
This study has had several interesting, if not surprising, outcomes. Unlike the
previous studies which were conducted in 1999, 2000 and 2005, this study has included a
manipulation with experimental and control groups to better understand whether
students’ performance improved in the course of their practice and interaction with the
Website and whether the worked examples helped in reducing the errors that occur in
answering the quiz items.
Performance is the success rate at which students in both the worked-example
group and the no-worked-example group answered the practice quiz items, eight-item
quizzes and surprise tests. The results of the study indicate that there no significant
differences found between the groups in answering the practice quiz items, eight-item
quizzes, and surprise tests.
Although students in the worked-example group were provided with worked
examples during their practice with the descriptive chemistry Website, there were no
significant differences found between the groups and the rate of errors did not decrease or
increase. In other words, both the groups had very similar learning patterns in all the
occasions in using the descriptive chemistry Website.
The researcher finds this outcome to be interesting and further infers the two
reasons that might have caused such a result. User interaction with the Website was
found to be very limited. A larger percentage of users interacted with the Website only
just before the actual AP examination. Also, a majority of users did not continuously
utilize the Website. There were a number of registrants who visited the Website just once
60
or twice and did not further revisit the Website. Early registration (for example, at least
more than two months use and interaction with the Website) might have improved users’
performance.
Extensive use of worked examples and tutoring components might have had
added value to improve the performance. A majority of the students did not access both
of the worked examples provided to them but instead accessed only one or none.
Likewise, users might have referred to the tutoring components more frequently for
answering the chemistry quiz items for understanding the chemistry concepts before
practicing the quiz items.
61
REFERENCES
Bangert-Downs, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. (1991). The instructional
effects of feedback in test-like events. Review of Educational Research, 61(2),
213-238.
Brooks, D. W., Schraw, G.,Crippen, K, I. (2005). Performance-related feedback: The
Hallmark of Efficient Instruction. Journal of Chemical Education, 82(4), 641-644.
Clark, R. C., & Mayer, R. E. (2003). E-learning and the science of instruction. San
Francisco, CA: John Wiley and Sons, Inc.
College Board, (2006). Bulletin for AP students and parents 2005-2006. Retrieved
August 15, 2006, from
http://www.collegeboard.com/prod_downloads/student/testing/ap/AP-bulletin.pdf
Cooper, G. (1998). Research into cognitive load theory and instructional design at
UNSW. Retrieved September 25, 2005, from
http://education.arts.unsw.edu.au/CLT_NET_Aug_97.HTML.
Crippen, K. J., & Brooks, D. W. (2005). The AP descriptive chemistry question: Student
errors. Journal of Computers in Mathematics and Science Teaching, 24(4), 357-
366.
Crippen, K.J. (2000). Analysis of learning at an advanced placement descriptive
Chemistry Website. Unpublished doctoral dissertation, University of Nebraska at
Lincoln.
Crippen, K. J., Brooks, D. W., & Abuloum, A. (2000). A Website supporting the AP
62
descriptive chemistry question. Journal of Chemical Education, 77(8), 1087-1088.
Dick, W., Carey, L., & Carey, J. O. (2001). The systematic design of instruction (5th ed.).
New York, NY: Addison Wesley, Longman.
Gagné, R. M., Briggs, L. J., & Wager, W. W. (2001). Principles of instructional design
(5th ed.). Fort Worth: Harcourt Brace Jovanovich College Publishers.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller. J., (2003). The expertise reversal effect.
Educational Psychologist, 38, 23-31.
Marzano, R.J., Pickering, D.J., & Pollock, J.E. (2001). Classroom instruction that works:
Research-based strategies for increasing student achievement. VA: ASCD.
Mason, B. J., & Bruning, R. (2000). Providing feedback in computer-based instruction:
What the research tells us. Retrieved September 25, 2005, from
http://dwb.unl.edu/edit/MB/MasonBruning.html.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia
learning. Educational Psychologist, 38(1), 43-52.
Miller, George A., (1956). The magical number seven, plus or minus two: some limits on
our capacity for processing information. Retrieved August 5, 2006 from
http://www.well.com/user/smalin/miller.html.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design:
recent developments. Educational Psychologist, 38(1), 1-4.
Renkl, A., & Atkinson, R. K., (2003). Structuring the transition from example study to
problem solving in cognitive skills acquisition: A cognitive load perspective.
Educational Psychologist, 38(1), 15-22.
Sweller, J. (1999). Instructional design. Camberwell, Vic.: ACER Press.
63
Sweller, J., Tuovinen, J. E. (1999). A comparison of cognitive load associated with
discovery learning and worked examples. Journal of Educational Psychology,
91(2): 334-341.
van Merrienboer, J. J. G., Kirschner, P. A.,Kester, L. (2003). Taking the Load Off a
Learner's Mind: Instructional Design for Complex Learning. Educational
Psychologist, 38(1): 5-13.
VanLehn, D. A. (1996). Cognitive skill acquisition. Annual Review of Psychology, 47,
513-39.
Walberg, H. J. (1999). Productive teaching. In H. C. Waxman & H. J. Walberg (Eds.),
New directions for teaching practice and research, 75-104. Berkeley, CA:
McCutchen.
64
APPENDIX A
INFORMED CONSENT FORM
65
Contd.
66
Contd.
67
68
APPENDIX B
SURPRISE TESTS
69
Surprise test 1
70
Surprise test 2
71
Surprise test 3
72
Surprise test 4
73
Surprise test 5