THE CATHOLIC UNIVERSITY OF AMERICA

Deconstructing Constructivism: Modeling Causal Relationships Among Constructivist Learning Environment Factors and Student Outcomes in Introductory Chemistry

A DISSERTATION

Submitted to the Faculty of the

Department of Chemistry

School of Arts & Sciences

Of The Catholic University of America

In Partial Fulfillment of the Requirements

For the Degree

Doctor of Philosophy

By

Regis Komperda

Washington, D.C.

2016

Deconstructing Constructivism: Modeling Causal Relationships Among Constructivist Learning

Environment Factors and Student Outcomes in Introductory Chemistry

Regis Komperda, Ph.D.

Director: Diane M. Bunce, Ph.D.

The purpose of this dissertation is to test a model of relationships among factors

characterizing aspects of a student-centered constructivist learning environment and student

outcomes of satisfaction and academic achievement in introductory undergraduate chemistry

courses. Constructivism was chosen as the theoretical foundation for this research because of its

widespread use in chemical education research and practice. In a constructivist learning

environment the role of the teacher shifts from delivering content towards facilitating active

student engagement in activities that encourage individual knowledge construction through

discussion and application of content.

Constructivist approaches to teaching introductory chemistry courses have been adopted

by some instructors as a way to improve student outcomes, but little research has been done on

the causal relationships among particular aspects of the learning environment and student

outcomes. This makes it difficult for classroom teachers to know which aspects of a

constructivist teaching approach are critical to adopt and which may be modified to better suit a

particular learning environment while still improving student outcomes.

To investigate a model of these relationships, a survey designed to measure student

perceptions of three factors characterizing a constructivist learning environment in online

courses was adapted for use in face-to-face chemistry courses. These three factors, teaching

presence, social presence, and cognitive presence, were measured using a slightly modified

version of the Community of Inquiry (CoI) instrument. The student outcomes investigated in this

research were satisfaction and academic achievement, as measured by standardized American

Chemical Society (ACS) exam scores and course grades.

Structural equation modeling (SEM) was used to statistically model relationships among

the three presence factors and student outcome variables for 391 students enrolled in six sections

of a general chemistry course taught by four instructors at a single university using a common

textbook. The quantitative analysis of student data was supported by investigating the instructor's

approach to teaching using instructor responses to a modified version of the Approaches to

Teaching Inventory (ATI), semi-structured interview questions, and information available in the

course syllabus.

The results of the SEM analysis indicate that incoming math ability, as measured by ACT

math scores, has the largest effect on student academic achievement in introductory chemistry

courses. Of the three presence factors, cognitive presence has the largest direct effect on

academic achievement and student satisfaction. Teaching presence has a direct effect on

satisfaction similar in size to the effect of cognitive presence. The relationship between social

presence and student outcomes is found to be relatively small. Given the role that both teaching

and social presence play in influencing cognitive presence, these results suggest that classroom

teachers should emphasize the development of a learning environment with a large degree of

cognitive presence where students take ownership of their own learning process. This type of

learning environment can be supported by specific instructor behaviors such as facilitating

discussions and implementing group work focused on collaboration and developing shared

understandings.

ii

This dissertation by Regis Komperda fulfills the dissertation requirement for the doctoral degree in Chemical Education approved by Diane M. Bunce, Ph.D., as Director, and by Gregory Miller, Ph.D., Marc M. Sebrechts, Ph.D., and Gregory R. Hancock, Ph.D., as Readers.

__________________________________________

Diane M. Bunce, Ph.D., Director

__________________________________________

Gregory Miller, Ph.D., Reader

__________________________________________

Marc M. Sebrechts, Ph.D., Reader

__________________________________________

Gregory R. Hancock, Ph.D., Reader

iii

Table of Contents

List of Tables ............................................................................................................................... viii

List of Figures ................................................................................................................................. x

Chapter 1 ......................................................................................................................................... 1

Foundations of Constructivism ................................................................................................... 2

Constructivism in Chemical Education ...................................................................................... 3

Constructivism in Online Education ........................................................................................... 6

Research Model ........................................................................................................................ 10

Instructor Approaches to Teaching ........................................................................................... 14

Research Questions ................................................................................................................... 16

Modification and Pilot Study of Survey Instruments ............................................................... 16

Methodology and Sample Size ................................................................................................. 19

Data Analysis ............................................................................................................................ 21

Summary of Results and Implications for Teaching ................................................................ 22

Limitations and Future Research .............................................................................................. 25

Chapter 2 ....................................................................................................................................... 27

Philosophical and Psychological Foundations of Constructivism ............................................ 28

Constructivism as a Model of Learning .................................................................................... 30

Constructivism in Education ..................................................................................................... 35

Chemical Education .............................................................................................................. 41

Online Education .................................................................................................................. 44

Measuring a Constructivist Learning Environment and Student Outcomes ............................. 48

Defining Constructivism ....................................................................................................... 48

iv

Development of the Community of Inquiry Student Survey Instrument .............................. 50

Measuring Student Outcomes in Constructivist Learning Environments ............................. 62

Modeling the Influence of a Constructivist Learning Environment on Student Outcomes .. 67

Measuring Instructor Approaches to Teaching ......................................................................... 72

Research Questions ................................................................................................................... 78

Chapter 3 ....................................................................................................................................... 80

Modifications to ATI and CoI Wording ................................................................................... 80

Instructor and Student Survey Pilot Studies ............................................................................. 86

Recruitment of Participants for Pilot Studies ........................................................................ 86

Instructor Survey Pilot Study Methodology ......................................................................... 87

Student Survey Pilot Study Methodology ............................................................................. 88

Pilot Study Results .................................................................................................................... 89

Instructor Survey ................................................................................................................... 89

Student Survey ...................................................................................................................... 98

Survey Instrument Validity Evidence ..................................................................................... 101

Test Content ........................................................................................................................ 104

Response Process ................................................................................................................ 105

Internal Structure ................................................................................................................ 107

Relationships with Other Variables .................................................................................... 108

Consequences of Use .......................................................................................................... 109

Power Analysis for Sample Size Determination ..................................................................... 111

Overall Data-Model Fit ....................................................................................................... 112

v

Testing Parameters within the Model ................................................................................. 116

Selection of model parameters ........................................................................................ 117

Simplification of model-implied correlation matrix ....................................................... 119

Construction of the model-implied correlation matrix ................................................... 122

Calculation of model fit function values ......................................................................... 124

Methodology ........................................................................................................................... 127

Data Analysis .......................................................................................................................... 130

Qualitative Data Analysis ................................................................................................... 130

Quantitative Data Analysis ................................................................................................. 132

Data cleaning .................................................................................................................. 132

Assumptions for CFA and SEM analysis ....................................................................... 135

Internal structure of the CoI instrument .......................................................................... 137

Average scale scores and scale reliability ....................................................................... 141

Two-phase SEM Analysis ................................................................................................... 142

Chapter 4 ..................................................................................................................................... 146

Instructor Survey and Interview Results ................................................................................. 147

Student Data Analysis Results ................................................................................................ 150

Descriptive Statistics and Assumptions for SEM Analysis ................................................ 150

Confirmatory Factor Analysis of CoI Data ......................................................................... 154

CoI Scale Scores and Reliability ......................................................................................... 158

Structural Equation Model Results ..................................................................................... 162

Addressing the Research Questions ........................................................................................ 176

vi

Research Question 1 ........................................................................................................... 176

Research Question 2 ........................................................................................................... 180

Research Question 3 ........................................................................................................... 184

Summary of Results ................................................................................................................ 186

Chapter 5 ..................................................................................................................................... 188

Contribution of Results to Existing Literature ........................................................................ 189

Instructor and Student Ratings of Learning Environment .................................................. 189

Influence of a Constructivist Learning Environment on Student Outcomes ...................... 191

Implication of Results for Teaching Introductory Undergraduate Chemistry ........................ 197

Limitations and Future Research ............................................................................................ 200

Appendix A – Original CoI Items and Loadings ........................................................................ 205

Appendix B – ATI Revised for Pilot Study ................................................................................ 209

Appendix C – Student Survey Items Used in Pilot Study .......................................................... 211

Appendix D – ATI Revisions After Pilot Study ......................................................................... 214

Appendix E – Student Survey Revisions After Pilot Study ........................................................ 217

Appendix F – R Program for Calculating Coefficient H ............................................................ 220

Appendix G – Path Tracing and Matrix Determination for Hypothesized Research Model ...... 221

Appendix H – LISREL Syntax and Output for Power Analysis ................................................ 226

Appendix I – CoI and Satisfaction Instrument Used for Student Data Collection ..................... 230

Appendix J – Interview Transcript and Course Syllabus Coding Rubric ................................... 232

Appendix K – Mplus Model Syntax ........................................................................................... 233

Appendix L – Student Variable Correlations from Mplus .......................................................... 239

vii

References ................................................................................................................................... 242

viii

List of Tables Table 1: Original and Revised CoI Items ..................................................................................... 82 Table 2: Notation Used in Full Structural Equation Model in Figure 13 ................................... 115 Table 3: Sample Sizes for Testing Overall Data-Model Fit ....................................................... 116 Table 4: Sample Size Necessary to Test Each Focal Parameter Arranged from Largest to

Smallest ................................................................................................................................. 126 Table 5: Semi-Structured Instructor Interview Questions .......................................................... 130 Table 6: Evidence of Teaching Approaches Aligned with Constructivism from Interviews and

Syllabi ................................................................................................................................... 149 Table 7: Descriptive Statistics for Student Academic Variables .................................................151 Table 8: Descriptive Statistics for Student Survey Variables ......................................................151 Table 9: Independent t-tests for Differences in Academic Variables Based on Missing Survey

Responses .............................................................................................................................. 153 Table 10: Nested Model Comparison for Teaching Presence Factor ......................................... 156 Table 11: Social Presence Items with Added Error Covariance Terms ...................................... 157 Table 12: Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument .... 160 Table 13: Reliability Values for the Presence and Satisfaction Scales ....................................... 162 Table 14: Satisfaction Items from Student Survey ..................................................................... 165 Table 15: Model Parameter Values and Standard Errors (SE) from Final Research Model ...... 166 Table 16: Decomposed Standardized and Unstandardized Effects Among Variables in Structural

Model .................................................................................................................................... 169 Table 17: Standardized Loadings for Satisfaction Items in the Current Research and Existing

Literature ............................................................................................................................... 174 Table 18: Item on Each Presence Scale with Highest Mean Rating ............................................179

ix

Table 19: Cronbach’s Alpha for CoI Presence Scales in the Current Research and Existing

Literature ............................................................................................................................... 183 Table 20: Original CoI Item Wordings and Published Loadings, Paths, or Correlations ........... 205 Table 21: ATI Items Used in Pilot Study, Revised ATI Items, and Rationale for Revision ...... 214 Table 22: Algebraic Statements from Path Tracing .....................................................................221 Table 23: Correlation Matrix from Mplus ...................................................................................239

x

List of Figures

Figure 1. The Community of Inquiry model ................................................................................... 9 Figure 2. The hypothesized structural model of relationships among the three CoI presence

factors, math ability scores, and student outcomes ................................................................. 11 Figure 3. The Community of Inquiry model ................................................................................. 47 Figure 4. A factor model with three measured variables and one latent factor ............................ 54 Figure 5. A component model with three measured variables and one latent factor (component)

................................................................................................................................................. 54 Figure 6. A factor model with three measured variables and one latent factor with the error terms shown for each measured variable ................................................................................ 55 Figure 7. A model of hypothesized relationships among the 34 items on the CoI student

survey and the three presence factors ..................................................................................... 57 Figure 8. A model of hypothesized relationships among the three CoI presence factors

and student satisfaction ........................................................................................................... 60 Figure 9. A model of hypothesized relationships among the three CoI presence factors and

student satisfaction indicating the nonsignificant path between social presence and satisfaction .............................................................................................................................. 60

Figure 10. The hypothesized structural model of relationships among the three CoI presence

factors, math ability scores, and student outcomes ................................................................. 68 Figure 11. Teaching presence as a single factor with 13 indicator variables ............................... 71 Figure 12. Two correlated factors taking the place of a single teaching presence factor ............. 71 Figure 13. The primary structural equation model with all parameters labeled ......................... 114 Figure 14. The primary research model conceptualized with all focal parameters existing among

latent variables ...................................................................................................................... 120 Figure 15. The three-factor model of the CoI survey instrument used to inform the two-phase

SEM analysis ........................................................................................................................ 140 Figure 16. The CFA model used in the measurement phase of the SEM analysis ..................... 143

xi

Figure 17. The structural model used in the second phase of the SEM analysis ........................ 145 Figure 18. The three-factor model of the CoI survey with standardized parameter values ........ 159 Figure 19. Standardized values for focal parameters in the structural model and R2 values for

endogenous variables ............................................................................................................ 164

xii

Acknowledgements

I am deeply indebted to many people for their support and guidance. The dissertation

process includes both moments of exhilaration and moments of self-doubt and I have been

fortunate to have someone looking out for me every step of the way. The following people

provided the pricks of brightness I needed in order to see the light at the end of the tunnel and

continue moving forward.

First, I would like to thank my advisor, Dr. Diane Bunce, for her patience with the

process of extracting a dissertation topic from the tangled web of my mind and for allowing me

the independence to pursue a project that, in the early stages, seemed impossible. Her feedback

and assistance at every stage of this process were invaluable, and I truly appreciate the time and

energy she invested in reading and providing edits on multiple drafts of this dissertation.

Similarly, I want to thank Dr. Gregory Miller, Dr. Marc Sebrechts, and Dr. Gregory Hancock for

generously giving their time to serve on my committee and for their valuable insights and helpful

suggestions to improve this research. Additionally, I gratefully acknowledge all the instructors

and students who provided the data that ultimately made this research possible.

To my statistics professors, Dr. Michaela Zajicek-Farber and Dr. Hancock, thank you for

being exceptional teachers and helping me to understand, appreciate, and truly enjoy statistics in

a way I never thought I could. I especially want to thank Dr. Hancock for answering my

seemingly endless stream of questions with his characteristically cheerful attitude no matter how

frantic I became. I must also recognize Brian Johnston for giving me the opportunity to use

statistics “in the wild” and for providing me with access to Qualtrics to use for data collection.

Given the unpredictable nature of both my schedule and mood at various stages in this process I

xiii

also want to thank my coworkers for their moral support and for allowing me to vent every time

something did not go according to plan. This dissertation also benefitted greatly from numerous

conversations with colleagues who pointed me towards relevant literature, provided excellent

critiques of my logical arguments, and above all kept me company when I had spent too much

time alone writing or analyzing data.

None of this would have been possible without the constant encouragement of my family

and friends, particularly their unwavering optimism that I would eventually finish. Finally, I

want to thank the best old man dog in the world, Tolle. He provided the perfect mix of snuggles

and walk breaks that kept me active enough to avoid becoming permanently attached to my

laptop, and I am forever grateful for his companionship.

1

Chapter 1 Meta-analysis of 225 studies focusing on student academic performance in undergraduate

science, technology, engineering, and mathematics (STEM) courses demonstrates that students

in courses utilizing active learning techniques performed almost half a standard deviation (0.47)

better on assessments as compared to students exposed to traditional lectures (Freeman et al.,

2014). A similar result (0.39) was obtained when isolating the 22 chemistry course studies

included in the meta-analysis (Freeman et al., 2014). As defined by the researchers, active

learning “engages students in the process of learning through activities and/or discussion in class,

as opposed to passively listening to an expert. It emphasizes higher-order thinking and often

involves group work” (Freeman et al., 2014, p. 814). Another way to define these student-

centered teaching practices emphasizing knowledge construction through active engagement is

as constructivist.

This research utilizes constructivism as a theoretical framework to integrate research

from chemical education and online education. A model is presented which demonstrates how

specific aspects of a constructivist learning environment are hypothesized to influence student

outcomes of academic achievement and satisfaction. The primary focus of this research is the

testing of a structural equation model depicting relationships among student perceptions of

aspects of the learning environment characterizing a constructivist approach to teaching and

student outcomes. The analysis of student data is supported by data collected from the course

instructor regarding his or her approach to teaching. Acceptable fit between the hypothesized

model and the student data provides introductory undergraduate chemistry instructors with more

detailed information regarding which aspects of a constructivist approach to teaching have the

greatest influence on student academic and affective outcomes.

2

Foundations of Constructivism While the term constructivist has been used as an adjective to describe a collection of

student-centered teaching practices, constructivism can also be considered a philosophy as well a

model of learning. As a philosophy, constructivism includes various epistemological and

ontological viewpoints. Generally, constructivists believe that knowledge is constructed by each

individual as a result of his or her experiences (Bodner, 1986; von Glasersfeld, 1981/1984). The

philosophical aspects of constructivism are interwoven in the psychological research of Piaget

(1964/1997), who considered himself a genetic epistemologist and focused his research on

identifying and describing the internal processes carried out by each individual while engaged in

the process of learning.

Psychological research by Piaget, Vygotsky, and Ausubel focused on understanding the

cognitive processes at work when an individual constructs knowledge. The result of these efforts

is a model of learning that describes knowledge construction in terms of situating new

knowledge within existing cognitive structures. Piaget called this process assimilation

(1964/1997). Building on Piaget’s work, Ausubel (1960) determined that new information can

only become part of an existing cognitive structure if the new information is relevant to the

existing structure. Vygotsky’s (1978) research focused on the role of social interactions in

knowledge construction and demonstrated that under the guidance of an adult or more capable

peer, children could solve problems above their current individual developmental level.

Constructivism as a model of learning suggests that teaching practices aligned with the cognitive

development work of Piaget, Ausubel, and Vygotsky should enhance the knowledge construction

process in individuals.

3

Though constructivism is a model of learning not a theory of teaching, it provides a

framework through which teaching practices can be analyzed to determine if they are consistent

with the constructivist model of learning. Teaching practices aligned with constructivism are not

new and many have been successfully implemented in the past before constructivism became

prevalent in education (von Glasersfeld, 1989). In this way, constructivism does not necessarily

represent a new way to teach, but rather a new way to understand which teaching practices

should be effective due their alignment with a constructivist model of learning.

Some teaching practices consistent with constructivism include the instructor (1) acting

as a facilitator of learning, (2) identifying existing student cognitive structures in order to make

the connections between existing knowledge and new knowledge more explicit, (3) creating

authentic problem solving tasks, (4) fostering active student involvement in problem solving, (5)

supporting students working and communicating in groups to socially construct understanding,

(6) encouraging discussion of and reflection on the learning process, and (7) assessing more than

arriving at a correct answer (Duffy & Cunningham, 1996; Hyslop-Margison & Strobel, 2008;

Piaget, 1973; von Glasersfeld, 1989; Windschitl, 2002). In general, teaching practices aligned

with constructivism are student-centered and shift the classroom focus away from an instructor

who is dispensing information. By examining which teaching practices are effective for various

types of students and the alignment of these teaching practices with a constructivist model of

learning, the constructivist model of learning can be tested and refined.

Constructivism in Chemical Education The influence of constructivist teaching practices on student outcomes in chemistry

courses can be seen most directly in research on the use of process-oriented guided-inquiry

4

learning (POGIL; Hanson, 2006, 2008) and peer-led team learning (PLTL; Varma-Nelson &

Banks, 2013; Varma-Nelson & Coppola, 2005). In both POGIL and PLTL, one or more weekly

lectures or recitation sections are replaced with workshop sessions in which groups of students

work together to construct an understanding of chemical concepts. Critically, both PLTL and

POGIL approaches warn against using “drill” or “plug and chug” type problems that encourage

memorization and application of algorithms and instead encourage problems emphasizing

application of concepts or synthesis of new ideas to encourage more active learning. In this way,

both POGIL and PLTL incorporate teaching practices aligned with a constructivist model of

learning.

Investigations of the effect of constructivist teaching practices on student learning are

complicated by the difficulty of directly measuring student learning. The chemical education

research community relies on exam grades, course grades, and standardized American Chemical

Society (ACS) exam scores as ways to measure student learning (Conway, 2014; Gosser,

Kampmeier, & Varma-Nelson, 2010; Gupta, Burke, Mehta, & Greenbowe, 2015; Hall, Curtin-

Soydan, & Canelas, 2014; Lewis & Lewis, 2005; Mitchell, Ippolito, & Lewis, 2012; Ruder &

Hunnicutt, 2008; Tien, Roth, & Kampmeier, 2002). The beneficial effect of POGIL and POGIL-

style instruction on both final exam and final course grades has been demonstrated in a one-

semester organic and biochemistry course for pre-health professionals and in large enrollment

general and organic chemistry courses (Conway, 2014; Ruder & Hunnicutt, 2008). Similar

results have been reported for implementations of PLTL in general and organic chemistry

courses from a variety of institutions including community colleges and research universities

(Gosser et al., 2010; Lewis & Lewis, 2005; Mitchell et al., 2012).

5

Implementing more general constructivist teaching practices, not specifically POGIL and

PLTL, has been shown to improve student academic achievement in chemistry. Hall et al. (2014)

describe a supplemental discussion-type section that has “roots in social constructivism and

borrows elements from a number of learner-centered pedagogies” (p. 37). This program recruited

students with lower SAT scores who, after enrollment in the program, earned exam scores in

both their general and organic chemistry courses that were not statistically different from peers

entering with higher SAT scores. Considering SAT scores is necessary to show the effectiveness

of teaching practices across different groups of students, since a demonstrated relationship exists

between math ability scores, as measured by SAT and ACT math scores, and grades in

introductory college science courses (Lewis & Lewis, 2005; Nordstrom, 1990; H. E. Spencer,

1996; Tai, Sadler, & Mintzes, 2006; Xu & Lewis, 2011).

In addition to improvements in academic achievement in chemistry, student satisfaction

with the constructivist learning environment has also been reported in the chemical education

literature (Conway, 2014; Hall et al., 2014; Ruder & Hunnicutt, 2008; Tien et al., 2002). Though

satisfaction and attitude are frequently used interchangeably, Xu & Lewis (2011) identify an

emotional satisfaction aspect of student attitudes towards chemistry. The courses in which the

students in the Xu & Lewis study were enrolled were not specifically described as constructivist,

but a correlation of 0.35 was found between emotional satisfaction and ACS exam scores

indicating that a relationship exists between student satisfaction and academic outcomes.

However, larger correlations (0.45 and 0.46) existed between ACS exam scores and math ability

scores as measured by SAT math and ACT math scores, respectively (Xu & Lewis, 2011),

confirming the relationship between math ability and academic achievement in chemistry.

6

Taken together, these studies in the chemical education literature demonstrate that student

academic achievement in introductory chemistry courses is related to both the teaching approach

employed by the instructor and the preexisting math ability of the student. In addition, a

relationship has been demonstrated between academic achievement and student satisfaction with

a course. Some of the teaching approaches utilized in these studies have explicit ties to

constructivism while other studies adopt more general student-centered approaches to teaching

that are also aligned with constructivism. However, none of the studies explores the specific

aspects of the teaching approach that may influence student outcomes. Without this level of

detail, it is difficult for classroom instructors to know which aspects of a constructivist teaching

approach are critically important to adopt to influence student outcomes and which aspects can

be modified to better suit the style of a particular instructor.

Constructivism in Online Education Research in online education has taken a complementary approach to investigating the

role of constructivism. Whereas chemical education has emphasized the student outcomes that

occur as a result of adopting student-centered constructivist teaching practices, online education

research has focused on identifying aspects of the learning environment that influence student

outcomes of satisfaction and persistence. For online educators, constructivism is a useful

framework for engaging students and fostering the development of knowledge (Vrasidas, 2000)

without the constant presence of a teacher.

The first cohesive model of online learning came from the development of the

Community of Inquiry (CoI) model. The CoI model was developed from analysis of computer-

conferencing transcripts (D. R. Garrison, Anderson, & Archer, 2000) using a grounded theory

7

approach of working from data to develop a theory (Creswell, 2013). From this analysis, three

types of presence emerged as the foundation of the CoI model: cognitive presence, social

presence, and teaching presence. The influence of constructivism can be seen in each type of

presence.

Cognitive presence is defined as “the extent to which the participants in any particular

configuration of a community of inquiry are able to construct meaning through sustained

communication” (D. R. Garrison et al., 2000, p. 89). Here the idea of knowledge construction by

constructing meaning indicates that cognitive presence is not simply a matter of providing

content to students but rather is related to the degree to which the content fosters mental activity

on the part of the students. In the CoI model, cognitive presence is described in terms of a four

stage inquiry learning cycle. This definition also highlights the social nature of the CoI model

since the role of both the other students in the course and the instructor is to provide someone to

communicate with to construct meaning.

The three types of presence in the CoI model work together to encourage deep learning,

as illustrated by the fact that “cognitive presence…is more easily sustained when a significant

degree of social presence has been established” (D. R. Garrison et al., 2000, p. 95). The CoI

definition of social presence as “the ability of participants in a community of inquiry to project

themselves socially and emotionally, as ‘real’ people (i.e., their full personality)” (D. R. Garrison

et al., 2000, p. 94) indicates that social presence is only linked to a successful educational

experience when it supports affective as well as cognitive outcomes. These affective outcomes

such as finding “the interaction in the group enjoyable and personally fulfilling” (D. R. Garrison

8

et al., 2000, p. 89) are described as important for keeping students engaged and enrolled in the

online course.

The final aspect of the CoI model, teaching presence, describes the responsibility of the

instructor to establish a learning environment that supports the development of both social and

cognitive presence. The role of the instructor as a facilitator of learning invokes the student-

centered emphasis of constructivism. The overlap of teaching presence and cognitive presence is

necessary in order to select content that will encourage students to follow the stepwise inquiry

cycle that comprises cognitive presence, and the overlap of teaching presence and social

presence is necessary to set a course climate that will foster the development of a learning

community. A visual description of the CoI model can be seen in Figure 1. In this model the

three presence factors are represented by overlapping circles and the indicators are represented

by arrows pointing to each presence factor. The steps in the inquiry cycle are joined by dashed

arrows. These indicators highlight the types of activities hypothesized to improve learning when

undertaken by the instructor and students.

The CoI model indicators were operationalized by online education researchers to create

a student survey instrument that could be used to determine the degree to which a learning

environment was perceived by students as fostering these three types of presence. Since these

types of presence describe various aspects of a constructivist learning environment, this survey

can also be used as a tool to measure the degree to which an instructor has created a

constructivist learning environment. Utilizing student perceptions of the learning environment is

an advantageous technique because students are the intended target of the teaching approach

employed by the instructor. Additionally, students directly experience the learning environment

9

Figure 1. The Community of Inquiry model. Indicators of each type of presence are given in the arrows. Adapted from D. R. Garrison et al. (2000) and Swan (2003). over an extended period of time and are therefore able to provide a more complete description

of the learning environment than an observer who may not attend all class sessions or fully

engage in activities and assignments.

The development and use of the CoI student survey has been documented in the online

education literature with both undergraduate and graduate students enrolled in fully online

courses from various disciplines from engineering to education to business (Arbaugh, Bangert, &

Cleveland-Innes, 2010; Arbaugh, 2008; Bangert, 2008; D. R. Garrison, Cleveland-Innes, &

Fung, 2010; Joo, Lim, & Kim, 2011; Shea & Bidjerano, 2009). After the initial instrument

10

development studies demonstrated the utility of the 34-item CoI instrument, researchers began to

look at relationships among the three CoI presence factors (cognitive, social, and teaching) and

student outcome variables such as satisfaction and persistence (Joo et al., 2011) utilizing the

statistical technique of structural equation modeling (SEM). The CoI instrument items and

literature values for relationships among factors are provided in Appendix A.

The structural equations in SEM describe hypothesized causal relations among variables.

Causal relations provide more specific information about the hypothesized influences of

variables than correlations. Correlations among the three CoI presences were expected to exist

due to their overlapping nature in the CoI model, but the correlations could not provide

information about causal paths among the variables including the influence of cognitive, social,

and teaching presence on student outcomes. The purpose of the current research using SEM will

be to investigate the causal relationships among the three CoI presence factors and student

outcomes in face-to-face introductory chemistry courses.

Research Model The previously discussed studies in online and chemical education can be synthesized into

a single model hypothesizing the influence of a constructivist learning environment on student

outcomes of academic achievement in chemistry and satisfaction. In this model, the

constructivist learning environment is measured by the three CoI factors of cognitive presence,

social presence, and teaching presence. Academic achievement in chemistry is measured by the

outcomes typically used in chemical education research such as ACS exam scores and final

course grades. Student satisfaction is measured using a survey instrument as is typical in both

online and chemical education research.

11

The model in Figure 2 provides a diagrammatic representation of the hypothesized

structural relationships among these variables. Relationships, or paths, between two variables are

indicated with directional arrows. Latent variables, also called factors, are shown as ovals and

represent variables that are not measured directly but will be identified by statistical analysis of

student responses to CoI and student satisfaction survey items. Measured variables of math

ability and academic achievement in chemistry are shown as rectangles and are determined based

on student scores. The portion of the model showing the individual CoI and satisfaction survey

items has been omitted for simplicity in presentation.

Figure 2. The hypothesized structural model of relationships among the three CoI presence factors, math ability scores, and student outcomes.

12

Based on prior research in online education, teaching presence is hypothesized to directly

influence both cognitive and social presence while also indirectly influencing cognitive presence

through social presence. The multiple influences of teaching presence are due to the role of the

instructor in both selecting course content and setting the tone of interactions between the

instructor and students (D. R. Garrison et al., 2010; Shea & Bidjerano, 2009; Swan, 2003). From

the chemical education literature, the cognitive presence and teaching presence aspects of a

constructivist learning environment are expected to directly influence academic achievement in

chemistry as measured by ACS exam scores and final course grades (Conway, 2014; Gosser et

al., 2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder & Hunnicutt,

2008). The direct influence of teaching presence on ACS exam scores and final course grades is

hypothesized because “the instructor serves as an expert who plans instruction to stimulate

students’ interest, motivates their participation in the learning process, and facilitates their

learning” (Swan, 2003, p. 8). Though no studies with the CoI instrument have specifically

examined the influence of teaching presence on student academic outcomes, the role of the

instructor as expert and facilitator of learning suggests a causal relationship between teaching

presence and academic achievement in chemistry.

Additionally, in the CoI model selecting appropriate content is at the overlap of teaching

and cognitive presence. The selection of content and learning activities that facilitate student

knowledge construction is expected to influence the demonstration of that knowledge

construction through ACS exam scores and final course grades. For this reason, cognitive

presence is also hypothesized to have a causal influence on academic achievement in chemistry.

13

In addition to the influence of teaching and cognitive presence, math ability is also

expected to have a direct influence on academic achievement in chemistry (Lewis & Lewis,

2005; Mitchell et al., 2012; Nordstrom, 1990; Tien et al., 2002; Xu & Lewis, 2011). A path is

included to account for the influence of ACS exam scores on final course grades. The paths

between social presence and both ACS exam scores and final course grades are omitted because

the influence of social presence on these two outcomes is hypothesized to be indirect and

mediated primarily by cognitive presence. This indirect influence is hypothesized because of the

small correlation seen between social presence and perceived learning in earlier studies

(Arbaugh, 2008) and the assumption that social presence only affects academic outcomes when

cognitive presence provides an academic context for the social interactions among students.

Finally, from both the online and chemical education literature, cognitive presence, social

presence, and teaching presence are all expected to directly influence satisfaction (Conway,

2014; Hall et al., 2014; Joo et al., 2011; Ruder & Hunnicutt, 2008). A two-headed arrow is

included to account for a relationship between final course grades and student satisfaction

beyond the relationships already present in the model, but no causal direction is proposed for the

relationship. The lack of causality reflects the ongoing debate as to whether students who are

more satisfied with a course perform better academically or if performing better academically

causes students to be more satisfied with a course (Greenwald & Gillmore, 1997; Howard &

Maxwell, 1982).

This hypothesized model represents the integration of research in both chemical education

and online education through their shared use of constructivism as a foundation for the

development of teaching practices aligned with how students learn. In this model, student

14

perceptions and student performance are the variables used to model relationships among

constructivist learning environment factors and outcomes of student satisfaction and academic

achievement in introductory chemistry. However, student measurements should not provide the

only source of information about the degree to which a learning environment incorporates

constructivist principles. The course instructor can also provide data to support or refute the

picture of the learning environment portrayed by student responses to the CoI survey instrument.

Instructor Approaches to Teaching Previous studies in online and science education have utilized interviews with instructors in

order to determine their approach to teaching (Arbaugh & Benbunan-Fich, 2007; Prosser,

Trigwell, & Taylor, 1994; Trigwell, Prosser, & Taylor, 1994). In the Arbaugh and Benbunan-

Fich (2007) study, instructors’ responses were compared with information in the course syllabi

or course websites in order to support the researchers’ classification of the course as either

objectivist or constructivist and group-centered or individual-centered. All instructor reports in

the Arbaugh and Benbunan-Fich (2007) study were found to be consistent with information

provided in the course syllabi or websites.

Interviews were also used to inform phenomenographic qualitative research investigating

the approaches to teaching adopted by 24 instructors of first-year undergraduate chemistry and

physics (Prosser et al., 1994; Trigwell et al., 1994). In later research, Trigwell & Prosser (2004)

used the interview transcripts to develop an inventory that could be administered to instructors in

order to determine their approach to teaching in a particular context. Statistical analysis of this

Approaches to Teaching Inventory (ATI) indicates that the ATI is an acceptable instrument for

identifying two distinct instructor approaches to teaching in specific contexts (Prosser &

15

Trigwell, 2006; Trigwell, Prosser, & Ginns, 2005). One approach identified by the ATI can be

described as an information transmission teacher-focused approach (ITTF) while the other

represents a conceptual change student-focused approach (CCSF).

The ATI has been used in chemical education research to examine how the teaching

approaches of new university chemistry professors change after attending a short teaching

workshop emphasizing the use of student-centered teaching approaches (Stains, Pilarz, &

Chakraverty, 2015). The instructors who attended this workshop had statistically significantly

higher CCSF scores one week after the workshop compared to their CCSF scores before the

workshop. They also had significantly lower ITTF scores than a control group who did not attend

the workshop. These results suggest that the CCSF scale measures approaches to teaching

aligned with constructivism.

Combining information from the perspectives of two classroom stakeholders, the

instructor and students, provides a more complete picture of the learning environment created by

the instructor and experienced by the students. Measuring the learning environment from both

perspectives also allows for an examination of how well the ATI and CoI instruments measure

the existence of different aspects of a constructivist learning environment. Comparing responses

to the ATI and CoI items also provides support for the acceptability of the CoI instrument for

measuring indicators of a constructivist learning environment from the students’ perspective.

Additionally, the psychometric properties of the CoI survey must be examined quantitatively as a

result of the small modifications in wording required for the instrument to be used with students

in a face-to-face class. Combining CoI responses with measurements of student outcomes will

provide the data necessary to test the model proposed in Figure 2. Testing this model provides

16

information on how a constructivist learning environment, as perceived by students, affects

student satisfaction and academic achievement in chemistry. These research goals are

summarized by the following research questions.

Research Questions

1. Are self-reported instructor approaches to teaching consistent with student

perceptions of the learning environment?

2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for

measuring student perceptions of the indicators of a constructivist learning

environment in a face-to-face introductory undergraduate chemistry course?

3. To what degree does a constructivist learning environment, as measured by

student CoI survey responses, affect outcomes of student satisfaction and

academic achievement in chemistry, as measured by ACS exam scores and final

course grades, when the effect of math ability on academic achievement is

considered?

Modification and Pilot Study of Survey Instruments

Both the ATI and CoI survey instruments required modifications to their wording before

they could be used to address the research questions. Initially, items on both were kept as similar

as possible to their original wording. In the ATI, changes were made to the original

European/Australian wording to more closely align the language with US usage by changing the

term “subject” to “course.” As an example, the item worded “In this subject, I provide the

students with the information they will need to pass the formal assessments” was revised to “In

17

this course, I provide the students the information they will need to pass the formal assessments.”

Similar revisions occurred for all items on the ATI. Four items on the CoI were reworded to

reflect the intended face-to-face focus of the current research. For these items the word “online”

was replaced with “face-to-face”. Two additional items on the CoI were modified to align the

instrument with best practices in survey design (Krosnick & Presser, 2010). One change was

made so that all items related to teaching presence started with the same question stem, “The

instructor…” The second change split the item reading “Reflection on course content and

discussions helped me understand fundamental concepts in this class” into two separate items,

one addressing reflections on course content and the other addressing reflections on discussions.

Changes were also made to the scales of both instruments to provide labels for each of

the five scale points above the corresponding number. In addition, a “Not Applicable” option

was added to the CoI instrument with a numerical value of zero for situations in which an item

did not apply to a specific student or course. It was anticipated that this situation might occur

when formal, structured in-class discussions were not part of the course but the item asked about

discussions.

The survey instruments underwent a pilot study to ensure the items were being

interpreted as intended due to the wording changes and the planned use of the instruments with a

research population that differed from the population reported in the literature. Two types of

satisfaction items were included on the student survey during the pilot study to provide a

comparison of responses to traditional satisfaction items from the online education literature

(Bolliger & Wasilik, 2012) with responses to satisfaction items using a semantic differential

18

scale from the chemical education literature (Xu & Lewis, 2011). The instruments used for the

pilot study with instructors and students are provided in Appendix B and C, respectively.

Permission was obtained from the university institutional review board (IRB) to recruit

instructor and student participants for the pilot studies. Five chemistry instructors and five

undergraduate chemistry students participated in the pilot studies of the instructor and student

survey instruments. The protocol for both pilot studies was a think-aloud interview in which each

participant read the items on the survey instrument aloud and verbalized his or her rationale for

selecting a particular response (Krosnick & Presser, 2010). Analysis of these think-aloud

responses was used to determine if the interpretation of the items aligned with their intended

interpretation or if particular items needed to be reworded to ensure more accurate and consistent

interpretation. Both the instructors and students used a specific introductory chemistry course,

either recently taught or completed, as a reference when responding to the survey items. The

syllabus of the relevant course was collected and analyzed to look for consistency between

responses to the surveys and the classroom practices listed in the syllabus. The instructor pilot

study also included semi-structured interview questions to provide a description of the

instructor’s approach to teaching in his or her own words to compare against responses to the

ATI.

As a result of the instructor pilot study, extensive revisions were made to the ATI to

make the items less vague and ensure that each item focused more closely on course design and

actual classroom practices instead of instructor beliefs about teaching. The extensive revisions to

the ATI led to the conclusion that ATI responses should not be used in isolation in future

research. Instead, the main research study combined data from ATI responses with a semi-

19

structured instructor interview and analysis of the course syllabus to provide a more complete

picture of the approach to teaching utilized by the course instructor.

The CoI items did not require extensive revisions, though a few small changes were made

based on student responses to particular items. Student responses demonstrated that the semantic

differential items were best for capturing overall satisfaction with the course, so these were

chosen for use in the main research project instead of the traditional satisfaction items. Appendix

D and E contain the instructor and student survey instruments after revisions based on the results

of the pilot studies.

Methodology and Sample Size

The design of the main research study utilized a mixed methods approach in which both

quantitative and qualitative data were collected and analyzed to answer the research questions

(Creswell, 2014). A mixed methods approach was chosen in order to minimize some of the

limitations of a purely quantitative approach by integrating qualitative data to provide a more

comprehensive understanding of the learning environment. For this research, the primary focus

of the data collection and analysis was quantitative data obtained from instructor responses to the

ATI, student responses to the CoI and satisfaction instrument, and student achievement data in

the form of math ability scores on the math portion of the ACT, ACS exam scores, and final

course grades. The qualitative portion of the research consisted of short semi-structured

instructor interviews and analysis of the course syllabi.

The first research question was addressed by comparing quantitative data from student

responses to the CoI and instructor responses to the ATI with qualitative data obtained from the

instructor interview and course syllabus analysis. The second and third research questions were

20

addressed through statistical analysis of quantitative data. Structural equation modeling (SEM)

was chosen as the primary statistical data analysis methodology over other techniques such as

hierarchical linear modeling (HLM) because of the complex causal relationships investigated in

which the three CoI presence factors were hypothesized to influence each other as well as

multiple student outcomes (Bauer, 2003; Huta, 2014; Kline, 2011). The second research question

can be considered a subset of the third research question in which only relationships among the

three CoI presence factors were examined. Since the third research question can only be

answered by testing the hypothesized model of relationships among all student variables, the

necessary sample size for the research was determined from the model illustrated in Figure 2.

Two separate power analyses were conducted to determine the sample size necessary to

test overall data-model fit and to test the specific model parameters of interest in this research.

The results of both a priori power analyses indicated that a sample size of approximately 80

students would be sufficient to test both overall data-model fit and specific model parameters of

interest with power = .80 and alpha = .05. The number of instructors participating in the research

was determined by how many instructors taught the students whose data was analyzed.

The student data analyzed in this research were obtained from an existing data set for an

administration of the modified CoI instrument collected for another project investigating

predictors of student success in general chemistry. This data set contained almost 400 usable

anonymized student responses to the CoI and satisfaction survey instrument in addition to scores

on the first-semester ACS general chemistry exam, final course grades excluding laboratory

scores, and ACT math scores. These students were enrolled in six sections of a first semester

general chemistry course taught by four different instructors at a large, public, primarily

21

undergraduate institution (Indiana University Center for Postsecondary Research, n.d.). Though

the student data can be considered as grouped by classroom, the selection of classrooms was

based on availability not random sampling, as would have been necessary for the use of HLM

(Huta, 2014). All four instructors completed the ATI, provided their course syllabus, and

participated in the semi-structured interview.

Data Analysis

The instructor data obtained for this research had three components. The first component

was quantitative responses to the ATI. The second was course syllabi provided by the instructors.

The final component of the instructor data was transcripts of the instructors’ responses to semi-

structured interview questions asking for a description of their approach to teaching the

introductory undergraduate chemistry course from which the student data were collected.

Qualitative analysis and coding of the transcripts of these instructor interviews and course syllabi

demonstrated a major theme of student-centered teaching practices consistent with

constructivism. Ultimately, this qualitative data was analyzed along with the results of instructor

responses to the ATI and student responses to the CoI survey in order to address the first

research question.

The student data analyzed for this research only consisted of the quantitative data

available in the anonymized data set. The quantitative analysis of the student data began by

cleaning the data set to remove unusable student responses. As a result of the data cleaning steps

the number of usable student participants was 391. The 391 responses represent approximately

89% of the total 439 responses collected. Exclusion of 11% of the initial sample still provided

22

roughly five times the minimum sample size necessary to examine the overall data-model fit and

model focal parameters with sufficient power.

Descriptive statistics were computed in order to check assumptions regarding missing

data and normality necessary for selection of the correct estimation technique for use when

testing the models of the hypothesized structure of the CoI instrument and the overall research

model presented in Figure 2. This descriptive statistical data is presented in Chapter 4. Mplus

software (version 7.0) was used to test the hypothesized models.

Initially the internal structure of the CoI instrument was tested with confirmatory factor

analysis (CFA). The CFA results provided information regarding the best overall model for the

individual items and three factors that comprise the CoI instrument. After the CoI CFA,

structural equation modeling (SEM) analysis was used to test the model presented in Figure 2.

This analysis followed the two-phase SEM analysis recommended by Mueller & Hancock

(2008).

Summary of Results and Implications for Teaching

The first research question was addressed by integrating the qualitative data collected

from the instructor interview and syllabus analysis with instructor responses to the ATI and then

comparing this analysis to student responses to the CoI items. Analysis of instructor and student

data indicated that student and instructor perceptions of the learning environment were generally

aligned. This alignment was demonstrated by the students perceiving indicators of a

constructivist learning environment in their CoI responses while the instructors described

approaches to teaching that were consist with a student-centered constructivist approach to

23

teaching in their ATI responses, course syllabi, and interviews. These results are presented in

Chapter 4.

The answer to the second research question was determined from evidence for the

validity and reliability of the CoI survey scores. Some validity evidence resulted from how well

the data fit a hypothesized three-factor model of the CoI instrument based on models of the CoI

survey found in the online education literature (Arbaugh et al., 2010; Arbaugh, 2008; Bangert,

2008; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). The model !"

provided an indication of data-model fit, where smaller values relative to the degrees of freedom

of the model (df) indicate better data-model fit. Data-model fit was also examined using fit

indices such as the comparative fit index (CFI), the root mean square error of approximation

(RMSEA) along with its 90% confidence interval (CI90), and the standardized root mean square

residual (SRMR). Scaled fit indices were computed as a result of corrections to the model !" due

to the nonnormal distribution of the data. A slightly modified three-factor model of the CoI

instrument was found to have the following data-model fit: !scaled,+,-./0

" = 1028.717; CFIscaled =

0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR = 0.061. Though the CFI is below the

target value of 0.95, possibly indicating a relatively small amount of variance and covariance in

the data, the model is a good fit for the data as indicated by the RMSEA and SRMR values based

on joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). This result indicates

that the tested model of the internal structure of the CoI is a viable representation of the true

underlying relationships present in the data. In addition to the overall data-model fit information,

reliability information calculated for the three CoI scales also provides evidence to support the

conclusion that the CoI is an acceptable instrument. Therefore, these results provide a positive

24

answer to the second research question indicating that the modified CoI survey is an acceptable

instrument for measuring student perceptions of the indicators of a constructivist learning

environment in face-to-face introductory undergraduate chemistry courses.

The third research question was addressed broadly by the overall data-model fit for the

full hypothesized research model in Figure 2. This model again had acceptable data-model fit

(!scaled,+,-120

" = 1429.111; CFIscaled = 0.892; RMSEAscaled = 0.053, CI90 = [0.049, 0.056]; SRMR

= 0.065) though not all hypothesized relationships among variables were found to be statistically

significant. Decomposing the relationships between two variables into direct and indirect effects

allowed for an examination of how the variables in the model influenced each other. These

results indicate that while constructivist learning environment factors of teaching presence and

cognitive presence do appear to have an influence on student satisfaction and academic

outcomes, the influence of math ability on academic outcomes is larger. The moderately large

and significant effects of cognitive presence on academic outcomes and affective outcomes

shows that a more general adoption of constructivist teaching approaches, as measured by

cognitive presence, positively influences student outcomes, even after the effects of math ability

are considered.

Additionally, social presence appears to have very minimal influence on either academic

outcomes or student satisfaction. The small effect of social presence on student outcomes

indicates that only group work that encourages the development of cognitive presence through

the use of group activities that foster active student engagement with material in support of

constructing explanations and understanding will ultimately influence student satisfaction and

academic achievement. For classroom instructors, these results suggest that group work should

25

be implemented with careful thought as to how it supports the development of a constructivist

learning environment.

As a result of this research, information is now available about the nature of relationships

among the variables in the research model. Understanding the nature and magnitude of the

relationships among specific factors indicating a constructivist learning environment and student

outcomes supports instructors in making informed decisions about how to most effectively

approach teaching in their own classrooms to maximize student outcomes given the ever-present

limitations of time and energy. By examining the effects of specific aspects of a constructivist

learning environment, rather than the effects of adopting an entire approach to teaching,

instructors should have a better sense of how to implement constructivist teaching practices in a

way that meets the needs and preferences of the individual instructor and the particular

classroom environment.

Limitations and Future Research

The small number of instructors involved in the research limited the ability to provide

additional evidence for the validity of the ATI scores given the extensive revisions that occurred

as a result of the pilot study. Some evidence for the validity of the ATI scores was available

based on instructors’ descriptions of their approaches to teaching provided during the semi-

structured interviews, but it would be beneficial to collect ATI responses from a larger set of

instructors so that the internal structure of the revised ATI instrument could be tested with CFA.

Similarly, the use of preexisting student data made it impossible to conduct think-aloud

interviews with students who had completed the CoI survey while they were still enrolled in the

course in order to provide evidence for the validity of the CoI survey scores based on the

26

response process of the students. Future research should consider conducting think-aloud

interviews with students who complete the CoI survey to ensure their interpretation of the items

is aligned with the intended item interpretations.

Using data from instructors teaching the same course at the same university in the same

semester using the same textbook provided some benefits in minimizing differences across

classrooms so that the student data could be combined into a single data set, but it also limited

the generalizability of this research. Since the current research established a relationship among

the CoI presence factors and student outcomes in a single chemistry course taught by multiple

instructors, future research should gather data from additional chemistry courses in which a

wider variety of approaches to teaching have been adopted. This would allow the research model

to be tested more broadly and provide evidence either supporting or modifying the relationships

examined in this study.

27

Chapter 2

For almost 30 years, chemical educators have developed and evaluated new teaching

practices encouraging a shift from teacher-centered to student-centered classrooms. An example

of student-centered teaching practices would be students working in groups during class time to

actively solve difficult problems and construct their own knowledge of chemical concepts.

Teaching practices emphasizing individual knowledge construction can be classified as

constructivist. One way constructivist teaching practices have entered the chemistry classroom is

through specific pedagogies such as process-oriented guided-inquiry learning (POGIL; Hanson,

2006, 2008) and peer-led team learning (PLTL; Varma-Nelson & Banks, 2013; Varma-Nelson &

Coppola, 2005). Improvement in final exam and final course grades as well as high levels of

student satisfaction with the course have been demonstrated when POGIL and PLTL were

employed in undergraduate general and organic chemistry courses (Conway, 2014; Gosser et al.,

2010; Gupta et al., 2015; Hall et al., 2014; Lewis & Lewis, 2005; Ruder & Hunnicutt, 2008;

Smith, Wilson, Banks, Zhu, & Varma-Nelson, 2014; Tien et al., 2002).

Student-centered teaching practices aligned with constructivism have also been more

broadly adopted by educators in both face-to-face and online classrooms (Duffy & Cunningham,

1996; Hyslop-Margison & Strobel, 2008; Partlow & Gibbs, 2003; Phillips, 1995; Vrasidas,

2000). An instrument that can be used to measure student perceptions of the presence of

constructivist teaching practices through the Community of Inquiry (CoI) model has been

developed by researchers in online education (Arbaugh et al., 2008; D. R. Garrison et al., 2010;

Swan, Garrison, & Richardson, 2009). Additionally, an instrument has been developed to

determine the degree to which an instructor has adopted a student-centered approach to teaching

based on self-reported frequencies of utilizing various classroom practices (Prosser & Trigwell,

28

2006; Trigwell et al., 2005; Trigwell & Prosser, 2004). The combination of these two

instruments allows for the identification of a constructivist learning environment utilizing both

instructor and student perspectives. The prevalence of constructivism in educational research and

practice across disciplines and delivery methods speaks to its acceptance among instructors as a

useful model of how students learn.

Philosophical and Psychological Foundations of Constructivism Contemporary writer Ernst von Glasersfeld traced constructivist beliefs back to eighteenth

century philosophers Giambattista Vico and Immanuel Kant. As understood by von Glasersfeld,

some of the earliest evidence for a belief that individuals construct knowledge as a result of

experiences comes from Vico’s writing that “human truth is what man comes to know as he

builds it” (as cited in von Glasersfeld, 1981/1984, p. 7). While von Glasersfeld adopted Vico’s

epistemology, he also adopted Kant’s ontological perspective on the nature of reality. This

perspective rejects metaphysical realism and the belief that knowledge represents objective

reality experienced in the same way by all people. To von Glasersfeld, knowledge is not a

reflection of objective reality but instead knowledge represents the truth as constructed to fit

experiences. Specifically, von Glasersfeld’s (1981/1984) interpretation is that Kant believed “our

mind does not derive laws from nature, but imposes them [laws] on it [nature]” (p. 3).

Von Glasersfeld was one of the first to create an accessible interpretation of constructivist

philosophies. He considered his interpretation of constructivism as radical because it “breaks

with convention” (1981/1984, p. 5) of knowledge as reflecting objective reality. This radical

interpretation of constructivism led to many concerns regarding the adoption of constructivism in

science education. Of particular concern to scientists were von Glasersfeld’s beliefs that “science

29

(1) cannot reveal ‘objective truth,’ (2) is forever fallible, and (3) is not the most important thing

in the field of human experience” (1993, p. 37). This interpretation of constructivism was

strongly criticized by Suchting (1992), and others who have not adopted the more radical aspects

of von Glasersfeld’s philosophy of constructivism. However, the version of constructivism

brought to the chemical education community by Bodner (1986) and Tobin (1999) was heavily

influenced by von Glasersfeld’s radical definition of constructivism (1989, 1993).

Despite the radical version of constructivism introduced to the chemical education

community, Staver (1998), a researcher and former high school chemistry teacher, believed it

was possible to adopt constructivist beliefs about how knowledge is constructed without

rejecting the ability of constructed knowledge to match the objective reality. He expressed this

belief saying “many constructivists, including myself, choose to remain silent on the issue of

knowledge as a correspondence with the facts of reality” (p. 505). This position is described by

Wink (2014) who believes that most chemical educators either implicitly or passively reject the

radical constructivism of von Glasersfeld and Bodner. The silence of chemical educators on the

philosophical aspects of constructivism may also be a result of debates in the Journal of

Chemical Education about the incompatibility of radical constructivism and commonly held

beliefs about science as a search for objective knowledge (Bernal, 2006; Scerri, 2003).

Consequently, it appears that at least within the chemical education community the acceptance of

constructivism does not require a belief about what knowledge represents but only an acceptance

that knowledge is the result of an individual interpreting and making sense of experiences in

order to construct understanding.

30

Though von Glasersfeld recognized influences from Vico and Kant, he primarily credited

Swiss developmental psychologist and philosopher Jean Piaget’s research on the cognitive

development of children as the foundation of constructivism. Piaget’s study of how knowledge

develops overlapped with the fields of psychology and philosophy (Bodner, 1986; von

Glasersfeld, 1974). Piaget considered himself a genetic epistemologist, and described his

constructivist philosophy by explaining “for the genetic epistemologist, knowledge results from

continuous construction” (as cited in von Glasersfeld, 1974, p. 8).

Piaget primarily focused on the role of the individual, while others investigated the role of

social interactions in knowledge construction. John Dewey, an American educational reformer,

wrote in the early twentieth century that “meanings do not come into being without language,

and language implies two selves [e.g., teacher and student] involved in a conjoint or shared

understanding” (as cited in Garrison, 1995, p. 722). In this way, Dewey’s educational philosophy

emphasized the role of language and social interactions in constructing knowledge. Similar

beliefs influenced the research of Lev Vygotsky, a Soviet psychologist, on the role of language

and social interactions in how children develop (1978). The emphasis of Dewey and Vygotsky

on the role of social interactions in advancing cognitive development has become the basis of

social constructivism in education. Thus, the combination of philosophical beliefs and

psychological research provided support for the transition of constructivism from a philosophical

belief to a model of learning.

Constructivism as a Model of Learning

The constructivist emphasis on learning as individual knowledge construction contrasts

with earlier behaviorist models of learning based on stimulus-response experiments that placed

31

the emphasis for learning on conditions external to the learner (Good, Wandersee, & St. Julien,

1993; Jonassen, 1991). Teaching approaches aligned with a behaviorist model of learning were

useful for memorization of facts and training of specific behaviors, but were not successful in

developing more advanced conceptual understandings desired by many educators (Abraham,

2008; Fosnot & Perry, 2005; Matthews, 1993). Constructivist educational models instead

focused on describing the internal cognitive processes that lead to knowledge construction.

Piaget described his model of learning as simply adding two steps to the stimulus-response

model “it is indeed a stimulus-response theory, if you will, but first you add operations and then

you add equilibration” (1964/1997, p. 27). Both operations and equilibration are internal

processes carried out by each individual while learning is occurring. An operation is “an

interiorised [sic] action which modifies the object of knowledge” (p. 20) and equilibration is “a

process of self-regulation”. Piaget described one type of operation as the incorporation of new

knowledge into existing cognitive structures, which he called assimilation. For Piaget, “learning

is possible only when there is active assimilation. It is this activity on the part of the subject

which seems to me underplayed in the stimulus-response schema” (1964/1997, p. 26). According

to Piaget, in situations where new knowledge cannot be fit into existing cognitive structures, a

change in cognitive structures is required. He called the adjustment of cognitive structures to fit

the new knowledge accommodation. Assimilation and accommodation are expressions of

constructivist principles because they describe knowledge in terms of cognitive building

processes. While current educational theory has moved beyond Piaget’s theory that children

progress in a defined way through four stages of cognitive development (Bunce, 2001; Staver,

1998), his theory of the cognitive processes an individual uses to construct knowledge has

32

become a key element of constructivism applied as a model of learning. Piaget’s work provided a

foundation for research to identify factors that influence the ability of an individual to construct

knowledge.

Educational psychologist David Ausubel (1960) studied the role existing cognitive

structures play in constructing new cognitive structures. According to Ausubel’s subsumption

theory, new information can only become part of an existing cognitive structure if the new

information is relevant to the existing structure. This theory is closely linked to Piaget’s concept

of assimilation, but specifies that the existing structures must be tied to the incoming information

in some way so that the new information can be classified under the existing structures. This led

to Ausubel’s belief that “the most important single factor influencing learning is what the learner

already knows. Ascertain this and teach him accordingly” (as cited in Abraham, 2008, p. 56).

Ausubel’s emphasis on supporting the individual cognitive development of students closely

aligns his research with constructivism.

Cognitive scientist Donald Norman (1980) echoes Ausubel’s belief that learning is an

individual process influenced by the preexisting cognitive structures of the learner. Norman had

initially advocated a “web learning” model (1973) in which the role of the teacher was simply to

make as many connections as possible between pieces of knowledge in order to help the student

develop a robust and redundant network of knowledge that could be accessed in numerous ways.

However, Norman later acknowledged that this theory was too simplistic because it treated the

student as a “passive receptacle” (1980, p. 42). Norman’s revised understanding of knowledge

acquisition falls more in line with Piaget’s idea of active assimilation.

Although I do not yet understand the specific way by which new knowledge is acquired, it does involve active interpretation on the part of the student. The student comes to the

33

learning situation with a large set of preexisting ideas, and the material that is presented is interpreted according to those ideas. You cannot prevent it: I have tried…It is the student who decides what aspects of the material are important, what aspects are not. (1980, pp. 42–43)

Though the work of Piaget, Ausubel, and Norman emphasized the construction of

knowledge within the mind of each individual, this construction is often the result of interactions

with other individuals such as teachers or peers. The role of these interactions is emphasized in

the form of constructivism known as social constructivism. Vygotsky’s research on the

development of knowledge focused on social factors that influence learning and development but

still emphasized the internal nature of the process. Vygotsky employs the analogy of using an x-

ray to “reveal to the teacher how developmental processes stimulated by the course of school

learning are carried through inside the head of each individual child” (1978, p. 91). Social

interactions are important to Vygotsky’s zone of proximal development (ZPD), which he

described as “the distance between the actual developmental level as determined by individual

problem solving and the level of potential development as determined through problem solving

under adult guidance or in collaboration with more capable peers” (1978, p. 86). A critical

component of the ZPD is the presence of someone with more advanced knowledge, either a

teacher or peer. This implies that collaborative groups cannot be successful at improving

problem-solving abilities if all the group members are at the same developmental level or if an

expert is not available to provide guidance. The presence of the expert teacher is especially

important for the development of the most advanced group member since this individual is not in

the company of more capable peers.

The constructivist model of learning helps explain which teaching practices, such as

identifying preexisting cognitive structures of students and well-structured group work, should

34

improve knowledge construction in individuals. This model also unites key aspects of research

on the development of cognitive structures. The constructivist model of learning provides simple

analogies for cognitive processes that are easy to understand without delving into specifics of

neurons and other biological functions related to memory and learning. The analogy of

constructing or building up knowledge is especially meaningful in disciplines such as chemistry

that are typically taught as a progression of cumulative facts and concepts starting from atoms

and gradually increasing in complexity to discuss properties of materials at the macroscopic level

and eventually complex systems of reactions as in biochemistry or chemical engineering.

Scientists in general, and chemists in particular, should be familiar with the use of models

due to their importance in teaching phenomena such as atomic structure. An analogy can be

made between the use of the Bohr model of the atom to understand electron energy levels and

the use of constructivism as a model to understand how learning occurs. The Bohr model of the

atom provides a way to understand the position and color of the lines in the hydrogen spectrum

even though the electrons causing the spectral lines cannot be directly observed. Similarly,

constructivism provides a way to understand the success of various student-centered pedagogies

even though the knowledge structures of students cannot be directly observed. However, the

nature of models is that they are “simplified representations of phenomena or ideas” (Coll,

France, & Taylor, 2005, p. 185). In this way, models are imperfect and cannot accurately

represent all aspects of a phenomena or idea simultaneously. In the same way that the Bohr

model of the atom is not appropriate for more complex multi-electron atoms, the constructivist

model of learning becomes strained when asked to function as a philosophy, model of learning,

and pedagogy simultaneously.

35

Constructivism in Education

Von Glasersfeld (1989) realized that “by supplying a theoretical foundation that seems

compatible with what has worked in the past, constructivism may provide the thousands of less

intuitive educators an accessible way to improve their methods of instruction” (p. 138). While

constructivism can be used as a foundation from which to develop teaching practices aligned

with the constructivist model of learning, constructivism is not a theory of teaching (Committee

on Developments in the Science of Learning, 2000; Windschitl, 2002). The constructivist model

of learning developed from philosophical debate and psychological research speaks only to the

internal processes at work when an individual constructs knowledge. In some sense the phrase

“constructivist teaching practice” is an oxymoron because it is impossible for any teacher, no

matter how skilled, to implant knowledge directly in the mind of the student (Norman, 1980).

Though the constructivist model of learning does not prescribe specific teaching practices,

there has been a general acceptance of constructivism as an appropriate foundation for

developing new pedagogies. However, the development of teaching practices aligned with

constructivism is complicated by the many types of constructivism that can be found in the

literature. These are often referred to as “faces” (Good et al., 1993; Phillips, 1995) or “forms”

(Bodner, Klobuchar, & Geelan, 2001) of constructivism and typically share a core belief that

knowledge is constructed by individuals, but differ in their underlying ontologies or

epistemologies. This has led to vigorous debate and some misconceptions surrounding the

implications of constructivism for educational practice. Phillips (1995) believes that with the

acceptance of constructivism, “a weak or at least a controversial epistemology has become the

basis for a strong pedagogical policy” (p. 11). Fox (2001) takes a more cynical stance on the

36

acceptance of constructivism warning “it is in danger of becoming a general term of approbation

with but little content and an incoherent underlying epistemology” (p. 23).

The epistemological issues identified by Fox (2001) stem from simplistic interpretations of

some of the basic principles upon which all constructivists can agree. These misinterpretations

may be the result of interpretations and reinterpretations of constructivism by different authors in

order to make constructivism more accessible for classroom teachers. As a result, constructivism

may have become “little more than an educational slogan in the absence of conceptual

understanding and clarification” (Hyslop-Margison & Strobel, 2008, p. 73). For example, the

constructivist model of learning holds that all knowledge construction is an individual process,

yet the role of social interactions is frequently cited as critical for knowledge development. At

the extremes, these beliefs would appear to be incompatible and in opposition to one another:

either learning happens in complete isolation or it happens in the presence of others.

To borrow an example from chemistry, a parallel can be drawn to the simplistic way in

which students are sometimes introduced to the idea of ionic and covalent bonding. Ionic

bonding is often introduced as a bond that forms when electrons are lost by one atom and gained

by another. Covalent bonding is then taught as a completely separate type of bonding in which

electrons are shared between two atoms. This presentation sets up a false dichotomy for students

and leads to confusion later when the more complex topic of electronegativity is introduced. In

discussing electronegativity, atoms are described as having differing levels of attraction to the

electrons comprising the bond. When different combinations of atoms form a bond, the possible

types of bonds that can form range from completely ionic to completely covalent. Additionally, a

middle ground is introduced in which the bond cannot be described as purely ionic or purely

37

covalent. This middle ground of unequally shared electrons allows for a richer understanding of

bonding and is critically important to understanding the intermolecular forces that exist between

various atoms, ions, and molecules.

In the same way, a false dichotomy is presented in simplifying constructivism into

opposing camps of individual and social knowledge construction. A combination of these two

ends of the spectrum can be described as an individual constructing his or her own knowledge

while interacting with others. This conceptual middle ground acknowledges the role of both the

individual and his or her social environment. Recognizing that both the individual and the social

environment play a role in knowledge construction opens the door to a richer understanding of

how an effective learning environment can be designed. Many misconceptions about

constructivism stem from incomplete or simplified understandings of how constructivism should

influence educational practice.

Considering learning to be an entirely individual process leads to misconceptions that

constructivist teaching must allow students to discover all knowledge for themselves, thus

prohibiting the teacher from ever directly instructing students. A more correct interpretation is

that all knowledge construction is an individual process since it occurs within the mind of each

individual but this process can occur in any number of situations. Often these situations are

assumed to require the students to be physically active, but this is a misinterpretation of Piaget’s

“active assimilation” (1964/1997, p. 26). The activity described by Piaget is the active process of

knowledge organization that occurs anytime a student is mentally engaging with the material. As

described by Bächtold (2013), “knowledge construction implies activity of the mind but not

necessarily activity of the body” (p. 2478). The long historical tradition of education clearly

38

demonstrates that learning can occur while listening to a lecture or reading a textbook

(Committee on Developments in the Science of Learning, 2000), as long as the student is

mentally engaged in the process of knowledge construction.

Another misconception surrounding constructivism is that by focusing on how each

individual student constructs knowledge, the role of the teacher is deemphasized or even ignored

(Bodner et al., 2001). This concern is predicated on an incomplete understanding of

constructivism. In courses designed using the constructivist model of learning, the role of the

teacher becomes less about disseminating information and more about facilitating the

construction of knowledge in individual students. This new, but critically important role of the

teacher is clearly articulated by Piaget:

It is obvious that the teacher as organizer remains indispensable in order to create the situations and construct the initial devices which present useful problems to the child. Secondly, he is needed to provide counter-examples that compel reflection and reconsideration of over-hasty solutions. What is desired is that the teacher cease being a lecturer, satisfied with transmitting ready-made solutions; his role should rather be that of a mentor stimulating initiative and research. (1973, p. 16)

Although Coll & Taylor (2001) noted a concern that constructivism undermines the expert status

of the teacher, this is likely to be true only in the case where the teacher is regarded as an expert

due solely to his or her ability disseminate facts with no consideration for how to best help

students learn. However, Piaget’s belief that “the teacher-organizer should know not only his

own science, but also be well versed in the details of the development of the child’s or

adolescent’s mind” (1973, pp. 16-17) signifies that both content and pedagogical knowledge are

equally important. In this way, the expert status of the teacher is heightened because content

knowledge expertise must be paired with expertise in the ability to design and manage learning

environments that will encourage knowledge construction.

39

The ontological confusion regarding the role of constructivism in education seems to stem

primarily from von Glasersfeld’s radical interpretation. Here the misinterpretation is that if

knowledge is constructed by each student to make sense of his or her experiences then all student

ideas be must considered as equally valid since they represent reality for that student (Bodner et

al., 2001; Committee on Developments in the Science of Learning, 2000; Windschitl, 2002). For

science educators in particular this misinterpretation led to an erroneous belief that students

should be allowed to hold misconceptions that appear supported by everyday experience, such as

objects requiring a constant force to stay in motion or a belief in pseudoscience such as

intelligent design (Matthews, 1993; Mugaloglu, 2014). These interpretations by Matthews and

Mugaloglu do reflect von Glasersfeld’s radical constructivist beliefs in rejecting the idea of

knowledge as reflecting an objective truth and emphasizing knowledge as construction of reality

for that individual. However, these interpretations miss a key point made by von Glasersfeld

(1984) and summarized by Bodner et al. that “knowledge is no longer true or false; it either

works or it does not” (2001, p. 5). Von Glasersfeld and Bodner’s interpretation of radical

constructivism focuses on the viability of knowledge rather than its correspondence with an

objective reality.

Regardless of whether knowledge reflects an objective reality, constructivism requires

knowledge to be validated by relevant experiences. In this way, it is similar to how scientific

knowledge must be validated by relevant experimentation and evidence. In the context of social

constructivism, knowledge can be validated through group consensus. In considering

misconceptions or pseudoscientific beliefs held by students, there is no foundation for the

knowledge of the student to supersede the knowledge of the teacher. This is especially true

40

because the teacher is responsible for developing curriculum to test knowledge and facilitate

discussions that lead to understandings. A teacher can use the existing knowledge of students,

including misconceptions and pseudoscientific beliefs, to design classroom activities that provide

evidence contradicting the existing knowledge. As students have their knowledge tested and

found to be unsatisfactory, it is hypothesized that they will adjust their cognitive structures to

align with the new evidence and dismiss misconceptions and pseudoscientific beliefs. To this

end, one of the first uses of constructivism in chemical education was as a theoretical framework

for identifying and addressing student misconceptions (Bodner et al., 2001).

Another issue that can arise from a simplified view of constructivism is interpreting social

constructivism to imply that any type of group work is beneficial to learning. This can occur

when the role of Vytogsky’s ZPD is not fully understood. If the teacher understands that students

are placed in groups not because social interaction itself improves learning but because social

interactions allows a transition from one developmental level to another in the presence of a

more capable peer or adult, the teacher is likely to recognize the importance of monitoring and

providing guidance to the groups. If the current group of peers cannot provide an opportunity for

all students to advance developmentally, then groups may need to be periodically rearranged or

the teacher may need to provide additional support for more advanced students.

Though constructivism itself is a model of learning not a theory of teaching, it provides a

framework through which teaching practices can be analyzed to determine if they are aligned

with the constructivist model of learning. As von Glasersfeld (1989) made clear, teaching

practices aligned with constructivism are not new and many have been successfully implemented

in the past before constructivism became important in education. In this way, constructivism

41

does not necessarily represent a new way to teach, but rather a new way to understand which

teaching practices should be effective based on their alignment with a constructivist model of

learning. These teaching practices are typically student-centered and shift the classroom focus

away from a teacher who is dispensing information. Examples of teaching practices aligned with

constructivism include the teacher (1) acting as a facilitator of learning, (2) identifying existing

student cognitive structures in order to make the connections between existing knowledge and

new knowledge more explicit, (3) creating authentic problem solving tasks, (4) fostering active

student involvement in problem solving, (5) supporting students working and communicating in

groups to socially construct understanding, (6) encouraging discussion of and reflection on the

learning process, and (7) assessing more than arriving at a correct answer (Duffy & Cunningham,

1996; Hyslop-Margison & Strobel, 2008; Piaget, 1973; von Glasersfeld, 1989; Windschitl,

2002). By examining which teaching practices are more or less effective for various types of

students and the alignment of these teaching practices with a constructivist model of learning, the

constructivist model of learning can continue to be tested and refined.

Chemical Education

The chemical education community has a long history of utilizing current learning theories

to inform research and practice. Starting with the learning theories of Piaget and Ausubel

(Abraham, 2008; Herron, 1975; Novak, 1984; Nurrenbern, 2001), chemical educators have

gradually adopted constructivism as a foundation for both teaching and research. As previously

discussed, the version of constructivism brought to the chemical education community by

Bodner (2001; 1986) and Tobin (1999) was heavily influenced by von Glasersfeld’s radical

definition of constructivism (1989, 1993) and was met with some criticism by scientists

42

uncomfortable with the rejection of science as a process of discovering objective truths about the

natural world.

Perhaps unsurprisingly there is little evidence of radical constructivism in the current way

constructivism in used as a framework for teaching and research by chemical educators. Yet, a

Google Scholar search conducted in April 2016 listed over 1000 combined citations of Bodner’s

1986 and Bodner et al.’s 2001 constructivism articles, which interpreted von Glasersfeld for

chemical educators. Wink (2014) provides one possible explanation for the omission of radical

constructivism from chemical education by proposing a separation of the pedagogical and

philosophical components of constructivism. Instead of continuing to debate the philosophical

components of constructivism, chemical educators appear to have instead embraced

constructivism as a model of learning and focused on the pedagogical implications of

constructivism by providing opportunities for students to construct their own knowledge in a

student-centered learning environment. Regardless of the debate about the nature of reality that

surrounds constructivism as a philosophy, constructivism has still provided a sound theoretical

framework around which to design effective new approaches to chemical education teaching and

research.

Two specific pedagogies in chemical education that emphasize the instructor as a facilitator

of learning in more student-centered environments are process-oriented guided-inquiry learning

(POGIL; Hanson, 2006, 2008) and peer-lead team learning (PLTL; Varma-Nelson & Banks,

2013; Varma-Nelson & Coppola, 2005). In both POGIL and PLTL, one or more weekly lectures

or recitation sections are replaced with workshop sessions in which groups of students work

together to construct an understanding of chemical concepts. In this way, both POGIL and PLTL

43

incorporate teaching practices aligned with a constructivist model of learning. Evidence for

improved student learning outcomes as a result of utilizing POGIL and PLTL will be discussed

later in the chapter more detail.

Group work is a key component in POGIL and PLTL classrooms and aligns both

approaches with social constructivism (Abraham, 2008; Smith et al., 2014). The foundations of

PLTL include working cooperatively in teams to construct knowledge through discussion and

debate (Varma-Nelson & Coppola, 2005). This aligns with Tobin’s (1999) description of

constructivism as encouraging “learning from diverse sources and providing social contexts in

which individuals can make sense and test their emerging understandings” (p. 238). For POGIL,

the role of social constructivism is not explicitly stated, but can be inferred from

recommendations highlighting the importance of a “diversity of perspective and skills that

produces a rich exchange of ideas” (Hanson, 2006, p. 22). These diverse skills allow students to

help each other progress through Vygotsky’s ZPD under the guidance of someone functioning at

a more advanced level.

During group work, POGIL activities utilize a guided inquiry learning cycle. Student

groups explore a topic, then formulate a conceptual understanding, and eventually apply this

understanding in a new context. This cycle invokes constructivist principles of first constructing

knowledge to make sense of experiences then testing the viability of that knowledge. In PLTL,

less emphasis is placed on a specific learning cycle. However, it is critical that PLTL workshop

materials are “challenging, intended to encourage active learning and to work with groups”

(Varma-Nelson & Coppola, 2005, p. 8). Vygotsky’s ZPD is used to explain how the challenging

materials should be just beyond the immediate ability level of the students. Without

44

appropriately designed materials, there is no reason for students to work together in groups to

solve problems because the problems will be easy enough to solve individually (Varma-Nelson

& Coppola, 2005, p. 5). To avoid this, both PLTL and POGIL approaches warn against using

“drill” or “plug and chug” type problems that encourage memorization and application of

algorithms and instead encourage problems emphasizing application of concepts or synthesis of

new ideas.

Online Education

The philosophical debate surrounding constructivism is much less evident in online

education. It may be that online educators from disciplines with less empirical foundations are

more comfortable with understanding each person to have a conception of reality shaped by his

or her own experiences. Or, like chemical educators, online educators may also have separated

the philosophical and pedagogical components of constructivism. Then, instead of having a

philosophical debate, online educators instead focus on using a constructivist model of learning

to inform pedagogy.

For online educators, constructivism is a useful framework for engaging students and

fostering the development of knowledge (Vrasidas, 2000). Since much of the research in online

education is concerned with keeping students engaged and learning without the constant

presence of a teacher, it is not surprising that “constructivist models of learning are almost

exclusively recommended as a guide for the design and delivery of Internet-based courses”

(Bangert, 2008, p. 28). The first cohesive model of online learning came from the Community of

Inquiry (CoI) model. The CoI model was developed from analysis of computer-conferencing

transcripts (D. R. Garrison et al., 2000) using a grounded theory approach of working from data

45

to develop a theory (Creswell, 2013). From this analysis, three types of presence emerged as the

foundation of the CoI model: cognitive presence, social presence, and teaching presence. The

influence of constructivism can be seen in each type of presence.

Cognitive presence is defined as “the extent to which the participants in any particular

configuration of a community of inquiry are able to construct meaning through sustained

communication” (D. R. Garrison et al., 2000, p. 89). Here the idea of knowledge construction by

constructing meaning is incorporated directly into the definition of cognitive presence. This

indicates that cognitive presence is not simply a matter of providing content to students but

rather is related to the degree to which the content fosters mental activity on the part of the

students. This definition also highlights the social nature of the CoI model since the construction

process is thought to result from sustained communication. In this way, the CoI model does not

isolate cognitive presence from social presence and teaching presence. The role of both the other

students in the course and the instructor is to provide someone to communicate with in order to

engage in the process of socially constructing meaning.

The link between cognitive presence and social presence is further solidified by the idea

that “cognitive presence…is more easily sustained when a significant degree of social presence

has been established” (D. R. Garrison et al., 2000, p. 95). This can be interpreted as a restatement

of social constructivism in the sense that knowledge construction benefits from social

interactions due to the role of social interactions in “indirectly facilitating the process of critical

thinking” (D. R. Garrison et al., 2000, p. 89). The CoI definition of social presence as “the ability

of participants in a community of inquiry to project themselves socially and emotionally, as

‘real’ people (i.e., their full personality)” (D. R. Garrison et al., 2000, p. 94) indicates that social

46

presence is only linked to a successful educational experience when it supports affective

outcomes as well as cognitive outcomes. These affective outcomes such as finding “the

interaction in the group enjoyable and personally fulfilling” (D. R. Garrison et al., 2000, p. 89)

are described as important for keeping students engaged and enrolled in the online course.

The final aspect of the CoI model, teaching presence, describes the responsibility of the

instructor to establish a learning environment that supports the development of both social and

cognitive presence. The role of the instructor as a facilitator of learning invokes the student-

centered emphasis of constructivism. The overlap of teaching presence and cognitive presence is

necessary in order to select content that will encourage students to follow the inquiry cycle, and

the overlap of teaching presence and social presence is necessary to set a course climate that will

foster the development of a learning community. According to the CoI model, deep learning

results from the overlap of all three types of presence. Additionally, each type of presence

manifests in specific activities or behaviors that serve as indicators of a student-centered

classroom. In this way, the CoI model provides information about the types of activities that are

hypothesized to improve learning when undertaken by the instructor and students. A visual

description of the CoI model can be seen in Figure 3 in which the three presence factors are

represented by overlapping circles and the indicators are represented by arrows pointing to each

presence factor. These indicators will be discussed in more detail later as they relate to the

development of the student survey instrument designed to measure the extent to which a learning

environment is aligned with the CoI model.

47

Figure 3. The Community of Inquiry model. Indicators of each type of presence are given in the arrows. Adapted from D. R. Garrison et al. (2000) and Swan (2003). As with POGIL in chemical education, the social component of the CoI model is used to

support inquiry-based learning activities in online courses. In the CoI model, the inquiry cycle

has four steps: a triggering event, exploration, integration, and resolution or application. The

steps in the inquiry cycle are joined by dashed arrows in Figure 3. While POGIL and CoI share

foundations in social constructivism and inquiry, an important distinction between the two is that

POGIL represents a specific pedagogy developed from educational theories while the CoI is a

model developed to explain data. Although the CoI model was developed using data from online

48

courses, it encourages the adoption of teaching practices aligned with a constructivist model of

learning and should be more broadly applicable to other disciplines and delivery methods

including chemistry courses delivered in a face-to-face environment.

Measuring a Constructivist Learning Environment and Student Outcomes

Defining Constructivism

In spite of Tobin’s (1999) declaration of “moving on” from constructivism over fifteen

years ago, constructivism has continued to find acceptance as a framework for chemical

education research and teaching. Ferguson (2007) identified multiple studies utilizing

constructivism as a framework to research teaching strategies and student conceptions of

chemical concepts. Articles published in the Journal of Chemical Education as recently as 2015

either reference writings on constructivism, or have constructivism as a keyword (DeFever,

Bruce, & Bhattacharyya, 2015; Flynn & Ogilvie, 2015; Gupta et al., 2015; Stoyanovich, Gandhi,

& Flynn, 2015; Talanquer, 2015). However, the level of understanding and application of

constructivism is inconsistent across these five articles.

Only two articles explicitly discuss social constructivism in teaching (Gupta et al., 2015)

and radical constructivism as a theoretical framework for research (DeFever et al., 2015).

Talanquer (2015) describes the need to consider existing student cognitive structures before

teaching new concepts that would cause those structures to change, aligning his view of

knowledge development with constructivist principles. Yet, Talanquer does not make the

connection between his beliefs and constructivism, even though he references a book chapter

titled Constructivism and Troublesome Knowledge. The other two articles with constructivism as

a keyword abstain from discussing constructivism in the article’s text (Flynn & Ogilvie, 2015;

49

Stoyanovich et al., 2015), though Stoyanovich et al. (2015) use Vygotsky’s ZPD to rationalize

the order in which to teach acid-base concepts, which provides a tangential link to social

constructivism. These recent articles highlight the inconsistent application or reporting of

constructivism in chemical education, even in situations where authors appear to hold views

consistent with constructivist principles.

One possible interpretation of these publications is that constructivism does not have a

common meaning for all chemical educators. In considering the role of constructivism in

educational practice, there is a need to define what is meant by “constructivism” in order to

provide a clear foundation for measurements of constructivist learning environments. In

subsequent discussions of constructivism as applied to pedagogy, constructivism will be used to

describe an approach to teaching in which teaching practices are aligned with a constructivist

model of learning. That is not to say that constructivist classrooms never utilize lectures,

memorization, or independent work. Instead, a constructivist learning environment will be

defined here as one in which teaching practices have been adopted that are more student-centered

and have shifted the role of the instructor from a lecturer to a facilitator for at least part of the

instructional time. This definition of constructivism is narrow in that it explicitly avoids much of

the philosophical debate surrounding radical interpretations of constructivism and instead

emphasizes commonalities among the various types of constructivism that can be directly

applied to educational practice.

With a clear definition of a constructivist learning environment established, it is necessary

to determine how to measure the degree to which a learning environment can be considered

constructivist. One way to determine the degree to which student-centered teaching practices are

50

being used is to employ observational protocols such as the Reformed Teaching Observation

Protocol (RTOP) or the Classroom Observation Protocol for Undergraduate STEM (COPUS;

Lund et al., 2015; Stains et al., 2015). However, these protocols are labor intensive and typically

the external observer only watches a few class sessions. Instead, the perceptions of the two

classroom stakeholders, the students and the instructor, may provide a richer description of the

learning environment since the stakeholders experience the learning environment over the whole

semester.

Development of the Community of Inquiry Student Survey Instrument

Student perceptions are critical to understanding the degree to which an instructor has

created a constructivist learning environment because students are the intended target of the

instructional techniques employed by the instructor. Additionally, students directly experience

the learning environment over an extended period of time and are therefore able to provide a

more complete description than an observer who may not attend all class sessions or fully engage

in activities and assignments. However, judging by the inconsistent way in which researchers

and instructors understand constructivism, it is highly unlikely that undergraduate students in

their first or second year would be able to speak directly about constructivism in their learning

environments. Therefore, it makes more sense to ask students about indicators of a constructivist

learning environment, not constructivism itself.

A survey was developed by Bangert (2008) in which students were asked to rate the

presence of indicators of an online constructivist learning environment. These indicators

included the development of a learning community fostering interaction and thoughtful

discussion among students and the level of comfort a student had interacting with other students

51

and the course instructor. Bangert’s (2008) survey was an initial step towards developing a

student survey to measure a constructivist learning environment using the Community of Inquiry

(CoI) framework. Bangert developed and validated his survey instrument using both exploratory

factory analysis (EFA) and confirmatory factor analysis (CFA).

Factor analysis techniques differ from other statistical techniques such as the t-test or

ANOVA. The goal of most factor analysis techniques is to identify underlying unmeasured

variables called latent variables or factors which may be responsible for observed patterns of

correlations in the data. This contrasts with t-tests and ANOVAs that look for differences

between group means. However, there are factor analysis techniques such as structured means

modeling that can be used to look for differences in group means on latent variables (Hancock,

1997). The goal of EFA is to identify underlying factors while the goal of CFA is to confirm the

presence of factors hypothesized by the researcher prior to the analysis. Using EFA (n=404),

Bangert was able to show that 23 of the original 26 items on his survey had mathematical

relationships to four latent factors related to student evaluations of online teaching effectiveness

including: (1) student-faculty interaction, (2) cooperation among students, (3) active learning,

and (4) time on task.

After EFA, Bangert (2008) used a new sample (n=403) for a CFA with the 23 items that

had previously been shown to be related to the four identified factors. The purpose of the CFA is

to provide a statistical description of how well the hypothesized model fits the provided data. A

variety of fit indices are used to evaluate data-model fit. The indices used by Bangert include the

root mean square error of approximation (RMSEA) and the comparative fit index (CFI), but the

model !" was not provided. Both the RMSEA and CFI values can range from 0 to 1. The

52

RMSEA is a parsimonious fit index that describes how well a model fits the data while taking

the simplicity of the model into account. Acceptable RMSEA values are below 0.06 (Hu &

Bentler, 1999) and indicate that the model explains a relatively large portion of the variance and

covariance among variables while also minimizing the number of relationships among variables.

The CFI is an incremental fit index that describes how much better the hypothesized model is

compared to a null model with no relationships among variables. Larger CFI values indicate that

the model explains a large amount of variance and covariance beyond what the null model

explains while smaller CFI values indicate that either the model is poor or that only weak

relationships are present in the data. Bangert’s CFA model had a RMSEA of 0.042 with the 90%

confidence interval (0.038 to 0.047) remaining below the cutoff of 0.06 and CFI equal to 0.99,

suggesting that the model with four latent factors was a good fit for the data.

Items similar to those used in Bangert’s (2008) study were utilized in developing a survey

specifically to measure the latent factors of cognitive presence, social presence, and teaching

presence in the CoI model (Arbaugh, 2008; Arbaugh et al., 2008). The items used in these

studies are provided in Appendix A. The indicators for each type of presence, previously seen in

Figure 3 (p. 47), were operationalized to create items designed to address the various aspects of

each of the three underlying presence factors. The four indicators for cognitive presence are

related to the use of the inquiry cycle in learning activities. The first stage of the inquiry cycle is

a triggering event. This was operationalized as items asking about the degree to which the

students experienced curiosity or interest related to course activities and problems. The second

stage, exploration, was measured by statements related to the students’ motivation to explore

questions posed in class by utilizing a variety of information sources. In the third stage of the

53

inquiry cycle, integration, students are asked about the degree to which they were able to

combine and reflect on the information obtained in the exploration stage. Items addressing the

final stage, resolution, described the ability of students to construct explanations and apply them

to solve problems in the course or in their daily lives.

The three indicators of social presence identified by the CoI model are the ability of the

students to express their emotions, engage in productive discourse with other students, and

collaborate. Productive discourse allows students to freely exchange ideas while collaboration

results in a cohesive group working together to solve a problem. The emotional aspect of social

presence is most prominent in the CoI survey items and is related to the affective domain in

which students feel they belong to the community of learners, are able to form impressions of

other students, and feel comfortable interacting and participating in discussions, including

disagreeing and acknowledging other viewpoints. Collaboration and discourse were

operationalized as items asking about the degree to which social interaction occurred during

communication and the development of a sense of collaboration through discussions. This

operationalization of social presence does not directly address the idea of group work or

collaborative assignments but rather treats social presence as a subtler sense of community

developed among students working towards a common goal.

Teaching presence items are most similar to those found on traditional student evaluation

forms. In the CoI model teaching presence has three indicators including instructional design,

facilitation, and feedback. Instructional design items cover traditional student expectations of an

instructor to communicate course topics, goals, due dates and instructions. Facilitation items are

more closely related to constructivism in asking about how well the instructor guided students

54

towards understanding, focused discussions, encouraged exploration, developed a sense of

community, fostered engagement, and kept students on task. Lastly, items related to instructor

feedback focused not on grades but rather on the ability of the instructor to help students identify

areas of strength and weakness and areas of agreement and disagreement related to course topics.

Early studies with the CoI survey instrument used principal component analysis (PCA) to

investigate the underlying structure of the instrument (Arbaugh, 2008; Arbaugh et al., 2008,

2010). In PCA, a large set of variables is reduced to a smaller set of new variables that explain as

much of the total variance in the original variables as possible. In PCA, these reduced variables

are more correctly called components, not factors, though the term factor is frequently used to

describe the results of PCA (Tabachnick & Fidell, 2007). PCA and factor analysis techniques can

produce mathematically similar results even though their corresponding algebraic operations on

the covariances of the set of variables are different (Velicer & Jackson, 1990). Conceptually, the

difference between PCA and factor analysis techniques such as CFA can be illustrated by

Figures 4 and 5, which show three measured variables, represented by boxes (V1, V2, & V3),

and their relationship to an underlying latent factor (or component), represented by an oval (F1).

Figure 4. A factor model with three measured variables and one latent factor.

Figure 5. A component model with three measured variables and one latent factor (component).

55

The primary difference between Figure 4 and Figure 5 is the directionality of the arrows, or

paths, between the measured variables and the factor or component; this directionally is the

critical conceptual difference between the two techniques. In the CFA model (Figure 4), the

arrows point from the factor to the variables because the factor is the underlying mechanism

thought to be causing the observed correlations, or covariance, among the measured variables. In

CFA the variables correlate, or covary, because they share variance in common with the factor.

In the PCA model (Figure 5), the arrows point from the variables to the factor (component)

because the variables are being combined in an optimal way to create the factor (component).

Another distinction between the two techniques is that in PCA the components are built to

explain all of the variance in the measured variables including variance due to measurement

error. In CFA the error is not included as part of the shared variance and is separated from the

measured variables and the factor, as seen in Figure 6. The benefit of CFA is that a factor is a

more pure representation of the underlying theoretical construct because it exists separately from

Figure 6. A factor model with three measured variables and one latent factor with the error terms shown for each measured variable.

the error. In PCA the component represents an optimally weighted combination of variables, not

any underlying theory, and is only as reliable as the variables that have been combined to form it.

56

The theoretical foundation of the CoI model suggests that factor analysis techniques are a

better match for instrument development than PCA since the three types of presence in the CoI

model are the underlying factors the survey items are attempting to measure. However, PCA can

be an appropriate technique in some situations because, unlike factor analysis, PCA allows for

the easy computation of summary or factor scores for each individual. These factor scores can be

calculated because the factor is a weighted composite of item scores. When PCA is used on the

CoI survey instrument, scores can be calculated for each survey respondent on cognitive, social,

and teaching presence. These scores can then be used in regression and group comparison

techniques as seen in the Arbaugh (2008) and Arbaugh et al. (2010) studies, respectively.

After the initial instrument development studies demonstrated the utility of the 34-item CoI

instrument in a variety of online course settings (Arbaugh, 2008; Arbaugh et al., 2008, 2010),

researchers began to look at relationships among the three CoI presence factors (cognitive,

social, and teaching), demographic variables such as gender, age, and academic level (Shea &

Bidjerano, 2009), and student outcome variables such as satisfaction and persistence (Joo et al.,

2011). Both studies (Joo et al., 2011; Shea & Bidjerano, 2009) used structural equation modeling

(SEM) to examine hypothesized causal relationships (paths) among the latent variables.

The structural equations in SEM describe the causal relationships among variables. Causal

relationships provide more specific information about the hypothesized influences of factors than

the correlations among factors seen in previous studies with the CoI survey instrument (Arbaugh

et al., 2008). Correlations between the three CoI presences were expected to exist due to their

overlapping nature in the CoI model, but the correlations could not provide information about

causal paths between factors. When using SEM, the relationships among measured variables and

57

factors must be specified a priori. Causal relationships should have a theoretical rationale for

either existing or being excluded from the model. As an example, the model typically proposed

for the 34-item CoI instrument and the three CoI presence factors is shown in Figure 7 (D. R.

Garrison et al., 2010; Shea & Bidjerano, 2009).

Figure 7. A model of hypothesized relationships among the 34 items on the CoI student survey and the three presence factors. Error terms have been omitted to minimize clutter in the model. Based on models in D. R. Garrison et al. (2010) and Shea & Bidjerano (2009).

This model can be described in terms of a measurement portion and a structural portion

(Mueller & Hancock, 2008). The measurement portion of the model shows the relationships

between the measured variables and the latent variables. In this case, the measurement portion of

the model shows how the 34 items on the CoI instrument are related to the three presence factors.

58

The structural portion of the model describes the causal relationships among the latent variables

of teaching presence, cognitive presence, and social presence. As seen in Figure 7, teaching

presence is hypothesized to have a causal influence on both social and cognitive presence and

social presence is hypothesized to have a causal influence on cognitive presence. Shea and

Bidjerano (2009) justify these relationships because the role of the instructor is to develop the

course environment which would cause teaching presence to influence both cognitive and social

presence. Additionally, Shea and Bidjerano believe that social presence acts as a mediator

between teaching presence and cognitive presence. The influence of social presence on cognitive

presence is also reinforced by the original description of cognitive presence as being supported

by social presence (D. R. Garrison et al., 2000). This hypothesized structure creates both a direct

effect of teaching presence on cognitive presence and an indirect effect of teaching presence on

cognitive presence through social presence.

In addition to finding evidence supporting causal relationships among the three CoI

presence factors, Shea and Bidjerano (2009) examined the causal influence of gender, age, and

academic level on student ratings of teaching presence. Their overall model contained the 34 CoI

survey items linked to the three presence factors, as shown in Figure 7, along with the three

additional measured variables of gender, age, and academic level. The fit statistics for the model

were acceptable, but not particularly good (!+,-1"3

" =11155.16; CFI = 0.95 and RMSEA = 0.08)

considering the customary threshold is at or above 0.95 for the CFI and at or below 0.06 for the

RMSEA. The large !" value is somewhat expected due to the large number of degrees of

freedom and the large sample size (n=2159). An absolute fit index called the standardized root

mean square residual (SRMR) was determined to be 0.05. This value is in line with

59

recommendations for SRMR to be at or below 0.08 (Hu & Bentler, 1999) and provides some

support for the authors’ claim of having acceptable data-model fit. Having acceptable data-model

fit allows for interpretation of the values determined for the paths between two variables. Only

the paths from gender to teaching presence and age to teaching presence were found to be

statistically significant at p < .05. Although statistically significant, the standardized paths, which

can be interpreted like regression coefficients, were very small for gender (0.04) and for age

(0.08). It is likely that these paths are statistically significant due to the large sample size utilized

in the research, but their practical significance is minimal within the overall CoI model. This

implies that gender and age have only a small influence on student perceptions of teaching

presence.

Also utilizing the CoI survey instrument, Joo et al. (2011) developed a model in which CoI

factors were hypothesized to have a causal bearing on student satisfaction and persistence in

courses at an online university in Korea (Shin, 2003). In this model, all three CoI presence

factors were hypothesized to have a direct influence on student satisfaction with the online

university as a whole. The structural portion of their model describing hypothesized relationships

among CoI presence factors and satisfaction is shown in Figure 8. A similar causal relationship

was hypothesized among teaching, social, and cognitive presence as in the Shea and Bidjerano

(2009) study. After initially testing their model with 709 students, Joo et al. (2011) found the

path between social presence and satisfaction to not be statistically significant. This

nonsignificant path is shown in Figure 9 with a lighter dashed arrow.

60

One reason the path from social presence to student satisfaction may have been found to be

nonsignificant is that the model may not have had enough power to detect a statistically

significant path. The power may have been low if the sample size was not appropriate for the

number of degrees of freedom in the model (Hancock, 2006). In SEM, degrees of freedom are

calculated by subtracting the number of parameters in the model from the number of unique

pieces of information provided by the measured variable variance/covariance matrix. The

degrees of freedom for this model (46) were relatively small because the researchers did not

allow individual CoI survey items to load on their respective factors. Instead, the 34 individual

CoI items and eight satisfaction items were reduced to two measured variables per factor through

item parceling, a technique which combines multiple measured variables into a composite by

taking either the average or sum.

Item parceling can be used to minimize the possibility of overweighting a latent factor that

has more paths to measured variables than other latent factors in the model (Little, Cunningham,

Shahar, & Widaman, 2002). Additionally, item parceling can reduce measurement error by

Figure 8. A model of hypothesized relationships among the three CoI presence factors and student satisfaction adapted from Joo et al. (2011).

Figure 9. A model of hypothesized relationships among the three CoI presence factors and student satisfaction indicating the nonsignificant path between social presence and satisfaction. Adapted from Joo et al. (2011).

61

reducing the number of variables in the analysis and increasing the likelihood that the

assumption of multivariate normality is met before running the statistical analysis. A drawback

of item parceling is that it decreases the degrees of freedom of the model by reducing the number

of unique pieces of information relative to the number of parameters. Reducing the number of

degrees of freedom can then reduce the power to detect a statistically significant path.

After finding the path between social presence and satisfaction to not be statistically

significant, the authors removed it from further analysis in order to improve their model fit.

However, no theoretical argument was presented to support the statistical argument for removal

of the path. Since each path in the model represents a belief structure based on a theoretical

understanding of the relationships among variables, the path should only have been removed for

a theoretical reason, not simply to improve model fit. Without a sound theoretical reason for

social presence not to influence student satisfaction, it is best to simply report the path as

nonsignificant and consider possible reasons for this outcome, such as having low power in the

analysis or the fact that the presence items were asked about a specific course and the satisfaction

items were related to satisfaction with the online university as a whole.

Through multiple studies, online education researchers have provided evidence for the

validity and reliability of the results obtained when using the CoI survey to measure student

perceptions of the three types of presence aligned with a constructivist learning environment

using diverse populations of online learners in a variety of disciplines (Arbaugh, 2008; Arbaugh

et al., 2008, 2010; Joo et al., 2011; Shea & Bidjerano, 2009). While the CoI survey instrument

has been used to measure three aspects of a constructivist learning environment for online

courses, there is little published research investigating the relationship among these three factors

62

and student outcomes beyond perceived learning or satisfaction (Arbaugh & Benbunan-Fich,

2007; Arbaugh, 2008; Joo et al., 2011). Currently, no online education research has investigated

relationships among the three presence factors thought to indicate a constructivist learning

environment and measured student outcomes of academic achievement as measured by grades or

standardized exam scores. This may be because comparing grades or locating appropriate

standardized exams is more difficult in online education research since the students surveyed are

typically from numerous disciplines and include both graduate and undergraduate students often

from different institutions (Arbaugh, 2008; Arbaugh et al., 2008, 2010). Additionally, the online

education literature is typically interested in student satisfaction with the online delivery method,

not student satisfaction with the learning environment or educational outcomes (Arbaugh &

Benbunan-Fich, 2007; Arbaugh, 2000, 2008; Joo et al., 2011).

Measuring Student Outcomes in Constructivist Learning Environments

In contrast, chemical education research has investigated student outcomes of both

academic achievement and satisfaction after implementation of specific teaching practices

aligned with social constructivism, such as POGIL and PLTL, or implementation of more

general teaching practices aligned with constructivism (Conway, 2014; Gosser et al., 2010;

Gupta et al., 2015; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder &

Hunnicutt, 2008; Tien et al., 2002). Utilizing academic achievement in chemistry and satisfaction

as dependent variables indicates an underlying assumption that aligning teaching practices with

constructivism should improve student learning and thus result in students who perform better

academically and are more satisfied with their learning experience. These studies have been

narrowly focused on introductory chemistry courses in general or organic chemistry, which

63

creates smaller differences between courses and makes comparisons across courses more

meaningful than comparing across multiple disciplines and academic levels.

While student learning is difficult to measure directly, exam grades, course grades, and

standardized American Chemical Society (ACS) exam scores are frequently employed in the

chemical education literature as ways to measure student learning (Conway, 2014; Gosser et al.,

2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder & Hunnicutt, 2008;

Tien et al., 2002). The beneficial effect of POGIL and POGIL-style instruction on both final

exam and final course grades has been demonstrated in a one-semester organic and biochemistry

course for pre-health professionals and in large enrollment general and organic chemistry courses

(Conway, 2014; Ruder & Hunnicutt, 2008). Similar results have been reported for

implementations of PLTL in general and organic chemistry courses from a variety of institutions

including community colleges and research universities (Gosser et al., 2010; Lewis & Lewis,

2005; Mitchell et al., 2012). However, instructors rarely discuss the psychometric properties of

their final exams or final course grades, so without more detailed information about these

achievement measures, the conclusions drawn from these studies must be interpreted cautiously.

Standardized ACS exams represent a more psychometrically sound tool available to

chemical educators for measuring student achievement. These exams have been in use since

1934 and are continuously revised and updated to reflect changes in teaching practices and

curricula (Brandriet, Reed, & Holme, 2015). The ACS exams are developed by national

committees independent of a specific classroom instructor, the validity and reliability of the

scores are evaluated, and the items on the version of the exam currently in use are not made

publicly available. Therefore, scores on these exams can be considered an acceptable measure of

64

students’ content knowledge (Lewis, 2014). In studies where ACS exams were used as final

exams, students experiencing PLTL showed similar or improved ACS exam scores relative to

students experiencing traditional lecture-based instruction (Lewis & Lewis, 2005; Mitchell et al.,

2012). These results indicate that PLTL instruction holds students to the same standard of

content knowledge as traditional instruction.

The benefits of implementing PLTL instruction were supported by taking initial

differences between students into account by controlling for SAT scores in the statistical analysis

(Lewis & Lewis, 2005; Mitchell et al., 2012; Tien et al., 2002). Considering SAT scores is

necessary to show the effectiveness of teaching practices across different groups of students,

since a demonstrated relationship exists between SAT math scores and grades in introductory

college science courses (H. E. Spencer, 1996; Tai et al., 2006). Since not all students take the

SAT in preparation for college, Nordstrom (1990) and Xu & Lewis (2011) looked at the

correlation between SAT math and ACT math scores for freshmen chemistry students and found

it to be approximately 0.70. This indicates that the ACT and SAT math scores are measuring

similar abilities in students. This is further supported by the similar standardized regression

coefficients of around 0.40 obtained by Nordstrom (1990) when predicting chemistry course

performance and by Lewis & Lewis (2005) and Xu & Lewis (2011) when predicting ACS exam

scores from the SAT and ACT math ability scores along with other variables.

Even implementing more general constructivist teaching practices, not specifically POGIL

and PLTL, has been shown to improve student academic achievement in chemistry. Hall et al.

(2014) describe a supplemental discussion-type section that has “roots in social constructivism

and borrows elements from a number of learner-centered pedagogies” (p. 37). This program

65

recruited students with lower SAT scores who, after enrollment in the program, earned exam

scores in both their general and organic chemistry courses that were not statistically different

from peers entering with higher SAT scores. The adoption of guided inquiry techniques and

collaborative group work was also found to increase critical thinking in a first semester general

chemistry laboratory course (Gupta et al., 2015). Here, social constructivism is invoked to

explain how “critical thinking develops in students through interactions with the teacher and

among students” (Gupta et al., 2015, p. 37). These two studies provide support for the idea that

the adoption of more general teaching practices aligned with constructivism, not just the

implementation of a specific constructivist pedagogy such as POGIL or PLTL, may improve

student academic achievement in introductory undergraduate chemistry courses.

In addition to improvements in academic achievement in chemistry, student satisfaction

with the constructivist learning environment has also been reported in the chemical education

literature (Conway, 2014; Hall et al., 2014; Ruder & Hunnicutt, 2008; Tien et al., 2002). In

contrast to the measurement of satisfaction with the online delivery aspect of the course seen in

the online education literature (Arbaugh & Benbunan-Fich, 2007; Arbaugh, 2000, 2008; Joo et

al., 2011), the chemical education literature typically reports student satisfaction and attitudes

towards a specific course learning environment. Students are generally positive about the

teaching practices employed in the learning environment, even though they often note an

increase in the amount of work done in class. The satisfaction instrument used by Hall et al.

(2014) highlighted the importance of student-student interactions by asking students to rate how

supportive they found their study group and how comfortable they felt contributing to

conversations about course materials. Students in the Tien et al. (2002) study reported that

66

working with peers helped their learning even more than the course lectures. Conway (2014)

reported students enjoyed working in groups and felt that they learned the material better than in

more traditionally taught courses. Similarly, Ruder & Hunnicutt (2008) reported students had

positive attitudes towards group work and a belief that they were learning from other students.

These studies reflect the affective outcomes of social presence as described by Garrison et al.

(2000).

Though satisfaction and attitude are frequently used interchangeably, Gardner (1975)

distinguishes satisfaction as one component of an attitude towards something. Similarly, Xu &

Lewis (2011) identify an emotional satisfaction component of student attitudes towards

chemistry. Xu & Lewis (2011) revised the Attitude towards the Subject of Chemistry Inventory

(ASCI) into a shorter version with the original semantic differential scale. In this semantic

differential scale, students indicate their position on a scale between two opposite words, such as

“satisfying” and “frustrating” instead of the more traditional response scale in which students

indicate their degree of agreement or disagreement with a particular statement. Though the

courses in which the students in the Xu & Lewis study were enrolled were not specifically

described as constructivist, a correlation of 0.35 was demonstrated between emotional

satisfaction and ACS exam scores indicating that a relationship exists between student

satisfaction and academic outcomes. However, a larger correlations (0.45 and 0.46) existed

between ACS exam scores and math ability scores as measured by SAT math and ACT math

scores, respectively (Xu & Lewis, 2011). This result confirmed the relationship between math

ability and academic achievement in introductory chemistry courses.

67

Modeling the Influence of a Constructivist Learning Environment on Student Outcomes

The previously discussed studies in online and chemical education can be synthesized into

a single model showing the influence of a constructivist learning environment on student

outcomes of academic achievement in chemistry and satisfaction. In this model, the

constructivist learning environment is measured by the three CoI factors of cognitive presence,

social presence, and teaching presence. Academic achievement in chemistry is measured by the

outcomes typically used in chemical education research such as ACS exam scores and final

course grades. Student satisfaction is measured using a survey instrument as is typical in both

online and chemical education research.

The model in Figure 10 provides a diagrammatic representation of the hypothesized

structural relationships among these latent and measured variables. The latent variables are

shown as ovals and represent variables that are not measured directly, but will be identified by

analysis of student responses to the CoI survey instrument and student satisfaction survey items.

The measured variables of math ability and academic achievement in chemistry are shown as

rectangles and are determined based on student scores. The portion of the model showing the

individual CoI and satisfaction survey items has been omitted for clarity.

Based on prior research in online education, the same relationship among cognitive

presence, social presence, and teaching presence is expected as seen previously in Figure 8 (p.

60). Teaching presence is hypothesized to directly influence both cognitive and social presence

while also indirectly influencing cognitive presence through social presence. The multiple

influences of teaching presence are due to the role of the instructor in both selecting course

68

Figure 10. The hypothesized structural model of relationships among the three CoI presence factors, math ability scores, and student outcomes. content and setting the tone of interactions between the instructor and students (D. R. Garrison et

al., 2010; Shea & Bidjerano, 2009; Swan, 2003). Building on existing research in online

education, the proposed model in Figure 10 adds causal relationships among the CoI presence

factors and student outcomes of academic achievement and satisfaction.

From the chemical education literature, the cognitive presence and teaching presence

aspects of a constructivist learning environment are expected to directly influence academic

achievement in chemistry as measured by ACS exam scores and final course grades (Conway,

2014; Gosser et al., 2010; Hall et al., 2014; Lewis & Lewis, 2005; Mitchell et al., 2012; Ruder &

Hunnicutt, 2008; Tien et al., 2002). The direct influence of teaching presence on ACS exam

scores and final course grades is hypothesized because “the instructor serves as an expert who

69

plans instruction to stimulate students’ interest, motivates their participation in the learning

process, and facilitates their learning” (Swan, 2003, p. 8). Though no studies with the CoI

instrument have specifically examined the influence of teaching presence on student academic

outcomes, the role of the instructor as expert and facilitator of learning suggests a causal

relationship between teaching presence and academic achievement in chemistry. Additionally, in

the CoI model selecting appropriate content is at the overlap of teaching and cognitive presence.

The selection of content and learning activities that facilitate student knowledge construction is

expected to influence the demonstration of that knowledge construction through ACS exam

scores and final course grades. For this reason, cognitive presence is also hypothesized to have a

causal influence on academic achievement in chemistry.

In addition to the influence of teaching and cognitive presence, math ability is expected to

have a direct influence on academic achievement in chemistry (Lewis & Lewis, 2005; Mitchell

et al., 2012; Nordstrom, 1990; Tien et al., 2002; Xu & Lewis, 2011). A path is also included to

account for the influence of ACS exam scores on final course grades. The paths between social

presence and ACS exam scores and final course grades are omitted because the influence of

social presence on these two outcomes is hypothesized to be indirect. This indirect influence is

hypothesized because of the small correlation seen between social presence and perceived

learning in earlier studies (0.19; Arbaugh, 2008) and the assumption that social presence only

affects academic outcomes when cognitive presence provides an academic context for the social

interactions among students. Finally, from both the online and chemical education literature,

cognitive presence, social presence, and teaching presence are all expected to directly influence

satisfaction (Conway, 2014; Hall et al., 2014; Joo et al., 2011; Ruder & Hunnicutt, 2008). A two-

70

headed arrow is included to account for a relationship between final course grades and student

satisfaction beyond the relationships present in the hypothesized model, but no causal direction is

proposed for the relationship. The lack of causality reflects the ongoing debate as to whether

students who are more satisfied with a course perform better academically or if performing better

academically causes students to be more satisfied with a course (Greenwald & Gillmore, 1997;

Howard & Maxwell, 1982).

The model proposed in Figure 10 represents only one possible relationship among this set

of variables. Even if this model is shown to have good fit with collected data, it does not

necessarily represent the only viable model. Early research using the CoI has shown that

teaching presence may not be a single factor, but may be best modeled as two factors (Arbaugh,

2007; Shea, Sau Li, & Pickett, 2006). In this alternate model, teaching presence is instead

conceptualized as a pre-course instructor activity factor and an in-course instructor activity

factor. Pre-course instructor activities are typically done outside of the class period and include

the design and organization of the course. Items 1–4 on the CoI instrument address the results of

pre-course activities such as communicating course goals, topic, due dates, and instructions. In-

course instructor activities are typically done during the class period and include facilitation of

student learning and direct instruction (Arbaugh et al., 2008). Items 5-13 on the CoI instrument

address the in-course instructor activity factor. A visualization of the difference between

teaching presence as a single factor and a two correlated factors can be seen in Figures 11 and

12. During the data analysis process, these two competing models can be tested and statistically

compared to see which provides a better fit for the data.

71

The primary model presented in Figure 10 relies on measurements of student perception

and student performance to model relationships among constructivist learning environment

factors and outcomes of student satisfaction and academic achievement in chemistry. This model

represents the integration of recent research in both chemical education and online education

through their shared use of constructivism as a foundation for the development of teaching

practices aligned with how students learn. Structural equation modeling (SEM) was chosen as

the primary statistical data analysis methodology because of the complex causal relationships

investigated in which the three CoI presence factors were hypothesized to influence each other as

well as multiple student outcomes (Bauer, 2003; Huta, 2014; Kline, 2011). While SEM is a

Figure 11. Teaching presence as a single factor with 13 indicator variables.

Figure 12. Two correlated factors taking the place of a single teaching presence factor.

72

sound statistical technique for analyzing relationships among measured and latent variables,

student measurements should not provide the only source of information about the degree to

which a learning environment incorporates constructivist principles. Information about the

learning environment provided by the course instructor can be used to support or refute the

picture of the learning environment portrayed by student responses to the CoI survey instrument.

Measuring Instructor Approaches to Teaching Though student perceptions of the learning environment can provide valuable information,

it is necessary to remember that “undergrads may not be sophisticated enough to distinguish

between facilitation and direct instruction” (D. R. Garrison & Arbaugh, 2007, p. 165), which is

an important distinction between constructivist and objectivist learning environments. For this

reason, the most complete description possible of a learning environment necessitates the

combination of student perceptions with information obtained from the course instructor.

However, given the various interpretations and implementations of constructivism previously

discussed in recent publications in the Journal of Chemical Education, it may not be possible to

directly ask instructors about constructivism and obtain information aligned with the more

general definition of constructivism proposed for this research. Additionally, it may be possible

that some chemical educators who hold constructivist beliefs about learning or teach in ways that

are aligned with constructivism may be unaware that they are in agreement with constructivist

principles. Other chemical educators may be aware that constructivism is currently popular, but

they may not understand the specific principles associated with constructivism. Therefore, it may

be more effective to ask instructors to describe their approach to teaching and then have the

researcher interpret the instructor responses through a constructivist lens.

73

Previous studies in online and science education have utilized interviews with instructors in

order to determine their approach to teaching (Arbaugh & Benbunan-Fich, 2007; Prosser et al.,

1994; Trigwell et al., 1994). As a portion of their research on the importance of interactions in an

online environment Arbaugh and Benbunan-Fich (2007) used semi-structured interviews to ask

MBA course instructors “whether their courses were based primarily in fact/concept

dissemination via online lectures, or based on knowledge construction by students” (p. 857).

Instructors were also asked about the relative amount of individual and group work in their

courses. Instructors’ responses were compared with information in the course syllabi or course

websites in order to support the researchers’ classification of the course as objectivist or

constructivist and group-centered or individual-centered. All instructor reports in the Arbaugh

and Benbunan-Fich (2007) study were found to be consistent with information provided in the

course syllabi or websites.

Interviews were also used to inform phenomenographic qualitative research investigating

ideas held about teaching and learning by 24 instructors of first-year undergraduate chemistry

and physics courses in order to see how these ideas influenced the approaches to teaching

adopted by the instructors (Prosser et al., 1994; Trigwell et al., 1994). The categories of

approaches to teaching developed in these studies were based on the interviews and were not

hypothesized in advance of collecting data as in the Arbaugh and Benbunan-Fich (2007)

research. The five categories that emerged from an analysis of the interviews were (1) a teacher-

centered information transmission approach, (2) a teacher-centered approach emphasizing

student acquisition of concepts, (3) an approach utilizing interactions between the teacher and

student to facilitate student acquisition of concepts, (4) a student-centered approach emphasizing

74

student development of concepts, and (5) a student-centered approach emphasizing students’

conceptual change. These five categories can be conceptualized as a continuum ranging from a

more objectivist approach focusing on the teacher acting as an external source of information to

a more constructivist approach focusing on changing the knowledge structures of individual

students.

In later research, Trigwell & Prosser (1996) used the interview transcripts to develop an

inventory that could be administered to instructors in order to determine their approach to

teaching in a particular context. Trigwell & Prosser clearly state that the inventory is context

dependent and should not be used to classify an instructor but rather to classify an instructor’s

approach to teaching for a particular course. Even within the same subject, instructors often

adopt different approaches to teaching for different classes depending on the situation. As an

example, the same instructor may be more likely to utilize a teacher-centered information

transmission approach with a large-enrollment introductory level course but utilize a student-

centered approach emphasizing conceptual change when working with upper level undergraduate

or graduate students on independent research projects which by their nature have smaller

enrollments.

From the interview responses, Trigwell & Prosser (2004) selected statements that served as

indicators of various teaching approaches. A principle components analysis (PCA) was

performed on a 39-item version of the inventory with responses from a total of 58 university

chemistry and physics instructors, 11 of whom had participated in the original interviews. As a

result of this analysis, the student-teacher interaction subscale was removed due to a high degree

of overlap with the student-centered subscale. Eventually, the inventory was shortened and a

75

confirmatory factor analysis (CFA) was undertaken with over 650 participants from universities

in 15 countries in disciplines representative of the variety typically taught at the university level.

Most recently, a CFA was undertaken for the 16-item version of the Approaches to

Teaching Inventory (ATI) with responses from over 1000 instructors at universities in the UK,

US, Scandinavia, and Hong Kong in a range of disciplines (Prosser & Trigwell, 2006). However,

this sample was heavily weighted towards engineering instructors from Sweden who comprised

over half the sample. Multiple CFA models were tested and the two models with the best fit were

found to be a four factor model with four covariance terms added between individual item errors

(CFI = 0.934, RMSEA = 0.041, and SRMR = 0.043) and a two factor model with three

covariance terms added between individual item errors and one item loading on both factors (CFI

= 0.931, RMSEA = 0.040, and SRMR = 0.043). No model !" values were provided. Though the

fit statistics for the two models are similar, Prosser & Trigwell state a preference for the two-

factor model due to the high degree of correlation between two of the subscales in the four-factor

model (over 0.90). In the two-factor model, one factor can be described as the information

transmission teacher-focused approach (ITTF) while the other represents a conceptual change

student-focused approach (CCSF).

It is not entirely clear from the Prosser & Trigwell (2006) article whether or not the error

covariance terms were hypothesized prior to beginning the analysis or whether they resulted

from modification indices provided by the software used in the analysis. Given the relatively

poor fit of the two factor model with no error covariances (CFI = 0.865, RMSEA = 0.055, and

SRMR = 0.056), it seems likely that the error covariances and additional item loading were

added based on modification indices provided by the software to show opportunities to improve

76

model fit. While the RMSEA and SRMR values are within an acceptable fit range, the low CFI

value indicates a possibility that even though the model did an acceptable job explaining

relationships among the data, the relationships were relatively weak to begin with. Though

nothing prohibits a researcher from modifying models after seeing the results, modifications

should be supported by a theoretical reason in addition to a mathematical reason (Mueller &

Hancock, 2008).

Prosser & Trigwell (2006) explain that the items linked by the error covariance terms were

somewhat redundant and therefore their linking is supported by their similar item wordings. The

item that was allowed to load both the ITTF and CCSF factors was related to the use of

textbooks to provide information. The rationale for this item being related to both approaches is

due to the inclusion of instructors from multiple disciplines in the research sample and the

differing ways in which instructors in the humanities and sciences regard the use of textbooks.

However, no evidence was provided for this in terms of conducting a separate analysis with

samples separated by discipline. Further work with the ATI removed some problematic items

and introduced new items, resulting in a 22-item inventory that had an improved CFI value

without the use of error covariance terms (CFI = 0.95; RMSEA = 0.06 CI90=[0.057, 0.072];

SRMR = 0.08). This model is based on data from only 318 instructors in a range of disciplines at

Australian and UK institutions (Trigwell et al., 2005).

Ultimately, the development of the ATI from qualitative interviews and the quantitative

results from these three studies (Prosser & Trigwell, 2006; Trigwell et al., 2005; Trigwell &

Prosser, 2004) indicate that the ATI is an acceptable instrument for identifying two distinct

instructor approaches to teaching in specific contexts. Cronbach’s alpha values for each

77

approach’s subscale range from 0.66 to 0.86 (Prosser & Trigwell, 2006; Trigwell et al., 2005;

Trigwell & Prosser, 2004) indicating good internal consistency of the subscales based on the

generally accepted, but somewhat arbitrary cutoff of 0.70 (Arjoon, Xu, & Lewis, 2013). The

results of the two factor CFA consistently demonstrate a negative correlation (–0.26; Prosser &

Trigwell, 2006; –0.35 Trigwell et al., 2005) between the factors representing the two approaches.

The negative values support the idea that these factors describe distinct instructor approaches to

teaching in a specific context. The ATI has also been used in chemical education research to

examine how the teaching approaches of new university chemistry professors change after

attending a short teaching workshop emphasizing the use student-centered teaching approaches

(Stains et al., 2015). The instructors who attended this workshop had statistically significantly

higher CCSF scores one week after the workshop compared to their CCSF scores before the

workshop, suggesting that the CCSF scale measures approaches to teaching aligned with

constructivism. The instructors also had significantly lower ITTF scores than a control group

who did not attend the workshop.

Combining information from the perspectives of two classroom stakeholders, the instructor

and students, provides a more complete picture of the learning environment created by the

instructor and experienced by the students. Measuring the learning environment from both

perspectives also provides information regarding how well the ATI and CoI measure the

presence of indicators of a constructivist learning environment. For example, if the instructor’s

approach to teaching in a particular course is closer to the conceptual change student-centered

approach, it would be expected that the student CoI responses would show a high degree of

teaching and cognitive presence. Additionally, if the instructor indicates on the ATI that he or

78

she sets aside class time for students to discuss course topics amongst themselves, this is likely to

be reflected in students’ agreement with CoI items related to course discussions.

Comparing responses to the ATI and CoI determines whether or not the CoI instrument,

when modified for face-to-face introductory undergraduate chemistry courses, is an acceptable

instrument for measuring indicators of a constructivist learning environment from the students’

perspective by providing evidence for the validity of the item responses. Additionally, the

psychometric properties of the modified CoI survey are examined quantitatively to provide

additional evidence for the validity and reliability of the scores. If the modified CoI survey

proves to be an acceptable instrument for this population of students, then combining CoI

responses with measurements of student outcomes will provide the data necessary to test the

model proposed in Figure 10 (p. 68). Testing the fit of this model allows for an examination of

how a constructivist learning environment, as perceived by students, affects student satisfaction

and academic achievement in chemistry. These research goals are summarized by the following

research questions.

Research Questions

1. Are self-reported instructor approaches to teaching consistent with student

perceptions of the learning environment?

2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for

measuring student perceptions of the indicators of a constructivist learning

environment in a face-to-face introductory undergraduate chemistry course?

3. To what degree does a constructivist learning environment, as measured by

student CoI survey responses, affect outcomes of student satisfaction and

79

academic achievement in chemistry, as measured by ACS exam scores and final

course grades when the effect of math ability on academic achievement is

considered

80

Chapter 3 This research relies on both student and instructor survey instruments to measure the

extent to which a constructivist learning environment exists in a given face-to-face introductory

undergraduate chemistry course. These instruments required slight modifications to their

wording to better align them with the intended research situation and population. Though the

literature contains documentation of prior use of the CoI and the ATI, modifications to the

wording of items necessitated a small study to pilot the reworded instruments prior to their use in

the main research project.

In addition to this preparation for the main research study, two separate power analyses

were conducted to determine the sample size necessary to test overall data-model fit and to test

the specific model parameters of interest for this research. The methodology for this research

primarily focuses on obtaining quantitative data in the form of survey responses and student

achievement data. A small qualitative strand is embedded in the collection of instructor data. For

this reason, the data analysis procedures utilized to answer the research questions are

predominately related to SEM analysis.

Modifications to ATI and CoI Wording

Though both the ATI and the CoI have been previously used in published research and

have information available on their development and use with various populations of instructors

and students (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al., 2010; Joo et al.,

2011; Prosser & Trigwell, 2006; Shea & Bidjerano, 2009; Trigwell et al., 2005; Trigwell &

Prosser, 2004), evidence for the validity of the instrument scores must be demonstrated in each

use (American Educational Research Association, American Psychological Association, &

81

National Council on Measurement in Education, 2014). For this particular study, the researcher

initially made small modifications to the wording of specific items on both the ATI and CoI in

order to better align the items with best practices in survey instrument design (Krosnick &

Presser, 2010) and ensure the wording was appropriate for the research population of instructors

and students enrolled in face-to-face introductory chemistry courses. These wording changes in

combination with the use of the instruments with this specific population necessitated an initial

pilot study of the survey instruments to check if the items were being interpreted as intended.

The pilot study procedures and results are described in greater detail after a discussion of the

modifications to particular survey items.

One of the authors of the ATI, K. Trigwell (personal communication, August 13, 2015),

was contacted for permission to use the ATI. The researcher was provided the revised 22-item

ATI-R (Trigwell et al., 2005) along with scoring directions. Initial changes were made to the

original European/Australian wording of the ATI-R by the researcher to more closely align the

language with US usage. As an example, item 11 on the original ATI-R is “In this subject, I

provide the students with the information they will need to pass the formal assessments.” Here,

the term “subject” is used in a way that would be more similar to the US usage of “course.”

Therefore, this item was initially revised to “In this course, I provide the students the information

they will need to pass the formal assessments.” This revision process occurred for all 22 items on

the ATI-R.

In this initial stage of revisions, the items were kept as similar as possible to their original

wording by only changing the word “subject” to “course” where appropriate. The response scale

was not changed, but the layout of the instrument was modified to label each of the five scale

82

points above its corresponding number. This modification was done to minimize the cognitive

effort necessary for respondents to select the most appropriate scale point by removing the need

for respondents to remember the labels or refer back to descriptions given in the instrument

directions (Krosnick & Presser, 2010). The revised instrument, response scale, and directions

used in the instrument pilot study with chemistry course instructors can be found in Appendix B.

A similar process was undertaken to modify the CoI instrument. The CoI items have been

published numerous times with only slight variations across research groups (Arbaugh et al.,

2010; Arbaugh, 2008; D. R. Garrison et al., 2010; Shea & Bidjerano, 2009). The version of the

CoI used as a starting point for the current research was published by Arbaugh, Cleveland-Innes,

and Diaz (2008). The CoI was developed by Canadian researchers and did not have the same

wording issues due to language differences as the ATI. However, since the CoI was originally

designed for students in online courses four items were reworded to reflect the intended face-to-

face population of the current research. These four original items and the modified version can

be seen in Table 1.

Table 1 Original and Revised CoI Items

Original CoI items (Appendix A)

Revised CoI items used in pilot study (Appendix C)

16. Online or web-based communication is an excellent medium for social interaction

Q16. Face-to-face communication is an excellent medium for social interaction

17. I felt comfortable conversing through the online medium

Q17. I felt comfortable conversing face-to-face in class

22. Online discussions help me to develop a sense of collaboration

Q22. In-class discussions helped me to develop a sense of collaboration

28. Online discussions were valuable in helping me appreciate different perspectives

Q29. In-class discussions were valuable in helping me appreciate different perspectives

83

In addition to modifications of the four items in Table 1 to remove references to an online

course environment, two other small wording changes were made to the instrument before it was

piloted with students who had completed face-to-face chemistry courses. These changes were

made so that the instrument reflected best practices in survey design (Krosnick & Presser, 2010).

Twelve of the thirteen items designed to measure teaching presence all started with the same

question stem, “The instructor…” but one item was originally worded “Instructor actions

reinforced the development of a sense of community among course participants.” The item was

changed to “The instructor reinforced the development of a sense of community among course

participants”. This change was made to maintain the consistency of the stem across all items

designed to measure teaching presence.

The second change was made to an item originally worded as “Reflection on course

content and discussions helped me understand fundamental concepts in this class.” As written,

this item is considered double-barreled because it asks two questions simultaneously (Krosnick

& Presser, 2010). This type of item poses a problem for respondents because if the respondent

agrees with only one part of the question but not the other it is difficult for the respondent to

choose a response that accurately reflects his or her opinion. For example, if reflections on

course content helped the student understand concepts but the student never reflected on course

discussions, the student may not know whether to select agree or disagree. Therefore, this item

was split into two separate items, one addressing reflections on course content and the other

addressing reflections on discussions.

The CoI instrument is most typically given using a five-point scale ranging from strongly

agree to strongly disagree (Arbaugh et al., 2008; D. R. Garrison et al., 2010; Shea & Bidjerano,

84

2009). The five-point scale has been found to be an acceptable number of scale points in both the

survey methodology literature and the structural equation modeling literature (Finney &

DiStefano, 2013; Krosnick & Presser, 2010). In structural equation modeling research,

approximately normal ordinal data with at least five categories can be treated as continuous data

without distorting the model fit indices (Finney & DiStefano, 2013). From a survey methodology

perspective, a five-point scale has performed better than shorter scales and similarly to seven-

point scales. Methodological studies have shown scale performance to worsen when the

instrument contains more than seven points (Krosnick & Presser, 2010). When given an

unlabeled scale, some research has shown that respondents naturally divide it into five scale

points (Krosnick & Presser, 2010).

Another benefit to the five-point scale is that it is easy to provide distinctive scale labels

for each point in a five-point scale. For the pilot study, these labels were provided as strongly

agree, agree, unsure, disagree, and strongly disagree. Survey research has demonstrated that

labeling all scale points improves reliability of the responses and respondent satisfaction with the

survey instrument (Krosnick & Presser, 2010). In addition, an option was added with a numerical

value of zero and the label “Not Applicable” for situations in which an item did not apply to a

specific student or course. It was anticipated that this situation might occur for courses that did

not use formal, structured in-class discussions since many CoI items reference course

discussions.

In addition to small differences in wording among the versions of the CoI found in the

literature, the items are sometimes grouped by type of presence (Arbaugh et al., 2008, 2010; D.

R. Garrison, Cleveland-Innes, & Fung, 2004; Shea & Bidjerano, 2009), sometimes randomized

85

(D. R. Garrison et al., 2010), and sometimes no information is provided on the order of item

presentation (Joo et al., 2011). The recommendations of Krosnick & Presser (2010) were

followed in grouping the items by type of presence, but these groupings were not explicitly

labeled to avoid biasing student responses to the items. With these groupings, the version of the

CoI instrument used in the pilot study with students had items 1-13 addressing teaching

presence, items 14-22 addressing social presence and items 23-36 addressing cognitive presence,

excluding item 28. Item 28 was designed as a check to catch respondents who may not be

reading carefully due to disinterest or fatigue by simply asking the respondent to select the

option corresponding to “Disagree”. Including this item allows for the exclusion of a set of

responses where “Disagree” was not selected since it is likely that the other options selected by

the respondent do not accurately reflect a thoughtful evaluation of the learning environment.

In addition to the CoI items, students in the pilot study were also provided with five

traditional satisfaction items on a five-point scale from Bollinger and Wasilik (2012) and four

satisfaction items on a five-point semantic differential scale from Xu & Lewis (2011). Prior

research with the traditional satisfaction items had used a five-point scale (Bolliger & Wasilik,

2012) but the semantic differential items were changed from the original seven-point scale used

by Xu & Lewis (2011) to a five-point scale in order to maintain consistency in response scale

length across all items on the student survey instrument used in the pilot study. The complete

instrument used for the pilot study with students is found in Appendix C.

86

Instructor and Student Survey Pilot Studies

Recruitment of Participants for Pilot Studies

The pilot study methodology was similar for both the ATI and the combined

CoI/satisfaction items on the student survey. The sample for the ATI pilot study was drawn from

a population of instructors of the classroom portion, not laboratory sections, of 100-level face-to-

face chemistry courses and the sample for the CoI/satisfaction item pilot study was drawn from a

population of students who had completed 100-level face-to-face chemistry courses. Both the

instructor and student samples were recruited from the population at a midsize private research

university (Indiana University Center for Postsecondary Research, n.d.). After obtaining

permission for the pilot study from the institutional review board (IRB) at the university,

recruitment emails were sent to both instructors and students informing them of the purpose of

the study and soliciting their participation.

Only instructors currently listed on the university chemistry department’s website who

had taught a 100-level chemistry course in the previous two years received the instructor

recruitment email. The student recruitment email was sent to all student members of the

Chemistry Club at the university. Chemistry Club members were selected as the target

population since these students were likely to have completed 100-level chemistry courses at the

university and their email addresses were available on the Chemistry Club website. Two

recruitment emails were sent to both instructors and students; the second email was sent

approximately a week after the first. Five instructors and five students participated in the pilot

study. All ten participants (instructor and student) gave permission to have their responses audio

recorded during the pilot study.

87

Instructor Survey Pilot Study Methodology

In the pilot study for the ATI, instructors were first asked a series of open-ended

interview questions related to the length of their teaching career, their general approach to

teaching chemistry, any changes they have made to their teaching approach over time, their

specific approach to teaching 100-level courses, and their approach to teaching in the 100-level

course they have most recently taught. The syllabus for the most recently taught 100-level course

was either provided by the instructor or brought to the interview by the researcher in order to aid

the instructor in recalling specific course details and to provide an artifact for the researcher to

use for comparison with the instructor’s responses. The answers to the open-ended interview

questions about the instructor’s approach to teaching were used to provide information about the

background of each instructor and to inform interpretation of the instructor’s responses to the

ATI.

The instrument testing phase of the pilot study was conducted after the instructors

responded to the open-ended interview questions. During this phase, instructors were provided

with the 22-item ATI instrument revised by the researcher as discussed earlier in this chapter

(see Appendix B). The instrument testing protocol was a think-aloud interview in which the

instructor read each item aloud and verbalized his or her rationale for selecting a particular

response (Krosnick & Presser, 2010). If the instructor indicated confusion or uncertainty in

selecting a response, additional follow-up questions were asked by the researcher to probe

possible reasons for this difficulty, including unclear wording of the item or the item not being

applicable in a particular classroom environment. Following completion of the survey

instrument, the instructor was asked to provide any general or overall feedback on the survey

88

instrument and asked if the survey instrument satisfactorily captured his or her approach to

teaching for the particular 100-level course that was the focus of the ATI responses.

Student Survey Pilot Study Methodology

The pilot study for the CoI and satisfaction items followed a similar format to the second

phase of the instructor pilot study. Prior to meeting with each student participant, the student was

asked to provide the course number and semester of enrollment for the first 100-level chemistry

course completed at the university. The researcher used this information to obtain a copy of the

course syllabus from the university’s online syllabus manager. As with the instructor pilot study,

this syllabus was provided to each student to aid the student’s recall of specific course details and

to provide an artifact for the researcher to use for comparison with the student’s interview

responses. The CoI and satisfaction item pilot study protocol started with the think-aloud

interview.

During the think-aloud interview students were provided with the 36-item CoI instrument

revised by the researcher along with the two sets of satisfaction items (see Appendix C). If the

student indicated confusion or uncertainty in selecting a response, additional follow-up questions

were asked by the researcher to probe possible reasons for this difficulty, including unclear

wording of the item or the item not being applicable in a particular classroom environment.

Following completion of the survey instrument, the student was asked if he or she had any

difficulty with the semantic differential set of satisfaction items, if the student had a preference

for the traditional or semantic differential satisfaction items, for any other general comments or

feedback on the CoI or satisfaction items, and if the items seemed to cover all relevant aspects of

89

the course that would help someone understand the student’s experience within the course

environment.

Pilot Study Results

Instructor Survey

The five instructors participating in the pilot study reported between four and 40 years of

experience teaching chemistry. Four instructors were male and one was female. The most

recently taught 100-level courses used for reference in the instructor interviews were first

semester general chemistry, GOB (general, organic, and biochemistry) for nursing students, and

a chemistry course for non-science majors. Typical enrollment for these courses ranged from

approximately 40 students in the GOB and non-science majors courses to approximately 100

students in the first semester general chemistry courses.

All instructors identified at least a few issues with item wording during the think-aloud

portion of the interview. During the think-aloud each instructor was directed to provide a short

explanation of why he or she chose a particular response to each ATI item and also asked to

elaborate on particular words or phrases that were unclear or confusing. The most frequent

comment from instructors was that particular phrases were unclear or too open to interpretation.

For example, the first item on the ATI asks about students focusing on “what I provide them” but

instructors were unsure if this meant only material created by the instructor such as lecture notes

and handouts, or if this included all course material selected by the instructor including the

textbook and supplemental materials.

Similarly, instructors were unclear on what the phrases “formal assessment items”,

“teaching time”, “key texts and readings”, “teaching sessions”, “good set of notes”, “good

90

presentation of information”, and “information base” meant in questions 2, 5, 6, 8, 9, 10, 11, 15,

16, 17 and 22. For these items, modifications were made to the wording to more closely align

each item with the interpretation adopted by most instructors. In items 5, 8, and 15 the phrases

“teaching sessions” and “teaching time” were removed to accommodate the broader

interpretation of teaching in the course to include communicating with students beyond the

scheduled class meeting times. This interpretation included office hours, email exchanges, and

course websites as places where instructors could “make available opportunities for students in

this course to discuss their changing understanding of the subjects” as described in item 13. This

interpretation is consistent with student responses to the CoI items in which the students

considered office hours and email exchanges as places where the instructor could communicate

important course information and provide feedback to students.

Other modifications included item 12 which was identified as too vague in asking about

“any questions that students may put to me” and was changed to “any questions about course

content that students may ask”. Additionally, item 14 was expanded to cover the frequently

mentioned idea that students could be annotating notes provided by the instructor and not

necessarily writing their own from scratch. In addition to the unclear and vague items, some

items used language that instructors found to be too loaded, such as the phrase “I deliberately

provoke debate and discussion” in item 8. This wording was interpreted as antagonistic and was

changed to the more neutral phrase “I encourage debate and discussion”. For the same reason,

the word “question” in item 15 was changed to “discuss”.

Some of the items also seemed to have implied judgments, such as the phrase “a lot of

facts” in item 4. Instructors generally agreed that students should be presented with facts as part

91

of the course but did not feel comfortable agreeing with the idea of presenting a lot of facts since

this seemed to have a negative connotation. For the same reason the qualifiers “good” or “a lot”

were removed from items 10, 15 and 16 to make them more neutral.

Another issue that arose during the think-aloud was the relevance of the frequency

response scale for certain ATI items. The frequently scale ranged from “only rarely” to “almost

always” and did not seem appropriate for items asking for instructor judgments using phrases

like “it is important” or “it is better”. As an example, the second item on the version of the ATI

used in the pilot study read “It is important that this course should be completely described in

terms of specific objectives that relate to formal assessment items.” On a frequency scale, it is

unclear if a response of “almost always” indicates that the instructor almost always believes it is

important to describe the course in terms of specific objectives related to formal assessment

items or if the course itself is almost always described in terms of specific objectives related to

formal assessment items. Due to the difficulty of answering these items on the frequency

response scale, they were reworded to remove the belief component and be more clearly focused

on actual classroom practices. As a result, the second item was changed to read “This course is

completely described in terms of specific objectives that relate to course assessments.” With this

new wording the item could now be answered with respect to how frequently specific objectives

are related to course assessments. This change and similar changes to items 4, 10, 14, 17, 18, 20,

and 21 better align the ATI with its purpose in the context of the current research, which is to

provide a description of the learning environment from the instructor’s perspective. Appendix D

contains the full list of ATI items, revisions made to the items after the pilot study, and the

specific rationale for each change.

92

While all but four items on the ATI required some degree of revision to their wording,

each instructor’s interpretation of the items during the think-aloud protocol indicated that

generally each instructor had an interpretation of the item that agreed with the intended focus of

the item based on whether it was designed to address the information transmission teacher-

focused (ITTF) or the conceptual change student-focused (CCSF) scale (Trigwell & Prosser,

2004). For this reason, scale scores were calculated for each instructor and compared to their

responses to the open-ended interview questions which asked about their approaches to teaching.

Responses to the open-ended interview questions were examined through the lens of the two

teaching approaches the ATI was designed to measure, ITTF and CCSF. The scale scores and

interview responses were also compared to the information available in the course syllabus.

The self-reported approaches to teaching varied for each instructor but a majority of the

instructors described the importance of having students solve chemistry problems both as a way

to learn chemistry and also as a way to demonstrate their knowledge. A subset of these

instructors also commented on wanting to provide students with class time to practice and

discuss problem solving with each other but felt limited in their ability to provide this class time.

These comments are interpreted as indicating that a majority of the instructors perceived a

conflict between wanting to adopt a more student-focused classroom environment, interpreted as

a more CCSF approach, and feeling limited by the need to use class time to provide information

to students, interpreted as a more ITTF approach. One instructor emphasized the role of

discussions in providing an impetus for students to learn content and as a way for students to

demonstrate their knowledge outside of solving problems, which provides another illustration of

a more CCSF approach.

93

Following the directions provided by Trigwell for use of the ATI-R, scores were

calculated on each scale for each instructor’s approach to teaching the particular course

discussed during the interview. These calculations were done by summing the responses to items

on each scale and dividing by the total number of items on the scale. This produced an average

score for each instructor on each scale ranging from 1 (Only Rarely) to 5 (Almost Always).

Examination of the scores showed that all five ITTF scores were in a relatively narrow range

from 3.5 to 4.4 while CCSF scores ranged from 3.0 to 4.8.

Additionally, there was no clear relationship between ITTF and CCSF scores. That is,

higher CCSF scores did not correspond to lower ITTF scores and higher ITTF scores did not

correspond to lower CCSF scores. This result was unexpected and may indicate that the two

approaches to teaching are not mutually exclusive for the 100-level chemistry courses described

by the instructors. One possible explanation for this result could be if the sample of five

instructors was atypical of those surveyed in other research with the ATI. Another possibility

may be that ITTF and CCSF scores have a negative correlation in aggregate across multiple

instructors but when focusing on an individual instructor describing his or her approach to

teaching a specific course, there is no relationship between ITTF scores and CCSF scores.

Addressing the first possible explanation requires looking at the samples used to develop

and refine the ATI. The initial phenomenographic interviews that led to the development of the

ATI were conducted with 24 instructors of first-year undergraduate chemistry and physics

courses at two Australian universities (Trigwell et al., 1994). The courses taught by these

instructors included courses for “engineers, life scientists, nurses and dentists as well as courses

for chemists and physicists” (Trigwell et al., 1994, p. 219). The instructors “held positions from

94

lecturers to professors, they all conducted lectures as part of their teaching and most were

involved in tutorials and/or laboratory teaching” (Trigwell et al., 1994, p. 219). The five

instructors interviewed for this pilot study comprised the entire population of instructors teaching

100-level chemistry lecture courses the semester the research was conducted and in that way are

fully representative of instructors at this midsize private university in the United States. In

addition, all interviewed instructors conducted lectures as part of their teaching and taught

courses targeted at the same population of students as in the Prosser, Trigwell & Taylor (1994)

study. Differences in US and Australian language were the primary reason for the small

modifications to the wording of the ATI items in changing “subject” to “course” for the pilot

study but the other similarities between the two groups indicate that they are analogous groups of

instructors.

While the group of instructors participating in the pilot study was comparable to the

group of instructors participating in the initial phenomenographic interviews used to develop the

ATI items, the ATI has also been tested with a much larger and more diverse population of

instructors. A majority of the research in the development and use of the ATI was conducted

with instructors in Europe, Australia, and Hong Kong who taught both in a variety of disciplines

in addition to chemistry and at various levels other than introductory courses (Prosser &

Trigwell, 2006; Trigwell et al., 2005). The use of the ATI with this range of instructors suggests

that it should be broadly applicable to other disciplines and that the sample of instructors

participating in this pilot study should be well within the population for which the ATI is

applicable. To confirm that the approaches to teaching 100-level chemistry courses described by

the sample of instructors participating in the pilot study are typical of instructors at other US

95

universities, future work with the revised ATI should be extended to include 100-level chemistry

course instructors at other US universities.

The second possible reason for the observation that ITTF and CCSF scores were not

related in this sample of instructors may be that almost all introductory chemistry courses are

heavily focused on content delivery. This emphasis on content delivery explains the high and

relatively narrow range of observed ITTF scores among all approaches to teaching 100-level

chemistry courses. The large amount of content in 100-level chemistry courses is frequently

discussed in the chemical education literature (J. N. Spencer, 1992; Talanquer & Pollard, 2010).

Given this environment for 100-level chemistry courses, it may be that the ITTF scale is

providing little relevant information about an instructor’s approach to teaching 100-level

chemistry courses since all instructors are concerned with covering a large amount of content.

This interpretation is further supported in the literature (Stains et al., 2015) by the lack of change

in ITTF scores after new chemistry professors attend a workshop emphasizing student-centered

teaching techniques, even though CCSF scores significantly increased for this group.

The chemical education literature also discusses the efficiency of lecture for delivering

content to large numbers of students even while recognizing that this information transfer

method of teaching is not as effective as more student-centered teaching approaches (Chambers

& Blake, 2007; Toto & Booth, 2008). The difficult balance between lecture and more student-

centered teaching approaches was apparent in the open-ended interviews with the instructors

participating in this study. All five instructors commented on wanting to spend less class time

lecturing but feeling constrained by the number of students enrolled in the course, the type of

material presented, or the limited amount of contact time with students. In spite of these

96

constraints, all instructors had been able to integrate some student-centered approaches into a

portion of their class time. The degree to which these student-centered teaching approaches had

been integrated into a particular course could be seen in interview responses, CCSF scale scores,

and the course syllabus.

The CCSF scale scores show that, on average, all five instructors’ approaches to teaching

could be considered as emphasizing a conceptual change, student-focused approach

approximately half the time. The instructors’ descriptions of their approaches to teaching in these

specific 100-level courses were compared with their responses to the ATI and the course

information available in their syllabi. This comparison indicated that the CCSF scale on the ATI

was able to detect a student-centered approach to teaching aligned with the definition used in this

research in which the role of the instructor was shifted from a lecturer to a facilitator for at least

part of the instructional time. The two instructors who described an approach to teaching that

was “student-centered” either in their interview or in their course syllabus corresponded to the

two highest CCSF scale scores. These two approaches to teaching still utilized lectures as the

primary method of delivering content to students but set aside some class time for small group or

whole class discussions. In this way these instructors adopted a teaching approach in which they

act as a facilitator for a portion of the instructional time which is reflected in the CCSF scores for

these courses.

Though these two instructors spent some class time utilizing a student-centered approach

to teaching, this did not mean that the instructors emphasized teaching content, as measured by

the ITTF scale, to any lesser degree than the other instructors who participated in the ATI pilot

study. While this sample of five instructors is relatively small, the results suggest that in the

97

context of approaches to teaching 100-level chemistry courses the ITTF and CCSF scales should

not be considered as describing mutually exclusive approaches to teaching. Clearly it is possible

for instructors to adopt aspects of a student-centered approach to teaching as part of an approach

that delivers content to students in a lecture format. However, it remains unclear whether this

result is due to the small sample size or the relative homogeneity of instructors and courses that

are the focus of the study. To address these concerns future research with the revised ATI should

be extended to instructors of chemistry courses beyond the 100-level to see if a similar

relationship is seen between ITTF and CCSF scores on the individual instructor level.

The scope of this research on constructivist learning environments in 100-level chemistry

courses did not necessitate the inclusion of a sample of instructors that would be large enough to

perform any statistical analyses, such as CFA, that could examine the internal structure of the

revised ATI instrument in Appendix D. Without the ability to demonstrate that the items in the

revised ATI are still related to the expected factors representing the ITTF and CCSF approaches

to teaching, it did not seem acceptable to use the revised ATI as the sole method of collecting

data on the degree to which a course instructor has created a constructivist learning environment.

For this reason, when the revised ATI was used in the main research project, a short semi-

structured interview was also conducted with the instructors to provide an opportunity for each

instructor to describe her or her approach to teaching. The interview was then used to inform

ATI scale score interpretation in the same way the open-ended question responses informed the

results of the pilot study. The results of the pilot study also indicated that only the CCSF scale

would provide information relevant to the degree to which a course instructor has created a

constructivist learning environment utilizing student-centered approaches to teaching.

98

Student Survey

The five students participating in the pilot study had all completed first semester general

chemistry courses. Four students had taken the course within one year of participating in this

pilot study and the other had taken the course three years prior to the interview. The four students

who had taken first semester general chemistry within the last year had to write a paper as part of

their course requirements. This group of four students also contained two pairs of students who

had been enrolled in the same course during the same semester. As a result of this overlap, the

five students represented courses taught by three of the five instructors interviewed in the

instructor portion of the pilot study. Two of the three female students and none of the male

students interviewed had been enrolled in an honors section of general chemistry. The class sizes

reported by the students ranged from approximately 15 students in the honors general chemistry

course to approximately 100 students in the regular section of general chemistry.

Due to recruitment of students in the Chemistry Club and the voluntary nature of

participation in the pilot study, these five students may not be representative of the entire

population of students enrolled in 100-level chemistry courses. Specifically, these students are

likely to have had more positive experiences in their chemistry courses and higher course grades,

both of which are possible motivations for joining the Chemistry Club. The students were not

asked to report their course grades as part of the pilot study, but the three students who

mentioned their course grade during the interview reported receiving grades of either B+ or A.

The other two students reported being happy with the grade they received. As a result, these five

students may have been more predisposed to provide positive answers to the items on the survey

than the average student enrolled in the courses. However, since the focus of the pilot study was

99

not on student ratings of the course but instead on how students interpreted the items, this

potential bias was not anticipated to influence the results of the think-aloud interviews.

Unlike the instructor pilot study, the student think-aloud interviews did not reveal many

issues with wording of CoI or satisfaction items. The only item that caused a problem for all five

students was item 5, a teaching presence item, which read “The instructor was helpful in

identifying areas of agreement and disagreement on course topics that helped me to learn.” The

students all indicated confusion on what was meant by “areas of agreement and disagreement”.

The most common student interpretation was related to agreement or disagreement on what

topics to include in the course or the order in which to present the course topics. These

interpretations appear to speak to the course design and organization component of teaching

presence. However, this interpretation is not supported by the literature discussing the

development of the CoI. Arbaugh (2008) describes identifying areas of agreement and

disagreement as one role of the instructor related to the facilitating discourse component of

teaching presence. Based on the intended focus of this item, it was reworded as “The instructor

was helpful in facilitating discussions on course topics that helped me to learn.”

Another wording issue occurred for item 27, a cognitive presence item, which initially

read “Brainstorming and finding relevant information helped me resolve content related

questions.” Student responses to this item indicated that brainstorming and finding relevant

information were not perceived as a single process but rather as two distinct phases of problem

solving. Students described “brainstorming” as the mental organization and planning that

occurred before “finding relevant information” which was described as looking up information

necessary to solve a problem. For most students these two processes were related, but some

100

students indicated that one half of the process was more helpful than the other half. Since some

students interpreted this item as double-barreled, it was split into two separate items. In addition,

a few students noticed inconsistent use of the phrases “course activities” and “learning activities”

so these were all changed to the phrase “course learning activities”.

The think-aloud interviews did not indicate any issues with student understanding of the

semantic differential satisfaction items. The follow-up question asking about student preference

for either the traditional or semantic differential set of satisfaction items revealed that students

preferred the traditional set of items because they felt those items addressed more specific

aspects of the course while the semantic differential items provided a more overall assessment of

the course. Even though students preferred the traditional satisfaction items, their description of

the semantic differential items as a more holistic view of their satisfaction with the course aligns

these items more closely with their purpose in this research project. The satisfaction items are

intended to function as indicators of the latent variable of overall student satisfaction with the

course and therefore the semantic differential items were chosen for use in conjunction with the

CoI items in the main research project.

Small changes were also made to the response scale options based on how students were

using the “Unsure” and “Not Applicable” options for the CoI items. Probing student use of the

unsure option revealed that students were treating it more as a midpoint between agree and

disagree than to communicate uncertainty in their response. For this reason, it was renamed as

“Neutral” in the revised version of the CoI. This change also increases the correspondence

between the CoI response scale and the response scale for the semantic differential satisfaction

items where the middle category functions a neutral midpoint between the two opposite words.

101

Initially a “Not Applicable” option was included as a zero on the CoI response scale, as

in the Shea & Bidjerano (2009) study, to accommodate situations where the course did not utilize

formal discussions. However, the students in the pilot study did not use this option consistently

and often were able to provide a response to items about course discussions even when the

course did not utilize formal structured discussions. The students typically tried to find anything

in the class that could be considered a discussion, such as the instructor asking questions of the

class during lecture.

Looking at pairs of students who were in the same course environment revealed that

when one student selected the “Not Applicable” option, the other student did not and instead

used the other scale options to describe his or her perception of the learning environment. This

inconsistent use of the “Not Applicable” option could have caused issues during the analysis and

interpretation of results when the CoI was used with a larger sample of students. For this reason,

the “Not Applicable” option was removed in order to encourage all students to select a response

that best described the learning environment. Encouraging respondents to provide their opinions

by removing don’t know and not applicable options is in alignment with best practices in survey

design (Krosnick & Presser, 2010). With the “Not Applicable” option removed, the response

scale for the CoI items was changed to a more standard appearance with 1 (Strongly Disagree)

on the left and 5 (Strongly Agree) on the right. The response scale for the semantic differential

items was also changed in this way, matching the original arrangement in Xu & Lewis (2011).

The full revised CoI and satisfaction student survey instrument is available in Appendix E.

Survey Instrument Validity Evidence

Much recent discussion of validity related to instruments used for measurement in

102

chemical education research utilizes the description of validity as a single concept with multiple

aspects (Arjoon et al., 2013; Wren & Barbera, 2013; Xu & Lewis, 2011). This conceptualization

of validity is also present in the most recent edition of the Standards for Educational and

Psychological Testing, known as the Standards, a collaboration of the American Educational

Research Association (AERA), the American Psychological Association (APA), and the

National Council on Measurement in Education (NCME). In the Standards, validity is described

as “a unitary concept. It is the degree to which all of the accumulated evidence supports the

intended interpretation of test scores for the proposed use” (AERA et al., 2014, p. 14). The

unitary nature of validity is also discussed at length by Messick (1989, 1995) who further

emphasizes that providing evidence for validity is a continuous process. Due to the continuous

nature of this process, Messick explains that “validity is a matter of degree, not all or none”

(1989, p. 13). Finally, both Messick (1995) and the Standards (2014) are clear that validity is not

a property that can be ascribed to a particular test but rather it is the test scores themselves that

must have evidence provided for the validity of their meaning in a particular context.

The unitary concept of validity replaced older descriptions of validity as having distinct

types including content validity, criterion-related validity, predictive validity, concurrent

validity, and face validity (AERA et al., 2014; Messick, 1989). Instead of recognizing distinct

types of validity, Messick (1989) defines construct validity as the overarching form of validity

because it provides “an integration of any evidence that bears on the interpretation or meaning of

the test scores” (p. 17). Messick (1995) describes six aspects of validity that can be used as

criteria to evaluate construct validity. These six aspects are content, substantive, structural,

generalizability, external, and consequential. Messick’s six aspects are integrated into the five

103

types of evidence for validity described in the Standards (2014) where the generalizability and

external aspects are integrated into a larger category of providing evidence for relationships

among the scores and other variables. The five types of evidence in the Standards (2014) are: (1)

evidence based on test content, (2) evidence based on response processes, (3) evidence based on

internal structure, (4), evidence based on relations to other variables, and (5) evidence for

validity and consequences of testing. These types of evidence for validity will be used as a

framework through which to evaluate the evidence that exists for the validity of ATI, CoI, and

student satisfaction scores in the context of the current research. While the Standards (2014)

specify that not all types of evidence are necessary in all situations, each type of evidence will be

discussed here and the presence or lack of evidence described for the ATI, CoI, and student

satisfaction items.

According to the Standards (2014), it is necessary to clearly define the proposed

interpretation of scores for which evidence is being provided. In the case of both the ATI and the

CoI, the underlying construct that both instruments are designed to measure is the degree to

which a constructivist learning environment exists for a particular 100-level chemistry course.

The ATI uses the instructor’s point of view to measure the frequency with which various

classroom activities and instructor behaviors occur. The proposed interpretation of ATI scores is

that these frequencies can be used to determine the degree to which an instructor’s approach to

teaching is aligned with constructivist principles of student-centered teaching. The CoI uses the

student’s point of view to measure the degree to which the student perceived various indicators

of a constructivist learning environment related to teaching presence, social presence, and

cognitive presence. The proposed interpretation of CoI scores is that the student perceptions can

104

be used as indicators of the three factors that encompass salient aspects of a constructivist

learning environment. Lastly, the student satisfaction items will be interpreted as indicators of

the latent variable of overall satisfaction with the course. With these proposed interpretations in

mind, the evidence for validity can now be described and evaluated. Since the ATI, CoI, and

satisfaction items were drawn from existing research, some evidence for validity will come from

reports by the test developers and test users in the literature (AERA et al., 2014) in addition to

evidence generated from the pilot study described in the previous section. Validity evidence from

use of the ATI, CoI, and satisfaction items in the main study will be reported in Chapter 5 along

with other interpretations generated from analysis of the survey response data.

Test Content The first source of validity evidence described by Messick (1995) and the Standards

(2014) relates to the content of the test. The test content must be both representative of the

construct and appropriate for the context. The development of the CoI provides the clearest

evidence of its representativeness in measuring a constructivist learning environment because the

survey items were specifically designed to measure the three presence factors described by the

Community of Inquiry model, which is a constructivist view of learning in online environments

(Swan et al., 2009). The ATI does not have an explicit link to constructivist learning

environments, but the pilot study interviews revealed that the student-centered aspect of the

conceptual change student-focused (CCSF) scale addressed many aspects relevant to a

constructivist learning environment such as the instructor acting as a facilitator and encouraging

active student involvement in discussions and explanations of problem solving. However, the

pilot study interviews also revealed that instructors were concerned that the ATI alone may not

105

fully address all aspects of teaching for a particular course. Since there is evidence for the

possibility of construct underrepresentation when using the ATI, it was used in combination with

instructor interviews and analysis of the course syllabus to provide a more complete description

of an instructor’s approach to teaching.

The original use of the CoI in measuring a constructivist learning environment in online

courses meant that some items were not appropriate for use with students in face-to-face

chemistry courses and necessitated rewording. However, a majority of the items addressed

indicators of a constructivist learning environment that could also be found in face-to-face

courses and were therefore appropriate for the current research context. The ATI was developed

from interviews of university chemistry and physics instructors and was therefore expected to be

appropriate for use in the current research with minimal changes to reflect the US language

conventions. While significant rewording of the ATI items was found to be necessary, the pilot

study interviews also revealed that the items on the information transmission teacher-focused

(ITTF) scale may not be relevant or appropriate for analysis in the current research context since

they did not provide information relevant to the development of a constructivist learning

environment. In contrast, the satisfaction items were developed specifically for use with college-

level chemistry students and are therefore representative of the satisfaction construct and

appropriate for the current research context (Xu & Lewis, 2011).

Response Process The second source of evidence for validity, described as substantive by Messick (1995)

and as evidence based on response process by the Standards (2014) speaks most directly to the

purpose of the pilot study. The results from the pilot study indicate that both the CoI and

106

satisfaction items were generally interpreted as intended, though the sample of students may not

have been fully representative of the entire population of students enrolled in 100-level

chemistry courses. During the pilot study interviews students demonstrated the ability to focus

only on the lecture portion of their chemistry course and had very few problems understanding

what each item was asking, providing evidence that the CoI and satisfaction items were in fact

measuring perceptions of the classroom environment. The notable exceptions to this were item 5,

“The instructor was helpful in identifying areas of agreement and disagreement on course topics

that helped me to learn,” and item 16, “Face-to-face communication is an excellent medium for

social interaction.” As previously discussed, item 5 was reworded to more clearly align it with its

intended function as an indicator of teaching presence related to facilitation of discussions. The

issue with item 16 was that students in the pilot study did not understand the relevance of this

item to the overall purpose of the survey since it seemed to be asking more about their beliefs

than any particular aspect of the classroom. However, item 16 was not removed from the

instrument to maintain comparability with previous administrations of the CoI. The functioning

of this item will be discussed in greater detail in Chapter 4 as part of the interpretation of the

results of administering the survey to a larger group of students in the main study.

Student responses to the two sets of satisfaction items indicated that the semantic

differential satisfaction items appeared to be measuring a more holistic sense of satisfaction with

the course. This understanding is in alignment with their intended interpretation as indicators of

overall student satisfaction both in prior research (Xu & Lewis, 2011) and the current research.

The pilot study with the ATI indicated extensive problems with instructors’ response

processes due to unclear wording of the items and the irrelevance of the frequency scale to items

107

worded more as measures of instructors’ beliefs than actual classroom practices. These problems

resulted in the rewording of 18 of the 22 ATI items to both clarify the items and improve their

ability to be answered on the frequency response scale. For the items that did not require

rewording, the pilot study indicated that instructors were able to answer each item by considering

their approach to teaching a specific chemistry course and were providing answers based on

actual classroom practices. This provides evidence that when the items are clear and appropriate

for the response scale, the instructor response process is in alignment with the intended goals of

the instrument to provide a measurement of classroom practices related to the ITTF and CCSF

scales.

Internal Structure Evidence based on internal structure primarily comes from existing research with the

CoI, satisfaction items, and ATI in which their internal structures were examined with factor

analysis techniques such as EFA and CFA. Evidence for the internal structure of all three

instruments was discussed extensively in Chapter 2 and was the primary reason the instruments

were selected for use in the current research. This evidence remains relevant for the satisfaction

items since they were unchanged in the pilot study. Given the small scale of the pilot study, it

was not possible to examine the results using any factor analysis techniques prior to

implementation of the modified instrument in the main study. The results of using the CoI and

satisfaction items with students in face-to-face chemistry courses in the main study are provide in

Chapter 4 and provide validity evidence based on the internal structure of the instrument in this

new context.

The substantial revisions to the ATI items limits the applicability of previous factor

108

analysis results to the revised version of the instrument. Since the primary research focused on

collecting data from students enrolled in the same class, too few instructors participated to

provide enough ATI responses for factor analysis of the results. Therefore, evidence for the

internal structure of the revised ATI will remain an open area for future research.

Relationships with Other Variables

The fourth type of evidence for validity described in the Standards (2014) is evidence for

relationships with other variables. This encompasses convergent, discriminant, predictive, and

concurrent relationships with other variables in addition to the generalizability of the validity

evidence. Due to the limited use of the semantic differential satisfaction items in the chemical

education literature, the modification of the CoI for use with students in face-to-face chemistry

courses, and the numerous revisions to the original ATI items, there is little evidence for the

generalizability of these instruments beyond the current study. Their use in the main research

study provides some evidence for their generalizability, discussed in Chapter 5, but this source of

evidence will remain minimal until the instruments are used more widely in chemical education

research.

However, some evidence for relationships between these instrument scores and other

variables can be found in the literature. The latent construct of emotional satisfaction measured

by the semantic differential items has been shown to correlate moderately with ACS exam scores

but less well with ACT or SAT math scores providing predictive and discriminant evidence for

the validity of the satisfaction item responses (Xu & Lewis, 2011). Similarly, the teaching

presence and cognitive presence factors on the CoI have been shown to have an influence on

student satisfaction (Joo et al., 2011). The existence of these relationships in the data collected

109

for the main research study will be presented in Chapter 4 and discussed in the context of

validity evidence in Chapter 5.

Research utilizing the ATI with new and first-year chemistry faculty teaching primarily

upper-level undergraduate and graduate chemistry courses at research intensive institutions who

were exposed to evidence-based instructional practices at a two-day summer workshop showed

an increase in CCSF scores one week after attending the workshop (Stains et al., 2015). Since

these evidence-based practices primarily emphasized student-centered teaching techniques, this

provides evidence for a relationship between CCSF scores and a pedagogical training variable.

Additionally, the workshop group was shown to use more student-centered teaching techniques

as measured by classroom observation scores from the Reformed Teaching Observation Protocol

(RTOP) and the Classroom Observation Protocol for Undergraduate STEM (COPUS; Stains et

al., 2015). This provides convergent evidence for the validity of the CCSF scores obtained with

the original form of the ATI. Additional convergent evidence was found in the pilot study based

on the alignment of CCSF scores with instructors’ descriptions of their approach to teaching and

information found in their course syllabi. Further validity evidence from instructor interviews

and syllabus analysis conducted for the main study will be discussed in Chapter 5.

Consequences of Use

The final category of evidence for validity comes from how the test scores will be used.

These consequences for use are related to how the creators of the instrument intended the scores

to be used, any claims that might be made beyond these intended interpretations, and any

unintended consequences of use. For all three instruments, care has been taken to trace the

instrument development back to its theoretical foundations, as discussed in Chapter 2, and ensure

110

that its use in the current research is in alignment with the intention of the creators. This is

especially a concern in using the ATI scores only as a measure of an approach to teaching a

particular course, not an assessment of any particular instructor (Prosser & Trigwell, 2006). The

decision to include instructor interviews in the main research study is in alignment with the

warning that the ATI “is not intended for use in gathering a full, rich self-report of teaching”

(Prosser & Trigwell, 2006, p. 405). The satisfaction and CoI items will only be used in their

intended capacity as indictors of the latent variables of student satisfaction with a chemistry

course (Xu & Lewis, 2011) and latent variables representing three aspects of a constructivist

learning environment (Arbaugh, 2008; Arbaugh et al., 2008; D. R. Garrison et al., 2000). Steps

will be taken to limit unintended consequences of using these instruments by clearly describing

the limitations of score interpretation in the context of the current research and protecting the

identity of all respondents when results are reported.

The constant generation of new evidence for the validity of CoI, satisfaction, and ATI

instrument scores means that the discussion of validity should be ongoing and revisited at every

stage of the research. As a result of the pilot study, there is clear evidence for the validity of both

the CoI and satisfaction instrument scores in the context of the current research. These two

instruments did not require substantial revisions and student interviews provided evidence for

alignment of student responses with the intended use of the instruments to provide indicators of

student perceptions of three aspects of a constructivist learning environment and overall student

satisfaction with the course. The pilot study provided less clear evidence for the validity of ATI

scores for use as the only means with which to determine the degree to which an instructor’s

approach to teaching is aligned with constructivist principles of student-centered teaching.

111

However, the alignment of the CCSF scale with instructor interviews and course syllabi provides

convergent evidence for the validity of this scale as does its use in other chemical education

research. Due to the significant revision of the ATI items, additional validity evidence was

examined after its use in the main research study. Though the CoI and satisfaction items were not

significantly reworded, their use in the main research study also provided an opportunity to

examine additional evidence for the validity of the scores in the context of the current research,

discussed in Chapter 5.

Power Analysis for Sample Size Determination In previous work with the CoI, recommendations for sample size in structural equation

modeling were based on rules of thumb suggesting that a sample size of 200 participants was

adequate for small to medium sized structural equation models with fair to good reliability (D. R.

Garrison et al., 2010; Tabachnick & Fidell, 2007). However, these general recommendations

overlook the importance of statistical power in the determination of sample size (Hancock &

French, 2013). Conducting a power analysis prior to collecting data allowed for the

determination of the sample size necessary to test both overall data-model fit and individual

parameters within the model.

Power analysis calculations assume that the data collected to test the model meet the

assumption of multivariate normality, that the model shows the correct relationships among

variables, and that correct parameters are used in the calculations. Previous research with the

CoI, satisfaction items, SAT and ACT math scores, and ACS exams have indicated that the data

meet normality assumptions (Joo et al., 2011; Shea & Bidjerano, 2009; Xu & Lewis, 2011) so it

was anticipated that the data analyzed for the current research met this assumption as well. Every

112

effort was made to ensure the correctness of the model and the parameters used in the power

analysis calculations. However, since the primary goal of the research was to test the model, it

was impossible to definitively know its correctness prior to collecting and analyzing data.

Overall Data-Model Fit

Power analysis for overall data-model fit was done utilizing the root mean square error of

approximation (RMSEA) fit statistic. As discussed in Chapter 2, smaller RMSEA values indicate

better data-model fit. When conducting power analysis for overall data-model fit an RMSEA

value of 0.05 is typically chosen as the cut-off between acceptable and unacceptable fit (Hancock

& French, 2013). The goal of the overall data-model fit power analysis is to reject a null

hypothesis that the RMSEA = 0.05 in favor of an alternate hypothesis where the RMSEA < 0.05.

The specific value of the RMSEA to be used in the alternate hypothesis must be specified prior

to conducting this power analysis. The RMSEA value chosen for the alternate hypothesis

represents the RMSEA value expected to be obtained when testing the fit between the

hypothesized model and the collected data. An expected RMSEA value of 0.00 indicates a

perfectly specified model and requires the smallest sample size to obtain the desired power

because the model perfectly represents the observed relationships in the data, but it is highly

unlikely that the data collected will have a perfect fit with the hypothesized model. At the other

extreme, an alternate RMSEA value of 0.04 is a much more conservative estimate of data-model

fit, but requires a much larger sample size to obtain the desired power due to the mismatch

between the relationships in the data and the model. Hancock and French (2013) suggest setting

the RMSEA value equal to 0.02 for the alternate hypothesis to provide a balance between

“unrealistic optimism” and “impracticality” (p. 127).

113

In addition to selecting an alternate RMSEA value, desired values for statistical power and

alpha level must be selected and the degrees of freedom for the model being tested need to be

determined. For this analysis the power was set to 0.80 and alpha was set to 0.05, as is typical in

hypothesis testing (Hancock & French, 2013). Calculating the degrees of freedom (df) of the

hypothesized model required knowing the number of unique variances and covariances of the

measured variables (u) and the number of model parameters (t) so that the number of model

parameters can be subtracted from the number of unique variances and covariances. This results

in the formula for degrees of freedom shown in Equation 1.

45 = 7 − 9 (1)

The number of unique variances and covariances can be determined from the number of

measured variables (p) in the model using the formula in Equation 2 (Mueller & Hancock, 2008).

7 =

:(: + 1)

2 (2)

For this research, the measured variables are the 36 items on the revised version of the CoI, the

four student satisfaction items, and the outcome variables of math ability, ACS exam scores, and

final course grades. This is a total of 43 measured variables, p, and following from Equation 2,

there must be a total of 946 unique variances and covariances.

To determine the value of t, it is helpful to construct the primary model used in this

research with all model parameters included and labeled. This full model is shown in Figure 13.

The model in Figure 13 is based on the model in Figure 10, provided in Chapter 2 on p. 68, but

includes all the hypothesized relationships among measured variables, latent variables, and

error/residual terms. The residual terms are called errors for measured variables and disturbances

114

Figure 13. The primary hypothesized structural equation model with all parameters labeled. Focal parameters of interest to this research are marked with asterisks.

115

for latent variables. The variables and error terms are labeled using the Bentler-Weeks labeling

convention and the model parameters are labeled using the abc system developed by Hancock &

Mueller (2006). A summary of the notation used in the model is provided in Table 2.

Table 2 Notation Used in Full Structural Equation Model in Figure 13

Symbol Meaning Rectangle Measured variable Oval Latent variable V Measured variable label F Latent variable/factor label E Error term/measured variable residual D Disturbance/latent factor residual bto,from Path between two variables where subscripts indicate where the path ends and

where it originated c Variance for a single variable or covariance between two variables T Nonstandard notation, used in this model to indicate a teaching presence item S Nonstandard notation, used in this model to indicate a social presence item C Nonstandard notation, used in this model to indicate a cognitive presence item SS Nonstandard notation, used in this model to indicate a student satisfaction item

Focusing first on the paths between variables (b) there are 36 paths representing survey

items loading on their respective factors and 13 paths between the factors and the measured

student variables. This comprises a total of 49 paths. Four paths between a survey item and its

respective factor have been fixed to 1 as a way to provide scale for the factor and are therefore

not parameters that will be estimated. There are a total of 48 variance and covariance

terms (c) representing the variances and error/disturbance variances of the 43 measured variables

and the four latent variables in addition to the covariance term between student satisfaction and

final course grades. Summing the 49 paths and 48 variances/covariances gives a total of 97

model parameters that must be estimated. Substituting the calculated values of u and t into

Equation 1 results in 849 degrees of freedom for the model in Figure 13.

Using the value of 849 for degrees of freedom, three power analyses were conducted in

116

order to estimate the sample size necessary to reject a null hypothesis of unacceptable data-

model fit (RMSEA = 0.05) with a power of 0.80 and alpha = 0.05. These calculations were

performed using a web-based power calculator for RMSEA (Preacher & Coffman, 2006). The

results were rounded up to the next whole number. The first calculation was done with the

alternate RMSEA value set at 0.00 to provide the most optimistic determination of sample size,

which was 53 participants. The second calculation was done with the alternate RMSEA value set

at 0.04 to provide the most conservative determination of sample size, which was found to be

178 participants. Finally, the calculation was done with the alternate RMSEA value set at 0.02 to

provide the most realistic determination of sample size (Hancock & French, 2013), which was

found to be 64 participants. These results are summarized in Table 3.

Table 3 Sample Sizes for Testing Overall Data-Model Fit

Alternate RMSEA value Resulting sample size 0.00 53 0.02 64 0.04 178

Note. For each analysis power = 0.80, alpha = 0.05, df = 849, and the null RMSEA = 0.05

Testing Parameters within the Model

Additional power analyses were conducted to determine the sample sizes necessary for

testing individual parameters within the model. While the whole model contains 97 parameters

that must be estimated, not all of these parameters are important in the context of the current

research questions. The primary purpose of the research is to examine relationships among the

three CoI presence factors (teaching presence, social presence, and cognitive presence), the

student satisfaction factor, and the measured variables of math ability, ACS exam scores, and

final course grades. These relationships include the 13 directional paths between measured

117

variables and factors as well as the covariance term between student satisfaction and final course

grades. In total, there are 14 parameters that can be considered the focal parameters for this

research. These are the same parameters explicitly shown as arrows in Figure 10 (p. 68) and

indicated by asterisks in Figure 13. The other parameters are termed peripheral parameters

(Hancock & French, 2013). The peripheral parameters will be estimated as part of the model, but

their values are not as important for addressing the primary research question.

Selection of model parameters

The steps outlined by Hancock & French (2013) were followed to calculate the sample

size necessary to determine if each focal parameter is non-zero with acceptable power. As with

the overall data-model fit power analysis, for each of the 14 single parameter tests the decision

was made to set power = 0.80 and alpha = 0.05. Next, numerical values were selected for all

parameters in the model, both focal and peripheral. The numerical values selected were all

standardized paths or correlations between variables, since these are most frequently available in

the published literature. In obtaining numerical values from the literature, preference was given

to path values obtained as a result of SEM analysis rather than correlations. This preference was

given because the path values that result from SEM analysis represent the magnitude of the

direct relationship between two variables after controlling for the effects of other variables while

correlations include the effects of other variables and are therefore less directly applicable to

being used as estimates for path values in the current research model.

Path values obtained from SEM analyses with the CoI included the loadings of the original

34 individual CoI items onto their respective factors (Arbaugh, 2008; Arbaugh et al., 2008, 2010;

D. R. Garrison et al., 2010; Shea & Bidjerano, 2009) and the paths among the three CoI presence

118

factors (D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). A table of the path

values obtained from the CoI literature can be found in Appendix A. Standardized paths to

satisfaction from teaching presence and cognitive presence were available from Joo et al. (2011),

but due to the removal of the path between satisfaction and social presence in the Joo et al.

analysis, a literature value could not be obtained for this path and instead the correlation between

social presence and satisfaction from Arbaugh (2008) was used.

Path values for the satisfaction items loading on the satisfaction factor were obtained from

the CFA performed by Xu & Lewis (2011) showing the loading of the semantic differential

satisfaction items on the emotional satisfaction factor. The covariance term between the residual

terms for final course grades and the satisfaction factor was estimated from SEM analysis by

Greenwald & Gillmore (1997) showing the relationship between an expected course grade factor

and a course evaluation factor. In situations where multiple analyses had been conducted and

multiple literature values were available for a path, the smallest value was chosen in order to

provide the most conservative estimate for the path value.

The value for the path between math ability and ACS exam scores came from the

standardized regression coefficient for SAT math scores presented in Lewis & Lewis (2005),

which was smaller than, but similar to, the correlations between ACT math, SAT math, and ACS

exam scores found in Xu & Lewis (2011). The relationship between math ability and final course

grades in 100-level chemistry courses was chosen from the lowest correlation found in the

literature (Craney & Armstrong, 1985; Nordstrom, 1990). The values for the paths among

teaching presence, cognitive presence, ACS exam scores, and final course grades are unique to

this research and were not available from the literature. Therefore, these paths were estimated

119

based on correlations among teaching presence, cognitive presence, and students’ perceived

learning in online MBA courses (Arbaugh, 2008). Finally, the value for the path between ACS

exam scores and final course grades was estimated using the average weight of final exam scores

based on the syllabi provided by the instructors participating in the pilot study interviews. This

decision was made based on the fact that ACS exams are often used as final exams (Lewis &

Lewis, 2005). All of the standardized values that were selected are provided in Table 4 along

with the results of the power analysis for testing model parameters.

Simplification of model-implied correlation matrix

Instead of using all 43 measured variables shown in the model in Figure 13 to calculate

the model-implied correlation or variance/covariance matrix the model was conceptualized as

containing seven latent variables instead of only four. This change allowed the model-implied

matrix to consist of only seven “pseudo” variables each loading on a single factor instead of 43

measured variables. The three new latent variables are the three achievement variables of math

ability, ACS exam scores, and final course grades each forced to load on a single latent variable

with a fixed loading of 1 and a fixed error variance of zero. In this way the new latent variables

are mathematically equivalent to the measured variables, but can be treated as latent

variables in analyses. These “dummy” latent variables are labeled “F5” (math ability), “F6”

(ACS exam scores), and “F7” (final course grades) in the model shown in Figure 14.

The four preexisting latent variables of teaching presence, social presence, cognitive

presence, and satisfaction were conceptualized a having a single measured variable loading on

each latent variable. This single measured variable serves as a placeholder for the multiple

measured variables loading on each factor. These placeholder variables are labeled “V1”, “V2”,

120

Figure 14. The primary research model conceptualized with all focal parameters existing between latent variables.

121

“V3” and “V4” in the model shown in Figure 14 and represent a combination of the individual

survey items loading on the teaching presence (“V1”), social presence (“V2”), cognitive

presence (“V3”), and satisfaction (“V4”) factors. The values for the loadings of these placeholder

variables and their error covariance terms are calculated using coefficient H, a value representing

the construct reliability for each factor calculated from the standardized loadings for each of the

measured variables previously loading on that factor (Hancock, 2001).

The formula for coefficient H is shown in Equation 3, where ℓ represents the

standardized loading for each of the measured variables (k) loading on a particular factor.

A =

1

1 +1

ℓB

"

(1 − ℓB

")

C

B-D

(3)

The calculation for HF4 is shown in Equation 4 using the loadings for each of the four semantic

differential satisfaction items provided in Xu & Lewis (2011).

AF4 =

1

1 +1

0. 74"

(1 − 0.74")+

0. 77"

1 − 0.77"+

0.83"

(1 − 0.83")+

0.48"

(1 − 0.48")

= 0.838 (4)

The value of HF4 determined from Equation 4 was then used to determine the values for the path

from F4 to “V4” and the error covariance for “E4” according to the formulas shown in Figure 14.

Similar calculations were performed for the three CoI presence factors utilizing a small function

written by the researcher for use with the statistical software R version 3.1.1 (R Core Team,

2014). The function takes the loadings as inputs and outputs a value for coefficient H. The code

for this function is available in Appendix F.

122

Construction of the model-implied correlation matrix

With values obtained for all the necessary model parameters, the next step of the power

analysis involved constructing the matrix containing the model-implied numeric relations among

the seven latent variables depicted in Figure 14. Since standardized values were used for the

parameter values, this matrix functions as a correlation matrix with values of 1 on the diagonals.

The mathematical determination of the matrix was done using path tracing rules to construct the

algebraic formulas that show how the total correlation between two variables results from

relationships between these two variables and other variables in the model. In other words, the

correlation between two variables is the sum of all possible paths between the two variables that

obey the path tracing rules developed by Wright (1934). In tracing the possible paths between

two variables any number of forward arrows (à) can be passed through, moving from tail to tip.

As an example, in Figure 14 there is a direct relationship between “F5” and “F7” represented by

the path bF7F5. There is also an indirect relationship between “F5” and “F7” through “F6”, this

trace would be represented as the product of the two arrows traveled along when moving from

“F5” to “F6” and “F6” to “F7”, represented as bF6F5bF7F6.

The path tracing rules allow for backwards movement along arrows from tip to tail (ß),

but once an arrow has been passed through in the forward direction, backwards movement is no

longer allowed. Additionally, each variable can only be passed through once in a particular trace.

Finally, only one two-headed arrow can appear in any trace. Since there are no more allowed

ways to move between “F5” and “F7” the total correlation between the two factors must be equal

to the sum of the two traces, that is bF7F5 + bF6F5bF7F6. However, the matrix is constructed to show

the model-implied correlations between variables, not factors. Therefore, an additional step was

123

added to each trace so that it proceeded from V5 (math ability) to V7 (final course grades).

Examination of Figure 14 shows that both traces now start with backwards motion through the

path connecting V5 to “F5” and proceed with the forward motion through the arrows previously

described, ending with forward motion from “F7” to V7. Since the paths connecting V5 to “F5”

and “F7” to V7 were fixed to 1, multiplying each trace by 1 twice will have no impact on the

numerical result obtained once the previously determined parameter estimates are substituted in

for bF7F5, bF6F5, and bF7F6.

A similar procedure was followed when starting or ending a trace with the composite

variables “V1” through “V4”, but in these cases the path between the factor and the composite

variable is equal to the square root of coefficient H for that factor. This path is equal to the

square root of coefficient H because coefficient H represents the construct reliability for that

factor, and the square root of the reliability is equal to the standardized loading for a single

indicator variable loading on that factor (Hancock & French, 2013). This value must be included

in the algebraic determination of each trace since it is not equal to 1 and will therefore affect the

calculated value for each trace.

Application of the path tracing rules described above resulted in the development of a

series of algebraic statements into which the previously estimated model parameters could be

substituted in order to calculate a model-implied correlation matrix for the seven variables

depicted in Figure 14. The full list of algebraic statements is available in Appendix G along with

the R code used to generate the matrix. Since there were no allowed ways to get from V5 to

“V1”, “V2”, “V3”, or “V4”, these correlations were set to zero.

Executing this code with the parameter estimates from the literature resulted in a

124

correlation matrix with values greater than one. Even though correlations greater than one are

impossible, this result was obtained because many of the literature values chosen for paths were

correlations and therefore overestimated the direct relationships between two variables when

substituted into the algebraic statements. Each correlation value obtained from the literature was

revised downward by an interval no greater than 0.05 until the matrix no longer contained values

greater than one. The resulting matrix is shown at the end of Appendix G. After obtaining this

matrix, its determinant was calculated to ensure it was positive since a non-positive definite

matrix would cause the software to be unable to run in later steps of the analysis. If a negative

determinant had been obtained, the literature correlations would have again been revised until the

matrix had a positive determinant (G. R. Hancock, personal communication, August 7, 2015).

Calculation of model fit function values

The final stage of the power analysis for testing individual model parameters utilized

structural equation modeling software to calculate the model fit function value (FML) each time a

focal parameter was constrained to zero. Constraining a parameter to zero is the mathematical

equivalent of removing it from the model, causing the fit of the model to degrade. For all models

tested in this stage, the sample size was arbitrarily set to 1001 because the actual sample size was

unknown but the modeling software required a sample size to perform the model fit calculations

(Hancock & French, 2013). The software used for this stage of the analysis was LISREL 9.10

Student Edition (Jöreskog & Sörbom, 2015).

Prior to constraining each focal parameter to zero the model-implied correlation matrix

was provided to LISREL along with relationship statements describing the model in terms of

relationships among variables and factors and setting path values and variance terms equal to the

125

values obtained from the literature and coefficient H calculations. The purpose of this step was to

check that the algebraic path tracing had been done correctly by showing that the data provided

in the model-implied correlation matrix exactly matched the model described by the relationship

statements. The result of this step was perfect data-model fit (χ2 = 0.00, p = 1.00), showing that

the specified model fit the provided data. Additionally, a visual inspection of the path diagram

output from LISREL indicated that the model had been specified correctly and matched the

model in Figure 14. The LISREL syntax used for this step and the corresponding text output are

provided in Appendix H.

With the accuracy of the model-implied correlation matrix generated by R and the

relationships statements entered as LISREL syntax confirmed, each one of the 14 focal

parameters was constrained to zero in turn. As a result, data-model fit results were obtained for

LISREL models where each model had a single focal parameter constrained to zero. Though 14

total models were run, two resulted in fatal errors in LISREL so only 12 sample size values were

obtained. The two missing sample size values were not expected to indicate the need for sample

sizes larger than the 12 obtained because the obtained sample sizes were for testing focal

parameters representing the largest and smallest hypothesized paths. As expected, constraining

each focal parameter to zero degraded the fit of model. This poor fit can be seen in the FML

values obtained for each model in which a focal parameter was constrained to zero (FML(θR)).

To calculate the sample size necessary to test each focal parameter, the value of the

noncentrality parameter (λ) must also be obtained. The noncentrality parameter follows a χ2

distribution and is determined based on the power level, alpha level, and degrees of freedom for

each test (Hancock & French, 2013). In this case of testing individual model parameters, the

126

degrees of freedom are equal to 1, the power level was set at 0.80 and alpha was set equal to

0.05. Using the table in Hancock & French (2013), λ was determined to be 7.85. The minimum

sample size necessary to test each focal parameter with power = 0.80 was then calculated using

Equation 5. The results of the twelve sample size determinations are provided in Table 4.

L = 1 +

M

NML(θR)

(5)

Table 4 Sample Size Necessary to Test Each Focal Parameter Arranged from Largest to Smallest

Focal parameter(s)

Standardized value used in correlation matrix

Minimum fit function value (FML(θR))

Sample size

bF3F2 0.30 0.103 78

bF6F1 0.30 0.147 55

bF6F3 0.35 0.194 42

bF2F1 0.52 0.231 35

bF3F1 0.49 0.261 32

bF6F5 0.414 0.292 28

bF4F2 0.30 0.320 26

bF7F6 0.18 0.349 24

bF7F1 0.30 0.712 13

bF7F3 0.35 0.870 11

bF7F5 0.414 1.171 8

cF4F7 0.38 1.233 8

bF4F3 and bF4F1 0.26 and 0.24 Fatal error obtained

Note. For each model, power = 0.80 and alpha = 0.05, and λ = 7.85

Examining the results of applying Equation 5 to the 12 FML(θR) values shows that as the

values of FML(θR) decreased the sample size necessary to detect each focal parameter increased.

This was the anticipated result because a smaller FML(θR) value for each parameter indicates that

127

constraining the parameter to zero, which is equivalent to removing it from the model, does not

influence data-model fit to a large degree. When the focal parameter has a smaller influence on

the model fit, more participants would be necessary to achieve a power of 0.80. The path from

social presence to cognitive presence (bF3F2) had the smallest FML(θR) value (0.103) and therefore

requires in the largest sample size (n = 78) to ensure it is tested with a power of 0.80 when alpha

= 0.05. The end result of this power analyses for model parameters is that a sample size of at

least 78 participants was targeted in order for all focal parameter tests of significance be

conducted with power = 0.80. A minimum sample size of 78 participants also exceeds the

sample size necessary to test overall data-model with power = 0.80 when the alternate RMSEA

value was set at 0.02.

Methodology

The main research study utilized a mixed methods approach in which both quantitative

and qualitative data were used answer the research questions (Creswell, 2014). A mixed methods

approach was chosen in order to minimize some of the limitations of a purely quantitative

methodology by integrating qualitative data to provide a more complete understanding of the

learning environment. Specifically, an embedded design was chosen for this research.

In an embedded design, one type of data collection is embedded within the larger

collection of a different data type (Creswell & Clark, 2007). For this research, the primary data

were quantitative data obtained from anonymized student responses to the CoI and satisfaction

instrument in addition to instructor responses to the ATI. Additional quantitative data were

student achievement measures: math ability scores on the math portion of the ACT (SAT scores

were not available), ACS exam scores, and final course grades. The qualitative portion of the

128

research consisted of short, semi-structured instructor interviews and analysis of the course

syllabus.

In an embedded design, the timing of the two types of data collection can be either

sequential or concurrent (Creswell & Clark, 2007). The collection of instructor data in this study

was concurrent. The instructor first completed the ATI and then participated in the semi-

structured interview. The timing of the instructor data collection and student data collection was

sequential. The student data analyzed in this research were from an existing data set collected by

another chemical education researcher for a separate project investigating predictors of student

success in general chemistry. The student data were collected prior to conducting instructor

interviews.

The student data were collected from six sections of a first semester general chemistry

course taught by four different instructors at a large, public, primarily undergraduate institution

(Indiana University Center for Postsecondary Research, n.d.). Though the student data can be

considered as grouped by classroom, the selection of classrooms was based on availability not

random sampling as would have been necessary for the use of a data analysis technique such as

hierarchical linear modeling (Huta, 2014). A total of 439 students are represented in the data set,

but complete data were not available for all students. The data set contained numeric responses

to the modified CoI and satisfaction survey instrument in addition to the number of items

answered correctly on the first-semester ACS general chemistry exam (Form GC15FG), final

course grades as a percentage of points earned after excluding laboratory scores, and ACT math

scores. Students completed the survey instrument during a regular class period using Scantron®

forms to record their responses. The survey was administered by the course instructor within two

129

weeks of the end of the semester during the normal end of course evaluation period for the

university.

To accommodate the Scantron® forms used at the participating institution the numeric

responses 1–5 (Strongly Disagree to Strongly Agree) were replaced with letters A–E and the

directions were modified slightly by the course instructor. The version of the instrument used at

the participating institution is provided in Appendix I. Student grades and ACT math scores were

not matched to survey responses by the course instructor until after final course grades had been

submitted. The collection and anonymization of this data was undertaken with institutional

review board (IRB) approval as an exempt research study by the university where the data were

collected. IRB permission from this researcher’s institution and IRB permission from the

institution where the data were collected were granted approving the use of the anonymous

student data set in the current research study.

Instructor data collection began by obtaining IRB permission from the home institution of

the researcher to conduct instructor interviews and collect ATI responses. In addition, consent

was obtained from the chair of the chemistry department where the student data were collected to

contact the instructors to invite them to participate in the current research. The four instructors

who taught the six sections of the course were invited participate in the research the semester

after teaching the class from which student data were collected. Each instructor completed the

revised ATI (items listed in Appendix D) as an online survey administered through Qualtrics

(http://www.qualtrics.com/) and responded to semi-structured interview questions during a video

conference. All four instructors consented to participate in the research.

Unlike the pilot study, the instructors were not asked to follow a think-aloud protocol to

130

respond to the ATI. After completion of the ATI, the instructors participated in the semi-

structured interview. With consent of the instructors, the interviews were audio recorded. The

interview questions were designed to provide the instructor with an opportunity to address any

issues that may have occurred when responding to the ATI and to allow the instructor to describe

his or her approach to teaching while providing details about the course. The questions used

during the semi-structured interviews are provided in Table 5. The researcher also requested a

copy of the syllabus of the introductory undergraduate chemistry course taught by the instructor

in which the students who provided data were enrolled.

Table 5 Semi-Structured Instructor Interview Questions

Question1. Are there any responses on the ATI you would like to provide an explanation for or

discuss in more detail? 2. In your own words, please describe your approach to teaching the course you used as a

reference for completing the ATI. For example, how do you decide what topics to cover or how to structure a class period?

3. In your own words, please describe how you prepare for a typical meeting of the course you used as a reference for completing the ATI.

4. Please describe the course you used as a reference for completing the ATI. For example, how long have you been teaching this course, what kind of topics are covered, and what type of students are enrolled in the course (chemistry majors, non-majors, etc.)?

Data Analysis

Qualitative Data Analysis

Analysis of the qualitative data collected for this research began with transcription of the

recorded instructor interviews. A second chemical education graduate student researcher listened

to a subset of the recorded interviews and spot checked the transcription both for accuracy and to

ensure that the semi-structured interview questions did not lead the instructor (Creswell, 2013, p.

131

259). Analysis and coding of the transcript and course syllabus used a major theme of student-

centered teaching practices consistent with constructivism.

Coding of the interview transcripts and course syllabi was done by hand using a rubric

developed specifically for this research, provided in Appendix J. The codes used in the rubric

were based on the focus of the interview and indicators of a constructivist learning environment.

These codes included (1) incorporation of student learning activities to replace a portion of

lecture time, (2) the role of the instructor as a facilitator, (3) instructor conceptions of student

learning, (4) use of group work, (5) emphasis on student understanding of concepts, (6)

incorporation of authentic problem solving tasks, (7) students being asked to test or apply

knowledge, and (8) use of discussions to probe student understanding.

To ensure validity of the coding process a second chemical education researcher coded

all the interview transcripts and syllabi using the same rubric. This chemical education researcher

was a faculty member familiar with constructivism but not directly involved in the current

research. Before beginning the coding process, the operationalized definition of a constructivist

approach to teaching utilized in this research was discussed with the researcher. This definition

of a constructivist approach to teaching describes an approach that is more student-centered and

shifts the role of the instructor from a lecturer to a facilitator for at least part of the instructional

time. The coding scheme was discussed to ensure both coders interpreted the codes in the same

way. After both coders had applied the rubric to each set of instructor data, scores were

compared to ensure consistency in interpretation of the indicators. If necessary, interpretations

were clarified and each coder had an opportunity to revise the score that had been assigned. After

coding all four sets of instructor data, the percent agreement of the two coders was 94%.

132

As part of the coding process both coders identified quotes from the interview transcripts

or course syllabi that provided evidence for the specific codes representing student-centered

constructivist approaches to teaching. Similarities in selection of quotes by both coders were

used to provide additional evidence for the validity of the coding. This qualitative data was

analyzed along with the results of instructor responses to the ATI and student responses to the

CoI survey in order to address the first research question.

Quantitative Data Analysis

Data cleaning Analysis of the quantitative data collected for this research began by checking the

instructor survey, student survey, and student achievement data to look for obvious errors in data

entry or otherwise corrupted data. This checking process included determining the minimum and

maximum value for each of the measured variables to ensure it was within an acceptable range.

For example, both instructor and student survey item responses were expected to be whole

numbers between 1 and 5. All four instructor survey responses were within the correct range. In

the student data set survey response values of 9 were present, these indicated missing values as

reported by the scanning software used at the institution where the data were collected.

Additionally, a few students had survey response values of 6 (F) which were from students who

were not paying attention to the fact that the scale ended at 5 (E). Both values of 9 and 6 were

recoded as missing data.

In the student data set, it was anticipated that the final course grade data would be a

percentage score between 0 and 100 but one student earned a final course grade of 100.02% and

this value was left in the data set. The range for the math ability score was expected to be

133

between 1 and 36 since ACT math scores (ACT, 2015) were present in the student data set. All

ACT math values in the data set fell between 16 and 35. The raw ACS exam scores could range

from 0 to 70 and the student data set contained values from 14 to 63.

The student survey responses were checked to see if any student did not select “disagree”

for item 29 as instructed (Please select “Disagree” for this item). Any student not selecting

“disagree” for this item had his or her data excluded from further analysis; a total of 17 students

were removed from the data set based on this criterion. Additional data cleaning involved

removal of 15 students from the data set for whom no course grade was available since these

cases represented students who withdrew from the course before the end of the semester. Finally,

an additional 16 students were removed from the data set because they were missing both ACS

exam scores and CoI responses. Missing the ACS exam score meant these students did not take

the final exam for the course and missing CoI responses meant that these students were also not

present on the day that the CoI was administered. These were likely students who did not

officially withdraw from the course but were no longer actively attending class meetings. This

interpretation is supported by the low course grades for these 16 students, ranging from 0.23% to

50.93%.

As a result of these data cleaning steps, the number of usable student participants was

391. The 391 responses represent approximately 89% of the total 439 responses collected.

Exclusion of 11% of the initial sample still provided five times the minimum sample size

necessary to examine the overall data-model fit and model focal parameters with sufficient

power. No instructor survey responses were excluded from the analysis since all four instructors

responded to all 22 items on the modified ATI survey instrument.

134

Of these 391 cases, only 307 students had complete data on all variables. Course grade

was the only variable for which all students had complete data. The missing data rate for ACS

exam scores was 1.5% and 6.6% for ACT math scores. Between 12% and 14% of students were

missing data on one or more of the survey items. More detailed information about the missing

data rate for each survey variable is provided in Chapter 4.

The next step of the quantitative data analysis was to look at outliers in the student data

set. Outliers were not examined in the instructor data set since it only contained four responses.

Univariate outliers were examined for the three continuous student variables: ACS exam scores,

ACT math scores, and course grades. Univariate outliers were considered to be any student with

scores more than three standard deviations above or below the average on any of the three

continuous variables. No outliers were identified for the ACS exam scores or ACT math scores.

Eight students were identified as having course grades three standard deviations below the

average due to earning less than 40% of the course points with laboratory points excluded.

However, since these 8 students only represented about 2% of the student data and a robust

estimation technique not requiring the assumption of normality was used in the SEM analyses

these negative outliers were left in the student data set.

Multivariate outliers were examined by calculating the Mahalanobis distance based on

the 40 student survey variables and three academic variables. Using the cutoff value of 77.42 for

a chi-square test with p = 0.001 and df = 43 there were 18 multivariate outliers identified. One

issue with identifying multivariate outliers using the Mahalanobis distance is that only

individuals with complete data on all 43 variables could be considered in the analysis whereas

the SEM analyses were conducted with all cases using a technique robust to missing data. For

135

this reason, and because the 18 cases represented less than 5% of the data set, the multivariate

outliers were not removed from the student data set.

Assumptions for CFA and SEM analysis

The typical estimation technique employed when performing CFA and SEM analysis is

full information maximum likelihood (FIML), often abbreviated as simply ML. This estimation

technique is robust to missing data and does not require deletion of incomplete cases or

imputation of missing values, unlike older statistical techniques used to deal with missing data

(Enders, 2013). Instead, iterative algorithms are used to find the most likely values for the

missing data given the assumption of multivariate normality and the relationships among other

values in the data.

The use of ML assumes that the missing data are random. This could mean either missing

completely at random (MCAR) or missing at random (MAR). For MCAR, missing values for a

variable, such as a response to a survey item, must not be missing because of what the value

would have been, such as a low rating, or because of a relationship to other variables in the

model, such as a low course grade. For missing at random (MAR), values must not be missing

because of what the value would have been, but may be missing because they depend on the

value of another variable. It is difficult to demonstrate MCAR, but MAR may be demonstrated if

examination of the data reveals that the missing survey data may be more likely for students with

high or low scores on academic achievement variables. If another variable looks like it may be

responsible for the missing data, this variable must be included in the model as an auxiliary

variable. As an auxiliary variable it must be allowed to correlate with the error variance terms for

the other measured variables in the model (Enders, 2013).

136

To check if the missing survey responses met the assumption of MAR, three separate

independent t-tests were conducted with one of the three academic variables (ACT math, ACS

exam, and course grade) as the dependent variable. The two groups being compared were

students with complete responses to all items on the survey instrument and students missing one

or more responses to survey items. The results of these t-tests, presented in Chapter 4, indicated

that students with incomplete responses to the survey items had statistically significantly (p <

0.05) lower scores on the academic variables than students who responded to all the survey

items. As a result, ACT math scores, ACS exam scores, and final course grades were included as

auxiliary variables in all analyses involving only the CoI instrument. Since the three academic

variables were included in the full research model used for the SEM analysis, they did not need

to be modeled as auxiliary variables in that analysis.

The responses to the survey were given on a five-point scale, and therefore they are

considered ordinal data and are inherently not normally distributed. However, research has

shown that with at least five scale points, ordinal data following an approximately normal

distribution can be treated as normally distributed continuous data for ML estimation and no

major distortion is seen in model-fit indices (Finney & DiStefano, 2013). However, slight

underestimation may occur for loadings and standard errors leading to an increase in Type I error

(Finney & DiStefano, 2013). If the categorical data are not approximately normal, this may lead

to an increase in the !" value for the model and a decrease in the CFI, increasing the likelihood

of finding the model to have unacceptable fit (Finney & DiStefano, 2013).

Skew of the survey items was checked by visually examining histograms created for each

of the survey variables and by calculating a z-score for skewness equal to the absolute value of

137

the skewness value divided by the standard error of the skewness value. All survey items except

one were found to have nonnormal distributions and skewness z-scores greater than 1.98

indicating a skewness value significantly different from 0 (p < .05) due to a majority of students

selecting responses of Agree and Strongly Agree to the CoI items and positive responses to the

satisfaction items. Descriptive statistics for the student variables are provided in Chapter 4.

As a response to the categorical nature of the data and the nonnormal distribution of the

survey responses a correction factor was applied to all CFA and SEM analysis. The Satorra–

Bentler correction is a commonly used robust estimation technique that corrects the model !"

value and standard errors but does not adjust parameter estimates since these are generally robust

to nonnormality (Finney & DiStefano, 2013). However, use of the Satorra–Bentler correction

requires either a complete data set or the use of listwise deletion to create a complete data set.

The limitations of LISREL made it impossible to deal with both the missing data and the

nonnormal distribution of the data simultaneously. Instead of reducing the sample size by using

the Satorra-Bentler correction and listwise deletion, the MLR estimator was used in Mplus

(version 7.0). The MLR estimator also corrects the model !" value and standard errors while not

adjusting parameter estimates, but MLR can be used with missing data (Muthén & Muthén,

1998-2015). Therefore, MLR was chosen to provide scaled estimates of model fit and parameter

significance for all CFA and SEM analysis.

Internal structure of the CoI instrument

The second research question was addressed by examining the internal structure of the

CoI instrument with a series of confirmatory factor analyses. First, a CFA was conducted with

the 13 teaching presence items to determine whether a single factor or two-factor model was a

138

better fit for the latent variable of teaching presence (Figures 8 and 9, p. 60). As discussed in

Chapter 2, some research with the CoI had indicated that the teaching presence factor may be

better described as two correlated factors instead of a single factor. In the two factor model one

factor, which items 1–4 were expected to load on, relates to pre-course activities performed by

the instructor that are typically accomplished in the syllabus or communicated via

announcements (Arbaugh, 2007; Shea et al., 2006). The second factor, which items 5–13 were

expected to load on, relates to in-course instructor activities such as facilitating discussions and

providing feedback.

These two models can be described as nested because the parameters of one model are a

subset of the parameters in the other model. In the case of the two competing models for teaching

presence, the single factor model can be considered as nested within the two-factor model

because the single factor model is a special case of the two-factor model where the correlation

between the two factors is constrained to 1. Due to this relationship, the two models can be

statistically compared by looking at the difference between their !" values and their respective

degrees of freedom (Mueller & Hancock, 2008). However, due to the use the MLR estimator

which produced a scaled !" value, the difference in !" values must be examined in the context of

the scaling correction factor. Each of the two models has its own scaling correction factor that is

simply the ratio of the uncorrected !" value to the corrected !" value. The difference test scaling

correction (cdiff) can be calculated from the scaling correction for each model and the degrees of

freedom for each model, shown in Equation 6 where the subscript 1 indicates the nested one-

factor model and the subscript 2 indicates the full two-factor model.

Odiff =

OD 45D − O" 45"

(45D − 45")

(6)

139

A !" difference test can then be conducted by subtracting the uncorrected !" value of the full

model from the uncorrected !" value of the reduced model and dividing by the difference scaling

correction. This can be seen mathematically in Equation 7. The degrees of freedom for the

∆!

dif

"=

!D

"− !

"

"

Odiff

(7)

resulting !" value are the difference in the number of degrees of freedom of the two models. The

result of conducting the !" difference test indicated that the two factor model of teaching

presence was a better fit for the data (!+,-D

"= 15.695, p < 0.001), and that an error covariance

term should be added between item 12 and item 13. The results of this analysis and specific

details of the !" difference calculation are presented in Chapter 4 and the syntax used to run the

models is provided in Appendix K. Since the in-course factor is most directly related to the

intended construct of classroom teaching practices additional analysis with the CoI instrument

only used items 5-13 as indicators of the teaching presence factor. The fit for this smaller nine-

item teaching presence model with three academic auxiliary variables and the error covariance

between item 12 and item 13 was acceptable (!scaled,+,-"1

"= 79.902; CFIscaled = 0.960;

RMSEAscaled = 0.073, CI90=[0.055, 0.091]; SRMR = 0.046) using the joint criteria of CFI ≥ 0.96

and SRMR ≤ 0.09 when the RMSEA is not considered (Hu & Bentler, 1999).

Next, a CFA was conducted to test the data-model fit for the full three factor CoI model

by including the items loading on the social presence and cognitive presence factors. The error

covariance between item 12 and 13 was included along with the three academic auxiliary

variables. After the addition of the error covariance terms between three pairs of satisfaction

items, shown in Figure 15, acceptable data model fit was obtained (!scaled,+,-./0

"= 1028.717;

140

Figure 15. The three-factor model of the CoI survey used to inform the two-phase SEM analysis.

141

CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR = 0.061) based on joint

criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 when the CFI is not considered (Hu & Bentler,

1999). This model of the CoI instrument provided an answer to the second research question,

discussed in more detail in Chapter 5, and was the model of relationships among CoI variables

used in the analysis of the full research model.

Average scale scores and scale reliability

After determining the item loadings for the three CoI factors, the next stage of the

analysis was to determine average scale scores for both CoI and ATI instruments. Average scale

scores for each of the three types of presence were determined by first computing an average

teaching, social, and cognitive scale score for each individual. The average scale score for each

individual was computed by adding the numeric responses for each item on a particular scale and

dividing by the total number of items on that scale. Then, the overall average scale score was

computed by adding all the individual scale scores and dividing by the total number of

individuals. For this analysis, only individuals with complete data for all CoI items were

included in the calculations. A similar analysis was undertaken for the satisfaction items, after

reversing the coding of the last item so that all four items had the positive response associated

with the number 1 on the scale. On the ATI only items representing the CCSF scale were

analyzed since the pilot study had indicated that only the CCSF scale score would be relevant to

the current research on constructivist learning environments.

Scale reliability was calculated in two ways for each of the three types of presence on the

CoI. Cronbach’s alpha was calculated using the raw student response data and provides

information about the reliability of each of the presence scales when item response values are

142

combined as either averages or sums to create scale scores and can be interpreted as the internal

consistency of responses to the survey items. Coefficient H provides information about the

reliability of the underlying latent factor representing each type of presence and can be

interpreted as the stability of the factor. Coefficient H was calculated, using the formula

previously presented in Equation 3, from the standardized loadings produced as a result of

conducting the three-factor CFA depicted in Figure 15. These results are presented in Chapter 4.

Scale reliability was not determined for the ATI since there were only four instructor responses.

Two-phase SEM Analysis To address the third research question, a two-phase SEM analysis (Mueller & Hancock,

2008) was implemented for analysis of the full model in Figure 13 (p. 114), after modifying it

based on the results of the CoI analysis shown in Figure 15. In the first phase, the measurement

portion of the model was tested. The measurement portion of the model focuses on the

relationships between the measured variables and their respective factors. In the measurement

phase of the analysis, the causal paths among factors are replaced with correlational

relationships. Only if the measurement portion of the model has acceptable data-model fit can

the structural portion of the model be tested in the second phase by replacing the correlational

relationships with the originally hypothesized relationships.

For the measurement phase, the overall research model was conceptualized as having

seven latent variables, similar to the model used in the power analysis for testing model focal

parameters. The first four latent variables are the three presence factors from the CoI plus the

student satisfaction factor. The indicator variables from the student survey are shown loading on

their respective factors in Figure 16. Note that the measurement model shown in Figure 16

143

Figure 16. The CFA model used in the measurement phase of the SEM analysis.

144

reflects modifications to the number of indicator variables for teaching presence and the addition

of error covariance terms as a result of the earlier CFA steps. The final three latent variables are

“dummy” latent variables for math ability, ACS exam score, and final course grade. These

“dummy” latent variables are mathematically equivalent to their respective indicator variables

because the loading from the latent variable to the indicator variable is constrained to one and the

error variance of the indicator variable is constrained to zero.

Unlike the model used for power analysis, the measurement model in Figure 16 allows all

seven factors to correlate with each other. This correlational model is a seven-factor CFA model.

This CFA model includes relationships between variables, like social presence and ACS exam

scores, that were not hypothesized to have a direct connection in the full structural model.

Allowing all seven factors to correlate with each other means that any poor data-model fit must

have been due to problems with how individual items are loading on the factors. The fit of this

model was acceptable (!scaled,+,-12X

"= 1374.826; CFIscaled = 0.899 ; RMSEAscaled = 0.051,

CI90=[0.047, 0.055] ; SRMR = 0.058) based on joint criteria of RMSEA ≤ 0.06, and SRMR ≤

0.09 (Hu & Bentler, 1999), and as a result, analysis proceeded to the second phase.

The second phase of the SEM analysis reintroduced the hypothesized paths among

variables and was, therefore, a test of the model in Figure 13 (p. 114), with the modifications that

were introduced during the earlier CoI survey analyses. This modified model is shown in Figure

17. As expected, the data-model fit degraded slightly after the introduction of the causal paths

between variables, but the overall fit of the model was acceptable (!scaled,+,-120

"= 1429.111;

CFIscaled = 0.892; RMSEAscaled = 0.053, CI90=[0.049, 0.056] ; SRMR = 0.065) so no additional

analysis took place. Detailed results of this analysis are presented in Chapter 4.

145

Figure 17. The structural model used in the second phase of the SEM analysis.

146

Chapter 4

The results of the previously described analyses are presented in this chapter, beginning

with a summary of the quantitative instructor data from responses to the Approaches to Teaching

Inventory (ATI) and qualitative instructor data from coding the instructor interviews and course

syllabi. A majority of the results presented in this chapter come from quantitative analysis of the

anonymized student data set. The student data analysis results begin with presentation of

descriptive statistics for the student academic variables and survey responses followed by the

results of three independent t-tests to support the inclusion of auxiliary variables in the

confirmatory factory analysis (CFA) of the Community of Inquiry (CoI) survey instrument.

Next, the results of using the student survey data to test models of the CoI instrument are

presented including rationale for modifications made to the initially hypothesized models. Then,

the results of using the entire student data set in the two-phase structural equation modeling

(SEM) analysis are provided.

The last part of this chapter interprets the results of the instructor and student data

analysis in the context of addressing the three research questions. The first research question is

addressed by examining the alignment of instructor and student descriptions of the learning

environment. The results of the CFA of the CoI instrument are used to address the second

research question by providing evidence for the validity and reliability of the CoI instrument

scores. Lastly, the results of the SEM analysis are used to address the third research question

regarding how a constructivist learning environment affects student outcomes of academic

achievement and satisfaction.

147

Instructor Survey and Interview Results

Following the directions provided by Trigwell for use of the ATI-R (personal

communication, August 13, 2015), scores were calculated on the conceptual change student-

focused (CCSF) scale for each instructor’s approach to teaching the general chemistry course

discussed during the interview. In the main study, only the CCSF scale score information was

calculated because the pilot study results indicated that the information transmission teacher-

focused (ITTF) scale did not provide information relevant to instructors’ development of a

student-centered constructivist learning environment. The average scale score calculations were

done by summing the responses to the 11 items on the CCSF scale (3, 5, 7, 8, 13, 14, 15, 17, 18,

20, 21) and dividing by the total number of items on the scale. This produces an average CCSF

score for each instructor ranging from 1 (Only Rarely) to 5 (Almost Always) with a midpoint of

3 (About Half the Time). The individual instructor CCSF scores range from 3.5 to 4.6 with a

mean score of 4.1 indicating that, on average, instructors could be considered as emphasizing a

conceptual change, student-focused approach in their classroom more than half the time.

Individual instructor CCSF scores are not provided since the focus of this research is not to

highlight instructor differences but instead to look at the correspondence between instructor and

student perceptions of the learning environment.

The instructor interview transcripts and course syllabi were coded using the rubric in

Appendix J following the procedure described in Chapter 3 with both the researcher and a

second chemical education researcher coding the instructor data. The coding process results in

two sets of scores for each instructor on eight indicators of a student-centered constructivist

learning environment. These indicators are (1) incorporation of student learning activities to

148

replace a portion of lecture time, (2) the role of the instructor as a facilitator, (3) instructor

conceptions of student learning, (4) use of group work, (5) emphasis on student understanding of

concepts, (6) incorporation of authentic problem solving tasks, (7) students being asked to test or

apply knowledge, and (8) use of discussions to probe student understanding. Each indicator was

rated on a scale ranging from 0 (no evidence) to 3 (multiple pieces of evidence). The percent

agreement between the two coders is 94% and disagreements between the coders are no larger

than one scale point out of the four scale points indicating a high degree of consistency of the

assigned codes. During the coding process both coders identified quotes from the interview

transcripts and course syllabi that provide evidence of the indicators of a student-centered

constructivist learning environment. Table 6 provides examples of this evidence for the two

indicators that most directly relate to the development of a student-centered constructivist

learning environment, namely, how the instructor uses class time and opportunities the instructor

provides for learning to take place. The evidence provided in Table 6 was was generated by

instructors self-reporting classroom practices and approaches to teaching in both the semi-

structured interview and in their own individually written course syllabus.

149

Table 6 Evidence of Teaching Approaches Aligned with Constructivism from Interviews and Syllabi Instructor Use of class time Opportunities for learning

A

“During most lectures, you will have the opportunity to work with your classmates in small groups.”

“A classroom environment in which students actively engage with the content promotes in-depth learning.”

B

“We would spend some time with me [instructor] presenting some information and then I’ll ask them [students] to do something with that information...where I ask them to work together in groups.”

“The teacher’s fundamental task is to get students to engage in learning activities.”

C

“I try to arrange the class so that there are at least two problem-solving periods in a class. I prefer not to talk for more than 30 minutes in one [75 minute] class period.” “These activities will be done in groups in most cases.”

No specific description of learning provided.

D

“Every day my students are all placed in groups so they work with those groups the entire semester and they sit with them in their class and there’s always some opportunity for them to work on something, discuss things, that’s a key piece.”

“My focus for teaching is structuring learning opportunities so that the students can construct their own understanding of concepts.”

All four instructors describe an approach to teaching that targets spending no more than

half of the class time presenting information to students in a lecture format. In support of this

approach, all four instructors utilize group work to some degree in their course, either by creating

formal structured groups or by encouraging students to create informal groups based on other

students sitting nearby that day. Additionally, three out of the four instructors describe learning

as requiring active engagement with the material or mention the construction of individual

knowledge either on their course syllabus or during the interview. The analysis of instructor data

provides evidence that the four instructors have an approach to teaching which shifts their role

150

from a lecturer to a facilitator for at least part of instructional time. The results from coding the

instructor interviews and course syllabi are in alignment with the average CCSF scale scores

which indicate that the instructors emphasize a conceptual change, student-focused approach in

their classroom more than half the time. This aligns the teaching approaches of all four

instructors with the definition of a constructivist approach to teaching used in this research which

is an approach where teaching practices have been adopted that are more student-centered and

have shifted the role of the instructor from a lecturer to a facilitator for at least part of the

instructional time.

Student Data Analysis Results

Descriptive Statistics and Assumptions for SEM Analysis

After undertaking the data cleaning steps described in Chapter 3, descriptive statistics

were calculated for all student variables in the data set to check if the data meet the assumptions

for conducting the CFA and SEM analyses. For the three continuous student academic variables

of final course grades, ACS exam scores, and ACT math scores, the descriptive statistics include

the response rate, mean, standard error of the mean, standard deviation, minimum, maximum,

skewness, and standard error of the skewness. A summary of these values is presented in Table

7. A similar table was constructed for the categorical student survey variables and is presented in

Table 8. Table 8 also includes the frequencies for each categorical response selected for each

survey item. A correlation matrix for the student variables is provided in Appendix L.

151

Table 7 Descriptive Statistics for Student Academic Variables

Variable n % missing Mean (SE) SD Min Max Skew (SE) Final course grades 391 0.0 77.51 (0.65) 12.91 25.75 100.02 –1.11 (0.12) ACS exam scores 385 1.5 39.35 (0.51) 9.94 14 63 –0.00 (0.12) Math ability (ACT) 365 6.6 25.58 (0.20) 3.80 16 35 –0.01 (0.13)

Note. SE = standard error Table 8 Descriptive Statistics for Student Survey Variables

Item n %

missing Mean (SE) SD Skew (SE) Response Frequencies 1 2 3 4 5

T1 (Q1) 345 11.76 4.39 (0.04) 0.82 –1.76 (0.13) 5 10 15 132 183 T2 (Q2) 345 11.76 4.30 (0.04) 0.81 –1.35 (0.13) 4 7 32 140 162 T3 (Q3) 345 11.76 4.40 (0.04) 0.75 –1.47 (0.13) 3 4 25 132 181 T4 (Q4) 345 11.76 4.61 (0.04) 0.67 –1.95 (0.13) 1 5 15 87 237 T5 (Q5) 345 11.76 4.19 (0.05) 0.88 –1.16 (0.13) 5 11 44 140 145 T6 (Q6) 344 12.02 4.17 (0.05) 0.95 –1.09 (0.13) 5 17 50 114 158 T7 (Q7) 345 11.76 4.12 (0.05) 0.88 –1.00 (0.13) 5 10 55 142 133 T8 (Q8) 345 11.76 4.14 (0.05) 0.87 –1.17 (0.13) 6 10 44 153 132 T9 (Q9) 345 11.76 3.96 (0.05) 0.87 –0.57 (0.13) 3 13 82 145 102 T10 (Q10) 345 11.76 3.85 (0.05) 0.92 –0.52 (0.13) 4 20 90 141 90 T11 (Q11) 345 11.76 4.08 (0.05) 0.91 –0.98 (0.13) 5 16 53 144 127 T12 (Q12) 345 11.76 3.78 (0.06) 1.03 –0.51 (0.13) 7 32 91 116 99 T13 (Q13) 345 11.76 4.10 (0.05) 0.97 –1.07 (0.13) 7 18 51 126 143 S1 (Q14) 345 11.76 3.71 (0.06) 1.03 –0.40 (0.13) 7 34 101 112 91 S2 (Q15) 345 11.76 3.81 (0.05) 0.95 –0.47 (0.13) 5 22 98 130 90 S3 (Q16) 345 11.76 4.39 (0.04) 0.76 –1.39 (0.13) 3 3 30 128 181 S4 (Q17) 345 11.76 4.09 (0.05) 0.92 –1.04 (0.13) 5 18 46 147 129 S5 (Q18) 345 11.76 3.79 (0.06) 1.03 –0.64 (0.13) 8 34 75 132 96 S6 (Q19) 344 12.02 4.16 (0.04) 0.82 –0.88 (0.13) 1 14 43 157 129 S7 (Q20) 344 12.02 3.85 (0.05) 0.88 –0.73 (0.13) 5 20 73 171 75 S8 (Q21) 343 12.28 3.92 (0.05) 0.87 –0.57 (0.13) 4 11 86 149 93 S9 (Q22) 343 12.28 3.84 (0.05) 0.88 –0.56 (0.13) 4 18 86 156 79 Note. SE = standard error; for T, S and C items 1 = Strongly Disagree and 5 = Strongly Agree

152

Table 8, continued Descriptive Statistics for Student Survey Variables

% missing

Response Frequencies Item n Mean (SE) SD Skew (SE) 1 2 3 4 5

C1 (Q23) 344 12.02 3.58 (0.05) 0.99 –0.41 (0.13) 9 38 103 132 62 C2 (Q24) 344 12.02 3.53 (0.05) 0.94 –0.38 (0.13) 8 36 113 138 49 C3 (Q25) 344 12.02 3.33 (0.06) 1.06 –0.16 (0.13) 15 58 120 100 51 C4 (Q26) 344 12.02 3.71 (0.06) 1.04 –0.63 (0.13) 9 44 65 147 79 C5 (Q27) 344 12.02 3.74 (0.05) 0.96 –0.59 (0.13) 7 30 84 149 74 C6 (Q28) 343 12.28 3.91 (0.05) 0.85 –0.83 (0.13) 5 15 65 180 78 C7 (Q30) 344 12.02 3.82 (0.05) 0.90 –0.64 (0.13) 7 14 92 151 80 C8 (Q31) 344 12.02 3.99 (0.04) 0.77 –0.59 (0.13) 2 8 68 181 85 C9 (Q32) 344 12.02 4.02 (0.04) 0.83 –0.97 (0.13) 6 7 59 175 97 C10 (Q33) 344 12.02 3.99 (0.04) 0.83 –0.88 (0.13) 5 10 61 176 92 C11 (Q34) 344 12.02 3.84 (0.05) 0.92 –0.66 (0.13) 7 15 88 149 85 C12 (Q35) 344 12.02 3.80 (0.05) 0.90 –0.70 (0.13) 7 19 83 163 72 C13 (Q36) 344 12.02 3.83 (0.05) 0.87 –0.76 (0.13) 5 21 70 179 69 C14 (Q37) 344 12.02 3.68 (0.06) 1.05 –0.58 (0.13) 12 36 84 131 81 SS1 (Q38) 338 13.55 1.85 (0.06) 1.02 –1.09 (0.13) 164 94 54 19 7 SS2 (Q39) 336 14.07 2.38 (0.07) 1.19 –0.59 (0.13) 91 111 70 42 22 SS3 (Q40) 337 13.81 2.13 (0.06) 1.09 –0.73 (0.13) 118 108 69 32 10 SS4 (Q41) 335 14.32 4.16 (0.06) 1.07 –1.31 (0.13) 11 23 33 101 167

Note. SE = standard error; for T, S and C items 1 = Strongly Disagree and 5 = Strongly Agree; SS items were on a semantic differential scale where 1 was positive word for SS1–SS3 (comfortable, satisfying, pleasant) and 1 was negative for SS4 (chaotic)

The skew value of –1.11 in Table 7 reflects the negative skew of the course grades as a

result of the decision to leave negative outliers with low course grades in the data set since the

MLR estimator used in Mplus was robust to violations of normality. The robustness of MLR to

nonnormality is also why none of the survey items are transformed even though they exhibited

skew. Table 7 also shows a relatively small amount of missing data for the three student

academic variables. There are no missing course grade data, a 1.5% missing data rate for ACS

exams scores and a 6.6% missing data rate for ACT math scores. The missing data rate is higher

for the student survey variables in Table 8, ranging from 12-14% missing. The missing data rate

153

for the student survey variables is addressed by testing for a relationship between missing student

survey responses and student academic variables.

Three separate independent t-tests were conducted with one of the three academic

variables (final course grade, ACS exam scores, and ACT math scores) as the dependent variable

to evaluate the assumption that the missing student survey data were either missing completely at

random or missing at random. The two groups being compared are students with complete

responses to all items on the survey instrument and students missing one or more responses to

survey items. The results of these t-tests are presented in Table 9.

Table 9 Independent t-tests for Differences in Academic Variables Based on Missing Survey Responses

Outcome variable Group n Mean (SE) SD

Mean difference (SE) t df p

Final course grades Complete survey 330 79.80 (0.59) 10.66 14.66 (2.21) 6.63 69.36a < .001

Incomplete survey 61 65.14 (2.13) 16.64 ACS exam scores

Complete survey 327 40.36 (0.53) 9.51 6.76 (1.38) 4.92 383 < .001 Incomplete survey 58 33.60 (1.37) 10.44

Math ability (ACT) Complete survey 310 25.76 (0.21) 3.71 1.26 (0.55) 2.27 363 .02

Incomplete survey 55 24.51 (0.56) 4.16 aEqual variances could not be assumed across groups

For each test the group of students with incomplete survey responses has statistically

significantly (p < .05) lower scores on the academic variable than students who responded to all

the survey items. These results demonstrate that the academic achievement variables could be a

possible mechanism responsible for explaining the missing student survey response data. That is,

students with lower scores on academic achievement variables are more likely to be missing

responses to the survey items. Since the academic achievement variables have a relationship with

154

the missing data, they are included in the model of the CoI instrument in order to meet the

assumptions for missing data in the maximum likelihood estimation technique used in analysis of

the CoI instrument. As a result, final course grades, ACS exam scores, and ACT math scores are

included as auxiliary variables in all analyses involving only the CoI instrument. Since the three

academic variables are included in the full research model used for the SEM analysis, their

relationship with the three CoI presence factors is already part of the model and therefore they do

not need to be modeled as auxiliary variables in that analysis.

Confirmatory Factor Analysis of CoI Data

The first aspect of the internal structure of the CoI instrument tested are the two

competing models of teaching presence. For all CFA and SEM analyses scaled fit indices were

computed as a result of using the MLR estimator. The single factor model has all 13 teaching

presence items loading on a single teaching presence factor. This model does not show good

data-model fit (!scaled,+,-1/

"= 219.044; CFIscaled = 0.916; RMSEAscaled = 0.083, CI90=[0.071,

0.095]; SRMR = 0.048) with only the SRMR value within an acceptable range based on target

values for each fit index of CFI ≥ 0.95, RMSEA ≤ 0.06, and SRMR ≤ 0.08 (Hu & Bentler,

1999). Examination of the modification indices suggests that adding an error covariance term

between items 12 and 13 would improve the data-model fit. This modification is justifiable since

both item 12 and item 13 start with the same stem related to the instructor providing feedback,

i.e., “The instructor provided feedback that helped me understand my strengths and weaknesses

relative to the course’s goals and objectives” and “The instructor provided feedback in a timely

fashion.” None of the other suggested modifications were found to have any theoretical

155

justification. As a result of adding the error covariance term between items 12 and 13 the fit of

the single factor teaching presence model improves slightly (!scaled,+,-1.

"= 204.752; CFIscaled =

0.928; RMSEAscaled = 0.075, CI90=[0.064, 0.087]; SRMR = 0.050). However, this model still

only meets the SRMR target criterion indicating that the single factor model of teaching presence

may not be a good fit for the data. The low CFI and high RMSEA combined with the acceptable

SRMR indicates that there may not be much variance or covariance in the teaching presence

items for the model to explain.

As described in Chapter 2 and 3, the literature indicates that teaching presence may be

better modeled as having two correlated factors. In this two-factor model, one factor represents

pre-course activities completed by the instructor, such as reminding students of due dates, and

the second factor represents in-course activities conducted by the instructor, such as facilitating

discussions. In this model, survey items 1 through 4 are associated with the factor representing

pre-course activities and items 5 through 13 are associated with the factor representing in-course

activities. The error covariance term between items 12 and 13 is also included based on the

results of testing the single factor model reported previously.

The two-factor model of teaching presence shows improved fit relative to the one-factor

model (!scaled,+,-13

"= 186.799; CFIscaled = 0.937; RMSEAscaled = 0.071, CI90=[0.059, 0.083];

SRMR = 0.048), but the values of the fit indices still indicate that there may not be much

variance or covariance in the teaching presence items for the model to explain. The competing

one-factor and two-factor models for teaching presence are then compared with the nested !"

comparison described in Chapter 3. This comparison uses the scaling correction factor (c) for

each model, computed as the ratio of the scaled and unscaled !" values, and the difference test

156

scaling correction (cdiff) calculated using the degrees of freedom. The resulting scaled !"

difference is 15.695, which is a statistically significant !" value with one degree of freedom at p

< .001. The calculations and result are summarized in Table 10.

Table 10 Nested Model Comparison for Teaching Presence Factor

Model Y2 df c cdiff Y

2diff p(df = 1)

Two-factor model – unscaled 244.258 63 1.308 1.544 15.695 < .001 Two-factor model – scaled 186.799

One-factor model – unscaled 268.491 64 1.311

One-factor model – scaled 204.752

It is determined from this comparison of the two models that the two-factor model is a

better model of teaching presence, indicating that teaching presence is best described as having

separate pre-course and in-course aspects. Before moving on to testing a model with all three CoI

presence factors, a final teaching presence model is tested which includes just the in-course

survey items (5–13) loading on a single teaching presence factor and maintaining the error

covariance between items 12 and 13. This model cannot be statistically compared to the other

teaching presence models since it contains a different number of items. This 9-item single factor

teaching presence model shows acceptable data-model fit (!scaled,+,-"1

"= 79.902; CFIscaled =

0.960; RMSEAscaled = 0.073, CI90=[0.055, 0.091]; SRMR = 0.046), particularly with regards to

the CFI and SRMR using the joint criteria of CFI ≥ 0.96 and SRMR ≤ 0.09 (Hu & Bentler,

1999). The values of the CFI and SRMR indicate that when considering only items 5–13, there

may be stronger relations in the data and the model did a good job explaining the variances and

covariances. As a result, this is determined to be an acceptable model of teaching presence for

use in additional data analysis steps. This model of teaching presence using only items

157

describing in-course instructor activities and behaviors also better aligns with the research goals

of examining aspects of the classroom learning environment created by the instructor.

Next, a model of the full CoI instrument is tested which includes all three presence

factors. All three presence factors are allowed to correlate because of their overlapping nature in

the original CoI model. The initial three factor model for the CoI instrument includes the error

covariance term between item 12 and 13. Based on modification indices provided by Mplus,

three additional error covariance terms are added between pairs of social presence items that

asked students to rate conceptually similar aspects of social presence. These items are provided

in Table 11.

Table 11 Social Presence Items with Added Error Covariance Terms

Item Wording S1 (Q14) Getting to know other course participants gave me a sense of belonging in the

course S2 (Q15) I was able to form distinct impressions of some course participants S3 (Q16) Face-to-face communication is an excellent medium for social interaction S4 (Q17) I felt comfortable conversing face-to-face in class S9 Q(22) In-class discussions helped me to develop a sense of collaboration

The relationship between the error variances of items S1 and S2 is likely due to both

items asking students about other course participants. Similarly, both items S3 and S4 ask the

students about face-to-face communication in the course. The relationship between the error

variances of S4 and S9 is supported by both items asking the students about in-class discussions.

As a result of adding these error covariance terms, the overall data-model fit of the three factor

CoI model (!scaled,+,-./0

"= 1028.717; CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052,

0.061]; SRMR = 0.061) is determined to be acceptable based on joint criteria of RMSEA ≤ 0.06,

158

and SRMR ≤ 0.09 (Hu & Bentler, 1999). The low CFI value may be a result of relatively weak

relations in the data. Figure 18 shows the standardized values for all model parameters and Table

12 contains values for both standardized and unstandardized model parameters.

The model of the CoI instrument in Figure 18 demonstrates the same three-factor

structure seen in online education research using the original instrument. In addition, the model

results in Figure 18 and Table 12 show relationships among the three presence factors, which is

expected based on the overlapping nature of the three types of presence in the originally

conceptualized CoI model (D. R. Garrison et al., 2000). However, the model of the CoI

instrument in Figure 18 differs slightly from what was previously described in the literature due

to the removal of the four teaching presence items addressing instructor activities outside of class

and the addition of the four error covariance terms.

CoI Scale Scores and Reliability Once the internal structure of the CoI instrument has been determined to generally agree

with what had been seen in the literature, except for the exclusion of survey items 1 through 4,

item scale scores and scale reliabilities are calculated. In the same way that the average CCSF

scale score was calculated from the instructor data, the student data are used to calculate average

scale scores for teaching presence, social presence, and cognitive presence. The average scale

scores are 4.04 for teaching presence, 3.95 for social presence, and 3.77 for cognitive presence

on a five-point scale where 1 is Strongly Disagree and 5 is Strongly Agree. These average scale

scores indicate that students generally agreed with CoI item statements, providing evidence for

the students’ perception of indicators of a constructivist learning environment.

159

Figure 18. The three-factor model of the CoI survey with standardized parameter values. Asterisks indicate values significant at p < 0.001

160

Table 12 Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument

Variable Standardized Unstandardized

From To Parameter

(SE) Variance/ Residual Parameter (SE)

Variance/ Residual

Teaching presence 1 (0) 0.506 (0.081) T5 (Q5) 0.798 (0.031) 0.363 (0.050) 1 (0) 0.288 (0.035) T6 (Q6) 0.830 (0.024) 0.311 (0.041) 1.121 (0.066) 0.287 (0.029) T7 (Q7) 0.720 (0.036) 0.482 (0.052) 0.897 (0.071) 0.379 (0.033) T8 (Q8) 0.798 (0.029) 0.364 (0.046) 0.996 (0.074) 0.287 (0.033) T9 (Q9) 0.614 (0.046) 0.622 (0.056) 0.758 (0.075) 0.479 (0.046) T10 (Q10) 0.546 (0.055) 0.702 (0.060) 0.703 (0.077) 0.590 (0.050) T11 (Q11) 0.749 (0.033) 0.439 (0.049) 0.969 (0.070) 0.372 (0.040) T12 (Q12) 0.617 (0.043) 0.619 (0.053) 0.903 (0.074) 0.670 (0.054) T13 (Q13) 0.623 (0.044) 0.612 (0.055) 0.855 (0.080) 0.583 (0.051) Social presence 1 (0) 0.447 (0.073) S1 (Q14) 0.651 (0.041) 0.577 (0.054) 1 (0) 0.608 (0.056) S2 (Q15) 0.584 (0.042) 0.659 (0.049) 0.827 (0.076) 0.592 (0.044) S3 (Q16) 0.538 (0.053) 0.710 (0.057) 0.608 (0.080) 0.405 (0.043) S4 (Q17) 0.756 (0.035) 0.428 (0.052) 1.038 (0.092) 0.360 (0.048) S5 (Q18) 0.766 (0.029) 0.413 (0.044) 1.181 (0.104) 0.438 (0.048) S6 (Q19) 0.786 (0.035) 0.383 (0.055) 0.965 (0.081) 0.258 (0.035) S7 (Q20) 0.733 (0.033) 0.462 (0.048) 0.967 (0.085) 0.359 (0.041) S8 (Q21) 0.733 (0.033) 0.463 (0.048) 0.953 (0.090) 0.350 (0.033) S9 (Q22) 0.709 (0.034) 0.498 (0.049) 0.934 (0.095) 0.386 (0.039) Cognitive presence 1 (0) 0.537 (0.075) C1 (Q23) 0.729 (0.030) 0.469 (0.044) 1 (0) 0.474 (0.038) C2 (Q24) 0.700 (0.045) 0.510 (0.063) 0.912 (0.064) 0.465 (0.056) C3 (Q25) 0.763 (0.026) 0.418 (0.040) 1.111 (0.063) 0.476 (0.043) C4 (Q26) 0.498 (0.055) 0.752 (0.055) 0.717 (0.099) 0.837 (0.065) C5 (Q27) 0.627 (0.046) 0.607 (0.058) 0.827 (0.077) 0.568 (0.054) C6 (Q28) 0.653 (0.047) 0.574 (0.062) 0.765 (0.077) 0.424 (0.039) C7 (Q30) 0.642 (0.044) 0.588 (0.056) 0.801 (0.076) 0.491 (0.045) C8 (Q31) 0.722 (0.040) 0.478 (0.058) 0.767 (0.063) 0.289 (0.029) C9 (Q32) 0.805 (0.029) 0.351 (0.047) 0.924 (0.071) 0.249 (0.026) C10 (Q33) 0.784 (0.027) 0.385 (0.042) 0.904 (0.060) 0.275 (0.027) C11 (Q34) 0.810 (0.028) 0.344 (0.045) 1.029 (0.076) 0.298 (0.034) C12 (Q35) 0.796 (0.027) 0.366 (0.043) 0.998 (0.068) 0.309 (0.027) C13 (Q36) 0.792 (0.026) 0.373 (0.041) 0.954 (0.067) 0.291 (0.025) C14 (Q37) 0.719 (0.031) 0.483 (0.044) 1.042 (0.071) 0.545 (0.043) Note. All parameters significant at p < 0.001

161

Table 12, continued Model Parameter Values and Standard Errors (SE) from CFA of CoI Instrument

Variables Standardized Unstandardized Between Parameter (SE) Parameter (SE)

Teaching Social –0.667 (0.049) –0.317 (0.050) Teaching Cognitive –0.794 (0.039) –0.414 (0.060) Social Cognitive –0.674 (0.049) –0.330 (0.052) T12 (Q12) T13 (Q13) –0.272 (0.065) –0.170 (0.043) S1 (Q14) S2 (Q15) –0.400 (0.058) –0.240 (0.042) S3 (Q16) S4 (Q17) –0.374 (0.074) –0.143 (0.038) S4 (Q17) S9 (Q22) –0.359 (0.072) –0.134 (0.029)

Note. All parameters significant at p < 0.001

Reliability values for the three presence scales are calculated in two ways. First,

Cronbach’s alpha values are calculated for each scale in order to allow for comparison with

existing literature and to provide information about the scale reliability when item responses are

combined either as averages or sums to create scale scores (Arbaugh, 2008; Arbaugh et al., 2008,

2010; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009). Second, construct

reliabilities are calculated from the standardized item loadings using coefficient H to provide

information about the reliability of the underlying latent presence factor. Both Cronbach’s alpha

and coefficient H values were greater than 0.89 for all three scales. These values are within the

generally accepted range of 0.70 or above (Arjoon et al., 2013; Mueller & Hancock, 2010).

These results are presented in Table 13. Both sets of calculations were performed using the

statistical software R (version 3.1.1; R Core Team, 2014). Cronbach’s alpha is calculated using

the alpha function in the psych package (version 1.4.8.11; Revelle, 2015), and coefficient H is

calculated using the function written in R by the researcher, provided in Appendix F.

162

Table 13 Reliability Values for the Presence and Satisfaction Scales

Scale Cronbach’s alpha Coefficient H Teaching presence 0.892 0.912 Social presence 0.894 0.904 Cognitive presence 0.932 0.944

The combination of the high reliability values for the CoI presence scales and the

acceptable overall-data model fit provides evidence that, with the sample of students in this

research, the CoI survey is able to measure student perceptions of indicators of a constructivist

learning environment. Other than the removal of items 1 through 4, the three-factor model of the

CoI instrument tested in this research corresponds to other research conducted with the CoI

instrument in online courses (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al.,

2010; Joo et al., 2011; Shea & Bidjerano, 2009). This relationship suggests that modifications to

the wording of the CoI instrument as a result of the pilot study did not impact its functioning.

Since these results demonstrate that the CoI instrument is an acceptable instrument for measuring

student perceptions of indicators of a constructivist learning environment, the CoI data are used

to test the full research model relating aspects of a constructivist learning environment to student

outcomes.

Structural Equation Model Results The full research model is tested in a two-phase SEM analysis process (Mueller &

Hancock, 2008). In the first phase of the SEM analysis, the measurement portion of the model is

tested in order to understand how the data fit the model under the least restrictive set of

conditions where all causal paths between variables are replaced with correlational relationships.

The measurement model includes modifications made to the CoI items such as removing survey

163

items 1–4 and adding the four error covariance terms. The fit of the measurement model is

acceptable (!scaled,+,-12X

"= 1374.826; CFIscaled = 0.899; RMSEAscaled = 0.051, CI90=[0.047,

0.055]; SRMR = 0.058) based on the joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu &

Bentler, 1999). None of the modifications suggested by the Mplus software are theoretically

justified so no changes are made before moving on to test the structural model.

The second phase of the SEM analysis reintroduces the hypothesized causal paths among

variables. As expected, the data-model fit degrades slightly after the introduction of the causal

paths between variables (!scaled,+,-120

"= 1429.111; CFIscaled = 0.892; RMSEAscaled = 0.053,

CI90=[0.049, 0.056]; SRMR = 0.065), but the overall fit of the model is acceptable based on joint

criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). The low CFI value

indicates a possibility that even though the model does an acceptable job explaining relationships

among the data, as demonstrated by the low RMSEA and SRMR values, the relationships were

relatively weak to begin with. Again, none of the modifications suggested by the Mplus software

are theoretically justified so this model is retained as the final research model. Having an

acceptable data-model fit indicates that the hypothesized model is a viable representation of the

true underlying relationships present in the data and provides support for the overall goal of the

research, which is to develop and test a model of relationships among aspects of a constructivist

learning environment and student outcomes of academic achievement and satisfaction. However,

not all of the originally hypothesized paths between variables, which are the focal parameters in

this research, are found to be statistically significant. Four previously hypothesized causal paths

and one nondirectional relationship are found to not be statistically significant at p < .05. These

four hypothesized causal paths are the direct effect of math ability on final course grades, the

164

direct effects of teaching presence on ACS exam scores and final course grades, and the direct

effect of social presence on student satisfaction. The hypothesized relationship between the

residual error terms for student satisfaction and final course grades is also found to not be

statistically significant. All other hypothesized paths are statistically significant. Figure 19 shows

the structural portion of the final research model with standardized values for all focal

parameters.

Figure 19. Standardized values for focal parameters in the structural model and R2 values for endogenous variables. Solid arrows and asterisks indicate paths significant at p < .05. The sign of the relationships with the satisfaction variable are reversed to reflect the scale of the satisfaction items. The standardized path values in Figure 19 can be interpreted similarly to standardized

regression coefficients. Dashed arrows indicate hypothesized paths that are found to not be

statistically significant at p < .05. The sign of relationships with the satisfaction latent variable

165

are reversed in Figure 19 to reflect the fact that the scale of three of the four satisfaction items,

provided in Table 14, is opposite from the scale for the other items on the survey instrument.

Table 14 Satisfaction Items from Student Survey THE CHEMISTRY

COURSE WAS… Middle

Q38(SS1). Comfortable 1 2 5 4 5 Uncomfortable Q39(SS2). Satisfying 1 2 5 4 5 Frustrating Q40(SS3). Pleasant 1 2 5 4 5 Unpleasant Q41(SS4). Chaotic 1 2 5 4 5 Organized

The scale of satisfaction items SS1, SS2, and SS3 presents a positive course description at the

low (1) end of the sale and a negative course description at the high end of the scale (5), but for

every other item on the survey instrument, a scale value of 5 corresponds to agreement with a

positive statement about the classroom environment. Due to these differences in the direction of

the scale, the original model output has a negative relationship between the three presence scales

and the satisfaction scale indicating that as the three presence scales increase the negative

responses to the satisfaction items decrease. To aid in interpretation of the model, the sign of

these relationships is reversed so that values can be more easily interpreted as when the three

presence scales increase the positive responses to the satisfaction items increase.

A full listing of all model parameters, both standardized and unstandardized, including

individual survey items loadings on their respective factors is presented in Table 15. In Table 15

the loadings of satisfaction items SS1, SS2, and SS3 onto the satisfaction factor are reversed

from the sign in the Mplus output so that the sign of their loading is consistent with the

relationships to the student satisfaction latent variable in the full model in Figure 19. Similarly,

the signs of the paths to satisfaction are reversed in Table 15 so that they are consistent with

Figure 19.

166

Table 15 Model Parameter Values and Standard Errors (SE) from Final Research Model

Variable Standardized Unstandardized

From To Parameter

(SE) Variance/ Residual

Parameter (SE)

Variance/ Residual

Teaching presence 1 (0) *0.502 (0.081) *T5 (Q5) 0.798 (0.032) 0.362 (0.051) 1 (0) 0.285 (0.035) *T6 (Q6) 0.836 (0.024) 0.302 (0.039) 1.133 (0.069) 0.279 (0.029) *T7 (Q7) 0.715 (0.038) 0.489 (0.055) 0.896 (0.074) 0.386 (0.034) *T8 (Q8) 0.797 (0.029) 0.365 (0.045) 0.991 (0.076) 0.284 (0.032) *T9 (Q9) 0.608 (0.047) 0.630 (0.057) 0.753 (0.076) 0.485 (0.045) *T10 (Q10) 0.540 (0.055) 0.708 (0.059) 0.700 (0.077) 0.597 (0.049) *T11 (Q11) 0.749 (0.033) 0.438 (0.049) 0.973 (0.073) 0.371 (0.039) *T12 (Q12) 0.612 (0.043) 0.625 (0.053) 0.893 (0.075) 0.667 (0.054) *T13 (Q13) 0.626 (0.043) 0.609 (0.054) 0.863 (0.083) 0.581 (0.051) Social presence *0.553 (0.063) *0.251 (0.044) *S1 (Q14) 0.654 (0.041) 0.573 (0.053) 1 (0) 0.608 (0.056) *S2 (Q15) 0.584 (0.042) 0.659 (0.049) 0.823 (0.076) 0.593 (0.044) *S3 (Q16) 0.541 (0.054) 0.707 (0.058) 0.608 (0.080) 0.405 (0.043) *S4 (Q17) 0.756 (0.035) 0.428 (0.053) 1.031 (0.090) 0.360 (0.048) *S5 (Q18) 0.770 (0.028) 0.407 (0.043) 1.183 (0.103) 0.436 (0.047) *S6 (Q19) 0.785 (0.035) 0.383 (0.055) 0.957 (0.079) 0.258 (0.035) *S7 (Q20) 0.735 (0.033) 0.460 (0.048) 0.964 (0.083) 0.359 (0.041) *S8 (Q21) 0.733 (0.033) 0.462 (0.048) 0.949 (0.088) 0.350 (0.033) *S9 (Q22) 0.712 (0.034) 0.493 (0.048) 0.936 (0.094) 0.385 (0.038) Cognitive presence *0.334 (0.056) *0.180 (0.033) *C1 (Q23) 0.731 (0.030) 0.466 (0.044) 1 (0) 0.469 (0.038) *C2 (Q24) 0.699 (0.046) 0.511 (0.064) 0.906 (0.065) 0.462 (0.054) *C3 (Q25) 0.769 (0.025) 0.409 (0.039) 1.123 (0.060) 0.470 (0.043) *C4 (Q26) 0.484 (0.057) 0.766 (0.055) 0.689 (0.099) 0.836 (0.064) *C5 (Q27) 0.626 (0.046) 0.608 (0.058) 0.827 (0.077) 0.571 (0.054) *C6 (Q28) 0.645 (0.049) 0.585 (0.063) 0.750 (0.078) 0.426 (0.039) *C7 (Q30) 0.636 (0.044) 0.596 (0.056) 0.789 (0.074) 0.495 (0.045) *C8 (Q31) 0.721 (0.040) 0.480 (0.057) 0.765 (0.062) 0.290 (0.029) *C9 (Q32) 0.804 (0.029) 0.353 (0.047) 0.926 (0.070) 0.252 (0.026) *C10 (Q33) 0.783 (0.027) 0.387 (0.043) 0.901 (0.061) 0.276 (0.027) *C11 (Q34) 0.806 (0.029) 0.350 (0.047) 1.020 (0.076) 0.302 (0.035) *C12 (Q35) 0.797 (0.027) 0.364 (0.044) 0.995 (0.068) 0.305 (0.028) *C13 (Q36) 0.792 (0.026) 0.373 (0.042) 0.949 (0.066) 0.289 (0.025) *C14 (Q37) 0.726 (0.030) 0.473 (0.043) 1.054 (0.070) 0.538 (0.042) Satisfaction *0.508 (0.077) *0.346 (0.062) *SS1(Q38) –0.804# (0.028) 0.353 (0.045) –1# (0) 0.372 (0.045) *SS2 (Q39) –0.868# (0.018) 0.246 (0.031) –1.271# (0.082) 0.359 (0.041) *SS3 (Q40) –0.892# (0.022) 0.204 (0.039) –1.189# (0.076) 0.247 (0.045) *SS4 (Q41) –0.573# (0.053) 0.672 (0.061) –0.745# (0.075) 0.774 (0.098)

167

Table 15, continued Model Parameter Values and Standard Errors (SE) from Final Research Model

Variables Standardized Unstandardized Variance / Residual (SE) Variance /Residual (SE)

Math ability 1 (0) *14.44 (1.037) *ACS exam scores 0.708 (0.039) 72.963 (5.351) *Final course grades 0.253 (0.022) 40.206 (4.255)

From To Parameter (SE) Parameter (SE)

Teaching presence *Social presence –0.668 (0.047) –0.635 (0.080)

*Cognitive presence –0.613 (0.077) –0.634 (0.100) *Satisfaction –0.388# (0.117) ––0.452# (0.142)

ACS exam scores –0.036 (0.107) –0.521 (1.534) Final course grades –0.023 (0.066) –0.408 (1.168)

Social presence *Cognitive presence 0.268 (0.072) –0.292 (0.075)

Satisfaction –0.053# (0.077) –0.065# (0.093) Cognitive presence

*Satisfaction 0.392# (0.114) 0.441# (0.130) *ACS exam scores 0.288 (0.097) 3.991 (1.389)

*Final course grades 0.140 (0.067) 2.416 (1.186) Math ability

*ACS exam scores 0.474 (0.035) 1.266 (0.104) Final course grades 0.042 (0.035) 0.141 (0.114)

ACS exam scores *Final course grades 0.790 (0.026) 0.981 (0.054)

Between Satisfaction Final course grades –0.080# (0.090) – –0.300# (0.334)#

T12 (Q12) T13 (Q13) –*0.273 (0.064) –*0.170 (0.043) S1 (Q14) S2 (Q15) –*0.401 (0.057) –*0.241 (0.042) S3 (Q16) S4 (Q17) –*0.374 (0.074) –*0.143 (0.038) S4 (Q17) S9 (Q22) *–0.362 (0.071) *–0.135 (0.029)

Note. Asterisks indicate significant at p < 0.05, #indicates sign change to reflect scale of satisfaction items

The values associated with the arrows in Figure 19 only provide information about the

direct effects between variables in the structural model, so indirect effects are requested from

Mplus in order to better understand how one variable may influence another variable by passing

through one or more additional variables. As an example, math ability has a significant direct

168

effect on ACS exam scores, but a nonsignificant direct effect on final course grades. As

described in Chapter 3, according to path tracing rules (Wright, 1934) the value of the indirect

effect of math ability on final course grades through ACS exam scores is mathematically equal to

the value of the direct effect from math ability to ACS exam scores multiplied by the value of the

direct effect from ACS exam scores to final course grades. The statistical significance of this

indirect effect, and all other indirect effects in the model, was determined by examining the

bootstrapped 95% confidence intervals calculated by Mplus. Bootstrapping procedures are used

to provide a more robust estimation of the significance of the indirect effects. This robust

estimation is necessary not only due to the nonnormality present in the data but also because

indirect effects are a product of multiple direct effects and are therefore unlikely to follow

normal distribution (Williams & MacKinnon, 2008). The bootstrapping procedures use the

student data as the pool from which to repeatedly draw random samples from the data to use in

calculating parameter estimates and confidence intervals (Field, 2013; Muthén & Muthén, 2010).

These results are presented in Table 16.

169

Table 16 Decomposed Standardized and Unstandardized Effects Among Variables in Structural Model Variables Standardized Unstandardized To From Direct Indirect Total Direct Indirect Total Final course grades (R2 = 0.747)

Teaching presence –0.023 0.263* 0.286* –0.408 4.681* 5.088* Social presence – 0.099* 0.099* – 1.849* 1.849*

Cognitive presence –0.140* 0.228* 0.368* –2.416* 3.917* 6.333* Math ability –0.042* 0.374* 0.416* –0.141* 1.242* 1.383* ACS exam scores –0.790* – 0.790* –0.981* – 0.981* ACS exam scores (R2 = 0.292)

Teaching presence –0.036* 0.228* 0.192* –0.521 3.272* 2.751* Social presence – 0.077* 0.077* – 1.165* 1.165*

Cognitive presence –0.288* – 0.288* –3.991* – 3.991* Math ability –0.474* – 0.474* –1.266* – 1.266* Student satisfaction (R2 = 0.492)

Teaching presence –0.388* 0.275* 0.663* –0.452* 0.321* 0.773* Social presence –0.053 0.105* 0.052 –0.065* 0.129* 0.064

Cognitive presence –0.392* – 0.392* –0.441* – 0.441* Cognitive presence (R2 = 0.666)

Teaching presence –0.613* 0.179* 0.792* –0.634* 0.185* 0.820* Social presence –0.268* – 0.268* –0.292* – 0.292*

Social presence (R2 = 0.447) Teaching presence –0.668* – 0.668* –0.635* – 0.635* *p < 0.05. Note. The sign of the paths to the satisfaction items are reversed.

In addition to the standardized values for the focal parameters of interest in this research,

Figure 19 and Table 16 also provide R2 values for all endogenous variables in the structural

model, which are the only variables for which R2 values could be calculated. The R2 values

indicate how much variance is explained by the causal paths pointing towards these variables.

Based on this data, the proposed research model is seen to explain almost 75% of the variance in

final course grades, 67% of the variance in cognitive presence ratings, approximately 50% of the

variance in student satisfaction, approximately to 45% of the variance is social presence ratings

and about 30% of the variance in ACS exam scores.

170

ACS exam scores represent a measurement of students’ chemistry content knowledge

based on a standardized national exam. About 30% of the variance in ACS exam scores (R2 =

0.292) is explained by the relationships included in this model. The data in Table 16 show that

only math ability and cognitive presence have statistically significant direct effects on ACS exam

scores. The direct effect of cognitive presence on ACS exam scores is about half the size of the

direct effect of math ability on ACS exam scores. This result indicates that the incoming math

ability of the students has more influence on their performance on the ACS exam than the degree

of cognitive presence in the classroom environment.

Teaching presence was hypothesized to have a direct effect on ACS exam scores in the

original research model, but the data do not support this hypothesis. However, the indirect effects

of teaching presence on ACS exam scores are statistically significant. There are multiple indirect

paths available in the model to move from teaching presence to ACS exam scores. The two

indirect effects with the largest values are the indirect effect of teaching presence through

cognitive presence to ACS exam scores (standardized = 0.177; unstandardized = 2.535) and the

longer indirect effect from teaching presence through social presence then through cognitive

presence to ACS exam scores (standardized = 0.052; unstandardized = 0.740). Both of these

indirect effects are statistically significant. Social presence was not hypothesized to have a direct

effect on ACS exam scores, but is found to have a significant, though small, indirect effect on

ACS exam scores due to the indirect path from social presence to cognitive presence then to ACS

exam scores.

Comparing the magnitude of the direct effect of math ability on ACS exam scores with

the magnitude of other direct and indirect effects on ACS exam scores demonstrates that math

171

ability has the largest influence on ACS exam scores. This relationship between math ability and

performance on the ACS exam is consistent with published research (Lewis & Lewis, 2005; Xu

& Lewis, 2011). The relationship between math ability and ACS exam performance for this

student data set is further supported by examination of the specific first semester general

chemistry ACS exam taken by the students (Form GC15FG) in which approximately one third of

the 70 items were classified by the researcher and an instructor involved in the research as

primarily emphasizing mathematical manipulation to arrive at the correct answer. Of the three

classroom environment factors, cognitive presence has both the largest influence on ACS exam

scores and the only significant direct effect on ACS scores of the three presence factors. The

influences of both teaching presence and social presence on ACS exam scores were only

statistically significant as indirect effects when passing through cognitive presence. These results

indicate that, in addition to math ability, creating a learning environment in which students are

engaged in constructing their own explanations and applying their knowledge has a beneficial

effect on students’ chemistry content knowledge.

The other measurement of students’ academic achievement used in this research is final

course grades. The final course grades in this research do not include the laboratory component

of the course, and in that respect are only an assessment of students’ performance in the

classroom portion of the course. Since the ACS exam is used as a final exam in the courses

studied in this research, students’ ACS exam scores are equal to between 15% and 25% of the

final course grade, depending on the instructor. Thus, the direct effect of ACS exam scores on

final course grades is by far the largest of all the variables. This explains why the model is able

to explain nearly 75% of the variance in final course grade (R2 = 0.747). While math ability does

172

not have a statistically significant direct effect on final course grades, the indirect effect of math

ability on final course grades through ACS exam scores is larger than any of the other total

effects on final course grades. These results indicate that for this sample of students, final course

grades are primarily influenced by the students’ scores on the final exam and their incoming

math ability and that the classroom environment factors have a smaller influence on final course

grades.

Similar to what was seen for ACS exam scores, cognitive presence is the only one of the

three presence factors that has a significant direct effect on final course grades. The large indirect

effect of cognitive presence on final course grades is a result of the path from cognitive presence

through ACS exam scores to final course grades. Teaching presence was originally hypothesized

to have a direct effect on final course grades, but the data do not support this hypothesis.

However, the sum of the multiple indirect effects of teaching presence on final course grades is

statistically significant. The largest indirect effect of teaching presence on final course grades is

the path from teaching presence to cognitive presence through ACS exam scores and ultimately

to final course grades (standardized = 0.140; unstandardized = 2.485). This indirect effect of

teaching presence is statistically significant. The next largest indirect effect of teaching presence

on final course grades is from teaching presence to cognitive presence to final course grades

(standardized = 0.086, unstandardized = 1.533) and is also statistically significant. These results

highlight the large influence ACS exam scores have on final course grades in this data set since

the size of the indirect effect of teaching presence falls almost in half when the path through ACS

exam scores is not included.

173

The third course environment factor, social presence, was not hypothesized to have a

direct effect on final course grades, so only the indirect effects of social presence on final course

grades are tested in the model. The significant indirect effect of social presence on final course

grades is a result of the indirect effect from social presence to cognitive presence through ACS

exam scores to final course grades (standardized = 0.061, unstandardized = 1.143). The indirect

effect from social presence through cognitive presence and then to final course grades, skipping

ACS exam scores, is not statistically significant. These results indicate that, for this sample,

social presence has a relatively small influence on final course grades and that this influence is

primarily as a result of the indirect effect through ACS exam scores to final course grades.

The large influence of ACS exams scores on final course grades, both directly and

indirectly, is mainly a result of the inclusion of ACS exams scores within the calculated final

course grades for this particular data set. The lack of a direct effect of math ability on final

course grades may indicate that the exams written by the instructors, which comprise between

30% and 45% of the final course grade, do not include as much mathematical content as the ACS

exam. Similarly to ACS exam scores, the only classroom environment factor that directly

influences final course grades is cognitive presence. Teaching presence and social presence only

significantly influence final course grades indirectly through cognitive presence and ACS exam

scores. These results highlight the importance of creating a learning environment where students

can engage in learning activities. These results also demonstrate that teaching presence and

social presence are influential only insofar as they support the types of learning activities that

develop cognitive presence.

174

Though the original model hypothesized a relationship between the residual error terms

for final course grades and student satisfaction, this relationship is not found to be statistically

significant. This result indicates that the correlations between final course grades and the four

individual satisfaction item responses, ranging from 0.24 and 0.52, were accounted for by the

causal relationships already present in the model. The hypothesized path from social presence to

student satisfaction is also found to not be statistically significant. While this result was

unexpected, the nonsignificant path from social presence to student satisfaction replicates results

of previous research (Joo et al., 2011). The implication of the nonsignificant path from social

presence to satisfaction in both this study and in the literature suggests that students’ comfort

level interacting with other students in the course does not directly influence students’

satisfaction with the course.

The magnitude of the individual survey item loadings for the student satisfaction factor

are larger than the values seen in previous research but trend in a similar way (Xu & Lewis,

2011), indicating that the satisfaction items in this study are functioning as intended. These

results are presented in Table 17.

Table 17 Standardized Loadings for Satisfaction Items in Current Research and Existing Literature

SS1 SS2 SS3 SS4 1 = comfortable

5 = uncomfortable 1 = satisfying 5 = frustrating

1 = pleasant 5 = unpleasant

1 = chaotic 5 = organized

Current research –0.80 –0.87 –0.89 0.57 Xu & Lewis (2011) –0.74 –0.77 –0.83 0.48

Note. The scale in Xu & Lewis (2011) ranged from 1 to 7; the sign of the loadings has been adjusted to be consistent with Figure 19 and Table 15. Additionally, the Cronbach’s alpha (0.857) and coefficient H value (0.903) of the satisfaction

scale are well above the generally accepted value of 0.70. This indicates an acceptable reliability

175

level of both the satisfaction scale and the latent variable of satisfaction (Arjoon et al., 2013;

Mueller & Hancock, 2010). These results suggest that the way satisfaction was measured in the

current research is in alignment with previous research and support the relationships student

satisfaction has with other variables in the research model.

Satisfaction is the only student outcome variable on which teaching presence had a

significant direct effect. The direct effects of teaching presence and cognitive presence on

student satisfaction were both similar in size and primarily responsible for the model being able

to explain nearly 50% of the variance in student satisfaction ratings (R2 = 0.492). The major

contributor to the indirect effect of teaching presence on student satisfaction is the path from

teaching presence to satisfaction through cognitive presence. The indirect effect of social

presence on satisfaction through cognitive presence is also significant, even though the total

effect of social presence on satisfaction is not significant. These results highlight the large

influence cognitive presence has on student satisfaction in this data set. Cognitive presence

influences satisfaction in two ways, directly and also indirectly as a mediator between other

variables and satisfaction. Combining these results with the nonsignificant relationship between

final course grades and satisfaction suggests that the students surveyed for this research are most

satisfied when the instructor facilitates the development of a learning environment where

students experience a high degree of cognitive presence. The role of the instructor as a facilitator

is important in creating the teaching presence that both directly and indirectly influences student

satisfaction. The results of this model also demonstrate that both teaching and cognitive presence

have a larger influence on student satisfaction than social presence. The smaller influence of

social presence on student satisfaction as compared to teaching presence and cognitive presence

176

suggests that individual engagement in the learning process has more influence on students’

satisfaction with the course than the students’ comfort level during group activities or

discussions.

Considering the results of the structural equation model as a whole indicates that while

constructivist learning environment factors of teaching presence and cognitive presence do

appear to have an influence on student satisfaction and academic outcomes, the influence of

math ability on academic outcomes is larger than either one. Cognitive presence has a greater

direct effect on the academic variables and satisfaction than teaching presence. However, the

direct effect of teaching presence on satisfaction is similar to the direct effect of cognitive

presence on satisfaction. Social presence appears to have minimal influence on both academic

outcomes and student satisfaction. In light of the nonsignificant connection between student

satisfaction and final course grades beyond the causal relationships already present in the model,

it appears that academic outcomes are strongly related to course learning activities while student

satisfaction is strongly related to both learning activities and specific instructor behaviors.

Addressing the Research Questions

Research Question 1 1. Are self-reported instructor approaches to teaching consistent with student perceptions of

the learning environment?

Measurement of the learning environment from the instructor’s perspective began with

modification of the Approaches to Teaching Inventory (ATI) for use in US classrooms. Pilot

testing of the ATI revealed that the conceptual change student-focused (CCSF) scale was most

177

closely aligned with the research goal of identifying student-centered teaching approaches where

the role of the instructor shifted to that of a facilitator for a portion of class time. During the

instructor facilitation portion of the course, students were given opportunities to actively

construct their own chemistry knowledge. This conclusion is a result of the pilot study

instructors who described an approach to teaching that was “student-centered”, either in their

interview or in their course syllabus, having the highest CCSF scores. Analysis of the ATI pilot

study results indicated the need to conduct a follow-up interview and analysis of the course

syllabus in order to provide a more complete description of an instructor’s approach to teaching.

Though a formal think-aloud interview was not conducted as part of the instructor data

collection process in the main study, the first question in the semi-structured interview provided

the instructors with an opportunity to explain or clarify their responses to the ATI if desired. All

instructors in the main study used this opportunity to clarify their interpretation or response to at

least one survey item. In all cases the instructor’s understanding of the item was in alignment

with its intended interpretation. In this way, the semi-structured interviews in the main study

provide evidence for the validity of the response process used to answer the extensively revised

ATI items.

When the four instructors in the main study provided responses to the ATI, the CCSF

scale scores calculated for each of the individual instructors range from 3.5 to 4.6 with a mean

score of 4.1. Since the response scale ranges from 1 (only rarely) to 5 (almost always) with the

midpoint (3) indicating something the instructor reports doing about half the time the mean score

of 4.1 indicates that, on average, the instructors self-report emphasizing a conceptual change,

student-focused approach in their classroom more than half the time. The items on the CCSF

178

scale include statements such as “I provide opportunities so that the students can discuss, among

themselves, key concepts and ideas in this course” and “My teaching in this course helps

students question their own understanding of the subject matter.” Agreement with these

statements and similar statements on the ATI is supported by the instructor interviews and

information available in the instructor-written course syllabus.

Student perceptions of the learning environments created by the instructors are examined

by computing the average scale scores for the three presence factors measured by the

Community of Inquiry (CoI) student survey instrument. The average scale scores are 4.04 for

teaching presence, 3.95 for social presence, and 3.77 for cognitive presence on a five-point scale

where 1 is strongly disagree and 5 is strongly agree. These averages indicate that the students in

the instructors’ classrooms generally have a positive perception of the three factors associated

with a student-centered constructivist learning environment. These results align the student

perceptions of the learning environment with instructor descriptions of their approach to teaching

the introductory chemistry course.

Closer examination of individual student survey item averages supports the instructors’

description of using an approach to teaching that encourages students to discuss course ideas

among themselves and construct their own understanding of the course material. The items on

each of the three presence scales with the highest average scores represent items the students rate

most favorability on the response scale from 1 (Strongly disagree) to 5 (Strongly agree). The

wording and average scores for these three items are provided in Table 18.

179

Table 18 Item on Each Presence Scale with Highest Mean Rating

Item Mean (SE) SD Wording

T5 (Q5) 4.19 (0.05) 0.88 The instructor was helpful in facilitating discussions on course topics that helped me to learn.

S3 (Q16) 4.39 (0.04) 0.76 Face-to-face communication is an excellent medium for social interaction.

C9 (Q32) 4.02 (0.04) 0.83 Course learning activities helped me construct explanations/solutions

Note. SE = standard error; SD = standard deviation; response scale from 1 (Strongly disagree) to 5 (Strongly agree)

The teaching presence item with the highest average score, T5, provides evidence from

the students’ perspective that the instructors utilize course discussions in a way that helped

students learn. The cognitive presence item with the highest average score, C9, relates directly to

the idea of having students construct their own understanding and aligns the students’

perceptions of the learning environment to their instructors’ own descriptions of how

opportunities for student learning are provided. The social presence item with the highest

average score, S3, is less directly applicable to understanding specific instructional strategies.

This is the same social presence item that was identified in the pilot study as the item students

described as asking more about their beliefs than any particular aspect of the classroom

environment. This item was left on the CoI instrument in order to provide an opportunity to

compare its functioning in this research with previous research using the CoI. Item S3 also shows

the smallest relationship to the social presence factor in the CFA of the CoI instrument,

indicating that while students generally agreed with the item, it is not highly representative of the

latent variable of social presence measured by the CoI instrument. Taken together, these results

demonstrate that self-reported instructor approaches to teaching, as measured by the CCSF scale

180

of the ATI, a short interview, and the course syllabus are consistent with student perceptions of

the learning environment as measured by the CoI at both the item and scale levels.

Research Question 2 2. Is the modified Community of Inquiry (CoI) survey an acceptable instrument for

measuring student perceptions of the indicators of a constructivist learning environment

in a face-to-face introductory undergraduate chemistry course?

The acceptability of the CoI survey as an instrument to measure student perceptions of

indicators of a constructivist learning environment is determined from evidence for the validity

and reliability of the CoI survey scores and underlying latent presence factors generated from

analysis of data in the main study. The analyses conducted in the main portion of this research

provide additional evidence for the validity and reliability of the student survey scores beyond

the evidence that was available after conducting the pilot study. Two types of additional validity

evidence generated in the main study are evidence for the internal structure of the CoI and

evidence based on relationships with CoI item responses and other variables. The reliability

evidence for the CoI survey scores and latent presence factors comes from determination of

Cronbach’s alpha and coefficient H values for the three presence scales of the CoI instrument.

Both the validity and reliability evidence are used to support the use of the CoI instrument as an

acceptable instrument for measuring student perceptions of indicators of a constructivist learning

environment in a face-to-face introductory chemistry course.

Evidence for the internal structure of the CoI items is primarily based on the results of the

confirmatory factor analyses. After implementing the modifications to the CoI instrument

181

including removing items 1–4 from the teaching presence factor and adding four error

covariance terms, the fit of the three factor model of the CoI instrument is found to be acceptable

(!scaled,+,-./0

"= 1028.717; CFIscaled = 0.895; RMSEAscaled = 0.057, CI90=[0.052, 0.061]; SRMR

= 0.061) based on joint criteria of RMSEA ≤ 0.06, and SRMR ≤ 0.09 (Hu & Bentler, 1999). All

items load on their intended factors, and no modifications are necessary which would have

resulted in cross loading items on multiple factors. This model of the CoI instrument differs

slightly from what was previously described in the literature due to the removal of the four

teaching presence items addressing instructor activities outside of class time. However, this

separation of the teaching presence factor into two separate but related factors was hypothesized

as a possible result of the analysis based on theory and previous research. The values of the

loadings for the remaining items on the CoI instrument are consistent with what has been

reported in the literature (Arbaugh, 2008; Arbaugh et al., 2008, 2010; D. R. Garrison et al., 2010;

Joo et al., 2011; Shea & Bidjerano, 2009). Therefore, it is concluded that the model for the

internal structure of the CoI shown in Figure 18 (p. 158) is a viable representation of the true

underlying relationships present in the data. This evidence for the validity of the revised CoI

items is particularly important to addressing the second research question given the

modifications to some of the items and the use of the instrument with face-to-face chemistry

students rather than students in an online course. In addition to evidence for the validity of CoI

scores as a result of the internal structure of the CoI instrument, additional evidence is provided

by relationships between CoI scores and other research variables.

Evidence for the validity of the CoI survey scores based on their relationships with other

variables was partially examined in the context of the first research question. The consistency of

182

student and instructor descriptions of the learning environment provides support for the

acceptability of the CoI instrument as a tool to measure student perceptions of the learning

environment. Triangulating the instructor and student quantitative survey response data with the

qualitative instructor data provides convergent evidence that student perceptions of the learning

environment are consistent with self-reported instructor approaches to teaching. The relationship

between the three types of presence measured by the CoI instrument and student satisfaction was

tested during the SEM analysis of the full research model. The results of this analysis show that

both teaching and cognitive presence directly influence student satisfaction, but that social

presence does not directly influence satisfaction. These relationships among the three CoI

presence factors and student satisfaction are consistent with previous research (Joo et al., 2011)

and therefore support the conclusion that the relationships the CoI variables have with other

variables in this research provide evidence for the validity of the CoI survey scores.

Reliability information about the three CoI presence scales, as determined from

Cronbach’s alpha and coefficient H values, also provides evidence to support the conclusion that

the CoI is an acceptable instrument for measuring aspects of a constructivist learning

environment. The alpha and coefficient H values for the three presence scales were all 0.89 or

above, within the generally accepted range of 0.70 or above (Arjoon et al., 2013; Mueller &

Hancock, 2010). The alpha values for the three CoI scales are also consistent with alpha values

reported in other studies utilizing the CoI instrument (Arbaugh, 2008; Arbaugh et al., 2008,

2010; D. R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009), as seen in Table 19.

183

Table 19 Cronbach’s Alpha for CoI Presence Scales in the Current Research and Existing Literature

Teaching presence Social presence Cognitive presence Current research 0.89 0.89 0.93 Arbaugh (2008) 0.95 0.87 0.90 Arbaugh et al. (2008) 0.94 0.91 0.95 Arbaugh et al. (2010) 0.96 0.91 0.95 D. R. Garrison et al. (2010) 0.93 0.87 0.91 Joo et al. (2011) 0.89 0.84 0.82 Shea & Bidjerano (2009) 0.96 0.92 0.95

The Cronbach’s alpha values in Table 19 from both this research using the modified CoI

instrument and from other research with the original CoI items designed for use in online courses

are all above 0.82 and indicate a high degree of internal consistency of the three presence scales.

Additionally, the range of Cronbach’s alpha values is similar across all seven studies listed in

Table 19 and provides evidence for the internal consistency of the three presence scales when

used with a wide variety of students enrolled in a range of different types of courses. The large

coefficient H values for the teaching, social, and cognitive presence factors calculated in this

research (0.912, 0.904, and 0.944, respectively) indicate a large degree of stability of the factors

and therefore greater anticipated reliability over repeated administrations (Mueller & Hancock,

2010). Taken together, both the Cronbach’s alpha and coefficient H values calculated in this

research demonstrate that modifying the CoI survey for use in a face-to-face environment does

not degrade its internal consistency relative to previously administered versions of the

instrument.

The previously described analysis of the student CoI responses provides evidence for

both the validity and reliability of the CoI scores from the internal structure of the CoI

instrument, from relationships among the three CoI presence factors and other research variables,

184

and from the calculated Cronbach’s alpha and coefficient H values. The three-factor structure of

the CoI instrument tested in the current research has acceptable fit with the student data and is

consistent with the structure of the CoI instrument seen in previous research with the CoI

instrument in online courses. The alignment of both individual CoI item averages and CoI scale

averages with data obtained from instructor ATI responses, interviews, and course syllabi

demonstrates that CoI scores can be used to provide an acceptable measurement of the learning

environment experienced by students. The calculated reliability information from Cronbach’s

alpha and coefficient H values suggests that the presence scales on the CoI instrument can be

considered reliable measurements of students’ perceptions of three aspects of a constructivist

learning environment. The combination of both validity and reliability evidence provides strong

support for the acceptability of the CoI as an instrument to measure student perceptions of

indicators of a constructivist learning environment in a face-to-face introductory undergraduate

chemistry course.

Research Question 3 3. To what degree does a constructivist learning environment, as measured by student CoI

survey responses, affect outcomes of student satisfaction and academic achievement in

chemistry, as measured by ACS exam scores and final course grades when the effect of

math ability on academic achievement is considered?

The primary goal of this research, as described by the third research question, is to use

the hypothesized model of relationships among the three CoI presence factors and student

outcomes to examine the influence of a constructivist learning environment on student

185

satisfaction and academic achievement. Since acceptable data-model fit was obtained, as

discussed on p. 163, the specific relationships among the three presence factors and student

outcomes of satisfaction and academic achievement are examined in order to provide detail

about the relative influence each variable had on student outcomes.

Four of the originally hypothesized direct effects between variables in the research model

are found to be not statistically significant at p < .05. These direct effects are the paths from math

ability to final course grades, from teaching presence to ACS exam scores, from teaching

presence to final course grades, and from social presence to student satisfaction. Not finding

these paths to be statistically significant does not mean that, for example, teaching presence does

not influence student academic outcomes in any way. Instead this result indicates that teaching

presence in isolation does not influence academic outcomes and instead teaching presence

influences academic outcomes through its relationship with other variables in the model, such as

cognitive presence. This multi-step influence describes the indirect effect of teaching presence on

academic outcomes through cognitive presence and supports the constructivist model of learning

that describes the role of the instructor as the person who influences the development of a

learning environment where students can experience cognitive presence by facilitating learning

activities where students can actively engage with the course material in order to construct their

own understanding.

Examination of the direct and indirect effects in the research model indicates that math

ability has the largest influence on academic outcomes. Of the three learning environment

factors, teaching presence and cognitive presence influence both student satisfaction and

academic outcomes directly and indirectly. In contrast, social presence does not have a large

186

influence on either satisfaction or academic outcomes. These results indicate that, of the learning

environment factors within the instructor’s control, the most essential aspect of the learning

environment for instructors to develop is cognitive presence. The large influence of cognitive

presence on student outcomes supports the constructivist model of learning in demonstrating the

importance of having students engage in learning activities that require them to explore content,

synthesize information, construct their own explanations, and apply their knowledge to solve

problems.

Summary of Results The results described in this chapter provide a description of the classroom environment

of six sections of an introductory undergraduate chemistry courses from both the instructors’ and

students’ perspectives. The four instructors participating in this research describe approaches to

teaching which emphasize the creation of a student-centered constructivist learning environment

more than half the time. Students in those sections positively rate three aspects of a constructivist

learning environment as measured by teaching presence, cognitive presence, and social presence

and those ratings are consistent with instructor descriptions of the learning environment. The

consistency between the instructors’ and students’ perceptions of the classroom environment

allows for the investigation of relationships among the three aspects of a constructivist learning

environment and student outcomes of academic achievement, as measured by ACS exam scores

and final course grades, and student satisfaction.

Using SEM analysis to test the hypothesized model of relationships among the three

presence factors describing aspects of constructivist learning environment, math ability, and

student outcomes of academic achievement and satisfaction demonstrates that math ability has

187

the largest influence on academic achievement and that only the cognitive presence aspect of the

learning environment directly influences academic achievement. While neither teaching presence

nor social presence directly influences academic achievement, both types of presence play a role

in supporting the development of cognitive presence and therefore both indirectly influence

academic achievement. Additionally, student satisfaction is directly influenced by both cognitive

presence and teaching presence and indirectly influenced by social presence to a small degree,

but satisfaction does not appear to have a relationship with final course grades beyond the causal

relationships present in the model. These results highlight the important role of the instructor in

creating the constructivist learning environment that fosters a high degree of cognitive presence

for the students which ultimately influences both academic outcomes and student satisfaction.

The influence of a constructivist learning environment on student outcomes will be examined in

Chapter 5 as it relates to implications for teaching introductory undergraduate chemistry courses.

188

Chapter 5

The primary goal of this research is to examine relationships among constructivist

learning environment factors and student outcomes in introductory undergraduate chemistry

courses. In order to examine these relationships, it was necessary to determine if the Community

of Inquiry (CoI) student survey instrument provides an acceptable measurement of student

perceptions of indicators of a constructivist learning environment. The acceptability of the CoI

student survey instrument is addressed by examining evidence for both the validity and the

reliability of the CoI survey scores. Validity evidence is generated from the internal structure of

the CoI instrument in addition to relationships between CoI responses and instructor data

collected in this research. Reliability evidence comes from determination of Cronbach’s alpha

and coefficient H values for the teaching presence, cognitive presence, and social presence

scales.

Once the acceptability of the CoI instrument is determined, the primary research model is

tested. This model contains hypothesized relationships among constructivist learning

environment factors and student outcomes of satisfaction and academic achievement. As shown

in Chapter 4, this model has an acceptable data-model fit though not all hypothesized

relationships among variables are found to be statistically significant. In this chapter, the

conclusions drawn from the results presented in Chapter 4 are discussed within the context of

existing literature related to constructivist learning environments. Implications for teaching,

limitations of the current research, and future research directions are also presented.

189

Contribution of Results to Existing Literature

Instructor and Student Ratings of Learning Environment

The two survey instruments used in this research, the Approaches to Teaching Inventory

(ATI) and the CoI, are drawn from existing instruments but required modifications in order to be

used in this particular research context with instructors and students in face-to-face introductory

undergraduate chemistry courses at US institutions. The ATI was originally developed for use in

Australia/Europe and the CoI was developed by Canadian researchers for use in online courses.

As a result of the instrument modifications, it was necessary to conduct pilot studies including

think-aloud interviews to ensure the items are being interpreted as intended by the target

population. These pilot studies resulted in additional modifications to the ATI and the CoI before

they could be used to collect data in the main research study.

At every stage of use, evidence for the validity of the ATI and CoI scores was examined

to test whether the scores were providing acceptable measurements of the learning environment

from both the instructors’ and students’ perspectives. Some evidence for the validity of the CoI

scores results from triangulating student responses with the course instructors’ descriptions of

their approach to teaching from their ATI survey responses. This investigation included

examining instructors’ conceptual change student-focused (CCSF) scale scores and interview

responses along with information available in instructors’ course syllabi. The results presented in

Chapter 4 indicate that student and instructor perceptions of the learning environment are

generally aligned. This alignment is demonstrated by the students perceiving indicators of a

constructivist learning environment while the instructors describe approaches to teaching that are

consistent with a student-centered constructivist approach to teaching.

190

Additional validity evidence is provided by testing the hypothesized internal structure of

the CoI instrument in the main study using confirmatory factor analysis (CFA). The

hypothesized model of the CoI instrument has acceptable data-model fit after only minor

modifications including the addition of four error covariance terms between CoI items and the

exclusion of four items found to be more closely related to activities performed by the instructor

outside of the classroom environment. In this model, the CoI items clearly load on the individual

factors they are hypothesized to be related to which were teaching presence, social presence, and

cognitive presence. Using this version of the CoI instrument, the reliability of the scales is

calculated in two ways. Cronbach’s alpha values assess the internal consistency of responses to

items on the presence scales while coefficient H values represent the reliability of each of the

underlying presence factors. Both measures of reliability are found to be well above the

generally accepted threshold values.

These results indicate that the modified CoI instrument and the CCSF scale of the

modified ATI are appropriate instruments for measuring aspects of a constructivist learning

environment created by the instructor and experienced by the students in face-to-face

introductory undergraduate chemistry courses. The ATI had previously been used in chemical

education research (Stains et al., 2015), but the current research indicates that perhaps only the

CCSF scale provides an acceptable measurement of student-centered approaches to teaching in

introductory chemistry courses since it measures the self-reported frequency of instructor

behaviors such as facilitating discussions and encouraging students to develop their own

understanding of course material. The modification of the CoI instrument for use with students in

a face-to-face learning environment represents the first measure of aspects of a constructivist

191

learning environment in chemical education research, and therefore provides a new tool for use

by researchers.

Influence of a Constructivist Learning Environment on Student Outcomes

The main goal of this research is to determine if it is possible to measure specific aspects

of a constructivist learning environment in order to provide more detailed information regarding

how a constructivist learning environment affects student outcomes. Prior to this research, the

existing literature had provided evidence that the adoption of constructivist teaching practices led

to improvements in student outcomes, but there was little evidence to support an explanation of

how the teaching practices were influencing student outcomes. As a result, constructivism had

become “little more than an educational slogan in the absence of conceptual understanding”

(Hyslop-Margison & Strobel, 2008, p. 73).

Constructivism is used as a theoretical framework in this research to build a model of

how aspects of a constructivist learning environment are hypothesized to affect student outcomes

of academic achievement and satisfaction. The model of a constructivist learning environment

described by the Community of Inquiry (D. R. Garrison et al., 2000) provides support for the

deconstruction of constructivism into three interrelated aspects: teaching presence, cognitive

presence, and social presence. These aspects of a constructivist learning environment are

hypothesized to influence each other as well as other student outcomes such as student

satisfaction with the course, content knowledge, and final course grades. While not all of the

hypothesized relationships are found to be present in the student data used in this research, the

relationships that are present support a constructivist model of learning in which students are

192

responsible for constructing their own understanding. The results also support the role of the

instructor in the constructivist model of learning which is to facilitate the knowledge

construction process by creating a learning environment in which students can actively engage

with the course material.

The results of testing the hypothesized research model with 391 students enrolled in six

sections of an introductory undergraduate chemistry course taught by four different instructors

demonstrate that math ability plays the largest role in influencing student academic outcomes as

measured by American Chemical Society (ACS) exam scores and final course grades. The strong

influence of math ability on student academic achievement in chemistry is well documented in

the chemical education literature (Lewis & Lewis, 2005; Mitchell et al., 2012; Nordstrom, 1990;

Tien et al., 2002; Xu & Lewis, 2011). This result is further supported by examination of the ACS

exam taken by the students (Form GC15FG) in which approximately one third of the 70 items

were classified by the researcher and an instructor involved in the research as primarily

emphasizing mathematical manipulation to arrive at a correct answer.

While the incoming math ability of the students is largely outside of the control of the

instructor, except in situations where a math requirement is placed on student enrollment in the

course, the instructor does have control over the type of learning environment experienced by the

students. Of the three presence factors measured by the CoI instrument used in the research

model, cognitive presence is shown to have the most influence on student academic outcomes

and satisfaction. Cognitive presence is the aspect of the learning environment most directly

aligned with the focus of the constructivist model of learning. The constructivist model

emphasizes individual knowledge construction through active engagement with information and

193

assimilation of that information within existing cognitive structures (Ausubel, 1960; Norman,

1980; Piaget, 1964/1997; Vygotsky, 1978). This large influence of cognitive presence on the

academic outcomes used in the research model provides evidence to support the constructivist

model of learning.

The research model also provides support for the role that teaching presence and social

presence play in influencing cognitive presence. In this way, both the instructor and students in

the learning environment indirectly influence student academic outcomes by supporting the

development of cognitive presence. These results highlight the importance of the instructor in

creating a learning environment that provides students with opportunities to construct their own

understanding of course material. As described in the constructivist model of learning, the role of

the instructor does shift to that of a facilitator of learning, but this is no way deemphasizes the

importance of the instructor, instead it speaks to the need for the instructor to have both content

and pedagogical knowledge of student learning (Bodner et al., 2001; Coll & Taylor, 2001;

Piaget, 1973).

The influence of both teaching presence and social presence on cognitive presence also

addresses some misconceptions surrounding how constructivist learning environments should be

structured to support the development of cognitive presence. The role of the instructor in

supporting cognitive presence demonstrates that developing a constructivist learning

environment does not imply that students should be left completely on their own to discover

knowledge (Bodner et al., 2001; Committee on Developments in the Science of Learning, 2000;

Matthews, 1993; Mugaloglu, 2014; Windschitl, 2002). Instead, in a constructivist learning

194

environment the instructor provides the structure and feedback that allows students to construct

their own knowledge, while still providing support and guidance when needed.

The role of the instructor in supporting the development of both social presence and

cognitive presence demonstrated by the research model is in alignment with Vygotsky’s (1978)

zone of proximal development (ZPD) in which the presence of a more capable adult or peer is

necessary to support a student solving a problem that is above his or her current development

level. The lack of a direct influence of social presence on student academic outcomes in the

research model addresses the common misconception that any type of group work will improve

student learning. In the research model, social presence only influences student academic

outcomes insofar as it supports the development of cognitive presence. This result is consistent

with Vygotsky’s theory of the ZPD in that group work can support students’ ability to solve

difficult problems. However, ultimately students must be able to solve the problems for

themselves in order to perform well on individual assessments. The research model shows that

instructors can influence the development of student’s problem solving ability directly through

their interactions with individual students but also indirectly by using social presence to create a

collaborative learning environment where students work together to create a community of

learners who help each other to construct knowledge. In addition to influencing student academic

outcomes, the research model also addresses how the three presence factors influence student

satisfaction.

The influence of the constructivist learning environment on student satisfaction in the

research model is primarily a result of the direct effects that both teaching presence and cognitive

presence have on student satisfaction. While social presence was hypothesized to directly

195

influence satisfaction no evidence to support this relationship is found from testing the model

with the data used in this research. One interpretation of these results is that because teaching

presence and cognitive presence have similar influences on student satisfaction, it may not be

necessary for instructors to be overly concerned if students are initially unhappy with taking

more control over their own learning in a constructivist learning environment. The results

suggest that the resulting increase in cognitive presence will ultimately have a positive influence

on student satisfaction. Examining the influence of various aspects of a constructivist learning

environment on student outcomes provides instructors with more specific information regarding

how changing specific aspects of their approach to teaching may ultimately influence student

satisfaction and academic achievement.

This research model demonstrates that cognitive presence is the primary aspect of a

constructivist learning environment that influences student academic achievement and

satisfaction. In this model, teaching presence is responsible for influencing the development of

cognitive presence, with social presence playing a smaller role in influencing cognitive presence.

These results suggest that the adoption of a constructivist approach to teaching primarily

influences student outcomes by encouraging students to actively engage with the material in

order to explore, discuss, construct, test, and apply their own understandings. This active

engagement is mainly facilitated by the instructor, but can also be supported by peers when

group work is designed to encourage supportive collaboration and discussion. The ability of this

research model to provide a more detailed understanding of how a constructivist learning

environment influences student outcomes can help explain improved student outcomes when

specific pedagogies such as process-oriented guided-inquiry learning (POGIL) and peer-led team

196

learning (PLTL) are implemented in chemistry classrooms (Gosser et al., 2010; Hanson, 2006,

2008; Lewis & Lewis, 2005; Mitchell et al., 2012; Tien et al., 2002; Varma-Nelson & Banks,

2013; Varma-Nelson & Coppola, 2005). The results of this study can also help explain the

effectiveness of the adoption of more general constructivist approaches to teaching in chemistry

courses (Freeman et al., 2014; Gupta et al., 2015; Hall et al., 2014) as well as recommendations

for the use of constructivist approaches to teaching online courses (Bangert, 2008; Vrasidas,

2000). Previously, it was unclear which aspects of pedagogies such as POGIL and PLTL are

responsible for improvements in student outcomes. This research indicates that cognitive

presence has the most influence on student outcomes. With this knowledge, instructors can

consider other teaching approaches, besides POGIL and PLTL, which can foster cognitive

presence which may provide for more gradual implementation of constructivist approaches to

teaching as opposed to the more encompassing constructivist approaches of POGIL and PLTL

that when implemented involve a major reorganization of an instructor’s approach to teaching.

The final contribution this research makes is to highlight the benefits that can be gained

from looking to other fields for theoretical foundations, instruments, and methodologies. The CoI

model and student survey instrument were originally developed by online education researchers

but can be applied to chemical education research due to the fact that the two fields share a

foundation in constructivism. The use of structural equation modeling (SEM) as methodology for

developing and testing a research model is much more prevalent in online education research (D.

R. Garrison et al., 2010; Joo et al., 2011; Shea & Bidjerano, 2009), but is becoming more widely

used in chemical education research, primarily as a way to develop and test survey instruments

(Xu & Lewis, 2011). SEM is a versatile analysis tool that can be used to examine complex

197

questions and causal relationships among variables with relatively simple data collection

procedures. Use of SEM in chemical education research could help examine many other

multifaceted relationships among variables which is becoming a more common analysis practice

in the field.

Implication of Results for Teaching Introductory Undergraduate Chemistry The results of testing the final structural equation model indicate that the adoption of

approaches to teaching corresponding to cognitive presence items on the CoI instrument could

contribute to improvements in student academic outcomes, even after the effects of math ability

are considered. These teaching approaches include having students construct their own

explanations and solutions rather than copy those provided by the instructor or textbook and

asking students to explore problems related to their life outside the classroom thus providing

opportunities for students to apply their chemistry knowledge to new situations. Though these

teaching approaches are often incorporated into specific pedagogies such as POGIL and PLTL,

the fact that none of the instructors in this study utilize these specific pedagogies demonstrates

that aspects of cognitive presence can be incorporated into the classroom without adopting a

completely new pedagogy. That is, this research suggests that a more general adoption of

constructivist teaching approaches positively influences student outcomes, as had been seen in

other studies (Freeman et al., 2014; Gupta et al., 2015; Hall et al., 2014), but this research is the

first to attribute the effect directly to the cognitive presence aspect of the learning environment.

While the adoption of a constructivist teaching approach must be supported and

facilitated by the course instructor, the nonsignificant direct effects of teaching presence on both

198

ACS exam scores and final course grades indicates that student academic success is not directly

influenced by specific instructor behaviors identified by students. Instead the significance of the

indirect effect of teaching presence on academic outcomes through cognitive presence indicates

that instructor behaviors influence the development of a constructivist learning environment by

creating the conditions for cognitive presence to occur. This interpretation is supported by the

large and significant direct effect of teaching presence on cognitive presence. For classroom

instructors, these results suggest that classroom behaviors such as facilitating discussions and

providing feedback are important only insofar as they contribute to creating an environment that

fosters student construction and application of knowledge as measured by cognitive presence. To

support construction and application of knowledge instructors could consider asking students to

explain the reasoning behind the processes used to solve problems in order to provide evidence

that students have internalized the knowledge necessary to solve the problem or instructors could

utilize problems that require student to apply multiple concepts simultaneously.

Satisfaction was the only student outcome on which teaching presence has a significant

direct effect. This direct effect is slightly smaller than the direct effect of cognitive presence, but

demonstrates that while specific instructor behaviors may make students feel more satisfied with

the course, students are also satisfied when they are engaged in constructing their own

knowledge, as measured by cognitive presence. The nonsignificant correlation between the

residual error terms for student satisfaction and final course grades illustrates that the causal

relationships between other variables in the model explains the correlation between satisfaction

and grades without requiring a direct link between satisfaction and grades. These results may

encourage instructors who initially receive negative student feedback while switching to

199

constructivist teaching approaches that place the instructor in the role of a facilitator. If students

are unfamiliar with being active participants in their own learning, they may at first be

uncomfortable with the instructor in this facilitator role but this initial dissatisfaction does not

necessarily preclude student academic success.

The minimal direct influence of social presence on satisfaction and student academic

outcomes indicates that, on its own, students’ comfort while interacting with their peers does not

play a large role in their affective or academic outcomes. This result was unexpected, but the lack

of direct influence of social presence on student satisfaction had been seen in previous research

with the CoI instrument (Joo et al., 2011). The largest effect of social presence is its direct effect

on cognitive presence. As a result, social presence indirectly influences satisfaction and

academic outcomes through cognitive presence. These results suggest that collaborative work or

discussions that encourage the development of cognitive presence through the use of activities

that foster active student engagement with material in support of constructing explanations and

understanding will ultimately influence student satisfaction and academic achievement. This

conclusion addresses a common misconception surrounding social constructivism, which is the

idea that simply having students work in groups will improve their achievement and satisfaction.

Based on the model tested in this research, classroom instructors should implement purposeful

group work or discussions with careful thought as to how the activity supports the development

of a constructivist learning environment.

This type of structured group work is used in both POGIL and PLTL teaching

approaches. The benefits to structured group work include the opportunity for students to

become comfortable working with various personalities and learn to work together towards a

200

common learning goal. A critical aspect of structured group work is holding students accountable

for the work that is done so that all members are encouraged to participate and all members have

an opportunity to engage in the learning process. Since it is ultimately the individual who must

construct his or her own knowledge, it is important that group work is structured to encourage

and support individual knowledge construction within the context of a larger group. This

encouragement and support speaks to the role of the instructor in facilitating structured group

work and connects back to the direct influence that teaching presence has on social presence in

the research model.

Limitations and Future Research The small number of instructors participating in this research limits the ability to provide

additional evidence for the validity of the ATI scores. Since the ATI has been extensively revised

from its original form it would be important to see if the internal structure of the revised

instrument matches its original two-factor structure. It is also important to test whether the

revised instrument scores still have a relationship to scores from observational protocols such as

the Reformed Teaching Observation Protocol (RTOP) and Classroom Observation Protocol for

Undergraduate STEM (COPUS). To test the internal structure of the ATI it would be best to

obtain a sample of chemistry instructors representing a variety of different chemistry courses

from the introductory undergraduate level through the graduate level. Comparing a larger set of

ATI scores to RTOP or COPUS scores would provide additional evidence regarding whether the

ATI is an accurate measurement of an instructor’s approach to teaching. This larger data set of

ATI responses would also provide more information on whether underreporting of adoption of

constructivist approaches to teaching is a common occurrence. The possibility of underreporting

201

a constructivist approach to teaching arose because the instructor in the main research study who

described the most student-centered constructivist approach to teaching did not have the highest

self-rating on the CCSF scale. Without a larger sample size, it is difficult to know if this result is

related to the particular instructor’s self-perception or if there is a more general issue with

instructors underreporting the adoption of student-centered approaches to teaching.

Limitations also exist related to providing evidence for the validity of the CoI scores. The

think-aloud interviews in the pilot study were conducted with the participation of Chemistry

Club students who may not have been representative of the more general population of students

enrolled in introductory undergraduate chemistry courses. Think-aloud interviews were not

conducted with the students in the data set whose responses to the CoI were analyzed in the main

portion of the research. It would be important for future research to conduct think-aloud

interviews with students enrolled in introductory chemistry courses at a variety of institutions in

order to provide validity evidence based on the response processes for a more diverse group of

students.

Another limitation of this research is the relative homogeneity of the approaches to

teaching utilized by the instructors whose teaching practices and student data were analyzed.

Though this homogeneity was helpful in minimizing differences in the university setting, course

materials, and student backgrounds so that the student data could be combined into a single data

set, it also limits the generalizability of this research. All four participating instructors actively

engage in chemical education research and are therefore familiar with the benefits of more

student-centered approaches to teaching. This is reflected in their CCSF scores, course syllabi,

and descriptions of teaching practices. Given this limitation, it is important for future research to

202

obtain an independent data set to cross-validate the results of this model. This cross-validation

would investigate whether or not the current research model is over fitted to the characteristics of

this particular sample.

While the size of the combined data set is large enough to test the model focal parameters

with sufficient power, there are not enough students represented in the data set to permit testing

the model separately for each instructor. Though the approaches to teaching described by the

instructors are relatively homogeneous, only one instructor uses formal structured student groups

which remained constant throughout the semester. With a larger sample of students and

instructors, it would be possible to test separate models for classrooms which utilize formal and

informal groups. This would provide information about whether the influence of social presence

changes based on the level of structure of student groups. Incorporating other measurement

techniques such as the RTOP or the COPUS could also help determine if the CoI scores are

sensitive to changes in approach to teaching across different learning environments, as was

demonstrated for the ATI in other literature (Stains et al., 2015).

Since the current research provides evidence for a relationship among the CoI presence

factors and student outcomes in a single chemistry course taught by multiple instructors, future

research should gather data from additional chemistry courses in which a wider variety of

approaches to teaching have been adopted. It would be especially useful to include classrooms

taught using known constructivist teaching approaches such as POGIL and PLTL and classrooms

taught using a more traditional lecture-based approach. This would allow the research model to

be tested more broadly and provide additional evidence either supporting or modifying the

relationships examined in this study.

203

If the relationships identified in this study are evident in a larger sample of classrooms,

then specific interventions could be developed to improve the aspects of the learning

environment that are identified as having the greatest influence on student outcomes, i.e.

cognitive presence and teaching presence. These interventions would not necessarily require the

development of new learning activities, but could simply help instructors identify specific

practices that would help them improve cognitive presence and teaching presence in their own

classrooms. As an example, many instructors recognize the importance of having students solve

problems during class, but may not realize that providing students an opportunity to solve

problems on their own, as opposed to watching the instructor solve the problem, encourages the

development of cognitive presence by providing students with an opportunity to engage with the

material by testing and applying their understanding. By helping the instructor understand why

problem solving is important, the instructor could be guided towards ways of implementing

problem solving that provide additional opportunities for the development of cognitive presence.

These opportunities could include asking more complex multi-stage problems that provide

students with opportunities to incorporate knowledge from a variety of topics or more open-

ended authentic problems based in real-world scenarios that may have more than one possible

solution. In this way the instructor would have a better understanding of how different types of

classroom activities influence student outcomes. This type of intervention would not require the

development of entirely new instructional materials, but simply provide the basis for a better

choice of teaching approaches. Ultimately, it is hoped that by providing more detailed

information about how aspects of a constructivist learning environment influence student

204

outcomes, instructors have a stronger research-based foundation available when selecting a

teaching approach that best fits their needs while also improving student outcomes.

App

endi

x A

– O

rigi

nal C

oI It

ems a

nd L

oadi

ngs

Tabl

e 20

O

rigi

nal C

oI It

em W

ordi

ngs a

nd P

ublis

hed

Load

ings

, Pat

hs, o

r Cor

rela

tions

O

rigi

nal C

oI In

stru

men

t Ite

m

Publ

ishe

d L

oadi

ngs*

Teac

hing

Pre

senc

e A

B

C

D

E

F G

1.

The

inst

ruct

or c

lear

ly c

omm

unic

ates

impo

rtant

cou

rse

topi

cs

0.71

0.

826

–0.8

79

–0.8

8 0.

80

0.80

3 0.

75

2.

The

inst

ruct

or c

lear

ly c

omm

unic

ates

impo

rtant

cou

rse

goal

s 0.

71

0.87

7 –0

.891

–0

.84

0.78

0.

829

0.71

3.

The

inst

ruct

or p

rovi

des c

lear

inst

ruct

ions

on

how

to p

artic

ipat

e in

co

urse

lear

ning

act

iviti

es

0.67

0.

592

–0.8

75

–0

.80

0.75

0.

722

0.70

4.

The

inst

ruct

or c

lear

ly c

omm

unic

ates

impo

rtant

due

dat

es/ti

me

fram

es fo

r lea

rnin

g ac

tiviti

es

0.

611

–0.8

71

–0.7

4 0.

69

0.52

5 0.

55

5.

The

inst

ruct

or is

hel

pful

in id

entif

ying

are

as o

f agr

eem

ent a

nd

disa

gree

men

t on

cour

se to

pics

that

hel

ped

me

to le

arn

0.

579

–0.8

76

–0.8

6 0.

87

0.69

7 0.

64

6.

The

inst

ruct

or is

hel

pful

in g

uidi

ng th

e cl

ass t

owar

ds u

nder

stan

ding

co

urse

topi

cs in

a w

ay th

at h

elps

me

clar

ify m

y th

inki

ng

0.83

# 0.

575

–0.8

36

–0.8

7 0.

90

0.74

0/0.

651+

0.86

/0.

72+

7.

The

inst

ruct

or h

elps

to k

eep

cour

se p

artic

ipan

ts e

ngag

ed a

nd

parti

cipa

ting

in p

rodu

ctiv

e di

alog

ue

0.86

0.

633

–0.7

00

–0.8

5 0.

88

0.68

5 0.

74

8.

The

inst

ruct

or h

elps

kee

p th

e co

urse

par

ticip

ants

on

task

in a

way

th

at h

elps

me

to le

arn

0.86

0.

579

–0.7

04

–0.8

7 0.

90

0.70

5/0.

758+

0.78

/0.

82+

9.

The

inst

ruct

or e

ncou

rage

s cou

rse

parti

cipa

nts t

o ex

plor

e ne

w

conc

epts

in th

is c

ours

e

0.52

3 –0

.555

–0

.77

0.85

0.

689

0.72

10.

Inst

ruct

or a

ctio

ns re

info

rced

the

deve

lopm

ent o

f a se

nse

of

com

mun

ity a

mon

g co

urse

par

ticip

ants

0.

67#

0.56

9 0.

591

–0.7

9 0.

87

0.64

5 0.

74

205

Ori

gina

l CoI

Inst

rum

ent I

tem

Pu

blis

hed

Loa

ding

s*

Teac

hing

Pre

senc

e A

B

C

D

E

F G

11.

The

inst

ruct

or h

elps

to fo

cus d

iscu

ssio

n on

rele

vant

issu

es in

a w

ay

that

hel

ps m

e to

lear

n 0.

81

0.42

5 –0

.682

–0

.74

0.85

0.

645

0.83

12.

The

inst

ruct

or p

rovi

des f

eedb

ack

that

hel

ps m

e un

ders

tand

my

stre

ngth

s and

wea

knes

ses (

rela

tive

to th

e co

urse

’s g

oals

and

ob

ject

ives

)#

0.80

# 0.

649#

–0.5

80

–0.7

5 0.

89

13.

The

inst

ruct

or p

rovi

des f

eedb

ack

in a

tim

ely

fash

ion

0.

513

–0.6

63

–0.7

5 0.

74

0.55

7 0.

44

Soci

al P

rese

nce

14.

Get

ting

to k

now

oth

er c

ours

e pa

rtici

pant

s giv

es m

e a

sens

e of

be

long

ing

in th

e co

urse

. 0.

55#

0.61

9 –0

.385

–0

.41

0.59

0.

576

0.68

15.

I am

abl

e to

form

dis

tinct

impr

essi

ons o

f som

e co

urse

par

ticip

ants

0.

55

0.47

3 –0

.440

–0

.40

0.58

0.

423

0.41

16.

Onl

ine

or w

eb-b

ased

com

mun

icat

ion

is a

n ex

celle

nt m

ediu

m fo

r so

cial

inte

ract

ion

0.63

0.

674

–0.6

07

–0.5

0 0.

62

0.56

2 0.

49

17.

I fel

t com

forta

ble

conv

ersi

ng th

roug

h th

e on

line

med

ium

0.

83

0.81

4 –0

.837

–0

.81

0.85

0.

789

0.68

18.

I fee

l com

forta

ble

parti

cipa

ting

in th

e co

urse

dis

cuss

ions

0.

79

0.78

8 –0

.854

–0

.87

0.86

0.

781

0.80

19.

I fee

l com

forta

ble

inte

ract

ing

with

oth

er c

ours

e pa

rtici

pant

s 0.

81

0.70

1 –0

.850

–0

.94

0.91

20.

I fee

l com

forta

ble

disa

gree

ing

with

oth

er c

ours

e pa

rtici

pant

s whi

le

still

mai

ntai

ning

a se

nse

of tr

ust

0.

620

–0.8

15

–0.7

8 0.

77

0.62

0 0.

59

21.

I fee

l tha

t my

poin

t of v

iew

is a

ckno

wle

dged

by

othe

r cou

rse

parti

cipa

nts

0.71

0.

556

–0.6

90

–0.7

8 0.

78

0.61

3 0.

67

22.

Onl

ine

disc

ussi

ons h

elp

me

to d

evel

op a

sens

e of

col

labo

ratio

n

0.56

1 –0

.643

–0

.75

0.77

0.

509

0.70

206

Ori

gina

l CoI

Inst

rum

ent I

tem

Pu

blis

hed

Loa

ding

s*

Cog

nitiv

e Pr

esen

ce

A

B

C

D

E F

G

23.

Prob

lem

s pos

ed in

crea

se m

y in

tere

st in

cou

rse

issu

es

–0

.785

0.

545

0.67

0.

72

0.79

1 0.

74

24.

Cou

rse

activ

ities

piq

ue m

y cu

riosi

ty

–0

.712

0.

671

0.75

0.

80

0.75

5 0.

77

25.

I fee

l mot

ivat

ed to

exp

lore

con

tent

rela

ted

ques

tions

–0.7

70

0.70

2 0.

79

0.81

0.

825

0.77

26.

I util

ize

a va

riety

of i

nfor

mat

ion

sour

ces t

o ex

plor

e pr

oble

ms p

osed

in

this

cou

rse

–0

.759

0.

681

0.72

0.

72

0.39

8 0.

50

27.

Bra

inst

orm

ing

and

findi

ng re

leva

nt in

form

atio

n he

lps m

e re

solv

e co

nten

t rel

ated

que

stio

ns

–0

.794

0.

751

0.74

0.

75

0.59

7 0.

66

28.

Onl

ine

disc

ussi

on w

ere

valu

able

in h

elpi

ng m

e ap

prec

iate

diff

eren

t pe

rspe

ctiv

es

–0

.699

0.

426

0.44

0.

72

0.55

9 0.

70

29.

Com

bini

ng n

ew in

form

atio

n he

lps m

e an

swer

que

stio

ns ra

ised

in

cour

se a

ctiv

ities

–0.7

16

0.69

8 0.

74

0.84

0.

654

0.71

30.

Lear

ning

act

iviti

es h

elp

me

cons

truct

exp

lana

tions

/sol

utio

ns

–0

.732

0.

717

0.76

0.

84

0.65

5 0.

73

31.

Ref

lect

ion

on c

ours

e co

nten

t and

dis

cuss

ions

hel

ped

me

unde

rsta

nd

fund

amen

tal c

once

pts i

n th

is c

lass

–0.6

40

< 0.

3 0.

75

0.85

0.

590

0.76

32.

I can

des

crib

e w

ays t

o te

st a

nd a

pply

the

know

ledg

e cr

eate

d in

this

co

urse

0.

77

–0.6

19

0.77

4 0.

81

0.78

0.

534

0.57

33.

I hav

e de

velo

ped

solu

tions

to c

ours

e pr

oble

ms t

hat c

an b

e ap

plie

d in

pra

ctic

e

–0.6

53

0.79

7 0.

84

0.78

0.

587

0.51

34.

I can

app

ly th

e kn

owle

dge

crea

ted

in th

is c

ours

e to

my

wor

k or

ot

her n

on-c

lass

rela

ted

activ

ities

0.

82

–0.6

87

0.74

5 0.

74

0.75

0.

689

0.58

207

Rel

atio

nshi

ps A

mon

g C

oI P

rese

nce

Fact

ors

Publ

ishe

d C

orre

latio

ns o

r St

anda

rdiz

ed P

aths

*

A

B

C

D

E

F G

H

Teac

hing

pre

senc

e an

d so

cial

pre

senc

e 0

–0.3

18

–0

.49

0.52

0.52

0.

81

Soci

al p

rese

nce

and

cogn

itive

pre

senc

e 0

–0.5

68

–0

.70

0.52

0.40

0.

30

Teac

hing

pre

senc

e an

d co

gniti

ve p

rese

nce

0 –0

.479

–0.6

9 0.

49

0.

51

0.65

Rel

atio

nshi

ps B

etw

een

CoI

Pre

senc

e Fa

ctor

s an

d St

uden

t Out

com

es

Teac

hing

pre

senc

e an

d sa

tisfa

ctio

n 0.

16

0.24

Soci

al p

rese

nce

and

satis

fact

ion

0.47

Cog

nitiv

e pr

esen

ce a

nd sa

tisfa

ctio

n 0.

04

0.26

Teac

hing

pre

senc

e an

d pe

rcei

ved

lear

ning

0.

51

Soci

al p

rese

nce

and

perc

eive

d le

arni

ng

0.19

Cog

nitiv

e pr

esen

ce a

nd p

erce

ived

lear

ning

0.

55

*A

= E

FA fr

om A

rbau

gh (2

008)

; B =

PC

A fr

om A

rbau

gh e

t al.

(200

8); C

= P

CA

from

Arb

augh

et a

l. (2

010)

; D

= P

AF

from

She

a &

Bid

jera

no (2

009)

; E =

SEM

from

She

a &

Bid

jera

no (2

009)

; F =

EFA

from

Gar

rison

et a

l. (2

010)

; G

= S

EM fr

om G

arris

on e

t al.

(201

0); H

= S

EM fr

om Jo

o et

al.

(201

1)

# Item

had

slig

ht w

ordi

ng d

iffer

ence

in th

is st

udy,

but

the

gene

ral i

nten

t was

sim

ilar

+ Som

e ite

ms w

ere

liste

d tw

ice

the

Gar

rison

et a

l. (2

010)

resu

lts w

ith d

iffer

ent l

oadi

ng v

alue

s

208

App

endi

x B

– A

TI R

evis

ed fo

r Pi

lot S

tudy

D

irec

tions

: Thi

s inv

ento

ry is

des

igne

d to

exp

lore

a d

imen

sion

of t

he w

ay th

at a

cade

mic

s go

abou

t tea

chin

g in

a sp

ecifi

c co

ntex

t or

subj

ect o

r cou

rse.

Thi

s may

mea

n th

at y

our r

espo

nses

to th

ese

item

s in

one

cont

ext m

ay b

e di

ffer

ent t

o th

e re

spon

ses y

ou m

ight

m

ake

on y

our t

each

ing

in o

ther

con

text

s or s

ubje

cts.

Pl

ease

con

side

r the

lect

ure

porti

on th

e 10

0-le

vel c

hem

istry

cou

rse

you

have

taug

ht m

ost r

ecen

tly a

t CU

A. U

se th

at c

ours

e as

a

refe

renc

e fo

r com

plet

ing

this

surv

ey. P

leas

e re

frai

n fr

om st

atin

g th

e na

me

of th

e co

urse

or y

our n

ame

alou

d.

For e

ach

item

ple

ase

circ

le o

ne o

f the

num

bers

(1–5

). Th

e nu

mbe

rs st

and

for t

he fo

llow

ing

resp

onse

s:

1.

this

item

was

onl

y ra

rely

or n

ever

true

for m

e in

this

cou

rse.

2.

th

is it

em w

as so

met

imes

true

for m

e in

this

cou

rse.

3.

th

is it

em w

as tr

ue fo

r me

abou

t hal

f the

tim

e in

this

cou

rse.

4.

th

is it

em w

as fr

eque

ntly

true

for m

e in

this

cou

rse.

5.

th

is it

em w

as a

lmos

t alw

ays o

r alw

ays t

rue

for m

e in

this

cou

rse.

Plea

se a

nsw

er e

ach

item

. Do

not s

pend

a lo

ng ti

me

on e

ach:

you

r fir

st r

eact

ion

is p

roba

bly

the

best

one

.

Only rarely

Sometimes

About half the time

Frequently

Almost always

1.

In th

is c

ours

e st

uden

ts sh

ould

focu

s the

ir st

udy

on w

hat I

pro

vide

them

.

1

2

3

4

5

2.

It is

impo

rtant

that

this

cou

rse

shou

ld b

e co

mpl

etel

y de

scrib

ed in

term

s of s

peci

fic

obje

ctiv

es th

at re

late

to fo

rmal

ass

essm

ent i

tem

s.

1

2

3

4

5

3.

In m

y in

tera

ctio

ns w

ith st

uden

ts in

this

cou

rse

I try

to d

evel

op a

con

vers

atio

n w

ith

them

abo

ut th

e to

pics

we

are

stud

ying

.

1

2

3

4

5

4.

It is

impo

rtant

to p

rese

nt a

lot o

f fac

ts to

stud

ents

so th

at th

ey k

now

wha

t the

y ha

ve

to le

arn

for t

his c

ours

e.

1

2

3

4

5

5.

I set

asi

de so

me

teac

hing

tim

e so

that

the

stud

ents

can

dis

cuss

, am

ong

them

selv

es,

key

conc

epts

and

idea

s in

this

cou

rse.

1

2

3

4

5

209

Only rarely

Sometimes

About half the time

Frequently

Almost always

6.

In th

is c

ours

e I c

once

ntra

te o

n co

verin

g th

e in

form

atio

n th

at m

ight

be

avai

labl

e fr

om k

ey te

xts a

nd re

adin

gs.

1

2

3

4

5

7.

I enc

oura

ge st

uden

ts to

rest

ruct

ure

thei

r exi

stin

g kn

owle

dge

in te

rms o

f the

new

w

ay o

f thi

nkin

g ab

out t

he su

bjec

t tha

t the

y w

ill d

evel

op.

1

2

3

4

5

8.

In te

achi

ng se

ssio

ns fo

r thi

s sub

ject

, I d

elib

erat

ely

prov

oke

deba

te a

nd d

iscu

ssio

n.

1

2

3

4

5

9.

I s

truct

ure

my

teac

hing

in th

is c

ours

e to

hel

p st

uden

ts to

pas

s the

form

al a

sses

smen

t ite

ms.

1

2

3

4

5

10. I

thin

k an

impo

rtant

reas

on fo

r run

ning

teac

hing

sess

ions

in th

is c

ours

e is

to g

ive

stud

ents

a g

ood

set o

f not

es.

1

2

3

4

5

11. I

n th

is c

ours

e, I

prov

ide

the

stud

ents

the

info

rmat

ion

they

will

nee

d to

pas

s the

fo

rmal

ass

essm

ents

.

1

2

3

4

5

12. I

shou

ld k

now

the

answ

ers t

o an

y qu

estio

ns th

at st

uden

ts m

ay p

ut to

me

durin

g th

is

cour

se.

1

2

3

4

5

13. I

mak

e av

aila

ble

oppo

rtuni

ties f

or st

uden

ts in

this

cou

rse

to d

iscu

ss th

eir c

hang

ing

unde

rsta

ndin

g of

the

subj

ect.

1

2

3

4

5

14. I

t is b

ette

r for

stud

ents

in th

is c

ours

e to

gen

erat

e th

eir o

wn

note

s rat

her t

han

copy

m

ine.

1

2

3

4

5

15. A

lot o

f tea

chin

g tim

e in

this

cou

rse

shou

ld b

e us

ed to

que

stio

n st

uden

ts’ i

deas

.

1

2

3

4

5

16. I

n th

is c

ours

e m

y te

achi

ng fo

cuse

s on

the

good

pre

sent

atio

n of

info

rmat

ion

to

stud

ents

.

1

2

3

4

5

17. I

see

teac

hing

as h

elpi

ng st

uden

ts d

evel

op n

ew w

ays o

f thi

nkin

g in

this

subj

ect.

1

2

3

4

5

18

. In

teac

hing

this

cou

rse

it is

impo

rtant

for m

e to

mon

itor s

tude

nts’

cha

nged

un

ders

tand

ing

of th

e su

bjec

t mat

ter.

1

2

3

4

5

19. M

y te

achi

ng in

this

cou

rse

focu

ses o

n de

liver

ing

wha

t I k

now

to th

e st

uden

ts

1

2

3

4

5

20

. Tea

chin

g in

this

cou

rse

shou

ld h

elp

stud

ents

que

stio

n th

eir o

wn

unde

rsta

ndin

g of

th

e su

bjec

t mat

ter.

1

2

3

4

5

21. T

each

ing

in th

is c

ours

e sh

ould

incl

ude

help

ing

stud

ents

find

thei

r ow

n le

arni

ng

reso

urce

s.

1

2

3

4

5

22. I

pre

sent

mat

eria

l to

enab

le st

uden

ts to

bui

ld u

p an

info

rmat

ion

base

in th

is su

bjec

t.

1

2

3

4

5

210

App

endi

x C

– S

tude

nt S

urve

y It

ems U

sed

in P

ilot S

tudy

D

irec

tions

: Ple

ase

cons

ider

the

lect

ure

porti

on o

f you

r firs

t und

ergr

adua

te c

hem

istry

cou

rse

at C

UA

. Use

that

cou

rse

as a

re

fere

nce

for c

ompl

etin

g th

is su

rvey

. Ple

ase

refr

ain

from

stat

ing

the

nam

e of

the

cour

se o

r the

cou

rse

inst

ruct

or a

loud

. Whe

n yo

u ar

e re

ady,

read

eac

h st

atem

ent a

loud

, the

n se

lect

the

num

ber t

hat b

est d

escr

ibes

you

r lev

el o

f agr

eem

ent w

ith e

ach

stat

emen

t. A

fter y

ou se

lect

a n

umbe

r, br

iefly

des

crib

e w

hy y

ou se

lect

ed th

at n

umbe

r. I m

ay st

op y

ou d

urin

g th

is p

roce

ss to

ask

fo

llow

-up

ques

tions

abo

ut th

e w

ay y

ou in

terp

rete

d an

item

.

Stro

ngly

A

gree

A

gree

U

nsur

e D

isag

ree

Stro

ngly

D

isag

ree

Not

A

pplic

able

Q

1. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt c

ours

e to

pics

5

4

3

2

1

0

Q

2. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt c

ours

e go

als

5

4

3

2

1

0

Q3.

The

inst

ruct

or p

rovi

ded

clea

r ins

truct

ions

on

how

to p

artic

ipat

e in

co

urse

lear

ning

act

iviti

es

5

4

3

2

1

0

Q4.

The

inst

ruct

or c

lear

ly c

omm

unic

ated

impo

rtant

due

dat

es/ti

me

fram

es fo

r lea

rnin

g ac

tiviti

es

5

4

3

2

1

0

Q5.

The

inst

ruct

or w

as h

elpf

ul in

iden

tifyi

ng a

reas

of a

gree

men

t and

di

sagr

eem

ent o

n co

urse

topi

cs th

at h

elpe

d m

e to

lear

n 5

4

3

2

1

0

Q6.

The

inst

ruct

or w

as h

elpf

ul in

gui

ding

the

clas

s tow

ards

un

ders

tand

ing

cour

se to

pics

in a

way

that

hel

ped

me

clar

ify m

y th

inki

ng

5

4

3

2

1

0

Q7.

The

inst

ruct

or h

elpe

d to

kee

p co

urse

par

ticip

ants

eng

aged

and

pa

rtici

patin

g in

pro

duct

ive

dial

ogue

5

4

3

2

1

0

Q8.

The

inst

ruct

or h

elpe

d ke

ep th

e co

urse

par

ticip

ants

on

task

in a

way

th

at h

elpe

d m

e to

lear

n 5

4

3

2

1

0

Q9.

The

inst

ruct

or e

ncou

rage

d co

urse

par

ticip

ants

to e

xplo

re n

ew

conc

epts

in th

is c

ours

e 5

4

3

2

1

0

Q10

. The

inst

ruct

or re

info

rced

the

deve

lopm

ent o

f a se

nse

of

com

mun

ity a

mon

g co

urse

par

ticip

ants

5

4

3

2

1

0

Q11

. The

inst

ruct

or h

elpe

d to

focu

s dis

cuss

ion

on re

leva

nt is

sues

in a

w

ay th

at h

elpe

d m

e to

lear

n 5

4

3

2

1

0

Q12

. The

inst

ruct

or p

rovi

ded

feed

back

that

hel

ped

me

unde

rsta

nd m

y st

reng

ths a

nd w

eakn

esse

s rel

ativ

e to

the

cour

se’s

goa

ls a

nd

obje

ctiv

es

5

4

3

2

1

0

Q13

. The

inst

ruct

or p

rovi

ded

feed

back

in a

tim

ely

fash

ion

5

4

3

2

1

0

Q14

. Get

ting

to k

now

oth

er c

ours

e pa

rtici

pant

s gav

e m

e a

sens

e of

be

long

ing

in th

e co

urse

. 5

4

3

2

1

0

211

St

rong

ly

Agr

ee

Agr

ee

Uns

ure

Dis

agre

e St

rong

ly

Dis

agre

e N

ot

App

licab

le

Q15

. I w

as a

ble

to fo

rm d

istin

ct im

pres

sion

s of s

ome

cour

se p

artic

ipan

ts

5

4

3

2

1

0

Q16

. Fac

e-to

-fac

e co

mm

unic

atio

n is

an

exce

llent

med

ium

for s

ocia

l in

tera

ctio

n 5

4

3

2

1

0

Q17

. I fe

lt co

mfo

rtabl

e co

nver

sing

face

-to-f

ace

in c

lass

5

4

3

2

1

0

Q

18. I

felt

com

forta

ble

parti

cipa

ting

in th

e co

urse

dis

cuss

ions

5

4

3

2

1

0

Q

19. I

felt

com

forta

ble

inte

ract

ing

with

oth

er c

ours

e pa

rtici

pant

s 5

4

3

2

1

0

Q

20. I

felt

com

forta

ble

disa

gree

ing

with

oth

er c

ours

e pa

rtici

pant

s whi

le

still

mai

ntai

ning

a se

nse

of tr

ust

5

4

3

2

1

0

Q21

. I fe

lt th

at m

y po

int o

f vie

w w

as a

ckno

wle

dged

by

othe

r cou

rse

parti

cipa

nts

5

4

3

2

1

0

Q22

. In-

clas

s dis

cuss

ions

hel

ped

me

to d

evel

op a

sens

e of

col

labo

ratio

n 5

4

3

2

1

0

Q

23. P

robl

ems p

osed

incr

ease

d m

y in

tere

st in

cou

rse

issu

es

5

4

3

2

1

0

Q24

. Cou

rse

activ

ities

piq

ued

my

curio

sity

5

4

3

2

1

0

Q

25. I

felt

mot

ivat

ed to

exp

lore

con

tent

rela

ted

ques

tions

5

4

3

2

1

0

Q

26. I

util

ized

a v

arie

ty o

f inf

orm

atio

n so

urce

s to

expl

ore

prob

lem

s po

sed

in th

is c

ours

e 5

4

3

2

1

0

Q27

. Bra

inst

orm

ing

and

findi

ng re

leva

nt in

form

atio

n he

lped

me

reso

lve

cont

ent r

elat

ed q

uest

ions

5

4

3

2

1

0

Q28

. Ple

ase

sele

ct “

Dis

agre

e” fo

r thi

s ite

m

5

4

3

2

1

0

Q29

. In-

clas

s dis

cuss

ions

wer

e va

luab

le in

hel

ping

me

appr

ecia

te

diff

eren

t per

spec

tives

5

4

3

2

1

0

Q30

. Com

bini

ng n

ew in

form

atio

n he

lped

me

answ

er q

uest

ions

rais

ed in

co

urse

act

iviti

es

5

4

3

2

1

0

Q31

. Lea

rnin

g ac

tiviti

es h

elpe

d m

e co

nstru

ct e

xpla

natio

ns/s

olut

ions

5

4

3

2

1

0

Q

32. R

efle

ctio

n on

cou

rse

cont

ent h

elpe

d m

e un

ders

tand

fund

amen

tal

conc

epts

in th

is c

lass

5

4

3

2

1

0

Q33

. Ref

lect

ion

on d

iscu

ssio

ns h

elpe

d m

e un

ders

tand

fund

amen

tal

conc

epts

in th

is c

lass

5

4

3

2

1

0

Q34

. I c

an d

escr

ibe

way

s to

test

and

app

ly th

e kn

owle

dge

crea

ted

in th

is

cour

se

5

4

3

2

1

0

Q35

. I h

ave

deve

lope

d so

lutio

ns to

cou

rse

prob

lem

s tha

t can

be

appl

ied

in p

ract

ice

5

4

3

2

1

0

Q36

. I c

an a

pply

the

know

ledg

e cr

eate

d in

this

cou

rse

to m

y w

ork

or

othe

r non

-cla

ss re

late

d ac

tiviti

es

5

4

3

2

1

0

212

Dir

ectio

ns: A

gain

con

side

r jus

t the

lect

ure

porti

on o

f you

r firs

t und

ergr

adua

te c

hem

istry

cou

rse

at C

UA

and

sele

ct th

e nu

mbe

r tha

t re

pres

ents

you

r lev

el o

f agr

eem

ent w

ith e

ach

stat

emen

t. Pl

ease

read

the

stat

emen

t and

you

r cho

ice

alou

d, a

nd b

riefly

des

crib

e yo

ur

reas

on fo

r sel

ectin

g th

at re

spon

se. A

s bef

ore,

I m

ay st

op y

ou to

ask

a fo

llow

-up

ques

tion

abou

t you

r int

erpr

etat

ion

of a

par

ticul

ar

item

.

St

rong

ly

Agr

ee

Agr

ee

Uns

ure

Dis

agre

e St

rong

ly

Dis

agre

e S1

. I w

as sa

tisfie

d w

ith th

e pa

cing

of t

he c

ours

e.

5

4

3

2

1

S2. I

was

satis

fied

with

the

leve

l of e

ffor

t thi

s cou

rse

requ

ired.

5

4

3

2

1

S3

. My

inte

rest

in th

e su

bjec

t mat

ter i

ncre

ased

bec

ause

of t

his

cour

se.

5

4

3

2

1

S4. I

was

satis

fied

with

my

lear

ning

in th

is c

ours

e.

5

4

3

2

1

S5. I

was

hap

py w

ith m

y fin

al g

rade

in th

is c

ours

e.

5

4

3

2

1

D

irec

tions

: A li

st o

f opp

osin

g w

ords

app

ears

bel

ow. R

ate

how

wel

l the

se w

ords

des

crib

ed y

our f

eelin

gs a

bout

the

lect

ure

porti

on

of y

our f

irst c

hem

istry

cou

rse

at C

UA

. For

eac

h lin

e, c

hoos

e a

posi

tion

betw

een

the

two

wor

ds th

at d

escr

ibes

exa

ctly

how

you

felt.

Th

e m

iddl

e po

sitio

n is

if y

ou a

re u

ndec

ided

or h

ave

no fe

elin

gs re

late

d to

the

term

s on

that

line

. As b

efor

e, p

leas

e re

ad th

e tw

o w

ords

and

you

r cho

ice

alou

d an

d br

iefly

des

crib

e yo

ur re

ason

for s

elec

ting

that

resp

onse

. Aga

in, I

may

stop

you

to a

sk a

follo

w-u

p qu

estio

n ab

out y

our r

espo

nses

.

TH

E C

HE

MIS

TR

Y C

OU

RSE

WA

S…

Mid

dle

S6

.

Com

forta

ble

5

4

3

2

1

Unc

omfo

rtabl

e S7

.

S

atis

fyin

g 5

4

3

2

1

Fr

ustra

ting

S8.

P

leas

ant

5

4

3

2

1

Unp

leas

ant

S9.

Cha

otic

5

4

3

2

1

O

rgan

ized

213

App

endi

x D

– A

TI R

evis

ions

Afte

r Pi

lot S

tudy

Tabl

e 21

AT

I Ite

ms U

sed

in P

ilot S

tudy

, Rev

ised

ATI

Item

s, an

d Ra

tiona

le fo

r Rev

isio

n A

TI i

tem

s use

d in

pilo

t stu

dy

AT

I Ite

ms r

evis

ed a

fter

pilo

t stu

dy

Rat

iona

le fo

r re

visi

on

1a.

In th

is c

ours

e st

uden

ts sh

ould

focu

s th

eir s

tudy

on

wha

t I p

rovi

de th

em.

1b. I

n th

is c

ours

e st

uden

ts sh

ould

focu

s th

eir s

tudy

on

mat

eria

ls li

sted

in

the

sylla

bus a

nd p

rovi

ded

by th

e in

stru

ctor

such

as t

he te

xtbo

ok a

nd

lect

ure

note

s.

Cha

nged

“w

hat I

pro

vide

them

” to

incl

ude

all

cour

se m

ater

ials

.

2a.

It is

impo

rtant

that

this

cou

rse

shou

ld b

e co

mpl

etel

y de

scrib

ed in

te

rms o

f spe

cific

obj

ectiv

es th

at

rela

te to

form

al a

sses

smen

t ite

ms.

2b. T

his c

ours

e is

com

plet

ely

desc

ribed

in

term

s of s

peci

fic o

bjec

tives

that

re

late

to c

ours

e as

sess

men

ts.

Rem

oved

bel

ief c

ompo

nent

, “fo

rmal

as

sess

men

t ite

ms”

mos

t fre

quen

tly in

terp

rete

d to

mea

n co

urse

ass

essm

ents

, not

ext

erna

l as

sess

men

ts

3a.

In m

y in

tera

ctio

ns w

ith st

uden

ts in

th

is c

ours

e I t

ry to

dev

elop

a

conv

ersa

tion

with

them

abo

ut th

e to

pics

we

are

stud

ying

.

3b.

In m

y in

tera

ctio

ns w

ith st

uden

ts in

th

is c

ours

e I t

ry to

dev

elop

a

conv

ersa

tion

with

them

abo

ut th

e to

pics

we

are

stud

ying

.

No

chan

ge. I

nstru

ctor

inte

rpre

tatio

n of

co

nver

satio

ns in

clud

e in

cla

ss a

nd o

ut o

f cla

ss

(off

ice

hour

s)

4a.

It is

impo

rtant

to p

rese

nt a

lot o

f fa

cts t

o st

uden

ts so

that

they

kno

w

wha

t the

y ha

ve to

lear

n fo

r thi

s co

urse

.

4b.

In th

is c

ours

e fa

cts a

re p

rese

nted

to

stud

ents

so th

at th

ey k

now

wha

t th

ey h

ave

to le

arn.

Rem

oved

bel

ief c

ompo

nent

. Rem

oved

“a

lot”

du

e to

impl

ied

nega

tive

asso

ciat

ion.

5a.

I set

asi

de so

me

teac

hing

tim

e so

th

at th

e st

uden

ts c

an d

iscu

ss,

amon

g th

emse

lves

, key

con

cept

s an

d id

eas i

n th

is c

ours

e.

5b.

I pro

vide

opp

ortu

nitie

s so

that

the

stud

ents

can

dis

cuss

, am

ong

them

selv

es, k

ey c

once

pts a

nd

idea

s in

this

cou

rse.

Rem

oved

“te

achi

ng ti

me”

to to

avo

id

rest

rictin

g qu

estio

n to

onl

y co

nsid

erin

g le

ctur

e tim

e.

6a.

In th

is c

ours

e I c

once

ntra

te o

n co

verin

g th

e in

form

atio

n th

at m

ight

be

ava

ilabl

e fr

om k

ey te

xts a

nd

read

ings

.

6a.

In th

is c

ours

e I c

once

ntra

te o

n co

verin

g th

e in

form

atio

n th

at is

av

aila

ble

from

ass

igne

d re

adin

gs.

Cha

nged

“ke

y te

xts a

nd re

adin

gs”

to a

lign

with

m

ost f

requ

ent i

nter

pret

atio

n as

ass

igne

d te

xtbo

ok re

adin

g, re

mov

ed “

mig

ht”

to m

ake

ques

tion

mor

e co

ncre

te.

214

Tabl

e 21

, con

tinue

d AT

I Ite

ms U

sed

in P

ilot S

tudy

, Rev

ised

ATI

Item

s, an

d Ra

tiona

le fo

r Rev

isio

n

AT

I ite

ms u

sed

in p

ilot s

tudy

A

TI I

tem

s rev

ised

aft

er p

ilot s

tudy

R

atio

nale

for

revi

sion

7a

. I e

ncou

rage

stud

ents

to re

stru

ctur

e th

eir e

xist

ing

know

ledg

e in

term

s of

the

new

way

of t

hink

ing

abou

t th

e su

bjec

t tha

t the

y w

ill d

evel

op.

7b.

I enc

oura

ge st

uden

ts to

rest

ruct

ure

thei

r exi

stin

g kn

owle

dge

in te

rms

of th

e ne

w w

ay o

f thi

nkin

g ab

out

the

subj

ect t

hat t

hey

will

dev

elop

.

No

chan

ge. R

estru

ctur

ing

know

ledg

e is

rela

ted

to c

onst

ruct

ivis

t ide

a of

mod

ifyin

g pr

eexi

stin

g kn

owle

dge.

Inst

ruct

ors d

isag

reei

ng e

ither

be

lieve

d th

e st

uden

ts e

nter

ed w

ith n

o kn

owle

dge

or e

xist

ing

know

ledg

e w

as g

ood

and

requ

ired

no c

hang

es.

8a.

In te

achi

ng se

ssio

ns fo

r thi

s sub

ject

, I d

elib

erat

ely

prov

oke

deba

te a

nd

disc

ussi

on.

8b. I

n th

is c

ours

e, I

enco

urag

e de

bate

an

d di

scus

sion

. R

emov

ed “

teac

hing

sess

ions

” an

d ch

ange

d “d

elib

erat

ely

prov

oke”

to b

e m

ore

neut

ral.

9a.

I stru

ctur

e m

y te

achi

ng in

this

co

urse

to h

elp

stud

ents

to p

ass t

he

form

al a

sses

smen

t ite

ms.

9b.

I stru

ctur

e m

y te

achi

ng in

this

co

urse

to h

elp

stud

ents

pas

s cou

rse

asse

ssm

ents

.

Cha

nged

“fo

rmal

ass

essm

ents

” to

cou

rse

asse

ssm

ents

.

10a.

I th

ink

an im

porta

nt re

ason

for

runn

ing

teac

hing

sess

ions

in th

is

cour

se is

to g

ive

stud

ents

a g

ood

set

of n

otes

.

10b.

Cla

ss ti

me

in th

is c

ours

e is

use

d to

gi

ve st

uden

ts a

set o

f not

es.

Rem

oved

bel

ief c

ompo

nent

. Rem

oved

teac

hing

se

ssio

ns. R

emov

ed “

good

”.

11a.

In th

is c

ours

e, I

prov

ide

the

stud

ents

the

info

rmat

ion

they

will

ne

ed to

pas

s the

form

al

asse

ssm

ents

.

11b.

In th

is c

ours

e, I

prov

ide

the

stud

ents

the

info

rmat

ion

they

will

ne

ed to

pas

s the

cou

rse

asse

ssm

ents

.

Cha

nged

“fo

rmal

ass

essm

ents

” to

cou

rse

asse

ssm

ents

.

12a.

I sh

ould

kno

w th

e an

swer

s to

any

ques

tions

that

stud

ents

may

put

to

me

durin

g th

is c

ours

e.

12b.

I sh

ould

kno

w th

e an

swer

s to

any

ques

tions

abo

ut c

ours

e co

nten

t th

at st

uden

ts m

ay a

sk.

Nar

row

ed fo

cus t

o qu

estio

ns o

ver c

ours

e co

nten

t.

13a.

I m

ake

avai

labl

e op

portu

nitie

s for

st

uden

ts in

this

cou

rse

to d

iscu

ss

thei

r cha

ngin

g un

ders

tand

ing

of th

e su

bjec

t.

13b.

I m

ake

avai

labl

e op

portu

nitie

s for

st

uden

ts in

this

cou

rse

to d

iscu

ss

thei

r cha

ngin

g un

ders

tand

ing

of

the

subj

ect.

No

chan

ge. O

ppor

tuni

ties i

nter

pret

ed to

mea

n in

cla

ss a

nd o

utsi

de o

f cla

ss in

off

ice

hour

s or

othe

r inf

orm

al d

iscu

ssio

ns.

14a.

It is

bet

ter f

or st

uden

ts in

this

co

urse

to g

ener

ate

thei

r ow

n no

tes

rath

er th

an c

opy

min

e.

14b.

Stu

dent

s are

enc

oura

ged

to

gene

rate

thei

r ow

n no

tes o

r mak

e an

nota

tions

on

min

e ra

ther

than

co

py m

ine

verb

atim

.

Rem

oved

bel

ief c

ompo

nent

. Add

ed id

ea o

f an

nota

ting

note

s.

215

Tabl

e 21

, con

tinue

d AT

I Ite

ms U

sed

in P

ilot S

tudy

, Rev

ised

ATI

Item

s, an

d Ra

tiona

le fo

r Rev

isio

n

AT

I ite

ms u

sed

in p

ilot s

tudy

A

TI I

tem

s rev

ised

aft

er p

ilot s

tudy

R

atio

nale

for

revi

sion

15

a. A

lot o

f tea

chin

g tim

e in

this

co

urse

shou

ld b

e us

ed to

que

stio

n st

uden

ts’ i

deas

.

15b.

Stu

dent

s’ id

eas a

re d

iscu

ssed

in

this

cou

rse.

R

emov

ed “

a lo

t” a

nd “

shou

ld”

to fo

cus o

n pr

actic

e no

t bel

ief,

“que

stio

ning

” se

emed

to

impl

y ju

dgm

ent a

nd w

as c

hang

ed to

“di

scus

s”,

rem

oved

“te

achi

ng ti

me”

to a

void

rest

rictin

g qu

estio

n to

onl

y co

nsid

erin

g le

ctur

e tim

e.

16a.

In th

is c

ours

e m

y te

achi

ng fo

cuse

s on

the

good

pre

sent

atio

n of

in

form

atio

n to

stud

ents

.

16b.

In th

is c

ours

e m

y te

achi

ng fo

cuse

s on

the

pres

enta

tion

of in

form

atio

n to

stud

ents

.

Rem

oved

“go

od”.

17a.

I se

e te

achi

ng a

s hel

ping

stud

ents

de

velo

p ne

w w

ays o

f thi

nkin

g in

th

is su

bjec

t.

17b.

My

teac

hing

in th

is c

ours

e he

lps

stud

ents

dev

elop

way

s of t

hink

ing

in th

is su

bjec

t tha

t are

new

to

them

.

Rem

oved

bel

ief c

ompo

nent

. Cla

rify

“new

” w

ays o

f thi

nkin

g as

new

to st

uden

ts.

18a.

In te

achi

ng th

is c

ours

e it

is

impo

rtant

for m

e to

mon

itor

stud

ents

’ cha

nged

und

erst

andi

ng o

f th

e su

bjec

t mat

ter.

18b.

In te

achi

ng th

is c

ours

e, I

mon

itor

stud

ents

’ cha

nged

und

erst

andi

ng

of th

e su

bjec

t mat

ter.

Rem

oved

bel

ief c

ompo

nent

.

19a.

My

teac

hing

in th

is c

ours

e fo

cuse

s on

del

iver

ing

wha

t I k

now

to th

e st

uden

ts.

19b.

My

teac

hing

in th

is c

ours

e fo

cuse

s on

del

iver

ing

wha

t I k

now

to th

e st

uden

ts.

No

chan

ge.

20a.

Tea

chin

g in

this

cou

rse

shou

ld h

elp

stud

ents

que

stio

n th

eir o

wn

unde

rsta

ndin

g of

the

subj

ect m

atte

r.

20b.

My

teac

hing

in th

is c

ours

e he

lps

stud

ents

que

stio

n th

eir o

wn

unde

rsta

ndin

g of

the

subj

ect

mat

ter.

Rem

oved

bel

ief c

ompo

nent

.

21a.

Tea

chin

g in

this

cou

rse

shou

ld

incl

ude

help

ing

stud

ents

find

thei

r ow

n le

arni

ng re

sour

ces.

21b.

My

teac

hing

in th

is c

ours

e in

clud

es h

elpi

ng st

uden

ts fi

nd

thei

r ow

n le

arni

ng re

sour

ces.

Rem

oved

bel

ief c

ompo

nent

.

22a.

I pr

esen

t mat

eria

l to

enab

le

stud

ents

to b

uild

up

an in

form

atio

n ba

se in

this

subj

ect.

22b.

I pr

esen

t mat

eria

l to

enab

le

stud

ents

to b

uild

up

a ba

se o

f co

nten

t kno

wle

dge

and

skill

s in

this

subj

ect.

Cla

rifie

d in

form

atio

n as

incl

udin

g co

nten

t kn

owle

dge

and

skill

s sin

ce th

is w

as a

freq

uent

in

stru

ctor

inte

rpre

tatio

n.

216

App

endi

x E

– S

tude

nt S

urve

y R

evis

ions

Afte

r Pi

lot S

tudy

D

irec

tions

: Ple

ase

com

plet

e th

is su

rvey

by

cons

ider

ing

only

the

lect

ure

porti

on o

f the

che

mis

try c

ours

e in

whi

ch y

ou a

re e

nrol

led.

Pl

ease

do

not c

onsi

der t

he la

bora

tory

por

tion

of th

e co

urse

.

Stro

ngly

D

isag

ree

Dis

agre

e N

eutr

al

Agr

ee

Stro

ngly

A

gree

Q

1. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt c

ours

e to

pics

1

2

3

4

5

Q

2. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt c

ours

e go

als

1

2

3

4

5

Q3.

The

inst

ruct

or p

rovi

ded

clea

r ins

truct

ions

on

how

to p

artic

ipat

e in

cou

rse

lear

ning

act

iviti

es

1

2

3

4

5

Q4.

The

inst

ruct

or c

lear

ly c

omm

unic

ated

impo

rtant

due

dat

es/ti

me

fram

es fo

r co

urse

lear

ning

act

iviti

es

1

2

3

4

5

Q5.

The

inst

ruct

or w

as h

elpf

ul in

faci

litat

ing

disc

ussi

ons o

n co

urse

topi

cs th

at

help

ed m

e to

lear

n

1

2

3

4

5

Q6.

The

inst

ruct

or w

as h

elpf

ul in

gui

ding

the

clas

s tow

ards

und

erst

andi

ng

cour

se to

pics

in a

way

that

hel

ped

me

clar

ify m

y th

inki

ng

1

2

3

4

5

Q7.

The

inst

ruct

or h

elpe

d to

kee

p co

urse

par

ticip

ants

eng

aged

and

pa

rtici

patin

g in

pro

duct

ive

dial

ogue

1

2

3

4

5

Q8.

The

inst

ruct

or h

elpe

d ke

ep th

e co

urse

par

ticip

ants

on

task

in a

way

that

he

lped

me

to le

arn

1

2

3

4

5

Q9.

The

inst

ruct

or e

ncou

rage

d co

urse

par

ticip

ants

to e

xplo

re n

ew c

once

pts i

n th

is c

ours

e 1

2

3

4

5

Q10

. The

inst

ruct

or re

info

rced

the

deve

lopm

ent o

f a se

nse

of c

omm

unity

am

ong

cour

se p

artic

ipan

ts

1

2

3

4

5

Q11

. The

inst

ruct

or h

elpe

d to

focu

s dis

cuss

ion

on re

leva

nt is

sues

in a

way

that

he

lped

me

to le

arn

1

2

3

4

5

Q12

. The

inst

ruct

or p

rovi

ded

feed

back

that

hel

ped

me

unde

rsta

nd m

y st

reng

ths

and

wea

knes

ses r

elat

ive

to th

e co

urse

’s g

oals

and

obj

ectiv

es

1

2

3

4

5

Q13

. The

inst

ruct

or p

rovi

ded

feed

back

in a

tim

ely

fash

ion

1

2

3

4

5

Q14

. Get

ting

to k

now

oth

er c

ours

e pa

rtici

pant

s gav

e m

e a

sens

e of

bel

ongi

ng

in th

e co

urse

. 1

2

3

4

5

Q15

. I w

as a

ble

to fo

rm d

istin

ct im

pres

sion

s of s

ome

cour

se p

artic

ipan

ts

1

2

3

4

5

Q16

. Fac

e-to

-fac

e co

mm

unic

atio

n is

an

exce

llent

med

ium

for s

ocia

l int

erac

tion

1

2

3

4

5

Q17

. I fe

lt co

mfo

rtabl

e co

nver

sing

face

-to-f

ace

in c

lass

1

2

3

4

5

Q

18. I

felt

com

forta

ble

parti

cipa

ting

in th

e co

urse

dis

cuss

ions

1

2

3

4

5

217

St

rong

ly

Dis

agre

e D

isag

ree

Neu

tral

A

gree

St

rong

ly

Agr

ee

Q19

. I fe

lt co

mfo

rtabl

e in

tera

ctin

g w

ith o

ther

cou

rse

parti

cipa

nts

1

2

3

4

5

Q20

. I fe

lt co

mfo

rtabl

e di

sagr

eein

g w

ith o

ther

cou

rse

parti

cipa

nts w

hile

still

m

aint

aini

ng a

sens

e of

trus

t 1

2

3

4

5

Q21

. I fe

lt th

at m

y po

int o

f vie

w w

as a

ckno

wle

dged

by

othe

r cou

rse

parti

cipa

nts

1

2

3

4

5

Q22

. In-

clas

s dis

cuss

ions

hel

ped

me

to d

evel

op a

sens

e of

col

labo

ratio

n 1

2

3

4

5

Q

23. P

robl

ems p

osed

incr

ease

d m

y in

tere

st in

cou

rse

issu

es

1

2

3

4

5

Q24

. Cou

rse

lear

ning

act

iviti

es p

ique

d m

y cu

riosi

ty

1

2

3

4

5

Q25

. I fe

lt m

otiv

ated

to e

xplo

re c

onte

nt re

late

d qu

estio

ns

1

2

3

4

5

Q26

. I u

tiliz

ed a

var

iety

of i

nfor

mat

ion

sour

ces t

o ex

plor

e pr

oble

ms p

osed

in

this

cou

rse

1

2

3

4

5

Q27

. Bra

inst

orm

ing

help

ed m

e re

solv

e co

nten

t rel

ated

que

stio

ns

1

2

3

4

5

Q28

. Fin

ding

rele

vant

info

rmat

ion

help

ed m

e re

solv

e co

nten

t rel

ated

que

stio

ns

1

2

3

4

5

Q29

. Ple

ase

sele

ct “

Dis

agre

e” fo

r thi

s ite

m

1

2

3

4

5

Q30

. In-

clas

s dis

cuss

ions

wer

e va

luab

le in

hel

ping

me

appr

ecia

te d

iffer

ent

pers

pect

ives

1

2

3

4

5

Q31

. Com

bini

ng n

ew in

form

atio

n he

lped

me

answ

er q

uest

ions

rais

ed in

cou

rse

lear

ning

act

iviti

es

1

2

3

4

5

Q32

. Cou

rse

lear

ning

act

iviti

es h

elpe

d m

e co

nstru

ct e

xpla

natio

ns/s

olut

ions

1

2

3

4

5

Q

33. R

efle

ctio

n on

cou

rse

cont

ent h

elpe

d m

e un

ders

tand

fund

amen

tal c

once

pts

in th

is c

lass

1

2

3

4

5

Q34

. Ref

lect

ion

on d

iscu

ssio

ns h

elpe

d m

e un

ders

tand

fund

amen

tal c

once

pts i

n th

is c

lass

1

2

3

4

5

Q35

. I c

an d

escr

ibe

way

s to

test

and

app

ly th

e kn

owle

dge

crea

ted

in th

is c

ours

e 1

2

3

4

5

Q

36. I

hav

e de

velo

ped

solu

tions

to c

ours

e pr

oble

ms t

hat c

an b

e ap

plie

d in

pr

actic

e

1

2

3

4

5

Q37

. I c

an a

pply

the

know

ledg

e cr

eate

d in

this

cou

rse

to m

y w

ork

or o

ther

no

n-cl

ass r

elat

ed a

ctiv

ities

1

2

3

4

5

---

surv

ey c

ontin

ues o

n th

e ne

xt p

age

-- 218

D

irec

tions

: A li

st o

f opp

osin

g w

ords

app

ears

bel

ow. R

ate

how

wel

l the

se w

ords

des

crib

ed y

our f

eelin

gs a

bout

the

lect

ure

porti

on

of y

our c

hem

istry

cou

rse.

For

eac

h lin

e, c

hoos

e a

posi

tion

betw

een

the

two

wor

ds th

at d

escr

ibes

exa

ctly

how

you

felt.

The

mid

dle

posi

tion

is if

you

are

und

ecid

ed o

r hav

e no

feel

ings

rela

ted

to th

e te

rms o

n th

at li

ne.

T

HE

CH

EM

IST

RY

CO

UR

SE W

AS…

M

iddl

e

S1.

C

omfo

rtabl

e 1

2

3

4

5

Unc

omfo

rtabl

e S2

.

S

atis

fyin

g 1

2

3

4

5

Frus

tratin

g S3

.

Ple

asan

t 1

2

3

4

5

Unp

leas

ant

S4.

Cha

otic

1

2

3

4

5

Org

aniz

ed

219

220

Appendix F – R Program for Calculating Coefficient H

coeff.H<-function(x){

denom <- 0

list<-length(x)

for (i in 1:list){

denom <- denom + ((x[i]^2)/(1-x[i]^2))

}

H <- 1/(1+(1/denom))

return(H)

}

221

Appendix G – Path Tracing and Matrix Determination for Hypothesized Research Model Table 22 Algebraic Statements from Path Tracing

Variables Trace

“V1”/”V2” Teaching presence and social presence

!F1bF2F1 !F2

“V1”/”V3” Teaching presence and cognitive presence

!F1bF3F1 !F3 + !F1bF2F1bF3F2 !F3

“V1”/”V4” Teaching presence and student satisfaction

!F1bF4F1 !F4 + !F1bF2F1bF4F2 !F4 + !F1bF3F1bF4F3 !F4 + !F1bF2F1bF3F2bF4F3 !F4

“V1”/V5 Teaching presence and math ability

0

“V1”/V6 Teaching presence and ACS exam score

!F1bF6F1 + !F1bF3F1bF6F3 + !F1bF2F1bF3F2bF6F3

“V1”/V7 Teaching presence and final course grade

!F1bF7F1 + !F1bF6F1bF7F6 + !F1bF3F1bF7F3 + !F1bF3F1bF6F3bF7F6 + !F1bF2F1bF3F2bF6F3bF7F6 +

!F1bF2F1bF3F2bF7F3

“V2”/”V3” Social presence and cognitive presence

!F2bF3F2 !F3 + !F2bF2F1bF3F1 !F3

“V2”/”V4” Social presence and student satisfaction

!F2bF4F2 !F4 + !F2bF3F2bF4F3 !F4 + !F2bF2F1bF4F1 !F4 + !F2bF2F1bF3F1bF4F3 !F4

“V2”/V5 Social presence and math ability

0

“V2”/V6 Social presence and ACS exam score

!F2bF3F2bF6F3 + !F2bF2F1bF6F1 + !F2bF2F1bF3F1bF6F3

“V2”/V7 Social presence and final course grade

!F2bF3F2bF7F3 + !F2bF3F2bF6F3bF7F6 + !F2bF2F1bF7F1 + !F2bF2F1bF6F1bF7F6 + !F2bF2F1bF3F1bF7F3 +

!F2bF2F1bF3F1bF6F3bF7F6

“V3”/”V4” Cognitive presence and student

satisfaction

!F3bF4F3 !F4 + !F3bF3F1bF4F1 !F4 + !F3bF3F2bF4F2 !F4 + !F3bF3F1bF2F1bF4F2 !F4 +

!F3bF3F2bF2F1bF4F1 !F4

222

Table 22, continued Algebraic Statements from Path Tracing

Variables Trace

“V3”/V5 Cognitive presence and math ability

0

“V3”/V6 Cognitive presence and ACS exam score

!F3bF6F3 + !F3bF3F1bF6F1 + !F3bF3F2bF2F1bF6F1

“V3”/V7 Cognitive presence and final course grade

!F3bF7F3 + !F3bF6F3bF7F6 + !F3bF3F1bF7F1 + !F3bF3F1bF6F1bF7F6 + !F3bF3F2bF2F1bF6F1bF7F6 +

!F3bF3F2bF2F1bF7F1

“V4”/V5 Student satisfaction and math ability

0

“V4”/V6 Student satisfaction and ACS exam score

!F4bF4F3bF6F3 + !F4bF4F1bF3F1bF6F3 + !F4bF4F1bF6F1 + !F4bF4F2bF3F2bF6F3 + !F4bF4F2bF2F1bF6F1 +

!F4bF4F2bF2F1bF3F1bF6F3 + !F4bF4F3bF3F1bF6F1 + !F4bF4F3bF3F2bF2F1bF6F1 + !F4bF4F1bF2F1bF3F2bF6F3

“V4”/V7 Student satisfaction and final course grade

!F4bF4F2bF3F2bF7F3 + !F4bF4F2bF3F2bF6F3bF7F6 + !F4bF4F2bF2F1bF7F1 + !F4bF4F2bF2F1bF6F1bF7F6 +

!F4bF4F2bF2F1bF3F1bF6F3bF7F6 + !F4bF4F2bF2F1bF3F1bF7F3 + !F4bF4F1bF7F1 + !F4bF4F1bF6F1bF7F6 +

!F4bF4F1bF2F1bF3F2bF6F3bF7F6 + !F4bF4F1bF2F1bF3F2bF7F3 + !F4bF4F1bF3F1bF6F3bF7F6 +

!F4bF4F1bF3F1bF7F3 + !F4bF4F3bF6F3bF7F6 + !F4bF4F3bF7F3 + !F4bF4F3bF3F2bF2F1bF6F1bF7F6 +

!F4bF4F3bF3F2bF2F1bF7F1 + !F4bF4F3bF3F1bF6F1bF7F6 + !F4bF4F3bF3F1bF7F1 + !F4cF4F7

V5/V6 Math ability and ACS exam score

bF6F5

V5/V7 Math ability and final course grade

bF7F5 + bF6F5bF7F6

V6/V7 ACS exam score and final course grade

bF7F6 + bF6F5bF7F5 + bF6F3bF7F3 + bF6F3bF3F1bF7F1 + bF6F3bF3F2bF2F1bF7F1 + bF6F1bF7F1 + bF6F1bF3F1bF7F3 +

bF6F1bF2F1bF3F2bF7F3

223

R code for generating the model-implied matrix from spreadsheet containing literature loading

library(openxlsx)

loadings<-read.xlsx("Power Analysis.xlsx", 2)

min<-lapply(abs(loadings), min, na.rm=T)

loadings<-rbind(loadings, min)

## F1: Teaching Presence

TP.H<-coeff.H(loadings[9, 1:13])

H1 <-sqrt(TP.H)

## F2: Social Presence

SP.H<-coeff.H(loadings[9, 14:22])

H2 <-sqrt(SP.H)

## F3: Cognitive Presence

CP.H<-coeff.H(loadings[9, 23:34])

H3 <-sqrt(CP.H)

## F4: Student Satisfaction

SATIS.H<-coeff.H(loadings[9, 35:38])

H4 <-sqrt(SATIS.H)

matrix<-matrix(NA, nrow=7, ncol=7)

b21<-0.52

b31<-0.49

b32<-0.3

b41<-0.24

b42<-0.30

b43<-0.26

b61<-0.30

b63<-0.35

b71<-0.30

b73<-0.35

b65<-0.414

b75<-0.414

b76<-0.18

224

c47<-0.38

cor12<-(H1*b21*H2)

cor13<-(H1*b31*H3 + H1*b21*b32*H3)

cor14<-(H1*b41*H4 + H1*b21*b42*H4 + H1*b31*b43*H4 +

H1*b21*b32*b43*H4)

cor15<-0

cor16<-(H1*b61 + H1*b31*b63 + H1*b21*b32*b63)

cor17<-(H1*b71 + H1*b61*b76 + H1*b31*b73 + H1*b31*b63*b76 +

H1*b21*b32*b63*b76 + H1*b21*b32*b73)

cor23<-(H2*b32*H3 + H2*b21*b31*H3)

cor24<-(H2*b42*H4 + H2*b32*b43*H4 + H2*b21*b41*H4 +

H2*b21*b31*b43*H3)

cor25<-0

cor26<-(H2*b32*b63 + H2*b21*b61 + H2*b21*b31*b63)

cor27<-(H2*b32*b73 + H2*b32*b63*b76 + H2*b21*b71 + H2*b21*b61*b76

+ H2*b21*b31*b73 + H2*b21*b31*b63*b76)

cor34<-(H3*b43*H4 + H3*b31*b41*H4 + H3*b32*b42*H4 +

H3*b31*b21*b42*H4 + H3*b32*b21*b41*H4)

cor35<-0

cor36<-(H3*b63 + H3*b31*b61 + H3*b32*b21*b61)

cor37<-(H3*b73 + H3*b63*b76 + H3*b31*b71 + H3*b31*b61*b76 +

H3*b32*b21*b61*b76 + H3*b32*b21*b71)

cor45<-0

cor46<-(H4*b43*b63 + H4*b41*b31*b63 + H4*b41*b61 + H4*b42*b32*b63

+ H4*b42*b21*b61 + H4*b42*b21*b31*b63 +

H4*b43*b31*b61 + H4*b43*b32*b21*b61 +

H4*b41*b21*b32*b63)

cor47<-(H4*b42*b32*b73 + H4*b42*b32*b63*b76 + H4*b42*b21*b71 +

H4*b42*b21*b61*b76 + H4*b42*b21*b31*b63*b76

225

+ H4*b42*b21*b31*b73 + H4*b41*b71 + H4*b41*b61*b76 +

H4*b41*b21*b32*b63*b76 + H4*b41*b21*b32*b73 +

H4*b41*b31*b63*b76 + H4*b41*b31*b73 +

H4*b43*b63*b76 + H4*b43*b73 + H4*b43*b32*b21*b61*b76 +

H4*b43*b32*b21*b71 + H4*b43*b31*b61*b76 +

H4*b43*b31*b71 + H4*c47)

cor56<-(b65)

cor57<-(b75 + b65*b76)

cor67<-(b76 + b65*b75 + b63*b73 + b63*b31*b71 + b63*b32*b21*b71 +

b61*b71 + b61*b31*b73 + b61*b21*b32*b73)

matrix[1,]<-c(1, cor12, cor13, cor14, cor15, cor16, cor17)

matrix[2,]<-c(cor12, 1, cor23, cor24, cor25, cor26, cor27)

matrix[3,]<-c(cor13, cor23, 1, cor34, cor35, cor36, cor37)

matrix[4,]<-c(cor14, cor24, cor34, 1, cor45, cor46, cor47)

matrix[5,]<-c(cor15, cor25, cor35, cor45, 1, cor56, cor57)

matrix[6,]<-c(cor16, cor26, cor36, cor46, cor56, 1, cor67)

matrix[7,]<-c(cor17, cor27, cor37, cor47, cor57, cor67, 1)

R code to print generated matrix for use in LISREL > print(matrix, digits = 5)

[,1] [,2] [,3] [,4] [,5] [,6] [,7]

[1,] 1.00000 0.44660 0.55940 0.48275 0.00000 0.49189 0.58043

[2,] 0.44660 1.00000 0.47200 0.47921 0.00000 0.32167 0.37957

[3,] 0.55940 0.47200 1.00000 0.49306 0.00000 0.50365 0.59431

[4,] 0.48275 0.47921 0.49306 1.00000 0.00000 0.34123 0.75055

[5,] 0.00000 0.00000 0.00000 0.00000 1.00000 0.41400 0.48852

[6,] 0.49189 0.32167 0.50365 0.34123 0.41400 1.00000 0.69956

[7,] 0.58043 0.37957 0.59431 0.75055 0.48852 0.69956 1.00000

226

Appendix H – LISREL Syntax and Output for Power Analysis OBSERVED VARIABLES V1-V7 Covariance Matrix 1 0.44660 1 0.55940 0.47200 1 0.48275 0.47921 0.49306 1 0.00000 0.00000 0.00000 0.000 1 0.49189 0.32167 0.50365 0.34123 0.41400 1 0.58043 0.37957 0.59431 0.75055 0.48852 0.69956 1 SAMPLE SIZE IS 1001 LATENT VARIABLES F1-F7 RELATIONSHIPS V1 = .935*F1 V2 = .919*F2 V3 = .926*F3 V4 = .916*F4 V5 = 1*F5 V6 = 1*F6 V7 = 1*F7 F6 = .30*F1 .35*F3 .414*F5 F7 = .30*F1 .35*F3 .414*F5 .18*F6 F2 = .52*F1 F3 = .49*F1 .30*F2 F4 = .24*F1 .30*F2 .26*F3 LET THE ERRORS OF F4 and F7 COVARY SET THE COVARIANCE OF F1 and F5 to 0 SET THE VARIANCE OF F1 to 1 SET THE VARIANCE OF F5 to 1 SET ERROR VARIANCE OF V1 TO .126 SET ERROR VARIANCE OF V2 TO .156 SET ERROR VARIANCE OF V3 TO .142 SET ERROR VARIANCE OF V4 TO .162 SET ERROR VARIANCE OF V5 TO 0 SET ERROR VARIANCE OF V6 TO 0 SET ERROR VARIANCE OF V7 TO 0 Options: ND=3 RS PATH DIAGRAM END OF PROBLEM Sample Size = 1001 CoI Model- Power Analysis

227

Covariance Matrix V2 V3 V4 V6 V7 V1 -------- -------- -------- -------- -------- ------- V2 1.000 V3 0.472 1.000 V4 0.479 0.493 1.000 V6 0.322 0.504 0.341 1.000 V7 0.380 0.594 0.751 0.700 1.000 V1 0.447 0.559 0.483 0.492 0.580 1.000 V5 - - - - - - 0.414 0.489 - - Covariance Matrix V5 -------- V5 1.000 Total Variance = 7.000 Generalized Variance = 0.0126 Largest Eigenvalue = 3.617 Smallest Eigenvalue = 0.047 Condition Number = 8.767 W_A_R_N_I_N_G : Both LX( 1, 1) and PH( 1, 1) are fixed non-zero values.

LISREL is unable to generate Starting Values for this model. The model will be estimated using the NS option.

W_A_R_N_I_N_G : Both LX( 3, 2) and PH( 2, 2) are fixed non-zero values.

LISREL is unable to generate Starting Values for this model. The model will be estimated using the NS option.

CoI Model- Power Analysis Number of Iterations = 2 LISREL Estimates (Maximum Likelihood) Measurement Equations V2 = 0.919*F2, Errorvar.= 0.156, R² = 0.844 V3 = 0.926*F3, Errorvar.= 0.142, R² = 0.858 V4 = 0.916*F4, Errorvar.= 0.162, R² = 0.838 V6 = 1.000*F6,, R² = 1.000 V7 = 1.000*F7,, R² = 1.000

228

V1 = 0.935*F1, Errorvar.= 0.126, R² = 0.874 V5 = 1.000*F5,, R² = 1.000 Structural Equations F2 = 0.520*F1, Errorvar.= 0.730 , R² = 0.270 Standerr (0.0414) Z-values 17.628 P-values 0.000 F3 = 0.300*F2 + 0.490*F1, Errorvar.= 0.517 , R² = 0.483 Standerr (0.0319) Z-values 16.211 P-values 0.000 F4 = 0.300*F2 + 0.260*F3 + 0.240*F1, Errorvar.= 0.542 , R² = 0.458 Standerr (0.0343) Z-values 15.803 P-values 0.000 F6 = 0.350*F3 + 0.300*F1 + 0.414*F5, Errorvar.= 0.480 , R² = 0.520 Standerr (0.0226) Z-values 21.268 P-values 0.000 F7 = 0.350*F3 + 0.180*F6 + 0.300*F1 + 0.414*F5, Errorvar.= 0.261 , R² = 0.739 Standerr (0.0129) Z-values 20.241 P-values 0.000 NOTE: R² for Structural Equations are Hayduk's (2006) Blocked-Error R² Error Covariance for F7 and F4 = 0.380 (0.0196) 19.407 Reduced Form Equations F2 = 0.520*F1 + 0.0*F5, Errorvar.= 0.730, R² = 0.270 F3 = 0.646*F1 + 0.0*F5, Errorvar.= 0.583, R² = 0.417 F4 = 0.564*F1 + 0.0*F5, Errorvar.= 0.681, R² = 0.318 F6 = 0.526*F1 + 0.414*F5, Errorvar.= 0.552, R² = 0.448 F7 = 0.621*F1 + 0.489*F5, Errorvar.= 0.376, R² = 0.624

229

Correlation Matrix of Independent Variables Note: This matrix is diagonal. F1 F5 -------- -------- 1.000 1.000 Covariance Matrix of Latent Variables F2 F3 F4 F6 F7 F1 -------- -------- -------- -------- -------- ------- F2 1.000 F3 0.555 1.000 F4 0.569 0.582 0.999 F6 0.350 0.544 0.373 1.000 F7 0.413 0.642 0.820 0.700 1.000 F1 0.520 0.646 0.564 0.526 0.621 1.000 F5 - - - - - - 0.414 0.489 - - Covariance Matrix of Latent Variables F5 -------- F5 1.000 W_A_R_N_I_N_G: Matrix above is not positive definite Log-likelihood Values Estimated Model Saturated Model --------------- --------------- Number of free parameters(t) 6 28 -2ln(L) 2631.209 2631.206 AIC (Akaike, 1974)* 2643.209 2687.206 BIC (Schwarz, 1978)* 2672.661 2824.651 *LISREL uses AIC= 2t - 2ln(L) and BIC = tln(N)- 2ln(L) Goodness of Fit Statistics Degrees of Freedom for (C1)-(C2) 22 Maximum Likelihood Ratio Chi-Square (C1) 0.00297 (P = 1.0000) Browne's (1984) ADF Chi-Square (C2_NT) 0.00297 (P = 1.0000) The Fit is Perfect ! Time used 0.060 seconds

23

0

App

endi

x I –

CoI

and

Sat

isfa

ctio

n In

stru

men

t Use

d fo

r St

uden

t Dat

a C

olle

ctio

n D

irec

tions

: Ple

ase

writ

e an

d bu

bble

in y

our G

Num

ber o

n th

e sc

antro

n, n

o ot

her i

nfor

mat

ion

is n

eces

sary

abo

ut th

e co

urse

, etc

…

Plea

se c

onsi

der t

he le

ctur

e po

rtion

of y

our C

HM

115

cou

rse

as y

ou a

nsw

er th

e fo

llow

ing

ques

tions

. Bub

ble

in th

e ap

prop

riate

cho

ice

on y

our

scan

tron.

Tha

nk y

ou fo

r you

r hel

p in

this

dat

a co

llect

ion.

St

rong

ly

Dis

agre

e D

isag

ree

Neu

tral

A

gree

St

rong

ly

Agr

ee

1. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt c

ours

e to

pics

A

B

C

D

E

2.

The

inst

ruct

or c

lear

ly c

omm

unic

ated

impo

rtant

cou

rse

goal

s A

B

C

D

E

3.

The

inst

ruct

or p

rovi

ded

clea

r ins

truct

ions

on

how

to p

artic

ipat

e in

cou

rse

lear

ning

act

iviti

es

A

B

C

D

E

4. T

he in

stru

ctor

cle

arly

com

mun

icat

ed im

porta

nt d

ue d

ates

/tim

e fr

ames

for

cour

se le

arni

ng a

ctiv

ities

A

B

C

D

E

5. T

he in

stru

ctor

was

hel

pful

in fa

cilit

atin

g di

scus

sion

s on

cour

se to

pics

that

he

lped

me

to le

arn

A

B

C

D

E

6. T

he in

stru

ctor

was

hel

pful

in g

uidi

ng th

e cl

ass t

owar

ds u

nder

stan

ding

cou

rse

topi

cs in

a w

ay th

at h

elpe

d m

e cl

arify

my

thin

king

A

B

C

D

E

7. T

he in

stru

ctor

hel

ped

to k

eep

cour

se p

artic

ipan

ts e

ngag

ed a

nd p

artic

ipat

ing

in

prod

uctiv

e di

alog

ue

A

B

C

D

E

8. T

he in

stru

ctor

hel

ped

keep

the

cour

se p

artic

ipan

ts o

n ta

sk in

a w

ay th

at h

elpe

d m

e to

lear

n A

B

C

D

E

9. T

he in

stru

ctor

enc

oura

ged

cour

se p

artic

ipan

ts to

exp

lore

new

con

cept

s in

this

co

urse

A

B

C

D

E

10. T

he in

stru

ctor

rein

forc

ed th

e de

velo

pmen

t of a

sens

e of

com

mun

ity a

mon

g co

urse

par

ticip

ants

A

B

C

D

E

11. T

he in

stru

ctor

hel

ped

to fo

cus d

iscu

ssio

n on

rele

vant

issu

es in

a w

ay th

at

help

ed m

e to

lear

n

A

B

C

D

E

12. T

he in

stru

ctor

pro

vide

d fe

edba

ck th

at h

elpe

d m

e un

ders

tand

my

stre

ngth

s an

d w

eakn

esse

s rel

ativ

e to

the

cour

se’s

goa

ls a

nd o

bjec

tives

A

B

C

D

E

13. T

he in

stru

ctor

pro

vide

d fe

edba

ck in

a ti

mel

y fa

shio

n A

B

C

D

E

14

. Get

ting

to k

now

oth

er c

ours

e pa

rtici

pant

s gav

e m

e a

sens

e of

bel

ongi

ng in

th

e co

urse

. A

B

C

D

E

15. I

was

abl

e to

form

dis

tinct

impr

essi

ons o

f som

e co

urse

par

ticip

ants

A

B

C

D

E

16

. Fac

e-to

-fac

e co

mm

unic

atio

n is

an

exce

llent

med

ium

for s

ocia

l int

erac

tion

A

B

C

D

E

17. I

felt

com

forta

ble

conv

ersi

ng fa

ce-to

-fac

e in

cla

ss

A

B

C

D

E

18. I

felt

com

forta

ble

parti

cipa

ting

in th

e co

urse

dis

cuss

ions

A

B

C

D

E

230

23

1

St

rong

ly

Dis

agre

e D

isag

ree

Neu

tral

A

gree

St

rong

ly

Agr

ee

19. I

felt

com

forta

ble

inte

ract

ing

with

oth

er c

ours

e pa

rtici

pant

s A

B

C

D

E

20

. I fe

lt co

mfo

rtabl

e di

sagr

eein

g w

ith o

ther

cou

rse

parti

cipa

nts w

hile

still

m

aint

aini

ng a

sens

e of

trus

t A

B

C

D

E

21. I

felt

that

my

poin

t of v

iew

was

ack

now

ledg

ed b

y ot

her c

ours

e pa

rtici

pant

s A

B

C

D

E

22

. In-

clas

s dis

cuss

ions

hel

ped

me

to d

evel

op a

sens

e of

col

labo

ratio

n A

B

C

D

E

23

. Pro

blem

s pos

ed in

crea

sed

my

inte

rest

in c

ours

e is

sues

A

B

C

D

E

24

. Cou

rse

lear

ning

act

iviti

es p

ique

d m

y cu

riosi

ty

A

B

C

D

E

25. I

felt

mot

ivat

ed to

exp

lore

con

tent

rela

ted

ques

tions

A

B

C

D

E

26

. I u

tiliz

ed a

var

iety

of i

nfor

mat

ion

sour

ces t

o ex

plor

e pr

oble

ms p

osed

in th

is

cour

se

A

B

C

D

E

27. B

rain

stor

min

g he

lped

me

reso

lve

cont

ent r

elat

ed q

uest

ions

A

B

C

D

E

28

. Fin

ding

rele

vant

info

rmat

ion

help

ed m

e re

solv

e co

nten

t rel

ated

que

stio

ns

A

B

C

D

E

29. P

leas

e se

lect

“D

isag

ree”

for t

his i

tem

A

B

C

D

E

30

. In-

clas

s dis

cuss

ions

wer

e va

luab

le in

hel

ping

me

appr

ecia

te d

iffer

ent

pers

pect

ives

A

B

C

D

E

31. C

ombi

ning

new

info

rmat

ion

help

ed m

e an

swer

que

stio

ns ra

ised

in c

ours

e le

arni

ng a

ctiv

ities

A

B

C

D

E

32. C

ours

e le

arni

ng a

ctiv

ities

hel

ped

me

cons

truct

exp

lana

tions

/sol

utio

ns

A

B

C

D

E

33. R

efle

ctio

n on

cou

rse

cont

ent h

elpe

d m

e un

ders

tand

fund

amen

tal c

once

pts i

n th

is c

lass

A

B

C

D

E

34. R

efle

ctio

n on

dis

cuss

ions

hel

ped

me

unde

rsta

nd fu

ndam

enta

l con

cept

s in

this

cl

ass

A

B

C

D

E

35. I

can

des

crib

e w

ays t

o te

st a

nd a

pply

the

know

ledg

e cr

eate

d in

this

cou

rse

A

B

C

D

E

36. I

hav

e de

velo

ped

solu

tions

to c

ours

e pr

oble

ms t

hat c

an b

e ap

plie

d in

pra

ctic

e

A

B

C

D

E

37. I

can

app

ly th

e kn

owle

dge

crea

ted

in th

is c

ours

e to

my

wor

k or

oth

er n

on-

clas

s rel

ated

act

iviti

es

A

B

C

D

E

Dir

ectio

ns: A

list

of o

ppos

ing

wor

ds a

ppea

rs b

elow

. Rat

e ho

w w

ell t

hese

wor

ds d

escr

ibed

you

r fee

lings

abo

ut th

e le

ctur

e po

rtion

of y

our

chem

istry

cou

rse.

For

eac

h lin

e, c

hoos

e a

posi

tion

betw

een

the

two

wor

ds th

at d

escr

ibes

exa

ctly

how

you

felt.

The

mid

dle

posi

tion

is if

you

are

un

deci

ded

or h

ave

no fe

elin

gs re

late

d to

the

term

s on

that

line

. T

HE

CH

EM

IST

RY

CO

UR

SE W

AS…

M

iddl

e

38.

C

omfo

rtabl

e A

B

C

D

E

Unc

omfo

rtabl

e 39

.

S

atis

fyin

g A

B

C

D

E

Frus

tratin

g 40

.

Ple

asan

t A

B

C

D

E

Unp

leas

ant

41.

Cha

otic

A

B

C

D

E

Org

aniz

ed

231

232

Appendix J – Interview Transcript and Course Syllabus Coding Rubric Scoring system: 0 = no evidence of indicator in either interview or syllabus 1 = some evidence of indicator, possibly indirect 2 = definite evidence of indicator in either interview transcript or syllabus 3 = definite evidence of indicator in both interview transcript or syllabus, or multiple mentions in either interview transcript or syllabus Instructor Code:

Indicator of student-centered constructivist learning environment Score Evidence/Quote

Instructor plans to spend some portion of class time not lecturing so that students can participate in some type of learning activity/problem solving

Instructor describes his or her role as providing guidance for or facilitating students’ own learning

Instructor understands student learning to be a process that involves students actively engaging with material and/or constructing their own knowledge

Instructor asks (or requires) students to work in groups during learning activities/problem solving

Instructor emphasizes students understanding underlying concepts in addition to being able to solve mathematical problems

Instructor incorporates authentic problem solving tasks

Learning activities/problem solving tasks require students to test or apply their knowledge

Instructor utilizes discussions as a way to probe student understanding

233

Appendix K – Mplus Model Syntax

One-factor teaching presence model with 13 items and auxiliary variables TITLE: One factor model for teaching presence with 13 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999);

AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T1-T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES; One-factor teaching presence model with 13 items, auxiliary variables, and error covariance TITLE: One factor model for teaching presence with 13 items add AUX variables and T12 T13 error covariance DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T1-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;

234

Two-factor teaching presence model with 13 items, auxiliary variables, and error covariance TITLE: Two factor model for teaching presence with 13 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T1-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: PRECOURSE BY T1-T4; INCURS BY T5-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES; One-factor teaching presence model with 9 items, auxiliary variables, and error covariance TITLE: One factor model for teaching presence with 9 items DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T5-T13; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; T12 WITH T13; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;

235

Three-factor CoI model with auxiliary variables and error covariance terms TITLE: Three factor model for CoI instrument with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = T5-T13 S1-S9 C1-C14; MISSING ALL (-999999); AUXILIARY (m) Course ACS ACT; ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;

236

Measurement model with error covariance terms TITLE: Measurement model with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;

237

Structural model with error covariance terms TITLE: Structural model with 9 item teaching presence DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS: ESTIMATOR IS MLR; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; SOCIAL ON TEACH; COGNITIVE ON TEACH SOCIAL; SATISF ON TEACH COGNITIVE SOCIAL; CHEM ON TEACH COGNITIVE MATH; GRADE ON TEACH COGNITIVE MATH CHEM; SATISF WITH GRADE; TEACH WITH MATH@0; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; OUTPUT: SAMPSTAT STANDARDIZED MODINDICES;

238

Structural model requesting bootstrapped confidence intervals for indirect effects TITLE: Structural model with 9 item teaching presence and bootstrapped CIs DATA: FILE IS Dissertation.data.csv; VARIABLE: NAMES ARE Course ACS ACT T1-T13 S1-S9 C1-C14 SS1-SS4; USEVARIABLES = Course ACS ACT T5-T13 S1-S9 C1-C14 SS1-SS4; MISSING ALL (-999999); ANALYSIS:

BOOTSTRAP = 250; MODEL: TEACH BY T5-T13; SOCIAL BY S1-S9; COGNITIVE BY C1-C14; SATISF BY SS1-SS4; MATH BY ACT; CHEM BY ACS; GRADE BY Course; SOCIAL ON TEACH; COGNITIVE ON TEACH SOCIAL; SATISF ON TEACH COGNITIVE SOCIAL; CHEM ON TEACH COGNITIVE MATH; GRADE ON TEACH COGNITIVE MATH CHEM; SATISF WITH GRADE; TEACH WITH MATH@0; ACT@0; ACS@0; Course@0; T12 WITH T13; S1 WITH S2; S3 WITH S4; S4 WITH S9; MODEL INDIRECT: CHEM IND TEACH; CHEM IND SOCIAL; GRADE IND TEACH; GRADE IND SOCIAL; GRADE IND COGNITIVE; SATISF IND TEACH; SATISF IND SOCIAL; COGNITIVE IND TEACH; OUTPUT:

SAMPSTAT STANDARDIZED MODINDICES CINTERVAL(bootstrap);

239

Appendix L – Student Variable Correlations from Mplus Table 23 Correlation Matrix from Mplus COURSE ACS ACT T5 T6 T7 T8 T9 T10 T11 T12 T13 COURSE 1.000

ACS 0.856 1.000 ACT 0.450 0.496 1.000

T5 0.230 0.160 0.062 1.000 T6 0.313 0.252 0.188 0.718 1.000 T7 0.140 0.084 -0.005 0.589 0.550 1.000 T8 0.288 0.203 0.114 0.623 0.684 0.644 1.000 T9 0.185 0.109 -0.019 0.474 0.415 0.474 0.446 1.000

T10 0.034 -0.030 -0.061 0.439 0.353 0.465 0.380 0.511 1.000 T11 0.198 0.110 0.057 0.558 0.649 0.503 0.617 0.462 0.386 1.000 T12 0.270 0.179 0.051 0.507 0.544 0.411 0.471 0.362 0.266 0.445 1.000 T13 0.215 0.169 0.125 0.471 0.568 0.425 0.460 0.400 0.334 0.454 0.553 1.000

S1 0.064 0.010 -0.008 0.283 0.291 0.310 0.289 0.286 0.441 0.314 0.205 0.319 S2 0.056 -0.007 -0.022 0.183 0.207 0.191 0.214 0.263 0.341 0.265 0.187 0.216 S3 0.115 0.063 0.019 0.295 0.317 0.259 0.334 0.275 0.325 0.291 0.261 0.401 S4 0.167 0.126 0.052 0.333 0.393 0.283 0.387 0.286 0.291 0.398 0.273 0.325 S5 0.194 0.166 0.041 0.517 0.487 0.405 0.505 0.322 0.359 0.475 0.361 0.347 S6 0.147 0.081 0.005 0.351 0.326 0.278 0.316 0.320 0.348 0.331 0.161 0.233 S7 0.178 0.134 0.054 0.322 0.301 0.292 0.332 0.346 0.342 0.415 0.228 0.172 S8 0.261 0.215 0.104 0.413 0.354 0.338 0.413 0.336 0.348 0.479 0.280 0.257 S9 0.209 0.148 -0.005 0.499 0.408 0.452 0.481 0.415 0.430 0.495 0.375 0.258 C1 0.304 0.254 0.112 0.424 0.455 0.390 0.427 0.459 0.323 0.405 0.369 0.379 C2 0.279 0.233 0.075 0.345 0.374 0.345 0.372 0.442 0.309 0.396 0.297 0.363 C3 0.300 0.241 0.072 0.390 0.415 0.381 0.424 0.463 0.347 0.397 0.413 0.336 C4 0.194 0.060 -0.099 0.236 0.236 0.223 0.266 0.377 0.276 0.228 0.233 0.236 C5 0.207 0.111 -0.046 0.371 0.362 0.310 0.299 0.384 0.337 0.313 0.308 0.347 C6 0.287 0.189 0.047 0.350 0.393 0.323 0.355 0.343 0.320 0.390 0.356 0.391 C7 0.170 0.066 0.002 0.459 0.459 0.462 0.419 0.382 0.402 0.437 0.314 0.391 C8 0.251 0.161 0.002 0.443 0.438 0.399 0.456 0.428 0.387 0.454 0.377 0.379 C9 0.275 0.196 0.094 0.543 0.570 0.469 0.510 0.473 0.380 0.507 0.457 0.461

C10 0.319 0.238 0.109 0.461 0.508 0.427 0.470 0.468 0.388 0.462 0.432 0.352 C11 0.269 0.175 0.025 0.479 0.576 0.469 0.505 0.495 0.414 0.551 0.472 0.453 C12 0.358 0.304 0.165 0.499 0.607 0.435 0.498 0.451 0.345 0.493 0.498 0.522 C13 0.368 0.295 0.164 0.422 0.508 0.420 0.482 0.467 0.332 0.456 0.448 0.465 C14 0.293 0.252 0.126 0.381 0.464 0.374 0.421 0.422 0.317 0.454 0.390 0.385 SS1 -0.521 -0.433 -0.219 -0.410 -0.482 -0.267 -0.453 -0.286 -0.190 -0.435 -0.364 -0.349 SS2 -0.501 -0.447 -0.222 -0.472 -0.565 -0.320 -0.495 -0.307 -0.202 -0.453 -0.384 -0.365 SS3 -0.479 -0.418 -0.237 -0.469 -0.505 -0.332 -0.457 -0.348 -0.238 -0.424 -0.391 -0.400 SS4 0.242 0.249 0.052 0.402 0.395 0.300 0.339 0.253 0.221 0.312 0.319 0.347

Note. COURSE = final course grade; ACS = ACS exam scores; ACT = ACT math scores; T = teaching presence item; S = social presence item; C = cognitive presence item; SS = satisfaction item

240

Table 23, continued Correlation Matrix from Mplus S1 S2 S3 S4 S5 S6 S7 S8 S9 C1 C2 C3 C4

S1 1.000 S2 0.626 1.000 S3 0.415 0.403 1.000 S4 0.467 0.433 0.625 1.000 S5 0.459 0.398 0.441 0.631 1.000 S6 0.575 0.540 0.484 0.671 0.574 1.000 S7 0.481 0.452 0.304 0.515 0.524 0.628 1.000 S8 0.435 0.421 0.318 0.489 0.541 0.561 0.636 1.000 S9 0.479 0.399 0.346 0.360 0.556 0.469 0.506 0.565 1.000 C1 0.398 0.264 0.267 0.305 0.430 0.338 0.310 0.378 0.422 1.000 C2 0.396 0.261 0.258 0.238 0.387 0.335 0.314 0.332 0.421 0.691 1.000 C3 0.400 0.303 0.279 0.313 0.413 0.338 0.359 0.338 0.415 0.706 0.715 1.000 C4 0.240 0.195 0.281 0.154 0.214 0.244 0.240 0.238 0.296 0.341 0.387 0.471 1.000 C5 0.367 0.302 0.276 0.250 0.326 0.362 0.331 0.325 0.400 0.444 0.487 0.486 0.452 C6 0.405 0.297 0.255 0.298 0.364 0.350 0.394 0.402 0.380 0.440 0.415 0.480 0.477 C7 0.458 0.341 0.268 0.311 0.411 0.442 0.422 0.498 0.509 0.455 0.425 0.422 0.305 C8 0.333 0.331 0.304 0.326 0.411 0.381 0.386 0.473 0.473 0.482 0.462 0.481 0.394 C9 0.413 0.293 0.308 0.351 0.398 0.380 0.360 0.442 0.464 0.538 0.505 0.557 0.363

C10 0.336 0.289 0.294 0.347 0.388 0.349 0.382 0.400 0.458 0.566 0.518 0.598 0.362 C11 0.409 0.318 0.325 0.418 0.454 0.377 0.371 0.460 0.529 0.521 0.512 0.580 0.356 C12 0.323 0.241 0.347 0.377 0.418 0.314 0.288 0.345 0.427 0.566 0.524 0.609 0.323 C13 0.330 0.262 0.319 0.352 0.391 0.342 0.321 0.363 0.440 0.543 0.498 0.590 0.388 C14 0.315 0.273 0.216 0.260 0.331 0.305 0.277 0.361 0.393 0.521 0.496 0.576 0.341 SS1 -0.297 -0.236 -0.224 -0.320 -0.416 -0.323 -0.349 -0.387 -0.412 -0.406 -0.391 -0.418 -0.155 SS2 -0.200 -0.132 -0.137 -0.214 -0.334 -0.205 -0.218 -0.277 -0.332 -0.471 -0.438 -0.482 -0.181 SS3 -0.232 -0.172 -0.165 -0.277 -0.379 -0.271 -0.275 -0.357 -0.365 -0.504 -0.429 -0.493 -0.219 SS4 0.208 0.083 0.209 0.218 0.284 0.236 0.200 0.223 0.227 0.318 0.234 0.304 0.082 Note. S = social presence item; C = cognitive presence item; SS = satisfaction item

241

Table 23, continued Correlation Matrix from Mplus

C5 C6 C7 C8 C9 C10 C11 C12 C13 C4 SS1 SS2 SS3 C5 1.000 C6 0.605 1.000 C7 0.408 0.456 1.000 C8 0.458 0.514 0.580 1.000 C9 0.481 0.547 0.560 0.664 1.000

C10 0.444 0.508 0.482 0.554 0.673 1.000 C11 0.506 0.489 0.524 0.572 0.674 0.710 1.000 C12 0.482 0.437 0.437 0.529 0.627 0.598 0.673 1.000 C13 0.505 0.528 0.444 0.560 0.647 0.603 0.617 0.727 1.000 C14 0.372 0.416 0.416 0.516 0.535 0.554 0.591 0.658 0.644 1.000 SS1 -0.315 -0.337 -0.360 -0.403 -0.409 -0.446 -0.401 -0.441 -0.474 -0.414 1.000 SS2 -0.325 -0.290 -0.317 -0.411 -0.466 -0.446 -0.441 -0.502 -0.463 -0.482 0.712 1.000 SS3 -0.325 -0.310 -0.367 -0.429 -0.429 -0.457 -0.469 -0.516 -0.491 -0.518 0.726 0.792 1.000 SS4 0.196 0.224 0.253 0.341 0.345 0.400 0.388 0.383 0.315 0.327 -0.499 -0.454 -0.505 Note. C = cognitive presence item; SS = satisfaction item

242

References Abraham, M. R. (2008). Importance of a theoretical framework for research. In D. M. Bunce &

R. S. Cole (Eds.), Nuts and bolts of chemical education research (pp. 47–66). Washington,

DC: American Chemical Society. doi:10.1021/bk-2008-0976.ch005

ACT. (2015). Understand your scores. Retrieved October 21, 2015, from

http://www.actstudent.org/scores/understand/

American Educational Research Association, American Psychological Association, & National

Council on Measurement in Education. (2014). Standards for educational & psychological

testing. Washington, DC: American Educational Research Association.

Arbaugh, J. B. (2000). Virtual classroom characteristics and student satisfaction with internet-

based MBA Courses. Journal of Management Education, 24(1), 32–54.

doi:10.1177/105256290002400104

Arbaugh, J. B. (2007). An empirical verification of the Community of Inquiry framework.

Journal of Asynchronous Learning Networks, 11(1), 73–85.

Arbaugh, J. B. (2008). Does the community of inquiry framework predict outcomes in online

MBA courses? International Review of Research in Open and Distance Learning, 9(2), 1–

14.

Arbaugh, J. B., Bangert, A. W., & Cleveland-Innes, M. (2010). Subject matter effects and the

Community of Inquiry (CoI) framework: An exploratory study. The Internet and Higher

Education, 13(1-2), 37–44. doi:10.1016/j.iheduc.2009.10.006

243

Arbaugh, J. B., & Benbunan-Fich, R. (2007). The importance of participant interaction in online

environments. Decision Support Systems, 43(3), 853–865. doi:10.1016/j.dss.2006.12.013

Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., &

Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of

the Community of Inquiry framework using a multi-institutional sample. Internet and

Higher Education, 11(3-4), 133–136. doi:10.1016/j.iheduc.2008.06.003

Arjoon, J. A., Xu, X., & Lewis, J. E. (2013). Understanding the state of the art for measurement

in chemistry education research: Examining the psychometric evidence. Journal of

Chemical Education, 90(5), 536–545. doi:10.1021/ed3002013

Ausubel, D. P. (1960). The use of advance organizers in the learning and retention of meaningful

verbal material. Journal of Educational Psychology, 51(5), 267–272. doi:10.1037/h0046669

Bächtold, M. (2013). What do students “construct” according to constructivism in science

education? Research in Science Education, 43, 2477–2496. doi:10.1007/s11165-013-9369-7

Bangert, A. W. (2008). The development and validation of the student evaluation of online

teaching effectiveness. Computers in the Schools, 25(1-2), 25–47.

doi:10.1080/07380560802157717

Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal

of Educational and Behavioral Statistics, 28(2), 135–167. doi:10.3102/10769986028002135

Bernal, P. J. (2006). Addressing the philosophical confusion regarding constructivism in

chemical education. Journal of Chemical Education, 83(2), 324–326.

doi:10.1021/ed083p324

244

Bodner, G. M. (1986). Constructivism: A theory of knowledge. Journal of Chemical Education,

63, 873–878.

Bodner, G. M., Klobuchar, M., & Geelan, D. (2001). The many forms of constructivism. Journal

of Chemical Education, 78(8), 1107. doi:10.1021/ed078p1107.4

Bolliger, D. U., & Wasilik, O. (2012). Student satisfaction in large undergraduate online courses.

Quarterly Review of Distance Education, 13(3), 153–165.

Brandriet, A., Reed, J. J., & Holme, T. (2015). A historical investigation into item formats of

ACS Exams and their relationships to science practices. Journal of Chemical Education,

150817104855004. doi:10.1021/acs.jchemed.5b00459

Bunce, D. M. (2001). Does Piaget still have anything to say to chemists? Journal of Chemical

Education, 78(8), 1107. doi:10.1021/ed078p1107.2

Chambers, K., & Blake, B. (2007). Enhancing student performance in first-semester general

chemistry using active feedback through the World Wide Web. Journal of Chemical

Education, 84(7), 1130–1135.

Coll, R. K., France, B., & Taylor, I. (2005). The role of models/and analogies in science

education: implications from research. International Journal of Science Education, 27(2),

183–198. doi:10.1080/0950069042000276712

Coll, R. K., & Taylor, T. G. N. (2001). Using constructivism to inform tertiary chemistry

pedagogy. Chemistry Education Research and Practice, 2(3), 215. doi:10.1039/b1rp90024b

245

Committee on Developments in the Science of Learning. (2000). How people learn: Brain, mind,

experience, and school. (J. D. Bransford, A. L. Brown, & R. R. Cocking, Eds.).

Washington, DC: National Academy Press.

Conway, C. J. (2014). Effects of guided inquiry versus lecture instruction on final grade

distribution in a one-semester organic and biochemistry course. Journal of Chemical

Education, 91, 480–483. doi:10.1021/ed300137z

Craney, C. L., & Armstrong, R. W. (1985). Predictors of grades in general chemistry for allied

health students. Journal of Chemical Education, 62(2), 127–129.

Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five

approaches (3rd ed.). Los Angeles: SAGE.

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods

approaches (4th ed.). Los Angeles: SAGE.

Creswell, J. W., & Clark, V. L. P. (2007). Choosing a mixed methods design. In Designing and

conducting mixed methods research (pp. 53–106). Thousand Oaks, CA: Sage.

DeFever, R. S., Bruce, H., & Bhattacharyya, G. (2015). Mental rolodexing: Senior chemistry

majors’ understanding of chemical and physical properties. Journal of Chemical Education,

92(3), 415–426. doi:10.1021/ed500360g

Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and

delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational

communications and technology (pp. 170–198). New York: Macmillan Reference USA.

246

Enders, C. K. (2013). Analyzing structural equation models with missing data. In G. R. Hancock

& R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 493–

519). Charlotte, NC: Information Age Publishing.

Ferguson, R. L. (2007). Constructivism and social constructivism. In G. M. Bodner & M. Orgill

(Eds.), Theoretical frameworks for research in chemistry/science education (pp. 28–49).

Upper Saddle River, NJ: Pearson Education.

Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics (4th ed.). Los Angeles:

SAGE.

Finney, S. J., & DiStefano, C. (2013). Non-normal and categorical data in structural equation

modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A

second course (2nd ed., pp. 439–492). Charlotte, NC: Information Age Publishing.

doi:10.1038/sj.bjc.6602399

Flynn, A. B., & Ogilvie, W. W. (2015). Mechanisms before reactions: A mechanistic approach to

the organic chemistry curriculum based on patterns of electron flow. Journal of Chemical

Education, 92(5), 803–810. doi:10.1021/ed500284d

Fosnot, C. T., & Perry, R. S. (2005). Constructivism: A psychological theory of learning. In C.

T. Fosnot (Ed.), Constructivism: Theory, perspectives, and practice (2nd ed., pp. 8–33).

New York: Teachers College Press.

Fox, R. (2001). Constructivism examined. Oxford Review of Education, 27(1), 23–35.

247

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., &

Wenderoth, M. P. (2014). Active learning increases student performance in science,

engineering, and mathematics. Proceedings of the National Academy of Sciences of the

United States of America, 111(23), 8410–8415. doi:10.1073/pnas.1319030111

Gardner, P. L. (1975). Attitudes to science: A review. Studies in Science Education, 2(1), 1–41.

doi:10.1080/03057267508559818

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based

environment: Computer conferencing in higher education. The Internet and Higher

Education, 2(2-3), 87–105.

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework:

Review, issues, and future directions. Internet and Higher Education, 10, 157–172.

doi:10.1016/j.iheduc.2007.04.001

Garrison, D. R., Cleveland-Innes, M., & Fung, T. (2004). Student role adjustment in online

communities of inquiry: Model and instrument validation. Journal of Asynchronous

Learning Network, 8(2), 61–74.

Garrison, D. R., Cleveland-Innes, M., & Fung, T. S. (2010). Exploring causal relationships

among teaching, cognitive and social presence: Student perceptions of the community of

inquiry framework. The Internet and Higher Education, 13(1-2), 31–36.

doi:10.1016/j.iheduc.2009.10.002

248

Garrison, J. (1995). Deweyan pragmatism and the epistemology of contemporary social

constructivism. American Educational Research Journal, 32(4), 716–740.

doi:10.3102/00028312032004716

Good, R. G., Wandersee, J. H., & St. Julien, J. (1993). Cautionary notes on the appeal of the new

“ism” (constructivism) in science education. In K. Tobin (Ed.), The practice of

constructivism in science education (pp. 71–87). Hillsdale: NJ: Lawrence Erlbaum

Associates.

Gosser, D. K., Kampmeier, J. A., & Varma-Nelson, P. (2010). Peer-led team learning: 2008

James Flack Norris award address. Journal of Chemical Education, 87(4), 374–380.

doi:10.1021/ed800132w

Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of

student ratings. The American Psychologist, 52(11), 1209–1217. doi:10.1037/0003-

066X.52.11.1209

Gupta, T., Burke, K. A., Mehta, A., & Greenbowe, T. J. (2015). Impact of guided-inquiry-based

instruction with a writing and reflection emphasis on chemistry students’ critical thinking

abilities. Journal of Chemical Education. doi:10.1021/ed500059r

Hall, D. M., Curtin-Soydan, A. J., & Canelas, D. A. (2014). The science advancement through

group engagement program: Leveling the playing field and increasing retention in science.

Journal of Chemical Education, 91(1), 37–47. doi:10.1021/ed400075n

249

Hancock, G. R. (1997). Structural equation modeling methods of hypothesis testing of latent

variable means. Measurement and Evaluation in Counseling and Development, 30(2), 91–

105.

Hancock, G. R. (2001). Effect size, power, and sample size determination for structured means

modeling and MIMIC approaches to between-groups hypothesis testing of means on a

single latent construct. Psychometrika, 66(3), 373–388. doi:10.1007/BF02294440

Hancock, G. R. (2006). Power analysis in covariance structure modeling. In G. R. Hancock & R.

O. Mueller (Eds.), Structural equation modeling: A second course (1st ed., pp. 69–115).

Charlotte, NC: Information Age Publishing.

Hancock, G. R., & French, B. F. (2013). Power analysis in structural equation modeling. In G. R.

Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed.,

pp. 117–159). Charlotte, NC: Information Age Publishing.

Hancock, G. R., & Mueller, R. O. (Eds.). (2006). Structural equation modeling: A second course

(1st ed.). Greenwich, CT: Information Age Publishing.

Hanson, D. M. (2006). Instructor’s guide to process-oriented guided-inquiry learning. Lisle, IL:

Pacific Crest.

Hanson, D. M. (2008). A cognitive model for learning chemistry and solving problems:

Implications for curriculum design and classroom instruction. In R. S. Moog & J. N.

Spencer (Eds.), Process oriented guided inquiry learning (POGIL) (Vol. 994, pp. 14–25).

Washington, DC: American Chemical Society. doi:10.1021/bk-2008-0994.ch002

250

Herron, J. D. (1975). Piaget for chemists. Journal of Chemical Education, 52(3), 146–150.

Howard, G. S., & Maxwell, S. E. (1982). Do grades contaminate student evaluations of

instruction? Research in Higher Education, 16(2), 175–188. doi:10.1007/BF00973508

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:

Conventional criteria versus new alternatives. Structural Equation Modeling: A

Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118

Huta, V. (2014). When to use hierarchical linear modeling. The Quantitative Methods for

Psychology, 10(1), 13–28.

Hyslop-Margison, E. J., & Strobel, J. (2008). Constructivism and education: Misunderstandings

and pedagogical implications. The Teacher Educator, 43, 72–86.

doi:10.1080/08878730701728945

Indiana University Center for Postsecondary Research. (n.d.). The Carnegie Classification of

Institutions of Higher Education, 2015 edition. Retrieved October 15, 2015, from

http://carnegieclassifications.iu.edu/

Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical

paradigm? Educational Technology Research and Development, 39(3), 5–14.

doi:10.1007/BF02296434

Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online university students’ satisfaction and

persistence: Examining perceived level of presence, usefulness and ease of use as predictors

in a structural model. Computers & Education, 57(2), 1654–1664.

doi:10.1016/j.compedu.2011.02.008

251

Jöreskog, K. G., & Sörbom, D. (2015). LISREL. Skokie, IL: Scientific Software International,

Inc. [Computer Software].

Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New

York: The Guilford Press.

Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J.

D. Wright (Eds.), Handbook of survey research (2nd ed.). Bingley, UK: Emerald Group

Publishing Limited.

Lewis, S. E. (2014). Examining evidence for external and consequential validity of the first term

general chemistry exam from the ACS Examinations Institute. Journal of Chemical

Education. doi:10.1021/ed400819g

Lewis, S. E., & Lewis, J. E. (2005). Departing from lectures: An evaluation of a peer-led guided

inquiry alternative. Journal of Chemical Education, 82(1), 135–139.

doi:10.1021/ed082p135

Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to

parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9(2),

151–173.

Lund, T. J., Pilarz, M., Velasco, J. B., Chakraverty, D., Rosploch, K., Undersander, M., & Stains,

M. (2015). The best of both worlds: Building on the COPUS and RTOP observation

protocols to easily and reliably measure various levels of reformed instructional practice.

CBE Life Sciences Education, 14(2). doi:10.1187/cbe.14-10-0168

252

Matthews, M. R. (1993). Constructivism and science education: Some epistemological problems.

Journal of Science Education and Technology, 2(1), 359–370. doi:10.1007/BF00694598

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–

103). New York: Macmillan.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’

responses and performances as scientific inquiry into score meaning. American

Psychologist. doi:10.1037/0003-066X.50.9.741

Mitchell, Y. D., Ippolito, J., & Lewis, S. E. (2012). Evaluating Peer-Led Team Learning across

the two semester General Chemistry sequence. Chemistry Education Research and

Practice, 13(3), 378–383. doi:10.1039/c2rp20028g

Mueller, R. O., & Hancock, G. R. (2008). Best practices in structural equation modeling. In J. W.

Osborne (Ed.), Best practices in quantitative methods (pp. 488–508). Thousand Oaks, CA:

Sage Publications, Inc. doi:10.4135/9781412995627

Mueller, R. O., & Hancock, G. R. (2010). Structural equation modeling. In G. R. Hancock & R.

O. Mueller (Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp.

371–383). New York: Routledge.

Mugaloglu, E. Z. (2014). The problem of pseudoscience in science education and implications of

constructivist pedagogy. Science and Education, 23, 829–842. doi:10.1007/s11191-013-

9670-x

Muthén, L. K., & Muthén, B. O. (2010). Mplus User’s Guide. Seventh Edition. Los Angeles, CA:

Muthén & Muthén.

253

Nordstrom, B. (1990). Predicting performance in freshman chemistry. In National Meeting of the

American Chemical Soceity. Boston.

Norman, D. A. (1973). Cognitive organization and learning. La Jolla, CA.

Norman, D. A. (1980). What goes on in the mind of the learner. New Directions for Teaching

and Learning, (2), 37–49. doi:10.1002/tl.37219800205

Novak, J. (1984). Application of advances in learning theory and philosophy of science to the

improvement of chemistry teaching. Journal of Chemical Education, 61(7), 607–612.

doi:10.1021/ed061p607

Nurrenbern, S. C. (2001). Piaget’s theory of intellectual development revisited. Journal of

Chemical Education, 78(8), 1107–1110.

Partlow, K. M., & Gibbs, W. J. (2003). Indicators of constructivist principles in internet-based

courses. Journal of Computing in Higher Education, 14(2), 68–97.

doi:10.1007/BF02940939

Phillips, D. C. (1995). The good, the bad, and the ugly: The many faces of constructivism.

Educational Researcher, 24(7), 5–12. doi:10.3102/0013189X024007005

Piaget, J. (1973). To understand is to invent: The future of education. New York: Grossman.

Piaget, J. (1997). Development and learning. In M. Gauvain & M. Cole (Eds.), Readings on the

development of children (2nd ed., pp. 19–28). New York: W. H. Freeman (Reprinted from

Piaget Rediscovered, pp. 7-20, by R. E. Ripple & V. N. Rockcastle, Eds., 1964).

Preacher, K. J., & Coffman, D. L. (2006). Computing power and minimum sample size for

RMSEA [Computer software].

254

Prosser, M., & Trigwell, K. (2006). Confirmatory factor analysis of the Approaches to Teaching

Inventory. British Journal of Educational Psychology, 76, 405–419.

doi:10.1348/000709905X43571

Prosser, M., Trigwell, K., & Taylor, P. (1994). A phenomenographic study of academics’

conceptions of science learning and teaching. Learning and Instruction, 4(3), 217–231.

doi:10.1016/0959-4752(94)90024-8

R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria:

R Foundation for Statistical Computing [Computer Software].

Revelle, W. (2015). psych: Procedures for psychological, psychometric, and personality

research. Evanston, IL: Northwestern University.

Ruder, S. M., & Hunnicutt, S. S. (2008). POGIL in chemistry courses at a large urban university:

A case study. In R. S. Moog & J. N. Spencer (Eds.), Process oriented guided inquiry

learning (POGIL). doi:10.1021/bk-2008-0994.ch012

Scerri, E. R. (2003). Philosophical confusion in chemical education research. Journal of

Chemical Education, 80(5), 468–474. doi:10.1021/ed080p468

Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster

“epistemic engagement” and “cognitive presence” in online education. Computers and

Education, 52, 543–553. doi:10.1016/j.compedu.2008.10.007

Shea, P., Sau Li, C., & Pickett, A. (2006). A study of teaching presence and student sense of

learning community in fully online and web-enhanced college courses. The Internet and

Higher Education, 9(3), 175–190. doi:10.1016/j.iheduc.2006.06.005

255

Shin, N. (2003). Transactional presence as a critical predictor of success in distance learning.

Distance Education, 24(1), 69–86. doi:10.1080/01587910303048

Smith, J., Wilson, S. B., Banks, J., Zhu, L., & Varma-Nelson, P. (2014). Replicating Peer-Led

Team Learning in cyberspace: Research, opportunities, and challenges. Journal of Research

in Science Teaching, 51(6), 714–740. doi:10.1002/tea.21163

Spencer, H. E. (1996). Mathematical SAT test scores and college chemistry grades. Journal of

Chemical Education, 73(12), 1150–1153. doi:10.1021/ed073p1150

Spencer, J. N. (1992). General chemistry course content. Journal of Chemical Education, 69(3),

182–186.

Stains, M., Pilarz, M., & Chakraverty, D. (2015). Short and long-term impacts of the Cottrell

Scholars Collaborative New Faculty Workshop. Journal of Chemical Education, 92(9),

1466–1476. doi:10.1021/acs.jchemed.5b00324

Staver, J. R. (1998). Constructivism: Sound theory for explicating the practice of science and

science teaching. Journal of Research in Science Teaching, 35(5), 501–520.

Stoyanovich, C., Gandhi, A., & Flynn, A. B. (2015). Acid–base learning outcomes for students

in an introductory organic chemistry course. Journal of Chemical Education, 92(2), 220–

229. doi:10.1021/ed5003338

Suchting, W. A. (1992). Constructivism deconstructed. Science & Education, 1(3), 223–254.

doi:10.1007/BF00430275

256

Swan, K. P. (2003). Learning effectiveness online: What the research tells us. In J. Bourne & J.

C. Moore (Eds.), Elements of quality online education, practice and direction (pp. 13–45).

Needham, MA: Sloan Center for Online Education.

Swan, K. P., Garrison, D. R., & Richardson, J. C. (2009). A constructivist approach to online

learning: The Community of Inquiry framework. In C. R. Payne (Ed.), Information

technology and constructivism in higher education: Progressive learning frameworks (pp.

43–57). Hershey, PA: IGI Global. doi:10.4018/978-1-60566-654-9.ch004

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn

& Bacon/Pearson Education. doi:10.1037/022267

Tai, R. H., Sadler, P. M., & Mintzes, J. J. (2006). Factors influencing college science success.

Journal of College Science Teaching, 36(1), 56–60.

Talanquer, V. (2015). Threshold concepts in chemistry: The critical role of implicit schemas.

Journal of Chemical Education, 92(1), 3–9. doi:10.1021/ed500679k

Talanquer, V., & Pollard, J. (2010). Let’s teach how we think instead of what we know. Chem.

Educ. Res. Pract., 11(2), 74–83. doi:10.1039/C005349J

Tien, L. T., Roth, V., & Kampmeier, J. a. (2002). Implementation of a peer-led team learning

instructional approach in an undergraduate organic chemistry course. Journal of Research

in Science Teaching, 39(7), 606–632. doi:10.1002/tea.10038

Tobin, K. (1999). Constructivism in science education: Moving on. In D. C. Phillips (Ed.),

Constructivism in education: Opinions and second opinions on controversial issues (pp.

227–253). Chicago: National Society for the Study of Education.

257

Toto, J., & Booth, K. (2008). Effects and implications of mini-lectures on learning in first-

semester general chemistry. Chemistry Education Research and Practice, 9(3), 259.

doi:10.1039/b812415a

Trigwell, K., & Prosser, M. (1996). Congruence between intention and strategy in university

science teachers’ approaches to teaching. Higher Education, 32(1), 77–87.

Trigwell, K., & Prosser, M. (2004). Development and use of the approaches to teaching

inventory. Educational Psychology Review, 16(4), 409–424.

Trigwell, K., Prosser, M., & Ginns, P. (2005). Phenomenographic pedagogy and a revised

Approaches to teaching inventory. Higher Education Research & Development, 24(4), 349–

360. doi:10.1080/07294360500284730

Trigwell, K., Prosser, M., & Taylor, P. (1994). Qualitative differences in approaches to teaching

first year university science. Higher Education, 27(1), 75–84.

Varma-Nelson, P., & Banks, J. (2013). PLTL: Tracking the trajectory from face-to-face to online

environments. In T. Holme, M. M. Cooper, & P. Varma-Nelson (Eds.), Trajectories of

chemistry education innovation and reform (Vol. 1145, pp. 95–110). Washington, DC:

American Chemical Society. doi:10.1021/bk-2013-1145.ch007

Varma-Nelson, P., & Coppola, B. P. (2005). Team learning. In N. J. Pienta, M. M. Cooper, & T.

J. Greenbowe (Eds.), Chemists’ guide to effective teaching (pp. 155–169). Saddle River, NJ:

Pearson.

258

Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor analysis:

Some issues in selecting an appropriate procedure. Multivariate Behavioral Research,

25(1), 1–28. doi:10.1207/s15327906mbr2501_1

von Glasersfeld, E. (1974). Piaget and the radical constructivist epistemology. In C. D. Smock &

E. von Glasersfeld (Eds.), Epistemology and education (pp. 1–15). Athens, GA: Follow

Through Publications.

von Glasersfeld, E. (1984). An introduction to radical constructivism. In P. Watzlawick (Ed.),

The invented reality: How do we know what we believe we know? (pp. 17–40). New York:

Norton (English translation of work originally published in 1981).

von Glasersfeld, E. (1989). Cognition, construction of knowledge and teaching. Synthese:

History, Philosophy and Science Teaching, 80(1), 121–140.

von Glasersfeld, E. (1993). Questions and answers about radical constructivism. In K. Tobin

(Ed.), The practice of constructivism in science education (pp. 23–38). Hillsdale: NJ:

Lawrence Erlbaum Associates.

Vrasidas, C. (2000). Constructivism versus objectivism: Implications for interaction, course

design, and evaluation in distance education. International Journal of Educational

Telecommunications, 6, 339–362. doi:10.1080/0031383890330103

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. M.

Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Cambridge, MA: Harvard

University Press. doi:10.1007/978-3-540-92784-6

259

Williams, J., & MacKinnon, D. P. (2008). Resampling and distribution of the product methods

for testing indirect effects in complex models. Structural Equation Modeling: A

Multidisciplinary Journal, 15(1), 23–51. doi:10.1080/10705510701758166

Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An

analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers.

Review of Educational Research, 72(2), 131–175. doi:10.3102/00346543072002131

Wink, D. J. (2014). Constructivist frameworks in chemistry education and the problem of the

“Thumb in the Eye.” Journal of Chemical Education, 91(5), 617–622.

doi:10.1021/ed400739b

Wren, D., & Barbera, J. (2013). Gathering evidence for validity during the design, development,

and qualitative evaluation of thermochemistry concept inventory items. Journal of

Chemical Education, 90(12), 1590–1601. doi:10.1021/ed400384g

Wright, S. (1934). The method of path coefficients. The Annals of Mathematical Statistics, 5(3),

161–215.

Xu, X., & Lewis, J. E. (2011). Refinement of a chemistry attitude measure for college students.

Journal of Chemical Education, 88(5), 561–568. doi:10.1021/ed900071q