co i evaluation
DESCRIPTION
TRANSCRIPT
Using the Community of Inquiry Framework Survey for Multi-Level Institutional Evaluation
Phil Ice, Ed.D.26th Annual Conference on Distance Teaching & Learning
CoI Framework
A process model of learning in online and blended educational environments
Grounded in a collaborative constructivist view of higher education
Assumes effective online learning requires the development of a community of learners that supports meaningful inquiry and deep learning
Three Presences
Social Presence
The ability of participants in a community of inquiry to project themselves socially and emotionally -- as ‘real’ people
The degree to which participants in computer mediated communication feel socially and emotionally connected
Social Presence - Elements
Affective expression (expressing emotion, self-projection)
Open communication (learning climate, risk free expression)
Group cohesion (group identity, collaboration)
Cognitive Presence
The extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry
Cognitive Presence - Elements
Triggering event (sense of puzzlement)
Exploration (sharing information & ideas)
Integration (connecting ideas) Resolution (synthesizing & applying
new ideas)
Cognitive Presence - Elements
triggering event (sense of puzzlement)
exploration (sharing information & ideas)
integration (connecting ideas) resolution (synthesizing & applying
new ideas)
Teaching Presence
The design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes
Teaching Presence - Elements
Design and organization (setting curriculum & activities)
Facilitation (shaping constructive discourse)
Direct instruction (focusing & resolving issues)
CoI Survey
9 social presence items (3 affective expression, 3 open communication, 3 group cohesion)
12 cognitive presence items (3 triggering, 3 exploration, 3 integration, 3 resolution)
13 teaching presence items (4 design & facilitation, 6 facilitation of discourse, 3 direct instruction)
CoI Survey - Validation
Tested in graduate courses at four institutions in the US and Canada
Principal component factor analysis Three factor model predicted by CoI
framework confirmed Arbaugh, Cleveland-Innes, Diaz,
Garrison, Ice, Richardson, Shea & Swan – 2008
Subsequent validation and cumulative n over 500,000
Community of Inquiry Survey Instrument (draft v15)Developed by Ben Arbaugh, Marti Cleveland-Innes, Sebastian Diaz, Randy
Garrison, Phil Ice, Jennifer Richardson, Peter Shea & Karen Swan
Teaching PresenceDesign & Organization1. The instructor clearly communicated important course topi2. The instructor clearly communicated important course goals.3. The instructor provided clear instructions on how to participate in course learning activities.4. The instructor clearly communicated important due dates/time frames for learning activities.
Facilitation of Discourse5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that
helped me to learn.6. The instructor was helpful in guiding the class towards understanding course topics in a way that
helped me clarify my thinking. 7. The instructor helped to keep course participants engaged and participating in productive dialogue.8. The instructor helped keep the course participants on task in a way that helped me to learn.9. The instructor encouraged course participants to explore new concepts in this course.10. Instructor actions reinforced the development of a sense of community among course participants.
Direct Instruction11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn.12. The instructor provided feedback that helped me understand my strengths and weaknesses. 13. The instructor provided feedback in a timely fashion.
Social PresenceAffective Expression
14. Getting to know other course participants gave me a sense of belonging in the course.15. I was able to form distinct impressions of some course participants.16. Online or web-based communication is an excellent medium for social interaction.
Open communication17. I felt comfortable conversing through the online medium. 18. I felt comfortable participating in the course discussions. 19. I felt comfortable interacting with other course participants.
Group cohesion20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust.21. I felt that my point of view was acknowledged by other course participants. 22. Online discussions help me to develop a sense of collaboration.
Cognitive PresenceTriggering Even23. Problems posed increased my interest in course issues.
24. Course activities piqued my curiosity.
25. I felt motivated to explore content related questions.
Exploration26. I utilized a variety of information sources to explore problems posed in this course.
27. Brainstorming and finding relevant information helped me resolve content related questions.
28. Discussing course content with my classmates was valuable in helping me appreciate different perspectives.
Integration29. Combining new information helped me answer questions raised in course activities.
30. Learning activities helped me construct explanations/solutions.
31. Reflection on course content and discussions helped me understand fundamental concepts in this class.
Resolution32. I can describe ways to test and apply the knowledge created in this course.
33. I have developed solutions to course problems that can be applied in practice.
34. I can apply the knowledge created in this course to my work or other non-class related activities.
TEACHING PRESENCE
1 2 3
1. The instructor clearly communicated important course topics. 0.826 0.088 0.067
2. The instructor clearly communicated important course goals. 0.877 -0.021 0.046
3. The instructor provided clear instructions on how to participate in course learning activities. 0.592 0.246 -0.035
4. The instructor clearly communicated important due dates/time frames for learning activities. 0.611 0.078 0.040
5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. 0.579 0.162 -0.138
6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking. 0.575 0.091 -0.281
7. The instructor helped to keep course participants engaged and participating in productive dialogue. 0.633 0.149 -0.160
8. The instructor helped keep the course participants on task in a way that helped me to learn. 0.579 0.042 -0.285
9. The instructor encouraged course participants to explore new concepts in this course. 0.523 0.099 -0.233
10. Instructor actions reinforced the development of a sense of community among course participants. 0.569 0.174 -0.176
11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn. 0.425 0.146 -0.374
12. The instructor provided feedback that helped me understand my strengths and weaknesses relative to the course’s goals and objectives. 0.649 -0.123 -0.201
13. The instructor provided feedback in a timely fashion. 0.513 -0.025 -0.103
SOCIAL PRESENCE
1 2 3
14. Getting to know other course participants gave me a sense of belonging in the course. 0.050 0.619 -0.233
15. I was able to form distinct impressions of some course participants. 0.172 0.473 0.013
16. Online or web-based communication is an excellent medium for social interaction. -0.181 0.674 -0.226
17. I felt comfortable conversing through the online medium. -0.039 0.814 0.015
18. I felt comfortable participating in the course discussions. 0.109 0.788 0.005
19. I felt comfortable interacting with other course participants. 0.286 0.701 0.038
20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust. 0.103 0.620 -0.034
21. I felt that my point of view was acknowledged by other course participants. 0.319 0.556 0.025
22. Online discussions help me to develop a sense of collaboration. 0.047 0.561 -0.340
COGNITIVE PRESENCE
1 2 3
23. Problems posed increased my interest in course issues. -0.099 0.172 -0.785
24. Course activities piqued my curiosity. 0.064 0.070 -0.712
25. I felt motivated to explore content related questions. 0.082 -0.031 -0.770
26. I utilized a variety of information sources to explore problems posed in this course. 0.078 -0.158 -0.759
27. Brainstorming and finding relevant information helped me resolve content related questions. -0.106 0.130 -0.794
28. Online discussions were valuable in helping me appreciate different perspectives. -0.096 0.286 -0.699
29. Combining new information helped me answer questions raised in course activities. 0.101 0.043 -0.716
30. Learning activities helped me construct explanations/solutions. 0.128 0.030 -0.732
31. Reflection on course content and discussions helped me understand fundamental concepts in this class. 0.008 0.237 -0.640
32. I can describe ways to test and apply the knowledge created in this course. 0.239 -0.097 -0.619
33. I have developed solutions to course problems that can be applied in practice. 0.147 0.026 -0.653
34. I can apply the knowledge created in this course to my work or other non-class related activities. 0.171 -0.041 -0.687
The CoI Survey at APUS
Used as the end of course survey since January 2009
45% return rate To date n = approximately 130,000 Data analyzed using comparative
descriptive statistics, regression analysis and factor analysis at the University, School, Program, Course and Instructor levels
Recognized as Sloan-C’s effective practice of the year 2009
Recommendations…Part II5. As part of a process of Continuous Quality
Improvement (CQI), work to strengthen data to improve decision-making.
6. When expanding existing systems for assessment and evaluation of student achievement, identify new measures that may be collected to more fully explain student retention. For example, the Community of Inquiry Framework survey has been used by other online institutions to help inform student retention, and items from this survey have been found to be significant predictors (both statistically and pragmatically) of student retention.
TeachingPresence
q12.38
q11.28
q10.24
q9.27
q8.18
q7.23
q6.19
q5.24
q4.53
q3.43
q2.40
q1.37
.89
1
.85
1
.87
1
.85
1
.90
1
.881
.901
.87
1.69
1
1
.78
1
.80
1
q13.46
.74
1
CognitivePresence
q34 .44
q33 .39
q32 .40
q31 .27
q30 .27
q29 .29
q28 .49
q27 .44
q26 .47
q25 .34
q24 ..36
q23 .48
.75
1
.78
1
.781
.851
.86
1.84
1
.721
.75
1
.72
1
.81
1
.80
1
.72
1
SocialPresence
q22
.40
q21
.38
q20
.41
q19
.18
q18
.29
q17
.27
q16
.62
q15
.67
q14
.65
.77
1
.78
1
.77
1
.91
1
.86
1
.85
1
.62
1
.58
1
.59
1
.52(.49)**
Gender Age AcademicLevel
.06(.0
4)*
.02(.08)**
.00(.01)
.06*
.00
.22**
.75
.52(
.52)
**
.49(
.47)
**
Implementation
Quarterly audit form revised to mirror the CoI
Annual review form revised to mirror the CoI with Focus on: Course Organization Content Knowledge/Learning for
Cognitive Presence Direct Instruction and Feedback Social Interaction and Discussion Work Agreement and Achievements Professional Development
Technology Integration
Difference between means in split testing
Difference in variance accounted for Discrete differences in CHAID
analysis SEM pattern alternation Qualitative instrumentation
derivative of the COI utilized where appropriate
Focus on efficacy and ROI
Testing
Feedback – audio and video implementations have proven very effective
Split testing used to assess differences
Analysis of differences in means Strengthen factor loadings on Social
Presence Impact on eight CoI items
Examples
Online or web-based communication is an excellent medium for social interaction. Text group / mean = 3.90 Audio group / mean = 4.27
The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn. Text group / mean = 3.90 Audio group / mean = 4.27
Instructional Design 1
CoI analysis informs optimal design strategy
Bifurcation of Teaching Presence is an indicator of need for review of “instructor voice”
Social Presence bifurcation is an indicator of the need for more collaboration
Strength of factor loadings can indicate areas where content review is needed
Instructional Design 2
Socio-epistemological orientation – objectivist vs. constructivist
Two factor loading pattern indicative of an objectivist orientation
Constructivist paradigm important for those most impacted – 38-47 years of age
Thank You!
Phil Ice, Ed.D.Director of Course Design, Research andDevelopmentAmerican Public University [email protected]
Vice President Sage Road [email protected]