a novel model of cognitive presence assessment using automated learning analytics methods

A Novel Model of Cognitive Presence

Assessment Using Automated

Learning Analytics Methods

Vitomir Kovanovic

School of Informatics

The University of Edinburgh

http://vitomir.kovanovic.info

[email protected]

#vkovanovic

2

Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info

Global Grand ChallengesShifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges

3


Four challenges for next decade

1. Affordable, sustainable energy.

2. Cures for HIV and neurodegenerative diseases like Alzheimer's.

3. Protection from future health epidemics.

4. Tools to provide a world-class education to all students.

4


1 GLOBAL NEED

FOR EDUCATION

Significant Challenges in Higher Education

2 INCREASE

STUDENT

ENGAGEMENT

3 DECREASING

UNIVERSITY

FUNDING

MORE AND MORE

UNIVERSITIES TURNING

TO TECHNOLOGY

MORE AND MORE

UNIVERSITIES TURNING

TO TECHNOLOGY

5


ONLINE &

BLENDED

Blended is the new norm.

Increasing interest in online learning.

MOOCs and new forms of delivery.

LEARNING

DATA

Large amounts of data collected.

Improve student experience.

Understand learning processes.

NEW MARKETS

AND MODELS

Workspace learning.

Lifelong learning.

Developing world.

Technology & Data TrendsThree important ways in which technology is shaping the future of education

6


Learning analytics is the measurement, collection, analysis and reporting of data about learners and their

contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.

CFP for the First Learning Analytics and Knowledge Conference

https://tekri.athabascau.ca/analytics/

What is Learning Analytics?

Learning AnalyticsMaking sense of the available learning data

Measurement Collection Analysis Reporting

7


What can Learning Analytics bringImprovement in teaching quality, student retention, learning outcomes, and understanding of learning processes

8


Traditional Online Learning

How can we use the educational data to understand student

learning?

Massive Open Online Courses

How to adapt existing models of online learning in MOOC

settings?

What are the challenges of MOOC pedagogies?

Learning Analytics for Online Learning

9


1 PedagogyThe use of Learning analytics must be aligned with the existing

pedagogical approaches used in particular learning context.

Driving PrinciplesFoundations for my learning analytics research

2 Data analysis

Use automated data mining techniques to process large amounts of

student-generated data to understand how student learn in online

setting

3 Tools

Develop automated learning analytics tools that can be used to assess

student learning and improve research in online and distance

education.

10


Community of Inquiry FrameworkDimensions of student online learning experience

Social-constructivist model of learning

- Students construct knowledge rather than receive it

- Discussion is an essential part of learning

- Inquiry basic learning activity

Widely used in “traditional” online

and distance education

- Strong teacher presence

- Up to ~ 30 students

Cognitive Presence

Teaching Presence

Social Presence

Student

Experience

Development of critical

and deep thinking skills

Social climate in the course

1. Group cohesion

2. Interactivity

3. Group affectivity

Instructor’s role in the course:

1. Design & organization,

2. Facilitation,

3. Direct instruction

(Garrison, Anderson, and Archer, 1999)

11


Cognitive PresenceOperationalization of critical and deep thinking skills

• Exploring and testing solutions

• Dilemma or problem identified

• Synthesis of relevant information

• Brainstorming

• Exploring ideas

2. Exploration

3. Integration

4. Resolution

1. Triggering

Event

Shared world of discourse

Private world of reflection

“an extent to which the participants

in any particular configuration of a

community of inquiry are able to

construct meaning through

sustained Communication”

(Garrison, Anderson, and Archer,

1999, p. 89)

Definition

12


1 Content analysisCoding scheme for each

presence

Assessment of three presencesHow do we measure levels of cognitive, social and teaching presence?

2 Self-reported survey34 items, 1-5 Likert scale

questions

• Experience with the coding scheme

• Inter-rater reliability issues

• Time-consuming

• Non-real time

• Primary use: research

• Fait accompli: results known after the

course is over

• Self-selection bias

• Invasive

(Arbaugh et al., 2008).

(Garrison, Anderson,

and Archer, 1999)

13


1 From the online

discussions

Assessment based on Learning AnalyticsAutomate as much as possible

2 Based on the use

of educational

technology

• Identifying different study

approaches

• How those approaches relate

to development of cognitive

presence?

• Clustering based on trace

data

Shared world learning

Private world learning

• Automating CoI coding

scheme for cognitive

presence

• Real-time analysis of student

cognitive presence

development

• Easier adoption and research

• Text mining of student online

discussion messages

1 Traditional online

courses

2 MOOCs

Athabasca University

fully online course

Delft University of

Technology MOOC

data

Types of data and analytics Contexts

14


Cognitive presence assessment frameworkDriven by Evidence-Centered Design (ECD) framework

ASSESSMENT IMPLEMENTATIONAutomated cognitive presence classification system

Technology use profiling system

ASSESSMENT FRAMEWORKEvidence-centered design (ECD) assessment

framework: Student model, Evidence model and Task

model

EDUCATIONAL TECHNOLOGYTechnology use trace data and online discussion

data from Learning Management Systems (LMS)

and MOOC platforms:

THEORY & PEDAGOGYSocial constructivist learning & Community of Inquiry

model

Kovanovic et al. (in-press)

15


Cognitive presence classification system

Five-class text classification problem:

1 – Triggering event,

2 – Exploration,

3 – Integration,

4 – Resolution,

0 – Other (non cognitive).

Data from six offers of a fully online course

1,747 messages coded for cognitive presence

Developed random forest classifier

Extracted features:

Linguistic Inquiry Word Count features (LIWC):

93 different counts indicative of different psychological processes (e.g.,

affective, cognitive, social, perceptual)

Coh-Metrix features:

108 metrics of text cohesion

LSA coherence:

Average LSA similarity of message’s paragraphs to each other.

LSA space is built from Wikipedia articles related to concepts extracted

from the topic start message (using TAGME).

Named entity count:

Number of concepts related to DBPedia computer science category (using

DBPedia spotlight)

Context features:

Number of replies

Message depth

Cosine similarity to previous/next message

Thread start/end Boolean indicators

Kovanovic et al. (2016)

16


Performance evaluation

• We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ.

• Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014)

and Waters et al. (2015) studies.

• The feature space ~ 100x smaller

• The feature space is also more generalizable

• We provided more detailed operationalization of the cognitive presence coding scheme.

Current work:

– Analysis of discussion messages from “Functional Programming” MOOC by TU Delft

– Investigation of the suitability of the same classification approach in various contexts

(Essay analysis, Twitter messages)


17


Feature importance

Phase

# Variable Description MDG* Other TE Exp. Int. Res.

1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68

2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18

3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58

4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00

5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78

6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24

7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38

8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39

9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16

10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99

11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05

12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22

13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84

14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29

15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67

16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73

17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97

18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19

19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99

20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47

* MDG - Mean decrease Gini impurity index, FPS - first person singular

Phase

# Variable Description MDG* Other TE Exp. Int. Res.

1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68

2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18

3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58

4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00

5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78

6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24

7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38

8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39

9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16

10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99

11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05

12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22

13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84

14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29

15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67

16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73

17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97

18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19

19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99

20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47


18


The higher the cognitive presence…

• The longer the message, the more paragraphs and

sentences.

• More concepts mentioned (more named entities).

• The lower the lexical diversity (both at content level

and in general).

• The later its position in the thread. Except non-

cognitive messages, they tend to occur closer to the

end as well.

• The fewer the question marks (except non-cognitive,

they have fewest question marks)

• The higher the average length of sentence and their

similarity to each other.

• The more money-related terms.

Operationalization of cognitive presence

• Exploring and testing solutions

• Dilemma or problem identified

• Synthesis of relevant information

• Brainstorming

• Exploring ideas

2. Exploration

3. Integration

4. Resolution

1. Triggering

Event



• Lowest message readability

• Syllabi count inconsistent

• Most replies

• Low similarity with the next message

• More replies than exploration and resolution

• Least replies

• Question marks more frequent than integration

2. Exploration

3. Integration

4. Resolution

1. Triggering

Event




19


Technology use profiling system

• Used 200,000 LMS trace data records of student learning activities

• Hierarchical clustering (Ward method, Euclidean distance)

• MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis

• We identified six different technology use profiles.

• Large number of students poorly regulated technology use.

• Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We

observed large effect size (partial eta2 = .54).

• Quality of interactions is more important than the quantity for the development of cognitive presence.


20


In progress: Communities of Inquiry for

Massive Open Online Courses

Used the data from five TU Delft MOOCs

2,446 survey responses

5-item Likert scale responses

Originally, CoI studies found 3-factor structure to best describe CoI survey instrument

Goal: can we replicate those findings in MOOCs?

21


Factor analysis

Original model holds in the MOOC context

CP6: Online discussions were valuable in helping me

appreciate different perspectives

Affectivity

Resolution

Course organization

& design

Better fit: 6 factor model:

22


How are they interactingStructural equation model between three presences

• Direct effect of Teaching Presence on Cognitive presence, and

• Mediating effect of social presence on teaching presence and cognitive presence relationship

Cognitive presence

Social presence

Teaching presence

Cognitive presence

Social presence

Teaching presence

Original SEM model MOOC model

0.52

0.51 0.40 0.60 0.22

0.35

23


Technology-use in MOOCs

• Adopted same methodology as in “traditional” online setting

• Significant MANOVA, (p=0.038).

• Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )

24


Summary

• Important to realize the tremendous potential educational data has for improving teaching and learning.

• How this potential of the data fits the existing models of online and blended learning?

• How can we design analytics that are flexible enough and provide meaningful insights about student learning?

• Can we better support research in online education by automating existing approaches used by researchers?

• Can we use the existing pedagogical models in MOOC setting?

• How can we make student learning in MOOCs les solitary?

25


Summary: cognitive presence assessment

framework

• We developed a framework for assessing the levels of cognitive presence

• We focused on two types of insights: from trace data & discussion data and

• We focused on two contexts: traditional online courses and MOOCs

• We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines

• Automated existing tool for measuring cognitive presence

• Enables easier adoption of Community of inquiry model

• Better operationalization of the phases of cognitive presence

• We developed a student profiling method that can be used to identify different profiles of student technology use

• Better understand online learning, especially related to student self-regulation of learning

• Insights into their private solitary learning not displayed in online discussion transcripts

26


Summary: Cognitive presence in MOOCs

• Explored the use of CoI model in MOOC context and use existing pedagogies at scale

• We conducted a factor analysis of Community of Inquiry model in the MOOC context:

• Original CoI model holds but with certain factors being more emphasized than others

• The role of social group is much weaker in MOOCs

• We identified profiles of students based on their technology use

• Identified profiles differ in the levels of their cognitive and social presence

• The differences are much smaller, primarily due to the use of survey-based instrument

27


Current and future work

• Enable provision of formative feedback based on their use of educational systems

• Possible to support instructional interventions by embedding the developed analytics into the learning platform

• We are currently working on a MOOC experimentation platform that will include cognitive presence analytics

• At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context

• Also, exploring the use of the same classification methodology for similar problems

Student reflection in discussions (OU)

Essay grading (USF)

Twitter and YouTube comments (QUT)

28


The End

THANK YOU FOR YOUR TIME

Q/A

29


REFERENCES

Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry

framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136.

Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer

Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105.

Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using

Automated Learning Analytics Methods (Analytics4Learning). SRI Education. Available at:

http://vitomir.kovanovic.info/publications/

Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content

analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning

Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950

Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of

learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–

89. https://doi.org/10.1016/j.iheduc.2015.06.002

Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In

Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics

and Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/

http://vitomir.kovanovic.info/publications/

https://doi.org/10.1145/2883851.2883950

https://doi.org/10.1016/j.iheduc.2015.06.002

http://ceur-ws.org/Vol-1137/

a novel model of cognitive presence assessment using automated learning analytics methods

Data & Analytics