difficulties evaluating cmoocs (open education conference 2013)

Difficulties evaluating cMOOCs: Negotiating Autonomy and Participation #DiffCMOOC

Christina HendricksUniversity of British Columbia, VancouverOpen Education Conference, November 2013

Presentation licensed CC-BY

Connectivist MOOCsNetwork: Facilitating connections between people and information, ideas (not transmitting knowledge from central source) (Siemens 2012 http://is.gd/K5JfXK )

Distributed: Takes place in multiple spaces (blogs, wikis, tweets, discussion boards, webinars, etc.): “A MOOC is a web, not a website” (Downes 2013a http://www.downes.ca/presentation/327 )

#OOE13 Open Online Experience 2013-2014http://www.ooe13.org

From a video on MOOcs by Dave Cormier & Neal Gillis (licensed CC-BY)

http://is.gd/cQwOSP

Autonomy: Participants decide when & how to participate; create own learning goals, choose own paths through course (McAuley et al. 2010 http://is.gd/6j1X1k; Downes 2009 http://is.gd/AYc84B)

Connectivist MOOCs

Open: free access available to anyone with reliable internet connection; curriculum open to alterations by participants

(Downes, 2013b http://is.gd/Downes2013 )

Evaluating cMOOC effectiveness

Do they achieve goals?

Which goals?

• Of designers

• Of participants

• Connectivism: making connections w/people & information

• What sort of entities cMOOCs are & whether fulfill purposes (Downes)

Goals of cMOOC designers

Participant autonomy:

•What happens in course depends on what participants do: “learners are expected to actively contribute to the formation of the curriculum through conversations, discussions, and interactions” (Cormier & Siemens, 2010 http://is.gd/nqTED )

•course may be successful (or fail) in ways designers never envisioned

Hub & Spoke, flickr photo by Antony_Mayfield, licensed

CC-BY

Goals of participants

•may not have any goals; just want to see what happens

•course may have other benefits not captured in participants’ goals; may miss this if focus on their goals

•benefits may take a long time to realize

E.g., Lane, 2013 http://is.gd/W0360s

•participants may have goals that don’t fit course; a problem if not fulfilled?

Huma Bird tweet analysis, #whyopen

http://is.gd/4h5CFq

Goals of participants

What one might do:

•Ask participants at the end what they got out of the course, with or without reference to their original goals

•Return to them six months or more later to ask again--perhaps see longer-term effects

•Consider how to support learners in being self-directed, working to achieve own goals in a cMOOC (e.g., Kop, Fournier and Mak, 2011 http://is.gd/KopEtAl2011 )

Connectivism: connections among people & info

• “Knowledge is defined as a particular pattern of relationships and learning is defined as the creation of new connections and patterns as well as the ability to maneuver around existing networks/patterns." (Siemens 2008 http://www.connectivism.ca/?p=116 )

• "At its heart, connectivism is the thesis that knowledge is distributed across a network of connections, and therefore that learning consists of the ability to construct and traverse those networks." (Downes, 2007 http://is.gd/Downes2007)

Connectivism: connections among people & info

Participation rates:

•surveys of participants: Milligan, Littlejohn and Margaryan, 2013 http://is.gd/MilliganEtAl2013

•log data from P2PU platform: Ahn, Weng and Butler, 2013 http://is.gd/AhnEtAl2013

•mixed methods:

• Waite, Mackness, Roberts and Lovegrove, 2013 http://is.gd/WaiteEtAl2013

• Kop, 2011 http://is.gd/Kop2011

Connectivism: connections among people & info

Negotiating autonomy & participation

•Tension: need at least some active participation, but participants must have autonomy

•Lurkers valued? Just b/c may become active participants?

#ds106zone May 25-June 6, 2013http://is.gd/o27mvc

Purposes of cMOOCs themselves

Downes 2013b http://is.gd/Downes2013

•look at what sorts of entities cMOOCs are, what purposes they serve, whether designed well for those purposes (rather than how they’re used)

•To evaluate a cMOOC, consider: “what a successful MOOC ought to produce as output, without reference to existing (and frankly, very preliminary and very variable) usage.” (Ibid.)

• output: “emergent knowledge”

Emergent knowledge:

In a successful cMOOC, “the structure of the interactions produces new knowledge, that is, knowledge that was not present in any of the individual communications, but is produced as a result of the totality of the communications, in such a way that participants can through participation and immersion in this environment develop in their selves new (and typically unexpected) knowledge relevant to the domain.” (Downes, 2013b; emphasis added)

See something or say something: Jakarta

, Flickr photo shared by Eric Fischer, licensed CC-BY

Blue dots tweets; red dots Flickr, white dots both

Networks that tend to produce emergent knowledge

1.Autonomy

2.Diversity

3.Openness

4.Interactivity/Connectedness

(Downes, 2013b)

http://is.gd/Downes2013Anek Rang, Ek Sang, Flickr photo

shared by Sanjay, licensed CC-BY

How evaluate cMOOCs acc to these criteria?

• Don’t measure each aspect of a cMOOC against these as if a checklist; rather, consider cMOOCs a “language” and a course as an expression in it

• These criteria should be considered “an aid, used to assist a person who is already fluent in MOOC design (or at least in the domain or discipline being studied) [to] recognize the quality (or lack of quality) of a MOOC” (Downes, 2013b).

Questions & concerns about this approach

• How can we determine if emergent knowledge has been produced? Where would we look? Whom would we ask?• Seemingly exclusive focus

on design and purpose of cMOOCs--doesn’t consider the experiences of participants

E.g., participant experiences in a cMOOC: Mackness, Mak and Williams, 2010 http://is.gd/MacknessEtAl2010Crowd, Flickr photo by James

Cridland, licensed CC-BY (altered)

Footprints of emergence• Williams, Karousou and Mackness, 2011

http://is.gd/WilliamsEtAl2011

- Emergent and prescriptive learning--need balance

• Williams, Mackness and Gumtau, 2012 http://is.gd/WilliamsEtAl2012

- Draw “footprints” of courses to map degrees of prescriptive and emergent learning

Footprint for CCK08

published in Williams, Mackness & Gumtau 2012

http://is.gd/WilliamsEtAl2012(licensed CC-BY)Centre: prescriptive

learningLight area: apex of emergent learningPeriphery: “edge of chaos”

Map points based on 24 factors, in four clusters

See wiki for factors, how to draw footprints, and more: http://footprints-of-emergence.wikispaces.com/

Full circle• Footprints are not meant to provide evaluations

of courses by themselves

• Instead, provides way to evaluate if course fits purposes (Footprints of emergence wiki: http://footprints-of-emergence.wikispaces.com/ )

• After have footprint, ask: “Is this appropriate, or fit, for the purpose and context of the course and for you, and/or the particular learners?”

Back to the beginning...

What now?

Suggestions?

THANK YOU!

Twitter: @clhendricksbc

Blog: http://blogs.ubc.ca/chendricks

Slides, video and bibliography at: http://is.gd/HendricksOpenEd2013