Open education: "Runnin' with the Devil"

Vivien Rolfe and David Kernohan@vivienrolfe @dkernohan

I am calling for a (radical?) pedagogy caucus, a core, self-identified group committed to placing pedagogy at the center of the OpenEd movement.(Robin De Rosa)

#opened15

At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old solutions (Kuhn 1970).

Are we being critical enough? Where arewe being critical?

What are we publishing and where, to explore facets of openness?

Are we critical in our thinking and writing?

How do we measure this?

su("Open education" OR "Open source technology" OR "Open university") AND su(Learning OR "Outcomes of education") AND su(Students OR "Adult students" OR "Independent study")

Systematic literature retrieval(Pubmed MeSH + ERIC Thesaurus)

186 (Pubmed) + 627 (ERIC) retrieved

REVIEW OF TITLES AND ABSTRACTS TO EXCLUDE

● Reports and other article types● Interventions that weren’t “open”● Those not an evaluation of learning

(but satisfaction)

REVIEW OF FULL PAPERS● Lost interventions that weren’t “open”● Not an evaluation of learning (but

satisfaction)

53 articles

5 evaluations of the impact of open education innovation on learning

Albeit a ‘pilot’ search - indications success there is not an established culture of evaluation.

A good proportion of abstracts didn’t contain the detail to judge the quality and content of the paper.

5 evaluations of the impact of open education innovation on learning

1) Publication bias.2) Evidence of critical reflection

and writing.3) Citation bias - many

introductions are just ‘shopping lists’ of past research.

4) Ball (2015) noted in his science study, 2.4% citations in papers were negative.

All 5 presented positive findings (main outcome, second outcome e.g. learning gain, test results)

All stated limitations of their methodology. Conclusions were positive but placed in context.

Of 62 citations within the introductions, 6 were within a negative critical context.

From 3Rs to the 5Rs!

At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old solutions (Kuhn 1970).

Are we being critical enough? Where arewe being critical?

What are we publishing and where, to explore facets of openness?

Are we critical in our thinking and writing?

How do we measure this?

“I found the simple life ain’t so simple”- Other ways of examining the literature.

Hypotheses:

1. Most of the *really good stuff* is in the blogs - blog citations are a significant part of the OER literature

2. Blog citations can be measured via the parts of the scholarly graph we have access to.

3. Citations of blog posts will be in papers concerning the kinds of topics that bloggers write about.

Hypotheses:

1. Most of the *really good stuff* is in the blogs - blog citations are a significant part of the OER literature

2. Blog citations can be measured via the parts of the scholarly graph we have access to.

3. Citations of blog posts will be in papers concerning the kinds of topics that bloggers write about.

Methods:

1. Collect citations and basic stats from Google Scholar using Harzing’s “Publish or Perish” tool (H1, H2)

2. Measure word frequency in citing paper titles using Voyant tools.

3. Measure word frequency in each blog corpus (via rss) using Timdream Wordcloud.

4. Compare (H3)

Blogs examined

David Wiley

http://www.opencontent.org/blog

George Siemens

http://elearnspace.org/blog

Audrey Watters

http://www.hackeducation.com

Martin Weller

http://nogoodreason.typepad.co.uk

Also

“Edupunk” (term)

“connectivism.ca” (former blog)

Notable findings.

1. The papers that cite George Siemens’ blog have a h-index of 38! (this is better than most education researchers)

2. The papers that cite Connectivism.ca, a former blog of his, has a still impressive h-index of 29.

3. Large amounts of the “edupunk” literature are in Spanish.

4. Siemens and Wiley are most likely to have their blogs cited together.

Semantometrics and citation metrics

Image from http://Semantometrics.org - no license indicated but assumed CC-BY due to funder requirements.

What is a citation anyway?

Citations serve a range of purposes, both technical and social. The reasons for citation are contested and

no comprehensive social theory of citation exists. Broadly there are two sets of theories around the

motivations for citation, normative, and social-constructivist.

Normative theories hold that citations are a means of expressing norms of the research community,

primarily the norm of assigning credit where it is due, and the norm of showing evidence

transparently. Normative theories focus on shared practice and purposes that are generally held to

be transparent and well understood.

Social constructivist theories focus on reasons for citation that are social or cultural, often differing

across communities, that are not (necessarily) associated with the normative aspects of

acknowledging intellectual debt.

https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/- Cameron Neylon

Semantic distance: data

Semantic distance: grouping

Semantic prediction

How good are the top five words from each blog at predicting the titles of citing papers?

David Wiley 0.7038

George Siemens 0.77373

Connectivism.ca 0.58621

Audrey Watters 0.12

Martin Weller 0.57639

Total instances of top five words in titlesNumber of titles

Lower number = less predictiveTherefore a greatersemantic distance betweenthe blog and citing papers.

Summary

The community called for critical debate on the direction of Open Education, but where do we do this? Robust research is lacking (as always identified by systematic approaches), and we are subject to publication and citation bias.

Social media brings critical discussions into the open as a new academic space. Some blogs are cited in formal literature, either within the literature they would be seen as a part of, or in wider literature.

Systematic review approaches, semantometrics and citation metrics are all different ways of examining a literature, each with strengths and weaknesses. Future work should draw on a multiplicity of methodologies.

In answer to Adam and Robin, “open textbooks ugh” - “is that what we meant”?

The comment reflects a common assertion that specificity must necessarily, in and of itself, betray the spirit of openness and informality. (Katz 1972)

At times, revolutions are misleading. Consider the spreading educational innovation which has come to be known as the British Infant School.The later is a concept which stands for a collection of educational approaches, all using the open classroom as a learning environment. Note that it is a collection of approaches, not just one theoretical or practical idea. (Nelson 1972)

Further Reading

Ball P (2015). Science papers rarely cited in negative ways. Nature News.

Croon A (2015). Is that what we meant? http://adamcroom.com/2015/11/is-that-what-we-meant/

Katz L G (1972). Research on Open Education: Problems and Issues.

Kuhn T (1970). Scientific Revolutions (2nd. ed., Enlarged), Chicago: The University of Chicago Press.

Nelsen J (1972). Open Minded, Thought-Filled Education.

De Rosa R (2015). Open textbooks. Ugh.http://robinderosa.net/uncategorized/open-textbooks-ugh/

Knoth P and Herrmannova D (2016) Semantometrics http://semantometrics.org/

Neylon C (2016) What constitutes research data? What is citation? https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/