A Systematic Analysis And Synthesis of the Empirical MOOC Literature

Published in 2013-2015

George Veletsianos - Canada Research Chair in Innovative Learning & Technology, Royal Roads University,Peter Shepherdson - University of Zurich, Laura Pasquini - University of North Texas, & Rich McCue - UVic

The 9th Collaboration for Online Higher Education & Research (COHERE) Conference October 2015, Victoria & Halifax, Canada

Rich McCueUniversity of Victoria

George Veletsianos, Canada Research Chair in Innovative Learning & Technology, Royal Roads University

Peter Shepherdson University of Zurich

Laura Pasquini University of North Texas

• What is the state of MOOC research (very briefly)?

• Where are there gaps & what gaps can we fill?

• Research Methods.

• Did we find anything interesting?

Overview

Introduction

We wanted to address a number of gaps in the scholarly understanding of MOOCs and present a comprehensive picture of the MOOC literature by examining:

geographic distribution,

publication outlets,

citations,

data collection and analysis methods, and

research strands of empirical research focusing on MOOCs.

Review of the Literature

cMOOCs and xMOOCs?

● cMOOCs are described as being “based on principles of connectivism, openness, and participatory teaching” (Jacoby, 2014, p. 76), “through a dynamic network of connections afforded by online technology” (Ebben & Murphy, 2014, p. 333).

● xMOOCs resemble “traditional teacher-directed course[s], yet automated, massive, and online” (Kennedy, 2014, p. 8).● Early MOOCs tended to follow the cMOOC model, whereas more recently the number of xMOOCs delivered has been growing

rapidly.

Expected impacts of MOOCs on education

The size and openness of MOOCs have the potential to disrupt conventional thinking about the role, value, and cost of higher education.

The discourse around MOOCs could diminish of the authority and importance of the educational leader.

MOOCs may threaten the business models of universities.

Demographics of MOOC users

• Vast majority male, 20-40 yrs old.

• College degree or higher.

• More than 1/2 of learners from outside the United States.

• Majority of participants are from North America and Europe, with a small minority being from Asia, Southeast Asia, or Africa.

Challenges for MOOC’s

Completion rates in MOOCs are less than 10%. Why? A lack of incentive, many courses free.

Insufficient prior knowledge (e.g., lack of math or language skills).

Failure to understand the content and having no one to turn to for help,

Lack of time due to having other priorities and commitments.

90%

Research Questions:

RQ1: How is MOOC research geographically distributed?

RQ2: Is MOOC research usually published in journal or conference proceedings? In which journals and conference proceedings is MOOC research currently being published?

RQ3: Which empirical MOOC studies are cited the most?

RQ4: What data collection methods and data analysis methods are used in empirical studies of MOOCs?

RQ5: What are the research strands of empirical MOOC research?

Research Methods

Literature discovery searches were conducted using the [NEXT] keywords “MOOC” or “Massive Open Online Course.” To be included articles had to be:

(1) empirical, (2) published in a peer-reviewed journal, in conference proceedings, or in Educause Review, (3) published or was available online as in press between January 2013 and January 2015, and (4) written in English.

All identified papers were examined by [NEXT] two researchers to ensure consensus that they fit the inclusion criteria.Forwarding references: In [NEXT] Google scholar looked up the papers and the looked at papers that cited those papers to discover new papers. Found 60 new papers.Completeness Search: Examined references of 17 papers published in 2015 in our corpus to identify any papers that we missed. They were more likely to reference literature published in 2013-2014 than the papers published in 2013 or 2014. 5 new papers found. [NEXT] The final number of published papers that constituted the corpus of this study was 183.

Data Collection

Search Keywords: “MOOC” or “Massive Open Online Course”

Published Papers: n=183

Authors: n=460

Limitations

1. Limited duration - 3 years.

2. Only English language data.

3. Quality of research not evaluated.

4. Some content analyzed, but not all.

Results

RQ1: How is MOOC research geographically distributed?

Country Corpus

USA 50.2% UK Kingdom 10%Australia 7.7%China 5.4%Spain 4.8%Canada 4.5%Germany 2.2%Switzerland 1.3%Netherlands 1.1%Other (29 countries) 12.8%

Author Affiliations

RQ2a: Is MOOC research usually published in journals or conference proceedings?

RQ2b: In which journals and conference proceedings is MOOC research currently being published?Outlet name # of papers TypeInternational Review of Research in Open and Distributed Learning (IRRODL) 18 JournalProceedings of the first ACM conference on Learning @ scale (L@S '14) 11 ConferenceJournal of Online Learning and Teaching (JOLT) 7 JournalConference on Empirical Methods in Natural Language Processing (EMNLP) 5 ConferenceDistance Education 5 JournalNIPS Workshop on Data Driven Education 5 ConferenceProceedings of the 2014 ASCILITE Conference 4 ConferenceEDUCAUSE Review 4 Journal

RQ3: Which empirical MOOC studies are cited the most?

Paper # of citations- Studying learning in the worldwide classroom research into edX’s

first MOOC. 124

- Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses.

117

- MOOCs and the funnel of participation. 83- Tuned Models of Peer Assessment in MOOCs. 73- Automated Feedback Generation for Introductory Programming

Assignments. 56

- Retention and Intention in Massive Open Online Courses: In Depth. 41- “Turn on, Tune in, Drop out”: Anticipating student dropouts in

Massive Open Online Courses.35

RQ4: What data collection methods & analysis are used in empirical studies of MOOCs?

Analytic method Frequency (%) of total papers

Descriptive statistics 93.4

Correlational 52.5

Basic qualitative study 38.8

Experimental and quasi-experimental 25.7

Grounded Theory 7.6

Natural Language processing 7.6

Social Network Analysis 6.6

Ethnography 4.4

Phenomenology 2.2

Discourse analysis 1.0

RQ5: What are the research strands of empirical MOOC research?

● [NEXT] 83.6% of the papers focused on student-related areas. ● [NEXT] Nearly half (46.4%) of the papers identified in the literature search had some focus on

topics relating to the design, creation, and implementation of MOOCs themselves. This included research pertaining to methods of assessment, the description of unique learning environments, the creation of MOOCs on specific topics, and the evaluation of course success.

● [NEXT] Only 8.2% of papers focused on topics related to instructors and teaching. Papers within this theme largely focused on academics’ awareness, perspectives of, and experiences with MOOCs.

● [NEXT] 10.9% of the corpus contained content pertaining to the context and impact of MOOCs. This included research into perceptions of MOOCs, their usefulness as an educational medium, and their economic impact .

● [NEXT] Other.

84%

46%8%

11%

Discussion & Conclusions

Dependence on particular research methods may restrict our understanding of MOOCs

Analysis suggests that researchers have favored a quantitative, if not positivist approach to the conduct of MOOC research.

Survey data and secondary data collected via automated methods were also favored.

While some interpretive research was conducted in MOOCs in this time period, it was often basic and only a handful of studies were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, focus groups).

Thus, even though results suggest that research on MOOCs focuses on student-related topics, learners’ voices are largely absent in the literature. Based on these results, we suggest that an expansion of the methodological approaches used in MOOC research is urgently needed.

Paucity of research examining instructor-related topics

Analysis shows that there is limited research reported on instructor-related topics. This is a rich area for future research. Topics of interest in this area may include instructor motivations, experiences, and perceptions.

Understanding Learner Subpopulations

Completing

Auditing

Disengaging

Sampling

No-shows

Observers

Drop-ins

Latecomers

Drop-in latecomers

Non-engaged

Engaged

Understanding Learner Subpopulations

Little research examines the experiences of different populations and how and why learning experiences differ between groups. Future research into this area for instance could examine why some learners disengage, how the learning experience of drop-ins differs from the learning experience of those who are non-engaged, and what interventions may scaffold different types of learners.

The geography of MOOC research

Our geographical analysis of author affiliations showed that over half of the authors conducted their research in the USA. Over 80% of authors were affiliated with institutions in North America or Europe.

Percentage of Research Published in USA

In contrast, according to a Scimago search of all citable documents published in 2013 across all disciplines only 19% came from the USA, and it takes the top 20 countries (including at least seven countries from outside of North America and Europe) to account for 80% of academic output. As such, current MOOC research is simply not a reflection of geographical contributions to academic output in general.

MOOC Resarch

All Other Research

0% 15% 30% 45% 60%

Future Directions

We hope that this systematic analysis enables researchers to make better sense of the empirical literature on MOOCs and its direction and limitations. There are many possibilities for future research in this area. Future systematic reviews of the literature may focus on:

Synthesizing the knowledge on particular areas of interest, (completion and retention in MOOCs; learner motivations in MOOCs).

Examining whether research methods used to understand MOOCs follow standard methods of inquiry.Follow methods that take into advantage the digital nature of learning and teaching in this context.

Final Thoughts

Finally, we hope that our results highlight the need for a critical reflection on the part of researchers as to the methods they have used to understand MOOCs to date and encourage a greater diversity in research methods.

Thank you!

Research available at: http://veletsianos/publications

This presentation: http://slideshare.net/richmccue

Contact: veletsianos@gmail.com @veletsianos on twitter @richmccue on twitter

Photo credit: Got Credit

Special Thanks to the Creators of the Creative Commons Images used in this presentation:

• https://www.flickr.com/photos/47572798@N00/8397808475 • https://commons.wikimedia.org/wiki/File:Wet_winding_road.jpg • https://pixabay.com/en/world-map-map-world-black-earth-297446/ • http://creativecommons.org/weblog/entry/26283 • https://en.wikipedia.org/wiki/Printing_press#/media/File:PrintMus_038.jpg • https://commons.wikimedia.org/wiki/File:2011_Library_of_Congress_USA_5466788868_card_catalog.jpg • https://upload.wikimedia.org/wikipedia/commons/7/72/Rhll_wire_rope.jpg • https://upload.wikimedia.org/wikipedia/commons/0/0a/Urval_av_de_bocker_som_har_vunnit_Nordiska_radets_litteraturpris_und

er_de_50_ar_som_priset_funnits_(3).jpg

• https://www.flickr.com/photos/mathplourde/8620174342/sizes/l/in/photostream/ • http://orig00.deviantart.net/34c5/f/2011/263/d/6/destruction_by_tbh_1138-d4af5vp.png • https://www.flickr.com/photos/68751915@N05/6355351769 • https://upload.wikimedia.org/wikipedia/commons/1/10/ECLA_of_Bard_Seminar,_2012.jpg • https://www.flickr.com/photos/carbonnyc/143186839 • https://pixabay.com/en/beer-quit-quitting-can-vintage-398742/ • https://pixabay.com/en/blueprint-ruler-architecture-964629/ • https://www.flickr.com/photos/keithroper/8139626676 • http://www.picserver.org/images/highway/phrases/results.jpg • https://upload.wikimedia.org/wikipedia/commons/4/4b/Colossus.jpg • http://www.apa.org/images/apastyle_logo.gif • https://upload.wikimedia.org/wikipedia/commons/3/39/Student_in_Class_(3618969705).jpg • https://upload.wikimedia.org/wikipedia/commons/e/e6/Avrocar_schematic_(high_resolution).jpg • http://teachers.net/teachers.jpg • https://pixabay.com/p-90781/?no_redirect • https://pixabay.com/en/people-sitting-chairs-in-front-690810/ • https://upload.wikimedia.org/wikipedia/commons/1/17/1895-Dictionary-Phrenolog.png • https://www.flickr.com/photos/safari_vacation/10362491406 • https://www.flickr.com/photos/16210667@N02/15617432129 • http://www.veletsianos.com/wp-content/uploads/2013/05/edtechmooc.jpg • https://www.flickr.com/photos/intersectionconsulting/7537238368/ • https://pixabay.com/en/compass-antique-map-of-the-world-429772/ • http://asiimdesgraphic.deviantart.com/art/Mirror-Ball-Reflection-214407739