International Review of Research in Open and Distributed Learning

Volume 19, Number 1

February - 2018

What Research Says About MOOCs - An Explorative Content Analysis

Olaf Zawacki-Richter¹, Aras Bozkurt², Uthman Alturki³, and Ahmed Aldraiweesh^3
1Carl von Ossietzky University of Oldenburg, Germany, ²Anadolu University, Turkey, and University of South Africa, South Africa, ³King Saud University, Saudi Arabia

Abstract

Since the first offering of a Massive Open Online Course (MOOC) in 2008, the body of literature on this new phenomenon of open learning has grown tremendously. In this regard, this article intends to identify and map patterns in research on MOOCs by reviewing 362 empirical articles published in peer-reviewed journals from 2008 to 2015. For the purposes of this study, a text-mining tool was used to analyse the content of the published research journal articles and to reveal the major themes and concepts covered in the publications. The findings reveal that the MOOC literature generally focuses on four lines of research: (a) the potential and challenges of MOOCs for universities; (b) MOOC platforms; (c) learners and content in MOOCs; and (d) the quality of MOOCs and instructional design issues. Prospective researchers may use these results to gain an overview of this emerging field, as well as to explore potential research directions.

Keywords: distance education, open and distance learning, massive open online courses, MOOCs, content analysis

Introduction

The 21^st century witnessed an educational paradigm shift, stemming from the widespread use of Information and Communication Technologies (ICT). With the proliferation of ICT, online, open, and flexible learning moved from the periphery to mainstream education. ICT improved the quality and capacity of the online delivery of educational content. Online networks are used as learning spaces that are distributed, flexible, accessible, and, most importantly, potentially open. Openness in education has evolved over time and has emerged in different forms (Weller, 2014): It is suggested that "there are three key strands that lead to the current set of open education core concepts: open access education, open source software and web 2.0 culture" (p. 34). Providing access to higher learning opportunities is the raison d'être of Open Universities (Tait, 2008). Massive Open Online Courses (MOOCs) are a recent development of this open learning movement, which have drawn much attention from both the academic and the public sphere. The first course in this format was offered in 2008 at the University of Manitoba and was entitled Connectivism and Connective Knowledge (Liyanagunawardena, Adams, & Williams, 2013).

MOOCs are not an independent phenomenon, isolated from other developments in the field of open and distance learning or educational technology. On the contrary, MOOCs are strongly tied to other developments in the field, having the potential to support lifelong learning, eliminate barriers in the learning process, provide equality of opportunity in education, and, most importantly, ensure the liberalization of knowledge.

MOOCs are a new and emerging, rapidly evolving field of practice and research. The body of literature about MOOCs has grown extremely rich. This article builds upon a previous study that investigated general publication and authorship patterns, research areas, and applied methods in MOOC research (see Bozkurt, Akgün-Özbek, & Zawacki-Richter, 2017). In light of this dynamic development, the aim of this study is to explore and to provide an overview of the key themes covered in MOOC research publications from 2008 to 2015 with the text-mining tool Leximancer™. The results provide a structure of themes and topics in MOOC research, which can be used to develop new research questions to be investigated in more in-depth content analysis, for example by means of systematic review (see Gough, Oliver, & Thomas, 2012).

Literature Review

A MOOC is defined as open, participatory, distributed, and as supporting lifelong network learning (Cormier, 2010). The first MOOC, belonging to the first generation, was given by George Siemens and Stephen Downes in 2008 (Downes, 2012). The success of first-generation connectivist MOOCs inspired other researchers; Sebastian Thrun and Peter Norvig gave the first extended MOOC in 2011, which belongs to the second generation (Martin, 2012). To differentiate between these two types of MOOCs, they were then called cMOOCs and xMOOCs respectively (Downes, 2012). The success of the first and second-generation MOOCs raised a lot interest in the public sphere, in academia, and in higher education institutions. This led to the innovative experimental idea of hybrid MOOCs, first delivered by a group of academics from the University of Edinburgh in 2013 (Roberts, Waite, Lovegrove, & Mackness, 2013; Waite, Mackness, Roberts, & Lovegrove, 2013; Ross, Sinclair, Knox, & Macleod, 2014; Bozkurt, Kilgore, & Crosslin, 2018).

Whilst the letters in the MOOC acronym represent one basic form (Diaz, Brown, & Pelletier, 2013), there are two different MOOC types according to the pedagogical approach they employ (Rodriguez, 2012). The first-generation cMOOCs embraced a decentralized, learner-centred approach; the second-generation xMOOCs were characterized by teacher-centred teaching and learning; the third-generation hybrid MOOCs took a more pragmatic approach by combining the two previous approaches; to diversify learning opportunities and to reach a broader audience.

There have been some efforts in academia to understand and analyse the MOOC phenomenon; several papers have examined MOOC research in academic journals (Ebben & Murphy, 2014; Gasevic, Kovanovic, Joksimovic, & Siemens, 2014; Kennedy, 2014; Liyanagunawardena et al., 2013; Raffaghelli, Cucchiara, & Persico, 2015; Sa'don, Alias, & Ohshima, 2014; Sangrà, González-Sanmamed, & Anderson, 2015; Veletsianos & Shepherdson, 2015, 2016; Bozkurt, Akgün-Özbek, & Zawacki-Richter, 2017). These papers examined aspects of MOOC research such as methodology, pedagogy, and theory. Furthermore, Bozkurt, Özdamar Keskin, and de Waard (2016) investigated theses and dissertations on MOOCs, focusing on methodological and theoretical issues, and representing MOOCs with a Gartner hype cycle.

Other papers have investigated MOOCs in the fields of broadcasting and social media (Bulfin, Pangrazio, & Selwyn, 2014; Deimann, 2015; Kovanovic, Joksimovic, Gasevic, Siemens, & Hatala, 2015; Shen & Kuo, 2015), taking a closer look at the phenomenon by focusing on discourses and sentiments on MOOCs, as well as identifying influencers in broadcasting and social media. Finally, some papers narrowed their scope in analysing MOOC research. For instance, Ossiannilsson, Altinay, and Altinay (2016) reviewed MOOC research with the aim of identifying factors that affect learner experience and quality issues in MOOCs. Similarly, Saadatdoost, Sim, Jafarkarimi, and Mei Hee (2015) examined MOOC studies from the perspective of education and information systems, and Calonge and Shah (2016) analysed MOOC literature in terms of graduate skills gaps and employability.

Similar to this research, but with a different scope, Chen (2014) identified 306 blog posts related to MOOCs published from January 2010 to June 2013 and analysed them using a text-mining technique. He noted that MOOCs provide many opportunities for learners, faculty members, universities, and MOOC providers. On the other hand, he also identified some challenges that MOOCs need to overcome, such as questionable course quality, high dropout rates, unavailable course credits, ineffective assessments, complex copyright issues, and the lack of necessary hardware required to join MOOCs.

Whilst previous bibliographic studies, literature reviews, and content analyses looked at theoretical, methodological, and pedagogical approaches, or specific aspects of MOOC research (e.g., quality or learner's perceptions), our study aims to provide an overview of the overall structure of themes and topics of research into MOOCs by means of a computer-assisted content analysis using a text-mining tool.

Method and Sample

This paper is a review study in nature. It uses document analysis to collect and identify relevant articles and content analysis using a text-mining tool to identify themes and concepts covered in the publications (Figure 1).

Figure 1. Phases of the research process.

The articles were selected by searching for the following keywords: MOOC, MOOCs, Massive Open Online Course, and Massive Open Online Courses. In the initial analysis, it was found that four academic databases provide the most comprehensive search results: EBSCO, ERIC, Google Scholar, and Scopus. A total of 888 papers were collected in the screening process and were analysed using the following inclusion criteria: published in a peer-reviewed journal between 2008 and 2015; written in English; online full-text accessibility; and searched keywords appearing in the title. Accordingly, 526 papers that were irrelevant or did not meet the inclusion criteria were excluded from the sample. Thus, 362 articles that met the criteria formed the corpus for further analysis. Figure 2 provides an overview of the growth and frequency of relevant research articles from 2008 to 2015.

Figure 2. Frequency of the sampled articles by year (N = 362).

Computer-based content analysis enables us to examine the conceptual structure of text-based information, so it can be used to identify the most important and most commonly occurring themes within large bodies of text (Krippendorf, 2013). For the purposes of this study, the software tool Leximancer™ was used to produce a concept map from the titles and abstracts of the 362 journal articles, as the titles and abstracts of peer-reviewed articles are usually lexically dense and focus on the core concepts, themes, and results of the research.

Leximancer™ has previously been used to analyse the content of academic journals such as Distance Education (Zawacki-Richter, & Naidu, 2016), the Journal of Cross-Cultural Psychology (Cretchley, Rooney, & Gallois, 2010), and the Journal of Communication (Lin & Lee, 2012). Moreover, it has been shown that computer-aided content analysis is an appropriate method to map out a research domain (see Fisk, Cherney, Hornsey, & Smith, 2012). The software tool creates so-called concept maps (see Figure 3) that display the core concepts within the text body (conceptual analysis) and show how these concepts are related to each other (relational analysis) by recording the frequency with which words co-occur in the text. Similar concepts that appear in close proximity are clustered together in the concept map (Smith & Humphreys, 2006): "The map is an indicative visualization that presents concept frequency (brightness), total concept connectedness (hierarchical order of appearance), direct inter-concept relative co-occurrence frequency (ray intensity), and total (direct and indirect) inter-concept co-occurrence (proximity)" (p. 264). Depending on the connectedness of concepts, thematic regions are identified, indicated by coloured circles, and named after the most prominent concept in the region.

Limitations

We acknowledge the limitation that the sample selection for the purposes of this content analysis is limited to publications in academic journals in the English language, even though much of the discussion about MOOCs also takes place at conferences and in their proceedings, on blogs, and social media. This choice of methodology was influenced by our aim to explore only fully-fledged research rather than non-evidence-based claims or opinions.

Journal publications are, of course, subject to various influences (Goldenberg & Grigel, 1991):

The most important of these is surely the gatekeeping role of editors, editorial boards, and reviewers of submissions to the journal. Quite aside from what one might prefer to do, publication responds to funding possibilities and publishing possibilities, and these in turn respond to connections and selection of a topic, a method, and a choice of potential journal most likely to lead to publication. (p. 436)

The text-mining tool Leximancer™ has been shown to produce stable and valid results for this kind of content analysis, as in Zawacki-Richter and Naidu (2016), who used this tool to map out research trends from 35 years of publications in the journal Distance Education. However, Harwood, Gapp, and Stewart (2015) highlighted that:

Leximancer is not a panacea, it still requires analytical sensitivity and judgment in its interpretation, but it is straightforward to probe the data and cross-check via the resultant maps. [...] Leximancer enables the analyst to make sense of large narrative data sets with minimal manual coding. The result is an efficient and impartial second opinion on open codes (concepts, categories and dimensions) and potential links between them. (p. 1041)

Thus, the generated concept maps require careful interpretation in light of exhaustive and profound knowledge of the subject matter under investigation.

Findings and Discussion

The concept map in Figure 3 depicts the major topics covered in the selected MOOC articles published between 2008 and 2015. The thematic summary includes a connectivity score to indicate the relative importance of the themes. The results reveal that the thematic region of courses has the most direct mentions within the text (i.e., titles and abstracts) with 599 (100% relative count), followed by MOOC / Massive Open Online Courses (83%), learners (23%), design (10%), analysis (9%), future (7%), and universities (6%). The following table provides an overview of the concepts in terms of their relative relevance in the concept map (see Figure 3).

Table 1

Ranked Concept List

In this section, the results of the text-mining analysis are described along four connected pathways that emerged from the selected MOOC articles: (a) the potential and challenges of MOOCs for universities; (b) MOOC platforms; (c) learners and content in MOOCs; and (d) the quality of MOOCs and instructional design issues. The selection of these four major research content areas in the MOOC literature is based on a qualitative interpretation of the central concepts (see Table 1) that are linked via the thematic regions in the overall concept map. In the following discussion, representative studies are chosen to illustrate the most prevalent research topics and themes covered in the publications.

Figure 3. Concept map of research articles on MOOCs (N=362).

The Potential and Challenges of MOOCs for Universities

The central theme of the papers is, unsurprisingly, open access to courses. The concepts of access, online, open, courses, and MOOCs are directly connected in the concept map; the potential of MOOCs in worldwide education development is discussed with a special emphasis on higher education opportunities. Many articles discuss the opportunities and challenges presented by implementing MOOCs at universities (see concept path: challenges - courses - online - open - access - education - potential - development - world - universities).

The authors acknowledge the potential of MOOCs to deliver education around the world. For instance, it has been reported that MOOCs can create opportunities for accessing quality higher education by building learning communities on a global scale (Mahraj, 2012) and reducing the cost of tuition (Ruth, 2012). There is also the possibility for innovative instructional designs to support self-regulated learning, unlike in traditional online courses (Bartolomé-Pina & Steffens, 2015). MOOCs also have potential in the field of corporate training, where they have been used to promote new recruiting techniques and innovative marketing and branding channels (Dodson, Kitburi, & Berge, 2015).

In addition to the many hopes for MOOCs and the benefits associated with them, the selected articles discuss and examine a number of challenges. High dropout and low completion rates in MOOCs are prominent topics in the publications (Kennedy, 2014). Conole (2015) argues that effective MOOC design is a key factor in combating challenges, naming three: (a) very high dropout rates; (b) learner authentication and cheating; and (c) providing support at an appropriate scale. Hew and Cheung (2014) list four key challenges with regard to teaching in MOOCs: (a) difficulty in evaluating students' work; (b) having a sense of speaking into a vacuum due to the absence of immediate feedback from students; (c) being burdened by the heavy demands of time and money; and (d) encountering a lack of student participation in online forums.

Other papers address the topic of licensing and intellectual property from the perspective of academic librarians (e.g., Mune, 2015; Gore, 2014). The business models on which MOOCs are based are important for their sustainability; Porter (2015) describes various models that are used by MOOC platforms and providers ("MOOConomics") and finds that most MOOCs are currently based on a freemium model, in which "a certain amount of a product is available to all, freely, whilst other parts of the product are charged for" (p. 57).

MOOC Platforms

The concepts MOOC and platforms are directly connected in the concept map (see concept path: MOOC - Massive Open Online Courses - platforms). This pattern is related to the popularity of xMOOCs, which are provided through learning platforms, as opposed to cMOOCs, which are provided in online, distributed, networked learning spaces.

Whilst Coursera, edX, and Udacity are the most established MOOC platforms, supporting very large numbers of learners, Ahn, Butler, Alam, and Webster (2013) explore alternative platforms "that promote more participatory modes of education production and delivery" (p. 160). They describe the platform of the Peer 2 Peer University, which invites any user to design and develop their own courses that can be taken by any other member of the community. The study explores how learners participated and engaged with online learning and course development using log data from the platform.

Other authors discuss MOOC platforms within specific content domains or national and cultural contexts; for instance, the Hasso Plattner Institute in Germany created the OpenHPI platform for special courses in information technology with a web tool for interactive software experiments (Neuhaus, Feinbube, & Polze, 2014). Adham and Lundqvist (2015) give an overview of Arab initiatives in the Middle East to launch their own country-specific MOOC platforms, such as Edraak in Jordan, Rwaq in Saudi Arabia, or MenaVersity in Lebanon. SkillAcademy, launched 2013 in Egypt, offers over 10,000 online courses at no cost. With regard to gender segregation in those countries, especially in Saudi Arabia, the authors believe that

MOOCs can help remove these cultural and social limitations and, that the social aspect should not be neglected. MOOCs can enable freedom of expression for women so they can communicate in a real world setting (mixed gender classes) meeting and interacting with others. (p. 134)

Finally, the institutional integration of MOOC platforms in the larger context of the digital learning and teaching infrastructure is an important topic. For example, Rocio, Coelho, Caeiro, Nicolau and Teixeira (2015) report on an open course on climate change at Universidade Aberta in Portugal, which was the largest MOOC course delivered in Portuguese. For this project, a technological solution was implemented to integrate the institutions open learning management system Moodle with open social software (Elgg).

Learners and Content in MOOCs

In order to produce effective learning experiences with quality learning materials, the analysis of learner characteristics and profiles is the starting point in the instructional design process (Morrison, Ross, Kalman, & Kemp, 2011; Stöter, Bullen, Zawacki-Richter, & von Prümmer, 2014). It is therefore not surprising that the concept path students - MOOC - learners - content forms a central backbone in the concept map.

The evaluation of student perceptions plays an important role in the course development process and the quality of e-learning in general. Regarding the evaluation of MOOCs, Li, Zhang, Bonk, and Guo (2015) integrated a MOOC into a traditional undergraduate course at a Chinese university and evaluated the perceived ease of use of the course environment, perceived interaction with peers, and overall learner satisfaction in order to derive suggestions on how to improve the course design. Zutshi, O'Hare, and Rodafinos (2013) examined student experiences with MOOCs through a content analysis of blog posts: "Results provided a glimpse of the student experiences, including why students take such courses, what elements of their experience are positive, and what can be improved from the student point of view" (p. 218).

Daza, Makriyannis, and Rovira Riera (2013) point out that, in open courses that are offered to thousands of students, it is very difficult to harmonize the different backgrounds of the participants given the diverse range of their prior knowledge, particularly with regard to mathematics. Phan, McNeil, and Robin (2016) investigated the association between learners' motivation for engagement, their prior knowledge, and course performance. Student motivation and its effects on course performance and completion are also investigated in several other studies (e.g., Stevanovic, 2014; Yang, 2014). Greene, Oswald, and Pomerantz (2015) found that "learners" expected investment, including level of commitment, expected number of hours devoted to the MOOC, and intention to obtain a certificate" (p. 925) are predictors of retention and achievement in MOOCs.

Online interaction patterns are a very prominent area of research in online and distance education (see Zawacki-Richter, & Anderson, 2014; Zawacki-Richter, Bäcker, & Vogt, 2009), and this issue is covered in several of the selected MOOC articles. For example, Gillani and Eynon (2014) used social network analysis to reveal when and how students interacted with one another and studied the relationship between forum participation and performance in terms of final marks. Clinnin (2014) also focused on interaction in discussion forums to understand how students presented their identities in forming learning communities.

Based on learners' needs and the content to be covered in a course, the development and reuse of learning materials in MOOCs are important topics in the analysed articles, and this is where open educational resources (OER) become a prominent issue. Atenas (2015) makes the point that, as "taxpayers are funding the development of these open and massive courses, access to the resources should be considered a right for all citizens who are interested in increasing their knowledge and improving their skills" (p. 10). In a more technical paper in the field of computer science, Piedra, Chicaiza, López, and Tovar (2015) propose an architecture and model for searching for OER for use in MOOCs. On the other hand, content creation has to be funded somehow, and the use and reuse of learning materials is part of the protected business model of the largest MOOC providers. Coursera, Udacity, EdX, and Future Learn have strict regulations in their terms and conditions that prohibit the reproduction, duplication, or redesign of any of their content. This is a major problem from the point of view of the Open Education Movement (Atenas, 2015).

Quality of MOOCs and Instructional Design Issues

The discussion about the quality of MOOCs is directly linked to research related to instructional design (see concept path: MOOC - study - quality - pedagogical - design) as evaluation and quality assurance is an integral part of the instructional design process (see Morrison et al., 2011). Around 2014, the first systematic MOOC quality assurance initiatives began to emerge; for example, Read and Rodrigo (2014) presented a quality model for MOOCs at UNED, the Spanish distance teaching university. In the European Excellence E-Learning Quality Project, Rosewell and Jansen (2014) developed a quality label based on benchmarks for MOOCs derived from the E-xellence label; an instrument for assessing the quality of e-learning in higher education. The European Foundation for Quality in E-Learning (EFQUEL) has also developed a special framework for the quality assurance of MOOCs (Creelman, Ehlers, & Ossiannilsson, 2014).

In contrast to these general quality frameworks, other authors elaborate in more detail on indicators of pedagogical or instructional quality. For example, in the context of teacher training, Aleman de la Garza, Sancho Vinuesa, and Gomez Zermeño (2015) administered a questionnaire with a set of indicators related to pedagogical, functional, technological, and time factors, in order to assess the quality of a MOOC on educational leadership with over 10,000 participants. Margaryan, Bianco, and Littlejohn (2015) compared and assessed the instructional design quality of xMOOCs and cMOOCs, concluding that "most MOOCs are well-packaged; [but] their instructional design quality is low" (p. 77). Admiraal, Huisman, and Van de Ven (2014) expressed particular concerns about the quality of self- and peer assessment in MOOCs. In a comparison of three MOOCs with 98,071 participants, they conclude that the quality of self- and peer assessment was only low to moderate, and that "both self-assessment and peer assessment should be used as assessment for learning instead of assessment of learning" (Admiraal, Huisman, & Van de Ven, 2014, p. 119).

Conclusion and Future Directions

This study provides an overview of the current state of research on MOOCs by analysing the titles and abstracts of publications in academic journals with a text-mining tool, in order to determine the prevailing themes and concepts in the MOOC studies. The research areas covered in these articles can be described along four major lines: (a) the potential and challenges of MOOCs for universities; (b) MOOC platforms; (c) learners and content in MOOCs; and (d) the quality of MOOCs and instructional design issues. These four broad research areas alternate between issues related to the institutional macro/meso level (opportunities and challenges of MOOCs for educational institutions, technological infrastructure, and platforms) and the micro level of teaching and learning in MOOCs (learner characteristics, content development, quality assurance, and instructional design). Zawacki-Richter and Naidu (2016) found a similar pattern of research in the last 35 years in the broader field of open, distance, and flexible learning.

MOOCs are but a new form of the open education phenomenon (cf. Weller, 2014); the content analysis revealed that open access to courses is a central theme in the publications. Open education should be open with regard to people, places, and methods. Online curriculum and course development, instructional design, quality assurance, student and faculty support, technological platforms, and infrastructure are - among other things - important issues to consider, not only in the context of MOOCs, but in open, online, and distance learning in general. Therefore, it is important to build upon the theory, research, and practice in the broader field of open, distance, and flexible learning, in order to prevent the research community reinventing the wheel.

Of course there are aspects that are unique to MOOCS, for example the obvious challenge to support and help very large numbers of students to succeed and to avoid dramatic drop-out rates. As Admiraal et al. (2014) discussed, carefully designed opportunities for peer support as well as self and peer assessment for learning (rather than assessment of learning) might be part of the solution, however more research is needed in this area.

In contrast, many MOOCs follow an instructional approach that leads to expository teaching and passive learning with poor student support. Research in the field of distance education has shown that student support and personal interaction, independent of time and space, is a critical factor in providing high quality learning opportunities. As the majority of MOOCs are organized as a series of video-based web-lectures, they can be compared with the development of video-conferencing in distance education in the 1990s. During this time, Daniel (1998) talked about a triple crisis of access, cost, and flexibility in a passionate keynote at a conference of satellite video-conferencing providers in the US:

Group teaching in front of remote TV screens? This is not only an awful way to undertake distance learning, but flies in the face of everything that we have learned while conducting successful open and supported learning on a massive scale for the past 27 years. Our lessons are the key to addressing the triple crisis of access, cost and flexibility now facing higher education world-wide. (p. 21)

Daniel criticized the synchronous mode of delivery in particular, which limits access and flexibility, but he also criticized video-conferencing as a very teacher-centred form of instruction. Given the huge demand for open access courses, Daniel's remarks about access, flexibility, and costs in higher education are obviously still relevant.

Furthermore, the evaluation of MOOCs and quality assurance is a very prominent and relevant topic in the publications. Rather than developing new quality frameworks for MOOCs from scratch, it is recommended to build upon quality models and instruments that were developed to measure the quality of multimedia applications, learning objects, and open educational resources (see Yuan & Recker, 2015).

After this first wave of MOOC hysteria, research and practice should focus on how best to harness the enormous opportunities that MOOCs might afford for providing access to knowledge and education, whilst equally addressing problematic issues like high dropout rates and the development of sustainable cost models. Major lessons learnt from the field of open, distance, and flexible learning (see Zawacki-Richter & Anderson, 2014), especially in the area of student support, instructional design, and quality assurance, should be kept in mind whilst moving forward.

Acknowledgement

The authors would like to extend their sincere appreciation to the Deanship of Scientific search at King Saud University for its funding this Research group NO. RG #---1435--003

References

Adham, R. S., & Lundqvist, K. O. (2015). MOOCs as a method of distance education in the Arab world - A review paper. European Journal of Open, Distance and E-Learning, 18(1), 123-139. doi: 10.1515/eurodl-2015-0009

Admiraal, W., Huisman, B., & Van de Ven, M. (2014). Self- and peer assessment in massive open online courses. International Journal of Higher Education, 3(3), 119-128. doi: 10.5430/ijhe.v3n3p119

Ahn, J., Butler, B. S., Alam, A., & Webster, S. A. (2013). Learner participation and engagement in open online courses: Insights from the Peer 2 Peer University. MERLOT Journal of Online Learning and Teaching, 9(2), 160-171.

Aleman de la Garza, L., Sancho Vinuesa, T., & Gomez Zermeño, M. G. (2015). Indicators of pedagogical quality for the design of a massive open online course for teacher training. RUSC. Universities and Knowledge Society Journal, 12(1), 104-118. doi: 10.7238/rusc.v12i1.2260

Atenas, J. (2015). Model for democratisation of the contents hosted in MOOCs. RUSC. Universities and Knowledge Society Journal, 12(1), 3-14. doi: 10.7238/rusc.v12i1.2031

Bartolomé-Pina, A. R., & Steffens, K. (2015). Are MOOCs promising learning environments? Communicar, 22(44), 91-99.

Bozkurt, A., Akgün-Özbek, E., & Zawacki-Richter, O. (2017). Trends and patterns in massive open online courses: Review and content analysis of research on MOOCs (2008-2015). The International Review of Research in Open and Distributed Learning, 18(5), 118-147. doi: 10.19173/irrodl.v18i5.3080

Bozkurt, A., Kilgore, W., & Crosslin, M. (2018). Bot-teachers in hybrid massive open online courses (MOOCs): A post-humanist experience. Australasian Journal of Educational Technology, 34(3), 39-59.

Bozkurt, A., Özdamar Keskin, N., & de Waard, I. (2016). Research trends in massive open online course (MOOC) theses and dissertations: Surfing the tsunami wave. Open Praxis, 8(3), 203-221.

Bulfin, S., Pangrazio, L., & Selwyn, N. (2014). Making MOOCs: The construction of a new digital higher education within news media discourse. The International Review of Research in Open and Distributed Learning, 15(5), 291-305. doi: 10.19173/irrodl.v15i5.1856

Calonge, D., & Shah, M. (2016). MOOCs, graduate skills gaps, and employability: A qualitative systematic review of the literature. The International Review of Research in Open And Distributed Learning, 17(5), 67-90. doi: 10.19173/irrodl.v17i5.2675

Chen, Y. (2014). Investigating MOOCs through blog mining. The International Review of Research in Open and Distributed Learning, 15(2), 85-106. doi: 10.19173/irrodl.v15i2.1695

Clinnin, K. (2014). Redefining the MOOC: Examining the multilingual and community potential of massive online courses. Journal of Global Literacies, Technologies, and Emerging Pedagogies, 2(3), 140-162.

Conole, G. (2015). Designing effective MOOCs. Educational Media International, 52(4), 239-252. doi: 10.1080/09523987.2015.1125989

Cormier, D. (2010, December 8). What is a MOOC? [Video file]. Retrieved from https://www.youtube.com/watch?v=eW3gMGqcZQc

Creelman, A., Ehlers, U., & Ossiannilsson, E. (2014). Perspectives on MOOC quality: An account of the EFQUEL MOOC Quality Project. International Journal for Innovation and Quality in Learning, 2(3), 78-87.

Cretchley, J., Rooney, D., & Galois, C. (2010). Mapping a 40-Year history with leximancer: Themes and concepts in the Journal of Cross-Cultural Psychology. Journal of Cross-Cultural Psychology, 41(3), 318-328.

Daniel, J. (1998). Can you get my hard nose in focus? Universities, mass education and appropriate technology. In M. Eisenstadt & T. Vincent (Eds.), The knowledge web - learning and collaborating on the net (pp. 21-29). London: Kogan Page.

Daza, V., Makriyannis, N., & Rovira Riera, C. (2013). MOOC attack: Closing the gap between pre-university and university mathematics. Open Learning: The Journal of Open, Distance and E-Learning, 28(3), 227-238. doi: 10.1080/02680513.2013.872558

Deimann, M. (2015). The dark side of the MOOC: A critical inquiry on their claims and realities. Current Issues in Emerging eLearning, 2(1), 3.

Diaz, V., Brown, M., & Pelletier, S. (2013). Learning and the massive open online course: A report on the ELI focus session [PDF file]. Retrieved from https://net.educause.edu/ir/library/pdf/ELI3029.pdf

Dodson, M. N., Kitburi, K., & Berge, Z. L. (2015). Possibilities for MOOCs in corporate training and development. Performance Improvement, 54(10), 14-21. doi: 10.1002/pfi.21532

Downes, S. (2012). Connectivism and connective knowledge: Essays on meaning and learning networks [PDF file]. National Research Council Canada. Retrieved from http://www.downes.ca/files/books/Connective_Knowledge-19May2012.pdf

Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: A review of nascent MOOC scholarship. Learning, Media and Technology, 39(3), 328-345. doi: 10.1080/17439884.2013.878352

Fisk, K., Cherney, A., Hornsey, M., & Smith, A. (2012). Using computer-aided content analysis to map a research domain: A case study of institutional legitimacy in postconflict east timor. SAGE Open, 2(4). doi: http://doi.org/10.1177/2158244012467788

Gasevic, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. The International Review of Research in Open and Distributed Learning, 15(5), 134-176. doi: 10.19173/irrodl.v15i5.1954

Gillani, N., & Eynon, R. (2014). Communication patterns in massively open online courses. The Internet and Higher Education, 23, 18-26. doi: 10.1016/j.iheduc.2014.05.004

Goldenberg, S., & Grigel, F. (1991). Gender, science and methodological preferences. Social Science Information, 30(3), 429-443.

Gough, D., Oliver, S., & Thomas, J. (Eds.). (2012). An introduction to systematic reviews. London; Thousand Oaks, Calif: SAGE.

Gore, H. (2014). Massive open online courses (MOOCs) and their impact on academic library services: Exploring the issues and challenges. New Review of Academic Librarianship, 20(1), 4-28. doi: 10.1080/13614533.2013.851609

Greene, J. A., Oswald, C. A., & Pomerantz, J. (2015). Predictors of retention and achievement in a massive open online course. American Educational Research Journal, 52(5), 925-955. doi: 10.3102/0002831215584621

Harwood, I. A., Gapp, R. P., & Stewart, H. J. (2015). Cross-check for completeness: Exploring a novel use of Leximancer in a grounded theory study. The Qualitative Report, 20(7), 1029.

Hew, K. F., & Cheung, W. S. (2014). Students' and instructors' use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45-58. doi: 10.1016/j.edurev.2014.05.001

Kennedy, J. (2014). Characteristics of massive open online courses (MOOCs): A research review, 2009-2012. Journal of Interactive Online Learning, 13(1), 1-16.

Kovanovic, V., Joksimovic, S., Gasevic, D., Siemens, G., & Hatala, M. (2015). What public media reveals about MOOCs: A systematic analysis of news reports. British Journal of Educational Technology, 46(3), 510-527. doi: https://doi.org/10.1111/bjet.12277

Krippendorf, K. (2013). Content analysis: An introduction to its methodology (3rd ed.). Thousand Oaks, CA: SAGE Publications.

Li, Y., Zhang, M., Bonk, C. J., & Guo, N. (2015). Integrating MOOC and flipped classroom practice in a traditional undergraduate course: Students' experience and perceptions. International Journal of Emerging Technologies in Learning (iJET), 10(6), 4. doi: 10.3991/ijet.v10i6.4708

Lin, J., & Lee, S. T. (2012). Mapping 12 years of communication scholarship: themes and concepts in the Journal of Communication. In H.-H. Chen & G. Chowdhury (Eds.), The outreach of digital libraries: A globalized resource network (Vol. 7634, pp. 359-360). Berlin, Heidelberg: Springer Berlin Heidelberg. Retrieved from http://link.springer.com/10.1007/978-3-642-34752-8_53

Liyanagunawardena, T., Adams, A., & Williams, S. (2013). MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distributed Learning, 14(3), 202-227. doi: 10.19173/irrodl.v14i3.1455

Mahraj, K. (2012). Using information expertise to enhance massive open online courses. Public Services Quarterly, 8(4), 359-368. doi: 10.1080/15228959.2012.730415

Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of massive open online courses (MOOCs). Computers & Education, 80, 77-83. doi: 10.1016/j.compedu.2014.08.005

Martin, F. G. (2012). Will massive open online courses change how we teach? Communications of the ACM, 55(8), 26-28. doi: 10.1145/2240236.2240246

Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2011). Designing effective instruction (6th ed.). Hoboken, NJ: Wiley.

Mune, C. (2015). Massive open online librarianship: Emerging practices in response to MOOCs. Journal of Library and Information Services in Distance Learning, 9(1-2), 89-100. doi: 10.1080/1533290x.2014.946350

Neuhaus, C., Feinbube, F., & Polze, A. (2014). A platform for interactive software experiments in massive open online courses. Journal of Integrated Design and Process Science, 18(1), 69-87.

Ossiannilsson, E., Altinay, F., & Altinay, Z. (2016). Analysis of MOOCs practices from the perspective of learner experiences and quality culture. Educational Media International, 52(4), 272-283. 10.1080/09523987.2015.1125985

Phan, T., McNeil, S. G., & Robin, B. R. (2016). Students' patterns of engagement and course performance in a massive open online course. Computers & Education, 95, 36-44. doi: 10.1016/j.compedu.2015.11.015

Piedra, N., Chicaiza, J., López, J., & Tovar, E. (2015). Seeking open educational resources to compose massive open online courses in engineering education: An approach based on linked open data. Journal of Universal Computer Science, 21(5), 679-711.

Porter, S. (2015). The economics of MOOCs: A sustainable future? The Bottom Line, 28(1/2), 52-62.

Raffaghelli, J., Cucchiara, S., & Persico, D. (2015). Methodological approaches in MOOC research: Retracing the myth of Proteus. British Journal of Educational Technologies, 46(3), 488-509. doi: 10.1111/bjet.12279

Read, T., & Rodrigo, C. (2014). Toward a quality model for UNED MOOCs. eLearning Papers, 37, 43-50.

Roberts, G., Waite, M., Lovegrove, E. J., & Mackness, J. (2013). x v c: Hybridity in through and about MOOCs. In Creating a virtuous circle: Proceedings of OER13. Milton Keynes: The Open University, Support Centre for Open Resources in Education. Retrieved from https://www.medev.ac.uk/oer13/file/79/9/

Rocio, V., Coelho, J., Caeiro, S., Nicolau, P., & Teixeira, A. (2015). iMOOC on climate change: Evaluation of a massive open online learning pilot experience. The International Review of Research in Open and Distributed Learning, 16(6), 152-173. doi: 10.19173/irrodl.v16i6.2160

Rodriguez, C. O. (2012). MOOCs and the AI-Stanford like courses: Two successful and distinct course formats for massive open online courses. European Journal of Open, Distance, and E-Learning, 15(2). Retrieved from http://www.eurodl.org/index.php?p=archives&year=2013&halfyear=2&article=516

Rosewell, J., & Jansen, D. (2014). The OpenupEd quality label: Benchmarks for MOOCs. Innoqual: The International Journal for Innovation and Quality in Learning, 2(3), 88-100.

Ross, J., Sinclair, C., Knox, J., & Macleod, H. (2014). Teacher experiences and academic identity: The missing components of MOOC pedagogy. Journal of Online Learning and Teaching, 10(1), 57-69. Retrieved from http://www.research.ed.ac.uk/portal/files/17513228/JOLT_published.pdf

Ruth, S. (2012). Can MOOCs and existing e-learning efficiency paradigms help reduce college costs? SSRN Electronic Journal. doi: 10.2139/ssrn.2086689

Sa'don, N. F., Alias, R. A., & Ohshima, N. (2014). Nascent research trends in MOOCs in higher educational institutions: A systematic literature review. 2014 International Conference on Web and Open Access to Learning (ICWOAL). doi: 10.1109/icwoal.2014.7009215

Saadatdoost, R., Sim, A. T. H., Jafarkarimi, H., & Mei Hee, J. (2015). Exploring MOOC from education and Information Systems perspectives: A short literature review. Educational Review, 67(4), 505-518. doi: 10.1080/00131911.2015.1058748

Sangrà, A., González-Sanmamed, M., & Anderson, T. (2015). Meta-analysis of the research about MOOCs during 2013-2014. Educación XX1, 18(2), 1-28. doi: 10.5944/educxx1.14808

Shen, C. W., & Kuo, C. J. (2015). Learning in massive open online courses: Evidence from social media mining. Computers in Human Behavior, 51, 568-577. doi: 10.1016/j.chb.2015.02.066

Smith, A. E., & Humphreys, M. S. (2006). Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping. Behavior Research Methods, 38(2), 262-279. 10.3758/bf03192778

Stevanovic, N. (2014). Effects of motivation on performance of students in MOOC. Paper presented at Sinteza 2014 - Impact of the Internet on Business Activities in Serbia and Worldwide. Serbia. doi: 10.15308/sinteza-2014-418-422

Stöter, J., Bullen, M., Zawacki-Richter, O., & von Prümmer, C. (2014). From the back door into the mainstream - the characteristics of lifelong learners. In O. Zawacki-Richter & T. Anderson (Eds.), Online distance education - Towards a research agenda (pp. 421-457). Edmonton, Canada: Athabasca University Press.

Tait, A. (2008). What are open universities for? Open Learning: The Journal of Open and Distance Learning, 23(2), 85-93. doi: 10.1080/02680510802051871

Veletsianos, G., & Shepherdson, P. (2015). Who studies MOOCs? Interdisciplinarity in MOOC research and its changes over time. The International Review of Research in Open and Distributed Learning, 16(3), 1-17. doi: 10.19173/irrodl.v16i3.2202

Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013-2015. The International Review of Research in Open and Distributed Learning, 17(2), 198-221. doi: 10.19173/irrodl.v17i2.2448

Waite, M., Mackness, J., Roberts, G., & Lovegrove, E. (2013). Liminal participants and skilled orienteers: Learner participation in a MOOC for new lecturers. Journal of Online Learning and Teaching, 9(2), 200. Retrieved from http://jolt.merlot.org/vol9no2/waite_0613.htm

Weller, M. (2014). The battle for open: How openness won and why it doesn't feel like victory. Ubiquity Press. Retrieved from http://www.oapen.org/search?identifier=533876

Yang, Q. (2014). Students' motivation in asynchronous online discussions with MOOC mode. American Journal of Educational Research, 2(5), 325-330. doi: 10.12691/education-2-5-13

Yuan, M., & Recker, M. (2015). Not all rubrics are equal: A review of rubrics for evaluating the quality of open educational resources. International Review of Research in Open and Distributed Learning, 16(5), 16-38.

Zawacki-Richter, O., & Anderson, T. (Eds.). (2014). Online distance education - towards a research agenda. Edmonton, Canada: Athabasca University Press. doi: 10.15215/aupress/9781927356623.01

Zawacki-Richter, O., & Naidu, S. (2016). Mapping research trends from 35 years of publications in Distance Education. Distance Education, 37(3), 245-269. 10.1080/01587919.2016.1185079

Zawacki-Richter, O., Bäcker, E. M., & Vogt, S. (2009). Review of distance education research (2000 to 2008) - analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distance Learning, 10(6), 21-50.

Zutshi, S., O'Hare, S., & Rodafinos, A. (2013). Experiences in MOOCs: The perspective of students. American Journal of Distance Education, 27(4), 218-227. doi: 10.1080/08923647.2013.838067

What Research Says About MOOCs - An Explorative Content Analysis by Olaf Zawacki-Richter, Aras Bozkurt, Uthman Alturki, and Ahmed Aldraiweesh is licensed under a Creative Commons Attribution 4.0 International License.