International Review of Research in Open and Distributed Learning

Volume 18, Number 5

August - 2017

Trends and Patterns in Massive Open Online Courses: Review and Content Analysis of Research on MOOCs (2008-2015)

Aras Bozkurt¹, Ela Akgün-Özbek², and Olaf Zawacki-Richter^3
1,2Anadolu University, Turkey, ³Carl von Ossietzky University of Oldenburg, Germany

Abstract

To fully understand the phenomenon of massive open online courses (MOOCs), it is important to identify and map trends and patterns in research on MOOCs. This study does so by reviewing 362 empirical articles published in peer-reviewed journals from 2008 to 2015. For the purpose of this study, content analysis and discourse analysis were employed to analyze the articles. Accordingly, the trend line showing the number of articles per year indicates that the extent of research on MOOCs is likely to increase in the coming years. In terms of research areas, the findings reveal an imbalance and three research areas out of fifteen constitute more than half of all research on MOOCs. With regard to types of MOOCs, related literature is dominated by research on xMOOCs. The discourse in MOOC articles takes a mostly neutral standpoint, articles with a positive outlook outweigh those that are negative, and there is an increase in a more critical discourse. Theoretical or conceptual studies are preferred by researchers, although MOOC research generally does not benefit from being viewed through theoretical or conceptual lenses.

Keywords: distance education, open and distance learning, Massive Open Online Courses, MOOCs, research trends

Introduction

The phenomenon of MOOCs has recently attracted considerable attention in the fields of higher education (HE), lifelong learning, and distance education (DE). In spite of the increasing demand and interest, many questions remain unanswered regarding what MOOCs really are and where they are heading in terms of their impact on educational institutions and educational opportunities. Among many published evaluations, researchers have used the following terms to refer to MOOCs: a disruptive innovation (Skiba, 2012; Billington & Fronmueller, 2013; Flynn, 2013); a digital tsunami (Auletta, 2012; Brooks, 2012; McKenna, 2012); an avalanche (Barber, Donnelly, Rizvi, & Summers, 2013); a revolution (Friedman, 2012); a global mega class (Bozkurt, 2016); an invasion (Krause & Lowe, 2014); a mania (Meisenhelder, 2013); and an educational buzzword (Daniel, 2012). In the light of such exuberant discussions about the "identity" of MOOCs, we believe that as a first step in exploring the phenomenon of MOOCs and trying to understand their past, present, and future, we should investigate research trends and patterns in the body of research on MOOCs.

In this context, the aim of this study is to provide a panoramic overview of MOOC research from 2008 to 2015 by identifying trends and patterns through a systematic review of the related literature. Within this perspective, the purpose of this study is to address the following research questions.

• What are the trends in research areas?
• What are the most researched MOOC types?
• What discourses are dominant in MOOC research?
• Which articles are cited the most in papers on MOOCs?
• What are the trends in methodology and research design (or models) in papers on MOOCs?
• What are the trends in theoretical backgrounds in MOOC research?

Literature Review

The increasing interest in network based lifelong learning models, that is, Massive Open Online Courses (MOOCs), has ignited efforts to identify trends in research on the topic. Although limited in number and scope, there have been valuable initiatives in reviewing MOOC research in scholarly articles, dissertations, and in the broadcast and social media.

Studies in Academic Journals

The first review study on MOOC research was conducted by Liyanagunawardena, Adams, and Williams (2013), who reviewed the published literature on MOOCs between 2008 and 2012. They identified and analyzed forty-five peer-reviewed papers. The results of their study reveal that research on MOOCs increased dramatically after they had been in existence in 2008; cooperative research efforts were popular, and research on MOOCs dealt with several domains of this topical field - from pedagogy and theory to technology.

Gaševi, Kovanovi, Joksimovi, and Siemens (2014) conducted an analysis of the 266 submissions to the MOOC Research Initiative (MRI) in 2013, which was funded by the Gates Foundation. They examined the main research themes and research methodologies used in those studies. They found that social learning as a theme received the greatest interest, and mixed methods was the most preferred research approach.

Ebben and Murphy (2014) analyzed 25 peer-reviewed articles to identify aspects of the scholarly discourse on MOOCs. They identified two major phases of scholarship on MOOCs, namely Connectivist MOOCs, Engagement, and Creativity from 2009 to 2011/2012 (phase 1); and xMOOCs, Learning Analytics, Assessment, and Critical Discourses about MOOCs from 2012 to 2013 (phase 2).

Sa'don, Alias, and Ohshima (2014) examined 164 papers published between 2008 and mid-2014 to identify emergent trends regarding MOOCs in higher educational institutions (HEIs). They reported that the top ten nascent research trends in MOOCs for HEIs (at that time) were pedagogical issues, assessment and accreditation, engagement or motivation, knowledge sharing, cultural diversity, technology, social interaction, participant retention, learning analytics, policy, and instructional design.

Kennedy (2014) identified the characteristics of MOOCs in informal and post-secondary e-learning with a review of research conducted between 2009 and 2012. After the elimination of several articles, six articles were used to identify the characteristics of MOOCs. She found that openness, barriers to persistence, and MOOC models were the main characteristics that dominated MOOC research at that time.

Veletsianos and Shepherdson (2015) conducted research by applying descriptive and inferential statistics to bibliometric data to investigate inter-disciplinarity in MOOC research. They examined 183 research papers published between 2013 and 2015. They reported that education and computer science disciplines were the most prevalent, with a trend towards more interdisciplinary approaches between 2013 and 2015 (Veletsianos & Shepherdson, 2015) compared to MOOC research published between 2008 and 2012 (Liyanagunawardena et al., 2013).

Raffaghelli, Cucchiara, and Persico (2015) discussed the methodological approaches in MOOC research between January 2008 and May 2014. Their analysis of 60 articles showed that the majority of research consisted of theoretical studies and case studies; and that there is a need for clear guidelines to identify research methodologies appropriate for the ontological and epistemological questions that address MOOCs.

Sangrà, González-Sanmamed, and Anderson (2015) investigated 228 studies that focused on MOOCs between 2013 and 2014. They found that pedagogical strategies, learner motivation, and implications for HE systems were the most popular focus areas.

Veletsianos and Shepherdson (2016) examined 183 papers on empirical studies of MOOCs published between 2013 and 2015, in order to identify gaps in the related literature. They found that most of the contributions to MOOC literature come from North America and Europe. They reported that the selected papers had a focus on students (83.6%), design (46.4%), context and impact (10.9%), and instructors (8.2%).

Dissertations and Theses

Bozkurt, Özdamar Keskin, and de Waard (2016) reviewed 51 theses and dissertations published between 2008 and 2015. They identified that MOOCs are on the verge of the "plateau of productivity" as described in the Gartner Hype Cycle. Additionally, they found that, though it is a multidisciplinary research avenue, MOOC research is dominated by the field of education; and researchers used qualitative (49%), quantitative (21%), mixed (18%), review (8%), and other (4%) research methodologies. They also highlighted the finding that nearly half of the theses and dissertations ignored any possible benefits from employing theoretical frameworks by not using them. The MOOC research in the theses and dissertations that were analyzed, focused on (extended) xMOOCs rather than on (connectivist) cMOOCs.

Broadcast and Social Media

Bulfin, Pangrazio, and Selwyn (2014) investigated 371 news media headlines over the preceding 24 months within mainstream news media sources in the United States, Australia, and the UK to identify how MOOCs are perceived in these sources. In their analysis, they found that MOOCs are considered to be a portentous development for HE.

Kovanović, Joksimović, Gašević, Siemens, and Hatala (2015) examined 3958 news articles, ranging from 2008 to the first half of 2014, to identify MOOC-related public discourse. By using topic-modeling technique, their research revealed that while the total number of news articles followed a declining trend, the quality of the discussions demonstrated an increasing trend.

Deimann (2015) examined the MOOC movement by conducting a discourse analysis of 58 articles published in the New York Times between 2012 and 2013. He indicated that the MOOC phenomenon is fueled by a net of power-knowledge relations and MOOCs contribute to a deeper understanding that is beyond pedagogical or economical perspectives.

Chen (2014) investigated 306 blog posts related to MOOCs published from January 2010 to June 2013, making use of text-mining. He reported that MOOCs provide opportunities to learners, faculty members, universities, and MOOC providers. He also found that challenges that MOOCs need to overcome include questionable course quality, high dropout rates, unavailable course credits, ineffective assessments, complex copyright issues, and necessary hardware required to join MOOCs.

Finally, Shen and Kuo (2015) performed a sentiment and influencer analysis based on Twitter data from June 2013 to May 2014 to explore public sentiment on social media towards MOOCs. They found that positive tweets outweighed negative tweets, even though a slight increase in the number of negative tweets was evident over that time period.

When these articles are examined in terms of their scope, it can be noticed that they covered different aspects of MOOC research, which makes it difficult to compare research findings with each other and conduct follow-up studies. The range of above review studies differ from sample size to issues covered. However, it is also observed that the methodological approaches, type of MOOCs, opportunities and challenges, use of technology in education, pedagogical approaches, social interaction, use of technology in education, HEIs, quality assurance, and dropout and retention rates were common interests in most of these MOOC reviews.

However, one of the common issues that was salient in MOOC review studies was the cultural relationship and geographical distribution of the participants or authors that were interested in MOOCs. Liyanagunawardena et al. (2013) reported that sampled studies in their research mostly presented participant demographics, which demonstrated that a large majority of participants were from North America and Europe. Similarly, in other MOOC review studies (Ebben & Murphy, 2014; Gašević et al., 2014; Veletsianos & Shepherdson, 2016), it was reported that majority of the authors of MOOC studies were mainly originated from North America and Europe; followed by authors from Australia, Asia, or Africa. This indicates a geographical pattern for the interest in MOOC research and might further indicate a linguistic or cultural relationship.

Another interesting point highlighted in MOOC review articles was the need for new methodological approaches resulting from complex and new nature of networked learning spaces. Thus, approaches such as data-mining, learning analytics, or social network analysis in MOOC research (Ebben & Murphy, 2014; Gašević et al., 2014; Kovanović et al., 2015; Raffaghelli et al., 2015; Sangrà et al., 2015) would be helpful to analyze and interpret massive, sheer volume of data; in other words, big-data, distributed across the networks and globe.

The number of sampled articles analyzed in the reviews presented above ranges from 6 to 266 articles. None of the above mentioned reviews regarding MOOC research analyzed the trends from the advent of MOOCs in 2008 all the way through to 2015. Therefore, to be able to identify and track research trends and patterns, there is a need for a longitudinal and inclusive review of MOOC research over that time period. With this in mind, this research aims to contribute to the MOOC literature by providing a comprehensive systematic analysis of research on MOOCs from 2008 to 2015.

Conceptual Background

Classification of Research Areas

In a systematic review study, it is vital to reflect what has been done in previous research studies and what has been omitted. Therefore, a framework of research areas in distance education, developed by Zawacki-Richter (2009), was used to identify the most prominent and the most neglected areas in MOOC research. Zawacki-Richter's (2009, p.7-9) framework consists of the following levels (an extended version is presented in Appendix A).

Macro level: Distance education systems and theories

1. Access, equity and ethics
2. Globalization of education and cross-cultural aspects
3. Distance teaching systems and institutions
4. Theories and models
5. Research methods in distance education and knowledge transfer

Meso level: Management, organization and technology

1. Management and organization
2. Costs and benefits
3. Educational technology
4. Innovation and change
5. Professional development and faculty support
6. Learner support services
7. Quality assurance

Micro level: Teaching and learning in distance education

1. Instructional design
2. Interaction and communication in learning communities
3. Learner characteristics

Reliability

Articles included in the sample were coded by the first author of this paper, and re-coded by the second author, according to above-mentioned framework of research areas in DE. The extent of agreement between the two raters was calculated using the Kappa statistic proposed by Cohen (1960), which yielded an inter-rater reliability of κ = 0.913. A value of between 0.81 and 1.00 reflects almost perfect agreement (Landis & Koch, 1977), or according to Altman (1991), a value within the same interval is regarded as being very good. Thus, the coding of the articles according to the DE research areas can be considered as being acceptable, with an inter-rater value of 0.913 for Cohen's Kappa statistic.

Classification of Research Method, Designs, and Models

Educational research is usually dominated by qualitative, quantitative, or mixed methods research. However, the advent of network technologies has enabled some innovative research methods based on specific data collection and analysis techniques such as the use of "big data" in learning analytics. In this sense, a new schema of research methods and models/ designs was introduced in this research. On these grounds, in addition to quantitative, qualitative, mixed and theoretical research methodologies, data mining and analytics was included. Additionally, two research methods - design-based research and action research - that don't fit into any of the standard research methodologies, were classified as "practice-based" methodologies.

Method and Sample

Research Method and Design

This paper used the method of systematic review (research synthesis) to arrive at a comprehensive and reliable overview of MOOC research. Systematic reviews involve three key activities: identifying and describing relevant research, critically appraising research reports in a systematic manner, and synthesizing research findings into a coherent statement (Gough, Oliver, & Thomas, 2012). Such reviews can provide guidance for researchers in planning future studies, as well as convenient summaries of the literature on a particular issue (Petticrew & Roberts, 2008). Two basic systematic research methodologies are aggregative and configurative reviews (Gough, Oliver, & Thomas, 2012). In this study, a configurative review was used, in which the synthesis is made predominantly by configuring data from the sampled studies to answer the review questions.

Sampling

The selected articles were found by searching for using the following keywords: MOOC, MOOCs, Massive Open Online Course, and Massive Open Online Courses. To screen the articles, multiple academic databases were used; however, EBSCO, ERIC, Google Scholar, and Scopus were found to provide the most comprehensive search results. Searches were conducted for each year separately, and recurring articles were removed from the list of sampled articles. The inclusion criteria for sampling were: published in a peer-reviewed journal between 2008 and 2015; written in English; online full-text accessibility; and searched keywords to appear in the title.

The search was limited to the time period from 2008 to 2015. The year 2008 was selected as a starting point since the first MOOC was run at this date, and the first example from the grey literature, that is to say non-conventional, non-commercial literature, was written in 2008 by Cormier (2008) who also invented the term "MOOC." Though there were some articles that used the searched keywords in their abstracts or list of keywords (or both), we deliberately selected only those that included the keywords in their titles, assuming that this would identify articles with MOOCs as their focal point.

After screening and examining 888 articles, a total of 362 articles (Figure 1) that met the inclusion criteria were further examined according the research questions of the study.

Figure 1. Frequency of the sampled articles by year.

Data Collection, Procedure, and Analysis

The study used document analysis to collect data, content analysis to identify research trends and patterns, and discourse analysis to identify the tone of the selected articles. The overall research flow is shown in Figure 2.

Figure 2. The overall research flow.

Document analysis was used to collect data and create a valid corpus based on the research questions and inclusion criteria stated above. Document analysis is a technique that involves skimming (superficial examination), reading (thorough examination), and interpretation (Bowen, 2009). During the initial searching and screening processes, a total of 888 papers were identified. This first corpus was analyzed through skimming, which yielded that 526 papers were irrelevant (articles that have searched keywords in the title, but do not address MOOCs in the main text), or did not meet the inclusion criteria; these were then excluded. Following the document analysis process, 362 empirical articles that were published in peer-reviewed journals were selected for further analysis.

After the identification of the 362 articles, the researchers used content analysis, which can employ different methodological approaches (qualitative or quantitative or both) and a variety of data types (Banks, Louie, & Einerson, 2000). The use of such approaches allows researchers to make replicable and valid inferences from data within their context, with the purpose of providing knowledge, new insights, a representation of the facts, and a practical guide to action (Krippendorff, 1980). The 362 articles or related sections were coded based on explicit rules of coding (Berelson, 1952), and according to pre-set categories (e.g., research method, model/ design etc.) defined by the researchers, keeping in mind the purpose of the research, and according to predefined research areas of DE. Some sections were based on counting the findings (e.g., citation analysis). The results were reported using trendline graphs or descriptive analysis such as frequency or percentage values.

In the final step, the researchers applied discourse analysis to analyze the viewpoints, perspectives, and aims hidden in the text, in order to reveal the position taken by the authors of each paper (Van Dijk, 1993). For this purpose, the conclusion sections of the papers were coded according to preset categories of positive, negative, neutral, and critical discourses, so as to identify the relationship between the text and the underlying message.

Significance and Limitations of the Study

This study provides a 360-degree evaluation of research on MOOCs by identifying trends and patterns in the field over an eight-year period. The aim is to examine the phenomenon from different aspects and thus provide a complete map of the field. Articles included in this research study were collected through an open search that provided a significant corpus of 362 articles and enabled the researchers to present a holistic perspective from the advent of MOOCs in 2008 to 2015. This time span is a sufficient period to allow the field to mature and provide sufficient data to identify trends and patterns. Lastly, the study not only identifies trends and patterns in MOOC research, but also provides a research agenda for future directions, which is important for the improvement of MOOCs in particular, and open, distance, and distributed learning in general.

In addition to its significance, this study has some limitations. First of all the research corpus is limited only to peer-reviewed articles published in journals between 2008 and 2015. Other studies such as conference proceedings were not included with an assumption that not all proceedings are filtered through a review mechanism. However, it is thought that conducting a similar analysis would contribute to the literature. Secondly, as lingua franca, only articles written in English were included to the research to reach a global perspective. Thirdly, the articles which have online full-text access were included in the research corpus and those that required payment to access full-text were excluded. Finally, articles that included searched keywords in their titles were analyzed. The rationale for such an approach is that: articles that used defined keywords in their title would specifically focus on MOOCs.

Findings and Discussion

This section explains research areas, patterns (discourse analysis, MOOC types, and citation analysis) and trends (methodology, research design/model, and theoretical frameworks) of MOOCs respectively.

Research Areas

The classification of research areas in distance education developed by Zawacki-Richter (2009) was used to identify research areas in MOOCs and provide a research agenda for future studies. MOOC studies were analyzed and coded according to three levels of classification and fifteen research areas (Figure 3). On examination, it is seen that there is an unbalanced distribution among the three levels of classification.

At macro level (overall 40.4%), theories and models (27%) is the strongest emerging research area, since more than a quarter of the articles studied MOOCs within angle of this research area. The inflation in this research area also increased the overall value of the macro level. Other macro level research areas such as research models in DE and knowledge transfer (4.2%), globalization of education and cross-cultural aspects (3.8%), distance teaching systems and institutions (2.8%), a ccess, equity, and ethics (2.6%) were identified as research areas that are rather neglected.

At the meso level (overall 25.6%), quality assurance (6.1%) is the most emerging research area, which includes topics such as dropout rates, accreditation and quality standards. Cost and benefits (5.6%), educational technology (4.2%), innovation and change (4%), and professional development and faculty support (3.3%) are less investigated research areas. Following these, management and organization (1.6%) and learner support services (0.7%) appear to be the least examined research areas at this level.

The micro level (overall 34.4%) is the second most studied level of classification. At this level, learner characteristics (15.7%) is the most examined research area, followed by instructional design (11%), and interaction and communication in learning communities (7.3%).

The three most studied research areas (theories and models at the macro level; learner characteristics and instructional design at the micro level) constitute 53.7% of the overall corpus, which clearly identifies the remaining twelve research areas as those that need to be studied more. These findings related to research areas have revealed not only those areas that are most researched, but also those that are most neglected and require a special focus to improve MOOC practices. Based on these findings, researchers and institutions can develop a research agenda and adjust their research interests accordingly.

Figure 3. Distribution of research areas by years.
*One study may include more than one research area

Discourse Analysis

The articles in the research corpus were coded (per year) according to the tone in the conclusion section, according to four categories: positive, negative, neutral, and critical (Figure 4). In a general sense, positive papers perceived MOOCs to be a hopeful innovation, while negative papers perceived them as hype. Neutral papers demonstrated no support for any side of the MOOC argument. Critical papers expressed the pros and cons of MOOCs within a holistic perspective, and provided careful judgements regarding their conclusions.

According to this classification, the nature of the discourse across all the articles is 27.1% positive, 1.1% negative, 56.4% neutral, and 15.5% critical. Based on the positive trend lines, it is possible to say that MOOCs will remain on the research agenda of open and distance learning. It is also noteworthy that very few of the papers take a negative perspective, while a considerable number of research articles take a critical perspective. Since critical researchers don't take the promise of MOOCs for granted, they produce valuable information that can contribute to the improvement of this lifelong learning model. The overall outlook provides an insight regarding how MOOCs are perceived in academia.

Figure 4. Distribution of discourse in MOOC articles by year.

On examination of Figure 4, it can be seen that the number of critical articles has been increasing from 2012 onwards. This may indicate that MOOCs are now being evaluated from more realistic perspectives, which is significant in order to build a robust research foundation. In terms of positive and negative discourses, the findings of this study demonstrate similar patterns with Shen and Kuo (2015). In their sentiment analysis, they found that public opinion in microblogging services generally favoured learning through MOOCs. Critical discourse in MOOC articles also has a similar pattern with previous research. For instance, Adams (2013) reported that the critical discourse became more apparent by 2013 because it was thought that MOOCs failed to achieve their promises. Ebben and Murphy (2014) also stated that a critical discourse about MOOCs started by 2012. In a similar vein, according to an analysis of public discourse surrounding MOOCs, Kovanović et al. (2015) analyzed 3,958 news articles and in line with the thoughts and findings of previous research, found that while there is a decrease in the number of MOOC-related news articles, the quality of the discussions in news articles appears to be increasing. They also noted that the discourse about MOOCs changed significantly.

As stated in previous research, year 2012/2013 is the beginning of critical discourse that can be linked to high dropout rate that was diagnosed with the second generation xMOOCs which followed the first generation cMOOCs. We also see that how an open, free learning model, that is to say cMOOCs, was transformed into something semi-open, that is to say xMOOCs, and adopted freemium business model resulting with criticism by 2012 onwards.

MOOC Types

The selected articles were examined according to MOOC types (Figure 5). For the analysis, research articles were coded if the data was gathered in a specific type of MOOC, and theoretical/conceptual articles were coded if they explained MOOC types in their literature review section and provided a synthesis accordingly.

Figure 5. Distribution of MOOC types by year.

Interestingly, across all the articles, the number of articles (n=27; 7.5%) that focus on cMOOCs is relatively small since their first appearance in 2008, while there was a sudden increase in the number of xMOOC articles from 2013 onwards (n=116; 32.1%). The number of articles that did not distinguish between the MOOC types or reported on both c- and xMOOCs (n=53; 14.7%) shows a similar pattern to xMOOCs. Although the number is small (n=6; 1.7%), there are some articles that report on hybrid/dual-layer MOOCs.

The letters in the MOOC acronym clearly define its meaning, and the initial letters such as "c" (connectivist) and "x" (extended/extension) describe the main distinctions among MOOC types. However, a great number of the articles (n=159; 44%) neither defined the type of MOOCs they studied, nor clarified it in their literature review section. This creates a problematic situation in MOOC research and hinders the development of the phenomenon by ignoring the inherent diversity of MOOC types, resulting in possible skepticism about the quality and depth of MOOC research. This finding also aligns with the discussion in the following methodology and research design/model section, namely that there are a great many loosely written papers in MOOC research without a specific methodological focus. Such obscurity in methodological frameworks and opacity in the definition of MOOC types could be construed as precluding MOOCs from being both a ripe research realm and a promising educational practice.

Findings of this study confirm those of both Liyanagunawardena et al. (2013) and Veletsianos and Shepherdson (2016), who highlighted the ambiguity in definition of MOOC types. There appears to be a need to define MOOC types more explicitly and explain emerging pedagogies, particularly since there are new experimental MOOC types such as hybrid/dual-layer MOOCs.

Another significant point is the relatively small number of articles between 2008 and 2011 period when cMOOC was the only MOOC type. Although there are other sources of data that are not included in this study (e.g., reports and conference proceedings), there is clearly a lack of cMOOC research in both periods 2008-2011 and 2012-2015. In contrast, after the first implementation of xMOOCs in 2011, this type of MOOC experienced a sudden increase in the number of research articles about it.

It was reported that the time period 2008-2011/12 was cMOOC dominant while 2012-2013 was xMOOC dominant (Ebben & Murphy, 2014). The findings of this study confirm Ebben and Murphy and raise an interesting fact, which indicates that the MOOC research sphere is xMOOC dominant and it seems that xMOOCs are being the default type. There might be many reasons regarding this trend. In an effort to explain the dominance of xMOOCs, Veletsianos and Shepherdson (2015) suggested that pedagogic approaches used in xMOOCs are well-known among the academics and very similar to those already have been used and widely adopted in HEIs and online learning practices, while the approaches used in cMOOCs are not much known and academics might be hesitant or skeptical about it. Accordingly, it is possible to suggest that among different MOOC types (cMOOCs, xMOOCs, or hybrid/dual-layer MOOCs), xMOOC research has matured and was adopted by mainstream education, while cMOOC and hybrid/dual-layer MOOC research needs more time to mature and potentially there are many issues to further explore.

The rise of empirical MOOC research is notable from 2012 (Figure 1). MOOCs originated from genuine and noble ideas such as openness, connectivism, critical pedagogy, and redistribution of the power in learning processes to the learners; however, elite universities and xMOOC platforms are now major players in the MOOC field, and consequently xMOOCs have come to dominate MOOC research (Figure 5). The current state of the art regarding the domination of xMOOCs highlights an ongoing discussion: Are MOOCs a form of disruptive or sustaining innovation? The first appearance of MOOCs can be regarded as a disruptive innovation and they attracted a lot of attention in both public and academic environments. However, strategic moves from elite universities and venture capitalists have changed the perspective, resulting in the transformation of MOOCs from a disruptive innovation to a sustaining innovation.

Citation Analysis

A total of 11,520 references were collected from the 362 articles published between 2008 and 2015, thus providing a map of the most cited works in MOOC research (See the five most cited works in Table 1). The 74 works that are cited at least 10 times are listed in Appendix B. The citation pattern reveals some interesting results. The 74 works are cited a total of 1,696 times which means that they constitute 14.7% of 11,520 references. This distribution of references exhibits a similarity with the Pareto Principle (Juran, 1975), which suggests that approximately 80% of outcomes originate in 20% of incomes. Similarly, 14.7% of the references in the MOOC articles provide a considerable amount of the knowledge for the MOOC research in general. Beyond descriptive findings in regard to citation analysis, this finding also provides insights about knowledge production and consumption patterns in this scholarly area. Though covering different time periods and reported in different ranks, the studies listed in most cited works also overlap with those reported by Gašević et al. (2014) and Veletsianos and Shepherdson (2016), which supports the idea proposed in Pareto Principle.

Considering the wide range of MOOC papers analyzed, the works listed in Appendix B constitute the touchstones of MOOC research from 2008 to 2015 and provide a robust reference guide for future researchers.

Table 1

The Five Most Cited Works

Note. An extended list is provided in Appendix B.

Methodology and Research Design/Model

The findings reveal that, of all the sampled studies (n=362), conceptual/descriptive studies constitute the most employed methodology (53.3%) (Figure 6) - more than half of the studies that examined MOOCs used this methodology. They took the form of literature reviews (24.3%), position papers (8%), opinion papers (6.1%), reports (5.5%) and other research models. Furthermore, conceptual/descriptive studies constitute the majority of papers in almost each year.

Unfortunately, though conceptual/descriptive studies have value on their own, many of the studies using this type of the methodology were poorly reported with a lack of empirical data, and did not contribute much to the literature or synthesize current literature; on the contrary, many are superficial reviews. In addition, the number of position and opinion papers is obtrusive. This indicates that many of the researchers at that time were still discussing the phenomenon of MOOCs, and deciding whether they were for or against it.

Currently, most of the MOOCs are provided through learning platforms such as Coursera and the data needed for research are confined to these platforms. Thus, the distinctly outnumbered conceptual/descriptive papers further indicate the obstacles to access MOOC data, which, in turn, ends up with 193 conceptual/descriptive papers out of 362.

The second most employed methodology is quantitative research (19.6%). The most commonly used quantitative methods were survey (12.2%), correlational (6.1%), or experimental (1.7%) studies. Quantitative methodology is important in being able to generalize research findings and improve the field horizontally; in particular, more correlational and experimental research studies are needed to explore the complex nature of MOOCs.

Figure 6. Distribution of research methodologies employed.

The third most employed methodology is qualitative studies (14.9%). Descriptive (4.1%), case studies (3%), and content analysis (2.2%) were the most preferred models. In contrast to horizontal quantitative methodology, vertical qualitative methodology provides an in-depth perspective in exploring a particular phenomenon. Considering the massive and open dimensions of the MOOC, we believe that there is a need for more qualitative research to investigate the undiscovered nature of this learning model.

Data mining and analytics follows as the next most prevalent research methodology (6.1%). Learning analytics (2.2%), social network analysis (1.4%), text-mining (1.4%), log analysis (1.4%), and Internet and traffic ranks (0.8%) were the preferred models. As an emerging research methodology, data mining and analytics enables a multi-layered examination of MOOCs, which are digital, networked, and online in nature.

Mixed methods ranked fifth (5.5%) among the research methodologies employed in the selected articles. Explanatory sequential (3%) and convergent parallel (1.9%) designs were the most used in this category. Considering the many aspects of MOOCS that may be studied, mixed methodology is a powerful approach for researchers to employ, building on the strengths of both quantitative and qualitative data. Nonetheless, the use of mixed methods in the selected articles is relatively low.

Practice-based studies constitute the least used type of research methodology (0.6%), with action research (0.6%) being the only model evident. Practice-based methodologies follow reflective, iterative, cyclical processes, which may be difficult for individual researchers to implement in studying MOOCs. Nevertheless, such studies would be favorable in contributing to promoting the success and sustainability of MOOCs.

These findings confirm those of Raffaghelli et al. (2015), who reported that theoretical/conceptual papers constitute the majority, that is to say 23.3% of MOOC research (n=60). They noted that mixed (20%), quantitative (15%), "others" (15%), qualitative (11.7%), and design-based research (8.3%) were other methodologies used. In their analysis, they classified empirical articles under a research paradigm based on their explicit findings, or their own interpretation when the methodology was not explicitly stated in a study. Because no articles in the scope of the present study were classified as design-based research, our findings do not match those of Raffaghelli et al. (2015) in the category of practice-based research models. Gašević et al. (2014), who categorized methodological approaches as qualitative, quantitative, mixed and other, reported a contrasting pattern with this study among the proposals (n=265) submitted for funding to the MOOC Research Initiative (MRI), with preferred methodologies being mixed (36.2%), qualitative (27.9%), quantitative (30.2%) and other (5.7%).

On examination of Figure 6, it can be seen that there was an increase in MOOC research from 2013 onwards, yet more than half of the research is taken up by conceptual/descriptive studies, which is thought to be a handicap. Undoubtedly, some of these papers provide valuable insights, yet it is clear that there is an imbalance in the distribution of research methodologies among the selected articles (Table 2).

Table 2

Description of Research Methodologies and Models/Designs (n=362)

As highlighted by Veletsianos and Shepherdson (2016), dependence on particular research methodologies may restrict our understanding of MOOCs. Raffaghelli et al. (2015) claim that it is necessary to define the main constructs needed to drive empirical research; however, the sparsity of empirical research on MOOCs indicates an inflated debate, poorly underpinned by empirical evidence. In any scientific endeavour, researchers should interpret and draw conclusions from empirical data and a synthesis of previous findings (or both), so as to contribute to the analysis and development of their field.

These research findings trigger new issues regarding research in general and MOOC research in particular. It is salient that the MOOC phenomenon has been studied not only by means of quantitative, qualitative, or mixed paradigms, but other promising methodological paradigms such as data mining and analytics and practice-based methodology have also been used to better understand MOOCs. It is clear that research methodology in the field is evolving and new methodologies are emerging, mostly due to advances in technology and needs stemming from the transformational effect of MOOCs on our viewpoints. There is an increasing tendency to use data mining and analytics also in other research fields to better understand the practices that are mostly online. Innovative techniques such as learning analytics, social network analysis, and data mining are influencing the emerging methodological paradigms. These techniques have been used as a methodology, research model, and data collection and analysis tool on their own. We also see that practice-based methodologies such as action research and design-based research create a research realm of their own.

Another issue that requires attention, in terms of MOOC research, is similar to what has been found regarding research in the field of distance education (Bozkurt et al., 2015a). A great deal of MOOC articles is weak or superficial in reporting methodology and/or findings, which undermines the validity and reliability of their empirical research findings. In addition, abstracts and keywords, which are supposed to be an overall indication of the content of an article, do not necessarily reflect the key features of the selected studies, such as research aims, methodology, and conclusions.

Theoretical Frameworks

The theoretical or conceptual frameworks used in articles about MOOCs were used as a lens to interpret findings, and were counted and ranked according to their frequency (Table 3). It was seen that 64 articles (17.7%) out of 362 used employed theoretical/conceptual frameworks, while 298 (82.3%) articles didn't benefit from any theoretical or conceptual frameworks, or both. Those 64 articles used 43 different frameworks, and some used more than one framework in a single study.

On examination of Table 3, it can be seen that theoretical or conceptual frameworks used in MOOC research relate mostly to the educational area, including technology, innovation, social learning, and independent learning.

Table 3

Description of Research Theoretical/Conceptual Frameworks (n=362)

Note : One study may employ more than one theoretical or conceptual framework.

According to Perraton (1988), research without a theoretical basis tends to be nothing more than data gathering, in many cases. Similarly, Bozkurt et al. (2015b) state that ignoring the use of theoretical or conceptual frameworks encourages researchers to report descriptive findings only. Anderson (2008) notes that theories allow researchers to make connections with other studies, provide deeper understanding of concepts, and facilitate the transformation of knowledge from one context to another. Referring to Immanuel Kant's saying "theory without practice is empty; practice without theory is blind,'' Morrison and van der Werf (2012, p.399) state that "there is a symbiosis between theory and practice, and, for educational research, they cannot flourish without each other, even though they may have difficulty in living both with and without each other." In spite of the importance of using theoretical or conceptual lenses when appropriate, this analysis shows that MOOC research suffers from a lack of application of such frameworks. Considering the fact that MOOCs rely on many already known, well-established practices, such as open and distance learning, open educational resources, and online learning, the application of appropriate theoretical frameworks would fortify the pillars of research on this evolving phenomenon.

Conclusion and Implications for Future Research

The results of this study reveal and describe research trends and patterns in MOOC research over the time period from 2008 to 2015. As can be seen in Figure 1, research on MOOCs started to expand from 2013 onwards, and the pattern indicates a positive trend in the coming years. The analysis regarding research areas in MOOCs has demonstrated that MOOC studies encompass three main research areas: theories and models (27%) (macro level), learner characteristics (15.7%), and instructional design (11%) (all at the micro level) (see Figure 3).

In terms of the types of MOOC that appear in the research, most of the selected articles deal with xMOOCs (32.1%), cMOOCs (7.5%), hybrid/dual-layer MOOCs (1.7%) or c/xMOOCs (14.7). However, many of the articles (44%) did not explain or clarify the type of MOOC that they were investigating. The discourse in the selected MOOC articles tends to be positive (27.1%), neutral (56.4%) or negative (1.1%), with a significant number of articles exhibiting a critical stance (15.5%).

The citation analysis reveals that the 74 most cited articles on MOOCs (n=362) constitute 14.7% of the citation corpus (n=11,520). Findings related to research methodology, model, and design demonstrate that articles employing a theoretical, conceptual, or descriptive methodology (53.3%) outnumber other research paradigms such as quantitative (19.6%), qualitative (14.9%), data mining and analytics (6.1%), mixed (5.5%), and practice-based methodologies (0.6%). With regard to the use of theoretical or conceptual frameworks, 298 out of 362 articles (82.3%) reported research findings without benefiting from such frameworks.

Findings of this research revealed that the least explored research areas are learner support services; management and organization; access, equity, and ethics. In this regard, the following implications can be considered for future research directions.

First implication is based on learner support services research area from meso level. High dropout and low completion rates seem to be one of the major concerns of MOOC providers. Though this issue occupies most of the negative discussions on MOOCs, the studies concerning this issue are insufficient. Research on learner support services in MOOCs can contribute to the literature by providing effective and efficient solutions, which in turn, can be one of the solutions for the above-mentioned concerns. Second implication is based on management and organization research area from meso level. Supposing that MOOC participants are self-directed, self-regulated learners does not mean that these processes are free from management and organization processes. Considering that recent online learning initiatives are based on networked practices, it is for sure that management and organization processes differ from traditional approaches. This untapped research area can lead researchers to an interesting research avenue. It is also important to generate knowledge for management and organization issues from the perspective of MOOCs to better understand the phenomenon.

Final implication is based on access, equity, and ethics research area from macro level. There is a need for clear policies to regulate ethical considerations in research, both for MOOCs and for other emerging networked practices. Although most of the data on MOOCs can be gathered from open environments and does not require Institutional Review Board (IRB) approval, there are still ethical issues that are beyond informed consent form and need to be explained explicitly to provide clear guidelines to future researchers.

In addition to above future research directions, considering the findings of this research study and the impressions gained from the studies reviewed, the following three additional recommendations are presented for future research.

The first recommendation is about open policy for MOOC data. MOOC platforms should be more transparent and should open up MOOC data to researchers (within ethical bounds) in order to develop the MOOCs as a lifelong learning model and surpass the limitations imposed by the current majority of descriptive studies.

The second recommendation is about cultural issues. The "M" letter in MOOC abbreviation refers to being massive and diverse in cultural, socioeconomic, demographic, and many other dimensions. As mentioned in MOOC literature, learners in MOOCs participate from diverse points of the globe; however, there is a tendency to participate from North America and Europe. Similarly, most of the MOOC research originates from these geographies. In contrast to high participation mostly from North America and Europe, participation from some continents shifts from moderate to low. Based on the idea that geography, language and culture are interrelated, MOOCs in particular and other networked learning spaces in general, future research can focus on geographical, linguistic, and cultural differences and participation.

The final recommendation is about future follow-up research on MOOCs. This study focused on trends and patterns in MOOC research by compiling empirical articles that meet the inclusion criteria published in journals. However, as seen in Figure 1, articles on MOOCs are sparse between 2008 and 2012. It is thought that articles published in journals require more refined and longer processes, which intend to present sound and solid research findings while retarding the dissemination of the knowledge quickly especially in emerging fields such as MOOCs. Considering that the inception of MOOCs dates back to 2008 and the interest arouses around 2012, a complementary follow-up research that focuses on proceedings would be helpful to know more about early days of MOOC phenomenon and would provide a deeper understanding and a comprehensive perspective.

In conclusion, these research findings raise another question: Should HE perceive MOOCs as a threat, a disruptive innovation, or a change agent for sustainable innovation that might serve as a catalyst for the transformation of higher education? This again provides an impetus for further investigation and debate.

References

Adams, S. (2013). Are MOOCs really a failure? Forbes. Retrieved from http://www.forbes.com/sites/susanadams/2013/12/11/are-moocs-really-a-failure/

Altman, D. G. (1991). Practical statistics for medical research. London: Chapman & Hall.

Anderson, T. (2008). Towards a theory of online learning. In T. Anderson (Ed.), The theory and practice of online learning, (pp. 45-74). Edmonton, Canada: Athabasca University Press.

Auletta, K. (2012, April 30). Get Rich U. The New Yorker. Retrieved from http://www.newyorker.com/magazine/2012/04/30/get-rich-u

Banks, S. P., Louie, E., & Einerson, M. (2000). Constructing personal identities in holiday letters. Journal of Personal and Social Relationships, 17(3), 299-327.

Barber, M., Donnelly, K., Rizvi, S., & Summers, L. (2013). An avalanche is coming. Higher Education and the revolution ahead. Institute for Public Policy Research Retrieved from http://www.ippr.org/publications/an-avalanche-is-coming-higher-education-and-the-revolution-ahead

Berelson, B. (1952). Content analysis in communication research. Glencoe, Ill: Free Press.

Billington, P. J., & Fronmueller, M. P. (2013). MOOCs and the future of higher education. Journal of Higher Education Theory and Practice, 13(3), 36-42.

Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40.

Bozkurt, A. (2016). Identifying interaction patterns and teacher-learner roles in connectivist massive open online courses. Doctoral dissertation. Anadolu University, Turkey.

Bozkurt, A., Akgun-Ozbek, E., Onrat-Yilmazer, S., Erdogdu, E., Ucar, H., Guler, E., … Aydin, C. H. (2015a). Trends in distance education research: A content analysis of journals 2009-2013. International Review of Research in Open and Distributed Learning, 16(1), 330-363.

Bozkurt, A., Genc-Kumtepe, E., Kumtepe, A. T., Erdem-Aydin, I., Bozkaya, M., & Aydin, C. H. (2015b). Research trends in Turkish distance education: A content analysis of dissertations, 1986-2014. The European Journal of Open, Distance and E-Learning (EURODL), 18(2), 1-22.

Bozkurt, A., Özdamar Keskin, N., & de Waard, I. (2016). Research trends in massive open online course (MOOC) theses and dissertations: Surfing the tsunami wave. Open Praxis, 8(3), 203-221.

Brooks, D. (2012, May 4). The campus tsunami. The New York Times, A29. Retrieved from http://www.nytimes.com/2012/05/04/opinion/brooks-the-campus-tsunami.html

Bulfin, S., Pangrazio, L., & Selwyn, N. (2014). Making MOOCs: The construction of a new digital higher education within news media discourse. The International Review of Research in Open and Distributed Learning, 15(5), 291-305.

Chen, Y. (2014). Investigating MOOCs through blog mining. The International Review of Research in Open and Distributed Learning, 15(2), 85-106.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.

Cormier, D. (2008). The CCK08 MOOC-Connectivism course, 1/4 way. Dave's Educational Blog. Retrieved from http://davecormier.com/edblog/2008/10/02/the-cck08-mooc-connectivism-course-14-way/

Daniel, J. (2012). Making sense of MOOCs: Musings in a maze of myth, paradox and possibility. Journal of interactive Media in education, 2012(3), Art-18.

Deimann, M. (2015). The dark side of the MOOC: A critical inquiry on their claims and realities. Current Issues in Emerging eLearning, 2(1), Article 3.

Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: A review of nascent MOOC scholarship. Learning, Media and Technology, 39(3), 328-345.

Flynn, J. (2013). MOOCS: Disruptive innovation and the future of higher education. Christian Education Journal, 10(1), 149-162.

Friedman, T. L. (2012, May 16). Come the revolution. The New York Times. Retrieved from http://www.nytimes.com/2012/05/16/opinion/friedman-come-the-revolution.html

Gašević, D., Kovanovi, V., Joksimovi, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. The International Review of Research in Open and Distributed Learning, 15(5), 134-176.

Gough, D., Oliver, S., & Thomas, J. (2012). An introduction to systematic reviews. London, England: Sage.

Juran, J. M. (1975). The non-Pareto principle; mea culpa. Quality Progress, 8(5), 8-9.

Kennedy, J. (2014). Characteristics of massive open online courses (MOOCs): A research review, 2009-2012. Journal of Interactive Online Learning, 13(1), 1-16.

Kovanović, V., Joksimović, S., Gašević, D., Siemens, G., & Hatala, M. (2015). What public media reveals about MOOCs: A systematic analysis of news reports. British Journal of Educational Technology, 46(3), 510-527.

Krause, S. D., & Lowe, C. (Eds.). (2014). Invasion of the MOOCs: The promises and perils of massive open online courses. Anderson, South Carolina: Parlor Press. Retrieved from http://www.parlorpress.com/invasion_of_the_moocs

Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park: Sage Publications.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.

Liyanagunawardena, T., Adams, A., & Williams, S. (2013). MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distributed Learning, 14(3), 202-227.

McKenna, L. (2012, May 11). The big idea that can revolutionize higher education: 'MOOC'. The Atlantic. Retrieved from http://www.theatlantic.com/business/archive/2012/05/the-big-idea-that-can-revolutionize-higher-education-mooc/256926/

Meisenhelder, S. (2013). MOOC mania. Thought and Action: The NEA Journal of Higher Education, 29, 7-26.

Morrison, K. & van der Werf, G. (2012). Editorial. Educational Research and Evaluation, 18(5), 399-401.

Perraton, H. (1988). A theory for distance education. In D. Sewart, D. Keegan, & B. Holmberg (Eds.), Distance education: International perspectives, (pp. 34-45). New York: Routledge.

Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. Oxford, The UK: Blackwell Publishing.

Raffaghelli, J., Cucchiara, S., & Persico, D. (2015), Methodological approaches in MOOC research: Retracing the myth of Proteus, British Journal of Educational Technologies, 46(3), 488-509.

Sa'don, N. F., Alias, R. A., & Ohshima, N. (2014). Nascent research trends in MOOCs in higher educational institutions: A systematic literature review. In Web and Open Access to Learning (ICWOAL), 2014 International Conference (pp. 1-4). IEEE.

Sangrà, A., González-Sanmamed, M., & Anderson, T. (2015). Meta-analysis of the research about MOOCs during 2013-2014. Educación XX1, 1-28.

Shen, C. W. & Kuo, C. J. (2015). Learning in massive open online courses: Evidence from social media mining. Computers in Human Behavior, 51, 568-577.

Skiba, D. J. (2012). Disruption in higher education: Massively open online courses (MOOCs). Nursing education perspectives, 33(6), 416-417.

Van Dijk, T. A. (1993). Principles of critical discourse analysis. Discourse and Society, 4(2), 249-283.

Veletsianos, G. & Shepherdson, P. (2015). Who studies MOOCs? Interdisciplinarity in MOOC research and its changes over time. The International Review of Research in Open and Distributed Learning, 16(3), 1-17.

Veletsianos, G. & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013-2015. The International Review of Research in Open and Distributed Learning, 17(2), 198-221.

Zawacki-Richter, O. (2009). Research areas in distance education: A Delphi study. International Review of Research in Open and Distributed Learning, 10(3), 1-17.

Appendix A

Research Areas of Distance Education (Zawacki-Richter, 2009)

Appendix B

Reference List of Most Cited Works

Trends and Patterns in Massive Open Online Courses: Review and Content Analysis of Research on MOOCs (2008-2015) by Aras Bozkurt, Ela Akgün-Özbek, and Olaf Zawacki-Richter is licensed under a Creative Commons Attribution 4.0 International License.