Volume 16, Number 6
>
Marco Kalz*, Karel Kreijns*, Jaap Walhout*, Jonatan Castaño-Munoz+, Anna Espasa^, Edmundo Tovar#
* Open University of the Netherlands
+ Institute for Prospective Technological Studies
^ Open University of Catalonia
# Technical University of Madrid
While MOOCS have emerged as a new form of open online education around the world, research is still lagging behind to come up with a sound theoretical basis that can cover the impact of socio-economic background variables, ICT competences, prior experiences and lifelong learning profile, variance in intentions, environmental influences, outcome expectations, learning experience, and economic return on taking and completing Massive Open Online Courses (MOOCs). The potential diversity of participants of MOOCs has been taken as a starting point to develop a theoretical model and survey instrument with the goal to establish a large-scale, cross-provider data collection of participants of (European) MOOCs. This article provides an overview of the theoretical model, the start-phase of the project, and reflects on first experiences with the cross-provider data collection.
Open educational formats have received more attention recently due to the hype around Massive Open Online Courses (MOOCs). Institutions all around the world are joining initiatives to provide external participants cost-free and barrier-free access to their online-courses. Open Universities around the world already have a tradition in supporting open learning practices for people who are unable to follow traditional formats of educational programs provided by higher educational institutions. We have earlier discussed the frictions and gaps in the discussion about MOOCs and have reframed it with regard to relations to the open educational resource initiatives, experiences with open educational practices and the potential benefit of learning network services for MOOCs (Kalz & Specht, 2013). In a meta-review Liyanagunawardena, Adams, and Williams (2013) summarize existing research on MOOCs up to 2012. The authors state that most studies to date have focused on case studies, the influence of MOOCs in higher education structure, or educational theory framing. Although MOOCs generated a plethora of data, the learner perspective is still underrepresented in current research. Fischer (2014) argues that, currently, we are still in an early development stage of MOOCs and he states that “both the hype and the underestimation [of MOOCs] are more based on assumption and beliefs than theoretical groundings and qualitative and quantitative data” (p.150). The situation has partially improved recently with several studies on MOOC participants using learning analytics or survey methodology (MOOCs@Edinburgh Group, 2013; Anderson, Huttenlocher, Kleinber, & Leskovec, 2014; Christensen, Steinmetz, Alcorn, Bennet, Woods, & Emanuel, 2014) and partially even open datasets mainly focused on U.S courses ( Ho, Reich, Nesterko, Seaton, Mullaney, Waldo, & Chuang, 2014; Jordan, 2014).
Recently, more empirical research has been conducted related to the learner perspective in MOOCs. This empirical research has treated topics like background of students and behaviour related to performance (DeBoer, Stump, Seaton, & Breslow, 2013), intentions of MOOC learners (Reich, 2014, Campbell, J., Gibbs, Najafi, & Severinski, 2014), learning paths across several MOOCs (Perna, Ruby, Boruch, Wang, Scull, Ahmad, & Evans, 2014) and last but not least professional learning in MOOCs (Milligan, & Littlejohn, 2014). These isolated initiatives and studies, however, did not provide the joint research instrument that is needed to collect MOOC participants’ data across Europe systematically. This large-scale data collection would inform both policy-making on a European level, and also higher education provider’s strategic decisions with respect to open online education.
In this article we introduce the theoretical model and survey instrument of the MOOCKnowledge project. This project is an initiative of the European Commission’s Institute of Prospective Technological Studies (IPTS) with the intention to build a database that can provide insights into the profile, experiences and behavior of participants of (European) open online courses. As a European project, the diversity of participants and participant profiles is an important focus aspect for the project that can only be addressed by a cross-provider data-collection. This approach will also enable the researchers to analyse the impact of open education for specific sub-populations, being it minorities, specific professions (e.g. teachers) or learners with specific cultural or socio-economic backgrounds. We first introduce in this contribution the European policy background of the project. Then we introduce the theoretical background and the research focus of the project. We briefly introduce the survey instrument and provide a summary of experiences with the cross-provider data-collection. Last but not least, we provide an outlook into future research and additional services for MOOC-providers and researchers to be developed in the future.
The European Commission's overall Europe 2020 strategy highlights the importance of smart, sustainable, and inclusive growth to remain competitive and to overcome the current economic crisis (European Commission, 2010). The modernization of the European Education and Training system, with a special focus on early school leavers and increasing tertiary education attainment, is one of the most important means to reach this goal. Linked to the broad objectives of this strategy the European Commission (2012) argues in the "Rethinking Education" Communication that efforts must be made to boost the full uptake of ICT, enhancing both the acquisition of digital competences and the modernisation of education to generate growth, employment, and social inclusion. Moreover, this communication emphasized the need to stimulate open and flexible learning in order to provide the skills needed in the 21st century society.
In 2013, the idea of the use of ICT and open and flexible learning in the European educational systems was further developed in the Communication “Opening up Education: Innovative teaching and learning for all through new technologies and Open Educational Resources” (European Commission, 2013). This Communication sets out an “agenda for stimulating high quality and innovative ways of learning and teaching through new technologies and digital content” (European Commission, 2013). The Communication responded specifically to the recent developments in the area of Open Education, such as the abundance of OER and MOOCs, and highlighted the need to develop economies of scale and remove barriers to access, use, and sharing of knowledge across borders for education. Although it focuses on Open Education, the Communication took a systemic approach and made recommendations ranging from general topics such as ICT access and skills to specific Open Education topics such as standards, or the use and integration of OER and MOOCs in formal and non-formal education.
In a direct or an indirect way, MOOCs have been signaled by policymakers as a potential solution to some of the challenges detected in European educational systems. The reasons are diverse. First, MOOCs could potentially expand the access to tertiary education offering free and online access to anyone with internet access, including to those who were early school leavers. Second, MOOCs could boost the modernisation of education by promoting innovative and efficient uses of online education, and by challenging the traditional roles of teachers. Third, MOOCs could be used as a means to overcome borders by the educational offer of the universities and to foster the interinstitutional collaborations in its development, use, and recognition (EADTU, 2014). The European Credit Transfer and Accumulation system (ECTS) (European Union, 2015) is a valuable and distinctive tool which positions Europe in a privileged situation at this regard, enabling the transfer and recognition of credits from one organization to the other across borders. Moreover, a cross-border MOOC offer guarantees the provision of education following one of the key European values: diversity (understood as diversity of culture, languages, methods, levels, topics, and profile of learners). Finally, MOOCs could have a positive impact on the skills of the individuals and could be useful to foster education and lifelong learning in areas where a lack of skills is detected ultimately contributing the generation of growth, employment, and social inclusion in the European societies.
In a recent study Tempelaar, Rienties, Giesbers and van der Loeff (2013) have analyzed the impact of cultural differences on learning dispositions. The authors argue that in Europe five of ten cultural world clusters are located and that all universities in Europe with an open inflow of students face these cultural differences. In their study, the authors show how the learning dispositions and learning approaches are influenced by cultural backgrounds of students.
On the other hand, MOOCs are not free of criticisms. It is often argued that MOOCs are not reaching disadvantaged individuals, and therefore these courses could even be contributing to increase educational inequalities. Additionally, the prominence of low quality instructional designs and high level drop-out rates have been also signaled as problems (Margaryan, Bianco & Littlejohn, 2015). Finally, some critics stress that the participation in the MOOC movement by the Higher Education institutions could be only a marketing strategy in the global arena, but without clear social returns.
While the potential benefits and risks of the MOOCs have been identified by literature and policy makers, there is scarce empirical evidence that allows to assess if they have actually been realised or avoided (Fischer, 2014). Since most studies have been focused in U.S context, the lack of evidence is specially an issue in the EU context. Therefore, better local knowledge is needed if Europe wants to implement effective evidence-based educational policies. Reich (2015) has recently criticized that research to date had little impact on educational practice. According to him, research needs to change focus and procedures on three levels: from studies of engagement to research about learning, from investigations of individual courses to comparisons across contexts, and from a reliance on post-hoc analyses to greater use of multidisciplinary, experimental design. To overcome the current trend of in-depth single-course or single-provider analysis is especially important in Europe due to the cultural, socio-economic, and motivational diversity of participants (Kalz & Specht, 2013) .
The MOOCKnowledge project is embedded in the three policy initiatives described above and aims to set up a large scale data collection on participants in European MOOCs. The data gathered by the project will allow to assess some of the above mentioned potentials and shortages of the MOOCS better on the basis of cross-national and paneuropean evidences. By doing this, the research will take into account diversity of courses and learner profiles in the EU (including minority learners), and will contribute to inform better the EU and Member States policies in Education. The research model behind the project is introduced next.
The MOOCKnowledge project addresses directly the underrepresentation of the learners in current MOOC research and aims to establish a large-scale cross-provider data collection on European MOOC participants. The study tender published by the European Commission’s Institute for Prospective Technological Studies (IPTS) defined the background to the research. The study contract then further elaborates this basic definition project, which is expected to deliver the following types of data on MOOCs participants:
On the basis of these components we have developed a research model for the project. In addition, two associated PhD projects extend the focus with respect to assessment and feedback practices in MOOCs and language learning.
For this purpose we have used two existing research frameworks. On the one hand, these frameworks can guide the construction of the survey instruments based on items validated earlier, and on the other hand, they also allow a systematic analysis of the data at a later stage (Kreijns, Vermeulen, Kirschner, van Buuren, & Van Acker, 2013; Kreijns, Vermeulen, Van Acker, & van Buuren, 2014). These two frameworks are the reasoned action approach elaborated by Fishbein and Ajzen (2010) and self-determination theory set out by Ryan and Deci (2000). These frameworks offer a basis for the prediction of human social behavior and consist of background factors (e.g., socio-economic status) that affect different variables and directly influence the behavioural intention to take and complete a MOOC.
By taking a MOOC we mean that a person enrolls in a MOOC to get access to all the course materials and teachers or support services. However, the completion of the MOOC is dependent on individual objectives. We define completion in this research model as the achievement of earlier set personal objectives, which may not be the same as completion in the sense of completing all learning activities, tests, and finally receiving a certificate. For instance, a MOOC taker (i.e., participant) may only be interested in one particular part of a MOOC about statistics, namely the part that explains confirmatory factor analyses. From the point of view of the MOOC taker, the MOOC is completed if that part is finished whereas the MOOC giver may see this as a drop-out. In this sense we define drop-out in the project as the gap between earlier set personal objectives and attainment of these objectives during the course. Grau-Valldosera and Minguillón (2014) have earlier discussed the complexity of the multidimensional phenomenon of drop-out and they have developed a possible definition of drop-out for the distance/online education context. Nonetheless, for the open education context, different approaches are needed. We believe that our approach to the phenomenon of “drop-out” in open education leads to more expressive and also more valid results.
Figure 1: Research model of the MOOCKnowledge project
Background variables are usually referred to as distal variables whereas the different variables that directly influence intention are referred to as proximal variables. Distal variables in our model exist on an individual level, a social level, and a task level. These are for example demographic data, the socio-economic status of the participants, their lifelong learning profile, previous experiences with open online courses and IT competences. The reasoned action approach identifies attitude, perceived norm, and perceived behaviour control (i.e., self-efficacy) as proximal variables as they directly predict behavioral intention. Attitude is the person’s favourable or unfavourable position as regards taking a MOOC. Attitude is generally formed by the outcome beliefs of a person and his/her evaluation of these beliefs. For example, a person may belief that taking a MOOC will give her/him more opportunities in the labour market and evaluate this as very important for that person. Consequently, the person may have a positive attitude towards taking a MOOC.
Perceived norm is the experienced social pressure to take a MOOC. This perceived norm is formed by a person’s important others, who may be colleagues, family members, supervisors, etc. For example, when a colleague advises a person to take a MOOC, that person may not be motivated to comply with this advice. However, when the boss gives the same advise, the same person may feel obliged to comply. Finally, perceived behaviour control defines whether or not a person is able to take a MOOC. Does the person have time to take a MOOC? Does she/he possess all the necessary equipment and software to follow a MOOC? All these questions will give indications to the perceived behavior control. All distal variables exert their influence on a person’s intention to take a MOOC through these three proximal variables. In other words, these proximal variables are mediating the influence of the distal variables on intention. Another element in the reasoned action approach is the intention behaviour gap. Not all intentions will result in actual behavior. There may be many reasons: for example, the MOOC may demand some prior-knowledge of the topic concerned which a person may not have. This is what Fishbein and Ajzen (2010) call actual behaviour control. Actual behaviour control moderates the relationship between intention and behaviour. Actual knowledge and skills are also moderate this intention-behavior relationship. Actual knowledge and skills refer to all the knowledge and skills that are needed to realise the intention. For example, a participant must know which codec has to be downloaded to view a footage that is part of the MOOC.
The more insight a person has about the actual behaviour control the more precise the perceived behavioral control will be. Consequently, the person’s intention to perform the behavior will be more accurately formed. This, in turn, means that the predictive power of behavioral intention will increase. At the same time, the moderating influence of both perceived and actual behavior control in the relationship between behavioral intention and actual behaviour will be positively altered (see the dotted arrows in Figure 1).
It is important to notice that until now, nothing has been said about the MOOC itself such as how it is organized, what it prerequisites are, if a certificate is obtained after completion, etc. These are aspects that the questionnaire will also address. Consistently with the theoretical model the project has two questionnaires planned during the course: a pre-course questionnaire which will assess the proximal variables and intention, whereas a post-course questionnaire which will also assess actual behavior. The reasoned action approach was earlier applied in many different domains like health, economics and the authors have applied this approach earlier in an educational context, namely to investigate motivations and intentions of teachers to use open educational resources (OER)/digital learning materials (DLMs) (Kreijns, Van Acker, Vermeulen & Van Buuren, 2013).
Self-determination theory (SDT) differentiates between intrinsic and extrinsic motivation and assumes that people have three basic needs namely: competence, relatedness, and autonomy. Autonomy is the feeling that people have when they are originators of their own actions (in harmony with the concept of the integrated self). Competence is the feeling people have when they are effective, and that there are sufficient opportunities to demonstrate efficacy. Relatedness is the feeling people have when they are connected and valued by others and that they experience a sense of belonging. If the social environment does not satisfy these basic psychological needs, then negative consequences will follow with respect to activity and development. However, the degree to which each of these needs are promoted and satisfied determine the types of motivations and the consequent regulations that are developed. These types may be categorized as self-determined versus controlled (i.e., non- self-determined) or as autonomous motivation versus controlled motivation. The Perceived Locus of Causality (PLOC) (Ryan & Connel, 1989) represent all the SDT motivational types ordered in from the most undesirable form of non-self-determined behavior (amotivation) to the most desired form of self-determined behavior (intrinsic motivation). Figure 2 depicts these types of motivations.
Figure 2: The different types of motivation according to SDT
In short, the different types of motivations are as follows: Autonomous motivation encompasses the preferred types of motivation, namely intrinsic, integrated, and identified motivation, the latter two are types of extrinsic motivation as well. Controlled motivation encompasses the two other types of extrinsic motivation, namely introjected and external motivation. Autonomous motivation is associated with self-determined regulation whereas controlled motivation is associated with coercive regulation (thus non-self-determined regulation). Intrinsic motivation is concerned with the active engagement with tasks solely because one finds them interesting and will give fun. Integrated motivation is the type of extrinsic motivation that is the closest to intrinsic motivation; it deals with doing the tasks because the activity is in compliance with the self. Identified motivation is the type of motivation in which humans identify the activity as important and useful and, thus, are willing to perform the activity. Introjected motivation represents a regulation that is driven by the feelings of guilt and shame as well as by factors such as self-esteem and ego-involvement. External motivation is the strongest type of extrinsic motivation in that the regulation of the activity is a function of expectations regarding reward and punishment. Finally, amotivation, also associated with non-self-determined regulation, is the type of motivation in which one performs the activity but intention is lacking.
From the perspective of the MOOCKnowledge project, it is important to have a clear picture about motivation because it makes a considerable difference if a participant is motivated intrinsically or externally. At the moment, very little is known about the motivational disposition of MOOC participants, how motivation is related to their behaviour, and the impact on their study success or career development.
To be able to analyse the intention-behavior gap more thoroughly we have integrated work by Gollwitzer about implementation intentions (Gollwitzer & Oettingen, 2013). Implementation intentions are concrete plans for how a specific goal can be achieved. This aspect is important, because the huge gap between numbers of participants that sign up for a MOOC and the numbers who actually start learning activities can point to a MOOC’s low perceived value or lack of implementation intentions among potential participants. These three theoretical foundations are combined with other aspects like usability aspect or interaction experiences in MOOCs taken.
In the long-term perspective, the MOOCKnowledge research model also aims to fill another identified research gap in the MOOC literature. Currently there are some studies on the economics and social returns of higher education and adult learning, but very little on online and open learning (Carnoy, Jarillo Rabling, Castaño-Muñoz, Duart, & Sancho-Vinuesa, 2013). Therefore, despite the increasing importance of the MOOC offer, little is known about the effects of these courses on formal study success and career development. This is a long term component of the MOOCKnowledge research model and the information will be gathered with a follow-up questionnaire that will be sent to the learners approximately one year after they have completed the post-questionnaire. Although the majority of MOOC learners have a high socioeconomic status and educational backgrounds (Hansen & Reich, 2015), it is hoped that the large scale of the MOOCs project will also obtain information on the effects of these courses on minority groups (such as unemployed people or learners without a degree). The project will also explore variations in the economics and educational returns of these courses depending on topic, the country, the certification, etc.
The survey-instrument of the MOOCKnowledge project was developed in several iterations. The pre-questionnaire consists of five chapters each with a special focus. The chapters of the pre-questionnaire are:
The post-questionnaire also consists of five chapters containing again several blocks of items with particular focus. The chapters of the post-questionnaire are:
After the post-questionnaire, participants will be recruited for voluntary participation in a follow-up-questionnaire sent out to participants one-year after finalization of the MOOC. In all chapters, reliability and validity is increased by using existing validated instruments, by adapting validated instruments, or by developing new instruments following the guidelines of Fishbein and Ajzen (2010).
In an initial analysis we have analyzed data from three different MOOCs. Table 1 provides an overview about the high-level differences between learner profiles participating in these MOOCs. MOOC A is related with professional training, MOOC B with entrepreneurship and MOOC C with personal development. Although some common trends were detected, the data showed how different MOOCs were linked to differences among the profile of the learners (Table 1).
Table 1
Summary of MOOC Takers Characteristics (n=1244)
|
MOOC A |
MOOC B |
MOOC C |
N |
173 |
316 |
715 |
Sex |
Male 56 Female 117 |
Male 194 Female 122 |
Male 248 Female 467 |
Mean age (SD) |
43.18 (9.09) |
39.36 (12.28) |
38.04 (12.49) |
Employment status |
employed 141 self-employed 15 looking for work 4 homemaker 1 student 5 unable to work 2 other 5 |
employed 105 self-employed 80 looking for work 65 not looking for work 4 homemaker 2 student 46 retired 2 other 12 |
employed 289 self-employed 57 looking for work 157 not looking for work 12 homemaker 7 student 143 retired 15 unable to work 9 other 26 |
Employer support |
no 16 yes 69 does not know 80 NA 8 |
no 20 yes 77 does not know 111 NA 108 |
no 42 yes 100 does not know 271 NA 302 |
Additionally, the data showed some unexpected particularities such as older learners than in US studies, high level of unemployed people participation in MOOCs (especially in MOOC B and C), or a high percentage of learners who have taken one or more MOOCs previously, pointing to the direction that the learners pick and choose from the MOOC offering in order to build personal training paths. The relative low importance of certification for appreciation of employees’ activities in open online education pointed to a possible increased awareness and appreciation of non-formal learning for professional development by employers.
The current data set at hand had an overrepresentation of participants from Spain. One of the methodological challenges of the project will be to come up with an adequate sampling strategy to overcome these biases. At the moment, more providers are joining the project and allowing the data-collection with the survey instrument of the project. This will contribute to a more representative sample. Potential providers can join the project via a cooperation agreement. The consortium collects the data for the providers and shares the data with the providers. In the future, the whole data set will be published in an anonymized and aggregated form for other researchers to conduct analyses with it.
The first three MOOCs have also been used to improve the data-collection process and to test an adequate length of the questionnaire. With initially around 40 minutes on average and 25 minutes for the post-questionnaire and a response rate between 5% and 10% the authors will explore several options to shorten the survey instrument without losing explanatory power, validity, and reliability.
In this paper we have introduced the theoretical model of the MOOCKnowledge project. We see the approach chosen as a promising option to address the shortcomings of the current research about MOOCs. The project is open for any MOOC provider or provider of open education with a specific focus on the situation in Europe.
The next step for the project will be to broaden the diversity of providers and cultural contexts with the help of the members of the OpenupEd initiative (Roswell & Jansen, 2014) and other partners. The more diversity we can achieve on the provider side, the richer our insights will be about the MOOC phenomenon on a European and global level. This will also allow the project to inform the policy building by the European Commission. Future analysis will focus on participant profiles and the relation to achievement, the intention-behaviour gap, or relation between employer support and motivation of MOOC participants.
To provide also feedback to the MOOC providers, at later stages of the project, benchmarking options will be explored to compare results of a single MOOC against several other MOOCs and data categories. This will enable also a comparison of MOOC investment against initiatives of other institutions. In addition, different approaches will be tested how the anonymized and aggregated dataset can be explored by the general public, ideally in the form of a set of linked data (Piedra, Chicaiza, Lopez, & Tovar Caro, 2014). For this purpose the consortium will also explore different data visualization services to be used.
For these services to become effective, a sufficient amount of data needs to be collected to deliver a meaningful benchmarking analysis. Attracting a sufficient amount of MOOC providers will be one of the upcoming challenges for the consortium. Potential threats arising from the chosen approach are related to a potential selection bias and survival bias. While the selection bias could exist on the level of the representativeness of the participating MOOCs, the survival bias will be related to the ratio of non-successful participants that fill out both questionnaires. To be able to explain the huge gap between subscribed learners and active learners, it will be critical to also collect a representative amount of answers from learners who did not reach their learning goal. While these problems might be addressed simply by the scale of the data-collection, the consortium has foreseen statistical corrections but also incentive mechanisms to attract higher response rates from this special target group.
This work has been partially funded by a tender (JRC/SVQ/2013/J.3/0035/NC) of the European Commission’s Institute for Prospective Technological Studies (IPTS).
Anderson, A., Huttenlocher, D,. Kleinber, J., & Leskovec, J. (2014). Engaging with Massive Online Courses. In Proceedings of the 23rd International Conference on World Wide Web, p. 687–698.
DeBoer, J., Stump, G. S., Seaton, D., & Breslow, L. (2013). Diversity in MOOC students’ backgrounds and behaviors in relationship to performance in 6.002 x. In Proceedings of the Sixth Learning International Networks Consortium Conference.
Campbell, J., Gibbs, A. L., Najafi, H., & Severinski, C. (2014). A comparison of learner intent and behaviour in live and archived MOOCs. The International Review of Research in Open and Distributed Learning, 15(5)
Breslow, L., Pritchard, D.E., DeBoer, J., Stump, G.S., Ho, A.D., & Seaton, D.T. (2013). Studying Learning in the Worldwide Classroom Research into edX’s first MOOC. Research & Practice in Assessment, 9(1), 13-25.
Carnoy, M., Jarillo Rabling, B., Castaño-Muñoz, J., Duart, J.M., & Sancho-Vinuesa,T. (2013). Does On-line Distance Higher Education Pay Off for Adult Learners? The Case of the Open University of Catalonia. Higher Education Quarterly, 66 (3).
Christensen, G., Steinmetz, A., Alcorn, B., Bennet, A., Woods, D., & Emanuel, E. J. (2013). The MOOC Phenomenon: Who Takes Massive Open Online Courses and Why? (November 6, 2013). Available at http://dx.doi.org/10.2139/ssrn.2350964
European Commission (2010). Europe 2020: a strategy for smart, sustainable and inclusive growth. Brussels: European Commission.
European Commission (2012). Rethinking education: Investing in skills for better socio-economic outcomes. France: Strasbourg.
European Commission. (2013). Opening up education: Innovative teaching and learning for all through new technologies and Open Educational Resources. Brussels: European Commission.
European Union (2015). ECTS user guide. Brussels, Belgium. Retrieved from http://ec.europa.eu/education/library/publications/2015/ects-users-guide_en.pdf
Fischer, G. (2014). Beyond hype and underestimation: identifying research challenges for the future of MOOCs. Distance Education, 35(2), 149–158.
Fishbein, M., & Ajzen, I. (2010). Predicting and changing behaviour: The reasoned action approach. Psychology Press. New York: Psychology Press.
Gollwitzer, P. M., & Oettingen, G. (2013). Implementation intentions. In M. Gellman & J. R. Turner (Eds.), Encyclopedia of Behavioral Medicine (pp. 1043-1048). New York: Springer-Verlag.
Grau-Valldosera, J., & Minguillón, J. (2014). Rethinking dropout in online higher education: The case of the Universitat Oberta de Catalunya. The International Review of Research in Open and Distributed Learning, 15(1).
Hansen, J. D., & Reich, J. (2015, March). Socioeconomic status and MOOC enrollment: enriching demographic information with external datasets. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 59-63).
Ho, A., Reich, B., Nesterko, S., Seaton, D., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, Fall 2012-Summer 2013. Massachusetts: Cambridge.
Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. International Review of Research in Open and Distance Learning, 15(1), 133–160.
Kalz, M., & Specht, M. (2013). If MOOCS are the answer, did we ask the right questions? Implications for the design of large-scale online-courses (pp. 1–16). Maastricht. The Netherlands: Maastricht School of Management.
Kreijns, K., Vermeulen, M., Van Acker, F., & Van Buuren, H. (2014). Predicting teachers’ use of digital learning materials: combining self-determination theory and the integrative model of behaviour prediction. European Journal of Teacher Education, 37(4), 465–478.
Kreijns, K., Van Acker, F., Vermeulen, M., & van Buuren, H. (2013). What stimulates teachers to integrate ICT in their pedagogical practices? The use of digital learning materials in education. Computers in Human Behavior, 29(1), 217–225.
Kreijns, K., Vermeulen, M., Kirschner, P. A., van Buuren, H., & Van Acker, F. (2013). Adopting the integrative model of behavior prediction to explain teachers’ willingness to integrate ICT in their pedagogical practices: A perspective for research on teachers’ ICT usage in pedagogical practices. Technology, Pedagogy and Education, 22(1), 55–71.
Liyanagunawardena, T., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distance Learning, 14(3), 202–227.
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83.
Milligan, C., & Littlejohn, A. (2014). Supporting professional learning in a massive open online course. The International Review of Research in Open and Distributed Learning, 15(5). 197-213.
MOOCs@Edimburgh Group. (2013). Report #1. Retrieved from https://www.era.lib.ed.ac.uk/bitstream/1842/6683/1/Edinburgh_MOOCs_Report2013_no1.pdf
Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., & Evans, C. (2014). Moving through MOOCs understanding the progression of users in Massive Open Online Courses. Educational Researcher, 43(9). 421-432. doi :0013189X14562423.
Piedra, N., Chicaiza, J., Lopez, J., & Tovar Caro, E. (2014). Supporting openness of MOOCs contents through of an OER and OCW framework based on Linked Data technologies. In IEEE Global Engineering Education Conference (EDUCON) (pp. 1112–1117). doi:10.1109/EDUCON.2014.6826249
Reich, J. (2014). MOOC completion and retention in the context of student intent. EDUCAUSE Review Online. Retrieved from http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent
Reich, J. (2015). Rebooting MOOC Research. Science, 347(6217), 34-35.
Roswell, J., & Jansen, D. (2014). The OpenupEd quality label: Benchmarks for MOOCs. International Journal for Innovation and Quality in Learning. 2(3). 88-100.
Ryan, R., & Deci, E. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78.
Ryan, R. M., & Connell, J. P. (1989). Perceived locus of causality and internalization: Examining reasons for acting in two domains. Journal of Personality and Social Psychology, 57, 749–761.
Tempelaar, D. T., Rienties, B, Giesbers, B., & van der Loeff, S. (2013). Cultural differences in learning dispositions. In P. Van den Bossche et al. (eds.), Facilitating Learning in the 21st Century: Leading 3 through Technology, Diversity and Authenticity, 5, Netherlands: Springer. DOI 10.1007/978-94-007-6137-7_1
Setting-up a European Cross-Provider Data Collection on Open Online Courses by Marco Kalz, Karel Kreijns, Jaap Walhout, Jonatan Castaño-Munoz, Anna Espasa, Edmondo Tovar is licensed under a Creative Commons Attribution 4.0 International License.