International Review of Research in Open and Distributed Learning

Volume 18, Number 6

September - 2017

The Pedagogic Architecture of MOOC: A Research Project on Educational Courses in Spanish

Elia Fernández-Díaz, Carlos Rodríguez-Hoyos, and Adelina Calvo Salvador
University of Cantabria

Abstract

This study has been carried out within the context of the ECO European Project (E-learning, Communication Open-Data: Massive Mobile, Ubiquitous, and Open Learning) which is being financed by the European Union over four years (2014-17). It analyses the pedagogic architecture of MOOC on pedagogic/educational subjects in Spanish over one academic year (September 2015-June 2016). The analysis focuses on five major dimensions from a qualitative perspective: subjects and the promoting institution, methodology, resources, type of videos, and evaluation. The results demonstrate the hegemony of subjects linked to the Society of Knowledge, such as the widespread use of traditional methodology based on video lessons (the "talking head" model). Communication tools are clearly underused and evaluation based on the use of questionnaires is dominant. The findings show the need for further research into MOOC based on a pedagogic approach such as the one adopted here.

Keywords: MOOC, higher education, qualitative research, open education

Introduction

This study has arisen from the E-learning, Communication Open-Data: Massive Mobile, Ubiquitous, and Open Learning project (ECO project) in which partners and universities from different European countries are participating. Its objectives are the analysis of pedagogic MOOC (massive open online courses) models, the review of studies and the documentation of practices aimed at overcoming the digital divide and finding more inclusive technological and educational strategies, the promotion of open training, and the creation of professional communities.

One area of work given priority by the European Commission is the promotion of open on-line training in order to democratize knowledge by making it more accessible and free. Spain is one of the countries which, together with the United Kingdom, has developed MOOCs more quickly (Oliver, Hernández-Leo, Daza, Martín, & Albó, 2014). The European map on these types of courses offered by Open Education (the European Commission) shows that out of the 1254 courses recorded on its database in April 2015, 348 were from Spanish Educational Institutions, followed by 307 in the United Kingdom, 170 in France, and 145 in Germany.

This article focuses on analysing MOOCs offered on pedagogic topics in Spanish. Our main objective is to review this training from an educational perspective in order to gain an in-depth understanding of the reality of these massive open courses given that research in this field usually focuses on the technological and economic dimension of these phenomena without in-depth analysis on their pedagogic aspects (Ossiannilsson, Altinay, & Altinay, 2015; Swan, Day, Bogle, & Van Prooyen, 2015). Not only is it necessary to be aware of the pedagogic, technological, and financial opportunities offered by these courses, but rather it is also important to know that not all practices developed obey the connectivist, democratizing, and collaborative theoretical foundations upon which they were conceived (Chiappe-Laverde, Hine, & Martínez-Silva, 2015; Méndez García, 2013; Ozturk, 2015). Thus we can overcome a "romantic vision of MOOCs" (Cabero, 2015, p.4) and reach a more critical and reflective analysis which does not only present the advantages of this way of learning, but also its weaknesses, limitations, and opportunities for improvements in the future.

Previous studies have identified some important educational issues requiring further research in order to improve training opportunities. The most notable are studies aimed at designing and promoting Good Practices (Bali, 2014; Spyropoulou, Pierrakeas, & Kameas, 2014) and those whose main objective is to assess the quality of courses through the use of various tools and rating scales (Aguaded & Medina-Salguero, 2015; Alemán, Sancho-Vinuesa, & Gómez, 2015; Baldomero, Salmerón, & López, 2015; Chapman, Goodman, Jawitz, & Deacon, 2016; Lowenthal & Hodges, 2015). On the other hand, some studies have raised the need to demand the democratization of this training offer through the application of the Open Educational Resources (OER) philosophy, given that not all platforms allow the review, re-use, redistribution, or remixing of contents, known as the 4 (5) Rs by Wiley (2007). Furthermore, strategies have been proposed for opening course content (Atenas, 2015; Chiappe-Laverde et al., 2015).

A number of studies focused on how to improve student assessment have emerged in the field of evaluation and accreditation. They include descriptions of the tools and evaluation models which are most used (quizzes, P2P) and propose new tools which, until now, have been used less frequently such as portfolios, rubrics, or group evaluation. Furthermore, they discuss the problems and tendencies which are emerging in the accreditation of courses (Chauhan, 2014; O´Toole, 2013; Sánchez-Vera & Prendes-Espinosa, 2015).

Finally, there are some studies focused on discussing and combatting participant dropout rates. They propose strategies such as improving levels of knowledge and management of the diversity of participants in the courses (De Waard et al., 2011; Kop, 2011; Walji, Deacon, Small, & Czerniewicz, 2016), achieving maximum involvement of all participants (García, Tenorio, & Ramírez, 2015) or questioning the system of rewards and accreditation established in MOOCs (Sánchez & Escribano, 2014).

Aims

The main objective of this study is to understand and critically analyse the pedagogic architecture of MOOCs offered in Spanish on different platforms under the category "education," "pedagogy," or similar. The specific objectives are:

• Understanding MOOC training platforms on education offered in Spanish and identifying the hegemonic topics within MOOCs on education/pedagogy.
• Analysing the curricular approach of these courses assessing their methodology, resources, and evaluation; demonstrating the strengths and weaknesses of this pedagogic architecture.

Methodology: Data Collection and Analysis

In this study we have used a qualitative approach to carry out an analysis of virtual training processes (Bonk, Lee, Reeves, & Reinolds, 2015; Gutiérrez Martín, Torrego, & Dornaleteche, 2015; Méndez García, 2013; Margaryan, Bianco, & Littlejohn, 2015; Park, Jung, & Reeves, 2015; Stewart, 2013).

This work is part of the ECO project in which 14 universities from all over Europe are participating (Spain, France, Portugal, Italy, etc.). Within this framework, the team from the University of Cantabria has developed a study focused on analysing the platforms offered by MOOCs, and within these, the courses on educational subjects which are offered in Spanish. The ultimate objective of this study is to carry out a review in educational code of this training in order to reach a more in-depth understanding of the reality of these courses. Thus, we will be contributing to the achievement of one of the ECO project objectives: collaborating from a local perspective to a global agenda. This study is led by three experts in virtual training, with a wide range of experience in teaching, and research into the incorporation of technologies in the processes of curricular development.

Our study was divided into four phases summarized in the following figure:

Figure 1. Research phases.

In the first phase, a comprehensive review of literature was carried out which enabled us to establish a "state of the question" on the main lines of research and practical work in the field of MOOC. In addition, studies in Spanish based on qualitative methodology, and not limited to a single platform or course, were identified as background to the research. With regard to previous research in this area, the study carried out by Torres (2013) which focused on analysing the main strengths and weaknesses of the UNED-COMA and UNX platform is particularly important. In addition, Ruiz Martín (2013) carried out a study on the complete offer of courses on the Coursera, Edx, MiríadaX, and Udacity platforms in which the platform, the name of the course, the subjects and areas of knowledge, the institution which supports the proposal, the country where the course is generated, and the vehicle language used can be identified.

We also found a study of the MiriadaX carried out by Medina-Salguero and Aguaded (2014), which exemplifies the elements that make up a MOOC. Later on, Gutiérrez Martín, Torrego, and Dornaleteche (2015) analysed the formal characteristics of MOOCs in Spain (with a sample of 109 courses) based on three categories (informative, aesthetic, and credibility) with the aim of analysing their impact on the actions of users, in the so called Zero Moment of Truth (ZMOT). Lastly, we would also like to highlight the analysis carried out by Gallego, Gámiz, and Gutiérrez (2015) of evaluation trends which were developed on different platforms using a sample of 87 courses.

In the second phase, educational platforms offering MOOCs on educational subjects in Spanish during the 2015/2016 academic year were selected, taking into account variables such as accessibility (open course or with accessible information), and the search category (education or similar, although courses not categorized as such but whose subject area was genuinely educational were included). Based on these criteria, we were able to select a total of 36 courses from five different platforms:

Table 1

Courses Analysed

In the third phase a system of categories and codes was devised through an inductive-deductive process in order to analyse the curricula of MOOCs. This enabled us to systematize the analysis of experiences in line with some of the studies reviewed in the first phase (Swan et al., 2015; Yuan & Powell, 2013, among others). This methodological procedure was defined by the focus of the study aimed at understanding which topics were being addressed in the courses offered, as well as their pedagogic or curricular architecture. The following table summarizes the categories and codes:

Table 2

Research Categories and Codes

The first category summarizes the basic course data such as title, subject, promoting institution, objectives, and hosting platform. The second category called Methodology refers to the definition of methodological strategies used in the courses. Three codes were used to define the types of strategies identified: 1) traditional, for those strategies aimed at reproducing the contents; 2) dialogic for those involving debate and exchange and 3) applied for those requiring decisions on different devices (pedagogic designs, projects, etc.) based on the contents of work done. The category called Resources involves the materials used in each training process, differentiating between videos, teaching guides, social networks, background reading, computer graphics/graphs, presentations, forums, Wikis, Hangouts, and others. The category Types of Videos refers to the characteristics identified in the audiovisual productions used in the courses. The codes used were: 1) video lessons which referred to videos created ad hoc by the teaching team for explaining the contents; 2) process which referred to videos created during the development of the course in order to clarify doubts, develop activities etc.; and 3) external, which referred to those videos which were not made by the teaching team but were used during the course. In terms of Evaluation, the assessment strategies used were reviewed: questionnaires, peer to peer activities, or others.

Finally, the courses offered between September 2015 and June 2016 were analysed in the fourth phase. For this purpose the research team enrolled in the courses, mentioned in Table 1, in order to carry out content analysis of the categories explained above. In addition, to find out the percentage of courses in which the codes appear, a descriptive statistical analysis (Bearman et al., 2012) was been carried out so that the dominant tendencies in the pedagogic design of these types of courses could be identified. The analysis of each course has also been discussed and compared in order to perform a process of triangulation between experts (Johnson & Turner, 2003).

Results

In this article we are focusing on the pedagogic analysis of MOOCs, so that we can present the results of the categories directly related to the design of the teacher-learner process in these training actions: subjects, methodology, resources, types of videos, and evaluation.

Subjects

The courses analysed show a field of study which is continuing to boom as a result of changes in education supposedly brought about by the Society of Knowledge and Information. Within this context, we can find 22 courses out of the 36 which were analysed (61.08%) that address diverse subjects ranging from some focused on tools or specific technologies (such as Moodle, mobile communication and learning, robots in digital education, or videogames) to other more general ones (such as the teaching and assessment of digital skills, working in learning networks, innovation through technologies, or the use of open educational resources, among others).

In the remaining subjects we can find issues such as the importance of creativity and ethics for teachers and researchers, leadership, sexuality, the prevention of violence in classrooms, metacognitive strategies, or museums in learning.

Methodology

With regard to methodology, all of them used traditional methodological strategies (100%), followed by applied strategies (61.10%) and lastly dialogic strategies (55.56%). Therefore, it can be seen that all the courses analysed promoted cognitive strategies which repeat contents.

Although these experiences are seeking to implement different types of methodological strategies in order to respond to student diversity, the reproduction of knowledge is predominant. An example of this type of strategy can be identified in the course Instructional Leadership, Global Perspective and Local Practices in which students are asked to respond to a series of items in each of the course modules following the viewing of different video lessons. This tendency can be interpreted from two perspectives. It could be related to the fact that we are faced with a very recent form of training, something which means that some teaching practices developed in classroom teaching are usually reproduced in one form or another. Likewise, it could also be related to the fact that a high number of people enrol in these types of courses, which is why strategies are designed that allow automatic data management in order to achieve an easy way of controlling of student performance.

Applied methodological strategies that promote more creative work focused on the analysis and/or resolution of cases or on project design can also be found. Thus, in the MOOC Robots in Digital Education: A Way of Teaching How to Think, students are given the task of designing their own robot in a contextualised and reasoned way.

Finally, dialogic methodology means that more than half of the courses proposed using different communication tools, such as forums and SNS (social network systems), even though the use of these resources varies greatly, as will be analysed in detail in the following section.

Resources and Types of Videos

With regard to the resources used, videos are present in all of the cases (100% of the courses), the next most used educational resource are forums (82.38%) followed by teaching guides and background reading (both 69.45%). This is followed by the use of social networks (44.43%), presentations (25.02%), and lastly computer graphics/graphs (16.64%), Hangouts (13.85%), and Wikis (6.01%).

Firstly, all courses used videos, although differences can be seen between video lessons, external videos and process videos. "Talking head" video lessons usually featuring the same person are predominant. In this type of video, the image of the teacher is projected onto a full screen; sometimes this is combined with other images related to the content. The quality of video lessons also varied considerably, from the use of simple slides converted into videos to other polimedia formats which even include the dynamism of VideoMakerFx. Furthermore, some of the video lessons attempted to incorporate different levels of interactivity, integrating multiple choice questions on the contents which the participant is required to answer in order to continue viewing, to check whether the student has maintained an adequate level of attention during the viewing (Towards a Constructivist Practice in the Classroom) or in other cases the inclusion of videos in which experts are interviewed (Learning to Learn).

In addition to video lessons, there are also external videos are hosted on other platforms, with YouTube being the most used. These videos are used to develop activities or provide further information. In this regard, videos are used for a range of different purposes on the same course (Educational Innovation and Professional Development: Opportunities and Limitations of ICT) such as exploring students’ previous ideas to demonstrating experiences shared by teachers. In some of the courses analysed we also found process videos, that is, videos that are created ad hoc in the process of tutoring the course in which the teachers and facilitators organised a session where a group of participants are given the floor in order to explain work done and share ratings on the development of the course. Hangout (a Google tool) is one of the most used formats. One of the most notable examples was seen in the course called Expanded Education with New Media, in which a hangout combined with other courses from the same platform takes place with the purpose of inviting the participants to interact and exchange experiences on pedagogic subjects. It should be noted that although the working language used by the courses selected was Spanish, the opportunity of watching the videos in another language or accessing the contents by using subtitles in various languages was available on some courses.

Secondly, the use of forums is quite varied. Although in some courses, forums promote dialogic action in order to create a learning community, in others it is reduced to being a place where participants ask specific questions on certain technical issues related to using the platform, obtaining certificates or activities for development. That is, real discussion or exchanging ideas are not core elements. Consequently, the level of participation in the forums differs a great deal between courses. Also, in other examples, groups are set up on social networks or on platforms outside the course, for example, in the MOOCs on the Educalab platform.

Thirdly, it is important to note the different ways the teaching guides were used in the MOOCs that were analysed. In this regard, on some platforms (like MiríadaX), the courses have a teaching guide, while on other platforms (like ECO), there is a specific guide for each module. From a curricular perspective, the guides are structured based on the elements that are traditionally used for designing training experiences: objectives, contents, etc. Similarly, with regard to the background reading, we have observed a variety of materials and uses, for example: 1) synthesis of contents developed ad hoc during the course (Education and Museums); 2) existing publications on subjects developed in each block (Strategies in the Management of On-Line Communities. The Community Manager); 3) links to other materials hosted on-line (Quality Education for all. Equality, Inclusion and Attention to Diversity, amongst others) and to platforms created previously by the teaching team, versions 1.0 of the course (Body Posture: Treatment in the School Environment). In another of the courses, which uses ad hoc materials and background reading, these resources are used to facilitate different levels of study into the contents addressed (sMOOC Step by Step course).

Fourthly, social networks are used frequently, with Facebook and Twitter being the most used, although others like Pinterest or Instagram are also used. The use of this type of resource in the training strategies analysed shows that, as in other classroom based training, the teaching teams that design and develop these experiences are trying to convert social networks into another educational resource, given their high levels of use among citizens.

However, it should be noted that the educational use of these types of devices differs substantially from one course to another, performing various roles in the development of these training processes. In courses, like in the MOOC Digital Literacy for People at Risk of Social Exclusion: Strategies for Socio-Educational Intervention, a social network like Facebook is used for developing the interaction produced in it. Thus, the platform used to host this process (ECO) serves as a repository in which some materials, such as videos, guides, or P2P activities, are available to students, while dialogic activities unfold in threads of discussion created in a closed group for students who form part of the course. The creation of closed groups in the courses developed on the platform has also been identified (Strategies for Managing Online Communities: the Community Manager). In other examples, although these groups are created they are only used to disseminate information on progress, provide information, make the course visible, etc., but they are not specifically used for recreating knowledge in a community (Quality Education for All. Equality, Inclusion and Attention to Diversity).

Strategies aimed at extending what happens on the platform to the outside world have been identified, with the objective of facilitating the creation of communities of participants interested in the same subject that can survive once they have completed their training process, as well as increasing the number of participants. In the course entitled Educational Innovation and Professional Development: Opportunities and Limitations of ICT or Open Educational Resources, social networks are used to develop a series of activities that complement those that unfold in the forums or in other communication areas on the platforms. In this regard, they are used as a means of encouraging others interested in the subjects to enroll in the courses later on.

Finally, the analysis of these training experiences has enabled us to discover some practical uses of existing resources in the courses which are contradictory. Although tools that facilitate collaborative work, such as Wikis, are available to students in some of the courses, in reality they are used to develop individual training activities instead of facilitating collaborative writing, and therefore serve to limit the possibility of exchange between participants (as in the case of Methodological Strategies for e-Learning Teaching). Also, the use of resources that are not used during the development process of the courses, or which remain inactive, can be seen, as in the course Construction of a Virtual Course on the Moodle Platform, in which the section "Documentation" remains inactive or in the MOOC Applied Educational Innovation, where there is an unused blog.

Evaluation

Evaluation is carried out mainly through questionnaires, although specific examples of courses that do not use questionnaires have been found: two on the ECO platform and one on Educalab. Evaluation questionnaires usually appear in each module, plus a final one with multiple choice or true/false questions. It is highly significant that on many of the platforms analysed sets of questions are not used which allow modification of the questions asked in the event that students do not respond to them adequately. Most of the questionnaires analysed on the different platforms reproduce the same questions if the correct score required to pass this test is not obtained, offering very poor feedback (for example through the use of icons that show whether they have responded correctly or incorrectly to any of the questions asked).

The second evaluation tool is the P2P. Some tasks to be developed for peer review could be: designing an innovative activity from a digital learning perspective, reviewing ethical aspects related to specific research cases, using a rubric to evaluate a digital resource, or designing an educational experience with ICT to promote critical digital literacy for people at risk of social exclusion.

Although the construction of an ad hoc rubric is used for each activity in those courses which use peer to peer evaluation processes, the characteristics of these kinds of tools vary greatly from one experience to another. In the course Construction of a Course on the Moodle Platform a rubric model is used in which only one criteria of evaluation is defined (for example: "the resource provides information related to module 1"). However, different degrees of fulfilment of this criteria are not established, but rather the students who correct the practice have to award a score and make comments which justify this rating. The sMOOC Step by Step course also uses rubrics, although a varying number of criteria and dimensions valued on a scale of 1 to 5 and which need to be qualitatively justified are defined on this course. The use of questionnaires to evaluate P2P should also be noted.

Finally, sporadic cases of the use of the portfolio as an evaluation tool have been found as in the case of courses analysed hosted on the Educalab (How to Teach and Evaluate Digital Competence?; The Meaning of Initiative and Enterprise in the Classroom; Expanded Education With New Media). In these examples, the potential of these tools is not fully exploited, given that evidence of the process is selected while reflection is at the discretion of the participant without adequate feedback or follow-up procedures being established by the facilitators.

A summary of the main findings are presented in the following table:

Table 3

Results of the Pedagogic Analysis: Findings and Limitations

Conclusions

In the following section we will highlight some tendencies detected in MOOCs which are being developed in Spanish and which address pedagogic/educational subjects.

Although in all of the courses traditional methodology predominates, it paves the way for more active methodologies, aimed at the creation of learning communities that strengthen learning feedback between the individual and the network. It is important to seek remixing and learning through the organisation of tasks that require more applied work and the organisation of authentic exchanges of ideas and experiences through the different communication tools available on the platforms that host the courses and also those available externally. As some studies have suggested, it is essential that beyond the possibility of connecting to a greater number of people we seek and design strategies aimed at promoting interaction between participants in these types of experiences (Mackness, Mak, & Williams, 2011).

With regard to the dialogic action generated in the MOOCs analysed, it should be noted that in some of the courses the use of forums (internal or external to the platform) is still very low, both in the volume of interactions and their quality. Although in some courses they are used for resolving doubts relating to technological issues, contents or study activities along the lines of e-learning (Buil, Hernández, Sesé, & Urquizu, 2012), in other MOOCs they are used as a genuine tool for exchanging knowledge and experiences. Nevertheless, the connections the teaching team and facilitators establish with students, as well as the strategies used to connect students to each other (Kop, Fournier, & Mak, 2011), are more relevant than the technological devices used. In this regard, taking into account the high number of students enrolled (Park et al., 2015) it is important to strengthen other types of strategies that favour the joint reconstruction of knowledge, fostering the creation of independent groups or peer assessment (Kellogg, Booth, & Oliver, 2014) given the difficulty of sustaining the presence of the teaching team and providing individualized monitoring. We have also identified the use of SNS as platforms that seek to foster communication between students (Lim & Ismail, 2010; McCarthy, 2013), although the functionality and variety found is notorious. Either way, the analysis of these resources highlights the need to rethink the potential of collaboration, exchange and group work. In order to strengthen connectivism, it is necessary to prioritise the creation of learning communities which facilitate the reconstruction of knowledge between participants facilitating the connection between these according to their interests and needs (Siemens, 2005; Downes 2012).

This research underlines the role of videos in the design of the MOOCs analysed, demonstrating in detail the existing typology in this area. There is a wide range of technical diversity and creativity in the video lessons, although the model called talking head and/or the sole protagonist, frequently used in virtual educational contexts (Schmid, 2004), still prevails. Although efforts have been made to make lessons more interactive, it is important to review their design in order to reduce dropout rates. Among the key factors identified for improving the development of these resources are: avoiding abrupt visual transitions; reducing their duration; providing links or other resources which students need to interact with; and summarizing the contents or facilitating access to tutorials through a single click (Kim et al., 2014). At the same time, videos created ad hoc during the training process as well as external videos with very different purposes are also used. This shows the flexibility of these resources, some of which were not even recorded for educational purposes.

In the field of evaluation, our study demonstrates the need to incorporate new tools. The clear preponderance of questionnaires in the courses analysed, linked to the hegemony of traditional methodology, means that the feedback received by the participant on their own learning is often very poor. Therefore, the heterogeneity of the rubrics used and the differences found in the feedback received by the participants on work carried out justifies the need to continue researching peer evaluation processes. Some strategies aimed at offering students multiple critical perspectives on their activities must be incorporated, above all in those activities which are more open or interpretive (O´Toole, 2013). Apart from the use of portfolios in the courses hosted on the Educalab platform, we have not found different evaluation tools on any of the other platforms analysed. Therefore, it appears that the variety of evaluation tools needs to be increased, as well as the level of feedback received by users with the aim of improving the collaborative dimension in all curricular elements (Gallego, Gámiz, & Gutiérrez, 2015).

We believe that the pedagogic architecture of a MOOC requires further research given that practices appear to suggest, that in reality, these strategies tend to reproduce the contents and that many of the communicative strengths that technological tools offer are clearly underused. Thus, we would be able to move towards the creation of more open, flexible, and emerging curricular models distancing ourselves from the reproduction of technical strategies commonly found in e-learning, rethinking the role of facilitators, promoting a social learning environment, supporting networks, and learning between peers (Walji et al., 2016).

Finally, we consider it appropriate to highlight the need to continue making progress so that this development and curricular innovation with regard to MOOCs takes place within a framework of inclusion and democratization. Despite the democratizing promises of MOOCs, the proliferation of courses has not resulted in inequalities in the field of education being overcome. Together with the opening strategies described earlier, reducing the digital and training divide requires strengthening personal autonomy and the ability to self-regulate and self-manage learning processes (Valverde Berrocoso, 2014). We have highlighted the benefits of designing courses based on a conception of differences and appreciate the value of technological tools for expanding the learning community that each course seeks to create. However, further research in this field must be given priority so that education becomes more accessible for a greater number of people (Caballo, Caride, Gradaille, & Pose, 2014; Chauhan, 2014; Jagannathan, 2015; Medina-Salguero & Aguaded, 2014; Rohs & Ganz, 2015).

Acknowledgments

This work was supported by the ECO project (E-learning, Communication Open-Data: Massive Mobile, Ubiquitous, and Open Learning) promoted by the European Union within the Competitiveness and Innovation Framework Program (CIP).

References

Aguaded, I., & Medina-Salguero, R. (2015). Quality criteria for the valuation & management of MOOC. RIED. Revista Iberoamericana de Educación a Distancia, 18(2), 119-143. doi: http://dx.doi.org/10.5944/ried.18.2.13579.

Alemán, L., Sancho-Vinuesa, T., & Gómez, M. G. (2015). Indicators of pedagogical quality for the design of a Massive Open Online Course for teacher training. RUSC. Universities and Knowledge Society Journal, 12(1), 104-119. doi: http://dx.doi.org/10.7238/rusc.v12i1.2260

Atenas, J. (2015). Model for democratisation of the contents hosted in MOOCs. RUSC. Universities and Knowledge Society Journal, 12(1), 3-14. doi: http://dx.doi.org/10.7238

Baldomero, M., Salmerón, J. L., & López, E. (2015). Comparative between quality assessment tools for MOOCs: ADECUR vs Standard UNE 66181: 2012. RUSC. Universties and Knowledge Society Journal, 12(1), 131-145. doi: http://dx.doi.org/10.7238/rusc.v12i1.2258

Bali, M. (2014). MOOC pedagogy: Gleaning good practice from existing MOOCs. MERLOT. Journal of Online Learning and Teaching, 10(1), 44-56. Retrieved from http://jolt.merlot.org/vol10no1/bali_0314.pdf

Bearman, M., Smith, C., Carbone, A., Slade, S., Baik, C., Hughes-Warrnington, M., & Neumann, D. (2012). Systematic methodology in higher education. Higher Education Research & Development, 31(5), 625-640. doi: http://doi.org/10.1080/07294360.2012.702735

Bonk, C. J., Lee, M., Reeves, T. C., & Reynolds, T. (Eds.). (2015). MOOCs and open education around the world. New York, NY: Routledge.

Buil, I., Hernández, B., Sesé, F. J., & Urquizu, P. (2012). Discussion forums and their benefits for e-learning: implications for effective use. Revista Innovar. Revista de Ciencias Administrativas y Sociales, 22(43), 131-143. Retrieved from http://www.fce.unal.edu.com/media/files/documentos/documentos/Innovar/v22n45.pdf

Caballo, M.B., Caride, J.A., Gradaílle, R., & Pose, H.M. (2014). Massive open on line courses (MOOCs) like university spread. Profesorado. Revista de Currículum y Formación Del Profesorado, 18(1), 43-61. Retrieved from http://www.ugr.es/local/recfpro/rev181ART

Cabero, J. (2015). Educational visions of MOOC. RIED. Revista Iberoamericana de Educación a Distancia, 18(2), 39-60. doi: http://dx.doi.org/10.5944/ried.18.2.13718

Chapman, S.A., Goodman, S., Jawitz, J., & Deacon, A. (2016). A strategy for monitoring and evaluating massive open online courses. Evaluation and Program Planning, 57, 55-63. doi: http://doi.org/10.1016/j.evalprogplan.2016.04.006

Chauhan, A. (2014). Massive Open Online Courses (MOOCS): Emerging trends in assessment and accreditation. Digital Education Review, 25, 7-17. Retrieved from: http://files.eric.ed.gov/fulltext/EJ1039698.pdf

Chiappe-Laverde, A., Hine, N., & Martínez-Silva, N. A. (2015). Literature and practice: a critical review of MOOC. Comunicar. Media Education Research Journal, 44(XXII), 9-18. doi: http://dx.doi.org/10.3916/C44-2015-01

De Waard, I., Albajian, S., Sean, M., Hogue, R., Keskin, N., Koutropolous, A., & Rodríguez, O. (2011). Using mLearning and MOOCs to Understand Chaos, Emergence, and Complexity in Education. The International Review of Research in Open and Distance Learning, 12(7). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1046/2026

Downes, S. (2012). Connectivism and connective knowledge: Essays on meaning and learning networks [Blog post]. Stephen Downes Web. Retrieved from http://www.downes.ca/post/58207

Gallego, M. J., Gámiz, V., & Gutiérrez, E. (2015). Trends in assessment in massive open online courses. Educación XX1, 18(2), 77-96. doi: http://dx.doi.org/10.5944/educxx1.12935

García, B. J., Tenorio, G. C., & Ramírez, M. S. (2015). Self-motivation challenges for student involvement in the Open Educational Movement with MOOC. RUSC. Universities and Knowledge Society Journal, 12(1), 91-104. doi: http://dx.doi.org/10.7238/rusc.v12i1.2185

Gutiérrez Martín, A., Torrego, A., & Dornaleteche, J. (2015). MOOC offer in Spain. Zero moment of truth (ZMOT). RELATEC. Revista Latinoamericana de tecnología Educativa, 14(2), 11-23.doi: http://dx.doi.org/10.17398/1695-288x.14.2.11

Jagannathan, S. (2015). Harnessing the power of open learning to share global prosperity and eradicate poverty. In C. J. Bonk, M. M. Lee, T. C. Reeves, & T. H. Reynolds (Eds.). MOOCs and open education around the world (pp. 219-231). New York: Routledge.

Johnson, R. B., & Turner, L. A. (2003). Data collection strategies in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.). Handbook of mixed methods in social and behavioral research, 297-319. Thousand Oaks, CA: Sage.

Kellogg, S., Booth, S., & Oliver, K. (2014). A social network perspective on peer supported learning in MOOCs for educators. The International Review of Research in Open and Distributed Learning, 15, 263-289.

Kim, J., Guo, P. J., Seaton, D. T., Mitros, P., Gajos, K. Z., & Miller, R. C. (2014). Understanding in-video dropouts and interaction peaks in online lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference, 31-40. Retrieved from http://www.eecs.harvard.edu/~kgajos/papers/2014/kim14video.pdf

Kop, R. (2011). The challenges to connectivist learning on open online networks: Learning experiences during a massive open online course. The International Review of Research in Open and Distance Learning, 12(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/882

Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distributed Learning, 12(7), 74-93. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1041/2042

Lim, T., & Ismail, J. T. (2010). The use of Facebook for online discussions among distance learners. Turkish Online Journal of Distance Education, 11(4), 72-81. Retrieved from http://tojde.anadolu.edu.tr/makale_goster.php?id=621

Lowenthal, P.R., & Hodges, C.B. (2015). In search of Quality: Using quality matters to analyze the quality of massive, open, online courses. International Review of Research in Open and Distributed Learning, 16(5) 83-101. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/2348

Mackness, J., Mak, S., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In L. Dirckinck-Holmfeld, V. Hodgson, C. Jones, M. De Laat, D. McConnell, & T. Ryberg (Eds.), Proceedings of the 7th International Conference on Networked Learning 2010 (pp. 266-275). University of Lancaster, Lancaster. Retrieved from http://eprints.port.ac.uk/5605/

Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83. doi: http://dx.doi.org/10.1016/j.compedu.2014.08.005

McCarthy, J. (2013). Learning in Facebook: First year tertiary student reflections from 2008 to 2011. Australasian Journal of Educational Technology, 29(3), 337-356. Retrieved from http://ajet.org.au/index.php/AJET/article/view/373

Medina-Salguero, R., & Aguaded, I. (2014). The MOOC in the educational platform MiriadaX. Profesorado. Revista de currículum y formación del profesorado, 18(1), 137-153. Retrieved from http://www.ugr.es/local/recfpro/rev181ART8.pdf

Méndez García, C. M. (2013). Design and implementation of Massive Online Open cursos (MOOCs): Expectations and practical recommendations. RED. Magazine about Distance Education, 39, 1-19. Retrieved from: http://www.um.es/ead/red/39/mendez.pdf

Ossiannilsson, E., Altinay, F., & Altinay, Z. (2015). Analysis of MOOCs practices from the perspective of learner experiences and quality culture. Educational Media International, 52(4), 272-283, doi: http://doi.org/10.1080/09523987.2015.1125985

O'Toole, R. (2013). Pedagogical strategies and technologies for peer assessment in Massively Open Online Courses (MOOCs). Discussion Paper. University of Warwick, Coventry, UK: University of Warwick. (Unpublished). Retrieved from http://wrap.warwich.ac.uk/

Oliver, M., Hernández-Leo, D., Daza, V., Martín, C., & Albó, L. (2014). MOOCs in Spain. An overview of massive open online courses in Spanish universities. Barcelona: Telefónica-UPF Chair. Retrieved from http://www.catedratelefonica.upf/España1.pdf

Ozturk, H. T. (2015). Examining value change in MOOCs in the scope of connectivism and open educational resources movement. International Review of Research in Open and Distributed Learning, 16(5), 1-25. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2027/3438

Park, Y., Jung, I., & Reeves, T. (2015) Learning from MOOCs: A qualitative case study from the learners’ perspectives, Educational Media International, 52(2), 72-87, doi: http://doi.org/10.1080/09523987.2015.1053286

Rohs, M. & Ganz, M. (2015). MOOCs and the claim of education for all: A disillusion by empirical data. International Review of Research in Open and Distributed Learning, 16(6). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2033

Ruiz Martín, P. (2013). Presente y futuro de los Massive Open Online Courses (MOOC). Análisis de la oferta completa de cursos de las plataformas COURSERA, EdX, MIRÍADA X y UDACITY [Present and future of Massive Open Online Courses. Analysis of all courses offered on COURSERA, EdX, MIRÍADA X & UDACITY] (Doctoral dissertation). Complutense University of Madrid. Retrieved from http://eprints.ucm.es/23502/2/MOOCs.pdf

Sánchez-Vera, M. M., & Prendes-Espinosa, M. P. (2015). Beyond objective testing and peer assessment: alternative ways of assessment in MOOCs. RUSC. Universities and Knowledge Society Journal, 12(1), 119-131. doi: http://dx.doi.org/10.7238/rusc.v12il.2262

Sánchez, E., & Escribano, J. J. (2014). Assessment tools classification for MOOC. EDUTEC. Revista Electrónica de Tecnología Educativa, 48, 20-21. Retrieved from http://edutec.rediris.es/Revelec2/Revelec48/pdf/Edutec-e_n48_Sanchez-Escribano.pdf

Schmid, A. M. (2004). Tutorships: The faces of online education. Educação e Contemporaneidade, 13(22), 275-285. Retrieved from http://nead.uesc.br/arquivos/biologia/scorm/educacao_e_contemporaneidade_-faeeba.pdf#page=35

Siemens, G. (2005). Connectivism: A learning theory for the digital age. Journal of Instructional Technology and Distance Learning, 1(2). Retrieved from http://www.ingedewaard.net/papers/connectivism/2005_siemens_ALearningTheoryForTheDigitalAge.pdf

Spyropoulou, N., Pierrakeas, C., & Kameas, A. (2014). Creating MOOC guidelines based on best practices. EDULEARN14 Proceedings, 6981-6990. Retrieved from http://eeyem.eap.gr/sites/default/files/EDULEARN14_paper%20draft.pdf

Stewart, B. (2013). Massiveness+openness=new literacies of participation? MERLOT. Journal of OnlineLearning and Teaching, 9, 228-238. Retrieved from http:/jolt.merlot.org

Swan, K., Day, S., Bogle, L., & Van Prooyen, T. (2015). A tool for characterizing the pedagogical approaches of MOOCs. In C. J. Bonk, M. M. Lee, T. C. Reeves, & T. H. Reynolds (Eds.), MOOCs and open education around the world (pp. 105-118). New York: Routledge.

Torres, D. (2013). Considerations and early results on MOOCs in Latin America: UNEDCOMA & UNX. Campus virtuales, 1(2), 64-73. Retrieved from http://www.uajournals.com/campusvirtuales/journal/25

Valverde Berrocoso, J. (2014). MOOCs: A critical view from the education sciences. Profesorado. Revista de Currículum y Formación Del Profesorado, 18(1), 93-111. Retrieved from http://www.ugr.es/local/recfpro/rev181ART6.pdf

Walji, S., Deacon, A., Small, J., & Czerniewicz, L. (2016): Learning through engagement: MOOCs as an emergent form of provision, Distance Education, 37(2), 208-223. doi: http://doi.org/101080/01587919.2016.1184400

Wiley, D. (2007). On the sustainability of open educational resource initiatives in Higher Education. OECD’s Centre for Educational Research and Innovation (CERI). OECD Publishing. Retrieved from http://www.oecd.org/dataoecd/33/9/38645447.pdf

Yuan, L., & Powell, S. (2013). MOOCs and open education: Implications for Higher Education. Retrieved from http://publications.cetis.org.uk/wp-content/uploads/2013/03 MOOCs-and-Open-Education.pdf

The Pedagogic Architecture of MOOC: A Research Project on Educational Courses in Spanish by Elia Fernández-Díaz, Carlos Rodríguez-Hoyos, and Adelina Calvo Salvador is licensed under a Creative Commons Attribution 4.0 International License.