International Review of Research in Open and Distributed Learning

Volume 18, Number 7

November - 2017

Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates

Inés Gil-Jaurena, Javier Callejo, and Yolanda Agudo
Universidad Nacional de Educación a Distancia (UNED), Spain

Abstract

The paper is a study about the MOOC experience at the Spanish National University of Distance Education (UNED), where we have collected initial and final information about learners' profiles and opinions, as well as enrolment, completion and certification rates. It is a survey-based study covering 17 MOOCs offered in UNED's own platform, and collects information from a sample of more than 24000 learners (initial survey) and 2003 learners (final survey). The paper first presents an overview of the MOOC experience at UNED, introducing the main features of these courses. Afterwards, it focuses on the methodology used in the study and in the information gathered in the second edition of UNED MOOCs, which took place from November 2013 until March 2014. Learners' average profile is a Spanish female, approximately 37 years old, with a University degree, and generally employed. For many of the participants, UNED MOOCs were their first experiences with these sorts of courses, and the main reasons for enrolment were the course topic and the perceived usefulness for professional development. The expectations regarding completion and certification where initially very high, but completion rates remain below 14%. In the final survey, the overall experience in the MOOCs and the different tools used in the courses were highly valued by learners, except the support figures (curator, facilitator, peers), which received lower ratings. These findings are of interest for the institution and further research, refining learning analytics, is encouraged.

Keywords: MOOC, open education, higher education, Spain

Introduction

The so-called Massive Open Online Courses (MOOCs) have become the most dominant emerging phenomenon in recent years in the field of higher education. Although the objectives have been different in different cases - institutional, promotional, advertising, etc., - the most prestigious universities in the world have created a growing and, above all, varied range of courses, both in terms of content and quality. Attentive to the development of this phenomenon, and assuming responsibility within the Spanish educational system for spearheading innovative initiatives, especially in the field of higher education distance learning, the Spanish National University of Distance Education (UNED, http://www.uned.es) has, from October 2012 onwards, launched a number of courses using this teaching modality and with significant success, in terms of the large response achieved and the number of people enrolled in the various courses that have been offered.

At the end of the second series of these courses (November 2013 to March 2014) and bearing in mind the commitment of the UNED to the quality of the courses it offers, we have made a first assessment of the experience, focusing specifically on: the analysis of the socio-demographic profile of the students of MOOCs offered by the UNED; the assessment of the MOOCs from the learners' perspective; and the analysis of the various activities of the students related to the courses (registration, completion, and certification rates).

Veletsianos and Shepherdson (2016) identify demographics of MOOC users as one of the themes that emerge across systematic analyses of the MOOC literature. This paper tries to fill one of the gaps in MOOCs literature, which is the geographic distribution - most studies about MOOCs proceed from North-American authors (Gašević, Joksimović, Kovanović, & Siemens, 2014) and refer to North-American or European (mainly UK) participants (Liyanagunawardena, Adams, & Williams, 2013), thus native English speakers. This is the first empirical analysis of one of the main MOOCs' offering university in Spain, i.e., UNED, through UNED MOOCs.

UNED MOOCs Overview

UNED's mission is the public service of higher education through the modality of distance education, characterised for utilising a specific didactical methodology by a conjoint use of printed, audiovisual, and technological media, along with face-to-face support from tutors in regional centres (more than 60 all over Spain) and communication means with professors (virtual fora, email, phone). It has the largest student population in Spain: more than 150000 registered students in Graduate programmes, almost 8000 in Master's programmes and more than 1000 in Doctorate studies, almost 13000 in the access to university course and more than 18000 in language courses in 2014-15 (UNED, 2016).

UNED has offered open educational content (especially in an audiovisual format) since its creation in 1972, joined the OpenCourseWare movement in mid-2000's and launched its first massive open online courses (MOOCs) in 2012 (Gil-Jaurena, 2014). The first institution offering MOOCs in Spain was UNED, in its pilot project launched in October 2012, UNED COMA (cursos online masivos abiertos - MOOCs), with 22 courses offered in the first edition and 25 - three of them new - in the second edition (November 2013-March 2014). The courses are in Spanish and remained open during approximately three months. Until 2015, UNED MOOCs were offered in its own platform (https://coma.uned.es/) using the openmooc software developed in Spain (https://github.com/OpenMOOC), which had the basics features for running a MOOC: infrastructure for hosting videos, online tests, peer-assessment, communication forums, and open digital badges. Given the difficulties for developing the software in order to incorporate new functionalities and keep it updated in a changing and evolving scenario, UNED analysed other alternative widespread and open source platforms; since 2015, UNED uses OpenEdx in its own server (https://iedra.uned.es/). The 17 courses on which this paper focuses were run in openmooc.

UNED MOOCs can be considered x-MOOCs, as they fall into the cognitive-behaviourist pedagogy and rely primarily on information transmission, computer marked assignments, and peer assessment (Rodriguez, 2013). Instructional design is based on providing content -mainly using videos- and proposing activities, mainly multiple choice question tests (computer marked) and other activities like peer-assessed essays, open questions, etc. According to Bates (2015), UNED MOOCs have the following design features supported by a specially designed platform software (openmooc in this case): video lectures, computer-marked assignments, peer assessment, supporting materials, a shared comment/discussion space, badges or certificates, and learning analytics. Regarding a remaining feature that this author identifies (Bates, 2015, p. 154), i.e., "no, or very light, discussion moderation," UNED MOOCs use different approaches to this aspect and two agents are in charge of supporting learners, as will be explained later, thus approaching to network-based MOOCs (Lane, 2012).

Regarding this mixture, there has been an opportunity to explore and experiment diverse pedagogical models through UNED MOOCs. Professors could use different approaches to tutoring and teacher-presence. Some professors have preferred to be actively involved in the delivery of the course, by encouraging debates in the fora, solving learners' doubts about content. Others have been actively involved in preparing material for self-learning and not so present in the delivery of the course. Regarding support to learners, besides professors-curators (authors of the contents, designers of the course, and in some MOOCs, actively involved in the implementation of the courses), there have been two different figures with active roles during the delivery of the courses: assistant tutors-facilitators (in charge of motivating and supervising debates along with professors, reordering messages, compiling FAQs, solving doubts about how to get access to the course content, about how to apply for completion certificates, etc.) and technical support staff (in charge of solving technical problems: registration, duplicated users, downloading badges and certificates, etc.) (Marauri, 2014; Gil-Jaurena, 2015).

Learners who complete the course (at least 80% of progress) have the following certification options:

• Digital badge: based on Mozilla Open Badges Infrastructure (OBI), it is free and it doesn't have academic effect.
• Digital credential: digital certificate with no academic effect, it costs 15€.
• Certificate: digital certificate for learners who, besides completing the course, take an exam in any of our regional centres. It costs 50€ and can have academic effect (recognition in UNED degrees).

Purpose of the Study

Taking into account a wide variety of MOOCs of different subjects and structures, the purpose of this study is to know which dimensions and elements of MOOCs provide a satisfactory experience. What is the place of the perception of methodological design of the MOOC in such an experience? Are personal and relational aspects preferable? Is technological support more important than personal support? And what about the offered material?

Methodological Strategy and Data Collection

In order to carry out a comprehensive evaluation of the experiences during the second series of the UNED MOOCs, the following objectives, dimensions of analysis and information sources (in parentheses) have been established:

1. Methodological evaluation, at a meta-level, especially designed to reflect on the empirical material and information which is currently available and which would be necessary for future evaluations.
    
2. Analysis and evaluation of records produced by the users of the courses (database of the UNED MOOCs platform)
1. Number of students registered.
2. Number of students who start the course.
3. Number of students who successfully finish the course.
1. Those who complete 85% of the course.
2. Those who obtain a credential or certificate.
    
3. Socio-demographic profile of the enrolled students (gender, age, education level, employment status, country of residence), previous experience of MOOCs, motivations, expectations, and assessments about the use and usefulness of the elements of the course was gathered by two online surveys. Initial survey was completed by registered students, given at the beginning to the students enrolled in each course, and it was composed by nine questions, three of which were multi-choice questions. The average time to complete this questionnaire was between two and three minutes. On the other hand, final survey was given at the end of the course and mainly replied by those students who had completed at least 85% of the course (it includes watching the videos, passing the tests and undertaking peer activities if it is the case) and voluntarily agreed to participate. This end survey was composed by socio-demographic variables and 26, 10-point (1-5: insufficient, poor, 5-7: enough, sufficient, 7-9: notable, good, 9-10: outstanding, very good) scale items (6 in Q7, 8 in Q8, 8 in Q9, and 4 in Q10). Learners were informed that the surveys are anonymous. Internal consistency reliability can be considered high: 0.938 Cronbach's coefficient alpha. Validity is obtained from the significance and strong correlation (0.701 Pearson's correlation coefficient, P<0.01 two tails) between additive score from the responses to Q7, Q8, and Q9 items and responses to recommendation of the course to other people interested in the subject (Q10 item 3). The average time to complete this second questionnaire was between six to eight minutes. Both online surveys were conducted by Lime Survey software and analysis was carried out using SPSS v.24 software for Windows. The level of statistical significance was established in P<0.01.
    
4. Analysis and evaluation of dimensions linked to the learning experience:
1. Disposition of the users (initial survey).
1. Reasons for enrolling in the course (Q8).
2. Course expectations (Q9).
2. Course experience (final survey).
1. Overall level of satisfaction.
2. Satisfaction with the platform (Q7.6).
3. Satisfaction with the organisation of the course (Q7.5).
4. Satisfaction with the length of the course (Q7.4).
5. Degree of fit with previous expectations (Q7.1).
6. Satisfaction with the methodology (Q7.3).
7. Satisfaction with the content (Q7.2).
3. Assessment of the use and usefulness of the elements of the course (final survey).
1. Videos (Q8.1 and Q9.1).
2. Complementary material (Q8.2 and Q9.2).
3. Tests (Q8.5 and Q9.5).
4. Self-assessments (Q8.10 and Q9.10).
5. Evaluation of the activities of peers (Q8.6 and Q9.6).
6. Support from the facilitator (Q8.7 and Q9.7).
7. Support from the curator (Q8.8 and Q9.8).
8. Support from peers (Q8.9 and Q9.9).
4. General assessments and future expectations (final survey).

The methodological strategy followed is based on two main principles:

• Given that the questionnaires have been customised to suit the needs of each course, our analysis only considers questions common to all courses;
• The focus is on the overall experience of the project, without going into a detailed analysis of each course.

The 17 courses analysed were the following (ordered by size: number of registered learners):

1. Start with English: learn the thousand most used words and their communicative possibilities.
2. Professional English.
3. Accounting: the language of business.
4. ICT for teaching and learning.
5. The human resources function in non-profit entities: competence-based management.
6. School tutoring.
7. Great works of Spanish art of the Renaissance and the Baroque in their international context.
8. The transversal skills of the entrepreneur.
9. Basics of circuits and practical electronics.
10. Entrepreneurship and social innovation.
11. Corporate social responsibility: path to sustainability.
12. Happiness, economics and business practice.
13. Modular teaching mini-videos: a critical element in the design of a MOOC.
14. Logic and uses.
15. Ibervirtual COMA: basic digital skills.
16. Non-profit organisations (NPOs) in the network society: analysis and challenges.
17. Basic analytical chemistry.

These 17 courses were selected among the 25 offered considering that they were 100% run by UNED with their professors. The other eight courses were developed in partnership with other institutions and had specific characteristics, this they are not part of this study.

The main characteristics of both surveys are summarised in Table 1.

Table 1

Population, Sample, Objectives and Dates of the Initial and Final MOOC Surveys

English courses were the most popular (Basic and Professional English courses). Financial accounting and ICT were also popular. More specialised courses, such as chemistry or logic, had fewer learners, but still more than 500 in the second edition.

Results and Discussion

Socio-Demographic Profiles

The gender distribution is the same in both the initial and the final surveys, being 36.4% men and 63.6% women (Figure 1). The equal distribution suggests that gender is not a relevant factor in determining whether or not the course is finished, or at least almost entirely completed. But the predominance of female learners challenges other studies that report that a majority (Ebben & Murphy, 2014) and more than 70% of participants in MOOCs (Ho et al., 2014) were males. But if we compare with learners registered in regular courses at UNED (graduate, master's, etc.), female are also a majority, as in the MOOCs (55.2% in the school year 2013-14)³. The skew towards a female majority in the UNED MOOCs could be also related to the knowledge area: most MOOCs in this study belong to Social Science and Humanities areas, where females are more present, as opposed to STEM courses where males are majority. As reported by Jiang, Schenke, Eccles, Xu and Warschauer (2016), only 23.89% females were enrolled in the Harvardx and MITx STEM MOOCs in 2012-13.

Figure 1. Distribution of samples by gender.

The average age of those completing the initial questionnaire is 36.17 years; while for those completing the final survey the average age rises slightly to 37.89 years, almost two years more. Worth mentioning the presence, even if the percentages are very low, of people under 18 and also of elderly people carrying out the courses, although the bulk of the students are found in the middle age ranges, especially between the ages of 31 and 45 (Figure 2). However, Table 2 shows that the age group that is most likely to complete a substantial part of the course -given indirectly by the completion of the final questionnaire of the course- is the group of students between 46 and 60 years of age.

Table 2

Age Groups of the Participants in the Surveys on the MOOCs

Figure 2. Distribution of respondents by age group and initial/final survey.

By nationality, almost four out of five of those responding to the initial questionnaire reside in Spain (79.5%). When referred to the general MOOC offer in the world, Oliver, Hernández-Leo, Daza, Martín, and Albó (2014), explain how "from the demand side, Spain also stands within the top-five countries with more participants, right after USA, UK, Canada and Brazil" (p. 5).

It is important to note the presence among those surveyed of residents from other Spanish-speaking countries: 2.8% Argentinians, 4.7% Colombians, 3.7% Mexicans, 1.4% Peruvians, 0.9% Chileans, 0.6% Venezuelans, and 0.7% from other Spanish-speaking countries. If these figures are added to the previous figure, it can be seen that residents of Spanish-speaking countries make up almost all of the students.

In the sample of students who completed the final questionnaire, the vast majority (89.3%) are resident in Spain, while the 1.9% residing in Mexico and 1.6% in Colombia are also worth highlighting.

The majority of people who have attended the UNED MOOCs are either currently studying or have completed higher, university-level education (Table 3). This result is in accordance with findings reported by other authors (Emanuel, 2013; Ho et al. 2014; Jordan, 2014; Bates, 2015). The persistence in MOOCs only achievable for privileged learners (those who have previously attained higher education qualifications) has been identified by Schuwer et al. (2015) as an example of inequality in access to education, and as one of the threats of MOOCs on the macro level.

Table 3

Percentage Distribution of Samples by Level of Education (Completed or in Progress)

In view of these distributions, and if we assume that those who have completed the final questionnaire have more or less satisfactorily completed their respective courses, we can see there is greater likelihood of this completion occurring for those who are either currently studying or have completed higher education than for those who have achieved a lower level of education (Figure 3).

Figure 3. Respondents by level of education and initial/final survey.

Regarding employment status, those making up the initial sample and, generalising, the body learners enrolled in the MOOC courses analysed, are mainly employees (35.2%) and unemployed (36.3%) (Figure 4). Students make up the third largest group with 19%. Self-employed represent just 7.7% of the total; but their high presence in some of the courses, such as the Mini-Videos and Social Responsibility courses, should be highlighted (Table 4).

Figure 4. Respondents by employment status in initial survey.

Table 4

Learners' Employment Status (%)

Summarising, concerning learners' profile, enrolment information and results from the initial survey say that there is a majority of women, with a high level of education and either unemployed or employed. Most of the enrolled learners are in Spain (more than 90%). Other learners are in Mexico, Colombia, Argentina, or European countries.

Previous Experience of MOOCs

Of the enrolled students who participated in the initial questionnaire, 70.5% stated that this was their first MOOC experience with the UNED open courses, while 8.3% had had only one previous experience. As little as 3% come to these courses with prior experience involving three or more courses of this sort. Therefore, it is fair to say that for the majority this opportunity represents their only point of reference regarding the MOOCs experience.

In terms of previous experience in COMA/MOOC courses, 9% have had this in different UNED platforms. However, 7% of the total sample have already tried the UNED-COMA courses themselves. This is followed by 4% for MiriadaX and just under 4% for Coursera.

Motivation and Reasons for Enrolment

Bearing in mind that an average of 2.5 reasons for embarking on the UNED MOOC courses analysed was identified by respondents, the most common of these are interest in course content, subject, and expected usefulness of this subject for professional development. The first of these reasons is given by almost 72% of participants in the initial survey; while the second is given by 60.6%. It should be borne in mind that a third (33.8%) give the two reasons together. In fact, between the two they account for 51% of the total collected answers. This is shown in Table 5.

Table 5

Reasons for Enrolment

*Note. a. Dichotomy group tabulated at value 1.

Looking beyond the subject of each specific course, it should be noted that 30% of participants in the questionnaire have selected the fact that the course methodology suited them as a reason for taking part in this experience. This is relatively independent of what is meant by the concept of methodology in this case: Distance learning? Free? Short courses? Participation and collaboration among peers? That is to say, there is something specific in this approach to teaching that attracts a significant number of those who have enrolled.

Given the importance of this specific reason, we will concentrate on it now. Thus, an analysis using binary logistic regression, inasmuch as this demonstrates the direct relationship with the dependent variable, putting the other variables in the model to one side and taking reason as the dependent variable, shows us the statistical significance of variables such as gender. Focusing on the latter, the probability that women indicate this option is 1.2 times more likely than it is for men. Age, however, although significant in the sense that the probability of selecting this reason increases with increasing age, is a factor of little importance. With regard to activity, it is the self-employed students who, compared with those involved in other activities, appear to be more motivated by the methodology of these courses. And, looking at the level of education, the probability of selecting this reason is significantly higher among those who have completed higher education, as would be expected. Thus, those with a university degree or equivalent, or who are studying for this, are more than twice as likely to select this option as respondents who have completed basic or primary education. Further detail and clarification is provided by Table 6, which shows the percentage of those who choose this as a reason within each of the different levels of education, independently of the regression analysis.⁴

Table 6

Level of Education (Completed or in Progress) by Reason for Taking this Course Cross-Tabulation: I Find That the Course Methodology Suits Me*

* Asymptotic significance (two tails) of Pearson's Chi-Square<.000

Finally, we would draw attention to the 28% who have selected the prestige of the UNED as a reason.

Expectations

What do the people enrolled in the UNED MOOCs expect from them? What are their expectations? The majority of the answers focus on obtaining the credential. In a credentialist society it is the certificate that counts, to the extent that four out of five people enrolled in the courses point to the credential as the objective for taking the course (Table 7).

Table 7

Expectations on Taking This Course

This pursuit of the credential accounts for as many as 92% of those enrolled in the MOOC on "School tutoring," 88% of those in the course "ICT for teaching and learning" and 87% of those in "Entrepreneurship and Social innovation." Where this expectation has least relative weight is in the "Corporate Social Responsibility" course, where it has not been selected by anyone, although in this case the sample base is very low. The next course where this aspect has relatively low importance is "Logic and uses," with 67% selecting it.

However, in contrast with these high expectations regarding completion of courses and requesting the corresponding certificate, the average course completion rate stands at 13.71% of those who enrol, and the application for certificates at 19.31% of those who complete the course, as reflected in Table 8. This difference between expectations and completion rates in MOOCs has been previously reported; as Reich (2014) stated, "the majority of students who intended to complete a course -in some cases the overwhelming majority of students - were not successful in doing so." (para. 22)

Learners' Behaviours and Rates

With regard to the behaviour of students, the most relevant data regarding enrolment, commencement, completion, and requesting of the certificate for each course are given in Table 8.

Table 8

Enrolment, Commencement, Completion and Certification Rates by MOOC

About dropout rates, the average of students who start the course but don't finish it is higher than 80%. Completion rate is 13.71% (considering whole enrolment) and 17.79% if we consider only those we started the course. Despite completion rates are low, they are higher than those reported in other studies, such as a rate between 2% and 10% (Reich, 2014), 12.6% (Jordan & Weller, 2013, as cited in Weller, 2014) or average as low as 5% (Jordan, 2014). The high dropout rate and low completion rate have been identified by Schuwer et al. (2015) as a threat of the MOOC movement at the micro level.

Learners' Evaluation and Satisfaction

In this final section on the final evaluations of the MOOCs we must begin by stating that 94% of those who responded have completed the course. Therefore, an overwhelming majority of the satisfaction ratings set out below are from people who have completed the courses.

The rating for what might be considered general aspects of the course is quite similar, being around eight. The most highly rated aspects are the operation of the virtual platform, the organisation and structure of the courses, and their length. The lowest rated aspect is course contents (Table 9).

Table 9

Evaluation of MOOC Elements in the Final Survey (from 1 to 10 )

The variability in the evaluation of the dimensions of the course is higher when the specific elements that have been used are addressed, as shown in Table 10. Videos, tests, and self-assessments are the most highly used dimensions or materials, the fact that the first of these is close to an average rating of nine (8.94) being noteworthy. However, the different forms of personal support were the lowest used elements, neither the support of the curator (4.87), the lowest rated of all, nor that of the facilitator (5.23), nor that of peers (5.01) having been convincing. Or at least it seems that more was expected of them. This is probably logical in the case of unfamiliarity with this kind of course and its comparison with "normal" courses.

Table 10

Degree of Use of MOOCs Tools (from 1 to 10)

Students were asked about their perception of the degree of usefulness of each of these tools for the implementation of the course. The result of the responses are shown in Table 11.

Table 11

Degree of Need and Usefulness of MOOCs Tools for Learning and Progress (From 1 to 10)

Thus we see that tools that have been used and appear to be highly rated (Table 10) are also considered very useful for learning and course development (Table 11). This is the case for the videos, supplementary materials, tests, and evaluations. Meanwhile, aspects that obtained a lower rating have been considered useful as support tools. This is the case for the support of the facilitator and curator. On a third level, the evaluation of activities by peers and peer support itself (Figure 5) received a relatively low rating, and its usefulness was less highly valued than the previously mentioned aspects. These results are in agreement with those obtained in the analysis of 10 courses in another platform that offers MOOCs in Spanish, Miriadax, as reported by González and Carabantes (2017). The use and satisfaction with various MOOC tools are aligned if we consider these two studies.

Figure 5. Assessment tools used and the degree of need for tools for course development (0-10).

In any event, the general experience seems to have been quite satisfactory, bearing in mind that, for example, the likelihood of recommending the course taken scored an average of 8.34, which can be interpreted as a high probability. Perhaps most important of all is the fact that the phrase "I would like to take more courses of this type at the UNED" was rated at 9.05. This indicates a high level of satisfaction (Table 12).

Table 12

Final General Evaluation (From 1 to 10)

Do these courses serve to promote the institution and the educational opportunities it offers, considering that this is one of the functions that might be assigned to the MOOCs? In the light of what is shown by the last row of Table 12, the table provides a very positive answer to this question, since 8.14 is the rating for the degree of agreement with the statement: "After completing this course, I would like to take another type of course at the UNED." Its advertising function appears to have been successful. That is to say, it serves as a point of entry -a sort of trial- for the UNED, as is also underlined by the average rating of 8.13 of the sentence: "The completion of this course has allowed me to find out about the training on offer at the UNED."

A minimal analysis of the behaviour of this dimension tells us that it has very little relation with age, with a low correlation coefficient of almost zero, with a greater weight among males (8.29) than among women (8.09), and, perhaps most important, with the highest value being reached among those with average levels of education, as shown in Table 13. Moreover, controlling the other socio-demographic variables available (sex and age), it can be said that there is a slight tendency for the rating of this aspect -the probability of taking other courses at the UNED- to decrease as level of education rises. This is something that Table 13 itself suggests.

Table 13

Future Expectations to take another course at UNED (official studies, continuing education, etc.) from 1 to 10 by Level of Education (completed or in progress)

Performing a factor analysis of the principal components (Appendices 2 and 3), focusing the analysis on the first two components, tells us that the assessment of these MOOC courses is structured around two axes. The first axis relates primarily to the general aspects that have been assessed: "I would recommend this course to others interested in the subject," "The course is well organised/structured," "I have found the methodology of the course motivating and appropriate," "The course matched my expectations," or "I think that the course contents have been suitable and sufficient." The second axis involves more specific aspects, especially related to the perception of the support received, highlighting this as a relatively key dimension when it comes to assessment, almost like a demand. Thus, in this axis the assessment of the role that the support of peers, facilitator, and curator played in the course has special weight.

Still bearing in mind these two axes, what is their relationship with fostering an interest in taking other courses at the UNED? Through the correlations in the matrix of components (Appendix 2), this relationship can be seen. That is to say, the correlation with the first axis, or general aspects, has a fairly high correlation (0.659, being p<0.01); while in the second axis it has much less weight. Meanwhile, the correlation of the variable that denotes interest in taking other courses at the UNED and variables that reflect the assessment of the support (facilitator, curator, and peers) received during the course is low, not reaching 0.3 under any circumstances. Also, if we take into account, not so much the evaluation of these tools in the course taken but the evaluation of their usefulness in this type of course, we can see that the correlation with the variable representing interest in taking other courses in the UNED is still low, reaching a peak of 0.321 for the support of the curator.

Conclusions

The main results have been presented in the previous section. Below is a summary of those aspects that are of particular interest for finalising the details of the next series of MOOCs and for their evaluation.

At the institutional level, the results of the analysis demonstrate the existence of a high level of interest in this type of course, as well as high expectations and the demand for more similar courses. As a way of attracting potential students, we must recognise and take advantage of MOOCs as a point of entry to the UNED, bearing in mind that it is the first contact with the institution for most of those who enrol in these courses. Linking these courses with other UNED academic courses (lifelong learning, regulated studies, etc.) could meet the demand of the students and might result in the sustainability of the project. In this sense, UNED regulated the approval of new MOOCs according to some criteria, among them the relation with other educational offer in the institution (Gil-Jaurena, 2015). It would be useful to continue the evaluation, analysis, and research on MOOCs offered by the UNED. In particular, aspects such as the difference between the expectations and the actual rates of completion of courses or the predominantly well-educated profile of the users are striking and require specific studies.

At a pedagogical level, although in general the methodology and materials in the MOOCs have been positively valued, we must rethink the personal support roles (curator, facilitator), which have been the least highly used and rated by the learners. Independent studies about each MOOC considering their pedagogical specificities (more video-based, more social interaction-based, etc.) would lead to refining the analysis and providing valuable recommendations for teachers, instructional designers, and the institution itself.

Another important issue relates to the identification and better knowledge of learners in different MOOCs or knowledge areas: who they are, which motivations and expectations they present when they approach MOOCs, which use they make of the resources, what other needs they show, etc. Besides the profile, information about the learning paths that students follow is also of great interest in order to improve the learning experience and reduce attrition rates. A more complex and accurate learning analytics system would be recommended. Clustering (Khalil, Kastl, & Ebner, 2016) is an example of analysis we would use to reveal patterns and study the correlation between learners' behaviours, engagement, and completion rates. In this regard, a reconsideration of the automated measures from a pedagogical perspective is needed, in line with the position held by DeBoer, Ho, Stump, and Breslow (2014). Finally, a deeper analysis of the dimensions that influence learners' evaluation and satisfaction with MOOCs would be of interest in order to design and offer effective MOOCs and other online courses in the near future.

Acknowledgment

This study has been supported by the UNED OER programme (UNED Abierta) and by the Office of the Vice President of Quality and Internationalisation, VIII Call for Proposals for Research Networks for Educational Innovation: Development of Pilot Projects to improve teaching quality in the European Higher Education Area (academic year 2014/2015).

Notes

1. No initial survey results are available
2. No final survey was conducted
3. Info obtained from the UNED statistical portal: https://app.uned.es/evacal/
4. The result of regression analysis is in Appendix 1.

References

Bates, T. (2015). Teaching in a digital age. Guidelines for designing teaching and learning. Vancouver: Tony Bates Associates Ltd. Retrieved from https://opentextbc.ca/teachinginadigitalage/

DeBoer, J., Ho, A. D., Stump G., & Breslow, L. (2014). Changing "Course" reconceptualizing educational variables for massive open online courses. Educational Researcher, 43(2), 74-84. doi: http://dx.doi.org/10.3102/0013189X14523038

Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly discourse: A review of nascent MOOC scholarship. Learning, Media and Technology, 39(3), 328-345. doi: http://dx.doi.org/10.1080/17439884.2013.878352

Emanuel, E. J. (2013). Online education: MOOCs taken by educated few. Nature, 503(342). doi: http://dx.doi.org/10.1038/503342a

Gašević, D., Joksimović, S., Kovanović, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. International Review of Research in Open and Distance Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1954/3099

Gil-Jaurena, I. (2014, April). Open educational policies and practices in the Spanish National Distance Education University: UNED Abierta (UNED OER programme). OCWC Global Conference 2014 Proceedings. Open Education for a multicultural world, Ljubljana (Slovenia). Retrieved from http://conference.ocwconsortium.org/2014/wp-content/uploads/2014/02/Paper_58-UNED1.pdf

Gil-Jaurena, I. (2015). MOOCs in Spain: Preliminary lessons from UNED MOOCs experience. In B. Kim (Coord.). MOOCs and educational challenges around Asia and Europe (pp. 191-204). Seoul: KNOU Press.

González de la Fuente, Á., & Carabantes Alarcón, D. (2017). MOOC: medición de satisfacción, fidelización, éxito y certificación de la educación digital [MOOCs: measuring satisfaction, loyalty, success and certification of digital education]. Revista Iberoamericana de Educación a Distancia, 20(1), 105-123. doi: http://dx.doi.org/10.5944/ried.20.1.16820

Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses. HarvardX and MITx Working Paper No. 1. doi: http://dx.doi.org/10.2139/ssrn.2381263

Jiang, S., Schenke, K., Eccles, J. S., Xu, D., & Warschauer, M. (2016). Females' enrollment and completion in science, technology, engineering, and mathematics Massive Open Online Courses. arXiv preprint. Retrieved from https://arxiv.org/pdf/1608.05131

Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open and Distributed Learning, 15(1). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1651

Khalil, M., Kastl, C., & Ebner, M. (2016). Portraying MOOCs learners: A clustering experience using learning analytics. Proceedings of the European MOOC Stakeholder Summit 2016, 265-278. Retrieved from http://emoocs2016.eu/wp-content/uploads/2016/02/proceedings-emoocs2016.pdf

Lane, L. (2012, August 15). Three kinds of MOOCs. Lisa's (online) Teaching & History Blog. Retrieved from http://lisahistory.net/wordpress/2012/08/three-kinds-of-moocs/

Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008-2012. The International Review of Research in Open and Distributed Learning, 1 4(3), 202-227. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1455/2531

Marauri, P. (2014). Figura de los facilitadores en los Cursos Online Masivos y Abiertos (COMA/MOOC): nuevo rol profesional para los entornos educativos abiertos [The figure of the facilitators in the mass and open online courses (COMA/MOOC): new profesional role for open educational environmentes]. Revista Iberoamericana De Educación A Distancia, 17. (1), 35-67. doi: http://dx.doi.org/10.5944/ried.17.1.11573

Oliver, M., Hernández-Leo, D., Daza, V., Martín, C., & Albó, L. (2014). MOOCs en España. Panorama actual de los Cursos Masivos Abiertos en Línea en las universidades españolas [MOOCs in Spain. Current overview of the Open Massive Courses Online at Spanish universities]. Barcelona: Cátedra Telefónica Universitat Pompeu Fabra. Retrieved from http://www.catedratelefonica.upf.edu/wpcontent/uploads/2014/02/MOOCsenEspa%C3%B1a1.pdf

Reich, J. (2014, December 8). MOOC completion and retention in the context of student intent. EDUCAUSE Review. Retrieved from http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent

Rodriguez, O. (2013). The concept of openness behind c and x-MOOCs (Massive Open Online Courses). Open Praxis, 5(1), 67-73. doi: http://dx.doi.org/10.5944/openpraxis.5.1.42

Schuwer, R., Gil-Jaurena, I., Aydin, C., Costello, E., Dalsgaard, C., Brown, M., … & Teixeira, A. (2015). Opportunities and threats of the MOOC movement for Higher education: The European perspective. The International Review of Research in Open and Distributed Learning, 16(6). doi: http://dx.doi.org/10.19173/irrodl.v16i6.2153

UNED (2016). Portal Estadístico Oficina de Tratamiento de la Información [Statistical Site of the Office of Information Processing]. Retrieved from https://app.uned.es/evacal/

Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013-2015. The International Review of Research in Open and Distributed Learning, 17(2). doi: http://dx.doi.org/10.19173/irrodl.v17i2.2448

Weller, M. (2014). The battle for open: How openness won and why it doesn't feel like victory. London: Ubiquity Press. doi: http://dx.doi.org/10.5334/bam

Appendix 1

Result of Logistic Regression Analysis About Selection of Methodology as a Reason for Enrolment: Independent Variables Standardised Coefficient: Initial Survey

Appendix 2

Component Matrix: Final Survey

*Note. Extraction method: principal component analysis
a. 4 extracted components.

Appendix 3

Percentage of Explained Variance in Principal Component Factor Analysis: Final Survey

*Note. Extraction method: principal component analysis.

Evaluation of the UNED MOOCs Implementation: Demographics, Learners' Opinions and Completion Rates by Inés Gil-Jaurena, Javier Callejo, and Yolanda Agudo is licensed under a Creative Commons Attribution 4.0 International License.