Volume 19, Number 2
Renata Marciniak
Autonomous University of Barcelona
The quality assurance of online Higher Education online programmes is one of the great challenges faced by Spanish universities. Regular assessment of these programmes is essential in order to take actions to improve their quality. The said assessment should be complex and include all of the components of the programme, as well as its planning and implementation stages and its effects. The purpose of this paper is to present a model designed to assess the quality of online Higher Education online programmes that includes the assessment of the quality of the programme itself, as well as its continuous assessment. In order to design the model, the author conducted a bibliographical analysis of different standards, models, and guides developed in Spain and other countries to assess online education. The model was validated by 23 international online education experts. The results of the validation were triangulated with specialized literature, thus allowing the author to make decisions regarding whether to change the model by keeping, reformulating, or removing a dimension or indicator. As a result, two variables, fourteen dimensions, and 81 indicators were obtained. In order to verify the utility of the model it was applied in the assessment of four online programmes. The model guides the persons in charge of the implementation of online programmes and allows to conduct a more comprehensive assessment of the programme in order to discover its strengths and weaknesses, and opportunities for its improvement. The model can be also applied by online programme designers as a guideline for creating other, high quality programmes.
Keywords: online education, quality of education, online programmes, assessment model
According to figures provided by the Spanish Ministry of Education, Culture, and Sports (2016) more than 15% of Spanish students study online. Given this high percentage, guaranteeing a high quality of online education is one of the main objectives of Spanish universities that offer online education programmes. This way, these universities seek to put an end to the assumption that the quality of online education cannot be as high as that of traditional education.
Furthermore, there is a discussion among Spanish academic groups regarding the appropriate method for assessing the quality of online education programmes with the purpose of answering the question of whether this quality should be assessed based on the same criteria as those applied to traditional programmes. The dominant view is that the assessment of the quality of these two types of programmes - online and traditional - is basically the same and only differs when it comes to the meaning given to the criteria and the instruments used to make measurements (Padilla, 2005; Marúm-Espinosa, 2011). As a result of this approach, the current trend in assessing online education programmes, especially when it comes to universities that offer both traditional and online education, is to assess them in the same manner as the traditional ones. Thus, online education programmes are assessed by using the criteria and indicators designed for assessing the quality of traditional education programmes without applying quality dimensions and indicators suitable to the context of online education. Moreover, Spanish accreditation organizations also assess and certify online programmes by applying the same models as the ones applied to traditional education programmes (Chmielewski, 2013).
Assessing online programmes in this way is a major drawback, as online education differs from traditional education when it comes to its organization and functioning. Online education is not equivalent to traditional models, and thus the same quality assessment mechanisms and models should not be applied to both types of education (Jung & Latchem, 2012; Veytia & Chao González, 2013).
Some models exist that seek to provide a response to the issue of quality assessment in online Higher Education programmes in Spain, such as those developed by AENOR (2012), ANECA (2012), AQU (2007). These models, however, combine a variety of approaches and sometimes respond to contradictory paradigms. They thus use divergent dimensions (and assign different meanings to these dimensions) to assess the quality of online education. Moreover, the indicators proposed by these models rarely highlight the need to assess all pedagogical and technological elements of the programm, its planning and implementation stages, and its impact, as postulated by Martínez Mediano (2013):
Programme evaluation is the systematic collection of information regarding a programme in order to meet specific needs, that is focused on 1) the quality of the programme itself, its basic elements, structure and coherence; 2) the planning of its launching, taking into consideration human, material and organizational resources, 3) the development of the programme and 4) the results of the programme. (p. 197)
The lack of knowledge in Spanish universities when it comes to the correct assessment of online education programmes, and the lack of models that could contribute to said assessment, inspired us to conduct our own research. The general purpose of the research was to design an applicative model for assessing online Higher Education online programmes in Spain by integrating the assessment of the quality of the programme itself and the ongoing assessment of the programme. This paper presents, from a panoramic point of view, the process of designing and validating this model, which is expected to become a useful tool for assessing and improving all of the components of education online programmes as well as the three stages of their existence - that is, the initial stage, the development stage and the final stage. It should be noted that this global approach has not been addressed so far (or has only been partially addressed), as has been noted by authors such as Butcher & Hoosen (2014), and Chmielewski (2013).
An online education programme is a document (computerized or not) that covers actions designed to achieve the educational objectives for a determined period of time and which, in turn, is composed of a set of elements necessary for the development of a learning/teaching process regarding a determined subject taught online. It is conducted in its entirety using communication and information technology with the technological support of an online platform and the support of online teachers who assist and guide the students' academic development through different interactive telecommunication systems (Marciniak, 2016).
It seems that there is not much point in separating the assessment of the components of the programme - the traditional purpose of assessment - from the ongoing assessment of the programme. These are two sides of the same coin, if we take into account that the assessment of an online programme - considered as an organizational scheme of pedagogical and technological components - only makes sense if we assess the quality of those components together with the implementation of the programme and the results obtained from its implementation.
As a result, except for the evaluative research and other purposes, it is perfectly possible to harmoniously integrate the assessment of the quality of the components of the programme (i.e., the assessment of the quality of the programme itself) and the ongoing assessment of the programme. In this way, the assessment will be complete, will allow for an in-depth diagnosis of the current quality of the programme and will allow to reflect on what needs to be modified in order to ensure the highest quality possible. Furthermore, it will make it possible to determine whether the programme has been designed, developed, and implemented in a way that will produce the desired effects.
Assessment of the quality of the online education programme itself. The assessment of the quality of the programme itself is defined as the assessment of all of the elements that constitute an online programme (Marciniak, 2016).
Based on a bibliographical review (Barnard & Echols, 2015; Berge, Collins & Dougherty, 2000; Higher Education Compliance and Quality [HECQ], 1997; Meza, 2012; Rushby & Surry, 2016; Shattuck, 2014, among others) it has been established that there are no formulas or schemes defining the components that should be included in online learning programmes or in the assessment of such programmes. This is due to the fact that each programme is unique and, thus, its structure and elements depend on its objectives, contents, resources, and so on. Nevertheless, the Higher Education Committee on Quality (1997) proposes the following elements that should be taken into account while designing and assessing online programmes: learning objectives, teaching strategies, learning assessment strategies, student profile, interactive teaching resources, learning resources, thematic contents, and activities schedule.
According to Barnard and Echols (2015), an online programme should include student profiles, thematic contents, learning strategies, characteristics of the educational technology, and strategies for the assessment of the learning process. Rushby and Surry (2016) propose different components of online programme: curricular data, general learning objectives, programme contents, learning activities, information sources, communication tools, media use, and strategies for the assessment of the learning process.
Based on the proposals made by the above cited authors, we can conclude that an online education programme should focus on clarifying and developing the following components:
These components define the quality of online Higher Education programmes and, thus, should be assessed in order to identify the strengths and weaknesses, and the opportunities for improvement, of each programme.
Ongoing assessment of the online education programme. The ongoing assessment of the programme provides constant information regarding its progress. This information allows for the improvement the programme through feedback.
Pérez Juste (2014) and Surry and Ensminger (2009) affirm that the assessment of an education programme (whether traditional or online) should be designed to be carried out in three stages (initial, development, and final). Furthermore, each stage should have its own entity and objectives and the three stages combined should contribute to the improvement of the programme as a whole.
The initial assessment of an online programme is carried out before the programme is launched and has two main purposes: (1) to launch the programme under favourable conditions and (2) to establish the level of readiness of the programme for its launch so as to minimize the risk of failure.
The processual assessment of the programme evaluates the performance of the programme in its development stage and the possibilities for its improvement. This assessment has two main purposes: (1) To facilitate the timely making of immediate decisions regarding the improvement of the programme based on the results of the assessment and (2) to collect data for subsequent decisions (Pérez Juste, 2014).
The final assessment of the programme is focused on the achievements made by the programme. The purpose of this assessment is to verify whether the objectives of the programme have been reached, measure its effects, and assess student satisfaction with the programme.
The specific nature of online education makes it necessary to draft, develop, and assess a series of educational actions for its correct development, and to determine unified guidelines in order to guarantee that these actions lead to high-quality results. To address these challenges, several authors have developed initiatives (standards, models, and guides) related to the quality assurance of online education. These initiatives can be used by universities as allies in their assessment of educational programmes. Some of the aforementioned initiatives are internationally recognised, while others have been designed to be applied in Spain.
International referents. It is worthwhile to consider the five pillars of quality online education model developed by the Online Learning Consortium (OLC, 2002) in the United States. The purpose of this model is to assist institutions in the identification of objectives related to online education and to measure their progress. This purpose can be fulfilled by assessing five pillars that constitute the quality of online education: (1) learning effectiveness; (2) faculty satisfaction; (3) student satisfaction; (4) scale; and (5) access.
In Latin America, the Latin American and Caribbean Institute for the Quality of Online Higher Education (CALED, 2010) developed the Guía de autoevaluación para programas de pregrado a distancia (guide for the self-assessment of distance undergraduate programmes) in order to improve the quality of the distance Higher Education offered by institutions in Latin America and the Caribbean. The guide includes a series of self-assessment guidelines within the certification process offered by CALED. The following areas of online education are assessed in this self-assessment: technology, training, instructional design, student information, and services.
While describing initiatives related to the assessment of the quality of online education, we must also mention the European Foundation for Management Development's (EFMD, 2006) technology-enhanced learning accreditation (EFMD CEL) programme. EFMD CEL is an accreditation process that assesses online education programmes with the purpose of improving the quality of online education programmes offered worldwide. The core of this process is the self-assessment of the programme by a group of internal auditors who assess the programme based on: programme profile, pedagogy, economy, technology, organization, and culture.
The aforementioned programmes are not the only programmes developed to assess the quality of online education. Table 1 contains the comparative analysis of some of the models analysed during the bibliographical research. The analysis has been arranged chronologically.
Table 1
Comparative Analysis of Some of the Models Developed to Assess the Quality of Online Education
| Model/author/year of creation | Model characteristics | Assessment dimensions |
| The e-learning maturity model (eMM; Marshall, 2006) | This model constitutes the framework for the improvement of the quality of online education programmes. It has been developed to guide organisations in their understanding of their capacities regarding e-learning, and to help them in the assessment of its quality. |
|
| CAPEODL Model (comprehensive approach to program evaluation in open and distributed learning; Khan, 2007) | This model focuses on the critical revision of e-learning products and services by collecting data related to all the aspects of e-learning programmes from its initial to final stages. |
|
| Evaluation logic model (University of Wisconsin, 2009) | This model helps to plan, implement, assess and communicate the programme to all the stakeholders. The model proposes five actions that describe online education programmes. |
|
| A conceptual model for measuring the quality of e-learning through knowledge sharing indicators (Raeesi, Qorbani, & Akhgar, 2010) | This model offers a series of indicators to assess the knowledge sharing process carried out through online education programmes. These indicators are divided in two groups: (1) indicators of quality measurement of e-learning with respect to knowledge sharing; and (2) indicators of knowledge sharing that directly affect the e-learning process. | Each group of indicators is assessed based on three perspectives:
|
| PDPP model (planning evaluation, development, process evaluation, and product evaluation; Zhang & Jiang, 2012) | The PDPP Model is based on the Stufflebeam assessment model and on the characteristics of online programmes. It proposes a system for assessing the quality of these programmes based on four dimensions and 24 indicators. |
|
| Conceptual framework for quality of e-learning (Ossiannilsson & Landgren, 2012) | The goal of the model is to guide institutions in the development, implementation, evaluation, and internalization of e-learning in Higher Education. |
|
| A model for the online education quality assessment (Giorgetti, Romero, & Vera, 2013) | This model proposes a series of dimensions and criteria to assess the educational quality in order to continuously improve it. |
|
| A model and an index for e-learning quality assessment (Petkova & Radeva, 2014) | This model is focused on the measures for the generalized quantitative assessment of the quality of distance learning platforms. It offers a geometric index to calculate different factors (characteristics) and subfactors, and to complete a final, generalised assessment of the quality of e-learning. |
|
These analysed models - as well as other models that could have been quoted, the utility of which is undeniable - enable the assessment of online education programmes through a series of dimensions that contain the information necessary to draft assessment plans. The main problem is that the variables and indicators proposed by these models rarely highlight the need to assess the quality of the programme, as well as the quality of its planning, implementation, and results.
Another weakness of the analysed models is the lack of consensus regarding the number of dimensions. Some models propose the assessment of only three dimensions, while other models propose to assess up to eight dimensions. It is further observed that we do not encounter any dimension common to all analysed models. Another weakness of the aforementioned models lies in the differing meaning given to the dimensions, as their interpretation differs depending on the author and the methodology.
Spanish referents. In Spain, few authors have developed initiatives (standards, models, guides) related to guaranteeing the quality of online education that can be used by Spanish universities as assessment allies as they move forward the assessment of educational programmes. Among others, we should mention the Association for Standardisation and Certification of Spain's (AENOR, 2012) UNE 66181:2012 standard, which sets out a quality model based on a set of indicators that represent three satisfaction factors of the recipients of online education. Each of these factors break down into key quality dimensions, based on which actions can be taken in order to improve the corresponding factor:
The Agency for the Quality of the University System of Catalonia (AQU, 2007), designed a guide for the internal self-assessment of online education, which is an assessment model for Higher Education online programmes. It is composed of five dimensions that do not differ much from the dimensions proposed for assessing traditional programmes:
The quality of online programmes is also assessed by the National Agency for Quality Assessment and Accreditation of Spain (ANECA, 2012) at three different stages. The first stage consists of an initial assessment of the education programme. ANECA assesses the project of the programme through the accreditation programme VERIFICA. Once the programme is implemented, ANECA monitors its implementation and development by the university, using for this purpose a specific accreditation programme called MONITOR. The third stage consists of a periodical renewal of the accreditation of the education programme. The said renewal is carried out by ANECA by using an accreditation programme called ACREDITA. Table 2 shows the criteria used by ANECA to assess programmes at the three abovementioned stages. It should be noted that the said criteria are used by ANECA to assess both traditional and online educational programmes.
Table 2
Criteria for the Assessment of Education Programmes Used by ANECA
| Accreditation programme | ||
| VERIFICA | MONITOR | ACREDITA |
| Programme description | Organization and development | Organization and development |
| Justification | Information and transparency | Information and transparency |
| Educational objectives | Internal quality guarantee system | Internal quality guarantee system |
| Accessibility by students | ||
| Teaching methods | Academic faculty | Academic faculty |
| Academic faculty | Didactic materials and student services | Support staff, didactic materials and student services |
| Didactic materials and student services | ||
| Expected teaching results | Efficiency indicators | Teaching results |
| Quality guarantee system | Implementation of recommendations, observations and suggestions | Indicators of the student satisfaction and efficiency |
| Programme implementation schedule | ||
From "Programas de Evaluación" [Assessment Programmes], by National Agency for Quality Assessment and Accreditation of Spain, 2017 (http://www.aneca.es/Programas-de-evaluacion). In the public domain.
The starting point for designing the model was a bibliographical/documental analysis of the components of e-learning education programmes and of guidelines regarding the assessment methods proposed for these programmes, as well as a specific bibliographical study of the standards, models and guides created in Spain and other countries in order to evaluate the quality of e-learning education (AENOR, 2012; ANECA, 2012; AQU, 2007; EFMD, 2006; European Foundation for Quality in eLearning [EFQUEL], 2012; Giorgetti, Romero & Vera, 2013; Marshall, 2006; OLC, 2002; and others).
Based on the results of the said analysis, a first draft of the model was created, composed of two variables, fourteen dimensions and ninety indicators. In order to facilitate the application of the indicators and determine its utility, a descriptive sheet was created for each of them which contained all of the components indicated by the standard UNE 66175 (see an example in Table 3).
Table 3
Example of the Descriptive Sheet of an Indicator Developed According to the Spanish Norm UNE 66175:2012
| Indicator | |
| Code | APEV-3 |
| Name | The educational goals of the online programme are adequate to the demands of the labour market. |
| Objective | To verify that the educational objectives are suitable to the demand for experts in the labour market. |
| Description | While defining the objectives, we should take into consideration the conditioning factors related to the demand for experts in the labour market. |
| Calculation formula | Option: Yes/No |
| Proposed evidences | The document containing the educational goals of the programme. Studies on the labour market and on the demand of specialists within the field of the assessed online programme. |
From "Autoevaluación de programas de educación universitaria virtual" [Self-assessment of higher virtual education programmes] (p. 311), by R. Marciniak, 2016, Barcelona, Spain: Autonomous University of Barcelona. Licensed under CC BY-NC-ND.
Taking into consideration that the model was the result of the author's own elaboration, it was necessary to validate it and determine its reliability through a methodological proceeding in order to determine the extent to which the model assessed the components it was supposed to assess.
According to Hernández, García, Padilla, García, and Leal (2012), validity refers to "the extent to which an instrument actually measures the variable that it intends to measure" (p. 201). Specialized literature proposes different types of validity and procedures to define the validity of a measurement instrument. All of these procedures apply formulas that allow to obtain reliable results and that guarantee the achievement of the set out goals (Pérez Juste, 1986). In our case, the process of validating the model was conducted through validation by experts and application of the model in the assessment of four online programmes.
Validation through expert judgement "consists, mainly, in asking several persons to cast judgement regarding an object, instrument, educational resource or to give their opinion regarding a particular element" (Cabero & Llorente Cejudo, 2013, p. 14). For this purpose, we asked 23 international experts from different countries (Spain, Portugal, Italy, Mexico, Chile, Paraguay, and Uruguay) to validate the model. The experts were selected based on their experiences in the field and on papers they authored. The experts were asked to determine the validity of each dimension and indicator of the model based on the assessment and evaluative judgment of its univocality, relevance, and importance. As for the indicators, the experts were asked to assess the suitability of the calculation formula and the relevance of the evidence.
An online form was designed in order to facilitate the validation of the model by the experts. The form was divided in three sections. In section 1 the experts were asked to provide certain personal data. Sections 2 and 3 were used to carry out the validation of the dimensions and indicators (see Figures 1, 2, and 3).
Figure 1. A screenshot of the online validation form (section 1).
Figure 2. A screenshot of the online validation form (section 2).
Figure 3. A screenshot of the online validation form (section 3).
Each dimension and indicator contained a comment field that could be used by the experts to suggest modifications. This field could also be used to suggest improvements of the proposed model. After validating all the dimensions and indicators, the expert saved and sent the form using the buttons appearing at the bottom of the form.
We contacted the experts via email in order to ask for their cooperation in the validation process of the provisional model. We attached two documents to the email message: (1) a brief description of the proposed model, its objective, and the operationalization of its variables; and (2) a document with instructions on how to fill in the online validation form. We asked the experts to respond via e-mail within one month.
The results of the validation provided by the experts through the online form were received in webpage format based on HTML5 technology, making its analysis more difficult. For this reason, the results were incorporated to a table generated by the Microsoft Excel program. Once the results were incorporated, the quantitative and qualitative validity of the model was verified.
The quantitative validity was verified using the Statistics programme in version 12 and included calculating the facial validity index, the contents validity index, and the interjudge reliability index for all the indicators composing the model. The qualitative validation of the model was verified by collecting all the comments made by the experts to justify their validation, as well as their suggestions for the improvement of the model.
Once the qualitative validation was completed, the results were triangulated with the results of the quantitative validation and specialized literature, which allowed us to make decisions regarding the maintenance, modification or removal of an indicator. For this reason, we created Tables with the specific findings related to the dimensions of the model and all the proposed indicators (see Table 4).
Table 4
Example of Triangulation of the Data for One of the Indicators of the Model
| Indicator: Accessibility and disclosure of the educational objectives | ||
| Findings based on the source | ||
| Quantitative validation | Qualitative validation | Theory |
| Indicator has a very high validation compared with all quantitative indexes: Index = 1 (maximum punctuation). Average CVR Index =1 (maximum punctuation). Average Fleiss' Kappa Index = 0.78. | Experts: Highly assessed indicator. Nevertheless, not all experts understand the accessibility of educational objectives. | According to Ardila (2011), the educational objectives of the programme should be duly described. According to ANECA (2015), the students enrolled in the online programme should be granted access to the relevant information regarding the programme and its educational objectives. |
| Researchers' decision: 1. The name of the indicator has been adjusted in order to facilitate its univocality. The new name is: The educational goals of the programme are available and accessible to all persons interested in the online programme. | ||
The definitive model for the assessment of Higher Education online programmes was designed based on the results of the triangulation process. In order to verify the utility of the model, it was applied in the assessment of four online programmes.
The main purpose of this part of the study was to validate the model in an actual context and to verify its suitability for the self-assessment of online Higher Education programmes. To achieve this goal, four self-assessment protocols were designed (one for each of the assessed programmes) to be used by researchers as observation and assessment guidelines. The data was collected from the evidence described in the descriptive sheets for each indicator. By way of example, Table 5 shows a fragment of the protocol for one of the aforementioned self-assessments.
Table 5
A Fragment of a Self-assessment Protocol
| DIMENSION: Learning activities | ||||
| Indicator | Assessment | Comments | Analysed evidences | |
| Yes | No | |||
| The learning activities proposed within the programme are of different types. | X | The programme offers a wide range of learning activities, which include, among others: participation in forums; drafting of essays; drafting of comparative tables; design of research projects; case studies; and watching and commenting on videos, etc. | The document "Planning of activities" posted in the virtual classroom of the programme. The document "Activities" posted for each unit of the programme. | |
| All the activities are coherent with the educational goals. | X | The educational objectives of the assessed programmes have not been defined and, thus, it is not possible to verify whether the learning activities are coherent with the objectives. | The document "Programme syllabus." Interview with the coordinator for the degree in logistics and transport. | |
| Instructions for each learning activity have been drafted. | X | Each activity includes guidelines that describe it, as well as its purpose, the Web tools to be used and the assessment of the activity. | The documents entitled "Learning activities for Unit 1," "Learning Activities for Unit 2," and "Learning Activities for Unit 3." | |
This application allowed us to verify the potential of the model while assessing the quality of the programmes through the detection of their strengths and weaknesses in order to design an action plan for their improvement.
The main result of this research is the development of a model composed of two variables, fourteen dimensions, and eighty-one indicators.
Variables of the model. A variable is a characteristic or quality, magnitude or quantity, that can undergo changes and that is subject to analysis, measurement, assessment, or control during a research project (Arias, 2012). In the field of the assessment of the quality of education programmes, a variable is a characteristic that expresses the feature of the quality to be assessed. There are two variables considered in the integrative model for the assessment of the quality of online Higher Education programmes developed here. The first variable is the quality of the online programme itself. We consider this variable to be crucial, as it enables the assessment of the quality of all of the components of an online education programme. The second variable refers to the ongoing assessment of the online programme, which provides continuous information regarding its progress. This information enables the improvement the programme through feedback and self-adjustment.
Dimensions of the model. "The dimensions indicate the direction of the actions and cover the hallmarks of the whole, as an integrated piece" (Careaga Butter, Meyer Aguilera, Graciela Badilla Quintana, Jiménez Pérez, & Sepúlveda Valenzuela, 2017, p. 276). A single dimension catches only one aspects of quality. The quality assessment model proposed in this study contains fourteen dimensions. The first eleven dimensions make it possible to assess the quality of all of the components of an education online programme, and the last three dimensions verify the quality of all stages of the programme. The working definition of each dimension is:
Quality indicators for online programmes . Indicators are a set of characteristics or features that constitute or allow for the description and assessment of certain dimensions of a variable. They can be presented as guiding questions or a checklist that make apparent the achieved degree of quality.
The system of 81 indicators developed to describe and assess the dimensions of our model can be used as a tool by Spanish universities to verify the extent of the application of the standards, parameters, and recommendations aimed at guaranteeing and assessing the quality of online education programmes.
Table 6 describes the distribution of the indicators based on the dimensions. The appendix to this paper describes the indicators selected to assess each dimension.
Table 6
Distribution of the Indicators Based on the Proposed Dimensions
| Variable 1: The assessment of the quality of the e-learning education programme itself | |
| Dimension | Number of associated indicators |
| 1. Justification of the online programme | 3 |
| 2. Educational goals of the online programme | 5 |
| 3. Student profile | 7 |
| 4. Thematic contents of the online programme | 5 |
| 5. Learning activities | 8 |
| 6. Online teacher's profile | 3 |
| 7. Learning materials | 9 |
| 8. Teaching strategies | 3 |
| 9. Tutoring | 7 |
| 10. Assessment of the students' learning | 4 |
| 11. Quality of the virtual classroom | 9 |
| Variable 2: The ongoing assessment of the online programmes | |
| Dimension | Number of associated indicators |
| 12. Assessment of the initial stage of the online programme | 4 |
| 13. Assessment of the development stage of the online programme | 7 |
| 14. Assessment of the final stage of the online programme | 7 |
| Total | 81 |
The purpose of the indicators of dimension 1 is to assess whether the online programme successfully explains and justifies itself so that students can see that the programme is useful, and can understand it in terms of their learning process and professional development.
The indicators of dimension 2 are proposed as ways of assessing whether the programme defines clear and measurable objectives and whether these objectives are coherent with the syllabus of the programme and are responsive to the demands of the labour market.
The purpose of the indicators of dimension 3 is to assess whether the programme clearly defines its access and graduation profiles, and whether these are made available to the students.
The indicators of dimension 4 assess the quality of the thematic content of the programme. They assess, among other things, whether the content is appropriate, taking into consideration the study load and whether the content is coherent with the educational objectives and the graduation profile of the programme, as well as whether the different topics and subjects are up to date, easily understandable, and arranged in a logical manner.
The indicators of dimension 5 allow for the assessment of whether the learning activities meet the following quality criteria: different typology; contribution to the achievement of the educational objectives of the programme; encouragement of teamwork; definition of guidelines, schedules, deadlines, and delivery methods for each activity.
The indicators of dimension 6 refer to the qualifications necessary to teach the programme. These qualifications include the pedagogical and technological competences required for the online teaching/learning process.
The indicators of dimension 7 serve to assess the quality of the teaching materials. Among other aspects. These indicators assess whether the teaching materials are appropriate, sufficient, up-to-date, motivating, and accessible by students.
The indicators of dimension 8 assess the quality of the teaching strategies applied by the online teacher, focusing on their typology and coherence with the educational objectives of the programme.
The purpose of the indicators of dimension 9 is to assess the quality of the individual and group tutoring sessions carried out by the online teacher. Moreover, they assess whether the teacher provides the students with appropriate feedback for each activity.
The indicators of dimension 10 assess the strategies applied for the continuous assessment of the learning process undertaken by the students. These indicators assess, among other things, whether the programme introduces clear assessment criteria and rules, and whether it applies them while taking into consideration the nature of each learning activity.
The indicators of dimension 11 assess the quality of the virtual classroom and whether it allows to manage all stages of the programme, such as the preparation of the programme content, the implementation of the programme, the assessment of the abilities acquired by the students, and the assessment of the learning process.
The indicators of dimensions 12, 13, and 14 serve to assess the three stages of the programme, that is, its initial, development, and final stages.
By generating a complete model for quality assessment that includes indicators for online education programmes, and allows for the assessment of the pedagogical and technological components of the programme and of its planning, development, and results, we intend to promote an improvement process for the continuous adequacy of the programmes based on their quality objectives.
Being aware of the limitations of this type of paper, below we present a summary of the results of the qualitative validation of the dimensions (Table 7) carried out by the experts who validated them in relation to their univocality (U), suitability (S), and importance (I). We also present some comments and suggestions regarding the indicators of the model.
In order to affirm that a dimension was sufficiently valued by the experts, the following criteria were established:
The type of validity sought was "positive" or "negative." Two scales of validation were used: the dichotomous validation (yes, no) and Likert validation (values from 1 to 4). In this former, a "yes" was considered positive; a "no" was considered negative. In the latter, a 3 or 4 was considered positive; a 1 or 2 was considered negative. Moreover, a high score was considered as a criterion to affirm that a dimension was sufficiently valued by the experts. To that end, a minimum score of 70% of positive responses was established. Qualitative feedback received from the experts, which was not mandatory, was also used for the reformulation of the dimensions.
Table 7
Results of the Qualitative Validation of the Dimensions of the Model
| No | Dimensions | Assessment obtained by the dimension (%) | ||
| U | S | I | ||
| 1. | Justification of the online programme | 92 | 83 | 83 |
| 2. | Educational goals of the online programme | 100 | 94 | 94 |
| 3. | Student profile | 92 | 94 | 86 |
| 4. | Thematic contents of the online programme | 92 | 100 | 100 |
| 5. | Learning activities | 100 | 92 | 100 |
| 6. | Online teacher's profile | 92 | 97 | 97 |
| 7. | Learning materials | 100 | 94 | 94 |
| 8. | Teaching strategies | 83 | 86 | 92 |
| 9. | Tutoring | 92 | 94 | 86 |
| 10. | Assessment of the students' learning | 100 | 94 | 92 |
| 11. | Quality of the virtual classroom | 92 | 97 | 100 |
| 12. | Assessment of the initial stage of the online programme | 100 | 89 | 92 |
| 13. | Assessment of the development stage of the online programme | 92 | 92 | 94 |
| 14. | Assessment of the final stage of the online programme | 100 | 96 | 96 |
As can be seen in Table 5, all 14 dimensions are considered clear, which means that they are precisely presented and understandable by the experts. Moreover, all dimensions are considered as highly suitable, that is, they allow, to a high extent, for the assessment of the quality of the online programme. Furthermore, the experts consider that all dimensions are very relevant, which means that they are considered as crucial for the assessment of the quality of online programmes.
Regarding this point, we would like to note that the differences in the percentages are due to the validation by the experts of the three validation criteria. The experts did not always give the same score for each validation criterion. For example, when it comes to dimension 1, 92% of the experts considered that the dimension was clearly defined (univocality), while only 83% of them considered it suitable and important for the assessment of the quality of the online programme. The percentages show the percentage of positive responses provided for each criterion.
Table 8 shows the qualitative feedback provided by the experts regarding the dimensions of our model.
Table 8
Comments Made by the Experts Regarding the Dimensions of the Model
| No | Dimension | Comments |
| 1. | Justification of the online programme |
|
| 2. | Educational goals of the online programme |
|
| 3. | Student profile |
|
| 4. | Thematic contents of the online programme |
|
| 5. | Learning activities |
|
| 6. | Online teacher's profile |
|
| 7. | Learning materials |
|
| 8. | Teaching strategies |
|
| 9. | Tutoring |
|
| 10. | Assessment of the students' learning |
|
| 11. | Quality of the virtual classroom |
|
| 12. | Assessment of the initial stage of the online programme |
|
| 13. | Assessment of the development stage of the online programme |
|
| 14. | Assessment of the final stage of the online programme |
|
As for the indicators of each dimension, all of them were assessed by the experts as unambiguous in their semantic definitions and relevant with respect to the dimensions in which they were included. Moreover, almost all of them were considered important for the assessment of Higher Education online programmes. Only nine indicators were assessed as unimportant; these were removed from the model:
The validation carried out by the experts allowed us to adjust and improve the model according to the comments made by them, which were incorporated in the definitive model.
The main constraints of this research are:
The issue of quality is a current concern for everyone involved in training and education, both traditional and online (Vlachopoulos, 2016). Some researchers (Grifoll, 2010, Ehlers, 2013; Butcher & Hoosen, 2014) point out that the assessment of online education programmes should be a permanent practice in order to provide a critical view of these programmes, as well as of the aspects relevant to ensuring a high quality of online education programmes.
Nevertheless, the assessment should be complete so that a detailed diagnostic of the current quality of the online programme, and a reflection on what should be modified and what should not be modified to guarantee its high quality, can be accomplished (Eby, 2015). The assessment of online programmes is complete if it is performed from a wide point of view, with the support of periodically collected data, and a strict analysis of this data that is focused on:
The model presented in this paper integrates two kinds of assessment and includes dimensions that serve to carry out the assessment of all of the components of the programme (programme justification, programme objectives, student profile, thematic contents, learning activities, online teacher profile, didactic material, learning strategies, learning assessment, tutorial, online classroom) and the assessment of its planning, application, and final stages. Various indicators were proposed for each dimension as an example of what should be assessed. The application of these indicators depends on the data collected and managed by the university, as well as on the university's available means.
The model presented here is addressed to the persons in charge of implementing online programmes, and to programme directors/coordinators. It can also be applied by online programme designers as a guideline for creating high quality programmes. It can also be useful to those who supervise the quality of Higher Education, as it is a technical document that allows for its revision and the drafting of a final report.
The model also provides a framework for future research, which will need to identify the elements of the model are stable, and those that can be adjusted to the specific context of each university in different countries. And given the variability of the conditions of the quality of online education, there is always a need for research that aims at these kinds of dimension and indicator adjustments.
Agency for the Quality of the University System of Catalonia. (2007). Guide to the self-evaluation of online degree programmes. Guide to institutional evaluation. Barcelona: AQU Catalonia.
Ardila Rodríguez, M. (2011). Calidad de la educación superior en Colombia, ¿Problema de compromiso colectivo? [Quality of Higher Education in Colombia. A collective commitment issue?] Revista Educación y Desarrollo Social, 5(2), pp. 44-55. Retrieved from http://www.unimilitar.edu.co/documents/63968/79819/nii_articulo3.pdf
Arias, F. G. (2012). The research project: Introduction to scientific methodology. Caracas: EPISTEME.
Association for Standardisation of Spain. (2012). Norma UNE 66181:2012. Quality management. Quality of virtual education. Madrid: AENOR.
Barnard, D., & Echols, J. (2015). The anatomy of K-12 online programs: Practical ideas and guidelines. London: Rowman and Littlefield.
Berge, Z., Collins, M., & Dougherty, K. (2000). Design guidelines for web-based courses. In B. Abbey (Ed.), Instructional and cognitive impacts of web-based education (pp. 32-40). London: Idea Group Publishing.
Butcher, N., & Hoosen, S. (2014). A guide to quality in post-traditional online higher education. Dallas, TX: Academic Partnerships.
Cabero Almenara, J., & Llorente Cejudo, M. C. (2013). La aplicación del juicio de experto como técnica de evaluación de las tecnologías de la información (TIC) [The application of expert judgment as an assessment method for information technology (ICT)]. Eduweb. Revista de Tecnología de Información y Comunicación en Educación, 7(2), 11-22. Retrieved from http://servicio.bc.uc.edu.ve/educacion/eduweb/v7n2/art01.pdf
Careaga Butter, M., Meyer Aguilera, E., Graciela Badilla Quintana, M., Jiménez Pérez, L., & Sepúlveda Valenzuela, E. (2017). Quality assurance for postgraduate programs: Design of a model applied on a university in Chile. The International Review of Research in Open and Distributed Learning 18(1), 267-292. doi: http://dx.doi.org/10.19173/irrodl.v18i1.2670
Chmielewski, K. (2013). Diagnoza stanu kształcenia na odległość w Polsce i wybranych krajach Unii Europejskiej [Diagnosis of distance education in Poland and selected EU countries]. Warsaw, PL: Demos Polska. Retrieved from http://www.koweziu.edu.pl/pliki/projekty/p4/diagnoza_st_kno_polska_ue.pdf
Eby, G. (2015). Identification, evaluation, and perceptions of distance education experts. Hershey, PA: IGI Global.
European Foundation for Management Development. (2006). Manual on EFMD CEL quality criteria. Retrieved from http://www.efmd.org/images/stories/efmd/downloadables/Manual_CEL_Quality_Criteria.pdf
European Foundation for Quality in eLearning. (2012). UNIQUe - Europen university quality in eLearning. Retrieved from https://www.efmd.org/index.php?option=com_content&view=article&id=158:unique-europen-university-quality-in-elearning&catid=126:completed-projects
Giorgetti, C. G., Romero, L., & Vera, M. (2013). Design of a specific quality assessment model for distance education. International Journal of Educational Technology in Higher Education 10(2), 301-315. doi: http://dx.doi.org/10.7238/rusc.v10i2.1742
Grifoll, J., Huertas, E., Prades, A., Rodríguez, S., Rubin, Y. ... Ossiannilsson, E. (2010). Quality Assurance of Online Education. Helsinki, FI: ENQA.
Hernández, M., García, M., Padilla, G., García, J., & Leal, F. (2012). Modelos de la tutoría virtual [Models of online tutoring]. In G. Padilla, F. Leal, M. Hernández & J. Cabero (Eds.), Un reto para el profesor del futuro: La tutoría virtual (65-90). La ciudad de México: SINED.
Higher Education Compliance and Quality. (1997). Guidelines for distance education. London: HECQ.
Instituto Latinoamericano y del Caribe de Calidad en Educación Superior a Distancia. (2010). Guía de autoevaluación para programas de pregrado a distancia. [Self-assessment guide for undergraduate distance learning programmes]. Loja: CALED.
Jung, I., & Latchem, C. (2012). Quality assurance and accreditation in distance education and online. New York/London: Routledge.
Khan, B. H. (2007). Program evaluation in e-learning. Unpublished manuscript. Retrieved from http://asianvu.com/bk/elearning_evaluation_article.pdf
Marciniak, R. (2016). Autoevaluación de programas de educación universitaria virtual [Self-assessment of higher virtual education programmes] (Doctoral dissertation). Autonomous University of Barcelona, Barcelona, Spain.
Marshall, S. (2006). eLearning maturity model. Version two. Process guide. Wellington, AU: Victoria University of Wellington.
Martínez Mediano, C. (2013). Assessment of the programmes, models and procedures. Madrid: UNED.
Marúm-Espinosa, E. (2011). Quality of service in higher distance education. A perspective from Mexico. Revista Iberoamericana de Educación a Distancia, 14(2), 49-62.
Meza, J. (2012). Modelo pedagógico para proyectos de formación virtual. [A pedagogical model for online training projects]. Bonn, DE: GIZ.
Ministry of Education, Culture and Sport. (2016 ). Educación superior universitaria. [Higher education]. Retrieved from http://www.mecd.gob.es/educacion-mecd/
National Agency for Quality Assessment and Accreditation of Spain. (2012). Guía de autoevaluación: Renovación de la acreditación de títulos oficiales de grado, máster y doctorado programa ACREDITA [Self-assessment guide: renewal of the certification of official undergraduate degrees, master's degrees and PhD degrees. ACREDITA program]. Madrid: ANECA.
National Agency for Quality Assessment and Accreditation of Spain. (2015). Guía de Autoevaluación: renovación de la acreditación de títulos oficiales de Grado, Máster y Doctorado Programa ACREDITA [Self-assessment guide: renewal of the certification of official Undergraduate degrees, Master's degrees and PhD degrees ACREDITA program]. Madrid: ANECA. Retrieved from http://www.aneca.es/Programas/ACREDITA/
National Agency for Quality Assessment and Accreditation of Spain. (2017). Programas de Evaluación [Assessment Programmes]. Retrieved from http://www.aneca.es/Programas-de-evaluacion
Online Learning Consortium. (2002). Five pillars of quality online education. Retrieved from https://onlinelearningconsortium.org/about/quality-framework-five-pillars/
Ossiannilsson, E., & Landgren, L. (2012). Quality in e-learning - A conceptual framework based on experiences from three international benchmarking projects. Journal of Computer Assisted Learning. Special Issue on Quality in e-learning 28(1), 42–51. doi: https://doi.org/10.1111/j.1365-2729.2011.00439.x
Padilla Vargas, I. (2005). Educación a distancia: Ofrecimientos con calidad y eficacia [Distance education: high quality and effective offers.]. Retrieved from http://www.uprm.edu/ideal/hermes2005/calidad.pdf
Pérez Juste, R. (1986). Pedagogía experimental. La medida en educación. [Experimental pedagogy. Measurements in education]. Madrid: UNED.
Pérez Juste, R. (2014). Evaluación de programas educativos [Assessment of education programes]. Madrid: La Muralla.
Petkova, Y., & Radeva, D. (2014). A model and an index for e-learning quality assessment. Proceedings of the International Conference on e-Learnin g, 2014, 176-181. Retrieved from http://elearning-conf.eu/docs/cp14/paper-26.pdf
Raeesi, I., Qorbani, D., & Akhgar, B. (2010). A conceptual model for measuring the quality of e-learning through knowledge sharing indicators. Proceedings of the 2010 International Conference on Information Knowledge Engineering (IKE), 2010, USA, 383-387. Retrieved from https://pdfs.semanticscholar.org/7df3/365eb74a07122aad2740a99150564b0ba223.pdf
Rushby, N., & Surry, D. (2016). Wiley handbook of learning technology. New York: John Wiley and Sons.
Shattuck, K. (2014). Assuring quality in online education: Practices and processes at the teaching, resource, and program levels. Sterling: Stylus Publishing.
Spanish Ministry of Education, Culture and Sports. (2016). Datos y cifras del sistema universitario español. Curso 2015-2016 [Data and numbers of the university system of Spain. Academic year 2015-2016.]. Madrid: Ministerio de Educación, Cultura y Deporte. Retrieved from http://www.mecd.gob.es/dms/mecd/servicios-al-ciudadano-mecd/estadisticas/educacion/universitaria/datos-cifras/datos-y-cifras-SUE-2015-16-web-.pdf
Surry, D. W., & Ensminger, D. C. (2009). Supporting the implementation of online learning. In P. L. Rodgers (Ed.), Encyclopedia of distance learning, (1994-1999). New York: IGI.
University of Wisconsin. (2008). Logic model. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
Veytia Bucheli, M., & Chao González, M. (2013). Las competencias como eje rector de la calidad educativa [Competences as the guiding principle for the quality of education.]. Revista electrónica de Divulgación de la Investigación, 4. Retrieved from http://mistareas.com.mx/redi/4/pdf/SABES_4_2MARIAPDF_V1.pdf
Vlachopoulos, D. (2016). Assuring quality in online course design: The roadmap. The International Review of Research in Open and Distance Learning 17(6), 183-205. doi: http://dx.doi.org/10.19173/irrodl.v17i6.2784
Zhang, W., & Cheng, Y, L. (2012). Quality assurance in e-learning: PDPP evaluation model and its application. The International Review of Research in Open and Distance Learning , 13 (3), 66-82. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1181/2205
| Variable 1: The assessment of the quality of the education online programme itself | ||
| No | Dimension | Indicator |
| 1 | Online programme justification | The online programme has been created in relation to the educational needs existing within the society. |
| The online programme responds to the labour market. | ||
| The reasons that justify the necessity of enrolling into the programme by the student are duly explained. | ||
| 2 | Educational goals of the online programme | The educational goals of the programme are adequate to the demands of the labour market. |
| The educational goals are drafted based on the skills to be acquired by the students after completing the programme. | ||
| The educational goals are coherent with the contents of the programme. | ||
| The educational goals of the programme are measurable. | ||
| The educational goals of the programme are available and accessible to all persons interested in the programme. | ||
| 3 | Access and graduation profile | Access profile |
| The access profile has been designed. | ||
| The access profile describes the previous education required in order to enrol in the online programme. | ||
| The access profile is accessible to all persons interested in the online programme. | ||
| Graduation profile | ||
| The programme contains the graduation profile. | ||
| The graduation profile has been defined in terms of the competences to be acquired by the students after completing the programme. | ||
| The graduation profile has been updated according to the needs. | ||
| Verification of whether the graduation profile of the programme is accessible to all persons interested in the programme. | ||
| 4 | Thematic contents of the online programme | The thematic contents of the programme are appropriate taking into account the subject of the programme. |
| The thematic contents of the programme are proportional to the study load indicated in the syllabus. | ||
| There is coherence between the thematic contents of the programme and its educational goals. | ||
| The themes and subthemes of the thematic contents of the programme are arranged in a logical manner. | ||
| The thematic contents of the programme are reviewed on an annual basis. | ||
| 5 | Learning activities | The learning activities proposed within the programme are of different types. |
| All the activities are coherent with the educational goals. | ||
| Instructions for each learning activity have been drafted. | ||
| A schedule of the activities has been provided. | ||
| A deadline for the delivery of each activity has been set out. | ||
| The method and format of delivery has been described for each learning activity. | ||
| The proposed activities promote collaborative learning. | ||
| The volume of the learning activities is suitable to the teaching load of the programme. | ||
| 6 | Online teacher profile | The online teacher has the appropriate professional profile according to the requirements of the programme subject. |
| The online teacher has the appropriate pedagogical competences in order to carry out the online teaching/learning process. | ||
| The online teacher has the technological competences necessary to carry out the teaching/learning process. | ||
| 7 | Educational material | The resources made available to the students are of different types. |
| The programme offers the basic learning resources. | ||
| The programme offers complementary learning resources. | ||
| The programme contains a set of Web-based learning resources. | ||
| All the learning resources are coherent with the educational goals of the programme. | ||
| The learning resources of the programme have been selected based on clear selection criteria. | ||
| The basic bibliography of the programme is described. | ||
| The complementary bibliography of the programme is described. | ||
| The basic and complementary bibliography is updated (30% of the bibliographical suggestions is dated no longer than five years prior). | ||
| 8 | Educational strategies | The online teacher uses different types of teaching strategies. |
| The teaching strategies used by the online teacher are coherent with the educational goals. | ||
| The online teacher promotes different learning strategies among the students. | ||
| 9 | Tutoring | The functions of the online teacher and of the persons involved in the development of the programme are defined and accessible by students. |
| The online programme includes instructions regarding the communication methods with the online teacher. | ||
| The timetable of the tutoring sessions has been defined. | ||
| The online teacher carries out individual tutoring sessions. | ||
| The online teacher carries out group tutoring sessions. | ||
| The tutoring sessions carried out by the online teacher are monitored. | ||
| The students receive feedback regarding each task. | ||
| 10 | Assessment of students' learning | Different strategies for the assessment of the students' learning process have been defined. |
| The assessment criteria of the learning process are accessible by the students. | ||
| The criteria to be used to grade the students' progress are detailed and accessible by the students. | ||
| The students participate in the process of determining and assessing the achieved progress. | ||
| 11 | Quality of the virtual platform | The virtual platform includes a tool to submit activities or files. |
| The virtual platform offers tools that allow to manage the learning activities. | ||
| The virtual platform includes tools that allow to create group tasks. | ||
| The virtual platform contains tools that allow the students to create their own personal learning environments. | ||
| The virtual platform offers tools for the asynchronous communication. | ||
| The virtual platform contains tools for the synchronous communication. | ||
| The students can view the results of the completed exercises and/or exams in the virtual platform. | ||
| The virtual platform contains a tool that allows the online teacher to monitor and manage the students. | ||
| The virtual platform contains a section describing the functionalities of all the tools available in itself. | ||
| Variable 2: The ongoing assessment of the education online programme | ||
| No | Dimension | Indicator |
| 12 | Initial assessment of the online programme | The programme is ready to be launched at least one week prior to its start. |
| All the resources (human, financial and technical) are sufficient to guarantee the quality of the development of the programme. | ||
| All the persons involved in the programme are ready for it to start. | ||
| The virtual classroom is ready to be used one week prior to the programme being launched. | ||
| 13 | Processual assessment of the online programme | The teaching strategies used by the online teacher are appropriate in order to achieve the educational goals of the programme. |
| The learning activities are appropriate for the students to acquire the competences described in the graduation profile. | ||
| The planning of the programme regarding the activities, the deadlines and the resources is met. | ||
| The online teacher is involved in the execution of the programme. | ||
| The teaching materials and resources are suitable to the students' expectations. | ||
| Student's motivation is stimulated. | ||
| Some of the programme goals have been achieved. | ||
| 14 | Final assessment of the online programme | Level of achievement of the educational goals set out for the online programme. |
| Level of fulfilment of the planned learning activities. | ||
| Level of students' satisfaction with the online programme. | ||
| Level of online teachers' satisfaction. | ||
| Impact rate of the online programme | ||
| Performance rate of the online programme. | ||
| Success rate of the online programme. | ||
Quality Assurance for Online Higher Education Programmes: Design and Validation of an Integrative Assessment Model Applicable to Spanish Universities by Renata Marciniak is licensed under a Creative Commons Attribution 4.0 International License.