A System of Indicators for the Quality Assessment of Didactic Materials in Online Education by Renata Marciniak and Cristina Cáliz Rivera is licensed under a Creative Commons Attribution 4.0 International License.
Volume 22, Number 1
Renata Marciniak1 and Cristina Cáliz Rivera2
1Faculty of Social Sciences at Manresa, University of Vic - Central University of Catalonia (UVic-UCC), 2Euncet Business School, Universitat Politècnica de Catalunya
The quality of didactic materials is a source of concern for teachers, users, and educational institutions that offer online education. There is a lack of indicators to help assess the quality of three key types of didactic materials commonly used in online education: didactic units (i.e., materials that contain program contents), didactic guides (i.e., materials that provide information), and additional didactic materials (materials to deepen knowledge). The objective of this article is to present a system of indicators designed to assess the quality of these types of didactic materials and guide their creation process. The system was developed based on a critical analysis of existing models designed to assess the quality of digital didactic materials. The system was validated by 16 international experts in online education, and a trial application of the system assessed five didactic guides and didactic units used by online universities in three different countries. Results of the validation process were triangulated with relevant literature, allowing the authors to make decisions regarding changes to the system in terms of maintaining, reformulating, or removing indicators. The resulting system comprises 43 assessment indicators and serves as a guide for designers, teachers, and users in the creation and selection of didactic materials for use in online education and in the assessment of their quality.
Keywords: online education, didactic materials, quality, indicators, system, assessment
There is no doubt that didactic materials are an essential factor in online education, where teachers cease to be the main transmitters of knowledge. In online education, didactic materials are the most important source of information, as Bautista Liébana et al. (2001) assert, “although all teaching uses them, online education in particular depends mainly on the materials to such a point that it cannot be understood without them” (p. 6). Didactic materials are the main instrument for the transmission of knowledge to students. These materials can also promote independent learning and influence the quality of the educational process. However, in order for didactic materials to fulfill these functions, they must be sufficient in quantity and quality.
In online education, there are a variety of didactic materials and different ways of classifying them (Shattuck, 2014). For example, Blanco Gil et al. (2010) classify didactic materials as conventional materials (e.g., textbooks, digital boards, and worksheets), audio-visual materials (e.g., slides, videos, and audio recordings), new technologies (e.g., videogames, presentations, interactive multimedia, Web pages, WebQuest, forums, and Wikis), materials used to present information (i.e., didactic units), materials to organize information (e.g., tutorials, guides, conceptual outlines, summaries, overviews, and flowcharts), and materials to develop skills (e.g., examples, analogies, questions, and exercises to apply content). In the development of the quality assessment system, we focused our attention on three types of didactic materials: didactic guides, didactic units, and additional didactic materials, as they are the most commonly used in online education (see García Aretio, 2014).
We define a didactic unit as a structured and organized material that contains all of the contents of a program (see García Martín et al., 2010). A didactic guide refers to all materials that provide information to familiarize students with the subject so that “they do not have to guess or look for clues” (Asinsten, 2011, p. 6), such as information about what they are going to study and for what purpose; how, when, with what, and with whom they will study; and how evaluation will be conducted. Additional didactic materials (also called complementary materials) constitute materials that allow students to deepen their knowledge, the reading of which is optional. According to Blanco Gil et al.’s (2010) classification, these materials may include conventional and audiovisual materials, new technologies, and materials to develop skills.
When these three types of didactic materials are of high quality, they contribute to improving students’ learning processes, as Hyla (2016) points out, “the main factor that contributes to the learning process in online education is the quality of the didactic materials” (p. 152). According to the Spanish Association for Standardization and Certification (AENOR, 2017), a didactic material of high quality “is capable of satisfying the needs of its users. This means, when it comes to online education (i.e., digital environments), that it is effective from the didactic, technological, and accessibility points of view” (p. 45). In order for the didactic materials to be of high quality, the professionals involved in the creation process (e.g., designers, teachers) must consider their quality from the very beginning. However, practice shows that educators often lack knowledge, guidelines, and specialized literature on how to create or select high quality didactic materials and which indicators to use to evaluate them (AENOR, 2017; Hyla, 2016; Marciniak, 2017; Padrón et al., 2006; Rushby & Surry, 2016). This gap has not been addressed so far (or has only been partially addressed), as scholars such as Rodríguez Rodríguez and Martinez Bonafé (2016) note:
A significant portion of recent research seems to highlight the lack of knowledge that currently exists when it comes to models and assessment guides for didactic materials, both in printed and digital formats, and their use in the educational context (p. 8).
A number of scholars and organizations have proposed different models to assess the quality of online education and the didactic materials used in this type of education (Marciniak, 2018; Vlachopoulos, 2016), such as Nielsen and Morkes (1998), Nesbit et al. (2002), Bravo Ramos (2005), Opdenacker et al. (2009), Marzal et al. (2008), Morales Morgado et al. (2010), Fernández-Pampillón Cesteros et al. (2012a), the National Distance Education University of Spain (UNED, n.d.), and AENOR (2017). These models focus on various contextual dimensions and include indicators that should be considered when designing or selecting didactic materials; however, the models combine a diversity of approaches and, at times, respond to contradictory and opposing paradigms. As a result, the proposed indicators refer to the quality of divergent didactic materials and, moreover, are assigned different meanings (Marciniak, 2016). In addition, the literature focuses on some aspects of online didactic materials (e.g., usability, visibility, flexibility, ease of use, and pedagogical and graphic design) but rarely highlights the need to assess the quality of the three types of didactic materials (i.e., didactic unit, didactic guide, and additional didactic materials). To address this gap, we conducted a study with the aim of designing a system of indicators applicable to the creation of quality online didactic materials, as well as their evaluation and improvement. As Fernández-Pampillón Cesteros et al. (2012b) propose,
In order to definitively address and promote the development of high quality digital didactic materials, it is essential to have systems to evaluate and recognize the quality of the teacher’s teaching production in addition to those already in place for scientific production. These systems must necessarily be based on quality assessment models and tools that are easy to use both by the teachers who create these materials and by the evaluators who are also usually teachers (p. 26).
Illum Hansen and Toke Gissel (2017) assert,
When we talk about good didactic materials and the quality of didactic materials, this presupposes that there is a number of fairly stable factors that apply to those who will be using the didactic materials (students, teachers and educators), and the contexts in which they will be used (teaching and lesson situations). In other words, it is implicit that they have been judged on specific didactic functions and purposes in a particular context (p. 129).
To address these challenges, several scholars have developed models related to the quality assurance of didactic materials intended for online education. Most of the models approach the evaluation of these materials from different perspectives. Table 1 presents a comparative analysis of 10 of the models analyzed in the bibliographic research conducted for this study. The analysis is presented in chronological order.
Table 1
Comparative Analysis of 10 Models Developed to Assess the Quality of Didactic Materials
| Author/Year | Model description | Assessment dimensions |
| Nielsen and Morkes (1998) | A collection of general rules for evaluating the usability of digital didactic materials. |
|
| Nesbit et al. (2002) | The learning object review instrument (LORI) model is a tool for evaluating and commenting on virtual teaching resources. The model aims to facilitate and support the evaluation of multimedia materials. |
|
| Bravo Ramos (2005) | A guide for evaluating materials with recommendations for their creation. |
|
| Espinoza and González (2006) | An evaluation tool that includes basic factors for consideration in the evaluation of printed didactic materials in a virtual context. |
|
| Opdenacker et al. (2009) | The quality assessment of digital educational material (QuADEM) model provides a comprehensive method for evaluating the quality of digital didactic materials used in blended learning. |
|
| Marzal et al. (2008) | A model based on a cognitive perspective of information processing and expected student competencies. |
|
| Morales Morgado et al. (2010) | The reusable learning teaching objects assessment tool (HEODAR) was developed at the University of Salamanca for the comprehensive assessment of didactic materials. |
|
| Fernández-Pampillón Cesteros et al. (2012a) | A model to assess the potential quality of a digital didactic material before its use by real users. The model comprises 10 criteria and includes an application guide. It can be applied in peer assessment, collaborative assessment, or self-assessment. |
|
| National Distance Education University of Spain (UNED, n.d.) | A protocol for the evaluation of distance didactic materials addressed to UNED students to obtain their opinions on the quality of didactic materials used during a course. |
|
| Spanish Association for Standardization and Certification (AENOR, 2017) | The UNE 71362 Standard is a model to define and evaluate, quantitatively and qualitatively, the quality of digital didactic materials. It includes 15 criteria to measure the didactic effectiveness of materials, their technological effectiveness, and their accessibility. |
|
The analysis presented in Table 1 shows that the 10 models focus on different aspects of the evaluation of online didactic materials, such as graphic and pedagogical design, suitability to recipients, content quality, learning activities, ease of use, visibility, accessibility, and flexibility. The authors of the analyzed models also propose evaluating other aspects that, in their opinion, define the quality of the materials to a greater or lesser degree. Yet, these models have certain disadvantages. One disadvantage is the lack of consensus among the models regarding the number of dimensions and indicators. Some models evaluate only four dimensions, while others propose up to 15. Furthermore, the models assign different meanings to the indicators used to assess these dimensions.
An in-depth study of the indicators proposed in these models revealed that their meanings differ depending on their author and the methodology used. While, for example, Nesbit et al. (2002) equate the “content quality” indicator with truthfulness, accuracy, a balanced presentation of ideas, and an adequate level of detail; Bravo Ramos (2005) regards content quality as both the scientific rigor of the expressive resources and techniques they use. According to Opdenacker et al. (2009), the authors of the QuADEM model, this indicator assesses the correspondence of a didactic material with the training objectives and the suitability of the material to its recipients.
It is important to note that none of the analyzed models proposes specific indicators to assess the quality of the three types of didactic materials commonly used in online education: didactic guides, didactic units, and additional materials. Although some of the proposed indicators can serve as a frame of reference to assess elements of these materials, they do not allow for a comprehensive evaluation of them. To address the disadvantages described above, it is necessary to develop a system of indicators to help those involved in online education (e.g., designers, teachers, or users) to create or select the three types of didactic materials and assess their quality.
The design of our indicator system was based on bibliographic research of existing models and standards used to evaluate didactic materials intended for both traditional and online education. Based on the results of this analysis, we identified the indicators that should be considered when creating or evaluating the didactic units, didactic guides, and additional materials that online teachers use to enrich the teaching and learning process. The results of the bibliographic research also helped us build a system of 45 indicators to measure each of these types of materials.
As the system was the result of our elaboration on existing models, it was essential to validate it through a methodological procedure to determine its reliability in terms of the degree to which the proposed indicators evaluate what they were expected to evaluate. For this purpose, a panel of 16 experts from different countries (i.e., Chile, Italy, Mexico, Paraguay, Poland, Portugal, Spain, and the United States) were selected based on their experience in the field of online education didactic materials. The experts included researchers, online teachers and students, and members of the different accreditation agencies for the quality of online higher education. The experts were asked to determine the validity of each of the indicators of the proposed system based on an evaluative judgment of its univocity, relevance, and importance. The quantitative and qualitative validation of the model was conducted based on the results of the expert panel. The qualitative validation was conducted based on a compilation of the experts’ comments on the reasons for their validation and their suggestions to improve the system.
To augment the experts’ qualitative validation, a trial application of the proposed system of indicators was conducted. The objective of the pilot was to delimit the reliability of the system, to determine the degree of internal consistency for each of the indicators and for the entire system. The trial application consisted of assessing five didactic guides and didactic units and a set of additional materials from five academic programs (i.e., fundamentals of administration, effective communication in health, quality systems, communication and financing management, and project management) offered by three online universities in different countries (i.e., Mexico, Poland, and Spain). Access to the virtual platforms of the participating universities was obtained, and the suitability of most of the system indicators was verified in the different contexts. Data was collected using assessment protocols for each material, which served as a guideline for observation and evaluation. By way of example, Table 2 shows a fragment of the protocol for the assessment of didactic unit.
Table 2
A Fragment of an Assessment Protocol
| Indicator | Assessment | Comments | |
| Yes | No | ||
| The didactic unit presents the objectives of the learning process to be achieved by its end. | X | The didactic unit does not present the educational objectives of the learning process. | |
| The didactic unit contains a set of learning activities. | X | The didactic unit contains a wide range of learning activities, which include, among others: drafting of essays, drafting of comparative tables, case studies, and watching and commenting on videos. | |
Results of the trial application of the system were triangulated with the experts’ validation and relevant literature, which allowed us to make decisions regarding each of the system indicators: whether they should be maintained, modified, or eliminated (see example in Table 3). This process resulted in a definitive system of 43 indicators for the assessment of the quality of didactic materials for online education.
Table 3
Triangulation of Results for Indicator 29: The Didactic Guide Includes the Thematic Contents of the Online Training Program
| Quantitative validation | Qualitative validation | Relevant literature |
| Indicator with high levels of validity across quantitative indexes: average index = 0.80; average CVR index = 1; average Fleiss Kappa index = 0.83. | Universities: All of the universities used this indicator. Experts: Indicator with maximum validity in all criteria. | García Martín et al. (2010) affirm that didactic guides must provide information on the contents of the subject and its grouping into didactic units. According to National Agency for Quality Assessment and Accreditation (ANECA, 2019), teaching guides should contain a description of each subject (e.g., objectives, competencies, bibliography, and syllabus), training activities, and evaluation systems. |
Note. Researchers’ decision: The indicator is maintained.
The system includes a total of 43 assessment indicators. Table 4 presents the system structure, in terms of the distribution of indicators across the didactic material types.
Table 4
Distribution of Indicators to Assess the Quality of Didactic Materials for Online Education
| Didactic material | Indicators (n) |
| Didactic unit | 23 |
| Didactic guide | 11 |
| Additional didactic materials | 9 |
| Total | 43 |
The 23 indicators chosen to assess the quality of didactic units allow us to assess, among other factors, whether the title of the material is clear and whether the material includes an index and an introduction, presents the learning objectives, contains all content on the subject, is well structured and organized, and presents basic and additional bibliography and other supporting elements. The 11 indicators proposed to assess the quality of didactic guides allow us to determine whether the material provides students with information about what they are going to study, why, how, when, with whom; and how they can evaluate what they have learned. In addition, these indicators allow us to assess whether the didactic guide includes all of the information necessary to familiarize students with the elements of the program and whether these elements are organized in a manner that allows for meaningful learning experiences. The nine indicators chosen to assess the quality of additional didactic materials allow us to assess whether there are additional materials on the subject, of different types, that are suitable for online education; and whether they are sufficient in quantity and quality. Furthermore, these indicators assess the technical aspects for all types of materials, such as ease of use, interactivity, navigability, interoperability, and accessibility. Table 5 presents the system of indicators chosen to assess the quality of the three types of didactic materials mentioned above.
Table 5
The System of Indicators for Quality Assessment of Didactic Materials in Online Education
| No. | Indicator | Description |
| Didactic unit | ||
| Title | ||
| 1 | The title of the didactic unit refers to its most relevant content. | The title of the didactic unit must make reference to its most relevant aspects. |
| 2 | The title of the didactic unit is easy to understand. | The title of the didactic unit must be clear and easy to understand, use simple language, and clear and direct terms according to the audience and the subject terminology. |
| Index | ||
| 3 | The index of the didactic unit is comprehensive. | The index must indicate the exact location of each topic and subtopic of the unit for the student (i.e., it is comprehensive). |
| 4 | The index of the didactic unit indicates the page on which each subject is explained. | The index should indicate the pages of each topic to facilitate navigation by the student. |
| Introduction | ||
| 5 | The didactic unit contains an introduction. | The didactic unit must contain an introductory text that helps familiarize the student with the subject, lexicon, and other aspects of the content to be studied. |
| 6 | The didactic unit contains a summary of its content. | The introduction should briefly explain each part of the unit (i.e., its topics) and present the objectives; expected results of the learning process; or, at least, its purpose. |
| 7 | The didactic unit briefly explains each part of the unit. | The introduction must capture the student’s attention from the start. It must invite the reader to continue reading. |
| Learning objectives | ||
| 8 | The didactic unit presents the objectives of the learning process to be achieved by its end. | The didactic unit must present the educational objectives, clearly describing what is to be achieved upon completion. |
| 9 | All the educational objectives of the didactic unit are appropriate to the student’s graduation profile. | All objectives of the didactic unit must be appropriate to the student’s graduation profile in terms of cognitive, procedural, and attitudinal competencies. |
| Development of content | ||
| 10 | A minimum of 25% of the presented information must have been generated in the past 5 years. | The contents of the didactic unit must be updated based on new advances in research, regulations, and current knowledge. A minimum of 25% of the references cited must have been published in the past 5 years. |
| 11 | The contents of the didactic unit match its learning objectives. | The contents of the didactic unit must be consistent with the learning objectives set therein. |
| 12 | The contents of the didactic unit are complete and guarantee the achievement of the learning objectives of the online program. | All contents of the didactic unit must be complete and comprehensively present the subject, while citing research, work, or publications on the subject. |
| 13 | The contents of the didactic unit include multimedia resources. | The contents of the didactic unit must take advantage of multimedia technology, rationally combining text with photos and/or audio, images, and videos, among other media. |
| 14 | The contents are presented in a visually attractive manner and attract the student’s attention. | The contents of the didactic unit must attract the student’s attention. The contents should be visually attractive (i.e., they should include charts, figures, and graphs in color). |
| 15 | The contents of the didactic unit respect copyright laws. | Plagiarism should be avoided in the contents of the didactic unit. Authors should not be cited without presenting the source of the cited information. |
| 16 | The didactic unit contains a closing summary of the contents. | The didactic unit must contain information that summarizes all of the studied subjects. |
| Bibliography | ||
| 17 | The didactic unit contains a basic bibliography. | The didactic unit must contain the references of the texts cited therein. |
| 18 | The didactic unit contains an additional bibliography. | The bibliography must be divided into basic bibliography and additional bibliography, so that the student can expand their knowledge on the topics covered in the unit. |
| 19 | A minimum of 25% of the recommended bibliography should have been published in the past 5 years. | A bibliographic review should identify what is currently known on the subject. A minimum of 25% of the additional bibliography should have been published in the past 5 years. |
| Other elements to support the learning process | ||
| 20 | The didactic unit contains a glossary. | At the end of the didactic unit, basic and new terms introduced in the unit must be defined and clarified. |
| 21 | The didactic unit contains a set of learning activities. | The didactic unit must propose activities in order for the student to verify the knowledge they have acquired. |
| 22 | The didactic unit contains a set of self-assessment activities. | The didactic unit must include self-assessment activities that allow the student to determine their progress in the learning process. |
| 23 | The didactic unit contains solutions to the self-assessment activities. | The didactic unit must contain solutions to the self-assessment activities, so that the student can resolve any doubts when completing the activities. |
| Didactic guide | ||
| 24 | The didactic guide presents information that allows the student to identify the program of the subject. | The cover of the didactic guide must indicate the title of the program, the center, the type (compulsory or optional), the number of European Credit Transfer and Accumulation System (ECTS) credits, the career (or training cycle) in which the program is included, and the online teacher in charge of the subject. |
| 25 | The didactic guide contains an index. | The didactic guide must include an index of the topics and the page numbers of each topic. |
| 26 | The didactic guide introduces the teacher(s) responsible for the development of the program. | The didactic guide must present the teacher(s) who will develop the online program. The presentation must include information, such as full name, academic training (bachelor’s, master’s, and doctoral degrees), and area of research. |
| 27 | The didactic guide includes an introduction of the program. | The didactic guide must briefly summarize the fundamental aspects of the online program. |
| 28 | The didactic guide presents the learning objectives and results to be achieved by the completion of the program. | The didactic guide must present the objectives and expected results, clearly describing what is to be achieved by the completion of the program. |
| 29 | The didactic guide includes the thematic contents of the program. | The didactic guide must include an index of the thematic contents of the program and an outline of the contents and develop the themes (or modules) of the program in more detail. |
| 30 | The didactic guide includes the teaching-learning methodology that will be implemented throughout the program. | The didactic guide must describe the methodology of the online program, the teaching and learning strategies, and the didactic materials that will be used and that will be available to the student throughout the program. |
| 31 | The didactic guide presents the learning activities that the student must complete to pass the program. | The didactic guide must include the learning activities, their schedule, and the methods of delivery for the student. |
| 32 | The didactic guide includes assessment strategies of the student’s progress. | The didactic guide must describe all assessment strategies of the student’s progress. In addition, evaluation and qualification criteria must be presented. |
| 33 | The didactic guide includes the basic and additional bibliography of the program. | The didactic guide must contain the bibliography of the program divided into basic (compulsory) and additional bibliography. |
| 34 | The didactic guide presents instructions on the forms of communication with the online teacher and tutoring schedule. | The didactic guide must provide students with specific instructions on the means of communication with the online teacher and the virtual and face-to-face tutoring schedule (if applicable). |
| Additional didactic materials | ||
| 35 | The course program offers a set of additional didactic materials. | Additional materials consist of materials that allow the student to deepen their knowledge on some of the studied topics. Reading these materials is optional. |
| 36 | The syllabus of the subject indicates a set of Web-based didactic materials. | In online education, a set of Web-based (or digital) didactic materials should be used to facilitate the teaching-learning process. |
| 37 | All of the program’s didactic materials have been chosen based on clear selection criteria. | Didactic materials should be chosen according to well-defined selection criteria. |
| 38 | All didactic materials are consistent with the learning objectives of the program. | The correspondence of the materials with the learning objectives indicates their validity. The materials are valid when they favor the desired learning process and enable the achievement of the results expressed in the learning objectives. |
| Technical aspects for all types of materials | ||
| 39 | Ease of use | The materials should be easy to use. |
| 40 | Functioning | All materials should function well. |
| 41 | Navigation | The materials must allow the student to comfortably navigate from one to the other depending on the course format. |
| 42 | Interoperability | The materials must allow use in multiple environments and computer systems. |
| 43 | Accessibility | The materials must be accessible from the virtual campus. They must also be adapted to students with visual, auditory, or motor disabilities in order to allow them to study. |
Given the limited scope of this paper, we present a summary of the qualitative validation of the indicators by the experts who determined the validity of each of the proposed indicators based on an evaluative judgment of their univocity, relevance, and importance. We also present some of the experts’ comments regarding the indicators.
The results indicate that the experts considered all of the indicators to be univocal in their semantic definitions and relevant to the dimensions in which they were included. However, based on the experts’ judgments of the importance of the indicators, two of the 45 indicators were eliminated because they were not considered to impact the quality of didactic materials in online education: “Variety of didactic materials and resources” and “Interactive content elements of the didactic unit.” According to the experts, a greater variety is not a sign of the high quality or success of didactic materials; and students are not always required to interact, as sometimes a linear behavioral structure is designed so that students must follow from beginning to end. Table 6 presents some of the experts’ comments regarding the indicators.
Table 6
Results of the Qualitative Validation of the System Indicators: Experts’ Comments
| Indicator | Experts’ comments |
| 1 |
|
| 2 |
|
| 4 |
|
| 5 |
|
| 6 |
|
| 9 |
|
| 10 |
|
| 13 |
|
| 14 |
|
| 15 |
|
| 17 |
|
| 21 |
|
| 25 |
|
| 27 |
|
| 28 |
|
| 30 |
|
| 31 |
|
| 32 |
|
| 33 |
|
| 35 |
|
| 36 |
|
| 38 |
|
The experts’ validation allowed us to refine and improve the system in accordance with their comments.
The quality of didactic materials is the primary factor that shapes the perception of online learning and content engagement among online course participants. In order for didactic materials to be of high quality, the persons involved in developing them (e.g., authors, designers, and teachers) should consider the quality aspect from the outset of the development process. Online education applies a variety of didactic materials; however, online teachers, students, and designers who create, modify, and use these materials may not know whether they are of high quality and whether they meet the minimum requirements for online education (Marciniak, 2018; Vizoso, 2018).
A number of authors have designed models to help create and assess didactic materials for online education. However, “there is no basic agreement among them regarding the indicators to be applied when creating and assessing these models” (AENOR, 2017, p. 45). Furthermore, the models do not include specific indicators to assess all of the elements of the didactic materials used in online education. A system of indicators to assess these didactic materials in a comprehensive manner is, therefore, needed. The system of 43 indicators presented in this paper allows users to assess the quality of three types of didactic materials most commonly used in online education: didactic guides, didactic units, and additional didactic materials. The trial application of this system to assess five online didactic materials used by universities in different countries allowed us to verify its utility and great potential to improve the quality of such materials. However, the system is not static; it allows for dynamic implementation based on its evolution through adaptation, removal, and/or incorporation of new indicators. Application of the system depends on the needs of each university; and, it can be applied by designers, teachers, and users as a guide to create, select, and assess the quality of the three types of didactic materials used in online education.
The proposed system provides a framework for future research. Some of the aims of this research are,
In all research processes, it is common to encounter constraints that the researcher must overcome in order to achieve the research objectives. A number of constraints were encountered in this research:
Nonetheless, the difficulties listed above did not hinder the development of the research or limit its thoroughness, and thus the results of the research are supported by the framework of the analysis conducted.
Asinsten, J. C. (2011). Producción de material didáctico en ambientes virtuales de aprendizaje [Production of didactic material in virtual learning environments]. Virtual Educa. https://virtualeduca.org/documentos/manual_del_contenidista.pdf
Bautista Liébana, J. R., Martínez Romero, R., & Sainz Ibáñez, M. (2001). La evaluación de materiales didácticos para la educación a distancia [The evaluation of didactic materials for distance education]. Revista Iberoamericana de Educación a Distancia, 4(1), 73-95. http://revistas.uned.es/index.php/ried/article/view/1192/1095
Blanco Gil, R., Casado García, L., Martín Ruiz, F. J., Mediano Villanueva, F. J., Ramos Ruiz, J., & Utiel Ortega, C. (2010). Didáctica de la tecnología [Teaching technology]. Grao.
Bravo Ramos, J. L. (2005). Elaboración de materiales educativos para la formación a distancia [Development of educational materials for distance learning] Instituto de Ciencias de la Educación (ICE) UPM. http://www.ice.upm.es/wps/jlbr/Documentacion/Libros/Elabora_mat_img.pdf
Cepeda Romero, O., ; Gallardo, I. M., & Rodríguez Rodríguez, J. (2017), La evaluación de los materiales didácticos Digitales [The evaluation of digital didactic materials]. Revista Latinoamericana de Tecnología Educativa, 16(2). https://doi.org/10.17398/1695-288X.16.2.79
Espinoza, O., & González, L. E. (2006). Procesos universitarios dinámicos. El modelo de gestión de la calidad total [Dynamic university processes. The total quality management model]. Calidad de la Educación, 1(24), 15-34. http://dx.doi.org/10.31619/caledu.n24.265
Fernández-Pampillón Cesteros, A. M., Domínguez Romero, E., & Armas Ranero, I. (2012a). Guía para la producción y evaluación de materiales didácticos digitales [Guide for the production and evaluation of digital didactic materials]. Universidad Complutense de Madrid. https://eprints.ucm.es/id/eprint/12533/1/COdAv1_1_07jul2012.pdf
Fernández-Pampillón Cesteros, A. M., Domínguez Romero, E., & Armas Ranero, I. (2012b). Diez criterios para mejorar la calidad de los materiales didácticos digitales [Ten criteria to improve the quality of digital didactic materials]. In A. Sanz Cabrerizo, J. A. López Orozco, & A. Baratas Díaz (Eds.). VII jornadas de campus virtual UCM: valorar, validar y difundir campus virtual (pp. 25-34). Universidad Complutense de Madrid. https://eprints.ucm.es/id/eprint/20297/
García Aretio, L. (2014). Bases, mediaciones y futuro de la educación a distancia en la sociedad digital [Bases, mediations, and future of distance education in the digital society]. Sintesis.
García Martín, A., Briones Peñalver, A. J., Busquier Sáez, S., García Cascales, M. S., De Miguel-Gómez, M. D., Mulas Pérez, J., Pérez García, J., Vicente Chicote, C., García Sanchez, A. J., Martínez Conesa, E. J., Serrano Martínez, J. L., Pérez Pérez, J., & Sánchez Pérez, L. Á. (2010). Manual de elaboración de guías docentes adaptadas al EEES. [Manual for preparing teaching guides adapted to the EEES]. Universidad de Cartagena. https://repositorio.upct.es/xmlui/bitstream/handle/10317/1747/isbn9788469350317.pdf
Hyla, M. (2016). Przewodnik po e-learningu. [A guide for e-learning]. Wolters Kluwer.
Illum Hansen, T., & Toke Gissel, S. (2017). Quality of didactic materials. IARTEM e-Journal, 9(1), 122-141. https://doi.org/10.21344/iartem.v9i1.601
Marciniak, R. (2016). Autoevaluación de programas de educación universitaria virtual [Self-assessment of higher virtual education programmes] (Doctoral dissertation). Autonomous University of Barcelona, Barcelona, Spain.
Marciniak, R. (2017). A methodological proposal for the Project design of the online course: Pilot application. APERTURA. Revista de Innovación Educativa, 9(2), 74-95. http://www.udgvirtual.udg.mx/apertura/index.php/apertura/article/view/991/797
Marciniak, R. (2018). Quality assurance for online higher education programmes: Design and validation of an integrative assessment model applicable to Spanish universities. The International Review of Research in Open and Distributed Learning 19(2), 126-154. http://www.irrodl.org/index.php/irrodl/article/view/3443
Marzal, M. A., Calzada-Prado, J., & Vianello, M. (2008). Criterios para la evaluación de la usabilidad de los recursos educativos virtuales: un análisis desde la alfabetización en información. [Criteria for evaluating the usability of virtual educational resources]. Information Research, 13(4), 387. http://informationr.net/ir/13-4/paper387.html
Morales Morgado, E. M., García Peñalvo, F. J., & Olmos Migueláñez, S. (2010). Diseño de objetos de aprendizaje para potenciar el desarrollo de competencias y su evaluación con HEODAR [Design of learning objects to enhance the development of skills]. J. Sánchez (Ed.). IE 2010, Segundo Congreso Iberoamericano de Informática Educativa (pp. 683-690). Universidad de Chile. http://www.tise.cl/volumen6/TISE2010/Documento97.pdf
National Agency for Quality Assessment and Accreditation (ANECA). (2019). Guide to self-evaluation: Ex post accreditation for official Bachelor’s Degrees, Master’s Degrees and Doctoral Degrees ACREDITA Procedure. ANECA. http://www.aneca.es/eng/content/view/full/12612
National Distance Education University of Spain. (n.d.). El protocolo de evaluación de materiales didácticos a distancia [The evaluation protocol for distance learning materials]. http://www.uned.es/iued/subsitio/html/documentos/Protocolo_MADI.pdf
Nesbit, J. C., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3), 105-120. https://doi.org/10.21432/T25C8C
Nielsen, J., & Morkes, J. (1998). Applying writing guidelines to Web pages. Nielsen Norman Group. https://www.nngroup.com/articles/applying-writing-guidelines-web-pages/
Opdenacker, L., Stassen, I., Vaes, S., Waes, L., & Jacobs, G. (Eds.). (2009). QuADEM:Manual for the quality assessment of digital educational material. University of Antwerp.
Padrón, C., Díaz, P., & Aedo, I. (2006). MD2 method: The didactic materials creation from a model based perspective. In: W. Nejdl & K. Tochtermann (Eds.), Innovative Approaches for Learning and Knowledge Sharing. Lecture Notes in Computer Science (vol. 4227, pp. 366-382). Springer. https://doi.org/10.1007/11876663_29
Rodríguez Rodríguez, J., & Martínez Bonafé, J. (2016). Libros de texto y control del currículum en el contexto de la sociedad digital. Revista Latinoamericana de Tecnología Educativa, 16(2), 79-95. https://doi.org/10.17398/1695-288X.16.2.79
Rushby, N., & Surry, D. (2016). Wiley handbook of learning technology. John Wiley and Sons. https://doi.org/10.1002/9781118736494
Shattuck, K. (2014). Assuring quality in online education: Practices and processes at the teaching, resource, and program levels. Sterling: Stylus Publishing.
Spanish Association for Standardization and Certification (AENOR). (2017). UNE 71362. Calidad de los materiales educativos digitales [UNE 71362. Quality of digital educational materials]. Madrid: AENOR.
Vizoso, C. M. (2018, November 29). Calidad de los MED: Materiales educativos digitales [Quality of the Digital Educational Materials]. Global Campus Nebrija. https://www.nebrija.com/medios/nebrijaglobalcampus/2018/11/29/calidad-de-los-med-materiales-educativos-digitales/
Vlachopoulos, D. (2016). Assuring quality in online course design: The roadmap. The International Review of Research in Open and Distance Learning, 17(6), 183-205. http://dx.doi.org/10.19173/irrodl.v17i6.2784
A System of Indicators for the Quality Assessment of Didactic Materials in Online Education by Renata Marciniak and Cristina Cáliz Rivera is licensed under a Creative Commons Attribution 4.0 International License.