International Review of Research in Open and Distributed Learning

Volume 24, Number 3

August - 2023

A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness

Lorena Sousa, Luís Pedro, and Carlos Santos
DigiMedia, Department of Communication and Art (DeCA), University of Aveiro, Portugal

Almost all open educational resources (OER) definitions encompass key concepts such as the 5R activities and open licenses. However, little attention is given to the technical aspects and tools that allow the user to interact with these resources. This study aims to answer five research questions regarding (a) 5R activities, (b) open licenses and intellectual property, (c) technical aspects, (d) tools for developing OER, and (e) the topic of sustainability. To answer these questions, a systematic review of systematic reviews on OER was conducted following the reporting checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Sixteen studies were eligible and included in this review. The main findings suggest that although most studies did not mention the term 5R exactly, they mentioned related terms, such as share and adaptation. There was also a tendency toward focusing on more legal issues than technical aspects. Besides, most of the studies that mentioned tools discussed them as platforms to access OER, not exactly tools that encourage users to develop or adapt resources in an easy way. In relation to sustainability, several studies highlighted the relevance of developing sustainable OER models, but only a few suggested approaches to sustain an OER project. Therefore, with this article, we hope to raise awareness of the importance of the technical openness and tools that might contribute to fostering users’ engagement with the OER, helping them to act as producers and contributors rather than mere passive receivers.

Keywords: open educational resources, legal openness, technical openness, systematic review, PRISMA

Introduction

With the advent of the Internet and information technologies, a vast number of digital resources have been created and made available at little or no cost. However, not all these resources available on the Internet are open and can be reused, modified, and re-shared. Most of them are released without clear license terms and are, automatically, protected by copyright, not being allowed to be copied without the author’s permission. On the other hand, resources that are made available under open licenses or in the public domain and enable legal use, adaptation, and redistribution are called open educational resources (OER).

In 2002, when the term OER was first coined by UNESCO in the Forum on the Impact of Open Courseware for Higher Education in Developing Countries, the group defined OER as “the open provision of educational resources, enabled by information and communication technologies, for consultation, use, and adaptation by a community of users for non-commercial purposes” (UNESCO, 2002, p. 24), emphasizing that OER is “a universal educational resource available for the whole of humanity” (UNESCO, 2002, p. 28).

Similarly, Wiley (2014) claimed that content is open not only when it is available to be used in other contexts, but also when it gives everyone permission to reuse it in different ways, known as the 5R, which means retain, reuse, revise, remix, and redistribute. Retaining is the right to make, own, and control copies of the content, such as downloading, duplicating, storing, or managing a resource. Reusing is the right to use the content in a wide range of ways, in a class, in a study group, on a website, in a video, and so forth. Revising is the right to adapt, adjust, modify, or alter the content itself as, for example, translating it into another language. Remixing is the right to combine the original or revised content with other material to create something new. Finally, redistributing is the right to share copies of the original content, its revisions, or its remixes with others.

In 2019, UNESCO updated this definition, saying that OER is “learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license, that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others” (p. 5). Creative Commons (n.d., 2020) defined OER as “teaching, learning, and research materials that are either in the public domain or licensed in a manner that provides everyone with free and perpetual permission to engage in the 5R activities.” Most of the OER definitions highlight the 5R and the open licenses in their concept, but only a few consider the technical infrastructure and tools necessary for the creation, revising, and sharing of content.

If people are given legal permissions with open licenses to interact with OER through the 5R activities, they should also be given technical tools to unlock the material so that they can revise and remix it according to their needs (Wiley, 2014). The ALMS analysis is a framework developed by Wiley (2014) and Hilton et al. (2010) that puts emphasis on the OER’s technical aspects. ALMS is an acronym for access to editing tools, level of expertise required to revise or remix, meaningfully editable, and source-file access.

Access to editing tools is known as access to software that enables users to edit the resource, not only open and visualize it. There is also the aspect related to the level of expertise required to revise or remix and, thus, the tools to develop the OER must be simple and easy to use. The OER must also be meaningfully editable, meaning that it must be shared in such a format that enables anyone to edit it. A source file must be accessible which means the file that the web developer edits and works with, for example, is the same one that the web browser displays and the user interacts with (e.g., an HTML file). Consequently, modifying it must be uncomplicated (Hilton et al., 2010; Wiley, 2014). Applying open licenses which allow users to engage with materials in the 5R activities as well as applying the ALMS analysis framework enable OER creators to maximize the openness of the materials they produce (Hilton et al., 2010).

Hylén et al. (2007) defined OER as “digitized materials offered freely and openly for educators, students and self-learners to use and reuse for teaching, learning and research” (p. 10). This definition includes not only the implementation resources, responsible for the intellectual property licenses which promote the openness of materials, and the content itself, such as the courses, modules or learning objects, and tools, but also the software used to support the development, usage, and sharing of content.

Although this definition encompasses the tools necessary for the creation and distribution of OER, the William and Flora Hewlett Foundation (2018) has recognized that there is a need for a better technical infrastructure to support OER. They have highlighted recurring topics recommended by the OER community as, for instance, the desire for better OER authoring tools, tools to manage the revision and adaptation of OER, and tools to better handle the import and export of OER across different file formats and platforms (Levin, 2017).

The culture of openness must extend far beyond the simple replacement of a textbook with an open textbook and the expansion of access to free or more affordable learning materials. Although cost savings are considered a potential benefit, OER adoption needs to consider other technical aspects and tools that involve students in the 5R activities and, thus, foster student engagement with the OER, helping them to act as producers and contributors rather than passive receivers in their learning process. According to Axe et al. (2020), in contexts where students create resources collaboratively, the platforms used remained traditional or are not mentioned at all. The technical issues were also considered one of the biggest barriers when adopting OER in an educational context by Dichev et al. (2011). They cited that availability and open access are important factors, but there is a need for supportive environments that foster discoverability and sharing of content, associated with tools for adaptation and redistribution.

Therefore, considering the OER definitions, which have focused more on legal openness and open licenses, neglecting the technical aspects, such as the tools and skills necessary to revise, remix, and redistribute the resource, this systematic review of OER systematic reviews aims at analyzing how the 5R activities and the open and technical aspects are addressed in the systematic reviews and identifying tools and practical examples that are beyond the simple use of OER. The research questions that guide this systematic review are:

Method

This systematic review was conducted according to the reporting checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Liberati et al., 2009) which consists of four phases: identification, screening, eligibility, and inclusion. In the first phase (identification), we began the process of identification of eligible studies by searching pre-selected terms in the Scopus database. The choice of Scopus lies in the fact that it is one of the largest abstract and citation databases of peer-reviewed literature in this field.

The search was conducted based on the title, abstract, and keywords of articles, using the following search terms: open educational resource, open educational resources, OER, OERs, review, and reviews. The reason for searching “review” and not “systematic review” was to avoid missing systematic literature reviews that did not use the term “systematic” in their titles, abstracts, or keywords and, therefore, expanding the results. The data collection was conducted on November 5, 2021, and this round of search yielded 1,023 results.

To maintain the quality and feasibility of the present analysis, a filtering process was performed, using the year of publication (2012-2021), language (English), and publication state (final), and 784 articles were retained. As many of the titles referred to OER as oxygen evolution reaction and not open educational resources, the keywords were filtered, limiting them to those related to education, learning, and teaching, resulting in the selection of 343 articles. The title, abstract, keywords, year, and authors of the identified records were exported to a Microsoft Excel spreadsheet to be screened by an individual reviewer.

In the second phase (screening), the titles of all articles were carefully screened to identify those still related to chemistry and oxygen evolution reaction, and 179 articles were excluded (out of 343). Then, 164 abstracts were read to determine their eligibility, and 56 articles that referred somehow to open educational resources and systematic reviews were selected.

In the third phase (eligibility), 56 full-text articles were assessed to check if they were systematic reviews on open educational resources. Five articles were not available online, one was not written in English, and 19 were excluded because they were not systematic reviews.

In the fourth phase (inclusion), a total of 31 articles had their full text read, and 15 were excluded due to not being systematic reviews on open educational resources. As open educational resource is a broad term and is related to other open terms, such as open educational practice, open pedagogy, and open education, many cited OER in the titles and abstracts but did not refer to OER exactly in their text.

Finally, 16 studies were included in this systematic review. The data collection procedures have been summarized in Figure 1.

Each study was then reviewed and examined based on the following items: if they mentioned the 5R activities, the licenses or intellectual property issues, technical aspects, open tools, and sustainability (see Table 1). These items provided information to answer the research questions and to conduct the synthesis that is presented in the next section.

As shown in Figure 2, 11 studies out of 16 were published in recent years: 2018 (n = 3), 2019 (n = 3), and 2020 (n = 5), which can indicate an increasing interest in systematic reviews on open educational resources, maybe due to the coronavirus pandemic and the transition to online learning. However, this tendency was inconspicuous in 2021 (n = 1). This may be explained by the fact that the data collection was conducted in November 2021, and there were still some articles to be published that year.

The systematic reviews that were selected used 71 keywords in total. Figure 3 presents these keywords, and the bigger the font size, the more frequently they appeared in the texts. The most frequently used keyword was undoubtedly open educational resources, which was mentioned 14 times, followed by OER, which appeared seven times. Open educational practices, systematic review, higher education, textbooks, disability, and accessibility appeared twice. The others appeared only once each.

Furthermore, the systematic reviews mentioned 10 other open terms in their titles, abstracts, and keywords besides the term open educational resources, which appeared 24 times and, as it is the focus of this study, it was not introduced in the word cloud below. Open educational practices was mentioned five times, massive open online courses and open course ware appeared three times, and open textbooks appeared twice. The other terms, as displayed in Figure 4, were mentioned only once.

Results

1. How are the 5R activities addressed in the systematic reviews on OER?

Only four texts mentioned Wiley’s 5R principles, as displayed in Figure 5. Three texts mentioned it before the presentation of the results, that is, in the introduction and/or theoretical background (Craig, 2020; Moon & Park, 2021; Tlili et al., 2019), and two texts mentioned the 5R in the results (Craig, 2020; Wong & Li, 2019). From these three and two texts that mention the 5R before and in the results, respectively, only one text cited the 5R both before and in the results (Craig, 2020).

Moon and Park (2021), for example, said that while most studies have emphasised the 5R, they hardly ever debate approaches to enhance learners’ engagement and interactions with the resource. They also said that OER-enabled pedagogy expands learners’ interactions with the OER beyond the simple use for information retrieval. Through this pedagogy, learners can create, modify, and share these resources.

In addition to this, Craig (2020) reported that learners could benefit more from the 5R if they had at least a basic understanding of editing tools. The access to editing tools and the level of expertise required to revise or remix materials are two aspects discussed in the ALMS framework (Hilton et al., 2010; Wiley, 2014) and essential to the technical openness requirements.

Most texts (n = 12) did not mention Wiley’s 5R exactly, but referred to some words related to it, such as share, dissemination, adaptation, copy, and combine (Arimoto & Barbosa, 2012; Hilton, 2016; Ivanova et al., 2020; King et al., 2018; Luo et al., 2020; Meng et al., 2020; Moreno et al., 2018; Paragarino et al., 2018; Wang et al., 2017; Yuan & Recker, 2015; Zhang et al., 2020). From these texts, seven mentioned related terms both before and in the results. Therefore, all the texts cited the 5R, directly or indirectly.

2. How often are the licenses or intellectual property issues mentioned?

In total, 14 out of 16 studies mentioned the licenses, such as Creative Commons, copyright, or intellectual property issues in their texts (Figure 6). Eleven mentioned them in the introduction and/or background theory (Arimoto & Barbosa, 2012; Clinton, 2019; Hilton, 2016; Ivanova et al., 2020; Meng et al., 2020; Moreno et al., 2018; Paragarino et al., 2018; Tlili et al., 2019; Wang et al., 2017; Wong & Li, 2019; Zhang et al., 2020), 11 mentioned them in the results (Arimoto & Barbosa, 2012; Clinton, 2019; Hilton, 2016; Ivanova et al., 2020; King et al., 2018; Luo et al., 2020; Meng et al., 2020; Moon & Park, 2021; Tlili et al., 2019; Wang et al., 2017; Wong & Li, 2019), and eight mentioned copyright issues both before and in the results (Arimoto & Barbosa, 2012; Clinton, 2019; Hilton, 2016; Ivanova et al., 2020; Meng et al., 2020; Tlili et al., 2019; Wang et al., 2017; Wong & Li, 2019).

Only two of the 16 studies selected for this review (Craig, 2020; Yuan & Recker, 2015) did not mention the licenses in their texts, revealing the high importance given to the legal aspects of OER, one of the key elements in the OER definition.

One of the texts that did not mention the licenses is about the implementation of OER in Canadian higher education during the pandemic (Craig, 2020). Although it talks about Wiley’s 5R principles, it does not cite any aspect regarding intellectual property and technical issues related to the reuse, sharing, and adaptation of resources. The other text that did not comment on licenses is about the use of rubrics to evaluate the quality of OER (Yuan & Recker, 2015). It used related terms to talk about the 5R activities and did not mention technical aspects (discussed in the next subsection).

3. How often are the technical aspects discussed?

Only three texts addressed concerns about the technical aspects of OER as, for example, tools that support the development and reuse of learning content, as visualized in Figure 7. Three mentioned the technical aspects in the introduction and/or background theory (Arimoto & Barbosa, 2012; Ivanova et al., 2020; Wong & Li, 2019) and only one mentioned the technical aspects in the results (Arimoto & Barbosa, 2012), being also the only one that mentioned them before and in the results.

When comparing these numbers from the technical aspects to the numbers from the legal openness discussed in the previous subsection, it can be noted that they are almost opposites. While 14 articles mentioned the legal openness and two did not mention it, 13 articles did not discuss the technical aspects and only three did.

These results supported few studies that have highlighted the importance of technical infrastructure and knowledge that enables technical openness in OER. The William and Flora Hewlett Foundation, for example, has recognized the need for “tools for authoring and automated feedback, tools for metadata management, analytical tools, [and] tools for import/export in different platforms and formats” (Ivanova et al., 2020, p. 67). Arimoto and Barbosa (2012, p. 6) argued that, “The integration of social tools encourages the active participation of developers and users in the construction of OERs, also important in distributed and collaborative development of such resources.” These tools can also facilitate the development of OER, besides contributing to the quality of the final resource.

4. Which open tools are pointed out in the reviews?

Six of the 16 texts pointed out open tools through their texts (Arimoto & Barbosa, 2012; Hilton, 2016; King et al., 2018; Moon & Park, 2021; Tlili et al., 2019; Wong & Li, 2019), as can be seen in Figure 8. Some open tools mentioned are Moodle, edX, Sina Weibo, XuetangX, CNMOOC, iCourse163, P2PU, Canvas Network, Coursera, EdX, Khan Academy, OpenCourseWare (OCW), Openstax, The Saylor Foundation, Washington State’s Open CourseLibrary, and The Minnesota Open Textbook Library.

Almost all these tools are platforms to access materials and content, not tools that facilitate the practice of the 5R activities, fostering the creation and adaptation of existing materials. Only three texts out of six mentioned tools intrinsically linked to the development of OER, such as blogs and wikis. These results are in close relation to the demand that exists in the OER field discussed by Ivanova et al. (2020). According to them, the William and Flora Hewlett Foundation has already recognized the need for a better infrastructure to support these resources.

As was mentioned in the previous subsection, Arimoto and Barbosa (2012) discussed the integration of social tools in the OER to encourage the active participation of users in the collaborative construction of these resources. Arimoto and Barbosa (2012) pointed out that the use of these tools tends to make the development of resources easy, contributing to the quality of the final resource.

5. How relevant is the topic of sustainability in these systematic reviews?

Only seven texts out of 16 (Craig, 2020; Luo et al., 2020; Meng et al., 2020; Paragarino et al., 2018; Tlili et al., 2019; Wang et al., 2017; Wong & Li, 2019) mentioned the sustainability issue regarding the OER (Figure 9). Sustainability refers to the ability of an OER to continue or be continued for the long term. According to Wang et al. (2017, p. 303), “Sustainability is a core issue and major challenge faced by not only end-users but also OER developers, foundations and policymakers.”

According to Paragarino et al. (2018), very few OER projects are continued after the end of funding. Few successful examples have impacted the general financial model of OER development, and designing new business models for the sustainability of OER is one of the main challenges in this area. In 2019, UNESCO released a recommendation on OER, and one of the areas of action is “nurturing the creation of sustainability models for OER” (Tlili et al., 2019, p. 11). Wong and Li (2019) also defended the position that developing policies to support OER is essential for the sustainability of their practices.

In Meng et al. (2020), the topic of sustainability is highlighted in the keyword analysis, highly cited publications, and OER practice. Similarly, the study also showed that many projects could not survive beyond their initial funding and argued that self-finance models should be explored to maintain OER projects. Meng et al. (2020) suggested two approaches to sustain an OER project: to advertise or offer other services to obtain extra income, and to extend the traditional mode of donation.

However, although sustainability is frequently considered from an economic perspective, it is not restricted to financial issues. According to Downes (2007), sustainable OER models can be categorized into four aspects: funding, technical, content, and staffing (Wang et al., 2017). Luo et al. (2020) found that sustainability is the most frequently cited barrier that stands in the way of OER, and suggested that partnerships among designers, e-learning staff, academic librarians, and teachers, for example, can contribute to the sustainability of OER.

Discussion and Conclusions

The majority of OER definitions are centered on the 5R activities and legal aspects, such as the open licenses, neglecting the technical aspects and tools that are necessary to fulfill the objectives of the OER movement, giving anyone not only legal permission but also technical support to reuse, adapt, and share materials.

This study performed a systematic review of systematics reviews on OER with the aim of analysing if these reviews mentioned and how often they mentioned the 5R activities, the licenses or intellectual property issues, technical aspects, open tools, and the topic of sustainability, and comparing if these studies also follow this tendency of emphasizing legal aspects over the technical ones.

The main findings revealed that all the texts mentioned the 5R or related terms, such as access, share, copy, and adaptation, and almost all the selected texts discussed the legal issues (n = 14), corroborating the definitions presented by UNESCO (2002, 2019), Creative Commons (n.d, 2020), and Hylén et al. (2007), which say that OER is any resource used in the teaching, learning, or research context, that resides in the public domain or is under an open license, giving anyone permission to engage with these materials through the 5R activities.

On the other hand, 13 texts out of 16 did not mention the technical aspects necessary to create, reuse, adapt, and share OER. Besides, the studies that talked about tools did not point out tools that facilitate the creation and adaptation of resources. Most of them were platforms used to access OER. As discussed by the William and Flora Hewlett Foundation (2018) and Levin (2017), there is a lack of technical infrastructure, such as tools, to support the revision and adaptation of OER. This technical challenge was also mentioned by Dichev et al. (2011), who cited that there is a need for environments that facilitate the sharing of content as well as tools for OER modification and redistribution.

There are some limitations to this systematic review. The first one is related to the database used, Scopus. Besides not covering all the publications, we did not search the articles on other databases to complement our research. Furthermore, research outcomes are published in several languages and this systematic review encompassed only those written in English. Because of this, we might have missed some relevant articles due to the database restriction and data collection process.

To conclude, having access to tools and technical knowledge that enable users to engage with resources through the 5R activities and knowing the legal issues that permit users to reuse or adapt these resources should have the same level of importance when talking about OER. If users are given legal permission to engage with OER through the 5R activities, they should also be given technical tools to unlock these resources so that they can interact with them as producers and contributors, and not only as passive consumers, maximizing the openness of content.

Therefore, with this study we hope to raise awareness of the importance of the technical openness and tools that might contribute to fostering users’ engagement with OER, helping them to reuse, remix, and redistribute these resources according to their needs.

References

Arimoto, M. M., & Barbosa, E. F. (2012). A systematic review of methods for developing open educational resources. In G. Biswas, L.-H. Wong, T. Hirashima, & W. Chen (Eds.), Proceedings of the 20th International Conference on Computers in Education, ICCE 2012 (pp. 262-269). Asia-Pacific Society for Computers in Education. https://apsce.net/conference_4_detail.php?id=710

Axe, J., Childs, E., DeVries, I., & Webster, K. (2020). Student experiences of open educational practices: A systematic literature review. Journal of E-Learning and Knowledge Society, 16(4), 67-75. https://doi.org/10.20368/1971-8829/1135340

Clinton, V. (2019). Cost, outcomes, use, and perceptions of open educational resources in psychology: A narrative review of the literature. Psychology Learning and Teaching, 18(1), 4-20. https://doi.org/10.1177/1475725718799511

Craig, C. D. (2020). 2020 pandemic: Resilient Canadian higher education institutions will integrate OER. In N. Callaos, B. Muirhead, L. Robertson, B. Sanchez, & M. Savoie (Eds.), 14th International Multi-Conference on Society, Cybernetics and Informatics (IMSCI 2020) (pp. 149-153). International Institute of Informatics and Systemics. https://www.proceedings.com/content/056/056306webtoc.pdf

Dichev, C., Bhattarai, B., Clonch, C., & Dicheva, D. (2011). Towards better discoverability and use of open content. In D. Dicheva, Z. Markov, & E. Stefanova (Eds.), Proceedings of S3T’11 Third International Conference on Software, Services and Semantic Technologies (pp. 195-203). Springer. https://doi.org/10.1007/978-3-642-23163-6

Hilton, J., III, Wiley, D., Stein, J., & Johnson, A. (2010). The four ‘R’s of openness and ALMS analysis: Frameworks for open educational resources. Open Learning: The Journal of Open, Distance and E-learning, 25(1), 37-44. https://doi.org/10.1080/02680510903482132

Hilton, J., III. (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64(4), 573-590. https://doi.org/10.1007/s11423-016-9434-9

Hylén, J., Pedro, F., Schuller, T., & Allen-Sinclair, A. (2007). Giving knowledge for free: The emergence of open educational resources. OECD. https://doi.org/10.1787/9789264032125-en

Ivanova, M., Grosseck, G., & Holotescu, C. (2020). Analysis and modeling the domain of open educational resources from learning analytics perspective. In I. Roceanu (Ed.), The 16th International Scientific Conference of e-Learning and Software for Education: E-Learning sustainment for never-ending learning (Vol. III, pp. 66-74). E-Learning and Software for Education. https://doi.org/10.12753/2066-026X-20-178

King, M., Pegrum, M., & Forsey, M. (2018). MOOCs and OER in the Global South: Problems and potential. The International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3742

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration. BMJ (Clinical Research Ed.), 339. https://doi.org/10.1136/bmj.b2700

Luo, T., Hostetler, K., Freeman, C., & Stefaniak, J. (2020). The power of open: Benefits, barriers, and strategies for integration of open educational resources. Open Learning, 35(2), 140-158. https://doi.org/10.1080/02680513.2019.1677222

Meng, X., Cui, C., & Wang, X. (2020). Looking back before we move forward: A systematic review of research on open educational resources. In Y. Xie, J. M. Pereira Barroso, & Y.-M. Huang (Eds.), Proceedings - 2020 9^th International Conference of Educational Innovation Through Technology, EITT 2020 (pp. 92-96). IEEE. https://doi.org/10.1109/EITT50754.2020.00022

Moon, J., & Park, Y. (2021). A scoping review on open educational resources to support interactions of learners with disabilities. The International Review of Research in Open and Distributed Learning, 22(2), 314-341. https://doi.org/10.19173/irrodl.v22i2.5110

Moreno, N., Caro, E. T., & Cabedo, R. (2018). Systematic review: OER and disability. In M. El Mohajir, M. Al Achhab, B. El Mohajir, & I. Jellouli (Eds.), 2018 IEEE 5^th International Congress on Information Science and Technology (CiSt) (pp. 428-431). IEEE. https://doi.org/10.1109/CIST.2018.8596659

Paragarino, V. R., Silveira, I. F., & Llamas-Nistal, M. (2018). Open educational resources: A brief vision from IEEE topics. In C. Gonzalez, M. Castro, & M. Llamas (Eds.), Proceedings of 2018 IEEE Global Engineering Education Conference (EDUCON) (pp. 2076-2081). IEEE. https://doi.org/10.1109/EDUCON.2018.8363495

Tlili, A., Huang, R., Chang, T. W., Nascimbeni, F., & Burgos, D. (2019). Open educational resources and practices in China: A systematic literature review. Sustainability (Switzerland), 11(18), Article 4867. https://doi.org/10.3390/su11184867

Wang, X., Liu, M., Li, Q., & Gao, Y. (2017). A bibliometric analysis of 15 years of research on open educational resources. In S. L. Wong (Ed.), Proceedings of the 25th International Conference on Computers in Education, ICCE 2017 - Main conference proceedings (pp. 296-305). Asia-Pacific Society for Computers in Education.

Wiley, D. (2014). Defining the “open” in open content and open educational resources. Opencontent.org. http://opencontent.org/definition/

Wong, B. T. M., & Li, K. C. (2019). Using open educational resources for teaching in higher education: A review of case studies. In Proceedings - 2019 International Symposium on Educational Technology, ISET 2019 (pp. 186-190). IEEE. https://doi.org/10.1109/ISET.2019.00046

Yuan, M., & Recker, M. (2015). Not all rubrics are equal: A review of rubrics for evaluating the quality of open educational resources. The International Review of Research in Open and Distributed Learning, 16(5). https://doi.org/10.19173/irrodl.v16i5.2389

Zhang, X., Tlili, A., Nascimbeni, F., Burgos, D., Huang, R., Chang, T. W., Jemni, M., & Khribi, M. K. (2020). Accessibility within open educational resources and practices for disabled learners: A systematic literature review. Smart Learning Environments, 7(1). https://doi.org/10.1186/s40561-019-0113-2

A Systematic Review of Systematic Reviews on Open Educational Resources: An Analysis of the Legal and Technical Openness by Lorena Sousa, Luís Pedro, and Carlos Santos is licensed under a Creative Commons Attribution 4.0 International License.

Author(s)	Topic Covered
Author(s)	5R	Legal	Technical	Tools	Sustainability
Moon & Park (2021)	yes	yes	no	yes	no
Meng et al. (2020)	related terms	yes	no	no	yes
Zhang et al. (2020)	related terms	yes	no	no	no
Luo et al. (2020)	related terms	yes	no	no	yes
Craig (2020)	yes	no	no	no	yes
Ivanova et al. (2020)	related terms	yes	yes	no	no
Tlili et al. (2019)	yes	yes	no	yes	yes
Wong & Li (2019)	yes	yes	yes	yes	yes
Clinton (2019)	related terms	yes	no	no	no
Moreno et al. (2018)	related terms	yes	no	no	no
Paragarino et al. (2018)	related terms	yes	no	no	yes
King et al. (2018)	related terms	yes	no	yes	no
Wang et al. (2017)	related terms	yes	no	no	yes
Hilton (2016)	related terms	yes	no	yes	no
Yuan & Recker (2015)	related terms	no	no	no	no
Arimoto & Barbosa (2012)	related terms	yes	yes	yes	no