International Review of Research in Open and Distributed Learning

Volume 20, Number 2

April - 2019

Effectiveness of OER Use in First-Year Higher Education Students’ Mathematical Course Performance: A Case Study

Juan I. Venegas-Muggli¹ and Werner Westermann^2
1Universidad Tecnológica de Chile INACAP, ²Library of National Congress of Chile

Abstract

This article examines the effect of two Open Educational Resources (OER) - Khan Academy Collection and a teacher-authored open textbook - on mathematical course performance and attendance amongst first-year higher education Chilean students. It also aims to find out about teachers’ and students’ views on the use of OER in order to understand how these resources are used and valued. To this end, quantitative and qualitative methods were employed. Findings indicate that students in face-to-face classes who used Khan Academy resources obtained better examination grades than students who used the open textbook or relied on traditional proprietary textbooks. Moreover, it was also found that students who used both types of OER had significantly lower attendance levels than students who relied on traditional proprietary textbooks. Finally, it was observed that teachers and students had very positive opinions on the use of both the Khan Academy Collection and open textbook resources.

Keywords: OER, Khan Academy, open textbook, higher education

Introduction

Education is a pivotal means of promoting development in any country. As nations seek to develop their human capital in order to participate in a society of global knowledge, there is increasing pressure on educational systems, particularly those in higher education, to meet growing demands for equal educational opportunities and to supply high quality, relevant, and efficient formal and informal educational processes.

Both equity and quality are major challenges for national educational systems in terms of the level of innovation and transformation required. UNESCO (2006) has coined the phrase Education for Sustainable Development (ESD) as an umbrella term for the many forms of educational practice that promote efforts to rethink educational systems in countries facing extreme educational challenges. ESD requires participatory teaching and learning approaches in order to motivate teachers and empower learners to change their behavior and take action to achieve sustainable development. It promotes competencies such as critical thinking, imagining future scenarios, making decisions, and solving problems in a collaborative way.

As a re-imagined education system is required to create a new set of skills and competencies for a burgeoning number of new learners, there appears to be the widespread consensus that new forms of educational provision must be available online and free of cost for learners. The European Commission (2012) states that digital technology “offers unprecedented opportunities to improve quality, access and equity in education and training,” and that it is a “key lever for more effective learning and for reducing barriers to education, in particular social barriers” (p. 9). It recognizes, however, that technology on its own does not assure innovation; it is, instead, the level of openness regarding the use of technology (European Commission, 2013) that enables the development of the capacity to stay current, promote innovation, and exploit the potential of new learning technologies and digital content.

In this context, recent trends in the use of Open Educational Resources (OER) - also referred to as “open content” (Downes, 2007) - are enabling fundamental changes and innovation in educational provision. New ways of learning, characterized by personalization, engagement, the use of digital media, collaboration, bottom-up practices, and an approach where the learner or teacher is a creator as well as a consumer of learning content, have been facilitated by the exponential growth of OER in recent years. OER are important for stimulating innovative learning environments where users can adapt content according to their needs (Keegan & Bell, 2011).

As previously noted, the need to study and evaluate OER initiatives emerges as a relevant field of research. If these types of action are to become widespread, their effectiveness - as well as possible measures for improvement - must be studied. As formerly stated, the aim of this research is to study the effect of OER by examining a specific case study. Focusing on first-year Chilean higher education students, it aims to observe whether the performance of students in mathematical courses taught using OER improves and how students and teachers perceive this same process.

The Chilean case is particularly relevant in terms of research since the country’s educational system has recently been challenged by demands from a civil society that wants access to quality education. Following a series of ongoing, student-led protests across the country, setting the foundation for a national social reform movement, the second presidential term of Michelle Bachelet's government (2014-2018) has embraced the challenge through complex structural educational reforms (Venegas, 2016). In this sense, the study of the effect of OER initiatives is a significant opportunity to contribute to Chile’s public policy debate about equity in education using empirical evidence.

The Relevance of Open Educational Resources (OER)

During the last few years, the adoption of Open Educational Resources has become a major trend in public education policy-making. A series of initiatives have emerged that have led to numerous institutional, local, regional, and national policies supporting OER throughout the world. Among them, the Policies for OER Uptake (POERUP), the European Open Education Policy Project, and the Creative Commons OER Policy Registry can be highlighted. In this context, the relevance of these types of resources concerning educational development has been growing.

According to Atkins, Brown and Hammond’s (2007) definition, open educational resources can be understood as “teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use or re-purposing by others” (p. 4). These include textbooks, course materials, videos, tests, podcasts, multimedia applications, and any other material designed to access to knowledge (Atkins et al., 2007; Butcher, 2015).

In sum, open educational resources are relevant because they facilitate learning processes by delivering diverse open access materials. To this effect, various authors have highlighted the specific benefits of these resources. Orr and van Damme (2015) state that OER positively impact educational processes in three different ways by a) harnessing the possibilities afforded by digital technology (in the case of e-learning OER) to address common educational challenges; b) acting as a catalyst for social innovation and new forms of interaction between teachers and learners; and c) promoting the idea of an extended lifecycle beyond their original design and purpose, with the process of their distribution, adaptation, and iteration improving access to high quality educational materials for all. Likewise, Lane and McAndrew (2010) point out how these resources benefit a specific institution such as the United Kingdom Open University. In this case, it is emphasized that OER have several benefits, such as enhancing university reputations, supporting broader participation, providing material, and accelerating the use of new technologies.

OER and Educational Research

Along with the rise in implementing OER practices, there have also been numerous research papers published based on understanding and evaluating the usefulness of these educational policies. In this respect, the OER agenda has evolved considerably in recent years.

Initially, global OER initiatives were focused on providing infrastructure and delivering mechanisms to implement policies, which resulted in OER research being primarily focused on measuring the deployment, access, and use of these resources (UNESCO, 2011). As the OER movement advanced, a second phase began in which these initial actions were consolidated and a new wave of studies emerged. These have been aimed at assessing the efficacy and impact of OER adoption and deployment (Santos-Hermosa, Ferrán-Ferrer, & Abadal, 2013; Smith, 2013). This is the trend that currently dominates the OER research agenda.

Studies on the effectiveness and impact of OER have focused predominantly on whether adopting these resources at an institutional level brings financial and academic benefits for students and institutions, and how these processes take place. In this context, most of the research on their impact has been focused on the cost-effectiveness of “packaging” OER into courseware or textbooks (Bliss, Hilton, Wiley, & Thanos, 2013; Wiley, Hilton, Ellington, & Hall, 2012, Chiorescu, 2017). For example, Hilton and Laman (2012) conclude that students from Houston Community College who used open textbooks in psychology classes got better grades, had a lower dropout rate, and did better in the final examination. Likewise, Mi Choi and Carpenter (2017) found that both exam grades and course grades in a Human Factors and Ergonomics class did not change when traditional textbooks were replaced by free-to-use materials.

Another point to be highlighted is that great diversity has also been observed among the educational resources currently referred to as OER, depending on their level of openness (Shear, Means, & Lundh, 2015). In the same way, research has also considered the various actors who influence a teaching resource’s ultimate success. Both tendencies thus imply that the research agenda on the effect of OER is still wide open and requires further studies on new educational materials and contexts to be developed.

Data and Method

This research aims to study the effect of OER on students’ academic performance. Likewise, it intends to understand teacher and students’ views on the use of these resources. To this effect, it employed a mixed method approach with two main phases of data collection and analysis. The first phase involved examining the effect of OER use on students’ performance in mathematical courses, as well as class attendance based on registry information. The second phase went into these initial results in more depth, using quantitative and qualitative methods focused on teacher and student views on the use of OER.

Both phases involved first year students and teachers from the Instituto Profesional Providenca (IPP) Chilean higher education institution. Specifically, this research considered students from the Schools of Education and Engineering taking several mathematics courses during 2014, both in face-to-face classes and through e-learning. A more detailed description of both phases is provided below.

Phase 1: The OER Effect on Students’ Performance

The first phase compared several groups of students to determine whether those taught with the help of OER produced better results than students whose education relied exclusively on traditional methods. Specifically, two scenarios considering different treatment and control groups were defined.

As described in Figure 1, Scenario 1 considered two treatment groups and one control group made up of face-to-face students in three different arithmetic classes from the IPP’s School of Education. The first class (Control Group, n=30) used a traditional proprietary textbook. The second class (Treatment Group 1, n=35) was taught with the help of the Khan Academy Collection. Finally, a third class (Treatment Group 2, n=31) was taught using a custom-designed open arithmetic textbook. This scenario took place during the second trimester of 2014 over a period of four months.

Scenario 2 compared two classes with a blended module of algebra and calculus classes in the School of Engineering, where students only came together in person for tests and the final exam. The first class (Control Group, n=41 students) relied on traditional proprietary (institutionally-produced) resources, while the second class (Treatment Group 3, n=21 students) used the Khan Academy Collection. This study was implemented during the second and third trimesters of 2014.

Figure 1. Overview of scenarios comprising the study.

As seen by the description of both specified scenarios, two types of OER were considered: the Khan Academy Collection and an open textbook. The Khan Academy is a Creative Commons Attribution, Non-Commercial (CC-BY-NC) licensed resource that delivers thousands of openly licensed resources through an unrestricted website. In this case, the Khan Academy operated as an additional resource that students were expected to use in order to fulfill their course requirements, alongside traditional course materials.

In the case of the second OER, an open textbook was developed. It was created by two participating teachers, based extensively on their own teaching materials and notes, and was published on Wikibooks (https://es.wikibooks.org/wiki/Matem%C3%A1ticas/N%C3%BAmeros_y_Operaciones). This open textbook was called Números y Operaciones (Numbers and Operations) and was provided to students in both printed and digital formats.

This textbook was specifically designed for this study and was exclusively used by students in Treatment Group 2 (see Figure 1). It was written by teachers and included numerous sources such as notes, study guides, assessments, and selected open content inspired or aligned with traditional/commercial textbooks. During classes, students were invited to build on and use exercises related to the content, thanks to the textbook’s participatory features and open context.

To estimate the effectiveness of both resources on students’ mathematical performance, the quasi-experimental Propensity Score Matching (PSM) methodology was used (Heckman, Ichimura, Smith, & Todd, 1999). This technique evaluates the impact of certain policies by estimating the probability of receiving a specific treatment. It was used because it allows the comparison of academic results between students that used OER and those that did not use these resources through a more unbiased method.

PSM was employed by using the Inverse Probability Weight method, which compares individuals by giving a greater weight in the estimations to people of the control group who have a higher probability of being treated (Imbens & Hirano, 2002). Specifically, the probability of having been taught with the help of an OER was initially modelled using students’ sociodemographic characteristics (age, family income, and number of years the mother was educated). Then, the results of the treated and control students (final exam grades, final course grades, and attendance level) were compared.

Phase 2: Educational Actors’ Views on the Use of OER

The second phase of the study examined students’ and teachers’ views on the process of using OER. This provided an opportunity to better understand the views of the end-user (learners and teachers) about the benefits and challenges related to their experience of using OER.

For this purpose, a qualitative approach was applied to teachers and students that used OER. This involved holding two semi-structured interviews with the teachers, two semi-structured interviews with the students, and one focus group with the students. An online survey was designed and applied to 49 students based on the qualitative results.

Using closed questions in two sections, the survey probed student perception of OER use, focusing specifically on their evaluation of a number of the resource’s characteristics, the problems identified, and recommendations for the better use of these resources. The first section consisted of a Likert scale of 39 items that measured the students’ evaluation of their OER experience, while the second section asked students to indicate the main perceived benefits and problems of this initiative, considering all the issues that had emerged in the qualitative phase. In terms of the survey’s reliability, Cronbach's alpha for the 39-item scale was 0.92, which indicates a high level of reliability.

Results

Effectiveness of OER Use on Students’ Performance

The first results section presents several analyses about the effect of using OER on students’ performance in mathematical courses This was done by comparing different treatment and control groups using the PSM methodology in the two previously specified scenarios (see Figure 1).

In the following tables, each coefficient in the first row describes the effect of the use of an OER regarding a specific comparison group in terms of standard deviations, while the second row shows the standard errors of these effects. These coefficients indicate the average difference between the groups compared regarding the specified result variables when controlling for other relevant variables, namely age, family income, and number of years the mother was educated.

The first analyses of students’ mathematical course performance only considered the academic results of freshmen enrolled in face-to-face arithmetic courses offered by the IPP’s School of Education in the second semester of 2014 (Scenario 1).

In relation to this, Table 1 shows the effect of using OER when comparing students that used the Khan Academy Collection 1 to students that used traditional textbooks. It is seen that the use of the Khan Academy Collection 1 had a negative effect on attendance (0.86 SD s), at a 5% significance level. Likewise, it can be seen that the use of these resources had a positive effect on final exam scores (0.54 SD s), at a 10% significance level. Finally, there were no significant differences between students’ final course scores.

Table 1

Estimation of the Effect of Using the Khan Academy Collection 1 Versus the Use of Traditional Textbooks (Scenario 1)

Note. ** = p < 0.05; * = p < 0.1; n=65.

This result suggests then that OER improve students’ examination performance but have a negative effect on their attendance levels. Students taught using the Khan Academy produced better final exam grades on average than those who relied on traditional textbooks, indicating that open resources helped improve students’ academic performance when considering this last indicator.

The second comparison considered Treatment Groups 1 and 2 from Scenario 1 (i.e., the class that used the Khan Academy Collection 1 and the class that used open textbooks). In this regard, Table 2 shows that students who used the Khan Academy Collection 1 had significantly lower attendance levels (at a 1% significance level) than those who used open textbooks. Additionally, it is seen that students who used the Khan Academy Collection 1 had significantly better exam results (at a 5% significance level) than those who used open textbooks (1.55 standard deviations). There were no significant differences between final course scores.

Table 2

Estimation of the Effect of Using the Khan Academy Collection 1 Versus the Use of Open Textbooks (Scenario 1)

Note. *** = p < 0.01; n=66

These results confirm that the Khan Academy Collection is an important resource for enhancing students’ performance. Students who used the Khan Academy Collection obtained better results in their final exams on average than those who used either traditional proprietary textbooks or open textbooks, thus highlighting the importance of the type of OER used to improve students’ skills.

The third analysis of Scenario 1 examined whether students taught using open textbooks had better results than those taught with traditional ones. In this respect, Table 3 shows that the only result where significant differences were found was the final exam grade, significant at a 1% level. This difference, however, was unexpected, as it can be seen that the use of open textbooks had a negative significant effect with a 0.08 standard deviation on students’ final exam grade. This means that students who were taught with traditional textbooks had, on average, higher exam grades than students who were taught with the help of open textbooks.

Table 3

Estimation of the Effect of Open Textbooks Versus the Use of Traditional Textbooks (Scenario 1)

Note. *** = p < 0.01; n=61.

The last analysis of OER effectiveness compared students from Scenario 2 - in other words, students from a blended module on algebra and calculus offered by the School of Engineering in the first and second trimesters of 2014. In this scenario, a Control Group (n=41) used a traditional proprietary resource, while Treatment Group 3 (n=21) used the Khan Academy Collection 2. Results from this comparison are presented in Table 4.

Table 4

Estimation of the Effect of Using the Khan Academy Collection 2 Versus Traditional Resources (Scenario 2)

Note. n=62.

As shown in Table 4, there were no significant differences between any of the result variables examined. This means that the use of OER did not result in any discernible improvement in students’ mathematical performance in blended courses.

Student and Teacher Views on the Use of OER

Having examined the effect of OER on students’ performance, this information must be complemented with data about the views of students and teachers on their experience of using OER. For this purpose, the findings presented are grouped into three main topics: overall evaluation, the positive aspects of OER use, and the negative aspects of OER use.

Overall Evaluation

Regarding an overall evaluation, it can be seen that students and teachers were very satisfied with the use of these resources. Qualitatively, it was pointed out that OER were important tools for them to develop their courses and that their use was beneficial to both students and teachers. It was stated that these resources provided vital support, helping to achieve different types of learning in the face-to-face classroom mode, as well as in the home environment. One student who were taught using the Khan Academy Collection, scenario 1, stated that “they are complementary, because they replace a teacher more efficiently. Because I can repeat, repeat and repeat and see the results and advance. Because sometimes texts are not the best motivation when you are tired.” Another student, from scenario 2, explains that “they were very pedagogical, didactic. I liked them better, because other algebra classes - or classes related to mathematics - that I have attended included very little support material other than documents or texts.”

Specifically, in the case of the Khan Academy Collection, both teachers and students positively highlighted that they included appropriate theoretical content, and the corresponding practical exercises, that allow students to easily understand the content. Moreover, students considered these resources to be user-friendly and felt that this platform facilitated the learning process.

As far as the use of open textbooks was concerned, this experience was also evaluated positively. Students indicated that they used open textbooks instead of traditional proprietary textbooks in order to study content covered in the course syllabus. Likewise, this resource was highlighted as a relevant means for continuing to study at home, and it was also emphasized that the use of this resource was voluntary and was never made compulsory by the teacher.

Our teacher gave them to us and each person decided what they wanted to do (...) Sometimes he also recommended using the book, saying that on a certain page there were exercises about what we had studied that day. However, it was not compulsory. He made us see that it was a kind of help.

Information about the overall evaluation of this OER experience is evident in the survey data as students were asked to indicate their level of agreement with 39 statements on the use of these resources. Based on this information, a scale of 1 to 10 was calculated for each person who replied where higher values meant a more positive evaluation. Only 37 of the 39 survey statements were considered for this scale since two of the items did not directly reference an evaluation of these resources.

Table 5 displays the average results obtained by different groups of students in this 10 point scale that evaluates student experiences with OER. It can be noted that, although the averages of the groups being compared were generally similar, there were some relevant differences. First, it can be seen that open textbook users gave a better evaluation than Khan Academy Collection users (7.17 versus 6.97). At the same time, it can be seen that younger respondents had a greater appreciation of the use of OER, since the group aged between 19-24 had an average index of 7.25, while the group aged over 25 had an average index of 6.91.

Table 5

Mean OER Use Evaluation Index by Resource Type, Age Group, and Income Category

* Note. n=47.

Table 5 also shows that students with higher incomes and with a higher level of use had a more positive evaluation of OER. This could hypothetically be explained by the fact that those with a higher income used OER more frequently as they had the resources to do so. This might be the case since it was discovered that when OER were used in a student’s home, those from higher incomes had more opportunities to engage with the resources.

Positive Aspects of OER Use

Regarding specific positive aspects of OER, the qualitative section data shows that responses vary significantly according to the resource used. In the case of the Khan Academy Collection, the website’s high level of accessibility and stability are highlighted as its main positive features as described by two participants: "It was always available and did not crash. It would have been terrible to be in the middle of an exercise and have the page crash. That was important to me," and "It let us edit.... So, in some cases, I simplified a few things. I added exercises or changed definitions or added missing content."

In the case of open textbooks, the ability to edit content is one of their most positive features. Consistent with this, students reported that the aspects they liked most included the ability to edit and upload exercises, the fact also that the supplementary printed book provided valuable support, and that there were a lot of exercises available for study.

Positive aspects of OER are highlighted in the survey data that complemented this study, which asked students to select their top three positive OER aspects from a list of 12 options. Table 6 shows the results from all the students, separated by resource type. The most important aspect identified is that OER contributed to the understanding of class content, as mentioned by 65% of students. Other positive features include the fact that explanations are delivered in a more didactic and entertaining way with the use of practical exercises.

Table 6

Positive Aspects Highlighted Regarding the Use of OER by Resource Type

*Note. n=49.

When these results were examined separately by resource type, important differences were observed. While dynamic/entertaining teaching and the fact that exercises could be carried out at home were more prominent for Khan Academy users, the possibility of using practical exercises and the level of teacher preparation was more important for users of open textbooks.

Main Perceived Problems of Using OER

In relation to the main perceived difficulties, the qualitative material shows that the lack of time to use these resources was a relevant factor. Moreover, teachers specifically highlighted that, in both scenarios, older students were not familiar with computers and did not know how to create or use these resources.

Students and teachers also highlighted problems with the IPP’s infrastructure, and it was specifically stated that often there were not enough computers for students and that some of them were in poor condition, which negatively influenced optimal use of the OER. Teachers in scenario 1 explained that “(n)ot everyone used them. I was able to directly check each student’s use of the Khan Academy.... I had to get angry and ask why. And it was because they didn’t have enough time," and “I had students in that class that were slightly older. That in some way were reluctant to use Wikibooks. Then, when it came to editing their uploaded exercises, there was a certain degree of refusal because it was complicated."

When analyzing views separately for both OER, certain difficulties can also be highlighted. Specifically, for Khan Academy, that some aspects of the course content covered in class were not included and that some students did not understand the practical exercises, was a criticism of this resource. These resources were presented to students in this platform, which was different than how students were taught in traditionally face-to-face classes. Likewise, students from the blended courses pointed out that initially they were not adequately informed that use of the platform was to be assessed, which resulted in their lack of interest at the start of the course.

I did not understand that it was another grade... so at first I didn’t take it seriously and got behind. In the end I had to start using it quickly. I used it in its entirety, but I wasn’t really aware of what the final goal was.

In the case of open textbooks, the main difficulty was associated with understanding the Wikibooks editing platform. This platform was built using the Latex programming language and some students, particularly the older ones, identified this as a problem.

As with the positive aspects, Table 7 shows the difficulties associated with use of the OER by resource type, based on the survey results. As seen in this table, the main difficulties identified include the time lag in loading resources, the IPP’s technological infrastructure, and, to a lesser extent, the lack of time for student use. What is more, it is important to note that only 2% of those who replied said that OER use required a level of knowledge they did not have.

Table 7

Main Perceived Difficulties of OER Use by Resource Type

Note. n=49.

Regarding the differences according to the type of OER used, issues related to IPP infrastructure were more prevalent for users of the Khan Academy Collection, while difficulties associated with the website and the lack of certain kinds of content were identified by open textbook users.

Discussion

The main aims of this research were to examine the effect of OER use among higher education students and to analyze teacher and student views on OER use in order to better understand how these resources are used and valued. This was justified by the fact that there is a lack of empirical evidence to support expanding the use of OER. Moreover, recent societal demands to improve education quality in Chile have made this a relevant case study environment in which to examine the potentials of OER.

In relation to the first aim, the most important result is that students in face-to-face arithmetic/statistics courses using Khan Academy resources achieved significantly better exam grades than students who did not use any extra resources (p < 0.05) or those who used open textbooks as an extra resource (p < 0.01). The fact that the final exam was the same for everyone makes this a valid comparative measure of students’ performance.

These findings indicate that OER could be valuable assets to Chilean students in a context where there is increased access to innovative resources and demand for higher quality education. It was observed that OER could be useful in specific contexts to improve students’ results, which indicates that Chilean education could benefit from offering OER to more students in formal educational settings. What is more, these results are coherent with other examined studies about OER effectiveness in other contexts (Feldstein et al., 2012; Hilton & Laman, 2012).

Another important finding was the fact that face-to-face students who used OER had significantly lower attendance levels than students who relied on traditional educational methods. Further research should be done on this issue. For example, research could be carried out on whether this can be explained by the fact that when students have access to these resources, they tend to work more from home.

All these results, however, should be treated with caution, since they were obtained from small samples that represent a very specific group of Chilean students only. Moreover, it was also found that the use of open textbooks did not have any effect and that, among students on e-learning courses, the use of OER did not make any difference. In terms of representativeness, these findings do not mean that OER cannot have a positive effect on other student groups; it does, however, need to be made clear that these findings have little external validity and that more research on the effect of OER is required to justify the use of these resources in a broader context.

Regarding the research’s second aim, the qualitative and quantitative materials examined reconfirmed the assumption that these resources can be a relevant asset to Chilean students. Qualitative data demonstrated that both actors had positive experiences when using these two OER types. These positive results were later ratified by the survey results, which demonstrated a very positive assessment of the OER used, since the majority of students indicated in the survey that OER made teaching more dynamic, that these resources were easy to use, and that they provided good explanations and practical exercises.

This positive evaluation of OER contrasts somewhat with the findings of the first part of this research, in which only one of the groups studied performed significantly better than those who did not use this type of resource. This discrepancy, however, should not be understood as an inconsistency of the examined data, since the results of and views about these results do not necessarily have to agree. The fact that, in some cases, OER were found to have a negative effect does not mean that these resources were not useful to students. It does mean that the positive effect perceived by students may not be reflected in higher grades, but could be shown in other ways, such as increased motivation or improved ICT skills. To prove this, however, would require further study that considers new outcome variables.

The second component of this research also showed some of the perceived problems regarding the use of these resources. It was highlighted that they did not work well when students lacked resources, such as adequate time and physical space. Furthermore, the fact that optimal use of resources relied on the IPP having computer laboratories in good condition was also noted. On the other hand, quantitative data identified areas for improvement, such as the institutional infrastructure challenge of learning how to use these resources properly and specific problems associated with OER websites. From these results, it is evident that any suggestions are aimed at enhancing the conditions in which OER strategies are implemented, rather than criticizing the usefulness of the resources, reconfirming the fact that students highly evaluated their OER experience.

Another important point addressed in the second part of this research relates to students’ perceptions according to income level, age group, and levels of student use. Younger students and those with higher levels of resources valued the experience of using OER more and students with higher income had higher scores on the evaluation scale. This last point in particular deserves certain attention as it shows that a digital gap could play a role in the usefulness of OER. Although this issue needs to be researched in more depth, these results seem to indicate that open access is not always enough to enable students to successfully engage with an open educational environment (Lane, 2009).

Conclusion

From a public policy standpoint, both OER selected for this study show that these types of resources can contribute to an increase in national educational challenges. As a public strategy, OER can provide an educational context with enriched possibilities: equal access, quality enhancement of educational knowledge and resources, professional teacher development, institutional innovation, cost-effectiveness and public accountability (Atkins et al., 2007, Lane & McAndrew, 2010).

In this context, having measured both the effect of OER on academic performance and understood the views on the use of these resources, an important contribution to the debate about the relevance of OER was made. As was previously stated, in order to promote the expansion of these resources, their usefulness has to be proved. To this effect, this research positively promotes the development of OER initiatives, since it empirically shows how these resources deliver innovative tools to address educational challenges, encouraging new forms of interaction between educational actors in the process of knowledge generation (Orr & van Damme, 2015).

Another important point is that, if OER are to accomplish their aims, educational institutions have to take on certain responsibilities and have the infrastructure needed to take full advantage of these resources. To be precise, if OER are expected to promote integral methods of curricular communication and have a transformative value (Butcher, 2015), educational institutions must install the necessary infrastructure for students to benefit from these open initiatives.

Given the Chilean context, this research shows evidence of the potential benefits of applying OER as a public strategy to improve equity and quality. Chile’s higher education system has been specifically described as having important equity problems because some institutions deliver low-quality education as a result of their limited resources. To this effect, OER could provide an important opportunity to tackle these problems, given their cost-effectiveness benefits. This would, however, require both the commitment of Chile’s public institutions to promote and lead the application of these types of resources and the desire of educational institutions to adopt them and provide the required facilities.

Acknowledgments

This article is based on findings from a research project financed by the Research on Open Educational Resources for Development (ROER4D).

References

Atkins, D. E., Brown, J. S., & Hammond, A. L. (2007). A review of the open educational resources (OER) movement: Achievements, challenges, and new opportunities. Report to the William and Flora Hewlett Foundation. Retrieved http://hewlett.org/wp-content/uploads/2016/08/ReviewoftheOERMovement.pdf

Bliss, T. J., Hilton III, J., Wiley, D. & Thanos, K. (2013). The cost and quality of online open textbooks: Perceptions of community college faculty and students. First Monday, 18(1). doi: 10.5210/fm.v18i1.3972

Butcher, N. (2015). A basic guide to open educational resources (OER). Retrieved from http://unesdoc.unesco.org/images/0021/002158/215804e.pdf

Chiorescu, M. (2017). Exploring open educational resources for college algebra. The International Review of Research in Open and Distributed Learning, 18(4). doi: 10.19173/irrodl.v18i4.3003

European Commission. (2012). Rethinking education: Investing in skills for better socio-economic outcomes. Retrieved from http://ec.europa.eu/education/policy/multilingualism/rethinking-education_en

European Commission. (2013). ‘Opening up education’: Innovative teaching and learning for all through new technologies and open educational resources. Retrieved from http://ec.europa.eu/education/news/doc/openingcom_en.pdf

Downes, S. (2007). Models for sustainable open educational resources. Interdisciplinary Journal of Knowledge and Learning Objects, 3(1), 29-44. Retrieved from http://www.learntechlib.org/p/44796/

Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open Distance and E-Learning, 2(1). Retrieved from http://www.eurodl.org/?p=archives&year=2012&halfyear=2&article&article=533

Heckman, J. J., Ichimura, H., Smith, J. A., & Todd, P. E. (1999). Characterizing selection bias using experimental data. Econometrica, 66(5), 1017-1098. Retrieved from https://www.jstor.org/stable/2999630

Hilton III, J.L., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open, Distance and e-Learning, 27(3), 265-272. doi: 10.1080/02680513.2012.716657

Imbens, G.W., & Hirano, K. (2002). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2(1), 259-278. doi: 10.1023/A:1020371312283

Keegan, H., & Bell, F. (2011). YouTube as a repository: The creative practice of students as producers of open educational resources. European Journal of Open, Distance and E-learning, 14(2), 1-10. Retrieved from http://www.eurodl.org/materials/special/2011/Keegan_Bell.pdf

Lane, A. (2009). The impact of openness on bridging educational digital divides. The International Review of Research in Open and Distributed Learning, 10(5). doi: 10.19173/irrodl.v10i5.637

Lane, A., & McAndrew, P. (2010). Are open educational resources systematic or systemic change agents for teaching practice? British Journal of Educational Technology, 41(6), 952-962. doi: 10.1111/j.1467-8535.2010.01119.x

Mi Choi, Y., & Carpenter, C. (2017). Evaluating the impact of open educational resources: A case study. Libraries and the Academy, 17(4), 685-693. doi: 10.1353/pla.2017.0041

Orr, D., Rimini, M., & van Damme, D. (2015). Open educational resources: A catalyst for innovation. Paris: OECD Publishing. doi: 10.1787/9789264247543-en

Santos-Hermosa, G., Ferrán-Ferrer, N., & Abadal, E. (2013). Recursos educativos abiertos: Repositorios y uso [Open Educational Resources: Repositories and use]. El profesional de la información, 21(2), 136-145. Retrieved from http://openaccess.uoc.edu/webapps/o2/handle/10609/13601

Shear, L., Means, B., & Lundh, P. (2015). Research on open: OER research hub review and futures for research on OER. Menlo Park, CA: SRI International. Retrieved from https://www.hewlett.org/wpcontent/uploads/2016/08/OERRH%20Evaluation%20Final%20Report%20June%202015.doc

Smith, M. (2013). Ruminations on research on open educational resources. Menlo Park, CA: William and Flora Hewlett Foundation. Retrieved from http://www.hewlett.org/wp-content/uploads/2016/08/OER%20Research%20paper%20December%2015%202013%20Marshall%20Smith_1.pdf

UNESCO. (2006). Framework for the UNDESP international implementation scheme. Retrieved from http://unesdoc.unesco.org/images/0014/001486/148650E.pdf

UNESCO. (2011). Activity report 2010-2011. Retrieved from http://iite.unesco.org/publications/3214690/

Venegas, J. I. (2016). ¿Por qué los jóvenes chilenos rechazan la política? Desafección política juvenil en el Chile postransición. [Why Chilean youth reject politics? Youth Political Disaffection in Chile’s post-transition] Santiago, Chile: Ril Editores.

Wiley, D., Hilton III, J., Ellington, S., & Hall, T. (2012). A preliminary examination of the cost savings and learning impacts of using open textbooks in middle and high school science classes. The International Review of Research in Open and Distributed Learning, 13(3), 262-276. doi: 10.19173/irrodl.v13i3.1153

Effectiveness of OER Use in First-Year Higher Education Students’ Mathematical Course Performance: A Case Study by Juan I. Venegas-Muggli and Werner Westermann is licensed under a Creative Commons Attribution 4.0 International License.