Abstract

Grant programs that foster the use of open educational resources (OER) significantly reduce undergraduate student spending on textbooks per semester. The Zero-Cost Course Materials (ZCCM) grant program at the University of California, Merced (UC Merced), eliminated text costs and ensured access to course materials by replacing required commercial materials with OER and library licensed e-resources. The present study applies the COUP framework (cost, outcomes, usage, and perceptions) to evaluate the ZCCM program. The ZCCM program resulted in student cost savings and did not negatively impact student outcomes. Students in ZCCM courses demonstrated higher rates of course completion than students enrolled in previous sections. For the outcomes of final course grade, passing with a C- or better, and number of credit hours enrolled in, findings were comparable between the cohorts. Student usage and perception of course materials were gathered using a survey. Though students reported favorable views of zero-cost materials, they reported using them less frequently than commercial texts. This research contributes to a growing body of literature that confirms beneficial cost savings for students using zero-cost materials without jeopardizing students’ success.

Keywords: open educational resource, OER, library e-resources, COUP framework, cost savings, student outcomes 

Introduction

The formal designation of open educational resources (OER) at the UNESCO Forum on the Impact of Open Courseware for Higher Education in Developing Countries (2002) launched OER as a compelling alternative to commercial educational materials. The definition of OER as “learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others” encapsulates the growing open movements at the time, including open courseware, open learning solutions, open access publishing, and open licensing (UNESCO, n.d.). While the 2002 forum brought participants together to examine worldwide applications of MIT OpenCourseWare, participants left with a collaborative plan to evaluate, use, and develop OER “for the whole of humanity” (UNESCO, 2002).

Within the United States, the development, curation, promotion, and use of OER in higher education curricula has been bolstered by funding at the federal, state, and institutional levels. The U.S. Department of Education allocated $47 million in grant funding starting in 2018 through the Open Textbooks Pilot program (Allen, 2022), and according to the SPARC (Scholarly Publishing and Academic Resources Coalition) OER State Policy Tracker, 28 states have enacted policies to promote OER use within higher education or at the K-12 level (Allen, 2022). Higher education institutions have launched their own programs to encourage faculty adoption, adaptation, and creation of OER. EDUCAUSE Horizon Reports (Brown et al., 2020; Pelletier et al., 2021) identify OER as a key technology and practice expected to significantly impact teaching and learning in higher education.

Encouraged by the success of initiatives like the Zero Textbook Cost (ZTC) degrees at California Community Colleges and the freely available online learning materials via the MERLOT (Multimedia Education Resource for Learning and Online Teaching) project initiated at the California State University system, the University of California, Merced (UC Merced) Library, and the UC Merced Center for Teaching and Engaged Learning (CETL) launched the Zero-Cost Course Materials (ZCCM) grant program as a three-year pilot project to explore the benefits and impact of OER. The ZCCM initiative provides faculty with grants to support their use of OER and/or library e-resources in place of commercial texts to ensure students have day-one access to materials at no cost. Including library e-resources, such as e-books with unlimited simultaneous user licenses along with OER, under the umbrella term “zero-cost materials,” provided faculty grant recipients with more flexibility when selecting alternatives to commercial texts. The grant program’s focus on costs and timely access to materials addresses challenges reported by UC Merced students acquiring course materials and recognizes a majority Pell-eligible undergraduate student population (UC Merced, Institutional Research & Decision Support, 2022a).

While many factors inform a faculty member’s selection of course materials, some hesitate to consider OER without more information about efficacy and quality (Belikov & Bodily, 2016). A growing body of research aims to assess OER’s impact. OER research in higher education frequently measures one or more elements in the COUP (cost, outcomes, usage, and perceptions) framework developed by the Open Education Group. This shared schema allows for current and future OER research to be compared (n.d.-a).

The present study assesses the impact of the ZCCM grant program by applying the COUP framework to answer the following questions:

1. What are the student cost savings of using zero-cost materials in place of commercial materials?
2. What is the impact of using zero-cost materials on student success?
3. How does the use of zero-cost materials influence unit enrollment patterns?
4. How do students perceive the quality and value of zero-cost materials?

Literature Review

Effects of Textbook Costs

Between 2006 and 2018, the average price of a college textbook increased by 190% (Bureau of Labor Statistics, U.S. Department of Labor, 2023; Smith Jaggars et al., 2019). While textbook prices have plateaued over recent years, students continue to take action to minimize textbook costs. The most recent National Association of College Stores (NACS) Student Watch Report finds that despite course material spending averaging $339 per year, the lowest since NACS started tracking this data in 1998, one in four students still skipped purchasing at least one course text, and 11% of students have pirated course materials (National Association of College Stores, 2022). Over 87% of respondents taking the Florida Virtual Campus (FLVC) Student Textbook and Instructional Materials Survey reported that they pursued measures to reduce textbook costs, such as renting a digital copy (50.5%) or sharing books with classmates (17%). Over half reported not purchasing the text at all (Florida Virtual Campus, 2022).

Students have expressed concerns about their ability to succeed academically when they do not purchase textbooks required for their classes. Sixty-five percent of respondents to a Student Public Interest Research Groups (PIRGs) survey indicated that they chose not to buy course materials due to costs, and of those respondents, 94% expressed concerns that this decision would negatively impact their course grade (Senack, 2014). Over a third (32.4%) of FLVC survey participants believed that high textbook costs resulted in poorer grades, and 24.2% reported dropping a course due to high textbook costs (Florida Virtual Campus, 2022). Additional literature suggests that these actions are not uncommon. In Jenkins et al.’s study (2020), 56% of surveyed students reported that they did not buy a textbook for a course due to costs and later felt like it had hurt their class performance; 12% dropped a class due to textbook costs, and 9% attributed failing a class for the same reason. Textbook costs can also influence students’ course selection and enrollment intensity (credit hour) decisions. In the FLVC survey, 38.5% of students reported not registering for a specific course, and 43.7% reported taking fewer courses due to textbook costs (Florida Virtual Campus, 2022).

Cost Savings

Instructors frequently point to student cost savings as the primary reason for adopting OER (Hilton III, 2020). Switching to an OER can quickly yield significant savings, especially in large enrollment courses requiring high-priced texts. Fialkowski et al.’s assessment of an OER program at the University of Hawai‘i at Mãnoa calculated savings of $39,000 for 311 students in an introductory nutrition course within a single semester (2020).

Researchers have employed various methods to calculate the cost savings of replacing a commercial text with an OER. One widely used method multiplies the sticker price of the previously assigned commercial text by the number of students enrolled in the OER course. Though Cozart et al. (2021) used the sticker price method, they note that cost savings can be difficult to determine precisely and may be inflated since not all students will purchase a new textbook at the list price. When determining cost savings of OER initiatives from 600 courses across 120 institutions, SPARC researchers (along with David Wiley) developed a formula to account for student purchasing behavior. Their calculation averaged the most expensive version of a text with the least expensive version, representing the text’s available format and delivery options, from which the cost of the OER, if any, is subtracted. In applying this formula, they determined $116.94 in cost savings per student when a commercial text was replaced with an OER (Nyamweya, 2018). Another method uses a standardized amount to represent textbook costs; the Open Education Network and Open Oregon Educational Resources both apply a rate of $100 per student to determine cost savings (Hofer, 2017).

Student Outcomes

Many studies assessing the impact of OER examine one or more of the following undergraduate student outcomes: course grades; passing with a C- or better; withdrawal, drop, failure, and course completion rates; and enrollment rates. They vary in terms of their methodologies, sample sizes, and ability to control for variables.

Final Course Grades

Researchers regularly, but not exclusively, use course grades as an indicator of student learning and success. Most studies find that students in OER courses have comparable learning gains (Clinton, 2018; Clinton & Khan, 2019), similar or better learning gains (Grewe & Davis, 2017; Suriano, 2023), differences that are not statistically significant (Allen et al., 2015; Croteau, 2017; Jhangiani et al., 2018; Ross et al., 2018; Winitzky-Stephens & Pickavance, 2017), or improved learning (Colvard et al., 2018).

In an analysis of final grades for eight courses at the University of Georgia from 2010 to 2016, students in OER courses had significantly improved course grades in contrast to students in non-OER courses (Colvard et al., 2018), while Fischer et al.’s examination of 15 undergraduate courses from 10 institutions found no significant differences in course grades between OER and non-OER courses when results were reported in aggregate. Robinson reported the opposite; students using OER had lower grades than those using commercial textbooks (2015). For data at the course level, results from Fischer et al. (2015) and Robinson (2015) were mixed.

Additional findings differ by student population. For instance, Winitzky-Stephens and Pickavance (2017) did not see a significant correlation between course grades and OER until they analyzed new first-year student and continuing student populations separately. When disaggregated, they found course grades positively impacted new students in OER courses. Colvard et al.’s (2018) research found that OER courses had even greater positive impacts for Pell-eligible students, part-time students, and historically underserved students.

Passing With a C- or Better

When examining three measures of student success, including the likelihood of passing with a C- or better, Winitzky-Stephens and Pickavance (2017) found that there were no adverse effects for students using OER compared with students using commercial texts. Fischer et al.’s (2015) work also investigated if students in OER and non-OER groups were more or less likely to earn a C- or better. Results were mixed, though for most courses (9 of 15), there were no significant differences between the two groups. Five OER courses and one commercial course yielded higher rates of students passing with a C- or better. While Robinson (2015) found no statistically significant differences between OER and non-OER groups earning a C- or better, analysis at the course level surfaced two of seven courses in which the OER groups had lower rates of earning a C- or better. While findings are not entirely uniform, the literature predominantly reports positive impacts or no negative impacts of OER usage on students’ ability to pass with a C- or better.

Withdrawal, Drop Rates, Failure Rates, and Completion Rates

Researchers have also examined students’ withdrawal rates, drop rates, failure rates, and rates of completion in OER versus non-OER courses as additional measures of student success. Most findings point to OER as having positive impacts on these measures, while others report findings with no statistical significance or findings with mixed results. Numerous researchers report lower withdrawal rates of students in OER courses (Bol et al., 2022; Clinton, 2018; Clinton & Khan, 2019; Feldstein et al., 2012; Hilton III & Laman, 2012; Hilton III, 2016; Pawlyshyn et al., 2013; Suriano, 2023), lower drop rates (Hilton III, 2016; Wiley et al., 2016), lower failure rates (Colvard et al., 2018; Feldstein et al., 2012), and higher completion rates (Ross et al., 2018).

Additional studies record a lack of statistically significant differences between the rates of OER and non-OER groups. Winitzky-Stephens and Pickavance (2017) found no statistically significant differences in withdrawal or pass/fail rates between students in OER and non-OER classes. Croteau (2017) concluded that there were no differences in either DFW (drop/fail/withdraw) or completion rates between students in courses pre- and post-OER adoption. Fischer et al. (2015) also compared completion rates of students in OER and non-OER sections in 15 courses and found few significant differences.

Enrollment Rates and Enrollment Intensity

There is a limited amount of literature on OER’s potential impact on student enrollment patterns. Some studies highlight students’ avoidance of classes due to textbook prices, and these costs may also have a greater impact on certain student populations. In their survey of students enrolled in a general psychology course, Hardin et al. (2019) found that non-White students and first-generation college students (FGCS) “reported that the cost of the book had a significantly higher impact on decisions to enroll in the course” in comparison to their non-FGCS or White peers; yet, the study’s larger conclusion pointed to textbook costs having a minimal effect on students’ decisions to enroll in courses of interest (p. 54). Similarly, Jenkins et al. (2020) reported that 27% of their student respondents avoided enrolling in a class due to textbook costs, and this rate of avoidance increased for Latinx and transfer students. Other studies have gone beyond student survey data to examine enrollment rates and enrollment intensity (the number of credits in which a student is enrolled each semester). In both cases, Fischer et al. (2015) and Robinson (2015) found that students enrolled in OER courses had a higher credit load on average.

Student Use and Perception of OER

Students have reported using OER at similar rates to commercial texts (Bliss et al., 2013; Hendricks et al., 2017; Jhangiani et al., 2018; Sack Illowsky et al., 2016). Gurung’s results (2017) differ, finding that students using OER read less than those using a commercial textbook.

In regards to OER quality, students are usually positive. When asked to compare the quality of an OER to that of commercial texts they had used previously, the majority of students in multiple studies responded that the OER was of better/higher or equal/similar quality to commercial texts (Bliss et al., 2013; California Open Educational Resources Council, 2016; Cooney, 2017; Hendricks et al., 2017; Hunsicker-Walburn et al., 2018; Ikahihifo et al., 2017; Jhangiani et al., 2018; Ross et al., 2018; Sack Illowsky et al., 2016; Watson et al., 2017). This finding is highlighted in Hilton’s synthesis of user perceptions of OER from 2015 to 2018 (Hilton III, 2020) and in Mullens and Hoffman’s 2023 systematic review of OER publications from 2002 through 2023, which included 48 studies that investigated student perceptions of OER quality. Additional studies indicate no significant differences in students’ perceptions of OER and commercial texts (Clinton, 2018; Nusbaum et al., 2020), while a limited number of studies report that students rated their standard commercial texts as higher quality than OER (Gurung, 2017; Lawrence & Lester, 2018).

Some studies have identified what students value about OER. Desirable characteristics include ease of use (Hilton III & Laman, 2012), ease of access (Brandle et al., 2019), greater readability (Jhangiani et al., 2018), better writing (Clinton, 2018), targeted content, organizational flexibility, and digital capabilities (Cooney, 2017; Ikahihifo et al., 2017).

Methodology

Participants

During the grant program, 1,870 students were enrolled in one or more ZCCM-grant-supported courses, which included large enrollment introductory courses, upper division courses, and one graduate level course. At each semester’s end, students in ZCCM courses were invited to complete a survey and consent to the release of specific student data to be linked to their survey responses. Of those enrolled in a ZCCM course, 426 provided their student ID number and completed the survey. The students who correctly recorded their ID number on the survey and were in a ZCCM course that could be matched to a comparison section comprised a treatment group of 198 students (ZCCM cohort; see Figure 1). Their survey responses were linked with demographic data and student success indicators. The same set of demographic data and student success indicators were obtained for a control group of 2,732 students (comparison cohort) who were enrolled in a prior offering of the course from the same instructor when commercial materials were required.

Figure 1

Cohort Eligibility Flowchart

Data Collection 

We worked with UC Merced’s Institutional Research & Decision Support (IRDS) and campus registrar to obtain final grade data, drop and withdrawal rates, and credit unit enrollment intensity. We obtained demographic and background data for students in both treatment and control groups, including gender, race/ethnicity, field of study (student major), year of study, grade point average, on-campus housing/commuter status, and first-generation college student status. For analysis, final grade data were recoded from a letter grade into a four-point scale, in which an F grade was valued at 0 and A and A+ grades valued at 4. Students’ course drop, withdrawal, and failure rates were used to calculate course completion. Student self-identified race and ethnicity data were categorized using Integrated Postsecondary Education Data System (IPEDS) classifications and described at the highest specificity that maintains student privacy. These variables were collected to control for non-treatment effects and differences between the cohorts, and we hoped to learn if any student populations had positive or negative outcomes based on zero-cost material or OER usage (Colvard et al., 2018; Nusbaum et al., 2020).

The survey asked ZCCM students about their use and perceptions of course materials (including zero-cost materials), their textbook purchasing behavior, and textbook costs. The survey was developed by reviewing surveys from the literature (Bliss et al., 2013; Dastur et al., 2015; Open Education Group, n.d.-b). Researchers obtained an exempt approval for this project from the UC Merced Institutional Review Board.

Data Analysis 

Demographic data describing the treatment and control groups were compared using a chi-squared test or t-test to identify significant differences between the groups. To examine the impact of zero-cost materials on final grade, final grade of C- or higher, enrollment intensity, and course completion, we applied a doubly robust (DR) treatment effects model with an inverse-probability-weighted-regression-adjustment (IPWRA) estimator, a statistical technique applied to estimate the effect of an intervention while effectively minimizing the selection bias via modeling the selection process and outcome mechanism simultaneously (Dudik et al., 2011; Funk et al., 2011; Hall et al., 2015; Wooldridge, 2010). Prior research found that such an approach consistently estimates the unbiased average treatment effects if the propensity score model or the outcome model is correctly specified (Bang & Robins, 2005; Sant’Anna & Zhao, 2020; Tan et al., 2022). All the DR models matched and controlled for students’ gender, race/ethnicity, on-campus housing/commuter status, first-generation college student status, major of degree, and semester. We analyzed survey data to examine students’ experiences and perceptions of zero-cost materials as compared with commercial texts using descriptive statistics, chi-squared tests, or t-tests. STATA 17 was used for all of the analyses.

Results and Discussion

Cost Savings

Between 2019 and 2021, the ZCCM grant resulted in student textbook savings of $202,899. This estimate uses the sticker price method to calculate savings for the total number of students enrolled in the 15 grant-supported courses. The highest cost savings were generated by two introductory courses: Biology 001 and Economics 001. Biology 001 enrolled 515 students and previously used a textbook that cost $81. Though Economics 001 enrolled about half that number of students, its prior textbook was $248. Cost savings are reported for the first semester in which zero-cost materials were implemented and do not represent continued savings over subsequent semesters.

Survey data point to the importance of cost savings for students. Ninety-one percent of student survey respondents indicated that potential cost savings of course materials are moderately to extremely important. During course registration, grant-supported courses were listed with a “no textbook costs associated with this section” label. Of the 24% of students who indicated seeing this label, 64% reported that this statement influenced their decision to register for the section. Students were later asked to imagine a future course with two identical sections, one using zero-cost materials and the other section using commercial texts. Eighty-four percent of students indicated that they would enroll in the section using zero-cost materials, with 15% indicating no preference and 1% preferring the section using commercial texts. 

Cohort Comparison

Before analyzing impacts on student outcomes, we ran descriptive statistics for both groups, which are displayed in Table 1. The ZCCM and comparison cohorts were found to be largely comparable, with no significant difference by gender, race/ethnicity, on-campus housing/commuter status, school year, or number of credit hours. However, students in ZCCM classes were less likely to be first-generation college students (p = .027). There were some differences in the declared major between the two cohorts; for example, the comparison cohort comprised a larger proportion of students majoring in biological sciences.

Table 1

Demographic Characteristics by ZCCM Treatment Group Versus Comparison Group

*Note. The sum of the percentages may not round up precisely to 100 due to the precision of one-digit decimals.

Impact on Student Outcomes 

When matched and controlled for student demographics and characteristics to isolate the treatment effect, students in the ZCCM group had outcomes that were better and no different than those in the comparison group, as summarized in Table 2.

Final Course Grades

The control and treatment groups did not differ in final grade (p = .091). These comparable or similar final grades align with the literature (Clinton, 2018; Clinton & Khan, 2019; Grewe & Davis, 2017). Since the same instructor taught the courses compared, one might expect little fluctuation in final grades despite different course materials being used. This finding may suggest that instructors’ selections of course materials were not significantly inferior or superior to their prior choices.

Passing With a C- or Better

No differences were found between ZCCM and control groups in the attainment of a final grade of C- or above (p = .076), which is similar to what studies with aggregated findings revealed (Robinson, 2015; Winitzky-Stephens & Pickavance, 2017). As with final grades, instructor consistency across comparison groups may have contributed to this finding. Grimaldi et al.’s 2019 work would expect this outcome. Their access hypothesis suggests that OER primarily benefits a subset of students who would not have purchased the course text due to the expense; therefore, any positive learning outcomes from OER adoption impact a smaller group of students and are unlikely to yield a measurable effect.

Course Completion Rates

In our statistical analysis, there was evidence that students enrolled in ZCCM courses had higher course completion rates (p = .002) than their peers in the control group. This finding was significant, though the effect size was small. This result aligns with the majority, though not all, of the literature on OER and course retention. However, this finding is not strongly supported by our survey data. Almost 92% of respondents reported that they had never (80.11%) or rarely (11.83%) dropped or withdrawn from a course due to the cost of the course texts. Still, this finding suggests that textbook costs may compromise students’ ability to complete a course.

Table 2 

Student Outcomes by ZCCM Treatment Group Versus Comparison Group

Credit Hours Enrolled

The two groups (p = .263) did not differ significantly in the number of credit hours enrolled, suggesting that students in ZCCM courses were not compelled to take on more credit hours due to the reduced cost of course materials, even though 20% of student survey respondents indicated that they had previously enrolled in fewer units due to the cost of the course text, and studies have found that students taking an OER course have, on average, enrolled in a higher number of units in contrast to a control group of peers (Fischer et al., 2015; Robinson, 2015). Other factors may be more influential in students’ enrollment decisions. A campus survey asked first-year students to report why they had enrolled in fewer than 15 units, and the most common reason (47%) was a lack of open seats in the section(s) needed (UC Merced, Institutional Research & Decision Support, 2022b). Other factors influencing enrollment choices may revolve around times courses are offered, perceived course difficulty, or commitments to activities such as employment or student research.

Student Use and Perception

ZCCM students were surveyed about their usage of required texts in their courses overall as compared with their usage of required zero-cost materials in their grant-supported course (see Figure 2). Students reported using the required texts in their other courses more frequently than they reported using zero-cost materials (3.7 [SD:1.2] vs. 3.4 [SD:1.4], p < .001). Fifty-one percent of respondents indicated that, on average, they used their required texts once a week or more, as opposed to 46% of students who reported using required zero-cost materials at the same frequency.

Figure 2

Frequency of Text Usage in All Courses Contrasted With a ZCCM Course*

*Note. The sum of the percentages may not round up precisely to 100 due to the precision of one-digit decimals. n = 198

Gurung’s 2017 analysis highlights a similar finding, in which students reported reading and studying less when using OER versus commercial textbooks. Gurung suggests this difference can be attributed more to student engagement patterns with print versus online materials than to the quality of the content. There may be several reasons students reported using zero-cost materials less frequently than the texts in other courses, such as zero-cost materials being adapted or created specifically to support in-class lecture and activities, or the implementation of zero-cost materials coinciding with fewer assessments that relied on course readings.

When asked about the percentage of materials used, 28% of students reported using a larger amount of commercial texts than zero-cost materials, while 25% reported using a larger amount of zero-cost materials than commercial texts. Forty-seven percent selected “Unsure” in response to this question. See Figure 3 for a summary of student responses on material usage.

Figure 3

Amount of Materials Used

n = 198

Students were also asked to rate the quality of the text(s) used in the zero-cost materials course in contrast to texts in other courses (see Figure 4). Over half of the students (52%) thought the zero-cost materials were of similar quality to other texts with costs while 33.8% reported that they were of better or much better quality, and under 8% of students reported the quality of the zero-cost materials as being worse or much worse.

Figure 4

Students’ Ratings of Quality of Zero-Cost Materials Compared With That of Commercial Texts

n = 198 

ZCCM students’ positive perception of zero-cost materials aligns with the literature, in which the majority of students surveyed have expressed favorable views of OER quality (Hilton III, 2020). While Jhangiani et al. (2018) propose that the free nature of OER can provoke a halo effect on quality ratings, these positive perceptions may also be driven by desirable features found in freely available resources (Brandle et al., 2019; Hilton III & Laman, 2012; Jhangiani et al., 2018). When asked about features of zero-cost materials, students indicated that the ability to access materials immediately (91%) and the convenience and portability of the digital format (87%) was moderately to extremely important.

In addition, students were asked to compare their experiences in the ZCCM course to those in previous courses. A small percentage of students reported negative impacts associated with the ZCCM course, while most students noticed a positive or neutral effect, as summarized in Table 3. In their systematic review, Mullens and Hoffman (2023) note six studies that found an increase in student engagement. The authors caution that engagement is not well defined through self-reported measurements but stipulate that access to materials and interactive features of OER could be driving engagement. Another driver could be changes in instructors’ pedagogical approach, associated with their adoption of OER.

Table 3

Experiences in Zero-Cost Courses as Compared With Previous Courses

n = 198

Limitations and Future Directions

Though we used the DR estimator to address potential selection bias in this observational study, some bias may be present due to recruitment challenges, which became more difficult when classes pivoted to online instruction in March 2020 due to COVID-19. This resulted in uneven levels of student participation across grant-supported courses. For example, biological science majors comprised 6.6% of our ZCCM cohort, while they made up 31.2% of the comparison group.

Many students who consented to participating in this research project incorrectly entered their student ID numbers on the survey, which prevented us from matching many survey responses to student data. This limited our ability to perform more granular analyses of the impact of zero-cost materials on specific student groupings, such as historically underrepresented students. Furthermore, this study only compared courses taught by the same instructor and did not account for variations in learning experiences (e.g., in-person vs. online). The current study did not attempt to assess the potential effects of the zero-cost material type (e.g., open textbooks, course-specific materials authored by the instructor, library e-resources) or how the instructors used the materials with their students. For future assessments, we plan to obtain students’ Pell Grant eligibility to examine how this group of students who qualify for low-income aid may be impacted by textbook costs and the use of zero-cost materials.

Conclusion

Our research using the COUP framework contributes to a growing body of literature confirming beneficial cost savings for students using zero-cost materials without jeopardizing students’ success. Students in our ZCCM cohort had higher course completion rates than their comparison group of peers, and no differences were found between these cohorts in final grades or passing with a C- or better. In addition, enrollment intensity did not vary. These results can alleviate any local concerns about the efficacy of using OER and/or library e-resources in course curricula. Though students’ perceptions of zero-cost materials were positive, more investigation is warranted into why students may have used these materials less frequently than other texts. Usage may have been impacted by the challenges related to unexpected distance education, a primary mode of instructional delivery for some of the classes in this study. While this less frequent engagement with the materials did not seem to negatively impact students’ academic success, it reminds us of the multiple dimensions that contribute to student learning and the need for additional research to uncover what elements of zero-cost materials—whether they are OER or library e-resources—may facilitate student learning. Also of concern, and another area for further study, is the potential impact of textbook costs on historically underserved student populations. These populations may be the ones most likely to lack access to course materials. Feedback from students confirms that they pay attention to textbook costs and seek to avoid or minimize them. Using zero-cost materials relieves some financial burden from our students while still supporting their academic success. These positive outcomes should encourage institutions, librarians, and faculty to strongly advocate for the use of OER and/or library e-resources in course curricula. Ongoing work should also consider how to promote further integration and engagement of course materials in curricula.

References

Allen, G., Guzman-Alvarez, A., Molinaro, M., & Larsen, D. (2015). Assessing the impact and efficacy of the open-access ChemWiki textbook project. EDUCAUSE. https://library.educause.edu/resources/2015/1/assessing-the-impact-and-efficacy-of-the-openaccess-chemwiki-textbook-project

Allen, N. (2022). U.S. open textbook pilot program renewed for sixth year. SPARC. https://sparcopen.org/news/2022/u-s-open-textbook-pilot-program-renewed-for-sixth-year/

Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models.  Biometrics,  61(4), 962-973. https://doi.org/10.1111/j.1541-0420.2005.00377.x

Belikov, O. M., & Bodily, R. (2016). Incentives and barriers to OER adoption: A qualitative analysis of faculty perceptions. Open Praxis, 8(3), 235-246. https://doi.org/10.5944/openpraxis.8.3.308

Bliss, T. J., Robinson, T. J., Hilton, J., & Wiley, D. A. (2013). An OER COUP: College teacher and student perceptions of open educational resources. Journal of Interactive Media in Education, 2013(1). https://doi.org/10.5334/2013-04

Bol, L., Monica, C. E., Ryan, D., & Kimmel, S. C. (2022). A comparison of academic outcomes in courses taught with open educational resources and publisher content. Educational Researcher, 51(1), 17-26. https://doi.org/10.3102/0013189X211052563

Brandle, S., Katz, S., Hays, A., Beth, A., Cooney, C., DiSanto, J., Miles, L., & Morrison, A. (2019). But what do the students think: Results of the CUNY cross-campus zero-textbook cost student survey. Open Praxis, 11(1), 85-101. https://doi.org/10.5944/openpraxis.11.1.932

Brown, M., McCormack, M., Reeves, J., Brooks, D. C., & Grajek, S. (2020). 2020 EDUCAUSE horizon report, teaching and learning edition. EDUCAUSE. https://library.educause.edu/-/media/files/library/2020/3/2020_horizon_report_pdf.pdf?la=en&hash=08A92C17998E8113BCB15DCA7BA1F467F303BA80

Bureau of Labor Statistics, U.S. Department of Labor. (2023). College textbooks in U.S. city average, all urban consumers, not seasonally adjusted. https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SSEA011

California Open Educational Resources Council (2016). OER Adoption Study: Using Open Educational Resources in the College Classroom. California OER Council. http://tinyurl.com/WPOERAdoption040116

Clinton, V. (2018). Savings without sacrifice: A case report on open-source textbook adoption. Open Learning: The Journal of Open, Distance and e-Learning, 33(3), 177-189. https://doi.org/10.1080/02680513.2018.1486184

Clinton, V., & Khan, S. (2019). Efficacy of open textbook adoption on learning performance and course withdrawal rates: A meta-analysis. AERA Open, 5(3). https://doi.org/10.1177/2332858419872212

Colvard, N. B., Watson, C. E., & Park, H. (2018). The impact of open educational resources on various student success metrics. International Journal of Teaching and Learning in Higher Education, 30(2), 262-276. http://files.eric.ed.gov/fulltext/EJ1184998.pdf

Cooney, C. (2017). What impacts do OER have on students? Students share their experiences with a health psychology OER at New York City College of Technology. The International Review of Research in Open and Distributed Learning, 18(4), 155-178. https://doi.org/10.19173/irrodl.v18i4.3111

Cozart, D. L., Horan, E. M., & Frome, G. (2021). Rethinking the traditional textbook: A case for open educational resources (OER) and no-cost learning materials. Teaching & Learning Inquiry, 9(2). https://doi.org/10.20343/teachlearninqu.9.2.13

Croteau, E. (2017). Measures of student success with textbook transformations: The Affordable Learning Georgia Initiative. Open Praxis, 9(1), 93-108. https://doi.org/10.5944/openpraxis.9.1.505

Dastur, F., Jhangiani, R., Le Grand, R., & Penner, K. (2015, November 18-20). Introductory psychology textbooks: The roles of online vs. print and open vs. traditional textbooks [Paper]. Open Education Conference, Vancouver, BC, Canada. https://www.slideshare.net/thatpsychprof/introductory-psychology-textbooks-the-roles-of-online-vs-print-and-open-vs-traditional-textbooks

Dudik, M., Langford, J., & Lihong, L. (2011). Doubly robust policy evaluation and learning. Proceedings of the 28 ^th International Conference on Machine Learning (ICML’11), 1097-1104. https://dl.acm.org/doi/10.5555/3104482.3104620

Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbook and increased student access and outcomes. European Journal of Open, Distance, and e-Learning, 15(2). https://scholars.fhsu.edu/learning_tech_facpubs/4/

Fialkowski, M. K., Calabrese, A., Tilinghast, B., Titchenal, C. A., Meinke, W., Banna, J. C., & Draper, J. (2020). Open educational resource textbook impact on students in an introductory nutrition course. Journal of Nutrition Education and Behavior, 52(4), 359-368. https://doi.org/10.1016/j.jneb.2019.08.006

Fischer, L., Hilton III, J., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159-172. https://doi.org/10.1007/s12528-015-9101-x

Florida Virtual Campus. (2022). 2022 student textbook and instructional materials survey results and findings. Florida Virtual Campus Office of Distance Learning & Student Services. https://assets.website-files.com/646e59f2d76c6e8c0c5223de/64de6132148ed7739bc186e4_FLVC%20Textbook%20Survey%20Report%20-%202022.pdf

Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173(7), 761-767. https://doi.org/10.1093/aje/kwq439

Grewe, K. E., & Davis, W. P. (2017). The impact of enrollment in an OER course on student learning outcomes. The International Review of Research in Open and Distributed Learning, 18(4), 231-238. https://doi.org/10.19173/irrodl.v18i4.2986

Grimaldi, P. J., Basu Mallick, D., Waters, A. E., & Baraniuk, R. G. (2019). Do open educational resources improve student learning? Implications of the access hypothesis. PloS One, 14(3), e0212508. https://doi.org/10.1371/journal.pone.0212508

Gurung, R. A. R. (2017). Predicting learning: Comparing an open educational resource and standard textbooks. Scholarship of Teaching and Learning in Psychology, 3(3), 233-248. https://doi.org/10.1037/stl0000092

Hall, C. E., Steiner, P. M., & Kim, J.-S. (2015). Doubly robust estimation of treatment effects from observational multilevel data. In  Quantitative Psychology Research: The 79^th Annual Meeting of the Psychometric Society, Madison, Wisconsin, 2014 (pp. 321-340). Springer International Publishing. https://doi.org/10.1007/978-3-319-19977-1

Hardin, E. E., Eschman, B., Spengler, E. S., Grizzell, J. A., Moody, A. T., Ross-Sheehy, S., & Fry, K. M. (2019). What happens when trained graduate student instructors switch to an open textbook? A controlled study of the impact on student learning outcomes. Psychology Learning & Teaching, 18(1), 48-64. https://doi.org/10.1177/1475725718810909

Hendricks, C., Reinsberg, S. A., & Rieger, G. W. (2017). The adoption of an open textbook in a large physics course: An analysis of cost, outcomes, use, and perceptions. International Review of Research in Open and Distributed Learning, 18(4), 78-99. https://doi.org/10.19173/irrodl.v18i4.3006

Hilton III, J. (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64, 573-590. https://doi.org/10.1007/s11423-016-9434-9

Hilton III, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open, Distance and e-Learning, 27(3), 265-272. https://doi.org/10.1080/02680513.2012.716657

Hilton III, J. (2020). Open educational resources, student efficacy, and user perceptions: A synthesis of research published between 2015 and 2018. Educational Technology Research and Development, 68, 853-876. https://doi.org/10.1007/s11423-019-09700-4

Hofer, A. (2017). Is the average cost of a textbook $100? OpenOregon Educational Resources. https://openoregon.org/is-the-average-cost-of-a-textbook-100/

Hunsicker-Walburn, M., Guyot, W., Meier, R., Beavers, L., Stainbrook, M., & Schneweis, M. (2018). Students’ perceptions of OER quality. Economics & Business Journal: Inquiries & Perspectives, 9(1), 42-55. https://nebeconandbus.org/journal/EBJIP2018v9WalburnGuyotetalOER_42-55.pdf

Ikahihifo, T. K., Spring, K. J., Rosecrans, J., & Watson, J. (2017). Assessing the savings from open educational resources on student academic goals. International Review of Research in Open and Distributed Learning, 18(7), 126-140. https://doi.org/10.19173/irrodl.v18i7.2754

Jenkins, J. J., Sánchez, L. A., Schraedley, M. A. K., Hannans, J., Navick, N., & Young, J. (2020). Textbook broke: Textbook affordability as a social justice issue. Journal of Interactive Media in Education, 1(3), 1-13. https://doi.org/10.5334/jime.549

Jhangiani, R. S., Dastur, F. N., Le Grand, R., & Penner, K. (2018). As good or better than commercial textbooks: Students’ perceptions and outcomes from using open digital and open print textbooks. The Canadian Journal for the Scholarship of Teaching and Learning, 9(1), 1-20. https://doi.org/10.5206/cjsotl-rcacea.2018.1.5

Lawrence, C. N., & Lester, J. A. (2018). Evaluating the effectiveness of adopting open educational resources in an introductory American government course. Journal of Political Science Education, 14(4), 555-566. https://doi.org/10.1080/15512169.2017.1422739

Mullens, A. M., & Hoffman, B. (2023). The affordability solution: A systematic review of open educational resources. Educational Psychology Review, 35(3), 72. https://doi.org/10.1007/s10648-023-09793-7

National Association of College Stores. (2022). NACS student watch report: Course materials spending dropped. National Association of College Stores. https://www.nacs.org/nacs-student-watch-report-course-materials-spending-dropped

Nusbaum, A. T., Cuttler, C., & Swindell, S. (2020). Open educational resources as a tool for educational equity: Evidence from an introductory psychology class. Frontiers in Education, 4, 1-8. https://doi.org/10.3389/feduc.2019.00152

Nyamweya, M. (2018). A New Method for Estimating OER Savings. SPARC. https://sparcopen.org/news/2018/estimating-oer-student-savings/

Open Education Group. (n.d.-a). The COUP framework. https://web.archive.org/web/20230601084102/http://openedgroup.org/coup

Open Education Group. (n.d.-b). OER Research Toolkit. https://web.archive.org/web/20230601093919/http://openedgroup.org/toolkit

Pawlyshyn, N., Braddlee, B., Casper, L., & Miller, H. (2013). Adopting OER: A case study of cross-institutional collaboration and innovation. https://er.educause.edu/articles/2013/11/adopting-oer-a-case-study-of-crossinstitutional-collaboration-and-innovation

Pelletier, K., Brown, M., Brooks, D. C., McCormack, M., Reeves, J., & Arbino, N. (2021). 2021 EDUCAUSE horizon report: Teaching and learning edition. EDUCAUSE. http://eric.ed.gov/ERICWebPortal/detail?accno=ED614350

Robinson, T. J. (2015). The effects of open educational resource adoption on measures of post-secondary student success. https://scholarsarchive.byu.edu/etd/5815

Ross, H. M., Hendricks, C., & Mowat, V. (2018). Open textbooks in an introductory sociology course in Canada: Student views and completion rates. Open Praxis, 10(4), 393-403. https://doi.org/10.5944/openpraxis.10.4.892

Sack Illowsky, B., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016). Examining student perception of an open statistics book. Open Praxis, 8(3), 265-276. https://doi.org/10.5944/openpraxis.8.3.304

Sant’Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101-122. https://doi.org/10.1016/j.jeconom.2020.06.003

Senack, E. (2014). Fixing the broken textbook market: How students respond to high textbook costs and demand alternative. PIRG. https://pirg.org/resources/fixing-the-broken-textbook-market-2/

Smith Jaggars, S., Rivera, M. D., & Akani, B. (2019). College textbook affordability: Landscape, evidence, and policy directions. Midwestern Higher Education Compact. https://eric.ed.gov/?id=ED598412

Suriano, Z. J. (2023). Impact of open educational resources on student performance in an introductory geography course. Journal of Geography in Higher Education, 1-10. https://doi.org/10.1080/03098265.2023.2255556

Tan, X., Yang, S., Ye, W., Faries, D. E., Lipkovich, I., & Kadziola, Z. (2022). When doubly robust methods meet machine learning for estimating treatment effects from real-world data: A comparative study. arXiv. https://doi.org/10.48550/arXiv.2204.10969

UC Merced, Institutional Research & Decision Support. (2022a). 2022 UC Merced undergraduate enrollments, demographics. UC Merced, Center for Institutional Effectiveness. https://cie.ucmerced.edu/undergraduate-enrollments

UC Merced, Institutional Research & Decision Support. (2022b). 2022 UC Merced new student survey dashboard, enrolled in less than 15 units. UC Merced, Center for Institutional Effectiveness. https://cie.ucmerced.edu/analytics-hub/surveys/new-student-survey-data

UNESCO. (2002). Forum on the Impact of Open Courseware for Higher Education in Developing Countries: Final report. https://unesdoc.unesco.org/ark:/48223/pf0000128515

UNESCO. (n.d.). Open educational resources. https://www.unesco.org/en/open-educational-resources

Watson, C. E., Domizi, D. P., & Clouser, S. A. (2017). Student and faculty perceptions of OpenStax in high enrollment courses. The International Review of Research in Open and Distributed Learning, 18(5), 287-304. https://doi.org/10.19173/irrodl.v18i5.2462

Wiley, D., Williams, L., DeMarte, D., & Hilton, J. (2016). The Tidewater Z-Degree and the INTRO model for sustaining OER adoption. Education Policy Analysis Archives, 24(41). 1-15. https://doi.org/10.14507/epaa.24.1828

Winitzky-Stephens, J. R., & Pickavance, J. (2017). Open educational resources and student course outcomes: A multilevel analysis. The International Review of Research in Open and Distributed Learning, 18(4), 35-49. https://doi.org/10.19173/irrodl.v18i4.3118

Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data (2nd ed.). The MIT Press.

Measuring the Impact of an Open Educational Resource and Library e-Resource Adoption Program Using the COUP Framework by Sara Davidson Squibb, Elizabeth Salmon, and Yueqi Yan is licensed under a Creative Commons Attribution 4.0 International License.