Measuring the Impact of an Open Educational Resource and Library e-Resource Adoption Program Using the COUP Framework
DOI:
https://doi.org/10.19173/irrodl.v24i4.7420Keywords:
open educational resources, OER, library e-resources, COUP framework, cost savings, student outcomesAbstract
Grant programs that foster the use of open educational resources (OER) significantly reduce undergraduate student spending on textbooks per semester. The Zero-Cost Course Materials (ZCCM) grant program at the University of California, Merced (UC Merced), eliminated text costs and ensured access to course materials by replacing required commercial materials with OER and library licensed e-resources. The present study applies the COUP framework (cost, outcomes, usage, and perceptions) to evaluate the ZCCM program. The ZCCM program resulted in student cost savings and did not negatively impact student outcomes. Students in ZCCM courses demonstrated higher rates of course completion than students enrolled in previous sections. For the outcomes of final course grade, passing with a C− or better, and number of credit hours enrolled in, findings were comparable between the cohorts. Student usage and perception of course materials were gathered using a survey. Though students reported favorable views of zero-cost materials, they reported using them less frequently than commercial texts. This research contributes to a growing body of literature that confirms beneficial cost savings for students using zero-cost materials without jeopardizing students’ success.
References
Allen, G., Guzman-Alvarez, A., Molinaro, M., & Larsen, D. (2015). Assessing the impact and efficacy of the open-access ChemWiki textbook project. EDUCAUSE. https://library.educause.edu/resources/2015/1/assessing-the-impact-and-efficacy-of-the-openaccess-chemwiki-textbook-project
Allen, N. (2022). U.S. open textbook pilot program renewed for sixth year. SPARC. https://sparcopen.org/news/2022/u-s-open-textbook-pilot-program-renewed-for-sixth-year/
Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4), 962–973. https://doi.org/10.1111/j.1541-0420.2005.00377.x
Belikov, O. M., & Bodily, R. (2016). Incentives and barriers to OER adoption: A qualitative analysis of faculty perceptions. Open Praxis, 8(3), 235–246. https://doi.org/10.5944/openpraxis.8.3.308
Bliss, T. J., Robinson, T. J., Hilton, J., & Wiley, D. A. (2013). An OER COUP: College teacher and student perceptions of open educational resources. Journal of Interactive Media in Education, 2013(1). https://doi.org/10.5334/2013-04
Bol, L., Monica, C. E., Ryan, D., & Kimmel, S. C. (2022). A comparison of academic outcomes in courses taught with open educational resources and publisher content. Educational Researcher, 51(1), 17–26. https://doi.org/10.3102/0013189X211052563
Brandle, S., Katz, S., Hays, A., Beth, A., Cooney, C., DiSanto, J., Miles, L., & Morrison, A. (2019). But what do the students think: Results of the CUNY cross-campus zero-textbook cost student survey. Open Praxis, 11(1), 85–101. https://doi.org/10.5944/openpraxis.11.1.932
Brown, M., McCormack, M., Reeves, J., Brooks, D. C., & Grajek, S. (2020). 2020 EDUCAUSE horizon report, teaching and learning edition. EDUCAUSE. https://library.educause.edu/-/media/files/library/2020/3/2020_horizon_report_pdf.pdf?la=en&hash=08A92C17998E8113BCB15DCA7BA1F467F303BA80
Bureau of Labor Statistics, U.S. Department of Labor. (2023). College textbooks in U.S. city average, all urban consumers, not seasonally adjusted. https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SSEA011
California Open Educational Resources Council (2016). OER Adoption Study: Using Open Educational Resources in the College Classroom. California OER Council. http://tinyurl.com/WPOERAdoption040116
Clinton, V. (2018). Savings without sacrifice: A case report on open-source textbook adoption. Open Learning: The Journal of Open, Distance and e-Learning, 33(3), 177–189. https://doi.org/10.1080/02680513.2018.1486184
Clinton, V., & Khan, S. (2019). Efficacy of open textbook adoption on learning performance and course withdrawal rates: A meta-analysis. AERA Open, 5(3). https://doi.org/10.1177/2332858419872212
Colvard, N. B., Watson, C. E., & Park, H. (2018). The impact of open educational resources on various student success metrics. International Journal of Teaching and Learning in Higher Education, 30(2), 262–276. http://files.eric.ed.gov/fulltext/EJ1184998.pdf
Cooney, C. (2017). What impacts do OER have on students? Students share their experiences with a health psychology OER at New York City College of Technology. The International Review of Research in Open and Distributed Learning, 18(4), 155–178. https://doi.org/10.19173/irrodl.v18i4.3111
Cozart, D. L., Horan, E. M., & Frome, G. (2021). Rethinking the traditional textbook: A case for open educational resources (OER) and no-cost learning materials. Teaching & Learning Inquiry, 9(2). https://doi.org/10.20343/teachlearninqu.9.2.13
Croteau, E. (2017). Measures of student success with textbook transformations: The Affordable Learning Georgia Initiative. Open Praxis, 9(1), 93–108. https://doi.org/10.5944/openpraxis.9.1.505
Dastur, F., Jhangiani, R., Le Grand, R., & Penner, K. (2015, November 18–20). Introductory psychology textbooks: The roles of online vs. print and open vs. traditional textbooks [Paper]. Open Education Conference, Vancouver, BC, Canada. https://www.slideshare.net/thatpsychprof/introductory-psychology-textbooks-the-roles-of-online-vs-print-and-open-vs-traditional-textbooks
Dudik, M., Langford, J., & Lihong, L. (2011). Doubly robust policy evaluation and learning. Proceedings of the 28th International Conference on Machine Learning (ICML’11), 1097–1104. https://dl.acm.org/doi/10.5555/3104482.3104620
Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbook and increased student access and outcomes. European Journal of Open, Distance, and e-Learning, 15(2). https://scholars.fhsu.edu/learning_tech_facpubs/4/
Fialkowski, M. K., Calabrese, A., Tilinghast, B., Titchenal, C. A., Meinke, W., Banna, J. C., & Draper, J. (2020). Open educational resource textbook impact on students in an introductory nutrition course. Journal of Nutrition Education and Behavior, 52(4), 359–368. https://doi.org/10.1016/j.jneb.2019.08.006
Fischer, L., Hilton III, J., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159–172. https://doi.org/10.1007/s12528-015-9101-x
Florida Virtual Campus. (2022). 2022 student textbook and instructional materials survey results and findings. Florida Virtual Campus Office of Distance Learning & Student Services. https://assets.website-files.com/646e59f2d76c6e8c0c5223de/64de6132148ed7739bc186e4_FLVC%20Textbook%20Survey%20Report%20-%202022.pdf
Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173(7), 761–767. https://doi.org/10.1093/aje/kwq439
Grewe, K. E., & Davis, W. P. (2017). The impact of enrollment in an OER course on student learning outcomes. The International Review of Research in Open and Distributed Learning, 18(4), 231–238. https://doi.org/10.19173/irrodl.v18i4.2986
Grimaldi, P. J., Basu Mallick, D., Waters, A. E., & Baraniuk, R. G. (2019). Do open educational resources improve student learning? Implications of the access hypothesis. PloS One, 14(3), e0212508. https://doi.org/10.1371/journal.pone.0212508
Gurung, R. A. R. (2017). Predicting learning: Comparing an open educational resource and standard textbooks. Scholarship of Teaching and Learning in Psychology, 3(3), 233–248. https://doi.org/10.1037/stl0000092
Hall, C. E., Steiner, P. M., & Kim, J.-S. (2015). Doubly robust estimation of treatment effects from observational multilevel data. In Quantitative Psychology Research: The 79th Annual Meeting of the Psychometric Society, Madison, Wisconsin, 2014 (pp. 321–340). Springer International Publishing. https://doi.org/10.1007/978-3-319-19977-1
Hardin, E. E., Eschman, B., Spengler, E. S., Grizzell, J. A., Moody, A. T., Ross-Sheehy, S., & Fry, K. M. (2019). What happens when trained graduate student instructors switch to an open textbook? A controlled study of the impact on student learning outcomes. Psychology Learning & Teaching, 18(1), 48–64. https://doi.org/10.1177/1475725718810909
Hendricks, C., Reinsberg, S. A., & Rieger, G. W. (2017). The adoption of an open textbook in a large physics course: An analysis of cost, outcomes, use, and perceptions. International Review of Research in Open and Distributed Learning, 18(4), 78–99. https://doi.org/10.19173/irrodl.v18i4.3006
Hilton III, J. (2016). Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development, 64, 573–590. https://doi.org/10.1007/s11423-016-9434-9
Hilton III, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open, Distance and e-Learning, 27(3), 265–272. https://doi.org/10.1080/02680513.2012.716657
Hilton III, J. (2020). Open educational resources, student efficacy, and user perceptions: A synthesis of research published between 2015 and 2018. Educational Technology Research and Development, 68, 853–876. https://doi.org/10.1007/s11423-019-09700-4
Hofer, A. (2017). Is the average cost of a textbook $100? OpenOregon Educational Resources. https://openoregon.org/is-the-average-cost-of-a-textbook-100/
Hunsicker-Walburn, M., Guyot, W., Meier, R., Beavers, L., Stainbrook, M., & Schneweis, M. (2018). Students’ perceptions of OER quality. Economics & Business Journal: Inquiries & Perspectives, 9(1), 42–55. https://nebeconandbus.org/journal/EBJIP2018v9WalburnGuyotetalOER_42-55.pdf
Ikahihifo, T. K., Spring, K. J., Rosecrans, J., & Watson, J. (2017). Assessing the savings from open educational resources on student academic goals. International Review of Research in Open and Distributed Learning, 18(7), 126–140. https://doi.org/10.19173/irrodl.v18i7.2754
Jenkins, J. J., Sánchez, L. A., Schraedley, M. A. K., Hannans, J., Navick, N., & Young, J. (2020). Textbook broke: Textbook affordability as a social justice issue. Journal of Interactive Media in Education, 1(3), 1–13. https://doi.org/10.5334/jime.549
Jhangiani, R. S., Dastur, F. N., Le Grand, R., & Penner, K. (2018). As good or better than commercial textbooks: Students’ perceptions and outcomes from using open digital and open print textbooks. The Canadian Journal for the Scholarship of Teaching and Learning, 9(1), 1–20. https://doi.org/10.5206/cjsotl-rcacea.2018.1.5
Lawrence, C. N., & Lester, J. A. (2018). Evaluating the effectiveness of adopting open educational resources in an introductory American government course. Journal of Political Science Education, 14(4), 555–566. https://doi.org/10.1080/15512169.2017.1422739
Mullens, A. M., & Hoffman, B. (2023). The affordability solution: A systematic review of open educational resources. Educational Psychology Review, 35(3), 72. https://doi.org/10.1007/s10648-023-09793-7
National Association of College Stores. (2022). NACS student watch report: Course materials spending dropped. National Association of College Stores. https://www.nacs.org/nacs-student-watch-report-course-materials-spending-dropped
Nusbaum, A. T., Cuttler, C., & Swindell, S. (2020). Open educational resources as a tool for educational equity: Evidence from an introductory psychology class. Frontiers in Education, 4, 1–8. https://doi.org/10.3389/feduc.2019.00152
Nyamweya, M. (2018). A New Method for Estimating OER Savings. SPARC. https://sparcopen.org/news/2018/estimating-oer-student-savings/
Open Education Group. (n.d.-a). The COUP framework. https://web.archive.org/web/20230601084102/http://openedgroup.org/coup
Open Education Group. (n.d.-b). OER Research Toolkit. https://web.archive.org/web/20230601093919/http://openedgroup.org/toolkit
Pawlyshyn, N., Braddlee, B., Casper, L., & Miller, H. (2013). Adopting OER: A case study of cross-institutional collaboration and innovation. https://er.educause.edu/articles/2013/11/adopting-oer-a-case-study-of-crossinstitutional-collaboration-and-innovation
Pelletier, K., Brown, M., Brooks, D. C., McCormack, M., Reeves, J., & Arbino, N. (2021). 2021 EDUCAUSE horizon report: Teaching and learning edition. EDUCAUSE. http://eric.ed.gov/ERICWebPortal/detail?accno=ED614350
Robinson, T. J. (2015). The effects of open educational resource adoption on measures of post-secondary student success. https://scholarsarchive.byu.edu/etd/5815
Ross, H. M., Hendricks, C., & Mowat, V. (2018). Open textbooks in an introductory sociology course in Canada: Student views and completion rates. Open Praxis, 10(4), 393–403. https://doi.org/10.5944/openpraxis.10.4.892
Sack Illowsky, B., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016). Examining student perception of an open statistics book. Open Praxis, 8(3), 265–276. https://doi.org/10.5944/openpraxis.8.3.304
Sant’Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122. https://doi.org/10.1016/j.jeconom.2020.06.003
Senack, E. (2014). Fixing the broken textbook market: How students respond to high textbook costs and demand alternative. PIRG. https://pirg.org/resources/fixing-the-broken-textbook-market-2/
Smith Jaggars, S., Rivera, M. D., & Akani, B. (2019). College textbook affordability: Landscape, evidence, and policy directions. Midwestern Higher Education Compact. https://eric.ed.gov/?id=ED598412
Suriano, Z. J. (2023). Impact of open educational resources on student performance in an introductory geography course. Journal of Geography in Higher Education, 1–10. https://doi.org/10.1080/03098265.2023.2255556
Tan, X., Yang, S., Ye, W., Faries, D. E., Lipkovich, I., & Kadziola, Z. (2022). When doubly robust methods meet machine learning for estimating treatment effects from real-world data: A comparative study. arXiv. https://doi.org/10.48550/arXiv.2204.10969
UC Merced, Institutional Research & Decision Support. (2022a). 2022 UC Merced undergraduate enrollments, demographics. UC Merced, Center for Institutional Effectiveness. https://cie.ucmerced.edu/undergraduate-enrollments
UC Merced, Institutional Research & Decision Support. (2022b). 2022 UC Merced new student survey dashboard, enrolled in less than 15 units. UC Merced, Center for Institutional Effectiveness. https://cie.ucmerced.edu/analytics-hub/surveys/new-student-survey-data
UNESCO. (2002). Forum on the Impact of Open Courseware for Higher Education in Developing Countries: Final report. https://unesdoc.unesco.org/ark:/48223/pf0000128515
UNESCO. (n.d.). Open educational resources. https://www.unesco.org/en/open-educational-resources
Watson, C. E., Domizi, D. P., & Clouser, S. A. (2017). Student and faculty perceptions of OpenStax in high enrollment courses. The International Review of Research in Open and Distributed Learning, 18(5), 287–304. https://doi.org/10.19173/irrodl.v18i5.2462
Wiley, D., Williams, L., DeMarte, D., & Hilton, J. (2016). The Tidewater Z-Degree and the INTRO model for sustaining OER adoption. Education Policy Analysis Archives, 24(41). 1–15. https://doi.org/10.14507/epaa.24.1828
Winitzky-Stephens, J. R., & Pickavance, J. (2017). Open educational resources and student course outcomes: A multilevel analysis. The International Review of Research in Open and Distributed Learning, 18(4), 35–49. https://doi.org/10.19173/irrodl.v18i4.3118
Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data (2nd ed.). The MIT Press.
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.
This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.
You are free to
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms below:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
