Volume 18, Number 4
Kim E. Grewe and Dr. W. Preston Davis
Northern Virginia Community College
Open Educational Resources (OER) are gaining acceptance as legitimate and effective teaching materials in higher education, particularly in 2-year institutions. Despite the steady growth in the availability and use of OER, there have been relatively few studies on the efficacy of OER and student achievement. This study analyzed the effect enrollment in an OER course had on student academic achievement when controlling for prior academic achievement in an introductory online history course at a large community college. Correlation analysis and simple linear regression were conducted. The results of this research indicate a significant positive moderate correlation between OER and student achievement. The study provided a process by which future, more rigorous efficacy studies can be conducted.
Keywords: open educational resources, OER, student learning outcomes, graduate student research, pilot research study
For the past decade, a more open approach to learning has emerged. The Open Educational Resources, or OER, movement is changing the way students learn and institutions operate. The most widely used definition of OER comes from the William and Flora Hewlett Foundation:
OER are teaching, learning, and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use and re-purposing by others. Open educational resources include full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools materials or techniques used to support access to knowledge. (2013, para. 2)
The Open Movement began in a wider context of a culture of open knowledge and peer collaboration. More recently, the OER movement has gained traction, particularly in American community colleges with their open admissions policies and inclusive philosophies. OER have proven to be a key component to student access and success, providing free or low-cost textbook alternatives to financially strapped students (Bradley, 2013). As more community colleges are adopting OER (Ashford, 2017), it is important to examine the impact of OER beyond cost saving to students.
To date, there have been relatively few studies on the efficacy of OER and student achievement. Robinson, Fischer, Wiley, and Hilton III (2014) used a quantitative quasi-experimental design to determine whether the choice of open textbooks had a significant impact on student science learning in public high school. After controlling for teacher efficacy and student characteristics, the researchers found that the use of OER did have a statistically significant effect on student learning in chemistry but not in biology or earth systems as demonstrated by student scores on end-of-year state standardized test scores. This study demonstrated that the adoption of OER can both dramatically reduce cost for students while at the same time improve student learning or, at the minimum, not negatively impact student learning. This study took place in a secondary setting, differentiating it from other studies reviewed here. Even so, the study does provide a framework for further efficacy studies, whether in secondary or post-secondary school settings. Robinson (2015) examined Open Educational Resource (OER) adoption as a correlate of post-secondary school students' academic success. Using an ex-post facto quasi-experimental research design, the study analyzed the academic performances of students using the open textbooks and those that were using the hardcopy traditional textbooks in seven Project Kaleidoscope post-secondary educational institutions. Having controlled for selection bias using propensity score matching (PSM), multiple regression and logistic regression models were used to analyze the effect of the OER on the students' academic performance. The findings showed that students using the traditional textbooks performed relatively better than those using the open textbooks in business and psychology courses. At the same time, students in this study who used open textbooks enrolled in more credits than students who used textbooks.
Fischer, Hilton III, Robinson, and Wiley (2015) analyzed the 2nd year of OER adoption in the same schools as did Robinson (2015) (who examined the first year of adoption). They sought to know whether the adoption of no-cost digital textbooks had an impact on post-secondary students' learning outcomes. The study examined course completion, class achievement, and intensity of enrollment during and after semesters in which students used OER and compared these to the same outcomes for students enrolled in the same courses that did not use OER. The sample was large, including 16,727 students, 4909 who were in the treatment group and 11,818 who were in the control group enrolled in 15 different undergraduate classes across 10 different institutions spread out across the United States. The size of this study makes the results more generalizable to a larger population than any other study of its kind conducted to date. There were statistically significant differences in enrollment intensity between students enrolled in courses which used OER and those that did not, with students who used OER taking more credits than their counterparts. There were also moderate differences in completion rates and final grades between the two groups, with those in OER courses performing as well as or better than their counterparts in courses where faculty used commercial textbooks.
Hilton III, Gaudet, Clark, Robinson and Wiley (2013) examined the implications of the adoption of OER in a Math Department of the Scottsdale Community College (SCC) in Arizona. The large sample size included 2,043 students. Specifically, the study examined the cost benefits of OER and the impact on the students' retention as well as the perception of the students and the faculty about the quality of the OER. The study was conducted in Fall 2012. The metrics used in measuring the students' learning outcomes include the withdrawal rate and C grade score or better reported by the college. Findings revealed that use of the open textbooks resulted in a significant cost saving. However, there was no significant change in the withdrawal rate and the students' retention level attributable to the use of the open textbooks. In terms of perception of the open textbooks' quality, both the students and the faculty perceived the books as being of good quality.
Hilton III and Laman (2012) investigated the efficacy of online open textbooks in improving students' academic performance at Houston Community College (HCC). The pilot study assessed the academic performance of students who used the traditional texts in the Spring 2011 semester and those who used open textbooks during the Fall 2011 semester. Grade point average, withdrawal rate, and departmental examination scores constituted the performance measures. Findings revealed that students who used the free online textbook had better overall outcomes than students who used the traditional textbooks. The study, however, could not establish a causal relationship. Future studies are needed to establish the use and adoption of OER as the driver of positive student learning outcomes.
Feldstein et al. (2012) also examined the impact of the adoption of the open textbooks on access and learning outcomes of students in the Virginia State University School of Business in a pilot study. All the participating students took courses that used Flat World Knowledge (FWK) open online textbooks between the Fall 2010 and Spring 2011. Students were also provided access to an array of supplementary online resources. Grades in the courses were used as measures of the students' learning outcomes. The researchers compared the grades of students in core courses using the FWK open text and materials to students enrolled in the same core courses that used traditional texts. The results revealed that there was a significant
Lovett, Meyer, and Thille (2008) evaluated the effectiveness of the hybrid Open Learning Initiative (OLI) Statistics course developed at Carnegie Mellon University (CMU) in what they called an "Accelerated Learning" study in the Spring of 2007. The study employed a quasi-experimental study design in which students were assigned to different groups: those who used OLI-Statistics to supplement classroom instruction or those who received the traditional classroom instruction by itself. Twenty-two students were randomly selected to use the OLI-Statistics course in hybrid mode, while 42 students were assigned to the control group. The control group received traditional classroom instruction. Participants for both groups were selected from a pool of volunteers. The learning outcomes measured by the in-class exams scores of the OLI-Statistics group were compared with those of the instructor-led groups. Findings revealed that the OLI group completed the course in about half the time of the control group and achieved similar learning outcomes.
Bowen, Chingos, Lack, and Nygren (2014) examined the learning outcomes of a hybrid machine-guided online interactive learning termed Interactive Learning Online (ILO). The study employed a randomized trial study design with two independent groups of public university students in six campuses in the United States. In essence, this study was a replication and extension of the Lovett et al. (2008) study. The first group was assigned to the traditional manual face-to-face teaching and learning while the other group received machine-guided instruction online. The researchers analyzed the learning outcomes of the two groups in the Carnegie Mellon University (CMU)-developed Statistics course. The course includes textual instructions and explanation, practice problems, and worked examples. They measured the groups' learning outcomes on a standardized statistical literacy metric including pass rates and final exam scores. Findings revealed similar learning outcomes from the two groups. A speculative cost simulation analysis conducted by the authors, however, revealed significant cost savings with the hybrid machine-guided instruction learning model.
Allen, Guzman-Alvarez, Molinaro, and Larsen (2015) assessed the impact and efficacy of the open-access ChemWiki Textbook in a general chemistry class at the University of California in a pilot study during the 2014 Spring semester. The ChemWiki Textbook Project was planned to promote the use of open online textbooks as an alternative to the traditional hardcopy textbooks. The study employed an experimental design with an experimental group using the ChemWiki Textook in one chemistry class and a control group using a traditional chemistry text in the other chemistry class. The researchers analyzed the performance of the two groups, controlling for possible instructor bias by using the same set of instructors to teach both classes. The researchers examined grades obtained during in-class assessments, pre/post exam scores, and student attitude and self-reported study habits as measures of the students' learning outcomes. Although there was a difference in the students' attitudes and beliefs between the two groups, the difference was not statistically significant. Likewise, no significant difference was found in the students' learning outcomes or study habits between the two groups.
The efficacy studies performed to date indicate that students who use OER do about as well or better as students who enroll in courses which use traditional materials. However, there are still limitations to the work that has been done to date. A particular weakness of more than half of the OER efficacy studies performed to date is that they make no effort to control differences in student characteristics or success metrics.
The present study seeks to address this gap in the literature by examining the academic achievement (grades) of students enrolled in an introductory online history course which utilized OER compared to students enrolled in the same introductory online history course which used traditional textbooks. Specifically, the study attempted to answer this research question: What effect does enrollment in an OER course have on student achievement in that course when controlling for prior academic achievement?
The data for this study was collected from the student information system at Northern Virginia Community College (NOVA), a large, multi-campus urban community college located near Washington, D.C. The sample includes 146 students. These students were enrolled in an online 8-week section of an introductory history class in the Fall of 2013 or in the Spring of 2014. In Fall 2013, there were two sections of the course which utilized OER and two sections which did not. In Spring 2014, there were three sections of the course which utilized OER and three which did not. Of the students enrolled across all sections of the courses, 64% were female and 36% were male. Since the study utilized student information after the fact, there was no way to control for selection bias.
The student information system allowed identification of the study's predictor variables: student cumulative previous GPA prior to taking the online introductory history course (prior academic achievement) and identification of the course as either an OER version or a non-OER version. A correlation analysis was computed to assess the relationship between prior academic achievement and student grade in the course (student achievement). The correlation analysis also assessed the relationship between enrollment in an OER version of the introductory history course and student achievement.
In addition, a multiple linear regression was calculated to predict student grade in the introductory history course (student achievement) based on OER and prior academic achievement. Dummy variables were created for course section identification with OER being the baseline group for comparison to non-OER sections of the introductory history course. Also, students' final course grades were converted from letters to numbers, i.e. A=4, B=3, C=2, D=1, and F=0. For this study, W grades were converted to 0.
There was a moderate positive correlation between prior academic achievement and student achievement (r = .30, p < .001). There was an even stronger positive moderate correlation between OER versus Non-OER and student achievement (r =.41, p < .001). These results show that there is a moderately positive relationship between taking an OER course and academic achievement.
A regression of student grade in the introductory history course on student GPA prior to taking the online introductory history course and identification of the course as either an OER version or a non-OER version explained 22% of the variance in student Number Grade in the introductory history course F (2, 143) = 20.30, MSE = 1.67, p < .001.
Student cumulative previous GPA was a significant predictor of student number grade in the introductory history course (β = .34, p < .01), which indicates that a .34 increase in previous student GPA led to a one standard deviation increase in student grade in the introductory history course after controlling for OER vs. non-OER. Enrollment in an OER course was also a significant predictor of student grade achievement (b = .37, p < .001).
The results indicate that there is a positive relationship between the use of OER and student academic achievement in an online history course. Because of the lack of demographic variables, we cannot definitively conclude that OER leads to increased student achievement. However, by controlling for prior college achievement, we show there may be evidence to support this claim. Students enrolled in the OER sections of the introductory history course performed as well or better than students enrolled in sections that used commercial textbooks. Usually, comparative studies examine the mode of delivery or instructional design as possible factors influencing student learning outcomes (Russell, 2015). One factor that this study was able to control for was the instructional design of the course. This study examined final course grades of students enrolled in 8-week sections of online courses with an identical instructional design.
Other factors were not controlled in this study. First, the study did not control for teacher effect. Even in online courses created from a master template, teacher effect can impact student learning. How often an instructor posts announcements, responds to class discussion forums, answers student email inquiries, and grades student work can have an impact on student success in a course. The study also did not control for student characteristics besides previous GPA, which may also have a significant impact on academic achievement. It would be useful to examine the interaction effect of these variables in future studies that examine the impact of OER on student academic achievement.
While the results of the study may not have implications for OER efficacy per se, the study is important for showcasing the ways that researchers in community colleges can begin to use data available at their institutions to study the effect of OER on student learning. While a more robust study, which includes a greater sample size and controls for confounding variables is needed, certainly this study provides the first step in developing an experimental research design that does have implications for future OER research.
Further research is needed within other contexts to determine which elements of OER seem to have the greatest impact on student learning outcomes. Also, since this study was limited to one online course over the duration of one academic year, similar experiments, which control for various potential confounding variables must be undertaken in larger contexts to substantiate or repudiate the findings of this study. Finally, future studies may also examine the ways in which OER interact with particular subject areas and what text characteristics make some OER better than others.
This study may be helpful for community college leaders who want to begin to use data more effectively or who seek to take on more of researcher/practitioner role at their institutions. By presenting a simple research model on which to build, this study can motivate community college leaders to conduct research of their own using data available from their own institutions to make decisions about OER use, adoption, and policy.
Allen, G., Guzman-Alvarez, A., Molinaro, M., & Larsen, D. (2015). Assessing the impact and efficacy of the Open-Access ChemWiki Textbook Project. Educause Learning Initiative Brief, 1–8.
Ashford, E. (2017, February 3). More colleges use open educational resources. Community College Daily. Retrieved February 13, 2017, from http://www.ccdaily.com/Pages/Academic-Programs/More-colleges-adopt-OER-degrees.aspx
Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). interactive learning online at public universities: Evidence from a six-campus randomized trial: Interactive learning online at public universities. Journal of Policy Analysis and Management, 33(1), 94–111. doi: https://doi.org/10.1002/pam.21728
Bradley, P. (2013, May 13). Cover story: Opening the books. Retrieved from http://ccweek.com/article-3414-cover-story-opening-the-books.html
Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance, and E-Learning, 15(2). Retrieved from http://www.eurodl.org/materials/contrib/2012/Feldsteint_et_al.pdf
Fischer, L., Hilton III, J., Robinson, J., & Wiley, D. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Interactive Media in Education, 27(3), 159-172. doi: 10.1007/s12528-015-9101-x
Hilton III, J. & Laman, C. (2012). One college's use of an open psychology textbook. Open Learning: The Journal of Open, Distance, and E-Learning, 27(3), 265–272. doi: https://doi.org/10.1080/02680513.2012.716657
Hilton III, J. L., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. The International Review of Research in Open and Distributed Learning, 14(4), 38–50.
Lovett, M., Meyer, O. & Thille, C. (2008). JIME-The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education, 2008(1). Retrieved from http://jime.open.ac.uk/article/view/142/jime-2008-14.pdf/
Robinson, T. J. (2015, May). The effects of open educational resource adoption on measures of post-secondary student success (Doctoral dissertation, Brigham Young University). Retrieved from http://scholarsarchive.byu.edu/etd/5815/
Robinson, T. J., Fischer, L., Wiley, D., & Hilton III, J. (2014). The impact of open textbooks on secondary science learning outcomes. Educational Researcher, 43(7), 341-51. doi: 10.3102/0013189X14550275
Russell, T. L. (2015). No significant difference. Retrieved from http://www.nosignificantdifference.org/
William and Flora Hewlett Foundation. (2013). Open educational resources [Web blog]. Retrieved from http://www.hewlett.org/programs/education/open-educational-resources
The Impact of Enrollment in an OER Course on Student Learning Outcomes by Kim E. Grewe and Dr. W. Preston Davis is licensed under a Creative Commons Attribution 4.0 International License.