Volume 18, Number 6
Scott Woodward, PhD, Adam Lloyd, and Royce Kimmons, PhD
Brigham Young University
Advocates for student voice in higher education believe students should have the right and power to engage in much of the decision-making traditionally dominated by instructors or administrators. This qualitative study examines the role of student voice in the evaluation of textbook quality. Evaluators included two graduate students enrolled in a project management course at a university in the western U.S. and their instructor. Evaluators used their own student-created metric to analyze the comparative quality of eight graduate-level project management textbooks, two of which were open and six copyright-restricted. The purposes of this study were to assess the comparative quality of low-cost open textbooks and traditional copyright-restricted textbooks and to identify key student-generated criteria wherein all textbooks may be improved to better serve student needs. The analysis revealed that textbooks can be rigorously and meaningfully evaluated by students, that open textbooks can compete with restricted textbooks in terms of quality, and that polyphonic approaches to textbook evaluation can be valuable for learning. We discuss the implications of open textbooks as viable, high-quality classroom options, and the importance of valuing both student voice and instructor perspectives to ensure the highest quality textbook selection for courses in higher education.
Keywords: open educational resources, open pedagogy, open textbooks
Student voice refers to the "collective contribution of diverse students' presence, participation, and power... [in] educational planning, research, and reform" (Cook-Sather, 2014, p. 132), and its advocates in higher education believe that students should have the right and power to engage in much of the decision-making traditionally dominated by instructors or administrators. In part, this has been exemplified in a shift from research on students to research with students (Cook-Sather, 2006; Fine, Torre, Bums, & Payne, 2007), and few areas of practice are as isolated from student voice, as are the processes surrounding textbook evaluation and adoption. Due to advancements in digital technologies, which allow content to be openly licensed and freely distributed, textbooks themselves are in a state of flux (cf. Denoyelles, Raible, & Seilhamer, 2015), and open textbooks have emerged as an alternative to traditional copyright-restricted textbooks. Open textbooks differ from traditional textbooks in that they: (1) can be distributed with no licensing costs, and (2) can be edited by instructors without permission from the original authors. As one example of an open educational resource (OER), open textbooks may include other OER within them (e.g., images) and also may be broken up themselves for partial redistribution as smaller, focused resources (e.g., individual chapters).
Open textbooks are of great interest to students and instructors in large part because high textbook costs have a discernible impact in the classroom. One study found that the persistence probability for poor and working-class students in higher education decreases by 16-19% for every 1,000 USD tuition increase (Paulsen & St. John, 2002), which was less than the average cost of textbooks in a single year (The College Board, 2015). Similarly, one university system found that 56% of its students "worried very often" about family financial circumstances, and 81% spent less money on food to meet college expenses (UCR VCSA, 2011). Another study found that many students decided against buying an assigned textbook due to cost, and almost all of these were concerned that doing so would hurt their grades and believed that access to a free textbook would help them do better in their courses (Senack, 2014). Such cost and academic concerns are of importance to online and distance learning programs as well as traditional face-to-face programs, because many online courses are developed around costly textbooks, which drives up associated costs, and also face additional enrollment challenges.
Introduction of open textbooks requires new evaluation mechanisms that are faster (i.e., responsive to a massive amount of possible resources), more inclusive (i.e., recognizing value from multiple sources), more contextually appropriate (i.e., allowing for adaptation for localized needs), and more sensitive to student needs (i.e., cost, learning needs). In response to this special issue's call for proposals, this study explores the third trajectory of student voice in research methodology by placing enrolled students as the primary evaluators of potential classroom textbooks for a graduate-level course for the purpose of: a) conducting valuable research on the comparative quality of low-cost open textbooks vs. traditional copyright-restricted textbooks, b) guiding departmental adoptions of high-quality textbooks, and c) identifying key criteria wherein all textbooks may be improved to better serve student needs. The role of students in this study is that of "active agents," rather than "silent witnesses" (cf. Smyth & McInerney, 2012), as students conducted research on their own learning experience while providing valuable findings on textbook quality and polyphonic evaluation. We will proceed by briefly exploring literature surrounding open textbooks and evaluation, highlighting our methods for student-based evaluations, providing results, and discussing implications of the findings.
Digital technologies allow content to be freely shared and distributed at an unprecedented scale that may challenge traditional thinking about textbooks and publishing. OER are textbooks, syllabi, course packets, readings, lesson plans, images, videos, audio, and courses "that are in the public domain or introduced with an open license" (UNESCO, n.d., para. 1). These materials may be freely reused, remixed, revised, and redistributed (cf. Bissell, 2009) without additional permission or licensing fees. Such freedom makes these resources especially useful for online and distance learning scenarios because online instructors and course developers can freely share, remix, and employ direct-linking to resources rather than operating within the strict limitations of fair use (Belikov & Kimmons, in press). Research into open textbooks has yielded neutral-positive results for learning outcomes, meaning that students using open textbooks generally perform as well as or better than students using restricted textbooks, and have also yielded positive results for cost savings (Bliss, Hilton, Wiley, & Thanos, 2013; Fischer, Hilton, Robinson, Wiley, 2015; Hilton & Laman, 2012; Mtebe & Raisamo, 2014; Prasad & Usagawa, 2014; Robinson, Fischer, Wiley, & Hilton, 2014; Walker, 2007), but there is need to expand the literature in the area of textbook quality and to enact a paradigmatic shift in how we think about educational resource quality assurance. This is especially pertinent for professionals in ODL as a shift to open textbooks will need to be supported by studies on resource quality to help bolster the perceived overall quality of online learning experiences.
Open textbook stakeholders recognize quality control to be an essential ongoing priority (Atenas, Havemann, & Priego, 2014). Limited research into open textbook quality in K-12 has found that open textbooks may be of higher quality than their copyright-restricted (and easily outdated) counterparts and that because open textbooks can be adapted, they can continuously improve in quality as educators edit and adapt them to their needs (Kimmons, 2015). In general, open textbook "quality control is in the hands of learners and teachers... instead of instructional [or content] experts" (Dinevski, 2008, p. 121), because open textbooks can be adopted at any time in a semester, can be utilized in a partial or scattershot manner, and can be adapted by instructors for ongoing improvement, thereby circumventing traditional vetting and approval mechanisms. Thus, new methods are needed for evaluating the quality of open textbooks and including student voice in this process. Research in this area is fledgling but should include more sophisticated means of evaluation than those offered by traditional peer review by content experts (Clements & Pawlowski, 2011; Kimmons, 2015; Kimmons, 2016). Meaningful evaluation of textbooks requires a deep understanding of contextualized educator and student needs, not just content knowledge (Carey & Hanley, 2007). Yet, textbook evaluations are generally instructor- or department-driven and altogether ignore student voice (cf. Mahmood, 2011; Stein, Stuen, Carnine, & Long, 2001). This means that textbook evaluations are currently monophonically shaped by the prejudices, attitudes, and beliefs of instructors.
From a hermeneutic perspective, such prejudices are expected and problematic when considered alone (Kinsella, 2006). Polyphony, or a range of voices, is altogether missing (Bakhtin, 1981), and since textbooks are selected for students, it seems reasonable that students should have voice in this decision. Widdershoven (2001) explains that "only if the evaluator tries to understand the values endorsed in a practice and is prepared to listen to the stories of all parties involved, can evaluation become a practice to which all participants contribute" (p. 253). In the current context, students can only contribute to meaningful selection of textbooks if they play a major role in the evaluation process, rather than being viewed as lacking "information or the capacity to take part in the process of deliberation" (Widdershoven, 2001, p. 260).
In this study, we seek to answer the following research questions:
This study applies a hermeneutic, student voice-driven lens to the evaluation of competing course textbooks and took place in a graduate-level project management course at a large, private, high research activity university in the U.S. Students in the course were full-time graduate students in the Instructional Psychology and Technology department at the institution, either at the Masters or Doctoral level, and enrolled students generally prepare for careers in instructional design, educational technology, or educational research. The first two authors of this study were enrolled students in the course, and the third author was the instructor. The instructor served as a facilitator to encourage students to thoughtfully shift between the roles of learner and evaluator and to mentor students through the evaluation process. Prior to this study, the student authors did not have training in course content or textbook evaluation, and the course was structured so that students would learn the subject matter while simultaneously conducting the research project. It was anticipated that this approach would: a) give students an authentic experience in project management, b) expose students to polyphonic perspectives on project management, c) mentor students in research and evaluation practices, d) empower students to exhibit voice and agency in their own learning and in the course structure, and e) provide valuable results on student voice, OER, and textbook evaluation.
The instructor began the class by introducing the course project and providing students with access to eight, de-identified electronic textbooks that were either commonly used in courses or by professionals in the field. The instructor selected textbooks by: a) consulting colleagues and syllabi from various peer institutions, and b) consulting Amazon sales ranks of books in the topic area. Of these, six textbooks were under copyright restrictions while two textbooks were openly released under a Creative Commons license. Restricted textbooks were made available to the students by licensing agreements through the institutional library.
Structure of the course beyond the first class period was directed by the student-directed project team as participants determined how to best approach the task of textbook evaluation and took turns serving as the project manager for each week. The team was given eight weeks to complete the project, and students elected to initially peruse all texts and to rank them according to their initial perceptions while memoing the process. Students then revisited the texts and articulated the pros and cons associated with their two "most liked" and two "least liked" textbooks. This information was then used to determine what criteria should be included in a more formal evaluation process, drawing upon their own initial reactions and using the existing literature for thematic comparison and common terminology. Recognizing that it would be unfeasible to deeply read all eight textbooks in a single course, students decided to conduct evaluations at two levels: broad and deep.
Broad evaluation consisted in the establishment of readily identifiable items that were valuable to the students and were applied to all eight textbooks at the book level. These broad evaluations included 24 questions representing four student-determined categories: Content, Design, Pedagogy, and Cost. Content included trustworthiness of the author and alignment with standards (i.e., PMBOK or project management body of knowledge); design included logical structure, visual layout, and aesthetics (e.g., font size, graphics); pedagogy included audience-appropriate word usage and inclusion of scaffolding and learning tools (e.g., glossaries, multimedia, summaries, and objectives); and cost included the general cost of the text. The included items were intentionally designed to consider only the criteria that students felt competent to evaluate, thereby excluding some possible but important criteria (e.g., content accuracy). Most items were evaluated on a 5-point Likert scale (ranging from Strongly Disagree to Strongly Agree with a neutral midpoint) by each student and then normalized to a 0.0 - 1.0 scale. Categories included both information- and value-based items, raw scores for value-based items were determined by averaging ratings together, and overall book scores were determined by weighting raw scores for each item according to each item's importance, determined by student consensus. Categories were also weighted to capture subjective importance to students: Content (25%), Design (33%), Pedagogy (33%), and Cost (8%). Weightings were agreed upon by students through a process of consensus regarding what they believed to be most important when selecting a quality textbook. A complete list of evaluation items along with weightings are available in the online supplement (Kimmons, n.d.). Results of the broad evaluation allowed students to then select the two highest-rated textbooks for deeper evaluation, one of which happened to be open and the other restricted.
Deep evaluation criteria were then constructed including the categories of Content (40%), Design (20%), and Pedagogy (40%) and were applied at the chapter level. Students created a reading schedule and evaluated each chapter by completing a 5-point Likert scale evaluation for each of the nine created questions. Overall deep evaluations were calculated by averaging weighted chapter-level evaluations. Deep evaluation criteria differed from broad evaluation items, because they required nuanced reading and immersion in the text. For instance, when considering visual design elements and the presence of illustrations, a broad item was worded as follows "the graphics look high-quality and seem helpful (rather than intimidating)," while a deep item was worded as follows "illustration/graphics/multimedia are effectively used to add to the instructional value of the material." Evaluating visual design elements in this manner allowed the evaluators both to account for initial impressions of visual elements on the reader and their potential value for learning.
Many considerations were made to help ensure qualitative rigor. First, evaluation team debriefs were conducted for 1.5+ hours, semi-weekly, in which student member checking occurred (Lincoln & Guba, 1985). Since students served as both learners and evaluators in this project, they had to regularly take time to thoughtfully switch roles, reflect on the process, and consider how results reflected their actual perceptions. Second, ongoing memoing occurred during team meetings, evaluations, and readings, and these memos were used as a data source for reflection and analysis (Birks, Chapman, & Francis, 2008). And third, triangulation was employed both in the use of multiple data sources (broad evaluations, deep evaluations, and memos) as well as in the use of multiple evaluators (two students and one instructor), and results represent consensus from these parallel triangulation processes (Creswell, 2002; cf. Figure 1).
Figure 1. Parallel triangulation of data sources and researchers to support rigor.
Furthermore, evaluators conducted deep evaluations at the chapter level to allow for the generation of highly-focused data points, thereby allowing for greater analytical rigor. When doing these deep evaluations, it was not possible for students to read and evaluate every chapter in the studied textbooks within the available timeframe, but saturation in these evaluations was reached once evaluators felt confident that new chapters would not influence overall perspectives (Creswell, 2002). In addition, subsequent, deep criteria chapter evaluations were regularly compared to the emerging average evaluation for each text and provided a guide for identifying reduced variations over time. For example, in its penultimate deep evaluation, the Amado et al. (2015) chapter differed only.02 points from its mean, and the last deep evaluation differed less than.01 point from the mean. We also took this as a sign of saturation.
We proceed by providing results for the broad, deep, and summative evaluations. Numeric results are provided in decimal form or as percentage points (i.e.,
Results of the broad evaluation indicated a wide range of overall scores: 35-72%,
Table 1
Broad Evaluation Results for All Eight Textbooks
| Rank | Type | Content | Design | Pedagogy | Cost | Overall | |
| Amado et al. (2015) | 1 | Open | .79 | .35 | .99 | 1.00 | .72 |
| Kuster et al. (2015) | 2 | Restricted | 1.00 | .70 | .51 | .00 | .65 |
| Olson (2003) | 3 | Restricted | .89 | .43 | .77 | .26 | .64 |
| Shaw (2015) | 4 | Restricted | .36 | .66 | .54 | .26 | .51 |
| Watt (2014) | 5 | Open | .38 | .54 | .34 | 1.00 | .47 |
| Crawford (2014) | 6 | Restricted | .89 | .44 | .09 | .00 | .40 |
| Taylor (2014) | 7 | Restricted | .29 | .48 | .32 | .26 | .36 |
| Kerzner (2015) | 8 | Restricted | .66 | .42 | .06 | .26 | .35 |
| Mean | .66 | .50 | .45 | .38 | .51 | ||
Results of the deep evaluation indicated a difference between the top open and top restricted textbook, favoring the open textbook by 5%. The open textbook performed better on content (15%) and pedagogy (12%), while the restricted textbook performed better on design (13%; cf. Table 2). These deep evaluation results corroborated results of the broad evaluation. Parallel reading schedules were developed between the texts so that student evaluators could read the same content as it was addressed in the separate texts. Differences in numbers of pages and chapters evaluated were a result of the writing style, layout, and authoring choices of each text, because authors chunked content differently or gave some topics more attention and because editors used different font sizes, graphical elements, and so forth that influenced how much text might appear on a page.
Table 2
Deep Evaluation Results for the Top Two Textbooks
| Type | Chapters evaluated | Pages | Words | Content | Design | Pedagogy | Overall | |
| Amado et al. (2015) | Open | 5 | 180 | 41K | .91 | .25 | .80 | .78 |
| Kuster et al. (2015) | Restricted | 13 | 85 | 20K | .76 | .38 | .68 | .73 |
Following triangulation design analysis (Creswell, 2002), evaluators reread all memos and found that most thoughts expressed therein were adequately captured in the broad and deep results. However, a few differences and nuances were noted that deserved additional attention. Overall, student evaluators felt that the two textbooks selected for deep evaluation were superior to the rest but that there were some nuanced strengths and weaknesses of each that should be more fully reported.
The open textbook by Amado et al. (2015), stood apart from the rest by providing many useful details and anecdotal illustrations, reducing the need for the instructor to clarify difficult concepts. However, some of its design decisions were distracting from the text. Anecdotal illustrations and case studies, for instance, were often poorly situated or seemed abrupt or out of place, and graphics typically used unhelpful stock photos rather than illustrative diagrams.
The restricted textbook by Kuster et al. (2015), exhibited a writing style that was tight and conceptually concise with useful checklists that served as quick reference guides for practitioners. However, the theories covered were not grounded in clear examples and illustrations from the field, which meant that students relied heavily on the instructor to explain difficult principles and to situate concepts in real-life scenarios. Illustrations also exhibited varying degrees of usefulness, as some expertly clarified concepts while others were overly complex or distracting.
Students evaluated textbook quality based upon the applicability of the content, the instructional value of the design, and the overall effectiveness of the pedagogical scaffolding in each chapter. Student perceptions of quality increased as more of these elements were present in a textbook, but students also recognized that they could not meaningfully evaluate all aspects of quality, such as content accuracy. Thus, though student voice revealed some important considerations for textbook quality, it was recognized that polyphony with an expert is necessary to ensure a fully-informed evaluation.
Interestingly, the two best textbooks identified as highest quality by students included an open textbook and the most expensive restricted textbook. Since cost was not a barrier in this study, cost-savings was not a factor in deep or summative analyses. While affordability is often the chief virtue lauded by advocates of open textbooks, this finding compellingly shows that when cost is neutralized in a comparative evaluation, open textbooks can compete with restricted textbook quality. However, each open textbook did not perform equally well, suggesting that future research should explore more generalizable patterns of comparative quality in these resources. One of the instructor's goals for this project, which he did not disclose to students until after analysis was complete, was to determine comparative open textbook quality when cost was not a factor. He will use results to guide textbook selection in future course sections.
Another finding is that of the eight textbooks analyzed in the broad evaluation, the top two selected were the only multi-authored options. This inadvertent selection may suggest the value of polyphony as an authoring decision as well as the value of openly licensed resources that allow for on-going revision, remixing, and collaboration. Future research should explore the effects of collaborative authorship on textbook quality at a larger scale.
As the evaluators reflected on their processes, they concluded that the best textbook is not based solely on the course to be taught, but ultimately upon the learning context and the relative presence of the instructor in the learning process. A textbook like Kuster et al. (2015), for example, which contains very few anecdotal or real-life illustrations, may actually be a better textbook for an instructor who intends to provide this illustrative scaffolding in class through personal anecdotes or real-life illustrations. If, however, instructor presence is weaker (such as in an online course), a textbook like Amado et al. (2015), with its extensive anecdotal illustrations to scaffold conceptual learning, may be of superior quality. In this regard, open textbooks could also provide value by allowing instructors to modify and personalize materials to fit their intended level of instructor presence. This is especially important in online and distributed learning scenarios where instructors may be more limited in their access to students or they might have to rely upon increased mediation via online tools. If online instructors cannot, for instance, provide on-the-fly, real-world examples to students as they would in traditional classroom discussions, then a textbook that provides many such examples (i.e., more scaffolding) might conceivably be more valuable for the student learning at a distance than for the face-to-face student.
Others may find this study's approach useful for: a) choosing a textbook, b) understanding student textbook perceptions, and c) guiding approaches to engaging students in analysis and evaluation. On this third point, it should be said that while the students believed the traditional approach of reading an instructor-assigned textbook to be a valuable way to learn the concepts of a field, they felt that engaging in a project to critically analyze multiple textbooks by their own standards fostered a much greater depth of learning, as it helped them simultaneously learn about content, research, writing, and learning. Additionally, as students would alternate managing the project, they were able to immediately transfer what they were learning from the textbooks to what they were doing on the project. The student authors believe this process allowed them to deepen their learning as they reflected together on a semi-weekly basis, both on their readings and their application of project management principles. Thus students felt that a deep critical analysis of multiple textbooks combined with the application of project management principles and regular group-reflection, created a rich and fertile context for deep learning to occur. Conversely, the instructor author believes that having students involved in this way provided insights on the strengths and limitations of different textbooks and also shed light on student expectations for learning within the context of the course and subject area. This allowed for a deep degree of reflective practice on the part of the instructor to consider what resources and teaching strategies are most suitable for achieving course learning outcomes with similar students in the future.
This study has a number of limitations and delimitations that should be addressed. First, as novices in the content area, students could not be expected to evaluate the accuracy of textbook content. This is a limitation of student voice itself and was minimized by focusing on items that students could reasonably be expected to evaluate. Second, this study was limited to an evaluation team of two graduate students who had to strategically approach project scope in a manner that was suitable for a formal classroom setting. This made evaluation results authentic but potentially incomplete or non-representative of students generally. Readers should counterbalance potential student biases with reported methodological rigor to determine result transferability.
Leveraging the potentials of digital technologies to explore the transformational value of open textbooks upon the traditional copyright-restricted paradigm, this study has sought to establish the possibility of meaningfully including student voice in the cloistered processes of textbook evaluation and adoption. Results indicate that textbook quality can encompass a variety of dimensions for students and that not all textbooks perform equally in addressing matters of content, pedagogy, and design. Comparative findings on open textbooks reveal that these alternatives can compete with their restricted counterparts, even when cost advantages are ignored. From this study, we hope that instructors and institutions will explore processes for including student voice in textbook evaluation by exploring open alternatives to traditional textbooks, identifying student textbook quality considerations, and polyphonically empowering students to participate in textbook adoption and adaptation.
Amado, M., Ashton, K., Ashton, S., Bostwick, J., Clements, G., Drysdale, J.,... Anonymous. (2015). Project management for instructional designers. Retrieved from http://pm4id.org/
Atenas, J., Havemann, L., & Priego, E. (2014). Opening teaching landscapes: The importance of quality assurance in the delivery of open educational resources. Open Praxis, 6(1), 29–43. doi: 10.5944/openpraxis.6.1.81
Bakhtin, M. (1981). The dialogic imagination (Michael Holquist, Ed.; Caryl Emerson & Michael Holquist, Trans.). Austin, TX: University of Texas Press. Retrieved from http://www.public.iastate.edu/~carlos/607/readings/bakhtin.pdf
Belikov, O., & Kimmons, R. (in press). What can I use? Copyright, fair use, and open licensing. In A. A. Piña, V. Lowell, & B. Harris (Eds.), Leading and managing e-learning: What the e-learning leader needs to know. Springer.
Birks, M., Chapman, Y., & Francis, K. (2008). Memoing in qualitative research probing data and processes. Journal of Research in Nursing, 13(1), 68-75. doi: 10.1177/1744987107081254
Bissell, A. (2009). Permission granted: Open licensing for educational resources. Open Learning, 24, 97-106. doi: 10.1080/02680510802627886
Bliss, T. J., Hilton, J., III, Wiley, D., & Thanos, K. (2013). The cost and quality of online open textbooks: Perceptions of community college faculty and students. First Monday, 18, 1–7. doi: 10.5210/fm.v18i1.3972
Carey, T., & Hanley, G. L. (2007). Extending the impact of open educational resources through alignment with pedagogical content knowledge and institutional strategy: Lessons learned from the MERLOT community experience. In T. Iiyoshi, & M. S. V. Kumar (Eds.) Opening up education: The collective advancement of education through open technology, open content, and open knowledge (pp. 181-196). Retrieved from https://mitpress.mit.edu/sites/default/files/9780262515016_Open_Access_Edition.pdf
Clements, K. I., & Pawlowski, J. M. (2011). User-oriented quality for OER: Understanding teachers' views on re-use, quality, and trust. Journal of Computer Assisted Learning, 28(1), 4-14. doi: 10.1111/j.1365-2729.2011.00450.x
The College Board. (2015). Average estimated undergraduate budgets, 2014-15. Trends in Higher Education. Retrieved from https://trends.collegeboard.org/college-pricing/figures-tables/average-estimated-undergraduate-budgets-2016-17
Cook-Sather, A. (2006). Sound, presence, and power: Exploring 'student voice' in educational research and reform. Curriculum Inquiry, 36(4), 359-390. doi: j.1467-873x.2006.00363.x
Cook-Sather, A. (2014). The trajectory of student voice in educational research. New Zealand Journal of Educational Studies, 49(2), 131-148. Retrieved from http://search.informit.com.au/documentSummary;dn=842480978608459;res=IELNZC
Crawford, J. K. (2014). Project management maturity model (3rd ed.). Boca Raton, FL: CRC Press.
Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (3rd ed.). Upper Saddle River, NJ: Pearson.
Denoyelles, A., Raible, J., & Seilhamer, R. (2015). Exploring students' e-textbook practices in higher education. Educause Review, 7. Retrieved from http://er.educause.edu/articles/2015/7/exploring-students-etextbook-practices-in-higher-education
Dinevski, D. (2008). Open educational resources and lifelong learning. In ITI 2008 - 30th International Conference on Information Technology Interfaces (pp. 117–122). IEEE. doi: 10.1109/iti.2008.4588393
Fine, M., Torre, M. E., Bums, A., & Payne, Y. (2007). Youth research/participatory methods for reform. In D. Thiessen, & A. Cook-Sather (Eds.), International handbook of student experience in elementary and secondary school (pp. 805-828). Dordrecht, The Netherlands: Springer.
Fischer, L., Hilton, J., III, Robinson, T. J., & Wiley, D. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159-172. doi: 10.1007/s12528-015-9101-x
Hilton, J., III, & Laman, C. (2012). One college's use of an open psychology textbook. Open Learning: The Journal of Open, Distance and e-Learning, 27(3), 265-272. doi: 10.1080/02680513.2012.716657
Kerzner, H. R. (2015). Project management 2.0. Hoboken, NJ: John Wiley & Sons.
Kimmons, R. (2015). OER quality and adaptation in K-12: Comparing teacher evaluations of copyright-restricted, open, and open/adapted textbooks. The International Review of Research in Open and Distributed Learning, 16(5).
Kimmons, R. (2016). Expansive openness in teacher practice. Teachers College Record, 118(9).
Kimmons, R. (n.d.). Online supplement for "Student voice in textbook evaluation: Comparing open and restricted textbooks." Retrieved from https://drive.google.com/open?id=0B4MRZPM74y16dTM1bm5Hd3RncFk
Kinsella, E. A. (2006). Hermeneutics and critical hermeneutics: Exploring possibilities within the art of interpretation. Forum: Qualitative Social Research, 7(3). Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/145/319&sa=U&ei=ssViU7yEHKet0QWql4CwBw&ved=0CCEQFjAB&usg=AFQjCNHJ_1emJ7oJ68R7uMqAZeo4UqOBIg
Kuster, J., Huber, E., Lippmann, R., Schmid, A., Schneider, E., Witschi, U., & Wüst, R. (2015). Project management handbook (N. Rosenthal, & J. Townsley, Trans.). New York, NY: Springer.
Lincoln, Y. S., & Guba, E. A. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Mahmood, K. (2011). Conformity to quality characteristics of textbooks: The illusion of textbook evaluation in Pakistan. Journal of Research and Reflections in Education, 5(2), 170-190. Retrieved from http://ue.edu.pk/jrre/articles/52006.pdf
Mtebe, J. S., & Raisamo, R. (2014). Challenges and instructors' intention to adopt and use open educational resources in higher education in Tanzania. The International Review of Research in Open and Distributed Learning, 15(1). doi: 10.19173/irrodl.v15i1.1687
Olson, D. L. (2003). Introduction to information systems project management (2nd ed.). Boston, MA: McGraw-Hill.
Paulsen, M. B., & St. John, E. P. (2002). Social class and college costs: Examining the financial nexus between college choice and persistence. The Journal of Higher Education, 73(2), 189-236. doi: 10.1353/jhe.2002.0023
Prasad, D., & Usagawa, T. (2014). Towards development of OER derived custom-built open textbooks: A baseline survey of university teachers at the University of the South Pacific. The International Review of Research in Open and Distributed Learning, 15(4). doi: 10.19173/irrodl.v15i4.1873
Robinson, T. J., Fischer, L., Wiley, D., & Hilton, J., III. (2014). The impact of open textbooks on secondary science learning outcomes. Educational Researcher, 43(7), 341-351. doi: 10.3102/0013189x14550275
Senack, E. (2014). Fixing the broken textbook market: How students respond to high textbook costs and demand alternatives. U.S. PIRG Education Fund & the Student PIRGs. Retrieved from http://www.uspirg.org/sites/pirg/files/reports/NATIONAL%20Fixing%20Broken%20Textbooks%20Report1.pdf
Shaw, G. (2015). Understanding project management: Skills and insights for successful project delivery. London, UK: Kogan Page.
Smyth, J., & McInerney, P. (2012). From silent witnesses to active agents: Student voice in re-engaging with learning. New York, NY: Peter Lang Publishing.
Stein, M., Stuen, C., Carnine, D., & Long, R. M. (2001). Textbook evaluation and adoption. Reading & Writing Quarterly, 17(1), 5-23. doi: 10.1080/105735601455710
Taylor, P. (2014). Real project management: The skills and capabilities you will need for successful project delivery. London, UK: Kogan Page.
UNESCO. (n.d.). What are open educational resources (OER)? UNESCO. Retrieved from http://www.unesco.org/new/en/communication-and-information/access-to-knowledge/open-educational-resources/what-are-open-educational-resources-oers/
University of California – Riverside, Undergraduate Education & Student Affairs. (2011). Survey brief: The impact of rising tuition on students: What can we do to help students at UC Riverside? UCR UESA. Retrieved from http://the-digital-reader.com/wp-content/uploads/2011/12/UCR-Student-Survey-Brief-Fall-20111.pdf
Walker, E. (2007). Evaluating the results of open education. In T. Iiyoshi, & M. S. V. Kumar (Eds.), Opening up education: The collective advancement of education through open technology, open content, and open knowledge, 77-88. Retrieved from https://mitpress.mit.edu/sites/default/files/9780262515016_Open_Access_Edition.pdf
Watt, A. (2014). Project management. BCCampus. Retrieved from http://opentextbc.ca/projectmanagement/
Widdershoven, G. A. M. (2001). Dialogue in evaluation: A hermeneutic perspective. Evaluation, 7(2), 253-263. doi: 10.1177/13563890122209676
Student Voice in Textbook Evaluation: Comparing Open and Restricted Textbooks by Scott Woodward, PhD, Adam Lloyd, and Royce Kimmons, PhD, is licensed under a Creative Commons Attribution 4.0 International License.