Using the Critical Incident Questionnaire as a Formative Evaluation Tool to Inform Online Course Design: A Qualitative Study
DOI:
https://doi.org/10.19173/irrodl.v23i2.5959Keywords:
student evaluation of teaching, Critical Incident Questionnaire, online course design, formative assessmentAbstract
The online instructor plays a prominent role in influencing how students respond to an online course, from designing the course structure, course activities, and assignments to encouraging interaction. Therefore, to develop effective online courses, instructors need robust feedback on their design strategies. Student evaluation of teaching (SET) functions as a summative evaluation of the course design and delivery. Yet, the feedback from SETs can only be integrated into the next iteration of the course, thereby failing to benefit the students who provide the feedback. One suggestion is to use midsemester formative evaluation to inform course design in real time. A qualitative research study was conducted to explore whether the Critical Incident Questionnaire (CIQ) could be an effective formative evaluative tool to inform real-time online course design and delivery. Thematic analysis was conducted on the midcourse evaluations obtained from 70 students in six fully online master’s level courses. There are three key findings from this study. First, CIQ use can provide opportunities for real-time adjustments to online course design and inform future redesign of online courses. Second, responses received via the CIQ prioritize the student voice and experience by focusing on factors that are critical to them. Finally, this deep-dive analysis reinforces the enduring factors that contribute to effective online course design and delivery. A recommendation for practice is to use the CIQ as an effective tool to gather formative feedback from students. This feedback can then be used to adjust course design as needed.
References
Anderson, T. (2011). The theory and practice of online learning (5th ed.). Athabasca University Press. https://auspace.athabascau.ca/bitstream/handle/2149/411/?sequence=1
Baldwin, S. J., & Trespalacios, J. (2017). Evaluation instruments and good practices in online education. Online Learning, 21(2). https://olj.onlinelearningconsortium.org/index.php/olj/article/view/913
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. https://doi.org/10.3102/0034654309333844
Blackboard. (2017, April 10). Blackboard exemplary course program rubric. https://www.blackboard.com/resources/are-your-courses-exemplary
Boring, A., Ottoboni, K., & Stark, P. (2016, January 7). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research, 1–11. https://www.scienceopen.com/document/read?vid=818d8ec0-5908-47d8-86b4-5dc38f04b23e
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brookfield, S. (1998). Critically reflective practice. Journal of Continuing Education in the Health Professions, 18(4), 197–205. https://doi.org/10.1002/chp.1340180402
California State University. (2019). CSU QLT course review instrument. https://docs.google.com/document/d/1ilqtDHjYfuJfjq1f8lG_bGhH9Xskcad-2a8aaPmnHG8/edit
Cohen, P. A. (1980). Effectiveness of student-rating feedback for improving college instruction: A meta-analysis of findings. Research in higher education, 13(4), 321-341.
Crews, T. B., Wilkinson, K., & Neill, J. K. (2015). Principles for good practice in undergraduate education: Effective online course design to assist students’ success. Journal of Online Learning and Teaching, 11(1), 87–103. https://uscdmc.sc.edu/about/offices_and_divisions/cte/instructional_design/docs/principles_good_practice_undergraduate_education_crews.pdf
Dancer, D., & Kamvounias, P. (2005). Student involvement in assessment: A project designed to assess class participation fairly and reliably. Assessment and Evaluation in Higher Education 30(4), 445–454. https://doi.org/10.1080/02602930500099235
Erikson, M., Erikson, M. G., & Punzi, E. (2016). Student responses to a reflexive course evaluation. Reflective Practice, 17(6), 663–675. https://doi.org/10.1080/14623943.2016.1206877
Finelli, C. J., Ott, M., Gottfried, A. C., Hershock, C., O’Neal, C., & Kaplan, M. (2008). Utilizing instructional consultations to enhance the teaching performance of engineering faculty. Journal of Engineering Education, 97(4), 397–411. https://doi.org/10.1002/j.2168-9830.2008.tb00989.x
Fisher, R., & Miller, D. (2008). Responding to student expectations: A partnership approach to course evaluation. Assessment & Evaluation in Higher Education, 33(2), 191–202. https://doi.org/10.1080/02602930701292514
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2, 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6
Gehringer, E. (2010, June 20–23). Daily course evaluation with Google forms [Paper presentation]. 2010 American Society for Engineering Education Annual Conference & Exposition, Louisville, KY, United States. https://doi.org/10.18260/1-2--16350
Glowacki-Dudka, M., & Barnett, N. (2007). Connecting critical reflection and group development in online adult education classrooms. International Journal of Teaching and Learning in Higher Education, 19(1), 43–52. https://eric.ed.gov/?id=EJ901286
Harasim, L. (2017). Learning theory and online technologies. Taylor & Francis. https://doi.org/10.4324/9781315716831
Hessler, H. B., & Taggart, A. R. (2011). What’s stalling learning? Using a formative assessment tool to address critical incidents in class. International Journal for the Scholarship of Teaching and Learning, 5(1), Article 9. https://doi.org/10.20429/ijsotl.2011.050109
Hurney, C., Harris, N., Bates Prins, S., & Kruck, S. E. (2014). The impact of a learner-centered, mid-semester course evaluation on students. The Journal of Faculty Development, 28(3), 55–62. https://cetl.uni.edu/sites/default/files/impact_of_learner-centered_by_hurney_1.pdf
Jacobs, M. A. (2015). By their pupils they’ll be taught: Using Critical Incident Questionnaire as feedback. Journal of Invitational Theory and Practice, 21, 9–22. https://eric.ed.gov/?id=EJ1163006
Jaggars, S. S., & Xu, D. (2016). How do online course design features influence student performance? Computers & Education, 95, 270–284. https://doi.org/10.1016/j.compedu.2016.01.014
Keefer, J. M. (2009). The Critical Incident Questionnaire (CIQ): From research to practice and back again. In R. L. Lawrence (Ed.), Proceedings of the 50th Annual Adult Education Research Conference (pp. 177–182). https://digitalcommons.nl.edu/cgi/viewcontent.cgi?referer=https://scholar.google.com/&httpsredir=1&article=1000&context=ace_aerc#page=191
Kyei-Blankson, L., Ntuli, E., & Donnelly, H. (2019). Establishing the importance of interaction and presence to student learning in online environments. Journal of Interactive Learning Research, 30(4), 539–560. https://www.learntechlib.org/p/161956/
Linstrum, K. S., Ballard, G., & Shelby, T. (2012). Formative evaluation: Using the Critical Incident Questionnaire in a graduate counseling course on cultural diversity. Journal of Intercultural Disciplines, 10, 94–102. https://www.proquest.com/docview/1033777245
Martin, F., & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learning, 22(1), 205–222. https://doi.org/10.24059/olj.v22i1.1092
Martin, F., Ritzhaupt, A., Kumar, S., & Budhrani, K. (2019). Award-winning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. The Internet and Higher Education, 42, 34–43. https://doi.org/10.1016/j.iheduc.2019.04.001
McKeachie, W. J., Lin, Y.-G., & Mann, W. (1971). Student ratings of teacher effectiveness: Validity studies. American Educational Research Journal, 8(3), 435–445. https://doi.org/10.3102%2F00028312008003435
Mitchell, K. M., & Martin, J. (2018). Gender bias in student evaluations. PS: Political Science & Politics, 51(3), 648–652. https://doi.org/10.1017/S104909651800001X
Moore, M. (1997). Theory of transactional distance. In D. Keegan(Ed.) Theoretical principles of distance education (pp.22-38) Routledge. http://www.c3l.uni-oldenburg.de/cde/found/moore93.pdf
Moore, M. G. (2013). Handbook of distance education. Routledge.
Phelan, L. (2012). Interrogating students’ perceptions of their online learning experiences with Brookfield’s Critical Incident Questionnaire. Distance Education, 33(1), 31–44. https://doi.org/10.1080/01587919.2012.667958
Picciano, A. G. (2017). Theories and frameworks for online education: Seeking an integrated model. Online Learning, 21(3). https://doi.org/10.24059/olj.v21i3.1225
Quality Matters. (2018). Specific review standards from the QM higher education rubric, sixth edition. https://www.qualitymatters.org/qa-resources/rubric-standards/higher-ed-rubric
Rao, V., & Woolcock, M. (2003). Integrating qualitative and quantitative approaches in program evaluation. In F. Bourguignon, & L. A. Pereira da Silva, (Eds.), The impact of economic policies on poverty and income distribution: Evaluation techniques and tools (pp. 165–190). World Bank and Oxford University Press.
Reid, L. D. (2010). The role of perceived race and gender in the evaluation of college teaching on RateMyProfessors.Com. Journal of Diversity in higher Education, 3(3), 137–152. https://doi.org/10.1037/a0019865
Rodin, M., & Rodin, B. (1973). Student evaluations of teachers. The Journal of Economic Education, 5(1), 5–9. https://www.jstor.org/stable/1734252
Samuel, A. (2020). Zones of agency: Understanding online faculty experiences of presence. The International Review of Research in Open and Distributed Learning, 21(4), 79–95. https://doi.org/10.19173/irrodl.v21i4.4905
Stark, P., & Freishtat, R. (2014, September 29). An evaluation of course evaluations. ScienceOpen Research, 1–7. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1
State University of New York. (2018). The SUNY online course quality review rubric: OSCQR. https://oscqr.suny.edu/
Steyn, C., Davies, C., & Sambo, A. (2019). Eliciting student feedback for course development: The application of a qualitative course evaluation tool among business research students. Assessment & Evaluation in Higher Education, 44(1), 11–24. https://doi.org/10.1080/02602938.2018.1466266
Subramanian, K., & Budhrani, K. (2020, February). Influence of course design on student engagement and motivation in an online course. In SIGCSE ’20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 303–308). Association for Computing Machinery. https://dl.acm.org/doi/abs/10.1145/3328778.3366828
Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication & Information, 2(1), 23–49. https://doi.org/10.1080/1463631022000005016
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007
Veeck, A., O’Reilly, K., MacMillan, A., & Yu, H. (2016). The use of collaborative midterm student evaluations to provide actionable results. Journal of Marketing Education, 38(3), 157–169. http://doi.org/10.1177/0273475315619652
Wolbring, T., & Treischl, E. (2016). Selection bias in students’ evaluation of teaching. Research in Higher Education, 57(1), 51-71. https://link.springer.com/content/pdf/10.1007/s11162-015-9378-7.pdf
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.
This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.
You are free to
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms below:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
