The Design and Psychometric Properties of a Peer Observation Tool for Use in LMS-Based Classrooms in Medical Sciences
DOI:
https://doi.org/10.19173/irrodl.v24i1.6689Keywords:
blended learning, virtual education, psychometrics, validity, reliabilityAbstract
In peer observation of teaching, an experienced colleague in the educational environment of a faculty member observes the educational performance of that faculty member and provides appropriate feedback. The use of peer review as an alternative source of evidence of teaching effectiveness is increasing. However, no research has been done in the field of tool design and development to peer review in classrooms that use a learning management system (LMS). This study used mixed methods. In the qualitative stage, after studying sources and interviewing professors active in virtual education, a question bank was prepared and a 26-item initial questionnaire created. In the quantitative stage, the psychometric properties of the developed instruments, such as the face, content, and structural validity, were examined, and reliability tests were performed. IBM SPSS Statistics (Version 20) was used for analysis. Five categories, including content preparation, content presentation, effective interactions, motivation management, and support services, and 26 subcategories were determined to be effective indicators in peer observation in LMS-based classes in medical sciences. During content analysis, 9 items were removed due to lack of necessary criteria. Then, using principal component analysis and varimax rotation in the present mode )Watkins, 2018), 5 components with eigenvalues higher than 1 were extracted, which explained a total of 70.55% of the total variance. The inter-cluster correlation coefficient (ICC) was 0.88. Thus, the peer observation measurement tool, designed with 17 expressions using the answer method “yes/no”, showed good validity and reliability. The research results demonstrate that the evaluation of virtual classes of professors by their peers is effective and that the results can be used in e-learning promotion plans.
References
Akkoyunlu, B., & Soylu, M. Y. (2006). A study on students’ views on blended learning environment. Turkish Online Journal of Distance Education, 7(3), 43–56. https://dergipark.org.tr/en/pub/tojde/issue/16925/176657
Al‐Qahtani, A. A. Y., & Higgins, S. E. (2013). Effects of traditional, blended, and e‐learning on students’ achievement in higher education. Journal of Computer Assisted Learning, 29(3), 220–234. http://dx.doi.org/10.1111/j.1365-2729.2012.00490.x
Amira, R., & Jelas, Z. M. (2010). Teaching and learning styles in higher education institutions: Do they match? Procedia-Social and Behavioral Sciences, 7, 680–684. https://doi.org/10.1016/j.sbspro.2010.10.092
Andone, I., & Sireteanu, N.- A. (2009). Strategies for technology-based learning in higher education. The FedUni Journal of Higher Education, 4(1), 31–42. https://ssrn.com/abstract=1337962
Bates, A. W. (2022). Teaching in a digital age: Guidelines for designing teaching and learning (3rd ed.). Tony Bates Associates Ltd. https://opentextbc.ca/teachinginadigitalage/
Benner, P. (2012). Educating nurses: A call for radical transformation— How far have we come? Journal of Nursing Education, 51(4), 183–184. https://doi.org/10.3928/01484834-20120402-01
Bernstein, D., Burnett, A. N., Goodburn, A. M., & Savory, P. (2006). Making teaching and learning visible: Course portfolios and the peer review of teaching (1st ed.). Jossey-Bass.
Butt, B. Z., & Ur Rehman, K. (2010). A study examining the students’ satisfaction in higher education. Procedia-Social and Behavioral Sciences, 2(2), 5446–5450. https://doi.org/10.1016/j.sbspro.2010.03.888
Clipa, O. (2011). The profile of the academic assessor. Procedia-Social and Behavioral Sciences, 12, 200–204. https://doi.org/10.1016/j.sbspro.2011.02.027
Collis, B., & Moonen, J. (2012). Flexible learning in a digital world: Experiences and expectations (2nd ed.). Routledge.
Cunningham, I., Johnson, I., & Lynch, C. (2017). Implementing peer review of teaching: A guide for dental educators. British Dental Journal, 222(7), 535–540. https://doi.org/10.1038/sj.bdj.2017.316
Dill, D. (2007). Quality assurance in higher education: Practices and issues. In P. P. Peterson, E. Baker, & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., pp. 377–383). Elsevier.
Donnelly, R. (2017). Blended problem-based learning in higher education: The intersection of social learning and technology. Psychosociological Issues in Human Resource Management, 5(2), 25–50. https://doi.org/10.22381/PIHRM5220172
Drisko, J. W., & Maschi, T. (2016). Content analysis: Pocket guides to social work research methods. Oxford University Press.
Fernandez, C. E., & Yu, J. (2007). Peer review of teaching. The Journal of Chiropractic Education, 21(2), 154–161. https://doi.org/10.7899/1042-5055-21.2.154
Fletcher, J. A. (2018). Peer observation of teaching: A practical tool in higher education. The Journal of Faculty Development, 32(1), 51–64. http://doi.10.13140/RG.2.2.19455.82084
Garrison, D. R. (2016). E-learning in the 21st century: A community of inquiry framework for research and practice (3rd ed.). Routledge.
Gehringer, E. F., Chinn, D. D., Pérez-Quiñones, M. A., & Ardis, M. A. (2005). Using peer review in teaching computing. ACM SIGCSE Bulletin, 37(1), 321–322. https://dl.acm.org/doi/10.1145/1047124.1047455
Henrie, C. R., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring intensive longitudinal measures of student engagement in blended learning. The International Review of Research in Open and Distributed Learning, 16(3), 131–155. https://doi.org/10.19173/irrodl.v16i3.2015
Ibrahimzadeh, I., Zandi, B., Alipour, A., Zare, H., & Yazdani, F. (2010). The kinds of e-learning and different forms of interaction on it. Interdisciplinary Journal of Virtual Learning in Medical Sciences, 1(1), 11–22. https://ijvlms.sums.ac.ir/article_46026.html
Johnston, A. L., Baik, C., & Chester, A. (2020). Peer review of teaching in Australian higher education: A systematic review. Higher Education Research & Development, 41(2), 390–404. https://doi.org/10.1080/07294360.2020.1845124
Kay, R. H., & Knaack, L. (2008). An examination of the impact of learning objects in secondary school. Journal of Computer Assisted Learning, 24(6), 447–461. https://doi.org/10.1111/j.1365-2729.2008.00278.x
Keig, L. (2000). Formative peer review of teaching: Attitudes of faculty at liberal arts colleges toward colleague assessment. Journal of Personnel Evaluation in Education, 14(1), 67–87. https://link.springer.com/article/10.1023/A:1008194230542#citeas
Keshavarz, M., & Ghoneim, A. (2021). Preparing educators to teach in a digital age. The International Review of Research in Open and Distributed Learning, 22(1), 221–242. https://doi.org/10.19173/irrodl.v22i1.4910
Keshavarz, M., Mirmoghtadaie, Z., and Nayyeri,S. (2022). Design and validation of the virtual classroom management questionnaire. The International Review of Research in Open and Distributed Learning, 23(2), 121–135. https://doi.org/10.19173/irrodl.v23i2.5774
Kintu, M. J., Zhu, C., & Kagambe, E. (2017). Blended learning effectiveness: The relationship between student characteristics, design features, and outcomes. International Journal of Educational Technology in Higher Education, 14(1), 1–20. https://doi. 10.1186/s41239-017-0043-4
Lacasse, Y., Godbout, C., & Sériès, F. (2002). Health-related quality of life in obstructive sleep apnoea. European Respiratory Journal, 19(3), 499–503. https://doi.org/10.1183/09031936.02.00216902
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975. Tb01393.x
Lee, J.- K. (2009). The effects of self-regulated learning strategies and system satisfaction regarding learner’s performance in e-learning environment. Journal of Instructional Pedagogies, 1, 30–45. https://aabri.com/manuscripts/08053.pdf
Min, H.-T. (2006). The effects of trained peer review on EFL students’ revision types and writing quality. Journal of Second Language Writing, 15(2), 118–141. https://doi.org/10.1016/j.jslw.2006.01.003
Missildine, K., Fountain, R., Summers, L., & Gosselin, K. (2013). Flipping the classroom to improve student performance and satisfaction. Journal of Nursing Education, 52(10), 597–599. https://doi.org/10.3928/01484834-20130919-03
Mohammadbeigi, A., Mohammadsalehi, N., & Aligol, M. (2015). Validity and reliability of the instruments and types of measurements in health applied research. Journal of Rafsanjan University of Medical Sciences, 13(12), 1153–1170. http://journal.rums.ac.ir/article-1-2274-en.html
Neuendorf, K. (2017). The content analysis guidebook. SAGE. https://dx.doi.org/10.4135/9781071802878
OECD (2021) The state of school education: One year into the Covid pandemic Paris, France: OECD
Opre, D., Calbaza-Ormenisan, M., & Opre, A. (2011). University teaching: Didactic expertise reflected by metacognitive abilities and emotional control. Procedia-Social and Behavioral Sciences, 29, 670–677. https://doi.org/10.1016/j.sbspro.2011.11.291
Patton, M. Q. (1983). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches [Review of the book Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches by E. G. Guba & Y. S. Lincoln]. The Journal of Higher Education, 54(3), 339–342. https://doi.org/10.1080/00221546.1983.11778201
Pinto-Llorente, A. M., Sánchez-Gómez, M. C., García-Peñalvo, F. J., & Casillas-Martín, S. (2017). Students’ perceptions and attitudes towards asynchronous technological tools in blended-learning training to improve grammatical competence in English as a second language. Computers in Human Behavior, 72, 632–643. https://doi.org/10.1016/j.chb.2016.05.071
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.10.1002/nur.20199
Ruiz, J. G., Candler, C., & Teasdale, T. A. (2007). Peer reviewing e-learning: Opportunities, challenges, and solutions. Academic Medicine, 82(5), 503–507. https://doi.10.1097/acm.0b013e31803ead94
Snyder, M. M. (2009). Instructional-design theory to guide the creation of online learning communities for adults. TechTrends, 53(1), 45–57. https://doi.org/10.1007/s11528-009-0237-2
Speer, S. (2010). Peer evaluation and its blurred boundaries: Results from a meta-evaluation in initial vocational education and training. Evaluation, 16(4), 413–430. https://doi.org/10.1177/1356389010382265
Stevens, J. P. (2012). Applied multivariate statistics for the social sciences (5th ed.). Routledge.
Twomey, A. (2004). Web-based teaching in nursing: Lessons from the literature. Nurse Education Today, 24(6), 452–458. https://doi.org/10.1016/j.nedt.2004.04.010
Waltz C. F., & Bausell R. B. (1981). Nursing Research: Design statistics and computer analysis. F. A. Davis.
Watkins, M. W. (2018). Exploratory Factor Analysis: A Guide to Best Practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807
Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: two simple measures. Journal of Personality and Social Psychology, 84(3), 608–618. https://doi.org/10.1037/0022-3514.84.3.608
Wood, D. F. (2003). ABC of learning and teaching in medicine. Problem-based learning. BMJ, 326, 328–330. https://doi.org/10.1136/bmj.326.7384.328
Yong, A. G., & Pearce, P. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. The Quantitative Methods for Psychology, 9(2), 79–94. https://doi.org/10.20982/tqmp.09.2.p079
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International Licence. The copyright of all content published in IRRODL is retained by the authors.
This copyright agreement and use license ensures, among other things, that an article will be as widely distributed as possible and that the article can be included in any scientific and/or scholarly archive.
You are free to
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms below:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
