What are the indicators of Student Engagement in Learning Management Systems? A Systematized Review of the Literature

Golchehreh Ahmadi; Aeen Mohammadi; Shadi Asadzandi; Mahsood Shah; Rita Mojtahedzadeh

doi:10.19173/irrodl.v24i1.6453

Authors

Golchehreh Ahmadi Department of e-Learning in Medical Education, Center of Excellence for e-Learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran https://orcid.org/0000-0003-0623-2534
Aeen Mohammadi Department of e-Learning in Medical Education, Center of Excellence for e-Learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran https://orcid.org/0000-0002-2745-3873
Shadi Asadzandi School of Health Management and Information Science, Iran University of Medical Sciences, Tehran, Iran https://orcid.org/0000-0002-1350-9629
Mahsood Shah Swinburne University of Technology, Sydney, Australia https://orcid.org/0000-0002-0007-6879
Rita Mojtahedzadeh Department of e-Learning in Medical Education, Center of Excellence for e-Learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran https://orcid.org/0000-0003-4560-6973

DOI:

https://doi.org/10.19173/irrodl.v24i1.6453

Keywords:

e-learning, student engagement, learning management system, LMS, log data

Abstract

Student engagement has an important role in academic achievement in all learning contexts, including e-learning environments. The extent of monitoring and promoting student engagement in e-learning affects the quality of education and is a determining factor for ensuring student’s success. Log data of students’ activities recorded in a learning management system (LMS) can be used to measure their level of engagement in the online teaching–learning process. No previous studies have been found stating a consistent and systematically raised list of LMS-based student engagement indicators, so this systematized review aimed to fulfill this gap. The authors performed an advanced search in the PubMed, Ovid, Google Scholar, Scopus, Web of Science, ProQuest, Emerald, and ERIC databases to retrieve relevant original peer-reviewed articles published until the end of June 2021. Reviewing the 32 included articles resulted in 27 indicators that were categorized into three themes and six categories as follows: (a) log-in and usage (referring to LMS, access to course material), (b) student performance (assignments, assessments), and (c) communication (messaging, forum participation). Among the categories, access to course material and messaging were the most and the least mentioned, respectively.

References

Al-Azawei, A., & Al-Masoudy, M. A. A. (2020). Predicting learners’ performance in virtual learning environment (VLE) based on demographic, behavioral and engagement antecedents. International Journal of Emerging Technologies in Learning, 15(9), 60–75. https://doi.org/10.3991/ijet.v15i09.12691

Atherton, M., Shah, M., Vazquez, J., Griffiths, Z., Jackson, B., & Burgess, C. (2017). Using learning analytics to assess student engagement and academic outcomes in open access enabling programmes. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 119–136. https://doi.org/10.1080/02680513.2017.1309646

Ayouni, S., Hajjej, F., Maddeh, M., & Alotaibi, S. (2021). Innovations of materials for student engagement in online environment: An ontology. Materials Today: Proceedings. https://doi.org/10.1016/j.matpr.2021.03.636

Ballard, J., & Butler, P. I. (2016). Learner enhanced technology: Can activity analytics support understanding engagement a measurable process? Journal of Applied Research in Higher Education, 8(1), 18–43. https://doi.org/10.1108/JARHE-09-2014-0074

Boulton, C. A., Kent, C., & Williams, H. T. P. (2018). Virtual learning environment engagement and learning outcomes at a “bricks-and-mortar” university. Computers & Education, 126, 129–142. https://doi.org/10.1016/j.compedu.2018.06.031

Chaka, C., & Nkhobo, T. (2019). Online module login data as a proxy measure of student engagement: The case of myUnisa, MoyaMA, Flipgrid, and Gephi at an ODeL institution in South Africa. International Journal of Educational Technology in Higher Education, 16, Article 38. https://doi.org/10.1186/s41239-019-0167-9

Chen, S.-B. A. (2020). Examining the effect of self-regulated learning on cognitive engagement in mastery-based online courses: A learning analytics perspective (Publication no. 28216212) [Doctoral dissertation, Ohio State University]. ProQuest Dissertations Publishing. https://www.proquest.com/openview/ec2b8156c96a026b68edc9ae9b058b67/1?pq-origsite=gscholar&cbl=18750&diss=y

Coussement, K., Phan, M., Caigny, A. D., Benoit, D. F., & Raes, A. (2020). Predicting student dropout in subscription-based online learning environments: The beneficial impact of the logit leaf model. Decision Support Systems, 135, Article 113325. https://doi.org/10.1016/j.dss.2020.113325

Critical Appraisal Skills Programme. CASP systematic review checklist, https://casp-uk.net/images/checklist/documents/CASP-Systematic-Review-Checklist/CASP-Systematic-Review-Checklist_2018.pdf (accessed 25 June 2021).

Cronin, P., Ryan, F., & Coughlan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), 38–43. https://doi.org/10.12968/bjon.2008.17.1.28059

Dixson, M. D. (2015). Measuring student engagement in the online course: The online student engagement scale (OSE). Online Learning, 19(4). https://doi.org/10.24059/olj.v19i4.561

Estacio, R. R., & Raga, R. C., Jr. (2017). Analyzing students’ online learning behavior in blended courses using Moodle. Asian Association of Open Universities Journal, 12(1), 52–68. https://doi.org/10.1108/AAOUJ-01-2017-0016

Franzoni, V., Milani, A., Mengoni, P., & Piccinato, F. (2020). Artificial intelligence visual metaphors in e-learning interfaces for learning analytics. Applied Sciences, 10(20), Article 7195. https://doi.org/10.3390/app10207195

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Green, R. A. (2017). The relationship between student engagement with online content and achievement in a blended learning anatomy course. Anatomical Sciences Education, 11(5), 471–477. https://doi.org/10.1002/ase.1761

Grubišić, A., Žitko, B., Stankov, S., Šarić-Grgić, I., Gašpar, A., Tomaš, S., Brajković, E., Volarić, T., Vasić, D., & Dodaj, A. (2020). A common model for tracking student learning and knowledge acquisition in different e-learning platforms. Journal of e-Learning and Knowledge Society, 16(3), 10–23. https://doi.org/10.20368/1971-8829/1135235

Henrie, C. R., Bodily, R., Larsen, R., & Graham, C. R. (2017). Exploring the potential of LMS log data as a proxy measure of student engagement. Journal of Computing in Higher Education, 30, 344–362. https://doi.org/10.1007/s12528-017-9161-1

Henrie, C. R., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring intensive longitudinal measures of student engagement in blended learning. The International Review of Research in Open and Distributed Learning, 16(3). https://doi.org/10.19173/irrodl.v16i3.2015

Holmes, N. (2018). Engaging with assessment: Increasing student engagement through continuous assessment. Active Learning in Higher Education, 19(1), 23–34. https://doi.org/10.1177/1469787417723230

Kew, S. N., & Tasir, Z. (2021). Analysing students’ cognitive engagement in e-learning discussion forums through content analysis. Knowledge Management & E-Learning: An International Journal, 13(1), 39–57. https://doi.org/10.34105/j.kmel.2021.13.003

Kim, D. (2017). The impact of learning management systems on academic performance: Virtual competency and student involvement. Journal of Higher Education Theory and Practice, 17(2), 23–35. http://www.na-businesspress.com/JHETP/KimD_Web17_2_.pdf

Kim, D., Lee, Y., Leite, W. L., & Huggins-Manley, A. C. (2020). Exploring student and teacher usage patterns associated with student attrition in an open educational resource-supported online learning platform. Computers & Education, 256, Article 103961. https://doi.org/10.1016/j.compedu.2020.103961

Kim, D., Park, Y., Yoon, M., & Jo, I.-H. (2016). Toward evidence-based learning analytics: Using proxy variables to improve asynchronous online discussion environments. The Internet and Higher Education, 30, 30–43. https://doi.org/10.1016/j.iheduc.2016.03.002

Kim, T., Yang, M., Bae, J.H., Min, B., Lee, I., & Kim, J. (2017). Escape from infinite freedom: Effects of constraining user freedom on the prevention of dropout in an online learning context. Computers in Human Behavior, 66, 217–231. https://doi.org/10.1016/j.chb.2016.09.019

Krasodomska, J., & Godawska, J. (2021). E-learning in accounting education: The influence of students’ characteristics on their engagement and performance. Accounting Education, 30(1), 22–41. https://doi.org/10.1080/09639284.2020.1867874

KunhiMohamed, B. B. V. (2012). Student participation and grade performance in an undergraduate online learning environment [Doctoral dissertation, Colorado State University]. CSU Theses and Dissertations. http://hdl.handle.net/10217/71599

Lee, Y., Choi, J. (2011). A review of online course dropout research: Implications for practice and future research. Educational Technology Research and Development, 59, 593–618. https://doi.org/10.1007/s11423-010-9177-y

Lee, J., Song, H.-D., & Hong, A. J. (2019). Exploring factors, and indicators for measuring student’s sustainable engagement in e-learning. Sustainability, 11(4), Article 985. https://doi.org/10.3390/su11040985

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. https://doi.org/10.1016/j.compedu.2009.09.008

Mansouri, T., ZareRavasan, A., & Ashrafi, A. (2021). A learning fuzzy cognitive map (LFCM) approach to predict student performance. Journal of Information Technology Education: Research, 20, 221–243. https://doi.org/10.28945/4760

Meyer, K. A. (2014). Student engagement in online learning: What works and why. ASHE Higher Education Report, 40(6), 1–114. https://doi.org/10.1002/aehe.20018

Mogus, A. M., Djurdjevic, I., & Suvak, N. (2012). The impact of student activity in a virtual learning environment on their final mark. Active Learning in Higher Education, 13(3), 177–189. https://doi.org/10.1177/1469787412452985

Moher, D., Liberati A., Tetzlaff J., & Altman D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ, 339, Article b2535. https://doi.org/10.1136/bmj.b2535

Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22–38). Routledge. https://doi.org/10.4324/9780203983065

Moore, M. G. (2018). The theory of transactional distance. In M. G. Moore & W. C. Diehl (Eds.), Handbook of distance education (4th ed., pp. 32–46). Routledge. https://doi.org/10.4324/9781315296135

Motz, B., Quick, J., Schroeder, N., Zook, J., & Gunkel, M. (2019, March 4–8). The validity and utility of activity logs as a measure of student engagement [Paper presentation]. In LAK19: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 300–309). Association for Computing Machinery. https://doi.org/10.1145/3303772.3303789

Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. American Journal of Distance Education, 34(2), 137–156. https://doi.org/10.1080/08923647.2020.1696140

Nkomo, L. M., & Nat, M. (2021). Student engagement patterns in a blended learning environment: An educational data mining approach. TechTrends, 65, 808–817. https://doi.org/10.1007/s11528-021-00638-0

Norze, J. (2020). Examining online student engagement in a program development course offered at a research university in the Southern Region of the U.S. Journal of Education and Human Development, 9(2), 13–16. https://doi.org/10.15640/jehd.v9n2a2

Peng, W. (2017). Research on model of student engagement in online learning. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 2869–2882. https://doi.org/10.12973/eurasia.2017.00723a

Rajabalee, Y. B., Santally, M. I., & Rennie, F. (2019). A study of the relationship between students’ engagement and their academic performances in an eLearning environment. E-Learning and Digital Media, 17(1), 1–20. https://doi.org/10.1177/2042753019882567

Ramesh, A., Goldwasser, D., Huang, B., Daume, H., III, & Getoor, L. (2014) Learning latent engagement patterns of students in online courses. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1). https://doi.org/10.1609/aaai.v28i1.8920

Raza, S. A., Qazi, W., Khan, K. A., & Salam, J. (2021). Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), 183–208. https://doi.org/10.1177/0735633120960421

Roach, V. A., & Attardi, S. M. (2021). Twelve tips for applying Moore’s theory of transactional distance to optimize online teaching. Medical Teacher. Advance Online Publication. https://doi.org/10.1080/0142159X.2021.1913279

Ross, S. M. (2019). Slack it to me: Complementing LMS with student-centric communications for the millennial/post-millennial student. Journal of Marketing Education, 41, 91–108. https://doi.org/10.1177/0273475319833113

Shah, M., & Cheng, M. (2019). Exploring factors impacting student engagement in open access courses. Open Learning: The Journal of Open, Distance and e-Learning, 34(2), 187–202. https://doi.org/10.1080/02680513.2018.1508337

Shah, R. K., & Barkas, L. A. (2018). Analysing the impact of e-learning technology on students’ engagement, attendance and performance. Research in Learning Technology, 26. https://doi.org/10.25304/rlt.v26.2070

Soffer, T., & Cohen, A. (2019). Student’s engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378–389. https://doi.org/10.1111/jcal.12340

Strang, K. D. (2017). Predicting student satisfaction and outcomes in online courses using learning activity indicators. International Journal of Web-Based Learning and Teaching Technologies, 12(1), Article 3. http://doi.org/10.4018/IJWLTT.2017010103

Summers, R. J., Higson, H. E., & Moores, E. (2021). Measures of engagement in the first three weeks of higher education predict subsequent activity and attainment in first year undergraduate students: A UK case study. Assessment & Evaluation in Higher Education, 46(5), 821–836. https://doi.org/10.1080/02602938.2020.1822282

Toro-Troconis, M., Alexander, J., & Frutos-Perez, M. (2019). Assessing student engagement in online programmes: Using learning design and learning analytics. International Journal of Higher Education, 8(6), 171–183. https://doi.org/10.5430/ijhe.v8n6p171

Tualaulelei, E., Burke, K., Fanshawe, M., & Cameron, C. (2021). Mapping pedagogical touchpoints: Exploring online student engagement and course design. Active Learning in Higher Education. Advance Online Publication. https://doi.org/10.1177/1469787421990847

Wang, F. H. (2017). An exploration of online behaviour engagement and achievement in flipped classroom supported by learning management system. Computers & Education, 114, 79–91. https://doi.org/10.1016/j.compedu.2017.06.012

Yassine, S., Kadry, S., & Sicilia, M.-A. (2016, April 10–13). A framework for learning analytics in Moodle for assessing course outcomes [Paper presentation]. In 2016 IEEE Global Engineering Education Conference (EDUCON) (pp. 261–266). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EDUCON.2016.7474563

You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement in online learning. The Internet and Higher Education, 29, 23–30. https://doi.org/10.1016/j.iheduc.2015.11.003

Young, S., & Bruce, M. A. (2011). Classroom community and student engagement in online courses. MERLOT Journal of Online Learning and Teaching, 7(2), 219–230. https://jolt.merlot.org/vol7no2/young_0611.pdf

Zaaruka, S., & Mosha, M. A. (2019, July 1–3). Learners’ communication tools preference—a comparison between institutional learning management systems and instant messaging. In EDULEARN19 Proceedings (pp. 2889–2893). International Academy of Technology, Education and Development. https://doi.org/10.21125/EDULEARN.2019.0777

Zanjani, N. (2015). Success factors of engaging higher education students and staff with e-learning tools within learning management systems (Publication no. 83940) [Doctoral dissertation, Queensland University of Technology]. QUT ePrints. https://eprints.qut.edu.au/83940/

Zepke, N. (2015). Student engagement research: Thinking beyond the mainstream. Higher Education Research and Development, 34(6), 1311–1323. https://doi.org/10.1080/07294360.2015.1024635