International Review of Research in Open and Distributed Learning

Volume 24, Number 1

February - 2023

What Are the Indicators of Student Engagement in Learning Management Systems? A Systematized Review of the Literature

Golchehreh Ahmadi¹, Aeen Mohammadi², Shadi Asadzandi³, Mahsood Shah⁴, and Rita Mojtahedzadeh^2
1MSc student, Department of e-Learning in Medical Education, Center of Excellence for e-Learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran; ²Department of e-Learning in Medical Education, Center of Excellence for e-Learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran; ³School of Health Management and Information Science, Iran University of Medical Sciences, Tehran, Iran; ⁴Swinburne University of Technology, Sydney, Australia

Abstract

Student engagement has an important role in academic achievement in all learning contexts, including e-learning environments. The extent of monitoring and promoting student engagement in e-learning affects the quality of education and is a determining factor for ensuring student’s success. Log data of students’ activities recorded in a learning management system (LMS) can be used to measure their level of engagement in the online teaching-learning process. No previous studies have been found stating a consistent and systematically raised list of LMS-based student engagement indicators, so this systematized review aimed to fulfill this gap. The authors performed an advanced search in the PubMed, Ovid, Google Scholar, Scopus, Web of Science, ProQuest, Emerald, and ERIC databases to retrieve relevant original peer-reviewed articles published until the end of June 2021. Reviewing the 32 included articles resulted in 27 indicators that were categorized into three themes and six categories as follows: (a) log-in and usage (referring to LMS, access to course material), (b) student performance (assignments, assessments), and (c) communication (messaging, forum participation). Among the categories, access to course material and messaging were the most and the least mentioned, respectively.

Keywords: e-learning, student engagement, learning management system, LMS, log data

What Are the Indicators of Student Engagement in Learning Management Systems? A Systematized Review of the Literature

Student engagement, being a multidimensional concept, is defined as the student’s amount of time and physical and psychological ambition devoted to fulfilling academic activities (Shah & Cheng, 2019). This includes students’ levels of attention, curiosity, interest, optimism, and passion while partaking in the teaching-learning process (Soffer & Cohen, 2019). Student engagement influences academic success regardless of the type of learning context and strategy; however; engagement is critical in an e-learning environment because of the physical distance between instructors and learners (Henrie et al., 2017). Moore (1993) explains his theory of transactional distance and points out that three elements—dialogue, course structure, and learner autonomy—are factors that affect students’ feelings of transactional distance. He argues that instructors, learners, and educational organizations can use these elements to plan for effective and deliberate learning (Moore, 1993). This theory of transactional distance has been used as the theoretical framework for research in online and technology enhanced learning, in which the medium of learner-instructor communication provides the chance for higher levels of interaction and engagement (Moore, 2018). This is important because the extent of monitoring and promoting student engagement in e-learning affects the quality of education (Henrie et al., 2017) and is a determining factor in whether an e-learning strategy is productive for educational institutions through ensuring students’ success (Meyer, 2014). Several studies have mentioned low levels of student engagement as the most important reason for student drop out in e-learning (Lee & Choi, 2011; Kim et al, 2017). Moreover, the level of student engagement shows the university’s level of commitment to academic activities and active learning (Lee et al., 2019). Implementing strategies to replace an e-student’s sense of isolation with relatedness and closeness facilitates their being more active in online courses, which results in the student’s higher satisfaction as well (Young & Bruce, 2011). Moreover, monitoring student engagement in e-learning environments helps in improving instructional events through recognizing the learners that need more support for following their studies on the path toward success (Henrie et al., 2015).

Studies on student engagement focus on two main aspects: learning behaviors and feelings of emotional belonging. In contrast to emotional belonging, learning behaviors are often measured quantitatively based on generic indicators of student engagement, which assess, report, and value the university’s performance. Such a qualitative analysis provides authorities and stakeholders with a feeling of certainty in understanding whether an educational process goes well or not (Zepke, 2015). Several studies have addressed the characteristics or constructs that make up student engagement assessment in a quantitative method, though a few have focused on the e-learning context (Lee et al., 2019). Most of these are cross-sectional studies, in which engagement self-reporting methods were used rather than continuous monitoring (Henrie et al., 2017). Meanwhile, an e-learning context with the possibility to instantly record indicators of student’s behaviors and learning activities within a learning management systems (LMS) provides a valid and approximate measure for student engagement in courses (Henrie et al., 2017; Motz et al., 2019).

LMSs are Web-based software used mainly for asynchronous interaction between instructors and students by delivering a course’s information, materials, and activities (Raza et al., 2021). Generated log data in LMS-based courses can be used as predictors of student achievement (Macfadyen & Dawson, 2010). Records of user’s activities within a software are labeled as log data, which include items such as number of clicks or page views, time spent on an action, and results of performed tasks or activities. Reviewing log data demonstrates a user’s real-time interaction with the software and can be analyzed to understand any changes in a user’s behavior (Henrie et al., 2017). The advantage of log data is that the data are automatically collected without any interference from instructors and staff. In addition, log data present objective information about aspects of a user’s behavior that are not easily measured in other ways (Macfadyen & Dawson, 2010; Zanjani, 2015). In fact, intelligent and effective analysis of LMS log data—that is, learning analytics—not only assists in promoting student’s success and retention rate and supporting at-risk learners (Atherton et al., 2017) but also provides the possibility to implement personalized learning, which is a revival of the learner autonomy concept in Moore’s (2018) transactional distance theory.The result of such analysis displays a student’s level of engagement and learning pattern, even at the initial stages of a course, and provides evidence for timely interventions to improve a student’s performance (You, 2016).

Hence, regarding the benefits of analyzing LMS log data to approximately measure student engagement, this systematized review of relevant literature was conducted to identify LMS indicators that can be used for this purpose.

Methods

The Cochrane Collaboration’s Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2009) was followed to conduct this systematized review. A systematized review follows the methodology of a systematic review and includes some of its elements; however, it cannot meet all the criteria for a full systematic review (Grant & Booth, 2009).

An advanced search was conducted in the PubMed, Ovid, Google Scholar, Scopus, Web of Science, ProQuest, Emerald, and ERIC databases on February 5, 2021, to retrieve relevant articles up to the end of June 2021. The search operators included Boolean operators (AND, OR, and NOT), parentheses, and truncation; the following keywords were searched as single terms or in combination with others: online learning, online education, distance learning, distance education, virtual learning, virtual education, e-learning, electronic learning, mobile learning, M-learning, distance study, distributed education, distributed learning, open learning, engagement, achievement, performance, progress, students, adult learners, learners, and users. Due to the limited number of articles dealing with the indicators of student engagement in LMSs, the search was not limited by keywords related to LMSs. The following is an example of a Web of Science search query:

TS = ((“online learning” OR “online education” OR “distance learning” OR “distance education” OR “virtual learning” OR “virtual education” OR “e-learning” OR “electronic learning” OR “mobile learning” OR “M-learning”) AND (“student engagement” OR “student achievement” OR “learners engagement” OR “learner’s achievement” OR “users engagement” OR “user’s achievement” OR “student involvement” OR “learner’s involvement”))

After extracting articles, duplicate ones were excluded using Endnote X8.2 (Clarivate Plc, London, UK).

The inclusion criteria for this review were as follows: (a) English language, (b) original articles, (c) student engagement in e-learning software being the subject of the study, and (d) availability of articles’ full texts. Studies that had used only special software or hardware facilities for monitoring student engagement, such as equipment for eye tracking, face recognition, or monitoring of mouse scrolling or movements, were not included. The reason was the focus of study was on data that were logged within routinely accessible e-learning software, independent of other external equipment or software. However, the articles including both types of monitoring were considered, and the results relevant to the aims of the review were retrieved. Studies on indicators of student engagement in online synchronous classes were omitted.

After article retrieval, two independent researchers reviewed the articles’ titles and abstracts using a standard checklist form to exclude irrelevant articles. Then they separately performed an in-depth assessment on articles’ full texts to determine their eligibility. At this stage, any inconsistencies between reviewers were resolved.

In the next step, the bibliographic data from each eligible article were extracted, and key findings and results were summarized and recorded. The PRISMA diagram (Figure 1) was considered for assessing the articles’ retrieval, extraction, and removal. Moreover, the Critical Appraisal Skills Programme checklist was used to investigate each article’s quality (Critical Appraisal Skills Programme, 2018).

Finally, two independent researchers thoroughly read the included articles in order to extract mentioned indicators of student engagement in an LMS log. As mentioned, any inconsistencies between these two researchers’ results were resolved by a third researcher’s review.

Results

Articles’ Retrieval and Bibliographic Information

After performing article retrieval steps, 32 articles were eligible to enter the study. Figure 1 shows the PRISMA diagram for this review, and Table 1 shows the search results based on databases.

Figure 1

PRISMA Flow Diagram for Retrieving Articles

Note. Adapted from Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement by Moher, D., Liberati A., Tetzlaff J., & Altman D. G., 2009, BMJ, p. 339 (https://doi.org/10.1136/bmj.b2535)

Table 1

Search Results

Table 2 includes information on included articles’ bibliographic characteristics. The 32 articles were published in 28 journals, among which Computers & Education had the highest number of publications (4 articles).

Table 2

Bibliographic Information of Included Articles in Chronological Order

Note. LMS = learning management system; OSE = online student engagement scale; VLE = virtual learning environment.

Identifying LMS Indicators for Student Engagement

After reviewing the articles, 27 indicators of student engagement in LMS log data were identified and classified into three themes and six categories based on their similarities. Table 3 includes these indicators and the article numbers (based on Table 2) that they are stated in.

Table 3

Student Engagement Indicators in LMS and Articles Stating Them

Note. LMS = learning management system.

Descriptions of each category are as follows:

Log-in and usage: This category and its indicators show not only the amount of time spent in an LMS but also the regularity and intervals of referring to it.

Access to course material: This category consists of indicators that demonstrate how much a student has interacted with course content and additional pages. Furthermore, the student’s preference to study the content before doing activities and their evaluation of the content are considered indicators of student engagement.

Assignments: Assignments are one of the main learning activities in an LMS-based course, so all potential LMS log data related to assignments are considered a category of student engagement indicators.

Assessments: LMS-based assessments provide objective data for estimating a student’s level of engagement with the course. Hence, in this category, all LMS log data related to assessments are listed.

Messaging: By default, students have access to the LMS messaging module to communicate with other LMS users. Sent and read messages show how much a student has interacted with instructors and peers in a course.

Forum participation: Indicators listed in this category estimate a student’s level of activity in forums or discussion groups in an LMS.

Figure 2 demonstrates the percentage of obtained indicators for each theme out of the total number of indicators, and Figure 3 depicts the frequency of articles stating each category. Moreover, among 32 articles, 13 (40.6%), 3 (9.4%), 3 (9.4%), and 13 (40.6%) used Moodle, Blackboard, Canvas, and other local LMSs, respectively. We calculated the number of citations of the indicators pertaining to each category in the related articles of each LMS type and conducted a Chi-square analysis to determine if there was any statistically significant difference among these LMS types in this regard. The results showed no significant difference except for in the messaging category ( p  = 0.033) (Table 4).

Figure 2

Student Engagement Themes

Note: LMS = learning management system.

Figure 3

Frequency of Articles Stating Each Category of Student Engagement Indicators in LMSs

Note: LMS = learning management system.

Table 4

Comparison of Indicators’ Number of Citations in the Articles Related to Each LMS Type in Total and for Each Category

Note. ^a Number of citations of the indicators pertaining to each category in related articles of each LMS type. *p < 0.05.

Discussion and Recommendations

In this systematized review on 32 articles, student engagement indicators based on LMS log data were identified and categorized into three themes of referring to LMS, student performance, and communication, which included six categories. Among these categories, access to course material was the most mentioned (23 articles), followed by forum participation (22 articles), log-in and usage (16 articles), assessments (14 articles), assignments (12 articles), and messaging (8 articles).

Despite the positive relationship between students’ activity levels within LMSs and their success in their courses (Grubišić et al., 2020), no study was found that cumulatively addressed LMS-based student engagement indicators. Hence, the results of this review provide insight into the indicators used for assessing student engagement in LMSs and their relative priority.

According to the findings of this study, access to course material in the LMS was the most mentioned indicator of student engagement in the literature. LMSs provide the option of uploading course materials to create a virtual learning environment in both fully online and blended courses (Chaka & Nkhobo, 2019). The opportunity to monitor and control students’ access to course material is one of the must-have features of LMSs and proves to be an indicator of student engagement level within the course (Krasodomska & Godawska, 2021). Students prefer having access to course material 24 hours a day, 7 days a week; and use this feature more than other LMS features. This results in increased engagement with the course and provides the possibility for students’ cognitive involvement, self-regulation, and self-paced learning (Chen, 2020). These results show the importance of monitoring such data as an indicator of student engagement in LMSs.

Forum participation was found to be the next considerable indicator for monitoring student engagement in LMSs in this study. In fact, LMSs have the feature of providing an environment for faculty and students communication, which helps in building a community of practice for collaborative learning rather than personalized individualized instruction (Moore, 2018). Moreover, students communicate with peers. A frequently used LMS communication tool is the discussion board or forum, which facilitates asynchronous collaboration among faculty members and students (Kew & Tasir, 2021). Analysis of data recorded in communication tools of LMS is supportive for assessing students’ behavioral, emotional, and cognitive engagement within e-learning environments (Henrie et al., 2017; Yassine et al., 2016). Forums allow for provision of feedbacks and comments on students’ work in order to promote academic goals (Kim, 2017). In addition, students’ active and passive participation in forum dialogues is positively associated with learning outcomes, and instructors and course designers concentrate on ensuring high levels of participation from students. Analysis of data gathered from forum participation helps in understanding its impacts on academic indicators, including the level of student engagement. Even though participation in forums can be obligatory, the level of students’ participation in forums may vary and shows their interest and engagement in course activities (Henrie et al., 2017; Yassine et al., 2016). In this regard, while students’ behavioral engagement is determined by the general use of the communication tools and platform (Mogus et al., 2012), their emotional engagement is analyzed through their self-expression in forums or discussion environments (Wang, 2017).

Log-in and usage, as a category within the referring to LMS theme, is the third ranked indicator for student engagement. Based on the existing research, LMS usage log data can be effectively used to measure student engagement (Wang, 2017; You, 2016). Student’s log-in and log-out data, as indicators of their behavioral engagement, have a strong positive correlation with their final grades (Mogus et al., 2012). In fact, spending more time within the LMS is associated with more engagement with the course’s activities and resources (Wang, 2017).

Other indicators extracted in this review are the categories under the themes of student performance, namely, assessments and assignments. Behaviors such as viewing and uploading assignments and participating in and completing quizzes play an important role in students’ academic performance and engagement (Franzoni et al., 2020). In fact, data such as the number of submitted assignments and completed online quizzes are used to quantify students’ regularity of participation in course activities and can show the level of students’ persistence in fulfilling learning expectations and engagement (Rajabalee et al., 2019). In other words, the more students are engaged in a course through revisiting and performing such activities, the more effectively they learn (Krasodomska & Godawska, 2021).

Messaging in the communication theme is the last ranked indicator of student engagement based on this review, with only 8 out of 32 articles stating it. The low number of mentions in these studies may be due to internal messaging not being a must-have feature of LMSs, and sometimes off-system messengers and social networks are preferred for this purpose (Ross, 2019; Zaaruka & Mosha, 2019). Meanwhile, communication activities in LMSs, such as number of messages, indicate students’ engagement in virtual environments (Ramesh et al., 2014). For example, in one study, students’ participation within an e-learning environment was analyzed by using the total number of students’ sent messages and total access; a moderate relationship was found between such participation and final grades (KunhiMohamed, 2012).

Finally, understanding such indicators is important because LMS usage is increasing in the academic sector, and students are spending more time and effort in these e-learning environments than ever before. Therefore, it is important to choose the appropriate LMS for a course, because it controls the way that learners engage with the course activities and interact with the material, their instructors, and their peers (Roach & Attardi, 2021). In this study, the number of citations of the indicators pertaining to each category in related articles showed that only messaging had a significant difference among LMS types, with institutions’ local LMSs having the highest number of citations of the indicator. This result may be due to the low number of the articles for each LMS type. However, even if LMS features are basic ones, it is possible to use them without compromising the quality of teaching through appropriate course design (Roach & Attardi, 2021). An effective course design needs gathering information about students’ participation and engagement through features offered by most LMSs that is helpful in substituting the insight that teachers gain about students’ learning in traditional classes.

Limitations

Despite the researchers’ efforts, this study has some limitations that must be considered. Although the literature search was conducted in multiple databases, as recommended for review articles (Cronin et al., 2008), possible bias in selecting databases or formulating search strategies may have resulted in missing relevant publications. Moreover, this review included only English-language articles. Articles in other languages and non-article publications, such as dissertations, may contain other indicators.

Future Research

Since research on student engagement in e-learning environment is still emerging, there are opportunities for future studies. Analyzing student engagement based on identified indicators of this study and comparing the results with the findings of the students’ self-reports of their engagement may expand knowledge in this regard. Furthermore, working on predictive models of student performance based on these indicators would provide the chance for early interventions to support students, prevent dropouts, and improve student retention. In spite of the usefulness of such quantitative analysis, in which course activities have the same weights, it is recommended to work on solutions to determine student engagement levels according to the importance and alignment of activities with learning objectives.

Conclusion

The results of this systematized review enrich the current literature. No previous studies have addressed a cumulative list of certain LMS-based indicators for measuring student engagement in e-learning environments. Hence, identifying such indicators has expanded the literature in this regard. Institutions and academics can use this list of indicators to (a) constantly monitor students’ engagement in asynchronous e-learning platforms, (b) determine strength and weaknesses of delivered e-courses, (c) identify and support students with low engagement levels, (d) plan for implementing personalized learning, (e) plan for faculty development programs to familiarize faculty with the activities that bring about higher levels of student engagement, and (f) compare institutions’ current LMS features and logs with the list of indicators to determine the ones that can be added to the software, if there are any, to promote the chance of engagement.

Author Note

We have no conflict of interest to disclose.

Correspondence concerning this article should be addressed to Rita Mojtahedzadeh. Department of e-Learning in Medical Education, Tehran University of Medical Sciences, Tehran, Iran. E-mail: r_mojtahedzadeh@tums.ac.ir

Acknowledgments

This article is a part of Golchehreh Ahmadi’s MSc thesis under the supervision of Dr. Rita Mojtahedzadeh, which was approved on May 3, 2020 by the research council at Tehran University of Medical Sciences, Teheran, Iran (IR.TUMS.VCR.REC.1399.331).

References

Al-Azawei, A., & Al-Masoudy, M. A. A. (2020). Predicting learners’ performance in virtual learning environment (VLE) based on demographic, behavioral and engagement antecedents. International Journal of Emerging Technologies in Learning, 15(9), 60-75. https://doi.org/10.3991/ijet.v15i09.12691

Atherton, M., Shah, M., Vazquez, J., Griffiths, Z., Jackson, B., & Burgess, C. (2017). Using learning analytics to assess student engagement and academic outcomes in open access enabling programmes. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 119-136. https://doi.org/10.1080/02680513.2017.1309646

Ayouni, S., Hajjej, F., Maddeh, M., & Alotaibi, S. (2021). Innovations of materials for student engagement in online environment: An ontology. Materials Today: Proceedings. https://doi.org/10.1016/j.matpr.2021.03.636

Ballard, J., & Butler, P. I. (2016). Learner enhanced technology: Can activity analytics support understanding engagement a measurable process? Journal of Applied Research in Higher Education, 8(1), 18-43. https://doi.org/10.1108/JARHE-09-2014-0074

Boulton, C. A., Kent, C., & Williams, H. T. P. (2018). Virtual learning environment engagement and learning outcomes at a “bricks-and-mortar” university. Computers & Education, 126, 129-142. https://doi.org/10.1016/j.compedu.2018.06.031

Chaka, C., & Nkhobo, T. (2019). Online module login data as a proxy measure of student engagement: The case of myUnisa, MoyaMA, Flipgrid, and Gephi at an ODeL institution in South Africa. International Journal of Educational Technology in Higher Education, 16, Article 38. https://doi.org/10.1186/s41239-019-0167-9

Chen, S.-B. A. (2020). Examining the effect of self-regulated learning on cognitive engagement in mastery-based online courses: A learning analytics perspective (Publication no.  28216212) [Doctoral dissertation, Ohio State University]. ProQuest Dissertations Publishing. https://www.proquest.com/openview/ec2b8156c96a026b68edc9ae9b058b67/1?pq-origsite=gscholar&cbl=18750&diss=y

Coussement, K., Phan, M., Caigny, A. D., Benoit, D. F., & Raes, A. (2020). Predicting student dropout in subscription-based online learning environments: The beneficial impact of the logit leaf model. Decision Support Systems, 135, Article 113325. https://doi.org/10.1016/j.dss.2020.113325

Critical Appraisal Skills Programme. CASP systematic review checklist, https://casp-uk.net/images/checklist/documents/CASP-Systematic-Review-Checklist/CASP-Systematic-Review-Checklist_2018.pdf (accessed 25 June 2021).

Cronin, P., Ryan, F., & Coughlan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), 38-43. https://doi.org/10.12968/bjon.2008.17.1.28059

Dixson, M. D. (2015). Measuring student engagement in the online course: The online student engagement scale (OSE). Online Learning, 19(4). https://doi.org/10.24059/olj.v19i4.561

Estacio, R. R., & Raga, R. C., Jr. (2017). Analyzing students’ online learning behavior in blended courses using Moodle. Asian Association of Open Universities Journal, 12(1), 52-68. https://doi.org/10.1108/AAOUJ-01-2017-0016

Franzoni, V., Milani, A., Mengoni, P., & Piccinato, F. (2020). Artificial intelligence visual metaphors in e-learning interfaces for learning analytics. Applied Sciences, 10(20), Article 7195. https://doi.org/10.3390/app10207195

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Green, R. A. (2017). The relationship between student engagement with online content and achievement in a blended learning anatomy course. Anatomical Sciences Education, 11(5), 471-477. https://doi.org/10.1002/ase.1761

Grubišić, A., Žitko, B., Stankov, S., šarić-Grgić, I., Gašpar, A., Tomaš, S., Brajković, E., Volarić, T., Vasić, D., & Dodaj, A. (2020). A common model for tracking student learning and knowledge acquisition in different e-learning platforms. Journal of e-Learning and Knowledge Society, 16(3), 10-23. https://doi.org/10.20368/1971-8829/1135235

Henrie, C. R., Bodily, R., Larsen, R., & Graham, C. R. (2017). Exploring the potential of LMS log data as a proxy measure of student engagement. Journal of Computing in Higher Education, 30, 344-362. https://doi.org/10.1007/s12528-017-9161-1

Henrie, C. R., Bodily, R., Manwaring, K. C., & Graham, C. R. (2015). Exploring intensive longitudinal measures of student engagement in blended learning. The International Review of Research in Open and Distributed Learning, 16(3). https://doi.org/10.19173/irrodl.v16i3.2015

Holmes, N. (2018). Engaging with assessment: Increasing student engagement through continuous assessment. Active Learning in Higher Education, 19(1), 23-34. https://doi.org/10.1177/1469787417723230

Kew, S. N., & Tasir, Z. (2021). Analysing students’ cognitive engagement in e-learning discussion forums through content analysis. Knowledge Management & E-Learning: An International Journal, 13(1), 39-57. https://doi.org/10.34105/j.kmel.2021.13.003

Kim, D. (2017). The impact of learning management systems on academic performance: Virtual competency and student involvement. Journal of Higher Education Theory and Practice, 17(2), 23-35. http://www.na-businesspress.com/JHETP/KimD_Web17_2_.pdf

Kim, D., Lee, Y., Leite, W. L., & Huggins-Manley, A. C. (2020). Exploring student and teacher usage patterns associated with student attrition in an open educational resource-supported online learning platform. Computers & Education, 256, Article 103961. https://doi.org/10.1016/j.compedu.2020.103961

Kim, D., Park, Y., Yoon, M., & Jo, I.-H. (2016). Toward evidence-based learning analytics: Using proxy variables to improve asynchronous online discussion environments. The Internet and Higher Education, 30, 30-43. https://doi.org/10.1016/j.iheduc.2016.03.002

Kim, T., Yang, M., Bae, J.H., Min, B., Lee, I., & Kim, J. (2017). Escape from infinite freedom: Effects of constraining user freedom on the prevention of dropout in an online learning context. Computers in Human Behavior, 66, 217-231. https://doi.org/10.1016/j.chb.2016.09.019

Krasodomska, J., & Godawska, J. (2021). E-learning in accounting education: The influence of students’ characteristics on their engagement and performance. Accounting Education, 30(1), 22-41. https://doi.org/10.1080/09639284.2020.1867874

KunhiMohamed, B. B. V. (2012). Student participation and grade performance in an undergraduate online learning environment [Doctoral dissertation, Colorado State University]. CSU Theses and Dissertations. http://hdl.handle.net/10217/71599

Lee, Y., Choi, J. (2011). A review of online course dropout research: Implications for practice and future research. Educational Technology Research and Development, 59, 593-618. https://doi.org/10.1007/s11423-010-9177-y

Lee, J., Song, H.-D., & Hong, A. J. (2019). Exploring factors, and indicators for measuring student’s sustainable engagement in e-learning. Sustainability, 11(4), Article 985. https://doi.org/10.3390/su11040985

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599. https://doi.org/10.1016/j.compedu.2009.09.008

Mansouri, T., ZareRavasan, A., & Ashrafi, A. (2021). A learning fuzzy cognitive map (LFCM) approach to predict student performance. Journal of Information Technology Education: Research, 20, 221-243. https://doi.org/10.28945/4760

Meyer, K. A. (2014). Student engagement in online learning: What works and why. ASHE Higher Education Report, 40(6), 1-114. https://doi.org/10.1002/aehe.20018

Mogus, A. M., Djurdjevic, I., & Suvak, N. (2012). The impact of student activity in a virtual learning environment on their final mark. Active Learning in Higher Education, 13(3), 177-189. https://doi.org/10.1177/1469787412452985

Moher, D., Liberati A., Tetzlaff J., & Altman D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ, 339, Article b2535. https://doi.org/10.1136/bmj.b2535

Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22-38). Routledge. https://doi.org/10.4324/9780203983065

Moore, M. G. (2018). The theory of transactional distance. In M. G. Moore & W. C. Diehl (Eds.), Handbook of distance education (4th ed., pp. 32-46). Routledge. https://doi.org/10.4324/9781315296135

Motz, B., Quick, J., Schroeder, N., Zook, J., & Gunkel, M. (2019, March 4-8). The validity and utility of activity logs as a measure of student engagement [Paper presentation]. In LAK19: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 300-309). Association for Computing Machinery. https://doi.org/10.1145/3303772.3303789

Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. American Journal of Distance Education, 34(2), 137-156. https://doi.org/10.1080/08923647.2020.1696140

Nkomo, L. M., & Nat, M. (2021). Student engagement patterns in a blended learning environment: An educational data mining approach. TechTrends, 65, 808-817. https://doi.org/10.1007/s11528-021-00638-0

Norze, J. (2020). Examining online student engagement in a program development course offered at a research university in the Southern Region of the U.S. Journal of Education and Human Development, 9(2), 13-16. https://doi.org/10.15640/jehd.v9n2a2

Peng, W. (2017). Research on model of student engagement in online learning. Eurasia Journal of Mathematics, Science and Technology Education, 13(7), 2869-2882. https://doi.org/10.12973/eurasia.2017.00723a

Rajabalee, Y. B., Santally, M. I., & Rennie, F. (2019). A study of the relationship between students’ engagement and their academic performances in an eLearning environment. E-Learning and Digital Media, 17(1), 1-20. https://doi.org/10.1177/2042753019882567

Ramesh, A., Goldwasser, D., Huang, B., Daume, H., III, & Getoor, L. (2014) Learning latent engagement patterns of students in online courses. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1). https://doi.org/10.1609/aaai.v28i1.8920

Raza, S. A., Qazi, W., Khan, K. A., & Salam, J. (2021). Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), 183-208. https://doi.org/10.1177/0735633120960421

Roach, V. A., & Attardi, S. M. (2021). Twelve tips for applying Moore’s theory of transactional distance to optimize online teaching. Medical Teacher. Advance Online Publication. https://doi.org/10.1080/0142159X.2021.1913279

Ross, S. M. (2019). Slack it to me: Complementing LMS with student-centric communications for the millennial/post-millennial student. Journal of Marketing Education, 41, 91-108. https://doi.org/10.1177/0273475319833113

Shah, M., & Cheng, M. (2019). Exploring factors impacting student engagement in open access courses. Open Learning: The Journal of Open, Distance and e-Learning, 34(2), 187-202. https://doi.org/10.1080/02680513.2018.1508337

Shah, R. K., & Barkas, L. A. (2018). Analysing the impact of e-learning technology on students’ engagement, attendance and performance. Research in Learning Technology, 26. https://doi.org/10.25304/rlt.v26.2070

Soffer, T., & Cohen, A. (2019). Student’s engagement characteristics predict success and completion of online courses. Journal of Computer Assisted Learning, 35(3), 378-389. https://doi.org/10.1111/jcal.12340

Strang, K. D. (2017). Predicting student satisfaction and outcomes in online courses using learning activity indicators. International Journal of Web-Based Learning and Teaching Technologies, 12(1), Article 3. http://doi.org/10.4018/IJWLTT.2017010103

Summers, R. J., Higson, H. E., & Moores, E. (2021). Measures of engagement in the first three weeks of higher education predict subsequent activity and attainment in first year undergraduate students: A UK case study. Assessment & Evaluation in Higher Education, 46(5), 821-836. https://doi.org/10.1080/02602938.2020.1822282

Toro-Troconis, M., Alexander, J., & Frutos-Perez, M. (2019). Assessing student engagement in online programmes: Using learning design and learning analytics. International Journal of Higher Education, 8(6), 171-183. https://doi.org/10.5430/ijhe.v8n6p171

Tualaulelei, E., Burke, K., Fanshawe, M., & Cameron, C. (2021). Mapping pedagogical touchpoints: Exploring online student engagement and course design. Active Learning in Higher Education. Advance Online Publication. https://doi.org/10.1177/1469787421990847

Wang, F. H. (2017). An exploration of online behaviour engagement and achievement in flipped classroom supported by learning management system. Computers & Education, 114, 79-91. https://doi.org/10.1016/j.compedu.2017.06.012

Yassine, S., Kadry, S., & Sicilia, M.-A. (2016, April 10-13). A framework for learning analytics in Moodle for assessing course outcomes [Paper presentation]. In 2016 IEEE Global Engineering Education Conference (EDUCON) (pp. 261-266). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EDUCON.2016.7474563

You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement in online learning. The Internet and Higher Education, 29, 23-30. https://doi.org/10.1016/j.iheduc.2015.11.003

Young, S., & Bruce, M. A. (2011). Classroom community and student engagement in online courses. MERLOT Journal of Online Learning and Teaching, 7(2), 219-230. https://jolt.merlot.org/vol7no2/young_0611.pdf

Zaaruka, S., & Mosha, M. A. (2019, July 1-3). Learners’ communication tools preference—a comparison between institutional learning management systems and instant messaging. In EDULEARN19 Proceedings (pp. 2889-2893). International Academy of Technology, Education and Development. https://doi.org/10.21125/EDULEARN.2019.0777

Zanjani, N. (2015). Success factors of engaging higher education students and staff with e-learning tools within learning management systems (Publication no. 83940) [Doctoral dissertation, Queensland University of Technology]. QUT ePrints. https://eprints.qut.edu.au/83940/

Zepke, N. (2015). Student engagement research: Thinking beyond the mainstream. Higher Education Research and Development, 34(6), 1311-1323. https://doi.org/10.1080/07294360.2015.1024635

What Are the Indicators of Student Engagement in Learning Management Systems? A Systematized Review of the Literature by Golchehreh Ahmadi, Aeen Mohammadi, Shadi Asadzandi, Mahsood Shah, and Rita Mojtahedzadeh is licensed under a Creative Commons Attribution 4.0 International License.