Abstract

As education has evolved towards online learning, the availability of learning materials has expanded and consequently, learners’ behavior in choosing resources has changed. The need to offer personalized learning experiences and content has never been greater. Research has explored methods to personalize learning paths and match learning materials with learners’ profiles. Course recommendation systems have emerged as a solution to help learners select courses that suit their interests and aptitude. A comprehensive review study was required to explore the implementation of course recommender systems, with the specifics of courses and learners as the main focal points. This study provided a framework to explain and categorize data sources for course feature extraction, and described the information sources used in previous research to model learner profiles for course recommendations. This review covered articles published between 2015 and 2022 in the repositories most relevant to education and computer science. It revealed increased attention paid to combining course features from different sources. The creation of multi-dimensional learner profiles using multiple learner characteristics and implementing machine-learning-based recommenders has recently gained momentum. As well, a lack of focus on learners’ micro-behaviors and learning actions to create precise models was noted in the literature. Conclusions about recent course recommendation systems development are also discussed.

Keywords: online learning, personalization, course recommender systems, course features, learner profiles

Introduction

The advancement of information and communication technology in recent decades have transformed traditional business models. Like other business sectors, the education industry has benefited from these developments, shifting from traditional classrooms to more online formats. Additionally, the COVID-19 pandemic emphasized the inevitability of transforming toward an online educational model. Although online education has been gaining popularity among students and instructors, there have been concerns about its effectiveness and efficiency in terms of learning outputs. For example, an average of less than 10% of registered MOOC participants actually complete their courses (Reparaz et al., 2020).

One of the biggest problems with e-learning platforms has been lack of personalization, defined as the tailoring of pedagogy, curriculum, and learning environments to meet learners’ needs (Baguley et al., 2014). With the recent growth in Web-based educational systems, delivering learning materials based on individual learners’ interests and competencies has become more challenging. E-learning systems produce a huge amount of data about students’ behavior, but it is impossible to analyze it manually. The exploitation of this data to personalize learning materials and extract meaningful insights about the learning process can benefit students, instructors, and institutions (Baker et al., 2016).

Recommendation systems (RSs) initially emerged as filtering methods to help users make decisions in the case of information overload. RSs discover the preferences of different users and predict items that correlate to their needs. These systems have been heavily employed by e-retailers to increase the reach and sales of their products. In the context of online education, recommender systems help facilitate decision making for students, instructors, and even institutions. RSs have been increasingly used for learning purposes with different applications ranging from recommending learning materials, to forum threads, or even peer recommendations (Khalid et al., 2020). With a shared interest in how educational data may be used to advance both education and the science of learning, learning analytics (LA) and educational data mining (EDM) communities have grown (Berland et al., 2014). LA and EDM are interdisciplinary areas providing solutions for recommender systems such as, among others, information retrieval, visual data analytics, domain-driven data mining, social network analysis, psychopedagogy, and so on (Romero & Ventura, 2020).

One pivotal choice that students often face is deciding which courses to enroll in. Opting for the most suitable courses that align with their interests while simultaneously advancing their preparation for future career prospects is an admirable trait. The abundance of courses offered by different educational institutions makes course selection a challenging task for learners. To help learners with this decision, RSs need to be adjusted for the educational context. E-learning recommender systems share similarities with well-known recommender systems used in e-commerce in the sense that they contain users, items, and ratings. In this study, when referring to the elements of e-learning recommendation systems, courses were considered as items and while learners were viewed as users, it is important to note that they should not be equated with typical users of e-commerce systems who are typically seen as potential buyers. However, using an advanced algorithm to predict learners’ perception of a recommended course is insufficient. To recommend courses with a higher probability of satisfactory completion, it is crucial to create elaborate learner models and pay attention to every detail about their static and dynamic data such as learning attitude and aptitude, background knowledge, skills, competence level, and so on (Abyaa et al. 2019).

In e-learning settings, the specific characteristics of users and items have led to proposals for different course recommendation systems (CRS) in recent years. To our knowledge, no previous study has intensively investigated learner and course characteristics and identified the features that recommender systems have considered to match courses to learners’ educational profiles. In this study, we reviewed recent advancements in CRS from 2015 to 2022 in order to identify different learner characteristics and course features for use in making recommendations. As well, the literature review distinguished trends and gaps in designing and implementing course recommender systems and generated future research directions in this field.

Previous Studies on Educational Recommender Systems

With increased research attention to e-learning, the number of publications that have proposed recommendation solutions to improve e-learning has escalated. Systematic literature reviews were conducted to shed light on e-learning recommender systems from different perspectives. Klašnja-Milicevic et al. (2015) conducted a comprehensive survey of e-learning environments recommender systems, and they analyzed 160 articles to find challenges in designing recommendation systems and their usefulness for personalized recommendations. Their focus was on collaborative tagging systems to extend the capabilities of recommendation systems for better delivery of learning objects. They investigated recommenders and summarized their results separately based on (a) matrix factorization methods, (b) collaborative filtering, (c) content-based approaches, and (d) association rule mining.

Recently, some review studies have been more concerned with the perception of learners as users of e-learning recommendation systems. For example, Yago et al. (2018) analyzed the role of competencies in proposed recommendation systems to discover their strength and weaknesses. They emphasized the importance of powerful learner modeling techniques to provide adaptive learning solutions, and they assessed the coverage, robustness, adaptivity, and scalability of proposed recommendation systems. They also analyzed the method of access (i.e., Web or desktop) and whether the individual who accessed the recommendation system was a student, instructor, lecturer, or professor. Yago et al. concluded that competence-based recommenders should consider factors related to learning resources, such as the representation taxonomies, besides other common drawbacks (e.g., overspecialization or cold start). In another remarkable study, Deschênes (2020) examined the effect of learning recommendation systems on learners’ agency, defined as their ability to set and follow through on their learning goals. Since learning paths restrict learners’ agency more than support it, the review focused on articles that introduced recommendations for learning resources and excluded those that recommended learning paths. The review categorized the presentation methods of recommendation results and how these affected learners’ satisfaction and performance. In a similar literature review, Salazar et al. (2021) investigated the relationship among recommendation systems, and learners’ emotional state and decision making. They focused on research in which emotions were used as a driving force to improve recommendations in a virtual education environment. They concluded that there were four main sources of emotion extraction (i.e., body gesture, facial expression, speech, and physiological sensors) in previous research, and that the methods to extract emotional state from these sources have not been depicted in the literature. Salazar et al. argued that more work was needed to improve the personalization and consequently engagement level of students in the learning process.

Recently, machine learning techniques have gained attention as ways to analyze educational data and create recommender systems. Khanal et al. (2020) conducted a literature review to classify different machine-learning algorithms applied to e-learning recommender systems. They categorized recommendation approaches into four groups: (a) content-based, (b) collaborative filtering, (c) knowledge-based, and (d) hybrid approaches. A key aspect of the work was an analysis of the datasets used for applying machine learning algorithms and dividing datasets into test and train subsets. They listed machine learning algorithms used in previous research and showed that clustering algorithms like k-means and k-nearest neighbor were frequently used in e-learning recommendation systems. They limited their review to journal articles published from 2016 to 2018. Tarus et al. (2018) reviewed ontology-based recommendations for education published from 2005 to 2014. They classified journal articles according to year of publication, recommendation technique, and type of learning resources for recommendations. They also explained different ontology representation languages such as OWL, DL, RDF, XML, and SWRL to integrate ontology representation with other knowledge-based recommendation techniques. Tarus et al. concluded that incorporating ontologies into the recommendation process can increase the accuracy of recommendations while also helping to solve issues with cold start and data sparsity.

Khalid et al. (2020) reviewed 89 articles to identify new trends in recommendation system applications for MOOCs. They categorized MOOC recommender systems based on the (a) course subject; (b) forum threads; (c) peers; (d) learning elements; (e) MOOC provider/teacher recommendation; (f) student performance; and (g) other recommender applications. They showed that 62% of MOOC recommenders focused on course or learning element recommendations. Kahid et al. stated that from 2017 onward, researchers applied data mining, neural networks, and deep learning techniques in processing data for MOOC recommendation applications. Also, they noted that authors frequently used receiver operating characteristics, recall, and precision metrics for recommender evaluations. Uddin et al. (2021) conducted another survey on MOOCs that illustrated the unavailability of a public dataset, with social information as a major gap in designing dynamic recommendation systems for MOOCs. They categorized MOOC recommender systems into nine different technology groups—namely (a) machine learning, (b) deep learning, (c) learning analytics, (d) hybrid approaches, (e) context-sensitive, (f) collaborative filtering, (g) knowledge-based, (h) ontology-based, and (i) content-based—and showed that machine learning solutions had received more research attention during the last few years. Moreover, Guruge et al. (2021) reviewed course recommender systems by conducting a classification of papers according to the year of publication and the techniques used for course recommendations. They concluded that hybrid and data mining techniques grew in popularity among course recommenders from 2016 to 2020. Their review stated that most current studies have estimated learner preferences based on their profiles, a static and one-directional form of user representation.

Relevant literature reviews have shown two major concentrations among e-learning recommender systems. One line of research has focused on the learner’s perception as the end user of recommendation; the other line has been more concerned with data mining techniques and technical solutions to predict recommendation ratings. Unlike previous literature review studies, we aimed to explore practical course recommendation proposals that emphasized personalization and solutions that considered different aspects of courses and learners. Table 1 summarizes the latest recommender systems literature reviews and the research questions they addressed. Our study investigated the latest developments in course feature extraction and learner modeling in online course recommendation applications.

Table 1

Articles Focused on e-Learning Recommendation Systems

Considering the two major research lines, there was a need to review the research on how proposed recommendations model learners and extract their characteristics from different perspectives. Besides extracting learner profiles, another input was the features of the items that are considered for recommendation. We focused on courses as the recommendation objects and analyzed how previous research extracted course information for the recommendation.

Search Methodology

This section explains the protocol we followed to search and select research articles for the review. Kitchenham (2004) has stated that a review protocol is essential as it defines the method to undertake the study. Primarily, we needed to identify the research goals and research questions. Recommender systems for online learning environments need to be elaborated carefully since learners remain passive to give explicit feedback. These recommendations mostly rely on the content of the items and the user behaviors collected as they use the online system. The goal of this study was to conduct an exhaustive literature review of course recommender systems. We focused only on course recommendations for students who were studying for their degrees at universities, or lifelong learners who were looking for skills helpful to their careers. We sought to answer the following two research questions.

1. Which course features were extracted and employed by recommender systems?
2. Which learner characteristics were used for creating their profiles and recommending courses?

The next step was to identify the keywords best suited for finding course recommendation systems. The literature search terms comprised words and combinations such as (a) course, (b) learning resource, (c) recommender or recommendation, (d) selection, and (e) system. Our search timeline spanned eight years from the beginning of 2015 until the end of 2022. We selected journal and conference articles from top-level electronic databases, namely (a) IEEE Xplore, (b) ACM Library, (c) Springer Link, (d) ERIC, (e) Wiley Online Library, (f) EBSCO, (g) ScienceDirect, (h) Taylor and Francis Online, (i) Scopus, and (j) Web of Science. We selected these online databases because they contain articles relevant to our literature review topic. In particular, ERIC focuses on education sciences of all kinds and their advancements. Additionally, to ensure our search was comprehensive, we checked Google Scholar for possible missing articles in the aforementioned databases. We conducted the initial search and retrieval of articles in April 2022 and updated it in April 2023. Since searching solely for keywords resulted in an immense draft of papers, we needed to restrict our search to specific phrases. Every database has its own tools and guidelines to narrow down the search results. Table 2 illustrates the search query and the number of articles for different databases.

Table 2

CRS Search Phrases and Initial Results

These searches resulted in 1,546 found articles, of which 570 results were repetitive and represented in multiple databases. For example, Esteban et al. (2020) was indexed by EBSCO, ScienceDirect, WoS, and Scopus databases simultaneously. We removed the duplicate articles based on the order of their appearance in our database searches. After eliminating duplicate results, the remaining number articles equaled 976. To further refine our results, we read the abstract and introduction for each article to determine its aim, application, and research focus. We applied multiple criteria to exclude articles from further review. These exclusion rules are summarized below:

• ExCrit1: recommender system not related to the education domain or course recommendation in particular
• ExCrit2: the article does not introduce a recommender system, but recommends optimal performance or analysis in education
• ExCrit3: other review research on educational recommenders
• ExCrit4: presentations, reports, magazine covers, or thesis
• ExCrit5: full text or abstract of the paper not available

Applying these exclusion criteria based on the content of the paper’s abstract and introduction narrowed our search results to 322 for extensive study. Figure 1 shows the number of papers selected for inclusion in the next phase and the articles excluded based on the criteria explained above.

Next, we read each article thoroughly to identify those that addressed our research questions. Papers with a lack of information about any of the research questions were removed from further analysis. We focused on the research that explicitly considered courses as recommendation items and used learner information for personalization purposes.

Figure 1

Number of Papers Included In and Excluded From Extensive Reading Phase

After this stage, we decided to include 48 articles for the course recommender literature review from 2015 until 2022. At the same time, to validate our findings, we shared the list of 322 articles and our research questions with an expert in the field to validate our findings. Expert opinion regarding the articles to be included was different on three occasions. Compared to the expert opinion, our filtering had an accuracy of 99.07%. The total number of articles in our final literature review equaled 51. Figure 2 illustrates the summary of articles selected for review, and Figure 3 shows the yearly distribution of this final list by publication type (i.e., journal or conference article).

Figure 2

Article Selection Process Adapted From the PRISMA Flowchart

Adapted from “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews,” by M.J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T.C. Hoffmann, C.D. Mulrow, L. Shamseer, J.M. Tetzlaff, E.A. Akl, S.E. Brennan, R. Chou, J. Glanville, J.M. Grimshaw, A. Hrobjartsson, M.M. Lalu, T. Li, E.W. Loder, E. Mayo-Wilson, S. McDonald, L.A. McGuinness, L.A. Stewart, J. Thomas, & D. Moher BMJ 2021, 272(71). 2021 by BMJ.

Figure 3

Article Types and Yearly Distribution

Results on Course Feature Extraction for Recommendation Systems

The two main elements of recommendation systems are users and items. In our analysis, we regarded items as courses or subjects that aligned with learners’ profiles, who represented the recommendation systems’ users. To recommend a course with a high possibility of matching a learner’s profile, we needed to understand how research in the literature extracted learner models and course features. This section presents an overview of previous publications that provided information about the features and information related to courses in the recommendation process. We proposed a framework that classified course features into major categories, namely, (a) course correlations and prerequisites; (b) static course information (e.g., university, department, instructor, language, fee, required hours); (c) course description and covered subjects; (d) comments and ratings; (e) enrolment history; and (f) combinations of features from these categories.

Course Correlations and Prerequisites

This group of features was mainly concerned with the relation among courses in terms of academics or semantics. Course difficulty level and complementary topics also fell into this group. As an example of the first category, Jing and Tang (2017) introduced a hybrid course recommendation system that calculated the transfer probability from course A to course B based on enrollment history. They extracted prerequisite relations between courses based on these probabilities. For example, if course A is a prerequisite to enrolling in course B, the transfer probability of A to B is much higher than the transfer probability of B to A. This probability calculation was used as a weighted input for their hybrid recommendation system. Similarly, Yin et al. (2020) introduced a recommendation model that used transition probability based on the learners’ enrolment history. They calculated the percentage of learners who take course A after attending course B. Additionally, they used the semantic structure of course topics and their connection to strengthen their hybrid recommendation model.

Huang et al. (2018) used an FP-growth association rule mining algorithm as part of the proposed hybrid recommendation system to find the relation between courses. Similarly, Yang et al. (2018) proposed an Apriori algorithm to find association rules among courses. They calculated the similarity between courses and integrated it into predicting learner results in a course and consequently recommending courses with the higher predicted score. Zhao et al. (2020) used concept-level relations to build course-level prerequisite relations. The method found similarity and concept-level relations with analyzing MOOC video captions to recommend which courses were better to take after a particular course. Chen et al. (2022) indicated that enrolled courses in the distant past are not as informative as recently enrolled courses. As a result, they used course enrolment sequence and course prerequisite information to construct a collaborative sequence graph for recommending relative courses.

Course Static Information

For the second category, there was an extensive range of static information sources for creating course feature models. For example, Elbadrawy and Karypis (2016) proposed grade prediction and course recommendation that exploited course subjects and levels based on the semester they were offered for degree students. Ibrahim et al. (2018) used course title, major subject, fee, university location, and even university rank to build an ontology-based course recommendation. Pardos and Jiang (2020) added course instructors and departments to the course representation of course2vec (Pardos et al., 2019), a model that represented student enrollment sequences chronologically, based on course contexts. H. Zhang et al. (2019) used the course resource library to create feature vectors with course grade, creator, and school. Xu and Zhou (2020) created multi-dimensional course features with historical data on course duration, number of video plays by learners, number of comments, and video or audio features of course content to illustrate what course factors attracted learners in online learning platforms.

Xu et al. (2021) considered hours required for course completion in addition to course subjects and instructors to build knowledge graphs. The extracted knowledge graphs found similarities among courses to combine with collaborative filtering for recommending courses. Urdaneta-Ponte et al. (2021) analyzed professional job databases and extracted information on the required skills to succeed in courses and new skills acquired after successful completion. They built related jobs for learners by using knowledge graphs and predicting the clusters that a course belonged to. Similarly, Yang and Cai (2022) used attributes such as instructor, industry, technical direction, and course form to construct a course knowledge graph. Sakboonyarat and Tantatsanawong (2022) considered information regarding the course institution, number of chapters, registration, and completion time to represent their input data for the course recommender system. These attributes were used to create course learning data groups and combined with user learning data to feed the recommendation deep neural network.

Course Description and Covered Subjects

Text mining, topic modeling, and semantic analysis of course descriptions and syllabus topics have all been used to construct course models. Ng and Linn (2017) analyzed course topics with a popular machine-learning algorithm called latent Dirichlet allocation (LDA), to identify topic distribution through a corpus of degree studies and within the course description. Similarly, Xia (2019) presented the contextual meaning of documents with vectors and calculated the similarity between learner query and course description vectors to recommend courses. Tan et al. (2020) used a long short-term memory (LSTM) network to extract course information from descriptions to predict the relevance of courses with a learner’s preferences.

Pang et al. (2018) defined a distance measure by fine-graining course videos. They used video properties like knowledge point, subject, and stage of the course at which the learner watched the video. Li and Kim (2021) proposed a model that embedded courses within their subjects from sparse data and extracted a course attribute module to represent the topics each course covered. Jung et al. (2022) proposed a graph-based model that considered the inclusion of keywords in the courses and embeds it with the keywords related to learners based on their interactions with the courses. Premalatha et al. (2022) mapped course contents in the curriculum with predefined domains suggested by experts. They classified elective courses into domains and recommended them based on the learner’s expertise domain.

Course Comments and Ratings

This category analyzed learners’ comments and their reactions to the courses in which they were previously enrolled. Chang et al. (2016) built a quality-control mechanism to prevent recommending courses with instructors that were rated poorly by learners. Bakhshinategh et al. (2017) assessed learners’ feedback with 28 sub-attributes of courses on a five-point scale. They perceived the given score as graduating values for students and used it for collaborative filtering purposes. Zhu et al. (2020) exploited learner ratings and textual comments about courses and teachers to predict their ratings for the classes they have not taken. They even analyzed learner lexical style in commenting on different courses to build learner relation networks. Likewise, to present their collaborative filtering course recommendation system, Man et al. (2022) calculated course similarities based on learners’ ratings and scores.

Course Enrolment History

Most research into course recommender systems has considered learners’ previous enrolments and obtained grades as the source of information to build their models. In early research, Khorasani et al. (2016) proposed a course recommending model based on historical enrollment data without considering course prerequisites or degree requirements. Al-Badarenah and Alsakran (2016) employed a clustering algorithm to build student groups based on their previous grades and find the closest group to the target student to recommend courses they succeeded in. Similarly, Bridges et al. (2018), Jiang et al. (2019), Morsy and Karypis (2019), Asadi et al. (2019), Yang and Jiang (2019), Salehudin et al. (2019), J. Zhang et al. (2019), Li et al. (2020), and Nguyen et al. (2021) focused on historical enrollment data for predicting grades and recommending courses. In another study, Ma et al. (2020) broke down learners’ previous enrollment data to measure how interesting and timely a course was in order to be recommended and additionally predicted what scores learners would achieve. Guo et al. (2022) used the votes each course received to recommend items based on learner interest and to prevent the cold start problem. Wang (2022) selected the number of participants in courses and their scores to define a measure of course popularity, and used course tests and completion rates to find learners’ recognition levels.

Combination of Course Features

In the literature, some research has combined different types of course-related information to extract desired features and construct their course models. Yanhui et al. (2015) used course major, category, and description similarity together with course ratings to create a group recommender system. Symeonidis and Malakoudis (2019) merged information about the skills to be covered, retrieved from the course title and description, with learner ratings. As a result, they built a course-skill matrix to fuse with the user-skill matrix to provide recommendations. Esteban et al. (2020) explained that an important factor for course recommendation is the coincidence of professors with courses learners liked in the past. Subsequently, these authors combined course professor information and knowledge area with course theoretical and practical content analysis to create course models. Cao and Chang (2020) used course static information like course department and school, together with course description word analysis, to build their recommendation systems its content-based filtering aspect.

The combination of different course features to build recommendations gained more attention during the last two years of our literature review coverage. Fan et al. (2022) integrated course reviews with course descriptions in different granularity of words and sentences to find personalized learning patterns and recommend courses based on the mined patterns. Zhou et al. (2022) proposed a method to build a course knowledge graph from demographic information such as teacher and school, fused with prerequisite relations and concepts a course covered. Jiang et al. (2022) integrated course name and teacher information with the segmented introduction to calculate the frequency of subject word appearance in course descriptions.

Agarwal et al. (2022) exploited course static attributes like name, provider, duration, language, and fee along with a set of elements from course videos and reading materials, as well as course enrollment and completion rate to create course ontologies for a knowledge-based recommendation system. In a similar work, Agrawal and Deepak (2022) modeled courses based on course titles and descriptions, and then combined them with course ratings, difficulty, and learner enrolment to build a massive course ontology. Wang et al. (2022) exploited several available information types including enrollment history, prerequisite restrictions, and other contextual data such as course meeting times, instructors, and instructional methods. Ahmad et al. (2023) used information from learner interaction with courses based on enrolment data and generated a bipartite network of learners and courses. For course nodes, they used attributes from course descriptions and fields.

Figure 4 illustrates the yearly distribution of the information sources used for course modeling. In the early years covered by our review, course recommenders mainly used single sources of information. Enrolment and grades history were the main focus of the research to create rating tables for recommendations, and used predictive methods in order to calculate learners’ interest and expected outcome for the courses that were not taken by them. These methods are widely used in the commerce world and ignore the different attributes of courses and learners. Our review suggested that the combination of course information resources to create course models have been investigated in the last years, and this trend will likely be a major focus in the foreseeable future. Five out of nine articles published in 2022 used a combination of course features. Still, more research is needed to combine information resources differently and create more precise course models.

Figure 4

Yearly Distribution of Course Feature Sources for Recommendation

Results on Learner Characteristics for Course Recommendation Systems

In the literature, we identified six types of major characteristics of learners for course recommenders to consider. The first category, learner static profile, consisted of static learner profile data such as identity number, name, age, occupation, and so on. The second source of information was learner ratings explicitly given to previous courses. The third type included learners’ previous enrolment and performance. The fourth group of attributes was related to learner activities and their interactions with the online learning platform. The fifth feature considered for creating learner profiles was learners’ skills and cognitive characteristics. The last attribute category covered learners’ motivation and interests, discovered implicitly or explicitly based on their behaviors captured via their interactions with the system. This category has usually been exploited together with different learner feature resources explained in the last part of this section. This last part also includes the previous works that considered a combination of multiple attributes to create learner profiles for course recommendations.

Learner Static Profile (Demographic Information)

A learner’s profile information contains static uninterpreted characteristics, such as demographic information, age, gender, and so on. Asadi et al. (2019) considered students’ age, gender, high school GPA, and university entrance exam scores to create learner profiles. They created learner clusters based on the similarity of these attributes. In another work, Huang et al. (2018) proposed a course recommendation model based on learners’ academic social networks and their ties with other learners.

Urdaneta-Ponte et al. (2021) proposed a lifelong learning course recommendation system that used learners’ demographic information, occupational information, and skills extracted from LinkedIn. Similar to the previous work they used this information to cluster entities in order to recommend courses collaboratively. Li and Kim (2021) used static information like users’ jobs, certificates, and language skills to match courses with learners’ profiles. In other research, students’ location, birthdate, gender, and level of education were used for MOOC recommendations (Sakboonyarat & Tantatsanawong, 2022).

Learner Ratings

Some works surveyed learners’ interests and opinions directly with questionnaires or asked them to input their interests and goals textually. For example, Yanhui et al. (2015) clustered similar learners into groups based on the similarity of their preferences and ratings over previous courses. Bakhshinategh et al. (2017) assessed students’ opinions about graduating attributes, defined as qualities and skills that universities aim to develop during students’ time in the institution. They recommended courses based on the weighted sum of five -point ratings given by students regarding 28 graduating attributes.

In addition to information about the course in general, some studies have captured learners’ opinions about details such as topics or course instructors. Through surveys, Ng and Linn (2017) asked learners about their preferences regarding the course level, desired topics, and professors’ ratings. In similar research, Xia (2019) explicitly asked about learners’ desired occupational positions and tried to recommend courses that supported students to be prepared for their career goals. Esteban et al. (2020) proposed a recommender system for university students majoring in computer science to choose elective courses based on their previous ratings of courses and branch of study. Zhu et al. (2020) created a model based on learners’ ratings and text comments about courses, teachers, achieved grades, and supervisors. Guo et al. (2022) extracted learners’ characteristics from submitted text and represented this information to build a six-dimensional learner model vector. Jiang et al. (2022) designed users’ interest models based on the labels they assigned to course topics. They used this information to determine learners’ preferences and their opinion about the online course quality.

Learners’ Previous Enrolment and Performance

Previous research on CRS has included numerous works that relied heavily on predictive analysis of historical enrolment and grade data. These methods are based on machine learning and data analytic techniques. Khorasani et al. (2016) introduced a course recommendation model based on historical enrollment data and no prior knowledge of the course prerequisites or degree requirements. Al-Badarenah and Alsakran (2016) proposed a learner clustering method based on course grades and applied a collaborative filtering method to recommend elective courses. Bridges et al. (2018) used learners’ historical grades and enrollment data to form a directed graph to show students’ transition possibilities from completing one course to the next. Yang and Jiang (2019) created a learners’ network based on registration and achieved score data. In the initial network, nodes represented learners, and edges between nodes meant the connected nodes enrolled in the same course before. Salehudin et al. (2019) used learners’ previous enrollment data along with their grades to calculate the similarity between learners. They use this similarity to recommend courses to a target student that had not taken the courses enrolled by similar students (neighbors).

Some studies examined learners’ previous scores to recommend courses for learners who were expected to perform well and acquire high marks. Yang et al. (2018) investigated learners’ course performance data based on their major, gender, and grades. Jiang et al. (2019) proposed a goal-based recommendation system to predict learners’ performance in upcoming courses based on their previous grades. Similarly, Morsy and Karypis (2019) introduced a grade-aware method to recommend courses in which students were expected to perform well. By learning from previous grades, they estimated the students’ grades in future courses. Ma et al. (2020) studied the reasons for course selection in universities; they found that getting relatively higher grades was one of the factors that influenced learners’ choices. Based on learners’ previous grades, they estimated how prepared they were for the upcoming courses. In a similar work Nguyen et al. (2021) showed that students selected courses that they thought would result in a better learning outcome. They used the students’ previous grades for predicting learning outcomes. Zhou et al. (2022) introduced a time-aware recommendation system that considered learners’ sequential enrolment data to recommend courses that matched temporal learner interests. Premalatha et al. (2022) proposed a learner domain expertise model that analyzed the number of elective courses students completed and the grades they achieved.

A line of research in recent years has used learners’ enrolment history alone to build course recommendation systems with machine learning and neural network techniques without considering learners’ personal traits and features (Chen et al., 2022; Li et al., 2020; Wang et al., 2022; Zhao et al., 2020).

Learner Activities

There has been a consensus in personalized learning that learners’ actions regarding different resource materials are an important factor in creating learner profiles. Most of the previous research combined learner activity data with other resources to demonstrate learner models. Pang et al. (2018) proposed adaptive MOOC recommendations that adopted learning duration as the key feature for creating learner models. Specifically, they calculated the time learners spent watching educational videos and calculated learners’ similarity based on the video topics they watched. Similarly, Xu and Zhou (2020) used learners’ video play and view records to determine their preferences. Agrawal and Deepak (2022) examined learners’ micro-actions, such as their clicks on the online platform, to build a recommendation model. They used unique terms and investigated learners’ navigation patterns.

Learner Skills and Cognitive Characteristics

Scant research has investigated learner cognitive skills or learning style directly to recommend courses. Symeonidis and Malakoudis (2019) exploited information from external resources like learner skills and matched them with the skills covered in the course topics. Agarwal et al. (2022) proposed a MOOC recommender that extracted learning styles based on the learner’s navigation through a course. They used this information to build learner clusters and created recommendation lists based on cluster-based collaborative filtering methods.

Learner Interests and Combination of Characteristics

This category included various combinations of different information sources of learners’ characteristics to build profiles. Some work has combined learners’ interests and motivation factors, acquired implicitly or explicitly, with other learner features. Jing and Tang (2017) explored learners’ navigation through learning Web pages to find the topics they were more likely to be interested in. They combined this information with learners’ demographic information to find similar learners and group them together for collaborative course recommendations. In similar work, Yin et al. (2020) analyzed learners’ behaviors by mining their visiting history to create an interest model. They infused the interest model information with learners’ demographic data to create learner clusters and avoid the well-known cold start problem. Pardos and Jiang (2020) introduced a university course recommendation system that surveyed students about their favorite courses taken. In their research, information about students’ majors, study years, and text comments on courses were used to understand learners’ opinions and feed the recommendation system.

A number of studies examined the correlations among students’ previous enrolments, achieved scores, and their ratings of courses. Chang et al. (2016) proposed a hybrid recommendation system that examined college students’ aptitude based on their previous scores. They also used students’ ratings of course instructors as a course recommendation quality control mechanism to avoid offering courses with poor instructor ratings. Tan et al. (2020) combined learners’ explicit ratings with their grades to obtain their preferences. Xu et al. (2021) calculated learner’s ratings and scores based on previous ratings collaboratively to build a personalized recommendation system. Fan et al. (2022) proposed a multi-attention MOOC recommender that used learners’ grade records with text reviews posted after course completion. They examined word-level learner reviews and compared them with course descriptions. Man et al. (2022) argued that the recommendation systems based on students’ course selection data used limited sources of data. So, to evaluate the similarity between courses, they used students’ ratings in addition to data from enrolments and grades.

The combination of different learner characteristics and profiling techniques was a major academic focus in the years covered by our review. In their pioneering work, Elbadrawy and Karypis (2016) used information from students’ fields of study, their academic level (year of study), and previous grades to define student groups. They showed how these groupings helped to predict grades and rank courses for recommendations. Ibrahim et al. (2018) extracted and integrated students’ data from different resources, like personal information, skills, and feedback. Additionally, they used ratings to tackle the new learner problem. J. Zhang et al. (2019) measured learners’ effort by calculating their video watch time. They presented learners’ profiles by combining learners’ enrolment and grades with their watch ratio, defined as the watch duration divided by the total duration of the video. Cao and Chang (2020) proposed a hybrid course recommendation model that took into account information like students’ department, duration of the study, registration history, and certificates. In a similar work, Ahmad et al. (2023) combined learners’ demographic information and educational background with their enrolment history to build a network and explore the relations between learners and courses.

Some studies made learners’ interactions with the learning system the focal point of their modeling and combined this data with other learner information. For example, H. Zhang et al. (2019) used multi-dimensional learner attributes ranging from age and gender to micro-activities like online video watch time, video pauses, post replies, and problem views to create an accurate recommendation model for MOOCs. Jung et al. (2022) created knowledge graphs to integrate learners and courses through their interaction with the keyword sets. By calculating the number of learner interactions with the keywords and combining them with the enrolment history, they estimated learners’ interests and skill levels. With a similar approach, Yang and Cai (2022) introduced a knowledge graph enhanced CRS that used information like learners’ age, job position, industry, and knowledge level beside learner click counts on course items. Wang (2022) detailed the necessity of a complete learner model to build an accurate course recommendation system and proposed a recommendation model based on learners’ emotional and psychological factors according to the educational content. The proposed method used learners’ personal information like age, gender, profession, education, and research direction together with their opinion about the curriculum. Furthermore, Wang (2022) exploited the information from chapter test scores, grades, certificates, and course registration time combined with actions like last landing time, number of studied chapters, visit time, and participation in the forums. Table 3 summarizes only the literature that proposed learner profiling solutions based on combining the information sources they used.

Table 3

Articles That Combined Multiple Sources of Information About Learner Characteristics

Discussion

In this study, the reviewed papers were sourced from high-quality journals and conferences spanning topics from information and computer science to education and learning. The review revealed a strong emphasis on innovative course recommender system solutions that relied heavily on artificial intelligence, data mining, and big data. We observed that course recommendation systems have gained significant attention in recent years due to the increasing demand for personalized learning experiences, and as a result, we retrieved many articles for review. To include all relevant articles, this study investigated scholarly databases in the fields of computer science and education, and relied on different search methods based on keywords and multi-layer filtering to examine 976 articles. This study clearly defined exclusion and inclusion criteria. However, the initial filtering of articles based on title and abstract may have resulted in the omission of some valuable information on course features and learner modeling. Another limitation was the availability of previously published course recommender articles. Despite their relevant title and abstract, 32 papers were not accessible for full-text exploration.

It is important to note that while the field of course recommendation systems has made significant advancements, there are certain aspects that require further development and elaboration. One notable aspect is the complexity inherent in the users and items of course recommendation systems, particularly in comparison to the traditional e-commerce domain. Course recommendation systems need to go beyond simple statistical models that predict ratings and consider the multi-faceted nature of learners and course materials. In contrast to earlier literature reviews, this study sought to delve into practical course recommendation approaches that prioritized personalization while encompassing various aspects of both courses and learners. Given these prominent research directions, it was essential to examine how the proposed recommendations extracted learners’ attributes and created course feature models.

To achieve successful course completion with higher grades, it is imperative that various aspects of the course, such as required hours, schedule, assignments, final exams, and overall difficulty, align with learners’ profiles. Simply relying on learners’ previous records is insufficient to create precise predictive models of their performances. Instead, a deeper semantic and contextual analysis of courses is necessary to match them with learner profiles, interests, and future accomplishments. Surprisingly, our review found that only Xu and Zhou (2020) focused on specific course topics and their video presentation, highlighting the need for more research in this area. Additionally, only 11 out of 51 articles combined different information sources to create course features, including six studies from the year 2022. This trend indicates that the research community has only recently addressed this issue; in the future, we can expect an increase in course recommendations incorporating multi-dimensional features.

Another compelling argument can be made for introducing more elaborate learner profiles that incorporate a hybridization of different information sources. The predominant research focus thus far has been on using information from previous registrations and grades (27 articles out of 51). However, in the e-learning context, users are more specific and complex. Therefore, creating comprehensive learner profiles that consider valuable learner actions during their time in the e-learning system deserves greater attention from academia. Only three articles focused solely on learner activities, and five other articles combined this information with other learner characteristics. Still, there is not enough emphasis on the actions the learner takes in the learning process. The exploitation of different activity logs is worth investigating in order to design precise and more personalized course recommender systems.

Furthermore, our review identified only one research study that considered the combination of learner cognitive skills with other features. Moving forward, it would be beneficial to prioritize learners’ cognitive skills in order to match courses to their individual levels. By incorporating cognitive skills into the recommendation process, course recommendation systems can better cater to the unique needs and abilities of learners, potentially leading to improved learning outcomes.

Figure 5 summarizes the findings of our literature review and presents a framework for classifying course features and learner models for recommendation systems.

Figure 5

Summary of Course Feature Extraction and Learner Models for Recommendation Systems

Direction for Future Work

In this literature review, we explored the field of course recommendation systems and discussed various approaches and techniques to extract course features, model learners, and design recommenders. The review revealed several important findings and highlighted the current state of research in this area.

While the field of course recommendation systems has seen significant advancements, there are important areas that warrant further exploration and refinement. Researchers should strive to develop more sophisticated models that go beyond traditional statistical approaches to consider the complexity of learners and the specificities of course materials in the e-learning context. In this line of research, there is a massive gap in measuring the effectiveness of course recommender systems in real-world online education settings. Studies have mainly measured their proposed methods with static data and made conclusions based on statistical numbers that may not represent the usefulness of recommendation systems to help learners achieve better outcomes. Moreover, integrating comprehensive learner models and prioritizing cognitive skills offer promising directions for future studies in course recommendation systems.

One of the downsides to course recommender systems is that there is no universal dataset to assess the effectiveness and accuracy of recommender systems. The availability and use of data play a crucial role in the effectiveness of course recommendation systems. During our research, we identified several data sources used for course recommendation, such as historical student data, course content information, and social interactions. Evaluating course recommendation systems poses challenges due to the absence of standardized evaluation protocols and the subjective nature of user preferences. While metrics such as precision, recall, and accuracy are commonly used, additional measures such as diversity, novelty, and serendipity are also important to capture the quality of recommendations. Also, examining learners’ perceptions of the recommended courses needs to be more focused on future research.

Finally, the use of deep learning techniques, such as neural networks and natural language processing, holds promise for improving recommendation accuracy and incorporating more complex features. Additionally, the integration of context awareness, such as considering temporal dynamics and user preferences in real time, can lead to more personalized and adaptive recommendations. Course recommendation systems have demonstrated significant potential in enhancing the learning experience for students by providing personalized and relevant course suggestions. Future research should focus on developing robust and scalable recommendation algorithms, exploring innovative data sources, and refining course or learner feature engineering. By addressing these challenges, course recommendation systems can contribute to the advancement of personalized education and lifelong learning.

Acknowledgments

This work was supported by the Fundació per a la Universitat Oberta de Catalunya’s (UOC) grant for doctoral thesis research.

References

Abyaa, A., Khalidi Idrissi, M., & Bennani, S. (2019). Learner modelling: Systematic review of the literature from the last 5 years. Educational Technology Research and Development, 67(5), 1105-1143. https://doi.org/10.1007/s11423-018-09644-1

Agarwal, A., Mishra, D. S., & Kolekar, S. V. (2022). Knowledge-based recommendation system using semantic Web rules based on learning styles for MOOCs. Cogent Engineering, 9(1), 2022568. https://doi.org/10.1080/23311916.2021.2022568

Agrawal, D., & Deepak, G. (2022). HSIL: Hybrid semantic infused learning approach for course recommendation. In Digital Technologies and Applications: Proceedings of ICDTA ’22 (Vol. 1, pp. 417-426), Fez, Morocco. Springer. https://doi.org/10.1007/978-3-031-01942-5_42

Ahmad, H. K., Qi, C., Wu, Z., & Muhammad, B. A. (2023). ABiNE-CRS: Course recommender system in online education using attributed bipartite network embedding. Applied Intelligence, 53(4), 4665-4684. https://doi.org/10.1007/s10489-022-03758-z

Al-Badarenah, A., & Alsakran, J. (2016). An automated recommender system for course selection. International Journal of Advanced Computer Science and Applications, 7(3), 166-175. https://dx.doi.org/10.14569/IJACSA.2016.070323

Asadi, S., Jafari, S., & Shokrollahi, Z. (2019). Developing a course recommender by combining clustering and fuzzy association rules. Journal of AI and Data Mining, 7(2), 249-262. https://dx.doi.org/10.22044/jadm.2018.6260.1739

Baguley, M., Danaher, P. A., Davies, A., George-Walker, L., Jones, J. K., Matthews, K. J., Midgely, W., & Arden, C. H. (2014). Educational learning and development: Building and enhancing capacity. Springer. https://link.springer.com/book/10.1057/9781137392848

Baker, R. S., Martin, T., & Rossi, L. M. (2016). Educational data mining and learning analytics. In Andre A. Rupp and Jacqueline P. Leighton (Eds.) The Wiley handbook of cognition and assessment: Frameworks, methodologies, and applications (pp. 379-396). Wiley. https://doi.org/10.1002/9781118956588.ch16

Bakhshinategh, B., Spanakis, G., Zaiane, O., & ElAtia, S. (2017, April). A course recommender system based on graduating attributes. In International Conference on Computer Supported Education (Vol. 2, pp. 347-354). SCITEPRESS. https://doi.org/10.5220/0006318803470354

Berland, M., Baker, R. S., & Blikstein, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. Technology, Knowledge and Learning, 19(1), 205-220. https://doi.org/10.1007/s10758-014-9223-7

Bridges, C., Jared, J., Weissmann, J., Montanez-Garay, A., Spencer, J., & Brinton, C. G. (2018, March). Course recommendation as graphical analysis. In 52nd Annual Conference on Information Sciences and Systems, CISS 2018 (pp. 1-6). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/CISS.2018.8362325

Cao, P., & Chang, D. (2020). A novel course recommendation model fusing content-based recommendation and K-means clustering for wisdom education. In LISS2019: Proceedings of the 9th International Conference on Logistics, Informatics and Service Sciences (pp. 789-809). Springer. https://doi.org/10.1007/978-981-15-5682-1_57

Chang, P. C., Lin, C. H., & Chen, M. H. (2016). A hybrid course recommendation system by integrating collaborative filtering and artificial immune systems. Algorithms, 9(3), 47. https://doi.org/10.3390/a9030047

Chen, W., Ma, W., Jiang, Y., & Fan, X. (2022, July). GADN: GCN-Based attentive decay network for course recommendation. In Proceedings of Knowledge Science, Engineering and Management: 15th International Conference, KSEM 2022 (Part 1, pp. 529-541), Singapore. Springer. https://doi.org/10.1007/978-3-031-10983-6_41

Deschênes, M. (2020). Recommender systems to support learners’ agency in a learning context: A systematic review. International Journal of Educational Technology in Higher Education, 17(1), 50. https://doi.org/10.1186/s41239-020-00219-w

Elbadrawy, A., & Karypis, G. (2016, September). Domain-aware grade prediction and top-n course recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems (pp. 183-190). https://doi.org/10.1145/2959100.2959133

Esteban, A., Zafra, A., & Romero, C. (2020). Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Systems, 194, 105385. https://doi.org/10.1016/j.knosys.2019.105385

Fan, J., Jiang, Y., Liu, Y., & Zhou, Y. (2022). Interpretable MOOC recommendation: A multi-attention network for personalized learning behavior analysis. Internet Research, 32(2), 588-605. https://doi.org/10.1108/INTR-08-2020-0477

Guo, Y., Chen, Y., Xie, Y., & Ban, X. (2022). An effective student grouping and course recommendation strategy based on big data in education. Information, 13(4), 197. https://doi.org/10.3390/info13040197

Guruge, D. B., Kadel, R., & Halder, S. J. (2021). The state of the art in methodologies of course recommender systems—A review of recent research. Data, 6(2), 18. https://doi.org/10.3390/data6020018

Huang, X., Tang, Y., Qu, R., Li, C., Yuan, C., Sun, S., & Xu, B. (2018, May). Course recommendation model in academic social networks based on association rules and multi-similarity. In 22nd International Conference on Computer Supported Cooperative Work in Design (CSCWD), IEEE 2018 (pp. 277-282). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/CSCWD.2018.8465266

Ibrahim, M. E., Yang, Y., Ndzi, D. L., Yang, G., & Al-Maliki, M. (2018). Ontology-based personalized course recommendation framework. IEEE Access, 7, 5180-5199. https://doi.org/10.1109/ACCESS.2018.2889635

Jiang, W., Pardos, Z. A., & Wei, Q. (2019, March). Goal-based course recommendation. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 36-45). https://doi.org/10.1145/3303772.3303814

Jiang, X., Bai, L., Yan, X., & Wang, Y. (2022). LDA-based online intelligent courses recommendation system. Evolutionary Intelligence, 16, 1619-1625. https://doi.org/10.1007/s12065-022-00810-2

Jing, X., & Tang, J. (2017, August). Guess you like: Course recommendation in MOOCs. In Proceedings of the International Conference on Web Intelligence (pp. 783-789). https://doi.org/10.1145/3106426.3106478

Jung, H., Jang, Y., Kim, S., & Kim, H. (2022). KPCR: Knowledge graph enhanced personalized course recommendation. In Proceedings of Advances in Artificial Intelligence: 34th Australasian Joint Conference, AI 2021 (pp. 739-750), Sydney, Australia. Springer. https://doi.org/10.1007/978-3-030-97546-3_60

Khalid, A., Lundqvist, K., & Yates, A. (2020). Recommender systems for MOOCs: A systematic literature survey (January 1, 2012—July 12, 2019). International Review of Research in Open and Distributed Learning, 21(4), 255-291. https://doi.org/10.19173/irrodl.v21i4.4643

Khanal, S. S., Prasad, P. W. C., Alsadoon, A., & Maag, A. (2020). A systematic review: Machine learning based recommendation systems for e-learning. Education and Information Technologies, 25, 2635-2664. https://doi.org/10.1007/s10639-019-10063-9

Khorasani, E. S., Zhenge, Z., & Champaign, J. (2016, December). A Markov chain collaborative filtering model for course enrollment recommendations. In 2016 IEEE International Conference on Big Data (pp. 3484-3490). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/BigData.2016.7841011

Kitchenham, B. (2004). Procedures for performing systematic reviews: Joint technical report (Keele University Technical Report TR/SE-0401 and NICTA Technical Report 0400011T.1). Keele University and National ICT Australia.. https://libguides.library.arizona.edu/ld.php?content_id=49906992

Klašnja-Milićević, A., Ivanović, M., & Nanopoulos, A. (2015). Recommender systems in e-learning environments: A survey of the state-of-the-art and possible extensions. Artificial Intelligence Review, 44, 571-604. https://doi.org/10.1007/s10462-015-9440-z

Li, Q., & Kim, J. (2021). A deep learning-based course recommender system for sustainable development in education. Applied Sciences, 11(19), 8993. https://doi.org/10.3390/app11198993

Li, X., Li, X., Tang, J., Wang, T., Zhang, Y., & Chen, H. (2020). Improving deep item-based collaborative filtering with Bayesian personalized ranking for MOOC course recommendation. In Proceedings of Knowledge Science, Engineering and Management: 13th International Conference, KSEM 2020 (Part I 13, pp. 247-258), Hangzhou, China. Springer. https://doi.org/10.1007/978-3-030-55130-8_22

Ma, B., Taniguchi, Y., & Konomi, S. I. (2020). Course recommendation for university environments. In Proceedings of the 13th International Conference on Educational Data Mining, EDM 2020. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_90.pdf

Man, M., Xu, J., Sabri, I. A. A., & Li, J. (2022). Research on students’ course selection preference based on collaborative filtering algorithm. International Journal of Advanced Computer Science and Applications, 13(5). https://dx.doi.org/10.14569/IJACSA.2022.0130583

Morsy, S., & Karypis, G. (2019). Will this course increase or decrease your gpa? Towards grade-aware course recommendation. arXiv preprint arXiv:1904.11798. https://doi.org/10.48550/arXiv.1904.11798

Ng, Y. K., & Linn, J. (2017, August). CrsRecs: A personalized course recommendation system for college students. In 8th International Conference on Information, Intelligence, Systems & Applications (pp. 1-6). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/IISA.2017.8316368

Nguyen, V. A., Nguyen, H. H., Nguyen, D. L., & Le, M. D. (2021). A course recommendation model for students based on learning outcome. Education and Information Technologies, 26, 5389-5415. https://doi.org/10.1007/s10639-021-10524-0

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J.M., Hrobjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-Wilson, E., McDonald, S., McGuinness, L.A., Stewart, L.A., Thomas, J., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021, 272(71).

Pang, Y., Liu, W., Jin, Y., Peng, H., Xia, T., & Wu, Y. (2018). Adaptive recommendation for MOOC with collaborative filtering and time series. Applications in Engineering Education, 26(6), 2071-2083. https://doi.org/10.1002/cae.21995

Pardos, Z. A., Fan, Z., & Jiang, W. (2019). Connectionist recommendation in the wild: On the utility and scrutability of neural networks for personalized course guidance. User Modeling and User-Adapted Interaction, 29, 487-525. https://doi.org/10.1007/s11257-019-09218-7

Pardos, Z. A., & Jiang, W. (2020, March). Designing for serendipity in a university course recommendation system. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (pp. 350-359). https://doi.org/10.1145/3375462.3375524

Premalatha, M., Viswanathan, V., & Čepová, L. (2022). Application of semantic analysis and LSTM-GRU in developing a personalized course recommendation system. Applied Sciences, 12(21), 10792. https://doi.org/10.3390/app122110792

Reparaz, C., Aznárez-Sanado, M., & Mendoza, G. (2020). Self-regulation of learning and MOOC retention. Computers in Human Behavior, 111, 106423. https://doi.org/10.1016/j.chb.2020.106423

Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1355. https://doi.org/10.1002/widm.1355

Sakboonyarat, S., & Tantatsanawong, P. (2022). Applied big data technique and deep learning for massive open online courses (MOOCs) recommendation system. ECTI Transactions on Computer and Information Technology, 16(4), 436-447. https://doi.org/10.37936/ecti-cit.2022164.245873

Salazar, C., Aguilar, J., Monsalve-Pulido, J., & Montoya, E. (2021). Affective recommender systems in the educational field. A systematic literature review. Computer Science Review, 40, 100377. https://doi.org/10.1016/j.cosrev.2021.100377

Salehudin, N. B., Kahtan, H., Abdulgabber, M. A., & Al-bashiri, H. (2019). A proposed course recommender model based on collaborative filtering for course registration. International Journal of Advanced Computer Science and Applications, 10(11). https://dx.doi.org/10.14569/IJACSA.2019.0101122

Symeonidis, P., & Malakoudis, D. (2019). Multi-modal matrix factorization with side information for recommending massive open online courses. Expert Systems with Applications, 118, 261-271. https://doi.org/10.1016/j.eswa.2018.09.053

Tan, J., Chang, L., Liu, T., & Zhao, X. (2020, October). Attentional autoencoder for course recommendation in mooc with course relevance. In 2020 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (pp. 190-196). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/CyberC49757.2020.00038

Tarus, J. K., Niu, Z., & Mustafa, G. (2018). Knowledge-based recommendation: A review of ontology-based recommender systems for e-learning. Artificial Intelligence Review, 50, 21-48. https://doi.org/10.1007/s10462-017-9539-5

Uddin, I., Imran, A. S., Muhammad, K., Fayyaz, N., & Sajjad, M. (2021). A systematic mapping review on MOOC recommender systems. IEEE Access, 9, 118379-118405. https://doi.org/10.1109/ACCESS.2021.3101039

Urdaneta-Ponte, M. C., Méndez-Zorrilla, A., & Oleagordia-Ruiz, I. (2021). Lifelong learning courses recommendation system to improve professional skills using ontology and machine learning. Applied Sciences, 11(9), 3839. https://doi.org/10.3390/app11093839

Wang, X., Cui, L., Bangash, M., Bilal, M., Rosales, L., & Chaudhry, W. (2022). A machine learning-based course enrollment recommender system. In Proceedings of the 14th International Conference on Computer Supported Education (Vol. 1, pp. 436-443). https://doi.org/10.5220/0011109100003182

Wang, Y. (2022). Research on online learner modeling and course recommendation based on emotional factors. Scientific Programming, 2022. https://doi.org/10.1155/2022/5164186

Xia, T. (2019, August). An e-learning support middleware with MOOC course recommendation. In Proceedings of the 14th International Conference on Computer Science & Education (pp. 596-600). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCSE.2019.8845533

Xu, G., Jia, G., Shi, L., & Zhang, Z. (2021). Personalized course recommendation system fusing with knowledge graph and collaborative filtering. Computational Intelligence and Neuroscience, 2021, 1-8. https://doi.org/10.1155/2021/9590502

Xu, W., & Zhou, Y. (2020). Course video recommendation with multimodal information in online learning platforms: A deep learning framework. British Journal of Educational Technology, 51(5), 1734-1747. https://doi.org/10.1111/bjet.12951

Yago, H., Clemente, J., & Rodriguez, D. (2018). Competence-based recommender systems: A systematic literature review. Behaviour & Information Technology, 37(10-11), 958-977. https://doi.org/10.1080/0144929X.2018.1496276

Yang, Q., Yuan, P., & Zhu, X. (2018). Research of personalized course recommended algorithm based on the hybrid recommendation. In MATEC Web of Conferences (Vol. 173, p. 03067). EDP Sciences. https://doi.org/10.1051/matecconf/201817303067

Yang, S., & Cai, X. (2022). Bilateral knowledge graph enhanced online course recommendation. Information Systems, 107, 102000. https://doi.org/10.1016/j.is.2022.102000

Yang, X., & Jiang, W. (2019). Dynamic online course recommendation based on course network and user network. In Proceedings of Smart City and Informatization: 7th International Conference (Vol. 7, pp. 180-196), Guangzhou, China. Springer. https://doi.org/10.1007/978-981-15-1301-5_15

Yanhui, D., Dequan, W., Yongxin, Z., & Lin, L. (2015, November). A group recommender system for online course study. In 7th International Conference on Information Technology in Medicine and Education (pp. 318-320). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ITME.2015.99

Yin, S., Yang, K., & Wang, H. (2020, May). A MOOC courses recommendation system based on learning behaviours. In Proceedings of the ACM Turing Celebration Conference: China (pp. 133-137). https://doi.org/10.1145/3393527.3393550

Zhang, H., Huang, T., Lv, Z., Liu, S., & Yang, H. (2019). MOOCRC: A highly accurate resource recommendation model for use in MOOC environments. Mobile Networks and Applications, 24, 34-46. https://doi.org/10.1007/s11036-018-1131-y

Zhang, J., Hao, B., Chen, B., Li, C., Chen, H., & Sun, J. (2019, July). Hierarchical reinforcement learning for course recommendation in MOOCs. In Proceedings of the AAAI conference on Artificial Intelligence (Vol. 33, No. 01, pp. 435-442). https://doi.org/10.1609/aaai.v33i01.3301435

Zhao, Z., Yang, Y., Li, C., & Nie, L. (2020). GuessUNeed: Recommending courses via neural attention network and course prerequisite relation embeddings. ACM Transactions on Multimedia Computing, Communications, and Applications, 16(4), 1-17. https://doi.org/10.1145/3410441

Zhou, J., Jiang, G., Du, W., & Han, C. (2022). Profiling temporal learning interests with time-aware transformers and knowledge graph for online course recommendation. Electronic Commerce Research, 23, 1-21. https://doi.org/10.1007/s10660-022-09541-z

Zhu, Y., Lu, H., Qiu, P., Shi, K., Chambua, J., & Niu, Z. (2020). Heterogeneous teaching evaluation network based offline course recommendation with graph learning and tensor factorization. Neurocomputing, 415, 84-95. https://doi.org/10.1016/j.neucom.2020.07.064

Extracting Course Features and Learner Profiling for Course Recommendation Systems: A Comprehensive Literature Review by Amir Narimani and Elena Barberà is licensed under a Creative Commons Attribution 4.0 International License.