International Review of Research in Open and Distributed Learning

Volume 20, Number 3

July - 2019

Patterns of Students' Utilization of Flexibility in Online Academic Courses and Their Relation to Course Achievement

Tal Soffer, Tali Kahan, and Rafi Nachmias
Tel Aviv University, Israel

Abstract

Online academic courses provide students with flexible learning opportunities by allowing them to make choices regarding diverse aspects of their learning process; hence, such courses support personalized learning. This study aimed to analyze the ways students make use of flexibility in online academic courses based on learning time, place, and access to learning resources, as well as to investigate how this relates to differences in course achievement. The study examined 587 students in four online courses. Educational data mining (EDM) methodology was used to trace students' behavior in the courses and to compute 34 variables, which describe their use of flexibility. The results show that students developed different patterns of learning time, place, and access to content, which indicates that flexibility was used substantially. Students' achievements were significantly related to patterns of learning time and access to learning resources. Understanding the different patterns of flexibility usage may support the design of personalized learning and increase collaboration among students with similar characteristics.

Keywords: Flexible learning, personalized learning, online academic courses, educational data mining, course achievement, learning behavior

Introduction

Online courses provide flexible learning opportunities to learners. Flexible learning is based on the recognition of differences among students, which are addressed by providing varying degrees of choice to learners regarding what, where, when, why, and how to learn (Bergamin, Ziska, Werlen, & Siegenthaler, 2012; Collis & Moonen, 2002; Hill, 2006). Hence, flexible learning supports personalized learning, wherein learners' needs, interests, backgrounds, and learning styles are central. It reflects a shift from teacher-centered pedagogies and practices towards more innovative, student-centered approaches (Wanner & Palmer, 2015). Studies have indicated that flexibility is perceived as beneficial to online instruction (Gillingham & Molinari, 2012; Soffer, Kahan, & Livne, 2017) and constitutes a key factor in students' enrollment in online courses (Jaggars, 2014).

The provision of flexible learning opportunities, however, does not necessarily lead to effective learning, as simply providing a range of options to learners does not bring with it deep learning (Goodyear, 2008; Willems, 2005). With the freedom comes responsibility, which requires real commitment and discipline (Hill, 2006). Hence, flexible learning requires that students take on more control of the learning, make their own decisions and invest a greater level of dedication (Grant & Hill, 2006; You, 2016). Studies indicated that students may need ongoing support in this process (Willems, 2005). Hence, it is essential to study how students make use of flexibility and how it relates to their achievements, in order to understand the best way to integrate it and support students' learning.

Educational data mining (EDM) is an evolving research methodology that enables the trace of students' behavior in online environments. Online learning environments, including learning management systems (LMS), automatically record students' learning activity on servers (Soffer, Kahan, & Livne, 2017; You, 2016). Among other things, analyzing this data using advanced techniques enables the discovery of meaningful patterns, the adjustment of instructional strategies, and the ability to make data-driven decisions rather than decisions based solely on students' self-reports (Dietz-Uhler & Hurn, 2013; Gašević, Dawson, & Siemens, 2015). Notably, few studies have used LMS data to analyze students' use of flexibility components in online courses and compare the results with course achievement. For example, Cornelius and Gordon (2008) reported on three different strategies for student engagement with flexible online resources, based on interviews with participants and examination of course statistics.

Consequently, the aim of this study was to use EDM to analyze patterns of undergraduate students' use of flexibility in four asynchronous online academic courses, and examine how they are related to differences in course achievement. The data mining methodology made it possible to trace students' actions in the courses and examine their use of flexibility in practice. The study examined the use of three flexibility dimensions: learning time, learning place, and access to learning resources. In addition, the study examined whether or not there are differences in course achievement among students who use flexibility differently. This study contributes to our understanding of the various patterns of flexibility usage by students, which may support better integration of flexibility into online courses, better design of personalized learning, and increased collaboration among students with similar characteristics.

Background

Flexible Learning

Flexible learning is not a new concept and has been a core issue in distance education for some time (Bates, 2001; Hill, 2006). A central element of flexible learning is the provision of choice to learners. Instead of the instructor or the institution making key decisions about learning dimensions, the learner has a range of options from which to choose (Collis & Moonen, 2002; Jochems, Van Merriënboer, & Koper, 2004). Thus, flexible learning involves loosening logistical and educational constraints, and is often related to student-centeredness as well as to individualization in teaching and personalization of the learning process (Collis & Moonen, 2001).

It is a challenge to define flexible learning, due to its manifold characteristics. As a result, diverse concepts have been developed around it (Bergamin et al., 2012; Collis & Moonen, 2002), which vary, in part, in terms of the flexibility dimensions to which they refer. Collis, Vingerhoets, and Moonen (1997), for example, referred to five sets of dimensions in which flexibility may be provided: (a) time, (b) content, (c) entry requirements, (d) instructional approach and resources, and (e) delivery and logistics. Boer and Collis (2005) distinguished between planning-type flexibility, which the instructor can designate before the course begins and which needs to be managed when the course is offered, and interpersonal flexibility, which relates more to the dynamics of the course as it is experienced by the learners. Hill (2006) distinguished between flexible delivery, which focuses on options regarding access for learners, and flexible learning, which focuses on options related to how learning occurs. Bergamin et al. (2012) identified seven categories of flexibility: (a) time, (b) space, (c) methods, (d) learning styles, (e) content, (f) organization and infrastructure, and (g) requirements.

Flexible Learning in Online Courses

Over the years, as distance education entered the online space, flexible and personalized learning became linked with the use of new information and communications technology (ICT), which give students greater diversity in their learning (Gedera, Williams, & Wright, 2015; Wanner & Palmer, 2015). Indeed, research affirms that online learning environments enable students to experience flexibility by allowing them to decide when, where, and what they can learn (Gedera et al., 2015; Hung, Chou, & Chen, 2010). Thus, among others, online learning environments support flexibility of time, place, and access to learning resources. The current study focuses on these three dimensions.

Flexibility of time relates to diverse time-related aspects, including: (a) when students can start and end a course, (b) times for interacting within the course, (c) frequency and pace of learning, (d) duration of learning, as well as (e) timing the moments of assessment (Boer & Collis, 2005). Online courses can provide learners with access to course content anytime. Harasim (2000) claimed that flexibility of time, which allows access to course content around the clock, enables students' participation all week, thus supporting continual knowledge building and allowing students to participate at their best learning readiness time. However, the advantages of taking a flexible course (e.g., doing the course work whenever it is convenient) might become disadvantages (e.g., there is never a convenient time to do the course work) (Hill, 2006).

Flexibility of place refers to where the learning occurs (Collis & Moonen, 2002; Hill, 2006). Flexibility in this regard means that learning can take place at locations that suit the learner (Goodyear, 2008). Online courses support flexibility in learning place by allowing access to course materials from anywhere (Glance, Forsey, & Riley, 2013). According to Goodyear (2008), flexibility in learning place can minimize disruptions and dissolve unhelpful boundaries between a learner's various activities (e.g., at work or at home) and learning.

Online learning also permits flexibility regarding learning resources (Hung et al., 2010). This includes flexibility in: (a) the modality and origin of study materials, (b) amount of content, (c) sequence of different parts of a course and study routes, (d) assessment standards, and (e) completion requirements (Collis & Moonen, 2002; Cornelius, Gordon, & Ackland, 2011; Hung et al., 2010). Unlike traditional learning environments, which typically require students to follow a linear sequence, online environments allow learners to follow a more individualized approach: viewing the instructional material in a sequence that best meets their needs, repeating or skipping sections, and following subjects regardless of the order in which they have been arranged (Hung et al., 2010). Moreover, audio and video formats enable students to control the pace at which they consume and review content (Neville et al., 2015). Flexibility regarding content may range from being completely open, where the learner is making all the choices, to providing options within a particular framework established by the instructor (Hill, 2006).

Previous studies have established that flexibility constitutes a key factor in students' enrollment in online courses (Gillingham & Molinari, 2012; Hung et al., 2010; Jaggars, 2014; Northrup, 2002). Students choose online courses for many reasons, including the opportunity to better balance their busy life schedule, which is full of responsibilities such as childcare and work (Gillingham & Molinari, 2012; Jaggars, 2014). Some students believe that online courses allow them to use their learning time outside of class more efficiently than in-class time. Others choose online courses in order to reduce the number of times they have to travel to campus. Jaggars (2014) reported that students appreciate the comfort of working at home (e.g., taking breaks, having snacks), and some feel that instructional materials in a flexible format offer them more control while still supporting course outcomes (Neville et al., 2015).

Nevertheless, flexible learning environments require students to exercise control over different aspects of their learning activities, such as time-management, type of media accessed, pace, and depth (Hung et al., 2010). Hence, flexible learning leads to a learning situation where students need to be organized, to set their own objectives and plan, and to regulate and evaluate the learning process themselves (Narciss, Proske, & Koerndle, 2007; You, 2016). Cornelius and Gordon (2008) expressed the need for further research to investigate what learning strategies learners adopt in a flexible online environment.

Several studies have addressed the relation between flexible learning and learning achievements. Some indicated a positive relation; Marton, Hounsell and Entwistle (1997) found that a wide range of learning options is pedagogically desirable and improves achievement. Brown and Smith (2013) argue that flexible learning, which adopts a student-centered approach, is more effective for improving learning achievement. Others found that the effect of greater student autonomy over the learning environment on academic achievement is ambiguous, with greater autonomy likely to improve academic achievement for some students but not others (Guest, 2005).

The aim of this study was to analyze the ways students use flexibility in online academic courses in practice. Following the literature review, for this study we defined flexibility in three dimensions: (a) learning time (i.e., during the semester, the week, and the day); (b) learning place (i.e., on campus and off campus); and (c) access to learning resources (i.e., re-access and sequence of access). In addition, the study examined whether there are differences in course achievement among students who make use of flexibility differently.

The Study

Research Questions

This study was framed by the following four research questions:

1. What are the patterns of students' use of learning time flexibility in online academic courses?
2. What are the patterns of students' use of learning place flexibility in online academic courses?
3. What are the patterns of students' use of flexibility of access to learning resources in online academic courses?
4. Are there significant differences in course achievement among students who exhibit different patterns of flexibility usage?

Research Population and Field

The study examined the behaviors of 587 undergraduate students who participated in one of four asynchronous online courses in the academic year 2015/2016. These courses are part of an academic program which is mandatory for all undergraduate students at the university. The program offers students the opportunity to study courses in fields unrelated to their major degree program. Thus, the courses are characterized by a wide range of participants from various disciplines.

The online courses that were examined in this study are from the faculties of humanities and arts. The one-semester (13-week) courses are two credit hours each and are taught via the Moodle learning management system (LMS). As part of the development of these courses, a consistent instructional course model was created and implemented, such that each course consisted of between 11 to 14 learning units, comprising the core of the online course. The learning units were built in a linear order, so that each unit is based on the previous one. Each learning unit covered a different topic and consisted of: (a) video lecture by the instructor; (b) text summary of the lecture; (c) course materials associated with the learning unit (e.g., textbook, presentations, articles, YouTube links, course dictionary); and (d) assignments, as some units were accompanied by an online, required assignment for students to practice certain topics and receive feedback from the instructors. The students were obliged to pass a final exam on campus. The final course grade was comprised of the assignment grades and the exam grade. Although the course design and structure were consistent, there were some differences among the courses, as presented in Table 1.

Table 1

The Structure of the Online Courses

During course development, explicit choices were made as to which flexibility dimensions would be provided to students. It was decided that all of the course content would be uploaded and opened for the students from the beginning of the course. Thus, flexibility was provided in terms of: (a) learning time (i.e., students could span their learning during the semester, the week, and the day, according to their preferences and needs); (b) learning place (i.e., students could learn on campus or off campus, according to their convenience); and (c) access to learning resources (i.e., all learning resources, including assignments, were available from the beginning of the course and students could re-access them, choose the sequence of access, and submit the assignments as they desired). It should be noted, however, that in order to provide a learning framework for students, assignment submission was limited by deadlines. Thus, flexibility in time was limited to some extent. Furthermore, the courses were structured with fixed start and end dates, finishing with a final exam on campus.

Method and Procedures

The data was collected automatically by the LMS and included a log file with over 204,000 records of students' activities in the course websites, as well as their grades. Each record documented a student's action in the course (e.g., entering a learning unit, playing a video lecture, accessing a course material) and contained information regarding the user id (anonymized), user IP address, action time, and action type. For ethical considerations, the data was anonymous and the students were informed upon their first entry to the LMS that the university might collect data regarding their online activity for research purposes.

Using SQL (Structured Query Language) queries, two different kinds of variables were computed. The first was from the student perspective—21 variables were computed per student, describing his/her activity in the course website in regard to the three flexibility dimensions examined, as well as the final course grade. The second was from the learning unit perspective—13 variables were computed per learning unit, describing the portion of students who accessed it every week, in order to examine the sequence of student accesses to the learning resources. Table 2 summarizes all the study variables.

Table 2

The Variables Computed

^a Percentage of student's actions in the course website was calculated as the number of student's actions in the course website during the time period examined, divided by the total number of the student's actions in the course website during the course.
^b Percentage of student's activity in the course website was calculated as the number of times in which the student was active in the course website on the day examined, divided by the total number of days in which the student was active in the course website.
^c Percentage of student's actions on and off campus was calculated based on the IP address from which the action was executed. Actions that were executed from IP addresses that belong to the range of the university's IP addresses (for wired or wireless communication) were considered on campus, whereas the remainder were considered off campus.

Next, in order to characterize the ways that students used flexibility, two approaches were used. From the student perspective, the variables were analyzed using two-step cluster analyses. Cluster analysis is an exploratory data mining approach for discovering the structure in data without an a priori idea of what should be found. This analysis enables the discovery of data points that naturally group together, splitting the data set into a set of clusters (Baker & Siemens, 2014). Hence, cluster analyses were used in this study to identify groups of students who exhibited similar patterns of using the flexibility aspect under investigation. Specifically, four cluster analyses were conducted, to identify patterns of using the flexibility of: (a) learning time during the semester, (b) learning time during the week, (c) learning time during the day, and (d) re-accessing learning resources. It should be noted that using flexibility of learning place was examined using descriptive statistics, because it was measured by two complementary variables.

Each cluster analysis was performed several times to explore and evaluate different numbers of clusters, in order to find the one that revealed the most insightful patterns and achieved a good silhouette coefficient score. The silhouette coefficient score, a measure of the cohesion within a cluster and separation between the clusters, was used to quantify the goodness of the clustering. The coefficient ranges from -1 to +1, such that a score is considered good if it is greater than 0.5, fair if it is between 0.2 and 0.5, and poor if it is less than 0.2 (Kaufman & Rousseeuw, 2009; Norusis, 2012). In addition, ANOVA tests were applied to determine if the differences between the clusters were significant. Significant differences were then assessed by Tukey post-hoc analyses. Since the research examined four different online courses, chi-squared tests were applied to determine if there was a significant dependence between the courses and the clusters.

From the learning unit perspective, heat map visualizations were used in order to examine the students' use of flexibility of sequence of access to the learning resources. The heat map displays the portion of students who accessed each learning unit every week, in order to analyze the ways students chose to access the course content, when flexibility of sequence was provided. Since the study examined four different courses, which consisted of different numbers of learning units, a separate heat map was created for each course.

Results

Patterns of Students' Use of Flexibility of Learning Time

Regarding learning time during the semester, examination of students' activity in the course websites revealed that they tended to span their activity quite evenly over the semester quarters. The average student activity in each quarter of the semester ranged from 21% to 26% of his/her total activity in the course website. However, the standard deviations ranged from 13% to 20% per quarter, which indicates that there was variance among the students in this regard.

Using cluster analysis, five patterns of behavior were identified: (a) 36.6% (n=215) of the students spanned their activity over all quarters quite evenly, on average; (b) 19.4% (n=114) of the students performed a major part of the their activity, 48% on average, during the third quarter; (c) 15.5% (n=91) of the students performed 42% of their activity, on average, during the second quarter; (d) 15.3% (n=90) of the students performed 60% of their activity, on average, during the fourth quarter; and (e) 13.1% (n=77) of the students performed 51% of their activity, on average, during the first quarter. The silhouette measure of the model was 0.4. ANOVA tests showed statistically significant differences among the clusters in regard to all the clustering variables (p<0.001), and in regard to course achievement (p<0.01). Specifically, significant differences were found in the course grade between clusters 3 and 5 (p=0.002). A chi-squared test found significant dependence among the course subject and the clusters χ²(12)=211.74, p<0.001. Table 3 presents the cluster sizes, the average and standard deviation of the variables in each cluster, and the results of the ANOVA tests.

Table 3

Clusters by Learning Time During the Semester

** p<0.01, *** p<0.001

Figure 1 provides a graphical presentation of the clusters and illustrates the different ways in which flexibility of learning time during the semester was used.

Figure 1. Patterns of use of the flexibility of learning time during the semester.

Regarding learning time during the week, analysis of the students' activity in the course websites revealed that they tended to be active to a similar extent on each day of the week. On average, a student was active on a specific day between 12% to 16% of his/her total days of activity in the course, with standard deviations ranging from 8% to 10%, per day. A cluster analysis did not yield any insightful results in identifying patterns of behavior during the week. Hence, Figure 2 displays the average of students' activity on each day of the week and the standard deviation.

Figure 2. Average of students' activity on each day of the week and the standard deviation.

Regarding learning time during the day, analysis of the students' activity in the course websites revealed that students tended to be more active during the noon and evening hours than during mornings and nights. On average, a student performed 41% of his/her total activity in the course website during noon hours, 36% of his/her activity during evening hours, 18% during morning hours, and only 4% during night hours. The standard deviations ranged from 9% to 18% per time of day.

Using a cluster analysis, four patterns of behaviors were identified: (a) 44.8% (n=263) of the students performed a major part of their activity, 51% on average, during evening hours; (b) 31.3%, (n=184) of the students performed a major part of their activity, 61% on average, during noon hours; (c) 14.7%, (n=86) of the students performed 45% of their activity, on average, during morning hours; and (d) 9.2% of the students (n=54) performed 27% of their activity on average during night time hours. The silhouette measure of the model was 0.5. ANOVA tests showed statistically significant differences among the clusters in regard to all clustering variables (p<0.001) but not in regard to course achievement. A chi-squared test found no significant dependence among the courses and the clusters. Table 4 presents the cluster sizes, the average and standard deviation of the variables in each cluster, and the results of the ANOVA tests.

Table 4

Clusters by Learning Time During the Day

*** p<0.001

Figure 3 provides a graphical presentation of the clusters and illustrates the different ways in which flexibility of learning time during the day was used.

Figure 3. Patterns of use of the flexibility of learning time during the day.

Patterns of Students' Use of Flexibility of Learning Place

Regarding learning place, analysis of the students' activity on and off campus revealed that the students tended to be more active in the course off campus than on campus. On average, 85% of a student's activity in the course website took place off campus. The standard deviation was 18% and the median was 91%.

Patterns of Students' Use of Flexibility of Access to Learning Resources

Regarding number of accesses to the learning resources, analysis revealed that the students tended to re-access a learning resource more than once. On average, a learning unit was accessed 4.01 times, a course material was accessed 1.94 times, and a video lecture was accessed 1.85 times. The standard deviations ranged from 1 to 3.

Using a cluster analysis, three patterns of behaviors were identified: (a) most students, 70.2%, (n=412) tended to re-access all learning resources a few times (between 1.49 to 3.29 times, on average); (b) 18.6% (n=109) of the students tended to re-access the course materials more than the other students (3.56 times per item, on average); and (c) 11.2% (n=66) of the students tended to re-access the learning units and the video lectures more than the other students (7.97 times per learning unit and 4.20 times per video lecture, on average). The silhouette measure of the model was 0.6. ANOVA tests showed statistically significant differences among the clusters in regard to the clustering variables and the course achievement variable (p<0.001). Specifically, significant differences were found in the course grade between clusters 1 and 2, and between clusters 1 and 3 (p=0.036, p<0.001, respectively). A chi-squared test found significant dependence between the course subject and the clusters, χ²(6)=133.97, p<0.001. Table 5 presents the cluster sizes, the average and standard deviation of the variables in each cluster, and the results of the ANOVA tests.

Table 5

Clusters by Number of Re-Accesses to the Learning Resources

*** p<0.001

Figure 4 provides a graphical presentation of the clusters and illustrates the different ways in which flexibility of re-accesses to learning resources was used.

Figure 4. Patterns of use of the flexibility of re-accesses to learning resources.

Analysis of the order in which the students accessed the learning units is presented in Figure 5. The heat maps display, per course, the portion of students who accessed each learning unit during each week. The dashed lines symbolize assignment deadlines. Several trends emerged from the maps. It is notable (mostly in courses 1, 2, and 3) that, in general, most students followed the learning units linearly during the semester, entering a learning unit (or its adjacent units) on the corresponding week of the semester (e.g., entering learning unit 1 on week 1, learning unit 2 on week 2). However, in all courses there were smaller groups of students who entered other learning units as well. This is especially notable in course 1, where during each week students entered more advanced or preceding learning units, and in courses 2 to 4, where students entered mostly preceding learning units. Notably (mostly in courses 1, 2, and 3) more students entered the learning units during weeks with assignment deadlines. Finally, during the last two weeks of the semester, there was an increase in the number of students who entered all the learning units.

Figure 5. Use of the flexibility of order of access to the learning units.

Discussion and Conclusions

This study examined how students used the flexibility of time, place, and access to learning resources in four online academic courses, using educational data mining methodology. EDM reveals meaningful patterns of students' behavior based on their actual activity in the LMS. The results demonstrated that flexibility was used significantly, and provides evidence for the variety of ways students used it.

In regard to flexibility of learning time during the semester, various students made use of the flexibility dimension differently. Five clusters were found: the largest cluster (37%) is characterized by students who spanned their activity quite evenly over the semester. The other four clusters are characterized by students who invested more intense activity in one quarter of the semester than in the others. The differences in learning time during the semester among students are in line with previous studies, which indicated that one of the main goals in providing flexibility of time is to help people integrate learning into their often rather complex lives (Hill, 2006). Indeed, it was found that learning occurred at multiple times. However, some differences in the learning time could be related to the course structure, such as different number of assignments and deadlines during the semester.

As for the learning place and time during the week, it was found that most of the students (92%) learned mostly off campus, and tended to span their activity quite evenly over the days of the week, without selecting a fixed day for learning. This may imply that the students divided their learning workload according to their timetables and needs. Regarding learning time during the day, as found in previous studies (Harasim, 2000), students made use of the opportunity to learn at all hours of the day. Four clusters were found: the largest cluster is characterized by students who tended to learn during the evenings (45%), followed by students who tended to learn during noon (31%). Only 15% of the students were significantly active in the mornings, and an even smaller group of students (9%) studied during the night. This may be related to students' other commitments, such as work.

In reference to flexibility of access to learning resources, while traditional learning environments typically require students to follow a linear sequence, online environments permit more flexibility by allowing learners to choose the sequence of learning as well as the amount of content they make use of (Hannafin, 1984; Reeves, 1993). Regarding the number of accesses to the learning resources, it was found that the students tended to use the possibility to re-access the learning resources. Three clusters were found: the majority of the students (70%) used the learning resources to a very limited extent, they re-accessed a learning resource between 1 to 3 times on average, whereas 19% of the students used it to a greater extent, specifically they accessed the learning units and the course materials between 3 to 4 times on average. The smallest cluster (11%) were students who used the learning resources the most. Specifically, they re-accessed the learning units and the video lectures between 4 to 8 times on average. The most re-accessed learning resources were the learning units, followed by the course materials and the video lectures. However, there were differences among the courses in this regard, which may be related to the characteristics of the course content, such as the level of difficulty of the learning resources and their importance to the course. For example, in one course, the course materials consisted of a digital textbook, which accompanied all learning units and constituted an integral part of the course. Some courses contained a dictionary, which was essential throughout the course, and so on.

As for the sequence of access to the learning resources, although all learning units were available from the beginning of the course, the vast majority of students followed through them linearly as the semester progressed. The linear trend is not completely surprising given the structure of the courses. Nevertheless, smaller groups of students accessed all the learning units all through the semester, which indicates that they may have used the possibility to access contents according to their preferred order and pace. In general, this pattern emerges in all four courses, but differently. Possible reasons may include different structures of course content, which in some courses may not have required linear sequencing of learning. Interestingly, there was higher students' engagement in courses with more assignments (Course 1 compared to Course 4). In all of the courses, engagement increased during weeks with assignments and towards the date for the final exam. These results emphasize that integrating scaffolding, such as assignment deadlines, can increase students' engagement.

Exploring the relation between use of flexibility and course achievement revealed significant differences among students who used the flexibility of learning time during the semester and access to learning resources differently. Interestingly, students who spanned their activity evenly over the semester quarters scored higher on average than those who tended to centralize part of their activity in one quarter. Notably, students who were more active towards the date for the final exam achieved higher grades. Moreover, students who tended to re-access the course content also achieved higher grades in the course. These findings are in line with previous research, which found a significant relationship between active participation in online courses and academic performance, and which indicated that success in online courses requires regular study during the course (You, 2016). It should be noted that course achievement was not found to be related to learning hours or places.

To summarize, this study found that most students adapted different patterns of flexibility use in learning time, place, and access to learning resources. This emphasizes that integrating flexibility in online courses enables students to learn according to their needs. Moreover, the results of this study indicate that the ways students use flexibility are related to their achievement. Specifically, the way students span their learning over the semester, as well as their level of engagement with the course content, are both related to achievement. This study contributes to the body of knowledge about flexible learning by analyzing students' behavior using EDM, and providing evidence for the ways in which students use flexibility. Practically, it may support course designers in developing flexible online courses according to students' preferences, while taking achievement into account. This could be done by providing suitable scaffolding for increasing students' engagement with the learning resources (such as assignments) along the semester and towards the end of the course. Furthermore, this analysis can increase collaboration among students with similar characteristics (e.g., same learning time, course content).

Limitations and Implications for Future Research

Several possible limitations should be noted. This study examined four online courses with a dedicated instructional course model. Future research should examine a larger sample of courses from diverse disciplines and a more heterogeneous population. Moreover, since a dependency was found among the courses and the clusters, further research to take a closer look into each course would be interesting. In addition, despite the differences that were found among different patterns of students' use of flexibility and course achievement, conclusions regarding causality should not be derived. Hence, further research should include a wider range of variables which could affect the use of flexibility, such as demographic factors. In addition, future research to examine the use of other flexibility dimensions, such as the duration of learning, may be worthwhile. Finally, the study relied solely on a data mining approach, which focused mainly on students' activities in the online environment. This approach disregards offline activities as well as students' perceptions and motivations. Thus, further research could use other reasearch methodologies (e.g., interviews with students, surveys) in order to shed light on the different patterns of flexibility usage that were found and obtain a more holistic understanding.

References

Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (2^nd ed., pp. 253-274). New York, NY: Cambridge University Press.

Bates, T. (2001). National strategies for e-learning in post-secondary education and training, In Fundamentals of educational planning, 70 (pp. 1-134). Paris: UNESCO.

Bergamin, P. B., Ziska, S., Werlen, E., & Siegenthaler, E. (2012). The relationship between flexible and self-regulated learning in open and distance universities. The International Review of Research in Open and Distributed Learning, 13(2), 101-123. doi: 10.19173/irrodl.v13i2.1124

Boer, W. D., & Collis, B. (2005). Becoming more systematic about flexible learning: Beyond time and distance. ALT-J, 13(1), 33-48. doi: 10.1080/0968776042000339781

Brown, S., & Smith, B. (2013). Research, teaching and learning in higher education. London: Routledge. doi: 10.4324/9781315041568

Collis, B., & Moonen, J. (2001). Flexible learning in a digital world: Experiences and expectations. London: Routledge. doi: 10.4324/9780203046098

Collis, B., & Moonen, J. (2002). Flexible learning in a digital world. Open Learning, 17(3), 217-230. doi: 10.1080/0268051022000048228

Collis, B., Vingerhoets, J., & Moonen, J. (1997). Flexibility as a key construct in European training: Experiences from the TeleScopia Project. British Journal of Educational Technology, 28(3), 199-217. doi: 10.1111/1467-8535.00026

Cornelius, S., & Gordon, C. (2008). Universalists, butterflies and changelings: Learners' roles and strategies for using flexible online resources. Proceedings of EdMedia: World Conference on Educational Media and Technology, 2008 (Vol. 1), 4052-4057.

Cornelius, S., Gordon, C., & Ackland, A. (2011). Towards flexible learning for adult learners in professional contexts: An activity-focused course design. Interactive Learning Environments, 19(4), 381-393. doi: 10.1080/10494820903298258

Dietz-Uhler, B., & Hurn, J. E. (2013). Using learning analytics to predict (and improve) student success: A faculty perspective. Journal of Interactive Online Learning, 12(1), 17-26.

Gašević, D., Dawson, S., & Siemens, G. (2015). Let's not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. doi: 10.1007/s11528-014-0822-x

Gedera, D., Williams, J., & Wright, N. (2015). Identifying factors influencing students' motivation and engagement in online courses. In C. Koh (Ed.). Motivation, leadership and curriculum design (pp. 13-23). Singapore: Springer. doi: 10.1007/978-981-287-230-2_2

Gillingham, M., & Molinari, C. (2012). Online courses: Student preferences survey. Internet Learning, 1(1), 36-45.

Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical foundations of massive open online courses. First Monday, 18(5). doi: 10.5210/fm.v18i5.4350

Goodyear, P. (2008). Flexible learning and the architecture of learning places. In M. Spector, D. Merrill, J. vanMerrienboer, & M. Driscoll (Eds.), Handbook of research on educational communications and technology (3^rd ed., pp. 251-257). Taylor & Francis e-Library.

Grant, M. M., & Hill, J. R. (2006). Weighing the risks with the rewards: Implementing student-centered pedagogy within high-stakes testing. In R. Lambert & C. McCarthy (Eds.), Understanding teacher stress in an age of accountability (pp. 19-42). Greenwich, CT: Information Age.

Guest, R. (2005). Will flexible learning raise student achievement? Education Economics, 13(3), 287-297. doi: 10.1080/09645290500073761

Hannafin, M. J. (1984). Guidelines for using locus of instructional control in the design of computer-assisted instruction. Journal of Instructional Development, 7(3), 6-10. doi: 10.1007/BF02905753

Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. The Internet and Higher Education, 3(1), 41-61. doi: 10.1016/S1096-7516(00)00032-4

Hill, J. R. (2006). Flexible learning environments: Leveraging the affordances of flexible delivery and flexible learning. Innovative Higher Education, 31(3), 187-197. doi: 10.1007/s10755-006-9016-6

Hung, M. L., Chou, C., & Chen, C. H. (2010). Learner readiness for online learning: Scale development and student perceptions. Computers & Education, 55(3), 1080-1090. doi: 10.1016/j.compedu.2010.05.004

Jaggars, S. S. (2014). Choosing between online and face-to-face courses: Community college student voices. American Journal of Distance Education, 28(1), 27-38. doi: 10.1080/08923647.2014.867697

Jochems, W., Van Merriënboer, J. J. G., & Koper, R. (2004). Integrated e-learning: Implications for pedagogy, technology and organization. London: Routledge.

Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: An introduction to cluster analysis, 344. NJ: John Wiley & Sons.

Marton, F., Hounsell, D., & Entwistle, N. J. (Eds.). (1997). The experience of learning: Implications for teaching and studying in higher education. Edinburgh: Scottish Academic Press.

Narciss, S., Proske, A., & Koerndle, H. (2007). Promoting self-regulated learning in Web-based learning environments. Computers in Human Behavior, 23(3), 1126-1144. doi: 10.1016/j.chb.2006.10.006

Neville, M. W., Palmer, R., Elder, D., Fulford, M., Morris, S., & Sappington, K. (2015). Evaluating the effects of flexible learning about aseptic compounding on first-year students in a pharmacy skills laboratory. American Journal of Pharmaceutical Education, 79(6), 91. doi: 10.5688/ajpe79691

Northrup, P. T. (2002). Online learners' preferences for interaction. Quarterly Review of Distance Education, 3(2), 219-26.

Norusis, M. J. (2012). IBM SPSS statistics 19 statistical procedures companion. Upper Saddle River, NJ: Prentice Hall.

Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39-46.

Soffer, T., Kahan, T., & Livne, E. (2017). E-assessment of online academic courses via students' activities and perceptions. Studies in Educational Evaluation, 54, 83-93. doi: 10.1016/j.stueduc.2016.10.001

Wanner, T., & Palmer, E. (2015). Personalising learning: Exploring student and teacher perceptions about flexible learning and assessment in a flipped university course. Computers & Education, 88, 354-369. doi: 10.1016/j.compedu.2015.07.008

Willems, J. (2005). Flexible learning: Implications of “when-ever,” “where-ever” and “what-ever.” Distance Education, 26(3), 429-435. doi: 10.1080/01587910500291579

You, J. W. (2016). Identifying significant indicators using LMS data to predict course achievement in online learning. The Internet and Higher Education, 29, 23-30. doi: 10.1016/j.iheduc.2015.11.003

Patterns of Students' Utilization of Flexibility in Online Academic Courses and Their Relation to Course Achievement by Tal Soffer, Tali Kahan, and Rafi Nachmias is licensed under a Creative Commons Attribution 4.0 International License.