International Review of Research in Open and Distributed Learning

Volume 17, Number 3

April - 2016

Participation in the Virtual Environment of Blended College Courses: An Activity Study of Student Performance

Cathy Cavanaugh¹, Jace Hargis², and John Mayberry^3
1Microsoft Worldwide Education, ²University of California, ³University of the Pacific

Abstract

This paper describes a study of success factors in the introductory semester of liberal studies blended courses offered at the bachelor of science level. The influence of student participation in the online course environment was examined, as measured by the number of times students logged into the learning management system (LMS) and average session length. These measures were correlated with final course grades to increase understanding of the participation patterns of successful students. The resulting patterns and their implications are identified. We observe that students with an intermediate number of logins and average session length tended to exhibit the optimal level of course performance with students who logged in near the low or high amount of times tending to receive lower grades.

Keywords: success factors, blended learning, online education, higher education, student engagement

Introduction

In an effort to provide students of a women's college in the Middle East with the highest quality learning experiences, flexible access to courses, and most accessible channels of communication among students and faculty, staff examined the potential for expanded use of open source software and platforms. The intent of the activity was to identify a rigorous and extensible system for teachers to build and share dynamic learning objects with students and between faculty. The pool of candidates for possible open Course or Learning Management Systems (LMSs) included Sakai and Moodle. Although these systems provided similar options, the college choose to pilot Sakai in an attempt to determine feasibility of use for blended courses, with the possibility of disseminating content and participation to other colleges in the region. Sakai has a modified approach to the traditional LMS, promoting the tool as a sharing or collaborative environment, hence the label of Collaborative Learning Environment (CLE). Sakai offers the core functions of prior LMSs with the addition of being open, which some interpret to mean free. Although there is no licensing for the software, the cost is in the human and technical resources required by hosting, maintenance, and programming. The college chose to outsource these responsibilities to a reputable hosting company in Europe, due to their proximity of time zone. The fee to host a two semester project for the college that enrolled 2,500 students on two campuses was $10,000 USD, which included all services as well as sufficient template modification to offer a branded site for the college. Both parties were aware that the cost was reduced as the intent was to produce proof of concept data, to enable full integration in subsequent years within the college, and potentially to expand use of the CLE throughout the multi-campus system.

The CLE hosting business's leaders and college staff worked via campus meeting and Web meetings, led by the host Project Manager and the college Technical Services Department Supervisor. The host provided training to the college technical staff, who were already experienced in managing LMS platforms. The goal of the pilot project was to create an effective and flexible system for student engagement, as well as instructor sharing of their resources, lessons, assessments, and so on. The anticipated outcome was that this system would be as good as, if not better than, the current LMS used in the college, which required a substantial licensing fee. The approach adopted for the pilot project was a quasi-experimental design in which both LMS platforms would be used in parallel for sufficient numbers of sections, instructors, and students to generalize the findings into other courses and campuses. Ultimately, the college found that Sakai could offer more than the commercial LMS, at a significantly reduced cost, and with no usability issues for faculty or students.

This paper describes key findings from the open CLE pilot program. This study focused on success factors in the introductory semester of liberal studies blended courses offered at the bachelor of science level. The influence of student participation in the online course environment was examined, as measured by the number of times students logged into the learning management system (LMS) and average session length. These measures were correlated with final course grades in order to develop recommendations for blended course design and instruction, which must apply evidence for how students learn to the micro and macro levels of content chunking (Keeler, 2015). This study addressed the blended learning success factor of iterative and recursive design (Ferdig, Cavanaugh, & Freidhoff, 2015), and provides a model for programs that seek to refine their blended course design approach.

Blended Learning for Liberal Studies

Tertiary education is increasingly being redefined from site to service (Cavanaugh & Hargis, 2010) as the emphasis in education shifts from being a location where students have similar experiences along similar timelines depending on system needs to being a personalized program provided as needed, where needed, and when needed depending on student needs. This site-to-service shift includes several changes that we see emerging as influences on the design and delivery of education, as shown in Table 1.

Table 1

Shifts from Site to Service

These shifts in the design and delivery of formal education have so far been largely made possible by technology, and their degree of priority has increased due to a range of economic and social conditions. Colleges are serving a wider range of students in degree programs, are under pressure to make efficient use of space and resources, are held accountable for each student's academic performance, are seen as key drivers in the nation's economic competitiveness, and thus, are depended upon to provide career-ready graduates. Because the technology now exists for differentiated instruction, these priorities are increasingly met through blended education programs that merge physical and virtual learning environments to serve students with greater flexibility and agility. This article describes a study of a semester-long college pilot program in which blended undergraduate courses were offered and data were analyzed to identify factors related to academic success in the courses. A blended approach to course design was especially appealing for the pre-professional students in the college who spent increasing time in professional settings during their baccalaureate degree programs, and could benefit from the added time flexibility afforded by blended course delivery.

In tertiary education, examples of technology-enabled blended education date from the beginning of educational technology. Classroom learning time has been enhanced, augmented, and partially replaced by technology for as long as the technology has been available (Ferdig & Kennedy, 2014). The past decade has seen a steady increase in the development and implementation of blended education programs as technology has become more powerful and ubiquitous. However, it has been only in the past few years that deliberate blended education programs have developed (Clark & Barbour, 2015). This progress has been spurred in large part by open source learning management systems and more readily available education resources.

In the Middle East and North Africa (MENA) region where the college participating in this study is located, open education resources are expanding, as shown in a recent survey in the United Arab Emirates (Khelifi, Abu Talib, Al Shabibi, Al Zaabi, & Al Marshoudi, 2010). Open source software and its associated benefits in terms of cost efficiency, flexibility, and participant content creation are regarded as positive supports for scaling up access to education in countries with comparatively new education systems: "an open learning environment and course should offer youth an opportunity to personalize their learning, to make it meaningful, authentic, and engaging. Open learning creates the opportunity to offer all networks a chance to connect and learn together regardless of age, experience, culture, or background" (Graham, Labonte, Roberts, O'Byrne, & Osterhout, 2014, p. 460).

Effective blended education approaches in higher education have been documented (Bonk & Graham, 2005; Garrison & Vaughan, 2008; Picciano & Dziuban, 2010), but none have centered in the Persian Gulf region and few report data on student performance. Those that have done so suggest that blended programs can contribute to learning gains (US Department of Education, 2009; Laumakis, Graham, & Dzubian, 2009). Further, very little research examines specific factors within blended courses and environments that correspond to student academic performance. The college program profiled here fills that gap by analyzing participation of all students enrolled in a semester of blended courses that used the Sakai CLE to enhance and expand classroom learning.

While each blended course has unique features that intersect in different ways with different types of students, findings of research on the characteristics of successful blended learners suggest a common set of characteristics likely to result in successful virtual learning. The expanded learning time afforded by a flexible blended course timeline has shown success in raising student achievement (Cavanaugh, 2009). Indeed, time was identified as the most significant factor in student success in 11 of 15 high enrollment blended courses (Liu & Cavanaugh, 2011). This finding encourages the design of courses to engage and motivate students to spend more time in the system. Specifically, Liu and Cavanaugh (2011) observed that the frequency with which students accessed online materials related to achievement in ways that seem to depend on the complexity of the course, in that more logins corresponded to higher achievement in introductory level courses, while fewer logins for longer sessions corresponded to higher achievement in advanced courses. These findings guided the college open source blended course data analysis in spring of 2012.

The research question guiding this study is "Does student level of participation in the online learning environment correspond to student academic performance in a blended college course? If so, how?"

Methods

Setting and Participants

Data were collected during a full semester when the blended course design was introduced in a mid-sized four-year women's college in the United Arab Emirates. During this semester, 590 students and 14 instructors were registered in course sections created in Sakai CLE for courses that included classroom and online experiences. Seven courses were designed in-house for the CLE. Of these courses, one was developed and taught in a way that integrated physical and virtual interactions. This course (LSS 2533—see Table 2) is part of the bachelor-level liberal studies curriculum, and was taught by seven experienced faculty members for a total of 477 students using the same instructional materials, assignments, and assessments as the fully face-to-face versions of the course. Students were registered in LSS 2533 according to their program plans and schedules. Three hundred twenty-one students completed the course and were included in this study. Because the college serves only female National citizens of the country where it operates, students in the study sample represent a narrow ethnic and socioeconomic range in one large city. Thus, demographic factors were not included as variables in the study. Because of the uniform backgrounds and entry levels of the students, differences among them in course performance may depend on their engagement in the course. Table 2 summarizes course information for students included in this study.

Table 2

Blended Course Participation

Instrument

The success of students in blended learning courses can be measured by the grades students earn in the courses and academic performance on standardized exams (Ronsisvalle & Watkins, 2005). In the present study, student final scores in the course were determined by performance in a range of course assessments developed by expert panels for use across the national college system for these courses. Assessment methods were reflective of an undergraduate survey course, and included discussion, essays, limited-response exams, and projects. The assessments have been validated for their use over time and at multiple campuses. Student final course grade was used as the measurement for success. Data logs generated by the Sakai CLE during the course included the number and length of student logins to the online course components.

Data Collection

The number of logins by each participant (a potential indicator of course presence during the weeks that the LMS was used) and average session length (a potential indicator of course engagement) were used as independent variables in our study. Login and final grade summary statistics are shown in Table 3 for the full 18-week semester of the course.

Table 3

Descriptive Statistics for Student Course Logins, Session Length, and Grades

Note. Min = minimum; 25^th% = lower quartile; 75^th% = upper quartile; Max = maximum

Notice that while there was a wide range of login attempts (1–80), the interquartile range was much smaller with 50% of all students logging in between 10 and 24 times with a median of 15 logins. Typical session lengths lasted about 1–1.5 hours. A positive correlation was found between number of logins and session length ( r = 0.19, p = 0.0006; Figure 1).

Figure 1. Plot of average session length versus number of logins along with the corresponding regression line.

Data Analysis

We looked at two nested models for predicting final course grade based on using number of logins ( Logins ) and average session length ( ASL ) as predictors. In Model 1, final course grade is linearly related to both predictive variables. In Model 2, we define the quadratic relationship:

Second order equations in the predictive variables were used to detect optimal performance at intermediate levels; for example, to allow us to determine if an intermediate number of logins predicts better performance than either too few or too many logins. All models were fit using the open-source software R. Due to the presence of extreme outliers (which could not be excluded on basis of incomplete information) and lack of normality in the distribution of residuals, the robust and resistant method of MM-Estimators, introduced by Yohai (1987) and implemented in R's MASS package (Venables & Ripley, 2002), was used to obtain estimates and standard errors for linear model coefficients. Significance of parameters was determined using asymptotic t tests. Estimates for the optimal logins and ASL were obtained via standard formulas for the vertex of a paraboloid. The collinearity between logins and ASL was addressed by checking the variance inflation factors (VIF) associated with each term in Model 1.

Results

Model 2 significantly outperformed Model 1 demonstrating a parabolic relationship between our response variable (course grade) and both predictors (logins and ASL) as shown in Table 4. Logins appears to be a more significant predictor than ASL although since these variables are collinear, we leave both terms in the model. Both quadratic coefficients in Model 2 are negative implying that course grade is predicted to be highest at intermediate values of both predictors, as shown in Figure 2. We approximate that the optimal combination of login times and session length was approximately 22 logins lasting around 63 minutes on average.

Table 4

Least Squares Estimates of Model Parameters

Note. *denotes results which were significantly nonzero with p < 0.05; *** denotes results with p < 0.001. To address the collinearity of predictors in Model 1, we computed the VIF of both terms and found a small impact on the resulting standard errors (VIF of Logins = 1.020; VIF of ASL = 1.020). Model 2 demonstrated a significant improvement over Model 1 (extra sum of squares F test; F 3,321 = 5.466, p = 0.001).

Figure 2. Left or top panel: plot of course grade versus number of logins along with fit from Model 2 using optimal ASL of 63.29 hours (solid black line). The optimal value of 22.33 logins is shown as the solid red line. Right or lower panel: plot of course grade versus ASL along with fit from Model 2 using optimal logins of 22.33 (solid black line). The optimal value of 63.29 hours is shown as the solid red line.

We propose possible explanations for our results. First, success at intermediate session lengths is aligned with common theories from learning sciences in that too much time spent per session may indicate cramming and lead to cognitive overload, while too little time may not contribute to sufficient time on task for retention (Cavanaugh, 2009; Liu & Cavanaugh, 2011). Since number of logins is positively correlated with ASL (Figure 1), this would also explain the dependence of course grade on number of logins. Alternatively, we conjecture that overly frequent logins may indicate a distracted student. Such a student may be logging into Sakai CLE frequently, but spending the bulk of their login time switching between the course Web site and non-course related Web browsing. As long as the student continued to stay logged in to Sakai, a long session length would have been recorded even though the student did not spend much of their login time on course related exercises. Such distracted sessions may, in turn, beget more distracted sessions, leading to a further increase in the frequency of logins. Finally, a third explanation is the classic correlation does not imply causation paradigm. It is possible that students who logged in too many times were less prepared students and the additional login time was not sufficient to overcome these initial aptitude gaps. Further study accounting for entry level skills is needed to parse this possibility.

Conclusions and Implications

We examined student participation in the online components of a blended undergraduate college course in order to identify the extent to which the number and length of logins corresponded with student grades in the course.

It was not the intent to compare the effects of learning in blended courses with other learning environments or to evaluate the specific design approach of the course. This study focused on student engagement in the online components of the course in order to guide students, instructors, and course designers. Each blended learning program is unique and designed for a specific purpose (Ferdig, Cavanaugh, & Freidhoff, 2012). This blended course was intended primarily to expand the learning time for native Arabic-speaking students who used the online tools and resources at the times of day and for the length of time they needed to enhance their learning, given the wide range of logins and session lengths recorded. For these students, who also met in class weekly, one to two logins per week of just over one hour in duration each related to the highest course grades. This result confirms the claim that variation theory underlies effective blended learning experiences because a range of activities maintains student engagement (Oliver & Trigwell, 2005). In keeping with variation theory and this study's findings, effective blended learning courses can be designed to alternate physical and virtual experiences that require an hour of online time.

Education in any learning environment is a complex, challenging professional practice; education across various learning environments multiplies the complexity and challenge. Formal research is needed in order to illuminate promising practices for each intersecting student and learning goal. This paper sheds light on one example. Data from practice must be coupled with data from research. Data collection on learning processes and student outcomes is identified as a success factor in blended courses (Ferdig, Cavanaugh, & Freidhoff, 2015), and is a key step toward application of analytics to inform content selection, course design, and learning pathways (Cavanaugh, Sessums, & Drexler, 2015). Thus examining data on student learning activities, content usage, and performance over time with prediction and prescription stand to dramatically improve education attainment.

We implore readers to do what they can to support research in blended education with open source tools: open your programs to researchers, engage in scholarship of teaching and learning (SoTL) to share your lessons, seek researcher funding and partnerships, disseminate research findings, participate in research conferences, read research journals, and assist researchers-in-training as mentors. We are all members of the community of practice and scholarship that advances outcomes for students as we learn together.

References

Bonk, C., & Graham, C. (2005). Handbook of blended learning: Global perspectives, local designs. San Francisco, CA: Pfeiffer. Retrieved from http://www.wiley.com/WileyCDA/WileyTitle/productCd-0787977586.html

Cavanaugh, C. (2009). Getting students more learning time online: Distance education in support of expanded learning time. Washington, DC: Center for American Progress. Retrieved from https://www.americanprogress.org/wp-content/uploads/issues/2009/05/pdf/distancelearning.pdf

Cavanaugh, C., & Hargis, J. (2010). Redefining school from site to service: Learning in and from K–12 online education. Distance Learning, 7(2), 1–5. Retrieved from http://www.infoagepub.com/dl-issue.html?i=p54c116b086919

Cavanaugh, C., Sessums, C., & Drexler, W. (2015). A call to action for research in digital learning: Learning without limits of time, place, path, pace…or evidence. Journal of Online Learning Research, 1(1) 9–15. Retrieved from https://www.editlib.org/p/149851/

Clark, T., & Barbour, M. (2015). Online, blended and distance education in schools. Sterling, VA: Stylus and Microsoft. Retrieved from https://sty.presswarehouse.com/Books/BookDetail.aspx?productID=391638

Ferdig, R., Cavanaugh, C., & Freidhoff, J. (Eds.). (2012). Lessons learned from blended programs: Experiences and recommendations from the field. Vienna, VA: iNACOL. Retrieved from http://www.inacol.org/resource/lessons-learned-from-blended-programs-experiences-and-recommendations-from-the-field/

Ferdig, R., Cavanaugh, C., & Freidhoff, J. (2015). Research into K–12 online learning. In T. Clark, & M. Barbour (Eds.), Online and distance education in schools: Global perspectives on policy and practice. Sterling, VA: Stylus and Microsoft. https://sty.presswarehouse.com/Books/BookDetail.aspx?productID=391638

Ferdig, R., & Kennedy, K. (2014). Handbook of research on K–12 online and blended learning. Pittsburgh, PA: Carnegie Mellon University Press. Retrieved from http://press.etc.cmu.edu/content/handbook-research-k-12-online-and-blended-learning-0

Garrison, R., & Vaughan, H. (2008). Blended learning in higher education: Framework, principles and guidelines. San Francisco, CA: Jossey-Bass. http://www.wiley.com/WileyCDA/WileyTitle/productCd-0787987700.html

Graham, L., Labonte, R., Roberts, V., O'Byrne, I., & Osterhout, C. (2014). Open learning in K–12 online and blended learning environments. In R. Ferdig & K. Kennedy (Eds.), Handbook of research on K–12 online and blended learning. Pittsburgh, PA: Carnegie Mellon University Press. Retrieved from http://press.etc.cmu.edu/content/handbook-research-k-12-online-and-blended-learning-0

Keeler, C. (2015). Instructional design. In T. Clark & M. Barbour (Eds.), Online, blended and distance education in schools. Sterling, VA: Stylus and Microsoft. Retrieved from https://sty.presswarehouse.com/Books/BookDetail.aspx?productID=391638

Khelifi, A., Abu Talib, M., Al Shabibi, A., Al Zaabi, A., & Al Marshoudi, N. (2010). Open source software in the UAE: A survey research. Paper presented at the 3rd International Symposium on Web Services, Zayed University, Dubai, U.A.E. Retrieved from http://brandengroup.com/files/OpenSource_UAE.pdf

Laumakis, M., Graham, C., & Dziuban, C. (2009). The Sloan-C pillars and boundary objects as a framework for evaluating blended learning. Journal for Asynchronous Learning Networks, 13(1). Retrieved from http://onlinelearningconsortium.org/read/journal-issues/

Liu, F., & Cavanaugh, C. (2011). Online core course success factors in virtual school: Factors influencing student academic achievement. International Journal of E-Learning, 10(4), 43–65. Retrieved from https://www.editlib.org/p/33040/

Oliver, M., & Trigwell, K. (2005). Can "blended learning" be redeemed? E-Learning and Digital Media, 2(1), 17–26. doi: 10.2304/elea.2005.2.1.2

Picciano, A., & Dziuban, C. (Eds.). (2010). Blended learning: Research perspectives. Needham, MA: The Sloan Consortium. Retrieved from http://olc.onlinelearningconsortium.org/node/921

Ronsisvalle, T., & Watkins, R. (2005). Student success in online K–12 education. Quarterly Review of Distance Education, 6, 117–124. Retrieved from http://www.infoagepub.com/qrde-issue.html?i=p54c3cb86a9a11

U.S. Department of Education. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Office of Planning, Evaluation, and Policy Development. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

Venables, W., & Ripley, B. (2002). Modern applied statistics with S (4^th ed.). New York, NY: Springer. Retrieved from http://link.springer.com/book/10.1007%2F978-0-387-21706-2

Yohai, V. (1987). High breakdown point and high efficiency robust estimates for regression. The Annals of Statistics, 15(20), 642–656. Retrieved from http://www.jstor.org/stable/2241331

Participation in the Virtual Environment of Blended College Courses: An Activity Study of Student Performance by Cathy Cavanaugh, Jace Hargis, and John Mayberry is licensed under a Creative Commons Attribution 4.0 International License.