|
|
|
Hedieh Najafi1, Rosemary Evans2, and Christopher Federico1, 2
1University of Toronto, Canada, 2University of Toronto Schools, Canada
We investigated how high school students taking a university preparatory economics course would engage with the learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course. Students were divided into two groups, MOOC-only, with no teacher support, and blended-mode, with weekly tutorials. MOOC only students scored slightly lower on a teacher designed knowledge test but scored slightly higher in a MOOC test. Although the MOOC-only students watched more unique videos, the blended-mode students stayed more on-track with the MOOC. The blended-mode students showed more persistence in retaking quizzes, yet they scored lower than the MOOC-only students.
Massive online open courses (MOOCs) have been used in various ways often in the form of a flipped classroom, to complement traditional classroom teaching through integration of a whole course or specific parts of a course both in K-12 and in higher education. High school students, for example, could use MOOCs for university preparation in the absence of available face to face or online courses or in addition to them (Vihavainen, Luukkainen, & Kurhila, 2013). Furthermore, contents of a live or an archived MOOC could be integrated into an existing course in a hybrid format (Bruff, Fisher, McEwen, & Smith, 2013).
In this research we investigated how high school students taking a university preparatory economics course would engage with learning and assessment components of a Behavioural Economics MOOC that was integrated into their school-based course. Of specific importance to this research was probing any potential difference in students’ use of various components of the integrated MOOC depending on receiving supplementary instructional support or studying on their own. For this purpose, three weeks of the “Behavioural Economics in Action” (BE101x) MOOC were chosen for integration into the school-based course “Analyzing Current Economic Issues” (ACEI). Twenty nine students from a Canadian independent university preparatory school who had enrolled in ACEI were randomly assigned to two groups for the BE101x integration period with one group studying on their own and the other receiving weekly tutorial sessions from their classroom teacher. Here, we report on how students in the two groups engaged with the learning and assessment components of BE101x and persisted throughout the three weeks of integration. Also, we compare students’ outcomes based on pre-post BE101x-integration content knowledge surveys and BE101x assessments.
Research on online university preparatory courses, MOOC-like university preparatory courses, and MOOC integration in credit courses informed our study.
Online advanced placement (AP) courses could increase students’ access when face to face options are not viable due to restricted budgets or geographical location. In Newfoundland and Labrador, for example, the percentage of students in rural schools who enrolled in advanced-level courses through the Centre for Distance Learning and Innovation was higher than the provincial average (Barbour, & Mulcahy, 2013). Similarly, students in Florida who enrolled in online AP courses available to all students, regardless of income level and district budget, scored higher than the state average in their AP exams (Johnston, & Barbour, 2013). Yet students who participated in a follow up interview reported a preference for face to face classes where they could readily communicate with a teacher and improve their performance based on teacher feedback. Teacher presence in online secondary school courses positively affected student outcomes in introductory mathematics courses (Liu & Cavanaugh, 2012). For more advanced courses, on the other hand, the amount of teacher feedback was negatively related to course outcomes which may indicate that to be successful in more advanced online courses, high school students need to be self-directed and less reliant on teacher feedback. Time spent in the online course environment was another factor that would have a positive effect on students’ assessment outcome.
Some MOOCs have imposed minimal prerequisites for registration, and high school students have already established themselves as a demographic group in MOOCs of various difficulty level (Breslow et al., 2013). 2.39% of survey respondents of an electronics MOOC, with suggested academic prerequisite of having advanced knowledge of electricity and magnetism, reported their highest education level to be below high school diploma (DeBoer, Stump, Seaton, & Breslow, 2013). In terms of course achievement, however, total points gained by those students with junior high school education level were comparable with students holding a bachelors degree.
Aside from MOOCs offered to the general public, specialized MOOCs have been developed for middle and high school students, either as university preparatory courses or as high school level courses. To address the paucity of pedagogically sound high school level computer science curricula, a mini course titled “Foundations for Advancing Computational Thinking” (FACT) was developed on the Stanford OpenEdX platform and piloted in a public middle school (Grover, Pear, & Cooper 2014). The FACT mini course has demonstrated promising results regarding students’ learning and it is being offered as a MOOC for either self-study or for teachers to use in their computer science classrooms. A similar project, MyCS, geared towards middle school students, was also considered for further development as a teacher-mediated computer science MOOC (Schofield, Erlinger, & Dodds, 2014).
To reinforce students’ knowledge of science and mathematics and to facilitate their entrance to universities, an Italian school board implemented high school specific online open courses by recording classroom lectures and making them available online (Cannesa & Pisani, 2013). Test results showed that students who watched online lecture videos in addition to their face to face classroom scored higher than those who only relied on face to face classroom learning. Although not a typical MOOC in terms of design, this project supports the notion that self-paced open access material, geared towards high school students may increase students’ depth of learning and academic success. In another study, a university preparatory mathematics MOOC was developed to increase mathematical proficiency of incoming students (Daza, Makriyannis, & Rovira Riera, 2013). First year university students were encouraged to participate in the MOOC voluntarily while the MOOC was also open to the public. Initial findings, showed students level of satisfaction with the contents of the MOOC, although the impact of MOOC learning on students’ success was not measured due to the scope of the study.
While several projects such as the University of Wisconsin’s College Readiness Math MOOC and the MIT physics MOOCs offer preparatory courses to secondary school students, research on the impact and implications of such initiatives is still in its infancy.
With respect to integrating existing MOOCs into regular classroom instruction, flipped classrooms could offer one model. In flipped classrooms, the online component, for example the MOOC, allows students to prepare for in-class discussion by reviewing relevant material beforehand and at their own pace. In other models of integration, even MOOC assessments could count towards students’ evaluation of the on-campus course. The instructor of a machine learning graduate course, for example, integrated an archived MOOC into his face to face course where students were required to take MOOC quizzes and write MOOC assignments, submitted to their instructor, in addition to covering learning components of the MOOC. Students responded positively to the experience and noted that the face to face classes would help them keep track of the self-paced MOOC component (Bruff et al., 2013).
In this study, we examined potential impacts of the presence or absence of teacher support on secondary school students’ behavior and outcome during a MOOC-integration initiative. Three research questions guided our study:
A case study encapsulates our research design in examining the impact of different levels of instructional support on students’ engagement with learning and assessment components of BE101x and their outcome as evident in MOOC quizzes and knowledge-based tests. To this end, BE101x integration in the ACEI course could be an instrumental case (Stake, 1995) that allows us to gain insight into the issue of using MOOCs in the context of a secondary school course.
This study was conducted in the context of a university preparatory economics course at a competitive independent Canadian school from October 14 2013 to November 21 2013. The two participating class sections were taught by an experienced teacher, also a co-investigator of the study. The age range of students enrolled in this course was between 15 to 17 years. Twenty nine students agreed to participate in the study out of a total of 32 students. Participation in this study was voluntary and had no impact on students’ formal evaluation and course outcomes.
BE101x, a six week MOOC on edX, was first offered in October 2013. Consequently, this study coincided with the live offering of this MOOC. Below we discuss curriculum design considerations, research timeline, participant recruitment, and grouping procedures.
Integration of BE101x in ACEI was temporally bound between October 14, 2013 and November 18, 2013, weeks 1 to 5 of the BE101x MOOC. Before the study started, the teacher compared the contents and objectives of BE101x and ACEI to find areas of overlap where BE101x could be most effectively integrated.
Weeks 1, 2, and 5 of BE101x MOOCs were selected for integration. ACEI’s start time coincided with the first week of BE101x. Timeline, themes, learning components and assessment components of weeks 1, 2, and 5 of BE101x were as follows. Note that only those learning components that preceded a quiz or a debate were accounted for.
Prior to the start of the study, students were randomly divided into two groups for research purposes: a MOOC-only group with 14 students and a blended-mode group with 15 students.
In the three weeks outlined above, regular classes were not held. Instead, students in the MOOC-only group would study BE101x independently with no support from their teacher. On the other hand the blended-mode group students met their teacher once a week in an hour-long tutorial session. Materials from these sessions were given to MOOC-only students after week 5.
Tutorials were structured to include three common sections: Questions and answers, Experiment/Business problem, and an Individual exercise. The purpose of these tutorial sessions was to elaborate on BE101x themes and engage students in applying what they had learned during that week. The dates of these tutorials were: week 1, October 21, 2013; week 2 , October 28, 2013; and week 5 , November 18, 2013.
Students would attend regular classes in weeks 3 and 4, between October 29 and November 10, 2013. Upon the culmination of week 5 of BE101x, regular classes were resumed for the rest of the school term.
Students received an email prior to the start of their course that explained the study and its goal. This email was followed by another orientation email sent from the teacher on the first day of BE101x asking students to sign up for the MOOC on edX and also recommending that they refrain from starting the course until October 16th where a face to face orientation class meeting would be held.
Time and date for knowledge-based pre and post tests were also shared with the students. On October 16, 2013 the teacher met all of the students where the students received further explanation about BE101x and the study and took a knowledge-base pre-test. The teacher explained that the students would not be evaluated on their participation in BE101x and that participation and the extent of engagement were absolutely the responsibility and choice of the students.
To maintain student anonymity, we refrained from using any data that could lead to revealing their identity including their contributions to BE101x discussion forums and weekly debates. We collected data from the following sources.
Three dimensions of student engagement, student persistence, and student outcomes during the duration of their MOOC participation were of importance to this research.
Student engagement related to students’ use of and access to BE101x learning components, specifically lecture videos that preceded each quiz, debate videos, and debate debrief videos. These learning components were located within the clickstream data by their unique identifiers. Four quizzes were preceded directly by either articles or slides. We could not track them in the clickstream data and thus did not include them in our analyses. For student engagement dimension, we compared the two groups on unique videos accessed, total number of videos accessed, and videos accessed on-track with BE101x pace.
Student persistence referred to the number of quizzes they took in weeks 1, 2, and 5 of the integration and if quizzes were retaken until a perfect score was achieved. The last dimension, student outcome, attended to differences between MOOC-only and blended-mode groups on their performance in BE101X test, overall quiz score, and in content knowledge pre-post tests. We would like to note that technical problems in extracting clickstream data largely affected the depth and scope of our analysis.
Students’ scores in content knowledge pretest, written at the start of BE101x integration, and posttest, written after the integration was over, showed an increase in their knowledge of Behavioural Economics. As illustrated in Figure 1, MOOC-only students scored lower in the pretest (M= 4.91; SD= 1.08) compared to blended-mode students (M= 6.06; SD= 1.86). In the posttest, despite performing better than the pretest (M= 7.75; SD= 1.42), MOOC-only students still gained lower scores than the blended-mode students (M= 8.53; SD= 1.68). A two sample t-test did not show a significant difference between the two groups on their posttest scores; t(25)=1.28, p=0.1.
A BE101x test written in week 3 provided another measure for students’ outcome. Unlike the content knowledge posttest results, students in the MOOC-only group scored higher (M= 12.73, SD=1.38) than the blended-mode group students (M=11.53, SD=3.06). A two sample t-test did not reveal a significant difference between the two groups, t(23)=0.94; p=0.17.
We investigated if the observed overall difference between the MOOC-only and the blended-mode groups in terms of their pretest and posttest scores also reflected in their level of persistence and engagement with learning and the assessment components of the integrated MOOC.
During the running of the course, students were free to login to BE101X whenever they desired. However, only student access to BE101X between October 14, 2013 and November 21, three days after the third tutorial session, was specifically relevant to our research. We started by, first, examining the number of students in both groups and in total who accessed the integrated BE101X at least once in each day between October 14, 2013 and November 21. Although weeks 3 and 4 of BE101X were not integrated into ACEI, we included these two weeks in our analysis as we assumed that students may have reviewed week 1 and week 2 material or have caught up with activities that they had missed.
The number of students who logged into the integrated BE101x in weeks 1, 2 and 5, in addition to weeks 3 and 4 in between, is shown in Figure 2. We corroborated high and low levels of activity with important dates during the integration period. Note that any activity before October 16 is negligible as students were not yet divided into groups.
The highest level of activity occurred on the two days when BE101X week1&2 and week5 tests were administered, October 31, 2013 and November 21, 2013 respectively. The day of the first tutorial showed another peak of activity with 7 MOOC-only and 9 blended-mode students online. A high activity level is observed on November 18 for the blended-mode group, which can be attributed to the third tutorial session. As expected, the level of activity was low for both groups after the October 31 test and remained relatively low throughout week 4. Both groups were comparable in the level of activity until November 16, 2013 when a peak of activity is evident for the blended-mode students that was sustained until November 18 and then dropped quickly in the next two days. In that three-day period, students in the MOOC-only group maintained a lower level of activity. However, their activity level steadily increased and eventually equated with the blended-mode students in the two days leading to test 2 on November 21, 2013.
We examined students’ access in both groups to the lecture videos that would precede a quiz in weeks 1, 2, and 5. First, we calculated the number of unique lecture videos, debate videos, and debate debrief videos students had accessed in the duration of BE101x integration, including weeks 3 and 4. Table 1 summarizes descriptive statistics relevant to unique videos that students in both groups watched.
MOOC-only students on average watched, that is, played, more unique videos than students in blended-mode group. Overall engagement level with lecture videos based on the average number of lecture videos watched was close to a maximum of 20 videos in both groups.
To better understand any noticeable differences in how students in the two groups accessed unique lecture videos of the integrated MOOC, we graphed the percentage of students in each group who watched lecture videos of week 1, week 2, and week 5 at least once any time during the integration and in the two weeks in between. As depicted in Figure 3, 80% of all lecture videos, were watched by more students from the MOOC-only group. For the rest of the lecture videos, the difference between the number of blended group students and MOOC-only group students who watched those videos was a maximum of two students. Throughout the three weeks and between the two groups, the percentage of students watching lecture videos fell slightly with only one of 7 videos being watched by more than 90% of the students. This number was 6 out of 6 for the week 1 video and 3 out of 7 for the week 2 lecture videos. Still, the overall lowest percentage of students watching a lecture video, which is a week 5 lecture video, was nearly 67% by students in the blended-mode group.
Students in both groups watched most of the lecture videos belonging to three weeks of BE101x integration. However, we questioned if students would have followed the course week by week or would have lagged behind and compensated later. Thus, for both groups, we examined if students watched lecture videos of a given week at least once during that week.
Table 2 summarizes the average percentage of lecture videos in weeks 1, 2, and 5 that were watched within those specific weeks. As Table 2 shows, on average, students in blended-mode group following the BE101x course pace watched lecture video more than their peers in the MOOC-only group. Week 2 engaged the highest number of students in both groups in watching lecture videos as MOOC-only students and blended-mode group students watched 43.88% and 63.81% of lecture videos between November 21, 2013 and November 27, 2013.
We, then, considered if there were students in both groups who consistently remained on-track during the three weeks of integration or consistently lagged behind. In weeks 1, 2, and 5, four MOOC-only students, 28.57%, completely stayed on track whereas six others, 48.85%, completely lagged behind and watched none of the lecture videos belonging to those weeks. These numbers for the blended-mode group were three students, 20%, and four, 26.66% respectively.
Figure 4 and Figure 5 show the difference between the two groups regarding following the pace of the integrated MOOC by watching lecture videos in Weeks 1, 2, and 5 on-track, in more detail. More students from blended-mode group followed the pace of the course for more than 80% of the lecture videos. Consequently, although a higher percentage of MOOC-only students watched all videos following the pace of the course, blended-mode students as a group remained on-track more consistently.
Students could take a total of 29 quizzes with a maximum total quiz points of 29. A quiz would only be graded if students chose to submit their answer. As a result, a zero quiz point would either mean that the student submitted an incorrect answer, or else, the student did not take the quiz at all. To examine students’ performance in quizzes in the MOOC-only and blended-mode groups we considered total points received in all of the 29 quizzes regardless of whether a quiz was actually submitted for grading. We factored such differences into our analyses of how students attempted to retake quizzes to achieve maximum points possible.
Students in the MOOC-only group received slightly more points in quizzes (M= 24.95; SD=5.98) than students in blended-mode group (M=23.74; SD=7.43). Regarding the number of quizzes taken, out of 29 quizzes, MOOC-only group submitted more quizzes for grading (M=26.42; SD=6.08) compared to blended-mode group students (M= 24.53; SD=7.69).
Finally, we investigated students' persistence while taking quizzes in terms of achieving maximum quiz points possible for the number of quizzes taken. Blended-mode group students outperformed MOOC-only students in persistence as 40% of the former versus 21.42% of the latter correctly answered the quizzes that they submitted for grading.
Of specific importance to this research was probing any potential differences in students’ use of learning and assessment components of the integrated MOOC depending on receiving supplementary instructional support or studying on their own. Students’ scores in the pretest, written before starting the integrated BE101x, and posttest, written after the integrated BE101x was over, showed an increase in their knowledge of behavioural economics. Thus, the integration had no adverse effect on students’ learning.
With the opportunity to expand on BE101x content in the three tutorial sessions, we would expect blended-mode students to watch more video lectures and be more engaged with these learning components. Surprisingly, MOOC-only students on average watched more unique videos than students in blended mode group. Regarding students’ performance in MOOC quizzes, MOOC-only students were at no disadvantage as they received more quiz points compared to blended-mode students. However, we noticed that students from the blended-mode group were more likely to retake quizzes to achieve complete quiz points.
MOOC quizzes, considered as data sources for this study, may only reveal students’ ability to recall information. The specific MOOC discussed in this study also contained knowledge transfer and application assignments and weekly debates that were not included as data sources. We also inferred from clickstream data that students participating in this study did not participate in the MOOC’s discussion forum. However, with the evidence that students would access MOOC resources and take quizzes with no direct contact with their classroom teacher, we posit that MOOCs could provide another means for reaching ahead and preparing for university for students who rely on online advance course for reasons such as geographical location (Barbour & Mulcahy, 2013).
In this study, we questioned if students would have followed the course week by week or would have lagged behind and compensated for it before the tests in week 3 and week 6 to cover lecture videos. Fewer students in the blended-mode group lagged behind, which could be attributed to the weekly face to face tutorials. Previous research confirms this finding (Bruff et al., 2013) that instructional presence may positively affect students following the expected pace of integrated content.
Although a higher percentage of MOOC-only students watched all videos following the pace of the course, blended-mode students as a group remained on-track more consistently. Knowing that they would discuss lecture videos of a given week during the tutorial session, these students would have preferred to attend the class prepared rather than postponing watching lecture videos to a few days before test 1 in week 3 and test 2 in week 6.
Empirical literature to investigate potential learning benefits of developing MOOCs or MOOC-like initiatives for specific age or grade level group is still in its infancy. Existing studies either report on preliminary findings in the form of overall satisfaction (Daza et al., 2013) or improved test results (Cannesa & Pisani, 2013) or propose the development of a MOOC based on pilot projects (Grover et al., 2014; Schofield et al., 2014). We contributed to this growing line of research, by comparing how high school students engage with a MOOC integrated into their school-based course, in a self-directed manner or with teacher support.
The results of our study are promising regarding integrating MOOCs in school-based courses in a self-study manner. Students engaged with learning and assessment components of an integrated MOOC in a self-study manner. It is important to note however, that participants of this study were high achieving and intrinsically motivated students which may have affected their level of engagement. Integrating MOOCs into school-based courses entails curriculum design challenges as classroom teachers need to find relevant content that enriches their existing curriculum. One implication would be for teachers to carefully examine the added cognitive value of MOOC integration. Another challenge is the persistence of an integrated MOOC over time. In case an integrated MOOC is removed from its provider platform, the teacher may have to look for a replacement or abandon the integration. MOOCs specifically developed for school courses may alleviate such problems.
Understanding students’ experience through interviews and detailed reflection notes would shed more light on their perception of the usefulness of the integration and the challenges they may have faced. This study only included quantitative measures but our next round of research will also consider in-depth qualitative data related to the MOOC integration design process, student interviews and reflection, and a knowledge integration project as data sources so that we are able to compare the intended and the implemented MOOC integration effort in more depth.
This project is part of the University of Toronto MOOC Research Initiative, funded by the Bill & Melinda Gates Foundation. We would like to acknolwdge professor Dilip Soman and Poornima Vinoo from Rotman School of Management (University of Toronto) whose collaboration with the curriculum design team made this study possible. We are immensely grateful to Laurie Harrison, Stian Håklev, and William Heikoop for their assitance with data extraction. We would also like to thank the anonymous reviewers for providing feedback on our submission.
Barbour, M. K., & Mulcahy, D. (2013). Examining enrollment trends in schools participating in online learning in Newfoundland and Labrador. The Morning Watch, 41(1-2), 66-76.
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying learning in the worldwide classroom: Research into edX’s first MOOC. Research & Practice in Assessment, 8, 13-25. Retrieved from http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF2.pdf
Bruff, D. O., Fisher, D. H., McEwen, K. E., & Smith, B. E. (2013). Wrapping a MOOC: Student perceptions of an experiment in blended learning. MERLOT Journal of Online Learning and Teaching, 9(2), 187-199. Retrieved from http://jolt.merlot.org/vol9no2/bruff_0613.htm
Cannesa, E., & Pisani, A. (2013). High school open on-line courses (HOOC): A case study from Italy. European Journal of Open, Distance and E-Learning, 16(1), 131-140.
Daza, V., Makriyannis, N., & Rovira Riera, C. (2013). MOOC attack: Closing the gap between pre-university and university mathematics. Open Learning: The Journal of Open, Distance and e-Learning, 28(3), 227-238.
DeBoer, J., Stump, G. S., Seaton, D., & Breslow, L. (2013). Diversity in MOOC students’ backgrounds and behaviors in relationship to performance in 6.002 x. In Proceedings of the Sixth Learning International Networks Consortium Conference.
Grover, S., Pea, R., & Cooper, S. (2014). Promoting active learning & leveraging dashboards for curriculum assessment in an OpenEdX introductory CS course for middle school. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 205–206). ACM.
Johnston, S., & Barbour, M. K. (2013). Measuring success: Examining achievement and perceptions of online advanced placement students. American Journal of Distance Education, 27(1), 16-28.
Liu, F., & Cavanaugh, C. (2012). Factors influencing student academic performance in online high school algebra. Open Learning: The Journal of Open, Distance and e-Learning, 27(2), 149-167.
Schofield, E., Erlinger, M., & Dodds, Z. (2014, March). MyCS: CS for middle-years students and their teachers. In Proceedings of the 45th ACM technical symposium on Computer science education (pp. 337-342). ACM.
Vihavainen, A., Luukkainen, M., & Kurhila, J. (2013, October). MOOC as semester-long entrance exam. In Proceedings of the 13th annual ACM SIGITE conference on Information technology education (pp. 177-182). ACM.
© Najafi, Evans, Federico