|
|
|
Jonatan Castaño-Muñoz, Teresa Sancho-Vinuesa, and Josep M. Duart
Universitat Oberta de Catalunya (UOC), Spain
Online interaction is considered to be a key aspect of effective e-learning and improved academic achievement. However, few studies have examined how effectiveness varies with the degree of interaction intensity. Using data for 17,090 students from three Catalan universities, in this paper we study the productivity associated with five different levels of interaction intensity in learning. We also compare the results obtained for fully online education with those for face-to-face learning. The analyses show that interaction in online education has diminishing returns, while in face-to-face learning it does not do so in a pure way. These results have implications for determining the optimum level of interaction that should be sought when designing courses and educational policies.
Keywords: Online interaction; academic achievement; higher education
Interaction is usually defined as a two-way communication process requiring two objects and two actions (Moore, 1989; Muirhead & Juwah, 2004; Wagner, 1994). With the focus on people’s learning, interactions can occur in formal and informal educational contexts alike (Anderson, 2003). Interactions in formal educational contexts mainly take place between any combination of two of the following three elements: students, lecturers, and content (Anderson, 2003; Gunawardena & McIsaac, 2004; Miyazoe & Anderson, 2010a).
The use of the Internet to strengthen student interaction is considered to be a key aspect of the effective incorporation of e-learning into higher education (LaPointe, 2003; Swan, 2004; Zhao, Lei, Yan, Lai, & Tan, 2005). Moreover, it has recently been shown that online interaction is the reason why students on face-to-face courses obtain better results (Castaño-Muñoz, Duart, & Sancho-Vinuesa, 2013; Means, Toyama, Murphy, Bakia, & Jones, 2009).
Despite the importance of online interaction, to date there have been few analyses of the varying effects of different levels of such interaction on academic outputs (e.g., satisfaction or academic achievement).
In an early approach to this topic, Anderson theorised on the possibility that by adding together degrees of interaction in distance education, students may gradually derive less satisfaction. This idea is reflected in Anderson’s equivalency theorem (2003), later renamed interaction equivalency theorem (Miyazoe & Anderson, 2010). Specifically, the second thesis of this theorem postulates that for distance education, “High levels of more than one of these three modes will likely provide a more satisfying educational experience, although these experiences may not be as cost or time effective as less interactive learning sequences” (Anderson, 2003).
Miyazoe empirically addressed this theorem in an unpublished study (see Miyazoe & Anderson, 2010b) on a sample of 236 students from Japan and Taiwan. This study analysed the preferred types of interaction depending on learner characteristics, learning modes, and content orientation. The results show that majority student satisfaction can only be achieved with just one type of interaction; when more types are added, fewer students are satisfied and returns diminish as a result. In addition, the study found different preferences in different learning modes, thus raising the need to differentiate between face-to-face, online, and blended learning in the analysis of online interaction.
Regarding academic achievement, Bernard et al. (2009) performed a meta-analysis of the results from 77 studies and applied the idea of diminishing productivity to the analysis of the impact of online interaction on academic achievement. The result of the aforementioned study provides empirical evidence of the fact that in courses with the highest levels of internet-mediated interaction, the academic achievement returns do not increase proportionately with the level of interaction. In fact, the returns increase to a lesser extent.
This result is especially important because of its extensive and pioneering nature. However, despite the methodological strength of the approach, there are three points of Bernard’s analysis that constitute limitations:
In this research paper, the aim is to overcome some of the aforementioned methodological limitations. To do that, we shall go a stage further and separately analyse how the academic achievement returns vary with the level of interaction on a scale of five intensities, incorporating into the analysis those courses that, while taught mainly in face-to-face mode, use internet-mediated interaction as a way of improving learning and making a distinction between study mode (fully online vs. face-to-face).
By doing so, the aim is to contribute to a field in which relatively little is known: the analysis of the productivity of internet use for interactive learning. In addition, we shall go a stage further and check the limits of the interaction equivalency theorem by providing empirical data on academic achievement.
However, unlike the study undertaken by Bernard et al., in this study, of the three types of interaction in learning that are habitually analysed (student-student, student-lecturer, and student-content), only interactions with people — and not with content — will be taken into account.
Having information available on this aspect may be of great importance when it comes to establishing the optimum amount of resources (including study time) that need to be invested in internet-mediated human interaction and — from the viewpoint of teaching staff and of institutions — in the facilitation and promotion of interaction to attain greater academic achievement. Despite that, it should be borne in mind that the decision to use internet-mediated interaction does not depend solely on a particular university.
An online survey was used to obtain the data for the research. In 2006, the survey was sent out to students attending three Catalan universities of different types. Two face-to-face universities — one generalist (University of Barcelona, UB) and the other technical (Universitat Politécnica de Catalunya, BarcelonaTech, UPC) — and one online (Open University of Catalonia, UOC). These data were complemented with information on academic achievement from the Government of Catalonia’s administrative registers for the face-to-face universities, and from the university’s own registers for the online university. It has to be noted that all the face-to-face universities analysed are public universities with such comparable characteristics as tuitions, so the effect that price could have in academic achievement is controlled when comparing between them. In online mode we only use one semi-public university.
The method for collecting data only allowed the selection of those students who had not dropped out of their studies and who were internet users. This served to filter out only those students who could be analysed.
Information was available on a total of 17,090 students, 8,046 from the face-to-face universities (5,452 from the UB and 2,594 from the UPC) and 9,044 from the online university. The characteristics of the students were similar to the distribution of the population (Table 1), except for academic achievement (due to the fact that the method for collecting data eliminated students who were not taking courses).
In order to measure online interaction, a set of dichotomous questions was employed to establish the purposes of students’ internet use. In particular, questions were asked on four topics. These were as follows:
When studying a course subject, what do you use the Internet for?
So, while adhering to the definition given earlier, we asked about every use regarding the active and collective creation of curricular knowledge through two-way communication with another person or other people, whether lecturers, students, or other internet users interested in a particular subject. The third potential agent in the interactive process — content — was therefore not considered (Anderson, 2003; Moore, 1989). The possible difference in quality of the content of face-to-face and online models doesn’t suppose a problem for the comparison between them because previous analysis has shown that mere interaction with content doesn’t have significant effects in academic achievement (Castaño-Muñoz; Duart, & Sancho-Vinuesa, 2013). It allows us to assume a similar quality level of the accessible material.
In order to draw up an index to reflect the level of interaction, the number of uses from those described previously was added up for each category, giving, as a result, an ordinal variable with five categories (zero to four uses). Underlying the creation of an index of this type is the assumption that the higher a student’s number of uses, the greater the interaction intensity in the teaching-learning process. The distribution of the intensity indicator by study mode is shown in Figure 1.
As shown in the figure, interactive uses of the Internet are not particularly widespread in face-to-face universities. Of the students at such universities, 9.67% do not use the Internet to communicate with anyone and 22.59% only make one of the proposed uses, mostly to communicate with lecturers, while in the case of individual uses there is practically no-one who does not make at least two. In addition, for the purpose of future analysis, it should be noted that in the case of face-to-face universities, the distribution of the variable measuring the level of interactive use of the Internet for education is similar to the normal distribution, whereas in the case of the online university, it is rising.
In order to measure academic achievement, information contained in the administrative registers of the universities was used: information on the number of credits (courses) taken and the number of credits passed per student on this course. Credit data were aggregated by year and therefore both semesters of this course count. In addition, a decision was taken not to measure credits for which official recognition (and therefore exemption) had been obtained.
An analysis of the number of credits for which the students had enrolled highlights the fact that those at face-to-face universities enrol for many more ordinary credits per course than students at the UOC. The mean among the former is 60.87 credits, whereas the figure at the UOC is half that number: 30.35 credits, with a lower standard deviation (Table 2).
In face-to-face mode, of the 60.87 taken credits, students pass a mean of 45.06, whereas at the UOC, of the 30.35 enrolled credits, students pass a mean of 24.3 (Table 3). Once again, the standard deviation is lower at the UOC owing to the fewer credits that students take.
Using the information about ordinary credits taken on the course and ordinary credits passed on the course, an academic achievement rate was calculated by dividing the credits passed by the credits taken and multiplying the result by 100, that is to say, the percentage of credits passed over the credits taken for each student.
Course academic achievement rate = (Ordinary credits passed on the course / Ordinary credits taken on the course) x 100
This academic achievement indicator was chosen because it is one of the official indicators that Catalan universities use. However, in order to ensure the reliability of the results, they were tested against the results that would have been obtained if the achievement indicator had been a measure of effectiveness: number of credits passed per course (variable that would measure effectiveness, whose distribution is similar to the normal distribution).
Table 4 shows the descriptions of the academic achievement rate for the two study modes presented. As shown, UOC students have a slightly greater academic achievement than face-to-face students. In other words, UOC students enrol on fewer courses and, on average, pass 78.26% of them, while students at face-to-face universities take a lot more credits and pass a slightly lower percentage of them (74.057%).
The distributions show that, among our universe of study of active students, academic achievement is high. In both modes, the bulk of students is concentrated in more than 90% of credits passed over credits taken. That is to say, it was found that the credits passed depend greatly on the credits taken, especially in the case of the UOC.
The figures shown suggest that there are two paces of study depending on the educational mode employed to take a degree course. On the one hand, students at face-to-face universities take twice the number of credits in comparison to students at the online university. On the other hand, students at the online university take fewer credits but manage to pass a slightly higher percentage of them. The fact that face-to-face students take more courses than in online learning explains why their pass rates may be lower given a fixed time interval.
In the productive process of any good or service, if there is any variation in terms of inputs quantities, then a variation in the output quantity is to be expected. The productivity of a factor can be defined as the change in the production of an output that occurs as a result of a variation in one of the inputs when the others remain stable.
In order to empirically check whether Anderson’s theorem works in online and face-to-face education, this article analyses the productivity of internet-mediated interaction in the academic achievement production process. In this respect, a regression was performed, where the dependent variable is academic achievement and independent variables are those considered to be fixed (detailed further on) plus the Intensity of interactive use of the Internet for education variable. The last of these variables was introduced as a set of dummy variables and takes the zero-use category as a reference.
Regarding the fixed inputs, several variables were controlled for, including those referring to sociodemographic characteristics, to the student’s relationship with technology, to the type of studies the student is taking, and, more indirectly, to the time available for study.
Taking account of the above, the analyses performed in this research are based on the following equation:
Where:
Achi = Academic achievement
Xij = A vector j of the students’ characteristics: Age, gender (male/female).
BDik = A vector of variables referring to the different relationships that the students have with technology, based on digital divide dimensions (Dimaggio, Hargittai, Celeste, & Shafer, 2004; van Dijk, 2005).
Estim = A set of dummy variables referring to the type of studies that the student is taking and a continuous variable referring to where the student is up to in those studies.
Tin = A vector including variables used as proxies to avoid, in conjunction with the sociodemographic variables, any possible biases stemming from not directly observing the time available for study. The number of credits taken during the course was controlled for in order to measure the student’s subject workload and a dummy variable distinguishing between students who combined their studies with work and those who did not.
IUIndip = Internet Use Index for individual learning (0-5)
Treat_Intensityiq: A series of dummy variables indicating the intensity of internet use for interaction. Intensity zero was taken as the reference to establish the returns of each intensity level, in comparison to students who did not interact via the Internet.
One possible weakness of the model is the lack of a direct measure of the motivation of the student. However it’s possible to affirm that when including a proxy as the grade in the exam of entrance to the university the results of the estimations don’t vary significantly (however it has to be noted that it can be tested only in face-to-face-mode).
To partially correct the possible selection bias in the calculation of effects, the propensity score matching technique was used to match, as closely as possible, the results to the causal effects and to reduce the selection bias and the dependence of parametric models (Ho, Imai, King, & Stuart, 2007).
The data analysed in this paper were divided into different levels of analysis (university, area, and type/duration of studies). If the existence of an intra-class correlation were to be ignored, then it might lead to erroneous results; that is why a decision was taken to use a cluster correction for the standard errors calculation.
By following the aforementioned steps, the increase in the students’ academic achievement brought about by one, two, three, or four interactive uses could be estimated in comparison to students who did not interact via the Internet (assumed to be a zero increase), partially avoiding the selection bias of the variables observed.
The estimates of the productivity of internet-mediated interaction for learning, separating online education from face-to-face education, are shown in Table 5 and, more graphically, in Figure 2.
The data show that the incorporation and promotion of internet-mediated interaction can have a significant repercussion on academic achievement and on the time students take to graduate, especially on longer degree courses. On these courses, students interacting via the Internet achieve a higher percentage of passes per course and these gradually accumulate over more courses.
When comparing the two study modes, it is possible to observe how internet-mediated interaction benefits the online education students. On the one hand, the estimates indicate that, on average, a student studying online who makes four uses passes 18.23% more courses than a student who does not make any, and 9.18% more than a student who only makes one. On the other hand, at face-to-face universities, the difference is lower, since an average student with a maximum-use intensity (four uses) passes 7.57% more credits than a student with a zero-use intensity.
An explanation for this difference is that at the online university the only way of interacting with lecturers and students is via the Internet, whereas at the face-to-face universities, such interaction is more of a complement than a necessary condition for interaction. Therefore, the returns of using it are lower in percentage terms. However, account should also be taken of the fact that online education students on average take on a lot fewer credits than face-to-face education students. Therefore, the number of credits that they pass as a result of online interaction is similar in both study modes.
When analysing the two study modes separately, it is found that in the online mode, there is evidence of diminishing returns: In the lowest intensities of use, the marginal achievements are higher than in the highest intensities.
Shifting from zero to one use significantly increases achievement with a confidence level of just 90%, due to the minimal number of individuals that make zero use or only one use in this study mode. The shift from one to two uses and from two to three uses occurs with a significance higher than 99% in both cases. Finally, the shift from three to four uses occurs with a significance lower than 95%, despite the fact that it is in these categories where almost 70% of the individuals studying at the online university are concentrated.
In the face-to-face mode, it is shown that internet-mediated interaction does not follow a pure diminishing return trend; it only does so from minimal use. The dynamic that breaks the diminishing trend is the fact that in the lowest intensity of use, using the Internet for interaction does not show itself to have any significant returns in comparison to the students who do not use it at all.
Therefore, it is found that productivity is low in a first stage, that the returns increase more than proportionately in a second stage, and, ultimately, that marginal productivity is lower in a third stage. As the results show, in the face-to-face mode, a simple increment of one to two uses is significant at 95% and even at 90%. However, it is worth noting that while neither shifting from two to three nor three to four uses has this degree of significance, increasing the intensity from two to four uses does indeed have it, which demonstrates that although the returns diminish, they do not stagnate.
By comparing the study modes, when going from the second level of intensity there is a trend towards diminishing returns. However, the main difference can be found in the first stage of use (from zero to one use).
In the online mode in the first stage, maximum marginal productivity is estimated at 9.05%. Since there is no possibility of interaction, it is the stage that distinguishes active students from those who either do not take any courses or who do so completely individually without any interaction during their learning process. It is hard to imagine a student actively taking courses fully online and learning in a way that is so individual that it would preclude all of the four proposed modes of interaction. In addition, as the online teaching-learning methodology usually fosters interaction by incorporating it into continuous assessment (Means et al., 2009), it is even harder to imagine the case of a student having zero interaction and, at the same time, good academic results.
In the face-to-face mode, on the other hand, the first stage does not bring any statistically significant returns. In line with the theory of production, this can be interpreted as evidence of the existence of two things: a) a stage of adaptation to internet use and b) an entry cost to the dynamic of using the Internet interactively for learning. This stage can be considered as one of approach, exploration, and learning about internet use for interactive learning, which opens doors and enables students to move towards the intensity of use of subsequent stages, where returns other than zero do indeed exist.
On the basis of the aforementioned results and in accordance with previous literature, it is possible to assert that internet use for interactive learning has a positive impact on academic achievement and that such impact depends on the intensity of such use (Bernard et al., 2009).
In the online mode, ceteris paribus, the productivity associated with interaction intensity follows a diminishing trend from the start. While our analysis only takes account of the interaction between humans and not with content, this trend coincides with the theorem proposed by Anderson for the impact of online interactions on satisfaction in distance education (Anderson, 2003; Miyazoe & Anderson, 2010) and, albeit with different methodologies, with existing empirical research on this topic (Bernard et al., 2009; Miyazoe & Anderson, 2010a), thus making the results obtained more robust.
The diminishing returns found in online education have two possible explanations: First, the limiting factor condition (necessary for production) of internet-mediated interaction for taking courses in the university studied, and, second, because greater knowledge is achieved in initial interactions than in subsequent ones. However, the main novelty that our analyses contribute is the postulation of the existence of differences between the online mode and the face-to-face mode. The possibility of transferring the hypothesis of diminishing returns to the face-to-face mode has been tested and shown to be unfeasible. In the face-to-face mode, the productivity of internet-mediated interaction takes the shape of an S, which is typical of production functions. Therefore, it is not possible to speak of diminishing production from the start. In fact, in the lowest intensities of internet use for interaction, there are no significant academic achievement returns.
The interpretation of this evidence is as follows: There is a stage of exploration and of adaptation to the incorporation of online interaction in education, which means that at the initial levels, students do not fully benefit from it. So this stage needs to be taken into account when it comes to developing the pedagogical design of courses. Indeed, previous literature shows that this stage can have a positive impact on satisfaction (Miyazoe & Anderson, 2010b).
The consequences of these results on policies for implementing the Internet as a tool for interactive learning are several.
Firstly, from the perspective of distance education institutions, account needs to be taken of the fact that, as shown in Figure 1, there are very few students with a low intensity of use for interaction (2.91% do not make any use and 8.6% make only one). However, they should not be dismissed because their weight (weighted by the greater return they would obtain from increasing their intensity) is greater than their numeric weight. In online education, all costs being equal, if a choice needs to be made between fostering the intensity of use among low-use students or high-use students, it is comparatively more effective to concentrate on the former. However, given that these are in the minority, if general policies aimed at fostering interaction among the student population as a whole are carried out, focusing on medium-to-high-use students (the largest group) may be more effective when it comes to improving learning. However, it should be borne in mind that this type of policy would increase inequality among the few students that least use internet-mediated interaction and the others (Castaño-Muñoz, 2010).
From the perspective of face-to-face universities that incorporate the Internet to improve their effectiveness, the results show that it is necessary to ensure that students do not limit themselves to simply discovering this methodology because it does not provide any significant returns. Rather, they need to make certain that students go beyond the minimum-use barrier so that they are at least situated in a medium intensity of use. In the case of such universities, a high number of students that make zero or minimal use certainly exists (9.67% and 22.59%, respectively). So focusing on getting these “soft” users to increase their use to a medium use may be a highly effective policy when it comes to improving the effectiveness of online education. In order to make best use of the Internet, these data show that universities should strive to find out which students make least use of the Internet for interaction in their learning processes. Having established who those students are, universities should then implement programmes to foster internet use. In this respect, compensatory courses focusing on the acquisition of basic digital competencies and the habit of using the Internet for interaction and study purposes might be useful for students lacking them. Likewise, adapting technological tools to the student population (for example, interaction tools that students already use in their daily lives outside the classroom) might be advisable to improve students’ learning.
A limitation on the results presented is that the costs of implementing online interaction do not form part of the analysis. If the costs of shifting from one intensity level to another were constant, then the preceding assertions would be true. But if they required an initial investment and then lower costs, then institutions that want to maximise both effectiveness and efficiency should foster the most intensive use of the Internet for this purpose. In this respect, there are no data available on the cost to students or to institutions of implementing such interaction. Studying such costs should therefore be a future line of research in the field of the effectiveness and efficiency of internet use in education.
Anderson, T. (2003). Getting the mix right again: An updated and theoretical rationale for interaction. International Review of Research in Open and Distance Learning, 4(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/149/230
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243 -1289. doi:10.3102/0034654309333844
Castaño-Muñoz, J., Duart, J. M., & Sancho Vinuesa, T. (2013). The Internet in face-to-face higher education: Can interactive learning improve academic achievement? British Journal of Educational Technology. doi: 10.1111/bjet.12007. Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/bjet.12007/abstract
Castaño-Muñoz, J. (2010). Digital inequality among university students in developed countries and its relation to academic performance. Universities and Knowledge Society Journal (RUSC), 7(1).
Dimaggio, P., Hargittai, E., Celeste, C., & Shafer, S. (2004). From unequal access to differentiated use. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.85.6001
Gunawardena, C. N., & McIsaac, M. (2004). Distance education. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15(3), 199-236. doi:10.1093/pan/mpl013
LaPointe, De. K. (2003). Effects of peer interaction facilitated by computer-mediated conferencing on learning outcomes. Presented at the 19th Annual Conference on Distance Teaching and Learning, Madison, Wisconsin.
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
Miyazoe, T., & Anderson, T. (2010a). The interaction equivalency theorem. Journal of Interactive Online Learning, 9(2).
Miyazoe, T., & Anderson, T. (2010b). Empirical research on learners’ perceptions: Interaction equivalency theorem in blended learning. European Journal of Open, Distance and E-Learning, 1. Retrieved from http://www.eric.ed.gov/ERICWebPortal/detail?accno=EJ911753
Moore, M. G. (1989). Three types of interaction. Editorial. Retrieved from http://www.cde.london.ac.uk/resources/documents/editorials/resource1426.htm
Muirhead, B., & Juwah, C. (2004). Interactivity in computer-mediated college and university education: A recent review of the literature. Educational Technology & Society, 7(1), 12-20.
Swan, K. (2004). Relationships between interactions and learning in online environments. The Sloan Consortium. Retrieved from http://sloanconsortium.org/publications/freedownloads
Van Dijk, J. (2005). The deepening divide: Inequality in the information society. Thousand Oaks, CA: Sage Publications.
Wagner, E. D. (1994). In support of a functional definition of interaction. American Journal of Distance Education, 8(2), 6-29.
Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H. S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107(8), 1836-1884.