Olaf Zawacki-Richter, Eva Maria Bäcker, and Sebastian Vogt
FernUniversität in Hagen (Germany)
This paper presents a review of distance education literature to describe the status thereof and to identify gaps and priority areas in distance education research based on a validated classification of research areas. The articles (N = 695) published in five prominent distance education journals between 2000 and 2008 were reviewed for this study. The conclusion is that distance education research is strongly dominated by issues related to instructional design and individual learning processes; whereas, other important areas (e.g., innovation and change management or intercultural aspects of distance learning) are dreadfully neglected. There is a significant trend towards collaborative research and more qualitative studies. Over 80% of all articles originate from only five countries.
Keywords: Review of distance education research; research areas; Delphi study.
Research on distance education has been subject to harsh and consistent critique (Berge & Mrozowski, 2001; Bernard et al., 2004; Perraton, 2000; Saba, 2000). Moore (1985) stated that there is “a massive volume of amateur, unsystematic, and badly designed research producing information of very little value” (p. 36). Panda (1992) analysed the Indian distance education literature and concluded that “most of the studies are either descriptive status surveys or experimental studies with poor methodological footing” (p. 322). Saba (2000) criticises the lack of theoretical underpinnings: “Research questions are rarely posed within a theoretical framework or based on its fundamental concepts and constructs” (p. 2). He is supported by Perraton (2000): “An examination of existing research shows that it is often atheoretical and predominantly descriptive” (p. 1). Is this really the case?
Lee, Driscoll, and Nelson (2004) emphasise that “understanding trends and issues in terms of topics and methods is pivotal in the advancements of research on distance education” (p. 225). The structure of a research discipline forms the foundation for identifying gaps and priority areas (Mishra, 1998, p. 281). Based on a validated classification of research areas in distance education through the systematic analysis of expert responses in a Delphi study (Zawacki-Richter, 2009), we conducted a literature review of previous distance education research published in five prominent distance education journals between 2000 and 2008 (N = 695 articles).
The aim of this paper is to address questions in the following three areas:
Based on the review of research areas and trends, the results can be used to explore the body of knowledge in distance education and to identify priority areas for future research projects.
Five journals were reviewed for this study: Open Learning (OL), Distance Education (DE), the American Journal of Distance Education (AJDE), the Journal of Distance Education (JDE), and the International Review of Research in Open and Distance Learning (IRRODL). They were selected because of their reputations as the most prominent and recognized journals in the field of distance education. With the exception of IRRODL, which was released for the first time in 2000, the journals have been used as data sources in previous studies (cf. Berge & Mrozowski, 2001; Lee, Driscoll, & Nelson, 2004; Koble & Bunker, 1997; Mishra, 1997; Rourke & Szabo, 2002; Scriven, 1991). Furthermore, the five journals were selected because of their wide scope of distance education research in contrast to more specialized journals such as the Online Journal of Distance Learning Administration or the International Journal of Distance Education Technologies.
All articles published between 2000 and 2008 in the five journals were reviewed (N = 695). Berge & Mrozowski (2001) presented a large scale review with articles published between 1990 and 1999. Therefore, the year 2000 was chosen as cut-off date for this study.
Twenty articles (2.9%) were published in French (in the Journal of Distance Education) and 675 articles (97.1%) were published in English.
In order to prepare this review, an international Delphi study was carried out in 2008 to develop a classification system for research areas in distance education. The Delphi technique was selected to develop a consensus among a group of experts on common areas that are or should be covered in distance education research. The essential element in the Delphi process is anonymity of participants when giving their opinion. The Delphi method, then, alleviates problems that could be caused by domination of the group by a few prestigious or powerful individuals (Charlton, 2004). According to Isaac and Michael (1995), the Delphi method of group interaction avoids the following disadvantages of face-to-face discussions: the bandwagon tendency, the vulnerability to manipulation, and the reticence on the part of individuals to change their minds in front of others.
The final expert panel comprised 25 individuals from 11 countries (Australia, Brazil, Canada, China, Fiji, Germany, Ireland, New Zealand, South Africa, UK, and USA). Based on a literature review and a qualitative analysis of the responses from the panellists, three broad meta-levels of distance education research were derived:
Within these three levels, the research issues that are considered important by the experts can be categorized into 15 research areas. They are briefly characterized below. A detailed description of the method and the results of this pilot study is published in Zawacki-Richter (2009).
All 695 articles published between 2000 and 2008 in the five journals (OL, DE, AJDE, JDE, and IRRODL) were coded according to this classification scheme. Table 2 provides an overview of sample studies. They are representative of a certain research area and research method.
The majority of distance education research can be classified broadly as quantitative, qualitative, or of mixed design (triangulation), which employs both quantitative and qualitative approaches. For this review the research methods were classified as quantitative, qualitative, triangulation, or other (cf. Grant, Ward, & Rong, 1987).
Articles that used statistical analysis were classified as quantitative, from simple methods such as chi-square analysis to multivariate techniques. Qualitative studies were data-based articles that did not quantify data beyond frequency counts. Included in this category were case studies, interpretive ethnographies, grounded theory, and phenomenological studies. Other articles were usually descriptive, not data-based, theoretical papers.
Thirty articles were randomly selected to evaluate the coding decisions of the three coders (A, B and C) to determine inter-rater reliability using Cohen’s kappa (κ) (Cohen, 1960), which is a coefficient for the degree of consistency among raters based on the number of codings in the coding scheme (Neumann, 2007, p. 326). Kappa values of .40 to .60 are characterized as fair, .60 to .75 as good, and over .75 as excellent (Fleiss, 1981; Bakeman & Gottman, 1997).
Coding consistency for the classification of research methods between rater A and B was κ = .855; between rater A and C, it was κ = .855; and between rater B and C, it was κ = .930 (median = .855). For the classification of research areas, coding consistency between rater A and B was κ = .672; between rater A and C, it was κ = .675; and between rater B and C, it was κ = .675 (median = .675). Therefore, inter-rater reliability can be considered as excellent for the coding of methods and as good for the coding of research areas.
The main source of coding discrepancies is the considerable overlap between some research areas so that a unambiguous classification of research areas was not easy in all cases. Some areas are investigated on different levels. Cross-sectional fields are those concerned with issues that refer to quality assurance and evaluation, educational technologies, and cross-cultural aspects (Zawacki-Richter. 2009). For example, cross-cultural aspects have to be considered in international co-operation in the global education market (macro level) and in the planning, implementation, and management of transnational programmes (meso level). Cross-cultural aspects also have implications for competencies required by distance educators (professional development and faculty support - meso level). Intercultural communication plays an important role in classes with mixed cohorts of students from all over the world (micro level).
In order to identify significant trends between the years 2000 to 2008 in the development of research areas, in the application of research methods, and with regard to cooperation among researchers (single vs. multiple-author papers), the frequencies of research areas, methods, and number of authors in the 695 articles under review were ranked for each year and correlated with the years. Significant rank correlations on the 5% confidence level are interpreted as significant linear positive or negative trends. Due to the small number of years (N = 9), Spearman’s rho (ρ) was chosen as the non-parametric correlation coefficient.
Table 3 reveals a strong imbalance of research areas covered in the publications: The micro-perspective (learning and teaching in distance education) is highly over-represented. Over 50% of all papers deal with the top three issues, i.e., interaction and communication in learning communities (17.6%), instructional design (17.4%), and learner characteristics (16.3%).
No significant linear positive or negative trend of research issues can be reported for the period between 2000 and 2008. With only 13 articles in research area 5 (research methods in distance education), the number of articles increased slightly, but significantly, at a low level (Spearman’s ρ = .69, p < .05). Research area 6 (management and organisation) is strongly correlated with area 7 (costs and benefits), ρ = .88, p < .01., which might indicate that these two categories could be merged.
Table 4 shows several noticeable peaks in certain research areas that were due to special issues on the topic, e.g., the special issue on “The Hybridzation of Higher Education: Cross National Perspectives” (area 3), edited by Peter S. Cookson in IRRODL (2002, Vol. 2, Issue 2), the special issue on “Challenges and Possibilities for Academics and Tutors at Open and Distance Learning Environments” (area 10), edited by Heather Kanuka in IRRODL (2006, Vol 7., Issue 2), the special issue on “Ethics in Open and Distance Learning” (area 1), edited by Anne Gaskell in Open Learning (2007, Vol. 22, Issue 2), and the special issue on “Technology, Policy, and the Right to Education” (area 1), edited by Barbara Spronk in IRRODL (2008, Vol. 9, Issue 1).
Distance education, in particular, and the teaching and learning process, in general, are complex matters. Many variables are involved in instructional settings, not to mention other elements involved in distance education, such as social, organizational, technical, and global issues affecting the theory and practice in the field.
Getting beyond the question of “significant differences,” experts make a plea for more qualitative approaches in distance education research (cf. Minnes, 1985; Saba 2000) to capture a deeper and richer range of data. Garrison and Shale (1994) make a case for mixed methods research: “Researchers are realizing that in practice the methodologies can be viewed as complementary....Researchers who advocate combining quantitative and qualitative methods are thus on solid epistemological ground” (p. 25). The link between qualitative and quantitative research is therefore often termed triangulation (cf. Neumann, 2007, p. 149). This approach has the advantage that a complex research field such as distance education can be explored from different perspectives (or angles), utilizing different instruments and methods, and the data gathered can be used to mutually validate the results.
However, what is the current status of the practice of distance education research? Table 5 shows the frequency tabulation regarding the methods applied in published studies. Interestingly there seems to be a trend towards more empirical research. Only 38.1% of all articles are descriptive in nature (“other”) and 12.9% followed a mixed-method design (triangulation). However, this trend is not significant for the investigated period between 2000 and 2008 in the five journals (see below). In contrast to the results of this study, Berge & Mrozowski (2001) classified 75.9% of articles published in journals as descriptive (N = 727 articles; AJDE, DE, JDE, and OL between 1990 and 1999). In his review of 361 articles published between 1991 and 1996 in AJDE, DE, JDE, and the Indian Journal of Open Learning, Mishra (1997) reported a percentage of 47.6 % of descriptive papers.
Table 5 furthermore reveals that AJDE prefers quantitative studies: 63.4% of all articles published between 2000 and 2008 in AJDE followed a quantitative design. The journal that accepted by far the highest percentage of qualitative studies is DE (29.5%). The journal with the most papers that applied a mixed-method approach is JDE (28.1%). IRRODL (56.6%) and OL (48.1%) are the journals with the highest number of descriptive or theoretical papers (“other”). The association between journals and research methods is highly significant: χ2 = 142.35, df = 12, p < .001. However, the association is modest at Cramer’s V of .26 (p < .001).
No significant trend towards more quantitative or mixed research designs or fewer descriptive studies can be reported (quantitative methods: Spearman’s ρ = -.43, p = .250; triangulation: ρ = -.10, p = .796; other: ρ = .24, p = .542). However, there is a significant positive trend towards more qualitative research (ρ = .73, p < .05).
Research into issues on the macro level (distance education systems and theories) is very descriptive, e.g., research area 3 (distance teaching systems and institutions, 77.4%), or research area 1 (issues of access, equity and ethics, 74.2%). As could be expected, the highest percentage of theoretical papers was found under research area 4 on theories and models of distance education (79.2%). The highest frequency of qualitative approaches can be reported for research area 14 (interaction and communication in learning communities, 33.6%). A large number of quantitative studies are carried out in research area 12 (quality assurance, e.g., development and application of instruments for program evaluation, analysis of drop-out, completion/non-completion in distance learning programs, etc., 53.7%), and in research area 15 (learner characteristics, e.g., studies on learning styles, learner preferences etc., 53.1%).
With 15 research areas on three broad levels, as described in the Classification of Research Areas section, the complexity of the distance education discipline is enormous, and it continues to grow with the application of new and emerging information and communication technologies and their implications for educational systems and organizational change. Researchers are more and more involved in international collaborative projects. This phenomenon has accelerated during the last decade through the massive proliferation of computer-mediated communication and the development of new web applications which facilitate collaboration, social interaction, and negotiation of meaning (e.g., via social software or Web 2.0 applications). As an indicator of the extent of collaboration among researchers, the number of contributors per paper was analyzed.
From 2000 to 2008, the percentage of single author articles decreased while the percentage of multiple author articles increased. There is a significant positive trend for multiple author articles (Spearman’s ρ = .70, p < .05) and a significant negative trend for single author articles (Spearman’s ρ = -.73, p < .05).
This positive trend since 2000 towards collaboration in distance education research is supported by the fact that Mishra (1997) reported a much lower percentage of multiple author papers of 38.5% for the period between 1991 and 1996 (N = 361 articles).
The total number of different authors who contributed to the 695 articles in this study was 1138. Table 9 lists the 48 authors who contributed to at least three articles. They are from Australia (5), Canada (17), China (5), Israel (2), Japan/South Korea (1), New Zealand (1), UK (8), and USA (9). Canada is very successful in distance education research, with 17 leading contributors coming from this country. Nine of these 17 researchers are affiliated with Athabasca University.
The analysis in Table 10 depicts that 385 (55.4%) of the first authors were men and 310 (44.6%) were women. The results seem to confirm the stereotypical view that female researchers (59.4%) are more likely than males (40.6%) to choose qualitative methods or to combine quantitative and qualitative methods (females, 61.1%). There is a highly significant association between gender and research methods: χ2 = 35.34, df = 3, p < .001. However, the association is modest at Cramer’s V of .23 (p < .001).
For the analysis of the country-wise distribution of articles, the country of origin of the first author was taken into consideration (53 countries). Table 11 shows that the vast majority of articles (80.3%) come from only five countries: USA, Canada, UK, Australia, and China. Given the national bias of journals, it is no wonder that journals tend to publish more from their own country of origin. For example, the stated intent of AJDE is to explore “the great new field of study, research, and practice that is distance education in the Americas” (Moore, 1987, p. 1). This goal is truly met: AJDE in particular, and also JDE, have a strong North American focus: 80.4% of papers published in AJDE and 71.1% of papers published in JDE are contributed by US-American or Canadian authors respectively. Koble and Bunker (1997) found that 69.8% of authors in AJDE between 1987 and 1995 were from the USA and 20.9% were from Canada. Similar results are also reported in earlier studies by Calvert (1995) and Mishra (1997).
The most international journal is IRRODL with only 18.9% of authors from Canada, followed by DE with 20.5% of authors from Australia, and OL with 42.2% of authors from the UK. Papers published in IRRODL come from 34 different countries, followed by DE (25 countries), OL (24 countries), JDE (13 countries), and AJDE (only 7 countries).
In 1985 Michael G. Moore, the editor of AJDE, lamented the lack of references to previous research and theoretical frameworks: “I believe the convention of reporting previous research as preface to current research is too neglected today” (Moore, 1985, p. 37).
The incidence of unreferenced articles and the number of references per article is taken as an indicator of the scholarliness of a journal (cf. Cline, 1982, p. 210; Mishra, 1997). Articles providing fewer than 10 references imply that “scholarship does not exist but is irrelevant or exists relevantly but is unknown” (Price, 1970, p. 8). Taking this as a benchmark, as was to be expected, all journals under review are on a high academic level, with a mean of 29 references per article. During the period of 2000 to 2008, only one paper out of 695 was published without any references in AJDE (Osiakwan & Wright. 2001). The record holder is McGreal (2004) with 128 references in a paper in IRRODL on copyright issues.
However, this point should not be pushed too hard. We do not claim that papers with 20 references are more scholarly than those with only 15, or that all articles with 20 references are of similar scholarliness.
This study examined (1) research areas in distance education, trends, priority areas, and gaps in distance education research; (2) research methods in distance education; and (3) authorship patterns. The results of this review convey certain implications for future research in distance education.
Major findings of this study may be summarized as follows:
According to the experts’ opinion in the Delphi study (Zawacki-Richter, 2009), there is a great need for more research on the role of culture and cultural differences in global distance learning programmes. Furthermore, co-operation among institutions should receive much more attention, including the impact of cultural differences on leadership and culturally complex student support systems, and there is a need for comparative research on distance learning systems (macro level). A lot of work still needs to be done on the meso level: In particular, experts on the panel highlighted aspects referring to leadership in distance education and strategy, management of change and innovation, costs, organizational development and infrastructure for online student and faculty support, professional development, and quality assurance. The experts claimed that empirical evidence is lacking on the pedagogical opportunities that Web 2.0 applications, mobile devices, and synchronous tools afford for teaching, learning, and assessment.
This review of 695 articles published in five leading distance education research journals confirms the results of the pre-study. It reveals a strong imbalance between the three research levels. Distance education research is highly dominated by issues that refer to the micro perspective (teaching and learning in distance education), with over 50% of all articles focusing on interaction and communication in learning communities, instructional design, and learner characteristics. As anticipated according to the results of the pre-study, those areas that were said to require much more attention take the last three places (globalisation of education and cross-cultural aspects, innovation and change, and costs and benefits) in the ranking of research areas with regard to the frequency of articles (Table 3).
A possible interpretation for this imbalance is that the selection of research themes might follow practical considerations, especially with regard to the availability of data. Of course, the analysis of interaction patterns in computer-mediated communication is a very interesting topic and the text-based data of synchronous communication in online conferences is saved in databases of learning management systems and is therefore readily available. In contrast, it is not surprising that educational institutions, as competitors in the global education market, are unwilling to freely share business models and data on their budgets and costs.
However, the dearth of articles dealing with issues especially on the meso level (management, organization, and technology) is disappointing. In order to guide practice, practitioners in the field should not rely on under-informed trial and error, but on sound research and empirical investigation on the effectiveness of managerial interventions for education innovation, diversity management, student and faculty support, quality assurance, course design, and intercultural communication.
Bakeman, R., & Gottman, J. M. (1997). Observing interaction - an introduction to sequential analysis. Cambridge: Cambridge University Press.
Berge, Z., & Mrozowski, S. (2001). Review of research in distance education. American Journal of Distance Education, 15(3). 5-19.
Bernard, R. M., Abrami. P. C., Lou. Y., & Borokhovski, E. (2004). A methodological morass? How we can improve quantitative research in distance education. Distance Education, 25(2), 175-198.
Calvert, J. (1995). Mapping knowledge in distance education. In D. Sewart (Ed.), One World. Many Voices: Quality in Open and Distance learning - 17th World Conference Proceedings (pp. 384-388). Milton Keynes: ICDE.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
Charlton, J. R. H. (2004). Delphi technique. In M. S. Lewis-Beck, A. Bryman & T. F. Liao (Eds.), The SAGE encyclopedia of social science research methods (Vol. 1). Thousand Oaks: Sage.
Cline, G. S. (1982). College and research libraries: Its first forty years. College and Research Libraries, 43(3), 208-232.
Fleiss, J. L. (1981). Statistical methods for rates and proportions. New York: Wiley.
Garrison. D. R., & Shale, D. (1994). Methodological issues: Philosophical differences and complementary methodologies. In D. R. Garrison (Ed.), Research perspectives in adult education (pp. 17-37). Florida: Krieger.
Grant. L., Ward. K. B., & Rong, X. L. (1987). Is there an association between gender and methods in sociological research? American Sociological Review, 52, 856-862.
Isaac, S., & Michael, W. (1995). Handbook in research and evaluation. San Diego: Educational and Industrial Testing Services.
Koble, M. A., & Bunker, E. L. (1997). Trends in research and practice: An examination of The American Journal of Distance Education, 1987-1995. American Journal of Distance Education, 11(2), 19-38.
Lee, Y., Driscoll, M. P., & Nelson, D. W. (2004). The past, present, and future of research in distance education: Results of a content analysis. American Journal of Distance Education, 18(4), 225-241.
McGreal, R. (2004). Stealing the goose: Copyright and learning. International Review of Research in Open and Distance Learning, 5(3).
Minnes, J. R. (1985). Ethnography, case study, grounded theory, and distance education research. Distance Education, 6(2), 189-198.
Mishra, S. (1998). Distance education research: A review of its structure. methodological issues and priority areas. Indian Journal of Open Learning, 7(3), 267-282.
Mishra. S. (1997). A critical analysis of periodical literature in distance education. Indian Journal of Open Learning, 6(1&2), 39-54.
Moore, M. G. (1987). Words of welcome and intent. The American Journal of Distance Education, 1(1), 1-5.
Moore, M. G. (1985). Some observations on current research in distance education. Epistolodidaktika, 1, 35-62.
Neumann, W. L. (2007). Social research methods: Qualitative and quantitative approaches. Boston: Pearson.
Panda, S. (1992). Distance educational research in India: Stock-taking, concerns and prospects. Distance Education, 13(2), 309-326.
Osiakwan, C., & Wright, D. (2001). Distance training for operating equipment: A cost-benefit and return-on-investment analysis. American Journal of Distance Education, 15(1), 69-79.
Perraton, H. (2000). Rethinking the research agenda. International Review of Research in Open and Distance Learning, 1(1).
Price, D. J. (1970). Citation measure of hard science, soft science, technology and non-science. In C. E. Nelson (Ed.), Communication among scientists and engineers (pp. 3-22). Lexington: Heth Lexington Books.
Rourke, L., & Szabo, M. (2002). A content analysis of the Journal of Distance Education 1986-2001. Journal of Distance Education, 17(1), 63-74.
Saba, F. (2000). Research in distance education: A status report. International Review of Research in Open and Distance Learning, 1(1).
Scriven. B. (1991). Ten years of 'distance education'. Distance Education, 12(1), 137-153.
Zawacki-Richter, O. (2009). Research areas in distance education: A Delphi study. International Review of Research in Open and Distance Learning, 10(3).
Al-Harthi, A. S. (2005). Distance higher education experiences of Arab Gulf students in the United States: A cultural perspective. International Review of Research in Open and Distance Learning, 6(3), 1-14.
Bernard, R. M., Abrami. P. C., Lou. Y., & Borokhovski, E. (2004). A methodological morass? How we can improve quantitative research in distance education. Distance Education, 25(2), 175-198.
Beyth-Maron,. R., Harpaz-Gorodeisky, G., Bar-Haim, A., & Godder, E. (2006). Identification, job satisfaction and work motivation among tutors and the Open University of Israel. International Review of Research in Open and Distance Learning, 7(2), 1-13.
Dearnley, C., & Matthew, B. (2000). A group of nurses experience open learning: Exploring the impact. Open Learning, 15(2), 191-206.
Garrison. D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.
Giguère, L. (2007). Benchmarking course completion rates: A method with an example from the British Columbia Open University. Journal of Distance Education, 22(1), 73-86.
Jeong, A. (2006). Gender interaction patterns and gender participation in computer-supported collaborative argumentation. American Journal of Distance Education, 20(4), 195-210.
Jones, J. G. (2008). Issues and concerns of directors of postsecondary distance learning programs regarding online methods and technologies. American Journal of Distance Education, 22(1), 46-56.
Jung, I. (2005). Cost-effectiveness of online teacher training. Open Learning, 20(2), 131-146.
Lara, L., Howell, R., Dominguez, J., & Navarro, J. (2001). Synchronous and asynchronous interactions of bilingual Hispanic pre- and in-service teachers in distance learning. American Journal of Distance Education, 15(3), 50-67.
Lee, M. J. W., & Chan, A. (2007). Pervasive, lifestyle-integrated mobile learning for distance learners: An analysis and unexpected results from a podcasting study. Open Learning, 22(3), 201-218.
Mishra, S. (2005). Roles and competencies of academic counsellors in distance education. Open Learning, 20(2), 147-159.
Moisey, S. D., Neu. C., & Cleveland-Innes, M. (2008). Community building and computer-mediated conferencing. Journal of Distance Education, 22(2), 15-42.
Morgan, J., Rawlinson, M., & Weaver, M. (2006). Facilitating online reflective learning for health and social care professionals. Open Learning, 21(2), 167-176.
Muilenburg, L. Y., & Berge. Z. L. (2005). Student barriers to online learning: A factor analytic study. Distance Education, 26(1), 29-48.
Rye, S. A., & Zubaidah, I. (2008). Distance education and the complexity of accessing the Internet. Open Learning, 23(2), 95-102.
Shea, P., Pickett, A., & Sau Li, C. (2005). Increasing access to higher education: A study of the diffusion of online teaching among 913 college faculty. International Review of Research in Open and Distance Learning, 6(2), 1-27.
Wang, T. (2005). Tensions in learner support and tutor support in tertiary web-based English language education in China. International Review of Research in Open and Distance Learning, 6(3), 1-18.
Williams, P. E. (2003). Roles and competencies for distance education programs in higher education institutions. American Journal of Distance Education, 17(1), 45-57.
Zhang, W.-Y., & Shin, N. (2002). Imported or indigenous? A comparative study of three open and distance education models in mainland China, India and Hong Kong. Open Learning, 17(2), 167-176.