|
|
Weiyuan Zhang and Y L Cheng
The University of Hong Kong
E-learning has become an increasingly important teaching and learning mode in educational institutions and corporate training. The evaluation of e-learning, however, is essential for the quality assurance of e-learning courses. This paper constructs a four-phase evaluation model for e-learning courses, which includes planning, development, process, and product evaluation, called the PDPP evaluation model. Planning evaluation includes market demand, feasibility, target student group, course objectives, and finance. Development evaluation includes instructional design, course material design, course Web site design, flexibility, student-student interaction, teacher/tutor support, technical support, and assessment. Process evaluation includes technical support, Web site utilization, learning interaction, learning evaluation, learning support, and flexibility. Product evaluation includes student satisfaction, teaching effectiveness, learning effectiveness, and sustainability. Using the PDPP model as a research framework, a purely e-learning course on Research Methods in Distance Education, developed by the School of Professional and Continuing Education at the University of Hong Kong (HKU SPACE) and jointly offered with the School of Distance Learning for Medical Education of Peking University (SDLME, PKU), was used as a case study. Sixty students from mainland China, Hong Kong, Macau, and Malaysia were recruited for this course. According to summative evaluation through a student e-learning experience survey, the majority of students were very satisfied/satisfied on all e-learning dimensions of this course. The majority of students thought that the learning effectiveness of this course was equivalent, even better, than face-to-face learning because of cross-border collaborative learning, student-centred learning, sufficient learning support, and learning flexibility. This study shows that a high quality of teaching and learning might be assured by using the systematic PDPP evaluation procedure. It is hoped that the PDPP evaluation model and its application can provide a benchmark for establishing a wider e-learning quality assurance mechanism in educational institutions.
Keywords: E-learning; quality assurance; evaluation; PDPP model
E-learning has become widely used in conventional education, continuing education, adult education, and corporate training because of its flexibility, richness, resource-sharing, and cost-effectiveness. United Nations Educational, Scientific and Cultural Organization (UNESCO) statistics show that over 455 million people around the world received education and training through the Internet in 2008. Over 70% of universities in the USA were providing e-learning courses, and more than 6.1 million university students were taking at least one e-learning course during the fall 2010 term, which accounted for over 31% of the total number of university students in the USA (Allen & Seaman, 2011).
The financial crisis in 2009 led to a significant increase in student enrollment in e-learning colleges in mainland China, some of which even had increases of 20%. The continuing education departments in China’s universities are gradually merging with e-learning colleges and providing continuing education programmes to their adult students through e-learning. With the rapid development of information and communication technology, student attitudes towards e-learning are becoming more positive. For example, according to the Survey on the Demand for Continuing Education in Hong Kong 2007-2008, 40.4% of the respondents showed positive attitudes to e-learning when they considered pursuing a continuing education programme, while 83.9% of them hoped to try the blended learning approach.
The economic benefits of e-learning are also becoming visible. For example, China Distance Education Holdings Limited in Beijing launched its first e-learning accounting course about 10 years ago in the name of China E-learning Accounting Website. Now with millions of students enrolled across the country every year, it generates a huge profit for the company, and about a year ago it was listed on the New York Stock Exchange (Chinaacc, 2009). Another example is that The Hong Kong and Shanghai Banking Corporation Limited (HSBC) transformed 30% of its training programmes into e-learning programmes in 2008, which cut its annual training expenditure by 15%; in 2009, HSBC planned to raise the percentage of e-learning training to 50 (Hu, 2009).
With the rapid development of e-learning, there is also an increasing interest in e-learning research. Among all the research topics, quality assurance of e-learning has attracted the greatest concern. Jung et al. (2011) found that various national, regional, and international initiatives have been undertaken with regard to quality assurance in e-learning. Endean et al. (2010) stated that those concerned about online learning have been developing and publishing ideas for over a decade about how to manage the quality of the learning experience of those trying to study through the Internet.
However, Jung and Latchem (2007) found that most institutions apply the same quality criteria for e-learning as for the other modes of delivery. Endean et al. (2010) pointed that new entrants to the e-learning field were unlikely to have existing internal procedures to cover quality assurance of this new mode of operation. The adoption of a set of quality standards that carry some wider recognition addresses the need for internal processes.
Course quality is assured by a series of evaluations, and e-learning should be no exception. In this paper, the authors propose an e-learning course evaluation model for quality assurance and analyze its concrete application using a cross-institutional purely e-learning course.
CIPP, which is a frequently used evaluation model in the field of social sciences, is the acronym for context, input, process, and product evaluation (Zhang & Jiang, 2007). Referring to the CIPP evaluation model and characteristics of e-learning courses, the authors propose a system for evaluating e-learning courses that consists of four evaluation activities: planning evaluation, development evaluation, process evaluation, and product evaluation; in short, the PDPP model.
Based upon the proposed PDPP model and in line with the components and e-learning characteristics, the e-learning evaluation model consists of 26 items (see Figure 1).
Figure 1 shows that the planning evaluation of e-learning courses begins with market demand analysis and feasibility analysis. Market demands refer to needs of target student groups for knowledge and skills in their careers. If courses are job-related, employer perspectives on the essential needs of their employees also need to be considered. If a course is to be exported to other countries, it is necessary to analyze their local educational import policies, regulations, and levels of technical support. Then one needs to analyze the target student group, course objectives, financial issues, and quality assurance mechanism. Analysis of target student groups includes age range, educational background, work experiences, work hours, study time availability, learning motivations, and job requirements. If the exported course is to be jointly launched with local educational institutions, the quality assurance system of the partner institution needs to be analyzed as well.
Development evaluation of e-learning courses involves analyzing every component of course development, including the course blueprint, e-learning platform, course Web site, instructional design, learning resources, assignments and examinations, and tutors.
Development evaluation is a process corresponding to eight activities of e-learning. According to the research findings of Zhang and Wang (2005), evaluating the e-learning teaching process should include the following eight dimensions: overall evaluation, technical support, Web site utilization, student-student interaction, resources utilization, learning evaluation, learning support, and flexibility.
Product evaluation measures the learners’ degree of satisfaction, teaching effectiveness, learning effectiveness, and any other possible additional outcomes. In the end, the sustainability of courses will depend on the results of the abovementioned analyses.
This section describes the application of the PDPP evaluation model through a case study of an e-learning course on Research Methods in Distance Education.
The e-learning course on Research Methods in Distance Education was developed by the School of Professional and Continuing Education at the University of Hong Kong (HKU SPACE) and jointly launched with the School of Distance Learning for Medical Education of Peking University (SDLME, PKU). HKU SPACE was mainly responsible for providing the learning materials including the textbook, e-learning study guide, study units, e-learning courseware, study timeline, selection and training of tutors, quality assurance and copyright protection, and the award of attendance certificates. The SDLME, PKU was responsible for e-learning platform design, development and management, student recruitment and management, tutor management, technical support, and learner support services.
The Research Methods in Distance Education course consists of five units with ten chapters (see Table 1).
The course lasts 10 weeks and the average study hours are about eight per week, 80 hours altogether. All the teaching, learning, activities, quizzes, assignment, and discussion sessions take place in e-learning mode. One tutor is allocated to every 20 students. The students’ learning procedure is as follows: read the study guide; understand the learning objectives; watch the video lectures (streaming media courseware); read the study unit; read related chapters in the textbook; participate in the e-learning discussion; read the summary of e-learning discussion prepared by the tutor, and write assignments.
It is necessary to analyze the market demand, feasibility, course objectives, target student groups, finance, and quality assurance when preparing an e-learning course plan. This is a standardized process in HKU SPACE, which involves writing a course development proposal and filling in various forms that will be discussed and decided upon by different academic boards, the registrar’s office, and the finance department.
This e-learning course has a huge potential market demand in Chinese language speaking areas, especially in mainland China. Considering the shortage of courses in this area in Chinese regions, the main target student group would be academic staff in distance education institutions. For example, under the China Open University System, there is one China Central Radio & Television University (CCRTVU), 44 Radio & Television Universities at provincial level, 1,103 Radio & Television Universities at prefecture and city level, and 1,853 study centres at county level (Development and Planning Office of CCRTVU, 2011).
With the rapid development of distance education and e-learning teaching mode, more and more distance education managers and practitioners are beginning to recognize the importance of research activities in enhancing teaching effectiveness and raising their university’s reputation. Academic research has also become an important index in the evaluation of distance education institutions, but high-quality academic research remains the weakest link in most such institutions. This is mainly because most distance education practitioners lack the necessary knowledge, skills, and experience of educational research. Therefore, they urgently need to acquire that knowledge and study the skills of distance education research in a systematic way.
Since the major market for this course is mainland China, HKU SPACE chose to collaborate with the School of Distance Learning for Medical Education of Peking University in order to achieve good cost-effectiveness. This school is one of the top e-learning education institutions in mainland China. Moreover, it has also had successful experience of cooperating in e-learning courses with the Open University UK.
Students come from Chinese speaking countries or areas like mainland China, Hong Kong, Taiwan, and Macau, among others. Researchers, managers, and teachers from radio and TV universities, e-learning colleges, open universities, and continuing education colleges who satisfy the following two requirements are qualified to study this course: having a college diploma and having two years working experience in the field of distance education.
In most open education institutions in Chinese regions, the academic staff members are encouraged to take on-the-job training in their professional areas. They could apply for sponsorship as well as leave from their own universities. The University could also assign some staff to take relevant courses as a part of staff training and development. It is expected the tuition fee of students would be paid by their institutions.
The objectives of this course are set based upon the analysis of the course content structure, market demand, and target student group. Generally speaking, by the end of the course, the learners will be able to understand the methodology of distance education research; understand the rationale of distance education research; understand the process of distance education research; collect and analyze literature in the field of distance education; conduct research design for distance education research; apply distance education research methods; write a distance education research proposal; and write distance education research papers.
In order to help students arrange study time before the course started, a timeline was proposed to guide students as follows.
The students were asked to use the e-calendar on the course Web site and arrange all 10 week’s study time at the beginning of the 1st week. The study time could be flexible on a weekly basis. The study time which students set would be automatically sent to them as a reminder.
This involves budgeting various expenditures and income. The expenditures consist of five categories, which are course materials development (course content writing, editing, use of copyright materials, translation); e-learning instructional design (working hours of instructional and graphic designers); multimedia design and production (expert teaching, video programme shooting, offline editing, dubbing, use of DV); interactive courseware development; technical development (setting up the e-course platform, designing the course Web site, upload learning resources); and textbook and tutor fees. The income is tuition fees from students. It was estimated that the tuition fee from 60 students of the 1st intake could cover all course design and development expenditures.
This follows the quality assurance systems of HKU SPACE. For a short course, the internal quality assurance includes six parts: course development and approval, course monitoring, course review, quality process review, teaching quality, and teaching and learning support (HKU SPACE, 2009).
The first step in e-learning development evaluation is to analyze the course blueprint, which is compiled and prepared by the programme director. The blueprint deals with the formation of the course team and its members’ roles, course background, course introduction, course objectives, learner analysis, requirements for learning facilities and skills, course modules/units, learning materials, assessment and examination, communication and collaboration in learning, learner support services, teaching model(s), course materials writing schedule, quality assurance, and copyright issues.
When the course blueprint evaluation has been completed, it is followed by analyzing the construction of the e-learning platform and course Web site, instructional design, learning resources, assignment and examination arrangements, and the recruitment and training of tutorial staff. Table 3 lists the e-learning course development and evaluation steps.
Since this course was developed by HKU SPACE, the evaluation of the course blueprint, instructional design, learning resources, assignment and examination procedures followed the course development procedures of HKU SPACE. Outside experts were invited to do an external evaluation. The requirements for external experts were professors in education with at least 10 years experience in distance education research and teaching.
In terms of instructional design, the ADDIE model was adapted with five phases: analysis, design, development, implementation, and evaluation (Jochems, van Merriënboer, & Koper, 2004, p. 55-56). The principle of seven types of interactions was also emphasized, including the interactions between students and interface, between students and teachers, among students, between students and learning content, between students and learning objectives, between students and multimedia learning resources, and between students and time management (Zhang, 2009).
Since this course was launched in collaboration with SDLME, PKU and students were mainly from mainland China, the e-learning platform was developed on Medtime Open Learning System (MOLS), which was designed and developed by SDLME, PKU. The MOLS functions could be classified into five categories: course content functions, communication and collaboration functions, feedback and evaluation functions, assignment and assessment functions, and administration and management functions. Utilizing the MOLS e-learning platform and its technical staff lowered the cost of course development.
Tutors were recruited from mainland China, but HKU SPACE assessed their qualifications.
Process evaluation refers to evaluating the process of course delivery, including the overall evaluation, technical support, Web site utilization, learning interaction, resource utilization, learner support, assessment, and flexibility. Process evaluation mainly uses three approaches: survey of students’ learning experience and feedback; survey of tutors’ opinions; and programme director’s monitoring of the e-learning tutorials.
The approach to understanding students’ learning experience and feedback is as follows: establishing a special feedback area on the course Web site, establishing email communication between tutors and students, and internal reviewer or programme director’s interviews with tutors and students. For example, the students were asked to familiarize themselves with various functions of the e-learning platform in the first week, referring to the course Web site guidance. The students needed to report to their tutor their degree of familiarity and time spent for this purpose. It was found that all the students learnt to use this course platform in two to four hours.
In the middle of this course, the reviewers and the programme director conducted formative evaluation. For example, at the sixth week of this course, individual interviews were conducted and a virtual classroom was organized for evaluators to gather students’ learning feedback, including overall evaluation, learning experiences, difficulties, and suggestions so that timely adjustments could be made. For example, in the original course schedule, students were required to study 10 research methods within two weeks’ time, which was very difficult for most of them on a part-time study basis. The students suggested “select and master a few research methods in detail that were closely related to the work requirement while achieving general understanding of the other research methods.”
Considering that this was a short on-the-job training course without study credits, the change was made by the course team. Instead of applying each research method in general, the students were asked to choose two research methods in writing a research proposal as a group project. The assignments were revised accordingly for the 2nd intake of students. However, for credit courses, changes must be approved by the quality assurance committees of HKU SPACE.
Monitoring e-learning tutorials is one of the most important tasks for the programme director, who needs to log into the course Web site at least once every two days to observe students’ learning progress and difficulties. If students’ questions are not answered promptly or only ambiguous answers are provided, or if there are not many posts in the discussion forum area, the programme director would take immediate action to contact the tutors and solve the problem and inspire student’s learning enthusiasm.
Process evaluation is a meticulous process which involves continuous evaluation throughout the course. The programme director and tutors need to plan carefully to maintain students’ learning enthusiasm and help them achieve the final learning objectives.
Product evaluation of an e-learning course is mainly conducted through quantitative research, supplemented with students’ feedback and suggestions. For the first intake of this e-learning course, the online questionnaire method was used and all 60 students were surveyed. Thirty-eight valid data sets were received; the response rate was 63%.
Tables 4 to 6 show the students’ evaluation of course effectiveness, teaching effectiveness, and learning effectiveness. Table 4 shows the results of overall feedback on course effectiveness; such feedback is required for all short courses at HKU SPACE. Tables 5 and 6 display the results of students’ evaluation of various e-learning components of the course.
In order to understand these results relative to those of the face-to-face teaching mode, we adopted the evaluation statistics labels used for face-to-face teaching in HKU SPACE and calculated the average percentage of each item in the questionnaire survey on a Likert-type scale. The scale of the grades is explained as follows: 0 – 39.9 %, Fail; 40%- 49.9%, Pass; 50%– 59.9%, Satisfied; 60% – 69.9%, Good; 70% – 74.9%, Very Good; 75% – 100%, Excellent.
Table 4 shows that students’ evaluation of learning effectiveness, teaching effectiveness, and course worth reached the level of “Excellent” using the same evaluation statistics method for face-to-face teaching at HKU SPACE.
It can be seen in Tables 5 and 6 that student evaluation of degree of satisfaction, teaching products and learning products in e-learning components reached the level of “Excellent” based on the course evaluation criteria used at HKU SPACE. Table 5 shows that satisfaction with tutors was highest with an average score of 92.1, but communication with students was lowest with an average score of 72.4. In this course, the tutors were required to answer student questions within 48 hours and it was very much appreciated by the students. However, there were no requirements for students to respond to other students’ enquiries in discussion forums. Therefore, interaction between students was not so active compared with the interaction between tutors and students.
The students also provided qualitative comments about the course in this e-learning course questionnaire. In summary, students mentioned two kinds of benefits besides the learning itself: personal experience of e-learning, which would be beneficial for their own development of e-learning courses or e-learning tutorials in the future, and acquaintance with many new friends from different places in their own teaching areas. The following are some of the most typical comments made by the students of this course:
I really enjoyed the course. It was developed by a well known expert in the field, and I had fellow students from various regions.
The knowledge and skills I learnt from this course could improve my research quality greatly. Although I learnt courses on education research methods in Australia, only from this course can I really understand how to conduct educational research properly.
I feel so lucky that I can study this course. Students are from different places. We can share our experiences with each other in the course website, with no restriction of time and place.
I met some of my classmates from Inner Mongolia and Malaysia at the AAOU (Asian Association of Open Universities) conference. We had had a lot of discussions on the internet, but the first sight of the real person was still so exciting, just like seeing old friends after a long time.
The e-learning study materials are so clear and interactive. It can serve as a good example for my own e-learning course development.
Thanks to the enthusiasm and good examples of the staff tutor and tutors, I really enjoyed the e-learning learning environment of this course.
Besides studying different research methods, I have also learned a lot about how to develop quality e-course materials and how to facilitate discussion forums effectively.
Many thanks for Professor to develop and chair this course. Also thanks a lot for the passion and hard work of the tutors and other staff. This course has achieved better teaching and learning effectiveness than face to face instruction. In addition to those different research methods, I have also learned a lot about how to develop quality e-course materials and how to facilitate e-learning discussion effectively. Meanwhile, I also got to know many friends in the same field from different places. This e-learning course is definitely a good model and worthy of studying.
The success of this e-learning course has built up a good reputation. Many institutions have reserved places for their staff, and the long waiting list of potential students shows the course’s success, which ensures its sustainability.
The PDPP model for evaluating e-learning courses was designed and proposed based upon the CIPP evaluation model (i.e., planning evaluation, development evaluation, process evaluation, and product evaluation.) In line with the characteristics and process of e-learning teaching and learning, 26 evaluation items were identified within the PDPP model. Using the PDPP model, the authors took the e-learning course on Research Methods in Distance Education as a case study to describe and analyze the series of evaluation activities. The research results show that this PDPP evaluation model could effectively ensure the quality of the e-learning course in terms of both teaching and learning effectiveness. However, the use of the PDPP model in this study measures only one purely e-learning course and further studies are needed. The authors hope that this model could contribute to the e-learning quality assurance literature in the Chinese context and could be one reference in establishing e-learning quality assurance models for other educational institutions.
This article is based on the paper presented at the 24th ICDE World Conference on Open and Distance Learning held on 2-5 October, 2011, on Bali, Indonesia. Some of the revisions made in the production of this article are based on discussions at the conference, for which I am very grateful to the participants.
Allen, I. E., & J. Seaman. (2011). Going the distance: Online education in the United States, 2011. Retrieved from http://www.onlinelearningsurvey.com/reports/goingthedistance.pdf.
Chinaacc (2009). About us. Retrieved from http://www.chinaacc.com/
Development and Planning Office of CCRTVU (2011). Education statistics communiqué of Radio & Television Universities in China. Journal of China Distance Education, 3, 72-76.
Education Life Website (2009). Economic crisis couldn’t prevent people from pursuing further education: E-learning education is blooming. Retrieved from http://www.yuloo.com/wangyuan/jyzc/2009-07-16/270066.html
Endean, M., Bai, B., & Du, R. (2010). Quality standards in online distance education. International Journal of Continuing Education and Lifelong Learning, 3(1), 53-73.
HKU SPACE (2010). Quality assurance manual. Retrieved from http://www.hkuspace.hku.hk/files/about-us/governance/quality-assurance/qa-in-hku-space/CoverPage_eng.pdf.
Hu, Xiaohong (2009). E-learning application: 50% of HSBC courses will be provided e-learning in 2009. Journal of China Distance Education, 6, 77.
Jochems, W., van Merriënboer, J., & Koper, R. (2004). Integrated e-learning: Implications for pedagogy, technology and organization. London: RoutledgeFalmer.
Jung, I., & Latchem, C. (2007). Assuring quality in Asian open and distance learning. Open Learning, 22(3), 235-250.
Jung, I., Wong, T.M., Chen, L., Baigaltugs, S., & Belawati, T. (2011). Quality assurance in Asian distance education: Diverse approaches and common culture. International Review of Research in Open and Distance Learning, 12(6), 63-83.
Zhang, W.Y. (2009). The theory and application of interaction-centered online interaction (in Chinese). Modern Distance Education Research, 2009(5), 10–13.
Zhang, W.Y., & Jiang, G. (2007). Evaluation research in distance education. In W.Y. Zhang & G. Jiang (Eds.), Distance education research methods (pp. 184-197). Beijing: Higher Education Press.
Zhang, W.Y, & Wang, L. (2005). International cooperation in the development of e-learning learning environment inventory. In W.Y. Zhang (Ed.). Management and learner support in modern distance education (pp. 1-18). Beijing: Beijing Radio and TV University Press.
Young, E., Zhang, W., Chan, A., & Cheung, D. (2009). Survey on the demand for continuing education in Hong Kong. International Journal of Continuing Education and Lifelong Learning, 1(2), 103-118.