Arundhati Mishra
BJB College, Bhubaneswar, India
Vijayshri
Indira Gandhi National Open University, India
Suresh Garg
National University of Lesotho, Southern Africa
The undergraduate science programme was launched at the Indira Gandhi National Open University (IGNOU) in 1991-92 with an enrolment of 1,210 students. The programme was well received, and enrolments increased over the years. However, the success rates have not kept pace with enrolment.
In this paper, the authors report the results of an evaluation of the undergraduate Physics programme at IGNOU. The evaluation, the first of its type for this programme, adapted the major tenets of the CIPP model. The findings are based on the responses from a randomly chosen sample of 509 learners across India. The methods employed for the study include records, document, and database analysis, surveys, and case studies.
Although the University has enhanced access to higher science education, the attrition rate is high (73%), and the success rate is low. The authors recommend that the University review and reorient its strategies for providing good quality, learner-centred higher education in science subjects. The programme should address the concerns of the learners about the effectiveness of the student support systems, the difficulty level, and the learner-friendliness of study materials with the goal of achieving long-term sustainability while maintaining parity with the conventional system. The need for improving the presentation of the courses and simplifying the mathematical details is emphasised.
Keywords: Physics; open learning; science education.
The undergraduate science programme aimed at providing higher science education through the distance mode at the national level was launched by the Indira Gandhi National Open University (IGNOU) in December 1991 with an enrolment of 1,210 students. The annual enrolment reached 9,973 in 2003 before stabilising at about 6,000 in 2006. The programme design is based on a flexible credit system and is characterised by a three-tier course structure: compulsory foundation courses (24 credits), elective courses to provide for core areas of study in a discipline (56 to 64 credits), and application-oriented courses (8 to 16 credits) to provide rudimentary skills that help to improve the employment prospects of the graduates. Learners who earn 48 credits in elective courses in any one discipline are awarded the B.Sc. degree in that subject (Vijayshri et al., 1998, p. 109).
The course options, the place, and the pace of learning in the programme are flexible and open. It is offered in five science disciplines (physics, chemistry, mathematics, life sciences or botany, and zoology), and students must opt for a minimum of 25% of total elective credits earned in disciplines with experimental components from laboratory courses. The instructional methodology comprises multiple-media self-instructional packages, with print as the mainstay. The assessment has two components: continuous and term-end. IGNOU provides support services to its B.Sc. students through a total of 1,653 study centres (SCs), out of which 150 (SCs) are activated for the B.Sc. programme. (SCs have been activated in reputed institutions [colleges and university departments] of higher learning to ensure that appropriate academic and other support is provided to the students.) The laboratory training and academic counselling for courses in the B.Sc. programme are carried out at all 150 SCs.
The programme has been well received as demonstrated by the increasing enrolments over the years, and it has been welcomed by those who missed the opportunity for higher science education. However, over the years, students, academic counsellors, and physics experts have expressed concerns to two of the authors about the quality of the curriculum transaction, including the laboratory, the effectiveness of the student support system, and the performance and satisfaction of the learners. Another motivation for the study was the decision taken by the Board of Management of IGNOU in 2002 to revise all programmes so that the shelf life of a programme is not more than seven years. To address these issues, it was considered essential to undertake a systematic evaluation of the physics programme, which is offered as a part of the bachelor’s degree programme in science at IGNOU. In the present study, the major tenets of the CIPP model have been adapted.
In distance education, programme evaluation encompasses the evaluation of programme objectives, course content, instructional design, support services, assessment practices, student achievements, and use/impact of technology (IGNOU, 2006) so as to improve the quality of teaching-learning, to enhance the relevance of the programme, to assess how the programme is perceived by its stakeholders and meets their expectations, and to provide regular feedback on the factors that affect outcomes. There exists substantial literature on the theory and methods of programme evaluation (Kellaghan & Stufflebeam, 2003, p. 1-79), which delineates the objectives, approaches, and methods for evaluating a programme (Mishra, 2008, p. 34-46). In the present study, we have adapted the major tenets of the CIPP model (Stufflebeam, 1983, p. 117). The CIPP evaluation model promotes the view that the purpose of evaluation is programme improvement. The process involves four stages of evaluation: context, input, process, and product.
A few studies on evaluation of programmes offered in the distance mode are available in the public domain. The PhD programme in Education at the University of the Philippines (UP) Open University has been evaluated using the CIPP model (Quimbo, 2002, p. 196). The study revealed that improvement in significant input and process variables made a positive impact on learner performance. The CIPP model was also used to monitor and develop an evaluation framework for technology-based community learning centres in India (Calder & Patel, 2002, p. 214).
Kapoor (2004) evaluated the Mental Health Nursing Care course – a competency-based course – offered in the B.Sc. Nursing programme at IGNOU and used a model akin to the CIPP evaluation model. The study revealed that the self-learning materials (SLMs) and laboratory experience for this course were interesting and useful but access to electronic media facilities at the study centres was poor, even though tele-conferencing had been made mandatory.
In a recent study, Fozdar et al. (2006) reported a variety of factors (personal, academic, and financial) that were responsible for the dropout of B.Sc. students at IGNOU. The study reported in this paper is comprehensive and the first of its kind on the evaluation of the B.Sc. Physics programme offered by IGNOU. Our evaluation model is essentially summative in intent and structure (Panda, 1991, p. 168). Figure 1 shows the conceptual framework developed for this study based on the theoretical and empirical evidence. Although many aspects of programme evaluation have been included in Figure 1, in the study reported here we have included only the objective set provided below.
The present study is limited to evaluating the programme for the following:
Findings pertaining to programme evaluation of other aspects listed in Figure 1, for example the quality of the student support services and the efficacy of the administrative and managerial system, will be reported in forthcoming papers.
The research design formulated vis-à-vis the objectives of programme evaluation encompasses studying the feedback of primary sources (learners, counsellors, programme designers and developers) on various aspects of the programme, collection of institutional data, and responses of various stakeholders. This study was initially formulated and designed in 2005 following the decision of the University in 2002 that every programme launched more than a decade ago should be reviewed. The period of the study was taken to be from the year 2001 to 2005 because the programme had stabilised by then, and the first batch of learners had completed the maximum period (eight years then but six years now) for completion of the programme.
The following methods were adopted to carry out this study:
The population for the study initiated in 2005 was distributed over 146 study centres activated by IGNOU for its B.Sc. programme. The prime consideration of the researchers was to select a sample representative of the system and to ensure that the selected centres represented an all-India picture. A survey of the institutional enrolment data of learners for the years 2001 to 2005 revealed that 45-50% of learners enrolled in the B.Sc. programme from the Northern region, 25-30% from the Eastern region, 15-20% from the Southern region, and 5-10% from the Western region.
In this study, questionnaires were administered to about 800 learners and responses were received from 509 students (64%). (One of the researchers personally visited 16 study centres in different parts of the country.) The maximum number of sampled students was from academic session 2005 (224, 43%), followed by academic session 2004 (125, 25%) and academic session 2003 (115, 23%). Only 45 learners in the sample (9%) were enrolled in 2001 and 2002. This indicates that most respondents were in the first, second, and third year of the programme and had the most recent experiences of the programme. Therefore, this sample can be taken as authentic in so far as providing feedback about various aspects of the programme is concerned. The sample size also meets the requirement of 99% confidence level and 5% confidence interval for the average number of students enrolling during the period 1997-2005.
The learner population was sampled at different stages of the study. The objectives of the study suggested that a sample needed to be determined for those who wished to major in physics and pursue higher education or those who were interested in upward mobility of their career in science laboratories (school/college/university/R&D). A sub-set of 199 learners (39%) who were pursuing the B.Sc. in Physics, out of the total sample of 509, was identified, and information on major trends emerging from the analysis of responses was verified in tele-interviews and face-to-face interactions. The institutional data was obtained from the Student Registration & Evaluation Division (SR&ED) of IGNOU.
The enrolment data in various physics courses for the period 2000 to 2006 and its comparison with the total enrolment data (sourced from institutional records) revealed that
This behaviour suggests that 3rd level physics courses are usually selected by those learners who are interested in pursuing a related master’s degree or a career in physics.
The success rates, retention rates, and performances of learners are important indicators of the acceptability and sustainability of a programme. To discover these for learners opting for physics electives (listed in Table 1) or opting to pursue a major, the institutional data was scrutinised. It revealed the following:
In Table 1, we have listed year-wise pass percentage in physics electives PHE-01 to PHE-14 (offered until 2005) through the years 2000 to 2005 based on the institutional records. The numbers in brackets indicate the credit weight of each course. The corresponding values of average pass percentage, standard deviations, and t-values are also given. We discovered the following:
The one-way ANOVA test, which can be thought of as an extension of unpaired student t-test, was carried out for analysis of variance for two sets of theory and lab courses. The calculated p and F values for the lab courses are respectively 0.02 and 5.31. The value of F is greater than the theoretical value of 3.98 indicating that the average pass percentages in lab courses are statistically significant. But in the case of the Mathematical Methods in Physics courses, p and F values are respectively 0.223 and 1.71 (< 3.89), indicating that average pass percentages are statistically not significant.
We now present the findings of our study in respect of the profile of respondents, learners’ feedback on course materials, and their validation by counsellors and programme developers. The results have been presented chiefly through computation of percentages.
The basic characteristics of 509 sampled learners revealed that the sampled population was predominantly urban (89%) and belonged to the non-Scheduled Caste/Scheduled Tribe category. (These have long been marginalised sections of Indian society, deprived economically, socially, and educationally.) A significant majority (84%) was over 21 years, male (70%), and lived far (25-50 km) from the study centres (68%); a learner might take 2-3 hours commuting each way, depending on the geographical location and availability of transport. The population of employed and not employed learners was nearly equal, and English as the medium of instruction was not perceived as a barrier.
To explore the reasons for the low success rates, learner responses on the time they devoted to their studies and on their study habits were examined. The findings are given below.
In view of their societal, familial, and personal commitments, distance learners invariably experience acute shortages of time. And a frequently asked question concerns the number of hours they must study everyday or in a week to successfully complete a programme in the minimum time without compromising the quality of learning. This has implications for course design, development, and student workload, as well. Our analysis showed that 51% of learners (252) put in more than 6 hours per week; however, half of the students spend, on an average, less than one hour per day.
Unfortunately, no document advises learners that spending 2-3 hours on an average everyday is necessary for successful completion of the programme in the minimum specified duration of three years. Although in all physics electives, the learner is advised about the estimated time s/he is expected to spend on each unit, block, and course, it is suggested that the programme developers/faculty members should also advise learners about the average time they must spend everyday to successfully complete the programme in minimum time. It is therefore desirable to include a detailed guide in the learners’ programme materials on ‘how to study on their own at a distance’ and reinforce this advice in counselling/orientation sessions through electronic media as well as through F2F counselling at the study centres.
In response to the specific question on how they study the print materials (e.g., a new block/unit), 361 learners (74%) reported that they began to read the unit from the beginning, and only 24 learners (5%) read the block introduction and the study guide to the block. Fifty-five learners (11%) began from the structure, 22 (4.5%) read the course objectives, and 14 (3%) gave comments such as lack of time, consulting other reference books, highlighting main points. This shows that the majority of sampled learners were serious about their studies, and they should be advised about the importance of the objectives in every unit as the yardstick against which they are tested and against which they should measure their learning achievements.
Printed self-instructional materials form the backbone of instruction in IGNOU. Other media (A/V and web-based) are used as supplements only. Therefore, evaluating various aspects related to teaching-learning through the course materials was a major concern of the researchers. The data analysis revealed that 99% of responding learners appreciated the courses and the majority of learners rated the courses as good (293, 66%), although the teacher only communicated with them sometimes (403, 82%) (in text), and the study material was partially self-instructional (316, 65%). Considering the breadth of course material, three out of four learners (374, 75%) opined that the material was adequate; whereas, 64 learners (13%) thought otherwise. In addition, 60 students (12%) found the material lengthy. But learners who did not find the courses adequate gave no suggestions on the topics to be included or deleted.
A majority of respondents (413 out of 496) reportedly needed help to learn the courses. The major reasons cited were high content density, difficulty in following the mathematical treatment, paucity of good diagrams, and lack of sufficient number of examples and exercises for practice. From these findings one may conclude that the learning materials do not fully substitute for the teacher, and it is prudent to revise them by addressing the concerns of learners.
A significant majority of respondents (400, 83%) perceived the course content as relevant, while 16 learners (3%) opined that the materials were not relevant, and 65 learners (13%) were undecided.
The analysis of the data on sequencing and presentation of printed study materials revealed the following:
The in-text self-assessment questions (SAQs) and terminal exercises (TQs) in the course materials are intended to clarify concepts further and to provide opportunities for self-learners to assess their understanding of the content. In our sampled population, 140 learners did not respond to the question on whether they attempted the self-check questions/exercises/activities included in their study materials. Of 369 learners, 347 (94%) reported that they attempted SAQs and TQs. This suggests that about 30% of learners in the sample were not self-evaluating. A large number of students (415, 86%), in spite of advice to the contrary, checked the SAQ answers sometimes before attempting them. However, SAQs were found most helpful in clarifying difficult concepts by 484 out of 496 respondents (97.6%) (Table 2). And only a few learners re-wrote answers if they were not satisfied with their responses.
Although few learners responded to the question about their preferred type of SAQ (objective type: multiple choice, fill in the blanks, matching, true/false; short answer type: sequencing, numerical; and long answer type: derivation, application), from personal interactions with the students, the researchers have formed the view that multiple choice questions in the objective type, numerical in the short answer type, and applications in the long answer type should help learners to gain mastery over the subject.
In Table 3, we have listed the responses to the open-ended question about the ways in which learners found the SAQs helpful. In a nutshell, SAQs and TQs play an important role in facilitating learning. The implications of these findings for course developers are that the number of in-text questions should be increased, and they should be diverse; the problems should correlate learning with real-life situations; and the questions should be sequenced in order of difficulty and of learning objectives.
The learners were asked to rate the difficulty level of concepts, mathematical derivations, examples and exercises, assignments, etc. on a four-point scale.
The data given in Table 4 suggests that more than 80% of responding learners consider the following as easy or slightly difficult: explanations/presentation of concepts, correlation of knowledge with immediate surroundings, quality of examples, and self-check questions (SAQs and TQs). However, 40-45% of responding learners find mathematical derivations and assignments difficult or very difficult. This could be the major reason for the lack of satisfaction of learners with materials. It would be worthwhile to provide complete derivations without skipping in-between steps.
Learner feedback on course materials obtained through questionnaires and interviews informs us that the level of materials should be moderated, and their presentation should be in tune with learner capabilities, expectations, and interests. It is prudent to improve the presentation by reiterating important definitions and highlighting them in boxes, for example, and by incorporating more illustrations, bar charts, tables, and solved and unsolved examples. Mathematical steps in the text should be explained in detail and continuous assessment through assignments should be in consonance with course contents. These findings may serve as general guidelines for developing revised versions of the courses. In the ultimate analysis, the revised version of each course should address course-specific learner feedback.
Learners in IGNOU come from varying backgrounds and constitute a heterogeneous group in competences, skills, and learning styles. But it seems that programme designers and developers have pitched the physics courses at a level higher than that expected by an average learner due to the over-riding concern for parity with conventional universities. This means that all physics courses should be moderated by simplifying language, by including more examples, diagrams, and pictures, and by solving intermediate mathematical steps. Modifications may also include regional requirements, practices, and variations. These findings have significant implications for learner performance and retention, programme revision, implementation, and sustainability. These provide crucial input for distance educators in their efforts to promote effective and active self-learning and to bridge the gap between the perceptions of programme designers/developers and learners.
In the sample of 509 learners, 199 learners were pursuing the B.Sc. in Physics degree. While this sub-group, in general, exhibited behaviour very similar to that of the sampled population, a few distinguishing features emerged. These are outlined below:
The credit completion status of 193 responding major learners at the time of the study is given in Table 5. For comparison, we have also included the status of 295 non-majoring and 488 of the sampled population. A closer examination of the status of 42 majoring learners who had completed 76-100% of credits revealed that 30 learners had completed the B.Sc. in Physics and 12 were about to complete all courses. Moreover, the percentage of learners earning more than 25% but less than 50% of the credits was more for the majoring group. So we can conclude that the completion rate of the learners intending to earn the B.Sc. in Physics within the stipulated time period of 3-6 years is better than that of non-majoring students.
In imparting instruction in science at a distance, practical training is construed as the major impediment by conventional peers. To provide an interactive, creative, and stimulating experience, the physics curriculum in IGNOU places greater emphasis on day-to-day work in the lab, which is accorded 70% of the final grade. The detailed findings have been presented elsewhere (Mishra et al., 2008). It may suffice to say that contrary to the expectations of the purists among academia, learners preferred to earn the maximum possible credits in practical/lab-based courses. But the basic considerations of learners seem to be full-time F2F guidance, a high success rate, and better grades. About 90% of distance learners opined that lab sessions were well planned and structured, lively and stimulating, and helped them to learn a lot of good physics; further, they looked upon practical sessions as an enjoyable experience. But some of these findings were not confirmed by the faculty. Moreover, there seems to be no correlation in the grades of learners in practical and theory courses (Khare et al., 2004)
To validate the patterns and characteristics emerging from the analysis of the responses of the majoring cohort, a research tool was administered on five learners in the form of case studies. Their written responses received by email were followed up by telephone interviews. Among the reasons they gave for joining IGNOU were a lack of time for full-time studies and the opportunity to pursue other courses simultaneously. The researchers encouraged them to give their frank assessment of the quality of course materials and of how the B.Sc. Physics degree helped them to shape their careers. They were also asked to suggest ways to improve systemic efficiency and to promote learner performance.
The programme-specific strengths identified by the respondents included flexibility in the choice of subjects and in the pace of learning, quality of learning materials, and equal opportunity to study science at the tertiary level. But limited use of ICTs in the teaching of physics and non-inclusion of project work and computer courses in the curriculum were identified as the major deficiencies.
Fifty-four counsellors were shortlisted to provide feedback: 34 sent their responses by post (group I), and 20 were interviewed in person (group II). The majority of respondents were male (45) and possessed a PhD degree; some of them were university professors/former heads of department/principals of colleges. All of them had good exposure to the courses being counselled, and many were counselling more than one course. The spread of their institutions ranged from leading institutions in cosmopolitan cities to lesser known colleges in towns. We can safely assume that their responses represent a rich pool of diverse practices, comparative positioning of courses, and learning environment/habits/styles of IGNOU learners enrolled in this programme.
To know how counsellors perceived physics materials, specific questions were raised about curriculum design, content presentation, difficulty level, and possible improvements. Twenty-one group I (91%) and 19 group II respondents (95%) found the content coverage to be appropriate, and, in their view, the curriculum did not warrant major changes. However, both groups recommended the inclusion of more examples (65%) and SAQs/TQs /unsolved exercises (74%), and that in-between mathematical steps should be worked out in detail to make materials more learner-friendly (94%). This validated learner feedback on different courses. Academic counsellors would like students to be trained in problem-solving skills that enable them to correlate their learning to their lives/surroundings. They also suggested that question banks should be developed for each course, and questions of different types, such as multiple choice questions in the objective type, numerical in the short answer type, and applications in the long answer type, should be included to help learners acquire mastery over the subject.
A significant majority of respondents in both groups (86%) found the format and style of presentation adequate and unanimously recommended that these be retained as such. However, their opinion on the need to give more explanations of concepts that were difficult to visualise or comprehend was sharply divided; although, the majority (60%) viewed them to be adequate. About 76% of responding counsellors consulted IGNOU materials while preparing their own classroom lectures. It was encouraging to discover that 84% of responding counsellors would like to procure these materials for their library so that their full-time students may use them. This is an indication of the national acceptability of the quality of the physics materials, and it may be regarded as a spin-off effect of IGNOU’s efforts to meet a felt need for good text materials at an affordable cost. This should help bridge the gap in the quality of higher education provided by different universities and colleges in India.
Eighty-three percent of learners reported that owing to higher standards, a self-learner needed outside help. This perception was confirmed by the majority of counsellors. When a question was raised about the books/reference materials consulted by the learners, it was astonishing to learn from the counsellors’ feedback that a significant number of learners looked for ‘physics made easy’ types of guidebooks. This has implications for the course developers to produce more learner-friendly study materials.
The majority of group I respondents (61.8%) rated the materials as very good. They found the language lucid and its comprehensibility high. As such, most of the figures were considered good; however, in their view, coloured or 3-D diagrams would improve quality further. Some very useful suggestions on deletion/reorganisation/inclusion of content were also made.
Separate tools were administered on programme designers/developers/experts. The programme designers and developers, including the in-house faculty, are subject experts, and they guide the process of material design/development by maintaining a fine balance between established and emerging bodies of knowledge. As a result, it is expected that learners are trained in the latest trends and practices in addition to the essentials. In these investigations, the researchers solicited responses from 35 experts. Their feedback revealed that
It may be pointed out here that IGNOU's Physics faculty has undertaken detailed revision of learning materials on the basis of the feedback obtained from different stakeholders. In this process, the course contents have also been revised with additions and deletions, where appropriate, to avoid repetitions and to update the curriculum. The revised study materials for the first-level courses are to be sent to learners for the January 2010 session, and their impact on learner performance shall be assessed in due course of time.
The learners in the sampled population were predominantly urban, adult males, who lived far from the study centres. Their seriousness about studying the courses is beyond doubt, though they need to be guided clearly about how best to study the courses. Since enrolment in the programme grew with time, one can say that the University has succeeded in providing access to higher science education to a diverse learner population across the country. However, a high attrition rate (73%) and a low success rate have significant implications for the students, the programme developers, and the University. Though course materials have been rated to be of high quality, relevant, useful, and on par with the best in the country, the need to improve the presentation of the courses and to simplify the mathematical details is also emphasised by all concerned.
The authors recommend that course developers include more and varied in-text questions. Also, the problems should correlate learning with real-life situations, and the questions should be sequenced in order of difficulty and of learning objectives. The level of available materials must be moderated, and important definitions should be reiterated and highlighted in separate text boxes. As well, more illustrations, tables, and solved and unsolved examples are needed, and all in-between mathematical steps should be worked out. Finally, question banks with answers should be developed and provided to learners to facilitate their preparation for term-end examinations and to improve their satisfaction levels.
Calder, J., & Patel, I. (2002). A monitoring and evaluation framework for technology-based community learning centres in India. In H.P. Dikshit, S. Garg, S. Panda, & Vijayshri (Eds.), Access and equity: Challenges for open and distance learning (pp. 214-224). New Delhi: Kogan Page.
Fozdar, B.I., Kumar, L.K., & Kannan, S. (2006). A survey of study on the reasons responsible for student dropout from the Bachelor of Science programme at Indira Gandhi National Open University. The International Review of Research in Open and Distance Learning, 7(3).
Pradhan, B., & Ramanujam, P.R. (2006). Handbook 12: Programme evaluation in open and distance education. New Delhi: STRIDE, IGNOU.
Kapoor, B. (2004). Development of an evaluation model for nursing programme through distance education (Unpublished doctoral thesis). Indira Gandhi National Open University, New Delhi, India.
Kellaghan, T., & Stufflebeam, D. L. (2003). International handbook of educational evaluation (Vol. 1). Boston: Kluwer Academic Publishers.
Khare, P., Garg, S., & Saxena, A. (2004). Performance analysis of IGNOU science students in theory courses through data mining. Indian Journal of Open Learning, 13, 29-49.
Mishra, A. (2008). Evaluation of undergraduate physics programme of IGNOU and implications for learners (Unpublished doctoral thesis). Indira Gandhi National Open University, New Delhi, India.
Mishra, A., Vijayshri, & Garg, S. (2008, in press). A preliminary evaluation of undergraduate physics laboratory instruction offered at IGNOU. AAOU Journal of Open Learning.
Panda, S. (1991). Programme evaluation in distance education: A perspective and proposed agenda of actions. Educational technology: Vol. 1 (pp. 168-181). New Delhi: AIAET.
Quimbo, Maria Ana T. (2002). A model for evaluating distance education programme. In H.P. Dikshit, S. Garg, S., Panda, & Vijayshri (Eds.), Access & equity: Challenges for open and distance learning (pp. 196-213). New Delhi: Kogan Page.
Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In G.F. Madaus, M. Scriven, & D.L. Stufflebeam (Eds.), Evaluation models (pp. 117-141). Boston: Kluwer-Nijhoff.
Vijayshri, Garg, S., & Panda, S. (1998). Teaching science at the Indira Gandhi National Open University. In S. Garg, S. Panda, & Vijayshri (Eds.), Science and technology through open and distance education (pp. 109-172). New Delhi: Aravali International.