Luis M. Villar
University of Seville
Spain
Olga M. Alegre
University of La Laguna
Canary Islands
The rapid growth of online learning has led to the development of faculty inservice evaluation models focused on quality improvement of degree programs. Based on current 'best practices' of student online assessment, the Online Faculty Development and Assessment System (OFDAS), created at the Canary Islands, was designed to serve the dual purpose of faculty development and classroom learning environment assessment. Results, as illustrated in this paper, show that the OFDAS encouraged faculty to reflect on the professionalism of their teaching skills. Implications are discussed in terms of emphasizing the process of online teaching, knowledge acquisition, and incorporating varying perspectives, all which yielded a comprehensive view of faculty teaching attitudes and their relationship to student's perceptions of their classroom environment.
Keywords: Faculty development; teaching professional skills; online student learning; environment assessment; online professional development
The Online Faculty Development and Assessment System (OFDAS) is a voluntary professional skill program. The course encompassed three components: 1) online faculty professional skill learning experiences; 2) faculty professional skills learning assessments; and 3) a student online learning environment assessment.
The critical issues in designing OFDAS included, among others, planning, organizing, structuring, tracking, reporting, and communicating assessments – efforts which took time and required orderliness on the part of the online program advisers. Based on their experiences, we found the following five-stage design process to be a highly effective strategy:
The OFDAS model recognizes faculty personal and professional objectives at various stages of their academic career. As such, it was necessary to create diverse training strategies. Accordingly, prior to the implementation of the OFDAS, the amount of time needed to complete the professional development program was negotiated with higher education institutions. A formal face-to-face workshop was delivered prior to starting the online course as well. Finally, in an attempt to ensure intrinsic motivation and to encourage participation among faculty, an official learning certificate was developed.
The authors developed a framework for training professional skills reflective of student-centred education (Villar, 2004); that is, a framework that focused student learning experiences and processes within the university social context (Badley, 2000). Prior to engaging in the OFDAS, it was expected that faculty have a deep understanding of their scientific field, as well as requisite pedagogical and didactic skills specific to their discipline. Accordingly, the concept of 'professional skill' was defined by the authors as: "an integrated set of knowledge, beliefs, abilities and attitudes that were basic for good performance in various university teaching settings." Common elements in the OFDAS program were to develop faculty competence in the design of curriculum and course material, and help faculty acquire didactic and guidance skills (Tigelaar, Dolmans, Wolfhagen, & Van Der Vleuten, 2004). Three basic principles predominated in OFDAS: helping faculty understand that, 1) academics and students are different, thus curriculum and implementation of classroom methods must be designed to respect student diversity and identity; 2) professoriate are dependent on one another in collegial and classroom interactions; and 3) online faculty development courses increase one’s own decision-making processes and learning by online student assessment. Consequently, ten professional skills were proposed (see Figure 1).
Online assessment of classroom teaching and learning processes has been the
focus of numerous studies that examine the degree this method of assessment
influences and facilitates changes within learning organizations. Online assessment
questionnaires should contain student demographic information such as age and
gender, as well as other items seeking students’ opinions about the quality
of the course (Lounsbury, Saudargas, Gibson, & Leong, 2005). Recommendations
for the online classroom climate assessment has components similar to conventional
assessments. For example, some scholars focused their attention on 'grading
procedures' and were very explicit when such tasks, projects, and tests should
be assigned (Summers, Waigandt, & Whittaker, 2005).
From an administrative point of view, research has been conducted examining
'why' online course evaluation should be used. According to this body of research,
online course evaluations lower costs, allow more time for teaching, lessens
administrative burden, and so on (Ballantyne, 2003). The question remaining
for us, therefore, was whether online student feedback gathered from questionnaires
(see Appendices A and B) actually does lead to improvements in teaching quality.
Until recently, these types of feedback tools (i.e., questionnaires) have formed
the basis upon which to compare teachers, departments, and even universities.
However, although correlations have been found to exist between teaching quality
and online student feedback, it does not necessarily imply causality, as other
factors could also affect the quality of teaching over time (Kember, Leung,
& Kwan, 2002).
Literature on students’ assessment of classroom environments has been on the rise for over a decade (Aldridge & Fraser, 2000) and the field of learning environments has undergone remarkable "diversification and internationalisation" (Fraser, 1998, p. 7). Evidence (largely derived from on-demand university teaching quality assessments) has accrued regarding the potential of classroom learning environment assessments and their ability to improve university-level teaching and learning, as well as staff development (Dallimore, Hertenstein, & Platt, 2004). For example, classroom climate questionnaires administered in a field-specific class were found to result in reflective changes in learning and instruction (Wildman, Hable, Preston, & Magliaro, 2000).
Following are some key features of the online course used in this study (see http://gid.us.es:8083):
Our broad research question was: "How did the online course elements and management activities affect professional skills learning and what were their impact on teaching attitudes and on the students’ classroom learning environment assessment?" This question was examined in light of three specific questions (see Figure 1). First, we asked was there a difference in faculty opinion about the quality of the online course? Second, did faculty learn professional skills? And third, after the course ended, was there a relationship between faculty teaching attitudes and students’ assessment of their learning environment?
Figure 1. The OFDAS model and sample variables
Twenty-four (n = 24) faculty members volunteered for this study and all met the following selection criteria: (a) employees of a university campus, (b) scientific field, and (c) professional merit. All respondents were full-time faculty employed by two public Canarian Universities: 11 from La Laguna (45.8%) and 13 from Las Palmas de Gran Canaria (54.2%). Of the 24 faculty respondents, ten (41.7%) were male and 14 (58.3%) were female. Nineteen (n = 19; 79.2%) respondents were considered 'experts' (i.e., with more than five years of teaching experience). Most had a doctorate (n = 14; 58.3%). When disciplines were broken down into scientific fields, eight respondents (33%) reported that they taught in the social sciences; five (20.8%) in experimental sciences; four (16.7%) in healthcare sciences; three (12.5%) in humanities, and four (16.7%) in technical sciences. The OFDAS program took place during the spring quarter of the 2006 academic year and spanned an 11 week period.
The AUTAQ was administered to 102 students enrolled in courses the two universities taught by the faculty listed above. Four hundred eighty (n = 480) students were asked to provide feedback on their classroom climate. The student sample was representative of gender, age, area of study, level of study, and other academic and social characteristics.
Professional Skills Scale
The Professional Skills Scale was used to measure
respondents’ ability to understand the knowledge and skill acquisition levels,
and the degree to which individuals or groups wished to use them (Cronbach's
alpha = 0.944). Consisting of ten items used to measure 'expert skills,' 'conditions,'
'technologies' and 'teaching practices' (i.e., "The competence was relevant
for my teaching"), this scale was designed to measure faculty's knowledge,
skills, and attitudes. For items 1 to 17, a five point Likert-like scale was
used: 1 = strongly agree; 2 = agree; 3 = average; 4 = disagree; 5 = strongly
disagree. Items 8 to 10 had specific five point scales. All ten items measured
various types of faculty 'opinions' over the duration of the course, such as professional
skills relevance, usefulness, appropriateness, adaptation, tips, structure,
pertinence, reading, impact, and time-consuming. This measure was developed
for use in this study.
Attitude Towards Course Learning Questionnaire (ACLQ)
Faculty were asked to make 'attitude judgments' on teaching practice
(Cronbach's alpha = 0.950). The measure consisted of 20 items (see Table 1,
and Appendices A and B) scored on a 5-point Likert-type scale, ranging from
1 = ‘strongly agree,’ to 5 = ‘strongly disagree.’ The measure was developed
for use in this study. Each scale consisted of four items.
Table 1. Description of Scales and a Sample Item for Each Scale
of the ACLQ
Assessment of University Teaching Activities Questionnaire (AUTAQ)
The AUTAQ was designed to appraise students’ perceptions of their classroom
environment (Cronbach's alpha = 0.958). This questionnaire consisted of 22 items
scored on a 5-point Likert-type scale, ranging from 1 ‘strongly agree’ to 5
‘strongly disagree’ (see Table 2, and the Appendix). The design of the AUTAQ
was guided by relationship, personal growth, and curriculum change dimensions
for conceptualizing university quality assurance (Villar, 2001).
Table 2. Description of Scales and a Sample Item for Each Scale of the AUTAQ
Data was collected online during and after the course. Faculty members explained to students the purpose of the AUTAQ and the research study, and assured them of full anonymity to encourage their participation.
Data analyses included descriptive statistical summaries, Alpha reliabilities of subscales of the two questionnaires, T-tests, analysis of variance (ANOVA), and intercorrelations among scales (Pearson product moment correlation coefficient).
Research Question 1 asked whether faculty opinion towards the quality of OFDAS was positive. In terms of the professional skills quality scale items, item means ranged from a high of 3.08 (Item 8, Reading: "I read websites and pdf documents which were linked to the professional skill") to 1.33 (Item 1, Relevance: "The competence was relevant for my teaching'). Standard deviations varied from 1.52 ( Item 8, Reading ) to .76 ( Item 1, Relevance ). All item mean scores exceeded the midpoint scale (3.00, normal), and Item 8, Reading, exceeded the midpoint scale (3.00, frequently).
T tests revealed significant differences with regard to gender in five quality items (usefulness, adaptation, tips, structure, and pertinence). Females held better opinions than males with respects to the quality of the professinal skills. As to degrees, significant difference was found in eight quality items (relevance, usefulness, appropriateness, adaptation, tips, structure, pertinence, and time-consuming). PhD prepared instructors/ teachers held better opinions than Bachelor's prepared instructors with respect to the quality of professional skills. With regard to the degree of teaching expertise, new faculty (those with less than four years teaching experience) and expert faculty (those with five years or more teaching experience) had different opinions with respect to five quality items (usefulness, appropriateness, adaptation, tips, and structure) (See Table 3).
Table 3. Significant t-Test Results for Demographic and Academic Factor Comparisons
Research Question 2 asked whether the online course stimulated faculty learning. This question was divided into two sub-categories: 1) facilitating learning activities, and 2) assessing the cognitive domain of professional skills learning.
Facilitating Learning Activities
Oliver and Herrington (2003) assert: "Designing a learning environment
by commencing with the design of learning activities creates a setting where
the focus of the planning centres on formulating the forms of learning outcomes
being sought, rather than considering what content will be covered" (p.
114).
Learning activities were developed to reflect the manner in which curriculum and didactic knowledge will be used in real-life university environments. Descriptive summaries detailed the ten professional skills being assessed during the learning activities. Results show that faculty respondents completed 1,587 learning activities (see Figure 2).
A principle of the learning process was peer assistance and peer review, which was provided via guidance and participant feedback. Online help was often needed. Thus, coaching and scaffolding of learning was provided by two OFDAS leaders assigned to diagnosed the strengths and weaknesses of each participant, and tailor any support needed. Figure 2 shows participant instructors’ changes in their interest in, and willingness to, respond to learning activities as the course progressed over the 11 week duration. Participation in the learning activities was found to be more intense in the earlier stages of the course, than during the final activities. Data collected shows that participants' time commitment was not equally distributed. While participants engaged heavily in Professional Skill 2 (awareness of students’ diversity in all its forms), they engaged very little in Professional Skill 6 (knowledge of area being supervised – i.e., learning tasks, research, assessment, etc.). The last module on 'evaluation' saw low rates of participation (Professional Skill 9 and Professional Skill 10). In spite of the ebb and flow of participation, learning was fluent as faculty participants were made aware of new possibilities concerning their teaching.
Figure 2. Participant Instructors’ Responses to Learning Activities
Content analysis was chosen as a methodology for analysing the online faculty learning activities; this involved comparing and contrasting the activities using a 10 point scale to code and interpret the results. Activity transcripts were scored by the researchers, and the scores were then actively discussed to arrive at a final version where the learning activities had been brought into alignment. Learning activities were scored as: “Maximum Distinction (9-10),” “Important for its Intensity (7-8),” “Suitable (5-6),” “Minimum Qualification (3-4),” and “Differed the Execution (0-2).” Highlights are provided in Figure 3, which demonstrates the ability of participant instructors to apply previously learnt solutions to learning activities. All 'professional skills' were passed by participants, with the exception of the Evaluation Module (Professional Skill 9, knowledge of formative and summative evaluation, and Professional Skill 10, competence to conduct own self-assessment process).
Figure 3. Learning Activity Qualifications
Assessing the cognitive domain of professional skills learning
Ten tests (10 multiple choice items) measured participants’ knowledge
and understanding of 'professional skills.' Test means varied from a high score
of 7.4 (knowledge of student motivation and ability to promote students’ positive
attitudes) to a very high score of 10 (teaching and didactic skills for large
groups). All ten test means exceeded score 7 on the ten-point scale used.
Figure 4. Self-Assessment Test Scores
Means and standard deviations on the ten self-assessment test scores are shown in Figure 4. It was found that faculty participants’ learning was effective. However, objective testing of professional skills showed that faculty participants’ performance was more effective in the 'competence of teaching and didactic skills for large groups,' than in the 'competence of knowledge of area being supervised' (i.e., learning tasks, research, assessment, etc.). There were significant differences in the learning of Professional Skill 3 (competence to solve students’ problems) between participants in regards to gender (t (15) = 2.520, p = .018). Female instructors had more successful results than male instructors. Also, significant differences were found between instructors with and without previous educational knowledge in Professional Skill 1 (knowledge of student motivation and ability to promote students’ positive attitudes) (t (15) = -3.119, p = .008), Professional Skill 3 (competence to solve students’ problems) (t (15) = -2.477, p =.027), Professional Skill 4 (competence to develop meta-cognitive skills in the trainee) (t (15) = -2.385, p =.032), Professional Skill 7 (teaching and didactic skills for large groups) (t (15) = -2.449, p =.028), and Professional Skill 8 (knowledge of questioning skills) (t (15) = -2.590, p =.022). All were in favor of the instructors with previous educational knowledge. Finally, in terms of measuring teaching experience, significant differences in learning Professional Skills 3 (competence to solve students’ problems) were found between new and expert faculty participants (t (15) = 2.800, p =.015).
Research Question 3 asked if was there a relationship between faculty members' teaching attitudes and students’ perceptions of their learning environment after completion of the OFDAS.
Differences between students’ perceptions of actual and preferred environments
Table 4 shows means, standard deviations, and a series of t
tests used for comparison between the two AUTAQ actual and preferred forms.
The findings revealed some clear patterns of differences in the suitability
of the current classroom environments. The scale for classroom climate showed
the highest means in both forms.
Table 4. Means, Standard Deviations, and t-Values for Actual and Preferred forms of AUTAQ
Figure 5 shows the comparison between students' actual and preferred forms. Students were found to be more satisfied with the actual classroom environment scales, than what they perceived from the other scales in the preferred classroom environment form. Moreover, climate scale means were the same in the actual and preferred forms.
Figure 5. Significant Differences Between Student Actual and Student Preferred Perceptions of the AUTAQ
Table 5. Between Scale Correlations Calculated on Items Grouped into their Initial Tentative Scales
This study was designed to assess faculty's professional teaching skills, which are believed to be useful for gaining a better understanding of teaching practice. The aim was to develop and validate a framework of professional skills taught in an online program. The three research questions are discussed below.
Faculty agreed that the professional skills taught in the OFDAS had a positive impact on their teaching skills. The leaders also had a positive impact on participants’ appreciation of the course and skills acquisition. This finding is supported by other researchers who have evaluated Web-based courses (Nijhuis & Collis, 2003). Based on gender, degree, and teaching experience, faculty gave different opinions on the usefulness of professional skills in terms of subject matter, resources, presentations, useful tips, learning goals for skills improvement, and the structure of these professional skills (purpose, uses, educational setting, and case study) for identifying, clarifying, and exploring educational situations. Thus, while the first research question was fully supported, it was found that participants’ opinions varied on the usage and ease of use of professional skills taught in the OFDAS.
Faculty completed a total of 1,587 learning activities over the 11 week duration of the OFDAS. Faculty reported that the learning activities to be generally useful. This finding answers a question raised by Caffarella and Zinn (1999): "Do professional development activities assist in a faculty member's professional success?" (p. 253). Based on this outcomes of the OFDAS, our answer to this question is a resounding "yes." Indeed, all but two professional skills were approved by faculty: the Evaluation Module (Professional Skill 9, knowledge of formative and summative evaluation, and Professional Skill 10, competence to conduct own self-assessment process). All ten Professional Skills test means exceeded 7 on the ten-point scale used. This finding supports our hypothesis that faculty can successfully acquire professional skills using an online program. However, it should be reiterated that significant differences in 'learning' was found between participants in three nominal variables: gender, previous educational knowledge, and teaching experience.
The results stressed two somewhat different but conceptually related measures, which brought about new perspectives on assessing learning environments in higher education settings. The Climate scale in particular emphasized the importance of developing mature, interpersonal relationships, friendships, social bonds, and connections with other students, as a vector of behaviour of student development (Lounsbury, Saudargas, Gibson, & Leong, 2005). Results from the AUTAQ went to each faculty participant, just as Kember, Leung, and Kwan (2002) had done with the student feedback questionnaire used in their study.
Overall, the findings of this study are encouraging. We found that all 'professional skills' taught were perceived by faculty to be useful and easy to implement, though at varying levels depending on the elements employed for the course (i.e., time available). As Fitzgibbon and Jones (2004) previously noted, the coordination of the online program is crucial to its success. Directed and purposeful course design efforts, coupled with activities best suited for various scientific fields, faculty are able to enhance their subject area teaching, with professional pedagogical skills that are easy to use and more importantly, used.
The Attitude Towards Course Learning Questionnaire (ACLQ); the constructivist-based format of the Assessment of University Teaching Activities Questionnaire (AUTAQ); and the other faculty demographic and academic variables explored in this study, all addressed ongoing concerns about the need to improve online training in higher education, as well as emphasizing new ideas about important variables that might be measured as alternatives to the more traditional approaches in evaluation of faculty development (Ellett, Loup, Culross, McMullen, & Rugutt, 1997). The ACLQ and AUTAQ online systems facilitated timely data collection, feedback, and online assessment, a finding supported by previous research undertaken by Tucker, Jones, Straker, and Cole (2003). Finally, the AUTAQ consisted of two sections (although only Section II was used for this study). Section I collected student demographic, academic and social information, but was not used -- as had occurred in the Barfield (2003) study.
An issue arose from this study regarding student online assessment. Our response rates were low, a finding that is supported by previous research by Ballantyne (2003). Nonetheless, faculty reported that collecting feedback online with the AUTAQ system to be convenient, a finding that is supported by earlier research undertaken by Bullock (2003) who investigated a similar online feedback system. Based on this research, we opine that a good starting point for training to enhance faculty scholarship of teaching is built into the design of the OFDAS. In short, the OFDAS is a good starting point for this type of training, because it encourages faculty to become fully involved in the development of online faculty courses. It achieves this by inclusion of learning materials faculty themselves deem necessary to their learning within the scope of the face-to-face workshop, activities, quizzes, grades, and in the direction dialog takes within the OFDAS forum. The selection of 110 learning activities was the organising element of the user/ faculty development-design process of the program. Our focus on learning tasks had also been already underlined by other researchers (Oliver & Herrington, 2003).
One of the limitations found in this study was the fact that it examined solely one faculty online course at only two public urban universities in the Canary Islands. Because faculty volunteered their classes, our pool of respondents were not randomly selected. For these reasons, the study is not intended to be, nor should it be, generalized to other universities.
Based on our experiences at two Canarian universities, we opine that the OFDAS is an effective training model that can be used to improve reflective practice on professional skills. The OFDAS online system is a mechanism used to both enhance online faculty development program management and provide evidence of a quality-improvement process in such online development. Because professional skills framework had been field-tested in this study, faculty knew the impact of professional skills. Finally, no correlation was found between faculty’s teaching attitudes and students’ learning environment at the conclusion of the course.
Aldridge, J. M., & Fraser, B. J. (2000). A cross-cultural study of classroom learning environments in Australia and Taiwan. Learning Environments Research, 3, 101-134.
Badley, G. (2000). Developing Globally-Competent University Teachers. Innovations in Education and Training International 37(3), 244-253.
Ballantyne, C. (2003). Online Evaluations of Teaching: An examination of current practice and considerations for the future. New Directions For Teaching And Learning, 96, 103-112.
Barfield, R. L. (2003). Students’ Perceptions of and Satisfaction with Group Grades and the Grouap Experience in the College Classroom. Assessment & Evaluation in Higher Education, 28(4), 49-64.
Blignaut, S., & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning environments-an exploratory investigation. Computers & Education, 41, 149-172.
Bullock, C. D. (2003). Online Collection of Midterm Student Feedback. New Directions For Teaching And Learning, 96, 95-101.
Caffarella, R. S., & Zinn, L. F. (1999). Professional Development for Faculty: A conceptual framework of barriers and supports. Innovative Higher Education, 23(4), 241- 254.
Dallimore, E. J., Hertenstein, J. H., & Platt, M. B. (2004). Classroom Participation and Discussion Effectiveness: Student-generated strategies. Communication Education, 53(1), 103-115.
Ellett, C. D., Loup, K. S., Culross R., McMullen, J. H., & Rugutt, J. K. (1997). Assessing Enhancement of Learning, Personal Learning Environment, and Student Efficacy: Alternatives to traditional faculty evaluation in higher education. Journal of Personnel Evaluation in Education, 11(2), 167-192.
Fitzgibbon, K. M., & Jones, N. (2004). Jumping the hurdles: challenges of staff development delivered in a blended learning environment. Journal of Educational Media, 29(1), 25-35.
Fraser, B. J. (1998). Classroom Environment Instruments: Development, validity, and applications. Learning Environments Research, 1, 7-33.
Kember, D., Leung, D. Y. P., & Kwan, K. P. (2002). Does the Use of Student Feedback Questionnaires Improve the Overall Quality of Teaching? Assessment & Evaluation in Higher Education, 27( 5), 411-425.
Lounsbury, J. W., Saudargas, R. A., Gibson, L. W., & Leong, F. T. (2005). An investigation of broad and narrow personality traits in relation to general and domain-specific life satisfaction of college students. Research in Higher Education, 46(6), 707-729.
Nijhuis, G. G., & Collis, B. (2003). Using a Web-based Course-Management System: An evaluation of management tasks and time implications for the instructor. Evaluation and Program Planning, 26(2), 193-201.
Oliver, R., & Herrington, J. (2003). Exploring Technology-Mediated Learning from a Pedagogical Perspective. Interactive Learning Environments, 11(2), 111-126.
Summers, J. J., Waigandt, A., & Whittaker, T. A. (2005). A Comparison of Student Achievement and Satisfaction in an Online versus a Traditional Face-to-Face Statistics Class. Innovative Higher Education, 29(3), 233-250.
Tigelaar, D. E. H., Dolmans, D. H. J. M., Wolfhagen, I. H. A. P., & Van Der Vleuten, C. P. M. (2004). The development and validation of a framework for teaching competencies in higher education. Higher Education, 48(2), 253-268.
Tucker, B., Jones, S., Straker, L., & Cole, J. (2003). Course Evaluation on the Web: Facilitating Student and Teacher Reflection to Improve Learning. New Directions For Teaching And Learning 96, 81-93.
Uhlenbeck, A. M., Verloop, N., & Beijaard, D. (2002). Requirements for an Assessment Procedure for Beginning Teachers: Implications from recent theories on teaching and assessment. Teachers College Record, 104(2), 242-272.
Villar, L. M. et al. (2001, ). Metaevaluación: Un inquietante modelo. Revista de Enseñanza Universitaria, 17, Junio, 43-76.
Villar, L. M. (2004). Programa para la Mejora de la Docencia Universitaria. Madrid: Pearson/ Prentice Hall.
Villar, L. M., & Alegre, O. M. (2004). Manual para la excelencia en la enseñanza superior. Madrid: McGraw-Hill.
1. Assessment of University Teaching Activities Questionnaire (AUTAQ)
Instructions
This questionnaire is about your perception of the classroom learning environment. Your opinion is required for each question. For each sentence select the score that best suits your perception. Please answer by circling the number with 1 = ‘strongly disagree’, 2 = ‘disagree’, 3 = ‘neutral’, 4 = ‘agree’ and 5 = ‘strongly agree’.
2. Attitude Towards Course Learning Questionnaire (ACLQ)
Instructions
This questionnaire is about your attitude towards the online learning course. Your opinion is required for each question. For each sentence select the score that best suits your attitude. Please answer by circling the number with 1 = ‘strongly disagree’, 2 = ‘disagree’, 3 = ‘neutral’, 4 = ‘agree’ and 5 = ‘strongly agree’.
