Richard F. Kenny
Athabasca University – Canada's Open University
Mark Bullen
British Columbia Institute of Technology, Canada
Jay Loftus
Athabasca University, Centre for Distance Education
This paper discusses an exploratory study to investigate the existence, and nature, of student problem formulation and resolution processes in an undergraduate online Problem-Based Learning (PBL) course in Agricultural Sciences. We report on the use of a content analysis instrument developed to measure problem formulation and resolution (PFR) processes in online asynchronous discussions (Murphy, 2004a, 2004b) to analyze students' text-based, online discussions. The results offer evidence that students do engage in problem formulation and resolution and that these processes appear to be consistent with the PBL process carried out in this course. However, the nature of the PBL pedagogy, at least in this instructional context, ties the PBL problems to be solved tightly to a marked assignment structure and, therefore, appears to restrict the PFR process in its early and late stages.
Keywords: online learning; problem-based learning; problem-solving; Constructivism; instructional design; content analysis
In a discussion on the International Forum of Educational Technology and Society, Nichols and Anderson (2005, ¶ 12) make two important points about instructional design for e-Learning:
1. E-Learning pedagogies must be defensible, used with some reference to proven educational practice and underpinned by accepted educational theory.
2. E-Learning pedagogies are evolving in the sense that new modes of practice and enhanced technological tools are continually emerging. E-Learning practice cannot remain static, but should instead seek to make the most of new opportunities.
In essence, in designing e-Learning, instructional designers must be guided by research and theory and must be willing to use it to guide them to new and justified instructional practices. In this paper, we examine the use in e-Learning of an established and well-researched pedagogy, Problem-Based Learning (PBL), an educational strategy in which complex, ill-structured problems serve as the context and the stimulus for learning and then we report on a study to investigate the existence and nature of student problem formulation and resolution processes in an undergraduate online Problem-Based Learning (PBL) course in Agricultural Sciences.
PBL contrasts with more traditional subject based approaches where students are first taught a body of knowledge and then may have an opportunity to apply what they have learned to sample problems. With PBL, students work collaboratively in groups to identify what they need to learn in order to understand the problem and to learn about the broader concepts and principles related to the problem. PBL, therefore, is designed to encourage active participation by the students by immersing them in a situation. It requires them to define their own learning needs within broad goals set by faculty, then to identify and search for the knowledge that they need to obtain in order to solve the problem.
PBL, as a pedagogical approach, was developed the 1960s and has been most widely used in Medical Education. It has also been employed in a range of other fields, however, including Nursing, Dentistry and Agriculture (Barrows, 1996, 1998; Boud & Faletti, 1991; Savery & Duffy, 2001).
Barrows (1998) articulated what has become one of the most widely used definitions of PBL. He termed it “authentic PBL” and argued that it has four key characteristics:
Research on PBL, especially as used in medical schools, has focused primarily on comparing the outcomes of PBL methods to more traditional instruction (Albanese, 2000; Albanese & Mitchell, 1993; Colliver, 2000; Smits, Verbeek & Buisonjé, 2002; Vernon & Blake, 1993). Much of this research has focused on the effectiveness of the pedagogy to foster learning.
A review of the literature on effectiveness on PBL in face-to-face instructional settings leads to mixed conclusions. Vernon and Blake (1993) used meta-analysis to compare 35 studies of PBL in medical education. The authors found that PBL was superior with respect to students’ clinical performance, but PBL and traditional methods did not differ substantially on tests of factual knowledge. Albanese and Mitchell (1993) produced similar findings. Students of conventional curricula outperformed PBL students on measures of basic science while PBL students scored higher on clinical examinations.
A more recent study (Dochy, Segers, Van den Bossche & Gijbels, 2003) produced similar overall results. They found a mild negative effect favouring traditional approaches for the assessment of student knowledge, although these differences were encountered in first and second year of medical school and evened out in the last two years. PBL students gained slightly less knowledge but remembered more of it over time (retention). The results for skills were consistently positive favouring the PBL curriculum.
Less work has been done on the specific learning processes occurring in students engaged in PBL (Arts, Gijselaers & Segers, 2002; Hmelo, Gotterer & Bransford, 1997; Kamin, O’Sullivan & Deterding, 2001; Norman & Schmidt, 1992). Gijbels, Dochy, Van den Bossche, & Segers (2005) evaluated 40 studies in order to examine the depth of student knowledge acquisition. They applied Sugrue’s (1995) integrated model of the cognitive components of problem-solving, which proposes that learners’ knowledge structures consist of three levels: 1) understanding of concepts; 2) understanding of the principles linking concepts; and 3) understanding the links from concepts and principles to conditions and procedures for application. The results supported PBL at all three levels but showed that it had the most positive effects when constructs were being assessed at the level of understanding principles that link concepts.
Research on the applicability of this approach in an online, Distance Education, context is also limited (e.g., Atan, Sulaiman & Idrus, 2005; Brown, Johnson, Lima, Boyer, Butler, et al., 2004; Chanlin & Chan, 2004; Ortiz, 2004), although there has been some more extensive work on blended learning or distributed problem-based learning (dPBL) (e.g., Barrows, 2002; Björck, 2002; Bowdish, Chauvin, Kreisman & Britt, 2003; Cheaney & Ingebritsen, 2005; Lehtinen, 2002; Lopez-Ortiz & Lin, 2005; Lou, 2004; Orrill, 2002; Pearson, 2006; Ronteltap & Eurelings; 2002).
Arts, Gijselaers, and Segers (2002) reported the redesign of a course in Business Education to offer PBL in a blended learning environment. Students accessed problem materials on CD-Rom and on the Internet, but met in face-to-face PBL groups. Scores on a knowledge application test indicated that the redesigned PBL-format contributed significantly to improved cognitive gains compared to the regular PBL-setting. However, this was not a fully online PBL course.
Brown et al. (2004) discuss a problem-based learning simulation delivered via the Web for middle and high school students during a five-week period. Both males and females significantly increased their knowledge scores after the completion of the simulation, but from the project description, it seems clear that this instruction did not meet Barrows’ criteria for authentic PBL.
Atan, Sulaiman, and Idrus (2005) compared the performances of students in an undergraduate Physics lesson using the Web-based PBL to that of the same students using a Web-based Content-Based Learning (CBL) in a subsequent lesson. Results significantly favoured the Web-based PBL approach, but are based on a brief, 7-item posttest. The PBL treatment was clearly quite limited in scope and likely could not be considered authentic PBL.
Finally, Pearson (2006) described the design, implementation and evaluation of a module in Business Education in which PBL was used to investigate the challenges associated with the adoption and use of ICT in Hong Kong secondary school classrooms. An evaluation examined five questions dealing with the implementation of PBL, the extent to which PBL facilitated academic discourse, the extent of new knowledge about ICT created, the role of the tutor, and the online learning environment provided, but did not assess the instructional effectiveness of the approach.
Overall, then, there appears to be some evidence that PBL is an effective pedagogy when used over time in whole curricula. However, given the mixed results, it is uncertain that it would make any difference in instruction of shorter duration and it is not yet clear if it can be effectively employed in an online context.
Nonetheless, experimental research studies and quantitative review methods may permit relatively strong statements of certainty about effectiveness, but these statements are typically quite broad – e.g., PBL facilitates the learning of clinical reasoning skills. Such conclusions still tell us little about the cognitive processes underlying learning in such contexts and how specific instructional strategies affect such processes. For instance, Barrows and other proponents of PBL have argued strongly that this instructional approach sets the conditions for effective and deep learning of both disciplinary knowledge and of problem-solving (e.g., Albanese, 2000; Barrows, 1998, Norman & Schmidt, 1992, 2000). Moreover, Barrows (1998) claimed that only “authentic” PBL could foster both the acquisition of a deeply understood knowledge integrated from a variety of disciplines and the development of effective clinical problem-solving [emphasis added].
While all the characteristics of PBL can be seen as important, problem-solving may be key. What does it mean to support problem-solving skills required in clinical or professional practice? What exactly are these skills? How does the instructor facilitate the application and development of effective problem-solving processes and how would one know that problem-solving was occurring?
This paper reports on an exploratory study to investigate the existence and nature of student problem formulation and resolution processes in an undergraduate online Problem-Based Learning (PBL) course in Agricultural Sciences. We accept Jonassen’s (1997) model for solving ill-structured problems, which holds that problem-solving consists of two main categories: Problem Formulation and Problem Resolution (the PFR process). We describe the use of a content analysis instrument developed to measure problem formulation and resolution processes in online asynchronous discussions (Murphy, 2004a, 2004b) to analyze students' text-based, online discussions and the modification of this instrument to more closely match the specific problem-solving processes occurring in PBL. The following research questions guided the investigation:
1. What evidence is there that undergraduate Agricultural Sciences students exhibit problem-solving behaviours and skills in an online PBL course?
2. What is the nature of the problem-solving process that students apply when engaged in online PBL activities?
The participants in this study consisted of the 11 students registered in the course and their instructor. The students were divided into two PBL groups of five and six students respectively and one student dropped the course part way through Case 1 (early October). The membership of the two groups was restructured after each case so that all students in the course worked with each other at some point in the course.
This study examined student and instructor interactions in an online course on AgroEcology, one of two online PBL courses taught in the Faculty of Land and Food Systems at a Western Canadian university. These courses were delivered using WebCT™ Campus Edition 3.8.
The use of incomplete case studies. Barrows (1998) states that a PBL approach must be problem-based – i.e., it should begin with the presentation of an ill-defined, complex, authentic problem. These problems usually consist of descriptions of sets of events that need explanation and provide only limited information. The course material in AgroEcology was introduced through the study of four cases impacting on the practice of Agroecology: 1) grazing ecosystems; 2) organic vegetable production; 3) tree fruit agroecosystems; and 4) genetically modified organisms and rural communities. Students were asked to play the role of consultants to “clients” presented in the case and the four assignments (one for each case) were structured as reports to these clients.
Each case was comprised of multiple rounds, each of which includes several disclosures. These disclosures presented students with the scenario that introduces the problem that they are being asked to address (See Figure 1) or else provides more information about it (supplementary disclosures). In most cases, supplementary disclosures are made available as learners discuss the scenario and identify further information that they require.
Figure 1. AgroEcology Case 2 problem statement.
In AgroEcology, each PBL group had two available discussion forums: a Process and Evaluation Forum (See Figure 2) used by the group to review and discuss ground rules for collaboration as well as the overall process for conducting work within each working round, and a Working (Discussion) Forum used by the group to carry out the actual PBL process itself.
Figure 2. AgroEcology discussion groups.
The Working Forum took the place of face-to-face meetings in which learners engage in various group processes including definition of the problem, development of working hypotheses, organization of the elements of the problem, agreement on research tasks and reporting back on research completed. The instructor participated by monitoring the discussions and making timely postings to encourage student participation, guiding the discussion of controversial points, ensuring that concepts were mastered, encouraging depth of thinking, and verifying the quality of resources used.
Garrison, Cleveland-Innes, Koole, and Kappleman (2006) argue that a sound theoretical framework and model is essential to address validity and to guide a transcript analysis. It is the research question and purpose of the discourse that should determine the model and coding scheme used. Several transcript analysis instruments have been developed recently to measure critical thinking in an online environment (e.g., Garrison, Anderson, & Archer, 2001; McLean, 2005; Meyer, 2004). However, while they appear to be closely related processes, it is not certain that problem-solving and critical thinking are the same thing. A common definition of critical thinking (e.g., Bullen, 1998; Bailin, Case, Coombs & Daniels, 1999) is that it is thinking that is reasonable and reflective and focused on what to believe or do. Garrison et al. (2001, p. 8) view the outcome of critical thinking to be the acquisition of deep and meaningful understanding and to include problem-solving. Bailin et al. (1999), on the other hand, consider problem-solving as an arena in which critical thinking may take place. Jonassen (2000), however, regards problem-solving as a more distinct process. A problem is an unknown entity in some situations (the difference between a goal state and a current state) and problem-solving is the process of finding this unknown (p. 65).
This analysis was carried out using a content analysis instrument recently developed to measure Problem Formulation and Resolution (PFR) processes in online asynchronous discussions (Murphy, 2004a, 2004b) because it was designed specifically to measure the problem-solving process which was the focus of the research questions in this study. This instrument is based on the conceptual framework of Jonassen’s (1997) model for solving ill-structured problems and consists of two main categories: Problem Formulation and Problem Resolution (the PFR process). Each category is in turn divided into several sub-processes and a series of 19 indicators for these processes (See Table 1).
In a previous pilot study (Kenny & Bullen, 2005), we conducted a post-hoc,
descriptive content analysis of all discussions of the Working Forums for both
PBL groups for Case 1. This consisted of 348 separate postings for Group 1 and
309 postings for Group 2. This initial analysis applied Murphy’s (2004b) revised
instrument. On the basis on this pilot analysis, we then further modified the
instrument to more closely match the PBL process occurring in this online course.
Murphy (2004b) points out that, to accurately measure the construct (problem-solving)
that they purport to measure, such instruments must adequately encompass important
aspects of the construct and eliminate aspects distinct from, or surplus to,
the intended construct being measured. To do so, the construct can be refined
empirically through its manifestations in real contexts. The version of the
instrument used in this study is shown in Table 1.
Table 1. Processes and Indicators for Identifying and Measuring
PFR in PBL.
Specifically, we discovered that several aspects of the instrument either did not apply in PBL as implemented in this particular course, or else we found the description of some processes and indicators confusing, missing or out of order. First, no indicator was provided for the process, articulating problem space. This process is a part of the overall PFR process, but, as was also the case in Murphy’s studies, PBL problems are given to the learners, so it is not necessary to include indicators in the instrument.
Second, we added a new indicator, Clarifying Information, because our analysis revealed that students spent a considerable part of their discussion clarifying the meaning of, the importance of, or the accuracy or currency of information. This indicator added specificity to the process of building knowledge. An example of a posting from Case 1 demonstrating this process is listed below:
At the end, AUM does mean the number of animals, but this is where the unit kg comes from.
I hope this is clear to everyone. Ask me more questions if anything doesn't make sense (it's 1:39am and i'm not thinking too clearly anymore . . .).
Third, we added a new process, Reaching Conclusions, to the second phase of Problem Resolution. In Murphy’s (2004b) most recent version of the instrument, this was included as an indicator under the process, Acting on Solutions. We considered this to be a process in and of itself and that it followed directly from the process of evaluating solutions, i.e., one which takes place prior to actually taking action to implement a solution. Murphy (2004b) added this category as a result of the analysis leading to the revisions of the first version of the instrument. She noted that “there were cases where participants indicated intentions, but did not explicitly state a plan of action. Instead, they may have simply reached a conclusion . . . (p. 350).”
Finally, we kept the last process, Acting on Solutions. In her first paper, Murphy (2004a) notes that Acting on Solutions represents the culmination of PFR “whereby individuals can apply the results of a problem in an actual context (p. 10).” Since we had moved Reaching Conclusions up to become a new process, this left only one indicator for this process, which was “planning to take action to resolve the problem.” For instance, Murphy (2004a, p. 12) gives the following example of Planning to Act: “Personally, I have decided to speak English the first day of classes.” This is clearly a statement about what the individual will do as a result of the problem solving process.
However, we concluded that this stage was not relevant for the PBL process as used in this course because students were not actually asked to go out on the farm and apply the solutions. The solution was, in effect, the final assignment for the case, the mock consultant’s report. Initially, the first author considered the following posting from Case 1 an example of planning to act if this process included creating the report:
I was hoping we'd all "vote" for one, although i know it's early, but I think Joan will need some cow math for tomorrow, so for now I am gonna go with my plan as described above and do herd calculations. It's not final, but just to get some numbers . . . :)
After discussion, we decided it did not fit into Planning to Act because it doesn't refer to how the student might apply a solution they worked out. Instead, it simply states what more he is going to do to reach a solution and refers to the organization of the PBL group activities in order to move forward with the class process. We eventually included postings of this sort into a catch all organizational category that we labelled PBL Organization. This may well be an issue of the “artificiality” of PBL problems in that they are realistic, but not actual real life activities.
Garrison et al. (2006) characterize transcript analysis as an exploratory, qualitative methodology and point out that the goal is descriptive, to attempt to understand the existing interactions. As such, they recommend the use of a negotiated agreement in which coders first code transcripts separately and then discuss their respective coding to arrive at a final version in which coded messages have been brought into alignment. This approach was applied to this analysis. The first author coded the discussions for both Group 1 Case 2 and for Group 2 Case 2, while a graduate assistant (the third author) also coded Group 1 Case 2 and the second author coded the Group 2 Case 2 discussion. To code, we followed the data analysis processes outlined in Murphy and Ciszewska-Carr (2005), in which the authors advocate the paragraph as unit of analysis and recommend a three level analysis process: 1) first to code units at the level of the category (i.e., is this unit an example problem formulation/understanding or is it problem solving/ resolution?); 2) next to re-code at the level of the process; and 3) to code a third time using the indicators. While the paragraph was taken as the unit of analysis, we coded each paragraph for instances of (one instance each of) multiple indicators. Finally, each pair of coders met in several sessions to discuss and reconcile disagreements and to arrive at a negotiated agreement as reported below.
As indicated above, we followed a three level analysis process: 1) to first code paragraphs at the level of the category, i.e., as problem formulation or as problem resolution; 2) to next re-code at the level of the process; and 3) to code a third time using the indicators. The total number of postings for the Case 2 Group 1 discussion was 237 and 230 for Group 2. Table 2 shows the results of the coding at the level of category for Case 2 Groups 1 and 2. Nearly half of the postings for Group 1 were viewed as problem formulation while nearly two thirds of the postings for Group 2 were placed in that category, while postings judged as problem resolution ranged around twenty percent. One third of all postings for Group 1 and 20 percent for Group 2 were judged as organizational or social and not representative of the problem-solving process.
Table 2. Coding of PFR Categories in AgroEcology Case 2 - First Pass.
Table 3 provides the results of the codings at the level of process for Case 2 Groups 1 and 2. Two thirds of the postings for both Group 1 and Group 2 were judged to involve the process of building knowledge. Within the category of Problem Formulation, the process of defining problem space was represented by only five percent of the postings for each group.
Table 3. Coding of PFR Processes in AgroEcology Case 2 Group 2- Second Pass.
Less than one third of the postings for each involved the category of Problem Resolution and these were fairly evenly divided between identifying and evaluating solutions. The two groups engaged in the process of reaching conclusions in less than five percent of their activities and there were no coded instances of acting on conclusions.
Table 4 provides the results of the codings at the level of indicator for Case 2 Groups 1 and 2. These findings necessarily mirror those of Pass 2 and provide detail about the nature of the problem-solving processes. Five percent of codings fell within the process of Defining the Problem Space. Of these, the majority (four percent of the total indicator codings) were assessed as identifying the extent of the problem. The majority of codings in Pass 2 were judged as being representative of the process of Building Knowledge.
Table 4. Coding of PFR Processes in AgroEcology Case 2 Group 2- Third Pass.
Most of these were seen as examples of mainly three indicators:
1. Accessing and reporting sources of information, which encompassed nearly twenty-five percent of the indicator codings for Group 1 and well over one third for Group 2.
2. Clarifying the meaning, importance or accuracy of information, which covered nearly eighteen percent of postings at this level for Group 1, and over ten percent for Group 2.
3. Seeking information, which involved nearly ten percent of the activities of both groups.
The processes of Identifying Solutions and Evaluating Solutions were the next most highly represented processes in Pass 2 and evenly represented. The process of Identifying Solutions was described by two indicators, proposing solutions and hypothesizing about these solutions. Both indicators were relatively evenly covered in this analysis and ranged from nearly six percent to twelve percent of the codings in Pass 3. The process of Evaluating Solutions included four indicators: agreeing with solutions proposed by others, weighing and comparing solutions, critiquing solutions, and rejecting solutions judged unworkable. The coverage of these indicators in our analysis was quite variable, ranging from a high of nearly five percent for agreeing with solutions for Group 1 to a low of less than one percent for Group 2.
The process of Reaching Conclusions was described by only one indicator, coming to agreement about solutions. This indicator was judged as occurring in four percent of the codings for Group 1, but the coders for Group 2 were unable to agree if the 12 instances of this process found in Pass 2 were described by this indicator. Rather, it was felt that a new indicator was needed.
The students in this course were engaged in a highly structured PBL process. In each of the four cases, they were presented with an explicit, if relatively ill-defined, problem situation to resolve and were asked to produce a solution in a specific format. For example, for Case 2, they were asked to produce a consultant’s report in the form of recommendation of the re-certification of a farm as organic and to provide a crop rotation plan (see Figure 1). Since this solution was also a course assignment (worth marks), they were not likely to deviate substantially from it. The overall results show that all aspects of the problem formulation and resolution process were being fostered within the parameters of the PBL process applied in this course with the exception of the process of Acting on Solutions.
As indicated previously, analysis of the PFR process occurred in three stages (passes): 1) first at the level of category (Problem Formulation and Problem Resolution); 2) at the level of sub processes for each category (see Table 1); and 3) at the level of indicators for each process. Nearly half of the postings for Group 1 and two thirds of the postings for Group 2 were viewed as Problem Formulation, while postings judged as Problem Resolution ranged around twenty percent. The PBL process is structured to direct learners, within their collaborative groups, to quickly determine what they do and do not know, then to conduct research to fill in the missing information and report back to the group. Only then do they attempt to come to conclusions or develop solutions. Therefore, where students are new to the subject domain, it is not unexpected that a substantial part of the activity of the group be focused on Problem Formulation, especially on the process of building knowledge. This appeared to be the case in this course since the course was offered at the second year undergraduate level (Agro 260) and the course instructor noted that these students were just getting used to the PBL model of instruction (K. Nolan, personal communication, July 15, 2005).
The analysis of the PFR processes occurred in the second stage (Pass 2). No indicator was provided for the process of Articulating the Problem Space and it was not coded. As in Murphy’s (2004a) study, while this process is recognized as a part of the PFR model, explicit and relatively detailed problems were given to the students and they were not required to engage in this activity. Moreover, for both PBL groups, the investigators found low activity (five percent) in the first category of defining the problem space. This is not surprising given the nature of the PBL process and the strong structure of this course. Since the students were required to produce written assignments for marks based on the problems as presented, they were unlikely to disagree with it as stated or to attempt to redefine it. Indeed, for both PBL groups, the greatest number of postings coded as representing this process fell under the indicator, Identifying the Extent of the Problem, which would indicate that the students were not engaged in redefining the problem, articulating new problems, or otherwise redefining the problem space. In PBL as represented in this course, the problem space comes pre-defined.
For each group, the investigators discovered the highest activity by a wide margin under the process of Building Knowledge. Whether conducted in a face-to-face instructional setting or online, the PBL process puts considerable onus on the students to decide what they do and do not know about the problem to be solved and then to conduct research on those topics which are unknown. Topics to be researched are typically divided up between group members, who then report back to the group with their findings. In Case 2 of this course at least, the students clearly focused mainly on seeking and reporting back information and on clarifying the meaning and the importance of what they had found.
After Building Knowledge, the next mostly highly coded processes were those of Identifying Solutions and Evaluating Solutions. Students in this course engaged in both processes nearly equally. The solutions to the problems the students were required to solve in this course, as in all PBL instruction, were complex and composed of multiple components. Group members needed to determine the nature of these sub-solutions and to agree on them. It was, therefore, to be expected that there would be evidence of the processes of identifying solutions, evaluating them and drawing conclusions. While they differed in the quantity of their assessments, both sets of coders found multiple instances of most indicators of these processes. The one exception was that, while they found 12 instances of the process, Reaching Conclusions, the investigators coding Group 2 did not code any instances of the indicator, Coming to Conclusions. Rather, after extensive discussion, we decided that a new indicator, Proposing Conclusions, was needed. The following postings from Group 2 members represent this indicator:
I agree . . . If any certification is granted, there are a lot of things that need to be substantiated. It is strictly against the guidelines to use animal manure from animals that are not 'organic' ones. Because this hasn’t been specified, I think it is safe to grant the farm a conditional certification as Joan has suggested.
And
SO as we have been discussing in the thread below on water quality, maybe the pH level in the water (resulting in it not being classified as acceptable water to rinse the veggies in) is reason enough for Friesen farms not to be recertified? What do you all think?
The most significant finding in this second analysis was that neither investigator coded any instances of Acting on Solutions. This is consistent with our concern about whether this stage is relevant for the PBL process because students were not actually asked to go out on a farm to actually apply the solutions. Rather, they had to write up their solutions as recommendations in a course assignment. Since these assignments were not available to the investigators, there was not a visible product to allow us to verify the existence of this process. This finding is also consistent with the nature of the problems that students were given in this case. That is, they did not require solutions to be implemented but only that the solutions are articulated in a “consultant’s” report. One would not reasonably expect to find examples of Acting on Solutions given this type of assignment.
There are two potentially related problems with the use of the PFR instrument as applied in our analysis: 1) the choice of the unit of analysis; and 2) the accuracy of the current instrument for representing the PFR processes in a PBL context. A third potential limitation relates to the possibility that the transcripts analysed may only be reflecting a restricted component of the PBL process.
Choice of the unit of analysis. As indicated above, we assumed the paragraph to be the unit of analysis. This procedure allowed us to code the same postings consistently among coders. Fahy (2001) pointed out that when the focus is on the meaning [original emphasis] of the interaction of the conference, the unit of analysis [original emphasis] must be something obvious and constant within transcripts. He concluded that this should be the sentence or independent clauses that could be structured as sentences if punctuated differently. The semantic or notional meaning may indeed transcend textual structures, but structural elements of text help form and convey [original emphasis] the notional relationships of the argument (Fahy, 2001, ¶12). Murphy (personal communication, October 4, 2005), however, disagrees with the choice of the sentence as the unit of analysis, arguing that it is insufficient to convey meaning and represents a potentially onerous analysis process. Having now completed two full analyses of PBL cases from this course, we would agree with Murphy that the sentence is rarely sufficient to convey full meaning in this circumstance. We also found, however, that it was often difficult to determine exactly when a part of a posting was a paragraph (e.g., many times, one sentence was separated from the others) and, even when the paragraph structure was clear, this unit was often still insufficient to convey meaning. We found many instances where the meaning of a posting carried over two or more paragraphs. For instance, on a number of occasions, the second author argued that the two indicators, Proposing Solutions and Hypothesizing about Solutions, were inseparable and should be one. Yet, all three coders found instances where a student proposed a solution in one paragraph and then explained it in the following paragraph.
Representing the PFR processes in a PBL context. A second issue concerns the suitability of the PFR instrument for measuring problem solving in a PBL context. Murphy and Ciszewska-Carr (2005) obtained a high level of inter-coder agreement (a kappa coefficient of 0.825 for the two categories of Problem Formulation and Problem Resolution), but they also used the instrument to analyze a discussion that was specifically structured to parallel the problem formulation and resolution process. The discussion was divided into eight tasks, each of which asked the learners to apply a step in the PFR model; e.g., the first task required participants to reflect on their initial knowledge of the problem and to post a message describing their understanding of the problem.
Our results indicate that the PBL process can be seen to broadly follow the PFR scheme. The process of Defining the Problem, however, is minimally represented and only then by one principle indicator (Identifying the Extent of the Problem) and there were no instances of the process, Acting on Solutions. The degree to which learners fully engage in the PBL process depends on guidance they receive via instructional materials and the interventions of the PBL instructor, their understanding of the process and the complexity of the material being engaged. In essence, PBL represents a real life problem-solving activity, but one which may not cleanly compliment the PFR model.
This being said, as discussed above, we did modify the original instrument to more fully match the PBL process on the basis of findings of our pilot study, (Kenny & Bullen, 2005), so one would expect better agreement. It may be necessary to modify the instrument further or else select a different instrument. In particular, the terminology of category, Problem Formulation, creates confusion because, on the face, it suggests that students would be engaged in defining what the problem is. The PBL approach, however, as manifested in this course, provides students with the problem and, through supplementary disclosures, much of the additional relevant information. Their task is to clearly identify the nature of the problem, identify what missing information they need, and to develop solutions. There was no need for the learners to formulate the problem in the sense represented by many of the PFR indicators, such as “specifying ways that the problem manifests itself” or “redefining problem within problem space.”
Rather than a question of clarifying terminology, this may require the recognition of the overlap in these problem-solving processes, that is, to accept that problem-solving is highly recursive in nature. Murphy (2004a) clearly views Problem Formulation as both understanding the problem within its context and building a body of knowledge about the problem area. In applying the instrument in this analysis, we frequently found that those postings we regarded as Building Knowledge fit more within the realm of Problem Resolution than Problem Formulation because they focused on gathering and clarifying information for constructing solutions rather than clarifying the nature of the problem itself. An example follows of a posting which might fit into either (or both) the problem formulation and problem resolution categories. The posting discusses the use of commercial and “home grown” manure on an organic farm.
Right now, I’m not sure what 'words of wisdom' I can offer Ann [the farmer in the case study for whom the report if being written] (I myself am still unsure about integration of all the research info into specific applications for the case). The only thing I can suggest from this info is that Ann is safer to use processed manure from a company, although this is likely to be more expensive than unprocessed manure from a producer. If economics are a concern, and Ann wishes to continue using manure from a producer (or from her own animals; this issue is still up in the air) she must ensure that it ages long enough to avoid the risk of pathogens and introduction of weeds. I think once we have a firm grasp on the soil condition, we can make a good decision on what type of fertilizer should be used (i.e. from what animal and amount it has been processed) and whether or not animals should be kept on the farm.
Clearly, when the students refer to integrating research into specific applications for the case, they are engaging in developing a solution and, therefore, Problem Resolution. Learning more about the issue of which form of manure to use, however, could be seen as either understanding the problem or elucidating a solution.
Possible use of other methods of communication in the PBL process. Did the students also use email or instant messaging, talk on the telephone, or meet in person? Did our analysis miss a significant part of the PBL group process, and hence, the problem-solving activity that occurred during the course? While we did not examine this question directly, there are several reasons to believe that the students in this course did not use other methods to collaborate and that they restricted their PBL deliberations to the working forum provided. First, the course instructions, which directed the students to use the discussion forum provided for group collaborations, were clear and very detailed. Moreover, the instructor was present from the start and very active in guiding the discussions throughout and she only used the discussion forum. The PBL groups appeared to closely follow her guidance in the PBL process and to restrict themselves to the working forum in particular (i.e., there was little use even of the Process and Evaluation Forum). Second, we found no instance in any of the transcripts of students asking fellow members of the PBL group to use a different communications medium (e.g., exchanging instant messaging usernames). In fact, in several postings, group members asked if another student was currently online, that is, the PBL group members were sometimes using the discussion forum as a form of instant messaging by posting and then waiting for a reply. And finally, we did find one instance where two of the PBL group members discussed encountering each other in a different, face-to-face, course. They mentioned how nice it was to meet in person, but made no reference to meeting in this way for the PBL process. In fact, one of the students in one group moved to Central America during the course and communicated from that location for half the course, so meeting in person was not in any case always feasible.
This was an exploratory study designed to examine two questions within the context of the specific, online PBL course examined: 1) what evidence is there that PBL fosters problem-solving behaviours and skills; and 2) what is the nature of the problem-solving process which students apply when engaged in PBL activities? From our analysis, it appears that online PBL can foster problem-solving behaviours in learners, at least in the sense that learners are required to engage in problem-solving activities. The more causal question of whether or not online PBL of this nature teaches or enhances problem-solving skills cannot be answered in an exploratory study and with this data. A future study examining the impact on student activities of both the instructional design of the course and the behaviour of the instructor might begin to shed some light on this question.
Based on our results, it also appears that problem-solving in the online PBL context, as represented by Jonassen’s (1997, 2000) PFR process, is constrained by the instructional design of the course, and, therefore, somewhat limited in nature. Learners were not asked to engage in Articulating the Problem Space and only minimally focused on Defining the Problem Space, nor did they employ the process of Acting on Solutions. Jonassen (2000) differentiates between well-structured and ill-structured problems. Well-structured problems have a clear initial state (what is known) and the nature of the solution is well defined, while ill-defined problems have solutions that are not predictable or convergent which may also require the integration of several content domains.
In this context, students were provided with limited information about the problems, but the problems also had well-defined initial states and required a clear and specific form of solution. Learners were presented with a clear problem statement (See Figure 1) and several further disclosures of information throughout the case. They were also presented on the second day with an extensive set of questions to consider, which helped the students considerably to determine the learning issues they had to address. In other words, they were well-structured problems. In order to more fully engage learners in the initial processes of the PFR model of problem solving, instructors and instructional designers may need to provide much less structure in terms of ill structured, open-ended problems and to allow for more flexibility in the directions that learners can take in arriving at solutions to the problems. It may, for instance, be advisable to avoid linking the PBL problems to specific marked assignments and, instead, base course assessments on other measures of the knowledge and skills acquired through the PBL process.
Finally, while the course instructor regarded the problems as “quite realistic” and, therefore, having some of the characteristics of ill-structured problems by virtue of being similar to those are encountered in professional practice (K. Nolan, personal communication, July 15, 2005), they are not actual, real life problems in which the solutions are to be put into practice. PBL problems tied to field or practicum type experiences might well engender the full range of PFR behaviours and lead to a more complete problem-solving process.
Barrows’ (2000) concept of “authentic” PBL is intended to support the application of problem-solving skills required in “clinical practice.” While this study provides some evidence that online PBL experiences can be designed which do foster problem-solving, more studies examining a range of online PBL contexts and instructional designs will be needed to confirm and detail this conclusion.
Albanese, M. A. (2000). Problem-Based Learning: Why curricula are likely to show little effect on knowledge and clinical skills. Medical Education, 34, 729-738.
Albanese, M. A., & Mitchell, S. (1993). Problem-Based Learning: A review of the literature on its outcomes and implementation issues. Academic Medicine, 68(1), 68-81.
Arts, J. A. R., Gijselaers, W. H., & Segers, M. S. R. (2002). Cognitive effects of an authentic computer-supported, problem-based learning environment. Instructional Science, 30(6), 465-495.
Atan, H., Sulaiman, F., & Idrus, R. M. (2005). The effectiveness of problem-based learning in the web-based environment for the delivery of an undergraduate physics course. International Education Journal, 6(4), 430-437.
Bailin, S., Case, R., Coombs, J. R. & Daniels, L. B. (1999). Conceptualizing critical thinking. Journal of Curriculum Studies, 31(3), 285-302.
Barrows, H. S. (1996). Problem-Based Learning in Medicine and Beyond: A brief overview. New Directions for Teaching and Learning, 68, 3 – 12.
Barrows, H. S. (1998). The essentials of problem-based learning. Journal of Dental Education, 62(9), 630-633.
Barrows, H. S. (2002). Is it truly possible to have such a thing as dPBL? Distance Education, 23(1), 119-122.
Björck, U. (2002). Distributed problem-based learning in social economy – key issues in students’ mastery of a structured method for education. Distance Education, 23(1), 85 – 103.
Boud, D., & Faletti, G. (1991). The challenge of problem-based learning. London: Kogan Page.
Bowdish, B. E., Chauvin, S.W., Kreisman, N., & Britt, M. (2003). Travels towards problem based learning in medical education (VPBL). Instructional Science, 31(4-5), 231-253.
Brown, S., Johnson, P., Lima, C., Boyer, M., Butler, M., Florea, N., & Rich, J. (2004, June). The GlobalEd Project: Problem-solving and decision making in a web-based PBL. World Conference on Educational Multimedia, Hypermedia and Telecommunications, Lugano, Switzerland. Retrieved June 15, 2006 from: http://dl.aace.org/15686
Bullen, M. (1998). Participation and critical thinking in online university distance education. Journal of Distance Education, 13(2), 1-32.
Colliver, J. A. (2000). Effectiveness of Problem-Based Learning Curricula: Research and theory. Academic Medicine, 75(3), 259-266.
Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of Problem-Based Learning: A meta-analysis. Learning and Instruction, 13, 533-568.
Fahy, P. J. (2001). Addressing some common problems in transcript analysis. International Review of Open and Distance Learning, 1(2). Retrieved October 11, 2005 from: http://www.irrodl.org/index.php/irrodl/article/view/321/530
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.
Garrison, D. R., Cleveland-Innes, M, Koole, M., & Kappleman, J. (2006). Revisiting Methodological Issue in Transcript Analysis: Negotiated coding and reliability. Internet and Higher Education, 9, 1-8
Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of Problem-Based Learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75(1), 27 – 61.
Jonassen, D. H. (1997). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63-85.
Jonassen, D. H. (2000). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65-94.
Kenny, R. F., & Bullen, M. (2005, October). A study of problem formulation and resolution in an online problem-based learning course. Paper presented at the annual meeting of the Association for Educational Communications and Technology, Orlando, FL.
Lehtinen, E. (2002). Developing Models for Distributed Problem-Based Learning: Theoretical and methodological reflection. Distance Education, 23(1), 109-117.
Lopez-Ortiz, B. I., & Lin, L. (2005). What makes an online group project work? Students’ perceptions before and after an online collaborative problem / project-based learning (PBL) experience. International Journal of Instructional Technology and Distance Learning, 2(2). Retrieved July 7, 2005 from: http://www.itdl.org/Journal/Feb_05/article04.htm
Lou, Y. (2004). Learning to solve complex problems through between-group collaboration in project-based online courses. Distance Education, 25(1), 49 – 66.
McLean, C. L. (2005). Evaluating Critical Thinking Skills: Two conceptualizations. Journal of Distance Education, 20(2), 1-20.
Meyer, K. A. (2004). Evaluating Online Discussions: Four different frames of analysis. Journal of Asynchronous Learning Networks, 8(2), 101-113.
Murphy, E. (2004a). Identifying and measuring ill-structured problem formulation and resolution in online asynchronous discussions. Canadian Journal of Learning and Technology, 30(1), 5-20. Retrieved January 26, 2005 from: http://www.cjlt.ca
Murphy, E. (2004b). Promoting construct validity in instruments for the analysis of transcripts of online asynchronous discussions. Educational Media International, 41(4), 347 – 354.
Murphy, E. & Ciszewska-Carr, J. (2005). Identifying sources of difference in reliability in content analysis of online asynchronous discussions. International Review of Open and Distance Learning, 6(2). Retrieved October 11, 2005 from: http://www.irrodl.org/index.php/irrodl/article/view/233/408
Nichols, M., & Anderson, B. (2005). Strategic e-learning implementation. Discussion paper of the International Forum of Educational Technology & Society. Retrieved July 7, 2005 from: http://ifets.ieee.org/discussions/discuss_july2005.html
Norman, G. R., & Schmidt, H. G. (1992). The Psychological Basis of Problem-Based Learning: A review of the evidence. Academic Medicine, 67, 557-565.
Norman, G. R., & Schmidt, H. G. (2000). The effectiveness of problem-based learning curricula: theory, practice and paper darts. Medical Education, 34, 721-728.
Orrill, C. H. (2002). Supporting Online PBL: Design Considerations for supporting distributed problem-solving. Distance Education, 23(1), 43-57.
Pearson, J. (2006). Investigating ICT using problem-based learning in face-to-face and online learning environments. Computers and Education 47, 56–73.
Ronteltap, F., & Eurelings, A. (2002). Activity and interaction of students in an electronic learning environment for problem-based learning. Distance Education, 23(1), 11-22.
Savery, J. R., & Duffy, T. M. (2001). Problem Based Learning: An instructional model and its constructivist framework (Tech. Rep. No. 16-01). Indiana University, Center for Research on Learning and Technology.
Smits, P. B. A., Verbeek, J. H. A. M., & de Buisonjé, C. D. (2002). Problem-Based Learning in Continuing Medical Education: A review of controlled evaluation studies. British Medical Journal, 324, 153-156.
Vernon, D. A., & Blake, R. L.(1993). Does Problem-based Learning Work? A meta-analysis of evaluative research. Academic Medicine, 68(7), 550-563.