Volume 17, Number 2
Francesca Pozzi, Andrea Ceregini, Lucia Ferlino, and Donatella Persico
Istituto per le Tecnologie Didattiche – CNR, Italy
The Peer Review (PR) is a very popular technique to support socio-constructivist and connectivist learning processes, online or face-to-face, at all educational levels, in both formal and informal contexts. The idea behind this technique is that sharing views and opinions with others by discussing with peers and receiving and providing formative feedback enriches the quality of learning. In this study, a class of trainee teachers conducts an online PR. The resulting interactions are analyzed and evaluated by the researchers through the application of an evaluation model based on both quantitative and qualitative data. In particular, two conditions are studied, namely the PR in groups versus the PR in dyads. Results show that students who carried out the PR in groups were less active from the cognitive point of view, while they devoted more effort to deal with organizational matters and discourse facilitation.
Keywords: peer review; computer supported collaborative learning (cscl); collaborative technique, evaluation
In the Computer Supported Collaborative Learning (CSCL) research field a lot of attention has been devoted to the use of either collaborative techniques, or “patterns” or “scripts,” which are all instruments able to provide – at different levels of granularity - guidance to students and a structure to the online collaborative activity (Dillenbourg, 2002; Fischer, Kollar, Mandl, & Haake, 2009; Hernández-Leo et al., 2005; Kanuka & Anderson, 1999; Jaques & Salmon, 2007; Pozzi & Persico, 2011).
A collaborative technique (i.e. Discussion, Peer Review, Role Play, Jigsaw, Case Study, etc.) can be defined as a structured content-independent path, which serves to scaffold a learning activity. These techniques usually specify the phase repartition and timing of the learning activity; the nature of the task to be performed and the work distribution among learners and groups; the social structure of the group(s) (in terms of size, composition, etc.); the mode of interaction among participants and groups.
Among other collaborative techniques, the Peer Review (PR) is being increasingly appreciated as a useful technique especially in higher education and in lifelong learning contexts, because of its ability to “model real world professional activities, providing learners with the opportunity to learn how to deal with criticism and how to provide constructive criticism to others” (Anewalt, 2005). In recent years, the rise of interest in peer review is also connected to its potential for increasing the scalability of online learning, especially in relation to Massive Open Online Courses (MOOCs).
In this paper a pilot study is described, which focuses on the use of the PR as a technique to structure collaboration during a CSCL activity. The dynamics and interactions raised by the PR in the experimental context are analysed and evaluated using a mixed approach (Persico, Pozzi, & Sarti, 2010), based on the quantitative and qualitative analysis of the messages exchanged among the students during the activity itself.
The aim of the study is to understand the impact of the proposed technique on the online learning process and, more specifically, to investigate whether different social structures adopted during the activity lead to any difference in the resulting interactions and learning processes.
As the name suggests, educational approaches based on the PR envisage the analysis, by someone, of an artefact produced by a peer. In educational settings, usually, one or more learners revise and provide feedback on the results of the work of their peers, with the aim of improving both the artefact and the learning process. The concept has been widely adopted by the academic community for several generations and has easily found its way into the classroom (Gehringer, 2001), taking from time to time the name of “peer review,’” “peer assessment,’” or even ‘peer grading.”
Whatever the terminology, the technique is based on a reciprocal teaching approach (Rosenshine & Meister, 1994), where one’s own interpretation of reality is to be faced and compared with those of others.
The technique has been applied in various disciplines, such as for example computer science (Anewalt, 2005), L1 and L2 writing (Cho & Schunn, 2007; Lundstrom & Baker, 2009), accounting, mathematics (Gehringer, 2001), psychology (Cathey, 2007), etc.
With the advent of the Internet, this technique has also been widely used in online learning contexts (Cathey, 2007; Kern, Saraiva, & Dos Santos Pacheco, 2003; Lin, Liu, & Yuan, 2001; Liou & Peng, 2009) and some studies have been conducted to compare face-to-face with online PR (Liu & Saddler, 2003; Ho & Savignon, 2007).
Using the 4Ts approach1 to describe this online collaborative technique (Persico & Pozzi, 2011), the PR can be defined as in Table 1.
Table 1
Description of the PR Technique (According to the 4Ts Approach)
| TASK | The task consists of two main phases. In the first phase, individual learners or teams produce an artefact; in the second, peers review it and provide formative feedback. A third phase may take place, during which the authors of the original artefacts revise them according to the received feedback. |
| TEAMS | The task nature does not impose strict requirements on team composition, because both phases can be carried out by individuals, dyads, groups, etc. Once the social structure has been defined, though, it should remain the same throughout the activity. |
| TIME | Since the task is usually carried out in a reciprocal way, individuals /dyads or groups must work synchronously in order to swap the products of their work. This imposes a high degree of structure in terms of time, typically involving two common deadlines: a first deadline to produce the artefact (phase 1) and a second to provide comments on the artefact produced by others (phase2). A third phase often follows, during which artefacts are revised according to the received feedback. |
| TECHNOLOGY | Technology may be synchronous or asynchronous, but ideally it should provide separate spaces for feedback provision on each artefact and interaction with its authors. |
As one may see from Table 1, the Task entails having two different outputs: the original artefacts produced in Phase 1; the feedback and comments provided to the authors of the artefacts in Phase 2. In a third, optional, phase the authors usually revise the artefacts based on the feedback received.
It should be noted that, in this activity, the Team structure is quite important: the PR is not - per se - associated to a specific social structure and can be organized in such a way that the feedback is provided either from an individual to another, or by a pair to another pair, or even among groups. Furthermore, the review process may be orchestrated as networked (any entity may provide feedback to any other entities), or reactive (entity A provides feedback to entity B, entity B provides feedback to entity C, etc.) or reciprocal (entity A provides feedback to entity B, which in turns provides feedback to A) (Strijbos, 2000). As discussed in the following, choosing the most adequate social structure in a PR is critical, and the choice can depend on many different variables, such as the target population and its size, the adopted technology and more in general the contextual constraints, etc. For this reason, it is important to investigate the pros and cons of each social structure and, in particular, understand what kind of learning dynamics are favoured by the different choices.
Since the Task is usually carried out in parallel by several entities (individuals or teams), all entities must work with the same deadlines in order to swap the products of their work and to provide feedback in time to allow the authors to revise the original product.
As to the Technology, it should allow for an easy flow of communications, with the possibility for each ‘entity’ to have a separate ‘space’ where interactions with their peers can take place.
As already mentioned, a number of studies have focussed on this technique and analysed different aspects of its educational value (Kern et al, 2003; Ho & Savignon, 2007; Pozzi, Ceregini, Ferlino & Persico, 2014).
Benefits and characteristics of the application of the PR in educational contexts have been widely debated (see Topping, 1998 for an exhaustive review; Gielen, Tops, Dochy, Onghena, & Smeets, 2010; Lundstrom & Baker, 2009; Mulder, Pearce, & Baik, 2014; Pond & ul-Haq, 1997; Van Den Berg, Admiraal, & Pilot, 2006).
According to Mulder et al. (2014), during the activity, reviewers develop problem-solving skills, while reviewees learn how to cope with diversity of opinions; both improve negotiation skills and critical thinking.
Nonetheless, one of the most critical issue in the PR is the quality and frequency of feedback (Liou & Peng, 2009). Some students do not like to comment on the work done by others, because they do not like to assume a “teaching role” or because of concerns relating to validity, reliability, bias and fairness (Cartney, 2010). For this reason, some authors think this technique better fits with consolidated groups which have already developed a strong sense of community (Rourke, Anderson, Garrison, & Archer, 2001).
Moreover, when feedback comes from a peer, it is probably poorer than a feedback coming from an expert in the discipline (Cho & Schunn, 2007) and poor feedback, or even no feedback at all, jeopardizes the quality of subsequent work. Besides, the ability to provide constructive feedback should not be taken for granted: collaborative and generative feedback should not pass judgment, but rather propose sound arguments, possibly based on ‘consolidated’ theories or criteria. For this reason, many academics stress the importance of students’ training on how to give feedback (Hansen & Liu, 2005; Min, 2006), while others explore the issue of improving learners’ ability to incorporate the feedback received (Liou & Peng, 2009). To face some of these limitations, some authors have developed ad hoc systems, to scaffold the PR process and somehow cope with related problems (Cho & Schunn, 2007).
In any case, a number of studies have demonstrated that even in presence of poor feedback among novice learners, they still scaffold each other’s learning (Teo 2006; De Guerrero & Villamil, 2000)
When the PR is conducted among groups, there is also a particular risk that theorists call “in-group/out-group bias”, namely the tendency by members to consider their group, its members and products, “better” than out-group members and products (Sherif, Harvey, White, Hood, & Sherif, 1961). The fact that another group (i.e. the members of the out-group) evaluate the result of what has been produced inside the in-group, may cause problems and seriously affect the overall process.
Another critical issue connected with the PR is related to the way groups are formed (Hansen & Liu, 2005). As it is easy to understand, reciprocal teaching and peer reviewing exploit pre-existing competence at its top and work better when participants have a variety of initial competences, because one’s own backgrounds and interpretations of reality are to be faced and compared with those of others. For this reason, coupling learners in an effective and balanced way is crucial. To tackle this issue, Crespo, Pardo, Somolinos Pérez, & Delglado Kloos (2005) have developed a matching algorithm to build pairs based on the learners’ profiles and according to a given pedagogical criterion.
Finally, yet importantly, from the point of view of the tutor/instructor, administrating the PR process may turn out to be a heavy task (White & Kirby, 2005). For this reason, systems have been developed through the years to manage the overall PR process, starting from the Daedalus Integrated Writing Environment (http://www.daedalus.com/default.asp), to more recent systems which use the web to manage peer interactions, such as OASIS (Ward, Sitthiworachart, & Joy,2004), PG (Gehringer, 2001) and WebCoM (Silva & Moreira, 2003).
This paper, by studying the nature of interactions that took place during a learning process based on a PR, investigates the learning dynamics ignited by this technique and focuses on the impact that different social structures have on the learning process. This study compares two different social structures adopted during the activity (dyads versus groups), so to evaluate their impact on the resulting learning process.
In the following sections, the context and method of the study are described in details and the results are discussed, aiming to address the following two research questions:
The study was carried out in the context of a real experience consisting of a module on “educational technology and media” proposed within a blended course delivered to trainee teachers in humanities at the University of Genoa in 2013.
The module involved 29 students (19 females; 10 males). Two teachers and a tutor were in charge of it, to support the face-to-face and the online phases respectively.
The students were divided in five groups (G1, G2, G3, G4 and G5). Each group had six members, except for G2, which had 5 members. The task they were requested to carry out in phase 1, after examining a set of resources on the web, was to collaboratively write an essay concerning advantages and drawbacks of the resources examined. In phase 2, they were required to provide feedback on the essay produced by other groups using the PR approach.
Although all the groups were facing the same Task, had the same Time constraints and used the same Technology, the PR script was differentiated as to the ‘Team’ dimension. In particular, two groups (G1 and G2) were asked to carry out the task (i.e. produce the assay and provide the feedback) collaboratively, as a team (see Figure 1), while the other three groups (G3, G4, G5) were organized in dyads (see Figure 2), with each dyad producing an essay and providing feedback to another dyad. The study was aimed at studying the nature of the interactions occurred within each group and detecting differences (if any) between the two social structures.
The university Moodle platform was used for the module and separated forums were made available for the activities of the different groups in each phase.
Figure 1. PR in group (G1, G2).
As one can see in Figure 1, in phase 1, G1 and G2 members interacted internally to their respective groups and the discussion took place in two separate forums, resulting into one artefact per group. In phase 2, G1 members jointly provided feedback to G2 and vice versa (reciprocal feedback; Strijbos, 2000). This happened in two separate forums. For the sake of brevity, in the following, this condition is referred to as “PR in group,” to indicate the overall process enacted by G1 and G2 members during the two phases.
Figure 2. PR in dyads (G3, G4, G5).
As one may see in Figure 2, in phase 1, G3, G4 and G5 members were coupled and each dyad discussed internally to jointly produce an artefact (for a total of nine artefacts). In phase 2, each dyad commented on the work done by another dyad (reactive feedback; Strijbos, 2000). Even if each dyad had a separate space where interactions occurred in both phases, all the interactions were visible to everyone belonging to the same group. In the following, this condition will be referred to as “PR in dyads” and will indicate the overall process enacted by G3+G4+G5.
In order to analyse the nature of the interactions that occurred while performing the PR, an evaluation model is needed. The CSCL literature provides a range of proposals (Henri, 1992; Hara, Bonk, & Angeli, 2000; Rourke et al., 2001; Lally, 2002; Lipponen, Rahikainen, Lallimo,, & Hakkarainen,, 2003; Martinez, Dimitriadis, Rubia, Gomez, & De La Fuente, 2003; Daradoumis, Martinez-Monés, & Xhafa, 2004; ICALTS Kaleidoscope JEIRP2; Schrire, 2006, Strijbos, Martens, Prins, & Jochems,, 2006; Weinberger & Fischer, 2006; Persico, et al., 2010). Among these, the approach adopted for this study (Persico, et al., 2010) was chosen because it provides, in addition to quantitative data on participation, a quantitative view on qualitative data concerning online interactions, fits in well with the constraints of the context of this study and had already been used by the authors in similar online contexts to evaluate other collaborative techniques (Pozzi, 2010; Pozzi, 2011).
According to this model, quantitative data are automatically tracked by the CMC system, while the qualitative data derive from the content analysis of the messages exchanged among participants, which is carried out by human agents. The model, inspired by Garrison and Anderson’s well known dimensions of online presence (Garrison & Anderson, 2003), has been widely tested and subsequently modified according to the results obtained (Pozzi, Manca, Persico, & Sarti, 2007; Persico, Pozzi, & Sarti, 2009a; Persico, Pozzi, & Sarti, 2009b), thus achieving a four-dimensional approach which includes the participative, social, cognitive and teaching dimensions. In the model, each dimension is operationalized by a set of indicators that can be used to evaluate it, summarized in Table 2. An extensive description of the proposed indicators can be found in Persico et al. (2010).
Table 2
Dimensions and Indicators of the Model Adopted (Persico et al., 2010)
| dimension | category | code | Indicators |
| PARTICIPATIVE | P1 - Active participation | P1.1 | sent messages |
| P1.2 | uploaded documents | ||
| P1.3 | attended chats | ||
| P2 - Reactive participation | P2.1 | read messages | |
| P2.1 | downloaded documents | ||
| P3 - Continuity | P3.1 | time distribution of session duration | |
| P3.2 | regularity in reading | ||
| SOCIAL | S1 - Affection | S1.1 | expressions of emotions that may be revealed either by verbal dissertation or through graphical/orthographical solutions, e.g. repetitions, use of punctuation, use of capital letters, emoticons |
| S1.2 | expressions of intimacy that may be revealed by the use of sarcasm, humour, irony, etc. | ||
| S1.3 | “self-disclosure” acts that may be revealed by presentations of personal anecdotes or by admission of self-vulnerability | ||
| S2 - Cohesiveness | S2.1 | occurrences of vocatives or more in general references to other people in the group | |
| S2.2 | expressions reinforcing group cohesion that can be revealed by either expressions of group self-efficacy, or use of inclusive pronouns or adjectives | ||
| S2.3 | greetings, phatics, salutations | ||
| COGNITIVE | C1 - Individual knowledge building | C1.1 | reporting of right contents |
| C1.2 | recognition of a problem or expression of doubts | ||
| C1.3 | explanation or presentation of a point of view | ||
| C1.4 | provision of information or ideas sharing e.g. description of events, accounts of personal experience or real-life stories, etc. | ||
| C1.5 | contribution to brainstorming e.g. by adding an item to a list | ||
| C2 - Group knowledge building | C2.1 | expressions of disagreement that can be revealed by contradicting others | |
| C2.2 | expressions of agreement that can be revealed by referring to others’ messages or by integrating others’ ideas | ||
| C2.3 | suggestions to others and/requests for confirmation e.g. through “explorative acts” such as: “Am I right?”, “Is that so?” | ||
| C2.4 | offers of knowledge or competence to others | ||
| C2.5 | connections between ideas or summarizations | ||
| C2.6 | creation or contribution to the creation of new, shared meanings | ||
| C3 - Meta-reflection | C3.1 | reflections on the learning process that may be revealed by attempting to evaluate one’s own knowledge, skills, limits, cognitive processes | |
| C3.2 | intentional control of the learning process, revealed by planning, monitoring or adjusting one’s own cognitive processes | ||
| TEACHING | T1 - Organizational matters | T1.1 | activity planning |
| T1.2 | methodological proposals e.g. suggestions about the division in groups, proposals of communication rules and netiquette | ||
| T1.3 | organizational proposals e.g. proposing to open a new conference, or organizing a meeting | ||
| T1.4 | offers or requests of logistical information | ||
| T2 - Facilitating discourse | T2.1 | identification of areas of agreement/disagreement, syntheses of discussion | |
| T2.2 | consensus seeking / achievement | ||
| T2.3 | creation of the climate for learning, encouragement, acknowledgement of participant contributions | ||
| T2.4 | solicitation of discussion and reminders of deadlines | ||
| T3 - Provision of instruction | T3.1 | presentation of contents, introduction of new knowledge | |
| T3.2 | in depth analysis of specific topics | ||
| T3.3 | description of new activities | ||
| T3.4 | confirmation of understanding or diagnoses of misconception through assessment and explanatory feedback |
As already mentioned, in this model, the indicators concerning the participative dimension are supposed to be gathered directly from the e-learning platform used (in this case, Moodle), whereas the analysis of the cognitive, the social and the teaching dimensions is based on “manual” content analysis.
However, in this study, the quantitative indicators concerning the participative dimension (first section of Table 2) are of limited significance, due to the fact that participation to the online activities and the production of the essays were course requirements. Furthermore, there were some limitations to the access to the log files and data base of the system which lead the researchers, among the indicators of the participative dimension, to focus on P1.1, the number of messages sent by participants. Given the aims of the study, that is, understanding the learning dynamics, it is the qualitative analysis of the messages that was expected to yield the most significant results.
According to the model, the unit of analysis chosen for the coding procedure of the qualitative analysis is the “unit of meaning,”3 with no limit to the number of units per message. Each unit could be assigned one indicator. This implied that, in case of doubt (where for example a unit could be considered “borderline” between two indicators), a choice was required.
The coding process was carried out by two independent coders who read the messages, segmenting them into units and classifying each unit as belonging to a certain indicator category (Persico et al., 2009a). Since one of the two coders had also been involved in the definition of the evaluation model (Pozzi et al., 2007), she trained the other one (the training lasted about 25 hours). After the training, a sample of messages (about 10% of the total) was selected and coded by both. The sample consisted of a selection of messages distributed in time (at the beginning, in the middle and at the end of the module) and in the various groups. The two coders worked separately on the sample and afterwards the inter-rater reliability was calculated, using Holsti coefficient, 0.85, and percent agreement 0.82. Disagreements were then resolved through discussion. After this tuning phase of the two coders, they worked independently on different sets of messages.
The analysis of the messages was carried out using an ad hoc tool, created for tagging units. The tool is a PHP-based software and allows to easily manage the overall coding procedure by the coders. In addition, it supports the comparison between the codings of two different raters (see Figure 3). The use of different colors for the three dimensions and of labels for the indicators sped up the comparison and the appreciation of the differences.
Figure 3. Screenshot of the tool for message coding (comparison between Coder 1 and 2).
In the following, the results obtained from the interaction analysis are reported.
In total, the number of messages exchanged by the students and analysed by at least one coder was 254. 73 messages were sent by the tutor, whereas 181 were the result of the students’ interactions. Quantitative data about the messages exchanged are reported in Table 3.
Table 3
Synopsys of Sent Messages (M=mean messages per student; MT=mean tutor’s messages per student)
| Students’ messages | Tutor’s messages | Total | |
| PR in groups (G1+G2) | 42 (M=3,82) | 17 (MT=1,55) | 59 |
| PR in dyads (G3+G4+G5) | 139 (M=7,72) | 56 (MT=3,11) | 195 |
| Total | 181 (M=6,24) | 73 (MT=2,52) | 254 |
The quantitative data about participation reported in Table 3 suggest that groups working in dyads were more active than those conducting the PR in groups. Even if the difference between the number of messages sent by the students is affected by the fact that participants in dyads where 18, while participants in groups were 11, the average number of sent messages per student is almost double for those who worked in dyads. Furthermore, the tutor’s contribution to the discussion was, at least quantitatively, higher with the dyads than with the groups.
A general view on the results of the qualitative analysis is reported in Figure 4, representing the number of units detected for each indicator category (S1 = Affection; S2 = Cohesiveness, C1 = Individual knowledge building; C2 = Group knowledge building; C3 = Meta-reflection; T1 = Organizational matters; T2 = Facilitating discourse; T3 = Provision of instructions).
In Figure 4, data are presented per ‘scenario’, whereby the two conditions are juxtaposed: ‘PR in groups’, carried out by G1 and G2 (in blue), and ‘PR in dyads’, by G3, G4 and G5 (in green). Data about the contribution by the tutor are also reported in the same Figure (in grey).
Figure 4. Units detected per indicator category and per condition (PR in groups vs. PR in dyads) + units by tutor in the two conditions.
According to the data reported in Figure 4, interactions were qualitatively quite rich, as all the dimensions are well covered.
The trends followed by the indicators in the two conditions are very similar, i.e. the dynamics triggered by the PR in the two conditions suggest that within the social dimension both S1 and S2 are present; C1 and C2 are definitely dominant as far as the cognitive dimension is concerned, while C3 is almost absent in both conditions; lastly, also T1, T2 and T3 follow a similar trend in the two situations.
In line with the quantitative data about participation, the tutor’s contribution to the discussion was stronger with the dyads than with the groups. As expectable, her contribution was mainly focused on the teaching dimension (especially T1 and T2), together with some social interventions (S2), while she didn’t contribute to the cognitive dimension at all. This aligns well with the rationale for the PR technique, where the cognitive contribution should be entirely a learner’s responsibility.
So far the analysed data have provided an overall picture of the interactions occurred during the activity, but a second aspect we wanted to explore is related to the social structures adopted.
To see if there are significant differences among the groups as far as the indicator categories are concerned (regardless of the phase of the activity), we firstly conducted single ANOVAs (one for each indicator category) (see Table 4). Furthermore, to verify if there was a difference between the two conditions (‘PR in groups’ vs. ‘PR in dyads’), we conducted a Contrast Test to compare G1+G2 and G3+G4+G5 (last column of the Table).
Table 4
Differences among Groups and Contrast Test for ‘PR in Groups’ Vs. ‘PR in Dyads’
| G1 | G2 | G3 | G4 | G5 | F | df | p | Contrast Test | ||||||
| M | SD | M | SD | M | SD | M | SD | M | SD | |||||
| S1 | .642 | .657 | .553 | .760 | 1.114 | .641 | .462 | .565 | 1.331 | .583 | 2.108 | 4 | .107 | -- |
| S2 | .988 | .731 | 1.143 | .976 | 1.096 | .662 | 1.059 | .311 | 2.238 | .521 | 3.502 | 4 | .020* | .118 |
| C1 | .831 | .816 | 2.285 | .799 | 2.604 | .551 | 2.578 | .386 | 2.213 | .501 | 8.588 | 4 | .000* | .001* |
| C2 | 1.325 | .798 | 2.043 | .825 | 2.314 | .379 | 1.997 | .486 | 2.505 | .289 | 3.506 | 4 | .020* | .012* |
| C3 | .000 | .000 | .355 | .462 | .481 | .548 | .000 | .000 | .000 | .000 | 3.321 | 4 | -- (Welch) | -- |
| T1 | .553 | .215 | .888 | .361 | 1.372 | .317 | .597 | .202 | 2.233 | .238 | 6.085 | 4 | .001* | .012* |
| T2 | .495 | .523 | .584 | .627 | 1.865 | .460 | .529 | .439 | 1.499 | .321 | 10.590 | 4 | .000* | .000* |
| T3 | .256 | .452 | .099 | .261 | .366 | .567 | .597 | .496 | 1.096 | .899 | 3.006 | 4 | .113 (Welch) | -- |
As one can see from Table 4, there are significant differences among the groups as far as S2, C1, C2, T1, T2.
As to S2, even if p is significant and the Tukey PostHoc reveals a difference between G1 and G5, the Contrast Test does not show any significant difference between ‘PR in groups’ (G1+G2) and ‘PR in dyads’ (G3+G4+G5).
More interestingly, the Contrast Test reveals significant differences between ‘PR in groups’ (G1+G2) and ‘PR in dyads’ (G3+G4+G5) for C1, C2, T1, and T2.
In particular, the Tukey PostHoc test reveals that G1 reports lower values than all the other groups for both C1 and C2.
As to T1, both G1 and G2 are lower than G5, but in this case, also G4 is lower than G5.
For T2, both G1 and G2 report lower values than G3 and G5; besides G4 is significantly lower than G3 and G5.
Overall, when the PR was organized in dyads, students’ interactions revealed higher levels of the cognitive dimension and this applies to the Individual knowledge building (C1) indicators, as well as to the Group knowledge building indicators (C2).
Furthermore, the PR in groups required a more intense dialogue as far as the teaching dimension is concerned, especially for organizational matters (T1) and facilitating discourse (T2).
In the following, we discuss the PR and its impact in the examined experience.
The study was aimed at understanding the nature of interactions sprung during a PR activity, and to check whether and how interactions varied depending of the social structure adopted during the activity itself. In particular, the study explored the two conditions of learners organized in groups vs. dyads.
First, we should come back to the design phase of our module and to the rationale behind choosing a PR as a technique to scaffold this online collaborative learning process.
One of the purposes of the activity was to solicit learners to explore a number of online resources (freely chosen within a given set) and teach them to critically analyze these resources by identifying their characteristics, strong points and weaknesses. For this reason, asking learners to collaboratively write an essay commenting the visited resources, and then providing feedback to the essay(s) produced by others, seemed a good way to increase the number and types of resources they would practice their critical skills. Furthermore, the target population in our case was composed of adults, i.e. people who should be somehow familiar with the idea of providing and receiving formative feedback.
Overall, our initial expectation was that proposing a PR in this context would trigger Group knowledge-building (C2) and that the quality of the essays would be enriched by the fact that learners were ‘forced’ to question their own opinions and to take into account the formative feedback received by others (Mei and Yuan, 2010).
Looking at the data obtained in the study regarding the cognitive dimension, we can confirm that the Group knowledge-building indicator (C2) played an important role in the overall process (meaning that people were triggered to share, collaborate, and comment on the work done by the others). At the same time, we should also recognize that even the Individual knowledge-building dimension (C1) played an important role, which means that people were triggered to express their own visions, points of view and personal opinions. This is a good point, as a balanced collaborative process typically starts from the presentation of personal points of view (Garrison & Anderson, 2003). The only ‘missing’ indicator is the Meta-reflection, which was almost absent in the interactions occurred among our participants during the PR. This confirms what was already demonstrated in other, similar studies, i.e. that, if this dimension is not explicitly triggered in online collaborative learning activities, it hardly takes place (Pozzi, 2011).
Furthermore, our learners already knew each other, as they were all participants in the same course, so we could count on some kind of sense of community already developed, which is recommended in the literature when a collaborative effort is proposed (Rourke et al., 2001). Our data about the social dimension show good levels of Cohesiveness (S2) and, to a lesser extent, of Affection (S1). The good social dynamics occurred during this activity were certainly very useful to allow a productive discussion, but should not be completely ascribed to the PR itself, as they are probably influenced by other variables of the context.
One of our main concerns as designers of this activity was related to the technology: this was imposed by the University and even if all the learners were familiar with it, we expected it to create some problems, because forums in Moodle are not particularly flexible when you have to manage parallel discussions in groups and sub-groups. This is somehow confirmed by the high number of messages exchanged regarding Organizational and logistical aspects (T1), especially due to people asking or giving information about where the discussion should take place, where to submit essays, etc.
As one may see from the data reported in the previous section, overall the PR solicited heterogeneous interactions, thus confirming its ability to benefit online collaborative learning processes: in both the studied conditions the social and the cognitive dimensions were triggered and the teaching contribution was also required. The trends emerged in the two conditions are very similar and reflect the nature of interactions that we might desire as a result of a PR.
Looking at the differences between the two different social structures adopted (groups and dyads), it seems that dyads worked better: people working in pairs were more active as to Individual (C1), as well as Group (C2) knowledge building. This confirms what is claimed in literature, that learners discussing in dyads feel a stronger sense of responsibility, as the lack of contribution when you work in pairs is much more evident and disruptive than a poor contribution in a group.
Dyads members were also forced to play more the teaching role and it seems that a stronger effort for organizing the work and facilitating the discourse was required to learners in pairs. Although this can be regarded as a positive point, as typically in CSCL contexts learners assuming the teaching role contribute to improve the overall process, we should also acknowledge that PR in dyads was more demanding also for the tutor. Besides, lack of feedback in dyads means no feedback at all so, when this risk is high (for example in MOOCs, were drop out rates are high) perhaps it is better not to run it altogether.
So, even if, overall, the data presented in this study indicate better results for dyads, before deciding to opt for a PR with such a social structure (as opposed to another organized in groups), one should take into account that managing dyads may turn out to be more demanding in terms of effort by the tutor and the students.
Lastly, we should mention one shortcoming of this study, which is the lack of the third phase of the PR that could not be proposed due to time restrictions imposed by the course. This prevented groups and dyads to adequately re-work their artifacts (even if some spontaneous attempts occurred); as a consequence, we are not in the position of evaluating whether and to what extent learners were able to integrate the received feedback and improve their original work. The limited amount of meta-reflection indicators might have been higher, had this phase been allocated an official slot of time.
The PR process is often advocated as one useful approach to be adopted in the educational contexts, as it allows to scaffold learners during a collaborative effort and make them consider points of view coming from their peers, at the same time being solicited to think critically and provide a formative feedback to others.
The contribution that this paper offers to the already rich literature in the CSCL field, concerns the nature of interactions that emerge during an online PR and the differences between groups and dyads while they are carrying such an activity.
The study confirms that the PR is a complex technique to be managed and that one of the most critical aspect has to do with the initial ability/familiarity of learners to give and receive feedback. This technique proved able to scaffold a rich cognitive process, made of personal expressions of ideas, accompanied by discussion and negotiation, thus confirming its ability to benefit online collaborative learning processes.
As far as the comparison between PR in dyads and PR in groups, our data show that dyads worked better, were more active and productive than groups.
Our results support the claim that implementing dyadic PR in online education can give good results in terms of quality of the online learning process. Organizing such technique in dyads, though, requires a number of conditions, such as having an adequate number of learners so to allow an easy management of the activity, having a sound technology, able to properly support numerous parallel communication streams, etc. Furthermore, with the present study we have also confirmed that the effort required by the tutor, as well as by participants, to manage the overall process (in terms of teaching dimension) is quite demanding and this seems to apply more to dyads than to groups. Additional attention could be devoted in the future to the role of the tutor, especially focusing on the way her contribution can influence dyads’ and groups’ interactions.
Given the specificity of the context of this study, further research directions should include similar studies with different targets and with different contextual characteristics, in order to understand whether and to what extent the observed features are generalizable. In particular, another possible research direction includes the analysis of the optional third phase of the PR, to see if metacognitive reflection is better fostered by the revision of artefacts.
1 The 4 Ts stand for Task, Teams, Time and Technology respectively, which have been found to be 4 key dimensions to define an online collaborative technique (Persico & Pozzi, 2011).
2 ICALTS (Interaction and Collaboration AnaLysis supporting Teachers and Students Self-regulation) is a Jointly Executed Integrated Research Project of the Kaleidoscope Network of Excellence, website at http://www.rhodes.aegean.gr/ltee/kaleidoscope-icalts/
3 The ‘unit of meaning’ within a message identifies a consistent “theme” or “idea” (Henri, 1992; De Wever, Schellens, Valke, Van Keer, 2006).
Anewalt, K. (2005). Using Peer Review as a vehicle for communication skill development and active learning. Journal of Computing Sciences in Colleges, 21 (2), 148-155.
Cartney, P. (2010). Exploring the use of peer assessment as a vehicle for closing the gap between feedback given and feedback used. Assessment & Evaluation in Higher Education, 35 (5), 551–64.
Cathey, C. (2007). Power of Peer Review: An Online Collaborative Learning Assignment in Social Psychology. Teaching of Psychology, 34 (2), 97-99.
Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the disciple: A web-based reciprocal peer review system. Computers & Education, 48, 409-426.
Crespo, R. M., Pardo, A., Somolinos Pérez, J. P., & Delglado Kloos, C. (2005). An Algorithm for Peer Review Matching Using Student Profiles Based on Fuzzy Classification and Genetic Algorithms. In M. Ali and F. Esposito (Eds.). IEA/AIE 2005, Lecture Notes in Artificial Intelligence, 3533, 685–694.
Daradoumis, T., Martinez-Monés, A., & Xhafa, F. (2004). An integrated approach for analysing and assessing the performance of virtual learning groups. Lecture notes in Computer Science, 3198, 289-304.
De Guerrero, M. C. M., & Villamil, O. S. (2000). Activating the ZPD: Mutual scaffolding in L2 peer revision. The Modern Language Journal, 84, 51–68.
De Wever, B., Schellens, T., Valke, M., & Van Keer, H. (2006). Content analysis schemes to analyse transcripts of online asynchronous discussion groups: A review. Computer & Education, 46, 6-28. DOI:10.1016/j.compedu.2005.04.005
Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with instructional design. In Kirschner P. A. (Ed.), Three worlds of CSCL. Can we support CSCL (pp. 61-91). Heerlen, NL: Open Universiteit Nederland.
Fischer, F., Kollar, I. Mandl, H. & Haake, J, M. (2009). Scripting Computer-Supported Collaborative Learning. Springer-Verlag New York.
Garrison, R., & Anderson, T. (2003). E-learning in the 21st century. A framework for research and practice. London and New York: RoutledgeFalmer.
Gehringer, E. F. (2001). Electronic peer review and peer grading in computer-science courses. Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, 139-143.
Gielen, S, Tops, L, Dochy, F, Onghena, P., & Smeets, S. (2010). A comparative study of peer and teacher feedback and of various peer feedback forms in a secondary school writing curriculum. British Educational Research Journal, 36 (1), 143–62.
Hansen, J. G., & Liu, J. (2005). Guiding principles for effective peer response. ELT: English Language Teachers Journal, 59 (1), 31–8.
Hara, N., Bonk, C. J., & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28, 115-152.
Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative Learning Through Computer Conferencing (pp. 115-136). The Najaden Papers, New York, Springer.
Hernández-Leo, D., Asensio-Pérez, J. I., Dimitriadis, Y., Bote-Lorenzo, M. L., Jorrín-Abellán, I. M., & Villasclaras-Fernández, E. D. (2005). Reusing IMS-LD Formalized Best Practices in Collaborative Learning Structuring. Advanced Technology for Learning, 2 (3), 223-232.
Ho, M. C., & Savignon, S. J. (2007). Face-to-face and computer mediated peer review in EFL writing. CALICO Journal, 24 (2), 269-290.
Jaques, D., & Salmon, G. (2007). Learning in groups: A Handbook for Face-To-Face and Online Environments, Routledge, London and New York.
Kanuka, H., & Anderson, T. (1999). Using Constructivism in Technology-Mediated Learning: Constructing Order out of the Chaos in the Literature. Radical Pedagogy, 1 (2).
Kern, V. M., Saraiva, L. M., & Dos Santos Pacheco, R. C. (2003). Peer review in education: promoting collaboration, written expression, critical thinking, and professional responsibility. Education and Information Technologies, 8 (1), 37–46.
Lally, V. (2002). Elaborating collaborative interactions in networked learning: a multi-method approach. Proceedings of the Networked Learning Conference 2002. University of Sheffield.
Lin, S. S. J, Liu, E. Z. F., & Yuan, S. M. (2001). Web-based peer assessment: feedback for students with various thinking-styles. Journal of Computer Assisted Learning, 17, 420-432.
Liou, H.-C., & Peng, Z.-Y. (2009). Training effects on computer mediated peer review. System, 37, 514-525.
Lipponen, L., Rahikainen, M., Lallimo, J., & Hakkarainen, K. (2003). Patterns of participation and discourse in elementary students’ computer-supported collaborative learning. Learning and Instruction, 13, 487-509.
Liu, J., & Sadler, R.W. (2003). The effects and affect of peer review in electronic versus traditional modes on L2 writing. Journal of English for Academic Purposes, 2, 193-227.
Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer’s own writing. Journal of Second Language Writing, 18 (1): 30–43.
Martinez, A., Dimitriadis, Y., Rubia, B., Gomez, E., & De La Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers and Education, 41 (4), 353-368.
Mei, T., & Yuan, Q. (2010). A case study of peer feedback in a Chinese EFL writing classroom. Chinese. Journal of Applied Linguistics, 33 (4), 87–98.
Min, H. T. (2006). The effects of trained peer review on EFL students’ revision types and writing quality. Journal of Second Language Writing, 15 (2), 118-141.
Mulder, R. A., Pearce, J. M., & Baik, C. (2014). Peer review in higher education: Student perceptions before and after participation. Active Learning in Higher Education, 15 (2) 157–171.
Persico, D., & Pozzi, F. (2011). Task, Team and Time to structure online collaboration in learning environments. World Journal on Educational Technology, 3 (1), 1-15.
Persico, D., Pozzi, F., & Sarti, L. (2009a). A model for monitoring and evaluating CSCL. In Juan, A.A., Daradoumis, T., Xhafa, F., Caballe, S., Faulin, J. (Eds.), Monitoring and Assessment in Online Collaborative Environments: Emergent Computational Technologies for E-learning Support (pp. 149-170). Hershey, PA: Information Science Reference. DOI:10.4018/978-1-60566-786-7.ch009
Persico, D., Pozzi, F., & Sarti, L. (2009b). Design patterns for monitoring and evaluating CSCL processes. Computers in Human Behavior, 25 (5), 1020-1027. DOI:10.1016/j.chb.2009.01.003
Persico, D., Pozzi, F., & Sarti, L. (2010). Monitoring collaborative activities in computer supported collaborative learning. Distance Education, 31 (1), 5-22. DOI:10.1080/01587911003724603
Pozzi, F. (2010). Using Jigsaw and Case study for supporting collaboration online. Computers & Education, 55, 67-75.
Pozzi, F. (2011). The impact of scripted roles on online collaborative learning processes. International Journal of Computer-Supported Collaborative Learning, 6 (3), 471-484.
Pozzi, F., Ceregini, A., Ferlino, L., & Persico, D. (2014). Learning dynamics of the Peer Review: social, cognitive and teaching aspects. In L. Gómez Chova, A. López Martínez, I. Candel Torres (eds.), Proceedings of the 7th International Conference of Education, Research and Innovation (ICERI2014) (pp. 2591-2598). Barcelona, Spain: IATED Academy.
Pozzi, F., Manca, S., Persico, D., & Sarti, L. (2007). A general framework for tracking and analysing learning processes in CSCL environments. Innovations in Education and Teaching International, 44 (2), 169-180. DOI:10.1080/14703290701240929
Pozzi, F., & Persico, D. (2011) (Eds.). Techniques for Fostering Collaboration in Online Learning Communities: Theoretical and Practical Perspectives. Hershey, PA: IGI Global. doi:10.4018/978-1-61692-898-8
Pond, K., & ul-Haq, R. (1997). Learning to assess students using peer review. Studies in Educational Evaluation, 23 (4), 331-348.
Rosenshine, B., & Meister, C. (1994). Reciprocal teaching: A review of the research. Review of Educational Research, 64 (4), 479–530.
Rourke, L., Anderson, T., Garrison, R., & Archer, W. (2001). Methodological Issues in the Content Analysis of Computer Conference Transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22.
Schrire, S. (2006). Knowledge building in asynchronous discussion groups: going beyond quantitative analysis. Computers & Education, 46, 49-70.
Sherif, M., Harvey, O. J., White, B. J., Hood, W. R., & Sherif, C. W. (1961). Inter-group conflict and cooperation. The Robbers Cave Experiment. Norman OK: Institute of Group Relations.
Silva, E., & Moreira, D. (2003). WebCoM: A tool to use peer review to improve student interaction. ACM Journal of Educational Resources in Computing, 3 (1).
Strijbos, J. W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: What are they talking about?. Computers & Education, 46, 29-48.
Strijbos, J. W. (2000). A classification model for group-based learning. EURODL Journal, Retrieved at: http://www.eurodl.org/materials/contrib/2000/strijbos/strijbos.html (01/04/2015).
Teo, A. K. (2006). Social-interactive writing for English language learners. The CATESOL Journal, 18, 160–178.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68, 249–276.
Van Den Berg, I., Admiraal, W., & Pilot, A. (2006). Design principles and outcomes of peer assessment in higher education. Studies in Higher Education, 31 (3), 341–56.
Ward, A., Sitthiworachart, J., & Joy, M. (2004). Aspects of web-based peer assessment systems for teaching and learning computer programming. In: IASTED International Conference on Web-based Education, 292–297.
Weinberger, A., & Fischer, F. (2006). A framework to analyse argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46, 71-95.
White, T. L., & Kirby, B. J. (2005). ‘Tis better to give than to receive: An undergraduate peer review project. Teaching of Psychology, 32, 259-261.
Dyads Versus Groups: Using Different Social Structures in Peer Review to Enhance Online Collaborative Learning Processes by Francesca Pozzi, Andrea Ceregini, Lucia Ferlino, and Donatella Persico is licensed under a Creative Commons Attribution 4.0 International License.