International Review of Research in Open and Distributed Learning

Volume 17, Number 1

January – 2016

Does Navigation Always Predict Performance? Effects of Navigation on Digital Reading are Moderated by Comprehension Skills

Johannes Naumann¹, Ladislao Salmerón²

¹Goethe-Universität Frankfurt, Germany, ²University of Valencia, Spain

Abstract

This study investigated interactive effects of navigation and offline comprehension skill on digital reading performance. As indicators of navigation, relevant page selection and irrelevant page selection were considered. In 533 Spanish high school students aged 11-17 positive effects of offline comprehension skill and relevant page selection on digital reading performance were found, while irrelevant page selection had a negative effect. In addition, an interaction between relevant page selection and offline comprehension skill was found. While the effect of relevant page selection was strong in good offline comprehenders, it was significantly reduced in weak offline comprehenders. The effect of offline comprehension skill was strong in students showing high rates of relevant page selection, while it was weak and insignificant in students showing low rates of relevant page selection.

Keywords: Hypertext, Navigation, Comprehension, Log-File Analysis

Introduction

In learning scenarios where materials are delivered online, e.g. in distance education programmes, students frequently need to progress on their task by navigating through a network of linked materials (or nodes). Specifically, during navigation they have to assess the relevance of the available hyperlinks, and decide which one may contain information useful to answer the question. If students access a node with relevant information, they may still have to assess if the information obtained from this node is enough, or if they need to access further relevant links to provide a complete answer. Think of a person who considers taking up studies in, e.g., Psychology in a distance learning setting, and accesses the website of, e.g., the Open University. Even before enrolling in a study program, they will need to access a number of hyperlinks, and read and integrate the documents opened by these, to find out about course contents, fees, terms of application, etc. After having enrolled, they will need to complete study assignments sent to them online. Again, completing these assignments, e.g., when searching for literature to complete an essay, they will need to engage with navigating documents online by accessing certain hyperlinks, and discarding others when they are not relevant to the task at hand. Thus, theoretically, navigation is a crucial process in online and distance learning. In addition, traditional or offline comprehension skills are needed to process the documents accessed through the navigation process (e.g., Salmerón & García, 2011). This means that in online learning scenarios as well, students need to decode words, parse the syntax of sentences, and execute local and global coherence processes to finally understand a document’s contents (e.g., Kintsch, 1998). This article aims at a contribution to clarifying how navigation processes and offline comprehension skills interact in online learning scenarios. In the following section, we first review the existing literature on this interaction.

Evidence for the Interaction of Offline Comprehension Skills and Navigation Processes

There is clear evidence that students’ navigation, as indicated by their traces in log files, play a major role in online question-answering tasks (Organization of Economic Co-Operation and Development [OECD], 2011). Specifically, a large scale study involving the adolescents participating in the OECD Programme for International Student Assessment (PISA)¹ 2009 electronic reading assessment revealed that students who displayed a more task-oriented navigation behaviour, as indicated by more visits to task-relevant pages, correctly responded to a higher number of questions. This result is in line with several other small scale studies that have looked at how adolescents and undergraduate students navigate in different learning tasks. These tasks include searching for information (Cress & Knabel, 2003; Hsu & Schwen, 2003), studying for a semester-long course (Puntambekar & Goldstein, 2007), reading for comprehension (Salmerón, Cañas, Kintsch, & Fajardo, 2005; Salmerón, Kintsch, & Cañas, 2006), reading to prepare a summary (Richter, Naumann, & Noller, 2003; Naumann, Richter, Flender, Christmann, & Groeben, 2007), or performing a science inquiry task (Goldman, Braasch, Wiley, Graesser, & Brodowinksa, 2012). In addition, OECD (2011) showed that students with good offline comprehension skills, as measured by a paper and pencil reading comprehension test, employed better navigation paths, which in turn improved even more their online performance. While this mediating role of navigation on the relationship between offline comprehension skills and online performance has been already reported in small scale studies (Naumann, Richter, Christmann, & Groeben, 2008; Salmerón & García, 2011), less is known about the potential moderator effect of offline comprehension skills on the relationship between navigation and performance (e.g., navigation may predict online performance only for students with sufficient offline comprehension skills). The goal of this study is to shed light on this aspect.

The fact that navigation, as measured by log files, predicts students’ performance in several online learning tasks has raised the attention of scholars due to its important implications on a theoretical and on a practical level (e.g. Güyer, Atasoy & Somyürek, 2015). On a theoretical level, navigation is important because it is highly interlinked with self-regulation learning (SRL) in online scenarios. SRL models describe students’ decisions in complex learning scenarios, such as deciding which hypertext sections to study and which one not. These decisions are based on students’ reflections on the constraints imposed by the scenario, their task goals, and the revision of their performance during the task (Winne & Hadwin, 1998). In addition, SRL models assume that learners try to optimize both achievement and how achievement is reached (Winne & Baker, 2013). In online learning, readers are provided with tasks that require a self-directed selection and organization of text materials. In terms of achievement, there is ample evidence that increased self-regulation in online learning goes together with better learning outcomes (e.g. Azevedo & Cromley, 2004; Azevedo, Greene, & Moos, 2007; Azevedo, Moos, Greene, Winters, & Cromley, 2008). In terms of how achievement is reached, there is evidence that better self-regulation goes together with better navigation, and that better navigation partly mediates effects of self-regulation on learning outcomes in online learning scenarios (Richter, Naumann, Brunner & Christmann, 2005; Salmerón, Kintsch, & Kintsch, 2010). Thus, altogether, the analysis of navigation behavior can provide rich information to self-regulated models of learning (Winne, 2010).

On a practical level, the fact that navigation, as measured by log files, predicts students’ performance in several online learning tasks makes it useful for intelligent online systems. Log files can be traced to build user models, and recommendations can be provided during the learning session on the basis of students’ navigation. For example, if the system detects that a student is navigating task-irrelevant nodes, it may react, prompting her to focus on more relevant ones (e.g. Puntambekar & Stylianou, 2005; Vidal-Abarca et al., 2014; Winne, & Hadwin, 2013).

The current study questions the widespread assumption that navigation univocally predicts performance in question-answering online tasks. Specifically, we tested the assumption that the positive relationship between navigation and performance may be moderated by students’ offline comprehension skills. To the best of our knowledge, the potential moderator effect of comprehension skill on the relation between navigation and performance has not yet been investigated. Theoretically, moderation takes place when offline comprehension skills and navigation interact statistically in predicting online reading performance. A student might lack offline comprehension skills, but still have the skill necessary to navigate a digital text. For example students may match words in the question to words in the text, without a thorough examination of the relevance of that particular text section (Cerdán, Gilabert, & Vidal-Abarca, 2011; Salmerón, Cerdán, & Naumann, 2015), or they may use typographical cues of hyperlinks to guide their navigation (Rouet et al., 2011). In this case, however, despite an apparently good navigation, comprehension outcomes will be poor. This is because a text that is ‘created’ through a readers’ self-guided selection and ordering of text materials, i.e. navigation, has to be read and understood eventually. Thus, according to this reasoning the effect of navigation on digital reading performance will be conditional on offline comprehension skills. In a student with good offline comprehension skills, a navigational path containing a large percentage of task-relevant materials will lead to good comprehension outcomes. A student with poor offline comprehension skills, in contrast, will not be able to utilize such a text.

Our study aimed at testing the hypothesis that the relation between navigation and performance is moderated by students’ offline comprehension skills. To do so we used a large (N=533) age-stratified sample comprising students aged eleven to 17 (see Table 1).

Method

Subjects

533 Spanish students participated in the study. Students were sampled from six different schools in the province of Valencia, Spain. Within each school, one complete classroom of 7^th, 8^th, 9^th, and 10^th graders was sampled. Schools and children were recruited to participate under specific agreement of collaboration between principals, regional educational authorities and the research team.

Table 1

Descriptive statistics and correlations of all variables in the study

Note. N = 533. Relevant page selection: Percentage of task-relevant pages visited. Irrelevant page selection: Percentage of task-irrelevant pages visited. Digital reading performance: Percentage of correctly answered questions in the digital reading test.

^a0 = No, 1 = Yes. ^b0 = female, 1 = male.

** p <.01, *** p <.001 (two-tailed).

As can be seen from Table 1, students’ age varied between eleven and 17 years (Table 1, row 8). Roughly half of the students were each male and female (Table 1, row 4). Most students had access to computers and the Internet in their homes (Table 1, rows 2-3). Also, most students had used a computer before (Table 1, row 1).

Materials and Measures

Digital reading performance. Digital reading performance was measured through a series of tasks from a test of digital reading skills developed for this project. The current study forms part of a project aimed at developing a test of digital reading skills for secondary school students. The test consists of 36 items covering the aspects identified in the PISA framework (OECD, 2009): accessing and retrieving (12 items), interpreting and integrating (13 items), and reflection and evaluation of information (11 items). Items are distributed into four scenarios that emphasized different uses of digital reading: educational (Wikipedia), multiple documents (Google), social (web forums) and navigation (web portal). The materials used in the test underwent a strict procedure of construction and validation (Salmerón et al., 2012): they were developed and subsequently tested in three pilot studies involving 60-80 students in each case. A final version of the test was tested in a sample of 555 students. As a measure of concurrent validity, we first correlated the scores students’ performance with the scores obtained with a standardized reading comprehension test (Llorens et al., 2011). Pearson correlation was.65, p < 001. Additionally, we correlated students’ performance and students’ grades on Spanish (.38), Social sciences (.39), Natural sciences (.39), and Mathematics (.31) (all correlations p <.01). Results from that sample indicated that question difficulty ranged from.18 to.89 (i.e. success rate). Finally, the reliability of the questions (Cronbach α) was.78.

In this study we used the web portal scenario of the test. Specifically, the materials consisted of a total of 12 items that students solved navigating through a web portal for youngsters. The web portal was organized around five main sections (Environment, Technology, Health, Sports, Courses), each of them including three subsections, which were further divided in three additional content pages. Some of the content pages included links to pages located outside the web portal. Thus, students navigated through a variety of web pages, including news pages, commercial sites and Google Maps mocks. Items assessed accessing and retrieving (5 items), integrating and interpreting (4 items), and reflecting and evaluating textual information (3 items). For example, one accessing and retrieving item requested: ‘The Youth Web has just released a report about endangered birds and mammals. What is the level of endangered species in Canada?’ To answer the question, students had to access the following route starting from the home page of the Youth Portal: Environment > Endangered species > Threatened animals across the world, while avoiding other irrelevant links such as ‘Endangered animals: Iberian Lynx’. In terms of the PISA framework, all content pages were authored texts; 66% tasks related to continuous texts and 33% related to non-continuous texts. In addition, all tasks required between two and six navigational steps to find the target information ( M = 2.92 steps, SD = 1.16), i.e., all tasks related to multiple web pages. All tasks were presented in a mimicked browser. Each item referred to information from a particular subsection that was not targeted by any other item before. The reliability of the items (Cronbach’s α) was.60.

Offline comprehension skills. Offline comprehension skills was measured through the Reading Literacy test for Secondary Education (CompLEC) by Llorens et al. (2011). The test is composed of five units, three continuous and two non-continuous texts. Topics covered included a textbook excerpt about global warming, an article from a popular science magazine about the language of the bees, a pamphlet about the postural ergonomics of work chairs, two letters to the editor about the pros and cons of nuclear energy, and a newspaper diagram about traffic accidents. Each unit included 3-5 questions covering the different aspects suggested by the PISA framework: accessing and retrieving (5 items), integrating and interpreting (10 items), and reflecting and evaluating textual information (5 items). The test has been validated with a sample of 1854 students from 42 different high-schools. Its reliability in the present sample was.80 (Cronbach’s α).

Navigation. Log files were used to analyze students’ navigation behavior. We used as an index of the navigation quality the percentage of task-relevant pages visited for each item (relevant page selection). The number of task-relevant pages visited reflects the process of task-accurate selection of learning or reading materials from online sources and can thus be assumed one crucial aspect of the navigation process, and has thus been shown to be associated with the quality of comprehension and learning (e.g. Cress & Knabel, 2003; Goldman et al., 2012; Naumann et al., 2008). Pages were considered relevant for a particular question if students needed to access them in order to locate information useful to answer that question. Thus, this index not only considers if a participant accessed or not the page or pages that contained useful information, but more precisely the extent to which she followed the path leading to those useful pages. To rule out the possibility that this index merely reflects the length of a navigational path, we considered in addition the percentage of irrelevant pages that were accessed by a student in each task (irrelevant page selection). While we predicted a positive effect of relevant page selection on digital reading performance, we predicted a negative effect of irrelevant page selection.

Procedure

Students were tested in their classrooms during school hours in two one-hour sessions. In a first session, students completed the offline comprehension skills test. After completion, they filled in two questionnaires asking for demographic information and about the frequency of their use of computers. In a second session, students completed the digital reading performance measure in the schools’ computer lab. Data was collected using USB devices.

Statistical Analysis

To test our hypothesis about the moderator role of offline comprehension skills on the relation between navigation and performance, we estimated a linear mixed model, using the R environment (R Development Core Team, 2012) with the package lme4 (Bates, Maechler, & Bolker, 2012). The predictor variables offline comprehension skill, relevant page selection, and irrelevant page selection were entered as fixed effects. In addition, the interactions of offline comprehension skill with relevant page selection and irrelevant page selection were entered as fixed effects. Since the predictor variable offline comprehension skill could be expected to covary with age, and thus be confounded with other age-related variables, we entered students’ age as a control variable. All predictor and control variables were z-standardized. Since complete classrooms were sampled, we entered a random classroom intercept to account for dependencies between students coming from the same classroom. As a measure of effect size, we report as a default the proportion of variance uniquely accounted for by each fixed effect (marginal Δ R ² following Nakagawa and Schielzeth, 2013). The significance of the random classroom intercept effect was evaluated using a simulation-based exact likelihood test with the package RLRsim (Scheipl, Greven, & Kuechenhoff, 2008).

Results

Descriptive statistics and correlations for all variables in the model are given in Table 1 (rows 5-9).

Recall that our hypothesis claims that the effect of relevant page selection on online comprehension is enhanced by offline comprehension skills, since without those appropriate skills, students displaying whatever navigation behavior will not be able to completely understand a digital text. This means that in a regression model with offline comprehension skills and relevant page selection as predictors, plus an interaction term, a positive regression coefficient should emerge for the interaction, indicating an increasing impact of relevant page selection on digital reading performance with increasing offline comprehension skills. In contrast, no such interaction effect was expected for irrelevant page selection. Selection of irrelevant pages can be seen as detrimental to digital reading performance, regardless of a student’s offline comprehension skill. Model results are summarized in Table 2.

Table 2

Model results

Note. R ²m: Marginal R ², variance explained by fixed effects. R ²c: Conditional R ², variance explained by fixed and random effects (Nakawa & Schielzeth, 2013). Effects sizes (decrements in variance explained by fixed effects) are reported for significant effects only.

* p <.05, ** p <.01, *** p <.001.

The random effect for classroom was significant, meaning that there was systematic between-classroom variance in digital reading performance over and above what was explained by the fixed effects in the model (see Table 2).

Analysis of fixed effects revealed a large positive effect of relevant page selection (Table 2, row 3), while irrelevant page selection had a negative effect as expected (Table 2, row 4), which was however small. Also as expected, offline comprehension skill had a medium-sized positive effect (Table 2, row 5). In addition to these main effects, there was a positive effect for the interaction between relevant page selection and offline comprehension skill (Table 2, row 6). The interaction between offline comprehension skills and irrelevant page selection was not significant (Table 2, row 7). The control variable age had no effect over and above the other predictors in the model (Table 2, row 8).

In line with our hypothesis, the positive sign of the regression coefficient for the interaction between relevant page selection and offline comprehension skills indicated that positive effects of relevant page selection on digital reading performance were accelerated with increasing offline comprehension skills. Accordingly, positive effects of offline comprehension skills were accelerated with increasing rates of relevant page selection. To further interpret this interaction, we conducted simple slopes analyses (see Aiken & West, 1991).

Firstly, we estimated the effects of relevant page selection on digital reading performance conditionally on high offline comprehension skills (one standard deviation above the mean, see the straight line in Figure 1a) and low offline comprehension skills (one standard deviation below the mean, see the dotted line in Figure 1a).

Figure 1. (A) Simple slopes for the moderated regression of digital reading performance on relevant page selection, conditional on high offline comprehension skill (+ 1 SD, straight line), and low offline comprehension skill (-1 SD, dotted line). (B) Simple slopes for the moderated regression of digital reading performance on offline comprehension skill, conditional on high rates of relevant page selection (+ 1 SD, straight line), and low rates of relevant page selection (-1 SD, dotted line). See text for significance tests.

For students high in offline comprehension skills, a strong effect of relevant page selection on digital reading performance was found, β = 0.17 ( SE = 0.01), t = 12.50, p <.001. For students low in offline comprehension skills, this effect was still significant, but reduced to about 60% of its size, β = 0.10 ( SE = 0.01), t = 9.44, p <.001.

Secondly, we estimated the effects of offline comprehension skill on digital reading performance conditionally on high rates of relevant page selection (one standard deviation above the mean; see the straight line in Figure 1b), and low rates of relevant page selection (see the dotted line in Figure 1b). For students selecting many relevant pages (one standard deviation above the mean), there was a strong effect of offline comprehension skill on digital reading performance, β = 0.08 ( SE = 0.01), t = 8.00, p <.001. For students selecting few relevant pages, in contrast, the effect of offline comprehension skill was not significant, β = 0.02 ( SE = 0.01), t = 1.73, p >.05.

Discussion

To the best of our knowledge, the present study is the first to provide clear evidence that the relationship between relevant page selection and performance in question-answering online tasks is moderated by students’ offline comprehension skills. While the results replicate the well-established relationship between navigation and performance in online tasks (see Naumann, 2008, for a review), they reveal that the influence of relevant page selection can drop to approximately 60% if students have low offline comprehension skills. They do also reveal that good offline comprehension skills are not sufficient in themselves to produce good digital reading performance. Rather, if students fail to comply with demands of relevant page selection, the otherwise strong and positive association of offline comprehension skill and digital reading performance is no longer significant.

One possible explanation for these results is a threshold model. According to such a model, when a critical threshold of offline comprehension skills is reached, students can benefit from their efficient navigation to improve their digital reading performance. It is a reasonable assumption that students would need to score on the PISA reading proficiency level 3 (out of 6) or above. This means, for instance, that they are able to, e.g., find multiple pieces of information in a text, or link information from different texts to one another (see OECD, 2010, for a detailed description). This assumption however needs to be corroborated by future research as the offline comprehension skills measure used in this study is not linked to the PISA reading scale.

It is strong comprehenders who benefit from efficient navigation of task relevant pages. Why do, in contrast, poor readers not benefit so much from their navigation? From the perspective of a developmental model, one can argue that these students still fail on matching their navigation goals to the available links by means of inferences. Thus, their assessments of a link’s relevance might rely on more superficial processes, such as links’ typography (Rouet et al., 2011) or word matching (Cerdán et al., 2011; Salmerón et al., 2015), e.g., if the goal and the hyperlink share a particular word. By using those cues to navigate, students may still access pages that are relevant for their goal, but nonetheless they won’t be able to make complete sense of the information on those pages. Log files provide information about the percentage of relevant pages visited by students, but they don’t give a complete picture of which cognitive operations students actually use to select those pages. A possible way to clarify this issue is to combine the information from log files with other online measures, such as students’ verbal protocols (e.g. van Gog, Paas, van Marriënboer, & Witte, 2005; Richter et al., 2005). Future research should clarify this issue.

In claiming a moderation effect of offline comprehension skills for the relationship between navigation and online performance, some precautionary remarks are in place as well. First, while it was one crucial property of navigation that was investigated in the study reported here, the selection of relevant (and irrelevant) text materials, this is by no means the only important property of a navigational path. For instance, coherence of the navigational path has been established as another variable that is predictive of comprehension and learning outcomes in reading and comprehending digital text (Amadieu & Tricot, 2008; Naumann et al., 2007; Richter et al., 2005; Salmerón et al., 2005; Salmerón, Kintsch, & Cañas, 2006), as has the use of navigation help devices such as topical overviews (Dias & Sousa, 1997; Salmerón & García, 2011; Su & Klein, 2006). For these properties of navigation, for the time being, we do not know whether, and how, their impact on performance is moderated by students’ skills.

Second, although in our study offline comprehension skills moderate the relationship between navigation and performance, navigation has a positive effect in most students. Other patterns of moderation can be observed. For example, a particular navigation pattern may be useful for students with or without a particular skill, but may have no effect on others. Although it is not directly linked to navigation, related evidence on the effect of overviews in hypertext exemplify these potential moderator patterns. In this line, Salmerón, Baccino, Cañas, Madrid, & Fajardo (2009, study 1) found that providing a thematic overview for a hypertext had a positive effect on students’ comprehension, only if the hypertext was difficult and if students had low background knowledge on the topic. Finally, while offline comprehension skills is a strong predictor of navigation, research has identified other students’ individual variables that are also important to articulate an efficient navigation, and thus their potential role as moderators should be studied in the future. One such important variable is background topic knowledge (e.g., Coiro, 2011). We could expect that, given a similar percentage of relevant pages visited, students with high background knowledge may fill in the existing gaps in their response, while students with low background knowledge may have difficulties overcoming an incomplete response generated through their navigation (cf., Taub, Azevedo, Bouchet, & Khosravifar, 2014).

Implications for Research on Navigation

What are the potential implications for the moderated relationship between navigation and online performance? Here, the present research carries two messages. First, at a theoretical level, our results imply that similar navigation traces may be reflecting different cognitive and metacognitive processes. This is a challenge for using navigation traces to inform and enrich self-regulation models of learning. A way to limit the number of potential processes underlying the selection of a particular navigation path is to include theoretically motivated hyperlinks, the selection of which can be easily linked to a unique theoretical explanation (Winne, 2010). For example, a recent question-answering study using Wikipedia (Salmerón et al., 2015) included three types of hyperlinks embedded in relevant sections of the hypertext: a) relevant uncued links that paraphrased the main idea of the question, and that led to a hypertext node with useful information to answer the question; b) irrelevant cued links that included a word that matched with a word in the question, although it was not representative of the main idea of the question, and the link led to an irrelevant node, and c) irrelevant uncued links that didn’t include a paraphrase or a word match to the question. Results from the navigation patterns followed by the students allowed to identify which relevance cues (semantic and/or word matching) they were using to navigate the hypertext.

Second, at a practical level, the existence of a moderation effect indicates that navigation alone, at least the navigation index used in this study, i.e., percentage of relevant pages accessed, can’t be used to accurately predict students’ performance in online question-answering tasks. An intelligent system that reacts to students’ navigation behavior, without considering important student variables, such as offline comprehension skills, will generate an inaccurate representation of the user, which may lead to less useful recommendations. For example, if a system identifies that a user is deviating from the optimal path to find relevant information, recommending an optimal route may be useful if the student has the necessary skills to realize that she is in an unfruitful path, but it may be less efficient if the student does not have the necessary reading skills to do so. Thus, in designing distance learning materials clearly usability in terms of navigation is important, i.e., supporting students to actually identify the relevant parts of the materials given the aims of study. What might however be equally important as suggested by the present results is to look at the text’s readability and difficulty itself. Since online learners’ offline comprehension skills might be measured before students engage with the materials, they cannot be changed in the short term. Thus, facing students with comprehension difficulties, for a course designer, it will be of importance to create materials as easy to comprehend as possible. Otherwise, as suggested by the present results, increasing the materials’ usability in terms of navigation won’t help much.

As we see it, the present analyses however already make a strong argument that while navigation does play a crucial role in digital reading performance, its effect is moderated by the more traditional offline comprehension skills. This also means that instructional programs which aim at strengthening students’ navigation processes, preparing them for a digital 21^st century world where open and distance learning scenarios are ubiquitous, should take students’ comprehension skills into account. Probably, students who struggle with comprehension will need support to strengthen their comprehension skills first, before they can benefit from instruction targeted at their preparedness to adequately select hyperlinks in online reading scenarios.

Limitations

Finally, some cautionary remarks about limitations of the present results are in place. First of all, our data base was correlational. This means that the present results should be backed, e.g., by training studies, where in the best case offline comprehension skill and navigation processes are manipulated experimentally. Second, while we analyzed data from high school students in this research, open and distance learning is frequently, and in the future will be, aimed at adults (e.g., Muñoz, Redecker, Vourikari, & Punie, 2014). While there appears to be some evidence that in adults comprehension skills interact with navigation support (Naumann et al., 2007), little is known though about possible interactions between comprehension skill in navigation behavior in adult distance learners. Cleary, future research will have to shed light on this issue.

Notes

¹ PISA is an international assessment of student skills that is carried out every three years since the year 2000 in all OECD and a number of partner countries and economies. The core assessment is in mathematics, science, and reading in students aged 15, which in most countries means the end of compulsory education. See OECD (2010, 2011) for details on PISA 2009.

References

Aiken, L. S. & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.

Amadieu, F., & Tricot, A. (2008). Effects of prior domain knowledge on comprehension of non-linear documents. In J. J. Cañas (Ed.), Workshop on Cognition and the Web: Information Processing, Comprehension and Learning (pp. 153-162). Granada, Spain: University of Granada.

Azevedo, R., & Cromley, J. G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of Educational Psychology, 96, 523–535. doi: 10.1037/0022-0663.96.3.523

Azevedo, R., Greene, J. A., & Moos, D. C. (2007). The effect of a human agent’s external regulation upon college students’ hypermedia learning. Metacognition and Learning, 2, 67–87. doi: 10.1007/s11409-007-9014-9

Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56, 45–72. doi: 10.1007/s11423-007-9067-0

Bates, D., Maechler, M., & Bolker, B. (2012). lme4: Linear mixed-effects models using S4 classes. R package version 0.999999-0. Retrieved January 14, 2014 from http://CRAN.R-project.org/package=lme4

Cerdán, R., Gilabert, R., & Vidal-Abarca, E. (2011). Selecting information to answer questions. Strategic individual differences when searching texts. Learning and Individual Differences, 21, 201-205. doi: 10.1016/j.lindif.2010.11.007

Cress, U., & Knabel, O. B. (2003). Previews in hypertext: Effects on navigation and knowledge acquisition. Journal of Computer Assisted Learning, 19, 517-527. doi: 10.1046/j.0266-4909.2003.00054.x

Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline comprehension skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43, 352-392. doi: 10.1177/1086296X11421979

Dias, P., & Sousa, A. P. (1997). Understanding navigation and disorientation in hypermedia learning environments. Journal of Educational Multimedia and Hypermedia, 6, 173-185.

Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska. K. M. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. doi: 10.1002/RRQ.027

Güyer, T., Atasoy, B., & Somyürek, S. (2015). Measuring disorientation based on the Needleman-Wunsch algorithm. The International Review of Research in Open and Distributed Learning, 16(2).

Hsu, Y.-C., & Schwen, T. H. (2003). The effects of structural cues from multiple metaphors on computer users' information search performance. International Journal of Human-Computer Studies, 58, 39-55. doi: 10.1016/S1071-5819(02)00125-8

Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press.

Llorens, A.C., Gil, L., Vidal-Abarca, E., Martínez, E., Mañá, A., & Gilabert, R., (2011). Evaluación de la competencia lectora: la prueba de Competencia Lectora para Educación Secundaria (CompLEC) [Assessment of reading skills: the Reading Literacy test for Secondary Education (ComLEC)]. Psicothema, 23, 808-817.

Muñoz, C. M., Redecker, R., Vuorikari, R., & Punie, Y. (2014). Open education 2030: planning the future of adult learning in Europe. Open Learning, 28, 171-186. doi: 10.1080/02680513.2013.871199

Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R² from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4, 133-142.

Naumann, J. (2008). Log file analysis in hypertext research: An overview, a meta-analysis, and some suggestions for future research. In J. J. Cañas (Ed.). Workshop on cognition and the web: Information processing, comprehension and learning (pp 53-56). Granada, Spain: University of Granada.

Naumann, J., Richter, T., Christmann, U. & Groeben, N. (2008). Working memory capacity and reading skill moderate the effectiveness of strategy training in learning from hypertext. Learning and Individual Differences , 18, 197-213. doi: 10.1016/j.lindif.2007.08.007

Naumann, J., Richter, T., Flender, J., Christmann, U., & Groeben, N. (2007). Signaling in expository hypertexts compensates for deficits in reading skill. Journal of Educational Psychology, 99, 791-807. doi: 10.1037/0022-0663.99.4.791

OECD. (2009). PISA 2009 assessment framework. Key competencies in reading, mathematics, and science. Paris: OECD.

OECD. (2010). PISA 2009 results vol I: What students know and can do. Paris: OECD. doi: 10.1787/9789264091450-en

OECD. (2011). PISA 2009 results vol VI: Student on line. Digital technologies and performance. Paris: OECD. doi: 10.1787/9789264112995-en

Puntambekar, S. & Stylianou, A. (2005). Designing navigation support in hypertext systems based on navigation patterns. Instructional Science, 33, 451-481. doi: 10.1007/s11251-005-1276-5

Puntambekar, S. & Goldstein, J. (2007) Effect of visual representation of the conceptual structure of the domain on science learning and navigation in a hypertext environment. Journal of Educational Multimedia and Hypermedia, 16, 429-459.

R Core Team (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved January 14, 2014 from http://www.R-project.org/

Richter, T., Naumann, J., Brunner, M., & Christmann, U. (2005). Strategische Verarbeitung beim Lernen mit Text und Hypertext [Strategic processing in learning from text and hypertext]. Zeitschrift für Pädagogische Psychologie, 19, 5-22. doi: 10.1024/1010-0652.19.12.5

Richter, T., Naumann, J., & Noller, S. (2003). LOGPAT: A semi-automatic way to analyze hypertext navigation behavior. Swiss Journal of Psychology, 62, 113-120. doi: 10.1024/1421-0185.62.2.113

Rouet, J.-F., Ros, C., Goumi, A., Macedo-Rouet, M. & Dinet, J. (2011). The influence of surface and deep cues on primary and secondary school students' assessment of relevance in Web menus. Learning and Instruction, 21, 205-219. doi:10.1016/j.learninstruc.2010.02.007

Salmerón, L., Baccino, T., Cañas, J.J., Madrid, R. I., & Fajardo, I. (2009). Do graphical overviews facilitate or hinder comprehension in hypertext? Computers & Education, 53, 1308-1319. doi: 10.1016/j.compedu.2009.06.013

Salmerón, L., Cañas, J. J., Kintsch, W. J. & Fajardo, I. (2005). Reading strategies and hypertext comprehension. Discourse Processes, 40, 171-191. doi:10.1207/s15326950dp4003_1

Salmerón, L., Cerdán, R., & Naumann, J. (2015). How adolescents navigate Wikipedia to answer questions. Infancia y Aprendizaje: Journal for the Study of Education and Development , 38, 435-471.

Salmerón, L., & García, V. (2011). Reading skills and children's navigation strategies in hypertext. Computers in Human Behaviour, 27, 1143–1151. doi: 10.1016/j.chb.2010.12.008

Salmerón, L., Kintsch, W. & Cañas, J. J. (2006). Reading strategies and prior knowledge in learning from hypertext. Memory & Cognition, 34, 1157-1171. doi: 10.3758/BF03193262

Salmerón, L., Kintsch, W. & Kintsch, E. (2010). Self-regulation and link selection strategies in hypertext. Discourse Processes, 47, 175–211. doi: 10.1207/s15326950dp4003_1

Salmerón, L., Vidal-Abarca, E., Fajardo, I., Cerdán, R., García, V., & Osma, I. (2012). WebLEC: a test to measure reading literacy skills in the electronic environment. European Association for Learning and Instruction SIG-6-7 2012 meeting, Bari (Italy).

Scheipl, F., Greven, S., & Kuechenhoff, H. (2008). Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models. Computational Statistics & Data Analysis, 52, 3283-3299.

Su, Y., & Klein, J. D. (2006). Effects of navigation tools and computer confidence on performance and attitudes in a hypermedia learning environment. Journal of Educational Multimedia and Hypermedia, 15, 87-106.

Taub, M., Azevedo, R., Bouchet, F., & Khosravifar, B. (2014). Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior, 39, 356-367. doi: 10.1016/j.chb.2014.07.018

van Gog, T., Paas, F., Van Merriënboer, J. J. G., & Witte, P. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11, 237-244. doi: 10.1037/1076-898X.11.4.237

Vidal-Abarca, E., Gilabert, R., Ferrer, A., Ávila, V., Martínez, T., Mañá, A., Llorens, A.C., Gil, L., Cerdán, R., Ramos, L., & Serrano, M. Á. (2014). TuinLEC, an intelligent tutoring system to improve reading literacy skills. Infancia y Aprendizaje, 37, 25-56. doi: 10.1080/02103702.2014.881657

Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45, 267-276. doi: 10.1080/00461520.2010.517150

Winne, P. H., & Baker, R. S. (2013). The potentials of educational data mining for researching metacognition, motivation and self-regulated learning. Journal of Educational Data Mining, 5, 1-8.

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. Metacognition in educational theory and practice, 93, 27-30.

Winne, P. H., & Hadwin, A. F. (2013). nStudy: Tracing and supporting self-regulated learning in the Internet. In R. Azevedo, & V. Aleven, International handbook of metacognition and learning technologies (pp. 293-308). New York: Springer.

Does Navigation Always Predict Performance? Effects of Navigation on Digital Reading are Moderated by Comprehension Skills by Johannes Naumann, Ladislao Salmerón is licensed under a Creative Commons Attribution 4.0 International License.