Chapter 4: Experiment 1
There were two objectives of Experiment 1. The first was to examine whether or not SK directs interaction. Accordingly, participants studied information about a website (i.e., acquired knowledge about the content of the website) by studying an outline and then attempted to find information within that website. Part of the website presented information that was congruent with what they learned previously and another part of the website presented information that was incongruent with what that they learned. Because it was hypothesized that SK directs interaction, it was predicted that when the website was congruent with participant’s SK their interaction would be more efficient than when the website was incongruent with SK.
In addition, a link analysis was used to examine the paths that participants’ took through the website. This analysis was then used to evaluate specific predictions that were made based on the HuWI cycle.
The second objective of Experiment 1 was to observe whether or not SK was modified by the website. Therefore, participants completed tests that measured their SK, both before and after they used the website. The analyses of these tests were then used to determine whether or not participants’ SK was modified by the website.
Additionally, participants completed sets of identical information-finding tasks (i.e., replicates) so that changes in their interaction efficiency over the replicates could be observed. Because the HuWI cycle predicts that SK directs interaction and SK is modified by the information that is sampled from the website, it was predicted that interaction would be more efficient with each information-finding replicate on tasks, at least on the task that involved incongruent SK.
Method
Participants
Thirty Kansas State University students (13 women and 17 men) participated in order to fulfill credit for psychology courses. They were all 18 to 26 years old (M = 19.40, SD = 1.85). The only prerequisite for participating in the experiment was that all participants had at least one year of experience using the web.
Participants completed a demographic survey (see Appendix A), which revealed that they had an average of 7.63 years of experience using a computer (SD = 3.02) and that they were spending, at the time, an average of 13.20 hours per week using a computer without using the Internet (SD = 10.16). In addition, all participants had at least 2 years of experience using the Internet (M = 5.53, SD = 2.01) and were spending, at the time, an average of 19.12 hours per week using the Internet (SD = 14.91). A more detailed description of the participants’ demographics is available in Appendix B.
Design
The experimental design was a 2 (Website Congruence: congruent and incongruent with SK) X 5 (Replicate: replications of a set of identical information-finding tasks) within-subjects design. The Website Congruence variable related to two parts of the website that all participants interacted with. Specifically, part of the website was consistent with participants’ SK and another part of the website was inconsistent with their SK. Replicates were repeated trials of the same information-finding tasks.
There were three primary dependent measures. The first measure is an interaction efficiency index (IEI), operationally defined as:
IEI = Actions Used - Minimum Number of Actions Needed
where an action is a click of a link on a webpage or the web browser’s back and forward buttons. Thus, IEI scores of 0 indicated perfectly efficient interaction (i.e., no mistakes) and higher scores indicated less efficient navigation.
In addition to the IEI, participants completed a content SK test (see Appendix C) at three times throughout the experiment. This test served two purposes. First, it was used as a dependent measure for evaluating changes in SK throughout the experiment. Second, it was used as a manipulation check to measure the extent to which participants knew what they were expected to know at various points throughout the experiment (i.e., prior to studying an outline of the website’s content, after studying the outline but prior to the information-finding tasks, and after all of the information-finding tasks).
The third dependent measure was the paths that participants took as they completed the information-finding tasks. These link-node path measures were subjected to a link analysis in order to observe how SK directed interaction. A subsequent section will describe how these data were quantified and analyzed.
Apparatus, Stimuli, and Procedure
The experiment took approximately 1 hour to complete. Participants completed the experiment in groups of 1 to 4. They began the experiment by first reading and then demonstrating understanding of the instructions to subjects (see Appendix D). Next, they completed the demographic survey (see Appendix A). After completing this questionnaire, they were given the first content SK test (see Appendix C) in order to establish a baseline of any prior knowledge of the website’s contents. All of the aforementioned materials and questionnaires were presented on a computer.
After taking the content SK test for the first time, participants studied the information about the website. They spent 15 min. studying an outline of the content information presented on the website while completing a worksheet to ensure that they studied the appropriate information (see Appendix E). The outline that they studied (see Appendix F) presented information about a fictitious planet called Cyrus that NASA discovered recently (Shapiro, 1999; Elgin et al., 2001). As the outline explained, NASA had been communicating with the intelligent life on Cyrus in order to exchange information about planetary wildlife. Participants were told also that the information that they were reading was 10 years old because, due to the long distance, it takes approximately 10 years in order to fully transmit an exchange of information with Cyrus’ inhabitants. The outline presented information about Cyrus’ wildlife within three major population categories: extinct, endangered, and populous. Within each of these population categories, there were three subcategory classifications of wildlife types: herding animals, birds, and meat eaters. Within each of these subcategory classifications, there was a description of two animals. After studying the planet Cyrus outline, participants took the content SK test again in order to measure their knowledge acquisition for the study period.
Next, participants interacted with the Cyrus website (see Appendix G), in order to complete the information-finding tasks. As explained on the homepage of the Cyrus website, the information contained in the website was from a new transmission with the intelligent life of Cyrus. What is not explained on the homepage was that the animals were all the same, but some had been reclassified into the extinct, endangered, and populous categories due to events such as animal hunting, discovery, and cloning. Therefore, participants’ content SK acquired during the study session could be either congruent with the website (i.e., the population status of the animals had not changed in 10 years) or incongruent with the website (the animal’s population status had changed).
In addition, two versions of the website were used in order to counterbalance the consistent and inconsistent parts of the website between subjects. This was done in order to ensure that finding particular animals, which could have varied in appeal or interest to participants, was not a confounding variable in the experiment. Figure 3 shows an example webpage and Figure 4 shows diagrams of both versions of the website, as they were structured in relation to participants’ SK.
For each information-finding task, participants located a specific webpage within the website. The minimum number of actions required was always 3. Therefore, IEI was always the number of actions taken minus 3. There was a total of 6 tasks (3 congruent and 3 incongruent) that participants completed in random order. The mean IEI score for the 3 similar tasks (i.e., the 3 congruent and 3 incongruent) was used as the single IEI score for congruent and incongruent tasks. Each group of 6 tasks was completed 5 times in order to have 5 replicates. Therefore, each participant completed a total of 30 information-finding trials during the experiment.
Figure 3. Screenshot of a webpage from the planet Cyrus website and the browser that was used in the study. The current task is displayed in the lower left status bar. In this figure, a dialog box would soon popup in order to notify the participant that the current task has been completed and to inform her of the next task. The only functional buttons on the browser are the back forward, stop, and refresh buttons. All of the menu items were disabled.
Figure 4. Diagrams of the planet Cyrus website structure. Boxes represent information about a category or animal on a webpage. Solid lines show congruent information. Dashed lines point to information in the website but not SK (i.e., hyperlink only). Dotted lines point to information in SK but not the website (i.e., SK only). For example, in the top panel, the helmet horn was extinct 10 years ago, but now is populous. Gray boxes are the ending webpages for congruent tasks and black boxes are the ending webpages for incongruent tasks.
The web browser was a simplified version of Microsoft Internet Explorer (see Figure 3) that was programmed in Microsoft Visual Basic. It collected the primary usage data for computing IEI scores and additional data such as what pages were visited, when each page was visited, how much time was spent on each page, and how much time was spent on each task. Also, the browser randomized the tasks, informed participants of their current tasks, and notified them when they had completed a task (i.e., found the appropriate page that contained the desired information). After completing all 30 trials, participants completed the content SK test again in order to test for SK modification.
Results
The following sections describe the methods and procedures used for data screening and the three primary analyses for the current experiment. The first analysis examined Website Congruence and Replicate effects using an ANOVA on IEI scores. The second analysis compared SK pretest and posttest scores using a Wilcoxon signed ranks test. Finally, a task analysis technique called link analysis provided a more detailed descriptive account of the participants’ interactions with the website.
Pre-Analysis Data Screening
Preliminary data screening identified outliers, which were defined as extreme values that were beyond the outer fence for each condition in a boxplot. Specifically, the outer fence was defined as:
Upper Bound = 3(IQR) + Q1
and
Lower Bound = 3(IQR) - Q3
where IQR (interquartile range) was the difference between the value for the 25th percentile (Q1) and the 75th percentile (Q3) . Scores greater than the upper bound of the outer fence and less than the lower bound of the outer fence were considered outliers. This procedure revealed that there were 12 outliers in the IEI scores.
However, 10 of the outliers came from the same two participants. In addition, the experimenter had noted during the experiment that one of these participants had fallen asleep during the outline study task and that the other had appeared extremely uninterested. Accordingly, both participants were eliminated from the dataset completely. Therefore, only 30 of the original 32 participants’ data were used in the analysis.
The final two outliers were then investigated. Because IEI scores are the mean of three separate scores, the individual trials that made up each IEI score were examined. This process revealed that there were individual trials with exceptionally large values that influenced the remaining two outlier IEI scores (i.e., individual trial scores of 6, 56 and 0 for one IEI score and 4, 8, and 26 for the other, respectively). The cause of these extreme values could not be determined. In both of these cases, the extreme value was eliminated and the participants’ IEI score was recalculated as the mean of the two remaining values.
After dealing with the aforementioned outliers, it was necessary to determine whether or not participants had learned the material adequately by studying the outline prior to using the website. A dependent sample t-test of test scores collected prior to studying the outline and test scores collected after studying the outline revealed that participants performed nearly at chance level for the pretest (chance value = 33.33%; M = 36.60%, SD = .15), and significantly better on the posttest (M = 82.20%, SD = .20), t (29) = 10.82, p < .001, h2 = .80[1]. This result demonstrates that participants did study and learn the outline, which was critical for the success of the Website Congruence manipulation.
Finally, several exploratory regression analyses were performed in order to determine whether or not any of the self-report measures from the computer experience and demographics survey predicted any of the dependent variables. Results of these analyses indicated that none of the measures predicted any of the dependent variables reliably (p > .05). Therefore, none of these self-report measures were used in subsequent analyses as control variables.
Family-Wise Error Rate
The analysis of IEI scores and SK test scores were conducted separately. For each of these analyses, the need arose to perform subsequent follow-up analyses, which raised the possibility of an inflated Type 1 error rate. In order to deal with this possibility, sets of sub-analyses were deemed families and alpha was controlled within each family. Families were defined as groups of sub-analyses that were used to investigate a particular effect (e.g., and interaction) in a particular way (e.g., one set on columns and another on rows to investigate an interaction).
For example, included in the following sections is the description of an ANOVA that was performed on IEI scores. This ANOVA required several sub-analyses in order to investigate an interaction. A Bonferroni correction was applied to each sub-analysis (e.g., a series of pairwise comparisons, a series of one-way ANOVAs, etc.) because all of these sub-analyses, although investigating the same effect, investigated the interaction in different ways. The following sections describe these families of analyses with an alpha (a) level set at .05. Bonferroni adjustments are shown as the p value divided by the number of tests (e.g., .05/2 for 2 tests at a = .05).
Analysis of IEI Scores
Assumptions
An analysis of IEI scores was performed in order to examine changes in web browsing performance attributed to 1) the congruence of the website with SK and 2) replicating the same task. However, before performing this analysis, the data were examined for violations of the assumptions of a repeated measured analysis of variance. Overall, the data appeared to violate the normality assumption and, to a greater extent, the sphericity assumption (i.e., violation of equality of variances of differences between all treatment pairs). Both of these violations appeared to be the result of a ceiling effect, especially at replicates 4 and 5 (see Appendix B for the normality statistics and the source tables for all statistical tests reported in this section).
Sphericity, on the other hand, can be much more problematic than normality if violated (Mertler & Vannatta, 2001). However, all current tests for sphericity violations are highly sensitive to departures from sphericity and from their respective null hypotheses (Davidson, 1972; Keselman, Rogan, Mendoza, & Breen, 1980; Rogan, Keselman, & Mendoza, 1979). Accordingly, it has been recommended that a correction to the degrees of freedom should be used at all times, rather than attempting to correct the data based on the results of these tests (Stevens, 1992). Of the possible remediations, the Huynh-Feldt adjustment of the degrees of freedom seems to be the best treatment for repeated measures designs (Maxwell & Delaney, 1990; Winer et al., 1991). Therefore, a Huynh-Feldt adjustment to the degrees of freedom of the F distribution was applied to all of the repeated measures ANOVAs reported here in order to compensate for any possible violations of the normality assumption.
Data Analysis
As can be seen in Figure 5, performance in the incongruent condition approached that of the congruent condition as replicates increased. This observation was corroborated with a 2 (Website Congruence) X 5 (Replicate) ANOVA indicating a significant Website Congruence X Replicate interaction, F (2.65, 76.87) = 15.21, p < .05, h2 = .34. In addition, there was a significant main effect for Website Congruence, F (1, 76.87) = 42.63, p < .05, h2 = .60, and Replicate, F (4, 76.87) = 72.78, p < .05, h2 = .72.
To explore the significant interaction, two sets of sub-analyses were performed. The first set was performed on each level of Replicates across Website Congruence in order to determine which Replicate conditions yielded a Website Congruence Effect. Conversely, the second set of sub-analyses was performed on each level of Website Congruence across Replicates in order to determine which Website Congruence conditions yielded a Replicate effect. When necessary, the second set of analyses was explored further with multiple comparisons of adjacent replicates (e.g., incongruent at replicate 1 vs. incongruent at replicate 2, but not incongruent at replicate 1 vs. incongruent at replicate 3) in order to determine the replicate at which performance stopped improving significantly within a particular condition. As discussed above, these sets of analyses investigated the interaction in different ways and were defined as different families of analyses. Therefore, each sub-analysis utilized a Bonferroni correction to control for family-wise error.
Figure 5. Mean IEI scores with 95% confidence intervals.
The first sub-analysis was a set of 5 dependent sample t-tests on each level of Replicates across Website Congruence. This group of analyses required a Bonferroni correction for 5 comparisons (i.e., a = .05/5 or .01). The results of these t-tests indicated significant Website Congruence effects for replicate 1, t (29) = 6.68, p < .05/5, h2 = .61, replicate 2, t (29) = 3.29, p < .05/5, h2 = .27, replicate 3, t (29) = 3.83, p < .05/5, h2 = .34, replicate 4, t (29) = 2.92, p < .05/5, h2 = .23, and replicate 5, t (29) = 3.10, p < .05/5, h2 = .25. Therefore, performance in the congruent condition was always better than performance in the incongruent condition.
The second sub-analysis was a set of 2 repeated measures ANOVAs on each level of Website Congruence across Replicates. This group of analyses required a Bonferroni correction for 2 comparisons (i.e., a = .05/2 or .025). The results of these ANOVAs indicated significant Replicate effects for the congruent condition, F (2.46, 71.29) = 13.58, p < .05/2, h2 = .32, and the incongruent condition, F (2.66, 77.03) = 66.41, p < .05/2. These results, in conjunction with examination of Figure 5, show that performance improved in both conditions as replicates increased.
In order to determine the replicate at which performance within each Website Congruence condition stopped improving, an additional set of sub-analyses of adjacent replicates for each Website Congruence condition was performed (i.e., replicates 1 vs. 2, 2 vs. 3, 3 vs. 4, and 4 vs. 5 for the incongruent and congruent condition). Therefore, this set of dependent sample t-tests required a Bonferroni correction for 8 comparisons (i.e., a = .05/8 or .006). The results of these analyses revealed that the incongruent condition yielded improved scores from replicate 1 to 2, t (29) = 5.83, p < .05/8, h2 = .54, and from replicate 2 to 3, t (29) = 4.04, p < .05/8, h2 = .36. However, none of the other comparisons were significant[2]. Therefore, the improvement over adjacent replicates was limited to the first two replicates in the incongruent condition. This corroborates what can be observed in Figure 5: performance improvement in the congruent condition was more subtle than that in the incongruent condition, which explains the obtained interaction.
Analysis of SK Test Scores
Assumptions
Before analyzing the SK test scores (i.e., pretest vs. posttest) in order to examine the changes in SK attributed to the web browsing task, the data were examined for violations of the assumptions of a dependent sample t-test. As was done for the IEI analysis, the distributions of the pretest and posttest scores were examined for normality by examining the 95% confidence intervals for the skewness and kurtosisness statistics. With this technique, it was revealed that the distribution of posttest scores violated normality, while the distribution of pretest scores did not (see Appendix B for the normality statistics and the source tables for all statistical tests reported in this section). Examination of the data suggests that the posttest normality problem was due to a ceiling effect caused by the high number of perfect scores on the test (i.e., 21 of the 30 scores were 100%)[3]. This was problematic because the dependent sample t-test is not robust with regard to violations of normality (Winer et al., 1991).
Because normality could not be corrected with several attempted data transformations, the nonparametric Wilcoxon signed ranks (WSR) test was employed. The WSR test does not assume normality. However, it does assume that 1) the paired values of scores are drawn independently of all other pairs, and that 2) the dependent variables have the properties of at least an ordinal scale of measurement (Myers & Well, 1995). The data collected for the current experiment meet these two assumptions.
Data Analysis
Overall, it appeared that the posttest scores (M = 91.07%, SD = 16.83%) were much higher than the pretest scores (M = 47%, SD = 12.53%). In addition, of the 30 pretest to posttest difference scores, there was only one negatively ranked score (i.e., lower score on the posttest than on the pretest). These observations were corroborated with a WSR test that demonstrated that there was a significant improvement in test scores from pretest to posttest, z = 4.85, p < .05.
Link Analysis
In order to acquire a more detailed outlook of the data, a link analysis was performed. Link analysis is a task analysis technique that is used in human factors studies for identifying the best configuration of a system by depicting relationships between system components (e.g., webpages) graphically in order to illustrate inefficient placement of the system components (Kirwan & Ainsworth, 1992). This analysis is usually descriptive and is used for summarizing observational data by displaying graphically the observed usage patterns between components based on, for example, frequency of component use.
Data Coding
Link analysis can be conducted on specific subtasks. For example, for the information-finding task used in Experiment 1, it was important to divide the task into two distinct subtasks: 1) selection of a link from the homepage, and 2) selection of a link from an animal population page[4]. This division of subtasks was necessary because the SK used in each subtask was expected to yield markedly different patterns of navigation. For Subtask 1 (selecting a link from the homepage) SK was not always congruent with the website. For Subtask 2 (selecting a link from an animal population page), SK was always congruent (e.g., birds are always birds), even though following the path using this SK information would not always lead to the right answer.
This is best illustrated with an example. Assume that a participant’s SK after studying the outline tells him that the Helmet Horn is an extinct herding animal (see Figure 6). However, since the creation of the materials that were studied initially, the Helmet Horn has been reclassified within the updated website as endangered, since a few specimens were located on a remote island (i.e., the website is incongruent with SK). In Subtask 1, the participant who was unaware of this change in the animal’s classification could follow a link presented on the homepage that was congruent with their SK (i.e., the “extinct” link), or one that was incongruent with their SK (i.e., the “populous” link, or where the animal is now, the “endangered” link). If they selected the extinct link, then they were correct, given the SK that was based on information that they acquired earlier, but not correct, given the current state of affairs. These types of actions were classified as “Correct SK.” If they selected the endangered link, they were correct, given the current state of affairs. These types of actions were classified as “Correct Webpage.” Finally, if they selected the populous link, then that action was classified as “Incorrect,” given both the SK and the current state of affairs.
Figure 6. Example link analysis showing the coding scheme for finding the helmet horn.
Subtask 2 (i.e., selecting a link from an animal population page) was quite different. Since the animal type (herding animal, bird, or meat eater) never changed, regardless of the animal’s population status, the participant could either select the link that was congruent with SK or an incorrect page. In the above example, an action would be classified as “Correct SK” if they selected herding animal from either the extinct or populous pages. If they selected the correct type of animal from the endangered page, then the action was classified as “Correct Webpage and SK,” denoting that they made the correct selection of animal type from within the “Correct Webpage” classification from Subtask 1. All other actions were classified as “Incorrect,” signifying the selection of a webpage that does not match SK or the website. Figure 6 illustrates this example.
In addition to dividing each information-finding task into two subtasks, each condition in the experiment was analyzed separately. Accordingly, a link analysis was performed for each Website Congruence X Replicate condition.
Because there were 3 tasks within each Website Congruence X Replicate condition (i.e., the 3 tasks that were averaged for each condition in order to compute the analysis of variance) and there were two counterbalanced versions of the website, these tasks were analyzed separately and later combined. Otherwise, any usage pattern differences would be washed out in the link analysis because participants’ goals on the tasks were to find different pieces of information. Accordingly, each node in the link analysis was categorized relative to where the SK given to participants (i.e., the information studied in the outline) indicated that the information was and to where the information actually was within the website (see Figures 7 and 8). This is denoted by the nodes labeled “Correct SK” for paths that lead to where the information was according to participants’ given SK, “Correct Webpage” for paths that lead to where the information was within the website and “Incorrect” for paths that lead to neither, as defined above.
Data Analysis
The results of the link analysis for the congruent and incongruent conditions are shown in Figures 7 and 8, respectively. The numbers beside the lines connecting the nodes represent the proportion of total actions from the left node to the connected right node. Each line contains five proportions, for replicates 1 through 5, respectively.
Figure 7. Results of the link analysis for replicates 1 through 5 in the congruent condition.
Figure 8. Results of the link analysis for replicates 1 through 5 in the incongruent condition.
As can be seen in Figures 7 and 8, it appears that the use of paths to the nodes that are labeled “Correct Webpage” increased with replicates, while the use of paths to “Incorrect” and “Correct SK” nodes decreased. In addition, it should be noted that, in general, the paths to Correct SK nodes were taken more frequently than the paths to Incorrect nodes.
Discussion
There were two primary questions addressed by the current experiment: 1) does SK direct interaction with the website and 2) does the available information in the website modify SK. The following sections describe the results of the current experiment with respect to each of these questions. Finally, the implications of these results for the HuWI cycle and website design are discussed.
Does System Knowledge Direct Interaction with the Website?
The HuWI cycle predicts that SK directs interaction. Accordingly, a user’s ability to find information efficiently (i.e., by clicking the fewest number of links in order to find a piece of information) on a website that is congruent with SK should be superior to performance when the website is incongruent with SK. There were two lines of evidence to investigate these anticipated performance differences. The first involved the analysis of IEI scores and the second was the link analysis.
IEI Analysis
For the analysis of IEI scores, the current experiment tested for differences in the efficiency of finding information when a website contained information that was either congruent or incongruent with participants’ SK. The prediction derived from the HuWI cycle was that performance in the congruent condition would be superior to that in the incongruent condition on the first replicate of the task. This prediction is limited to the first replicate where the website had little influence on modifying the SK being used.
The results of the analysis of IEI scores suggest that SK does direct interaction with the website. Performance within the first replicate in the congruent condition was more efficient than it was in the incongruent condition. In addition, performance in the congruent condition remained superior to the incongruent condition across all five replicates. This suggests that, even though performance appeared to improve with extended interaction with the website, the use of the initial SK (i.e., the unmodified SK that one has when first viewing a particular website) to direct interaction had a lasting influence that could not be overcome even after several replications of the same tasks. This finding has implications for the HuWI cycle and website design, which will be discussed later.
Link Analysis
A second line of evidence that SK directs interaction with the website comes from the link analysis (see Figures 7 and 8). First, if one compares the paths that point to Correct SK nodes (i.e., paths that would be correct if the website matched SK) against paths that point to Incorrect nodes (paths that were not correct with the website or SK) within each subtask, then it appears that participants tended to go to webpages that would be correct if the website was congruent with SK. In fact, a simple rule emerges from this examination: paths that were consistent with SK were always taken more frequently than paths that were not[5]. In addition, this pattern held for every condition in the study (i.e., the congruent and incongruent conditions across all replicates). This suggests that there was a tendency to select webpages that match the initial SK and that this tendency was rather persistent.
In addition, if one compares the paths that lead to the Correct Webpage nodes (i.e., paths that were correct within the website, regardless of SK) for the congruent condition against the paths to the Correct Webpage nodes for the incongruent condition, then it appears that finding information in the incongruent condition was more problematic than it was for the congruent condition. As with the previous comparison, this pattern holds for every replicate in the study. This implies that even though the website might modify SK, as indicated by the increasing proportions of paths to the Correct Webpage nodes across replicates, interaction with the website was still influenced by the initial SK.
Summary of System Knowledge Directing Interaction
Overall, the results of the analysis of IEI scores and the link analysis revealed two findings. First, because participants had a bias toward going to webpages that matched their SK on the first replicate, it appears that SK did direct interaction with the website. Therefore, because this bias was toward webpages that match SK and not to other webpages, the prediction derived from the HuWI cycle that SK directs interaction with the website appears to be supported. In addition, this supports Neisser’s (1976) contention that schemata (e.g., SK) direct exploration (e.g., interaction).
The second finding was that the bias to go to webpages that match an initial SK was somewhat persistent. In other words, there was always a tendency to go to webpages that matched the information that was learned originally. Even though this tendency was reduced with each replicate, it still persisted across all replicates. This suggests that SK is modified, as opposed to replaced, by the website. The following section further explores this possibility.
Does Available Information in the Website Modify System Knowledge?
The HuWI cycle predicts that the information available in a website modifies SK. This section describes three lines of evidence for this prediction that are available from the current experiment: 1) the analysis of IEI scores, with a specific focus on the replicate effect, 2) the analysis of pretest and posttest scores on the SK test, and 3) information from the link analysis.
IEI Analysis
Based on the HuWI cycle, IEI scores should improve across replicates, at least in the incongruent condition. That is, if SK was modified by the information in the website, and the initial SK was incongruent with the website, then repeating the same task should yield improvements in IEI scores. The results of the IEI analysis supported this prediction. Although both the incongruent and congruent conditions improved over replicates, the significant interaction revealed that the improvement in IEI scores was a function of website congruence and replicates. Specifically, the mean differences between congruent and incongruent trials decreased as replicates increased. This Website Congruence X Replicate interaction suggests that participants modified SK as they used the website and that SK directed their interactions with the website. Thus, the interaction effect was only possible if exploring the website modified participants’ SK.
As mentioned in previous sections, there was a persistent tendency for participants to go to webpages that matched the initial SK. This finding, when combined with the finding that performance improved over replicates, suggests that the website modified SK, rather than replaced it. If SK would have been replaced, then the effect that the initial SK (i.e., the congruence manipulation) had on subsequent replicates would not have been so persistent. Therefore, in accord with Neisser’s (1976) perceptual cycle, the process appears to be one that is similar to Piaget’s (1952) concept of accommodation, where cyclical interactions modify the initial schema.
SK Test Analysis
In addition to the analysis of IEI scores, an analysis of the content SK tests provided a second line of evidence that SK was modified by the information in the website. The SK test was a more direct, albeit less detailed, measure of SK modification than the aforementioned IEI analysis. These data indicated that there was an improvement in SK as a result of using the website. Thus, it appears that participants’ SK was modified when interacting with the website.
Link Analysis
The third and final line of evidence that SK was modified by the information in the website comes from a pattern found in the link analysis results (see Figures 7 and 8). This pattern is apparent in the changes in proportions of actions taken as replicates increased (i.e., the number string on each line in the figures). Specifically, the proportions of paths that lead to the Correct Webpage nodes always increased across replicates. In addition, with the exception of a few cases around .03 and below, the proportions of paths leading to an incorrect answer (i.e., Incorrect and Correct SK nodes) decreased. Therefore, it appears that participants’ usage patterns changed in a relatively consistent and predictable way, which could be attributed to SK modification from interacting with the website.
Finally, it should be noted that examination of the link analysis reveals that the initial SK appeared to be modified, rather than replaced. For example if one examines the last replicate in the incongruent condition with subtask one (the left column of Figure 8), it can be seen that participants were still selecting the webpage that was consistent with the initial SK (i.e., the path to nodes labeled Correct SK) more than the page that was not consistent with SK or the website (the path to nodes labeled Incorrect). It appears that participants used their SK to interact with the website and, although SK changed, it also appears that they never fully modified their SK to match what it could have been if given a consistent initial SK. This finding, which was a common theme for the entire set of analyses for this experiment, has important implications for the HuWI cycle and website design, which will be discussed in the following sections.
Implications
The HuWI Cycle
The results demonstrated that SK directed participants’ interaction with a website. This finding might explain the results of other studies that show that if users have an inadequate understanding on a website’s content, then their interaction with the website will be inferior to that of users with a better understanding of a website’s content (Carmel, et al., 1992; Korthauer & Koubek, 1994; McDonald & Stevenson, 1998; Symons & Pressley, 1993). That is, if a user has inadequate SK (e.g., a novice within the website’s domain area), they will be looking for information in the wrong areas of the website or they will be looking for areas of the website that do not exist.
The results demonstrated also that SK was modified by the information in the website. In other words, participants learned the contents of the website. This result is consistent with the findings of many other studies that have analyzed learning on the web and found that users acquire information about a website’s contents (e.g., Elgin, et al. 2001; Leventhal, et al., 1993; Rouet, 2003; Shapiro, 1999). However, akin to these studies, the reported experiment only examined content SK, whereas the HuWI cycle proposes the existence of content SK and structure SK (i.e., knowledge of how webpages are interlinked). Accordingly, Experiment 2 will address this issue.
Beyond substantiating two tenants of the HuWI cycle, the results indicate the presence of a persistent influence of the initial SK[6] on participants’ interaction. Because the effect of the initial SK was apparent in most of the analyses, it is suggested that the initial SK is an important component for providing a structure for SK that can later be modified, but not replaced . One could speculate that SK could change eventually (i.e., the effect of initial SK on interaction no longer exists), but, according to the HuWI cycle, this would be caused by a modification in SK, not a replacement of SK. However, a more important implication for the HuWI cycle is explaining this SK effect.
Recall that SK is a schema that is used to interact with a particular website. Accordingly, the literature on schema development might explain the persistency of the congruency effect across replicates in the reported experiment.
Although there are numerous ways to conceptualize schemas, for the current discussion they are conceptualized as networks of related knowledge elements. In addition, schemas can be embedded within a larger schema (Marshall, 1995). In order to clarify the following discussion, schemas that are embedded within a larger schema will be referred to as subsets of a schema.
In addition, it is assumed for the present purposes that multiple elements in the schema can be associated with individual congruent or incongruent aspects of the website. For example, the incongruent nature of the helmet horn being an extinct animal may be associated with multiple knowledge elements that are then associated with various other aspects of the schema, e.g., the helmet horn is profitable to hunt to extinction because its horns are valuable and there are prey to several known predators. In other words, even though the helmet horn is no longer extinct in the schema, there are still knowledge elements within that subset about the helmet horn that suggests that it would be extinct.
When a schema is called upon for a particular situation, one or more subsets of the schema are activated. In other words, schemas are called upon for groups of related activities (e.g., the Cyrus schema for exploring they Cyrus website), and particular subsets of that schema are activated in order to direct behavior in particular situations (e.g., the extinct animal subset and the preyed upon animal subset in order to find the helmet horn). The question, therefore, is how a schema and its subsets of knowledge elements deal with a situation that is incongruent with it.
When the initial schema does not fit the situation adequately, a new one must be found or the current one must adapt (Marshall, 1995). Piaget (1952) referred to this as accommodation. According to Marshall (1995), the two kinds of modification that must be considered are 1) enlarging the schema with new knowledge elements and 2) changing the knowledge elements that are already a part of it. Note that the latter is not assimilation, which would be changing the information to fit into a schema, but rather it is a form of accommodation, which changes the schema as a result of new information.
If information adds knowledge to a schema constantly, then the resulting schema, after modification, is a memory structure that will contain a great deal more knowledge than can be used in one particular situation. As a result, only a subset of knowledge elements within a schema is needed in any particular instance. In addition, most situations “…will involve unique sets of [knowledge] elements, because each instance, with its own set of unique features, will activate slightly different elements” (Marshall, 1995, p 45). For the reported experiment, it is possible that incongruent knowledge elements remained in the schema and were accessed occasionally. Therefore, at each replicate there could be instances where a participant accessed a subset of the schema that contained knowledge elements that were a part of the initial SK and other instances where they accessed elements that were later added to SK.
The reduced frequencies of errors in the incongruent condition across replicates might be attributed to some sort of change in SK that resulted from strengthened activation levels for correct knowledge elements. Alternatively, the reduction of error may be due to an inhibition of incorrect knowledge elements as a result of adding several correct knowledge elements into SK. Regardless of whether the specific process increased activation levels of correct knowledge, inhibition of incorrect knowledge, or both[7], it appears that SK was enlarged and changed in the reported experiment.
In order to illustrate this point, Figure 9 depicts these enlarging and changing processes within the HuWI cycle when incongruent knowledge exists. Each face in Figure 9 represents different elements of knowledge within SK. Depending upon the specific knowledge, the new information, which is always congruent with the website, arrives (left panel of insert) and either enlarges SK when the information is congruent with SK (upper middle panel) or simultaneously enlarges and changes SK when the information is incongruent with SK (lower middle panel). Because congruent information only enlarges SK, the end result of acquiring this new knowledge (upper right panel) is more congruent elements and retention of the same incongruent elements. However, because information that is incongruent with SK enlarges and modifies SK, the end result of acquiring this new knowledge (lower right panel) is more congruent elements and retention of fewer incongruent elements. The rest of this section describes the reported experiment’s results within this context.
Figure 9. Enlarging SK on congruent trials (upper panels) and simultaneous enlarging and changing processes on incongruent trials (lower panels) within the HuWI cycle when incongruent knowledge exists.
Examination of the IEI scores in Figure 5 reveals that performance in the congruent condition improved across replicates. Because all of the available information in the website was congruent with the initial SK, it is assumed that participants did not learn the information perfectly. In other words, there was always room for knowledge to be added to SK when it was not acquired through previous interactions. This is the SK enlargement process shown in the upper panels of Figure 9. Therefore, the replicate effect for the congruent condition shown in Figure 5 represents SK enlargement.
The stronger replicate effect for the incongruent condition in Figure 5 is attributed to enlarging and changing SK simultaneously, as in the lower panels of Figure 9. That is, two different SK modification processes were working to increase the number of congruent elements and to reduce the number of incongruent elements. Accordingly, the learning curves for congruent trials were more pronounced.
Also, the SK modification process can be used to explain the persistent influence of the initial SK on performance. Recall that we have assumed imperfect learning. Therefore, replicates will not change SK perfectly, thus leaving remnants of the old incongruent knowledge, as shown in far right panels in Figure 9. If these remnants are accessed for directing interaction with the website, then the user will look for information in the wrong place. Until these incongruent knowledge remnants are completely overrun by higher activation levels for congruent elements or sufficiently inhibited by other elements, they will continue to influence interaction with the website, therefore exhibiting a persistency effect.
In summary, it appears that adding new knowledge to the initial SK (enlarging SK) leaves incongruent knowledge elements within SK that can still accessed. In addition, it appears that the frequency of the influence of incongruent elements is reduced (changing SK), but remnants remain in SK and can be accessed, which creates the persistent influence of the initial SK on interaction. There are design implications for this persistent effect, which will be discussed in the following section.
Web Design
Many researchers have expressed the need to build accurate user models in order to design effective website architectures (e.g., McDonald & Stevenson, 1998; Otter & Johnson, 2000). This study is no different in that respect. In order to design effective website architectures, we must have a good understanding of how users think about the displayed information (i.e., their SK). However, one interpretation of the results and the preceding discussion of the HuWI cycle might give cause for concern over this policy.
One important aspect of user modeling is the process of using cognitive task analysis techniques to understand how users think about information. For example, this can be done by having a sample of users sort cards with names of concepts to figure out how users categorize information (Seamster, Redding, & Kaempf, 1997). The results of this process are then used to make recommendations about the design of websites (Mayhew, 1999). The current experiment does not dispute the validity of this process. Indeed, the experiment and the HuWI cycle largely support it. That is, according to the HuWI cycle, collecting and applying data from user models probably works well for designing new websites because SK directs interaction. If the website is congruent with SK, then users should have little problem locating the information that they desire.
However, one interpretation of the current experiment and the HuWI cycle is that the user model for a domain area might not always be the best model to use when making design recommendations for a website in that domain area. Specifically, recall that the information acquired from the website modifies SK, which is a website schema. The potential problem, therefore, is that one could make design recommendations based on how users think about a domain area (i.e., an initial SK that has not been modified by interacting with websites), and not how users think about websites in that domain area. In other words, it is possible that a user would visit a website about a particular domain area for the first time and use their knowledge of that domain to direct their interaction. If the website is congruent with the domain area knowledge, then the user will engage in an enlarging SK process. However, if the website is incongruent with the domain area knowledge, then the user will engage in simultaneous enlarging and changing processes. Consequently, when visiting other websites later on, the changed knowledge elements may direct interaction incorrectly if they are activated. Therefore, the best user model may not be a user model for a domain area, but rather for websites in the particular domain area. The following example illustrates this point.
Imagine that several websites for selling furniture already exist when a consultant is commissioned to make recommendations for a new online furniture store. The consultant correctly and appropriately applies several task analysis techniques and acquires a good target customer user model of furniture. However, mistakenly he does not audit similar websites in order to look for other organization themes that might exist, and makes recommendations to the client based solely on the user model that he acquired. For discussion purposes, we will assume that the consultant’s recommended user model is one based on function (i.e., categories of things you sit on, things you rest other objects on, etc.) and that the company designs the website around this classification scheme. However, users of the online furniture store are likely to have done some shopping around on similar websites. In addition, all of these other websites have used a categorization scheme based on traditional furniture store groupings (e.g., dining room, bedroom, etc.), where “things you can set on” are located in nearly every room of the house. Assuming that the user has visited a lot of these websites, then the new website could be more difficult to use than the other websites. This is because the furniture website SK that users have developed through interaction with other websites is actually full of knowledge elements that are incongruent with the furniture user model that the consultant measured. Therefore, although the user model was correct for the product (i.e., domain area), it was not a model that would be appropriate for the product’s website.
In light of this finding, it is recommended that designers acquire user models and audit the website’s competition in order to look for inconsistencies. If large inconsistencies exist, one could either conform to the standard, thus making a website consistent with other websites in order to appease those that have visited already competing websites, or create the website around the user model in order to appease the users that happen upon the new website first. Perhaps the best procedure would be to attempt to incorporate both models into the design. For example, stating explicitly that users can sort through furniture based on functionality or traditional room vignettes might be a compromise.
[1] The assumptions of the dependent sample t-test were tested before conducting this analysis. Distributions of the pretest and posttest scores were examined for normality by examining the 95% confidence intervals around the skewness and kurtosisness statistics. With these criteria, the distributions of both sets of scores were normal (see Appendix B for the normality statistics and the source tables for this analysis).
[2] Differences for the incongruent condition in replicate 3 vs. 4, t (29) = 2.68, p > .05/8, and replicate 4 vs. 5, t (29) = 2.23, p > .05/8, were not significant. In addition, differences for the congruent condition in replicate 1 vs. 2, t (29) = 2.30, p > .05/8, replicate 2 vs. 3, t (29) = 2.16, p > .05/8, replicate 3 vs. 4, t (29) = 1.03, p > .05/8, and replicate 4 vs. 5, t (29) = 1.71, p > .05/8, were not significant.
[3] These high numbers of perfect scores made any non-perfect score an outlier by using the outer fence criteria explained above. However, these scores were left in the dataset because removing them removed all error variability, which was necessary for the statistical test.
[4] The third subtask (Subtask 3), selecting a link from an animal type page, was not analyzed, as it is uninteresting for the purposes of the proposed study that participants would select a link to a webpage for an animal that they are not supposed to be looking for.
[5] If it is assumed that participants learned where information was located with each replicate, then the Correct Webpage nodes might also be considered Correct SK nodes at replicates 2 - 5. Therefore, this statement and comparison excludes paths that lead to the dubious Correct Webpage nodes.
[6] It should be noted that an initial SK is not the same as a global schema, as defined by Dillon et al. (1993). A global schema is a general “web schema” that includes knowledge like links are underlined, links are located at the top and side of the page, content is located in the middle of the page, etc. Initial SK, on the other hand is the knowledge that one arrives at a particular website with. Accordingly, the modified SK is not the same as Dillon et al.’s (1993) instantiated schema because an instantiated schema is the modified global schema.
[7] The HuWI cycle does not specify a particular theory of memory or learning, so long as it is compatible with schema theory (i.e., behaviorist theories will not work). However, this discussion tends to recruit terms and concepts from contemporary theories of learning that are specified by associative network models (e.g., Collins & Quillian, 1969) and connectionist models (e.g., McClelland & Rumelhart, 1981).
Document Last Updated December 31 1969 19:00:00.
Top
If you are in need of consulting services, we recommend Atlantic Human Factors
for human factors consulting and
usability consulting.
Or Search for Human Factors and Usability Consultants
Copyright © 2003 - 2007 by HuWI.org