Chapter 5: Experiment 2

Table 1 Means, 95% Confidence Intervals for the Means, and Expected Chance Values for Experiment 2

In order to calculate the expected chance values, recall that the SK tests required one response (i.e., possible favorable outcome) of two possible answers (pertinent possibilities). Therefore, the expected chance values of the Structure SK, Content SK test, and the C&S SK tests were .50.

The matrix tests, on the other hand, had 120 cells to choose from. However, each test occupied a certain number of cells. If one assumes that the responses on all of the other tests were correct, then the number of pertinent possibilities for each test must be reduced by the number of possibilities for a favorable outcome on the other tests[3]. Therefore, the Structure Matrix and the Content Matrix tests had 8 possibilities for a favorable outcome, and 96 pertinent possibilities. The 96 was derived from the number of cells in the matrix (120) minus the number of cells in the matrix that were assumed to have been occupied by responses from the other tests. Since the Structure Matrix and Content Matrix both required 8 cells to be filled and the C&S Matrix required 16 cells to be filled, the denominators are 120-8-16, or 96, thus making their expected chance values .0833. Similarly, the C&S Matrix had 16 possibilities for the numerator and 120-8-8, or 104, for the denominator, making the expected chance value .1538.

As can be seen in Table 1, the expected chance value fell within the 95% confidence intervals for the means for both groups on the Structure Matrix test. This suggests that the Structure Matrix test was not a good measure and/or it was too difficult for participants to complete. Therefore, it was eliminated from any further analyses. All of the other measures had at least one group with a confidence interval that did not include the expected chance value, therefore warranting their inclusion in the analyses.

Tests of Assumptions

All of the following analyses utilized independent sample t-tests. To test the assumptions of each t-test, Levene’s test, with alpha set at .05, was used to evaluate homogeneity of variance. In addition, each variable was examined for normality by examining the 95% confidence intervals around the skewness and kurtosisness statistics (Mertler & Vannatta, 2001). Using these criteria, all of the variables met the assumptions of an independent sample t-test. The source tables for all analyses along with the results of their respective Levene’s tests and normality statistics are reported in Appendix H.

Family-Wise Error Rate

For the current experiment, three different theoretical constructs (i.e., Content, Structure, and C&S SK) were measured. In addition, each construct was measured in two ways (with a SK test and a matrix). Therefore, there were six measures taken from each participant. Five of these measures met the criterion for analysis (i.e., one was eliminated as described above). The large number of independent tests that were necessary to evaluate this high number of dependent measures caused concern over the possibility of an inflated Type 1 error rate. In order to mitigate this issue, families of tests were identified and the family-wise error rate for each was set to .05.  Specifically, since each construct was identified a priori, it was determined that all analyses that were related to a particular construct would be considered a family because they test different predictions derived from the HuWI cycle. Therefore, the following section describes three families of analyses, two of which required a Bonferroni adjustment in order to control the family-wise error rate when more than one measure was used.

Data Analysis

Table 1 shows the means and 95% confidence intervals for each of the dependent variables. One should note first the low scores obtained for the matrix tests relative to the SK tests. This is to be expected given the lower expected values due to chance (see the chance levels reported in Table 1). However, as noted above, dependent variables were analyzed only if one of the groups performed above chance.

As can be seen in Table 1, the mean for the webmaster group appears slightly higher than the mean for the storywriter group on the Structure SK variable. However, there was not a significant difference between these groups, t (39) = 0.97, p > .05[4].

The content SK family of analyses, on the other hand, yielded differences between the groups. The storywriter group scored higher than the webmaster group on the Content SK Test, t (39) = 2.66, p < .05/2, h² = .15, and on the Content Matrix Test, t (39) = 2.50, p < .05/2, h² = .14.

Similarly, the C&S family of analyses yielded differences between the groups. Specifically, the storywriter group scored higher than the webmaster group on the C&S SK Test, t (39) = 4.55, p < .05/2, h² = .35, and on the C&S Matrix Test, t (39) = 2.50, p < .05/2, h² = .14. 

Discussion

Experiment 2 tested whether or not interaction with the website sampled goal-relevant information and whether or not SK was modified by this information, as predicted by the HuWI cycle. Specifically, it was predicted that if interaction sampled goal-relevant information and that information modified SK, then 1) the webmaster group would score highest on the structure SK test and 2) the storywriter group would score highest on the content SK test. In addition, the possibility that particular kinds of SK were easier to acquire than others was explored. It was expected that, if content SK was easier to acquire than structure SK, as posited by Farris et al. (2002), and participants sampled goal-relevant information, then the storywriter group should outperform the webmaster group on the C&S SK test. The following sections describe each of these hypotheses with respect to the obtained results. Finally, implications for the HuWI cycle and web design are discussed.

Goal-Relevancy and System Knowledge Modification

Based on the HuWI cycle, it was predicted that participants who used the website in order to make plans to alter its content (i.e., the storywriter group) would acquire more content SK than participants who used the same website in order to make plans to alter its structure (i.e., the webmaster group). Results of the analysis of content SK scores and the content SK matrix revealed that the storywriter group scored higher than the webmaster group on both of those tests. Therefore, it appears that if a user’s goal was related to a website’s content, then content SK was modified more than if the user’s goal related to structure.

These results suggest two things. First, they suggest that content information can be sampled and that it can modify SK. This is consistent with other research that has shown that participants do learn content from websites when given a post website navigation test (e.g., Elgin, et al., 2001; Leventhal, et al., 1993; Rouet, 2003; Shapiro, 1999). Second, as predicted by the HuWI cycle, the results suggest that the sampling of content information and subsequent modification of content SK depends on the user’s goal. Specifically, if users’ goals relate to content, then they will acquire more content SK than if their goals relate to a website’s structure. This highlights the importance of users having a content-related goal in order to acquire content SK, and suggests that the acquisition of at least content SK was goal-dependent. However, this was not the case when the goal related to structure.

Results of the analysis of structure SK scores revealed that the acquisition of structure SK did not depend on whether or not the goal related to structure. That is, based on the HuWI cycle, it was predicted that participants who used the website in order to make plans to modify its structure (i.e., the webmaster group) would sample more structure information and subsequently modify structure SK more than participants who used the same website in order to make plans to modify its content (i.e., the storywriter group). The analysis of structure SK test scores revealed that there were no differences between the webmaster and storywriter groups on the structure SK variable.

There are at least three potential explanations for this outcome. First, it is possible that the manipulation did not focus the participants’ attention on structural information. However, this appears to be the least likely explanation for several reasons. Specifically, the analysis of self-report measures and the number of actions taken while exploring the website suggest that the webmaster group attended to structure more than the storywriter group. In addition, as indicated by the differences in content SK, the webmaster group did not appear to have attended to content as much as would have been possible. Therefore, although it is still possible that participants did not attend to structure adequately, it appears to be less likely than the following explanations.

Alternatively, it is possible that the employed measure of structure SK was not sensitive enough to capture any true difference that might have existed. Although the structure SK test, unlike the structure matrix test, revealed scores for at least one group that were above chance, examination of the means in Table 1 reveals that the scores on the test were only 7.74 % above chance for the webmaster group and 4.38% above chance for the storywriter group. Scores that appear to be so near chance level raise the suspicion that the test was either too difficult or insensitive to structure SK. Future research should try alternative measures of structure SK (e.g., drawing or card-sorting techniques) in order to determine whether or not this potential insensitivity explains why differences were not observed between groups with widely different goals.

A third candidate for not detecting a difference between the groups on structure SK is that structure SK is not easily modified, making detecting changes in structure SK difficult regardless of the way it is measured. In other words, structure SK might not accommodate information as easily as does content SK. This explanation would be consistent with research that suggests that structure SK is difficult, if not impossible to acquire (i.e., Farris et al., 2002). The following section delves further into this possibility.

Goal-Relevancy and Structure vs. Content System Knowledge Modification

As mentioned previously, Farris et al. (2002) challenged the notion that users can acquire knowledge about a website’s structure. Participants in their study drew the structure of a recently explored website. The researchers noted that the drawings did not resemble the website’s structure, but rather the drawings reflected the relations between the concepts that were presented within the website (i.e., the content SK). Therefore, they concluded that users do not develop a strong understanding of a website’s structure. However, one potential shortcoming of their study was that the specific goals of the participants appeared to have related to content, rather than structure. According to the HuWI cycle, if the goal had related to structure, then it is possible that they could have acquired structure SK.

Based on Farris et al.’s (2002) results, along with predictions derived from the HuWI cycle, the current study investigated the possibility that users could acquire structure SK if their goal related specifically to the website’s structure. The results of the reported study were consistent with the results obtained by Farris et al., but not with the predictions derived from the HuWI cycle. That is, the results suggested that both groups of participants learned very little about the website’s structure.  However, it appears that one might be able to modify structure SK.  This is suggested by the above chance structure SK test performance in the webmaster group. Accordingly, it appears that structure SK is more difficult to acquire than content SK.

In anticipation that this issue might need to be addressed, measures of both content and structure SK (i.e., the C&S SK test and C&S matrix test) were administered. The items on these tests required participants to identify relations between characters in the website that existed in both the content and the structure, as opposed to only one or the other as was the case earlier. Analyses of these tests revealed that the storywriter group, whose task related more to content, recalled more C&S relationships than the webmaster group. This suggests that when the goal related to content, SK was modified more than when the goal related to structure. In other words, it appears that content SK accommodates information more readily than structure SK. Because it is possible that it is difficult to modify structure SK, future research should focus on finding more sensitive measures to assess structure SK modification

Beyond finding ways to assess structure SK, one must ask why it might be more difficult to accommodate information into structure SK than it is for content SK. Recall that SK is a schema, and a schema is a pattern of knowledge that is frequent in a situation. In addition, the pattern of knowledge is a group of knowledge elements for a situation, which are general memories. When developing a schema, therefore, one must encode information into memory that will later be recalled. Accordingly, one should not only examine the information that is going to be encoded into memory, but also how that information is encoded and later recalled.

Numerous studies have shown that elaborative rehearsal will lead to deep processing of information, whereas maintenance rehearsal will lead to shallow processing of information (Craik & Lockhart, 1972; reviews of empirical findings presented in Baddeley, 1990 and Horton & Mills, 1984). In addition, recall of information that is encoded into memory using deep processing is far superior to that which is encoded using shallow processing, supposedly because deep processing creates more connections in memory to facilitate recall (Craik & Watkins, 1973).

Based on this, one could argue that structure information is not meaningful enough to facilitate deep processing, which makes it less memorable than content information. In fact, the only meaning that can be found in structure information, without creating meaning as a mnemonic, is that a webpage either is or is not connected to another webpage. Content information, on the other hand, can be meaningful enough to facilitate deep processing, which would make it more memorable than structure information. For example, knowing that two characters are married, and one is having an extramarital love affair with their spouse’s sibling has much more meaning to be encoded into SK than simply the fact that all of these characters webpages were linked. Therefore, it is possible that structure SK is more difficult to recall than content SK because it is not encoded as well.

Alternatively, one could argue, as suggested by Farris et al. (2002), that typically users do not process structure information when they use the web. Recall that Farris et al. (2002) demonstrated that users could not acquire structure SK on a task where the goal was not specific to structure. One of their conclusions was that users do not normally process structure information because they use the web with goals that relate to processing content. In order to address this issue, the reported experiment utilized a goal (i.e., being a webmaster) that would require the processing of structure information. Although the issue of experience with processing structure information remains unanswered, it does not appear to be likely that simply having a goal that relates to structure will make users acquire structure SK.

Because having a goal that relates to structure does not influence the processing of structure information, then what other reason could there be for not acquiring structure SK?  One likely answer was suggested by Farris, Jones, and Elgin (2001, 2002) and Boechler (2001). That is, the concept of “structure” in hypermedia research may be the product of the “spatial” metaphor that has been used to simplify discussions of hypermedia.[5] The term “structure” assumes spatial properties such as distance, direction, and shape exist. However, Farris et al. (2001, 2002) and Boechler (2001) argue that websites are not spatial, thus they contain no structure information. In other words, websites do not have the properties that we associate with a physical structure, so there is no reason to assume that structural information can be sampled from them. There is no distance, direction, or shape on a website, so no information can exist about these non-existent properties.

Implications

The HuWI Cycle

In general, the current study demonstrated that content SK was modified by the content information in the website and that this modification was dependent upon whether or not the participants’ goal related to content. However, assuming that structure information can be processed, it appears that content SK was more accommodating to new information than structure SK. Consistent with other research (e.g., Farris et al., 2002), there appears to be little if any modification of structure SK.

This finding is inconsistent with the “plausibility hypothesis,” which, as defined by Tergan (1997), states that users will acquire the structure of a website simply by using it, and by learning the structure of a website they will learn the relationships between concepts. In other words, there appears to be no support for Kearsly’s (1988) supposition that the web will improve learning because it focuses attention on structural information since it is either not possible or difficult to modify structure SK. However, there is evidence to suggest that users will learn the relationships between concepts and those conceptual relationships will be construed as the website’s structural relationships (Farris et al., 2002). In other words, based on the available evidence, content is the only information that is learned, and it is the only information used to guide exploration. 

In addition, an implicit assumption of the plausibility hypothesis is that users will “serendipitously” sample goal-irrelevant information (Cunningham et al., 1993), like structure SK. The reported study showed that content SK was best modified by the information in the website when the goal relates to content and that structure SK does not appear to be modified even when the goal relates to structure. Therefore, there appears to be no means by which information presented on the web will be sampled serendipitously in order to modify structure SK.

Although, in general, the current study demonstrated that content SK was modified by the content information in the website and that this modification was dependent upon users’ goals, there still are several uninvestigated issues, some of which have been mentioned already. For example, the issue of SK accommodation might be explored further by investigating what makes SK more or less able to accommodate additional information. For example, will different types of website structures (e.g. linear, hierarchical, etc.) be more or less susceptible to structure SK modification because they vary inherently in ease of use (Larson & Czerwinski, 1998)? Are there ways to promote deeper processing of structure SK so that it can be accommodated? Addressing issues such as these might lead to models that will be able to predict the effectiveness of learning on the web.

Web Design

Web design, in general, is an iterative process of gathering information about target users and incorporating this information into the design and redesign of the website (Mayhew, 1999). The current study highlights the need of not only understanding how users think about information (i.e., their mental model), but also understanding the relationship between the user, the system, and the goal. The former is what is known as user-centered design and is championed by many usability professionals (Mayhew, 1999, Nielsen, 2000). The latter is known as use-centered design and is a less commonly found methodology in the HCI literature[6]. In general, user-centered design, as the name implies, focuses on the user and the user’s goals in order to design a system. Use-centered design does not focus on the user or the goal as separate entities, but rather on the relationships between these entities so that the user, system, and goal can be integrated in order to complete a task (Flach & Dominguez, 1995).

The current study found evidence in support of the use-centered design methodology. Specifically, it was demonstrated that it is important to understand the relationship between the user, system, and the goal. One could imagine instances in which the user’s goal is incongruent with the website’s goal, thus causing problems for the user. For example, users searching for information about how to install additional memory into their computers might have problems finding the information on a website that sells memory upgrades, even though the information is there. In this example, the relationship between the goal of the user (i.e., to locate memory installation instructions) and the system (to sell computer hardware) is incongruent. Many technology websites deal with this problem by having a “support” section of the website, which contains information for existing customers (e.g., installation information). This type of strategy separates clearly the website into sections that center on different possible user goals (e.g., seeking information about a purchased product or seeking a product to purchase). Therefore, whether it is explicitly known or not, many websites do organize around user goals. However, the process to reach this design decision is unknown. One could speculate that it would have required several iterations of tedious user testing in order to arrive at the same design that would have been conceived of by using a use-centered design methodology.

In addition to supporting the use-centered design methodology, the current study has implications for the design of distance education courseware. As mentioned, there was no evidence in support of the plausibility hypothesis because it was not apparent that structure SK was modified, even when the goal was to study structure information. Since structure SK was not modified, it is assumed that users do not sample structure information. Taken with Farris et al.’s (2002) results, it is suggested that users acquire content SK and construe the relationships in content SK as the structure for the website.

The lack of evidence for the plausibility hypothesis is consistent with research that suggests that the non-linear structuring of subject matter in hypertext format does not improve comprehension and retention of subject matter compared to linear text, as assumed by the plausibility hypothesis (Tergan, 1997). In other words, because there is no evidence that users acquire a substantial amount of structural SK, there is no reason to assume that a hierarchical presentation will be more or less effective than a linear presentation, such as that in a textbook. Therefore, developers of web courseware should focus their efforts on other ways to use technology in order to more effectively present information to the user (e.g., the use of sound and motion, etc.; e.g., Najjar, 1998), rather than assume that the website will be effective for learning simply because it is nonlinear.