Usability Test Plan: Graphic History Browser

Doug Bowman

Deirdre Devane

David Koller

Don Turnbull

CS 6752

,

Overview of the Application

This system is not a stand-alone application. Rather, it is completely integrated to work with NCSA Mosaic for X Windows. To use the system, the user selects a menu item titled "Graphic History" in the "Navigate" menu of Mosaic. This brings up a separate window in which the graphic history will be displayed.

In the history browser, each page is symbolized as a node in a tree. This node can be viewed in many different ways, as we will explain later. As the user navigates through the Web, the history browser presents the following information:

* The current node is shown highlighted with reverse video.

* Each time the user follows a link to a new page, a new node is added to the current tree, with an arrow linking the previous node to the new one.

* Each time the user goes to a new page by specifying a URL or jumping to a page on his/her hotlist, a new tree is begun with this page as the root node.

* If the user follows a link which leads to a page already represented in the tree, a downward arrow is attached to the node being linked from, and an upward arrow is attached to the node being linked to. The node is not added as a new node to the tree again.

As stated above, the user has several options regarding the appearance of the nodes. Using check boxes in the "Options" menu, or using the "Preferences" dialog box, the user can specify whether or not to view the node's title, its URL, and/or a thumbnail image of the page. The user also has some control over the size of the nodes and whether full or abbreviated titles will be used. The user can "zoom in" or "zoom out" to view more or less of the tree at once, and also has the option of collapsing or expanding any subtree, in order to focus on certain sections of the tree.

All of these options aid the user in visualizing the structure of the WWW, and particularly the specific path taken through the nodes. However, much of the real power of the history browser lies in its ability to aid navigation as well. Besides simply seeing where they are, where they've been, and how to get back to a certain page, the users of this system can use it interactively to quickly return the browser to any previous location.

To do this, the user must simply locate the desired node in the tree that was automatically built as they navigated, and either double-click with the left mouse button or single-click with the right button. Either action will cause the Mosaic browser to return immediately to the specified page. This facility is valuable because the only way to accomplish the same task using Mosaic alone is to use the "Back" and "Forward" buttons. When many nodes have been visited in a session, getting back to a certain page can be memory taxing and time consuming.

The history browser also provides interactive feedback to help the user find the desired node. If the user has set the option for no title or a shortened title, it may be difficult to find the correct node. However, passing the pointer over a node causes its full title and URL to appear in text boxes at the bottom of the display. Also, passing the pointer over an upward or downward arrow causes the linked-to or linked-from nodes to be highlighted. This may be helpful in finding a node that the user remembers relative to its link to/from some other node in the tree.

Additionally, the history browser system supports saving the tree to an ASCII file. This file can then be reopened in a later session, so that the user can resume exploration/navigation at the same point where they left off, without having to start from the beginning again. This is more flexible and powerful than saving each important page in a hotlist entry.

Finally, some online help is offered to users of the system. Choice of the help button or menu item causes a new Mosaic window to appear, which jumps to a help page which is, itself, a WWW page.

Statement Of Purpose

More specifically, we want to learn, first and foremost, what value, if any, the graphical history browser's representation and navigation tools add to the basic features provided by Mosaic. The questions that we intend to answer are: Does the tree structure help users better understand the path they have taken through the Web? Can users more easily navigate to a desired location using this tool?

To determine the answers to these questions, some comparative tasks are required. In other words, our participants will perform a given benchmark task using the Mosaic standard Window History feature, and then perform the same task using Mosaic with the graphic history browser. Contrasting the differences and comparing the similarities between the timings and the number of unnecessary node visits for these tasks will give us some indication of the added value that the history browser gives to Mosaic.

In addition, we want to evaluate whether the physical layout and appearance of the nodes are conducive to--on inspection--searching for a desired node and navigating between the nodes in the tree. Simple benchmark tasks isolating these features will be helpful in evaluating these features of the system. Also, our study should include some analysis of advanced feature usage. We would like to know which features of the history browser will be accessed by a typical user. To get some information about the advanced features, we should include at least one benchmark task which forces the user to use an advanced feature. This will give us information on whether the feature is easy to use. Also, to determine the overall coverage of features, we should include some time for the participant to freely play with the system and take note of which features are accessed.

If certain features are never used, or are used incorrectly, redesign of the interface with respect to those features--or their complete removal from the system--may be in order.

As an aside, we will attempt to establish whether the layout and organization of the GVU WWW pages are conducive to helping visitors to the pages find the information that interests them. It might also be interesting to assess whether the use of the history browser aids in these searches. To meet these objectives, we will use the GVU WWW pages as input data for our tasks. We will use the answers from the questions on the feedback questionnaire to assess the users' opinions of the GVU WWW pages. Other subjective data such as talking out loud protocols and user expressions or frustration will also be used.

Usability Specifications

Usability Attribute:                 Learnability                                   
Measuring Instrument:                "Build up a tree" task per B-1                 
Value to be Measured:                Whether participant is able to figure out      
                                     any feature of the GHB (or WH)                 

Usability Attribute:                 Complexity                                     
Measuring Instrument:                "Revisit a page " task per B-2                 
Value to be Measured:                Number of non-progressive node visits          
Methodology:                         Measure for WH and GHB for early task and      
                                     late task, respectively and compare            

Usability Attribute:                 Complexity                                     
Measuring Instrument:                "Retrace  path" task per B-3                   
Value to be Measured:                Number of non-progressive node visits          
Methodology:                         Measure for WH and GHB for early task and      
                                     late task, respectively and compare            

Usability Attribute:                 Effectiveness                                  
Measuring Instrument:                "Sketch structure of tree traversed" task      
                                     per B-4                                        
Value to be Measured:                Percentage of missing, incorrect, and          
                                     wrongly-placed nodes                           
Methodology:                         Measure for WH and GHB for early task and      
                                     late task, respectively and compare            

Usability Attribute:                 First Impression, Initial Performance          
Measuring Instrument:                "Save Tree" task per B-5                       
Value to be Measured:                Length of time required to save tree           
Worst Acceptable Level:              1 minute                                       
Planned Target Level:                30 seconds                                     
Best Possible Level:                 10 seconds                                     

Usability Attribute:                 First Impression, Initial Performance          
Measuring Instrument:                "Load Tree" task per B-6                       
Value to be Measured:                Length of time required to load tree           
Worst Acceptable Level:              1 minute                                       
Planned Target Level:                30 seconds                                     
Best Possible Level:                 10 seconds                                     

Usability Attribute:                  Advanced Feature Usage                        
Measuring Instrument:                "Configure Display"  task per B-7              
Value to be Measured:                Length of time required to find option         
Worst Acceptable Level:              1 minute                                       
Planned Target Level:                10 seconds                                     
Best Possible Level:                  5 seconds                                     

Usability Attribute:                 Advanced Feature Usage                         
Measuring Instrument:                "Use multiple features to locate URL" task     
                                     per B-8                                        
Value to be Measured:                Number of features used                        
Worst Acceptable Level:              1 feature                                      
Planned Target Level:                2 feature                                      
Best Possible Level:                 6 features                                     

Usability Attribute:                 Learnability                                   
Measuring Instrument:                "Figure out arrows meaning" task per B-9       
Value to be Measured:                Number of incorrect answers given in           
                                     describing functionality                       
Worst Acceptable Level:               1 incorrect answer                            
Planned Target Level:                 0.5 incorrect answers                         
Best Possible Level:                  0 incorrect answers                           

Usability Attribute:                 Overall Impression                             
Measuring Instrument:                Feedback Questionnaire                         
Value to be Measured:                Average score (numerical range -2 to 2)        
Worst Acceptable Level:              -2.0                                           
Planned Target Level:                1.5                                            
Best Possible Level:                 2.0                                            

Benchmarking Task Details

The evaluators will be divided into two groups: Group A and Group B. Group A evaluators will use the standard Mosaic Window History (WH) feature before the GHB, and Group B will use the GHB before the WH. The following details of benchmark tasks are in the order that Group A will perform them. Appendix F contains the complete task script for both Group A and Group B.

Benchmark Task 1 (B-1): Have the user build up a path

For this benchmark task, we seek to gauge how well the participant can figure out to use the GHB (or WH) and locate basic features.

Benchmark Task 2 (B-2): Revisit a page

The purpose of this benchmark task is to observe whether the evaluator--recognizing that this is a revisit--uses the GHB (or WH) to get back to a page.

Benchmark Task 3 (B-3): Retrace a path

The purpose of this benchmark task is to observe whether the evaluator--recalling the path that she has traversed--uses the GHB (or WH) to locate this page. This differs from the previous one in that this page has not been visited before; however, a previously visited page is in the path to the page to be visited in this task.

Benchmark Task 4 (B-4): Sketch structure of tree traversed

Assess whether or not the GHB (or WH) aids the evaluator in establishing a mental model of the path she has traversed.

Benchmark Task 5 (B-5): Save a tree

We anticipate feedback from the user on the interface of the "Save" pop-up window. Hence, this benchmark task is included to assess whether or not the participant confirms or does not confirm our criticism.

Benchmark Task 6 (B-6): Load in a tree

We anticipate feedback from the user on the interface of the "Open" pop-up window. Hence, this benchmark task is included to assess whether or not the participant confirms or does not confirm our criticism.

Benchmark Task 7 (B-7): Have the participant configure the display of a tree

This benchmark task is intended to measure a variety of things. First, we expect feedback on the layout of the "Preferences" pop-up window. Next, we want to see whether or not the participants are able to figure out how to locate and use various advanced features such as `Name-length,' `Zoom Out,' and `Display URL.' Finally, we hope to assess whether or not the participants are able to utilize the GHB in various display modes.

Benchmark Task 8 (B-8): Use multiple advanced features to locate URL

We want to observe whether the "Title and URL Fields" of the GHB "confuses" the participants or whether they are able to figure out the functionality of them. Secondly, we want to observe the techniques that the participants use to browse a tree. Finally, we would also like to observe whether the participants are able to select the features that will aid them in locating a URL. Hence, this benchmark task is seeks to fulfill these objectives.

Benchmark Task 9 (B-9): Find out what the arrows mean to the user

The group is not sure whether the meaning and functionality of the arrows will be apparent to the users. This benchmark task is meant to assess this.

Description Of Laboratory Sessions

Environment and Lab Setup

For this study, the equipment necessary will be fairly minimal. To run the software, a Sun workstation running SunOS 4.1.3 or better is necessary, and 24-bit color is preferable. If the machine does not have 24-bit color, the thumbnail images on the history browser nodes will have colormap problems and will change colors as new nodes are added and new pages are accessed.

This could be considered as a valid part of the study, however, since it is likely that many, if not most, of the users of such a system will not have access to a 24-bit color machine. Since the users would have to deal with these color problems in a real life situation, so why not deal with them in the experiment?

For video, our needs are fairly limited as well. Because of the placement of the Sun workstation in the lab, the cameras will be placed to the left of the user, one slightly in front and one behind. With the camera in front, the user's facial expressions can be taped, which is a helpful subjective measure of satisfaction or frustration. With the camera behind, we can capture the screen events, which are the most important thing, and also be able to see the use of the mouse and mouse buttons. We feel this will be important, since in our guideline-based evaluation we pointed out several problems with the mappings of the mouse-buttons. On our video, we would like to be able to see whether users use the wrong buttons or the wrong number of clicks for a certain task. The timer on the video will be used to confirm stopwatch clockings for measuring time elapsed for various user tasks.

The laboratory intercom will be used for communication with the participants during the lab sessions. In addition, a microphone, placed on a separate table nearby the workstation, will be used to record the comments of the usability tester.

Other Materials

* Appendix B -- Pre-Session (User Background) Questionnaire

* Appendix C -- Post-Session (User Feedback) Questionnaire

* Appendix D -- Instructions given to each evaluator

* Appendix E -- Consent Form given to each evaluator

Session Structure and Duration

First, each participant will get a brief tour of the usability lab and its components, on both sides of the mirror. This helps to assure them that they are not the ones being tested, and takes away some of the feeling of the unknown. Next, the users will be asked to read and sign the consent form, and will answer the pre-test questionnaire. Finally, participants will receive a brief introduction to the graphic history browser system - its purpose and features, not its exact usage. This introduction/instruction will be scripted, so that it is the same for each person.

The user then begins work on the benchmark tasks. Throughout this period, they will be strongly encouraged to talk out loud about what they are thinking and doing as they perform the tasks with the system. This section will take the majority of the time, and it will be broken up into three short parts. Each part will each contain some generic tasks, but will also contain some specific tasks for use in our comparative analysis. In the first and second parts, the user will use the standard Mosaic history browser and the graphic history browser (the order will vary among participants, as we are doing a comparative study). The third set of user tasks will concern the usability of the GVU Center Web pages.

After the three task sections are completed, the participant will answer the post-session questionnaire, and will be informally interviewed to obtain any final comments or suggestions he/she might have. We will thank the user, and most likely have some food as a small payment for helping us in our study.

Task Script

Selection Process And Criteria For Participants

Also, since we plan to get feedback on the organization of the GVU WWW pages, our users should not be intimately familiar with the layout and content of these pages. If our participants had already frequently accessed these pages, their times would be significantly lower and their errors significantly fewer than the average visitor to the site. However, there should be no shortage of people, even here on campus, who have used Mosaic but are not familiar with these specific pages.

It shouldn't be too difficult to round up users with the correct attributes to participate in our study. These users could be pulled from the CoC, the Tech campus in general, or our personal friends. The only thing to be careful about with users from the CoC is to make sure that they are not too familiar with the GVU Web Space, since that would bias the results of the search tasks on those pages.

Potential users will be informally screened by team members to make sure that they meet the two criteria outlined above.

Appendix A: Diagram of Usability Lab Setup
Appendix B: Pre-Session (User Background) Questionnaire

1) Please specify your job title, if any. If you are a student, please specify your classification.

2) How often do you use a computer? (Circle the best answer)

a) Daily b) Once a week c) 1 - 3 times per month d) Rarely e) Never

3) Please rate your familiarity with the World Wide Web (WWW). (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

unfamiliar familiar

4) What WWW browser(s) have you used?

5) How often do you use Mosaic? (Circle the best answer)

a) Daily b) Once a week c) 1 - 3 times per month d) Rarely e) Never

6) When you use Mosaic, how often do you use the standard "Window History" feature? (Circle the best answer)

a) Often b) Sometimes c) Rarely d) Never

7) If Mosaic is not the WWW browser that you use most often, please specify the WWW browser that you do use most often and rate the frequency in which you use it.

Name of WWW browser:

Frequency of use: (Circle the best answer)

a) Daily b) Once a week c) 1 - 3 times per month d) Rarely e) Never

8) Please list any college degrees--if any--that you have or are currently working towards.

9) How often have you used the GVU World Wide Web pages?

a) Daily b) Once a week c) 1-3 times per month d) Rarely e) Never

Appendix C: Graphical History Browser for Mosaic Feedback Questionnaire

1) What features of the Graphical History Browser (GHB), if any, were most useful to you and why?

2) What features of the GHB, if any, seemed useless to you and why?

3) What features, if any, would you add to the interface that would make the application more useful to you?

4) Visually, how would you rate the interface? (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

not pleasing

pleasing

5) Visually, what improvements would you suggest be made to the interface to make it more pleasing?

6) How often did you "get lost" navigating through the WWW while using the GHB? (Circle one of the following)

a) often b) sometimes c) rarely d) never

7) How often did you "get lost" navigating through the WWW while using the standard Mosaic Window History feature?

a) often b) sometimes c) rarely d) never

8) How often did you consult the "help" feature when using the GHB?

a) often b) sometimes c) rarely d) never

9) Please rate the ease of using the GHB. (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

difficult easy

10) Please rate the usefulness of the GHB. (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

useless useful

11) Please rate the usefulness of the standard Mosaic Window History feature. (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

useless useful

12) If the GHB were a permanent feature to Mosaic, would you use the GHB?

a) yes b) no

Please elaborate on your answer.

13) If you answered "yes" to question 12 and if Mosaic is not the WWW browser you use most often, would addition of the GHB to Mosaic make you inclined to use Mosaic more often than your current choice of WWW browser?

a) yes b) no

14) Visually, how would you rate the GVU Web pages? (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

not pleasing

pleasing

15) How often did you "get lost" while navigating through the GVU Web pages?

a) often b) sometimes c) rarely d) never

16) How comfortable were you in using the Glimpse search facility to locate specific information in the GVU Web pages? (Indicate your answer by circling a "dot" on the scale below.)

.----------. ----------.----------.----------.

uncomfortable comfortable

17) What general comments/suggestions do you have regarding the usability of the interface for navigating and finding information in the GVU Web pages?

Appendix D: Evaluator Instruction Script

You will be given a list of tasks to perform using the system. Work through them as efficiently as you can, but do not rush. Before beginning each task, please read the instructions out loud. Please continue to talk out loud as you work. Describe your thoughts about the system, why you are performing certain actions, how you make your decisions, and so forth.

If you feel you are stuck and would like to ask us a question, please stop work and press the call button on the intercom behind you. We will help you in any way we can.

There will be three sessions of tasks, each of which should take you about 20 minutes to complete. You will be instructed when to begin.

Do you have any questions at this time?

Special Instructions For Session 1 (Window History Feature)

Special Instructions For Session 2 (Graphic History Browser)

Appendix E: Informed Consent Form

Your rights as a participant are as follows:

1. You have the right to withdraw from the session at any time for any reason.

2. At the conclusion of the session, you may see your data, if you so desire. If you decide to withdraw your data, please inform one of the evaluators immediately.

3. You are requested not to discuss this session with other people who might be in the group from which other participants could be drawn.

4. Finally, we greatly appreciate your time and effort for participating in this evaluation. Remember, you cannot fail any part of this session, and there are no right or wrong answers. The session is to identify usability problems with the software. Your signature below indicates that you have read this consent form in its entirety and that you voluntarily agree to participate.

NAME ___________________________________________

SIGNATURE _______________________________________

PHONE _____________________ DATE _________________

Appendix F: Task Script and Instructions

Group A
Part 1

1. From the GVU homepage, find out who will be the speaker at the May 25 "Brown Bag" program.

2. Find out the title of the 1995 GVU Technical Report numbered 95-06.

3. Benjamin Watson is a graduate student at the GVU Center. Find out from what university Ben received his B.S. degree.

4. Determine the names of the two lead authors of the 1995 GVU Technical Report numbered 95-14.

5. Determine the number of GVU Technical Reports that were filed in 1994.

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 2

When you finish drawing your sketch, please verbally indicate so and wait for further instructions.

Part 3

1. Start up the Graphic History Browser (GHB) by selecting its menu choice under the "Navigate" menu of Mosaic.

2. Consult the on-line help for the GHB, and skim the help information for a minute or two.

3. From the GVU Home Page, explore the "People" section of the GVU Web. Spend 5-7 minutes freely exploring this section of the Web, noting the function and behavior of the Graphic History Browser (GHB) as you navigate about. Feel free to experiment with some of the features of the GHB during this time, and consult the on-line help if you need any assistance.

4. Save your resulting GHB tree to the file "~bowman/scivis/histbrowser/my.tree".

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 4

1. Open the Graphic History Browser window.

2. Starting from the GVU homepage, use Mosaic to find out which German-based company is a GVU Industrial Affiliate.

3. Determine the first "Keyword" in the abstract of the GVU Technical Report numbered 95-07.

4. Dr. John Stasko is a GVU faculty member and an Associate Director. What is his full Associate Director title/position?

5. Determine the names of the two authors of the 1993 GVU Technical Report numbered 93-21.

6. Determine from what university Dr. John Stasko (GVU faculty member and Associate Director) received his PhD degree.

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 5

When you finish drawing your sketch, please verbally indicate so and wait for further instructions.

Part 6

1. Open the Graphic History Browser window.

2. Load the GHB tree file "/net/students/general2/koller/CS6752/test.tree"

3. Locate the node in the GHB window with the title "Dynamic Simulation of Splashing Liquids." Read aloud the URL corresponding to this node.

4. Locate the node in the GHB window with the title "GVU Center -- Multimedia Educational Delivery System." Determine the title of this node's parent node.

5. State the complete title of the currently highlighted node in the GHB window.

6. Count or otherwise determine the total number of nodes currently displayed by the GHB.

7. Demonstrate by example how to change the GHB node titles from full length to abbreviated length, and vice versa.

8. Using the GHB, find a node whose URL contains the number "50." State the complete title of this node.

9. Demonstrate by example how to turn "thumbnail" images on and off within the GHB.

10. Determine and then describe the function of the arrows that connect the nodes in the GHB.

11. Collapse and then uncollapse the largest GHB subtree in the "GVU Research" tree.

12. Determine what corporation holds the copyright on the GHB.

13. GHB has a number of "preferences" and other display configuration options. Configure these options so that it is most useful to you for navigating through the WWW.

Task Script and Instructions

Group B
Part 1

1. Start up the Graphic History Browser (GHB) by selecting its menu choice under the "Navigate" menu of Mosaic.

2. Consult the on-line help for the GHB, and skim the help information for a minute or two.

3. From the GVU Home Page, explore the "People" section of the GVU Web. Spend 5-7 minutes freely exploring this section of the Web, noting the function and behavior of the Graphic History Browser (GHB) as you navigate about. Feel free to experiment with some of the features of the GHB during this time, and consult the on-line help if you need any assistance.

4. Save your resulting GHB tree to the file "~bowman/scivis/histbrowser/my.tree".

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 2

1. Open the Graphic History Browser window.

2. Starting from the GVU homepage, use Mosaic to find out which German-based company is a GVU Industrial Affiliate.

3. Determine the first "Keyword" in the abstract of the GVU Technical Report numbered 95-07.

4. Dr. John Stasko is a GVU faculty member and an Associate Director. What is his full Associate Director title/position?

5. Determine the names of the two authors of the 1993 GVU Technical Report numbered 93-21.

6. Determine from what university Dr. John Stasko (GVU faculty member and Associate Director) received his PhD degree.

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 3

When you finish drawing your sketch, please verbally indicate so and wait for further instructions.

Part 4

1. Open the Graphic History Browser window.

2. Load the GHB tree file "/net/students/general2/koller/CS6752/test.tree"

3. Locate the node in the GHB window with the title "Dynamic Simulation of Splashing Liquids." Read aloud the URL corresponding to this node.

4. Locate the node in the GHB window with the title "GVU Center -- Multimedia Educational Delivery System." Determine the title of this node's parent node.

5. State the complete title of the currently highlighted node in the GHB window.

6. Count or otherwise determine the total number of nodes currently displayed by the GHB.

7. Demonstrate by example how to change the GHB node titles from full length to abbreviated length, and vice versa.

8. Using the GHB, find a node whose URL contains the number "50." State the complete title of this node.

9. Demonstrate by example how to turn "thumbnail" images on and off within the GHB.

10. Determine and then describe the function of the arrows that connect the nodes in the GHB.

11. Collapse and then uncollapse the largest GHB subtree in the "GVU Research" tree.

12. Determine what corporation holds the copyright on the GHB.

13. GHB has a number of "preferences" and other display configuration options. Configure these options so that it is most useful to you for navigating through the WWW.

Part 5

1. From the GVU homepage, find out who will be the speaker at the May 25 "Brown Bag" program.

2. Find out the title of the 1995 GVU Technical Report numbered 95-06.

3. Benjamin Watson is a graduate student at the GVU Center. Find out from what university Ben received his B.S. degree.

4. Determine the names of the two lead authors of the 1995 GVU Technical Report numbered 95-14.

5. Determine the number of GVU Technical Reports that were filed in 1994.

When you finish these tasks, please stop and do not proceed any further until directed to.

Part 6

When you finish drawing your sketch, please verbally indicate so and wait for further instructions.