Introduction and Background

The Classroom 2000 project has used several different web-based presentation techniques to provide students offline with as much of the actual classroom environment as possible. For this critique, the class notes that will be used are from the Fall 1997 CS 6751 course. The main evaluation criteria for this critique are the usability and usefulness of the web notes interface. An example set of class notes used for this critique can be found at URL: http://c2000.gatech.edu/zenpad/classes/cs6751_97_Fall/Oct.23.1997.1/html/

In order to evaluate this notes system, a heuristic evaluation will be performed. However, due to the limited time frame for this assignment, only one evaluator will be used. Normal heuristic evaluations are performed by 3 to 5 independent evaluators. After the evaluation, all evaluators collate their findings and rank each problem on a scale from 0 to 5. This severity ranking is based mainly on four criteria: frequency, impact, persistence, and market impact. The first is the number of times the usability error is encountered within the system. The second is a measure of the effect the error has on the user. Third is how difficult the error is to undo or fix, and the last measure is of the effect the error will have on the marketability of the system.

Heuristic evaluations require the evaluator(s) to examine the system using a set of guidelines or heuristics to determine problems which should be addressed by the design team. There are several ways to produce the set of guidelines for a heuristic evaluation. Jakob Nielsen has published a list of 10 heuristics which apply to most interactive systems. These heuristics are:

1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation

In addition to (and overlapping) Nielsen's list, there are three general usability criteria that should be met by interactive systems. First, the system should be learnable. New users should be able to learn the complete functionality of the system quickly and efficiently. The second usability principle is flexibility. Users should be able to interact with the system in a variety of ways which are capable of meeting the individual needs of each user. Finally, interactive systems should be robust. Robustness measures the level of support users have in both achieving and assessing their goals.

The Heuristic Evaluation

For the evaluation of the Classroom 2000 notes system, the evaluator will focus on a subset of these guidelines and principles. This subset has been chosen based on previous use of the system and will hopefully uncover the most errors possible within the limited scope of this evaluation. The following heuristics will be used:

• Visibility of system status
• Match between system and the real world
• Recognition rather than recall
• Flexibility and efficiency of use

The task for the evaluation will be to use the class notes system to prepare for an exam in the class. This task would normally involve looking back over written notes from the past few lectures and trying to mentally organize that material. With the Classroom 2000 notes system, a richer combination of media can be used to recall the material from previous classes. Some of the problems with the C2000 interface should be uncovered by this evaluation.

The unranked raw data from the heuristic evaluation is available. The findings were then ranked according to severity and given a value from 0 to 5 based on the rating criteria above. The resulting ranked comments are:

• (5) Two timelines are confusing/redundant
• (5) No clear mapping for blue squares and red squares
• (5) Playing audio does not affect the browser at all
• (4) Slides don't scroll with audio
• (4) No clear feedback on where the audio is in the lecture
• (4) Switching to another set of notes is annoying
• (3) Two scrollbars that don't have the same meaning for moving through the notes
• (3) Trial and error to figure out clicking on strokes in images
• (2) Two frames are confusing
• (2) Wanted to jump to a slide, but still had to scroll the left frame to do it... might as well have scrolled down the right frame
• (2) Markings on audio timeline are good, but units are not obvious (sec,min,hr?)
• (2) Slides with a lot of pen strokes do not work well for playing audio (minor movements of the mouse change the starting time of the audio greatly)
• (1) Postscript versions of slides are good to have, but some text doesn't show up
• (1) Why can't I click on the black part of the second timeline?
• (0) Usage tracking/login screen works well

Conclusions and Suggestions

Based on the results of the heuristic evaluation, some suggestions for improving this online notes system have been generated. The main problem with the system is the synchronization of the media. Using the audio is difficult and marginally useful. It is extremely difficult to follow the audio and the slides simultaneously. If this problem were solved, the system would be much more useful. Another problem with the audio is availability. Current users of the system most likely will be using it in a lab setting where loud audio is not acceptable. The quality of the audio is too low to be able to leave the volume at a low setting and still understand what is being said. With these considerations in mind as well as the results of the heuristic evaluation, the evaluator produced the following list of improvements:

• Add a marker that moves down the left frame as the audio plays
• Change the blue and red squares for slides and web pages to red and blue arrows that point into the functional timeline (i.e. remove the second timeline)
• Add another small frame at the top that allows users to jump to any slide
• Change the login screen to allow the back operation (add a delay to give the user time, or come up with a way to override the browser's default action)
• Perhaps in addition to a marker on the audio, actually redraw the keystrokes on the slide as the audio progresses. In other words, clear off the correct slide, and at the corresponding times of each keystroke in the audio track, redraw that keystroke on the slide.

Further Resources

Gregory Abowd's Guide to Heuristic Evaluations

Back to the Follow Me Page