CS6751 : Topic Report
Heuristic Evaluation

Introduction

There are basically four ways to evaluate an user interface : Formally by some analysis technique, automatically by a computerized procedure, emperically by experiments with test users, and heuristically by simply looking at the interface and passing judgement according to ones own opinion. Formal analysis models have not yet reached the stage where they can be generally applied in real software development projects. Automatic evaluation is completely infeasible except for a few primitive checks. Therefore, current practice is to do emperical evaluation if one wants a good and thorough evaluation of a user interface. Unfortunately, in most practical situations, people actually do not conduct empirical evaluations because they lack the time, expertise, inclination, or simply the tradition to do so.

As its name suggests, heuristic evaluation is not a pure analysis technique. It might be thought of as "analysis by a team of analysists using a variety of informed models". A more descriptive term applied to heuristic evaluation is Usability Inspection : an inspection is carried out and a list of problems affecting usability is drawn up. For its effect, the heuristic evaluation method relies on two techniques in combination. First, it employs a team of evaluators rather than relying on one person to carry out the evaluation. Second, a set of design heuristics is used to guide the evaluation.

Usability Heuristics

Molich and Nielsen [1990a] have listed nine heuristics which can be used to generate ideas while critiquing the system. These nine principles seem to be well suited as the basis for practical heuristic evaluation. These nine principles correspond more or less to principles that are generally recognized in the user interface community and almost all usability problems fit well into one of these categories.

• Simple and natural design. Dialogs should not contain irrelevant or rarely needed information. Every extraneous unit of information in a dialog competes with the relevant units of information and diminishes their relative visibility. All information should appear in a natural and logical manner.
  
  
• Speak the users language. The dialog should be expressed in words, phrases, and concepts familiar to the user rather than in system oriented terms.
  
  
• Minimize the user's memory load. The user's short term memory is limited. The user should not have to remember information from one part of the dialog to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. Complicated instructions should be simplified.
  
  
• Be consistent. Users, should not have to wonder whether different words, situations, or actions mean the same thing. A particular system action when appropriate - should always be achievable by one particular user action. Consistency also means coordination between sub systems and between major independent systems with common user population.
  
  
• Provide feedback. The system should always keep the user informed about what is going on by providing him or her with appropriate feedback within reasonable time.
  
  
• Provide clearly marked exits. A system should never capture users in situations that have no visible escapes. Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialog.
  
  
• Provide shortcuts. The features that make a system easy to learn such as verbose dialogs and few entry fields on each display are often cumbersome to experienced users. Clever shortcuts, unseen by the novice user, may often be included in a system such that the system caters to both inexperienced and experienced users.
  
  
• Provide good error messages. Good error messages are defensive, precise, and constructive. Defensive error messages blame the problem on system deficiencies and never criticize the user. Precise error messages provide the user with exact information about the cause of the problem. Constructive error messages provide meaningful suggestions to the user about what to do next.
  
  
• Error prevention. Even better than good messages is a careful design that prevents a problem from occuring in the first place.
  
  

• Visibility of system status.
• Match between system and the real world.
• User control and freedom.
• Consistency and standards.
• Error prevention.
• Recognition rather than recall.
• Flexibility and efficiency of use.
• Aesthetic and minimalist design.
• Help users recognize, diagnose, and recover from errors.
• Help and documentation.

Team of evaluators

In order to test the practical applicability of heuristic evaluation, Nielsen and Molich [1990b], conducted four experiments where a number of evaluators were presented with an interface design and asked to comment on it. They found that individual evaluators were mostly bad at doing heuristic evaluations and that they only found between 20% and 51% of the usability problems in the interfaces they evaluated. On the other hand, they found that the overall result can can be dramatically improved by forming aggregates of evaluators since the "collected wisdom" of several evaluators is not just equal to that of the best evaluator in the group. Aggregates of evaluators are formed by having several evaluators conduct a heuristic evaluation and then collecting the usability problems found by each of them to form a larger set.

They also found that three to five evaluators in an aggregate would be able to detect about two third of the usability problems. The result can be seen in figure 1.

Figure 1 : Curve showing the proportion of usability problems in an interface found by heuristic evaluation using various numbers of evaluators. The curve represents the average of six case studies of heuristic evaluation.

They also found that in order to produce better results than the individual evaluatons, the evaluators should do their evaluations independently of each other and only compare results after each of them has looked at the design and written his/her evaluation report. This is so that the evaluators don't bias each other towards a certain way of approaching the analysis and therefore only discover certain usability problems.

In another study carried out by Nielsen[1992], he found out that usability specialists were much better than those without usability expertise at finding usablilty problems by heurisitic evaluation. Furthermore, usability specialists with expertise in the specific kind of interface being evaluated did much better than regular usability specialists without such expertise, especially with regard to certain usability problems that were unique to that kind of interface.

Figure 2 : Average proportion of usability problems found as a function of number of evaluators in a group performing the heuristic evaluation

In figure 2, `Novice evaluators' refer to evaluators with no usability expertise, "regular specialists" refer to usability specialists, and "double specialists" refer to usability specialists who also have experience with the particular kind of interface being evaluated. As can be seen from figure 2, the double specialists found significantly more usability problems than did the regular usability specialists. If double specialists are used a smaller group size can be used to evaluate the interface. A size of two to three has been recommended.

He also found that major usability problems have higher probability of being found than minor problems by heuristic evaluation. Problems with the lack of clearly marked exits are harder to find than problems violating the other heuristics, and additional efforts should be taken to identify these usability problems.

Conclusions

A number of advantages are claimed for heuristic evaluation. They include