CS 4753 -- Human Factors in Software Development

Assignment 1: Usability Evaluation

Due November 4

Part 1 -- Questionnaire Design

Objectives

In the course of their professional careers, computer scientists will often be faced with the task of gathering data from future users of the computer systems they are planning to build. These instances of data gathering will grow as the use of computers spreads to a larger audience, as the tasks that computers are applied to becomes more extensive and as the requirements for well designed human-computer interfaces grows. Without training in data gathering techniques, the computer scientist is left in a position of potentially designing questionnaires or conducting interviews that, at best, provide no useful data, and at worst, give erroneous information.

This component of the assignment is designed to give you practice writing, testing and administering a questionnaire to a true user population. Although it does not train you in the very fine points of questionnaire design, it does alert you to the basic problems in obtaining valid responses from humans.

In addition to practice in valid questionnaire design and questionnaire administration, the assignment asks you to focus information about a user interface to a computer system. Its intent is to help you develop probing skills (through good question design). These skills can then be used to find out what failures and successes users are having with a system and even the underlying causes for these successes and failures.

Directions

This part of the assignment will have 7 steps. They are:

Selection of a computer interface/system as your topic area.
Selection of an existing interface to study.
Preparation of a draft questionnaire (1-2 pages).
Piloting of the draft questionnaire.
Preparation of a final questionnaire.
Administration of the final questionnaire.
Analysis of the results.

Description of Study Problem

You are being asked to develop a questionnaire to find out information about a user interface which relates to the interface design you will be asked to develop in the series of course assignments that follow. The first task for you to do is to select the user interface you intend to develop. Once you have selected your design task, you will be in a position of collecting information about potential users of the interface.

One of the methods of collecting this information is to look at existing user interfaces that have things in common with the interface you are designing, i.e. the computer program accomplishes the same or similar tasks, or you believe that the task that the program supports is in many ways similar to the task you will be supporting with your interface design. For example, if you were building a design for an interface which helped users find out which books were available in the school library system, you might look at the existing interface for accomplishing this task.

In selecting the interface you will design in this course, it is important not to make the interface too big or too complicated. Designing a word processor is out of the question. You will also later be developing this interface with a graphical prototyping tool, so choose something that you believe will be tractable to build.

Below are some potential interface designs to examine. Feel free to come up with your own ideas too.

Design of a library book search system for a university.
Design of a new automatic teller interface.
Design of an interface for a VCR to program recording TV shows.
Design of an interface to a system for making plane reservations.
Design of an interface for managing home lighting, both indoor and outdoor.

The type of information you are to obtain about the user interface through the careful design of your questionnaire is the following:

How easy has the system been for them to learn?
What are the particular parts of the system that they are having the most trouble with?
What kinds of recommendations do they have for improving the system?
How useful are the manuals for the system?
How much time are they spending on learning the system?

Unfortunately, the questions above can't be used directly because they won't generate very good answers. Question 1 is too ambiguous. Question 2 is much too broad to get useful answers. Question 3 is too difficult for new users. Question 4 is again ambiguous and users do not have the information to answer question 5. Also, since the amount of difficulty a person has with the system depends on that person's previous experience, whether they are computer science majors, whether they are highly motivated, whether they have a good friend who is helping them out a lot and whether they are very intelligent, questions have to be asked about all these factors as well.

Design a questionnaire to administer to the users of the system of your choice. Administer that questionnaire to about 3 compatriots to determine if they understand the questions in the same way you meant the questions. You do this by giving them the questionnaire to fill in and then asking them what their answers mean and what they thought your question meant. (This is called pilot testing the questionnaire.) Once you have received feedback from your trial respondents, use this to redesign your questionnaire. If the design changes drastically, it is a good idea to test your questionnaire again on a couple additional friends.

When you feel your questionnaire has been tested enough and will work on the targeted set of users, find 4 or 5 users who fit the eligibility requirements for your survey. Ask these users to fill out one of your questionnaires. (We wouldn't use such a low number of respondents in a real study, but it's OK here.)

Summarize the data collected from your questionnaires. The structured question answers are usually presented as percentages, e.g., 25 percent responded ``strongly disagree'' to the question, ``Should the system always have menus available?'' Often the percentages are presented across demographic data, e.g., ``30 percent of the women and 35 percent of the men would like to have less commands to learn.'' A clear way to present this information is in tables.

Use the data results of your questionnaire to suggest changes that might be made to the user interface to make it easier for users to learn and use the system. These can be changes in manuals and training as well as detailed changes to the interface commands and the documentation.

Part 2 -- Observing Users

Objectives

The objective of this component of the assignment is to give you practice in evaluating human-computer interfaces. This practice is intended to help you develop evaluation criteria appropriate to particular interfaces you may be designing. The evaluation, in turn, can guide future design decisions.

This part has three sub-objectives. The first of these is to give you practice in the detailed observation of users at work with interfaces. Analysis of the user behavior is intended to give you skills in observing humans at the micro-level needed for understanding user problems.

The second sub-objective is to alert you to very basic differences in user behavior and to the multiple ways users will interpret information presented to them and generate what is to them, very intelligent guesses about what is needed to use the interface.

Once of the basic assumptions that humans have about their counterparts is that we all think alike. This is far from the truth, especially with the differences in skilled training that occur in today's society.

The third sub-objective is to give you practice in distinguishing between the functionality of a computer system and its usability. The computer system may do all the tasks that the user needs to perform with the system, but the user may be required to organize the steps to perform these tasks in an order which is extremely foreign to the way the user thinks about the tasks.

Note: This part of the assignment is focused only on the usability analysis of the interface not on the functions that might be needed for the new design. It is assumed that this functionality was captured in the first part.

Directions

This portion of the assignment will have 3 steps. They are:

Develop at least 3 criteria for evaluating your existing analogous interface.
Conduct the evaluation of the interface.
Analyze the results of the evaluation.

Hints for Selecting Evaluation Criteria

Below are a series of suggested evaluation mechanisms that you might consider in performing this assignment. The list is far from complete and although you are welcome to use these hints, it is important to develop your own criteria as well.

LEARNING:

How easy is the analogous interface to learn?
Do the commands have a structure or organization that makes learning easier?
Do the commands relate to English words which describe the same task?
Are the commands discriminable, e.g., are they whole words or single letter commands?

TASK USABILITY:

Are various functions of the interface easily done by the users or does a simple task take long and involved steps?
Are there functions that users would never use because they are infrequent and need to be looked up each time?
Can the user build in shortcuts once the interface is learned?
Are there functions that do not match any task a user might wish to do?

DOCUMENTATION:

How easy is it to read the documentation? Is the manual so long and cumbersome that a user will never take the time to read it?
Is it easy to find the commands to perform a particular task or do you have to reference many different places in the manual?
Does the manual use words which have a different meaning for the user?
Does the interface provide reminders of what to do next so that the manual does not have to be constantly referred to?
Is the on screen help slow and cumbersome? Does it help?

OVERALL USABILITY:

Is the interface cluttered with too much detail so that it is difficult for the user to see what is going on?
Does the interface provide feedback to the user indicating that a command has been received and is now being executed?
Can the user get into situations in using the interface that allow no easy exit?

Hints for Implementing Evaluation Criteria

In addition to setting up the evaluation criteria, you must use it. Here are some hints for implementing the first LEARNING item into evaluation tests.

Set up a test to see how long it takes to learn part of the user interface:
Build a set of tasks, e.g., in searching for books in the card catalog system, it might be handwritten notes on titles, authors and subjects to look at in researching a course paper. You would learn a subset of the system's commands. Do this by mentally rehearsing the command set. Time yourself. When you think you have mastered the command set, look at your set of handwritten notes and write down beside each one, the set of commands you would do to perform the search. Try to do the same thing the next day.
Have one of your colleagues from class learn an equivalent set of commands for the another tool. The colleague should perform the same tests on the same set of commands. Compare the times and errors.
Measure how many things there are to learn:
List a set of basic tasks that your analogous interface must do for anyone beginning to use it. Count the number of commands you will need to learn to perform these tasks.
For a more detailed measurement, add 10 to your count every time a different format for the command needs to be learned.
Measure how easy the commands are for novices to learn:
Teach a friend who knows nothing about computing two of the interface commands. Ask them to explain how the commands work after you feel that you have adequately described the command. Ask them to describe as many situations as they can in which they believe the commands can be used. Write down any conceptual errors they make in expressing their understanding of the commands.

What to Turn In

The results of your questionnaire and usability examination must be written up and submitted by the specified due date. Only one report should be turned in per team. Within the team, you must negotiate how much work and which tasks the different individuals will be responsible for. The grade you receive will apply to all team members. The members will be polled, however, for their opinions on how much each individual has contributed to the team. We reserve the right to adjust individual scores with respect to this feedback.

You will be using HTML and the WWW to write up your assignment and submit it to us. Your write-up should be the equivalent of about 10-12 pages when printed out in hard-copy. DO NOT make your write-up much longer than this. On the specified due date, turn in a hard-copy of the report and also a path to a readable version of the file(s) involved in the report. We will copy these files over to the class file space for safe keeping.

With respect to the first part you should include:

A write-up on the results of the survey with the final portion recommending any changes to be made to the studied user interface. The write-up should include a discussion of the reasons for each question or set of questions in your questionnaire. The write-up should also contain a brief description of your target interface/system and the existing system you chose to study.
The draft questionnaire.
The final questionnaire.
A section describing why you changed any questions on the final questionnaire.

In regard to the second part of the assignment, you should include:

A discussion of the evaluation criteria. The criteria should be related to how the interface is used or perceived, e.g., one obvious use of the card catalog system is to search for books.
A discussion of the tests you took to perform the comparisons, e.g., you could calculate how long it took each of the subjects to perform a typical task in the interface.
A report on the evaluation conducted. What was good about the design of the interface and what functions appear to cause great difficulties. Your interface evaluation should be tied to the information you gathered not just to your intuitive impression of the interface. You need justification for your critique.
Recommendations for your own design based on the evaluation. It may be that your analogous interface performed superbly. In this case, you should say so and indicate what good features you should bring over to your design.

Items such as the actual questionnaires, figures, references and materials of that type can be included in an appendix, and need not be counted against your page total.

What you will be evaluated on

You will be judged on both the content of your submission and its presentation. In terms of content, you will be judged on creativity, thoroughness, adhering to good human-computer design principles, etc. We will judge how good your questions were, and whether they were well planned to elicit the type of information you sought. We will evaluate whether the evaluation and tasks you ran were appropriate. The success or failure of your experiments don't really matter (whether the interface was ``good'' or ``bad.'') What matters is if your evaluation process was well-designed to uncover the type of information you were seeking.

In terms of presentation, you will be judged on quality of writing, expressiveness, and grammatical correctness. It is important not only to conduct the right types of usability evaluations, but also to do a good job of communicating your techniques and methods.