CS 6751 -- Human-Computer Interface

Assignment 1: Usability Evaluation

Due January 31

Throughout this term you will be involved in a group project focusing on interface evaluation and design. You will be choosing some computer application to develop a new (and improved!) user interface for. One way to assist with that task is to study and evaluate an existing interface/system for this application area, which is the first assignment of three making up the project. You will be able to choose the application and the existing system, so you can make a choice that is interesting to you. This first evaluation assignment will give you practice with summative evaluation, and is itself made up of three parts which are described further below.

Part 1 -- Choosing an Application

The first part of the project allows you to choose the computer application or domain for which you will be developing an interface. You also will choose an existing interface/system in this domain to evaluate. Be careful not to choose something too complex (e.g, a word processor) or something too rigid (e.g., an ATM machine). Try to choose an application that is interesting or innovative, and one that you will be able to adequately redesign and simulate in a later assignment. Applications such as an electronic whiteboard, an information/directions kiosk, or an on-line class registration system would appear to be good. Avoid computer games--they are difficult to redesign.

Part 2 -- Questionnaire Design

In order to design a new user interface for the domain/application you have chosen, you need to find out about its users, its capabilities, its current strengths and weaknesses, and its most essential attributes. One way to do that is by surveying current users of an existing system/interface in the domain.

The type of information that you are to obtain about the user interface through the careful design of a questionnaire could include the following:

Unfortunately, the questions above can't be used directly because they won't generate very good answers. Question 1 is too ambiguous. Question 2 is much too broad to get useful answers. Question 3 is too difficult for new users. Question 4 is again ambiguous and users do not have the information to answer question 5. Also, since the amount of difficulty a person has with the system depends on that person's previous experience, whether they are computer science majors, whether they are highly motivated, whether they have a good friend who is helping them out a lot and whether they are very intelligent, questions have to be asked about all these factors as well.

To address this part of the assignment, design a questionnaire to administer to the users of the system of your choice. Administer that questionnaire to about 3 compatriots to determine if they understand the questions in the same way you meant the questions. You do this by giving them the questionnaire to fill in and then asking them what their answers mean and what they thought your question meant. (This is called pilot testing the questionnaire.) Once you have received feedback from your trial respondents, use this to redesign your questionnaire. If the design changes drastically, it is a good idea to test your questionnaire again on a couple additional friends.

When you feel your questionnaire has been tested enough and will work on the targeted set of users, find 4 or 5 users who fit the eligibility requirements for your survey. Ask these users to fill out one of your questionnaires. (We wouldn't use such a low number of respondents in a real study, but it's OK here.)

Summarize the data collected from your questionnaires. The structured question answers are usually presented as percentages, e.g., 25 percent responded "strongly disagree" to the question, "Should the system always have menus available?" Often the percentages are presented across demographic data, e.g., "30 percent of the women and 35 percent of the men would like to have less commands to learn." A clear way to present this information is in tables, charts or graphs. We strongly encourage you to do that.

Use the data results of your questionnaire to suggest changes that might be made to the user interface to make it easier for users to learn and use the system. These can be changes in manuals and training as well as detailed changes to the interface commands and the documentation.

Part 3 -- Observing Users

This part of the assignment is intended to give you practice evaluating interfaces and observing users. The previous part focused on subjective (questionnaire) data, but this part focuses on objective data. You should design appropriate benchmark tasks that will help you assess the interface relative to evaluation criteria such as learnability, performance, errors made, retainability, etc. You are seeking answers to questions such as You should assess at least three different criteria by conducting a number of benchmark tasks. For example, if your chosen application is a calendar management system, you could have participants schedule appointments, change appointments, seek help about a particular command, and so on. Be sure to do two things:
  1. Gather objective, quantitative data about the tasks such as time to complete operations, number of errors made, number and identity of commands used, etc. In a later assignment you must compare this data to that taken from a new system.
  2. Qualitatively study users performing tasks to further evaluate the existing interface and identify its strengths and weaknesses.

What to Turn In

The results of your application selection, questionnaire and usability examination must be written up and submitted by the specified due date. Only one report should be turned in per team. Within the team, you must negotiate how much work and which tasks the different individuals will be responsible for. The grade you receive will apply to all team members. The members will be polled, however, for their opinions on how much each individual has contributed to the team. We reserve the right to adjust individual scores with respect to this feedback.

You will be using HTML and the WWW to write up your assignment and submit it to us. Your write-up should be about 10 pages when printed out in hard-copy (Simply print your HTML file, no need to do another report). DO NOT make your write-up much longer than this. On the specified due date, turn in a hard-copy of the report and also a path to a readable version of the file(s) involved in the report. We will copy these files over to the class file space for safe keeping.

The form and clarity of your report are extremely important. Consider the professor and TA as potential business clients and you are "selling" your plan and evaluation. A strong report should include

Items such as references, the actual questionnaires, task description forms and materials of that type can be included in an appendix, and need not be counted against your page total. You need not include completed questionnaires.

What you will be evaluated on

You will be judged on both the content of your submission and its presentation. In terms of content, you will be judged on creativity, thoroughness, adhering to good human-computer design principles, etc. We will judge how good your questions were, and whether they were well planned to elicit the type of information you sought. We will evaluate whether the evaluation and tasks you ran were appropriate. The success or failure of your experiments don't really matter (whether the interface was "good" or "bad.") What matters is if your evaluation process was well-designed to uncover the type of information you were seeking.

In terms of presentation, you will be judged on quality of writing, expressiveness, and grammatical correctness. It is important not only to conduct the right types of usability evaluations, but also to do a good job of communicating your techniques and methods.