The objective of this assignment is to give you practice in and have you understand some of the procedures and issues in conducting a usability study with trial users and benchmark tasks. You will design a study of a website and have some sample users participate in the study. You will create all the appropriate materials and will come to some conclusions about the usability and design of the website based on what you observe.

In this assignment, you are to evaluate and compare two websites by observing trial users completing benchmark tasks using the sites. The two sites that you will compare are both airlines: Delta Airlines and American Airlines (the same airlines whose telephone-based systems you evaluated recently, and whose websites your looked at briefly earlier in the course).

Design your study: Obviously you cannot analyze the whole website, nor can you study all the actions that all types of users might perform. Decide on a category of users, define some tasks that they would do frequently, and choose four benchmark tasks to help you compare the two sites on those tasks. Carefully select these tasks to simulate what actual users of the site would do.

Once you have your tasks, design the experiment: decide on how the participants will be grouped (within or between?), the procedure, informed consent, debriefing, etc.

Demographics: You must design a pre-experiment questionnaire to give to each participant. It should gather appropriate background information that you will find useful in analyzing the performance data later.

Gather data: You must recruit four colleagues, friends, relatives, etc., to be participants in your study. Do not use other students in the class. Also, the participants should not have participated in the same project for another person in the course (either section). These participants will need to read and sign the following Informed Consent [.doc, 32k]. Then gather your data in whatever manner is appropriate, given your design, your hypotheses, etc. You should include subjective and objective measures.

You will conduct an experimental session with each of your four participants in which you deploy the appropriate materials and have the participants perform the benchmark tasks. Carefully observe each session as well because you should take notes about the participants' interactions with the site. Which tasks were performed successfully? How long did they take? Did participants make errors? What problems occurred? Did the participants have a conceptual model of the site? Was it correct?

Post-task questionnaire: You must create and administer a written questionnaire (4 or 5 questions) to be given to the participants after the tasks are completed. This questionnaire should gather some more subjective data, and should contain quantifiable inquiries. Finally, plan a few open-format interview questions to ask each participant at the end of the session. These should elicit more overall, qualitative impressions of the website.

Clearly, you should interact with the site yourself ahead of time to become familiar with its functionality, including functionality outside of what your task includes (users may get off course, lost, distracted, etc., and you'll need to be prepared to help them recover).

Analyze your data and make conclusions: Inspect your data and determine if/how they address your hypotheses. Look at descriptive statistics primarily; if you are able to, you should also look at appropriate inferential statistics (e.g., t-test). NOTE: inferential stats will not be required in this assignment, simply because of the small number of participants. However, they are very important in real-world analyses, as well as in your Group Project Part 4. In reporting your findings in this homework, you can assume that observed differences are real (i.e., that if you had had many more participants, the results would have yielded statistically significant tests).

What to turn in: You should turn in all the materials you designed and created, ie, the pre-questionnaire, the benchmark tasks, the post-questionnaire, and the interview questions. Also indicate why you made the decisions you made. You should also write a relatvely brief usability study of the site. Include and report on the quantitative data you collected, both objective and subjective. Furthermore, provide some more reflective analysis of the site. What aspects are designed well? What problems exist? Put yourself in the situation of being a person hired by the company to do an evaluation of the site. Report back to me as you would do so to the company. Remember, all of this exercise is intended to help make the Web sites better, basing any decisions on a variety of evidence (data sources).

Evaluation: We will evaluate you on the quality and appropriateness of the materials you design as well as your overall analysis of the usability of the site. Good tasks have users perform representative operations on a site and expose important issues. Insightful questions uncover the kind of data than an evaluator seeks. A clear, well-prepared report summarizes the quantitative data gathered, draws some conclusions from it, and provides a high-level evaluation of the overall design and usability of the web site.