A Usability Evaluation of Our Redesigned Banner Registration System:
A 6751 Group Project

by

Zongming Fei
Chad Jenkins
Harish Kotbagi

Table of Contents:


Introduction

 The Banner Student Registration System, currently in place at Georgia Tech, provides an essential means of accessing the registration system, via the World Wide Web. Banner is an viable system that provides students with a plethora of registration services. However, the system has several quirks that are less than pleasurable to the humans that interact with it. Among these shortcomings are CRN's, user login, and lackluster navigational abilities. Consequently, the driving 1 behind our redesign of the Banner System centered around removing or diluting these shortcomings. This will allow the full potential of Banner to be realized by the Georgia Tech community. While this is a well-intentioned goal, does the redesigned Banner fulfull this purpose? If so, does the system live up to the promises held by the rationale for the redesign?

Brief Summary of the Redesigned Banner System

The redesigned Banner system is an evolution of the current Banner system at place at Georgia Tech. The current Banner system allows students to perform their registration and administration functions through the World Wide Web. The details of this system are discussed in our previous usability evaluation. The redesigned Banner system is very similar to its predecessor in structure. This structure is such that the user login is followed by a main menu consisting of two submenus. The two submenu contain all of the system's registration and administration functions. The registration and administration functions are differentiated by the time frame a function is used for. More specifically, registration functions are used only for a specific quarter, while administrative functions are quarter independent. The redesigned Banner system, however, has a number of differences from its predecessor that are considered necessary to improve the users interaction with the system. These changes are centered around the general philosophy discussed in the Introduction. During the course of this study, the changes made for the redesigned Banner will be discussed in greater detail.

User Session Methodology

The application of the survey was performed similarly to our methodology for the previous study of the existing Banner system. The only difference was in the demographics of the two user groups we used for this study. For this study, we differentiated user into two groups by providing the users of one information regarding reasons for improving the system and short descriptions of new features. This information was provided to these users in order to determine if the users of the redesigned Banner will take advantage of its improved features.

Changes in the Task List and Questionnaire

Several changes were made to the user questionnaire used for the existing Banner system. The changes were meant to be substantial enough to gauge the user's attitudes about the new feature of the system, yet still maintain enough similarity to the previous questionnaire for comparisons between the two. For the most part, the questions were left the same or slightly reworded for clarity. The current questionnaire removes Questions #3 and #11 from the previous study questionnaire. Question #3 inquires about the user's gender, which had little or no impact on the results from the previous survey. Question #11 is regarding the of the on-line manual, with on-line manual being the set of notes provided to the user before logging into the Banner system. The redesigned Banner has no such facility because the information is not necessary or is already covered in the help features in the system. For any questions that have been added or reworded, an explanation is provided in the study questionnaire. The bulk of the task list was retained from the previous study task list. The only change was the elimination of Task #10, relating to the users ability to update their forwarding address. This task was eliminated because of the inability of the redesigned Banner to handle this feature. Due to the similarity of the task lists for our usability studies, the motivation for the tasks are explained by the task list for the existing Banner system.

Task List Observational Data Results


Average time (seconds) for performing each task:

Task # Old Value Worst Value Best Value Expected Value New Value
1: Access to the system. 19 50 6 15 28.0
2: Login. 45 80 5 15 15.5
3: View schedule. 24.6 60 5 15 14.8
4: View transcript. 48.8 80 5 20 21.6
5: Check holds. 31 70 5 15 14.3
6: View registration status. 24 60 5 20 21.3
7: Add a class. 23 80 5 20 47.8
8: View address. 16.6 50 10 15 23.1
9: View accounting. 16 60 6 15 11.5
10: Exit system 11 30 4 8 5.6



The average number of mouse clicks for users performing each task given:

Task # Old Value Worst Value Best Value Expected Value New Value
1: Access to the system. 5 15 1 2 1.3
2: Login. 5 15 1 2 2.0
3: View schedule. 3.2 15 2 3 3.1
4: View transcript. 12.4 20 3 5 4.1
5: Check holds. 6.2 20 2 3 2.1
6: View registration status. 5 15 2 3 4.0
7: Add a class. 3.4 10 2 3 6.0
8: View address. 5.6 10 2 3 4.3
9: View accounting. 5.6 10 2 3 3.3
10: Exit system 1.6 8 1 1.5 1.6



Average user ranked difficulty for each task (1 indicates very difficult, 5 indicates very easy) :

Task # Old Value Worst Value Best Value Expected Value New Value
1: Access to the system. 4.8 1.0 5.0 4.8 4.8
2: Login. 4.2 1.0 5.0 4.5 4.6
3: View schedule. 4.8 1.0 5.0 4.8 4.1
4: View transcript. 3 1.0 5.0 4.0 4.3
5: Check holds. 3.6 1.0 5.0 4.5 5.0
6: View registration status. 4.2 1.0 5.0 4.5 4.0
7: Add a class. 4.4 1.0 5.0 4.5 4.3
8: View address. 4.6 1.0 5.0 4.8 4.6
9: VIew accounting. 4 1.0 5.0 4.5 4.8
10: Exit system 5 1.0 5.0 5.0 4.3


Questionnaire Data Results


User satisfaction of the system (5 indicates best and 1 indicates the worst. )

Questions Old Value Worst Value Best Value Expected Value New Value
5a New system is better than old one. 3.4 1.0 5.0 4.0 4.4
8 The options are well-structured. 3.0 1.0 5.0 4.0 3.9
9 Easy to correct errors. 3.2 1.0 5.0 3.5 3.4
10 The help are useful. 3.2 1.0 5.0 3.5 3.5
12 Screen layout is appealing. 3.4 1.0 5.0 4.0 4.0
13 Overall impression of the system. 3.0 1.0 5.0 3.5 3.9
14 Easy to access and enter. 2.2 1.0 5.0 4.0 4.2
15 Course number is better. 1.8 1.0 5.0 4.0 5.0

Conclusions and Analysis

In order to find the exact problems of the existing system, we evaluated the system's learnability, retainability and overall usability. Learnability is an important criterion as it allows us to measure the time taken by different people to learn the system. It provides with a measure of the "user friendliness" of the system.

High retainability is also important for this system because the user typically uses the system once a quarter for enrollment. If the user has to learn to use the system every time s/he accesses it, the design is not a good one. As the system performs several tasks, it should be also be effective and usable. Usability enables us to measure how easy, effective and capable the system is for use.

Learnability

Determining the learnability of the interface was important to our evaluation primarily because we wanted to find out how the system is first perceived and how well the user can determine the functionality and overall interaction. We included six participants in our survey. The profile of the subjects is presented below :

4 CS Majors, 2 Science Majors, 3 Master's candidates, 3 PhD candidates.

The time taken by people on this exercise ranged from around 90 to 360. The major difference in time data was due to the varied times required by subjects to Add/Remove classes. Most subjects were unhappy about this particular aspect of the system. Moreover, we found that the time taken reflected to a certain extent the academic background of the subjects. Students majoring in CS required less time than the Science majors.

The timings taken by the different subjects was also largely determined by whether they briefed about the system prior to the tasks. We found that subjects with prior knowledge of the system took much less time to complete the tasks than those who weren't briefed. This is because subjects who aren't briefed, expect the sytem to behave like the older one. One of the subjects didn't use the back button as it is prohibited in the older system where it results in a relogin. No communication was of this new feature.

The maximum amount of time taken by the subjects was to Add/Remove classes from the list. The subjects also took considerable time to view the current local address, because they had to navigate through more pages to get there. Some of the subjects took more time to view the time limits in which they are allowed to register because the field descriptor (Registration Status) was not very intuitive.

The number of mouse clicks gave us a measure of the "depth" at which the options could be found. There seems to be a direct correlation between the time taken and the number of mouse clicks in most of the tasks. The first task of logging into the system was an exception as the time required to execute the task is dependent on the speed of the network and not on the design per se.

The questionnaire reveals some insightful results. We asked a few questions initially to find out the profile of our subjects. This allowed us to analyze the dependence of performance on subject profile. As expectec, subjects with more computing background, completed the tasks faster than those with little background in computing. This was also reflected by theie usage of web browsers.

We found from our study that subjects are strongly influenced by any similar systems that they have previously used. In such cases, the subjects expect to find similar fucntionality as also the same relative locations of functions.

All subjects agreed that the newer registration system was an improvement over the older one. This shows that most of the changes made to the system really eased the execution of the tasks outlined. The system exhibited good learnability as the subjects had used the existing BANNER system before. Thus, learnability is highly correlated with prior experience.

Retainability

Some of the tasks were put in specifically so that we could get feedback on retainability. Subjects who had not been briefed took a lot of time in carrying out the task of viewing their current address because they didn't remember the locations of the options relative to the current screen. As a result they didn't know the best way to navigate to perform the desired task.

The subjects found logging into the sytem easier as they didn't have to remember digital passwords. This shows that the retention of words is easier than numbers, which was a strong case for replacing the older banner system. Carrying this further, the course registration numbers (CRN), were removed in favour of using course numbers for registration. However, this change didn't reflect in better performance as the integration of Add/Remove/Update classes resulted in making this option more difficult to use. This shows that having too much functioanlity in one page is not particularly desirable, as it requires the subject to "learn" the sytem.

Tasks which were designed to test the retainability of the subjects, showed that they took longer time for tasks that asked for retention.

Usability

Usability testing gives us a measure of how easy the system was to use. The results of the tasks show that although there is an improvement in overall system performance, there is scope for further improvement. The graphs for the time taken and the number of mouse clicks show that our design has done much better than the BANNER system currently in use.

The usability tests showed that the subjects took lesser time to perform the tasks when compared to the older system. Although there was an overall improvement in performance, the performance was worse while trying to Add/Remove classes. This shows that putting in a lot of functionality on a single page, impedes the perform of the subject rather than aid him.

The time taken to access the system was greater mainly due to the slower speed of the network. Another reason could also be the fact that the URL was not easy to remember. The elimination of digital passwords, lead to a faster login. The time taken to view the schedule was significantly lesser than with the older system. This was because the Main Menu of the program detailed the options available under each menu. The time to View Holds and Registration status were vastly improved as the options under the submenus displayed the place where they could be found. There is a slight increase in the timing from task 3 to task 4, as they fall under different menus (Registration and Administrative). The graphs clearly show a strong correlation between the number of pages navigated and the time taken, with tasks separated by more pages, taking greater time. The number of mouse clicks also show a direct dependence on the number of pages navigated, as expected .

The questionnaire survey showed that the subjects were happy with the new design (Survey results). Note that in the graph, 1 indicates difficult and 5 indicates easy. The subjects didn't need to look at the online help as the questions were self explanatory. Hence the uesfulness of the online help couldn't be evaluated.

Conclusion

The feedback from the subjects strongly indicated that they wanted the login and password to be the same as their OIT account. The elimination of digital passwords helped in reducing the retainability necessary on the subject's part.

Feedback from the subjects included a lot of suggestions:

Security: introduce timeout

Activate use of Return at the login screen

Rename Registration status as Time ticket

Prompt the user before making any changes

Reference printout (calendar printout)

Fee payment receipt screen

Have links from the Main menu

Have direct links between other menu items

Although most of the suggestions can be incorporated, it is not possible to have links from the Main menu as it is a limitation of the underlying HTML/CGI interface. The timings can be further improved by keeping links in each page that connect to all other menu items.

User awareness of the sytem was found to be crucial in making them more efficeint in their use of the system. For instance, people who were briefed about the system, used the back button on the browser effectively. The most important change that needs to be made is to the Add/ Remove Classes option. It is better to keep the two distinct as this will lead to less cluttering of the page and good readability. The replacement of CRNs with course numbers met with consent from the subjects - this was a strong suggestion made during our pilot interviews.

The usability study showed that the performance of the subjects depended strongly on any previous experiences with similar systems that they have. The subjects expected behavior identical to any similar systems that they have used. The total time taken to perform all the tasks is also dependent on the ordering of the tasks. Hence, these timings could be different if the order of the tasks were changed.

Ths study showed that the greatest benefit can be had from a new design by making it have some similarity with any existing systems, so that subjects with prior experience have better learnability. This was one of the main reasons for the design being better.

Appendix A: Survey Task List

Thank you for volunteering to participate in our study. We greatly appreciate the time and effort you are giving to us. The purpose of this study is to evaluate the usability of a revision of Banner, the World-Wide Web based registration system currently in place at Georgia Tech. Before you continue, let us stress that participation in this study is completely voluntary. Please, FEEL FREE TO STOP AT ANY TIME if you feel uncomfortable or just do not want to continue. With that said, we will ask you to perform various tasks that simulate an actual user's interaction with system. Attempt, to the best of your ability, each task one at a time and in the order that they are given. If you are stuck on any of the tasks presented, just let us know that you are stuck and going on to the next task. If you have any question, please, do not hesitate to ask one of us, although we may not be able to give you a complete answer to you question. Remember, we are evaluating the performance of the registration system, not your performance.

Below are ten tasks that we would like for you to perform. After you have attempted each task, please indicate the ease of task by circling one of the numbers associated with the task, with 1 indicating very difficult and 5 indicating very easy. If you have any comments about the registration system, feel free to speak your mind. We are very interested in your thoughts about the system. After you have completed these tasks, we have a few exit questions to ask you. Again, thank you for participating in our study.

1. Using the WWW browser provided, access to the Banner Registration system located at http://it.cc.gatech.edu:3455/face.html 

 
       1     2     3     4     5


2. Login into the registration system using (valid ID: 123456789) as you student ID and (valid PIN: abc) as your PIN number or the login information provided. 

 
       1     2     3     4     5


3. View your schedule for the Spring 1997 quarter by day and time. 

 
       1     2     3     4     5


4. View your current transcript. 

 
       1     2     3     4     5


5. Check for any holds that may prohibit you from registering. 

 
       1     2     3     4     5


6. View the times when you are allowed to register. 

 
       1     2     3     4     5


7. Attempt to add a class to your schedule. 

 
       1     2     3     4     5


8. View your current local address without updating the address. 

 
       1     2     3     4     5


9. View a summary of your accounting information. 

 
       1     2     3     4     5


10. Leave the registration system. 

 
       1     2     3     4     5



Appendix B: Survey Questionnaire and Motivations

The purpose of this questionnaire is to gather opinions of users regarding their experiences with the redesigned BANNER registration system.

1. What is your major ?

Motivation for Question 1: This question was taken directly from the previous questionnaire Question 1. Since all of our users were Georgia Tech students, there was no reason for asking the question "Are you a Georgia Tech student".

2. What is your academic standing ?

Motivation for Question 2: This question was taken directly from the previous questionnaire Question 2.

3. Have you previously participated in a study similar to this.

Motivation for Question 3: This question was used to determined if the user had either taken part in our study of the existing Banner system or another similar study. Users experienced with such surveys may have advantages and insights that impact the resulting user session data.

4. How often do you use a web browser ?

Motivation for Question 4: This question was taken directly from the previous questionnaire Question 4.

5. Have you used the older registration system (OSCAR) ?

5a. If you answered "Yes" for Question 5, do you agree with the

statement: "The redesigned Banner system is an improvement over the OSCAR registration system." Motivations for Questions 5 and 5a: These questions were based on Questions 5 and 6 from the previous survey. Question 5 from the previous survey is used as a comparison with Question 5a. The comparison indirectly measures how Banner and the redesigned Banner compare with respect to the old OSCAR system.

6. Have you used the Banner registration system currently used by the students of Georgia Tech ?

6a. If you answered "Yes" for Question 6, do you agree with the statement: "The new redesigned Banner registration system is an improvement over the current Banner system".

Motivations for Questions 6 and 6a: These questions are a direct comparison of Banner and the redesigned Banner.

7. Have you used a web based registration system previously, other than Banner ?

7a. If you answered "Yes" to question 7, how would you rate the redesigned Banner against the other Web system ?

Motivations for Questions 5 and 5a: These questions were based on Questions 7 and 8 from the previous survey. Question 8 from the previous survey is used as a comparison with Question 5a. The comparison indirectly measures how Banner and the redesigned Banner compare with respect to another WWW-based registration system.

8. The options in the redesigned Banner registration system are well structured and intuitive.

Motivation for Question 8: This question was taken directly from the previous questionnaire Question 9.

9. The system provides a means to easily correct all errors.

Motivation for Question 9: This question was based on the previous questionnaire Question 10. This question was reworded from "It is easy to correct an error." to give clarify the meaning of the question.

10. The help features provided by the system are useful.

Motivation for Question 9: This question was based on the previous questionnaire Question 12. This question was reworded from "The context sensitive help is useful." because the term "context sensitive" may be unknown to the participant.

11. The screen layout is appealing.

Motivation for Question 11: This question was taken directly from the previous questionnaire Question 13.

12. What is your overall impression of the system ?

Motivation for Question 12: This question was based on the previous questionnaire Question 14. The responses allowed for this question were expanded from 3 to 5.

13. The system is easy to access and enter.

Motivation for Question 13: This question was based on the previous questionnaire Question 15. This question was reworded from "The new numeric passwords are better than the older ones." because we wanted to gauge the entire experience of logging in to the system, not just the passwords. However, the numeric passwords and student ID's are the central piece of logging in.

14. Having a CRN number is preferable to having a course number and department abbreviation.

Motivation for Question 14: This question was based on the previous questionnaire Question 16.

15. The schedule modification feature (Add/Remove/Change Classes) was intuitive and made changing your schedule easier.

Motivation for Question 15: This question measures the user's attitude toward the schedule modification featured used by the redesigned Banner to eliminate CRN's.

16. Have you noticed any redundant features in the system ? If yes, what are they ?



17. Do you have any suggestions for improving the system ?



18. Are there any questions which you feel haven't been addressed ?

Motivation for Questions 16, 17, and 18: These questions was taken directly from the previous questionnaire Questions 17, 18, 19.

Appendix C: Graphs of Performance and Questionnaire Data