User Interface Software and Technology

Introduction

In addition to understanding how to design and evaluate human-computer interfaces, you must also understand how to implement them effectively. Moreover, you must understand the structure of artifacts (window systems and toolkits) that enable programmers to build and maintain complex applications. As with any other area of computer software, there is a body of techniques, algorithms and approaches that have been built up over the years. This is especially true of the 2D WIMP (Windows, Icons, Mice, Pointers) interfaces found on all commonly available systems these days (ie. Windows, the Macintosh, and X). The goal of this section of the qualifier is to ensure that you are familiar with the body of knowledge related to user-interface software.

When looking at the reading list below, two things should strike you: first, most of the papers listed are quite old in "computer time" (over a decade, in some cases), and second, they focus almost entirely on building WIMP interfaces and say little about the newer, "off the desktop" user interfaces (that go by names such as Ubiquitous Computing, Information Appliances, Intelligent Environments, Augmented Reality, Wearable Computing, Tangible Media, and so forth). The papers are mostly old because the techniques and approaches on which current toolkits are built was the subject of research a decade or more ago; the best descriptions of these techniques is the papers that describe that research. The papers focus on 2D WIMP interfaces because current research on newer paradigms is still in the early stages; we will not know for a few more years which of the techniques being proposed by researchers is best.

While little definitive literature exists to describe the software and technology for non-WIMP paradigms, the abstractions and methodologies developed for building WIMP interfaces provide a framework for thinking about how one might approach the task of building software to support newer, non-WIMP paradigms. You should look at some examples of newer paradigms, perhaps by looking in the proceedings of the conferences discussed in the next section, and think about how these WIMP techniques do and do not transfer to these new paradigms.

General Resources

Surveys and detailed coverage of many user interface software and technology techniques are covered in CS 6456: Principles of User Interface Software.

Many papers in the ACM SIGGRAPH/SIGCHI conference series on User Interface Software and Technology, also known as the UIST conference, include significant examples of both toolkits and systems. This is also true to a lesser extent of the other ACM and IEEE sponsored conferences, especially CHI (Human Factors in Computing Systems). Many other ACM/IEEE conferences also contain examples relevant to their particular domain, although they are much rarer. Examples of such conferences include CSCW (more the ACM CSCW conference than the European ECSCW series), ISWC (Wearable Computing), HUC (Handheld and Ubiquitous Computing), ISAR/IWAR (Augmented Reality) and so on.

The Java Foundation Classes, aka the Swing Toolkit, are an excellent example of a state-of-the-art 2D UI software toolkit architecture. You should be familiar with this toolkit; a good place to start is the JavaSoft Swing documentation, including the Getting Started with Swing tutorial and the Swing Architecture document.

A good, introductory text on build UI software is:

Keep in mind that this is a very introductory book; it is a quick read for people with UI programming experience, and will provide a good background on elementary 2D graphics and UI programming for those with little experience.

Historical and Broad Coverage Papers

Brad Myers has written a number of survey papers covering the history of user interface software. A good example is: Recently, he and some co-authors wrote a paper summarizing the important milestones in UI software, and included a discussion of why certain techniques failed to catch on, and what some promising techniques might be for the future:

Toolkit Issues

A good paper that describes one of the first toolkits to use composition of widgets (via container widgets and layout management) to form complex nested layouts: An important issue for toolkits is how to create an API that allows programmers to develop applications that degrade gracefully when the displays become complex. Tang and Linton describe an architecture for graceful degredation in the Interviews toolkit in: There is growing interest in creating interfaces that are more lively and engaging. There are plenty of examples of such interfaces in recent UIST and CHI proceedings. Here is a paper that describes how to integrate animation into a user interface toolkit. Their technique is described in the context of their toolkit (ARTKIT), but is easily adapted to any toolkit: One approach to flexible interface layout that has not caught on is constraints. A good paper on simple, one-way constraints (which are common in UI toolkits that try to use constraints), is: Finally, for a discussion of the issues surrounding the design space of input devices, see:

Higher Level Tools

You should be familiar with UI Builders, aka Layout Editors, as every integrated development environment comes with one these days. There are other more powerful tools that are not common, however. For example, many folks have explored tools to create interfaces by demonstration: There was a significant amount of work in the late-80s/early-90s and so-called User Interface Management Systems. These tools were aimed at making interfaces easier to build by allowing the user to specify characteristics of the interface and then having the system build it. One of the earliest is systems generated simple menu-based dialog box interfaces from the annotated function prototypes in the programmers code: A good example of a much more advanced UIMS is the UIDE system. There are many papers describing many interesting aspects of the system; one good summary paper on it is: While UIMS research has largely ceased, it is important to understand it because of the growing interest in adaptive interfaces; those that adapt and change based on their user, the context of their use, or the characteristics of the device they are being used on. In order to create complex, adaptive interfaces, it is likely that many of the techniques developed for UIMSs will need to be used. You should think about how these techniques might be useful in such cases; how can we create an application that can present itself effectively on a wall-sized display, desktop display, or palm-sized display? Even more challenging, how can we have such applications adapt to non-visual displays? Retrofitting existing GUI-based applications to audio-only environments without cooperation from the application is extremely challenging, as discussed here:
Blair MacIntyre
Last Modified: Thu Aug 17 19:34:40 Edt 2000