GVU Technical Report Number: GIT-GVU-04-03

Title: An Empirical Study of the Effect of Agent Competence on User Performance and Perception

Authors: Jun Xiao, John T. Stasko, Richard Catrambone

Abstract:
We studied the role of the competence of a user interface agent/assistant that helped users to learn and use a new text editor. Participants in the study made a set of prescribed changes to a document via the editor with the aid of one of four interface agents. Participants could ask questions out loud to the agent and the agent would respond using a synthesized voice; the agent would also make proactive suggestions. The agents varied in the quality of responses and suggestions made. One group of participants were provided with a help screen as well as the agent. We focused on assessing the relation between users' objective performance, interaction style, and subjective experience. Results revealed that the perceived utility of the agent was influenced by the types of errors made by the agent, while participants' subjective impressions of the agent related to the perceptions of its representation. In addition, allowing participants to choose their preferred assistance style(s) (agent vs. online-help) improved objective performance. We correlate quantitative findings with qualitative interview data and discuss implications for the design and the implementation of systems with interface agents.

Keywords: Embodied conversational agents, interface agent, empirical evaluation, qualitative study, quantitative study

You can access this technical report via: PDF