From lara@cc.gatech.edu (Lara Catledge)
Overheads from 11/23/94

	The Evaluation of 
	Adaptive Systems
	
	P. Totterdell and E. Boyle
	
	Presented by
	Lara Catledge
	
	
	Adaptive User Interfaces
	Fall 1994
	11/23/94

Evaluation

	For Adaptivity Performance &
	For the Design Process
	
	Different Types of HCI Evaluation
	Problems in Evaluative Adaptive Systems
	Techniques for Evaluating AID
	Application to Other Systems

Types of HCI Evaluation

	Formative 
	Summative
	Comparative
	Diagnostic
	Iterative Design

Steps for Evaluation
	Objective
	Experimental Design   *Difficult for Adaptive Systems
	Data Collection
	Analysis of Data
	Conclusions

Problems in Evaluation of Adaptive Systems
	Comparison with Adaptive Systems
-	Prime with User Data
	Novel Applications
	Intervals of Comparison
-	Final vs. In Media Res
	Context of Use
	Both user and system adapt

Criteria
	Criteria for Adaptive Behaviors
-	One adaptation, not others
-	One adaptation and others
-	One adaptation -> others
	Self-Evaluation
-	System -- Log files, error rates
*	Human
*	Wider, Summative evaluations
*	Within Social Context
* 	Design specifications of experimental design to be encoded within
	system architecture P Migration of responsibility from human
	to system

Evaluating Adaptive Systems I

	Metrics
-	Trigger - Event
-	Theory Assessment - Basis for change
-	Recommendation - Characterizes recommendations
-	Implementation - General functioning of system
-	Objective - Captured by evaluator
-	General - Unknowns and human centered
	Other components
-	User Model/Theory based Reasoning
-	Recommendations
-	Control
-	Monitor

Niche Descriptions

	Tease out assumptions about the world
-	Transforming
-	Operational

Evaluating AID

	Data Collected
	Assessment of User Model
-	Against users own statements - 40% detected
	Assessment of changes made at interface
-	expert assesses chances made - 7% helpful
	Comparative Performance with Static System
-	Questionnaires
-	Command Entry, Task completion, & goal achievement rates

Stability and Change

	Telephone Lookup System
	Static vs. Adaptive system
	Take 2: increased number of items
	Swapping by evaluation
-	Self-Evaluation
-	Fall-Back Strategy
	Consistency of Interface vs. Memory

Affect on Design Processes

	Formative Evaluation
-	Iterative Design
-	Check Assumptions
-	Objectives & Metrics
	Summative Evaluation
-	Verify pre-build checks
-	Adaptive vs Static System Evaluation
-	Meeting objectives of designer
-	Field trials or longitudinal studies

Questions for Discussion

	Advantages of splitting evaluation and adaptation between user and
	human evaluator?

	comparative and non-comparative methods?
	Will any of these methods approach true usability of the system?

	How would you compare or complement this research with sociological
	methods?

-- 
					
					-Lara D. Catledge (lara@cc.gatech.edu)