Design and Evaluation
|(Email is best way to reach me.)|
|Office Hours:||Tuesdays 1-2 pm|
|Teaching Assistant: ||
- Interviewing as Qualitative Research by Irving Seidman.
New York: Teacher's College Press, 1991.
Available at the Georgia Tech
bookstore, or click h
to buy it from Amazon books with the profit from the sale going to buy
pizza for students in this and other classes.
- Articles online
- Becoming Qualitative
Researchers by Corrinne Glesne. White Plains, NY: Longman Publishing
Available for purchase
In this class, you immediately begin applying your knowledge of the
theory of educational technology to design a piece of software.
Consequently, you must already know something about it! To take this
class, you must have completed CS 6460: "Educational Technology:
Conceptual Foundations" (or its quarter equivalent, CS 6397.) If you
have equivalent experience from classes in another department or at
another institution, speak to the instructor to request permission to
enroll. No one without a solid theoretical background in this
research area may take this class.
Focus of the Course
The course has two parts, each with a set of questions that we will be
What is learner-centered design?
What are student needs for learning? How do you facilitate learning with
software? What design principles are in common use for educational software
and which should be used for a given project?
How do we evaluate educational software? What variables should we be
considering (e.g., teachers, learning style, social context)? What methods
for evaluation should we use (e.g., ethnography, clinical interviews, log
Objectives of this course are for students to be able to:
- Design software about which you are willing to make explicit claims
about the intended student audience, the needs of that audience, how the
software meets those needs (including choice of media, structure, and
interface), and what should be learned from the software.
- Evaluate the software and describe which (of many possible) variables
were studied, why these variables were studied, how these variables were
studied (what methods), and why those methods were used.
So what are we going to do?
During the course of the semester, students will (alone or in groups of at
- Design a piece of educational software,
- Implement that software, and
- Evaluate it (with at least four users, preferably a whole class if
There are no restrictions on the KIND of software (e.g., microworlds,
construction kits, intelligent tutors) implemented, as long as the
are willing to claim that the software facilitates learning and to evaluate
Most classes will be a discussion of one or two papers.
Wherever possible, I have provided links to the authors' home pages.
You are encouraged at least to browse through those pages.
Papers will be available either as handouts
on the web.
You may wish to print web papers out before reading them.
Your grade will be based on these assignments:
- Preliminary software design (5%)
- Revised software design: (15%)
- Results of field work
- Revised software design
- Human subjects proposal
- Software development plan
- Midterm design assignment: (35%)
- Paper: your design and its pedogical foundations
- Presentation of software and its design
- Evaluation plan (5%)
- Final evaluation assignment: (35%)
- Paper: Results of your evaluation
- In-class presentation of results
- Class participation (5%)
Suggested assignment lengths are approximate. Some people will do a good
in less and some will require many more pages.
Please don't play games with the margins or fonts to try to make it come out
a certain length--no one is counting.
Don't worry about the page
count--just make sure that you've completed the assignment well.
Please remember to double space papers so I can write comments.
You will be graded primarily on the thoughtfulness of your design
and evaluation process, as documented in your papers and
presentations--the emphasis is not on
the quality of your software. If you are working in a group, each
team member must submit separate midterm and final papers. For
other assignments, one per team is sufficient. Each team member must
also participate in giving the in-class presentations.
If you registered for the class and you already
have a CoC account, you have been given access to College of
Computing machine clusters (if you didn't have access already). If you
register late, you should be added later automatically. If not, mail
asking to be given access.
If you need a CoC account, you can get a request form in front of CoC 213.
Complete the form and return it there.
CoC computing asks that:
- If your work can be done on an OIT machine, please do so. CoC machines
tend to be over-used. But if you need special software not available at
using machines in this building is fine.
- Please do not use the SGIs unless you need SGIs specifically.
- Booklets on available CoC computing resources are available outside
of room CoC 140.
- CoC does not currently have a Mac cluster, but there are ones run by OIT
the Student Center (2nd floor) near the Music Listening Room and the
Library (basement). However, those may not have the development software
that you need. If you absolutely need to work on a Mac, come talk to me
and we'll see if we can think up a work-around.
- If you need special software that is not currently available to
you, please let me know immediately. It may take a while to get, or
may in fact not be possible to get.
All readings are subject to change.
Introduction: Learner-Centered Design
Introduction, meet me and each other, start making teams.
- What is design?
- What is learner-centered design?
- How is designing educational software different from designing other
kinds of software?
- Can you think of a learning experience of yours that was special?
What made it special?
- Can you draw on that experience in your design work?
- (1/12) Introduction to Evaluation
While evaluation is primarily covered in the second half of
you need to understand something about it in order to be able to design
software that you will be able to evaluate. This class will give
you an overview of the evaluation techniques that we will study in
detail in the second half of the course.
- "Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings" by
Ann L. Brown (Handout.)
- (1/17) MLK Day Holiday
- (1/19) Class cancelled
Come to class prepared to discuss your project plans and
progress so far.
Due: Assignment 1, Preliminary Software Design.
Please submit a roughly five-page project proposal including:
- The names and email addresses of your team members,
- A general description of what role each person will play within the
- What learning goal you hope to meet with your software,
- A basic description of what your software will look like, including
sketches (which may be hand drawn),
- A description of the social context in which your software is designed
to be used, and
- A list of milestones for completing your project within the available
Please bring enough copies of your project proposal for the whole
- (1/26) Early Field Work/Needs Analysis
- Glesne, Corrine. "Being There: Developing Understanding
Through Participant Observation." Chapter 3 in Becoming Qualitative
Researchers. White Plains, NY: Longman Publishing Group, 1999.
- (1/31) Research Ethics/Human Subjects Issues
- (2/2) Discussion of Project Designs in Progress
Come to class today prepared to talk about how your design work
- (2/7) Designing for an Audience
- How do you design a good construction kit?
- What do kids learn from working with
software like Logo, StarLogo, MOOSE, and the programmable brick?
- (2/14) Scaffolding
In class, I'll give a short talk on how to give a short
- (2/16) Revised Design Proposals
Due: Assignment 2: Revised Software
- (2/21) Case Study of an Educational Software Project:
- "Implementation and Evaluation of Genscope Learning Environment:
Issues, Solutions, and Results." Paul Horwitz, Joyce Schwartz, et al.
Procceedings of ICLS 98. Charlottesville, VA: AACE Press, p.
- The Genscope Website,
- (2/23) Learning from Simulation
- "Seductions of Sim" by Paul Starr. The American Prospect, Spring 1994,
p. 19-29. (Handout.)
- (2/28) Special Guest: Nichole Pinkard
- (3/1) Quantitative vs Qualitative Methods
- Papert, Seymour. "Computer Criticism vs. Technocentric Thinking."
Educational Researcher, Jan-Feb 1987, pp. 22-30. (Handout.)
- Pea, Roy D. "The Aims of Software Criticism: Reply to Professor
Papert." Educational Researcher, June-July 1987, pp. 4-8. (Handout.)
- Walker, Derek F. "Logo Needs Research: A Response to Professor
Papert's Paper." Educactional Researcher, June-July 1987,
p. 9-11. (Handout.)
- (3/13) Clinical Interviews
- "Evaluating Instruction: The Complementary Use of Clinical Interviews"
by Fred N. Finlay. Journal of Research in Science Teaching 23:7,
635-650 (1986). (Handout.)
- Chapter 6, "Technique isn't everything, but it is a lot" in Seidman.
- (3/15) Ethnography
- "Thick Description: Towards an Interpretive Theory of Culture" by
- (3/20) Student Design Presentations
Due: Assignment 3, Midterm Papers and
- (3/22) In-Class Design Exercise
- (3/27) Portfolio Scoring
- Chi, Michelene T. (1997). Quantifying Qualitative Analyses of Verbal
Data: A Practical Guide. Journal of the Learning Sciences, 6(3), 271-315.
- (3/29) Evaluation Discussion
Due: Assignment 4, Evaluation Plan.
Your evaluation plan should be double-spaced, in 12-pt font, and
approximately 5 pages long. Describe in detail how you will conduct
your evaluation. If you are working in a team, describe what each
team member will do. One plan per team is sufficient for this
assignment (but you must hand in separate midterm and final papers).
Please come to class prepared to talk about how you will evaluate your
software. Each group should present its ideas as succinctly as
possible--preferably in about 3 minutes, leaving another 2 minutes for
Log File Analysis
will talk about her work with
data from the MOOSE Crossing project.
- (4/5) Attitudinal Inventories
will talk about use of attitudinal inventories in his American Timewarp
- (4/10) Pre/post Tests
- Clements, Douglas H. and Dominic F. Gullo. (1984) "Effects of
Computer Programming on Young Children's Cognition." Journal of
Educational Psychology. 76:6. 1051-1058. (Handout.)
- Clements, Douglas H. "Effects of Logo and CAI Environments on
Cognition and Creativity." Journal of Educational Psychology. 78:4.
- (4/12) In-Class Interview Practice
In class we'll practice interviewing technique on each
- (4/17) Case Study of an Educational Software
Project: Rappin Reader
- Pinkard, N. (in press) Lyric Reader: Architecture for Creating
Intrinsically Motivating and Culturally Relevant Reading Environments.
- Say Say Oh
- (4/19) Student Evaluation Presentations
Due: Assignment 5,
Final Papers and Presentations
- (4/24) No class (AERA)
- (4/26) No class (AERA)
Each student may have a total of three late days over the course of the
semester. Once you've used those up, late work will be penalized.
I always make an effort to return papers promptly; however,
late papers may be returned substantially later.
Your presentations may not be late. Your final papers may not be late.
Hints for Successful Projects
- In designing your project think small. The semester is
short. The important thing is that you are willing to make some
learning claim about your software, and to evaluate that claim.
- If you are not a programmer, team with someone who is.
- If you work with someone else, make sure you define your roles clearly.
Make sure each person feels they have equal say in the
- If you are a non-programmer working with a programmer, don't make your
programmer partner feel like hired help/slave labor.
- If you are a programmer
working with a non-programmer, share control--sure you'll have new ideas as
you code, but don't make major changes without consulting your partner.
- As you work on your project, keep careful notes about your design
process. Your papers need to document not just your final
design decisions, but the earlier decisions you made and why
you changed your mind.
- Make something your target audience
will enjoy. Learning is not like swallowing castor oil.
But remember to make the act of learning itself fun--don't sugar coat
around the learning.
- Have fun with your project. It's a privilege
to have the time to
make something interesting, and the time to think
seriously about what you're doing and why.
Questions welcome--email firstname.lastname@example.org