This page is being updated--readings are subject to change
Design and Evaluation
|(Email is best way to reach me.)|
|Office Hours:||Find me after class or email for an appointment|
|Teaching Assistant: ||
- Educational Research by L.R. Gay and Peter Airasian. Sixth edition.
New Jersey: Prentice Hall, 2000.
- Articles online
- Becoming Qualitative
Researchers by Corrinne Glesne. White Plains, NY: Longman Publishing
- Interviewing as Qualitative Research by Irving Seidman.
New York: Teacher's College Press, 1991.
Available at the Georgia Tech
bookstore, or click
to buy them from Amazon books with the profit from the sale going to buy
pizza for students in this and other classes.
In this class, you immediately begin applying your knowledge of the
theory of educational technology to design a piece of software.
Consequently, you must already know something about it! To take this
class, you must have completed either:
If you have equivalent experience from classes in another department or at
another institution, speak to the instructor to request permission to
enroll. No one without a solid theoretical background in this
research area may take this class. (The issue is not just that you
will have difficulty--it's also difficult to have class discussions
if we have to stop and explain the basics to class members without
background in this area.)
- CS 4660: Educational Technology
- CS 6460: Educational Technology: Conceptual Foundations
Course requirements are the same for graduate and undergraduate students,
but I expect more depth in papers and projects by graduate students.
Focus of the Course
The course has two parts, each with a set of questions that we will be
What is learner-centered design?
What are student needs for learning? How do you facilitate learning with
software? What design principles are in common use for educational software
and which should be used for a given project?
How do we evaluate educational software? What variables should we be
considering (e.g., teachers, learning style, social context)? What methods
for evaluation should we use (e.g., ethnography, clinical interviews, log
Objectives of this course are for students to be able to:
- Design software about which you are willing to make explicit claims
about the intended student audience, the needs of that audience, how the
software meets those needs (including choice of media, structure, and
interface), and what should be learned from the software.
- Evaluate the software and describe which (of many possible) variables
were studied, why these variables were studied, how these variables were
studied (what methods), and why those methods were used.
So what are we going to do?
During the course of the semester, students will (alone or in groups of at
- Design a piece of educational software,
- Implement that software, and
- Evaluate it (with at least four users, preferably a whole class if
There are no restrictions on the KIND of software (e.g., microworlds,
construction kits, intelligent tutors) implemented, as long as the
are willing to claim that the software facilitates learning and to evaluate
Most classes will be a discussion of one or two papers.
Wherever possible, I have provided links to the authors' home pages.
You are encouraged at least to browse through those pages.
Papers will be available either as handouts
or on the web.
You may wish to print web papers out before reading them.
Your grade will be based on these assignments:
- Preliminary software design (5%)
- Revised software design: (15%)
- Results of field work
- Revised software design
- Human subjects proposal
- Software development plan
- Midterm design assignment: (35%)
- Paper: your design and its pedogical foundations
- Presentation of software and its design
- Evaluation plan (5%)
- Final evaluation assignment: (35%)
- Paper: Results of your evaluation
- In-class presentation of results
- Class participation (5%)
Suggested assignment lengths are approximate. Some people will do a good
in less and some will require many more pages.
Please don't play games with the margins or fonts to try to make it come out
a certain length--no one is counting.
Don't worry about the page
count--just make sure that you've completed the assignment well.
Please remember to double space papers so I can write comments.
You will be graded primarily on the thoughtfulness of your design
and evaluation process, as documented in your papers and
presentations--the emphasis is not on
the quality of your software. If you are working in a group, each
team member must submit separate midterm and final papers. For
other assignments, one per team is sufficient. Each team member must
also participate in giving the in-class presentations.
If you registered for the class and you already
have a CoC account, you have been given access to College of
Computing machine clusters (if you didn't have access already). If you
register late, you should be added later automatically. If not, mail
asking to be given access.
If you need a CoC account, you can get a request form in front of CoC 213.
Complete the form and return it there.
CoC computing asks that:
- If your work can be done on an OIT machine, please do so. CoC machines
tend to be over-used. But if you need special software not available at
using machines in this building is fine.
- Please do not use the SGIs unless you need SGIs specifically.
- Booklets on available CoC computing resources are available outside
of room CoC 140.
- CoC does not currently have a Mac cluster, but there are ones run by OIT
the Student Center (2nd floor) near the Music Listening Room and the
Library (basement). However, those may not have the development software
that you need. If you absolutely need to work on a Mac, come talk to me
and we'll see if we can think up a work-around.
- If you need special software that is not currently available to
you, please let me know immediately. It may take a while to get, or
may in fact not be possible to get.
All readings are subject to change.
- (1/8) Introduction: AquaMOOSE 3D as a case study in Design and Evaluation
Prof. Bruckman regrets that she will be at a conference this week--
Computer-Supported Collaborative Learning (CSCL) in Boulder, CO.
Syllabi will be distributed in class today, and Jason Elliott will
be available to answer questions. Should you have any questions about
the class, email the instructor at firstname.lastname@example.org. You can also
include a phone number in your email. Prof. Bruckman will try to answer
all questions promptly, and will give you a call on the phone if you
Jason Elliott will show his "AquaMOOSE 3D" software and discuss
it as an example of educational technology design and evaluation.
- (1/10) No class
Introduction, meet me and each other, start making teams.
- What is design?
- What is learner-centered design?
- How is designing educational software different from designing other
kinds of software?
- Can you think of a learning experience of yours that was special?
What made it special?
- Can you draw on that experience in your design work?
- (1/17) Introduction to Evaluation
While evaluation is primarily covered in the second half of
you need to understand something about it in order to be able to design
software that you will be able to evaluate. This class will give
you an overview of the evaluation techniques that we will study in
detail in the second half of the course.
- "Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings" by
Ann L. Brown (Handout.)
Come to class prepared to discuss your project plans and
progress so far.
Due: Assignment 1, Preliminary Software Design.
Please submit a roughly five-page project proposal including:
- The names and email addresses of your team members,
- A general description of what role each person will play within the
- What learning goal you hope to meet with your software,
- A basic description of what your software will look like, including
sketches (which may be hand drawn),
- A description of the social context in which your software is designed
to be used, and
- A list of milestones for completing your project within the available
Please bring enough copies of your project proposal for the whole
- (1/24) Early Field Work/Needs Analysis
- Glesne, Corrine. "Being There: Developing Understanding
Through Participant Observation." Chapter 3 in Becoming Qualitative
Researchers. White Plains, NY: Longman Publishing Group, 1999.
- (1/29) Research Ethics/Human Subjects Issues
- Chapters 4 ("Establishing Access to, Making Contact With, and Selecting
Participants") and 5 ("Affirming Informed Consent") in Seidman.
- MOOSE Crossing permission forms
- "The ethics of research," and "gaining entry to the research site";
Gay and Airasian, pp. 93-104
Designing for an Audience
- (2/5) Revised Design Proposals
Due: Assignment 2: Revised Software
- How do you design a good construction kit?
- What do kids learn from working with
software like Logo, StarLogo, MOOSE, and the programmable brick?
- (2/12) Scaffolding
In class, I'll give a short talk on how to give a short
Case Study of an Educational Software Project:
- "Implementation and Evaluation of Genscope Learning Environment:
Issues, Solutions, and Results." Paul Horwitz, Joyce Schwartz, et al.
Procceedings of ICLS 98. Charlottesville, VA: AACE Press, p.
- The Genscope Website,
Learning from Simulation
- "Seductions of Sim" by Paul Starr. The American Prospect, Spring 1994,
p. 19-29. (Handout.)
- (2/21) Usability versus Learning Value
- "HCI for Kids" by Amy Bruckman and Alisa Bandlow (Handout.)
Student Design Presentations
Due: Assignment 3, Midterm Papers and
- (2/28) In-class design exercise
- (3/5 & 3/7) Spring Break
Quantitative vs Qualitative Methods
- Gay and Airasian, "Quantitative and qualitative research," and
"guidelines for classification"; pp. 8-23.
- Papert, Seymour. "Computer Criticism vs. Technocentric Thinking."
Educational Researcher, Jan-Feb 1987, pp. 22-30. (Handout.)
- Pea, Roy D. "The Aims of Software Criticism: Reply to Professor
Papert." Educational Researcher, June-July 1987, pp. 4-8. (Handout.)
- Walker, Derek F. "Logo Needs Research: A Response to Professor
Papert's Paper." Educactional Researcher, June-July 1987,
p. 9-11. (Handout.)
- "Evaluating Instruction: The Complementary Use of Clinical Interviews"
by Fred N. Finlay. Journal of Research in Science Teaching 23:7,
635-650 (1986). (Handout.)
- Chapter 6, "Technique isn't everything, but it is a lot" in Seidman.
- "Thick Description: Towards an Interpretive Theory of Culture" by
Due: Assignment 4, Evaluation Plan.
Your evaluation plan should be double-spaced, in 12-pt font, and
approximately 5 pages long. Describe in detail how you will conduct
your evaluation. If you are working in a team, describe what each
team member will do. One plan per team is sufficient for this
assignment (but you must hand in separate midterm and final papers).
Please come to class prepared to talk about how you will evaluate your
software. Each group should present its ideas as succinctly as
possible--preferably in about 3 minutes, leaving another 2 minutes for
- Gay & Aairasian, Chapter 4
- (3/28) Measuring Instruments
- Gay & Airasian, Chapter 5
will talk about use of attitudinal inventories in his Palaver Tree Online
- Jay, Gina M. & Willis, Sherry L. (1992) Influence of direct computer
experience on older adults' attitudes toward computers. Journals of
Gerontology, Vol 47(4), pp. 250-257 (handout).
- (4/4) Descriptive Research
- Gay & Airasian, Chapter 8
- (4/9) Correlational & Causal-Comparative Research
- Gay & Airasian, Chapters 9 & 10
Log File Analysis
- (4/16) Experimental Research
- Gay & Airasian, Chapter 11
- (4/18) Experimental Research, continued
Early Experimental Ed Tech Research: Logo
- Clements, Douglas H. and Dominic F. Gullo. (1984) "Effects of
Computer Programming on Young Children's Cognition." Journal of
Educational Psychology. 76:6. 1051-1058. (Handout.)
- Clements, Douglas H. "Effects of Logo and CAI Environments on
Cognition and Creativity." Journal of Educational Psychology. 78:4.
- (4/25) Student Evaluation Presentations
Due: Assignment 5,
Final Papers and Presentations
Each student may have a total of three late days over the course of the
semester. Once you've used those up, late work will be penalized.
I always make an effort to return papers promptly; however,
late papers may be returned substantially later.
Your presentations may not be late. Your final papers may not be late.
Hints for Successful Projects
- In designing your project think small. The semester is
short. The important thing is that you are willing to make some
learning claim about your software, and to evaluate that claim.
- You are responsible for finding your own study subjects
for your evaluation in the second half of the course.
The easiest thing would be for you to design for college or graduate school age
students and ask your peers to be subjects. If you decide to
design for young children, you need to know a teacher
of that age group who is willing to work with you.
- If you are not a programmer, team with someone who is.
- Project teams should be all undergraduate or all graduate,
becaues the grading criteria for undergrads and grads are
grad/undergrad teams require permission of instructor.
- If you work with someone else, make sure you define your roles clearly.
Make sure each person feels they have equal say in the
- If you are a non-programmer working with a programmer, don't make your
programmer partner feel like hired help/slave labor.
- If you are a programmer
working with a non-programmer, share control--sure you'll have new ideas as
you code, but don't make major changes without consulting your partner.
- As you work on your project, keep careful notes about your design
process. Your papers need to document not just your final
design decisions, but the earlier decisions you made and why
you changed your mind.
- Make something your target audience
will enjoy. Learning is not like swallowing castor oil.
But remember to make the act of learning itself fun--don't sugar coat
around the learning.
- Have fun with your project. It's a privilege
to have the time to
make something interesting, and the time to think
seriously about what you're doing and why.
Questions welcome--email email@example.com