Design and Analysis
of Educational Software
Office: CoC 255
(Email is best way to reach me.)
Wednesdays 4:00-5:30, Thursdays 3-4:30, and by appointment.
Teaching assistant: Shahin Shadfar
Help session time: TBA
Time: TR 1:30-3:00
Location: CoC 201
Text: Articles online, on electronic
reserve, and on regular library reserve.
Focus of the Course
The course has two parts, each with a set of questions that we will be addressing:
What is learner-centered design?
What are student needs for learning? How do you facilitate learning with software? What design principles are in common use for educational software and which should be used for a given project?
How do we evaluate educational software? What variables should we be
considering (e.g., teachers, learning style, social context)? What methods
for evaluation should we use (e.g., ethnography, clinical interviews, log file
Objectives of this course are for students to be able to:
- Design software about which you are willing to make explicit claims about the intended student audience, the needs of that audience, how the software meets those needs (including choice of media, structure, and interface), and what should be learned from the software.
- Evaluate the software and describe which (of many possible) variables were studied, why these variables were studied, how these variables were studied (what methods), and why those methods were used.
So what are we going to do?
During the course of the quarter, students will (alone or in groups of at most three):
- Implement a piece of educational software and
- evaluate it (with at least three users, preferably a whole class if available).
There are no restrictions on the KIND of software (e.g., microworlds,
construction kits, intelligent tutors) implemented, as long as the developers
are willing to claim that the software facilitates learning and to evaluate
Most classes will be a discussion of one (at most, two) papers.
Wherever possible, I have provided links to the authors' home pages.
You are encouraged at least to browse through those pages.
Papers will be available either on the web or on electronic reserve.
You may wish to print them out before reading them.
Grading will be based on two papers and two presentations:
- October 28th students will present their software (demos and/or screenshots are required) and submit a paper on the design of their software and the claims made for it. (Presentations continue in class on October 30th.)
- 10% of final grade on presentation
- 40% of final grade on paper
- On December 2nd, students will present the result of their evaluation and submit a paper describing the evaluation (subjects, variables studied, methods, data, analyses, and conclusions). (Presentations continue in class on
- 10% of final grade on presentation
- 40% of final grade on paper
You will be graded primarily on the thoughtfulness of your design
and evaluation process, as documented in your papers and
presentations--the emphasis is not on
the quality of your software. If you are working in a group, each
team member must submit a separate paper.
If you registered for the class and you already
have a CoC account, you have been given access to College of
Computing machine clusters (if you didn't have access already). If you
register late, you should be added later automatically. If not, mail help@cc
asking to be given access.
If you need a CoC account, you can get a request form in front of CoC 213.
Complete the form and return it there.
CoC computing asks that:
- If your work can be done on an OIT machine, please do so. CoC machines
tend to be over-used. But if you need special software not available at OIT,
using machines in this building is fine.
- Please do not use the SGIs unless you need SGIs specifically.
- Booklets on available CoC computing resources are available outside
of room CoC 140.
- CoC does not currently have a Mac cluster, but there are ones run by OIT in
the Student Center (2nd floor) near the Music Listening Room and the
Library (basement). However, those may not have the development software
that you need. If you absolutely need to work on a Mac, come talk to me
and we'll see if we can think up a work-around. I may be able to find
creative solutions, but only for a small number of people.
- If you need special software that is not currently available to
you, please let me know immediately. It may take a while to get, or
may in fact not be possible to get.
(Some readings may change.)
Introduction: Learner-Centered Design
Introduction, meet me and each other, start making teams.
- What is design?
- What is learner-centered design?
- How is designing educational software different from designing other
kinds of software?
- Can you think of a learning experience of yours that was special?
What made it special?
- Can you draw on that experience in your design work?
- How do you design a good construction kit?
- What do kids learn from working with
software like Logo, StarLogo, MOOSE, and the programmable brick?
- (10/2) Introduction to Evaluation
While evaluation is primarily covered in the second half of the course,
you need to understand something about it in order to be able to design
software that you will be able to evaluate. This class will give
you an overview of the evaluation techniques that we will study in
detail in the second half of the course.
- "Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings" by
Ann L. Brown
Come to class prepared to discuss your project plans and progress so far.
Please submit a roughly two to three page project proposal including:
- The names and email addresses of your team members,
- A general description of what role each person will play within the team,
- What learning goal you hope to meet with your software,
- A basic description of what your software will look like,
- A description of the social context in which your software is designed to be used, and
- A list of milestones for completing your project within the available time frame.
Please bring enough copies of your project proposal for the whole class.
These preliminary proposals will not be graded. This is an opportunity for you to get some feedback on your ideas from me and from your classmates.
- (10/9) Discussion of project proposals, continued
- (10/14) Designing for an Audience
- (10/16) Scaffolding
- (10/21) Designing Goal-Based Scenarios
- "The Design of Goal-Based Scenarios" by
Roger Schank, Andrew Fano,
Benjamin Bell, and Menachaem Jona. Journal of the Learning Sciences
- (10/23) Intelligent Tutors
- "Cognitive Tutors: Lessons Learned" by John R. Anderson, Albert
T. Corbett, Kenneth R. Koedinger, and Ray Pelletier. In
Journal of the Learning Sciences 4(2), 167-207 (1995).
- (10/28) Student Design Presentations
Design papers due.
- (10/30) Student Design Presentations (Continued)
- (11/4) Research Ethics
AERA code of conduct
- MOOSE Crossing permission forms
- Chapters 4 ("Establishing Access to, Making Contact With, and Selecting
Participants") and 5 ("Affirming Informed Consent") from "Interviewing as
Qualitative Research" by IE Seidman. New York: Teacher's College Press, 1991.
(Handout; also attached is chapter 6, which is for 11/13.)
- (11/6) Ethnography
Guest lecturer: Wendy Newstetter
- (11/11) Evaluation Discussion
Please come to class prepared to talk about how you will evaluate your
software. Each group should present its ideas as succinctly as
possible--preferably in about 3 minutes, leaving another 2 minutes for
Optional: if you submit a description of your evaluation plan, I will
send you comments on it. You can also come to office hours for more
- (11/13) Formative Evaluation
- (11/18) Clinical Interviews
- "Evaluating Instruction: The Complementary Use of Clinical Interviews"
by Fred N. Finlay. Journal of Research in Science Teaching 23:7,
- Chapter 6, "Technique isn't everything, but it is a lot" from
"Interviewing as Qualitative Research" by IE Seidman. New York: Teacher's
College Press, 1991. (Handed out with chapters for 11/4)
- (11/20) User Interface Design and Evaluation
Log File Analysis
Colleen Kehoe will discuss her work with log file analysis in the
- (12/2) Student Evaluation Presentations
Evaluation papers due.
Presentations continue at 7 pm, in room 201. Celebration to follow.
- (12/4) No class.
Here are a couple project ideas:
A few hints for successful projects:
- Kids and seniors:The popular press has spread the myth
that on the Internet, every kid can email a Nobel Prize winner. (That's
actually true... just don't expect a response!) Kids don't need
Nobel-laureate-quality advice-- they just need interaction with thoughtful
adults who take them seriously. Older people often report that since they
retired, no one asks for their opinion about anything any more. They want to
feel needed, to share their expertise, and to make a positive contribution.
(But they want to do it on their own time, and with flexible time
commitments.) Could you design an activity to get kids and seniors learning
from one another online?
- Sick kids: What kind of computer activity would you design
for a chronically ill child in the hospital? Could you use the network
to combat their isolation and loneliness?
- In designing your project think small. The quarter is
short. The important thing is that you are willing to make some
learning claim about your software, and to evaluate that claim.
- If you are not a programmer, team with someone who is.
- As you work on your project, keep careful notes about your design
process. Your papers need to document not just your final
design decisions, but the earlier decisions you made and why
you changed your mind.
- Make something your target audience
will enjoy. Learning is not like swallowing castor oil.
But remember to make the act of learning itself fun--don't sugar coat
around the learning.
- Have fun with your project. It's a privilege
to have the time to
make something interesting, and the time to think
seriously about what you're doing and why.
Question welcome--email firstname.lastname@example.org