Design and Analysis
of Educational Software
Office: CoC 255
(Email is best way to reach me.)
Mondays 4:30-6:00, and by appointment.
Time: MW 3-4:30
Location: CCB 101
- Interviewing as Qualitative Research by Irving Seidman.
New York: Teacher's College Press, 1991.
- Course packet of articles for sale at the bookstore
- Articles online
Focus of the Course
The course has two parts, each with a set of questions that we will be addressing:
What is learner-centered design?
What are student needs for learning? How do you facilitate learning with software? What design principles are in common use for educational software and which should be used for a given project?
How do we evaluate educational software? What variables should we be
considering (e.g., teachers, learning style, social context)? What methods
for evaluation should we use (e.g., ethnography, clinical interviews, log file
Objectives of this course are for students to be able to:
- Design software about which you are willing to make explicit claims about the intended student audience, the needs of that audience, how the software meets those needs (including choice of media, structure, and interface), and what should be learned from the software.
- Evaluate the software and describe which (of many possible) variables were studied, why these variables were studied, how these variables were studied (what methods), and why those methods were used.
So what are we going to do?
During the course of the quarter, students will (alone or in groups of at most two):
- Implement a piece of educational software and
- evaluate it (with at least three users, preferably a whole class if available).
There are no restrictions on the KIND of software (e.g., microworlds,
construction kits, intelligent tutors) implemented, as long as the developers
are willing to claim that the software facilitates learning and to evaluate
Most classes will be a discussion of one (at most, two) papers.
Wherever possible, I have provided links to the authors' home pages.
You are encouraged at least to browse through those pages.
Papers will be available either in the course packet or
on the web.
You may wish to print web papers out before reading them.
Grading will be based on two papers and two presentations:
- October 26th students will present their software (demos and/or screenshots are required) and submit a paper on the design of their software and the claims made for it. (Presentations continue in class on October 28th.)
- 10% of final grade on midterm presentation
- 40% of final grade on midterm paper
- On December 2nd, students will present the result of their evaluation and submit a paper describing the evaluation (subjects, variables studied, methods, data, analyses, and conclusions). (Presentations continue in an evening session starting at 7 pm. Attendance is mandatory. Please see me
in advance if you have an unavoidable
- 10% of final grade on final presentation
- 40% of final grade on final paper
You will be graded primarily on the thoughtfulness of your design
and evaluation process, as documented in your papers and
presentations--the emphasis is not on
the quality of your software. If you are working in a group, each
team member must submit a separate paper.
If you registered for the class and you already
have a CoC account, you have been given access to College of
Computing machine clusters (if you didn't have access already). If you
register late, you should be added later automatically. If not, mail help@cc
asking to be given access.
If you need a CoC account, you can get a request form in front of CoC 213.
Complete the form and return it there.
CoC computing asks that:
- If your work can be done on an OIT machine, please do so. CoC machines
tend to be over-used. But if you need special software not available at OIT,
using machines in this building is fine.
- Please do not use the SGIs unless you need SGIs specifically.
- Booklets on available CoC computing resources are available outside
of room CoC 140.
- CoC does not currently have a Mac cluster, but there are ones run by OIT in
the Student Center (2nd floor) near the Music Listening Room and the
Library (basement). However, those may not have the development software
that you need. If you absolutely need to work on a Mac, come talk to me
and we'll see if we can think up a work-around. I may be able to find
creative solutions, but only for a small number of people.
- If you need special software that is not currently available to
you, please let me know immediately. It may take a while to get, or
may in fact not be possible to get.
Introduction: Learner-Centered Design
Introduction, meet me and each other, start making teams.
- What is design?
- What is learner-centered design?
- How is designing educational software different from designing other
kinds of software?
- Can you think of a learning experience of yours that was special?
What made it special?
- Can you draw on that experience in your design work?
- How do you design a good construction kit?
- What do kids learn from working with
software like Logo, StarLogo, MOOSE, and the programmable brick?
- (9/30) Introduction to Evaluation
While evaluation is primarily covered in the second half of the course,
you need to understand something about it in order to be able to design
software that you will be able to evaluate. This class will give
you an overview of the evaluation techniques that we will study in
detail in the second half of the course.
- "Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings" by
Ann L. Brown (In course packet.)
Come to class prepared to discuss your project plans and progress so far.
Please submit a roughly two to three page project proposal including:
- The names and email addresses of your team members,
- A general description of what role each person will play within the team,
- What learning goal you hope to meet with your software,
- A basic description of what your software will look like,
- A description of the social context in which your software is designed to be used, and
- A list of milestones for completing your project within the available time frame.
Please bring enough copies of your project proposal for the whole class.
These preliminary proposals will not be graded. This is an opportunity for you to get some feedback on your ideas from me and from your classmates.
- (10/7) Discussion of project proposals, continued
- (10/12) Designing for an Audience
- (10/14) Scaffolding
- (10/19) Research Ethics
- (10/21) Learning from Simulation
In class, I'll give a short talk on how to give a short talk.
- "Seductions of Sim" by Paul Starr. The American Prospect, Spring 1994,
p. 19-29. (In course packet.)
Human subjects applications and consent forms due.
- (10/26) Student Design Presentations
Design papers due.
- (10/28) Student Design Presentations (Continued)
- (11/2) Quantitative vs Qualitative Methods
- "The Case for Qualitative Research" by Gareth Morgan and Linda
Smircich. Academy of Management Review 1980, 5:4, 491-500.
(In course packet.)
- (11/4) Formative Evaluation
- (11/9) Evaluation Discussion
Please come to class prepared to talk about how you will evaluate your
software. Each group should present its ideas as succinctly as
possible--preferably in about 3 minutes, leaving another 2 minutes for
Optional: if you submit a description of your evaluation plan, I will
send you comments on it. You can also come to office hours for more
- (11/11) Clinical Interviews
- "Evaluating Instruction: The Complementary Use of Clinical Interviews"
by Fred N. Finlay. Journal of Research in Science Teaching 23:7,
635-650 (1986). (In course packet.)
- Chapter 6, "Technique isn't everything, but it is a lot" in Seidman.
- (11/16) Ethnography
- "Thick Description: Towards an Interpretive Theory of Culture" by Clifford Geertz
- (11/18) No class
Log File Analysis
Colleen Kehoe will discuss her work with log file analysis in the
will talk about her work with
data from the MOOSE Crossing project.
- (11/25) No class
- (11/30) In-Class Design Exercise
In class we'll spend twenty minutes designing a piece of educational
software in small groups, and then discuss everyone's designs.
- (12/2) Student Evaluation Presentations
Evaluation papers due.
Presentations continue at 7 pm.
Attendance at the evening session is mandatory, unless you have an
unavoidable conflict and contact me in advance.
Celebration to follow.
Each student may have a total of three late days over the course of the
quarter. Once you've used those up, late work will be penalized.
I always make an effort to return papers promptly; however,
late papers may be returned substantially later.
Hints for Successful Projects
- In designing your project think small. The quarter is
short. The important thing is that you are willing to make some
learning claim about your software, and to evaluate that claim.
- If you are not a programmer, team with someone who is.
- If you work with someone else, make sure you define your roles clearly.
Make sure each person feels they have equal say in the
- If you are a non-programmer working with a programmer, don't make your
programmer partner feel like hired help/slave labor.
- If you are a programmer
working with a non-programmer, share control--sure you'll have new ideas as
you code, but don't make major changes without consulting your partner.
- As you work on your project, keep careful notes about your design
process. Your papers need to document not just your final
design decisions, but the earlier decisions you made and why
you changed your mind.
- Make something your target audience
will enjoy. Learning is not like swallowing castor oil.
But remember to make the act of learning itself fun--don't sugar coat
around the learning.
- Have fun with your project. It's a privilege
to have the time to
make something interesting, and the time to think
seriously about what you're doing and why.
Questions welcome--email firstname.lastname@example.org