This page is being updated--readings are subject to change
CS 4665/7465:
Educational Technology:
Design and Evaluation
Time: | TTh 1:30-3:00 |
Location: | CCB 101 |
Instructor: |
Amy
Bruckman |
Office: | CoC 255 |
Tel: | 894-9222 |
Email: | asb@cc.gatech.edu |
| (Email is best way to reach me.) |
Office Hours: | Find me after class or email for an appointment |
Teaching Assistant: |
TBD |
TA Email: | |
Texts:
Required:
- Educational Research by L.R. Gay and Peter Airasian. Sixth edition.
New Jersey: Prentice Hall, 2000.
- Articles online
- Handouts
Recommended:
- Becoming Qualitative
Researchers by Corrinne Glesne. White Plains, NY: Longman Publishing
Group, 1999.
- Interviewing as Qualitative Research by Irving Seidman.
New York: Teacher's College Press, 1991.
Available at the Georgia Tech
bookstore, or click
here
to buy them from Amazon books with the profit from the sale going to buy
pizza for students in this and other classes.
Prerequisite
In this class, you immediately begin applying your knowledge of the
theory of educational technology to design a piece of software.
Consequently, you must already know something about it! To take this
class, you must have completed either:
- CS 4660: Educational Technology
- CS 6460: Educational Technology: Conceptual Foundations
If you have equivalent experience from classes in another department or at
another institution, speak to the instructor to request permission to
enroll. No one without a solid theoretical background in this
research area may take this class. (The issue is not just that you
will have difficulty--it's also difficult to have class discussions
if we have to stop and explain the basics to class members without
background in this area.)
Course requirements are the same for graduate and undergraduate students,
but I expect more depth in papers and projects by graduate students.
Focus of the Course
The course has two parts, each with a set of questions that we will be
addressing:
- DESIGN:
What is learner-centered design?
What are student needs for learning? How do you facilitate learning with
software? What design principles are in common use for educational software
and which should be used for a given project?
- EVALUATION:
How do we evaluate educational software? What variables should we be
considering (e.g., teachers, learning style, social context)? What methods
for evaluation should we use (e.g., ethnography, clinical interviews, log
file
analysis, pre-test/post-test)?
Objectives
Objectives of this course are for students to be able to:
- Design software about which you are willing to make explicit claims
about the intended student audience, the needs of that audience, how the
software meets those needs (including choice of media, structure, and
interface), and what should be learned from the software.
- Evaluate the software and describe which (of many possible) variables
were studied, why these variables were studied, how these variables were
studied (what methods), and why those methods were used.
So what are we going to do?
During the course of the semester, students will (alone or in groups of at
most two):
- Design a piece of educational software,
- Implement that software, and
- Evaluate it (with at least four users, preferably a whole class if
available).
There are no restrictions on the KIND of software (e.g., microworlds,
construction kits, intelligent tutors) implemented, as long as the
developers
are willing to claim that the software facilitates learning and to evaluate
that claim.
Most classes will be a discussion of one or two papers.
Wherever possible, I have provided links to the authors' home pages.
You are encouraged at least to browse through those pages.
Papers will be available either as handouts
or on the web.
You may wish to print web papers out before reading them.
Grading
Your grade will be based on these assignments:
- Preliminary software design (5%)
- Revised software design: (15%)
- Results of field work
- Revised software design
- Human subjects proposal
- Software development plan
- Midterm design assignment: (35%)
- Paper: your design and its pedogical foundations
- Presentation of software and its design
- Software
- Evaluation plan (5%)
- Final evaluation assignment: (35%)
- Paper: Results of your evaluation
- In-class presentation of results
- Class participation (5%)
Suggested assignment lengths are approximate. Some people will do a good
job
in less and some will require many more pages.
Please don't play games with the margins or fonts to try to make it come out
a certain length--no one is counting.
Don't worry about the page
count--just make sure that you've completed the assignment well.
Please remember to double space papers so I can write comments.
You will be graded primarily on the thoughtfulness of your design
and evaluation process, as documented in your papers and
presentations--the emphasis is not on
the quality of your software. If you are working in a group, each
team member must submit separate midterm and final papers. For
other assignments, one per team is sufficient. Each team member must
also participate in giving the in-class presentations.
Computing Resources
If you registered for the class and you already
have a CoC account, you have been given access to College of
Computing machine clusters (if you didn't have access already). If you
register late, you should be added later automatically. If not, mail
help@cc
asking to be given access.
If you need a CoC account, you can get a request form in front of CoC 213.
Complete the form and return it there.
CoC computing asks that:
- If your work can be done on an OIT machine, please do so. CoC machines
tend to be over-used. But if you need special software not available at
OIT,
using machines in this building is fine.
- Please do not use the SGIs unless you need SGIs specifically.
- Booklets on available CoC computing resources are available outside
of room CoC 140.
- CoC does not currently have a Mac cluster, but there are ones run by OIT
in
the Student Center (2nd floor) near the Music Listening Room and the
Library (basement). However, those may not have the development software
that you need. If you absolutely need to work on a Mac, come talk to me
and we'll see if we can think up a work-around.
- If you need special software that is not currently available to
you, please let me know immediately. It may take a while to get, or
may in fact not be possible to get.
Course Outline
All readings are subject to change.
- (1/8) Introduction: AquaMOOSE 3D as a case study in Design and Evaluation
Prof. Bruckman regrets that she will be at a conference this week--
Computer-Supported Collaborative Learning (CSCL) in Boulder, CO.
Syllabi will be distributed in class today, and Jason Elliott will
be available to answer questions. Should you have any questions about
the class, email the instructor at asb@cc.gatech.edu. You can also
include a phone number in your email. Prof. Bruckman will try to answer
all questions promptly, and will give you a call on the phone if you
wish.
Jason Elliott will show his "AquaMOOSE 3D" software and discuss
it as an example of educational technology design and evaluation.
- (1/10) No class
- (1/15)
Learner-Centered Design
Introduction, meet me and each other, start making teams.
Questions:
- What is design?
- What is learner-centered design?
- How is designing educational software different from designing other
kinds of software?
- Can you think of a learning experience of yours that was special?
What made it special?
- Can you draw on that experience in your design work?
- (1/17) Introduction to Evaluation
While evaluation is primarily covered in the second half of
the course,
you need to understand something about it in order to be able to design
software that you will be able to evaluate. This class will give
you an overview of the evaluation techniques that we will study in
detail in the second half of the course.
Reading:
- "Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings" by
Ann L. Brown (Handout.)
- (1/22)
Project Plans.
Come to class prepared to discuss your project plans and
progress so far.
Due: Assignment 1, Preliminary Software Design.
Please submit a roughly five-page project proposal including:
- The names and email addresses of your team members,
- A general description of what role each person will play within the
team,
- What learning goal you hope to meet with your software,
- A basic description of what your software will look like, including
sketches (which may be hand drawn),
- A description of the social context in which your software is designed
to be used, and
- A list of milestones for completing your project within the available
time frame.
Please bring enough copies of your project proposal for the whole
class.
- (1/24) Early Field Work/Needs Analysis
Reading:
- Glesne, Corrine. "Being There: Developing Understanding
Through Participant Observation." Chapter 3 in Becoming Qualitative
Researchers. White Plains, NY: Longman Publishing Group, 1999.
- (1/29) Research Ethics/Human Subjects Issues
Read carefully:
- Chapters 4 ("Establishing Access to, Making Contact With, and Selecting
Participants") and 5 ("Affirming Informed Consent") in Seidman.
- MOOSE Crossing permission forms
- "The ethics of research," and "gaining entry to the research site";
Gay and Airasian, pp. 93-104
Skim:
- (1/31)
Designing for an Audience
Reading:
- (2/5) Revised Design Proposals
Due: Assignment 2: Revised Software
Design.
- (2/7)
Constructional Design.
Reading:
Questions:
- How do you design a good construction kit?
- What do kids learn from working with
software like Logo, StarLogo, MOOSE, and the programmable brick?
- (2/12) Scaffolding
In class, I'll give a short talk on how to give a short
talk.
Reading:
- (2/14)
Case Study of an Educational Software Project:
Genscope
Reading:
- "Implementation and Evaluation of Genscope Learning Environment:
Issues, Solutions, and Results." Paul Horwitz, Joyce Schwartz, et al.
Procceedings of ICLS 98. Charlottesville, VA: AACE Press, p.
6-10. (Handout.)
- The Genscope Website,
http://genscope.concord.org
- (2/19)
Learning from Simulation
Reading:
- "Seductions of Sim" by Paul Starr. The American Prospect, Spring 1994,
p. 19-29. (Handout.)
- (2/21) Usability versus Learning Value
Reading:
- "HCI for Kids" by Amy Bruckman and Alisa Bandlow (Handout.)
- (2/26)
Student Design Presentations
Due: Assignment 3, Midterm Papers and
Presentations.
- (2/28) In-class design exercise
- (3/5 & 3/7) Spring Break
- (3/12)
Quantitative vs Qualitative Methods
Reading:
- Gay and Airasian, "Quantitative and qualitative research," and
"guidelines for classification"; pp. 8-23.
- Papert, Seymour. "Computer Criticism vs. Technocentric Thinking."
Educational Researcher, Jan-Feb 1987, pp. 22-30. (Handout.)
- Pea, Roy D. "The Aims of Software Criticism: Reply to Professor
Papert." Educational Researcher, June-July 1987, pp. 4-8. (Handout.)
- Walker, Derek F. "Logo Needs Research: A Response to Professor
Papert's Paper." Educactional Researcher, June-July 1987,
p. 9-11. (Handout.)
- (3/14)
Clinical Interviews
Reading:
- "Evaluating Instruction: The Complementary Use of Clinical Interviews"
by Fred N. Finlay. Journal of Research in Science Teaching 23:7,
635-650 (1986). (Handout.)
- Chapter 6, "Technique isn't everything, but it is a lot" in Seidman.
- (3/19)
Ethnography
Reading:
- "Thick Description: Towards an Interpretive Theory of Culture" by
Clifford Geertz
- (3/21)
Evaluation Discussion
Due: Assignment 4, Evaluation Plan.
Your evaluation plan should be double-spaced, in 12-pt font, and
approximately 5 pages long. Describe in detail how you will conduct
your evaluation. If you are working in a team, describe what each
team member will do. One plan per team is sufficient for this
assignment (but you must hand in separate midterm and final papers).
Please come to class prepared to talk about how you will evaluate your
software. Each group should present its ideas as succinctly as
possible--preferably in about 3 minutes, leaving another 2 minutes for
discussion.
- (3/26)
Sampling
Readings:
- Gay & Aairasian, Chapter 4
- (3/28) Measuring Instruments
Readings:
- Gay & Airasian, Chapter 5
- (4/2)
Attitudinal Inventories
Guest lecturer:
Jason Ellis
will talk about use of attitudinal inventories in his Palaver Tree Online
Readings:
- Jay, Gina M. & Willis, Sherry L. (1992) Influence of direct computer
experience on older adults' attitudes toward computers. Journals of
Gerontology, Vol 47(4), pp. 250-257 (handout).
- (4/4) Descriptive Research
Readings:
- Gay & Airasian, Chapter 8
- (4/9) Correlational & Causal-Comparative Research
Research
Readings:
- Gay & Airasian, Chapters 9 & 10
- (4/11)
Log File Analysis
Reading:
- (4/16) Experimental Research
Readings:
- Gay & Airasian, Chapter 11
- (4/18) Experimental Research, continued
- (4/23)
Early Experimental Ed Tech Research: Logo
Readings:
- Clements, Douglas H. and Dominic F. Gullo. (1984) "Effects of
Computer Programming on Young Children's Cognition." Journal of
Educational Psychology. 76:6. 1051-1058. (Handout.)
- Clements, Douglas H. "Effects of Logo and CAI Environments on
Cognition and Creativity." Journal of Educational Psychology. 78:4.
309-318. (Handout.)
- (4/25) Student Evaluation Presentations
Due: Assignment 5,
Final Papers and Presentations
Late Policy
Each student may have a total of three late days over the course of the
semester. Once you've used those up, late work will be penalized.
I always make an effort to return papers promptly; however,
late papers may be returned substantially later.
Your presentations may not be late. Your final papers may not be late.
Hints for Successful Projects
- In designing your project think small. The semester is
short. The important thing is that you are willing to make some
learning claim about your software, and to evaluate that claim.
- You are responsible for finding your own study subjects
for your evaluation in the second half of the course.
The easiest thing would be for you to design for college or graduate school age
students and ask your peers to be subjects. If you decide to
design for young children, you need to know a teacher
of that age group who is willing to work with you.
- If you are not a programmer, team with someone who is.
- Project teams should be all undergraduate or all graduate,
becaues the grading criteria for undergrads and grads are
different. Mixed
grad/undergrad teams require permission of instructor.
- If you work with someone else, make sure you define your roles clearly.
Make sure each person feels they have equal say in the
fun/important/interesting parts.
- If you are a non-programmer working with a programmer, don't make your
programmer partner feel like hired help/slave labor.
- If you are a programmer
working with a non-programmer, share control--sure you'll have new ideas as
you code, but don't make major changes without consulting your partner.
- As you work on your project, keep careful notes about your design
process. Your papers need to document not just your final
design decisions, but the earlier decisions you made and why
you changed your mind.
- Make something your target audience
will enjoy. Learning is not like swallowing castor oil.
But remember to make the act of learning itself fun--don't sugar coat
around the learning.
- Have fun with your project. It's a privilege
to have the time to
make something interesting, and the time to think
seriously about what you're doing and why.
Questions welcome--email asb@cc.gatech.edu