EVALUATING SOFTWARE VISUALIZATION USE FOR COMPUTER SCIENCE EDUCATION
CS7390 WINTER '98
ASHLEY TAYLOR
INDEX
- Evaluation Methods
- Stasko and Lawrence
- Prolog SV's: Mulholland
- Gurka and Citrin
- Recent Work at Tech.
Empirical/Experimental
- Make Hypothesis
- Choose two or more equally sized groups of similar ability
- Provide experimental material to one group, conventional material to
control group for (usually) equal time
- Administer post-test
- Analyze Statistics
Observational/Naturalistic
- Record, analyze user behaviour
- Avoid artificial settings
- No control group
Prev Index
Next
Empirically Assessing Algorithm Animations as Learning Aids
Stasko and Lawrence
Study 1
Stasko, et.al. (1993) compared use by grad students of
- Text-only description of pairing heap alg. (45 mins)
- Text (30 mins max.) + Animation (remaining 15 mins or more)
Post-test showed non-significant advantage for animation group.
Animation group finished post-test slightly faster.
DESIGN FLAW?
"Most of the test items require the ability to accurately carry
out the main procedures of the algorithm, and neither presentation seemed
likely to give participants that ability."
Prev Index
Next
Study 2
Lawrence and Stasko (1994) compared several usage modes for undergrads
with Prim's minimum spanning tree alg.
- Lecture (video) + Animation vs. Lecture (video) + Static Figures
- No sig. diff. on post-test
- Lecture (each mode above) followed by either:
- Passive Lab - student runs animation using supplied input
- Active Lab - student interactively specifies input to anim.
- Significant advantage for Active Lab over Lecture-only group on free-response
questions.
Prev Index
Next
Issues Raised In Other Studies
- Stronger students benefit more than weaker students? [Whitney90]
- Students will utilize animations as resource, if made available [Kehoe96]
- Student animation designs are often different from 'traditional' designs
[Douglas95]
- Students like designing animations, think they help them understand
algorithm [Stasko96]
Prev Index
Next
A Principled Approach to the Evaluation of SV: A Case Study in Prolog
Paul Mulholland
Combines Empirical and Observational Approaches
"Much of the empirical work ... suffers from the problem of trying
to find global generalisations that are not there."
"Many of the studies derive performance results without deriving the
information necessary to explain them. Observations are usually not made
of how the subjects approached the task and what features they found confusing."
Prev Index
Next
STUDY
Compare four Prolog SV's using empirical + observational + cognitive approach.
- Cognitive Psychology students use Prolog SV's for learning, debugging
for a week.
- Students knew prolog previously
Test task - given program and source code, identify differences with
modified program (output only, no source code)
- Program designed to check backtracking misconceptions
- Pairs of students worked together to promote discussion
- Discussion recorded, analysed
- Limit of 5 min per problem
Program Differences:
- Change relation name
- Change atom name
- Change Data flow
- Change Control flow
Prev Index
Next
METHOD
8 pairs of students used each of four SV's:
| Spy |
Trace execution using Unification model |
| PTP |
(Prolog Trace Package )
Give more execution details than Spy
|
| TPM |
Transparent Prolog Machine Show execution graphically using depth first
AND/OR trees Provide overview, detailed views |
| TTT |
Textual Tree Tracer Trace using close format to source code |
Common Execution Environment (PPVL) used
for all SV's:
- Common interface, navigation
- Records user activity
Prev Index
Next
RESULTS
Main Types of Misunderstanding:
- Clause-Goal
- Data flow
- Control flow
- Timing
Total Instances of Misunderstanding:
| Least |
PTP |
|
SPY/TTT |
| Most |
TPM |
However, TPM best for Data Flow.
Prev Index
Next
Number of Problems Solved in Given Time:
| Most |
PTP |
|
TTT |
|
Spy |
| Least |
TPM |
SOME
CONCLUSIONS
- PTP use of indentation for control flow works well
- Spy provides less control info
- TPM good for execution overview, poor for details
- TTT caused clause/goal difficulties, time misunderstandings
Prev Index
Next
Testing Effectiveness of Algorithm Animation
Gurka and Citrin
Discusses
Issues related to evaluation of Algorithm Animation and Software Visualization:
"...it is possible that animations, as they are now created and used,
are not particularly effective for teaching algorithms. "
"Since
intuition suggests otherwise, it is important to continue research until
definitive results are obtained.... if we can show ineffectiveness, perhaps
the same research methods can suggest new modes for using animation that
will succeed."
Propose that animations can be used for:
- Lecture examples
- Studying
- As a tool for code reading and grading, discussion
- For students to construct own visualizations
Prev Index
Next
Observational Studies
- Do not usually address effectiveness directly
- Tend to rely on student perceptions, which are also mixed with usability
issues
Empirical Studies
- Often have no pedagogical substitute for animation in the control group.
- Any improvement may be attributed to the extra or alternative methodology
used.
- Use a human tutor for the control group?
Prev Index
Next
Animation factors for design and evaluation
1. Usability difficulties.
2. Animation quality, discussed below.
3. System training, including tutorials on exploration techniques.
4. System availability.
5. Type of animation use, i.e. lecture, lab, self-study.
6. Individual differences among learners, particularly exploratory and visual learning ability.
7. Algorithm complexity. Some require multi-level presentation?
Prev Index
Next
Each algorithm should be examined to see
- What is hard to grasp about it
- What special cases need to be highlighted
- What concepts need to be presented
- The best sequence to present the components of the algorithm Graphical
Design Principles from Art/Cinema should be used.
Multimedia design study
[Faraday and Sutcliffe 97] supports this.
Prev Index
Next
RECENT WORK AT TECH.
Grad Students try to learn Complex Data Structure
(Binomial Heap)
- Use Homework Format
- No time limit
- Given Problems, all Materials at outset
- Observe use of materials, use to solve questions
Materials
Text + Pseudocode + Anim
OR
Text + Pseudocode + Static Figures
PRELIMINARY RESULTS
- Animation Students did better on operational questions
- Non-animation students were more likely to give up
Prev Index
Next
Student Animation Construction
(Stasko, Newstetter, Taylor)
- Still need to understand Algorithm first (usually)
- Can take lots of time
- Students seem to remember algorithm
- Viewing Good Animation better than writing Poor Animation
Prev Index