Discussion questions from CS 7390 lecture

Date: 2/16/96

Presented by: Jeremy Heiner

Topic: Empirical Studies of Software Visualization

• Both of these papers discuss the benefit of soft viz/algorithm animation in terms of learning. However, many of the papers about software visualization systems have also mentioned their potential benefit in research. How well does software visualization (and, to the extent it applies, algorithm animation) potentially aid researchers? It this testable?
• I have a few complaints and comments bout the study that was conducted in the Stasko paper. First, why are graduate students the subjects of this experiment? Is this the level of education sought after? Second, why wasn't the algorithm animation explained thoroughly for the subjects that were to use the animation? Under a timed exposure, some orientation could approximate a normally paced self-taught session. And finally, why were the scores on the operation section so poorly answered by all subjects? This seems to imply that this section was not valid for comparison. Could the section have been easier?
• What are the effects of algorithm animation with respect to long term retention? If I view an algorithm, will remember it after some time?
• Is there any empirical study on how software visualization assists in learning?
• Would it be possible to give partial credit on the test given on the algorithms? It is possible that the group with the animations gor more of the problem correct, but not 100%, so it was counted as totally wrong.
• The bug correction/suggestion methods were very simplistic (read() not in loop, etc.). What would it take to be able to do this for more complicated bugs involving pointers or other difficult programming constructs?
• The Stasko paper says that maybe if the students had been provided with sample questions before learning the algorithm they would have known what to look for and done better on the test. Isn't this just an artificial way of boosting the overall scores as I don't see how knowing the kinds of questions in advance helps them actually learn the algorithm better.
• I don't understand how the figure in the Eisenstadt paper (figure 7b) actually helps a student figure out what their bug is.
• How can we evaluate the correctness of an algorithm animation a novice studetn builds during learning the algorithm? The evaluation process may bring in the supervised learning element into the algorithm learning process, which defies the purpose of using the animation.
• Toy programs and scaled up programs require different level of detail in visualization. How does one automatically decide which level should be used? Maybe the system can provide visualization tools at all levels and allow the user to decide what to use.
• Isn't it true that even if a study was done based on a single algorithm animation showing theat the animation helped people learn the algorithm better (than without the animation) conclusions could only be drawn about this particular aniamtion, not a general sweeping statment about all algorithm animations? Proving or showing that results are conclusive is hard enough in the single animation case, and that much harder when speaking about animation in genral. Is there any study or test that could be performed to allow generalizations? How is this type of similar problem handled in other fields?
• A specific question about a remark in the Stasko et al paper. One of the comments about how to improve test results of the participants was to give them an idea of what types of questions will be asked in the test. Would this unfairly skew the results, making them somewhat less valid?
• Have any empirical studies been done of the effectiveness of algorithm animation in areas besides teaching? (eg program understanding, documentation, or algorithm development). What about the effect of developing algorithm animations on algorithm understanding?
• The Eisenstadt paper talks about the need to provide visual support that will be meaningful and useful both at the introductory (learning) and advanced (practical, large system implementation) levels. Is this really necessary? If animations and visualizations help people with abstractions and mappings, isn't that more easily transferrable even without the visualization software?
• In the Eisenstadt paper, the while loop examples they gave were pretty simple. Does this concept carry over easily to tree recursion, abstract data structures, and pointers?
• I don't know if I agree that building software that points out where the bugs are helps students learn programming. Conventional debugging tools may help to some extent, but I feel that students learn more by finding the bugs themselves.
• Concerning the Stasko study, wouldn't it have been useful to gather results from a variety of "exposure" times, eg. 30, 45, 60 and 90 minutes---it seemed that 45 minutes wasn't enough for most of the students.