CS 7390 - Software Visualization

Discussion Questions - Evaluation of SV: Experiments and Studies

• Even after looking at these studies, how do we *really* know whether or not these viz's really work?
• We still have the "appropriateness" question. How do we know what areas we can/can't use viz's in? And why are they used predominantly in algorithms animations and nothing else? And predominantly in computer science and no other fields?
• Is the lack of non-computer-related viz's due more to the difficulty of programming them, or the fact that they haven't been shown, to some degree, to be effective in an inter-disciplinary type setting?

• It would be interesting to see how a group of professors who are experts in the topic would evaluate the instructions of the two different lectures having one with animation. Perhaps after gathering the feedback and re-designing the lectures, deliver them to students and conduct the experiment again and see what kind of results we would get.
• Besides education, are there empirical studies done in other domains, for example in real-world systems?
• In regards to the experiment of different prolog systems, do you think the results may also depend on how the students learn prolog in the first place?

• Could a better animation toolbox help overcome some of the pessimism in using algorithm animations or is this a problem in pedagogy which will require a review of techniques and procedures using these techniques ?
• Could detailed usability studies using controlled groups of subjects provide better empirical evidence rather than depending on classroom subjects ?

• It is believed that using SV techniques in lab assignments will significantly help students to better understand lectures.  Is it widely adopted in practice now ?  What has been done for it ?
• I did not get the ideas of some experiments conducted in evaluating Prolog SV packages, such as the ones for information content and strategies.

• Before empirically assessing algorithm animations as learning aids, it is important for researchers to screen out students and algorithms, so that the study is assured on a fairly reasonable baseline. At the same time, it will also simplify the upcoming result analysis and quantification. However, for a generality concern, can this study also be carried out in such a way? That is, 1) At the beginning, all the students (no matter what level they stand in the class) are asked to finish a set of questions prepared for this study; 2) Instructor or researchers collect the questions for later assessment; 3) Instructor or researchers show the animation in the lecture, or the students play with the animation in the lab; 4) The students are asked to do the same questions again; 5) Instructor or researchers collect the answer; 6) Two sets of answer sheets are graded, analyzed and compared. This methodology may offer a comparative study of the roles the animations play in aiding learning. Of course, some assumptions should stand, such as students do the questions cooperatively based on their knowledge, and the questions should be well designed to reflect different levels of understanding.
• It is a good practice to have students involved in developing algorithm animations, since it will surely enhance their understanding. The question is, what is the better way to incorporate it into the curriculum so that their time can be utilized better?

• Why haven't more empirical studies of software visualization been done to determine where and how it is useful? Is this because empirical studies in software engineering in general aren't too common yet?
• Wouldn't it be interesting to see if, although the students may not do significantly better, if the algorithm animations facilitate discussions through common images, and if they facilitate longer remembering of the essence of algorithms? It seems to me like these would be true.

• On page 435 it is stated that "... stronger students ... appeared to benefit more form the animations than poorer students." I am wondering if animations tend to reinforce or disprove pre-conceived notions rather than evoke some fundamental understanding. Poorer students have less to reinforce whereas better students simply true-up their understanding.
• In the statistical evaluations, was there any consideration give to the quality of the UI or the familiarity of the student with such UIs? Could these factors affect the statistics significantly?