In many domains, especially engineering and computer science domains, practitioners are required to make complex connections between an object in the world and a conceptual representation of that object, called a model. In order to make those predictions, the practitioner needs to have a deep understanding of the relationship between the model and the external object. Despite the ubiquity and usefulness of modeling in actual practice, students are rarely explicitly taught modeling skills. One reason why modeling skills are not presented to students is because of the complexity and potential expense involved. The problem addressed by this research is the inability of students to perform modeling tasks, and the complexity of current modeling and simulation tools that are supposed to improve student’s modeling. The approach of this research is to create a modeling and simulation environment that focuses student activity on the most critical aspects of model building: the connection between the model and the object it represents, and the implications of the model for future behavior of the real-world object. Students using these tools will have their performance on modeling tasks compared to the performance of students who did not use the tools. In addition to performance, measures of student’s modeling process will also be taken. The general contribution of this research is a theory for creating modeling and simulation environments that increase student learning by emphasizing the interactions between a system and its model.

In many domains, especially engineering and computer science domains, practitioners are required to make complex connections between an object in the world and a conceptual representation of that object, called a model. The practitioner creates this model because it has properties that make the model easier to work with than the real object. These properties may include a formal representation structure, simplification or removal of parts of the real object, or a more easily grasped visual representation. This enables the practitioner to make predictions about the behavior of the original object based on the behavior of the model.

In order to make those predictions, the practitioner needs to have a deep understanding of the relationship between the model and the external object. The model may not encapsulate all of the behavior of the object, or the representation of the model may need to be translated to make sense in terms of the real world. An understanding of the connections between model and object is critical for successful use of the model.

Despite the ubiquity and usefulness of modeling in actual practice, students are rarely explicitly taught modeling skills [1]. Even activities that are designed to simulate authentic activities, such as labs and design projects, often fail to focus on modeling skills. As a result, even students who have a strong theoretical understanding of a domain are often unable to apply their knowledge in practice. Manifestations of these misunderstandings could include designs that are more expensive then necessary, designs that will fail due to unaccounted-for constraints, or software designs that are brittle and hard to maintain.

The most effective way for students to learn modeling skills is to be explicitly engaged in creating models, a process called model-building. Model-building focuses the student’s activities on the important tasks of defining a model and making the connections between the model and the original object.

One reason why modeling skills are not presented to students is because of the complexity and potential expense involved. This expense comes in several forms:

• Cost of equipment used in creating, displaying, or simulating models
• Cost of environment needed to support model-building
• Student time needed to learn model-building and to create and modify models
• Instructor time needed to create class curriculum to support modeling

Computer environments have been used in an attempt to reduce equipment cost by simulating the activity of equipment that would otherwise be too expensive or dangerous. The critical cost of using a computer-based model-building environment is student time. Model-building and simulation are a very complex tasks, and without some form of simplifying support, they have too much overhead to be integrated into an existing course – the students are forced to learn modeling formalisms that are secondary to their actual learning goals.

This research explores the creation of computer based modeling and simulation environments that are simple enough for students to be able to use them within the framework of existing courses, and yet complex and rich enough for the students to be able to learn from the modeling. Two such environments are discussed. DEVICE is a modeling and simulations environment for Chemical Engineering students that has already had success in improving student performance. BOOST is modeling and simulation environment for undergraduate College of Computing students, which will support the students in Object-Oriented modeling.

A model is a conceptual representation of a real object. Models are necessarily simplifications, however they can be used to make predictions about the behavior of a real object. Computer-based modeling environments attempt to connect complex systems in the world to the computational environment in which a student is learning. The student is engaged in model-building, the explicit creation of a model in order understand a real world object or system. The model can then be executed, creating a simulation of the original object. The simulation can be used to induce properties and behavior of the object in a variety of situations.

Because the creation of a model requires a deep understanding of the real-world object, modeling and simulation environments hold great promise as learning tools. Previous work has shown that learning is not encouraged if students merely observe the workings of a simulation environment [2]. They must be able to inspect, create and modify the underlying model. However, the creation of a computational model requires the student to learn some sort of formalism to communicate the model to the computer system. This is a very difficult task in itself – roughly equivalent to learning a new programming language, then writing a complex program in it [3]. Usually, learning the programming language or formalism is not the goal of the instruction, but is just distracting overhead. This research addresses the question of how the model-building process can be simplified while still providing a powerful enough environment from which students can learn.

This research makes the following claims:

• Students using model-building environments will improve performance and learning in domains where it is important to connect to the external world.
• A successful modeling and simulation environment can focus student activities on two areas of the model building environment – the connection between model and object, and the composition of model pieces into a more complex whole.
• Other parts of the modeling process, such as the initial decomposition of the model, can be simplified without affecting student learning.

In order to evaluate these claims, two computer-based modeling and simulation environments will be studied.

DEVICE and BOOST are the modeling and simulation environments that will be evaluated during this research project. Both will be described in more detail below.

DEVICE (Dynamic Environment for Visualization in Chemical Engineering) is a computer environment designed to support undergraduate Chemical Engineering students in modeling tasks. DEVICE has gone through two iterations, and the current version has been used in two separate offerings of the same Chemical Engineering course. Student activity in the current version of DEVICE is centered on linking pieces of a mathematical model to the real-world attributes that make up that model. Student’s using DEVICE improve performance on sample problems by about 30%. However, the earlier version of the program, where students did not explicitly connect model and object, did not increase student learning.

BOOST (Basic Object-Oriented Support Tool) is a modeling and simulation environment to be designed and built to support students in learning object-oriented modeling. Students using BOOST will learn a model building process that will allow them to connect their real-world problems to the formal methods of Object-Oriented analysis and design. Further, the learning of this process will improve the quality of student models and designs in ways laid out below. The improvement in design quality will make the students better Object-Oriented programmers.

The goals of BOOST are for students to learn:

• The basics of Object-Oriented modeling. The different types of interactions between objects. The way those interactions are represented in a formal notation.
• How to create good Object-Oriented models. The implications for design and reuse are for a set of interactions. Why certain types of interactions lead to better models. What we mean when we say that a model is better.

The open research questions answered by this project are:

• Can a modeling process like the one used in DEVICE be applied to Object-Oriented programming?
• Are the type of simplifications used in model-building with DEVICE applicable to other modeling domains? Are they effective in those domains?
• Do students who use this modeling approach to Object-Oriented design create better designs?
• Do students who use this modeling approach to Object-Oriented design write better programs?

Discussion of work related to BOOST will be divided into three categories.

• Non-computational efforts to use modeling and design in education
• Computational model-building environments
• Commercial and semi-commercial environments designed to help practitioners with Object-Oriented modeling.

The purpose of this discussion is to situate BOOST as a model-building environment and to elaborate on the contribution of this research.

The term model is a heavily overloaded term, (for example, see Guzdial 93)[4]. For the purposes of this research, the unmodified word model will always refer to the concrete models being built by the students. The model of the domain that is being created in their head will be referred to as a cognitive model. Beliefs about the nature of student learning will be referred to as the learning model.

Any model is "a surrogate object, a conceptual representation of a real thing."[5] There are four parts of a model:

1. A set of names for the objects and agents represented within the model.
2. A set of descriptors or attributes that represent properties of the objects.
3. A set of formal descriptions of the behavior of the model.
4. An interpretation relating the pieces of the model (and its output) to the real world properties that the object represents.[6]

A simulation extends a model by allowing it to be executed, usually in a computer-based environment. The student can see the results of the simulation, which should represent the behavior of the object being modeled. In most cases, the student has access to details of process that would be difficult or impossible to gather from a real-world experiment – the instantaneous velocity and acceleration of a moving object, for example. The simulation should allow the student to change parameters and attributes more easily than in the real world, to understand system performance under different conditions. This is changing the system at the simulation level, or tweaking. A tweak is a more superficial change to the system than a change at the model level, which would involve actually changing one of the four parts of the model listed above.

One effort to integrate modeling into teaching without technological support was proposed by Hestenes for physics instruction[7]. He claims that physics professors take the basic grounding of the field for granted, and fail to teach them, not realizing that those basics are essential for developing models and theories based on those models. In particular, he claims that it is critical for students to be explicitly exposed to the interpretation of a model – the meaning of a set of equations, for example. Without the interpretation, Newton’s Laws are merely algebra.

Another familiar non-computational modeling domain is algebraic graphing of functions – the graph is a model of the underlying equations. A survey of research in that domain again suggests the critical importance of connections between the model and the thing being modeled. This can include the definitions of line and point, as well as the fact that a line on the graph is equivalent to an algebraic function[8].

Cognitive Apprenticeship is a framework for learning environments that includes modeling as a key component.[9] In that framework, modeling refers to a skill being carried out by an expert, while describing their thought process, in order that a novice might build an internal model of the task being performed. Although that modeling process does not map explicitly to this research, the Cognitive Apprenticeship framework also discusses methods for supporting students in complex cognitive tasks. These methods include coaching, offering immediate feedback to students as they perform their task, or offering heuristic strategies that the student can apply to the domain. Another method is scaffolding, or explicit support intended to allow students to perform tasks they would otherwise be unable to do, with the expectation that the scaffolding is not permanent.

Design, in and of itself, is a powerful activity for learning. Since design is a nearly ubiquitous activity, especially in engineering domains, it makes sense for students to participate in design activities.[10] Design and modeling tasks complement each other, with a design activity often acting as the motivator for a model to be built. There have been a number of research efforts showing that the design of artifacts for communication causes student learning to increase dramatically.[11] Such design necessarily includes the construction of a model as at least a first step in the design.

There are three types of computational environments that are relevant to BOOST:

1. Programming environments that are designed to be easy to use or to specifically build a certain kind of program. These environments, such as Boxer and StarLogo, can be used for model-building even though they do not explicitly support it.
2. Pure simulation environments, where the user does not have direct access to the underlying model. These systems, such as SimCity, are not model-building environments. However, they can allow a student to explore a model that might be too complex for the student to build.
3. Model-building environments, where the student can create and execute a model. These environments typically need to manage the complexity of the model-building in some way. Examples of model-building environments include Emile, SimLife, Model-It, Stella, and DEVICE.

The following table illustrates the differences between the three groups.

A comparison of the two Maxis products, SimCity and SimLife, show an example of the difference between changes at the simulation level and changes at the model level. In SimCity, the user can tweak the simulation in a variety of ways – adding buildings to the city, changing the tax rate, connecting power lines and so on. However, the user cannot make changes in how the system models cities – changing the reaction to a tax hike, creating new types of buildings, for example. In SimLife, on the other had, the user has a much wider amount of latitude in the creation of new environments and new organisms, all the way down to the "DNA" of the simulated organism.

The distinction between a programming environment and a model-building environment is somewhat blurry. Certainly a computer program can be a model, however not all programs are models. I will be calling an environment model-building if it provides semantic support for creating models in a given domain or domains. Programming environments such as Boxer[12] and StarLogo[13] provide a great deal of flexibility for student projects, but they do not explicitly support modeling in a domain. Even though StarLogo was designed to allow exploration of a specific class of models, namely decentralized ones, it does not have any kind of support for a modeling process or domain knowledge. As a result, teaching students to use a new programming environment can be a very time consuming process.

Simulation environments take the opposite approach. In a pure simulation environment, such as SimCity, the student has no access to the underlying model. The student cannot even see the model, which describes the behavior of the real-world object. Although this can make for an entertaining game, previous research has indicated that restricting students to simulation-level changes in the model does not cause them to learn the properties of the underlying model (although it may act as a cue if the student already understands the model.)[14]

Model-building environments, then, are a middle ground between the complexity of programming environments and the passivity of simulation environments. A successful model-building environment must avoid being caught with the negative features of the other two groups, either by making its model-building component too complex, or by making the changes available to the student too trivial.

Model-It is a "tool for creating models and simulations of dynamic systems", specifically stream ecosystems.[15] Model-It provides a set of high-level objects, and student activity in Model-It is focused on defining the interaction between these objects (for example, the relationship between the quality of the stream water and the amount of oxygen in the water). Model-It scaffolds this process by allowing students to define these interactions qualitatively, and allowing the student to quickly explore a variety of different interactions.[16] High school students have been successful building and testing models using Model-It. Model-It is similar to BOOST in that the interactions between objects are important. However, students using BOOST will not be given objects specific to their particular problem. Instead, BOOST students will be given templates of common interaction patterns, and asked to apply them to the problem at hand.

STELLA (Structural Thinking Experimental Learning Laboratory with Animation) is a modeling program that allows students and practitioners to use a system dynamics approach to building models. STELLA uses a flow analogy to model systems, where flow represents the rate of increase of a value, and is designed to be applicable to a wide variety of domains. Although STELLA is easier for students to use than a plain programming language, it is not always clear to students how to translate from their domain to the flow model. There is no scaffolding in STELLA for student process or other features to help the student manage modeling complexity. STELLA has been used with some success in high school classrooms. The primary difference between BOOST and STELLA is that BOOST’s domain-specific structure makes it easy to include process support and components that make the initial complexity of the task less daunting.[17]

Emile is a modeling environment that supported students building physics kinematics models using HyperCard. Emile provided extensive scaffolding, both in process support, and in a rich component library that students could use to construct their models. The effect of the support was that students were able to program models with much less difficulty than they would have had in a plain programming language or even in STELLA. In fact, students were able to build complex systems without learning much programming – which is not a problem since programming is not among the student’s learning goals. BOOST is different from Emile in the nature of the modeling task – Emile is text-based and looks very much like programming, while BOOST will be a more visual model. BOOST is also different in the learning goal, teaching a form of design, while Emile taught a more constrained domain.

A summary of the previous work on modeling and simulation environments shows that the amount and nature of support needed to encourage learning is still an open question. On one end, it is known that too much support keeps the students from being engaged in the problem enough to learn from the experience. However, if the environment is too complex, the student is also unable to use it effectively for learning. Exactly which portion of a model-building activity is most salient for learning is unclear, as is the exact nature of support needed to focus student activity in those areas.

There are a number of tools aimed at expert software designers to assist them in Object-Oriented modeling. These tools range from Object International’s freeware tool Playground, to their commercial tool, Together/C++ to Rational Software’s Rational Rose. Without doing an exhaustive survey of these tools at this time, this section will focus on their common features. All of these environments (except Playground, which merely has the diagram) allow some sort of connection between Object-Oriented diagrams and working code, usually in both directions. Between these programs they have an exceptional range of features, and all these products claim to be easily usable by practitioners.

The difference between these tools and BOOST has to do with the target audience. BOOST is explicitly aimed at students, who are novices in the domain (even if the students have been exposed to an Object-Oriented programming language, they are most likely still novices in the design component). Because of that focus, BOOST needs to provide kinds of support that the tools aimed at experts will not need to have – basic definitions, for example. Students who have learned on BOOST would be able to transition to the more complex environments with much less difficulty than they would have starting out in the more complex tool.

DEVICE (Dynamic Environment for Visualization in Chemical Engineering) is a system designed to teach modeling techniques to Chemical Engineering undergraduates. The problem in Chemical Engineering was that although the students understood the algebraic equations that represented a chemical engineering system, they were not able to connect those equations to real world solutions. Students would design systems that were orders of magnitude more powerful and expensive than needed or systems that would explode if built due to real world constraints not measured by the equations.

DEVICE went through two versions. The first version, which was abandoned after initial pilot testing, assumed that merely being able to inspect the underlying model would be enough to allow students to gain an understanding of it. Students were shown the equation governing the system, and could see where the values in the equation had come from (although in practice, none did). They could not alter that behavior in any way. The display, as shown in Figure 1, allowed students to see the values of the equation, and also the values of critical partial steps towards the final calculation.

Lab studies with potential users suggested that this approach was ineffective in getting the students to explore the underlying properties of the model. In fact, the students quickly forgot all about the model, and used the above screen as an output device only rather than exploring its relationship with the system they were designing.

In creating the second version of DEVICE, we decided to force the students to be explicit about the connection between the equation and the system. Rather than teach the students a full-fledged programming language to describe their model, students were provided a modeling environment that used the algebraic equations with which they were already familiar. The DEVICE system allowed them to solve design problems in building pump systems using these algebraic models.

The students were asked to perform a four step modeling process. There is nothing specific to Chemical Engineering in this process; it is a domain-independent approach to model-building.

Although the second version of DEVICE took the students much longer to learn and master, their performance improved in pencil and paper questions when compared to students who did not use DEVICE.

The second version of DEVICE had a number of features that made learning from modeling and simulation possible. Those features that can be abstracted to BOOST and other modeling and simulation environments include:

1. DEVICE 2.0 provided students with an existing decomposition of the problem into equations. This worked partially because the specific decomposition used in this domain is reasonably standard. However, even in other domains where the problem decomposition options are more varied, it is useful to suggest possible decompositions.
2. Attempting to make the program easier by hiding the mechanisms of the simulation makes it nearly impossible for a student to learn those mechanisms. Ease of use can hinder learning in modeling and simulation systems.
3. Allowing students to explicitly connect the model to the real-world system is an effective method of eliciting reflection about how the two are related.
4. Students can learn to follow a modeling process with minimal support from the program.
5. Explicitly following the modeling process improves students performance and may allow them to better make connections to the real world.

BOOST is the modeling and simulation environment that will be constructed and evaluated as part of this research. BOOST will be deployed in undergraduate College of Computing courses that teach Object-Oriented Modeling. BOOST will enable the student to create Object-Oriented models in a language independent way and create the connections between the pieces of that model and the real-world problem.

In BOOST, the student is creating an Object-Oriented model of a real-world situation that will be the basis for the student’s program. BOOST models have the following properties:

Since the students using BOOST will be creating every aspect of their models, BOOST is a modeling environment, placing it in the lowest row of the table in section 2.3 above.

A simulation in BOOST is the execution of a student-created scenario of interaction between objects in the student’s model. For example, a point-of-sale system may have "conduct a transaction" as a scenario. The purpose of running such a scenario is to see if the interaction patterns that the student has set up in the model are capable of handling the interactions actually called for by the model.

Possible negative results of the simulation include trying to access a service that does not exist, or attempting to access a class where no connection has been specified in the model. Based on the above definitions, this is process output of the simulation. Therefore, BOOST is placed in the rightmost column of the table.

The BOOST environment, then, will allow students to create their own models within the domain in a way that has been successful in increasing learning in other systems.

In DEVICE, students used equations to model pumping problems, in the hope that learning the modeling would make them better at understanding the real-world constraints on choosing pumps.

In BOOST, students will use OO Diagrams to model a real world system, in the hope that learning the modeling process will make them better OO programmers.

BOOST extends DEVICE in the following ways:

1. BOOST takes the DEVICE process and applies it to a non-mathematical domain.
2. In DEVICE, students already understood the formalism. In BOOST, students will simultaneously be learning the formalism and learning to apply it.
3. In DEVICE, students were using an existing problem decomposition, and individual pieces of the model were more or less independent. In BOOST, students will need to provide their own problem decomposition, and the way a student handles any given piece will have a strong effect on the rest of the problem.
4. In DEVICE, students only worked on one type of problem, pumping systems. BOOST will have two levels of support. One, where the problem is unknown to the system, will allow the student to draw the diagrams. The second level, which would involve problem creation by a domain expert, could support the student in the OOA task in a more problem specific way.
5. BOOST will most likely offer both process support and coaching, pointing out common problems in OOA decomposition.
6. BOOST may also offer support for the student in programming – an activity with no real analog in DEVICE.

BOOST will take three separate approaches to mitigating complexity.

• Process support
• Component libraries
• Use of already familiar formalisms.

Supporting a student in a task that the student would otherwise not be able to do is known as scaffolding. In BOOST, scaffolding takes two forms. First, students need to know where they are within the process, what has and has not been done, and what a reasonable next step might be. Useful process steps will be placed in a "process library" that will be presented to the students. Secondly, the students should have some way of telling whether their models are flawed. There are two ways in which this might be accomplished, depending on how much knowledge the program has. The program may present a checklist of things for the student to watch out for, or it may actually analyze the design and make suggestions for improvement.

In Emile[18], students were presented with an extensive library of primitive pieces that they could use to build simulations of simple kinematics systems. These pieces took the form of HyperCard code that the students could cut and paste into their models. In BOOST, the component pieces will be Design Patterns. A Design Pattern is a set of object interactions that has been successful in a real world problem, presented such as to provide a template for future use.[19] A design pattern can be as simple as two objects. Students in BOOST will be able to insert patterns into their models.

One way to lower the overhead of teaching students a formalism is to use one that they already know and can easily work with. In DEVICE, students were asked to create models using formalisms that they were already familiar with, namely, algebraic expressions. Students were able to create models using that formalism, and performance on modeling tasks improved.

BOOST takes a slightly different approach. Whereas in DEVICE, the students were already conversant with the formalism when they entered the classroom for the first time, the students using BOOST learn their formalism – Object Oriented diagrams – as part of their normal classroom activity during the course.

This is an interaction scenario to show how a student might work with BOOST in CS 2390. Kermit is a 2390 student working on an assignment.

Kermit steps up to a web browser and hits the BOOST start page. The start page prompts for a login. Kermit enters a name and password. The system recognizes that he has not used BOOST before, and it prompts him to choose a project. On subsequent uses, BOOST will remember where Kermit left off, and will ask him if he wishes to resume from that point, or start a new project.

Kermit chooses a problem that the system already knows about, the Video Store design problem that has been assigned in class.

Kermit is then shown the main boost screen. The center area, where Kermit’s design will go, is blank (right now, I’m calling that the Playing Field). Outside the Playing Field on the left are boxes labeled things like Problem Component, Interface Component, etc... The boxes are also empty. Outside the Playing Field on the right is a box labeled Pattern Library, filled with things like "Participant -> Transaction". Kermit can’t make any sense out of that at the moment. Below the Pattern Library is a box labeled Strategy Library. Kermit can’t make any sense of that one. Above the Playing Field is what looks like a toolbar for drawing OO diagrams.

Kermit isn’t quite sure where to start. Luckily, there is a button labeled "Where to Start?". Kermit presses it. A dialog box suggests that since the purpose and general specifications of the assignment have already been entered that it would be a good idea to start selecting objects. Had Kermit not been working on a problem already known by the system, Where to Start? would have pointed to strategies for problem definition.

Kermit goes to the Strategy Library and clicks on a strategy for selecting initial objects. A box comes up prompting Kermit to enter a list people, places and things in the system. Kermit enters "customer, manager, clerk" for people, "store" for places, and "register, movie, CD-ROM, Nintendo" for things. Kermit misses a strategy for defining abstract objects like transactions, however.

Kermit returns to the main screen and notices that the lists he has entered have been added to the Problem Component box outside the Playing Field. Kermit’s clicks on the "what next" button. The system tells him that his goal is to create his OOA/D by creating classes based on the objects listed in the PC box (out-of-bounds). It suggests that he look through the Pattern Library for ideas on how to connect the objects.

Kermit starts scrolling through the Pattern Library. One pattern catches his eye. Participant->Transaction. That seems like it would be relevant. He selects it. A dialog pops up showing the two pieces of the pattern and suggesting possible applications (generally, no specific knowledge). Kermit applies the pattern.

Two classes, Participant and Transaction, show up in the playing field, in some way it is clear that they have not been connected to the design yet. Kermit connects customer to the Participant. He sees that he was wrong not to include a Transaction class, so he creates one tied to the Transaction part of the pattern. He reapplies the pattern for Clerk and Transaction. He decides that rentals are a slightly different transaction, so he presses a button in the toolbar to create a specialization class, named RentalTransaction. The system asks him what the specialization class knows that the base class did not. The new class shows up in the playing field.

Kermit continues to search the patterns, and applies the Transaction->Item pattern, and creates a new Item class. He then tries to create subclasses for movie, etc..., but stops when the system asks him what new things those classes know and he cannot add anything. This would be an example of failure in the design section. In the out-of-bounds area, those objects (movie, CD-ROM, Nintendo) are tagged as instances, rather than classes.

Kermit has now created classes for all the items on his list except store. He looks at the pattern marked Transaction->Place, but it doesn’t seem to add anything to his system so he decides not to include it. He tags the store as a "not in system" object.

At this point, Kermit hits the "What next" button again. The system suggests that he look at the attributes and services provided by the patterns to determine if they all apply. Kermit does this, and is able to simplify his model somewhat. The system then suggests that Kermit begin testing scenarios.

In this case the system already knows about a couple of scenarios. In particular, it knows about a customer renting video scenario, which starts with a transaction, and returns the price and how long the rental will take.

Kermit describes a series of messages to the system, specifying the sender, the receiver, the message, the arguments and the return value... Obviously a better format is needed for this...

This is a fairly simple scenario.

After Kermit enters in the scenario, he runs it. There is some sort of visual representation of the scenario working it’s way through the design in the playing field. The system will stop on getReturnDate, because transactions do not offer that service. In this case, that would most likely prompt Kermit to add a getReturn date service to the RentalTransaction object.

After going through this scenario and a couple of others, Kermit hopefully comes out of the system with a fairly robust blueprint for his programming. The option is here to create stub code, as well.

This scenario does not take collaboration into account. Kermit will be able to save his diagram as an html page that he can point to using WebCaMILE, as an anchor to collaboration using that tool.

CS 2390 teaches Object-Oriented design and programming. The students are presented with a process by which to create Object-Oriented programs. However, observations of the class show that the students do not appreciate the full value of the process presented. Further, even at the end of the quarter, students still exhibit misunderstandings about object models.

Students in CS 2390 are presented an object oriented modeling process as follows:[20]

1. There is an analysis activity (OOA), the purpose of which is to decide on a set of classes that models the problem domain, and the attributes and services for each class
2. There is a design activity (OOD), the purpose of which is to decide on interactions between classes, and scenarios of activity within the program
3. There is a programming activity (OOP), where the program is constructed.

Although there is no set order to the activities, the suggested practice is to do some OOA to get a rough idea of the classes, some OOD to understand the object interaction, then start programming. The programming is supposed to inform refinements of the OOA and OOD.

Based on informal interviews with students who took CS 2390 during Winter Quarter 1997, the most common student process looks like this.

1. The student performs a brief OOA. The amount of work the students put in at this point varies widely. Some students claim to do the entire OOA process in their head, but they do claim to do the process. Other students do draw simple diagrams. Relatively few students did detailed OOA diagrams at this point. Based on written work, the students have some difficulty giving principled reasons for the class distinctions they have made.
2. The student begins programming. Few students do much in the way of detailed OOD (scenario) work before starting to program.
3. The student goes back and writes up the OOA and OOD, reverse engineering as necessary. Well over 75% of students admitted to producing most of their handed in OOA and OOD work after the program was complete. One student posted to CaMILE the night before an assignment was due that "his program was 70% done, and his analysis was 30% done."

Based on these interviews, the CS 2390 students do not, in general, see the OOA and OOD activities as useful in their own right. (Further data about the reaction of CS2390 students to the OO process will be gathered as part of this research). Rather, to the extent that its not just something that they have to do to get a grade, they see the process as a way of convincing their instructors that they understand their own program. They do not seem to use the OOA or OOD as blueprints for their eventual program. In particular, they do not see the value of planning out scenarios. Student discussion indicated that they had trouble working at the proper level of detail for scenarios – they tended to work at too low a level, making the scenarios tedious.

Figure 4: A sample Coad format OOA design

The Coad & Nicola process defines three kinds of interactions between objects.

• Generalization-Specialization (inheritance): The specialization class inherits all the attributes and services of the generalization class, and adds new attributes and services of its own.
• Part-Whole (composition): The part class is conceptually needed to complete the whole class. An instance of the part class knows about at least one whole. An instance of the whole knows about at least one part.
• Sender-Receiver (generic connection): The two classes have some knowledge of each other, but are not conceptually part of each other.

The current trend in Object-Oriented modeling is to favor composition over inheritance, because it is a more flexible mechanism in most languages, allowing reuse from smaller objects.[21] 2390 students, however, along with most Object-Oriented programmers, tend to overuse inheritance as a communication mechanism.

Analysis of student final exams from the Winter ’97 quarter suggest that although students appear to have a basic understanding of both inheritance and composition, they do not have a clear understanding of how to combine the mechanisms to create a more complex, flexible model.

In the final exam, students were given an Object-Oriented model with 6 classes and 7 interactions. They were asked to correct flaws in the model. The results show:

• Just under 30% of the students did not create an abstract class in one of the two cases where one would have been useful, and just under 20% didn’t do it in either case. In another case, where an abstract class might have been useful (but where the student was creating the entire hierarchy), only 6% of the students did so. Subjectively, many of the students did not understand why an abstract class might be desirable in a design.
• About 20% of the students were willing to create two separate classes that each had attributes for name, email, and other personal information – a clear candidate for one reusable class. None of the students used composition in this case, even though it made for a far more flexible design.
• Students had a great deal of difficulty dealing with a flaw where inheritance had created a redundant attribute in a class. Only about 15% of the students handled the flaw optimally (by removing the redundant attribute). As many as 30% of the students made corrections that left the redundancy intact.
• Only about 60% of the students chose to use composition to add flexibility to the model.

All of these problems have a common root: an inability to understand, debug and create designs with complex interactions of objects. In addition there are the following concepts that student appear to have difficulty learning through the course of the quarter.

• The students tend to over factor and create too many classes. Often there is a one to one correspondence between classes and instances.
• It takes the students a long time to understand simple interaction between objects. The functionality of passing objects as parameters or holding other objects as attributes is difficult for the students to master.

This section discusses the evaluation mechanisms that will be used to answer the research questions listed above:

The BOOST experiment will compare the learning outcomes of two groups of students. The control group will be made up of students taking CS 2390 during Spring and Summer Quarter, 1997. These students will have been taught using current teaching methods and tools, without using BOOST. The experimental group will consist of students taking CS 2390 during Winter and Spring Quarter, 1998. These students will use BOOST to complete modeling assignments. A pilot study will be performed Fall Quarter, 1997, where only a small number of students in CS 2390 will use BOOST. This test will evaluate the consistency and usability of the BOOST software, as well as provide cues for important factors in the larger tests.

Pre-test data will be collected from all groups, to show whether the classes start from the same lack of knowledge about Object-Oriented design. Post-test data will be compared between the control and experimental groups, to show any differences in the final outcome of the experimental group. The nature of that data is described below.

In the DEVICE user test, three mechanisms were used to gauge students understanding of the modeling process:

• Students were observed informally using the program, and their paths through the system were observed.
• Log files of students’ interaction with DEVICE were taken, monitoring every action students took with the tool. This information was used to monitor the amount of time students spent in any one step of the program, as well as the order of student tasks.
• Students were given a post-test to recreate the steps used in solving a problem in DEVICE.

All three of these methods will be used in BOOST (informal student observation will be augmented with more formal observation and interview techniques).

In addition, students will be surveyed at the end of the course about the design process used during class assignments, to test if they are transferring the modeling process.

Data that would indicate a positive answer to this question might include:

• Observation showing that students use the program mostly by following the paths intended.
• Log files showing that students tend to order the process in coherent ways, and that they do not ignore useful system functionality.
• Post-test data indicating that students were able to internalize the modeling process.
• Another thing to watch would be the pattern of student usage of BOOST. If the students follow the modeling process, there should be substantial BOOST usage a several days before the assignment is due. If many of the students are reverse engineering their programs, then there probably won’t be much usage until immediately before the assignment is due.

There are two basic methods of evaluating student designs, neither of which is perfect. One approach is to create a normative model representing an expert conception of a problem, and explicitly compare student designs to that model. Another approach is to make several small quantitative observations about student designs, and compare one set of student data to another. The BOOST evaluation will use both of these methods.

You are designing an employee tracking system for a large corporation. The system will need to keep track of employees and projects. Specifically, the system must know for each employee:

• basic personal information (name, email, etc...)
• all projects that the employee is on or supervises (can be one or more)
• who the employee’s immediate manager is for each project.
• a list of any employees that report to this employee (if the employee is a manager)

Note that while not all employees are managers, all managers are employees (that is, all employees to report to somebody). Also, there is a hierarchy of projects as well – each project is part of a higher level.

For purposes of discussion, Figure 5 above shows a pretest problem given to students in the first control group.

The first evaluation would be to create an expert model and compare student models with that. An advantage of this method is that it gives each student design one number, making claims about individual designs possible. It is also the method that most explicitly measures student design performance.

A disadvantage of this method is that it is only as good as the expert model and the grading heuristics. Also, the baseline testing that has already been done suggests that student answers to a problem will range far beyond solutions that can easily be compared or anticipated beforehand. A related evaluation approach would be to identify a set of potential design flaws, and mark student work based on which flaws appear in a given design.

Figures 6 through 8 show an example of this process. Figure 6 is the expert model created for the test problem. The grading heuristic ranks student designs on a 0 – 100 scale based on the kinds of concrete classes the student creates, the relationships between those classes (including the creation of abstract classes as parents), the distribution of responsibility among classes, and mistakes in the actual drawing of the diagram. Certain classes and connections (such as an Employee class) are considered more important than others.

Figure 7 shows a student model that is missing a important class from this design. Among other problems, the diagram distinction between abstract and concrete classes is not observed. This diagram scored a 43 on the current version of the grading heuristic.

Figure 8 shows a different student design. This design does contain all the major classes, however the student has created an extraneous class, PartialProject, and has left off a couple of minor classes. The grading heuristic suggests that this design is somewhat better than the first one, scoring this design a 66.

The second method of evaluation would gather simple objective data such as the number of classes created, number and type of connections, and whether the connections accurately reflect the. The disadvantage of this method is that it does not allow for comparisons between individual student designs. However, it does allow comparisons to be made between groups of student designs. The research hypothesis would predict that BOOST students would have more and smaller classes (and more abstract classes), and more connections (and a higher percentage of composition connections. The table below suggests how common student flaws may be evaluated.

Subjective measures would include student surveys about whether they found BOOST useful, whether they felt that they learned something from it, etc. Another measure would be the frequency with which students use the program on their own, even when not required. Utility of specific features will also be queried.

Modeling is a skill needed by practitioners in a wide variety of domains. Students in those domains, however, do not acquire modeling skills as part of their formal education. Modeling and simulation environments have great potential as learning tools. However, modeling is a very complex task, and in order for a computer environment to successfully support learning through modeling, it needs to balance the difficulty of the student tasks carefully. Offering examples of good practice in the domain, as well as focusing student attention on the interpretation of a model in terms of the object it represents, are strategies that have been successful in modeling and simulation environments.

DEVICE is a modeling environment based on those principles that was built to support undergraduate Chemical Engineering students. Students using DEVICE have shown increased performance on Chemical Engineering Tasks. BOOST is a successor environment to DEVICE. It will support undergraduate Computing students in Object-Oriented modeling.

There are two contributions of this research

• The application of a modeling and simulation environment to Object-Oriented modeling, a domain which has been difficult for novices to learn well.
• A methodology for creating modeling and simulation environments that focus student activity on the steps most necessary for learning, without either making the environment to simple to be interesting, or too difficult to be useful.

During Spring Quarter, 1997, baseline data will be gathered from the 2390 class. This data will include.

• A pretest which gauges previous programming knowledge and tests basic awareness of terminology.
• Design problems on the midterm and final that will measure student ability to diagnose and repair flawed designs.
• Think aloud sessions with selected students at the midterm and at the final. The students will be asked to create an OO model. An interview session will also be conducted.
• All students in the class will complete a survey about design process

On the other track, this proposal will be approved, and design and coding of the first version of BOOST will begin.

Baseline data evaluated.

BOOST first version completed (this version may not be totally feature complete).

During Fall Quarter, a pilot test will be conducted with selected students from CS 2390 (perhaps around 10). These students will run through the intended 2390 BOOST curriculum. The purpose of this pilot test will be to gauge the learning value of the program, determine how best to deploy it in class, and find the most nasty bugs before testing on an entire class.

During Winter Quarter, a full test of BOOST will be conducted using all the students in CS 2390. These students will be expected to use BOOST to complete the design portion of their assignments. All of the data gathered from the baseline group will be gathered. In addition, log file and partial save information will allow measurement of how students use BOOST. Students will also be surveyed specifically about BOOST.

During this time, bug fixes will be made to the program. Functionality changes that become necessary based on the students response to the program will also be made.

Spring Quarter will be a repeat of the Winter Quarter test, using the newer version of the program.

During Summer Quarter 1998, final data analysis will be performed, and hopefully defense of research will occur.