Toymaker: Learning by Designing

---

Objectives

These disparate subjects are tied together by giving the student a job to do that combines elements of all these subject areas. The student is told that he or she is CEO and head designer of a major toy company. Giving the student this kind of executive role gives the student control over the environment and allows for many different kinds of problems to solve. (Pea, 1987) Control over the final goals of the program was considered important to the kind of learning desired. Also, since all of these types of problems would come up naturally in the course of this job, the variety of subjects seems authentic and not forced.

The role also contains an implicit goal; although whether that goal is to make quality toys, make money, or something else entirely is deliberately left up to the student. The design task given to the students -- making a toy -- was highly motivating, and the self-directed goal added to this motivation. (Schank, 1992, Malone, 1981) When designing a toy, the student takes into account the internal stress caused by the weight of the toy's components, and creates a toy that is structurally sound and yet interesting enough to be purchased. The student also decides where in the United States to sell the toy and how best to market it. The student receives authentic feedback in the form of sales charts as well as letters from customers, satisfied and otherwise. Using that feedback, the student is then free to go back and iterate the toy design. The program emphasizes the process of design, rather than focusing solely on the end product. A preliminary version of Toymaker, containing most of the functionality described above, was tested in a second grade class in April, 1994.

Methods

The prototype used in the April test started by prompting the students for their names, after which they were given a brief written explanation of their task (Figure 1).

Figure 1: The startup screen

Upon entering the program fully, the students were placed in the main design screen, where they were able to put their toy together by assembling components. The assembly was constrained to force the toys to be physically realistic, at least in the sense that the bottom of the window was the "floor" and toy components were not allowed to float in mid-air. An attempt to place a component that violated this constraint resulted in no component being placed. The students were able to modify the color and other characteristics of the toy (Figure 2).

Figure 2: Toymaker design screeen

Before leaving the design screen, the students were required to test the stability of their toy. This was done by clicking on a component of the toy with a testing tool. The students were told if the toy was in danger of breaking at that point, or whether that component was solid. This information was calculated by simulating the forces and torques of the various components and comparing them to a value representing the maximum strength of the components (Figure 3).

Figure 3: Stress analysis

When their design was complete to their satisfaction, the students were shown a map of the United States, and asked to choose three regions in which to sell their toy. The map showed weather conditions and population in each region, and both pieces of information were significant in the computing how well the toy sold.

Figure 4: Marketing Map

After making these decisions, the students were shown a graph representing the sales of their toy, and a figure showing how popular the toy was among those that had bought it.

Figure 5: Visual Feedback

They were also shown one or two letters from customers. These letters were intended to hint at various strengths and weaknesses of their design by showing the toy's effect on hypothetical customers.

Figure 6: Feedback from a satisfied customer

The students were then able to start again, either on a new toy or modifying the existing toy. Future versions of the program could incorporate other subject matters by allowing the students to set the sale price of the toy, writing advertising copy, and responding to customer letters.

Data Source

The first screens that the students encountered took their names, and then gave them a brief description of what they would be doing in Toymaker. The students had no

problems typing in their names. Those students who were comfortable reading did tend to actually read the information screen and understand what it said. Their future interaction with the program, however, seems to indicate that this information was not

retained . Any verbal instructions given to the students before the program were also not utilized when interacting with the program.

The students were tested in pairs. We used the conversation between the students as a substitute for a talk-aloud protocol in trying to assess their understanding of the program, believing that the students would be too self-conscious if they were alone. The teacher chose the parings deliberately, allowing for student needs and often pairing higher ability students with lower ones. After each child had had one turn, students that had particularly interesting interactions with the program were asked to play solo, while they were questioned in some detail about their understanding of the program.

The turns lasted a half-hour each. This was enough time to gauge student interest in the program, but not enough time to accurately assess any learning that came about because of it. The test was conducted in a corner of their normal classroom. During their turn, the students were monitored by one of the designers, with the teacher occasionally checking on progress. The monitor functioned as a manual help system, answering student questions and providing assistance if the students got stuck. Participating students were interviewed informally following their turn, and were asked what they remembered about the program, as well as their likes and dislikes. The whole class discussed the program once all members had had the opportunity to use the program. They offered suggestions for the next version, and commented on parts of the program they enjoyed.

The program also kept a log of every action performed by a student during their session. This log allowed for a qualitative analysis of student performance and strategies.

Student Activity

On a superficial level, the students did not have much trouble manipulating the program and in most instances could figure out how to get the tools to do what they wanted. The students were very familiar with drawing programs such as Kid Pix and the graphics workshop in MediaText. The design screen was seen by them as a similar type of drawing program, with a few extra rules. The design screen enforced constraints on the placement of objects to ensure some degree of physical reality. Joints had to either be on the "floor" or touching a stick -- no floating in mid-air. The rules for other objects were similar. Students often placed their first joints in mid-air, much as they would in a drawing program. The program responded by doing nothing. This was somewhat confusing at first, but with a bit of help, the children quickly learned the rules for placement of objects.

Student performance with the constraints improved more or less steadily over time. This is backed by analysis of log files which recorded student actions as well as by. The students tended to violate the constraints rather grossly at the beginning, attempting to place objects in mid-air. After the first couple of minutes the errors fell into a couple of categories. Off by one pixel errors were common in placing sticks and rectangles, caused by a misconception of what touching another object actually meant. Another common error was an attempt to jump out of the system, best shown by the girl who tried to put a smiley face in the upper corner of the screen to represent the sun. A clever use of the tools, and she was not violating the physical constraints as she saw them; but

the program still wouldn't let her do it. In one case, a program bug forced a girl to recreate a toy that she had just finished making. According to her log file, the second pass was done in half the time, and with far fewer errors. Additionally, those students who got a second chance at the game remembered the rules and performed better their second time through.

Although the students became adept at working around the constraints, they saw the rules as arbitrary. We feel that a flawed interface was the primary reason the students were unable to perceive the physical reality that would have given meaning to the rules. Since the objects on the screen were completely static, the students had no reason to think of them as anything other than a drawing. A more dynamic screen could establish that reality. Adding a floor or table graphic to the bottom of the screen to establish a ground point might also alleviate part of the problem.

The bottom of the design screen displays the stress and cost of the toy. Most of the second graders ignored this display completely, despite reading a description of it before starting the design. The only students to be conscious of these figures were the fifth graders. That group found the display funny, since they had decided to build "a computer that does your homework", and after getting the outline of the computer (a bunch of sticks and joints) on the screen, their toy only cost about a dollar. The fact that the stress reading on this display told the students if their toy was structurally okay was not used by any group, despite the fact that the display turned red when it was showing a stress overload. This lack of awareness of the screen environment may be a problem in continuing development of Toymaker for second graders. One possible solution would be to follow the model of games such as Wagon Train and Oregon Trail where decisions pop up and must be handled before students are allowed to proceed. This would force students to think about the issues ignored during testing while allowing them to stay in the design screen where they apparently felt most comfortable.

Supporting the idea that the students responded to Toymaker as a paint program, the students were very reluctant to leave the design screen. Even students who were explicitly told that there were other parts of the program to explore refused to leave the design screen. One student who had been told this responded to leaving the design screen by saying, "Do we make another toy now?" Some student groups were given gentle hints to move on, also with little effect. This may be due to a lack of trust in the program, that it would allow them to come back. It may also indicate that they saw the design screen as the entire reality of the program. The students did not know where else they would go, knew they had a limited time, and had been anxiously waiting for their turns. They felt comfortable in this screen and perhaps did not want to learn something new and not be able to master it.

Toymaker was supposed to teach an iterative design process. The students were to create a design, place it out in the world, and then refine it in response to the feedback received. However, no student group ever went back to revise a previously created toy after leaving the design screen. In fact, the students iterated their design in subtler ways during one use of the design screen. Students were anxious to change the colors in their toys. They were less excited about changing material, since they saw little point to that. They also added components, discussed with their partners, and changed the designs during the one session. Many groups deleted there designs all the way back to nothing at some point during their session. Once they exited the design screen, though, they were pleased with their revisions, considered the toy a finished product, and were anxious for another chance to start on a new one.

The standard Macintosh color changer was used. The students thought it was "cool" and liked exploring it. They were frustrated because there was no way to match colors to other objects. One girl got around this difficulty by clearly choosing colors she would be able to find again, on the right angles of the circle, or on clear color boarders. Some groups had problems with the order of the actions needed to change color, after deciding to change the color of an object it was unclear where the color choices would come from. This may be because while other aspects of the design screen resembled known programs the color changer was new to them, as was the experience of having the colors on a different screen from the object being created.

Another major component of the Toymaker process involved the students testing their toys to discover where they were having stress problems. When students went to test their designs, they tested the joints with a cursor shaped like a hammer and the sticks with one shaped like a barbell. They would receive a message that the object being tested was "OK" or that there was too much stress on that part of their toy. In response to this screen the most confident answer Erin received in post-interview about the meaning of "stress" was that it had something to do with messed up hair.

The students did not test their toys often. The second graders probably did not notice the button labeled "Test Toy", and would not have tested their toys at all except that they could not exit the design screen without doing so. Again, following the Wagon Train model would ease this problem, using a pop up test screen that would have to be dealt with before proceeding. No second grader spontaneously tested their toy during their first session with Toymaker. The fifth grade group, however, did do this, as did some of the second graders who got to try the game again. No group was able to use the stress display that was always on the screen as a cue to test their toy.

Once they did enter the test phase, most groups either randomly tested one or two things, then left or they methodically tested each stick and joint at least once, sometimes forgetting which ones had already been done. Students did notice the difference between the okay and not okay messages, but were confused by the meaning of a toy that "could break". The students did not have any idea at all what to do in response to a not okay message. Often, they showed no interest in modifying their toys at all. They liked their toys and had already spent quite a bit of their allocated time working on and revising them and the stress message was too abstract for them to change in response to. Related to not seeing the toys as physical objects, the students also did not see the "could break" message as related to anything they had done, as shown by the following conversation:

Observer:  Why are you testing your toy?	
Student: To see if it might break. 
Observer: Why would it break? 
Student:  Someone might drop it. 
Observer:  What cold you do that would make the toy more likely to break? 
Student: Hit it with a hammer. 
[Student gets a could break message on a joint] 
Observer:  What will you do with that joint? 
Student: Fix it. 
Observer: How? 
Student:  Take it off. 
Observer:  What else? 
Student:  Glue it? 
Observer:  You can say I don't know if you don't know. 
Student:  I don't know. 
Observer:  Which other joints might be about to break? 
Student:  I don't know.

The "I don't know" response of the student indicates her inability to think of her toy as a three dimensional model. If this student had been playing with a Tinkertoy set, similar in many ways to the stick and joint tools, she would have been able to fix a problem if her toy was about to break. The same student responded to an earlier message about breaking by saying that "changing the color" of the object would fix it. Another student was closer in saying that one side was more stressed because "the stick was longer". The students had no idea to respond to these stress messages, and very little idea of their meaning. One solution would be for students to do some hands on work in conjunction with this program. Using a set of Tinkertoys or even marshmallows and straws would allow for concrete visualization of the items on the design screen and help the students handle "could break" messages as a part of construction not an external problem.

After leaving the design screen, students were asked to name their toy, and to tell the system if the toy was used indoors or outdoors. Students then looked at a map of the U.S. and planned where to market their toys. Even though it was explained that these would be places that the toys would be sold, and that the weather and population on the map would affect sales, most students went about placing their factories haphazardly. Almost every group recognized their home state on the map and, therefore, chose the South as a factory location. From there choices were somewhat random. Only about half of the second graders even looked at the weather information and even fewer at the population figures. Only one group saw the relationship between this information and the placement of their factories, and was able to make optimal choices. In post-interviews, the students were able to remember where they had placed their factories, often remembering the states and not the regions named in the program. The map was a good reinforcement of geography topics covered in second grade.

At the beginning of their next turn the students saw a start screen like the one in Figure 8, showing sales and popularity for each of their toys. The students did not react to the admittedly imperfect graphs on this screen at all. It did not seem to be a lack of understanding , but that they were uninterested in further suggestions. They were already happy or unhappy with the toy they had made, and didn't think the computer's opinion was that important. This indicates that students were unaware of the greater scope of the program. A better understanding of the purpose of Toymaker may come as the program expands to include elements initially proposed.

Students were prompted to check their mail at this point in the program. They would not have seen this button on their own, but almost all students could later recall precise details from the letters. Many of them were entertained by the contents of their mail. However, despite the retention of the information, the letters did not lead the students to reconsider any previous decisions regarding their toys. Nor were they able to extract the information contained in the letters regarding improvements. Students had already done as much editing as they wanted to do before leaving the design screen the first time, and they were not interested in further revisions. Some saw it as an opportunity to start over because they did not like the first toy created.

Results and Conclusions

The tools given by the program were easily usable to the students. Despite constraints on component placement which may have seemed arbitrary, the students were quickly able to use the tools to do build what they wanted.

There was some difficulty on the part of the students in believing that the toy had a physical reality. They had a lot of trouble believing that these objects had simulated weight, and were confused by the nature of the "stress" that they were testing their toys for. This is a serious problem in continuing development on this program. Although these were concepts that the students were unfamiliar with, we believe that the real problem was in the program itself. The static display of the design screen gave the student no reason to assume that the objects had weight. In the next version of the program, objects that are overstressed or placed in midair will dynamically fall to the ground. Allowing these objects to behave as though gravity applies to them should go a long way towards creating the illusion that they are real objects. (Laurel, 1991)

The students were reluctant to leave the design screen to explore other parts of the program. In part this is because the other parts of the program were relatively underdeveloped in the program as tested. Mostly, this seems to have been a result of their interest in the design itself; they didn't want to leave an exciting environment. In the next version of the program, the students won't have to leave the design screen to make other decisions. Instead, the decisions will be brought to them by "employees", while they are designing. This will also help solidify the students position as head of a major company.

The students did not want to go back and revise a toy they had already finished. This was not because they were not revising their designs. Rather, it was that they were revising their designs while they were in the design screen. Once they made the choice to leave, the toy was exactly what they wanted, and they were not interested in the computer's opinion of whether it sold well or not.

References

Duckworth, E. (1987) The Having of Wonderful Ideas. Teachers College Press, New York.

Laurel, B. (1991) Computers as Theater. Addison-Wesley Publishing Co. New York.

Malone, T. (1981) "Toward a Theory of Intrinsically Motivating Instruction" Cognitive Science 4.

Papert, S. (1980) Mindstorms: Children, Computers and Powerful Ideas. Basic Books.

Pea, R. (1987) "Putting Knowledge To Use".

Schank, R. (1992) "Goal-Based Scenarios" Northwestern University Institute for the Learning Sciences Technical Report #36.