Janet L. Kolodner

College of Computing

Georgia Institute of Technology

Atlanta, GA 30332-0280

jlk@cc.gatech.edu

In Learning by Design™ projects, students iteratively reflect upon their hands-on experiences to achieve design challenges as scientists and engineers do. In addition to learning science contents, they learn how to design fair tests, justify test results with evidence, and explain their findings scientifically. In this paper, we explain how and why our project separates itself from other hands-on projects, where learning tends to be more superficial, and propose that the iteration of on collaborative design challenges plays a key role for thein learning both content and scientific skills learning.

Our design of LBD’s classroom sequencing and ritualized activity structures is based on advice offered by Case-Based Reasoning (CBR) (Kolodner, 1993, 1997; Schank, 1982, 1999), an approach to building intelligent computer programs that can reason based on their experiences. CBR was inspired by human reasoning. The many software prototypes built by the CBR community helped us to understand two things — how to build more intelligent computer systems and the reasoning needed to learn effectively from experience and reuse lessons learned in new situations. The ability to learn for transfer is a chief aim in education; hence, we believed that applying lessons learned in our CBR research to science education could result in more deep and lasting learning, both of science content and of the skills and practices engaged in while learning. CBR’s advice about learning is consistent with the literature on learning from transfer (see, e.g., Bransford et al., 1999) and adds to it specificity about processes involved in learning. Learning by Design is consistent with the advice offered by CBR and the advice offered by the learning for transfer literature. The LBD experience provides an example of how what we know about human learning can be used to design learning environments that promote deep and lasting learning of science content and practices.

Learning by Design™ has been piloted and field-tested with over 3000 students and two dozens teachers in the Atlanta area, covering a full spectrum of student backgrounds and capabilities. We have developed a year’s worth of LBD units spread over physical and earth sciences. Our physical science units, which cover forces and motion and mechanical advantage, focus on fair testing in the context of designing and running experiments and designing working artifacts. Our earth science units, covering a wide variety of topics in geology, focus on investigation through modeling and interpreting real-world cases. Learning by Design has been piloted and field-tested with over 3000 students and two dozen teachers in the Atlanta area, covering a full spectrum of student backgrounds and capabilities. These are an example of how what we know about human learning can be used to design learning environments that promote deep and lasting learning of the science practices.

In this paper, we specifically focus on how this curriculum helps students develop science skills. We first schematize the activity sequence in LBD, and then explain it in the context of a concrete curriculum challenge (the Balloon Car Challenge in the Vehicles in Motion unit), showing how the iteration of towards better design solutions challenge designs enhances students’ acquisition of scientific skills. Our assessments reveal that LBD students’ ability to engage in science and collaboration skills is quite a bit more advanced than that of their comparisons (Kolodner et al., 2002, 2002a2003a, 2003b). Under space constraints of this paper, we will not explain Case-Based Reasoning, our base learning model, nor the computer software for scaffolding relevant activities. Fuller discussion of these will can be found in Kolodner et al. (2002a).

Figure 1 shows how activities in LBD are sequenced with each other. Notice that students engage in design challenges (left cycle in Figure 1) to motivate the need to investigate, and that investigation (right cycle in Figure 1) is engaged in when a need arises as they are designing. The solid arrows in the counter-clockwise direction show that sequencing is normally in that direction; the white arrows in the other direction show that sometimes there is cycling backwards and between several steps before going on.

We have identified seven novel features in LBD (see the complete list in Kolodner et al., 2002, 2003aa), from which we focus on the following two for the purpose of this paper.

A sample walkthrough with of the Balloon Car Challenge will illustrate the LBD’s features. Notice in the walkthrough how iteration towards better solutions provides opportunities for iteration towards better understanding and how sharing experimental results, design ideas, and design experiences promotes focus on learning of scientific reasoning.

The Balloon Car Challenge comes two months into the school year and is the second sub-challenge of our Vehicles in Motion unit, where students learn about forces and motion in the context of designing a vehicle and its propulsion system that can navigate several hills and beyond. The Vehicles in Motion unit has five mini-challenges, the Coaster Car Challenge, the Balloon Car Challenge, the Rubber-band and Falling-weight Challenge, and Final Car & Antarctica Challenge. In the Balloon Car Challenge, the students are challenged to design and build a propulsion system from balloons and straws that can propel their coaster car as far as possible on flat ground. For each challenge, the following sequence iterates to provide the students with ample chances to learn science content and skills.

Addressing the challenge begins with gaining a better understanding of it (top of the Design/Redesign cycle), first through "messing about" in small groups and then "whiteboarding" as a class to identify ideas.

Messing about is exploratory activity with materials or devices with mechanisms similar to what will be designed to quickly identify important issues that will need to be addressed. Here students mess about with a variety of balloon-and-straw engines. Some have larger balloons, some smaller; some have longer straws, some shorter; some have wider straws, some narrower. Having experienced messing about several times in prior challenges and becoming adept at it, the students are expected to know that they should quickly examine how each material work in several different ways.

Next the teacher helps the class decide which of the issues are the most important to investigate. They might then have a discussion about good experimentation, reminding themselves of what they’ve learned about experimentation during earlier activities. Indeed, tThhere may be a poster on the wall with "fair test rules of thumb" generated during previous activities and with such entries as "To insure a fair test, make sure to run procedures exactly the same way each time," and "To insure a fair comparison, keep everything the same except for one variable."

Each group of students now takes responsibility for investigating one of the questions and then designs and runs an experiment to find an answer. It is usually the end of a class period at this point, and for homework, the teacher assigns individuals the responsibility of designing an experiment that will assess the effects of the variable they are investigating. As they are designing their experiments, students use a "Design Diary" page that prompts them on what to pay attention to in designing and running an experiment and collecting data (Puntambekar & Kolodner, 1998) (Figure 3).

Students get together in small groups the next day, comparing and contrasting the experiments they have each designed as individuals and designing an experiment that takes the best of each. One student may have remembered that multiple trials are needed while another grappled with which variables to control. It is rare for a single student to consider everything needed for a fair test. The teacher also makes her way around the room, providing help as groups need it.

Students spend a day or two designing and running their experiments and collecting and analyzing their data, and at the beginning of the following day, each group prepares a poster to present to the class. They show their experimental design, data, and data interpretations, and they try to extract out advice for the class from their results. Each group presents to the class in a "poster session." Because students need each others’ investigative results to be successful balloon car designers, they listen intently and query each other about experimental design and procedures and gathering and interpretation of data (much as in a professional poster session). This provides an opportunity to discuss the ins and outs of designing and running experiments well. For example, in running balloon car experiments, groups often fail to make sure that they blew up their balloons exactly the same amount each time. The class might discuss ways of making sure that balloons are inflated the same way each time — by counting breaths, by measuring diameter of the balloon, and so on. Often, even though students have already had some previous experience designing and running experiments, some groups’ results are not trustworthy yet, and they redo their experiments and the cycle of activities just described. Figures 4 and 5 show typical posters.

When the class agrees that results most groups have come up with are believable, the teacher helps students abstract over the full set of experiments and experimental results to notice abstractions and extract out "design rules of thumb," e.g., "By using double walled balloon engines, the car goes farther because a larger force is acting on the car." Experiments provide learners the opportunity to experience and record phenomena, but they don't necessarily understand why those phenomena are happening. To learn the explanations behind these phenomena, the teacher makes some relevant reading available about the science content involved, performs demonstrations that exemplify the science concept in another context, and reviews and discusses student generated examples of the science concept, usually from everyday life experiences. Here the issue is why a bigger force would cause the vehicle to go farther if the total amount of air coming out of the balloon is the same no matter how many balloons are used. The balloon car works best when a high velocity is achieved because most of the distance gained is during coasting (after the balloon engine has exhausted its air). If you are coasting to zero velocity, then the higher your velocity is when you start to coast, the greater the distance you will travel, provided that the floor has low and near-constant friction during the coasting. Thus, students need to obtain achieve high acceleration with their engines. To understand this, the class spends time discussing both Newton's Third law - equal and opposite forces - and Newton's Second Law - about how changes in force and mass can change motion - and use that science to explain how their balloon-powered vehicles behave. Later iterations to the rule of thumb produce more informed and complete statements, e.g., "By using double walled balloon engines, the car goes farther because a larger force is acting on the air inside, so then an equally large force from the air acts on the car". (Example of how Third Law gets woven into the context of the Balloon Car.)

Upon returning to the design/redesign cycle, the class briefly revisits the whiteboard and specifies the constraints and criteria of this challenge (understanding the challenge). Next, students plan their balloon car designs, using the combination of experimental results the class has produced, design rules of thumb extracted out, and scientific principles read about and discussed as a class. The teacher makes her way around the room making sure that design decisions the groups are making are based on being able to justify based on the experimental results they’ve seen and what they know about forces and not simply on the loudness or popularity of a group member. (In general, students tend to design cars with two or three or even four balloon engines, often the balloons are doubled, and they tend to use wide straws or several narrow ones attached to each balloon).

Each group prepares another poster, this time presenting their design ideas along with the evidence that justifies each decision and their predictions about how it will perform. They present to their peers in a "pin-up session." During this activity, justifying decisions using evidence and making predictions are the primary foci, and after groups present to their peers and entertain their peers’ questions and suggestions, the class as a group discusses not only the ideas that everyone has presented, but also the practice of justifying, what it means to identify and use good evidence, and making predictions (They’ve had much experience with this while working in small groups).

Justifications during this pin-up session tend to refer to both the experiments that have been done and the principles about combining forces that were discussed earlier (e.g., "We decided to use two balloon engines because Group 3’s experimental results showed that the more balloon engines, the more force will be exerted, and the farther the car will go. We didn’t use more than two because we couldn’t figure out how to blow up more than two balloons at a time. We also decided to double the balloons on each engine because Group 4’s experiment showed that double balloons make the car go farther. We think this is because a double balloon exerts more force on the air inside the balloon, providing more force in the direction we want the car to go.") If justifications are truly are qualitatively better than during a previous pin-up session, the teacher might point that out to the class and ask them if they know why they were better and use the results of that discussion to update "justification rules of thumb" or the "justification working definition," e.g., "Justifications can refer to experimental results or to scientific principles; if they refer to both, they will convince people more than if they refer to just one."

Students now move to the construction and testing phase, modifying their designs based on what they’ve discussed in class and heard from their peers, and then constructing and testing their first balloon-powered engines. They use another design diary page here, this time with prompts helping them to keep track of their predictions, the data they are collecting as they test, whether their predictions are met, and explanations of why not (Figure 96).

None of their balloon cars work exactly as predicted, sometimes because of construction problems and sometimes because of incomplete understanding of scientific principles and the results of experiments. After working in small groups to try to explain their results, the class engages in a "gallery walk," with each group’s presentation focused on what happened when their design was constructed and tested, why it worked the way it did, and what to do next so that it will perform better. The teacher helps students state their explanations scientifically and calls on others in the class to help as well. Hearing the explanations students are generating allows the teacher to identify misconceptions and gaps in student knowledge, and discussions about those conceptions and gaps might follow the gallery walk, along with readings or a short lecture or set of demos to help promote better understanding. This gallery walk is also followed by classroom discussion abstracting over the set of experiences presented, and revisiting and revising design rules of thumb, correcting them and doing a better job of connecting to them explanations of why rules of thumb work. Discussion also focuses on the explanations students made, and the class might generate the beginning of a working definition of "scientific explanation," e.g., "Explaining scientifically means providing reasons why something happened that use science terminology and include the science principles we learned about earlier."

Following the gallery walk and ensuing discussions, students make their way again around the Design/redesign cycle, revising their designs, based on explanations their peers have helped them develop and the new things they’ve learned. They construct and test their new designs, each time using another "Testing My Design" Design Diary page, and each time participating in a gallery walk with the rest of the class. and tThey iterate in this way towards better and better solutions until they or the teacher decide they have gotten as far as they need to. Discussions after second and third gallery walks often focus on "fair testing," this time not in the context of an experiment but rather in the context of comparing results across different designs. Students will not be able to explain why a new design works better than an old one if they have changed more than one thing in their design since the earlier time. Nor will they be able to believe their own results if they don’t follow the same testing procedures each time. Sometimes teams discover these issues as they are testing a later design and report them to the class; sometimes a peer notices that a test was done differently than last time or, if results are confusing, asks how a test was done and helps a group discover inconsistencies in their procedures. The "fair test rules of thumb" may be consulted and refined, as might the working definition of "scientific explanation."

The entire activity takes ten to twelve class periods. At the end, the class holds a final gallery walk and a contest, and then they compare and contrast across designs to better understand the scientific principles they are learning, going back to the rules of thumb to revise and explain them better. They finish up, as well, by discussing their collaboration experience, their design process, their use of evidence, and so on, and revising the rules of thumb and working definitions about each. Following all of this group work, each student writes up and hands in a project report — including a summary of the reasoning behind their group’s final design and what they’ve learned about collaboration, design, use of evidence, and so on.

If you go back to the details of activity sequence in Section 3, you will find several supports to scaffold this learning.

We’ve evaluated learning among LBD students using a variety oftwo methods. (1) We’ve tested students on physical science content knowledge and general science skill knowledge using a standardized written test format. It was administered in a pre-/post- implementation design in LBD™ classrooms and comparison classrooms. Items on the test were drawn from a number of normed and tested sources, including the National Assessment of Educational Progress (NAEP), released items from the Third International Mathematics and Science Study (TIMSS), the Force Concept Inventory and the Force and Motion Conceptual Evaluation, as well as items of our own. (2) We have designed the test items so that, by cross referencing subsets of the questions, we can pinpoint the conceptual model used by the students and see how it has evolved. We’ve also been testing students in both LBD and comparison classrooms for changes in content knowledge, science skills, and the ability of students to solve problems in group settings using two performance-based assessment tasks, both adapted from the PALS database (SRI, 1999) with the addition of an initial design activity on each assessment. These assessments were extensively videotaped. We have developed a tool for analyzing the tapes for both collaboration issues and use of appropriate scientific knowledge to solve the challenge (Gray, Camp and Holbrook, 2001).

Our results are quite interesting. Comparing LBD students to matched comparisons, we’ve found in two separate years of data collection that LBD students do at least as well as comparison students in content mastery and sometimes often better (Kolodner et al., 2002, 2003a, 2003b). When we analyze the results from the various individual teachers, we find that the largest gains among our LBD students were in those classes that are the most socio-economically disadvantaged and who tested lowest on the pre-test. Interestingly, these classes also had teachers with less background in physical science. We also see a trend towards re-engaging girls in science. Preliminary scoring on our 1998-99 data shows that while girls score lower than boys, as a rule, on pre-tests, they are equal with boys or ahead of them on post-tests. While we would have liked to have seen LBD students master the content far better than non-LBD students, and we believe they have, we don’t yet have the data to show that, as the kinds of questions asked on short answer and multi-choice tests often don’t provide good ways of distinguishing different levels of content mastery.

* = p < .03; ** = p < .02; ***= p < .01

N= groups where most groups consisted of 4 students each.

(Means are based on a likert scale of 1 - 5, with 5 being the highest rating)

Reliability for the coding scheme ranged from 82-100 percent agreement when two coders independently rated the tapes. For this set of data, a random sample of four - five tapes were coded for each teacher from one class period. Approximately 60 group sessions are represented in this table, representing 240 students.

The "we" in this paper includes many wise people who have helped create, evaluate, and extract lessons from Learning by Design™, including Jackie Gray, Jennifer Holbrook, David Crismond, Paul Camp, Barbara Fasse, Mike Ryan, Lisa Prince, and the many teachers we’ve worked with. Our funding for this work came from the National Science Foundation, the Woodruff Foundation, and Georgia Tech’s College of Computing. Thanks to Naomi Miyake for making sure this paper got written.