Ten Years of the AAAI Robot Competition and Exhibition: Looking Back and to the Future Tucker Balch and Holly A. Yanco (to appear, AI Magazine, Spring 2002) "This won't be a slick, polished competition. There will be a certain amount of chaos, but I can guarantee there will be a lot of excitement and enthusiasm." -- Tom Dean 1992 Origins In 1992 Tom Dean and Pete Bonasso convinced AAAI to host a robot competition at the National Conference on AI; the AAAI Mobile Robot Competition and Exhibition was born. The event has endured to become the oldest AI-centric robotics competition in the world. As we near the end of our first decade it seems worthwhile to to reflect on the event's origins, how it has evolved, and where it is headed. Pete Bonasso recalls that the entertainment industry had laid some of the groundwork for the contest by building expectations and excitement about robots in many people's imaginations. Characters like Star Trek's Commander Data and Bishop the "artificial person" in James Cameron's Aliens made us all think about what might be possible. The concrete idea for a competition took root after a panel at AAAI-91 on household robots. "Neats and scruffies alike were mesmerized by the animal-like responses of the robots demonstrated there," says Bonasso. "At the end of that panel a bunch of us got with Tom to talk up the idea of a competition." Then president of the AAAI, Patrick Hayes hoped the contest would highlight the "cognitive possibilities for mobile robots," and demonstrate "the longstanding symbiotic relationship between AI and robots." Of course some felt there is more than a symbiosis between AI and robotics; that there could be no intelligence without embodiment [Brooks 1991]. Over the years the event and AI Magazine have served as a venue for this and several other intellectual debates, including: sensing versus modeling, color-based versus shape-based object recognition, and reactive control versus symbolic planning for robot navigation [Mason 1993, Miller 1993, Balch et al 1995, Buhmann et al 1995, Firby et al 1996, Sargent et al 1997, Dudek 1998]. Towards the end of the 1980s, the apparently stalled progress of symbolic AI for robot control, contrasted with Brooks' and Arkin's successful reactive strategies, seemed to foreshadow a dominance of behavior-based approaches in the 90s. In 1992, David Miller and his son Jacob Milstein made the point that symbolic representation may not always be necessary with their stark robot Scarecrow. Scarecrow was controlled by a simple circuit of wires and relays (no computer controller). While Scarecrow didn't win the competition, it did well, making some competitors quite nervous. Miller relishes the memory: "For at least five years after '92, I received email and letters from researchers around the world asking for more details about Scarecrow's spatial representation and planning system. I would tell them its spatial representation system was 100\% accurate and full fidelity ... the representation was quite large and was stored mostly off-board the robot." In fact, robots following reactive strategies often claimed the winner's circle (e.g. [Balch et al. 1995, Balch 1998]). However, in other years, robots that relied on representation and planning stole top honors (e.g. [Buhmann et al 1995]). The end of a decade of competitions has seen a mellowing of the "religious" fervor on these issues, as well as a realization among many that reactivity, planning, and representation will all play a role in the successful robots of the new millenium. Evolution The contest immediately took on two important but apparently conflicting roles: First, it provided a target for research in AI and robotics; in Pete Bonasso's words, the event was cast "in the spirit of trying to develop as animate, responsive, and intelligent robot behavior as possible." Second, the contest provided a venue where people could demonstrate their work to the AI community. Contest organizers faced the challenge of devising compelling problems worthy of world-class research (and the funding that goes with it) while balancing risk. The risk arises from the danger of setting the bar so high that no robot can complete the task. In the worst case, critics might interpret this as a systemic failure of AI robotics itself. The contest chairs have wrestled with these issues ever since. The competition's evolution reflects the organizers' efforts to keep the bar at the right level while also providing relevant contest events that inspire and drive research. For the first contest, Bonasso and Dean designed a competition with progressively more challenging trials. The contest centered on a task where the robots were expected to explore a large arena containing easily detected obstacles along with conspicuously marked objects to be located by the robots. Initial trials were conducted behind closed doors, before the general public was admitted to the competition hall. This enabled the judges to assess the capabilities of the robots so they could tailor the final, public, trials for success. And a success it was -- the 1992 competition was a big hit that ensured a place for the competition in subsequent conferences. The competition grew to include multiple events, since some researchers might find one task or the other a better match for their funded research. Since that time the Robot Competition has included about three problems every year (sometimes four, sometimes two). Over the next three years spanning 1993 to 1995, some form of navigation task was maintained, along with a simple manipulation task. The complexity of the competition tasks, however, was kept at about the same level as the first contest. Another aspect of the contest that has evolved over the years bears mentioning. The types of participants have changed over the years. In the early 1990s a large portion of the competitors came from research universities like CMU, Stanford, Georgia Tech and MIT. The entries from these universities often reflected the research of Ph.D. students and their advisors. In the mid to late 1990s the mix of participants evolved to include a larger proportion of undergraduates from a broader selection of universities, including for instance, winning entries from teaching colleges like Swarthmore. This trend probably reflects the pervasiveness and accessibility of AI and robotics technologies. Recently, with the addition of new events we are seeing research unversities return to the competition. "Where is the science?" --Tom Dean, 1994 "The science is what you do [back at the lab, but] throw away the night before the competition." -- Jim Firby, 1994 The robots were providing an exciting show, but the researchers were still struggling to build robust systems that would convincingly complete the competition tasks. According to Tom Dean, one consequence of promoting such competitions and their co-location with scientific conferences is that the organizers and participants were constantly asked to explain the science under the hood, by peers, fellow conference attendees, by the powers that sponsor such events (DARPA, NSF, NASA, AAAI, IJCAI), and by the press. "Indeed," Dean says, "I was often placed in the position of 'spinning' comments made by participants that were picked up by the press to the effect that the real competitive advantage was a result of finessing tough problems, taking advantage of loopholes in the rules, and last minute hacks." Consequently, Dean took it upon himself to play devil's advocate and to challenge the community to explain their contributions to science, engineering and society. With regard to Firby's quip, Dean says "I'm sure that elegance suffered in the dark dawn prior to the final rounds of several competitions ... but Jim's achievements were still due in large part to a deep understanding of very basic principles and years of hard won experience cached out in well designed systems." Then, in 1995, the community reached an important milestone. According to David Miller, that year's organizer, "all the robots that competed in the events accomplished the tasks." The successes of 1995 were enabled by corresponding research progress in the technology areas the contest was pushing, including: overcoming ultrasonic sensor noise for navigation, building maps from noisy range sensors, real-time vision, and planning in a more or less structured environment. 1996 marked a move to more dynamic and unstructured tasks. The "Tennis Court Cleanup" task required robots to collect numerous tennis balls strewn about the arena and to deposit them in a bin. The problem was further complicated by the addition of battery-powered, quickly moving "squiggle balls" that must also be captured and delivered to a bin. One of the competitors, M1 from Newton Labs, was able to chase a squiggle ball down and capture it (see Alan Alda's quote in the sidebar). This competition was challenging and visually appealing as well. This year's event also brought the reactive versus deliberative control debate to the fore; of the two top entries, one, M1 from Newton Labs was purely reactive while the other, Jeeves from CMU, used symbolic reasoning. Others outside the community were taking note of the competition's success. Pete Bonasso recalls "some of my favorite recollections were seeing people like Ian Horswill, Maja Mataric and Jon Connell having to patiently explain to CNN and other press reps that there really was no one behind the curtain." In 1996, the PBS television show Scientific American Frontiers asked if they could visit and tape the contest. The show's host, Alan Alda, attended the competition in Portland, Oregon, and spent many hours observing and asking questions with his camera crew. The resulting program presented AI robotics in a very positive light. By 1997, the robots were reliable enough that another research challenge could be added. The "Hors d'Oeuvres, Anyone?" event debuted in that year's competition held in Providence. The general idea was to have robots serve food to attendees at the conference's banquet. This competition would drive research in manipulation, navigation in dynamic environments (e.g. crowds of people), but most notably, human-robot interaction. Because evaluation for scoring included an audience vote component, robots would be rewarded for compelling interaction with people. In addition to motivating research within the AAAI community, the competition has had a strong impact on AI robotics research outside the competition arena. Two of the most visible examples include the tour guide robots developed by Illah Nourbakhsh's group and Sebastian Thrun's group [Nourbakhsh 99][Thrun 2000]. According to Thrun, "our tour-guide work was clearly strongly motivated by similar tasks in the mobile robot competition -- in fact, without the competition, we would never have had the resources to do the museum tour-guide work." Research in robotic rescue is another example. The AAAI Robot Competition hosted resue competitions in 2000 and 2001. Some of the same robots that participated in these events were employed at the World Trade Center site in September 2001 [crasar2001]. Variations on maze navigation problems (e.g. the office delivery task) played a prominent role in the Robot Competition from the first contest through 1998. In 1999, however, the organizers deemed this a solved problem and dropped it from the competition. At about the same time, some felt the competition had drifted from its original objective of showcasing and driving AI robotics research. It seemed that robots participating (and winning) the completion had shifted from "state of the research" to "state of the art." In other words, the competitors were building their robots by integrating proven technologies -- they weren't being forced to develop new technology. At the same time, graduate student teams demonstrating their thesis research were being supplanted by undergraduate students demonstrating their engineering skills. These observations led to the development of a new event, the Mobile Robot Challenge. The intent of this event is to set the bar again at an appropriate level to draw cutting edge research back to the contest. The Challenge task is to develop a robot that can attend the National Conference on AI and present a talk about itself. Components of the problem include: human-robot interaction, navigation in dynamic environments with and without a map, and accepting and scheduling prioritized sub-tasks (e.g. "please take this paper to the registration desk"). So far the Challenge has drawn only a few participants in comparison to the other events, but there are indications participation will increase for 2002. Perhaps the most exciting recent change in the Robot Competition was the addition in 2000 of a Robot Rescue event. The task for robots in this event is to explore an arena that simulates a post earthquake environment for surviving humans. The rather sizable arena for this event was designed by Adam Jacoff and his team at NIST. The relevance of research in this area was brought home by the events of September 11, 2001. We expect substantial growth in this event over the next few years. The Exhibition Since the first year of the event, the Exhibition has been a forum for robotics demonstrations which fall outside the year's specified Competition tasks. In the early years, when most of the competition tasks involved solving indoor navigation, people interested in other areas of robotics research found a place to bring their work. The first year, the event was actually called the "AAAI Robot Exhibition and Competition," putting the exhibition of robots ahead of the competitive aspects [Dean and Bonasso 1995]. Most frequently, the exhibitions are given by graduate students, who bring their latest work to the event. In the first year, the exhibitors jumped into the ring whenever the competition wasn't going on; in later years, the exhibition has had rings of its own for demonstrations. The program has expanded over the years to include posters and video loops as well as live demonstrations, allowing people with non-traveling robots to participate. In 1997, the event gained its own chair, resulting in larger exhibitions. The wide variety of systems that have been demonstrated in the past include robotic wheelchairs, robotic sculptures, one-legged hopping robots, tube-like rolling robots, "dramatic robots", prototypes for Mars rovers, and humanoid robots, as well as the more typical looking research robots. The exhibition has served as a forum for researchers to give demonstrations of their latest work, sharing research more quickly with the community. The spirit of open discussion of research is present in all of the events and is further encouraged through the Robot Workshop held at the end of the Competition and Exhibition. A great deal of cutting-edge research has been presented at the exhibition, including the following systems. The first exhibition in 1992 was the site for the first demonstration of robots that learned lanuages for communication [Yanco and Stein 1993]. In 1994, the exhibition was host to the first 1000 pound autonomous hovercraft from the University of Maryland. In 1995, Dave Miller's enthusiasm resulted in the demonstration of several robotic wheelchair systems in Montreal. To add a small bit of competition to this exhibition, the wheelchairs competed in a "doorway limbo." At the exhibition in 1997, Robin Murphy demonstrated her lab's research on robotics for Urban Search and Rescue (which then debuted as a competition in 2000). Also in 1997, speech and gesture recognition for human-robot interaction were demonstrated on Coyote from the Naval Research Laboratories [Perzanowski et al 1998]. In 1998, Kurt Konolige demonstrated his Small Vision System, a small, compact stereo head [Konolige 1997]. At this year's exhibition, the Nils Nilsson Prize for Integrating AI Technologies was awarded to an exhibition participant: Cerebus from Northwestern University. Cerebus gave an interactive talk about itself to interested passers-by. This talk was the hit of the Robot Workshop that followed the Robot Competition and Exhibition. (See the article in this issue for further information on the research behind Cerebus.) The 2001 Competition The tenth annual AAAI Robot Competition and Exhibition was held at IJCAI-2001 in Seattle, Washington. This year, the events included two competitions, "Hors d'Oeuvres, Anyone?" and Robot Rescue (held in conjunction with RoboCup-2001), and a well attended exhibition. We also welcomed the Botball National Tournament again this year in the AAAI exhibition hall, even experiencing cross-over when a high school student demonstrated a snake robot called Slither that he had built with his father. The winners of this year's events are listed in Figure 1. See the articles in this issue for discussions of the events and the research approaches of the participants. Towards the Future As of this writing the slate of events for the 2002 AAAI Mobile Robot Competition and Exhibition is still under discussion. This discussion, informed by our look back over the last ten years, has led us to reflect on the objectives of the Competition. Whatever the original objectives were, it has grown to become many more things to many people. For instance, on the one hand, while we don't see as much cutting edge research at the competition as we would like, we see many, many motivated and excited undergraduates building successful robots. So, in any realignment of objectives and component events, we want to be sure we preserve what is right with the Competition. There is no question that the AAAI Mobile Robot Competition should promote and showcase {\it AI} robotics: those aspects of robotics that require AI and those aspects of AI that enable successful robotics. However, the Competition should also inspire and involve undergraduate students and provide a venue for graduate students to demonstrate their work. We feel these multiple goals can be met by providing several events that target different groups. The Robot Rescue event will certainly be included as it offers a wide range of research challenges as well as extraordinary relevance in these times. The "Hors d'Oeuvres, Anyone?" and Challenge events have been the primary drivers of human-robot interaction research over the last few years. Both Robot Rescue and "Hors d'Oeuvres, Anyone?" offer entry level possibilities for undergraduates, yet offer a deep enough problem that graduate students and researchers in the area will find interesting problems to solve. However, the overlap of human-robot interaction in "Hors d'Oeuvres, Anyone?" and in the Challenge suggests that we might find a way to combine the two into a new competition that could drive the field even farther. This leaves us with the opportunity to establish a new event.... What will it be? Acknowledgments Thanks to the Defense Advanced Research Projects Agency, the Naval Research Lab and Nils Nilsson for supporting the 2001 event. John Blitch, Doug Gage, Jim Hendler, Jean Schultz and Mark Swinson at DARPA have provided critical advice and support this year and in the past. ActivMedia, iRobot and Probotics donated robots at cost for prizes this year. The authors thank Tom Dean, Pete Bonasso, David Miller, Reid Simmons and Alan Schultz for their recollections. The AI community owes a debt to the AAAI Mobile Robot Competition and Exhibition Chairs: Tom Dean and Pete Bonasso (1992); Kurt Konolige and Terry Weymouth (1993); Reid Simmons (1994); David Miller (1995); David Kortenkamp, Illah Nourbakhsh and David Hinkle (1996); Ron Arkin (1997); Robin Murphy, Gregory Dudek and David Kortenkamp (1998); and Alan Schultz (1999, 2000). Finally, the competition is made possible every year through the hard work of the AAAI staff, especially Carol Hamilton, Keri Harvey and Richard Skalsky. References Ronald C. Arkin. "The 1997 AAAI Mobile Robot Competition and Exhibition." AI Magazine, 19(3): Fall 1998, pp. 13-17. Tucker Balch, Gary Boone, Thomas Collins, Harold Forbes, Doug MacKenzie, and Juan Carlos Santamaria. "Io, Ganymede and Callisto - a Multiagent Robot Trash-collecting Team." AI Magazine, 16(2), Summer 1995, pp. 39-51. Tucker Balch. "Profile of a winner: Georgia Tech." AI Magazine, 19(3), Fall 1998, p. 23. Pete Bonasso and Thomas Dean. "A Retrospective of the AAAI Robot Competitions." AI Magazine, 18(1): Spring 1997, pp. 11-23. Rodney A. Brooks. "Intelligence Without Reason." Proceedings of 12th Int. Joint Conf. on Artificial Intelligence, Sydney, Australia, August 1991, pp. 569-595. Joachim M. Buhmann, Wolfram Burgard, Armin B. Cremers, Dieter Fox, Thomas Hofmann, Frank E. Schneider, Jiannis Strikos, and Sebastian Thrun. "The Mobile Robot RHINO." 31-38, AI Magazine, 16(2), Summer Thomas Dean and R. Peter Bonasso. "The 1992 AAAI Robot Exhibition and Competition." AI Magazine, 14(1): Spring 1993, pp. 34-48. 1995, pp. 31-38. Gregory Dudek. "Profile of a Winner: McGill University." AI Magazine, 19(3), Fall 1998, p. 21. James Firby, Peter N. Prokopowicz, Michael J. Swain, Roger E. Kahn, and David Franklin. "Programming CHIP for the IJCAI-95 Robot Competition." AI Magazine, 17(1), Spring 1996, pp 71-81. Karen Zita Haigh and Tucker Balch. "AAAI-98 Robot Exhibition." AI Magazine, 21(1): Spring 2000, pp. 67-76. David Hinkle, David Kortenkamp, and David Miller. "The 1995 Robot Competition and Exhibition." AI Magazine, 17(1): Spring 1996, pp. 31-45. Kurt Konolige. "Designing the 1993 Robot Competition." AI Magazine, 15(1): Spring 1994, pp. 57-62. Kurt Konolige. "Small Vision Systems: Hardware and Implementation." Eighth International Symposium on Robotics Research, Hayama, Japan, October 1997. David Kortenkamp, Illah Nourbakhsh, and David Hinkle. "The 1996 AAAI Mobile Robot Competition and Exhibition." AI Magazine, 18(1): Spring 1997, pp. 25-32. Matthew T. Mason, "Kicking the Sensing Habit." AI Magazine, 14(1), Spring 1993, pp. 58-59. Lisa Meeden, Alan Schultz, Tucker Balch, Rahul Bhargava, Karen Zita Haigh, Marc Bohlen, Cathryne Stein, and David Miller. "The AAAI 1999 Mobile Robot Competitions and Exhibition." AI Magazine, 21(3): Fall 2000, pp. 69-78. David P. Miller. "A Twelve Step Program to More Efficient Robotics." AI Magazine, 14(1), Spring 1993, pp. 60-63. [crasar2001] Center for Robot-Assisted Search and Rescue website, http://www.csee.usf.edu/robotics/crasar/ Robin R. Murphy. "Using Robot Competitions to Promote Intellectual Development." AI Magazine, 21(1): Spring 2000, pp. 77-90. "An Affective Mobile Educator with a Full-time Job," Illah Nourbakhsh, Judith Bobenage, Sebastien Grange, Ron Lutz, Roland Meyer and Alvaro Soto. Artificial Intelligence, 114 (1-2), pp. 95-124. October 1999. D. Perzanowski, W. Adams, and A. Schultz. "Communicating with a Semi-Autonomous Robot." In Workshop on Integrating Robotics Research: Taking the Next Leap, AAAI 1998 Spring Symposium Series, Menlo Park, CA: AAAI Press, 1998, pp. 55-59. (Also available as NCARAI Report AIC-98-015.) Randy Sargent, Bill Bailey, Carl Witty and Anne Wright. "Dynamic Object Capture using Fast Vision Tracking." AI Magazine, 18(1), Spring 1997. Alan C. Schultz. "The 2000 AAAI Mobile Robot Competition and Exhibition." AI Magazine, 22(1): Spring 2001, pp. 67-72. Reid Simmons. "The 1994 AAAI Robot Competition and Exhibition." AI Magazine, 16(2): Summer 1995, pp. 19-30. S. Thrun, M. Beetz, M. Bennewitz, W. Burgard, A.B. Cremers, F. Dellaert, D. Fox, D. Haehnel, C. Rosenberg, N. Roy, J. Schulte, and D. Schulz. "Probabilistic Algorithms and the Interactive Museum Tour-Guide Robot Minerva." International Journal of Robotics Research, 19(11):972--999, 2000. Holly A. Yanco. "The 1997 AAAI Mobile Robot Exhibition." AI Magazine, 19(3): Fall 1998, pp. 39-47. Holly A. Yanco and Lynn Andrea Stein. "An Adaptive Communication Protocol for Cooperating Mobile Robots." In From Animals to Animats 2: Proceedings of the Second International Conference on the Simulation of Adaptive Behavior, edited by J.-A. Meyer, H.L. Roitblat and S.W. Wilson. The MIT Press/Bradford Books, 1993, pp. 478-485. ------------------------------------------------------------------------ Figure 1: The list of winners from the 2001 AAAI Robot Competition and Exhibition 2001 AAAI Robot Competition Winners Nils Nilsson Award for Integrating Artificial Intelligence Technologies Cerebus, Northwestern University Robot Rescue Technical Award for Artificial Intelligence for Rescue Swarthmore College Technical Award for Advanced Mobility for Rescue Sharif University Hors d'Oeurvres Anyone? First Place: University of British Columbia Second Place (Tie): Swarthmore College and Seattle Robotics Club Third Place: Universidad de Aveiro ------------------------------------------------------------------------ **Note** These quotes should be put in a sidebar with the article. No figure number is needed. Quotes over the years: "This won't be a slick, polished competition. There will be a certain amount of chaos, but I can guarantee there will be a lot of excitement and enthusiasm." -- Tom Dean 1992 "This event is cast ... in the spirit of trying to develop as animate, responsive, and intelligent robot behavior as possible." -- Pete Bonasso, 1992 "We wanted this contest to point up the higher-level cognitive possibilities for mobile robots and demonstrate the longstanding symbiotic relationship between AI and robots." -- Patrick Hayes, 1992 "Perhaps my greatest surprise in judging the competition was how much I learned about an area I thought I knew something about." -- Jim Hendler, 1992 "Robots bring people back to the core of what AI is aimed at: learning how to build intelligent creatures." -- Ben Kuipers, 1992 "Where is the science?" -- Tom Dean, 1994 "The science is what you do [back at the lab] and throw away the night before the competition." -- Jim Firby, 1994 "The hardest part is still the sensing and acting." -- David Kortenkamp, 1997 "When that machine (Newton Labs' Vision Car) caught that tennis ball today at the last second when I didn't think it was going to get it and it did, it was like Rocky. I was cheering for it. And the combination of AI and a machine that almost seemed to have a human will was exciting and that alone can make people, I believe, want to do more in both these fields." -- Alan Alda, host of Scientific American Frontiers, 1996