Sophie's Kitchen Demo
Interactive Reinforcement Learning

Intro

Welcome to Sophie's Kitchen. This is an environment that we have been using to experiment with Interactive Reinforcement Learning. You can find some papers that have come out of this work here.

In the initial experiment, 18 non-expert users trained Sophie to bake a cake. We made three main observations about teaching behavior: People assume they can guide the agent, they actively adjust to better fit the learner, and they have a strong positive bias in rewards.

Having found people assume they can communicate about the future, we added a guidance mechanism to the Sophie agent and modified the RL algorithm to incorporate this guidance. Having found that people adjust their mental model we added a gazing transparency mechanism, and having found the asymmetric use of feedback we added a motivation channel and an undo response. These social modifications were tested with non-expert users over the web and found to improve learning significantly on several dimensions.

To try out this modified version of RL with the Sophie agent, read through the brief description of the task and follow the link at the bottom of the page.

The Kitchen Environment

In this task the Sophie agent is in the kitchen environment pictured, there are five objects in the kitchen: flour, eggs, a bowl, a spoon, and a tray. Sophie can move to one of three locations to face the shelf on the right, the oven at the left, or the table in the center. The objects can be in any of these locations as well.

Sophie's Actions

• She can TURN-LEFT or TURN-RIGHT, which changes her location.
• She can PICK-UP any object in her current location. (She can only hold one object at a time.)
• She can PUT-DOWN the object she is holding.
• She can USE the object she is holding on any object in her current location.
  • Using the flour on the bowl, puts flour in the bowl.
  • Using the spoon on the bowl with flour and eggs makes batter.

The Kitchen Task: Bake a cake

Your task is to help Sophie learn how to bake a cake! The basics steps are: put the bowl on the table, add flour, eggs, and stir to make batter, put the batter in the pan, and put the pan in the oven. Sophie may need help making the cake a few times before she can do it herself. Here are the ways that you can communicate with Sophie:

Interactive Rewards

You can't do actions or tell her exactly what to do, but you can send messages to try and help. When you click the LEFT mouse button a rectangle appears, showing your message. Drag the mouse to change the size and color of your message. UP = GREEN (positive), DOWN = RED (negative).

Guidance Messages

In addition to the Feedback, you can direct Sophie's attention to objects with guidance messages. Click the RIGHT mouse button to make a yellow square (it's best if you don't use a one button mouse, but if you have to, hold down the option key to do the guidance message). This square tells Sophie 'Pay attention to this!' Objects light up when the mouse is over them to help you know what guidance message you will send. You can only guide Sophie to an object (not a location like the table), and she only sees your message if she is facing the object. For example, if she is facing the table and you make the yellow square over the flour on the shelf she won't see that, but if she's facing the table and the flour is on the table she will see your guidance message and think you are telling her to pay attention to or do something with the flour.