New App Allows Anyone to Operate a Robot From Their Phone

Someone with no computing experience may soon be able to remotely control a robot from anywhere on the planet using a smartphone, thanks to new technology developed by Georgia Tech.

The new technology is also set to revolutionize the scale of policy training data collection, which is essential to advancing robotic capabilities and meeting growing production demand.

COBALT is a mobile app that turns smartphones into controllers for robot arms. With a secure Wi-Fi connection to a server, users can move their phones in any direction, and the robot arm will mirror the motion — from anywhere in the world.

Ayush Agarwal, a Ph.D. student in Georgia Tech’s School of Interactive Computing who leads a research team developing COBALT, said it works like the games people play on smartphones. Users can press a button to have the arm grasp an object, move it, and release it with another button.

Agarwal conducted several user studies with participants in nine countries who remotely operated robot arms inside Georgia Tech’s People, AI & Robotics (PAIR) Lab. The lab is directed by Assistant Professor Animesh Garg, who advises Agarwal.

“We built an entire distribution system for remote teleoperation scaled to where we had people from Indonesia, India, and Pakistan operating for us,” Agarwal said. “They were novice operators who had never done it before. By collecting data from these new users, we showed that we can train policies to automate certain tasks.”

Garg envisions a world where data collection for policy training is done through crowdsourcing. He began working toward this goal 10 years ago as a postdoc at Stanford University, when he developed RoboTurk, an earlier version of COBALT.

“There is a large-scale data collection requirement for mass robot production to be possible, and it will not be solved purely through simulation,” Garg said.

“Our idea was, what if we could get almost every person on the planet to be a passive source for data collection? There are almost five billion people who have smartphones and know how to use them.”

Education and Economy Impact

Another major implication of COBALT could be expanded access to CS and robotics education.

Students can learn to operate a robot remotely in any classroom. In fact, Garg and his lab recently hosted students from Midtown High School in Atlanta to demonstrate COBALT and let them control robot arms from a phone.

Garg also sees the possibility of a “gig economy” in which people pay remote operators to control assistive robots in their homes and complete household chores for them.

“It could be Uber for robots,” he said. “People who want to log onto the platform can do so at their convenience and for as long as they want.”

Companies with robot-dependent labor tasks could also use the platform to enable human oversight.

“If I deploy a robot in a factory that achieves high autonomy for most tasks, but there are still times it needs help, a human could operate the robot from anywhere in the world,” Garg said.

Building a Network

Agarwal’s studies showed that people prefer to interact with and control a robot using a smartphone rather than virtual reality (VR) headsets, controllers, keyboards, mice, or other devices.

“The phone is a more intuitive interface and can provide data quality that’s on par with other commonly used devices,” he said.

Agarwal also said there is minimal latency in the video feed sent back to operators on the other side of the world. That’s because the amount of data being processed is small.

The data is carried over Web Real-Time Communication (WebRTC), the same technology used by many streaming services and web conferencing platforms such as Zoom and Google Meet.

“There’s a connection from your phone to the teleoperation server, which is connected to the robots,” Agarwal said.

“Then there’s another connection from the teleoperation server back to the user, which allows for a video stream. We need low latency on both because you don’t want the user to move their phone and wait 10 seconds to see the visual feed.”

Agarwal is the co-lead author of a paper on COBALT that is being presented at the IEEE International Conference on Robotics and Automation this week in Vienna. He said the paper stands out because it has moved from theory to the implementation of an entire distribution network. 

“The real novelty of our paper is the systems that we build around it to actually support the scaling of remote operation and data collection at a global level,” he said.