Cobalt: Researchers develop smartphone app for remote robot training

A research team led by Georgia Tech (with participation from UC Berkeley, NYU Abu Dhabi, University of Toronto and Nvidia) has developed the Cobalt app to control robotic arms that can be controlled with a smartphone from anywhere in the world. In principle, anyone can operate a robot without any prior technical knowledge. The goal behind this is to collect and use the resulting control data for training control algorithms for various autonomous tasks of the robot.

The Cobalt app for smartphones works comparatively simply: via a secure WiFi connection, it establishes a connection via the Internet to a teleoperation server through which a robot arm can be controlled. To control it, users simply need to move the cell phone around the room. The movements are mapped via the internal sensors and imitated by the robot arm. Users can view the result immediately via a live, low-latency video connection. Data transmission is implemented via WebRTC (Web Real-Time Communication), the researchers write in the study “COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphones,” which is published in the preprint on Arxiv. The lead author, Ayush Agarwal, also presented the results at the IEEE International Conference on Robotics and Automation in Vienna, which ends on June 5, 2026.

The teleoperation server can be accessed from anywhere in the world, enabling a large user base to be acquired. After all, around five billion people worldwide have a smartphone, according to the researchers.

Users can use the Cobalt app to instruct the robot arm to grab, move and release an object. A variety of object manipulations are possible. This happens more intuitively via a smartphone than via other input devices.

The idea behind this is that even inexperienced people can work with a robot. The scientists record the control data they generate. They want to train robot control algorithms in order to be able to automate the execution of certain tasks with a robot.

Crowdsourcing robot training data

The researchers imagine that such a system could enable data collection for training robots using crowdsourcing. This would solve the problem of getting to grips with the large-scale collection of data needed to train robots for mass manufacturing. The training of autonomous robots cannot be achieved with the required quality through simulations alone.

The scientists tested their system with test subjects from nine different countries. These included people who had no experience with robots and students from Midtown High School in Atlanta. The data collected showed that they are suitable for robot training in terms of quality and are in no way inferior to those recorded with professional virtual reality headsets, controllers, keyboards or computer mice.

(olb)