Robot dog with ChatGPT-4: AI navigation for blind people

Robot dogs have repeatedly made headlines over the past decade. This is mainly thanks to the Spot model from Boston Dynamics, which has been used in various areas since it was introduced in 2015: by the US police, for example, as a mine detector in Ukraine and even by Deutsche Bahn to detect graffiti sprayers.

But four-legged robots are not only suitable as guard dogs, but possibly also as guide dogs. In the past, there have been several studies and attempts to provide blind people with a robot dog that can safely navigate them from A to B. A new work from Binghamton University in the US state of New York is based on this, which was presented in January at the AAAI Conference on Artificial Intelligence and is available as a preprint paper.

“Very strong language skills”: Where a robot dog with ChatGPT-4 is superior to guide dogs

The special feature of the robot dog is that it can not only guide its owners, but can also communicate with them and describe the environment thanks to large language models. An interaction that is low-threshold in itself, but can be effective, as Shiqi Zhang, co-author of the study, says: “Real dogs can understand around 20 commands at best. With robotic guide dogs, however, it is possible to program GPT-4 so that the system has very strong language skills.”

The team didn’t use a Spot as a robot, but rather a Unitree Go2 Pro – a comparatively simple model from China that is available to private individuals for less than 4,000 euros. A trained guide dog can cost five to ten times as much.

The researchers then equipped the model with a specially developed dialogue system, which essentially consists of two modules: “plan verbalization,” in which humans and robots agree together what the task is and where they want to go. And “scene verbalization”, in which the robot communicates information about the environment to its owner as it moves, so that the owner can react if necessary and adapt the originally agreed plan.

How the robot guide dog and its owner work together

Both work via voice commands: If the owner says she is thirsty, the robot guide dog first checks whether it already has appropriate targets: It checks whether there is a kitchen, a drinks machine or a water dispenser nearby. If there are several valid options, the system creates detailed directions and tells you whether there are obstacles on the way, such as doors or escalators, and how many: “A door needs to be opened for the kitchen; the path takes about three minutes.” If the task is not clear, the system asks.

During navigation, the dialog interface then uses the camera and sensors integrated in the robot to provide descriptions of the environment “to improve the operator’s spatial imagination and facilitate joint decision-making,” as the study states.

The robot guide dog still has numerous limitations

To test the system, the researchers from New York had seven blind people navigate through a large office building with the robot dog under different conditions: both with minimal verbalization, only with scene verbalization and in the combination mentioned. The feedback was generally positive, they say. The participants particularly liked the combination of plan and scene description.

The team sees potential in the technology, but also points out limitations in the study: The system was explicitly not tested in strange places or in complex situations, such as in the hustle and bustle of a big city. Instead, the language model was provided with information about the premises in advance. Therefore, it is impossible to say how it will perform in other situations. This will be examined in a follow-up study.

In addition, an AI-supported guide dog can understand more commands than an actual four-legged friend. But a guide dog is more than a navigator for its owners. A trained guide dog can assess dangerous situations, react to falls and also refuse the owner’s commands. Last but not least, it is a psychological support: Even the best robot dogs may not be able to simulate the feeling of having a real, trustworthy companion at your side so quickly.