ISS robotics experiment "Surface Avatar" successfully completed

The astronaut allowed the various robots to act together and partially or fully autonomously as required. A new feature that was also tested is the AI chatbot assistant 'NealAI', which assisted the astronaut with questions. "The Surface Avatar Experiment is a milestone in the collaboration between humans and robots in space. We have thus achieved all the technical prerequisites for controlling complex robotic missions on Mars, including towards a future permanent lunar research station," explains Prof. Alin Albu-Schäffer, Director of the DLR Institute of Robotics and Mechatronics.

The core of the technology demonstration is to enable a person to command and deploy a team of different robots without intensive training. The Surface Avatar experiment series, which will be launched in 2022, is being led by the DLR Institute of Robotics and Mechatronics and is being carried out in collaboration with the European Space Agency ESA and the German Space Operations Center GSOC.

"Through the experiments with the astronaut in microgravity and the connection via relay satellites, we have mastered the technical hurdles in the remote control of robots. We are therefore confident that we will be able to manage future scenarios, be it lunar gateway, lunar habitats or Earth to robot. With these experiments and technologies, we have built up unique expertise in Europe that is also useful for applications on Earth," says Dr. Thomas Krüger, Team Lead ESA Human-Robot Interaction Lab.

More robots, more difficult tasks

As a step up from the last Surface Avatar experiment in July 2024, not only have the tasks become more difficult and the robots more independent - another robot has also been added. In addition to the humanoid DLR robot Rollin' Justin, ESA rover Interact and the four-legged DLR robot Bert, the four-legged ESA robot Spot now completes the exploration team. Spot is larger than Bert and equipped with a gripper arm. Jonny Kim gave him the task of independently finding sample containers and bringing them to the transfer station. The NASA astronaut then switched directly to the DLR robot Rollin' Justin with the task of taking the incoming containers from the transfer station to the lander. The two robots independently collected several 'Mars samples' for transportation back to Earth, while the astronaut was able to devote himself to the next tasks.

Robot quadruped learns to limp

The Surface Avatar team demonstrated the robots' special teamwork and learning abilities in the next challenge: to explore a cave, Jonny Kim commanded the Interact rover, which in turn had loaded the DLR robot Bert in its transport basket - the world's first robot-robot transport. Telecommanded by the astronaut in space, the rover drove to the cave, carefully lifted the little four-legged friend out with its gripper arm and set him down on the ground.

But Bert had 'injured' himself - one of his legs was stuck, so the four-legged friend was unable to explore the cave as intended. The experiment team on Earth had intended this malfunction so that the astronaut and robot would have to spontaneously solve an unforeseen problem - just like on a real space mission. For this purpose, the DLR robot Bert is equipped with 'reinforcement learning', a method of machine learning. In order to find a stable gait for Bert with three legs, Jonny Kim carried out 'training': He had the robot try out different gaits and evaluated them until Bert had found a strategy that worked. After the successful learning unit, the DLR robot completed its cave exploration. Here, the astronaut opted for manual control - using Bert's camera eyes, he was able to see what the robot was seeing and control it remotely using the joystick on the Robot Command Terminal.

AI-supported preparation

The collaboration between NASA astronaut Jonny Kim on board the ISS and the robots on the simulated Mars habitat in Oberpfaffenhofen went so well that all tasks, including the additional options, were successfully completed in the tight time window of two and a half hours. For example, Jonny Kim steered the rover manually to pick up a sample container and place it in the transfer station. The US-American controlled the robots to drive around, look around and explore, touch and feel the environment. He used them as intelligent employees to plan and carry out mission tasks on site. They served as a physical extension of him from a distance - like avatars, in keeping with the mission.

To prepare, the astronaut used the new AI chatbot assistant 'NealAI', which is based on a Large Language Model (LLM for short) from the European provider Mistral AI. The development team at the DLR Institute of Robotics and Mechatronics trained the chatbot specifically for the Surface Avatar mission. During the training phase, it answers Jonny Kim's questions about the functionalities of the robots, the operation of the control unit and the user interface. The idea behind this is that in future missions to Mars, the signal delay between Mars and Earth can be up to 40 minutes. During these communication breaks between the exploration team and the control center, an AI assistant like Neal AI could immediately help astronauts with questions.

"Surface Avatar's final experiment was the perfect conclusion to the mission. It showed how robotics can support our astronauts in their exploration of space. Following this fruitful collaboration with ESA, I look forward to celebrating further joint successes in the future," summarizes Principal Investigator Dr. Neal Y. Lii from the DLR Institute of Robotics and Mechatronics summarizes.