Flexible automation through human demonstration

Despite significant advances in automation, programming robots for new or modified tasks is a time-consuming and costly process that requires careful programming and calibration for each variant. A particularly promising method for simplifying the programming effort is imitation learning, in which a human shows the robot how to perform a task and the robot can then carry out these processes independently. This approach also offers the advantage of being able to adapt flexibly to the variances observed in the demonstration process, for example to variations in the component positions. This means that instead of rigid automation stations, adaptable, human-centered work environments can be created.

During the demonstrations, data on the robot's condition (e.g. robot position, forces acting on it) and camera images are automatically recorded. These data sets are then used to train an AI model, where the robot learns to reproduce the observed movements. The training can also be automated to such an extent that no robotics or AI expert knowledge is required to teach new assemblies. Once the training is complete, the execution phase follows, whereby the robot executes the learned process independently using live image data and can be used productively.

Application example of automated cable assembly

The Fraunhofer Institute for Production Systems and Design Technology (IPK) is working in the field of automation technology on the further development of imitation learning in robotics and is a partner for the transfer of the latest research results to industry. As part of the "Converging" research project, imitation learning was integrated and evaluated for the automated insertion of cable connections (flat receptacles) with a collaborative robot in electrical assembly (see Figure 1). The investigations focused on the robustness of the approach with regard to process variants, such as changed component and robot start positions. The number of demonstrations required for reliable execution was also investigated. It was shown that with fixed component positions and small variances in the robot start position (up to 2 cm), just ten demonstrations are sufficient to teach the cable insertion task and execute it reliably (100% success rate over ten executions).

At the same time, the investigations also reveal the current limitations of imitation learning. Hidden states and changes in the camera field of view that are not depicted in the demonstrations are problematic. The addition of further variances, such as the random shift of the component position from 0 - 7 cm, increases the number of demonstrations required. Success rates of 43.8 % / 62.5 % / 75.0 % were achieved after 20 / 30 / 40 demonstrations. For precise processes with tolerances in the sub-millimeter range, such as the insertion of flat receptacles examined here, further measures are therefore necessary to ensure assembly quality. Fraunhofer IPK is also working with industrial partners to investigate which precise processes with lower tolerance requirements (e.g. palletizing, crate handling, machine loading, order picking or assembly with larger fits) can be implemented with imitation learning.

Prerequisites for practical implementation

All that is needed for integration into industrial systems is a robot, a PC with a commercially available modern graphics card for AI training, a camera and a simple teleoperation interface (e.g. 3D mouse). If a robot cell or a workstation for human-robot collaboration already exists, the process can also be retrofitted.

Imitation learning makes robot training more intuitive and efficient by simply demonstrating instead of programming. As the research example shows, just a few demonstrations are enough for simple assembly processes to ensure reliable execution by the robot. New variants can therefore be taught in just a few hours, even by existing production staff without robotics or AI expertise. At the same time, imitation learning is not a panacea; the method can reach its limits, especially with high-precision processes and high process variants. Companies already have the opportunity to be proactive and integrate imitation learning into suitable existing processes in order to secure a head start in the field of AI-based manufacturing.

Fabian Tiele and Valentyn Petrichenko, Fraunhofer IPK

This project has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No 101058521 - CONVERGING.