Thermography for hand detection in the MRK

In industrial environments, humans and collaborative robots are increasingly working closely together, which brings with it new challenges in the area of occupational safety. Protecting people from potential collisions or accidents is of central importance here. In particular, the detection of human hands in the shared workspace is critical for safe collaboration.

Reliably recognize hands

A research project at the Bremen Institute of Structural Mechanics and Production Systems (bime) is investigating the use of thermography and machine learning to reliably detect human hands in a shared workspace. Thermographic cameras offer the advantage of capturing temperature-based images and precisely detecting objects with a temperature that deviates from the environment, such as human hands. Unlike conventional camera systems that rely on color or texture information, this method avoids misinterpretation in complex environments. It focuses solely on temperature differences and is therefore more resistant to interference from visually similar objects.

To further optimize recognition accuracy, an algorithm based on neural networks is being developed. Convolutional neural networks (CNNs) are ideal for image processing tasks as they are able to learn complex patterns such as shapes and contours. The Yolo (You Only Look Once) model used for this work makes it possible to recognize objects in image data in real time. This is particularly relevant as industrial applications rely on high processing and response speeds. The model is specially adapted to analyze thermographic images and segment hands reliably and accurately. Hands are localized in the image data and then precisely delimited from their surroundings to ensure efficient further processing.

Recognition of the hand using different tools. © bime

The algorithms developed are tested extensively, both in simulated and real working environments. The results show a high recognition rate, even under difficult conditions such as different hand positions, varying temperatures or partial occlusions. Particularly noteworthy is the real-time capability of the system, which achieves frame rates of up to 60 frames per second. This ensures that the application can be used effectively in safety-critical areas, as is common in HRC.

Training and validation of the model

An essential part of the research is the development of a dataset of thermographic hand images that covers multiple variations of human hands under different conditions. This data set is used to train and validate the model and enables precise tuning of the recognition algorithm. Different hand postures, temperature changes and occlusion by objects are taken into account. The implementation of this system brings clear advantages for safety in HRC scenarios and reduces the risk of robots injuring humans through unforeseen movements.

This work makes a significant contribution to current research in the field of safe human-machine collaboration. Thermography-based approaches offer significant advantages over classical image-based methods, as they are robust against visual disturbances and effectively utilize temperature differences to detect human extremities. The use of modern neural networks enables efficient processing of image data and guarantees a powerful real-time application, which is crucial for industrial use.

Patrick Rückert, Lorenz Marhenke