The demand from global markets for customized products presents manufacturers of production systems with the challenge of having to develop flexible yet economical systems. Robotics has found a firm place in this environment due to its flexibility. Methods of artificial intelligence and machine learning promise a new leap in efficiency for the programming of robot systems.

Virtual commissioning

Virtual commissioning (VIBN) is an established tool for programming production systems. It uses a virtual test environment to test the programming before commissioning the real system, eliminate errors and carry out optimizations. The virtual test environment allows tests to be carried out earlier and with less risk. X-in-the-loop systems are used as the test environment. The systems consist of the coupling of a simulation of the production system and the control technology. With software-in-the-loop, the control technology is emulated. Both the VDMA and VDI/VDE offer comprehensive knowledge on VIBN in the form of a guideline and a directive respectively.

Test automation

Parallelization during the VIBN merely pushes the commissioning effort forward, but does not reduce the overall time required. For this reason, the ISW has been researching an efficiency method - test automation - that reduces the actual effort required to test the production system^[3]. The automated execution and evaluation of test runs saves time. Each test with subsequent troubleshooting corresponds to an optimization loop of the control system programming or the overall system, which is driven by the control system developers. Reinforcement learning (RL) is an approach that enables the demanding work involved in programming and optimization to be supported by tools.

Reinforcement Learning

RL stands for a special class of problems for which a variety of methods for automatic solution finding from the field of machine learning are available. An adaptive program, a so-called agent, interacts with an environment. The agent perceives its environment in the form of a state space and can exert influence on the environment via defined actions. The agent attempts to perform a specific task and continuously learns by receiving a reward or punishment as numerical feedback for each iteration and adapting its behavior accordingly.

Various studies at the ISW have shown that software-in-the-loop test environments are a suitable learning environment for RL agents. The agent can assume the role of the controller and send the target signals as actions to the production system. Alternatively, it can use its actions to program and parameterize an existing industrial control system.

The agent learns according to one of the selected scenarios in the virtual test environment, as this is cheaper to set up and also allows errors to be made during learning without risk. A central point for RL is to define a suitable reward function for the agent. The development of a reward function is currently only possible by RL experts. In addition, some reward functions are problem-specific, meaning that once a reward function has been found, it cannot be transferred.

The solution: test-driven learning

The processes involved in testing control programming and learning in RL are comparable. The crucial point is to consider the defined test cases from the control programming as a reward function for the agent. By coupling the automated tests with the learning process, the agent receives the test results as a reward. Assuming that the tests for the control programming have been defined in advance, a method based on test-driven development is created. Initial results show that classic control tests are suitable for allowing the agent to learn the required automation processes in the form of control programs.
Dipl.-Ing. Karl Kübler; Florian Jaensch, M.Sc.; Univ.-Prof. Dr.-Ing. Dr. h.c. mult. Alexander Verl

Briefly explained: The ISW at the University of Stuttgart

The Institute for Control Engineering of Machine Tools and Manufacturing Units (ISW) at the University of Stuttgart is equally committed to basic research and application-oriented development, which has led to successful cooperation with both public project sponsors and industry. In the course of its more than 50 years of existence, the main research areas and fields of application have been constantly expanded. Further information is available at: www.isw.uni-stuttgart.de

Briefly explained: The MHI e.V.

The Wissenschaftliche Gesellschaft für Montage, Handhabung und Industrierobotik e.V. (MHI e.V.) is a network of renowned university professors - institute directors and chair holders - from German-speaking countries. The members conduct both fundamental and application-oriented research on a wide range of current topics in the fields of assembly, handling and industrial robotics. Further information on the society, its members and activities: http://www.wgmhi.de