The touchscreen revolutionizes the HMI
Twelve years ago, Apple's iPhone caused a sensation with a new operating concept. The keyboard was replaced by a touchscreen and the entire front of the device was filled with a huge screen with an unprecedented pixel density. It is operated via a touchscreen with a projected capacitive field, which can also recognize and evaluate several touch events simultaneously. This technology can only be used sensibly with a completely new type of user guidance via software. Tapping, sliding and other gestures with one or two fingers control different functions.

Since this product launch, the world of input and output has changed. A single touch on the touchscreen, which triggered exactly one action, has given way to gestures with one or more fingers that trigger complex actions. "Drag and drop" or "pinch" work from the wrist without having to laboriously select objects via menus and assign an action. This change requires completely new underlying software and therefore a new operating concept.

Do actions always have to be initiated by touching a touchscreen in front of the screen? Is it not also conceivable to evaluate movements in front of the screen? Yes, several technologies make use of this idea. 3D touch systems that determine positions by means of a change in an electrical field are suitable for gestures that do not place any particular demands on accuracy. The resolution of these systems is sufficient for a qualitative determination of "louder/quieter", "higher/lower", "zoom in/zoom out".

Alternatives to the touchscreen
If the position needs to be evaluated more precisely, cameras are used that follow the operator's hands, for example. Control units for AR/VR glasses offer perfect position resolution, allowing sensitive control in the system.

Eye tracking" takes a different approach. After a short learning phase, the system is able to track the operator's pupils with the help of cameras. The "mouse click" is made by blinking. After familiarization, the operator can operate the system without using their hands. It can be used wherever hands are not free or must remain sterile.

Voice control is also a trend. Apple, Google and Amazon are leading the way: Calling up information from the internet or controlling devices in the living room at home is simply done by voice. Behind this is artificial intelligence, which is located in the provider's server farm.

Somewhat simpler is voice input, which follows more or less strict syntax rules. We find it in the navigation system of our car, "Please enter the address in the form city, street and house number". Parsing, i.e. assigning the words entered to fields in the database, can be carried out by a local controller and the desired action, be it navigating to the destination address or changing the radio station, can be executed.

Visualization
On the other side of the interaction is the output, which is usually handled by a display. Here, too, technology is advancing. Again driven by mass market quantities, OLEDs are appearing with brilliant displays and outstanding image quality thanks to their enormous contrast. In certain applications, they appear to be outstripping LCD technology, even if they are subject to an ageing process due to their principle, which becomes visible through a reduction in brightness.

© Hy-Line

TFT counters with quantum dots, which are entering the market under names such as "QLED" and promise a high color gamut with vibrant hues. Screens with micro-LEDs are still at the laboratory stage and are helping LED technology to enjoy a renaissance, moving it from a pure light source for TFTs to a frontline image source. The production of large screens with high resolution has not yet been solved for series production, but prototypes show the great potential of this technology.

Back to the HMI: today's users can expect a consistent concept. It is not one technology alone that leads to success, but the combination. The illustration clearly shows the involvement of the senses. A human-machine interface must offer a coherent input function that appeals to many sensory organs.

In addition to the touch systems described above, there is also haptic feedback, which directly signals successful input to the operator via the sense of touch. It could also be provided acoustically via a beeper, but in noisy environments this may be drowned out by the noise level. The high-resolution display shows the required information ergonomically. However, the user guidance provided by the software ("GUI" = Graphical User Interface) plays a decisive role. It determines how information is presented, through colors, shapes, arrangement on the screen and presentation in relation to each other.

HMI 5.0 is the next revolution
What does HMI 5.0 mean? The term describes the comprehensive interaction of people with the system using all their senses. New methods are coming onto the market, such as 3D gesture recognition, speech recognition and eye tracking for input. For output, the importance of haptic feedback, which compensates for the shortcomings of conventional touchscreens, and three-dimensional visualization with holograms or glasses is increasing.

More powerful graphics cards have made glasses for AR and VR affordable. Initially driven by computer games, VR glasses have already found their place in simulator training, where they visually and acoustically transport the operator into a realistic scene. The communication channel between man and machine has become broader: Seeing and being seen - with image output and eye-tracking; hearing and being heard - with sound output and voice input; touching and feeling - with touchscreen and haptic feedback. Only smelling and tasting are still missing in this roundelay.

Conclusion
Is touch operation still state of the art? The answer to this question is a resounding "yes". Despite many new technologies, the touchscreen will continue to set the tone for some time to come. In addition, comprehensive, multi-sensory communication between humans and computers, which we call HMI 5.0, will play an important role and will not be limited to keyboard/touchscreen and monitor. Even if some technologies have not yet made the breakthrough into the mainstream, they are ready as tools to provide the user with a new user experience. With their support, they are ideally prepared when it comes to implementing software for artificial intelligence (AI) or machine learning (ML).

Hy-Line has been intensively involved with technologies relating to the interface between man and machine for more than ten years. It all started with the introduction of projected capacitive touchscreen technology (PCAP) in the industry. Today, the focus is on expert knowledge relating to input and output media, from consulting to the finished design, which is also imparted to all interested parties in seminars in Germany, Austria and Switzerland.