Images

Picture-in-picture function of GStreamer. © Creative Commons Attribution 3.0' Peach Open Movie Project

Modern vision applications contain a large number of different elements. Each individual function block must be well selected, configured and validated before it is then connected to other components in applications that are as durable as possible. In addition to the lowest possible barrier to entry into a new technology, the longevity of the architecture and the largest possible number of low-cost or free features should round off the package.

Use of pipelines for maximum flexibility

Prototype of a pipeline. © Aries Embedded

GStreamer is based on the pipeline model and offers the possibility of combining one or more multimedia sources with each other, modifying, filtering, encoding or decoding them using suitable filters and, after processing, feeding them into one or more data sinks. The pipeline can be branched and thus offers a wide range of options for further processing content.

Each filter element also represents a data sink for receiving the data and a data source for providing the data for the following filter element or the data sink.

This representation of the pipeline is greatly simplified and is intended to illustrate the basic principle of GStreamer. For example, multimedia files, an image that is delivered directly from the camera to the pipeline or a data stream that is received via the network can be used as a data source. Filter blocks are used to filter content, split or merge data streams, encode or decode data, superimpose other information on existing content or modify it in other ways.

If embedded Linux is used as the operating system for the embedded application, GStreamer can be used very easily to create prototypes of new or modified functions. The command line is used in a terminal to start GStreamer and to string together and use the individual elements. In GStreamer, the elements of the pipeline are called "elements". A plug-in usually comprises several elements. For example, there is a plug-in for "vaapi" or "rtp". A typical entry in the command line could look like this:

gst-launch-1.0

videotestsrc \

! video/x-raw-yuv, framerate=30/1, width=800, height=480 \

! vaapih264enc \

! mp4mux \

! filesink location=testfile.mp4

This command generates a test image using the test image generator, formats the image to 800x480 pixels and a frame rate of 30 FPS, encodes this video content in H.264 format, generates the MP4 format from the compressed content and saves the result in a file. In function blocks of a GStreamer pipeline, the function shown would look like Figure 4.

GStreamer text and image overlay. © Aries Embedded

With the videotestsrc element, it is possible to provide test videos in many different formats for a pipeline. If these are then combined with the media players available in Linux distributions, initial prototypes of a later video track or another application can be realized using only standard Linux on-board resources, without having to configure a camera sensor or switch on a display beforehand.

In practice, plug-ins are used as shared libraries to provide a variety of video and audio functions. Plug-ins help to realize data sources and sinks, add overlays as text or images, encode and decode many common audio and video formats, display streams and images and much more.

Codecs and hardware acceleration

Many formats for files, streams and codecs are natively supported by hardware acceleration in CPU derivatives, graphics or audio cards. These architectures offer functions for encoding and decoding popular formats such as H.264, H.265 or MJEPG in the CPU or chipset. This is particularly interesting for the resource-saving implementation of embedded systems, as the use of these accelerator blocks places little or no load on the hardware. In practice, this helps to realize leaner systems, save time, money and resources for the product and also reduce the development risk. Using the command line can be the preferred solution for quickly testing the feasibility of a desired functionality. For final products and releases, GStreamer is used in C#, C++, Python, Perl and other environments. The GStreamer framework itself is available as source code and can also be used as a precompiled binary on Android, iOS and Windows.

Aries Embedded already supports its customers in many smart vision projects. This often begins with advising customers on the CPU or FPGA architecture to be used, and continues with the development of hardware, software and IP through to the qualification and delivery of complete systems in series production.

Andreas Widder, Managing Director of Aries Embedded