Working with 3D cameras and their data is complex and requires a lot of preparation and set-up time when developing an application. Especially in multi-camera applications or in connection with robotics, complex calibrations of several coordinate systems are necessary before the data can be used effectively. Due to the strong system dependency, these activities often have to be carried out directly on the target system. The actual application development (image processing or software algorithms) must also often be developed directly on the system in order to generate usable data. In addition, the field of view and resolution of many 3D cameras are not sufficient for applications with larger workspaces. These requirements have been incorporated into the development of the new Ensenso SDK 2.2 and the new 5 MP version of the Ensenso-X series from IDS.

The 3D cameras are suitable for both static and moving applications. For bin picking or high-precision object comparisons, their Flexview technology in conjunction with the SC algorithms (sequence correlation) optimizes the result accuracy of the camera models N35, X36 1.3 MP, X36 5 MP. For this purpose, a movable high-performance projector projects a random pattern onto the test object, generating image material with different surface structures of the test object. The SC algorithms use this to calculate the 3D object data, which benefits from each additional image pair (up to 16) to increase accuracy.

Ensenso's N30, X30 1.3 MP and X30 5 MP camera models are also prepared for applications with moving objects, such as on continuous conveyor belts, or cases where the camera itself is moving. Optimized SGM algorithms (Semi-Global Matching) can achieve an accuracy of depth information even from a single recorded image pair.

Virtual scene with simulated raw images and depth information in the demo program NxView. © IDS

Larger eyes with a wider field of view
The integration of two Sony IMX264 5 MP image sensors increases the 3D image resolution of the Ensenso 3D camera family by around 35 percent compared to the previous 1.3 MP version, while at the same time increasing the field of view by around 20 percent. In order to completely capture a packed Euro pallet with a volume of 120 x 80 x 100 centimetres, the distance between the camera and the test object can be reduced from 1.5 meters to just 1.25 meters. In conjunction with the lower pixel noise of the Sony sensors, this results in an improvement in the calculated depth information from 0.43 millimetres to 0.2 millimetres.

More powerful sensors naturally generate larger amounts of data and therefore potentially longer processing times until results are available. Reference studies with a 5-MP model confirmed the approximately four times longer processing times for matching the images of an image pair or the time required to obtain a complete 3D image. Nevertheless, the times for the complete 3D image calculation of a sequence correlation with 16 high-resolution 5 MP image pairs are only around 2.5 seconds. These times are perfectly adequate for most applications.

In order to counteract the larger data volumes and the associated time losses, important calculations for Cuda have been optimized. Depending on the GPU used and the parameterization of the corresponding algorithms, the additional computing capacity of Nvidia graphics processors accelerates the processing by a factor of five. With Cuda support, 3D applications also become interesting for the embedded environment.

Multiple cameras in one application
The Ensenso software libraries offer useful functions for using multiple cameras in one application. Due to different views and positions, the coordinate systems of several cameras must be aligned with each other or with fixed points in the real world and calibrated to a uniform object coordinate system. If cameras are to work together with a robot and its movements need to be coordinated with the camera data, a hand-eye calibration can also be performed. An integrated calibration wizard supports the user during the process. The SDK also enables the integration and calibration of monocular 2D uEye cameras in an application. The quality of inspection and measurement results in 3D applications can be significantly improved by the capabilities of 2D cameras. Where stereo cameras have difficulties identifying objects in border areas, 2D cameras ideally provide support through edge detection or color recognition. Furthermore, 2D cameras also make it possible to capture additional information such as the content of barcodes. The Ensenso software therefore supports the integration of both technologies.

Application developers should particularly benefit from the "File Cameras" and "Virtual Cameras" extensions. To improve algorithms and processes, it is essential to be able to debug identical data multiple times. A file camera behaves like a real camera on the system, with the difference that its image material comes from a local folder with saved data records. This means that application sequences can be simulated again and again using the data without having to access the real system or recreate situations. This is also an ideal tool for error detection: users can save problematic data sets to make them available to image processing specialists.

With "virtual cameras", simulations can be carried out in an offline environment to evaluate the data quality, completeness, resolution and noise of a scene with different camera models. For this purpose, objects can be imported in STL or PLY format, rendered and positioned as required. This allows performance evaluations to be carried out using different variants of an inspection process without having to set up a real system.

Heiko Seitz / as