Recently, although the topic of 3D image processing has been dominated by hardware, there have also been significant developments in software, resulting in the emergence of many different application areas.

In terms of hardware, 3D cameras can be divided into groups such as Stereo Vision Cameras, Time of Flight Cameras, and Line Profile / Sheet of Light (laser line) cameras, regardless of their working principle. Regardless of the working principle, typical Cartesian images that provide X-Y and Z-axis information are obtained from a 3D camera, and images that provide the real position of each pixel in space with the z-distance, which we refer to as Disparity Map/Point Cloud, are obtained.

A typical 3D image processing software used for industrial purposes includes, in addition to 2D operations,

3D Surface Matching: Finding parts in a box or on a conveyor, determining their center positions and rotation angles. Robot Pick and Place applications are usually an example of this topic.

3D Measurement: Measurement not only in the X-Y axis but also in the Z-axis (usually height). (Height control of parts such as studs, collapse or bulge errors, surface indentations and protrusions that need to be measured from above but cannot be seen from the side, etc.)

They are used in operations such as. In some cases, some of these operations may be performed on the camera side, but generally, they will provide more project-specific and realistic results if performed by an external proprietary software. Most cameras come with their own SDK, but operations that can be performed with such SDKs will either be too standard or too limited. Therefore, to be able to develop a 3D application that can work properly in the real world, one must have full mastery of both the hardware and software sides.

3D Surface Matching:

Similar to 2D matching/object recognition process, in 3D shape matching, the shape is first trained with all its surfaces. This can be achieved by either using a CAD file or multiple images of the same part. The part is then subjected to a training process, and the system learns every point of the part. Later, the trained points are compared with the acquired point cloud. (Surface scanning is performed.) Depending on the matching amount, the system extracts the coordinates and rotation angles of the entire part in space.

Modeling 3D parts by reading from a 3D CAD file.

Usually, parts moving on a box or a conveyor will only show a single surface from the camera's perspective. The system must be able to match the entire part from this visible surface.

As shown in the image above, parts can come in different positions, such as upright, lying down, or at an angle, and sometimes partially overlap each other. In such cases, it is necessary to match and capture all the parts.

In a typical robot application, the coordinates and rotation angles of the center of each part or the gripper's holding point are determined and transmitted to the robot. The robot picks up the part from this position and performs the appropriate assembly operation.

Measurement: Since 3D cameras also provide Z information, it is possible to make height measurements and real-world coordinate axis measurements.

Normal 2D cameras work in pixels, and to convert pixel values to real-world coordinates (usually in mm), a tedious process called calibration is required. 3D cameras usually provide values directly in the real-world coordinate system. In this case, it is possible to make 3D measurements according to the real coordinate system.

As seen in the image above, 3D image processing is often considered in conjunction with robotics applications. After the acquired image is matched and the coordinates and angle of the center or holding point are determined, this information is transmitted to the robot. The robot picks up the part and processes it accordingly (mounts or places it in the appropriate station, etc.)

In real-life, these applications require other controls. The gripper's ability to grasp the part may depend on other conditions (such as the part's surroundings being empty). A good 3D image processing software must also control all of these situations.