Images

The Jacobi Vision library provides a set of classes to handle different image data types; color images, depth images, and combined RGBD images, providing functionality for loading, saving, visualizing, and processing image data, as well as performing projections between 2D and 3D spaces using camera intrinsics.

The image classes are structured hierarchically to handle different types of image data:

  • ColorImage: Handles RGB color images.

  • DepthImage: Handles depth images, providing methods to convert depth data into 3D point clouds.

  • RGBDImage: Combines both color and depth data into a single image object.

Throughout this tutorial, we will load images from files in order to display them and perform basic operations such as converting depth data to point clouds. Later on, we will see how to stream images from a camera in a live environment using drivers.

Working with Color Images

To load a color image from a file, you can use the ColorImage.load_from_file method. This method reads an image file and creates a ColorImage object.

color_image = ColorImage.load_from_file('path/to/color_image.png')

You can display the loaded color image using the show method.

color_image.show()

To save a ColorImage object to a file, use the save method.

color_image.save('path/to/save_color_image.png')

Working with Depth Images

Depth images can be loaded from a NumPy array file using the DepthImage.load_from_file method. They are a simple 2D array of floating-point values representing distances from the camera at each pixel.

depth_image = DepthImage.load_from_file('path/to/depth_image.npy')

Depth images represent distances and may not be directly viewable as standard images. You can render the depth image into a visual format using the render_depth method and then display or save it.

depth_visual = depth_image.render_depth()
depth_visual.show()

You can convert depth data into a 3D point cloud using the to_point_cloud method.

point_cloud = depth_image.to_point_cloud()

Point clouds may be used for collision checking while planning robot motions.

Working with RGBD Images

An RGBDImage combines both color and depth data, allowing synchronized processing of both types of information.

To load an RGBDImage, you need to provide the path to the color image. The depth image path is inferred by replacing 'color' with 'depth' in the filename and changing the extension to .npy.

rgbd_image = RGBDImage.load_from_file('path/to/color_image.png')

Note: The depth image file should be in NumPy .npy format and located in the same directory, with 'color' replaced by 'depth' in the filename.

You can access the color and depth components separately using the color and depth properties.

color_image = rgbd_image.color
depth_image = rgbd_image.depth

You can display both the color and depth components side by side using the show method.

rgbd_image.show()

To save an RGBDImage, use the save method with the desired path for the color image. The depth image will be saved automatically with a corresponding filename.

rgbd_image.save('path/to/save_color_image.png')

Camera Intrinsics and Projections

All image classes can optionally take camera intrinsics, either as a camera matrix or an Intrinsics object. This allows for projecting 3D points to 2D pixels and deprojecting 2D pixels back to 3D space.

To project 3D points onto the image plane:

color_image.camera_matrix = intrinsics.as_matrix()
pixels = color_image.project(points_3d)

To deproject 2D pixels back into 3D space:

points_3d = depth_image.deproject(pixels_2d)

In the next tutorial, we will take a look at camera drivers and how to stream images from real cameras using the Jacobi Vision library.