Techno Blender
Digitally Yours.

Camera Radial Distortion Compensation with Gradient Descent | by Sébastien Gilbert | Dec, 2022

0 51


How to characterize the radial distortion of a camera-lens pair, based on a simple model

Photo by Charl Folscher on Unsplash

Consumer-grade cameras and lenses are cheap and ubiquitous. Unfortunately, unlike their industrial counterparts, they were not designed to be used as tools for precise measurements in computer vision applications.

Among the various types of distortion, the most visible one affecting a low-grade camera and lens is radial distortion. Radial distortion is a nonlinearity between the viewing angle of an object in the scene and the pixel where that object appears in the image. In the vicinity of the optical center, the effect is hardly perceptible, but the distortion gets more pronounced as we move away radially from the optical center. Typically, a pixel far from the optical center will appear closer to the center than it should. The corners of the image appear to have been pulled toward the center. This phenomenon is known as barrel distortion because a rectangular object seen perpendicularly by the camera will appear as a rounded “barrel” (see Fig .1).

Figure 1: The image of a flat checkerboard showing radial distortion. Image by the author.

The purpose of this story is to characterize the radial distortion of a camera-lens pair, based on a simple model. Once we know the distortion parameters, we’ll be able to compensate for the pixel location of an object and get an undistorted pixel location.

Undistorted?! Is that a word?

I’m not sure. But for conciseness, I will use the [possibly fake] verb “to undistort” to mean “to compensate for the radial distortion”.

You can clone the code and the sample images in this repository.

The checkerboard image of Figure 1 will provide us with colinear feature points. In the absence of radial distortion, the pixel location of colinear points in the scene should be colinear. Since they are visibly not colinear, we’ll build a model with adjustable parameters, that will map the distorted pixel points onto undistorted points. Our objective function will be the colinearity of undistorted pixel points belonging to a line on the checkerboard.

Feature points

The first step is to extract the feature points in Figure 1.

# Find the checkerboard intersections, which will be our feature points that belong to a plane
checkerboard_intersections = checkerboard.CheckerboardIntersections(
adaptive_threshold_block_side=adaptive_threshold_block_side,
adaptive_threshold_bias=adaptive_threshold_bias,
correlation_threshold=correlation_threshold,
debug_directory=output_directory
)
intersections_list = checkerboard_intersections.FindIntersections(checkerboard_img)

The object of type CheckerboardIntersections converts the color image into a grayscale image and then applies an adaptive threshold. The result is a binary image where the square intersections are made very crisp. The inside of the squares has no value for the rest of the processing.

Figure 2: The thresholded pattern. Image by the author.

The thresholded image of Figure 2 gets correlated with synthetic pattern images designed to emphasize the two types of intersections.

Figure 3: The two synthetic patterns. Image by the author.

The two correlation images are thresholded to get the highest peaks (see Figure 4).

Figure 4: One of the thresholded correlation images. Image by the author.

The blobs in the thresholded correlation images are detected, and the center of mass is computed for each one, yielding the list of intersection points.

Figure 5: The small color circles show the found feature points. Image by the author.

The radial distortion model

We’ll consider a basic radial distortion model — as simple as it gets: a quadratic correction factor as a function of the distance from the optical center. The undistorted radius will be the product of the distorted radius and the correction factor.

Figure 6: The correction factor as a function of the radial distance. Image by the author.

Such a distortion model has only three parameters:

  • The optical center (cx, cy). It doesn’t necessarily coincide with the image center (w/2, h/2).
  • The quadratic coefficient α. When α > 0, we have barrel distortion (see Figure 1). When α < 0, we have pincushion distortion (the image corners appear stretched outwards). When α = 0, there is no radial distortion.

Model optimization

We are facing a nonlinear optimization problem: finding the optimal parameters (cx, cy) and α that will project the intersection points that we found (see Figure 5) in such a way that they form straight lines. To do this, we’ll create a PyTorch model that stores our three distortion parameters.

class DistortionParametersOptimizer(torch.nn.Module):
def __init__(self, center, alpha, image_sizeHW, zero_threshold=1e-12):
super(DistortionParametersOptimizer, self).__init__()
self.center = torch.nn.Parameter(torch.tensor([center[0]/image_sizeHW[1], center[1]/image_sizeHW[0]]).float())
self.alpha = torch.nn.Parameter(torch.tensor(alpha).float())
self.image_sizeHW = image_sizeHW
self.zero_threshold = zero_threshold

This class also needs to know the image size, since the pixel coordinates will be normalized in (-1, 1) for numerical stability. The DistortionParametersOptimizer.forward() method returns a batch of mean squared error, each corresponding to the residual error of a line of feature points projected on the corresponding best fit line. In the ideal scenario, if the radial distortion gets perfectly compensated for, the forward() method will return a batch of zeros.

We interact directly with the distortion compensation class, RadialDistortion. When we call its Optimize() method, it will internally instantiate an object of type DistortionParametersOptimizer, and run a determined number of optimization epochs.

# Create a RadialDistortion object, that will optimize its parameters
radial_distortion = radial_dist.RadialDistortion((checkerboard_img.shape[0], checkerboard_img.shape[1]))
# Start the optimization
epoch_loss_center_alpha_list = radial_distortion.Optimize(intersections_list, grid_shapeHW)
Plot([epoch for epoch, _, _, _ in epoch_loss_center_alpha_list],
[[loss for _, loss, _, _ in epoch_loss_center_alpha_list]], ["loss"])

Within 100 epochs, the mean squared error drops by a factor of 100:

Figure 7: The MSE loss as a function of the epoch. Image by the author.

Wow! That was easy¹!

The found parameters are (cx, cy) = (334.5, 187.2) (i.e. North-North-East of the image center (320, 240)) and α = 0.119, corresponding to the correction factor of barrel distortion, as expected.

Undistorting the points

Now that we have characterized the radial distortion, we can apply the correction factor to our feature points.

Figure 8: The feature points, after compensation. Image by the author.

Figure 8 shows the feature points, in blue, after having been compensated for the radial distortion. The central points are mostly unchanged, while the periphery points get pushed further away from the center. We can observe that the scene points that belong to a straight line on the checkerboard look much better aligned in the image.

Although we probably would not want to do that in a real-time application, we can undistort the whole image by projecting each pixel in the original image onto its corresponding undistorted location.

Figure 9: Undistorted image. Image by the author.

Since we are progressively pushing pixels radially away from the optical center, we regularly encounter pixels in the projected space that have not gotten mapped by a pixel in the original image, giving rise to the annoying black traces in Figure 9. We can eliminate those by replacing the black pixels with the median color in their neighborhood (Cf. Figure 10, right).

Figure 10: Left: The original image. Right: The corresponding image after compensating for the radial distortion. Image by the author.

Conclusion

We considered the problem of radial distortion that affects most consumer-grade cameras and lenses. We assumed a simple distortion model that pushes or pulls pixels radially according to a quadratic law. We optimized a set of parameters using the feature points of a checkerboard, through gradient descent. These parameters allowed us to compensate for the radial distortion.

It is critical to recognize that the distortion parameters are intrinsic to the camera-lens pair. Once they are known, we can compensate for the radial distortion of any image, as long as the camera-lens pair is fixed.

I invite you to play with the code, and let me know what you think!


How to characterize the radial distortion of a camera-lens pair, based on a simple model

Photo by Charl Folscher on Unsplash

Consumer-grade cameras and lenses are cheap and ubiquitous. Unfortunately, unlike their industrial counterparts, they were not designed to be used as tools for precise measurements in computer vision applications.

Among the various types of distortion, the most visible one affecting a low-grade camera and lens is radial distortion. Radial distortion is a nonlinearity between the viewing angle of an object in the scene and the pixel where that object appears in the image. In the vicinity of the optical center, the effect is hardly perceptible, but the distortion gets more pronounced as we move away radially from the optical center. Typically, a pixel far from the optical center will appear closer to the center than it should. The corners of the image appear to have been pulled toward the center. This phenomenon is known as barrel distortion because a rectangular object seen perpendicularly by the camera will appear as a rounded “barrel” (see Fig .1).

Figure 1: The image of a flat checkerboard showing radial distortion. Image by the author.

The purpose of this story is to characterize the radial distortion of a camera-lens pair, based on a simple model. Once we know the distortion parameters, we’ll be able to compensate for the pixel location of an object and get an undistorted pixel location.

Undistorted?! Is that a word?

I’m not sure. But for conciseness, I will use the [possibly fake] verb “to undistort” to mean “to compensate for the radial distortion”.

You can clone the code and the sample images in this repository.

The checkerboard image of Figure 1 will provide us with colinear feature points. In the absence of radial distortion, the pixel location of colinear points in the scene should be colinear. Since they are visibly not colinear, we’ll build a model with adjustable parameters, that will map the distorted pixel points onto undistorted points. Our objective function will be the colinearity of undistorted pixel points belonging to a line on the checkerboard.

Feature points

The first step is to extract the feature points in Figure 1.

# Find the checkerboard intersections, which will be our feature points that belong to a plane
checkerboard_intersections = checkerboard.CheckerboardIntersections(
adaptive_threshold_block_side=adaptive_threshold_block_side,
adaptive_threshold_bias=adaptive_threshold_bias,
correlation_threshold=correlation_threshold,
debug_directory=output_directory
)
intersections_list = checkerboard_intersections.FindIntersections(checkerboard_img)

The object of type CheckerboardIntersections converts the color image into a grayscale image and then applies an adaptive threshold. The result is a binary image where the square intersections are made very crisp. The inside of the squares has no value for the rest of the processing.

Figure 2: The thresholded pattern. Image by the author.

The thresholded image of Figure 2 gets correlated with synthetic pattern images designed to emphasize the two types of intersections.

Figure 3: The two synthetic patterns. Image by the author.

The two correlation images are thresholded to get the highest peaks (see Figure 4).

Figure 4: One of the thresholded correlation images. Image by the author.

The blobs in the thresholded correlation images are detected, and the center of mass is computed for each one, yielding the list of intersection points.

Figure 5: The small color circles show the found feature points. Image by the author.

The radial distortion model

We’ll consider a basic radial distortion model — as simple as it gets: a quadratic correction factor as a function of the distance from the optical center. The undistorted radius will be the product of the distorted radius and the correction factor.

Figure 6: The correction factor as a function of the radial distance. Image by the author.

Such a distortion model has only three parameters:

  • The optical center (cx, cy). It doesn’t necessarily coincide with the image center (w/2, h/2).
  • The quadratic coefficient α. When α > 0, we have barrel distortion (see Figure 1). When α < 0, we have pincushion distortion (the image corners appear stretched outwards). When α = 0, there is no radial distortion.

Model optimization

We are facing a nonlinear optimization problem: finding the optimal parameters (cx, cy) and α that will project the intersection points that we found (see Figure 5) in such a way that they form straight lines. To do this, we’ll create a PyTorch model that stores our three distortion parameters.

class DistortionParametersOptimizer(torch.nn.Module):
def __init__(self, center, alpha, image_sizeHW, zero_threshold=1e-12):
super(DistortionParametersOptimizer, self).__init__()
self.center = torch.nn.Parameter(torch.tensor([center[0]/image_sizeHW[1], center[1]/image_sizeHW[0]]).float())
self.alpha = torch.nn.Parameter(torch.tensor(alpha).float())
self.image_sizeHW = image_sizeHW
self.zero_threshold = zero_threshold

This class also needs to know the image size, since the pixel coordinates will be normalized in (-1, 1) for numerical stability. The DistortionParametersOptimizer.forward() method returns a batch of mean squared error, each corresponding to the residual error of a line of feature points projected on the corresponding best fit line. In the ideal scenario, if the radial distortion gets perfectly compensated for, the forward() method will return a batch of zeros.

We interact directly with the distortion compensation class, RadialDistortion. When we call its Optimize() method, it will internally instantiate an object of type DistortionParametersOptimizer, and run a determined number of optimization epochs.

# Create a RadialDistortion object, that will optimize its parameters
radial_distortion = radial_dist.RadialDistortion((checkerboard_img.shape[0], checkerboard_img.shape[1]))
# Start the optimization
epoch_loss_center_alpha_list = radial_distortion.Optimize(intersections_list, grid_shapeHW)
Plot([epoch for epoch, _, _, _ in epoch_loss_center_alpha_list],
[[loss for _, loss, _, _ in epoch_loss_center_alpha_list]], ["loss"])

Within 100 epochs, the mean squared error drops by a factor of 100:

Figure 7: The MSE loss as a function of the epoch. Image by the author.

Wow! That was easy¹!

The found parameters are (cx, cy) = (334.5, 187.2) (i.e. North-North-East of the image center (320, 240)) and α = 0.119, corresponding to the correction factor of barrel distortion, as expected.

Undistorting the points

Now that we have characterized the radial distortion, we can apply the correction factor to our feature points.

Figure 8: The feature points, after compensation. Image by the author.

Figure 8 shows the feature points, in blue, after having been compensated for the radial distortion. The central points are mostly unchanged, while the periphery points get pushed further away from the center. We can observe that the scene points that belong to a straight line on the checkerboard look much better aligned in the image.

Although we probably would not want to do that in a real-time application, we can undistort the whole image by projecting each pixel in the original image onto its corresponding undistorted location.

Figure 9: Undistorted image. Image by the author.

Since we are progressively pushing pixels radially away from the optical center, we regularly encounter pixels in the projected space that have not gotten mapped by a pixel in the original image, giving rise to the annoying black traces in Figure 9. We can eliminate those by replacing the black pixels with the median color in their neighborhood (Cf. Figure 10, right).

Figure 10: Left: The original image. Right: The corresponding image after compensating for the radial distortion. Image by the author.

Conclusion

We considered the problem of radial distortion that affects most consumer-grade cameras and lenses. We assumed a simple distortion model that pushes or pulls pixels radially according to a quadratic law. We optimized a set of parameters using the feature points of a checkerboard, through gradient descent. These parameters allowed us to compensate for the radial distortion.

It is critical to recognize that the distortion parameters are intrinsic to the camera-lens pair. Once they are known, we can compensate for the radial distortion of any image, as long as the camera-lens pair is fixed.

I invite you to play with the code, and let me know what you think!

FOLLOW US ON GOOGLE NEWS

Read original article here

Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.
Leave a comment