What is Camera Calibration?

Geometric camera calibration (also called camera resectioning) is the process of estimating the parameters of a camera model (usually a pinhole camera model) and the lens used, in order to correct for manufacturing imperfections and effects of wear and tear.

With a calibrated camera it is possible to convert from camera units [pixels] to physical world coordinates [meters] and thus measure the size of objects in the image, determine the position of the camera in the environment, or correct the image for lens distortion.

These tasks are used in many areas of automation, robotics, VR/AR, metrology, medical imaging and autonomous driving. Camera calibration is often used as an early stage in computer/machine vision.

What is actually calibrated?

Camera calibration is primarily about determining the intrinsic parameters of a camera that affect the imaging process. 
With the pinhole model, the parameters that are determined are listed below:

• Focal Length: Most important parameter which relates directly to the focus of your camera and is defined by the distance between the image sensors center and the center of projection. 
• Optical Center: Due to manufacturing tolerances, the center of the image sensor is generally never perfectly aligned with the optical axis. The calibration takes this effect into account and calculates the optical image center.
• Scale Factor: This factor accounts for the possibility that the shape of the physical pixels on the image sensor is not quadratic but rectangular (one side is longer than the other). Usually this scale factor is expressed by introducing two different focal length f_x and f_y instead of one common f.
• Skew Factor: This factor accounts for the possibility that the physical pixels on the image sensor are not perfectly rectangular, but rather have the shape of a parallelogram. Note: This parameter is only used in very rare cases, as almost every modern device has rectangular pixels (that is why OpenCV omits this parameter).
• Lens Distortion Coefficients: Especially cheap lenses cause a lot of distortions that have to be corrected to get good calibration results. Most modern calibration applications distinguish between radial and tangential distortion coefficients.

The pinhole model explained

So far we have mentioned the pinhole model a few times, but what exactly is it? The pinhole model assumes the simplest setup of a camera you can imagine, a camera without a lens and with a single small aperture. It makes the assumption, that light enters the camera through this infinitely small pinhole (aperture) and project an inverted image on the opposite side of the camera. 

The pinhole camera matrix has two components: intrinsic and extrinsic parameter. When calibrating a camera, the calibration algorithm calculates the pinhole camera matrix using the extrinsic and intrinsic parameters.

The intrinsic parameters of the pinhole model can be expressed in a 3x3 Matrix: 

The extrinsic parameters represent the location of the camera in the 3D scene and consist of a rotation R and a translation t, which can be determined for every calibration image. They are represented in homogenous coordinates by a 4x4 matrix:

Putting the matrix equations for intrinsics and extrinsics together we get the distortion-free projective transformation of 3D points in world coordinates to 2D points in the image plane: 

with:

• s, the projective transformation's arbitrary scaling (not part of the camera model)
• p, a 2D point (pixel) in the image plane
• K, the intrinsic camera matrix
• [R|t], the extrinsic matrix   
• P_w, a 3D point in homogenous world coordinates

We can rearrange the equation above to get the distortion-free image coordinates (x, y):

Lens distortion

The intrinsic camera matrix of the pinhole camera model does not take lens distortion into account. This is simply due to the pinhole camera definition, which states that such a camera has only an infinitesimally small aperture through which light enters the camera chip. In order to focus an appropriate amount of incident light rays onto the image sensor, a system of lenses is usually installed in front of each camera.

To accurately represent a real camera, the camera model should consider radial and (sometimes) tangential lens distortion.

Radial distortion

Radial distortion occurs when light rays bend more near the edges of a lens than they do at its optical center. Usually the smaller the lens, the greater the distortion.

Radial distortion is commonly modeled with the even ordered radial model by Brown. This model has three polynomial coefficients (k1, k2, k3) and is described as follows:

with p being an image coordinate (x or y) and r being the distance from the optical image image center calculated as follows: 

Typically, two coefficients (k1 and k2) are sufficient for calibration. For severe distortion, such as in wide-angle lenses, one should select three coefficients to include k₃.

Tangential distortion

This kind of distortion is caused by manufacturing inaccuracies in lens production that result in the lens and image plane not being exactly parallel to each other.

Tangential distortion is described in almost all calibration applications by the following mathematical model with two parameters (p₁, p₂):

Pinhole Camera Model with Distortion

To include distortion in the pinhole camera model, we simply plug the formulas for radial and tangential distortion into the pinhole camera model: