Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

Camera Projection Matrix

For camera calibration we will use the data which consists of points from world co-ordinates to co-ordinates on image plane. We will specifically form the below shown matrix:

where [X1, Y1, Z1] are the world co-ordinates, [u1, v1] are image plane co-ordinates. We will then use SVD to find the solution for the equations we obtain. The matrix V i.e. the right singular vectors after reshaping into 4 × 3 and taking the transpose will give us the projection matrix M. The matrix M can be seen as (Q|m4). The camera center is given by -Q-1m4. The projection matrix M obtained is:

The residual obtained is 0.0445 and the camera center C is <-1.5127, -2.3517, 0.2826>. Below is the visualization of the solution obtained for the equations using SVD:

Shown below is the visualization of the points in the real world 3D space and the location of camera center:

Fundamental Matrix Estimation

For estimating the fundamental matrix we will use the below shown equation:

where fij are entries of the fundamental matrix and [u1, v1] and [u'1, v'1] are the co-ordinates of the matched pair of points in the two images. We will get one equation for every pair of matched points. Similar, to part 1 we can form a matrix from these equations and perform the SVD to solve the equations. Since, the fundamental matrix is a rank 2 matrix we will set the last element of the diagonal matrix S to be 0 and calculate U * S * V' to get the rank 2 fundamental matrix. This gives reasonable performance. However, the performance can be improved by normalizing the co-ordinates. To normalize we perform the following transformation:

where u and v are unnormalized co-ordinates and u' and v' are normalized and T = is the transformation matrix. Here, -cu and -cv are the means for u and v respectively. s defines scale and out of the number of different options for s, I chose s to be the standard deviation obtained after subtracting the means. We obtain then obtain the original fundamental matrix by TbT * Fnorm * Ta. The estimated fundamental matrix using normalization trick is:

This gives very good performance as can be seen by the epipolar lines and the matches on the below shown pair of images:

Fundamental Matrix with RANSAC

Using a pipeline like SIFT for finding matched pairs often gives spurious matches. For this purpose the least squares method fails as it is sensitive to outliers. We can use RANSAC for this purpose. For RANSAC we will choose some small number of points and estimate the fundamental matrix. The fundamental matrix has the property that x' * F * x = 0 for points that completely agree with the fundamental matrix. We define a threshold and find all pairs of matched points for which the absolute value of x' * F * x value lies within the threshold. These points are known as inliers. We perform RANSAC for 2000 iterations randomly picking some points each iteration. The fundamental matrix giving maximum inliers is the fundamental matrix obtained from the RANSAC method. Below is the output (top 30 matches) for various images after using the normalization trick when estimating the fundamental matrix:

Effect of normalization

Below is a comparison of outputs for Episcopal Gaudi by unnormalized and the normalized methods. We can clearly see that normalization gives us much better results.