Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

RANSAC based matches on two images of the Episcopal Gaudi in Spain

The goal of this project was to introduce us students to the concepts of camera and scene geometry. This was done in three steps:

  1. Estimating the Camera Projection Matrix, which maps 3D coordinates to image coordinates
  2. Estimating the Fundamental Matrix which relates points in 2 different scenes to one another, using epipolar lines
  3. Using RANSAC to combine parts 1 and 2 to achieve much better image matching than what can be done using SIFT.

Additionally, I implemented coordinate normalization to improve the estimates for the Fundamental Matrix for extra credit.

Part 1: Camera Projection Matrix

The goal of this section was to compute the projection matrix that goes from 3D coordinates to 2D image coordinates. This can be done using the following equation:

and a slight modification of changing m34 to 1 to adjust the scale for the matrix. Once the Camera Projection Matrix was calculated, calculating the center of the camera was as simple as carrying out the following equation . After implementing both of those equations in the appropriate files, I was able to come up with the results shown below for the two provided point sets.

1st Point Set: 2D Projection Matrix Evaluation 2nd Point Set: 2D Projection Matrix Evaluation
1st Point Set: 3D Projection Matrix Evaluation & Camera Center 2nd Point Set: 3D Projection Matrix Evaluation & Camera Center

Part 2: Fundamental Matrix Estimation

The goal of this section was to estimate the mapping of points in one image to epipolar lines in another image using the Fundamental Matrix. This was done by using SVD decomposition to solve the following equation:

Similarly to the first part of the project, I had to set m33 as 1 to adjust the scale correctly. Due to this, I only needed to find 8 matching points to create the fundamental matrix. My results are shown below:

Image 1: Epipolar lines and matches Image 2: Epipolar lines and matches

Part 3: Fundamental Matrix with RANSAC

The final portion of this project involved using RANSAC in conjunction with the Fundamental Matrix Estimation to correctly match points in images based on unreliable point correspondences computed with SIFT. By using RANSAC to get the best estimation, and filtering those matches down to the 30 best ones, I was able to produce the results below:

Mount Rushmore Image 1: Epipolar lines and 30 best matches Mount Rushmore Image 2: Epipolar lines and 30 best matches
Mount Rushmore RANSAC 30 best matches

It is quite clear, that using RANSAC, we can do a extremely good job at matching points across the 2 images. Based on a quick visual scan of the matched points, they all seem to be correct and valid matches for the Mount Rushmore image. For the sake of being thorough, I also ran it without limiting the number of matches to produce the following results:

Mount Rushmore Image 1: Epipolar lines and all the matches Mount Rushmore Image 2: Epipolar lines and all the matches
Mount Rushmore RANSAC with all the matches

It is much harder to tell if all the points are matched correctly due to the density of the lines, but they seem to be quite accurate.

Extra Credit: Normalizing the Fundamental Matrix

In order to improve the accuracy of RANSAC on the Episcopal Gaudi image, I wrote additional code to normalize the Fundamental Matrix for extra credit. This was done using the following equation:

Normalizing the Fundamental Matrix showed quite a difference in the resulting matches for the Episcopal Gaudi image pair. Without the normalized Fundamental Matrix, the results were as follows:

Episcopal Gaudi Image 1: Epipolar lines (non-normalized) and 30 best matches Episcopal Gaudi Image 2: Epipolar lines (non-normalized) and 30 best matches
Episcopal Gaudi RANSAC (no normalization) 30 best matches

Unlike the Mount Rushmore example, running RANSAC alone on the Gaudi image pair was entirely accurate. You can see multiple incorrect matches in the images, my favorite one being the match between a cloud and a window. After normalizing the Fundamental Matrix, I was able to produce the results shown below:

Episcopal Gaudi Image 1: Epipolar lines (normalized) and 30 best matches Episcopal Gaudi Image 2: Epipolar lines (normalized) and 30 best matches
Episcopal Gaudi RANSAC (normalized) 30 best matches

As you can see, running RANSAC when using a normalized Fundamental Matrix drastically improves the results and matching on the Episcopal Gaudi pair of images.