Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

For this project we were tasked with doing camera calibration and Fundamental Matrix estimation in order to find the epipolar lines for an image pair. The project had 3 different parts listed below.

  1. Estimate the center of the camera
  2. Compute a Fundamental Matrix for a set of points
  3. Use Ransac to find the best Fundamental Matrix

Estimating Camera Center

For this part we needed to compute a projection matrix. The way I choose to do this was by constructing the problem as a least squares problem, which is standard for projections and point regression estimation. I did this by creating a matrix A such that Ax = 0. I then minimized this by using the SVD method to obtain a series of matrices that project from 2D to 3D. We take V since it is the matrix doing the actual transformation in space. Then we take V can calculate the center of our camera by taking -1*(M(:, 1:3)\M(:,4)). For the normalized points in part one I got <-1.5127, -2.3517, 0.2826> which seems to be within floating point error of the suggested <-1.5125, -2.3515, 0.2826> for the normalized points. The

Computing the Fundamental Matrix

For this part of the project we were tasked with computing the Fundamental Matrix given a set of points in image A that match to their respective points in image B. To start we set this up as another least squares regression problem; however, we only want a rank 2 matrix since we know that our epipolar points will lie on the epipolar lines themselves. To do this we take our singular values matrix, S, from the SVD and set S(3,3) to 0. This removes the less important 3rd rank when we do USV' to obtain our actual fundamental matrix.

RANSAC

For this part we are tasked with finding the best Fundamental Matrix for a given image pair's sift matches. RANSAC is a fairly simple algorith, I started by randomly sampling 8 point pairs and use our method from part 2 in order to construct a fundamental matrix. I then found which points were inliers by computing error = abs(point_b * fmatrix * (point_a')). Then I checked that error against a threshold and if it was below the threshold I added it to the inliers. I did this 10,000 times in order to make sure that my Fundamental Matrix would be a good one, and returned the best out of the 10,000 iterations. For the Mount Rushmore and Notre Dame image pairs I used an error threshold of 0.1. This provided a good balance of accuracy and a good number of points considered as inliers. You can see where there are one or two points that aren't on a line but are still considered inliers, this would likely be fixed if my program did the point normalization that was discussed as extra credit. For the Episcopal Gaudi image pair I choose to be more strict with my error in order to make thet best matches possible so I restricted it to 0.05 instead of 0.1.

Results in a table

Overall this project shows how SIFT can be made to be more useful if we constrain our feature matches to be along a transformation. This leads to overall better matching pairs even without doing things such as point normalization; however, there are definitely still matches that don't necessarily belong and the point normalization would likely take care of this issue.