Project 3
Camera Calibration and Fundamental Matrix Estimation with RANSAC

In this project we'll estimate the camera projection matrix, mapping 3D world coordinates to camera coordinates. We will also estimate the fundamental matrix which maps points in one scene to epipolar lines in the other. In case we don't have ground truth correspondences, we will use keypoints identified by SIFT and use RANSAC to estimate a good enough fundamental matrix, that we will then use to reject spurious point matches.

Part 1: Camera Projection Matrix

We set up the linear regression to solve for elements of the projection matrix. The matrix that we obtained is a scaled version of the matrix in the Project page, (specifically by a factor of -0.5968). The total residual error comes out to be 0.0445 and the camera center is estimated to be at (-1.5126, -2.3517, 0.2827).
Projection Matrix:
0.7679 -0.4938  -0.0234 0.0067
-0.0852 -0.0915 -0.9065  -0.0878
0.1827 0.2988 -0.0742 1.0000


3D Plot of the camera center and the 3D points used to calculate projection matrix
Plot showing the projected points using the calculated matrix and the actual 2D points

Part 2: Fundamental Matrix Estimation

We set up the linear regression to solve for elements of the fundamental matrix similar to how we did in the previous part. To normalize, we compute the mean of all the 2D points for each image, and estimate a translation matrix to translate each point by the respective amount to bring the mean 2D point for each image to (0,0). By calculating the average length of these translated points, we compute a scaling factor to multiply each point with so as to bring the average length of the resultant set to 2. We then solve for elements of the fundamental matrix by performing svd on the matrix that we set up. Once we obtain that, we enforce the singularity constraint by calculating the svd of F and setting its lowest singular value to zero. Thereafter, we undo the scaling and translation operations that we performed initially.
Fundamental Matrix (No Noise; Normalized):
-0.0000  0.0000 -0.0004
0.0000 -0.0000 0.0032
-0.0000 -0.0044 0.1035
The following sets of images clearly depict the effect of normalizing the points before computing the estimate of fundamental matrix.

No Noise, No Normalization

The epipolar lines don't exactly pass through the corresponding point in the other image. However they are only slightly off

Noisy Points, No Normalization

The epipolar lines are way off corresponding (noisy) point in the other image.

No Noise, Normalized

The epipolar lines seem to exactly pass through the corresponding point in the other image. Normalization improved the result by a huge amount.

Noisy Points, Normalized

The epipolar lines exactly pass through the corresponding (noisy) point in the other image. It doesn't look too perfect though. Normalization improved the result by a huge amount.

Part 3: Fundamental Matrix with RANSAC

In the RANSAC method, we randomly select 8 points from the matches that SIFT returns us and fit a fundamental matrix to them using the function we wrote in the previous part. Over a certain number of iterations, we select the F matrix that has the highest fraction of inliers. We classify a point pair as inlier or outlier based on a threshold for x2' * F * x1 (which should be zero for perfect correspondence). We have used #iterations = 4000 and threshold to be 0.01
In the table below, we have compared results by varying the #iterations and threshold. We selected a value of 0.001 for threshold because it gives a very low inlier error while still keeping a substantial fraction of points as inliers. We select 4000 as #iterations because, beyond that, increasing #iterations doesn't seem to be bringing significant improvement in results.
# iterations threshold total inlier error total outlier error #inliers/#total points
2000 0.1 34.8290 94.7551 0.7673
2000 0.01 2.925 161.523 0.7527
2000 0.001 0.1551 153.9273 0.3927
         
1000 0.001 0.1550 151.2474 0.4000
2000 0.001 0.1551 153.9273 0.3927
4000 0.001 0.1373 163.6863 0.3467
6000 0.001 0.1447 151.4771 0.3552

For the best model that we obtain above, we then sort all the inliers based on the same error metric we used above to classify as inlier and outlier. We select the 30 best matches from these inliers. Some results (30 best matches) with #iterations = 4000, threshold = 0.001

Mt. Rushmore

The 30 points that do get returned seem to be perfect matches.

Notre Dame

Except for one pair, all the matches seem to be perfect.

Episcopal Gaudi

The 30 points that do get returned seem to be perfect matches.

Woodruff

Except for one pair, all the matches seem to be perfect.