Project 3: Camera Calibration and Fundamental Matrix Estimation with RANSAC

Example of a local features.

Overview

This project uses Matlab to perform camera calibration and to estimate the fundamental matrix. Ransac is then used with the fundamental matrix code to find the best possible fundamental matrix.

Camera Calibration

The camera calibration algorithms calculate the projection matrix and estimate the camera's locaton.
Projection Matrix
The projection matrix algorithm proceeds by setting up systems of equations between 2D and 3D:
  1. Calculate u, v, X, Y, and Z
  2. Setup systems of equations between aforementioned variables
  3. Use SVD to solve for the projection matrix
The resulting projection matrix for the test data is:
[0.0069 -0.0040 -0.0013 -0.8267
0.0015 0.0010 -0.0073 -0.5625
0.0000 0.0000 -0.0000 -0.0034]
Residual Total
<15.5450>

Esimated Camera Location
The camera location is estimated by using the projection matrix to determine the center of the camera. The algorithm calculates Q, the the first 3 columns of the projection matrix. It also calculates m4, the last column. It then calculates the center by the formula:
Center = -inverse(Q) * m4
The estimated camera center for the more difficult test data was estimated as:
<303.1000, 307.1843, 30.4217>
Camera Estimate Visualization Actual & Projected Points




Estimated Fundamental Matrix

The fundamental matrix is used to relate corresponding points across images. The algorithm I wrote is used to estimate the fundamental matrix. The algorithm operates by:
  1. Calculating U, U', V, and V'
  2. Constructing a system of equations for the fundamental matrix with the aforementioned components
  3. Using SVD to solve for a rank 2 estimate matrix
  4. Setting smallest singular value to zero to reduce rank
  5. Using SVD again to calculate the fundamental matrix
For the base image pair (pic_a.jpg and pic_b.jpg), the estimated fundamental matrix was:
[-0.0000 0.0000 -0.0001
0.0000 -0.0000 0.0007
-0.0000 -0.0010 0.0235]

Estimated Fundamental Matrix Extra Credit

For the extra credit, I completed the extra credit of normalizing coordinates. This was achieved by:
  1. Taking the average of the points for both images
  2. Calculating C_u and C_v for both images
  3. Calculating the average distance between the points and C_u/C_v for both images
  4. Calculating the scales for both images
  5. Generating the transforms for both images
  6. Applying the transforms for both images to the base image coordinates
  7. After the fundamental matrix has been calculated, applying T_a & and T_b to the fundamental matrix to scale the matrix correctly.
The normalization result in immediate results in the accuracy of the epipolar lines in the sample images. While previously the epipolar lines were slightly off, the normalization accounted for the error. It also substantially increased the number of coordinate matches in part 3 of the assignment resulting in more point correspondance.
Sample Epipolar Lines Image 1 Sample Epipolar Lines Image 2




RANSAC

RANSAC is used with the fundamental matrix estimation function to estimate the best possible fundamental matrix. SIFT is implemented through an external library to provide a list of potential matches. Throughout the algorithm, an outlier ratio of 0.50, a p value of .99, and a theshold of 0.01 was used to filter results. The algorithm used is:
  1. Compute random sample indexes
  2. Solve for the fundamental matrix of these samples using part 2's code
  3. Calculate correspondance distances using the fundamental matrix
  4. Discard outliers
  5. Proceed only if more inliers than previously seen
  6. Note new max amount of inliers
  7. Save the inliers in A and B
  8. Save the new best fundamental matrix


Notre Dame Simple Images

Algorithm run on Notre Dame images.

Potential Correspondances Matched Correspondances
Image 1 Epipolar Lines Image 2 Epipolar Lines



Mount Rushmore Medium Images

Algorithm run on Mount Rushmore images.

Potential Correspondances Matched Correspondances
Image 1 Epipolar Lines Image 2 Epipolar Lines



Episcopal Gaudi Complex Images

Algorithm run on Episcopal Gaudi images.

Potential Correspondances Matched Correspondances
Image 1 Epipolar Lines Image 2 Epipolar Lines