Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

The assigned project focused on the projection of coordinates from a 3 dimensional space to a 2 dimensional one as well as using epipolar geometry to solve the point correspondence problem. The biggest lesson that this project taught me was that if you set up your given data in a format that an algorithm can use (SVD in my case), then you can solve for the missing parts fairly accurately. The parts that needed to be completed where:

  1. Camera Calibration/Centering
  2. Finding the fundamental matrix
  3. Using RANSAC for SIFT matching

Camera Calibration

For the camera calibration problem, I set up my system of linear equations as non-homogenous so I could use SVD. After I found the projection matrix to transform the 3D world to 2D, I was able to center the camera based on the matrix that was calculated by taking the negative inverse of the first 3 columns then multiplying it by the fourth column. I found that the most challenging part of this problem was just setting up the matrix. I did it like so:


rows = size(Points_3D);
rows = rows(1);
A = zeros(2 * rows, 12);
Y = zeros(2 * rows, 1);
%Setting up matrix
for i=1:rows
    row_3D = Points_3D(i, :);
    row_2D = Points_2D(i, :);
    ind = i * 2;
    Y(ind - 1, 1) = row_2D(1);
    Y(ind, 1) = row_2D(2);
    A(ind - 1, 4) = 1;
    A(ind, 8) = 1;
    A(ind - 1, 12) = -row_2D(1);
    A(ind, 12) = -row_2D(2);
    for j = 1:3
        item = row_3D(j);
        A(ind - 1, j) = item;
        A(ind, j + 4) = item;        
        A(ind - 1, j + 8) = item * -row_2D(1);
        A(ind, j + 8) = item * -row_2D(2);
    end
end

Fundamental Matrix Estimation

This problem initially felt very similar to the previous one as the challenging part was setting up the matrix. After I set it up, I used SVD to estimate the fundamental one. I noticed that every row of the fundamental matrix always has [u' v' 1] at every 3 indexes, so I saved that and multiplied it to generate the matrix. The code used to set up the initial matrix is below.


rows = size(Points_a);
rows = rows(1);
A = zeros(rows, 9);
for i=1:rows
    row_a = Points_a(i, :);
    row_b = Points_b(i, :);
    mB = [row_b(1) row_b(2) 1];
    A(i, 1:3) = mB .* row_a(1);
    A(i, 4:6) = mB .* row_a(2);
    A(i, 7:9) = mB;
end
[U, S, V] = svd(A);
f = V(:, end);
F_matrix = reshape(f, [3 3]);

Epipolar lines using the matrix.

Matrix with RANSAC and SIFT matching correspondences

We used RANSAC to grab random subsets of corresponding pairs to be able to build our matrix. We build our matrix several times until we come up with a correct one, which makes the algorithm fairly robust to outliers. I iterated 5000 times and used a threshold of .05. The code is depicted below.


finalInliers = 0;
Best_Fmatrix = zeros(3, 3);
rows = size(matches_a, 1);
finalList = zeros(rows, 1);
for i=1:5000
    sampleIndex = ceil(rand(10, 1) .* rows);
    M = estimate_fundamental_matrix(matches_a(sampleIndex, :), matches_b(sampleIndex, :));
    curr = 0;
    currList = zeros(rows, 1);
    for j = 1 : rows
        dis = abs([matches_b(j, :) 1] * M * [matches_a(j, :) 1]');
        if (dis < .05)
            curr = curr + 1;
            currList(j) = dis;
        else
            currList(j) = 2^1500;
        end
    end
    if curr > finalInliers
        Best_Fmatrix = M;
        finalList = currList;
        finalInliers = curr;
    end
end
[~, d] = sort(finalList);
if (finalInliers > 200)
    inliers_a = matches_a(d(1:200), :);
    inliers_b = matches_b(d(1:200), :);
else
    inliers_a = matches_a(d(1:finalInliers), :);
    inliers_b = matches_b(d(1:finalInliers), :);
end

RANSAC Results