Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

For this project, I worked towards estimating the fundamental matrix, from which we are able to obtain the estimated epipolar lines for an image. I sought to find the best fundamental matrix for different images using point correspondance found with SIFT - the algorithm explored in the previous project. The 3 parts of this project are as follows

  1. Finding the Camera Projection Matrix
  2. Estimating the Fundamental Matrix
  3. Finding the Best Fundamental Matrix using RANSAC

Finding the Camera Projection Matrix

To find the projection matrix M, I used the known 2 and 3 dimensional locations and set up my system of equations to solve for M by setting the product equal to 0 and using singular value decomposition as discussed in lecture.


for i=1:size(A,1)/2
    j = 2*i-1;
    k = 2*i;
    A(j, :) = [X(i) Y(i) Z(i) 1 0 0 0 0 ...
        -u(i)*X(i) -u(i)*Y(i) -u(i)*Z(i) -u(i)];
    disp(A(j,:))
    A(k, :) = [0 0 0 0 X(i) Y(i) Z(i) 1 ...
        -v(i)*X(i) -v(i)*Y(i) -v(i)*Z(i) -v(i)];
end

[U,S,V] = svd(A);
M = V(:,end);
M = reshape(M,[],3)';

For the example given I found the projection matrix to be

    0.4583   -0.2947   -0.0140    0.0040
   -0.0509   -0.0546   -0.5411   -0.0524
    0.1090    0.1783   -0.0443    0.5968

the camera center: <-1.5127, -2.3517, 0.2826>, and a residual of 0.0445.

Estimating the Fundamental Matrix

To find the fundamental matrix I used the fact that its product with the transpose of the points in one image and the points in the other should equal 0 and set up my system of equations as follows.


for i=1:size(Points_a, 1)
    A(i, :) = [u(i)*u_prime(i) v(i)*u_prime(i) u_prime(i) u(i)*v_prime(i) ...
        v(i)*v_prime(i) v_prime(i) u(i) v(i) 1];
end

I then took the singular value decomposition of the resulting matrix and used the left singular vectors as my fundamental matrix. I found that the matrix produced was already rank 2, so I did not set the smallest eigen value to 0. The matrix produced was: 

   -0.0000    0.0000   -0.0019
    0.0000    0.0000    0.0172
   -0.0009   -0.0264    0.9995

The epipolar lines for the points generated from the fundamental matrix go through the center of each point and represent the points actual location.

Finding the Best Fundamental Matrix using RANSAC

To find the best F, I used RANSAC and randomly sampled 8 points then computed the fundamental matrix using those, and then evaluated F according to the number of pFp' pairs < .01 (i.e. the number of inliers). 


for i=1:limit
    rnd = randsample(size(matches_a,1), N_1);
    F = estimate_fundamental_matrix(matches_a(rnd,:), matches_b(rnd,:));
    y = evaluate_fundamental_matrix(matches_a, matches_b, F);
    thresholded = abs(y) < .01;
    inliers = sum(thresholded);
    disp(inliers);
    if inliers > Best_inliers
       Best_inliers = inliers;
       Best_y = y;
       inliers_idx = find(thresholded);
       Best_Fmatrix = F;
    end
end


function y = evaluate_fundamental_matrix(Points_a, Points_b, F)
    u = Points_a(:, 1);
    v = Points_a(:, 2);

    u_prime = Points_b(:, 1);
    v_prime = Points_b(:, 2);

    A = zeros(size(Points_a, 1), 9);

    for i=1:size(Points_a, 1)
        A(i, :) = [u(i)*u_prime(i) v(i)*u_prime(i) u_prime(i) u(i)*v_prime(i) ...
            v(i)*v_prime(i) v_prime(i) u(i) v(i) 1];
    end
    
    y = A * reshape(F, [9,1]);
end

With this I was able to obtain a good number of correct matches on the pair of Mt. Rushmore images.

With other image pairs, the points were matched well also even without normalization. However, I believe with normalization, I would have much fewer incorrect matches.