Nikhil Kulkarni

Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

The first part of this project was to compute the projection matrix given 2d and 3d coordinates. The second part was to compute the fundamental matrix of a camera given 2d coordinates from 2 different images. The 3rd part used SIFT features, RANSAC, and the method from part 2 to compute the fundamental matrix of the camera and draw epipolar lines on 2 images.

Part 1

The first part of the project took a set of 2d points and corresponding 2d points to compute the projection matrix from x = k[R t]X. I first set up a homogenous system of equations as described in the project and then used linear least squared to solve for the projection matrix. I also tried out solving using svd. Both results were the same. I ended up with a residual of 0.0445 and a camera center of <-1.5126, -2.3517, 0.2827>.

Projection matrix vs actual points

Part 2

The second part of the project took two corresponding sets of points from 2 images and computed the fundamental matrix from them. The following steps were taken:

1. Setup a system of equations using the form: uu'f11 + uv'f12 + uf13 + vu'f21 + vv'f22 + vf23 + u'f31 + v'f32 + f33
2. Use SVD to compute fundamental matrix

Epipolar lines using Fundamental Matrix

Part 3

The third part used SIFT features, RANSAC, and estimate_fundamental_matrix to draw epipolar lines. We were given 2 sets of images from which to extract SIFT features and run RANSAC. The following steps were taken.

1. Use VLFeat to find all possible corresponding points between the two images.
2. For every iteration, choose a random sample of 8 corresponding points (From 8 points algorithm) from both images
3. Compute the fundamental matrix using those 8 points
4. x*F*x' = 0. So use fundamental matrix and 8 points to compute the inliers. I used a threshold of 0.02. Goal is to get as close as possible to 0.
5. Choose fundamental matrix with highest number of inliers

The RANSAC loop is shown below

% The RANSAC loop
for i=1:1000
    corr = randperm(n,8); % Choose 8 points
    Points_A = matches_a(corr, :);
    Points_B = matches_b(corr, :);
    F_matrix = estimate_fundamental_matrix(Points_A, Points_B); % Compute fundamental matrix for these 8 points
    
    % x'*F*xT = 0
    % Compute Inliers
    threshold = 0.02;
    in_a = [];
    in_b = [];
    for i=1:n
        if (abs([matches_b(i, :) 1]*F_matrix*[matches_a(i,:) 1]') < threshold) % Get to as close as zero as possible
           in_a = [in_a; matches_a(i,:)];
           in_b = [in_b; matches_b(i,:)];
        end
    end

    % Choose max outlier code and prob code below ...

Epipolar lines using RANSAC for Mt Rushmore

This set of images (Mt Rushmore) was relatively easy to find epipolar lines for. This is because most of the initial matches are correct.

Epipolar lines using RANSAC for Notre Dame

This set of images (Notre) was harder but still not bad. Keypoints were on the same plane for this pair. The fundamental matrix did a decent job of filtering out incorrect matches.

Epipolar lines using RANSAC for Episcopal Gaudi

This set of images (Episcopal Gaudi) was hard. Almost no correct matches were found.