Computer Vision Project

Project 3: Camera Calibration and Fundamental Matrix Estimation with RANSAC

Goal

The goal of this project is to implement the geometrical aspects of camera and images such as projection matrix and fundamental matrix. With the help of such new camera calibration techniques, our goal is to improve our SIFT matching by filtering spurious matches based on epipolar lines.

Part I: Camera Projection Matrix

First, we will find the projection matrix (also called camera matrix). This is a 3x4 matrix which described the mapping of a pinhole camera from 3D points in the world to 2D points in an image. Using this matrix, we can find the position of the camera respective to the world. The following equation described the algebraic approach to finding projection matrix.

In order to solve the equation for the projection matrix, least square regression method was used. Here is the code snippet from calculate_projection_matrix.m file:


n = size(Points_2D, 1);
A = zeros(n * 2, 11);
b = zeros(n * 2, 1);

for i=1:n
    A(2 * i - 1, :) = [Points_3D(i, 1), Points_3D(i, 2), Points_3D(i, 3), 1, 0, 0, 0, 0, ...
        -Points_2D(i, 1)*Points_3D(i, 1), -Points_2D(i, 1)*Points_3D(i, 2), ...
        -Points_2D(i, 1)*Points_3D(i, 3)];
    A(2 * i, :) = [0, 0, 0, 0, Points_3D(i, 1), Points_3D(i, 2), Points_3D(i, 3), 1, ...
        -Points_2D(i, 2)*Points_3D(i, 1), -Points_2D(i, 2)*Points_3D(i, 2), ...
        -Points_2D(i, 2)*Points_3D(i, 3)];
    b(2 * i - 1, :) = Points_2D(i, 1);
    b(2 * i, :) = Points_2D(i, 2);
end;

X = transpose([A\b;1]);
M = [X(1:4); X(5:8); X(9:12)];
M= -M / sqrt(sumsqr(M));

The results I got for the projection matrix were slightly different than the actual expected results. This is because of some floating point rounding done by Matlab when solving the equation. Once I obtained the projection matrix, I calculated the camera center using the projection matrix using the below formula:

Here is the Matlab equivalent of the above equation:


Q = M(:, 1:3);	
m4 = M(:, 4);
Center = -inv(Q) * m4;

Because of slight rounding error when calculating the projection matrix, my camera center is also slightly shifted (by very small fraction). This can be described from results below:

Part II: Fundamental Matrix Estimation

The core of this part is to find the fundamental matrix, which is a 3x3 matrix. It will be use to generate epipolar lines which relates corresponding points in stereo images. The mathematical definition of fundamental matrix is as follows:

In order to improve the accuracy of fundamental matrix, I also normalized the coordinate points (part of the extra credit). The code snippet describes my_normalize function used later when estimating fundamental matrix.

c = mean(pts(1:2, :),2);
d = (pts(1:2, :) - repmat(c, [1, size(pts, 2)]));
d = sqrt(2)/mean(sqrt(sum(d.*d, 1)));
	       
M = [d,         0,      -d*c(1);
     0,         d,      -d*c(2);
     0,         0,          1];

Heres my implementation for estimating fundamental matrix using normalization.

% Converting 2d to 3d
A = [Points_a, ones(length(Points_a), 1)];
B = [Points_b, ones(length(Points_b), 1)];

% Normalizing the points
N1 = my_normalize(A);
N2 = my_normalize(B);
x1 = N1*A';
x2 = N2*B';

% Constructing the M matrix.
M = [];
for i = 1:length(x1)
    x = x2 (: , i )* x1 (: , i )';
    M(i,:) = x(:)';
end

% SVDing
[~,~,V] = svd(M);
v = V(:,end);
Fn = reshape (v ,[3 3]);

% Enforcing rank 2 contraint
[U,S,V] = svd(Fn);
Fn = U*diag([S(1,1) S(2,2) 0])*(V');

% Denormalizing
F_matrix= N2'*Fn*N1;

Here are the epipolar lines found using fundamental matrix.

Part III: Fundamental Matrix with RANSAC

Compared to project 2 where we only used our SIFT implementation, I used RANSAC (Random Sample Concensus) along with SIFT to filter some mismatches generated by SIFT. Here is the code snippet for RANSAC

% main loop
for i = 1:iterations
    sample = randsample(size(matches_a, 1), 8);
    FM = estimate_fundamental_matrix(matches_a(sample, :), matches_a(sample, :));
    err = zeros(1, size(A,2));
    for row = 1:size(err, 2)
        err(row) = B(:, row)'*FM*A(:,row);
    end
    FM_A = FM*A;
    FM_B = FM'*B;
    dist = err.^2./(FM_A(1,:).^2 + FM_A(2,:).^2 + FM_B(1,:).^2 + FM_B(2,:).^2);
    inliers = find(dist < delta);
    if length(inliers) > bestScore
        bestScore = length(inliers);
        bestModel = inliers;
        Best_Fmatrix = FM;
    end
end

Here are the results using RANSAC fundamental matrix when drawing epipolar lines