Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

With the knowledge of camera calibration and Epipolar Geometry, we can now improve the result we get from project 2 for keypoints matching. In this project, we are going to calculate the camera projection matrix and fundamental matrix. For Part I, we are provided with the corresponding 3d and 2d points, with which we can solve for the camera projection matrix using least squares regression. We also visulize the camera center in 3d space in this part. In Part II, given the corresponding 2d points in two images, we can solve for the fundamental matrix with least squares regression. In Part III, with RANSAC, we estimate for the "best" fundamental matrix, with which we can apply the Epipolar Constraint when we do keypoints matching. We test the image pairs from Project 2 and we and get much better results now.

Part I: Camera Projection Matrix

Based on the nonhomogeneous provided in the slides, I implement the least square regression and figure out the unknown camera parameters. In this method, we need to constrain m_34 to be 1.

Relationship between 3d ad 2d points

Algorithm in calculating

The result I get for the M matrix. It is a scaled equivalent of the providing matrix in the homework description

Result of M matrix

Groudtruth and prediction

The with matrix M, we can calculate for center of camera

Result of camera center

Part II: Fundamental Matrix Estimation

Basic Implementation


%Basic implementation
n=size(Points_a,1);
A=zeros(n,9);
Y=zeros(n,1);
for i=1:n
    a=Points_a(i,:);
    b=Points_b(i,:);
    A(i,:)=[a(1)*b(1) b(1)*a(2) b(1) b(2)*a(1) b(2)*a(2) b(2) a(1) a(2) 1];
    Y(i,:)=-1;
end
[U,S,V] = svd(A);
f=V(:, end);
F=reshape(f,[3 3])';
[U,S,V]=svd(F);
S(3,3)=0;
F_matrix=U*S*V';

Fundamental Matrix gained with the basic implementation

Epipolar Line of the left image

Epipolar Line of the right image

Grduate Credit Implementation


%graduate credit implementation
mean_a=mean(Points_a,1);
sub_a=pdist2(mean_a,Points_a);
s_a=1/max(sub_a); %using the maximum value as scale
%s_a=1/std(sub_a);  %using standard deviation
Ta=diag([s_a s_a 1])*([1 0 -mean_a(1);0 1 -mean_a(2);0 0 1]);
new_Points_a=[Points_a ones(size(Points_a,1),1)]*Ta';

mean_b=mean(Points_b,1);
sub_b=pdist2(mean_b,Points_b);
s_b=1/max(sub_b);
%s_b=1/std(sub_b);
Tb=diag([s_b s_b 1])*([1 0 -mean_b(1);0 1 -mean_b(2);0 0 1]);
new_Points_b=[Points_b ones(size(Points_b,1),1)]*Tb';

n=size(Points_a,1);
A=zeros(n,9);
for i=1:n
    a=new_Points_a(i,:);
    b=new_Points_b(i,:);
    A(i,:)=[a(1)*b(1) b(1)*a(2) b(1) b(2)*a(1) b(2)*a(2) b(2) a(1) a(2) 1];
end
[U,S,V] = svd(A);
f=V(:, end);
F=reshape(f,[3 3])';
[U,S,V]=svd(F);
S(3,3)=0;
F_matrix=U*S*V';

F_matrix=Tb'*F_matrix*Ta; 

Fundamental Matrix gained with the extra credit implementation

Epipolar Line of the left image

Epipolar Line of the right image

Part III: Fundamental Matrix with RANSAC

In this part, we will use RANSAC conjunction with the algorithm I implemented in Part II to figure out a best fundamental matrix for the given images pairs. With the fundamental matrix, we can apply more constrains on the keypoints matching and get better matching result. We use the VLFeat in this part for finding matching candidates efficiently.


%RANSAC implementation of finding the best fundamental matrix
N=size(matches_a,1);
threshold=0.0005;
Best_num=0;
iter_num=3000;
for i=1:iter_num
    sample_index=randsample(N,9);
    F_matrix=estimate_fundamental_matrix(matches_a(sample_index,:), matches_b(sample_index,:));
    record_num=0;
    for j=1:N
        cost=abs([matches_b(j,:) 1]*F_matrix*[matches_a(j,:) 1]');
        if (costBest_num)
        Best_Fmatrix=F_matrix;
        Best_num=record_num;
    end
end

%calculate the inliners based on the best fundamental matrix
temp=ones(1,N)*2*threshold;
for j=1:N
    temp(j)=abs([matches_b(j,:) 1]*Best_Fmatrix*[matches_a(j,:) 1]');
end

%select the top 30 inliners with smallest error.
index=find(temp < threshold);
[~,ind] = sort(temp, 'ascend');
num=min(length(index),30);
inliers_a = matches_a(ind(1:num),:);
inliers_b = matches_b(ind(1:num),:);

Results with Basic implementation of Part II

Result gained with basic implementation of Part II

Results with Graduate Credit implementation of Part II

We can find that we get much better result by normalizing the coordinates before computing the fundamental matrix. We only show the top 30 matching keypoints here for better visualization

Result for Episcopal Gaudi



Result for Mount Rushmore



Result for Notre Dame



Result for Woodruff Dorm