With the knowledge of camera calibration and Epipolar Geometry, we can now improve the result we get from project 2 for keypoints matching. In this project, we are going to calculate the camera projection matrix and fundamental matrix. For Part I, we are provided with the corresponding 3d and 2d points, with which we can solve for the camera projection matrix using least squares regression. We also visulize the camera center in 3d space in this part. In Part II, given the corresponding 2d points in two images, we can solve for the fundamental matrix with least squares regression. In Part III, with RANSAC, we estimate for the "best" fundamental matrix, with which we can apply the Epipolar Constraint when we do keypoints matching. We test the image pairs from Project 2 and we and get much better results now.
Based on the nonhomogeneous provided in the slides, I implement the least square regression and figure out the unknown camera parameters. In this method, we need to constrain m_34 to be 1.
Relationship between 3d ad 2d points |
Algorithm in calculating |
The result I get for the M matrix. It is a scaled equivalent of the providing matrix in the homework description
Result of M matrix |
Groudtruth and prediction |
The with matrix M, we can calculate for center of camera
Result of camera center |
%Basic implementation
n=size(Points_a,1);
A=zeros(n,9);
Y=zeros(n,1);
for i=1:n
a=Points_a(i,:);
b=Points_b(i,:);
A(i,:)=[a(1)*b(1) b(1)*a(2) b(1) b(2)*a(1) b(2)*a(2) b(2) a(1) a(2) 1];
Y(i,:)=-1;
end
[U,S,V] = svd(A);
f=V(:, end);
F=reshape(f,[3 3])';
[U,S,V]=svd(F);
S(3,3)=0;
F_matrix=U*S*V';
Fundamental Matrix gained with the basic implementation |
Epipolar Line of the left image |
Epipolar Line of the right image |
%graduate credit implementation
mean_a=mean(Points_a,1);
sub_a=pdist2(mean_a,Points_a);
s_a=1/max(sub_a); %using the maximum value as scale
%s_a=1/std(sub_a); %using standard deviation
Ta=diag([s_a s_a 1])*([1 0 -mean_a(1);0 1 -mean_a(2);0 0 1]);
new_Points_a=[Points_a ones(size(Points_a,1),1)]*Ta';
mean_b=mean(Points_b,1);
sub_b=pdist2(mean_b,Points_b);
s_b=1/max(sub_b);
%s_b=1/std(sub_b);
Tb=diag([s_b s_b 1])*([1 0 -mean_b(1);0 1 -mean_b(2);0 0 1]);
new_Points_b=[Points_b ones(size(Points_b,1),1)]*Tb';
n=size(Points_a,1);
A=zeros(n,9);
for i=1:n
a=new_Points_a(i,:);
b=new_Points_b(i,:);
A(i,:)=[a(1)*b(1) b(1)*a(2) b(1) b(2)*a(1) b(2)*a(2) b(2) a(1) a(2) 1];
end
[U,S,V] = svd(A);
f=V(:, end);
F=reshape(f,[3 3])';
[U,S,V]=svd(F);
S(3,3)=0;
F_matrix=U*S*V';
F_matrix=Tb'*F_matrix*Ta;
Fundamental Matrix gained with the extra credit implementation |
Epipolar Line of the left image |
Epipolar Line of the right image |
In this part, we will use RANSAC conjunction with the algorithm I implemented in Part II to figure out a best fundamental matrix for the given images pairs. With the fundamental matrix, we can apply more constrains on the keypoints matching and get better matching result. We use the VLFeat in this part for finding matching candidates efficiently.
%RANSAC implementation of finding the best fundamental matrix
N=size(matches_a,1);
threshold=0.0005;
Best_num=0;
iter_num=3000;
for i=1:iter_num
sample_index=randsample(N,9);
F_matrix=estimate_fundamental_matrix(matches_a(sample_index,:), matches_b(sample_index,:));
record_num=0;
for j=1:N
cost=abs([matches_b(j,:) 1]*F_matrix*[matches_a(j,:) 1]');
if (costBest_num)
Best_Fmatrix=F_matrix;
Best_num=record_num;
end
end
%calculate the inliners based on the best fundamental matrix
temp=ones(1,N)*2*threshold;
for j=1:N
temp(j)=abs([matches_b(j,:) 1]*Best_Fmatrix*[matches_a(j,:) 1]');
end
%select the top 30 inliners with smallest error.
index=find(temp < threshold);
[~,ind] = sort(temp, 'ascend');
num=min(length(index),30);
inliers_a = matches_a(ind(1:num),:);
inliers_b = matches_b(ind(1:num),:);
Result gained with basic implementation of Part II |
We can find that we get much better result by normalizing the coordinates before computing the fundamental matrix. We only show the top 30 matching keypoints here for better visualization
Result for Episcopal Gaudi |
Result for Mount Rushmore |
Result for Notre Dame |
Result for Woodruff Dorm |