Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

Introduction

In this project I try to use stereo images to do camera calibration using least squares method. I implement Fundamental Matrix estimation to filter SIFT-matches.

Part 1: Camera calibration

To compute the projection matrix I solve the nonhomogenous linear system using least squares. The data used is 20 corresponding 3d points and their projections into the camera plane.


	% Mx = X
	M = X \ x;
	M = [M; 1];
	M = reshape(M, [], 3)';
From the projection matrix I then compute the camera center by computing -R^-1K^-1m4.

	Q = M(:, 1:3);
	m4 = M(:, 4);
	Center = -Q\m4;
The result yielded is as follows:

	The projection matrix is:
	    0.7700   -0.4975   -0.0172    0.0080
	   -0.0581   -0.1129   -0.9467   -0.0805
	    0.1863    0.3003   -0.0495    1.0000


	The total residual is: <0.6894>

	The estimated location of camera is: <-1.5167, -2.3415, 0.2872>

Part 2: Estimating the Fundamental Matrix

In second section of the project I estimate the Fundamental Matrix by doing SVD on a matrix contaning equations between corresponding 2d point from two images. The points should share an epipolar plane and by computing the Fundamental Matrix we should be able to determine the change in camera movement between the two images.


	[U, S, V] = svd(A);
	f = V(:, end);
	F = reshape(f, [3 3]);

	[U, S, V] = svd(F);
	S(3, 3) = 0;
	F_matrix = U*S*V';
This gives the follow Fundamental Matrix for the two examples images:

	F_matrix =

	   -0.0000    0.0000   -0.0019
	    0.0000    0.0000    0.0172
	   -0.0009   -0.0264    0.9995
Table 1: Shows the epipolar lines found for the two examples images.

Part 3: Improving the Fundamental Matrix with RANSAC

In the last part of the project I try to compute the Fundamental Matrix for two images whose corresponding points are extracted using the SIFT-library VLFeat. Since the library yields many matches we use RANSAC to find the 8 that yields the best Fundamental Matrix.

The best Fundamental Matrix is defined as the matrix which yields the highest amount of RANSAC inliers. The error for points is the result of computing x^TFx', since this should be 0 for a perfect Fundamental Matrix.


	% The code for scoring a Fundamental Matrix

	for i=1:size(matches_a, 1)
			% calculate distance matrix, perfect fit should be trans(x) * F * x' = 0
			res = abs([matches_a(i, :) 1] * F * [matches_b(i, :) 1]');
			if res <= inliers_threshold
					inliers = [inliers, i];
					residues = [residues, res];
			end
	end
	score = size(inliers, 2) / size(matches_a, 1);

Once the best Fundamental Matrix is selected the inliers to that matrix is returned as good correspondences. Below is the results of running this pipeline on several image pairs.

It should be noted that the result of RANSAC heavily depends on the tuning of the inlier threshold value. Since a too strict threshold will only count almost perfect matches to the model and a too lax will not properly exclude outliers. In the end I settled values between 0.01 and 0.04 depending on the picture set.

As seen above using the RANSAC with the Fundamental Matrix works pretty well though this can certianly be improved. For example by using the normalized eight-point algorithm.