Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

In this example from part 3, we could see a very good correlation in data points from both images due to the algorthms I built. More explanation will be added below.

This project required me to use algorithms to estimate the camera projection matrix which mapped 3D world coordinates to image coordinates. The Fundamental matrix will also be implemented using RANSAC. Using RANSAC, we will be able to filter away bad matches to achieve an almost near perfect result. Below will be the 3 main parts and how I implemented them.

Algorithm

Part One: Calculating camera projection matrix

The first of this part required me to implement the camera projection matrix. To construct this algorithm, I needed to setup a homogenous set of equations using the 2d and 3d points given. Once setup, setting up the linear square regression was quite easy. I noticed that if the last value in the matrix was not 1, the matrix would have repeated values and so it was not different and so I made it a 1 so they are all differnt. Below will be my projection matrix and total residual value as well as a code snippet.


...
for index = 1:size(Points_2D,1)
    twoD_P1 = Points_2D(index,1);
    twoD_P2 = Points_2D(index,2);

    x = Points_3D(index,1);
    y = Points_3D(index,2);
    z = Points_3D(index,3);

    left(leftCounter,:) = [x y z 1 0 0 0 0 -(twoD_P1)*x -(twoD_P1)*y -(twoD_P1)*z];
    leftCounter = leftCounter + 1;

    left(leftCounter,:) = [0 0 0 0 x y z 1 -(twoD_P2)*x -(twoD_P2)*y -(twoD_P2)*z];
    leftCounter = leftCounter + 1;

    right(rightCounter,:) = twoD_P1;
    rightCounter = rightCounter + 1;

    right(rightCounter,:) = twoD_P2;
    rightCounter = rightCounter + 1;
end
...

The camera center matrix was the next part of this section. The estimated camera location of the camera is above. The code is pretty simple for this part. The code will be below.

Center = (-1*M(1:3, 1:3)) *  M(1:3, 4);

Part Two: Calculating the fundemental matrix given corresponding point pairs

This part of the project will estimate the mapping of the points on an image to lines in another image. This matrix that we will get will be used to move the searching from a 2D image to a 1D so that it increases speed and efficiency. This part will have very similar parts to as part 1. Below will be snippets of the code.


	...
	for i = 1:Numberofpoints
	    valOne = Points_a(i,1);
	    PointAColOneSum = PointAColOneSum + valOne;

	    valTwo = Points_a(i,2);
	    PointAColTwoSum = PointAColTwoSum + valTwo;

	    valOne = Points_b(i,1);
	    PointBColOneSum = PointBColOneSum + valOne;

	    valTwo = Points_b(i,2);
	    PointBColTwoSum = PointBColTwoSum + valTwo;
	end

	...
	pointAMatrix = [pointAFinalVal 0 0; 0 pointAFinalVal 0; 0 0 1] * ...
    [1 0 -(PointAColOneSum/Numberofpoints); 0 1 -...
    (PointAColTwoSum/Numberofpoints); 0 0 1];
	Points_a = pointAMatrix*transpose([Points_a ones(Numberofpoints,1)]);
	Points_a = transpose(Points_a);
	...

	//now the same for point B matrix

	currentAValue = 1;
	for n = 1:Numberofpoints
	    A(currentAValue,:) = [Points_a(n,1)*Points_b(n,1) ...
	        Points_a(n,2)*Points_b(n,1) ...
	        Points_b(n,1) ...
	        Points_a(n,1)*Points_b(n,2) ...
	        Points_a(n,2)*Points_b(n,2) ...
	        Points_b(n,2) Points_a(n,1) Points_a(n,2)];
	    currentAValue = currentAValue + 1;
	end

	...
	F_matrix = U*S*transpose(V);

Part Three: Calculate the fundamental matrix using RANSAC

This part will use RANSAC to match potential matching points. We will gather the interest points, and randomly sampling the interest points. Using the RANSAC algorithm below, we calculated the best F and then matched those up. The reason we chose RANSAC is because it works really well with the outliers! If we used the result from the VLFeat, it would be really inaccurate since the input would not be right. As described in the lectures, a small threshold must be used so I used a .01 threshold. Also, I ran the algorithm 5000 times since i need to get the best modal possible! The RANSAC algorithm that I used is this:

  1. Sample (randomly) the number of points required to fit the model
  2. Solve for model parameters using samples
  3. Score by the fraction of inliers within a preset threshold of the model

below will also be a shortened algorithm to show how i went about this part:


numberOfIteration = 5000;
probFreeOfOutlier=0.99;

sizeOfMatchesA = size(matches_a,1);
inlierCount = zeros(numberOfIteration,1);
Best_Fmatrix = zeros(3,3);

MaxCurrentInlier = 0;
inlierList = [];

for index = 1:numberOfIteration

    ...

    for m = 1:1:sizeOfMatchesA
        matchAValues = [matches_a(m,1);matches_a(m,2);1];
        matchB = [matches_b(m,1);matches_b(m,2);1];
        matchBValues = transpose(matchB);
        matchList(currentIndex,1) = sum((matchBValues * EFMValues * ...
            matchAValues).^2);
        currentIndex = currentIndex + 1;
    end

    ...

    if lengthOfInliners > MaxCurrentInlier
        Best_Fmatrix = EFMValues;
        inlierList = find(matchList - (1-probFreeOfOutlier) <= 0);
        MaxCurrentInlier = lengthOfInliners;
    end

end

inliers_a = matches_a(inlierList,:);
inliers_b = matches_b(inlierList,:);

end

below are the image outputs from this step. As we could tell, mount rushmore did very well. The notre Dame photo performed okay and still saw good results but some were not. The Gaudi also performed not so great. The amount of SIFT detectors for mount rushmore had these results: found 5583 SIFT descriptors in pic a, found 5249 SIFT descriptors in pic b, Found 825 possibly matching features.