Computer Vision Project

1. Intro

This project is an implementation of the RANSAC algorithm, along with camera calibration techniques. We estimate the camerica projection and fundamental matrices utilizing 2d and 3d points.

We calibrate the camera given the points, solved through decomposition and linear regression. We get the coordinates of the camera, and solve for the fundamental matrix of rank 2. Then, we implement RANSAC with the fndamental matrix,solving matching features.

2. Code

2a. Camera Calibration

The projection of the 3D world onto the 2D is represented by 'x = K [ R | t ] X', notable the instrinsic and extrinsic matrix for camera positioning and calibration.

This is the calculate_project_matrix main code. We solve the equations.


%Create matrix
M = zeros(3,4);
%set num of points for the algorithm
numPoints = size(Points_2D,1);

a = zeros(numPoints*2, 11);
b = zeros(numPoints*2, 1);


%we use this in order to figure out the camera position and angle.
%this is solved in a least squares regression linear equation

for i= 1:numPoints
   
   X = Points_3D(i,1z);
   Y = Points_3D(i,2);
   Z = Points_3D(i,3);
   
   u = Points_2D(i,1);
   v = Points_2D(i,2);
   
   %odd rows of matrix a
   a(2*i-1,:) = [X Y Z 1 0 0 0 0 -u*X -u*Y -u*Z];
   %even rows
   a(2*i,:) = [0 0 0 0 X Y Z 1 -v*X -v*Y -v*Z];
   
   b(2*i-1) = u;
   b(2*i) = v;
end


M = a\b;
M = [M;1];
M = reshape(M,[],3)';

This is code for figuring out the camera center.


Center = -M(:,1:3)\M(:,4);

2b. Fundamental Matrix Estimations

Now, we estimate the F matrix based on 2 sets of corresponding matches between the two images. Because we have 9 unknowns in the system of the current fundamental matrix, we need 8+ points to solve for the free variables. There is similarity with the camera calibration with the linear equations solving.

This is from estimate_fundamental_matrix. The code both solves the F matrix as well as having an implementation of matrix normalization. The normalization follows the following:



Points_num = size(Points_a,1);
A = [];
B = -ones(Points_num,1);

%this manages some matrix normalization to transform the matrices and
%rescale them back to penalize the parameters of F in the least squares estimate.


%means of points.
u_a = sum(Points_a(:,1))/numPoints;
v_a = sum(Points_a(:,2))/numPoints;
s = numPoints/sum(((Points_a(:,1)-u_a).^2 + (Points_a(:,2)-v_a).^2).^(1/2));
transA = [s 0 0; 0 s 0; 0 0 1]*[1 0 -u_a; 0 1 -v_a; 0 0 1];
%rescaling the matrix
Points_a = transA*[Points_a ones(numPoints,1)]';
Points_a = Points_a';

u_b = sum(Points_b(:,1))/numPoints;
v_b = sum(Points_b(:,2))/numPoints;
s = numPoints/sum(((Points_b(:,1)-u_b).^2 + (Points_b(:,2)-v_b).^2).^(1/2));
transB = [s 0 0; 0 s 0; 0 0 1]*[1 0 -u_b; 0 1 -v_b; 0 0 1];
%rescaling the matrix
Points_b = transB*[Points_b ones(numPoints,1)]';
Points_b = Points_b';


%this solves for the f Matrix

for n = 1:numPoints
    u1 = Points_a(n,1);
    v1 = Points_a(n,2);
    u2 = Points_b(n,1);
    v2 = Points_b(n,2);
    A(end+1,:) = [u1*u2 v1*u2 u2 u1*v2 v1*v2 v2 u1 v1];
end

F_matrix = A\B;
F_matrix = [F_matrix;1];
F_matrix = reshape(F_matrix,[],3);
F_matrix = transA'*F_matrix*transB;
F_matrix = F_matrix';
[U,S,V] = svd(F_matrix);
S(3,3) = 0;
F_matrix = U*S*V';

2c. Ransac

Now, the code has keypoints, descriptors, matches, camera calibration, and knows how to estimate. We now implement the RANSAC algorithm. We select some random matches/pairs, compute a solution, and interate, eventually finding the best fit given data and outliers.



%various variables to adjust the threshhold and iterations. I chose a large 10k iterations, and a low threshhold.
threshold = 0.001;
bestPoints = 0;
iterations = 10000;

i = 1;
while i < iterations
    count = 0;
    iter = 1;
    k = randsample(size(matches_a, 1), 8);
    pointsA(1:8, :) = matches_a(k, :);
    pointsB(1:8, :) = matches_b(k, :);
    fMatrix = estimate_fundamental_matrix(pointsA, pointsB);
    i = i + 1;

    for j = 1:size(matches_a, 1)
        score = abs([matches_b(j, :), 1]*fMatrix*[matches_a(j, :), 1]');
        if score < threshold
            count = count + 1;
            tempA(iter, :) = matches_a(j, :);
            tempB(iter, :) = matches_b(j, :);
            iter = iter + 1;
        end
    end

    if(count > bestPoints)
        Best_Fmatrix = fMatrix;
        bestPoints = count;
        inliers_a = tempA;
        inliers_b = tempB;
    end
end