Nikhil Kulkarni

Project 4 / Scene Recognition with Bag of Words

In this project, we used the bag of visual words model to classify images into one of 15 categories. We started with a simple sample of the image with tiny images which takes a 16x16 patch of an image and uses the nearest neighbor classifier to find the closest test image to assign its class to. Then we moved to implement bag of sifts which creates sift features for every image and used the nearest neighbor classifier once again to find the closest match for each image. Finally we used the svm classifier along with bag of sifts to achieve the best results.

Building Vocabulary

For building a vocabulary, I simply used vl_sift and vl_kmeans to group sift features together. We had a vocabulary size of 200. I started with using a step size of 3 for vl_sift. This took >8 hours to run. So I changed to a step size of 10 for building vocab but found that smaller SIFT features tend to have slightly better result. I had a slightly better accuracy with step size of 3 than with step size of 10. (0.631 vs 0.667). The code included in the project uses a step size of 10 for building the vocabulary because step size of 3 was taking too long.

Tiny Images + Nearest Neighbor

The tiny images + nearest neighbor pipeline was the worst performing pipeline which is unsurprising since it doesn't do anything very intellgent during image sampling. It simply creates a 16x16 patch of the image. This pipeline achieved 0.201 accuracy. The steps followed are listed below:

1. Create 16x16 patch of image
2. For every patch find the nearest neighbor (euclidean) in the image space.
3. Assign that image the class of it's nearest neighbor.

One advantage of 1-nearest neighbor is that it's very simple. It requires no training and can be used to created any boundary. However, the downside is that it's not very flexible.

% SIFT + closest visual word + histogram creation
for i=1:n
   [locations, SIFT_FEATURES] = vl_dsift(single(imread(image_paths{i})), 'step', 8);
   dist = vl_alldist2(single(SIFT_FEATURES), vocab);
   % sift x vocab
   % Row-wise sort bc of vl_alldist2 - Entries in 1st column are closest
   % cluster centers
   [S, I] = sort(dist, 2);
   histogram = zeros(1, k);
   
   for ind=1:size(I, 1)
       histogram(1, I(ind, 1)) = histogram(1, I(ind, 1)) + 1;
   end
   image_feats(i, :) = histogram/norm(histogram, 2);  
end