Random Chance

As is expected, performance from guessing the categories randomly hovers around 6-7% (1 / 15 ~= 0.667).

Tiny Images and KNN

We now employ the strategy of tiny images with k-nearest neighbors. We resize the image to a smaller 16x16 image, adjust the mean of the pixels to 0, and normalize the pixel values to unit length. We will use these normalized pixel values as features and find our nearest neighbors using a mininum L2 distance. A higher value of k means we consider more nearby images. Will this improve performance?

Example large and tiny images side by side (enlarged):

Performance after varying the number of nearest neighbors:

k	1	5	10	20	30	50	100
Performance	22.5%	21.5%	21.3%	22.2%	21.3%	20.3%	19.4%

As seen above, a larger k does not necessarily mean better performance, at least while using a L2 distance measure. Good performance using knn requires a direct correspondence between the distance measure and the classification categories. Clearly, spatially located pixel intensity is not necessarily a fool-proof indicator of a category placement, although it is easy to imagine that this could work for images where the pixel intensities are correlated (such as a bed being in the middle of the image every time).

Bag of SIFT and KNN

The next step is to use SIFT features as our defining image features. Our procedure is to identify SIFT features for our training images, cluster them with kmeans into a vocabulary of a fixed size, then count how many SIFT features have a closest proximity to each cluster for a given image. We can then use these resulting histograms to classify with k-nearest neighbors. It took a good amount of tweaking to reach a performance of greater than 50%. Notable improvements were increases in knn neighbors (up to a point) and adjustments of SIFT sampling from the images for vocab and training/testing. Below are some of the iterations of tweaking.

Vocab size	All SIFT width	Vocab SIFT step size	Train/Test SIFT step size	Max # of vocab SIFT features per image	Max # of SIFT features per image	Nearest Neighbors	Performance
200	16	5	5	50	50	1	24%
200	16	50	25	50	300	1	43.9%
100	16	15	15	50	500	25	50.3%
200	16	22	10	75	250	20	53.1%

Scene classification results visualization

Accuracy (mean of diagonal of confusion matrix) is 0.531

We ultimately ended up with a larger sampling for our vocabulary formation which seemed to be a catalyst for our improvments. Further tweaking would surely improve results even more, but we are now going to move on to our main focus for the assignment.

Bag of SIFT and Support Vector Machine

We now utilize the same SIFT cluster feature histograms as before, except this time we run them through one vs all support vector machines and take the most positive result as our classification. Like the recent methodology, this one took some parameter tuning. First we will briefly discuss our cross validation methodology.

Cross Validation

The regular Performance column below reports the best performance found from running the SVM approximately 5 times. However, a better approach to this is using some type of cross validation. Currently, we use the same train/test split for our training and evaluation. To get a better measure with a single run, I randomly shuffled this split 10 times and reported the average and standard deviation of the overall performances. This was a bit more methodical than the previous method of reporting performance which gives us a more reliable metric.

Now we will move on to the results. One note is about our "features per image" parameter. This parameter samples the SIFT features for an image. The number of SIFT features produced is normally quite high, so we make sure to just get a sampling that is representative of the image to save computation time. Here are just a few of the different parameter runs I performed where I saw leaps in performance:

Vocab size	All SIFT width	Vocab SIFT step size	Train/Test SIFT step size	Max # of vocab SIFT features per image	Max # of SIFT features per image	SVM Lambda	Performance (best of 5)	Cross Validation (standard deviation)
100	8	25	10	100	200	0.1	39.6%	36.72% (0.02836)
100	32	40	20	10	50	0.1	48.1%	47.5% (0.02220)
200	16	22	10	75	250	0.0002	63.0%	62.7% (0.01365)
10000	16	22	10	75	250	0.0002	67.9%	67.69% (0.01040)

It became very clear through parameter tweaking that the parameters wildly affected the accuracy of the classification. Performance was very sensitive to the SIFT feature width and SVM lambda, especially. I am confident that there are even better parameter configurations, but I ended up finding the best performance with a SIFT width of 16, a larger vocab size, and a smaller lambda. Early on, it was difficult to determine which parameter was the parameter to change to boost performance. For example, I tested both 8 and 32 SIFT feature widths as seen above. I saw a performance increase with this, but a feature width of 32 ended up taking far too long (~45 minutes vs ~15 minutes). Later, I discovered that a proper small lambda value actually produced significant accuracy boosts to all configurations at the cost of only an extra second or two. The vocab size was also an important performance booster but at the cost of inceased computation time (we will visit this later). The SIFT sample size for the vocab and train/test images actually turned out to be less important than I thought; tweaking these values did not have a huge impact on accuracy or computation time (although of course a very small step size took significantly longer). I still tried to keep with a small sample size for vocab construction and a larger one for train/test features as the vocabulary aims to be a broad sampling of the categories rather than a specific train/test image classification.

Finally, here are our detailed results for the top performing configuration: