Project 1: Image Filtering and Hybrid Images
Image from Apple's Developer website.
How does image filtering work? Suppose you are given an image filter, a 3x3 matrix in the example on the right, and an image, which can be represented as a matrix of dimension x by y by channel. You can then place the filter over the image and perform the operation in a sliding window manner until the filter's center has passed over every pixel in the image. The operation is a weighted sum of itself and the neighboring pixels. In the example, image(0,0) is multiplied by the filter(0,0). Then, image(0,1) is multiplied by the filter(0,1) and so forth until all nine values of one matrix have been multiplied by all nine values of the other and the resulting values have been summed. But, what about the edges of the image? How do we handle when the filter needs to be centered on a corner of the image? This is solved by padding, which means including a border around the image. Padding can be done in a few ways; one can simply add a band of zeros around the image, one can copy the edge of the image and place that as the border, etc. The padding method I chose was mirroring because it led to the most natural looking results. With a padding that utilizes a border consisting of zeros, when the different filters are performed, it leads to issues like a dark black ring forming around the perimeter of the image as a remnant of that padding method.
The general method I used to perform the filter is as follows:
- Create a mirrored padding around the image.
- Iterate through the image pixel by pixel, applying the filter to each pixel.
In the code given to us to test the hybrid image, the low frequencies were extracted using a Gaussian filter. While at first I left it as is, I realized it was taking too long to run. So, rather than leaving it a large N x N filter, I looked at how we might be able to make the process faster.
With filters that are separable, when you separate them and perform the process, it's much faster. Let's say we have a filter F and it can be separated into two filters, F1 and F2. You can tell if a filter is separable if the rank of the filter is 1. Then, the filter can be separated in matlab using the svd function, which returns a diagonal matrix with the same dimensions as the input filter, D, and two other matrices, M1 and M2, both with the same dimensions as the input filter. Here, the filter is equal to M1*D*M2'. We can actually get the horizontal and vertical vectors from the first columns of M1 and M2, scaled by the square root of the first value in D (the only nonzero value in the matrix since the filter is of rank 1). For a more complete explanation, Matlab has created a tutorial on how to do this.
The code for how I separated filters automatically is below. The automatic portion of this used the rank function in matlab to check what the rank of the filter was and from there, I used the equations mentioned above to get the vertical and horizontal vector (1xN and Nx1) separations of the filter matrix. Then, I recursively call my imfilter function twice so that I'm filtering with both the 1xN and Nx1 separations of the filter. I checked this code on a test gaussian filter to see if it worked and it did, with a few errors due to precision handling. Any advice with better precision handling going forward would be appreciated.
if rank(filter) == 1 && filter_row>1 && filter_col>1
[M1,diag_mat,M2] = svd(filter);
vertical = M1(:,1) * sqrt(diag_mat(1,1));
horizontal = M2(:,1)' * sqrt(diag_mat(1,1));
output = my_imfilter(image, vertical);
output = my_imfilter(output, horizontal);
return;
end
|
Can you tell which Gaussian blur is the unseparated version and which version has been separated by my addition? Hover below for the answer:
Results of my imfilter function
For comparison, here's the original image being filtered:
|
|
Top Row: Identity Filter, Box Blur, Gaussian Blur.
Bottom Row: High Pass Filter, High Pass Discrete Laplacian Filter, Oriented Filter (Sobel Operator).
Results of Hybrid Images
Cutoff rate used: 6.
|
|
|
Top Row: Original image of the dog, low frequency image.
Middle Row: Original image of the cat, high frequency image.
Bottom Row: The hybrid image at various scales.
Another Hybrid Example
I also decided to try this out on an image of Britney Spears and I. I realized after testing this out on several different variations (me and a husky, me and a chicken, etc.) that I needed a photo where I didn't have bangs, otherwise my black hair would show through no matter whether I made my image the high or low frequency and despite changing the cutoff rate multiple times. I finally settled with this, which actually looked okay. Cutoff rate used: 10.
|
|
|
Top Row: Original image of Britney, high frequency image.
Middle Row: Original image of me, low frequency image.
Bottom Row: The hybrid image at various scales.