Introduction
From 1909-1915, with support from Tsar Nicholas II, scientist and photographer Sergei Mikhailovich Prokudin-Gorskii set out to document the russian empire and did so by taking thousands of color photographs. There was no way to take color photographs however he was convinced that color photography was the wave of the future. He had the idea to record three separate exposures onto glass plates using a red, green and blue filter. At the time there was also no way of printing or viewing these images, but he envisioned a way to project these images on top of each other to view them in color. Although his visions never came to life his glass plate images survived and were purchased by the library of congress and made available online. The goal of this project is to take these digitized glass plate images and align and combine them to produce color images.
Glass Plate Images with Red, Green and Blue Filters
Approach
Each digitized image contains three glass plates stacked on top of each other in blue, green, red order. The
first step in colorizing these images is to split the image vertically in three equal parts to separate these
color channels.
Next, in order to align the images, one can fix one of the channels, in my case the blue channel, and displace
the other channels to align with the fixed channel. The simplest way to achieve this is to exhaustively search
over a window of possible displacements and score each displacement according to alignment metric. At first
I tried to use normalized cross-correlation (NCC) for my alignment which computes a dot product between two
flattened and normalized images providing a score that represents the images similarity, however in practice
I found this metric performed poorly. Afterwards I tried using the L2 Norm (Euclidean Distance), which instead
computes a sum over pixel values, and serves the same purpose in terms of scoring. I found this metric to produce
much better results, and thus decided to use it as my alignment metric going forward.
Although I found the L2 Norm to perform better as a metric, due to the nature of glass plate images a lot
of noise is present at the edge of the images that skew the results of the metric if they are included. To solve
this I computed the L2 Norm on the image after its edges had been cropped by 20% of the size of the image (a number
which I found through trial and error).
This method works well when performed on small image formats like jpg, however the exhaustive search proves to be
extremely costly when performed on larger image formats such as tif. To speed up the search I implemented an algorithm
that uses image pyramids. Image pyramids represent the image at multiple scales, in my case each level represent a
scale by a factor of two. I implemented my algorithm in an iterative manner that first creates an image pyramid by scaling
down the image until it reaches a size roughly equivalent to the size of the smaller jpg images that performed well. I
then iterated backwards over this pyramid I created so I start at the furthest scaled down image and work up until reaching
the original image. Performing a search on a scaled down version of the image is less computationally intensive,
and we can use this search to create a guess of the correct displacement that is used as a starting place
to search scaled up images. A key realization in achieving further speedup was that after the intial scaled down search,
assuming the initial search window was large enough to find the correct displacement at that level, subsequent levels
can only be one further pixel away in their displacement.
With these optimizations the image alignment algorithm performed well on all images except for the image of Emir, which
contains too much blue within the image, leaving very little data in the other color channels to align on and thus I was
not able to produce a perfect alignment. Alternative alignments are explored later in this project that correct this.
Results
Below are the results of colorizing a set of Prudukin-Gorskii images using my base alignment algorithm with image pyramids. I used a search window of [-50, 50] for the initial coarse search and [-1, 1] for all subsequent scales. The initial window was determined through trial and error as a window that was large enough to find the correct displacement at the coarsest scale. Before displaying the images I also applied a fixed crop of 8% of the image, which I found to be just enough to get rid of any artifacts on the edges of the image without removing too much of the contents.
| Small Images | ||
|---|---|---|
 Cathedral |
 Monastery |
 Tobolsk |
| Large Images | ||
|---|---|---|
 Church |
 Emir |
 Harvesters |
 Icon |
 Lady |
 Melons |
 Onion Church |
 Sculpture |
 Self Portrait |
 Three Generations |
 Train |
|
Additional Images
I also performed my base alignment algorithm on an additional selection of Prokudin-Gorskii images that appealed to me.
| Additional Images | ||
|---|---|---|
 Canal Green: (24, 28), Red: (48, 47) |
 Capri |
 Conservatory |
 Estate |
 Lastochinko |
 Milan Cathedral |
Bells and Whistles
Sobel Edge Detection
As mentioned previously, the L2 Norm did not perform well enough as a metric in the case of the image of Emir. To fix this I implemented Sobel edge detection which applies the Sobel kernels to a grayscale version of the image through convolution and computes the magnitude of the gradient at each pixel. This performed much better than L2 in this case.
Before
Green: (24, 48), Red: (43, 60)
After
Green: (24, 48), Red: (40, 106)
White Balancing
Many of the images have a warm hue to them, and in order to correct this I decided to apply white balancing. In order to achieve this I used the gray-world assumption. This assumption speculates that in an image with a variety of colors, the average color in the image is gray, or in other words that on average each color channel is equal in magnitude. To implement this I first calculate the average intensity across all pixels and color channels in the image. Then dividing by the average rgb values yields a scalar for each color channel that I can element wise multiply each channel by and effectively white balance the image.
Before
After
