Image Inpainting

For this project, I designed and implemented process for inpainting regions in images by interpolating structure (image gradients) using a Poisson solver and color values via PDE’s (anisotropic diffusion) in C++.

With the recent popularity of generative AI (diffusion models) for a variety of visual tasks including inpainting, the goal was to take a deeper dive into classical methods for image inpainting.

Poisson Solver

The first step for this project was to implement a Poisson solver. Below, you can see a test case that runs the Poisson solver for seamless compositing given a background (water), foreground (bear), and mask. The blended result is shown in the bottom right image.

Laplacian

Next, I had to compute the Laplacian of an image. The test case below runs the code to compute the Laplacian by convolving with the appropriate kernel. Drag the slider on the image to reveal the original image and its corresponding Laplacian. The Laplacian is a second derivative and should highlight regions of the image with drastic changes in pixel intensity (i.e. edges).

Structure Tensor

I also compute the structure tensor of an image. The test case below runs the code to compute the structure tensor while ignoring pixels within the masked area. Drag the slider on the image to reveal the original image with the mask and its corresponding structure tensor. The red and blue coloring indicates a change in intensity in primarily the horizontal and vertical directions respectively. Also, note how the tensor output is correctly blacked out in the masked regions.

Structure Interpolation

With these components, the next step is to interpolate the structure of the masked regions. To do this, I first compute the structure tensor outside the masked area. Then, I run the Poisson solver with the initial structure tensor as the background and a black image as the foreground. The test case below runs the code to interpolate the structure tensor within the masked area of the image. The image on the left is the full structure tensor output and the image on the right depicts just the interpolated parts in the masked region.

Color Interpolation

Now that I have an interpolated structure tensor, I can use it to interpolate color values within the masked area. To do this, I iterate over the image and perform iterative image inpainting updates. I interleave these updates with a pass of anisotropic diffusion every third iteration. I only update pixels within the masked region. Below are the results of the algorithm on a few sample images.