This was the project I worked on for my undergraduate thesis Reflection Removal: Improvements of the Ghosting Model (2015) at Tsinghua University.
We aimed to remove the undesired reflections when taking pictures through transparent glasses. The existing “ghosting model” failed to process pictures with repetitive patterns. After I modified the model and proposed a new regularization term in the optimization problem, my algorithm achieved better performances than the original one on both artificial images and some of the natural images.
/code/origin
contains the code of the original "ghosting model" I started with, credited to YiChang Shih (https://www.yichangshih.com/)
/code/new
constains the code of my own work based on the original model.
When taking pictures through transparent glasses, we often obtain images with undesired reflections. This is a common issue in daily lives and much effort has been devoted to separate the transmitted background layer and the undesired reflection layer.
One natural idea is to view the input image as a linear superposition of two different layers:
where
A large amount of earlier work takes multiple images or a video clip as input to effectively explore the relationship between
As pointed out by Reflection Removal Algorithms (Hu, 2016), compared to algorithms that use a single input, those who operate on a series of images usually have much better performances since they have imposed a higher level of prior knowledge. Most of the practical post-processing problems, however, appear with only one single image, and these multiple-image based methods are not applicable to general cases. Therefore, this paper focuses on developing a robust single-image based method.
The question is, to effectively separate the two layers from only one single input, what kind of prior would be essentially helpful? One straightforward idea is that the two independent layers we extract are natural images. A. Levin and Y. Weiss (2007) utilizes the sparse distribution property of natural images, using a Laplacian mixture model to constrain the distribution of the input gradients. R. Wan et al. (2017) also adopts the gradient sparsity prior to regularize
Furthermore, researchers have tried to break the symmetry between
Recently a new model proposed by Y. Shih (Reflection removal using ghosting cues, 2015) has outperformed many existing single-image based methods. Depending on the ghosting phenomenon that the reflection always appears with a shifted and attenuated version of itself, they formulate the model as
and combine it with a patch-based GMM (Gaussian Mixture Model) prior. Here
where
This model, however, fails to process images with repetitive pattern in the transmission layer. Therefore, Y. Huang et al. (2019) replaced the GMM prior by a more detailed prior containing information about the differences of the repetitive pattern between
Motivated by Y. Shih and Y. Huang et al., this paper aims to improve the reflection removal performance based on their ghosting model by the following three ways: polish the estimation of the ghosting kernel
The original optimization problem was formed as follows:
where
Our goal is to obtain better estimation of
The rest of this chapter will describe how we achieve our goals mentioned above based on the methods proposed by Y. Shih and Y. Huang et al..
Given an input image, the very first step is to estimate the ghosting kernel
The original method is to first estimate
$$\mathbf{d}k = \arg\max{\mathbf{d}} \mathcal{R}_{E_{\mathbf{Y}}}(\mathbf{d}).$$
To estimate
When the transmission layer
So far most of the previous works take the input digital image as the direct addition of several digital images. When implementing the primary model
The physical formation of the ghosting model
Therefore, we aim to find a new model
where
As shown in the following figure, The image on the left side is the input image, we combine the new model
This chapter will describe how we implement the new ideas stated in Chapter \ref{cha:idea}, present and discuss the results of our experiments.
We aim to improve the estimation of
where
Now we let
We set
Implementing alternating minimization, we split this problem into two subproblems as follows:
Each subproblem is a standard
As for initialization, let
We test our new algorithm on both synthetic images and real images. In the following diagrams, the figure on the left is the input image, the figure in the middle is the recovered result of the original algorithm, and the figure on the right is the recovered result of our new algorithm. They both show that we obtain much better results when processing images with simple transmission layer.
We also find that when dealing with more complicated
Based on the ghosting model used by Y. Shih and Y. Huang et al., we have made the following three improvements.
-
We polish the estimation of the ghosting kernel
$\mathbf{k}$ , so that the new algorithm could also process images with strong repetitive pattern in the transmission layer$\mathbf{T}$ and finally give a more accurate estimation of$\mathbf{k}$ . -
We replace the original model
$\mathbf{Y} = \mathbf{T} + \mathbf{R} \otimes \mathbf{k}$ by$\mathbf{Y} = \delta\mathbf{T} + (1-\delta)(\mathbf{R} \otimes \mathbf{k})$ , so that the new model is closer to the real physical model. -
We propose a new regularization term to minimize the correlation between the two layers, so that the algorithm could achieve better performances in terms of "separation".
Although our algorithm achieve clearer separation than many previous methods, it cannot keep the details in