Cross-Field Joint Image Restoration via Scale Map
Summary (2 min read)
1. Introduction
- Images captured in dim light are hardly satisfactory.
- This enables a configuration to take an NIR image with less noisy details by dark flash [11] to guide corresponding noisy color image restoration.
- In the second row, edge gradients have opposite directions in the two images, which cause structural deviation.
- These issues are caused by inherent discrepancy of structures in different types of images, which the authors call crossfield problems.
- The authors propose a framework via novel scale map construction.
2. Modeling and Formulation
- The authors system takes the input of a noisy RGB image I0 and a guidance image G captured from the same camera position.
- Other cross-field configurations are allowed in their framework, presented in Section 4.
- Pixel values in each channel are scaled to [0, 1].
- Its additional benefit is the special role as latent variables to develop an efficient optimization procedure.
2.1. Data Term about s
- It controls the penalty when computing si for different pixels.
- The final cost resulted from |si∇Gi − ∇Ii| is dependent on the value of ∇Gi.
- If ∇Gi and ∇Ii are doubled simultaneously, although s remains the same, the cost from |si∇Gi −∇Ii| will get twice larger.
- It removes the unexpected scaling effect caused by ∇Gi. max(|∇kGi|, ε) returns the larger value between |∇kGi| and ε.
2.2. Data Term for I
- E2(I) requires the restoration result not to wildly deviate from the input noisy image I0 especially along salient edges.
- The robust function ρ helps reject part of the noise from I0.
2.3. Regularization Term
- The authors regularization term is defined with anisotropic gradient tensors [13, 4].
- S values should change smoothly or be constant along an edge more than those across it.
- The authors anisotropic tensor scheme preserves sharp edges according to gradient directions of G. When ∇Gi is much smaller than η, Eq. (8) degrades to 0.5 · 1 and the structure tensor is therefore isotropic.
- Stronger smoothness is naturally imposed along edges.
2.4. Final Objective Function
- This objective function is non-convex due to the involvement of sparsity terms.
- Joint representation for s and I in optimization further complicates it.
- Naive gradient decent cannot guarantee optimality and leads to very slow convergence even for a local minimum.
- The authors contrarily propose an iterative method, which finds constraints to shape the s map according to its characteristics and yields the effect to remove intensive noise from input I0.
3. Numerical Solution
- To solve the non-convex function E(s, I) defined in Eq. (14), the authors employ the iterative reweighted least squares (IRLS), which make it possible to convert the original problem to a few corresponding linear systems without losing generality.
- This process, however, is still nontrivial and needs a few derivations.
- To ease derivation, the authors re-write Eq. (14) in the vector form by taking the expression in Eq. (15) into computation.
- Cx and Cy are discrete backward difference matrices that are used to compute image gradients in the x− and y−directions.
3.1. Solver
- It yields zero cost for E3(s), a nice starting point for optimization.
- This initialization also makes the starting ∇I same as∇G with many details.
- Usually, 4-6 iterations are enough to generate visually compelling results.
- The authors solved it using pre-conditioned conjugate gradient (PCG).
3.2. Why Does It Work?
- According to the linear system defined in Eq. (21), the resulting si for pixel i is a weighted average of pi,x∇xIi ≈ ∇xIi/∇xGi and pi,y∇yIi ≈ ∇yIi/∇yGi, whose weights are determined by (Ax)ii and (Ay)ii.
- Ii is larger than the other term, in solving for I according to Eq. (23), si reduces the gradient in the x-direction and increases the other so that ∇Ii lies close to s∇Gi.
- Eventually when the two estimates meet each other, s converges; I is accordingly optimal.
- (f)(g) are maps produced in two iterations, and (h) shows the final s. Initially the map is noisy because of confusing or contradictive gradient magnitudes and directions in the input images.
- As of more iterations being taken, it becomes better regularized while not overly smoothed.
4. Experiments
- Suppose the two input images – one is noisy and the other is clean – are aligned.
- The authors method, by only taking the IR flashed image as G, accomplishes the comparable result shown in Fig.
- By applying their method to singleimage dehazing result that is noisy and the NIR input, the authors can improve the quality.
- The authors restoration result with an NIR image as guidance G is more visually pleasing.
- More results from their system are available in the project website (see the title page), including those of depth image enhancement using Kinect.
5. Conclusion and Limitation
- The authors have presented a system effective for cross-field joint image restoration.
- Unlike transferring details or applying joint filtering, the authors explicitly take the possible structural discrepancy between input images into consideration.
- The authors objective functions and optimization make good use of the guidance from other domains and preserve necessary details and edges.
- The limitation of their current method is on the situation that the guidance does not exist, corresponding to zero∇G and non-zero ∇I∗ pixels.
- Because the guidance does not exist, image restoration naturally degrades to single-image denoising.
Did you find this useful? Give us your feedback
Citations
267 citations
Cites background or methods or result from "Cross-Field Joint Image Restoration..."
...For a target image, the guidance image can either be the target image itself [10,6], highresolution RGB images [6,2,3], images from different sensing modalities [11,12,5], or filtering outputs from previous iterations [9]....
[...]
...Image filtering with a guidance signal, known as joint or guided filtering, has been successfully applied to a variety of computer vision and computer graphics tasks, such as depth map enhancement [1,2,3], joint upsampling [4,1], cross-modality noise reduction [5,6,7], and structure-texture separation [8,9]....
[...]
...(c) Restoration [5] (d) Ours (g) Restoration [5] (h) Ours...
[...]
...The filtering results by our method are comparable to those of the state-of-the-art technique [5]....
[...]
...Guided by a flash image, the filtering result of our method is comparable to that of [5], as shown in Figure 8(e)-(h)....
[...]
183 citations
165 citations
Cites background from "Cross-Field Joint Image Restoration..."
...from the reference image to the target image for color/depth image super-resolution [15], [16], image restoration [36], etc....
[...]
162 citations
123 citations
References
12,560 citations
"Cross-Field Joint Image Restoration..." refers background in this paper
...Our regularization term is defined with anisotropic gradient tensors [13, 4]....
[...]
8,738 citations
"Cross-Field Joint Image Restoration..." refers background in this paper
...Simple joint image filtering [18, 8] could blur weak edges due to the inherent smoothing property....
[...]
7,912 citations
3,668 citations
2,215 citations
"Cross-Field Joint Image Restoration..." refers background in this paper
...Simple joint image filtering [18, 8] could blur weak edges due to the inherent smoothing property....
[...]
Related Papers (5)
Frequently Asked Questions (13)
Q2. Why did Krishnan and Zhang develop a method to enhance color images?
because of the popularity of other imaging devices, more computational photography and computer vision solutions based on images captured under different configurations were developed.
Q3. What is the simplest way to solve the non-convex function E(s,?
To solve the non-convex function E(s, I) defined in Eq. (14), the authors employ the iterative reweighted least squares (IRLS), which make it possible to convert the original problem to a few corresponding linear systems without losing generality.
Q4. What is the effect of the iterative method?
The authors contrarily propose an iterative method, which finds constraints to shape the s map according to its characteristics and yields the effect to remove intensive noise from input I0.
Q5. What is the result of the restoration of the d image?
The restoration result shown in (d) is with much less highlight and shadow, which is impossible to achieve by gradient transfer or joint filtering.
Q6. What is the s map of the image?
Their estimated s map shown in (c) contains large values along object boundaries, and has close-to-zero values for highlight and shadow.
Q7. What is the method for restoring a color image?
Since the two input images are color ones under visible light, the authors use each channel from the flash image to guide image restoration in the corresponding channel of the nonflash noisy image.
Q8. What is the key to the structure of G?
The authors introduce an auxiliary map s with the same size as G, which is key to their method, to adapt structure of G to that of I∗ – the ground truth noise-free image.
Q9. What is the limitation of their current method?
The limitation of their current method is on the situation that the guidance does not exist, corresponding to zero∇G and non-zero ∇I∗ pixels.
Q10. What is the main advantage of using flash to restore a color image?
This enables a configuration to take an NIR image with less noisy details by dark flash [11] to guide corresponding noisy color image restoration.
Q11. What is the function used to remove outliers?
Further to avoid the extreme situation when∇xGi or∇yGi is close to zero, and enlist the ability to reject outliers, the authors define their data term asE1(s, I) = ∑i( ρ(|si −pi,x∇xIi|)+ρ(|si−pi,y∇yIi|) ) , (4)where ρ is a robust function defined asρ(x) = |x|α, 0 < α < 1. (5)It is used to remove estimation outliers.
Q12. What is the main advantage of using flash?
In previous methods, Krishnan et al. [11] used gradients of a dark-flashed image, capturing ultraviolet (UV) and NIR light to guide noise removal in the color image.
Q13. What is the simplest way to solve a linear system?
The final linear system in the matrix form is((CTx (Px) 2At+1,tx Cx + C T y (Py) 2At+1,ty Cy) + λB t+1,t ) I= (CTx PxA t+1,t x + C T y PyA t+1,t y )s + λBt+1,tI0. (23)The linear system is also solved using PCG and the solution is denoted as I(t+1).