Saliency Detection via Dense and Sparse Reconstruction

TL;DR: A visual saliency detection algorithm from the perspective of reconstruction errors that applies the Bayes formula to integrate saliency measures based on dense and sparse reconstruction errors and refined by an object-biased Gaussian model is proposed.

Abstract: In this paper, we propose a visual saliency detection algorithm from the perspective of reconstruction errors. The image boundaries are first extracted via super pixels as likely cues for background templates, from which dense and sparse appearance models are constructed. For each image region, we first compute dense and sparse reconstruction errors. Second, the reconstruction errors are propagated based on the contexts obtained from K-means clustering. Third, pixel-level saliency is computed by an integration of multi-scale reconstruction errors and refined by an object-biased Gaussian model. We apply the Bayes formula to integrate saliency measures based on dense and sparse reconstruction errors. Experimental results show that the proposed algorithm performs favorably against seventeen state-of-the-art methods in terms of precision and recall. In addition, the proposed algorithm is demonstrated to be more effective in highlighting salient objects uniformly and robust to background noise.

Summary

1. Introduction

• The main motivation of the present paper stems from the current debate on the weak productivity performance of European countries as opposed to that recorded, especially during the second half of the 1990s, in the US.
• Among the largest European countries, Italy has experienced, over the last decade, a slowdown in total factor productivity (TFP) that has been particularly evident in manufacturing.
• These findings support the need of a Lisbon-type of policy by suggesting that, if European countries want to attain the same productivity enhancement recorded among US industries, they must invest much more resources in R&D.
• According to their explanation, the weak relationship between R&D and TFP arising for Italy is not due to structural (and, as such, not easily modifiable) features but is the outcome of a decade of slowdown in R&D investment (the Nineties) after a decade of remarkable expansion (the Eighties).
• Section 6 performs a supplementary analysis of the Italian case.

2. The productivity slowdown in the EU and Italian manufacturing: explanations and remedies

• The Lisbon strategy, launched by the European Council in 2000, is mainly based on the idea that the slow rate of productivity growth recorded during the last decade by EU countries (as opposed to the productivity revival experienced in the US) depends primarily upon their lower endowments of knowledge capital.
• A large body of empirical evidence, either among countries (European Commission, 2005) or across industries and firms (Griliches, 1995; Wieser, 2005), supports the above arguments: no matter the units of observation, the returns from R&D investments are substantial and provide to the performing units permanent rather than transitory advantages.
• By extending the analysis up to 2003, Daveri and Jona-Lasinio (2006) confirm the above findings: the labour productivity slowdown experienced in Italy during the last decade is mainly due to TFP and the growth of the latter declined particularly (albeit not only) in manufacturing.
• The authors employ data on twelve manufacturing industries over a period of twenty-three years (i.e. from 1980 to 2002) to perform separate econometric analyses for five of the major OECD states: namely, US, Germany, France, Spain and Italy.
• On average, the four European countries account for two thirds of the EU-15 levels of R&D expenditure and GDP, so that, although a special focus shall be devoted to Italy, the findings and policy implications of their analysis will refer to a large portion of Europe.

3. Analytical framework and data description

• It must be stressed that, as far as the labour and capital inputs specifically devoted to R&D are already included in L and K (i.e. the inputs on the right-hand side of equation [1] are not corrected for double counting), θ must be interpreted as the excess elasticity of value added with respect to R&D capital.
• For Italian industries, instead, the authors used the R&D data coming from ISTAT (the Italian Statistical Office), mainly because3 they are available since the late sixties, and then allowed us to extend the analysis of the Italian case to a longer period of time (see section 6).
• Net and gross fixed capital stocks are built by OECD through a permanent inventory method accounting for the age and efficiency profile of different capital assets.
• To be added is that the R&D stock in 1973 is computed by applying the same procedure described above although, in this case, the authors assume a depreciation rate of 15 per cent, constant across industries.

4. Comparing R&D performances across industries and countries

• To highlight the differences among industries and countries, the authors first employ the share of R&D expenditures on value added (at current prices).
• On the contrary, in all the industries considered Italy attained a much lower intensity of R&D outlays: only in Transport Equipment and only when opposed to Germany the Italian gap is not pronounced.
• It must be pointed out that the above aggregate trends are the result of quite different performances at industry level.
• The US, as opposed to Germany and France, have maintained the lead in manufacturing R&D by concentrating, over the last years, their research efforts in the ICT industry.
• Spain, albeit remaining a backward country in terms of R&D intensity, has behaved as a typical technological follower might do, i.e. in almost all the industries it has continuously increased its R&D investment.

5. Estimation procedure and results across countries

• In this section the authors perform an estimation of the long-run relationship between TFP and R&D capital stock.
• Cameron (2005) uses a similar framework to estimate the impact of R&D and human capital on the productivity gap between eleven Japanese and US industries observed during 1963-1989.
• From the bottom part of Table 4 it emerges that when the presence of structural breaks is not taken into consideration, the statistics τN rejects the null hypothesis of no cointegration for the US, France and Italy, but not for Germany7 and Spain.
• Moving to the SUR estimate of the ECM described in equation [4], it should be added that the authors make a small-sample adjustment to calculate the covariance matrix of residuals.
• Using data for twenty-two manufacturing industries in fourteen OECD countries over the period 1972-94, Frantzen (2002) estimates an elasticity of TFP with respect to R&D equal to 0.34.

Figures

Figure 8. Evaluation of Bayesian integrated saliency. (a) Precision-recall curves of Dense-Sparse and Sparse-Dense Bayesian models. (c) F-measure curves of the proposed Bayesian integrated saliency SB and four other integrated saliency of MDEPG and MSEPG.

Figure 7. Evaluation of saliency via reconstruction error. (a) Based on the context-based propagation. (b) Based on the multi-scale reconstruction error integration. (c) Based on the object-biased Gaussian refinement. DE: dense reconstruction error; DEP: propagated DE; MDEP: multi-scale integrated DEP; MDEPG: Gaussian refined MDEP. SE: sparse reconstruction error; SEP: propagated SE; MSEP: multi-scale integrated SEP; MSEPG: Gaussian refined MSEP.

Figure 1. Main steps of the proposed saliency detection algorithm.

Figure 10. Performance of the proposed algorithm compared with other methods on the MSRA and SOD databases.

Figure 9. Performance of the proposed method compared with seventeen state-of-the-art methods on the ASD database.

Figure 2. Saliency maps based on dense and sparse reconstruction errors. Brighter pixels indicate higher saliency values. (a) Original images. (b) Saliency maps from dense reconstruction. (c) Saliency maps from sparse reconstruction. (d) Ground truth.

Figure 11. Comparisons of saliency maps. Top, middle and bottom two rows are images from the ASD, SOD and MSRA data sets. DSR: the proposed algorithm based on dense and sparse reconstruction. DSR cut: cut map using the generated saliency map. GT: ground truth.

Figure 4. Saliency maps with the multi-scale integration of propagated reconstruction errors. (a) and (b) are original images and ground truth. (c) and (d) are propagated dense reconstruction errors without and with integration. (e) and (f) are propagated sparse reconstruction errors without and with integration.

Figure 3. Saliency maps with the context-based error propagation. (a) and (b) are original images and ground truth. (c) and (d) are original and propagated dense reconstruction errors. (e) and (f) are original and propagated sparse reconstruction errors.

Figure 5. Comparison of center-biased (Gc) and object-biased (Go) Gaussian refinement. Ed and Es are the multi-scale integrated dense and sparse reconstruction error maps, respectively.

Figure 6. Bayesian integration of saliency maps. The two saliency measures via dense and sparse reconstruction are respectively denoted by S1 and S2.

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Saliency detection via dense and sparse reconstruction" ?

In this paper, the authors propose a visual saliency detection algorithm from the perspective of reconstruction errors. 

Q2. What are the eigenvectors from the normalized covariance matrix of B?

The eigenvectors from the normalized covariance matrix of B, UB = [u1,u2, ...,uD′ ], corresponding to the largest D′ eigenvalues, are computed to form the PCA bases of the background templates. 

Q3. How does the reconstruction error of a pixel be assigned?

The reconstruction error of a pixel is assigned by integrating the multiscale reconstruction errors, which helps generate more accurate and uniform saliency maps. 

Q4. How does the Bayesian integration method perform?

To combine the two saliency maps via dense and sparse reconstruction, the authors introduce a Bayesian integration method which performs better than the conventional integration strategy. 

Q5. How many superpixels are generated for multi-scale reconstruction errors?

For multi-scale reconstruction errors, the authors generate superpixels at eight different scales respectively with 50 to 400 superpixels. 

Q6. How do the authors evaluate the performance of Bayesian integrated saliency map SB?

The authors evaluate the performance of Bayesian integrated saliency map SB by comparing it with the integration strategies formulated in [5]:Sc = 1 Z ∑ i Q (Si) or Sc = 1Z ∏ i Q (Si), (15)where Z is the partition function. 

Q7. What is the weight of the reconstruction error of a segment?

The authors integrate multi-scale reconstruction errors and compute the pixellevel reconstruction error byE(z) =Ns∑ s=1 ωzn(s) ε̃n(s)Ns∑ s=1 ωzn(s) , ωzn(s) = 1 ‖fz − xn(s)‖2 , (7)where fz is a D-dimensional feature of pixel z and n(s) denotes the label of the segment containing pixel z at scale s. Similarly to [14], the authors utilize the similarity between pixel z and its corresponding segment n(s) as the weight to average the multi-scale reconstruction errors. 

Q8. what is the weight of the reconstruction error of a segment in cluster k?

The propagated reconstruction error of segment i belonging to cluster k (k = 1, 2, ...,K), is modified by considering its appearance-based context consisting of the other segments in cluster k as follows:ε̃i = τ Nc∑ j=1 wikj ε̃kj + (1− τ) εi, (5)wikj = exp(−‖xi−xkj‖22σx2 ) (1− δ (kj − i))Nc∑ j=1 exp(−‖xi−xkj‖ 2 2σx2 ), (6)where {k1, k2, ..., kNc} denote the Nc segment labels in cluster k and τ is a weight parameter.