scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Image compression with Stochastic Winner-Take-All Auto-Encoder

TL;DR: This paper addresses the problem of image compression using sparse representations with a variant of autoencoder called Stochastic Winner-Take-All Auto-Encoder (SWTA AE), which performs variable rate image compression for images of any size after a single training, which is fundamental for compression.
Abstract: This paper addresses the problem of image compression using sparse representations. We propose a variant of autoencoder called Stochastic Winner-Take-All Auto-Encoder (SWTA AE). “Winner-Take-All” means that image patches compete with one another when computing their sparse representation and “Stochastic” indicates that a stochastic hyperparameter rules this competition during training. Unlike auto-encoders, SWTA AE performs variable rate image compression for images of any size after a single training, which is fundamental for compression. For comparison, we also propose a variant of Orthogonal Matching Pursuit (OMP) called Winner-Take-All Orthogonal Matching Pursuit (WTA OMP). In terms of rate-distortion trade-off, SWTA AE outperforms auto-encoders but it is worse than WTA OMP. Besides, SWTA AE can compete with JPEG in terms of rate-distortion.

Summary (2 min read)

Introduction

  • Image compression, sparse representations, auto-encoders, Orthogonal Matching Pursuit.
  • Auto-encoders are powerful tools for reducing the dimensionality of data.
  • But all image patches have the same rate and therefore different distortions due to the texture complexity variety in image patches.
  • This work has been supported by the French Defense Procurement Agency (DGA).
  • Therefore, during training, the WTA parameter that controls the rate is stochastically driven.

1.1. Notation

  • Vectors are denoted by bold lower case letters and matrices by upper case ones.
  • The authors now present their Stochastic Winner-Take-All AutoEncoder (SWTA AE) whose architecture is shown in Figure 1.
  • The authors justify below two of the most critical choices for the SWTA AE architecture.

2.1. Strided convolution

  • A compression algorithm must process images of various sizes.
  • This imposes to train one architecture per image size.
  • Each layer i ∈ J1, 4K consists in convolving the layer input with the bank of filters W(i), adding the biases b(i) and applying a mapping g(i), producing the layer output.
  • For the borders of the layer input, zero-padding of width p(i) is used.
  • Indeed, if the encoder contains a maxpooling layer, the locations of maximum activations selected during pooling operations must be recorded and transmitted to the corresponding unpooling layer in the decoder [12, 13].

2.2. Semi-sparse bottleneck

  • The authors propose to apply a global sparse constraint that povides control over the coding cost of Z. gα only applies to the output of the convolution in the second layer involving the first 64 filters in W(2), producing the first 64 sparse feature maps in Z. Figure 1 displays these sparse feature maps in orange.
  • Varying α leads to various coding costs of Z. Note that [14] uses WTA, but their WTA rule is different and gα does not apply to specific dimensions of its input tensor as this constraint is not relevant for image compression.
  • The authors have noticed that, during the training in Section 4.2, SWTA AE learns to store in the last feature map a subsampled version of its input image.

2.3. Bitstream generation

  • The coefficients of the non-sparse feature map in Z are uniformly quantized over 8-bits and coded with a Huffman code.
  • The position along z is coded with a fixed-length code and, for each pair (x, y), the number of non-zero coefficients along z is coded with a Huffman code.
  • The difference is that SWTA AE computes the sparse representation of an image by alternating convolutions and mappings whereas OMP runs an iterative decomposition of the image patches over a dictionary.
  • For the sake of comparison, the authors build a variant of OMP called Winner-Take-All Orthogonal Matching Pursuit (WTA OMP).
  • The support of the sparse representation of each patch has therefore been changed.

4.1. Training data extraction

  • The RGB color space is transformed into YCbCr and the authors only keep the luminance channel.
  • For SWTA AE, the luminance images are resized to 321×321. σ ∈ R∗+ is the mean of the standard deviation over all luminance images.
  • The authors remove the DC component from each patch.

4.2. SWTA AE training

  • If α is fixed during training, all the filters and the biases of SWTA AE are learned for one rate.
  • This justifies the prefix “Stochastic” in SWTA AE.
  • The training objective is to minimize the mean squared error between these cropped images and their reconstruction plus l2-norm weights decay.
  • The authors implementation is based on Caffe [18].

4.3. Dictionary learning for WTA OMP

  • Η∑ i=j ‖Zj‖0 ≤ γ × n× η (4) (4) is solved by Algorithm 2 which alternates between sparse coding steps that involve WTA OMP and dictionary updates that use stochastic gradient descent.
  • Dictionary learning for WTA OMP, also known as Algorithm 2.
  • For SWTA AE, the same values for m and n are used for training D via Algorithm 2. 1K-SVD code: http://www.cs.technion.ac.il/ elad/software/ 5. IMAGE COMPRESSION EXPERIMENT.
  • After training in Section 4, the authors compare the rate-distortion curves of OMP, WTA OMP, SWTA AE, JPEG and JPEG2000 on test luminance images.

5.1. Image CODEC for SWTA AE

  • Each input test luminance image is pre-processed similarly to the training in Section 4.1.
  • The mean learned image M is interpolated to match the size of the input image.
  • Then, the input image is subtracted by this interpolated mean image and divided by the learned σ.

5.2. Image CODEC for OMP and WTA OMP

  • A luminance image is split into 8×8 non-overlapping patches.
  • The DC component is removed from each patch.
  • The DC components are uniformly quantized over 8-bits and coded with a fixed-length code.
  • OMP (or WTA OMP) finds the coefficients of the sparse decompositions of the image patches over D ′ (or D).
  • The non-zero coefficients are uniformly quantized over 8-bits and coded with a Huffman code while their position is coded with a fixed-length code.

5.3. Comparison of rate-distortion curves

  • In the literature, there is no reference rate-distortion curve for auto-encoders.
  • Furthermore, the authors compare SWTA AE with its 2JPEG and JPEG2000 code: http://www.imagemagick.org/script/index.php non-sparse Auto-Encoder counterpart (AE).
  • Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, “ImageNet classification with deep convolutional neural networks,” in NIPS, 2012. [9].

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

HAL Id: hal-01493137
https://hal.archives-ouvertes.fr/hal-01493137
Submitted on 21 Mar 2017
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Image Compression with Stochastic Winner-Take-All
Auto-Encoder
Thierry Dumas, Aline Roumy, Christine Guillemot
To cite this version:
Thierry Dumas, Aline Roumy, Christine Guillemot. Image Compression with Stochastic Winner-Take-
All Auto-Encoder. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP 2017), Mar 2017, New Orleans, United States. �hal-01493137�

IMAGE COMPRESSION WITH STOCHASTIC WINNER-TAKE-ALL AUTO-ENCODER
Thierry Dumas, Aline Roumy, Christine Guillemot
INRIA Rennes Bretagne-Atlantique
thierry.dumas@inria.fr, aline.roumy@inria.fr, christine.guillemot@inria.fr
ABSTRACT
This paper addresses the problem of image compression us-
ing sparse representations. We propose a variant of auto-
encoder called Stochastic Winner-Take-All Auto-Encoder
(SWTA AE). “Winner-Take-All” means that image patches
compete with one another when computing their sparse rep-
resentation and “Stochastic” indicates that a stochastic hy-
perparameter rules this competition during training. Unlike
auto-encoders, SWTA AE performs variable rate image com-
pression for images of any size after a single training, which
is fundamental for compression. For comparison, we also
propose a variant of Orthogonal Matching Pursuit (OMP)
called Winner-Take-All Orthogonal Matching Pursuit (WTA
OMP). In terms of rate-distortion trade-off, SWTA AE out-
performs auto-encoders but it is worse than WTA OMP. Be-
sides, SWTA AE can compete with JPEG in terms of rate-
distortion.
Index Terms Image compression, sparse representa-
tions, auto-encoders, Orthogonal Matching Pursuit.
1. INTRODUCTION
Auto-encoders are powerful tools for reducing the dimension-
ality of data. Deep fully-connected auto-encoders [1] are tra-
ditionally used for this task. However, two issues have so far
prevented them from becoming efficient image compression
algorithms: they can only be trained for one image size and
one compression rate [2, 3].
[4] attempts to solve both issues. The authors train an
auto-encoder on image patches so that images of various sizes
can be compressed. Their auto-encoder is a recurrent [5]
residual auto-encoder that performs variable rate image com-
pression after a single training. But all image patches have the
same rate and therefore different distortions due to the texture
complexity variety in image patches. In addition, recurrence,
which is equivalent to scalability in image compression, is not
optimal in terms of rate-distortion trade-off [6, 7].
Instead, we propose to perform learning on whole images
under a global rate-distortion constraint. This is done through
This work has been supported by the French Defense Procurement
Agency (DGA).
Winner-Take-All (WTA), which can be viewed as a competi-
tion between image patches when computing their represen-
tation. Furthermore, auto-encoders architecture must adapt
to different rates. Therefore, during training, the WTA pa-
rameter that controls the rate is stochastically driven. These
contributions give rise to Stochastic Winner-Take-All Auto-
Encoder (SWTA AE).
1.1. Notation
Vectors are denoted by bold lower case letters and matrices
by upper case ones. X
j
denotes the j
th
column of a matrix
X. kXk
F
is the Frobenius norm of X. kXk
0
counts the
number of non-zero elements in X. The support of a vector x
is supp (x) = {i | x
i
6= 0}.
2. STOCHASTIC WINNER-TAKE-ALL
AUTO-ENCODER (SWTA AE)
We now present our Stochastic Winner-Take-All Auto-
Encoder (SWTA AE) whose architecture is shown in Figure 1.
SWTA AE is a type of auto-encoder. An auto-encoder is a
neural network that takes an input and provides a reconstruc-
tion of this input. We justify below two of the most critical
choices for the SWTA AE architecture.
2.1. Strided convolution
A compression algorithm must process images of various
sizes. However, the most efficient neural networks [8, 9] re-
quire that all images have the same size. Indeed, they include
both convolutional layers and fully-connected layers, and the
number of parameters of the latters directly depends on the
image size. This imposes to train one architecture per im-
age size. That is why our proposed SWTA AE only contains
convolutional layers. Its encoder has two convolutional lay-
ers and its decoder has two deconvolutional layers [10]. Each
layer i J1, 4K consists in convolving the layer input with
the bank of filters W
(i)
, adding the biases b
(i)
and applying
a mapping g
(i)
, producing the layer output. For the borders of
the layer input, zero-padding of width p
(i)
is used.
Max-pooling is a core component of neural networks [11]
that downsamples its input representation by appling a max

Fig. 1: SWTA AE architecture.
filter to non-overlapping sub-regions. But max-pooling in-
creases the rate. Indeed, if the encoder contains a max-
pooling layer, the locations of maximum activations selected
during pooling operations must be recorded and transmitted
to the corresponding unpooling layer in the decoder [12, 13].
Instead, for i J1, 2K, we downsample using a fixed stride
s
(i)
> 1 for convolution, which does not need any signaling.
2.2. Semi-sparse bottleneck
The bottleneck is the stack of feature maps denoted Z
R
h×w×65
in Figure 1. Z is the representation of the input
image that is processed in Section 2.3 to give the bitstream.
We propose to apply a global sparse constraint that po-
vides control over the coding cost of Z. This is called
Winner-Take-All (WTA). Let us define WTA via a mapping
g
α
: R
h×w×64
R
h×w×64
, where α ]0, 1[ is the WTA
parameter. g
α
keeps the α × h × w × 64 most representa-
tive coefficients in its input tensor, i.e. those whose absolute
values are the largest, and sets the rest to 0. g
α
only applies
to the output of the convolution in the second layer involving
the first 64 filters in W
(2)
, producing the first 64 sparse fea-
ture maps in Z. Figure 1 displays these sparse feature maps
in orange. Varying α leads to various coding costs of Z. Note
that [14] uses WTA, but our WTA rule is different and g
α
does not apply to specific dimensions of its input tensor as
this constraint is not relevant for image compression.
A patch of the input image might be represented by a por-
tion of the first 64 sparse feature maps in Z that only con-
tains zeros. We want to ensure that each image patch has a
minimum code in Z to guarantee a sufficient quality of recon-
struction per patch. That is why the last feature map in Z is
not sparse. Figure 1 displays it in red. We have noticed that,
during the training in Section 4.2, SWTA AE learns to store in
the last feature map a subsampled version of its input image.
2.3. Bitstream generation
The coefficients of the non-sparse feature map in Z are uni-
formly quantized over 8-bits and coded with a Huffman code.
The non-zero coefficients of the 64 sparse feature maps in Z
are uniformly quantized over 8-bits and coded with a Huff-
man code while their position is coded as explained here-
after. Figure 1 defines a coordinate system (x, y, z) for Z.
The non-zero coefficients in Z are scanned along (x, y, z)
where z changes the fastest. The position along z is coded
with a fixed-length code and, for each pair (x, y), the num-
ber of non-zero coefficients along z is coded with a Huffman
code. This unequivocally characterizes the position of each
non-zero coefficient in Z. We have observed that this pro-
cessing is effective in encoding the position of the non-zero
coefficients.
3. WINNER-TAKE-ALL ORTHOGONAL
MATCHING PURSUIT (WTA OMP)
SWTA AE is similar to Orthogonal Matching Pursuit (OMP)
[15], a common algorithm for image compression using
sparse representations [16]. The difference is that SWTA AE
computes the sparse representation of an image by alternat-
ing convolutions and mappings whereas OMP runs an iter-
ative decomposition of the image patches over a dictionary.
More precisely, let x R
m
be an image patch. Given x and
a dictionary D R
m×n
, OMP finds a vector of coefficients
y R
n
with k < m non-zero coefficients so that Dy equals
to x approximatively.
For the sake of comparison, we build a variant of OMP
called Winner-Take-All Orthogonal Matching Pursuit (WTA
OMP). More precisely, let X R
m×p
be a matrix whose
columns are formed by p image patches of dimension m and
Y R
n×p
be a matrix whose columns are formed by p vec-

tors of coefficients of dimension n. WTA OMP first decom-
poses each image patch over D, see (1). Then, it keeps the
γ × n × p coefficients with largest absolute value for the n-
length sparse representation of the p patches and sets the rest
to 0, see (2). The support of the sparse representation of each
patch has therefore been changed. Hence the need for a final
least-square minimization, see (3).
Algorithm 1 : WTA OMP
Inputs: X R
m×p
, D R
m×n
, k < m and γ ]0, 1[.
For each j J1, pK, Y
j
= OMP (X
j
, D, k) (1)
I = f
γ
(Y) (2)
For each j J1, pK, Z
j
= min
zR
n
kX
j
Dzk
2
2
st.
supp (z) = supp (I
j
)
(3)
Output: Z R
n×p
.
4. TRAINING
Before moving on to the image compression experiment in
Section 5, SWTA AE needs training. Similarly, a dictionary
D R
m×n
must be learned for WTA OMP.
4.1. Training data extraction
We extract 1.0 × 10
5
RGB images from the ILSVRC2012
ImageNet dataset [17]. The RGB color space is transformed
into YCbCr and we only keep the luminance channel.
For SWTA AE, the luminance images are resized to
321 ×321. M R
321×321
denotes the mean of all luminance
images. σ R
+
is the mean of the standard deviation over
all luminance images. Each luminance image is subtracted by
M and divided by σ. These images are concatenated into a
training set R
321×321×
(
1.0×10
5
)
.
For WTA OMP, η = 1.2 × 10
6
image patches of size
m ×
m are randomly sampled from the luminance im-
ages. We remove the DC component from each patch. These
patches are concatenated into a training set Γ R
m×η
.
4.2. SWTA AE training
As explained in Section 2.2, α tunes the coding cost of Z. If α
is fixed during training, all the filters and the biases of SWTA
AE are learned for one rate. That is why we turn α into a
stochastic hyperparameter during training. This justifies the
prefix “Stochastic” in SWTA AE. Since there is no reason to
favor some rates during training, we sample α according to
the uniform distribution U[µ , µ + ], where µ > 0 and
µ + < 1. We select µ = 1.8 × 10
1
and = 1.7 × 10
1
to make the support of α large. At each training epoch, α is
drawn for each training image of .
As shown in Section 2.1, SWTA AE can process images
of various sizes. During training, we feed SWTA AE with
random crops of size 49×49 of the training images of . This
accelerates training considerably. The training objective is to
minimize the mean squared error between these cropped im-
ages and their reconstruction plus l
2
-norm weights decay. We
use stochastic gradient descent. The gradient descent learn-
ing rate is fixed to 2.0 × 10
5
, the momentum is 0.9 and the
size of mini-batches is 5. The weights decay coefficient is
5.0 × 10
4
. Our implementation is based on Caffe [18]. It
adds to Caffe the tools introduced in Sections 2.2 and 2.3.
4.3. Dictionary learning for WTA OMP
Given Γ, k < m and γ ]0, 1[, the dictionary learning prob-
lem is formulated as (4).
min
D,Z
1
,...,Z
η
1
η
η
X
j=1
kΓ
j
DZ
j
k
2
2
st. j J1, ηK, kZ
j
k
0
k
st.
η
X
i=j
kZ
j
k
0
γ × n × η
(4)
(4) is solved by Algorithm 2 which alternates between sparse
coding steps that involve WTA OMP and dictionary updates
that use stochastic gradient descent. Given Γ and p N
+
,
let φ be a function that randomly partitions Γ into η
p
=
η / p mini-batches
X
(1)
, ..., X
(η
p
)
, where, for i J1, η
p
K,
X
(i)
R
m×p
. Mini-batches make learning very fast [19].
Algorithm 2 : dictionary learning for WTA OMP.
Inputs: Γ R
m×η
, k < m, γ ]0, 1[, p N
+
and ε R
+
.
D R
m×n
is randomly initialized.
j [|1, n|], D
j
D
j
/ kD
j
k
2
For several epochs do:
h
X
(1)
, ..., X
(η
p
)
i
= φ (Γ, p)
i [|1, η
p
|], Z
(i)
= WTA OMP
X
(i)
, D, k, γ
D D ε
X
(i)
DZ
(i)
2
F
D
j [|1, n|], D
j
D
j
/ kD
j
k
2
Output: D R
m×n
.
For OMP, given Γ, a dictionary D
0
R
m×n
is learned
using K-SVD [16]
1
, and the parameters m and n are opti-
mized with an exhaustive search. This leads to m = 64 and
n = 1024. For SWTA AE, the same values for m and n
are used for training D via Algorithm 2. Moreover, k = 15,
γ = 4.5 × 10
3
, p = 10 and ε = 2.0 × 10
2
.
1
K-SVD code: http://www.cs.technion.ac.il/ elad/software/

Fig. 2: Evolution of PNSR with the rate.
(a) LENA luminance 512 × 512. (b) BARBARA luminance 480 × 384.
5. IMAGE COMPRESSION EXPERIMENT
After training in Section 4, we compare the rate-distortion
curves of OMP, WTA OMP, SWTA AE, JPEG and JPEG2000
on test luminance images.
5.1. Image CODEC for SWTA AE
Each input test luminance image is pre-processed similarly to
the training in Section 4.1. The mean learned image M is
interpolated to match the size of the input image. Then, the
input image is subtracted by this interpolated mean image and
divided by the learned σ. The encoder of SWTA AE computes
Z. The bitstream is obtained by processing Z as detailed in
Section 2.3.
5.2. Image CODEC for OMP and WTA OMP
A luminance image is split into 8×8 non-overlapping patches.
The DC component is removed from each patch. The DC
components are uniformly quantized over 8-bits and coded
with a fixed-length code. OMP (or WTA OMP) finds the co-
efficients of the sparse decompositions of the image patches
over D
0
(or D). The non-zero coefficients are uniformly
quantized over 8-bits and coded with a Huffman code while
their position is coded with a fixed-length code.
Then, for WTA OMP only, the number of non-zero coef-
ficients of the sparse decomposition of each patch over D is
coded with a Huffman code.
5.3. Comparison of rate-distortion curves
In the literature, there is no reference rate-distortion curve
for auto-encoders. We compare SWTA AE with JPEG and
JPEG2000
2
even though the image CODEC of SWTA AE is
less optimized. Furthermore, we compare SWTA AE with its
2
JPEG and JPEG2000 code: http://www.imagemagick.org/script/index.php
non-sparse Auto-Encoder counterpart (AE). AE has the same
architecture as SWTA AE but its Z only contains non-sparse
feature maps. Note that, to draw a new point in the AE rate-
distortion curve, AE must be first re-trained with a different
number of feature maps in Z.
Figure 2 shows the rate-distortion curves of OMP, WTA
OMP, AE, SWTA AE, JPEG and JPEG2000 for two of the
most common images: LENA and BARBARA. In terms
of rate-distortion trade-off, SWTA AE outperforms AE and
WTA OMP is better than OMP. This highlights the value
of WTA for image compression. When we compare SWTA
AE with WTA OMP, we see that iterative decomposition
is more efficient for image compression using sparse repre-
sentations. Moreover, SWTA AE can compete with JPEG.
We also ran this image compression experiment on several
crops of LENA and BARBARA and observed that the rel-
ative position of the six rate-distortion curves was compa-
rable to the relative positioning in figure 2. The size of
the test image does not affect the performance of SWTA
AE. More simulation results and a complexity analysis for
OMP, WTA OMP and SWTA AE can be found on the
web page https://www.irisa.fr/temics/demos/
NeuralNets/AutoEncoders/swtaAE.htm.
6. CONCLUSIONS AND FUTURE WORK
We have shown that, SWTA AE is more adaptated to image
compression than auto-encoders as it performs variable rate
image compression for any size of image after a single train-
ing and provides better rate-distortion trade-offs.
So far, our work has focused on the layer of auto-encoders
which is dedicated to coding. Yet, many avenues of research
are still to be explored to improve auto-encoders for image
compression. For instance, [20] proves that removing a max-
pooling layer and increasing the stride of the previous con-
volution, as we do, harms neural networks. This has to be
addressed.

Citations
More filters
Journal ArticleDOI
TL;DR: Autoencoders (AEs) as mentioned in this paper have emerged as an alternative to manifold learning for conducting nonlinear feature fusion, and they can be used to generate reduced feature sets through the fusion of the original ones.

209 citations

Journal ArticleDOI
Jiahao Li1, Bin Li2, Xu Jizheng2, Ruiqin Xiong1, Wen Gao1 
TL;DR: This paper proposes using a fully connected network to learn an end-to-end mapping from neighboring reconstructed pixels to the current block to generate better prediction using traditional single line-based methods.
Abstract: This paper proposes a deep learning method for intra prediction. Different from traditional methods utilizing some fixed rules, we propose using a fully connected network to learn an end-to-end mapping from neighboring reconstructed pixels to the current block. In the proposed method, the network is fed by multiple reference lines. Compared with traditional single line-based methods, more contextual information of the current block is utilized. For this reason, the proposed network has the potential to generate better prediction. In addition, the proposed network has good generalization ability on different bitrate settings. The model trained from a specified bitrate setting also works well on other bitrate settings. Experimental results demonstrate the effectiveness of the proposed method. When compared with high efficiency video coding reference software HM-16.9, our network can achieve an average of 3.4% bitrate saving. In particular, the average result of 4K sequences is 4.5% bitrate saving, where the maximum one is 7.4%.

143 citations


Cites methods from "Image compression with Stochastic W..."

  • ...In [23], a variant of autoencoder is proposed by using sparse...

    [...]

Journal ArticleDOI
TL;DR: The proposed PixelMotionCNN (PMCNN) which includes motion extension and hybrid prediction networks can model spatiotemporal coherence to effectively perform predictive coding inside the learning network and provides a possible new direction to further improve compression efficiency and functionalities of future video coding.
Abstract: One key challenge to learning-based video compression is that motion predictive coding, a very effective tool for video compression, can hardly be trained into a neural network. In this paper, we propose the concept of PixelMotionCNN (PMCNN) which includes motion extension and hybrid prediction networks. PMCNN can model spatiotemporal coherence to effectively perform predictive coding inside the learning network. On the basis of PMCNN, we further explore a learning-based framework for video compression with additional components of iterative analysis/synthesis and binarization. The experimental results demonstrate the effectiveness of the proposed scheme. Although entropy coding and complex configurations are not employed in this paper, we still demonstrate superior performance compared with MPEG-2 and achieve comparable results with H.264 codec. The proposed learning-based scheme provides a possible new direction to further improve compression efficiency and functionalities of future video coding.

116 citations


Additional excerpts

  • ...on and optimized a convolutional auto-encoder with an incremental training strategy [6]. Dumas et al. incorporates a stochastic hyperparameter to control a competition mechanism between image patches [20]. As a fast-growing architecture in the field of neural network, GANs have also been proved to be effective on image compression in practice [21], [22]. In the works of [4], [7], [23], [24], a discrete...

    [...]

Journal ArticleDOI
TL;DR: Deep Learning Video Coding (DLVC) as discussed by the authors is a deep learning-based video coding framework, which is based on convolutional neural network (CNN) and block adaptive resolution coding (BARC).
Abstract: The past decade has witnessed great success of deep learning technology in many disciplines, especially in computer vision and image processing. However, deep learning-based video coding remains in its infancy. This paper reviews the representative works about using deep learning for image/video coding, which has been an actively developing research area since the year of 2015. We divide the related works into two categories: new coding schemes that are built primarily upon deep networks (deep schemes), and deep network-based coding tools (deep tools) that shall be used within traditional coding schemes or together with traditional coding tools. For deep schemes, pixel probability modeling and auto-encoder are the two approaches, that can be viewed as predictive coding scheme and transform coding scheme, respectively. For deep tools, there have been several proposed techniques using deep learning to perform intra-picture prediction, inter-picture prediction, cross-channel prediction, probability distribution prediction, transform, post- or in-loop filtering, down- and up-sampling, as well as encoding optimizations. In the hope of advocating the research of deep learning-based video coding, we present a case study of our developed prototype video codec, namely Deep Learning Video Coding (DLVC). DLVC features two deep tools that are both based on convolutional neural network (CNN), namely CNN-based in-loop filter (CNN-ILF) and CNN-based block adaptive resolution coding (CNN-BARC). Both tools help improve the compression efficiency by a significant margin. With the two deep tools as well as other non-deep coding tools, DLVC is able to achieve on average 39.6\% and 33.0\% bits saving than HEVC, under random-access and low-delay configurations, respectively. The source code of DLVC has been released for future researches.

47 citations

Journal ArticleDOI
TL;DR: A broad view of what an AE is, how they are used for feature fusion, a taxonomy gathering a broad range of models, and how they relate to other classical techniques are provided.
Abstract: Many of the existing machine learning algorithms, both supervised and unsupervised, depend on the quality of the input characteristics to generate a good model. The amount of these variables is also important, since performance tends to decline as the input dimensionality increases, hence the interest in using feature fusion techniques, able to produce feature sets that are more compact and higher level. A plethora of procedures to fuse original variables for producing new ones has been developed in the past decades. The most basic ones use linear combinations of the original variables, such as PCA (Principal Component Analysis) and LDA (Linear Discriminant Analysis), while others find manifold embeddings of lower dimensionality based on non-linear combinations, such as Isomap or LLE (Linear Locally Embedding) techniques. More recently, autoencoders (AEs) have emerged as an alternative to manifold learning for conducting nonlinear feature fusion. Dozens of AE models have been proposed lately, each with its own specific traits. Although many of them can be used to generate reduced feature sets through the fusion of the original ones, there also AEs designed with other applications in mind. The goal of this paper is to provide the reader with a broad view of what an AE is, how they are used for feature fusion, a taxonomy gathering a broad range of models, and how they relate to other classical techniques. In addition, a set of didactic guidelines on how to choose the proper AE for a given task is supplied, together with a discussion of the software tools available. Finally, two case studies illustrate the usage of AEs with datasets of handwritten digits and breast cancer.

46 citations


Cites background from "Image compression with Stochastic W..."

  • ...A less specific goal can be found in [122]....

    [...]

References
More filters
Posted Content
TL;DR: In this paper, a deconvolution network is proposed to identify pixel-wise class labels and predict segmentation masks in an input image, and construct the final semantic segmentation map by combining the results from all proposals.
Abstract: We propose a novel semantic segmentation algorithm by learning a deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer net. The deconvolution network is composed of deconvolution and unpooling layers, which identify pixel-wise class labels and predict segmentation masks. We apply the trained network to each proposal in an input image, and construct the final semantic segmentation map by combining the results from all proposals in a simple manner. The proposed algorithm mitigates the limitations of the existing methods based on fully convolutional networks by integrating deep deconvolution network and proposal-wise prediction; our segmentation method typically identifies detailed structures and handles objects in multiple scales naturally. Our network demonstrates outstanding performance in PASCAL VOC 2012 dataset, and we achieve the best accuracy (72.5%) among the methods trained with no external data through ensemble with the fully convolutional network.

2,601 citations

Book ChapterDOI
15 Sep 2010
TL;DR: The aim is to gain insight into different functions by directly comparing them on a fixed architecture for several common object recognition tasks, and empirical results show that a maximum pooling operation significantly outperforms subsampling operations.
Abstract: A common practice to gain invariant features in object recognition models is to aggregate multiple low-level features over a small neighborhood. However, the differences between those models makes a comparison of the properties of different aggregation functions hard. Our aim is to gain insight into different functions by directly comparing them on a fixed architecture for several common object recognition tasks. Empirical results show that a maximum pooling operation significantly outperforms subsampling operations. Despite their shift-invariant properties, overlapping pooling windows are no significant improvement over nonoverlapping pooling windows. By applying this knowledge, we achieve state-of-the-art error rates of 4.57% on the NORB normalized-uniform dataset and 5.6% on the NORB jittered-cluttered dataset.

1,409 citations


"Image compression with Stochastic W..." refers background in this paper

  • ...Max-pooling is a core component of neural networks [11] that downsamples its input representation by appling a max...

    [...]

Journal ArticleDOI
TL;DR: This paper demonstrates that the restricted isometry property (RIP) can be used for a very straightforward analysis of OMP and concludes that the RIP of order K+1 (with isometry constant δ <; [ 1/( 3√K)]) is sufficient for OMP to exactly recover any K-sparse signal.
Abstract: Orthogonal matching pursuit (OMP) is the canonical greedy algorithm for sparse approximation. In this paper we demonstrate that the restricted isometry property (RIP) can be used for a very straightforward analysis of OMP. Our main conclusion is that the RIP of order K+1 (with isometry constant δ <; [ 1/( 3√K)]) is sufficient for OMP to exactly recover any K-sparse signal. The analysis relies on simple and intuitive observations about OMP and matrices which satisfy the RIP. For restricted classes of K-sparse signals (those that are highly compressible), a relaxed bound on the isometry constant is also established. A deeper understanding of OMP may benefit the analysis of greedy algorithms in general. To demonstrate this, we also briefly revisit the analysis of the regularized OMP (ROMP) algorithm.

560 citations


"Image compression with Stochastic W..." refers methods in this paper

  • ...SWTA AE is similar to Orthogonal Matching Pursuit (OMP) [15], a common algorithm for image compression using sparse representations [16]....

    [...]

Posted Content
TL;DR: A general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional LSTM recurrent networks are proposed, which provide better visual quality than (headerless) JPEG, JPEG2000 and WebP, with a storage size reduced by 10% or more.
Abstract: A large fraction of Internet traffic is now driven by requests from mobile devices with relatively small screens and often stringent bandwidth requirements. Due to these factors, it has become the norm for modern graphics-heavy websites to transmit low-resolution, low-bytecount image previews (thumbnails) as part of the initial page load process to improve apparent page responsiveness. Increasing thumbnail compression beyond the capabilities of existing codecs is therefore a current research focus, as any byte savings will significantly enhance the experience of mobile device users. Toward this end, we propose a general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional LSTM recurrent networks. Our models address the main issues that have prevented autoencoder neural networks from competing with existing image compression algorithms: (1) our networks only need to be trained once (not per-image), regardless of input image dimensions and the desired compression rate; (2) our networks are progressive, meaning that the more bits are sent, the more accurate the image reconstruction; and (3) the proposed architecture is at least as efficient as a standard purpose-trained autoencoder for a given number of bits. On a large-scale benchmark of 32$\times$32 thumbnails, our LSTM-based approaches provide better visual quality than (headerless) JPEG, JPEG2000 and WebP, with a storage size that is reduced by 10% or more.

432 citations

Journal ArticleDOI
TL;DR: The design for these extensions represents the latest state of the art for video coding and its applications, including work on range extensions for color format and bit depth enhancement, embedded-bitstream scalability, and 3D video.
Abstract: This paper describes extensions to the High Efficiency Video Coding (HEVC) standard that are active areas of current development in the relevant international standardization committees. While the first version of HEVC is sufficient to cover a wide range of applications, needs for enhancing the standard in several ways have been identified, including work on range extensions for color format and bit depth enhancement, embedded-bitstream scalability, and 3D video. The standardization of extensions in each of these areas will be completed in 2014, and further work is also planned. The design for these extensions represents the latest state of the art for video coding and its applications.

420 citations


"Image compression with Stochastic W..." refers background in this paper

  • ...In addition, recurrence, which is equivalent to scalability in image compression, is not optimal in terms of rate-distortion trade-off [6, 7]....

    [...]

Frequently Asked Questions (15)
Q1. What are the contributions in "Image compression with stochastic winner-take-all auto-encoder" ?

This paper addresses the problem of image compression using sparse representations. The authors propose a variant of autoencoder called Stochastic Winner-Take-All Auto-Encoder ( SWTA AE ). For comparison, the authors also propose a variant of Orthogonal Matching Pursuit ( OMP ) called Winner-Take-All Orthogonal Matching Pursuit ( WTA OMP ). 

Given Γ and p ∈ N∗+, let φ be a function that randomly partitions Γ into ηp = η / p mini-batches { X(1), ...,X(ηp) } , where, for i ∈ J1, ηpK, X(i) ∈ Rm×p. 

The position along z is coded with a fixed-length code and, for each pair (x, y), the number of non-zero coefficients along z is coded with a Huffman code. 

The training objective is to minimize the mean squared error between these cropped images and their reconstruction plus l2-norm weights decay. 

Max-pooling is a core component of neural networks [11] that downsamples its input representation by appling a maxfilter to non-overlapping sub-regions. 

Each layer i ∈ J1, 4K consists in convolving the layer input with the bank of filters W(i), adding the biases b(i) and applying a mapping g(i), producing the layer output. 

η∑ i=j ‖Zj‖0 ≤ γ × n× η(4)(4) is solved by Algorithm 2 which alternates between sparse coding steps that involve WTA OMP and dictionary updates that use stochastic gradient descent. 

The non-zero coefficients are uniformly quantized over 8-bits and coded with a Huffman code while their position is coded with a fixed-length code. 

for WTA OMP only, the number of non-zero coefficients of the sparse decomposition of each patch over D is coded with a Huffman code. 

For each j ∈ J1, pK,Yj = OMP (Xj ,D, k) (1) The author= fγ (Y) (2)For each j ∈ J1, pK,Zj = min z∈Rn ‖Xj −Dz‖22 st.supp (z) = supp (Ij) (3)Output: Z ∈ Rn×p.4. 

For instance, [20] proves that removing a maxpooling layer and increasing the stride of the previous convolution, as the authors do, harms neural networks. 

CONCLUSIONS AND FUTURE WORKThe authors have shown that, SWTA AE is more adaptated to image compression than auto-encoders as it performs variable rate image compression for any size of image after a single training and provides better rate-distortion trade-offs. 

Gary J. Sullivan, Jim M. Boyce, Ying Chen, Jens-Rainer Ohm, C. Andrew Segal, and Anthony Vetro, “Standardized extensions of high efficiency video coding (HEVC),” IEEE Journal of Selected Topics in Signal Processing, vol. 7 (6), pp. 1001–1016, December 2013.[8] 

it keeps the γ × n × p coefficients with largest absolute value for the nlength sparse representation of the p patches and sets the rest to 0, see (2). 

Given Γ, k < m and γ ∈ ]0, 1[, the dictionary learning problem is formulated as (4).min D,Z1,...,Zη1η η∑ j=1 ‖Γj −DZj‖22st. ∀j ∈ J1, ηK, ‖Zj‖0 ≤ kst.