Title
Image reconstruction for thin observation module
by bound optics by using the iterative
backprojection method
Author(s)
Nitta, Kouichi; Shogenji, Rui; Miyatake,
Shigehiro; Tanida, Jun
Citation Applied Optics. 45(13) P.2893-P.2900
Issue Date 2006-05-01
Text Version publisher
URL http://hdl.handle.net/11094/2900
DOI 10.1364/ao.45.002893
rights
Note
Osaka University Knowledge Archive : OUKAOsaka University Knowledge Archive : OUKA
https://ir.library.osaka-u.ac.jp/
Osaka University
Image reconstruction for thin observation module
by bound optics by using the iterative
backprojection method
Kouichi Nitta, Rui Shogenji, Shigehiro Miyatake, and Jun Tanida
A method for reconstructing high-spatial-resolution images in an imaging system known as thin
observation module by bound optics (TOMBO) is reported. We investigate a novel procedure combin-
ing a pixel-rearrangement method and iterative backprojection (IBP). Pixel rearrangement has been
used until now in TOMBO, and IBP is a digital superresolution technique. We verify the effectiveness
of the combined procedure with simulated and experimental results. © 2006 Optical Society of
America
OCIS codes: 110.2970, 110.4190, 100.3010, 100.3020, 230.0250, 230.3120.
1. Introduction
Recently, owing to the remarkable progress made in
the fields of solid-state imaging devices, data storage,
and image processing, the adoption of digital image-
capturing systems has grown dramatically, and they
are now in widespread use around the world. This, in
turn, has led to significant expansion of the field of
digital imaging. Such systems are considered to be a
key technology in the future. To accelerate the ubiqui-
tous adoption of these technologies, for example, in
portable information terminals, more compact and
lightweight imaging systems are required. However,
current compact imaging systems such as microlenses
have insufficient imaging quality, and therefore im-
proved compact optical components, in addition to
novel system architectures, will be required to meet
these demands.
A system architecture known as thin observation
module by bound optics (TOMBO) was proposed to
construct compact, low-profile image-capturing de-
vices.
1
This system consists of an optical module
that uses compound-eye imaging technology and
digital signal processing for image reconstruction.
Compound-eye imaging imitates the visual organs
of some arthropods.
2–4
One of the attractive fea-
tures of this system is that the optical module can
be made quite thin. Another is that digital signal
processing can be used to improve the quality of the
image captured by the optical module.
Various issues were identified to improve the per-
formance of the TOMBO system. In terms of the op-
tical hardware, high imaging performance of each
microlens, uniformity of the lens array, and accurate
alignment between the lens array and the imaging
device are required. On the other hand, the digital
signal processing should be capable of retrieving de-
tailed information about the target object. In partic-
ular, restoration of the high-frequency components in
the target image is one of the most important issues.
The pixel rearrangement method presented in Ref. 5
is not, by itself, an effective solution for this problem,
even though the method is very useful for compensa-
tion of misalignment in the optical module.
In this paper, therefore, we present a novel method
for image reconstruction in the TOMBO system. In
this method the pixel-rearrangement method and it-
erative backprojection (IBP)
6
are used complementa-
rily. IBP is a digital superresolution technique that
can generate an image with high spatial resolution
from a set of images with low spatial resolution. We
demonstrate with simulated and experimental re-
sults that the proposed method improves the spatial
resolution of the reconstructed image.
K. Nitta (nitta@kobe-u.ac.jp) is with the Faculty of Engineering,
Kobe University, 1-1 Rokkodai-cho, Nada, Kobe 657-8501,
Japan. R. Shogenji and J. Tanida are with the Graduate School of
Information Science and Technology, Osaka University, 2-1
Yamadaoka, Suita, Osaka 565-0871, Japan. S. Miyatake is with
Konica Minolta Technology Center, Inc., 1-2 Sakura-machi,
Takatsuki, Osaka 569-5803, Japan.
Received 12 August 2005; revised 9 January 2006; accepted 10
January 2006; posted 13 January 2006 (Doc. ID 64076).
0003-6935/06/132893-08$15.00/0
© 2006 Optical Society of America
1 May 2006 兾 Vol. 45, No. 13 兾 APPLIED OPTICS 2893
2. Thin Observation Module by Bound Optics
A. Architecture
The TOMBO system consists of an optical module
that uses compound-eye imaging and electronic sig-
nal processing for image reconstruction. In the opti-
cal module a set of images with low spatial resolution
are captured simultaneously to form a single image
called a compound image. The signal processing then
converts the compound image to a single image with
high spatial resolution.
Figure 1 shows a schematic diagram of the optical
module in the TOMBO system. Components of the
optical module are a microlens array, a signal sep-
arator, and a photodetector array. As shown in Fig.
1, an elemental optical system used to acquire one
low-spatial-resolution image is called a unit, and
the image obtained is called a unit image. The op-
tical module is a collection of these units, and the
set of unit images forms the compound image. Since
this kind of compound-eye imaging system acquires
visual information with a set of small lenses, a com-
pact, lightweight imaging system can be con-
structed.
In the visual organs of certain arthropods, an op-
tical signal obtained by a single lens is detected by a
single detector. On the other hand, TOMBO has flex-
ibility in terms of the number of photodetectors used
per lens system. The number of the photodetectors N,
the number of units , and the number of pixels in a
single unit are the system parameters. The relation
among these three parameters is represented by
⫽N兾. (1)
The number of pixels in a single unit corresponds
directly to the aperture size of the imaging lens and
the working distance. Therefore the larger the value
of , the thinner the system. In particular, in the case
of ⫽N, the system is equivalent in architecture to
the arthropod vision system mentioned above.
The functionality of the TOMBO system can be
expanded by modifying the architecture of the optical
module and the digital processing. In Ref. 7 two
methods for color imaging were reported. One is color
separation by pixels, which is used in commercial
image sensors, and the other is color separation by
units, which gives superior multispectral imaging
performance.
8
Also, fingerprint capturing has been
reported as an application of TOMBO.
9
B. Pixel-Rearrangement Method
As the signal processing used for image reconstruc-
tion in TOMBO, a pixel-sampling method and a
pseudomatrix method were studied in Ref. 1. How-
ever, the quality of the reconstructed images obtained
by these methods was not suitable for practical use.
The main reasons for the poor image quality were
considered to be (1) misalignment between the mi-
crolens array and the photodetector array and (2)
undersampling due to compound-eye imaging.
In the optical module for the TOMBO system,
accurate alignment is required to obtain a recon-
structed image with high quality, but slight errors
are unavoidable. Also, each unit image has fewer
pixels than those of conventional single-eye imag-
ing systems. Thus compound-eye imaging cannot
acquire high-frequency components of the target
object. Post-signal processing should be capable of
compensating for these alignment errors and of
achieving operations equivalent to oversampling in
order to retrieve the original information.
The pixel-rearrangement method was proposed to
solve problem (1) above. Figure 2 shows a schematic
diagram of this method. All pixels in the unit images
are remapped onto a virtual plane, and the geometric
relations between the unit images and the virtual
plane are described by registration parameters. In
the processing, the relations are described with an
affine transformation model, whose parameters are
estimated statistically.
5
The first step of the pixel-rearrangement method is
to compensate for the shading effects caused by the
limited aperture of the microlenses and spatial vari-
ations in the photodetectors. Thus the captured com-
pound image is corrected in intensity. The corrected
image is then divided into a set of unit images. The
registration parameters are determined based on the
Fig. 1. Schematic diagram of optical module in the TOMBO
system.
Fig. 2. Schematic diagram of the pixel-rearrangement method.
2894 APPLIED OPTICS 兾 Vol. 45, No. 13 兾 1 May 2006
scheme described above, and the pixels in all unit
images are mapped onto the virtual image plane on
the basis of the registration parameters. If blank pix-
els remain in the virtual image, interpolation opera-
tions are performed to determine these pixel values.
The pixel-rearrangement method is suitable for
high-speed processing because its principle is very
simple. Although we confirmed that this method
could compensate for misalignment to improve the
quality of the output image,
5
the method cannot re-
solve the problem caused by undersampling because
the pixels in the unit images, which have low spatial
resolution, are mapped onto the corresponding pixels
on the virtual plane without any oversampling pro-
cess.
3. Limitations of the Pixel-Rearrangement Method
In this section we describe a frequency analysis of the
image reconstruction with the pixel-rearrangement
method. For this analysis, a simple one-dimensional
model that satisfies the sampling theorem is as-
sumed. Figures 3(a) and 3(b) show the models for
conventional single-eye imaging and for the pixel-
rearrangement method used in TOMBO, respec-
tively. In the models the fill factor of the pixels is
assumed to be 100%. In this figure f(x), p, and a
represent the target object, the pixel pitch, and the
observation area of a pixel, respectively. In the con-
ventional imaging model the values of a and p are the
same. On the other hand, in the TOMBO model the
value of a is larger than that of p.
In both models, pixel values of an image g(x) are
represented by
g
共
x
兲
⫽ f
共
x
兲
comb
共
x兾p
兲
ⴱ rect
共
x兾a
兲
, (2)
where ⴱ indicates the convolution operator and
comb共x兾p兲 and rect共x兾a兲 are terms caused by the ef-
fects of pixel sampling and the pixel window, respec-
tively.
The Fourier transform of g(x) is represented by
G
共
x
兲
⫽
ⱍ
ap
ⱍ
F
共
x
兲
ⴱ comb
共
p
x
兲
sinc
共
a
x
兲
, (3)
where
x
is the spatial frequency of the target image.
Figure 4 shows profiles of G共
x
兲 for the two cases. In
conventional single-eye imaging, the Nyquist fre-
quency
N
is given by
N
⫽ 1兾2p. (4)
From Fig. 4(a), G共
x
兲 becomes positive for |
x
| ⬍
N
.
This means that the effect of the pixel window in g(x)
Fig. 3. One-dimensional model [g(x)] for (a) single-eye imaging and (b) the pixel-rearrangement method in TOMBO.
Fig. 4. G(
x
) of both analysis models: (a) single-eye imaging and (b) the pixel-rearrangement method in TOMBO.
1 May 2006 兾 Vol. 45, No. 13 兾 APPLIED OPTICS 2895
can be corrected with simple spatial-frequency filter-
ing. In the case of compound-eye imaging, at
a ⬍ 2p [as shown in Fig. 4(b)], G共
x
兲 becomes negative
for 1兾a ⬍ |
x
| ⬍
N
. This means that the high-
frequency components of f(x) are inverted in phase
and that the components at the zero-crossing points
are lost.
Consequently, it is clear that the TOMBO method
based on the pixel-rearrangement method cannot
achieve the same performance as conventional
single-eye imaging. Therefore we should investigate
digital superresolution methods that will be suitable
for compound-eye imaging.
4. Iterative Backprojection Method
A. Processing Procedure
We investigated IBP
6,10
as a useful digital superreso-
lution algorithm in the TOMBO system. In the IBP
method a reconstructed image is generated with an
iterative process of error estimation and an inference
of a deblurred image. Figures 5 and 6 show a sche-
matic diagram and a flow chart of the IBP method,
respectively. In these figures f shows the target ob-
ject, which is unknown; f
共n兲
is the inferred image gen-
erated after n iterations; and g
u
x
,u
y
and g
u
x
,u
y
共n兲
indicate
the observed low-resolution image and the simulated
one obtained by computation. In the TOMBO system,
g
u
x
,u
y
represents a unit image at position 共u
x
, u
y
兲.
As the first operation of the iterative process, g
u
x
,u
y
共n兲
are generated by using
g
u
x
,u
y
共
n
兲
⫽
关
T
u
x
,u
y
共
f
共
n
兲
兲
ⴱ h
兴
↓ s, (5)
where T
u
x
,u
y
describes a geometric transformation be-
tween the target object and the unit at 共u
x
, u
y
兲 and h
and 2 s represent a blur kernel determined by the
lens system and a downsampling operator, respec-
tively.
The next process is estimation of the error function
e
共n兲
, which is given by
e
共
n
兲
⫽
冋
共
1兾N
2
兲
兺
u
y
⫽1
N
兺
u
x
⫽1
N
共
g
u
x
,u
y
⫺ g
u
x
,u
y
共
n
兲
兲
2
册
1兾2
. (6)
If e
共n兲
is less than a threshold value, f
共n兲
is output as the
final result and the process is terminated. If e
共n兲
is
larger than the threshold value, however, f
共n兲
is up-
dated by using
Fig. 5. Schematic diagram of the IBP method.
2896 APPLIED OPTICS 兾 Vol. 45, No. 13 兾 1 May 2006
