IEEE Transactions on Consumer Electronics, Vol. 49, No. 4, NOVEMBER 2003
Contributed Paper
Manuscript received April 21, 2003 0098 3063/00 $10.00 © 2003 IEEE
1256
Motion Adaptive Interpolation with Horizontal Motion Detection
for Deinterlacing
Shyh-Feng Lin, Yu-Ling Chang, and Liang-Gee Chen, Fellow, IEEE
Abstract
—
A motion adaptive deinterlacing algorithm is
presented in this paper. It consists of the ELA-Median
directional interpolation, same-parity 4-field horizontal
motion detection, morphological operation for noise reduction
and adaptive threshold adjusting. The edges can be sharper
when the ELA-Median directional interpolation is adopted.
The same-parity 4-field horizontal motion detection detects
faster motion and makes more accurate determinations about
where objects are going to move. The morphological
operation for noise reduction and adaptive threshold
adjusting preserve the actual texture of the original objects.
The proposed method achieves cost-efficient hardware
implementation with low complexity, low memory usage, and
high-speed processing capability. In addition, it consumes less
time in producing high-quality images and allows the
audience to enjoy a high-quality TV sequence on their
progressive devices. The experimental results show that the
proposed algorithm is more cost-effective than previous
systems.
Index Terms — Deinterlacing, ELA-Median,
Morphological, Progressive.
I. I
NTRODUCTION
Deinterlacing is an important process which converts
ordinary TV sequences into progressive sequences for the sake
of displaying on progressive devices (e.g. Computers, Plasma
Display, and Projection TV). Some defects such as edge
flicker, jagged effects, blur, and line-crawling will cause
uncomfortable visual artifacts, if deinterlacing is not done
perfectly.
Two low-complexity deinterlacing methods are BOB[1] and
Weave[1], which are commonly adopted in the software
approach. BOB is an intra-interpolation method, which uses a
single field to reconstruct one progressive frame. However, the
vertical resolution is halved and the image is blurred. Weave is
a simple deinterlacing method combining directly two
interlaced fields into one progressive frame. However, the
line-crawling effect will occur in the motion area.
1
Some motion adaptive techniques [9]-[13] [15]- [24] have
been presented to improve image quality. Sugiyama and
Nakamura [9] proposed a method of motion-compensated
adaptive interpolation. They used motion-estimation and
1
This work was supported in part by the AVermadia Inc., R.O.C. under
Grant No. 90-S-B59.
Shyh-Feng Lin, Yu-Ling Chang, and Liang-Gee Chen are with the
National Taiwan University, Department of Electrical Engineering and
Graduate Institute of Electronics Engineering, Taipei 106, Taiwan, R.O.C. (e-
mail: shyh@video.ee.ntu.edu.tw, doraemon@video.ee.ntu.edu.tw,
lgchen@video.ee.ntu.edu.tw).
adaptive interpolation to reconstruct the missing field with the
information obtained from the backward and the forward
fields. Hilman [10] and Haan [20] proposed a motion-
compensated frame-rate conversion algorithm with
interpolation to reduce the 3:2 pull-down artifacts. Ville [11]
proposed a motion adaptive technique on a fuzzy motion
detector. Schutten and de Haan [21] proposed an object-based
true-motion estimation algorithm to eliminate the 2-3 pull-
down sequences. Sun [23] proposed a shortest path technique
of the motion information to re-align the fields of a video
image. Patti [24] proposed using global motion estimation to
reconstruct the video for solving the panning or zooming
situation. All of these methods can provide better deinterlacing
quality. However, these methods demand a lot of
computational power and are very expensive for consumer
electronic products.
In this paper, a motion adaptive deinterlacing method with
Edge Line Average (ELA), 4-field horizontal motion detection
in same parity, noise reduction, and threshold adjusting is
proposed. The overview of intra-field and inter-field
deinterlacing will be discussed in Section 2. The fundamentals
of the proposed method will be described in Section 3. Section
4 shows the experimental results and comparison with
previous methods. Section 5 gives the conclusion and remarks.
II. D
EINTERLACING
O
VERVIEW
Fundamentally, deinterlacing could be characterized into
four categories: intra-field deinterlacing, inter-field
deinterlacing, motion adaptive deinterlacing, and motion-
compensated deinterlacing.
A. Intra-Field Deinterlacing
The most cost-efficient method is intra-field deinterlacing
by the same field. It is widely used in software
implementations, because it needs less computational power
and only one delay line buffer. The most common method of
the intra-field deinterlacing is line doubling, which is used in
small LCD panels. However, the jagged effect will occur in the
oblique line and flicker will be seen in the texture of the detail.
Some methods of interpolation use upper and lower line pixels
to reduce such flickers and alias.
ELA is the most popular algorithm in this category. It is a
kind of directional edge interpolation [5]. Three pixels in the
previous scan line and the next scan line are referenced to
determine the obvious edge in the image as shown in Fig. 1.
This method can eliminate the blurring effect of the bilinear
interpolation and gives sharp/straight edges.
Although intra-field interpolation is very cost-efficient and
needs one line buffer only, the resolution of the picture is half
S.-F. Lin et al.: Motion Adaptive Interpolation with Horizontal Motion Detection for Deinterlacing 1257
of the original. In addition, some defects may occur when an
object only exists in the same-parity field.
nn+1n+2n-2 n-1
k-1
k+1
original
scan line
original
scan line
New line
Fig. 1. Edge directional interpolation
B. Inter-Field Deinterlacing
Inter-field interpolation means merging two fields into one
frame, so it needs one-field buffer. The video quality is better
than that of intra-field interpolation in static area, but the line-
crawling effect, as shown in Fig. 2, will occur in the motion
area.
Fig. 2. Line-crawling effect occurs when merged directly.
C. Motion Adaptive Deinterlacing
The motion adaptive deinterlacing combines the advantages
of both intra-field deinterlacing and inter-field deinterlacing. It
detects the motion areas first, and then adopts intra-field
deinterlacing in motion areas and inter-field deinterlacing in
static areas. The block diagram is shown in Fig. 3. High-
resolution and flicker-free picture can be realized in both static
area and motion area. The image quality is the same as that of
intra-field interpolation. Motion adaptive deinterlacing relies
on accurate motion detection. Any erroneous detection will
cause artifacts as spots in the video.
Motion
Detection
Field Storage
Intra-Field
Interpolation
interleaving switch
per line
Progressive
output
Interlace
input
previous field
select
current field
current intra-field
interpolation
Fig. 3. Block diagram of motion adaptive deinterlacing.
Current Field
Reference Field
Frame
Separated
into 2 field
Fig. 4. Block matching by motion estimation method.
D. Motion-Compensated Method
The motion-compensated method utilizes motion estimation
to find the most similar blocks in the neighboring fields and
calculates its motion vectors, as shown in Fig. 4. Then a new
field is reconstructed from neighboring field. Block matching
needs a large buffer size to locate the current macroblock and
reference macroblock. Additionally, it also needs a lot of
computational power to calculate the Sum of Absolute
Difference (SAD) value. Although motion-compensated
methods have greater potential to produce better result, they
are also more complex since they require a motion estimation
engine.
In the next section, the fundamental of proposed
deinterlacing is described. This new deinterlacing scheme
using pixel-based motion detection is very cost-effective. It
contains the noise reduction engine and can get reliable motion
information to achieve high image quality.
III. P
ROPOSED
M
OTION
-A
DAPTIVE
D
EINTERLACING
The block diagram of the proposed method is shown in Fig.
5. Three-field buffers are used to store the reference data. The
enhanced ELA module does the directional edge interpolation
according to the current-field information. The same-parity 4-
field horizontal motion detection calculates the difference
between the forward-forward field and the current field, and
the difference between the forward field and the backward
field. After the morphological operation for noise reduction,
the field difference will be sent to the threshold-adjusting
module. According to the pixel value of the current field, the
IEEE Transactions on Consumer Electronics, Vol. 49, No. 4, NOVEMBER 2003 1258
threshold-adjusting module will provide an adaptive threshold
value to produce the motion information. The decision block
receives the motion information, and then selects the forward
field in the static area and the output of ELA in the motion
area. Finally, the current field and the interpolation field are
merged into the progressive frame.
Field
storage
Field
storage
Field
storage
4-field
horizontal
motion
detection
&
3:2 pull-
down
detection
Enhanced
ELA
Progressive frame out
Field in
Forward Field
Current Field
Backward Field
Forward-Forward Field
Forward Field
Current Field
Current Field by ELA
Decision Block
Interpolation Field
Morphological
operation
Threshold
adjusting
Current Field
Motion Information
Interleave
by line
Current Field
Fig. 5. Block diagram of proposed deinterlacing method.
A. Enhanced Directional Interpolation - with Median ELA
The directional interpolation algorithm using ELA [5] can
generate straight and high contrast image. However, some artifacts
will occur due to erroneous detection in a non-dominant
directional edge region. The modified ELA [6] can eliminate the
artifacts by interpolating the missing pixel according to the
classification of the edge region. First, five differences are
calculated along the five directions as shown in Fig. 6 and ( 1 ).
n n+1 n+2n-2 n-1
k-1
k+1
original
scan line
original
scan line
New line
X
Y
minmum difference
ab c
d
e
Fig. 6. ELA with median processing.
The minimum difference is chosen to be the highest
correctional edge, and the opposite direction of the highest
correction is used to detect a non-dominant directional edge
region. As shown in Fig. 7, if the highest correction is a, then d
and e are used for detecting the dominant edge region. The
differences of directional difference are calculated as D
d1
, D
d2
as shown in ( 2 ).
()()
[]
()()
[]
()()
[]
()()
[]
()()
[]
1212
1111
11
1111
1212
+−−−+=
+−−−+=
+−−=
++−−−=
++−−−=
,knX,knXe
,knX,knXd
n ,kXn ,kXc
,knX,knXb
,knX,knXa
( 1 )
cdeab
Fig. 7. The direction of the highest correlation is d or e.
eaD
daD
d
d
−=
−=
2
1
( 2 )
Td is the threshold value determined empirically. If Dd1>
Td and Dd2 > Td, the region is dominant directional edge, a
directional interpolation will perform along the edge. At last,
the median of three pixels ( 3 ) - the produced pixel, the
corresponding pixels in the previous line and in the next line-
is calculated to prevent the occurrence of bursting pixels. For a
non-dominant edge region, a bilinear interpolation was
performed along the vertical region. The flowchart of
enhanced ELA is shown in Fig. 8.
()()
[]
ELAknXknXMedianY ,1,,1, +−=
( 3 )
Two scan lines
Difference of edge
directional calculation
(a,b,c,d,e)
Dominant edge
Vertical bilinear
interpolation
Directional
interpolation
Median filter
No
Yes
Fig. 8. Flowchart of enhanced ELA.
S.-F. Lin et al.: Motion Adaptive Interpolation with Horizontal Motion Detection for Deinterlacing 1259
B. 4-Field Motion Detection in Same Parity
1) Same-Parity Field Detection
For interlaced video, the vertical position of the even field
and the odd field is slightly different. Same-parity field [25]
detection detects the motion area by the even-field-to-even-
field or odd-field-to-odd-field difference. The benefits can be
seen in the static area. Using successive-field difference for
motion detection will lead to some problems. For example, the
static letter “T” in the progressive frame can be separated into
two fields as shown in Fig. 9. The motion detection by
successive-field difference will determine the horizontal edge
of the letter T to be different. The wrong decision will cause
the static letter T in motion and produce the flicker line in the
horizontal edge. With same-parity field detection, the even-to-
even-field difference and odd-to-odd-field difference will be
the same. The correct decision can be made and the static
letter T can be reconstructed perfectly.
C. 4-Field Horizontal Motion Detection
The 2-field difference [14] will lead to a wrong decision, if
the object is moving too fast. As shown in Fig. 10, a person is
moving very fast between the forward field and the backward
field, crossing the current-field position. Both the forward field
and backward field are background at the position of the
person in the current field. So the 2-field difference between
the forward field and backward field is very small and the
person is detected to be static. The line-crawling effect will
occur to this erroneous detection.
y
t
Even field
t-2
Odd field
t-1
Even field
t
Odd field
t+1
different different
sam e
sam e
successive-field difference
same-parity-field difference
x
different
Progressive frame
Fig. 9. 4-field motion detection in same parity.
Same-parity 4-field motion detection involves two more
fields than 2-field motion detection, as shown in Fig. 11; the
current field and the forward-forward field. It also has same-
parity field detection ability, which can solve the detection
error caused by the different positions in the even field and
odd field. The extra field difference between the current field
and forward-forward field can detect more motion information
than same-parity 2-field motion detection. So the line-crawling
effect can be eliminated by this method.
Current Field Backward Field
Field
Difference
Forward Field
Progressive Out
Fig. 10. The 2-field motion detection.
Current FieldForward FieldForward-Forward Field Backward Field
Field
Difference
Field
Difference
OR
P rogressiv e
Out
Fig. 11. The 4-field motion detection.
a b c d e
Forward
Backward
&
FForward
Current
Forward (c) Forward (b) Forward (d) Forward (a) Forward (e)
Backward
Direction c
is minimum
Direction b
is minimum
Direction d
is minimum
Direction a
is minimum
Direction e
is minimum
Forward
Fig. 12. 4-field horizontal motion detection.
Usually, adaptive motion deinterlacing will adopt intra-field
interpolation for pan sequences, resulting in lower resolution.
IEEE Transactions on Consumer Electronics, Vol. 49, No. 4, NOVEMBER 2003 1260
Five-directional temporal interpolation is added into the same-
parity 4-field motion detection to achieve resolution as high as
inter-field interpolation as shown in Fig. 12. Block matching is
done between the forward field and the backward field by 1×3
blocking size. If the minimum difference of block matching is
smaller than the threshold, and the pixel difference between
the forward-forward field and the current field is also smaller
than the threshold, the temporal prediction will be adopted.
The proposed same-parity 4-field horizontal motion detection
can eliminate the detection error due to vertical position of
interlaced sequence and fast motion, and can achieve high-
resolution result to process the pan sequences.
D. 3:2 Pull-Down Detection
1) 3:2 Pull-Down algorithm
The 3:2 pull-down method means transforming the sequence
rate from 24 frames/second into 60 fields/second. An example
is shown in Fig. 13, frame 1 is separated into three fields; even
field/ odd field/ even field. The first even field and the third
even field of frame 1 are the same. Frame 2 is separated into
two fields as described above. So two frames will be translated
into five fields, which means the original 24 frames/second
will be translated into 60 fields/second.
E O E O E O E O E O
Progressive Frame
Field
Frame
1
Frame
2
Frame
3
Frame
4
Fig. 13. 3:2 Pull-down translation.
2) 3:2 Pull-Downed Recovery
E1 O1 E2 O2 E3 O3 E4 O4 E5 O5
Frame
1
Progressive Frame
Field
Frame
2
Frame
3
Frame
4
Fig. 14. 3:2 pull-down recovery.
Same-parity field detection, which means detection from
even-to-even field or from odd-to-odd field, can also detect the
3:2 pull-downed sequences. If the same-parity field detection
detects that two fields resemble each other periodically, then
the 3:2 pull-down sequence is detected. The redundancy of an
even field or an odd field will be removed. As shown in the
Fig. 14, E1 and O1 are merged into Frame 1, and E2 is
discarded. O2 and E3 are merged into Frame 2. E4 and O4 are
merged into Frame 3, and O3 is discarded. E5 and O5 are
merged into Frame 4. So five fields will be translated into two
frames, which means 60 fields/second will be translated into
24 frames/second. Then the progressive frame will be
reconstructed perfectly. The circuits of same-parity field
detection can be used both in the 3:2 pull-down detection and
4-field motion detection, saving the hardware cost and meeting
low-power consideration.
E. Morphological Operation for Noise Reduction
Video signal after coding and transmission will involve
some noise, which will cause erroneous motion detection. The
morphological operation scheme will eliminate the noise of
field difference to preserve the reliable motion information.
Morphological operation, which includes the erosion process
and dilation process, can reduce the high-frequency noise of
field difference. The purpose of erosion is to reduce the high-
frequency noise, and its equation is shown in ( 4 ). The 3×3
window is shown in Fig. 15, if one of the nine pixels is black,
then the center pixel is black. After the erosion process, we go
through a dilation process. The purpose of dilation is
extending the motion area, because the erosion process will
erode the edge of the motion area. If one of the nine pixels in
the 3×3 window is white, then the center pixel is white.
F(j+1,k-1)F(j,k-1)F(j-1,k-1)
F(j+1,k)F(j,k)F(j-1,k)
F(j+1,k+1)F(j,k+1)F(j-1,k+1)
Fig. 15. The 3×
××
×3 window of morphological operation.
1)]k1, F(j1),k, F(j1),k1,- F(j
),k 1, F(j),k , F(j),k 1,- F(j
1),-k1, F(j1),-k, F(j1),-k1,-MIN[F(jk)G(j,
++++
+
+=
( 4 )
G(j,k) is the mask result of each pixel after erosion. F(j,k) is
the original same-parity field difference. The 3×3 dilation of
gray scale difference is defined as ( 5 ).
1)]k1,G(j 1),kG(j, 1),k1,-G(j
),k 1,G(j ),k G(j, ),k 1,-G(j
1),-k1,G(j 1),-kG(j, 1),-k1,-MAX[G(jk)D(j,
++++
+
+=
( 5 )
Here, D (j, k) is the final mask result after dilation and
erosion. G (j, k) is the mask result after erosion. We can easily
discover in Fig. 16 (a) that the original-field difference (shown
in Fig. 16 (b)) contains a lot of noise and we may not know
where the real motion areas are. Black represents the static
area, and white represents the motion area. The difference
after “erosion” is shown in Fig. 16 (c), where there is a great
improvement in noise reduction. But we lose the difference
information on the edge of the objects. So the dilation process
is utilized after applying the erosion process and obtains a
