0018-9456 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIM.2019.2963555, IEEE
Transactions on Instrumentation and Measurement
>Paper ID: IM-19-22905R Title: Automated Visual Defect Detection for Flat Steel Surface: A Survey<
1
Abstract—Automated computer-vision-based defect detection has
received much attention with the increasing surface quality
assurance demands for the industrial manufacturing of flat steels.
This paper attempts to present a comprehensive survey on surface
defect detection technologies by reviewing about 120 publications
over the last two decades for three typical flat steel products of
con-casting slabs, hot- and cold-rolled steel strips. According to
the nature of algorithms as well as image features, the existing
methodologies are categorized into four groups: Statistical,
spectral, model-based and machine learning. These literatures are
summarized in this review to enable easy referral to suitable
methods for diverse application scenarios in steel mills.
Realization recommendations and future research trends are also
addressed at an abstract level.
Index Terms—Automated visual inspection (AVI), automated
optical inspection (AOI), surface defect detection, flat steel,
survey.
I. INTRODUCTION
S A DOMINANT steel product, flat steel occupies more
than 65% of all the products in the iron and steel industry,
which is the vital fundamental material for the related planar
industries, including without limitation, architecture, aerospace,
machinery, automobile, and so on. Any quality problems
suffering on flat steel would lead to huge economic and
reputation losses to steel manufacturers. For thin and wide flat
steel, surface defects are the greatest threat to the product
quality. Even for occasional internal defects, morphological
changes will arise on the surface with large probability.
Automated visual inspection (AVI) instrument targeting on
surface quality emerges as a standard configuration for flat steel
mills to improve product quality and promote production
efficiency.
A general AVI instrument provides two main functions of
This work was supported in part by the National Natural Science Foundation
of China under Grant 51704089 and Grant 61973323, in part by the Anhui
Provincial Natural Science Foundation of China under Grant 1808085QF190.
Q. Luo and C. Yang are with the School of Automation, Central South
University, Changsha 410083, China. (Corresponding Author, Chunhua Yang,
Email: ychh@csu.edu.cn).
X. Fang is with the School of Electrical and Automation Engineering, Hefei
University of Technology, Hefei 230009, China.
L. Liu is with Center for Machine Vision and Signal Analysis, University of
Oulu, Oulu 90014, Finland. She is also with the College of System Engineering,
National University of Defense Technology, Changsha 410073, China.
Y. Sun is with the School of Engineering and Computer Science, University
of Hertfordshire, Hatfield ALl0 9AB, U.K.
Fig. 1. The contribution of defect detection in a typical AVI instrument.
defect detection and classification [1-4]. The former detection
process recognizes defective regions from normal background
without identifying what types of defects they are. This step is
the foundation of the “quality problem close loop”, earlier
defect detection allows less economic losses. The latter process
is dedicated to identify and label detected defects to support
finishing product grading. In this context, the flat steel covers
three categories of con-casting slabs, hot- and cold-rolled steel
strips, where slabs are rolled into hot strips and then to cold
strips. Taking hot strip as an example, Fig. 1 briefly gives the
flow chart of AVI processes. In general, defect detection is
required to be in strict real-time while defect classification can
be handled in quasi real-time. The total performance of AVI
system is mainly limited by the accuracy, time-efficiency and
robustness of the arithmetical methods in the defect detection
process which is the very focus of this paper.
However, on-site surface defect detection in real-world steel
mills is seriously challenging: 1) Unsatisfactory imaging
environments. Continuous casting and rolling production lines
involve multiple sufferings of high temperature, dense mist,
heavy cooling water drops [5], uneven illumination, stochastic
noises [1, 2], and aperiodic vibration [6]. The undesirable
image quality requires preeminent detection algorithms to resist
large intra-class variation and minor inter-class distance [1-4].
2) Eternally continuous image streams. The online dual-surface
quality measurement for average flat steel mills requires the
surface AVI instrument to continuously process about 2.56
Gbps of image flows [5] to identify defective regions, which
Automated Visual Defect Detection for Flat Steel
Surface: A Survey
Qiwu Luo, Member, IEEE, Xiaoxin Fang, Li Liu, Chunhua Yang, Member, IEEE, and Yichuang Sun,
Senior Member, IEEE
A
0018-9456 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIM.2019.2963555, IEEE
Transactions on Instrumentation and Measurement
>Paper ID: IM-19-22905R Title: Automated Visual Defect Detection for Flat Steel Surface: A Survey<
2
force the detection algorithms to achieve excellent balance
between accuracy, computational complexity and reliability.
Over the years, industry and academia devote themselves to
address the aforementioned challenges from hardware upgrade
to algorithmic optimization. The hardware architecture based
on either server expansion [7-9] or ASIC acceleration [5] has
been opened in some recent reports. Furthermore, it is not easy
to see dramatic hardware breakthroughs within relatively short
time due to the limitation of Moore’s law [10]. This review thus
focuses on the latest theoretical and algorithmic advances of
automated visual defect detection over the past two decades to
enable easy referral to suitable methods for diverse application
scenarios in steel mills. Especially the literatures over the last
five years accounted for nearly 50%.
The structure of this context is as follows. After the
introduction in Section I, some relevant prior survey papers are
briefly reviewed in Section II. Typical defect morphologies on
flat steel surfaces are illustrated vividly in Section III. The four
categories of defect detection approaches are presented in
Section IV in detail. This paper is ended in Section V with the
conclusion and comments on the realization recommendations
and future research trends.
II. PRIOR LITERATURE REVIEW
A number of AVI surveys (such as [11-13]) with a wide
coverage of inspection problems can be available successively.
Recently published surveys gradually pay more attention to
specific planar materials like fabric [8] and semiconductor [14].
Notably, a brief but rare AVI review covering defect detection
and classification techniques for steel products was reported [9],
where nearly all types of steel products (slab, billet, plate, hot
strip, cold strip, rod/bar) are involved at an overview level. It is
widely recognized that AVI techniques are more suitable to
inspect surface defects on sheet materials than on wire rod/bar
with minor diameter or even special-shaped structures [15]. To
further narrow the scope of [9], that is, concentrate on the vital
defect detection process on only flat steel products, this paper
attempts to present a first Transactions survey on this focused
topic, so as to support the AVI applications for the relevant
industrial manufacturing.
III. DEFECT MORPHOLOGIES ON FLAT STEEL SURFACE
Various defects on flat steel surface are generally caused by
mechanical or metallurgical imperfection during the industrial
manufacturing. To save paper space, we only take some surface
defect image samples for hot-rolled steel strips and con-casting
slabs by using the AVI instrument designed in [5] for
illustration. Fig. 2(a) lists four raw defective images
(4096×1024 pixel) acquired by the equipped line-scan camera.
And Fig. 2(b) presents eighteen typical defect samples with
256×256 pixel obtained from raw images after defect detection
process. These are roller marks, longitudinal scratches,
horizontal scratches, inclusions, scarring, holes, waves, pitting,
air bubbles, peeling, water droplets, convex bags, reticulations,
star cracks, foreign bodies, heavy leather, wrinkles and
longitudinal cracks, respectively. Finally, in Fig. 2(c), some
longitudinal crack image samples of con-casting slabs are
presented (512×512 pixel), and this defect type is with high
probability of occurrence on continuous casting line, which has
great threat to the quality of downstream products. Besides the
diversity and complexity of these defects, nearly all the
challenges mentioned in Sec. I can be encountered in these
image samples. For example, some pseudo defects of water
droplets and mill scales are pretty commonly distributing on the
surfaces of hot-rolled strips and casting slabs, which would
trigger false detection. Another example, the image intensity is
fairly inhomogeneous and varies actively.
IV. TAXONOMY OF DEFECT DETECTION METHODS
This section presents a review on the prior techniques and
models for defect detection of flat steel surfaces. In general,
researchers categorize previously proposed methods into
different groups based on the distinct features, while the
taxonomy also varies from person to person. Timm et al. [16]
broadly separated texture defect detection approaches into local
and global groups. According to different technique roadmaps,
defect detection methods are summarized as classification-,
local-abnormalities-, and template-matching-based methods in
[17]. Youkachen et al. [18] classified defect detection methods
into probabilistic, statistical, proximity-based, deviation-based
and network-based models. At the microscopic level, the flat
steel surface inspection problem is essentially a texture analysis
problem [8]. Normally, texture analysis problem can be solved
by statistical-, spectral- and model-based methods. Notably,
machine learning enjoys its popularity in computer vision in
recent years, especially in texture analysis. Thus, as shown in
Fig. 3, this paper classifies defect detection methods for flat
steel surfaces into four categories: conventional statistical,
spectral, model-based and emerging machine learning.
A. Statistical
Statistical approaches are frequently used to detect defects of
flat steel surface by evaluating the regular and periodic
distribution of pixel intensities. Eight representative statistical
methods are briefly introduced as follows.
1) Thresholding
Thresholding methods are usually used to separate the
defective regions on flat steel surfaces, which have been widely
applied in online AVI systems [19, 20]. The traditional
thresholding methods identify defects by comparing the value
of image pixels to a fix number and turn the test image into a
simple binary frame, which is sensitive to random noises and
non-uniform illuminations. Djukic et al. [21] first estimated the
probability distribution of pixel intensities from some
defect-free hot-rolled steel images, which was considered as a
basis for adaptively determining threshold. The dynamical
thresholding procedure can then discriminately separate true
defects from random noise. Further, Nand et al. [22] calculated
the local entropy of defective and defect-free images
respectively and extracted defective region of image by using
background subtraction method by comparing their entropy, it
is reported to perform better on detecting defective blocks of
0018-9456 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIM.2019.2963555, IEEE
Transactions on Instrumentation and Measurement
>Paper ID: IM-19-22905R Title: Automated Visual Defect Detection for Flat Steel Surface: A Survey<
3
(a)
(b)
(c)
Fig. 2. Typical defect image samples. For hot-rolled steel strips: (a) are typical defective raw images of steel surface (4096×1024 pixel) acquired by line-scan
camera and (b) are a series of typical defect samples with 256×256 pixel. For con-casting slabs: (c) are typical longitudinal cracks acquired by area-scan camera.
Markov random
field model
Weibull model
Active contour model
Other latest reported
model-based
Defect Detection
Statistical Spectral Model-based Machine Learning
Thresholding
Clustering
Edge-based
Fractal Dimension
Gray-level statistic
Co-occurrence matrix
Local binary pattern
Morphological
Fourier transform
Gabor filters
Optimized FIR filters
Wavelet transform
Multiscale geometric
analysis
Hough transform
Supervised
learning
Unsupervised
learning
Reinforcement
learning
Sec. IV.A Sec. IV.B Sec. IV.C Sec. IV.D
Fig. 3. The overall structure of detection method taxonomy.
0018-9456 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIM.2019.2963555, IEEE
Transactions on Instrumentation and Measurement
>Paper ID: IM-19-22905R Title: Automated Visual Defect Detection for Flat Steel Surface: A Survey<
4
low-quality steel surface than former dynamical thresholding
method. To obtain a better global detection performance, Neogi
et al. [23] proposed a global adaptive percentile thresholding
scheme based on gradient images. It can selectively segment
defective region and effectively preserve the defect edges
regardless of the size of defects. In order to further accomplish
the task of defect detection, it is promising to obtain the optimal
thresholds or design smarter dynamic thresholding mechanism.
2) Clustering
Based on the similarity among image pixels, clustering
method is specialized in mining information implicitly existing
in texture images, then defect detection can be achieved by the
multiple-class defect classification. Real-time and anti-noise
capability are always the basic requirements of industrial defect
detection, Bulnes et al. [24] detected the defects occurring
periodically by clustering the characteristics (i.e., position, type)
of each defect. This method can timely detect periodical defects
even in noisy environment. However, some interferences like
stochastic industrial liquids increase the detection difficulty.
Zhao et al. [25] proposed a two-level labeling technique to
solve the above problem based on superpixels. The pixels are
clustered into superpixels and then superpixels are clustered
into subregions, the superpixel boundaries are updated
iteratively until pixels with similar visual senses are clustered
into one superpixel, subregions after many rounds of growth
will converge towards defects. This method achieved an
average correct detection rate of 91% when applying on cold
strips. Further, Wang et al. [26] proposed an entity sparsity
pursuit (ESP) method to detect surface defects. Defect image
can be segmented into several superpixels to realize entity
sparsity pursuit of defects, while defects do not satisfy the
sparsity assumption in pixel level. The ESP method is
insensitive to noise and computationally efficient. For the
nature of clustering, it is more suitable for defect classification
than defect detection.
3) Edge-based
The purpose of edge detection is to identify points with
obvious brightness changes in digital images. Researchers
often use local image differentiation technique to obtain edge
detection operator, the commonly used edge detection
templates for flat steel surface are Kirsch, Sobel and Canny
operator. It is investigated that Sobel is specialized in weighing
the influence of pixel position to reduce the ambiguity of edge,
but it is sensitive to uneven illumination on flat steel surface,
which easily leads to false edge detection. In order to avoid the
false detection, Borselli et al. [27] modified Sobel operator by
applying thresholding to convert the grayscale image to binary
matrix. Further, Shi et al. in [28] developed eight directional
templates to obtain more comprehensive edge information than
the original Sobel operator which only has horizontal and
vertical directions. Fig. 4 illustrates the technique details of
these two Sobel operators including template topology, detect
performance, etc. The easily trigged false edge detection was
well suppressed by the eight-directional Sobel operator. With
270˚
1
2
1
0 0 0
-1 -2 -1
2
1
0
1 0 -1
0 -1 -2
0
-1
-2
1 0 -1
2 1 0
1
0
-1
2 0 -2
1 0 -1
-2
-1
0
-1 0 1
0 1 2
1
2
1
0 0 0
-1 -2 -1
-1
0
1
-2
0 2
-1 0 1
0
1
2
-1 0 1
-2 -1 0
Y
X
135˚
180˚
45˚
225˚
315˚
1
2
1
0 0 0
-1 -2 -1
-1
0
1
-2
0 2
-1 0 1
Gx
Traditional Sobel
Optimized Sobel
Raw image
Traditional Sobel Optimized Sobel
Scaling factor
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Edge detection ratio
1
0.05
1.1
1.3 1.4
1.5
1.6
1.7
1.8
1.9 2
1.2
Optimized Sobel
Traditional Sobel
Gy
Fig. 4. Comparison of the traditional and the optimized Sobel operator.
the weighted factor and multiple templates, Kirsch is more
noise-robust for tiny defect detection among flat steel images
especially suffered with uneven illumination. While the eight
directional templates bring large computation amounts to
Kirsch. Bo et al. [29] simplified the original Kirsch operator by
choosing some partial templates on the premise of little
influence on edge extraction. Compared with the first-order
Kirsch and Sobel operator, Canny possesses better
signal-to-noise ratio and detection accuracy due to its
second-order feature. However, it suffers with low adaptive
ability and sometimes is easy to blur the noise-free region.
Hence, it is not a wise choice to directly apply existing edge
detection operator on steel surface defect inspection until
appropriate algorithm is imported to enhance its edge detail
retaining ability. Furthermore, many edge detection operators
have not been used to detect surface defects of flat steel, such as
Prewitt, Laplacian and Log. Specifically, Prewitt has been used
for object enhancement and extraction. Laplacian sharpening
template and Log operators have been reported performing well
in determining edge position. So, it is highly recommended to
explore other edge detection operators on the task of steel
surface inspection in the near future.
4) Fractal Dimension
Fractal Dimension (FD) has the desirable self-similarity
which means the overall information can be expressed by
partial features. It is reported that statistical gray value of defect
images practically possesses some features of FD, especially in
self-similarity. Zhiznyakov et al. [30] employed fractal features
of digital images to detect defects of flat steel surfaces by
characterizing the internal distribution of self-similarity and the
image segments with the highest similarity. The experimental
results are basically consistent with inspected data from a
non-destructive testing inspector. Similarly, multifractal
dimension is utilized by Yazdchi et.al [31] to detach and
specify the defective region for five typical defects of steel
surfaces. It should be pointed out that the application of FD has
some limitations because it is only suitable for self-similar
defect image detection.
0018-9456 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TIM.2019.2963555, IEEE
Transactions on Instrumentation and Measurement
>Paper ID: IM-19-22905R Title: Automated Visual Defect Detection for Flat Steel Surface: A Survey<
5
TABLE I
LIST OF SOME OF TYPICAL STATISTICAL METHODS OF DEFECT DETECTION
Ref.
Year
Methods
Applications
Objects
Difficulties
Image source
Performance
[19]
2008
Fixed threshold
Hot-rolled strip
Multi-type defects
Uneven
illumination
Raw images
TPR = 0.929
FPR, FNR, T = NA
[20]
2008
Double threshold and Hough
transform
Steel sheet
Multi-type defects
Complex texture
characteristics
Database
(PRI)
TPR = 0.868
FPR, FNR, T = NA
[22]
2014
Histogram thresholding
Hot-rolled steel
Multi-type defects
Uneven
illumination
Raw images
All not given
[23]
2017
Global Adaptive Percentile
Thresholding
Hot-rolled steel
Blister defect
water-deposit
Size uncertainty
Database
(PRI)
TPR = 0.942
FPR = 0.026
FNR = 0.155
T: = NA
[63]
2017
Double threshold
Steel plates
Multi-type defects
Surface noise
and uneven defect
positions
Raw images
TPR = 0.88-1.00
FPR = 0.00-0.15
FNR, T = NA
[64]
2017
Double threshold
Steel slab
Pinhole
Small size and
pseudo-noise
interference
Raw images
TPR = 0.962
FPR = 0.0131
FNR, T = NA
[24]
2012
Clustering Algorithm
Hot-rolled strip
Periodically defects
Complex texture
characteristics
Raw images
F-Measure: 0.86
TPR, FPR, FNR = NA
T = NA
[25]
2016
Two-level labeling
technique
Cold-rolled strip
Cracks
scratches
Pseudo-noise
interference
Raw images
TPR = 0.91
FPR, FNR, T = NA
[26]
2019
LBP-spired and superpixel
segmentation
Hot-rolled plates
Multi-type defects
Pseudo-noise
interference
Database
(PUB)
FPR = 0.088,
FNR = 0.266,
MAE = 0.143
TPR, T = NA
[27]
2010
Sobel method
Flat steel
Inclusions
rolled in defect
Complex texture
characteristics
Database
(PRI)
TPR = 0.87
FPR, FNR, T = NA
[31]
2009
Multifractal Dimension and
Temporal Fourier
transformation
Cold-rolled mill
Multi-type defects
Defects with
irregular shapes
Database
(PRI)
TPR = 0.979
FPR, FNR, T = NA
[39]
2017
HOG and GLCM
Steel surface
Distributed defects:
scale
Complex texture
characteristics
Raw images
TPR = 0.909
T: 19.79 ms per image
FPR, FNR = NA
[41]
2015
SLBP and GLCM
Strip steel
Multi-type defects
Random noise and
uneven illumination
Database
(PRI)
TPR = 0.916±0.02
T: 7.45 ms per image
FPR, FNR = NA
[2]
2013
Adjacent evaluation
completed local binary
patterns (AECLBPs)
Hot-rolled strip
Multi-type defects
The variations of the
intra-class changes,
the illumination and
grayscale changes
Database
(PUB)
TPR = 0.989±0.37
FPR, FNR, T = NA
[50]
2011
Gabor and morphological
Steel slab
Pinhole
Small size and
uneven illumination
Raw images
TPR = 0.871
FPR = 0.038
FNR, T = NA
[52]
2016
Genetic algorithm and
mathematical morphology
Strip steel
Multi-type defects
Non-uniform
illumination
Database
(PRI)
T: 7.48 ms per image
TPR, FPR, FNR = NA
Notes:
Image source.
PUB: Public, PRI: private
Performance criteria.
TPR: True positive rate, FPR: False positive rate, FNR: False negative rate,
MAE: Mean absolute error, T: Detection time
5) Gray-Level Statistic
Using thresholding methods for defect detection directly
may be ineffective in low contrast images, so it is necessary to
analyze the distribution of image gray level before threshold
operation. Yang et al. [32] utilized the features (i.e., mean value
and distribution of pixels) from steel surface background to
separate bright and dark defect objects simultaneously. Further,
to be insensitive to noise, Choi et al. [33] first estimated the
distribution of background by a spectral-based approach and
then locally refined the defective regions to obtain the
probabilistic estimation. This method is superior to the previous
defect detection methods and gives robust results even in noisy
environment. However, the above methods for surface defect
detection are limited by application scenarios due to the
diversity of surface defects. Ma et al. [34] proposed a
neighborhood gray-level difference method using the
multidirectional gray-level fluctuation which combined the
advantages of global and local characteristics. The proposed
algorithm not only enhances the generalization also improve
the accuracy of surface defects inspection.
6) Co-occurrence Matrix
Gray level co-occurrence matrix (GLCM) is a common mean
to describe texture by studying the spatial correlation of gray
level. In 1973, Haralick et al. [35] first presented GLCM, the
