UC Berkeley
UC Berkeley Previously Published Works
Title
Robust lane detection and tracking in challenging scenarios
Permalink
https://escholarship.org/uc/item/50n0c8cg
Journal
IEEE Transactions on Intelligent Transportation Systems, 9(1)
ISSN
1524-9050
Author
Kim, ZuWhan
Publication Date
2008-03-01
Peer reviewed
eScholarship.org Powered by the California Digital Library
University of California
16 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 9, NO. 1, MARCH 2008
Robust Lane Detection and Tracking
in Challenging Scenarios
ZuWhan Kim, Member, IEEE
Abstract—A lane-detection system is an important component
of many intelligent transportation systems. We present a robust
lane-detection-and-tracking algorithm to deal with challenging
scenarios such as a lane curvature, worn lane markings, lane
changes, and emerging, ending, merging, and splitting lanes. We
first present a comparative study to find a good real-time lane-
marking classifier. Once detection is done, the lane markings are
grouped into lane-boundary hypotheses. We group left and right
lane boundaries separately to effectively handle merging and split-
ting lanes. A fast and robust algorithm, based on random-sample
consensus and particle filtering, is proposed to generate a large
number of hypotheses in real time. The generated hypotheses
are evaluated and grouped based on a probabilistic framework.
The suggested framework effectively combines a likelihood-based
object-recognition algorithm with a Markov-style process (track-
ing) and can also be applied to general-part-based object-tracking
problems. An experimental result on local streets and highways
shows that the suggested algorithm is very reliable.
Index Terms—Collision warning, computer vison, lane detec-
tion, part-based object tracking.
I. INTRODUCTION
D
ETECTING and localizing lanes from a road image is an
important component of many intelligent-transportation-
system applications. There has been active research on lane
detection [1]–[9], and a wide variety of algorithms of various
representations (including fixed-width line pairs, spline rib-
bon, and deformable-template model), detection and tracking
techniques (from Hough transform to probabilistic fitting and
Kalman filtering), and modalities (stereo or monocular) have
been proposed.
Due to a real-time constraint and, then, slow processor speed,
the lane markings have been detected based only on simple
gradient changes, and much of the older work has presented
results on straight roads and/or highways with clear lane mark-
ings or with an absence of obstacles on the road.
Many commercial lane-detection systems are available and
show good performance in many challenging road and illumi-
nation conditions. However, they do not provide lane-curvature
information but just lane positions to deliver robust results.
Although lane positions are sufficient for some applications,
Manuscript received December 27, 2006; revised April 19, 2007, July 14,
2007, and July 31, 2007. The Associate Editor for this paper was U. Nunes.
The author is with California Partners for Advanced Transit and Highways,
University of California, Berkeley, Richmond, CA 94804-4698 USA (e-mail:
zuwhan@berkeley.edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TITS.2007.908582
Fig. 1. False-alarm scenario of a collision-warning system. Without knowing
the lane curvature, the system will generate a false alarm for the postbox.
such as lane-departure warning, there are other applications
which require lane-curvature information.
For example, a collision-warning system can generate false
alarms when the lane curvature is not known. An example sce-
nario is shown in Fig. 1. Without knowing the road curvature,
the system cannot distinguish objects on the sidewalk (e.g., the
postbox) from the objects on the road, and it may generate a
false alarm. As an alternative to a vision-based approach, one
may use a global-positioning system (GPS) with a geographic-
information system (GIS). However, the GPS has a limitation
on the spatial and temporal resolution, and detailed information
is often missing or not updated frequently in GIS. For example,
it is important to detect the road curvature at an off-ramp
because it can generate a false-collision warning, but most
GPS-based systems suffer from even discriminating whether
the vehicle entered an off-ramp or not.
Recent efforts deal with curved roads [5], [7]–[9], and robust
detection results on challenging images, such as distracting
shadows or a leading vehicle, have been reported. Some of them
work in real time, and some do not.
We present a real-time lane-detection-and-tracking system
which is distinguished from the previous ones in the follow-
ing ways.
1) It uses more sophisticated lane-marking-detection algo-
rithm (than gradient- or intensity-bump-based detection)
to deal with challenging situations, such as worn lane
markings and distracting objects/markings, for example,
at an intersection and on a road surface.
2) It detects the left- and right-lane boundaries separately,
whereas most of the previous work uses a fixed-width
lane model. As a result, it can handle challenging sce-
narios such as merging or splitting lanes and on- and off-
ramps effectively.
3) It combines lane detection and tracking into a single
probabilistic framework that can effectively deal with
1524-9050/$25.00 © 2008 IEEE
KIM: ROBUST LANE DETECTION AND TRACKING IN CHALLENGING SCENARIOS 17
Fig. 2. Flow diagram of the algorithm.
Fig. 3. Example image and a rectified image.
lane changes, emerging, ending, merging, or splitting
lanes. Much previous work has focused on lane tracking
and usually uses a time-consuming detection algorithm
to initialize the tracking. We introduce a fast and robust
lane-detection algorithm that can be applied in every
frame in real-time.
Our algorithm follows the “hypothesize and verify” par-
adigm. In the “hypothesize” step, lower level features are
grouped into many higher level feature hypotheses, and they
are filtered in the “verify” step to reduce the complexity of the
higher level grouping. Fig. 2 shows the flow diagram. First, the
image is rectified, assuming that the ground is flat.
1
An example
image and the rectified image are shown in Fig. 3. Possible lane-
marking pixels are detected in the rectified image. Then, the
detected lane-marking pixels are grouped into lane-boundary
hypotheses. A lane-boundary hypothesis is represented by a
constrained cubic-spline curve. A combined approach of a
particle-filtering technique (for tracking) and a RANdom SAm-
ple Consensus (RANSAC) algorithm (for detection) is intro-
duced to robustly find lane-boundary hypotheses in real-time.
Finally, a probabilistic-grouping algorithm is applied to group
lane-boundary hypotheses into left- and right-lane boundaries.
Note that we generate left- and right-lane-boundary hypotheses
separately (unlike much of the previous work which has a lane
model of uniform width) to deal with various scenarios such as
on/off-ramps or an emerging lane.
In Section II, a comparative study of both classification
performance and computation time on various lane-marking-
1
However, a nonflat case is also addressed at a later stage (lane-boundary
grouping).
Fig. 4. Example road images.
classification methods is presented. In Section III, we present
our approach to hypothesize lane boundaries. The probabilistic-
grouping algorithm is proposed in Section IV. Experimental
results are presented in Section V, and we present the summary
and future work in Section VI.
II. L
ANE-MARKING DETECTION
Sample road images are shown in Fig. 4. Many of the pre-
vious algorithms simply look for “horizontal intensity bumps”
to detect lane markings, which shows reasonably good perfor-
mance in many cases, but it cannot distinguish false intensity
bumps caused by leading vehicles and road markings/textures
from weak lane markings. For example, worn yellow markings
often have similar grayscale intensity to the road pixels. In
addition, we sometimes need to deal with a poor image quality,
for example, when we need to postprocess an MPEG data.
To deal with such problems, we apply machine learning. We
applied various classifiers to the lane-marking-detection task
and present a comparative analysis. Since the size of the lane
marking changes dramatically with respect to its distance from
the car, we need to normalize them to apply a standard classifier.
Therefore, we first rectify the original image, as shown in Fig. 3.
When we assume that the ground is flat (for this stage only), we
can apply a plane homography to find an image rectification.
A point (x, y) on the rectified image corresponds to the point
(u, v) in the original image, where
λx
λy
λ
= H
u
v
1
and H is a homography matrix. A homography matrix can
easily be obtained by applying a simple external camera cal-
ibration with four reference points. Details on the plane ho-
mography can be found in many computer-vision textbooks, for
example [10].
When a plane homography is given, image rectification is
done in the following manner: For each pixel (x, y) of the
rectified image, its correspondence (u, v) on the original image
is obtained. Since u and v are not integer numbers in most cases,
the pixel value of the rectified image is calculated by linearly
18 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 9, NO. 1, MARCH 2008
Fig. 5. Example image patches of lane markings and nonmarkings.
interpolating the intensity values of the four neighboring pixels
(by flooring and ceiling the u and v) in the original image.
Once we have a rectified image, a lane-marking classifier
is applied on a small image patch around each and every
pixel. A typical width of the lane marking on the rectified
images are about three pixels. Therefore, raw pixel values of
a9× 3 window were used as inputs (total 27 features for
a grayscale image and 81 RGB values for a color image).
To find a suitable classification algorithm, we tested various
classifiers.
Applying a stereo algorithm [3], [4] can further improve
the lane-marking-detection performance, but we focus on a
monocular image in this paper.
For learning, we have gathered image patches of 421 lane
markings and 11 124 nonmarkings. Fig. 5 shows example
image patches. We observe a variety in colors, textures, and
width. We compared the classification performances and the
computation requirements of various classifiers on the data set.
The following classifiers were considered.
1) Intensity-Bump Detection: Intensity-bump detection is
the most popular method in the lane-detection litera-
ture. It is the simplest and fastest detection method,
and it can also be applied to nonrectified images. We
use an implementation by Ieng et al. [6]. We applied
various values for the gradient threshold (s
0
) to control
the tradeoffs between the detection rates and the false-
alarm rates.
2) Artificial Neural Networks (ANNs): We tested two-
layer neural networks with various numbers of hid-
den nodes. Training an ANN (a back-propagation al-
gorithm was applied in our experiment) requires sig-
nificant computation, but the actual classification time
is relatively small. When there are n features (inputs)
and m hidden nodes, it requires nm multiplications,
nm + m additions, and m sigmoid-function calculations
to classify a hypothesis (n =27or 81 and m =7in our
examples).
3) Naive Bayesian Classifiers (NBC): NBCs show good
classification performances, in spite of their unrealistic
conditional independence assumption. We compare the
discrete and the unimodal Gaussian representations of
the conditional probability. For both representations, the
learning time is linear to the number of the examples
(fastest). A discrete NBC requires very little computa-
tion for classification. The Gaussian representation re-
quires computation of the exponential function n times.
However, we can avoid calling the exponential func-
tion by using a logarithm of the probability instead of
the actual one. In fact, for both representations, it is
Fig. 6. Classification performance of the classifiers.
necessary to use a logarithm to minimize the numeri-
cal errors, particularly when the number of features is
large. For the discrete NBC, we can precalculate the
logarithms of all the probability table entries to save
computation.
4) Support Vector Machine (SVM): During the last
decade, SVMs have rapidly gained popularity. They pro-
vide a good framework for incorporating kernel methods.
We tested the second-order polynomial kernel, which
requires the smallest computation. Learning requires sig-
nificant computation, but it is bounded in polynomial
time. The classification involves a large number of mul-
tiplications: O(mn), where m is the number of support
vectors. The number of support vectors is at least n +1,
and it can be much greater when the data is not clearly
separable (in the transformed feature space) or when a
small tuning parameter is given. For training, we used the
implementation by Collobert et al. [11] (SVMTorch) with
the tuning parameter of 100.
Details on most of the above classifiers can be found in the
machine-learning literature, for example, in [12].
Fig. 6 shows the classification performances of the presented
classifiers. We followed the evaluation scheme presented in
[13]. We repeated stratified fivefold cross-validation ten times
and showed the receiver operating-characteristic (ROC) curves
with the confidence intervals. For all the classifiers, we obtained
the ROC curves by changing only the threshold values (no
relearning with different parameters).
For all the classifiers, we applied various parameters and
chose the best ones. For ANN, we compared the ones with
seven, 10, and 15 hidden nodes, but we present the result of the
one with seven hidden nodes because it is the fastest, whereas
the performances among them are not significantly different.
For the discrete naive Bayesian network, we used seven-level
discretization. The SVM was learned with the tuning parameter
of 100.0.
We observe that all the classifiers show superior performance
than the intensity-bump detector. In fact, intensity-bump de-
tectors introduce too many false alarms, given an acceptable
detection rate. Therefore, applying any of the above classifiers
will deliver much better lane-detection performance. The SVM
KIM: ROBUST LANE DETECTION AND TRACKING IN CHALLENGING SCENARIOS 19
TAB LE I
C
OMPUTATION TIME OF THE CLASSIFIERS
Fig. 7. Classification performance with neural networks when directly using
color pixels (81 features), gray-level pixels with 5 : 4 : 1 weights, and gray-
level pixels with equal weights. Using a gray image with 5 : 4 : 1 weights gives
competitive performance to that of using a color image.
shows far better performance than any other classifiers, and
then, the ANN follows.
We also compared the classification computation for the
classifiers. We have applied the classifiers on images of a
70 × 250 size and summarized the computing time in Table I.
The algorithms ran on an Intel Core 1.83-GHz processor. For
fair comparison, all the classification algorithms were imple-
mented in C++ inline functions and optimized to bring maxi-
mum performance.
Unfortunately, the SVM was not fast enough for real-time
classification, and we chose to use the ANN. To further reduce
the computation time, we applied a cascade classification: First,
a simple gradient detector and an intensity-bump detector with
loose (low) threshold values are successively applied to quickly
filter out nonlane markings, and then, the ANN classifier is
applied to the remaining samples (much smaller in number).
As shown in Table I, it significantly reduces the classifica-
tion time.
We used gray-level lane-marking images for the above
classification result and the computation-time analysis. The
gray-level images were generated by weighted-summing RGB
values (0.5 for red, 0.4 for green, and 0.1 for blue) to
better detect worn yellow lane markings. Applying such
weights outperformed the equal-weight conversion, as shown
in Fig. 7. We have tested various different weight combi-
nations, and the proposed weights showed the best perfor-
mance. One may apply the classifier directly to the color
pixels (total 81 features), but it introduces too much com-
putation in image-rectification classification, whereas it does
not improve the performance significantly, as also shown in
the Fig. 7.
Fig. 8. (a) Detected lane-marking pixels. (b) Smoothed lane-marking score.
(c) Line-segment grouping. (d) Selected hypotheses from the particle-
filtering/RANSAC algorithm.
III. LANE-BOUNDARY-HYPOTHESES GENERATION WITH
PARTICLE FILTERING AND RANSAC
Once possible lane-marking pixels are detected [an example
is shown in Fig. 8(a)], they are grouped into uniform cubic-
spline curves of two to four control points. Splines are smooth
piecewise polynomial functions, and they are widely used in
representing curves. Various spline representations have been
proposed, and we use a cubic spline among them. In a cubic-
spline representation, a point p on a curve between the ith and
(i +1)th control point is represented as
p =(x
i
(t),y
i
(t))
where
x
i
(t)=a
i
+ b
i
t + c
i
t
2
+ d
i
t
3
y
i
(t)=e
i
+ f
i
t + g
i
t
2
+ h
i
t
3
where the parameters a
i
,...,h
i
are uniquely determined by the
control points so that the curve is smooth. (x
i
(0),y
i
(0)) is the
ith control point, (x
i
(1),y
i
(1)) is the (i +1)th control point,
and 0 ≤ t ≤ 1.
A cubic-spline curve enables fast fitting, because the control
points are actually on the curve. We use this property to apply
a RANSAC algorithm [14]. A RANSAC algorithm is a robust
