THIS IS THE AUTHOR VERSION OF ARTICLE PUBLISHED AT IEEE MED’19 CONFERENCE (
c
IEEE) 1
Comparison of Mono Camera-based Static Obstacle Position Estimation
Methods for Automotive Application
Peter Bauer
1
, Antal Hiba
2
, Akos Zarandy
2
Abstract— This paper presents the comparison of four dif-
ferent mono camera-based steady obstacle position and size
estimation algorithms focusing on automatic emergency braking
application. Three methods are well known in the automotive
field, the fourth is the author’s own method successfully
applied in aerospace until now. The first contribution is the
extension of all methods to consider multiple data points and
variable velocity (where possible). The second contribution is an
extensive simulation testing of the methods considering constant
and variable speeds, attitude uncertainties and the braking
characteristic of real vehicles. The methods are evaluated based-
on the worst case hitting speed of the obstacle and the precision
of obstacle side distance and size estimation. The maximum
speeds of applicability are determined for all methods and the
results are commented in detail.
I. INTRODUCTION
In the past decades automotive systems approach higher
and higher levels of autonomy starting with cruise control
(CC) until the first Level 2 autopilot systems (such as Tesla
autopilot). In these systems several tasks require the estima-
tion of distance and speed of the obstacles and vehicles ahead
and current systems have limitations in their operational
range regarding the own vehicle speed. Currently EuroNCAP
tested ten vehicles equipped with driver assistance systems
(see [1]) and experienced that there are several limitations
and user alert and intervention were required several times.
Regarding stopping before a stationary target at 50km/h all
systems performed well and stopped meanwhile at 80km/h
six systems failed out of the ten. This means that there is
a possibility and need for developments even considering
stationary obstacles.
The author targets to compare existing mono
camera-based methods without radar assistance as
this is an extensively researched field in automotive
([2],[3],[4],[5],[6],[7],[8],[9],[10],[11]) and a good
opportunity to evaluate his mono camera-based obstacle
avoidance algorithm ([12],[13]) used in aerospace until
now. The final goal is to select the best algorithm which
can be further evaluated and possibly tested in automotive
application.
This paper was supported by the J
´
anos Bolyai Research Scholarship of
the Hungarian Academy of Sciences. The research presented in this paper
was funded by the Higher Education Institutional Excellence Program.
1
author is with Systems and Control Laboratory, Institute for Computer
Science and Control, Hungarian Academy of Sciences, H-1111 Budapest,
Kende utca 13-17, Hungary bauer.peter@sztaki.mta.hu
2
author is with Computational Optical Sensing and Processing Labora-
tory, Institute for Computer Science and Control, Hungarian Academy of
Sciences, H-1111 Budapest, Kende utca 13-17, Hungary
Part of the literature sources consider fiducial markers with
known size such as license plates [5], [7] and artificial tags
[4]. The problem with small markers is their detectability
from larger distances as [3] points out (above 30m its hard
to detect them) and the possible change of size in different
countries. Other sources assume known lane width or vehicle
size such as [3], [8]. The problem here is that several different
vehicle widths and lane widths [14] can be observed so such
methods can give uncertain results. That’s why known size
based methods are not considered in this study.
Scale change of the obstacle image and system sampling
time are considered in [6] and [11] to estimate time to
collision (TTC). Assuming known constant velocity the
range of the obstacle can be estimated and with known range
the size and side distance can also be estimated as pointed
out in subsection III-A. This is the first method considered
in comparison and denoted by SC (scale-based).
Scale change and traveled distance are considered in [10]
from which the range and again the size and side distance
can be determined. As the traveled distance is measurable
through odometer and GPS this is the second method con-
sidered as SCD (scale and distance-based).
Point of contact of the vehicle with ground and known
camera height are considered in [2], [8] from which the range
can be directly obtained and again the size and side distance.
This is the third tested method as G (ground-based).
The fourth method is the author’s aerospace method [12]
which considers scale change with constant velocity [13]
and estimates TTC and the closest point of approach (CPA)
which is the ratio of side distance and obstacle width. It is
denoted by SAA (originally aircraft sense and avoid).
Its worth to mention that some sources use Kalman
filtering to smooth results ([9], [8]) but such methods will
be targeted in another work after detailed comparison of the
basic methods here.
Besides comparison the contribution of this work is to
present the selected four methods on a common basis and
extend them to the use of N data samples and time-varying
speeds where possible. The further structure of the paper is
as follows: section II introduces the considered simulation
setup, test scenarios and the evaluation method of the test
results. Section III describes the selected four methods and
extends them where possible, section IV evaluates the results
of the test campaign and finally section V concludes the
paper.
THIS IS THE AUTHOR VERSION OF ARTICLE PUBLISHED AT IEEE MED’19 CONFERENCE (
c
IEEE) 2
II. TEST SCENARIOS AND EVALUATION
The simulated test scenario is the approach of a static
obstacle (vehicle) either in the own lane or in the neighboring
one as shown in Fig. 1. So either the center lines of the
vehicles are aligned (center case) or there is one lane width
difference between them (on the right side of the own,
side case). The considered vehicle widths in lane design
are 1.75m for car and 2.55m for truck as shown in [14]
and a typical lane width can be 3m. These parameters are
considered in the simulations. A straight flat road without
slopes and curves is assumed.
A pinhole camera projection model was applied consider-
ing the f
H
= 1373px and f
V
= 1925px focal lengths of the
Bosch MPC2 automotive camera and pixelization errors. The
camera is assumed to be aligned with the center line of the
own vehicle with coordinate axes Z forward and horizontal,
X rightward, Y downward. The considered image parameters
are the x
1
left corner of obstacle, S = x
2
− x
1
size (see Fig.
1) and y
g
ground contact point (see Fig. 2). Side position
estimation targets the position of the obstacle left corner (see
Fig. 1). Camera fps is assumed to be 10 as a realistic value
with onboard systems.
Fig. 1. Tested situations
Fig. 2. Side view with ground contact point
The simulated scenarios include constant speed approach
of the obstacle between 20 and 130km/h, variable speed
approach which means a sinusoidal variation of the lon-
gitudinal speed and addition of a sinusoidal lateral speed
component and ramp up/down (between 20 and 130km/h)
speed approach. Additionally, pitch or yaw angle sinusoidal
disturbances were applied to test the camera angular align-
ment sensitivity of the methods. Vehicle speed and position
were assumed to be perfectly known at every time step in
the implementation of the algorithms.
The added longitudinal velocity disturbance is: ∆V
z
=
1.34 sin
2π
T
+
π
2
[m/s] with T = 3sec period time while
the lateral is: ∆V
x
= 0.4 sin
2π
T
+
π
2
[m/s]. The amplitude
of the former was determined considering the 9-10sec speed
up of an average car from 0 to 100km/h which means
about 2.8m/s
2
acceleration. The amplitude of the latter was
determined to give about 0.4m maximum side motion of the
own vehicle which can be realistic inside the lane.
The pitch angle disturbance is given as ∆θ = 1 sin
2πZ
10
so it is distance dependent as the pitching caused by road
errors and its amplitude is 1
◦
. The yaw disturbance is the
same when applied.
As the main attempt is to evaluate the methods regarding
emergency braking the braking distance of the own vehicle
from different speeds should be considered. A braking model
is set up in [15] giving the jerk as a
a
= −20m/s
3
and the
brake system delay as t
2
= 0.18s. A more detailed braking
model is used in [16] as: S = (t
1
+ t
2
+ 0.5t
3
)V
0
+
V
2
0
2a
x
including t
1
as driver reaction time, t
3
as deceleration
increase time and a
x
as settled deceleration (V
0
is the initial
speed). Considering automatic emergency braking the t
1
driver reaction time can be neglected and only the other
times considered. The braking model in [16] was modified
to have three parts: distance traveled during system delay
S
2
= V
0
t
2
, distance traveled until deceleration builds up
S
3
= V
0
t
3
+ a
a
t
3
3
6
and distance traveled during constant
deceleration S
4
= V
3
t
4
+ a
x
t
2
4
2
. The settled deceleration was
selected to be a
x
= −7.6m/s
2
as the general case in [16]. t
3
is either the time until standstill t
3lim
=
q
2V
0
|a
a
|
or the time
until deceleration settles t
3
=
a
x
a
a
whichever achieved first.
V
3
= V
0
+ a
a
t
2
3
2
is the speed achieved during deceleration
build up. t
4
=
V
3
|a
x
|
is the time until standstill. The overall
braking distance results as S = S
2
+ S
3
+ S
4
+ 1 where the
last term 1m is a targeted safety distance from the obstacle
at standstill. This S distance is calculated between 20 and
130km/h and the method of system evaluation is to check
if the estimated obstacle distance
ˆ
Z is below S. This case
emergency braking should be initiated and the difference
of the real distance Z and the braking one S at this time
gives the final distance between obstacle and own vehicle
∆S = Z − S. ∆S > 0 means successful stop before hitting
the obstacle. Difference of estimated and real obstacle size
and side distance are also calculated and compared at this
time.
The evaluation criteria for the methods was set as below:
• Size or side position estimation is acceptable if the error
is in the ±0.2m range. With this error range assuming
at least 0.6m targeted distance between vehicles in case
of an avoidance maneuver the worst case distance on
the left side will be 0.4m while on the right 0.2m.
• Considering the 50km/h test speed with full width rigid
barrier of Euro NCAP [17] the obstacle hitting speed
in case of non-complete stop can safely be allowed
to be 20 or 30km/h (probability of severe injury very
low). 20km/h hitting speed means 2m remaining braking
distance ∆S ≥ −2, while 30km/h means 4.57m ∆S ≥
−4.57.
• Methods are evaluated based-on the satisfaction of the
20km/h (Limit
20
or Lim
20
) or 30km/h (Limit
30
or
Lim
30
) braking distance limit together with the accept-
able estimation of size and side position. The highest
speed at which all limits are satisfied is chosen as the
limit of applicability of the given method in the given
scenario. Detailed test results are summarized in Section
THIS IS THE AUTHOR VERSION OF ARTICLE PUBLISHED AT IEEE MED’19 CONFERENCE (
c
IEEE) 3
IV.
III. OVERVIEW OF METHODS
Methods referenced in the introduction are collected here
and extended to use N data points to smooth results and
consider time-varying velocity if possible. At first, all meth-
ods are presented for constant velocity then the extensions
are described.
A. Time to collision from scale change (SC)
Probably the simplest method is the one published in [6].
This estimates TTC from the scale change of the obstacle
size in the camera and the sampling time (∆t) (fps of
camera) as follows:
ˆ
T T C
k
= ∆t
S
k−1
S
k
− S
k−1
(1)
From the TTC estimate the absolute distance, side distance
and real size can be estimated with known longitudinal
velocity V
z
:
ˆ
Z
k
= V
z
ˆ
T T C
k
,
ˆ
X
0
=
1
f
H
x
k
ˆ
Z
k
,
ˆ
W =
1
f
H
S
k
ˆ
Z
k
(2)
This TTC estimation method can be easily extended to
consider N data samples as:
ˆ
T T C
k
= (N − 1)∆t
S
k−N +1
S
k
− S
k−N +1
(3)
Considering a longer horizon can suppress high errors
caused by sudden local changes. Unfortunately, the case of
time-varying velocity can not be included into this frame-
work as it does not comply with TTC estimation.
B. Distance from scale change and traveled distance (SCD)
This method is published in [10]. Knowing the image
coordinates of a point on the object at two time steps
x
k−1
, y
k−1
and x
k
, y
k
and the traveled 3D distances during
that time T
x
(k − 1) = X
k
− X
k−1
, T
y
(k − 1) = Y
k
−
Y
k−1
, T
z
(k − 1) = Z
k
− Z
k−1
one can obtain the system of
equations below from the pinhole camera projection model.
Assuming T
y
(k − 1) = 0 (no vertical displacement) the
system can be further simplified.
x
k
y
k
=
"
f
H
(X
k−1
+T
x
(k−1))
Z
k−1
+T
z
(k−1)
f
V
(Y
k−1
+T
y
(k−1))
Z
k−1
+T
z
(k−1)
#
=
Z
k−1
Z
k−1
+ T
z
(k − 1)
"
f
H
X
k−1
Z
k−1
f
V
Y
k−1
Z
k−1
#
| {z }
h
x
k−1
y
k−1
i
T
+
f
H
Z
k−1
+ T
z
(k − 1)
T
x
(k − 1)
0
(4)
Extension of this method to N data points and variable
velocity is straightforward as points further in time can be
considered and the traveled distances can be calculated from
the related positions.
x
k
y
k
=
Z
k−N +1
Z
k−N +1
+ T
z
(k − N + 1)
| {z }
s
1
x
k−N +1
y
k−N +1
+
f
H
Z
k−N +1
+ T
z
(k − N + 1)
| {z }
s
0
T
x
(k − N + 1)
0
T
x
(k − N + 1) = X
k
− X
k−N +1
T
z
(k − N + 1) = Z
k
− Z
k−N +1
(5)
If one can determine s
1
, s
0
than the distance Z
k
=
Z
k−N +1
+ T
z
(k − N + 1) can be determined with them:
ˆ
Z
k
=
s
1
T
z
(k − N + 1)
1 − s
1
+ T
z
(k − N + 1)
ˆ
Z
k
=
f
H
− s
0
T
z
(k − N + 1)
s
0
+ T
z
(k − N + 1)
(6)
s
1
, s
0
can be determined from a system of equations:
x
k
y
k
=
x
k−N +1
T
x
(k − N + 1)
y
k−N +1
0
s
1
s
0
(7)
If T
x
(k − N + 1) is close to zero than the matrix is close
to singular so its better to solve the 2 equations only for s
1
and then average:
ˆs
1
=
1
2
x
k
x
k−N +1
+
y
k
y
k−N +1
(8)
After determining
ˆ
Z
k
ˆ
W and
ˆ
X
0
can be determined
similarly as in (2). A possible problem with this method
is that x can be yaw and y can be pitch sensitive. This will
be examined during the tests.
C. Distance estimation based-on ground contact point (G)
This method is described in [2]. It is based-on the known
height of the camera mounted on the own car H and the
vertical coordinate of the ground contact point of the obstacle
in the camera image y
g
k
(see Fig. 2). The distance can be
directly calculated from them:
ˆ
Z
k
=
f
V
H
y
g
k
(9)
After determining
ˆ
Z
k
ˆ
W and
ˆ
X
0
can be determined
similarly as in (2). This method does not include any assump-
tion about the speed so time-varying speed does not cause
problem. However, it is very sensitive to the pitching motion
of the car (if it is not compensated). Possibly some smoothing
can be applied by considering the traveled distance and
solving a system of equations by simply averaging the left
hand side for
ˆ
Z
k
:
THIS IS THE AUTHOR VERSION OF ARTICLE PUBLISHED AT IEEE MED’19 CONFERENCE (
c
IEEE) 4
f
V
H
y
g
k
f
V
H
y
g
k−1
+ T
z
(k − 1)
.
.
.
f
V
H
y
g
k−N +1
+ T
z
(k − N + 1)
=
1
1
.
.
.
1
ˆ
Z
k
(10)
D. Method based-on TTC and CPA estimation (SAA)
This method was developed by the author for aircraft sense
and avoid application as published in [12] and [13]. The basic
formulas relate forward (Z) and side distances (X) and the
object size (W) to the measurable image parameters (x,S):
1
S
k
=
Z
k
f
H
W
,
x
k
S
k
=
X
k
W
(11)
Considering V
x
= 0 side velocity (X
k
= X
0
= const),
V
z
< 0 forward velocity and the camera projection model in
Fig. 1:
1
S
k
= −
V
z
f
H
W
T T C
k
=
V
z
f
H
W
|{z}
a
t
k
−
V
z
f
H
W
t
C
| {z }
b
x
k
S
k
= CP A
(12)
The first expression gives a possibility to fit line on the
points 1/S
k
, t
k
and so obtain the absolute time of collision
as t
C
= −b/a and
ˆ
T T C
k
= t
C
− t
k
. At least two points are
required to do this but more points will give better results as
the tests will show.
ˆ
CP A =
X
0
W
can be simply calculated as
the average of the x
k
/S
k
ratios.
Extension of this method to time-varying speed is possible
but requires a completely different solution method. With
time-varying speed the distances can be calculated as follows
(Z
0
, X
0
initial distances):
Z = Z
0
+
Z
t
t
0
V
z
dt, X = X
0
+
Z
t
t
0
V
x
dt
(13)
Substituting these into (11) and grouping the known and
unknown terms gives the following system of equations:
1
S
k
=
1
f
H
Z
0
W
+
R
t
k
t
0
V
z
dt
f
H
1
W
x
k
S
k
= 1
X
0
W
|{z}
CP A
+
Z
t
k
t
0
V
x
dt
1
W
(14)
This system of equations includes three unknown constant
parameters Z
0
/W, 1/W, CP A. It is advisable to multiply
the first equation with f
H
to make it better conditioned
and to solve it first for Z
0
/W, 1/W considering multiple
measured points. Then CPA can be determined from the
second equation with known 1/W and averaging if multiple
points are considered. Finally, the side distance can be
calculated as X
0
= CP A · W
IV. TEST RESULTS
First, the applicability of the constant velocity formulas
was tested simulating constant vehicle speed. Beside the
normal undisturbed case (see Fig. 3) cases with pitch (see
Fig. 4) and yaw (see Fig. 5) disturbances are also considered
to estimate sensitivity to small angular errors (1
◦
disturbance
only). The figures show the limit speeds of applicability
of each method as bar plots. Numbers on the horizontal
axis belong to 1 = car/center, 2 = truck/center, 3 =
car/side and 4 = truck/side evaluated scenarios. Four bars
are plotted for every scenario representing respectively SC,
SCD, G and SAA methods. The 5km/h bar value means
that the method is not applicable even for 20km/h vehicle
speed. The overall summary of applicability limits is given
in Table I.
Fig. 3. Constant velocity normal test case
Fig. 4. Constant velocity pitch disturbance test case
Fig. 5. Constant velocity yaw disturbance test case
THIS IS THE AUTHOR VERSION OF ARTICLE PUBLISHED AT IEEE MED’19 CONFERENCE (
c
IEEE) 5
Table I shows that in the normal case the best is the
G method applicable at the 130km/h maximum test speed
also. The second best is the SAA method applicable until
110km/h with the 30km/h hitting speed limit. However,
considering the really small 1
◦
pitch disturbance the G
method becomes inapplicable while the SAA method still
performs well with slightly decreased 100km/h speed limit.
This case the performance of the SC method is also ac-
ceptable. This result is not surprising as SC and SAA
methods do not use the y coordinate which is disturbed by
pitching. Considering the small yaw disturbance the SCD
method becomes inapplicable and the applicability of SC
and G become severely limited while the limits for the
SAA method stay the same as for the pitch case. The G
method is limited only by the side distance estimate the
forward distance and obstacle size estimates are as good
or better than with SAA. The side distance from SAA is
calculated in a different way (X
0
= CP A · W ) so this
can cause the difference. As a summary, it can be stated
that the SAA method outperforms the others if there is a
possibility of angular disturbances which is the usual case
for moving vehicles even with attitude angle compensation
as its precision is rarely below 1
◦
.
TABLE I
FINAL COMPARISON OF CONSTANT VELOCITY TEST RESULTS
Test case Limit SC SCD G SAA
CONST normal Lim
20
80 80 130 90
Lim
30
90 110 130 110
CONST pitch Lim
20
80 20 N/A 90
Lim
30
80 20 N/A 100
CONST yaw Lim
20
30 N/A 30 90
Lim
30
30 N/A 30 100
Second, the applicability of the variable velocity formulas
was tested simulating variable vehicle speed. Beside the
normal undisturbed case (see Fig. 6) cases with pitch (see
Fig. 7), yaw (see Fig. 8) and pitch and yaw disturbances
are also considered to estimate sensitivity to small angular
errors (1
◦
disturbance only). The pitch and yaw plot is very
similar to the pitch one with slightly different numerical
values which are shown in Table II. Here an extra case is
the application of the variable speed formulas with constant
vehicle speed in Fig. 9 as this is a realistic situation with
such system. The concept of figures is the same as for the
constant velocity, but only three bars are plotted for every
scenario representing respectively SCD, G and SAA as the
SC case was not applicable for variable speed. The overall
summary of applicability limits is given in Table II.
TABLE II
FINAL COMPARISON OF VARIABLE VELOCITY TEST RESULTS
Test case Limit SCD G SAA
VAR normal Lim
20
60 130 90
Lim
30
60 130 90
VAR pitch Lim
20
N/A N/A 90
Lim
30
N/A N/A 90
VAR yaw Lim
20
N/A 50 70
Lim
30
N/A 50 100
VAR pitch/yaw Lim
20
N/A N/A 70
Lim
30
N/A N/A 100
VAR CONST Lim
20
80 130 90
Lim
30
80 130 100
Fig. 6. Variable velocity normal test case
Fig. 7. Variable velocity pitch disturbance test case
Fig. 8. Variable velocity yaw disturbance test case
Table II shows that in the normal case the best is the
G method applicable at the 130km/h maximum test speed