Journal of the
OPTICAL
Of
SOCIETY
AMERICA
VOLUME 61, NUMBER 1 JANUARY 1971
Lightness and Retinex Theory
EDWIN H. LAND* AND JOHN J. MCCANN
Polaroid Corporation, Cambridge, Massachusetts 02139
(Received 8 September 1970)
Sensations of color show a strong correlation with reflectance, even though the amount of visible light
reaching the eye depends on the product of reflectance and illumination. The visual system must achieve this
remarkable result by a scheme that does not measure flux. Such a scheme is described as the basis of retinex
theory. This theory assumes that there are three independent cone systems, each starting with a set of re-
ceptors peaking, respectively, in the long-, middle-, and short-wavelength regions of the visible spectrum.
Each system forms a separate image of the world in terms of lightness that shows a strong correlation with
reflectance within its particular band of wavelengths. These images are not mixed, but rather are com-
pared to generate color sensations. The problem then becomes how the lightness of areas in these separate
images can be independen t of flux. This article describes the mathematics of a lightness scheme that generates
lightness numbers, the biologic correlate of reflectance, independent of the flux from objects
INDEX HEADINGS: Vision; Color.
Most of us assume that, subject to a variety of com-
pensatory factors, we see in terms of the amount of the
light coming from objects to our eye; we think that in
a particular scene there is more light coming from
white objects than from black objects; we think there
is more long-wave light (so-called red light) coming
from red objects than from blue objects.
Yet, when we measure the amounts of light in the
world around us, or when we create artificial worlds in
the laboratory, we find that there is no predictable
relationship between flux at various wavelengths and
the color sensations associated with objects. Accord-
ingly, we believe that the eye must have evolved a
system which, though using light as the communication
medium with the world, has become as nearly inde-
pendent of energy as is biophysically possible. In short,
color sensations must be dependent on some as yet
undefined characteristic of the field of view, a char-
acteristic that can be communicated to us by the light
with which we see, even though the amount and com-
position of the light are everywhere variable and un-
predictable; the eye must have evolved around such a
permanent characteristic of the field of view. This
paper describes our search for that characteristic.
A major visual phenomenon is that objects with low
reflectance look dark, objects with high reflectance
look light, objects with reflectance higher in the long-
wave portion of the spectrum than in the short-wave
look reddish, objects with reflectance higher in the
short-wave portion than in the long-wave look bluish,
and so on. It is with reflectance that sensation of color
is strongly correlated when we view the world around
us.'-
8
Yet ascertaining reflectance in any of the familiar
ways requires an operational step which the eye cannot
take. For example, the eye cannot insert a comparison
standard next to the object which it is regarding.
Furthermore, what reaches the eye from each point
is clearly the product of the reflectance and the illumi-
nation. The illumination from the sun is modulated by
clouds, atmosphere, water, mountains, trees, houses,
etc. As every photographer knows, the sun and sky
produce every conceivable combination of sunlight and
skylight. Even less uniform illumination is provided by
artificial light because the distance from the light
source drastically affects the illumination falling on
any point.
We are then left with the circular logical problem
that, because the light coming to our eye is the product
of the reflectance and illuminance, our eye could not
determine reflectance unless the illuminance is uniform
and the eye could not determine illuminance unless
the reflectance is uniform. In general, across the field
i
Copyright ©D 1971 by the Optical Society of America.
E.
H.
LAND
AND
J.
J.
McCANN
FILTER
TRANSMISSION
BANDS
Lu
0
z
Ca
H-
Z
450
550
650
WAVELENGTH
IN
MILLIMICRONS
FIG.
1.
Spectral
transmittances
of bandpass
interference
filters.
of
view,
neither
reflectance
nor
illuminance
is
known;
and
neither
is
uniform.
INDEPENDENCE
OF
COLOR
SENSATION
FROM
FLUX-WAVELENGTH
DISTRIBUTIONS
To
demonstrate
the
extent
to
which
color
sensation
is
independent
of
flux-wavelength
distribution,
we
will
describe
a
simple
quantitative
laboratory
experiment.
An
extended
array
of
rectangular,
colored
papers
is
arranged
to
look
like
the
paintings
of
the
artist
Mondrian.,"'
0
To
reduce
the
role
of
specular
reflectance,
the
papers
are
not
only
matte,
but
are
also
selected
to
have
a
minimum
reflectance
as
high
as
or
higher
than
10%
for
any
part
of
the
visual
spectrum.
The
Mondrian-
like
pattern
is
illuminated
by
three
illuminating
pro-
jectors
with
sharp-cut
bandpass
filters,"
one
passing
long
waves,
one
middle-length
waves,
and
one
short
waves
(Fig.
1).
The
flux
from
each
of
the
three
pro-
jectors
is
changed
by
a
separate
variable
transformer.
The
filters
are
selected
on
two
bases:
first,
to
minimize
the
diversity
of
color
sensations
from
the
array
of
colored
papers
when
only
one
projector
is
turned
on;
second,
while
satisfying
the
first
condition,
to
transmit
as
wide
a
band
of
wavelengths,
and
as
much
light
as
possible.
With
all
three
illuminating
projectors
turned
on,
the
variable
transformers
are
set
so
that
the
whole
array
of
variegated
papers
is
deeply
colored,
and
so
that,
at
the
same
time,
the
whites
are
good
whites.
This
is
not
a
critical
setting.
Then,
using
one
projector
at
a
time,
and
hence
only
one
waveband
at
a time,
we
measure
with
a telescopic
photometer
the
luminance
at
the
eye
from
any
particular
area,
say
a
white
rectangle.
Thus,
we
obtain
from
a
white
rectangle
three
numbers
that
are
proportional
to
the
three
luminances
at
the
particular
location
of
the
eye.
(The
subsequent
pro-
cedures
constitute
a
null
experiment.
The
radiance-vs-
luminance
function,
the
particular
units
of
measure,
the
wavelength
sensitivity,
and
the
linearity
of
the
meter
are
not
significant
in
the
experiment.)
In
one
example,
the
readings
from
a
white
area
were
6
long-wave
units,
35
middle-wave
units,
and
60
short-
wave
units.
We
turned
the
photometer
to
another
area,
such
as
a
dark
brown.
We
then
separately
adjusted
the
transformers
to
settings
such
that
the
three
luminances
at
the
eye
were
6,
35,
60.
Thus
the
luminances
from
the
new
area
were
identical
to
the
three
luminances
previously
reaching
the
eye
from
the
first
rectangle.
The
color
sensation
from
the
new
area
remained
es-
sentially
unchanged
(dark
brown)
despite
the
fact
that
the
wavelength-luminance
composition
for
that
area
had
changed
from
whatever
it
might
have
been
to
6,
35,
60.
We
then
pointed
the
photometer
towards
a
series
of
different
areas:
bright
yellow,
blue,
gray,
lime
green,
and
red.
The
illumination
of
each
area
was
readjusted,
in
turn,
so
that
the
three
luminances
coming
to
the
observer's
eye
were
6,
35,
60.
After
each
of
the
new
illuminations
was
adjusted
so
that
the
photometer
read
6,
35,
60
for
the
long,
middle,
and
short
wavelengths,
each
area
appeared
essentially
un-
changed.
Thus,
the
observers
reported
that
the
color
sensations
from
the
series
were
yellow,
blue,
gray,
green,
and
red.
When
the
variable
transformers
were
changed
in
this
way
to
produce
the
standard
set
of
three
luminances
for
any
square,
then
all
the
other
areas
nevertheless
continued
to
generate
their
original
color
sensation
(although
in
a
few
areas
there
were
some
slight
changes).
Dramatically,
the
retention
of
the
color
sensations
was
related
to
the
reflectances
of
the
papers-not
to
the
product
of
reflectance
times
illumination,
although
this
product
appears
to
be
the
only
information
reaching
the
eye.
Therefore,
the
color
sensations
in
the
display
have
a
completely
arbitrary
relation
to
the
composition
of
light
in
terms
of
wavelength
and
luminance
of
any
one
point.
The
luminance-vs-wavelength
distribution
of
each
object
in
the
world
around
us
cannot
tell
us
whether
an
object
is
white,
gray,
or
black;
the
ratio
of
fluxes
at
various
wavelengths
cannot
determine
whether
a
point
on
an
object
is
reddish,
greenish,
bluish,
or
grayish.
The
mystery
then
is
how
we
can
all
agree
with
such
precision
about
blacks,
whites,
grays,
reds,
greens,
browns,
yellows,
when
there
is
no
obvious
physical
quantity
with
which
to
describe
how
we
know
at
all
the
color
of
the
objects
we
are
seeing.
It
might
occur
to
the
reader
that
such
a
large
change
of
relative
luminance,
a
change
such
as
we
produce
by
altering
the
output
of
the
long-wave
projector
relative
to
other
projectors,
is
countered
by
a
compensatory
adaptation
in
the
eye.
If,
in
the
previous
experiments,
changes
of
adaptation
compensated
for
the
changes
of
flux
coming
to
the
eye,
then
deliberately
causing
changes
of
adaptation
should
have
a
significant
effect
on
the
color
appearance
of
objects.
To
produce
an
extremely
large
difference
between
the
state
of
adaptation
to
long
waves
and
the
state
of
adaptation
to
middle
and
short
waves,
we
asked
observers
to
wear
deep-red,
dark-adaptation
goggles,
described
by
Hecht,"
for
2
h,
Vol.
61
LIGHTNESS
AND
RETINEX
THEORY
to
allow maximum
regeneration
of middle-
and short-
wave
visual
pigments.
In
order
to insure
an ample
domain
for
adaptation,
the
level of
illumination
of
the
display
was
maintained
at
a sufficiently
high
level.
(The
white
areas had luminances
for
the middle-
and short-
wave bands
between
100 and
1000 times
higher than
the
threshold
for cone
response
after 30
min in
the
dark.)
When
the
observers
removed
the goggles,
they
reported
at the
first
instant,
as well
as
later,
that
the
colors of
the paper
squares
in the
Mondrian
were
es-
sentially
unchanged.
The
experiment
was
repeated
with
the deep-red,
dark-adaptation
filter
over
only one
eye.
At
the end
of the adaptation
period, the
observers,
using
the
binocular-comparison
technique,
reported
slight
shifts
of the
color
sensations
but
none
so large
as to
change
the
color
names.
(Indeed,
in
our
theory
a
change
of photochemical
adaptation
is unimportant,
for the
same
reasons
that
a
change
of the
flux
of one
of
the illuminants
is
unimportant.
Similarly,
reasonable
variations
of
the native
concentrations
of
visual
pig-
ments are
not important
from time to time,
or from
individual
to individual.)
In another
set of adaptation-related
experiments
the
6,
35, 60
Mondrian
experiments
were repeated
with
the observers
seeing
the
Mondrian
for less
than 1/100
s.
The experimental
procedure
was exactly
the
same as
in
the first
6, 35,
60
Mondrian
experiments,
with
the
exception
that
the
observers
looked
at the
Mondrian
through
a photographic
shutter.
The projectors
were
set
so that
the
long-,
middle-,
and
short-wave
lumi-
nances from
a white area
were 6, 35, 60
and the ob-
servers
reported
that
the
area appeared
white.
The
projectors
were then
set so
that other
areas
had lumi-
nances
of 6, 35, 60
and the observers
reported
that
these areas,
as before,
produced sensations
of brown,
yellow,
blue, gray,
green, and
red.
These experiments
are significant
because
they show
that the
visual system
uses
a processing
mechanism
that
is not merely
independent, but
instantaneously
independent
of the
wavelength-luminance
composition
of
the light coming
to the
eye. These
mechanisms
are
not controlled
by processes
that are
time dependent,
such as
the changes
of
the visual
pigments
that
are due
to
differences of
duration or intensities
of adapting
illumination.
If a particular
rectangle is
moved to various
positions
in the
Mondrian,
where
it is
surrounded
by
new sets
of colored
rectangles, the
color sensation
does
not
change
significantly.
The color sensation
depends
only
on the
long-, middle-,
and
short-wave
reflectances
of
the rectangle
and not
on the properties
of the neighbor-
ing rectangles.
This independence
of
the neighboring
rectangles
holds for
all flux settings
of the
illuminating
projectors.
Because
all these
experiments,
which show that
any
given
wavelength-luminance
combination,
within
limits
as
wide as the reflectance
variations
of these
papers,
3
H
<C1
z
D
Zc
cc
H
0
2
D
-J
l00
80
60,
40
2
BOUNDARY
BETWEEN
40%
& 80% PAPERS
I
DISTANCE
ACROSS
DISPLAY
FIG. 2. Luminance
vs position
for two-squares-
and-a-happening
experiment.
can produce
any color sensation,
and because
of many
other kinds
of laboratory
experiments,'
3
-
17
we came to
the conclusion
that
a color sensation
involves the inter-
action
of at least three
(or four) retinal-cortical
systems.
Each
retinal system
starts with a
set of receptors
peak-
ing, respectively,
within the
long-, middle-,
or short-
wave portion
of the
visible spectrum.
Each system
forms a
separate image
of the world;
the images are
not mixed
but are
compared. Each
system must
dis-
cover
independently,
in spite of
the variation and
un-
knowability of
the illumination,
the reflectances
for
the band of wavelengths
to which
that system
responds.
We invented
a name,
retinex, for each
of these
systems.
A retinex employs
as much
of the structure
and
function of the
retina and cortex
as is necessary
for
producing
an image
in terms of a
correlate of reflectance
for a band of wavelengths,
an
image as nearly
indepen-
dent
of flux as is
biologically possible.
It is convenient
to refer
to the differences
in this
image as
steps of lightness,'
8
-
20
the whites
being called
light, and the
blacks being called
dark. Unfortunately,
as Evans
2
'
points out, dark
is also used to
describe the
quite different
family
of experiences
associated with
change
of illumination.
Nevertheless,
following
him,
we shall
call the steps
in the scale from
black to white,
steps
of lightness.
In our theory,
it is an image in
terms
of lightness, which
is produced
by each retinex
for the
portion of
the spectrum to
which its pigment
responds.
The color
sensation for
any area is
determined by
the three
lightnesses
that are arrived
at independently
by
the three retinexes.
Because
the lightnesses
of an
area are here
defined as the biologic
correlates
of three
reflectances,
it follows within
this conceptual
frame-
work that
the color sensation
is not dependent
on il-
lumination
or flux,
but on reflectance.
Our original
problem is converted
into a
new one: How
does each
retinex generate
for each
area the appropriate
light-
3
January 1971
E. H. LAND
AND J. J. M~cCANN
FIG. 3. Picture
of two-squares-and-a-happening
experiment.
Place
a pencil over
the boundary
between
the
two gray areas.
ness,
the biologic
correlate
of
reflectance
that is
inde-
pendent of
illuminance?
The scheme
that we are about to
describe for answer-
ing this question
is one of
a number of
approaches
that
we
have been
investigating.
All these
schemes
are
designed
to solve
the same
problem,
namely:
For one
retinex,
given
the flux
from each
point
in an
entire
scene,
and assuming
that
nothing
is known
about
the
pattern
of illumination
and nothing
is known
about
the
reflectances, how
can the biologic
system
generate a
set
of values that
we experience
as lightness?
The
particular
scheme
we will describe
is the first
that we
have found to satisfy our
criteria.
EDGES
The experiment
that we call two squares and a
happening
provides
striking
evidence
of edges
as the
source
of lightness
information.
A piece
of paper
that
reflects
80%
of the light
that falls on
it is placed
to the
right
of a piece
of paper that
reflects
46% of
the light.
A fluorescent tube
is mounted in front
and to the left
of
the papers.
The tube
is carefully
positioned
so
that
twice as much light
falls on the center of
the 40% paper
as
falls on
the center
of the
80% paper.
The
light,
being
a line source,
produces
an
approximately
linear
gradient across
the papers,
and the
reflected luminances
at
each corresponding
point
of the two
papers
are
equal.
The graph
of
luminance
vs position
on the
display
is
shown in
Fig. 2. The
40%
paper on
the left
looks
darker than
the 86% paper
on the
right. Figure
3
is a photograph
of the experiment.
What
increases
our interest
is that
when
a long
narrow object, a happening,
obstructs the boundary
between
the left and
right areas,
the two
areas are then
perceived
as having
the
same lightness.
Long narrow
strips
of colored papers in
parallel, or three-dimensional
objects
such as
a pencil
or a piece
of yarn,
make the
two
areas
change
from looking
uniform
and
different
to looking uniform
and indistinguishable; yet, the only
alteration
of the display is the obscuration
of the edge.
We can see this by placing
a pencil on the boundary
between the areas in Fig. 3.
The
experiment
was important
in the
development
of our
ideas of how the visual system
could generate
lightnesses. The
fact that obscuring
the edge informa-
tion could
change the appearance
of these
areas meant
that the
edges are
a very important
source
of informa-
tion. It suggested
that the
change of luminance
at the
junction
between areas
both constituted
an edge
and
also
led to the visual
difference between
the whole
two
areas. The word
edge suggests
a sharp, in-focus
boundary. Experiments, however, show that
the sharp-
ness or focus
of the
boundary is
not at all
critical. For
example,
Fig. 3 can be viewed
through optometric
lenses
to change
the boundaries
from
being sharp
and
in focus
to a variety of fuzzy
out-of-focus
stages. Areas
with
boundaries
quite out
of focus
look essentially
the
same as when
they are
in sharp focus.
What mechanism can we
imagine that would discover
edges and characterize
adjacent areas in a way consis-
tent
with our experiences with the happening, a mecha-
nism that will also
discover the reflectances
in the
Mondrian even
when it is in nonuniform
illumination?
Let
us imagine two light detectors
placed to measure
the luminance from
two different places on
a piece of
paper. If the illumination
is nonuniform, then
the
luminances
at these two positions
will, of course, be
different.
When the two detectors
are placed closer and
closer
together, the luminances
approach the same
value
and the ratio of the outputs
approaches unity.
This will be
true of almost
any two
adjacent points.
However,
if the two detectors
bridge the
boundary
between two areas of
differing reflectance, then the
ratio of the outputs of
these detectors will approach
the ratio
of the reflectances. Thus, the
simple pro-
cedure of taking
the ratio between
two adjacent
points
can both detect an edge and eliminate
the effect of
nonuniform illumination.
Processing the entire image
in terms of the
ratios of luminances at closely adjacent
points
generates dimensionless numbers that are inde-
pendent of illumination. As the distance
between de-
tectors is decreased, each
number approaches a limit
equal
to the ratio of the reflectances, the reflectances
themselves having not yet
been ascertained.
ENTIRE FIELD OF VIEW
Given a procedure for determining the
ratio of re-
flectances between adjacent areas,
the next problem is
to obtain the ratio of
reflectances between any two
widely
separated areas
in an entire
scene. We solve
the problem in the following way: Find
the ratio of
luminances at the edge between
a first and a second
area, and multiply this by the
ratio of luminances at
the edge
between the second and a third area.
This
product of sequential ratios approaches
the ratio of
reflectances between the
first and third areas, regard-
Vol. 61
LIGHTNESS
AND
RETINEX
THEORY
less
of the distribution
of illumination.
Similarly,
we
can obtain the ratio of reflectances of any two areas in
an image,
however remote they are from each other,
by multiplying
the ratios
at all the boundaries
between
the starting area and the remote area. We can also
establish
the ratio of the reflectance
of any area on the
path to the reflectance of the first area by tapping off
the sequential product at that area.
Consider
a Mondrian
similar
to the colored
one in
complexity
and randomness, but consisting of black,
gray, and white papers (see bound transparency, Fig. 4).
That is, in
this Mondrian each piece of paper has the
same reflectance
for all wavelengths. The reflectances
of each area along one
path between the top and
the
bottom are shown in Fig. 5. If we apply the sequential-
multiplication
technique to these reflectances, we can
determine the ratio of the top reflectance to the bottom
reflectance, as shown in Fig. 5. Note that the number
we get by sequential multiplication, 75/12, equals the
number we would get if the bottom area were contigu-
ous
to the top area and
we took the ratio of
their
luminances. We are now coming close to the solution
of the problem that we
defined at the beginning of our
discussion. How can the eye
ascertain the reflectance
of an area without, in effect, placing a comparison
standard next to the
area? The sequential product can
be used as
a substitute for the placement of two areas
adjacent
to each other, thus defining a photometric
operation conceptually
feasible
for the eye.
We placed a fluorescent
tube to illuminate the
Mondrian from below so that more light fell on the
bottom of the display
than on the top. We adjusted
the position of the light so that exactly the same
luminance was coming to the eye from a high-reflectance
area at the top of the display and a low-reflectance area
near the bottom. If the luminance determined the
lightness of an area, the low-reflectance area and the
high-reflectance
area should look essentially alike; in
fact, they do not. Although
the luminances of the two
areas are equal, the high-reflectance area at the top
looks dramatically lighter than the low-reflectance area
at the bottom (see areas indicated by arrows in bound
transparency, Fig. 6).22 Clearly, the visual processes
that determine the lightness of an area are not governed
by the luminance of that area.
Figure 7 shows the luminances along a path from the
top of the Mondrian to the bottom. Note that the
luminance at the center of the top area is the same as
the luminance at the center of the bottom area. Con-
sidering the top area alone, note that the luminance
(in arbitrary units) increases from 118 at its center to
140 at its lower edge. The ratio between the bottom
edge of the first area and the top edge of the second
area is 140 to 80. The luminance of the second area
increases from 80 to 115 from upper edge to lower edge.
The ratio of the second area to the third is 115 to 150.
As we continue down the path, we obtain the ratios.
shown at the bottom of Fig. 7. The product of all the
75 X 43 X
53 X 20 X 58 =
75 = 6.25
43 53 20 58 12 12 1
FIG. 5. Reflectance along one path between the top and bottom
of a black-and-white Mondrian. The numbers at the bottom
indicate the ratios of reflectances at adjacent edges along the
path.
ratios along the path from the high-reflectance area at
the
top to the low-reflectance area
at the bottom is
6.25. This number is equal to the ratio of reflectances
of the top and bottom areas. Thus, without determin-
ing the reflectances and without determining the illumi-
nation, however it varies,
we have determined a number
exactly equal to the ratio of reflectances of these two
areas. Yet the two areas have the same luminance as
each other and are remote from each other by the
whole width of the display. Furthermore, this pro-
cedure of sequential multiplication of edge ratios can
generate values equivalent to relative reflectance for
all areas along the path.
CONSISTENCY OF SEQUENTIAL PRODUCTS
ON DIFFERENT PATHS
Let a number
of different paths start from a given
area and wander back and forth over the display, all
to arrive finally at a distant area, which we wish to
evaluate with respect to the starting area. If we com-
pute the sequential products along each of these paths,
we obtain
the ratio of reflectances of the remote area
to the starting area for each path. In this case the
starting and remote areas for all the paths are the
same, therefore the terminal sequential products are
identical.
If, instead of having all the paths start from a
single area, the paths start from different areas, wander
over the display, and all terminate in a single remote
5
January 1971