Recent advances in retinex theory and some implications for cortical computations: color vision and the natural image

TLDR: A series of experiments demonstrated that in images neither the wavelength of the stimulus nor the energy at each wavelength determines the color, and this departure from what the authors expect on the basis of colorimetry is complete.

Abstract: In the Color Vision Symposium at the Academy in April 1958, we showed a series of experiments which demonstrated that "4whereas in color-mixing theory the wavelengths of the stimuli and the energy content at each wavelength are significant in determining the sense of color . . . in images neither the wavelength of the stimulus nor the energy at each wavelength determines the color. This departure from what we expect on the basis of colorimetry is not a small effect, but is complete . .. .' (1, 2). The initial and most engaging experiment comprised taking two black-and-white photographs of the same scene, one through a red filter and one through a green filter, and projecting these two black-and-white pictures in superposition on the screen to yield a single black-and-white panchromatic image of the scene. When a red filter was placed in the path of the light from the projector that contained the picture taken through a red filter, the whole scene became dramatically colored as if in many respects it were a standard full-color photograph. The first paradox was that the radiation coming to the eye of the observer consisted only of various ratios of red light to white light which should have yielded only a variety of pinks. The second paradox was that the overall ratio of light from the one projector to light from the other projector could be changed markedly without changing the color names of the objects in the colored picture: the colors of the individual objects must be determined by the ratio of red light to white light, but a change in the overall ratio of red light to white light did not change the colors. In light of the understanding which we now have, this simple experiment, which was a shock to the intuitive understanding of all of us, turns out to be the most sophisticated experiment we could have undertaken. For the flavor of the many experiments described at the Symposium, I refer you to the two papers (1, 2) at that time. Here, I want to turn to the quantitative procedures which we now use. We prepared a laboratory display which we dubbed a "Mondrian" (although it actually is closer to a van Doesburg), utilizing about 100 colored papers. A paper of a given color would appear many times in different parts of the display, each time having a different size and shape and each time being surrounded by a different set of other colored papers. One reason for the design was to prohibit the superposition of afterimages of areas onto other areas (3), and another reason for the design was to obviate explanations of results in terms of the size or shape or surrounding of any given paper. The Mondrian is illuminated by using three 35-mm slide projectors with no slides in the slide holder. The output of each projector/illuminator is controlled independently. An interference filter passing long waves is placed in the path of one projector, a middle wave filter, in the path of the second, and a short-wave filter, in the path of the third (Fig. 1). One may think of these as relating roughly to the three visual pigments. A telescopic photometer (Spectra Pritchard photometer, model 1980A), placed roughly where the observers will be, receives and measures radiation from about 1/16th of a square inch on each chosen area of the Mondrian when it is pointed at that area. The instrument is calibrated so that at any wavelength it reports directly in watts per steradian per square meter. Let me call your attention to these four papers: yellow, white, green, and blue. The telescope is pointed at a yellow paper. The short-wave and middle-wave illuminators are turned off, and the whole Mondrian is illuminated with the long-wave illuminator. The output of this projector is then changed until the meter reads exactly "one" (0.1 W per Sr2 per m2). The longwave illuminator is turned off and the middle-wave illuminator is turned on. Its output is adjusted until the meter reads one. This ensures that the amount of middle-wave energy now reaching the meter from that small patch is equal to the amount of long-wave energy. Finally, after the middle-wave illuminator is turned off, the short-wave illuminator is turned on, and its output is set so that the meter (which we must remember is reading the radiation to our eyes) reads one. All three illuminators are now turned on. While looking at the Mondrian as a whole, we note that the yellow paper looks yellow. We now turn our attention to the white paper, pointing the telephotometer at it. We go through the same procedure of illuminating with one illuminator at a time and of setting each illuminator so that the light coming this time from the white paper to the meter, and hence to our eyes, measures one for the long wave and one for the middle wave and one for the short wave. Thus, we have arranged to have coming to our eye from the piece of white paper exactly the same flux-the same wavelength composition, the same energy composition-which a moment earlier we had arranged to have coming to our eye from the piece of yellow paper. The somewhat indigestible question is "what color will the piece of paper be which was white in the Mondrian previously?" Keep in mind that the information now coming to our eye from that piece of paper dictates classically that, if one, one, and one coming to our eye gave yellow, then one, one, and one must again be yellow. This conviction dates back to Newton's proposition V (4):