Models for image editing: Display-referred and scene-referred
This article uses the xyY reference color space to explain the similarities and differences between display-referred and scene-referred image editing. Even though the two models serve very different image editing goals, both models work with bounded RGB data. Display-referred RGB data is bounded by Color, which is to say by both Luminance and Chromaticity. Scene-referred RGB data is bounded only by Chromaticity.
Written May 2014. Updated August 2014.
Introduction: Display-referred and scene-referred image editing both use bounded RGB data
Display-referred editing and scene-referred editing are two different models for editing images. Even though the two models serve very different image editing goals, both models work with bounded RGB data. Display-referred RGB data is bounded by Color, which is to say by both Luminance and Chromaticity; scene-referred RGB data is bounded only by Chromaticity.
This article uses the xyY reference color space to explain the similarities and differences between display-referred and scene-referred image editing.
Separating Luminance and chromaticity: XYZ and xyY
RGB ICC profile working color spaces are defined by the locations of their primaries (red, green, and blue profile colorants) in the XYZ reference color space. In the XYZ reference color space, Y carries Luminance information, and X, Y, and Z together carry information about color.
In the present discussion, it's convenient to separate Luminance information from color information, which is what the xyY reference color space allows us to do. xyY is calculated from XYZ by the following equations:
- x = X / (X+Y+Z)
- y = Y / (X+Y+Z)
- Y = Y
In the xyY reference color space:
- xy specifies color, more technically speaking "chromaticity", less technically speaking, whether a color is red, green, blue, etc, and also how saturated the color might be, but carries no information about how bright the color might be:
- Less saturated colors have xy values that are closer to the D50 white point xy values.
- More saturated colors have xy values that are farther from the D50 white point xy values.
- Y, also called "Luminance", specifies how intense (bright) a color might be.
- The higher Y is, the brighter and more intense the color is.
- The Luminance of a color never goes below zero because there is no such thing as a displayed color that's less bright than solid black (R=G=B=0.0). This is a mathematical reflection of the fact that out there in the real world, there's no such thing as "less than zero reflected light" (black holes out in space aren't relevant to the present discussion).
ProPhotoRGB and sRGB in the xyY reference color space
Figure 1 shows the ProPhotoRGB and sRGB color spaces displayed on the same chromaticity diagram:
As already stated, the farther the xy coordinates of an RGB color are from the D50 white xy coordinates, the more saturated the color is. Comparing the sRGB "reddest red" xy coordinates to the ProPhotoRGB "reddest red" xy coordinates, ProPhotoRGB's reddest red is redder, that is, more saturated than sRGB's reddest red. The same is true of ProPhotoRGB's greenest green and bluest blue, compared to sRGB's greenest green and bluest blue.
As an aside, ProPhotoRGB's greenest green and bluest blue are so saturated that they are completely outside the realm of real colors; this was done deliberately so that the ProPhotoRGB color space would be large enough to accomodate all possible print- and film-related colors. As another very important aside, ProPhotoRGB predates today's digital cameras and is not large enough to accomodate all possible colors produced by interpolating camera raw files and applying an appropriate camera input profile.
Comparing the sRGB and ProPhotoRGB working color spaces in terms of their xyY values:
- In terms of Luminance (Y)
- The Luminance of ProPhotoRGB's reddest red (Y=0.29) is slightly higher than the Luminance of sRGB's reddest red (Y=0.22). Speaking in less technical term, ProPhotoRGB reddest red is just a tiny bit brighter than sRGB red. Speaking more technically, ProPhotoRGB reddest red is slightly more intense than sRGB reddest red.
- The Luminance/intensity of ProPhotoRGB's greenest green (Y=0.71) is almost the same as the Luminance of sRGB's greenest green (Y=0.72).
- The Luminance/intensity of ProPhotoRGB's bluest blue (Y=0.00007629) is much less than the Luminance of sRGB's bluest blue (Y=0.06).
- In terms of saturation (xy)
- Because its xy coordinates are farther from the D50 xy coordinates, ProPhotoRGB's reddest red is redder, that is, more saturated, than sRGB's reddest red.
- Because its xy coordinates are farther from the D50 xy coordinates, ProPhotoRGB's greenest green is greener, that is, more saturated, than sRGB's greenest green.
- Because its xy coordinates are farther from the D50 xy coordinates, ProPhotoRGB's bluest blue is bluer, that is, more saturated, than sRGB's bluest blue.
- In terms of boundedness, both the ProPhotoRGB and the sRGB working spaces are bounded:
- sRGB colors are bounded by the triangle of xy coordinates that's created by the sRGB chromaticities.
- ProPhotoRGB colors are bounded by the triangle of xy coordinates that's created by the ProPhotoRGB chromaticities.
Display-referred editing is bounded image editing in a user-chosen RGB working space
RGB values below are given as floating point values. To calculate the corresponding integer values, multiply by 255 or 65535 as appropriate and round to the nearest whole number.
What is display-referred image editing?
The phrase "display referred" comes from the fact that ICC profile color managed image editing revolves around displaying images on devices. The displaying device might be a monitor, or an image printed on paper, or some other display technology. Regardless of the technology, when you display an image on a device, that device has a maximum and minimum brightness. The maximum and minimum brightnesses are the display-referred colors "white" and "black".
Correspondingly, when editing an image in a normal ICC profile color managed workflow using an RGB working space such as sRGB or ProPhotoRGB, "white" means the RGB color (1.0, 1.0, 1.0). This color has the very special significance that there's no such thing as "brighter than white". So in display-referred image editing, all RGB channel values are less than or equal to 1.0 and no color is brighter than "solid white", (1.0, 1.0, 1.0).
Similarly, "black" means the RGB color (0.0, 0.0, 0.0). This color has the very special significance that there's no such thing as "less bright than black". So in display-referred image editing, all RGB channel values are greater than or equal to 0.0 and no color is less bright than "solid black", (0.0, 0.0, 0.0).
When working with 8- and 16-bit integer display-referred RGB data, editing operations are coded to clamp RGB channel data to keep channel values within the display-referred range between 0.0 and 1.0 (multiplied by 255 or 65535 as required to produce integer values). For example, if you add solid white to solid white, you get what you started with, which is solid white: (1.0, 1.0, 1.0) plus (1.0, 1.0, 1.0) = (2.0, 2.0, 2.0), which is immediately clamped to (1.0, 1.0, 1.0). And if you add solid red to solid red, you get solid red: (1.0, 0.0, 0.0) plus (1.0, 0.0, 0.0) = (2.0, 0.0, 0.0), which is immediately clamped to (1.0, 0.0, 0.0).
As will be shown below in terms of the xyY reference color space, the colors that can be represented in display referred image editing are always bounded by the color gamut of the RGB working space in which the values are encoded.
Display-referred RGB image editing and the resulting xyY values
Summarizing color-managed, display-referred RGB image editing
- No editing operation performed on clamped and bounded RGB data can create RGB values that are out of gamut with respect to the image's ICC profile color space. Expressed in terms of RGB values, editing operations that produce RGB channel values that are less than 0.0 or greater than 1.0 are simply clipped to 0.0 or to 1.0 in the affected channel(s). Expressed in terms of the xyY reference color space:
- No editing operation produces a color that's so dark that its Y value is less than 0.0.
- No editing operation produces a color that's so bright that its Y values extend past the "canopy" of Y values defined by calculating the Luminance of every possible combination of bounded RGB values. In other words, display-referred RGB data is bounded by the working space's "Y canopy".
- No editing operation produces a color that's saturated enough to be located past the edges of the xy triangle created by the xy coordinates (chromaticities) of the RGB working space's colorants. In other words, display-referred RGB data is bounded by the working space's "in gamut" xy values.
- In theory, any RGB working space can be used to edit clamped and bounded integer RGB data. However, in practice:
- For 8-bit integer clamped and bounded RGB data, the RGB color space needs to be perceptually uniform and small enough to avoid posterization. The regular sRGB is a good choice. With care, AdobeRGB can be used. Using linear gamma versions of any RGB color space will cause serious posterization in the shadows of an image, which means you can't do radiometrically correct image editing in an 8-bit integer color space.
- For 16-bit integer clamped and bounded RGB data, even the very large gamut linear gamma ProPhotoRGB color space can be used for editing without fear of posterization, which means 16-bit integer clamped and bounded RGB data does allow for radiometrically correct image editing. Except, of course, if you perform an operation that would otherwise send the RGB data outside the bounded gamut of the color space (eg adding solid white to solid white).
Scene-referred editing is also bounded image editing in a user-chosen RGB working space
Why scene referred image editing?
With display-referred RGB data you have roughly two and half stops of head room above middle gray and maybe six and a half useable stops below middle gray, at which point the RGB data is too dark to visually discern the difference between solid black and "just barely gray". So at best you have 9 stops of dynamic range, compared to the 20 or more stops of dynamic range you might find in some (certainly not all!) real world scenes.
The usual solution to the dynamic range limitations of display-referred, hence clamped and bounded, RGB data is to allow the RGB channel values to be however high or low as is needed to encode the scene data. When using the OpenEXR file format's 32-bit floating point file format, as many as 30 stops of dynamic range are available, which is roughly equivalent to allowing an image's RGB values to go from a low of R=G=B=0.001 (just barely discernible from solid black), all the way up to R=G=B=1,000,000.
When working with color-managed scene-referred high dynamic range RGB data, the chromaticities from any RGB working space can be used. But the tone reproduction curve does need to be linear. Manipulating nonlinear RGB data produces radiometrically incorrect results, breaking the scene-referred nature of the data.
Editing scene-referred high dynamic range RGB data of course requires that there isn't any clamping code in the RGB editing operations. This means you can add (1.00, 1.00, 1.00) and (1.00, 1.00, 1.00) and get (2.00, 2.00, 2.00) or twice as bright as the maximum brightness allowed in display-referred editing.
And you can add what in clamped RGB editing would be referred to as solid red to what in clamped RGB editing would be referred to as solid red and get (2.0, 0.0, 0.0), which is twice as bright as the display-referred maximally saturated and bright red (1.0, 0.0, 0.0).
xyY values that result from unclamped editing operations with scene-referred RGB data
Summarizing color managed scene-referred image editing
- The user can choose the RGB color space chromaticities in which to encode and edit the scene-linear RGB data. Posterization because of the size of the RGB working space isn't an issue with high bit depth floating point RGB data.
- Unlike display-referred image editors, scene-referred image editors are not programmed to automatically clamp the resulting RGB channel values to the range between 0.0 and 1.0, as such clamping code would destroy the scene-referred nature of the data. Nonetheless, the resulting RGB channel values always represent colors that are bounded by the range of xy values enclosed in the triangle of xy values formed by the user-chosen RGB working space's chromaticities.
- Scene-referred RGB data can contain, and editing operations can produce, RGB colors with corresponding Y values that greatly exceed the "canopy" of an RGB working space's "maximum channel value=1.0" Y values. For example, Y might be 2.00, 10.00, 1000.00, etc.
- The particular RGB color (1,0, 1.0, 1.0) still has the color "D50", but so do the RGB colors (2.0, 2.0, 2.0), (1000.0, 1000.0, 1000.0), and etc. So there is no such thing as "solid white". Correspondingly, the color (1.0, 1.0, 1.0) has no special significance beyond being on the RGB working color space's gray axis.
- "Solid black" still means "no light" and is still represented by the channel values (0.0, 0.0, 0.0). As an aside, it should be noted that some scene-referred image editors do allow user-initiated editing operations that produce RGB channel values that are less than 0.0 (for example subtract blue from 100% saturated yellow). In such cases it's up to the user to decide how to deal with any negative channel values that s/he might produce. It behooves the user to get herself out of this situation as quickly as possible because many editing operations on such data produce entirely meaningless results.
Conclusion: Display-referred and scene-referred — two models for image editing serving very different editing goals
Summarizing the differences and commonalities between display- and scene-referred image editing:
- For both display- and scene-referred image editing, the minimum RGB channel value is 0.0, corresponding to "zero percent of the light" being emitted or reflected by a surface (hard to acheive in practice, but light traps come close; a light-tight box with no light source inside doesn't count because by definition there's no light inside the box to be reflected off of any surfaces that might be inside the box).
- Display-referred image editing is oriented towards editing images for display on devices. Devices have a maximum brightness determined by the device itself. Correspondingly display-referred image editing has a maximum brightness represented by the color white, or R=G=B=1.0. The dynamic range of real world scene data can easily exceed the roughly 9 stops of useable dynamic range that are available in display-referred image editing. So for scene-referred image editing, in order to accomodate high dynamic range, scene-referred RGB image data, "white" has no special significance and there is no maximum RGB channel value.
- In both display- and scene-referred image editing, the range of meaningful RGB values is bounded by the user-chosen RGB working space and specifically by the triangle of xy values created by drawing lines on the xy plane connecting the user-chosen working space's Reddest Red, Greenest Green, and Bluest Blue chromaticities.
- For display-referred image editing the range of meaningful RGB values is further bounded by the "canopy" of allowable Y values for each of the enclosed xy values, with the maximum Y value being 1.00 for the color "white" where R=G=B=1.0.
- For scene-referred image editing Y isn't bounded by the upper value 1.0, and hence the corresponding R, G, and B channel values are also not bounded by 1.0. "Solid white" becomes a meaningless concept and the color R=G=B=1.0 is merely another point on the neutral axis that extends from 0.0 theoretically to infinity. However, all Y values and the corresponding RGB channel values are always positive or zero, except in the case of user-initiated editing moves such as subtracting red from solid black or saturated blue from saturated yellow. In such user-initiated cases the user should know what s/he is doing and be prepared to deal with the resulting "colors". Adding and subtracting these "colors" is OK, but trying to multiply or divide them by a (non-neutral) color will produce meaningless results.